Control Behavior of Conversational Agent about haystack

[ss2342]

In this case, I discovered that because I was not using the [INST] prompt [/INST] format for the mistral model, it gave me the strange outputs. Once I added those tags, it fixed the responses!

[anakin87]

The mentioned tutorial uses the free rate-limited Hugging Face Inference API.

Creating the PromptNode with the following command works for me:

prompt_node = PromptNode(
    model_name_or_path="mistralai/Mistral-7B-Instruct-v0.1",
    api_key=model_api_key,
    max_length=256,
    stop_words=["Human"]
)

It correctly stops text generation when encounters the stop words.

---

If you want to load the model locally (you should have sufficient RAM and GPU VRAM),
you can try something like the following (untested):

prompt_node = PromptNode(
    model_name_or_path="mistralai/Mistral-7B-Instruct-v0.1",
    max_length=256,
    stop_words=["Human"],
    model_kwargs={"load_in_4bit":True}  # this line is not strictly needed but can help reducing GPU usage with quantization
)

I am closing this issue now because everything seems working.
Feel free to reopen it if you still encounter problems.