Getting json.decoder.JSONDecodeError: about alpaca.py

[muelletm]

Hi!

So first of all, the API is supposed to be used with a special binary that is a version of chat that is stateless and uses a JSON API. The binary can be built inside the cpp directory. If you use the dockerfile everything will be done for you.
Otherwise:

cd cpp
make
cd ..
export ALPACA_CLI_PATH="$PWD/cpp/build/alpaca"

Now ALPACA_CLI_PATH points to the right binary.

The locks are just there to make the client thread-safe so that we don't get into race conditions when we use it from e.g. FastAPI.

[robinnarsinghranabhat]

Thanks It's working now. I thought I could run it using the original binary from alpaca.cpp.

I was curious. When new improvements/optimizations are made to original alpaca.cpp, like faster loading, memory enchancements e.t.c how difficult would it be to keep this repo in track with those ?

I am not a C/C++ person but will love to help out in any way I can.

[rg089]

@muelletm I'm getting the same error even after making the new alpaca file as described within cpp/build. I'm on macOS 13.3 Ventura (M1 chip).

[muelletm]

@rg089, can you share how you run the client and the output you are getting?

[rg089]

@muelletm I first compiled the binary and ran using multiple options. When I run using python as follows, I 'm getting this error: json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

from alpaca import Alpaca, InferenceRequest
import os

alpaca_cli_path = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'cpp/build/alpaca')
model_path = os.path.join(os.path.dirname(os.path.realpath(__file__)), '../alpaca_cpp/ggml-alpaca-7b-q4.bin')
alpaca = Alpaca(alpaca_cli_path, model_path)
try:
    output = alpaca.run_simple(InferenceRequest(input_text="Are alpacas afraid of snakes?"))["output"]
    print(output)
finally:
    alpaca.stop()

When I directly ran the compiled alpaca file inside cpp/build using ./alpaca -m ../../../alpaca_cpp/ggml-alpaca-7b-q4.bin' and gave the input {"input_text": "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\n\nAre alpacas afraid of snakes?### Response:\n\n", "top_k": "40", "top_p": "0.95", "temp": "0.1", "repeat_penalty": "1.3", "repeat_last_n": "64", "n_predict": "128"}, there was a segmentation fault.

Please let me know if you need any further details. I had another query regarding this: If I wanted to continue the conversation with the model, would using alpaca.run_simple in a loop load the model multiple times as the generation params are also being passed in the message, or would it just pass the prompt to the loaded model? Is there a better way to interact with the model in a multi-turn conversation?