Error wrong number of tensors when serving vinai/PhoGPT-4B-Chat with llama.cpp about phogpt

[nviet]

That is because phoGPT uses tensors with bias parameter in addition to weight parameter and llama.cpp currently does not support MPT-trained model with such feature. I created a fork of llama.cpp here some months back to make it able to run phoGPT. Perhaps applying same method will also work with latest version of llama.cpp

[datquocnguyen]

Thanks. I will take a look and respond back soon.

[datquocnguyen]

Thanks @nviet and @xtfocus
Here is the PR I just made for llama.cpp to address this issue, following @nviet's commit with minor modifications to make 'bias' parameters optional (i.e. now it works for both MPT and PhoGPT):
ggerganov/llama.cpp#5638

[datquocnguyen]

PR was merged.
I will close this issue.

[nntadotzip]

@xtfocus I faced the same issue. Then I found @nviet's solution on Huggingface, and it worked!

cd llama-cpp-python
git checkout e9bc4c4baf3f121a178dec215770ccd0ac86c28e
cd vendor
git clone https://github.com/nviet/llama.cpp
cd ..
pip install .