model is not None about llama2-webui

[jamartinh]

It is the new gguf format that llama.cpp is designen for. I replaced the code to download the gguf versión and it loads it well, however the output seems to be in hexadecimal which I do not understand why. The other thing is to stick to the old version of llama.cpp , but I think that is not the idea. El jue, 28 sept 2023, 17:35, AIGoldenfinger ***@***.***> escribió:

…

There so many issues in the code... but if this is you main struggle... What worked for me as workaround,, before I stumble to the next issue is: Replace in .env file , BACKEND_TYPE = "llama.cpp" with "gptq" it will download the model and continued to the next error ... Then errors appears to be in: nn_modules\qlinear\qlinear_cuda_old.py line 83, in *init* self.autogptq_cuda = autogptq_cuda_256 NameError: name 'autogptq_cuda_256' is not defined And then I surrender :) — Reply to this email directly, view it on GitHub <#71 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA3NNMYOJJJ3WJJ3E77QWEDX4WKK5ANCNFSM6AAAAAA5AZH254> . You are receiving this because you commented.Message ID: ***@***.***>

[jamartinh]

Same here !

[liltom-eth]

Thanks for the issue. Might be some issue from llama.cpp. Will have a look.

[talarari]

https://stackoverflow.com/questions/76986412/assertionerror-when-using-llama-cpp-python-in-google-colab

need to use llama-cpp-python==0.1.78
the format changed

or better yet, use the gguf models

[HyperUpscale]

There so many issues in the code... but if this is you main struggle...

What worked for me as workaround,, before I stumble to the next issue is:
Replace in .env file , BACKEND_TYPE = "llama.cpp" with "gptq" it will download the model and continued to the next error ...

Then errors appears to be in: nn_modules\qlinear\qlinear_cuda_old.py
line 83, in init self.autogptq_cuda = autogptq_cuda_256
NameError: name 'autogptq_cuda_256' is not defined

And then I surrender :)

[liltom-eth]

@jamartinh @HyperUpscale Thanks for the feedback. I believe the issue is mainly due to the new GGUF format.
A quick solution is try install from requirements like

git clone https://github.com/liltom-eth/llama2-webui.git
cd llama2-webui
pip install -r requirements.txt

to stick the old version of llama-cpp-python== 0.1.77.

[liltom-eth]

@jamartinh @HyperUpscale hi now the new release llama2-wrapper=0.1.13 will lock on llama-cpp-python = "0.1.77" to support old ggml models.
New release will support gguf models.

[liltom-eth]

#73 now supporting gguf models.