llmware.exceptions.ModelNotFoundException: about llmware

[doberst]

@abhi-0907 - thanks for clarifying - we will dig in over the next couple of days and figure out what is going wrong.

[doberst]

@abhi-0907 - please check out the file posted on Huggingface:
--repository: llmware/bonchon
--file: meditalk_4_Q_M_042124.gguf

Hope that this resolves the issue.... 😄

[doberst]

@abhi-0907 - thanks for sharing this. Your code looks spot-on, and would expect this to work. I have tested it locally, and can recreate the issue, e.g., I get the same error. It looks as if the gguf engine is not loading the model successfully. Just confirming that it is a Llama-7b base model, and was converted/quantized using a current build of llama-cpp? Any insights on the base model and the gguf build environment will definitely us to recreate the environment and figure out what is going wrong.

[abhi-0907]

Yeah, It was Llama2 - 7b basemodel and it was quantized using a current build of llama-cpp on windows.

[doberst]

@abhi-0907 - FYI, I have tried recompiling our llama cpp libs, and am still having trouble getting the Meditalk2 Q4_K_M file to successfully load. Not seeing an issue with other llama GGUF models in testing. In experimenting, I re-quantized your original Pytorch meditalk model in Q4_K_M and it is working well (really nice finetuned output!) on both Mac and Windows CUDA. Perhaps there is some small, but significant, difference in the llama cpp build you used to quantize (?). I have posted the re-quantized version in a private HF repo - I didn't want to put in a public repo unless you said OK ... please confirm that you are OK, and I will post it in llmware/bonchon - or can upload it to you directly if you prefer ...

[abhi-0907]

Thanks for resolving. You can post it on public repo.