LLama.Exceptions.RuntimeError: Failed to load model about llamasharp

[SignalRT]

The models supported are models with GGUF format. Please review the documentation:

It seems that you are trying to load directly the model downloaded from meta repositories https://github.com/facebookresearch/llama that don´t use GGUF format. If this is the case you need to convert previously the model to GGUF format or download a GGUF model (for example in this repository https://huggingface.co/TheBloke you will find a lot of models converted to GGUF).

If the library fails to load a GGUF model, please indicate the link to download the model to be able to verify the issue. You also should have in consideration that you will need a lot of memory to load a 70b parameter model.

[TonyWang666]

I did run it successfully after load from a gguf file. Thanks~

[TonyWang666]

Hi jlsantiago, thank you very much. I plan to use gguf file from https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGUF/blob/main/llama-2-7b-chat.Q5_K_M.gguf. Will update the result later in this thread.

[SignalRT]

@TonyWang666, Did you have time to test some gguf model with LLamaSharp ?

[TonyWang666]

@TonyWang666, Did you have time to test some gguf model with LLamaSharp ?

Yeah, I can do the test today. Sorry for the delay.