What is a good local model for generating Embeddings? about llamasharp

[martindevans]

Maybe I'm misunderstanding something, but "new york" doesn't seem to be mentioned anywhere in the saved information?

[martindevans]

In that case I'd recommend opening a discussion on the main llama.cpp repo, you'll probably get a better response over there (just because it's a bigger repo, with more people who can answer your question).

[martindevans]

I was thinking a more general discussion asking for embedding model recommendations.

[AshD]

Good catch Martin. I missed one text addition, "my family is from New York"

[AshD]

I tried to use https://huggingface.co/intfloat/e5-mistral-7b-instruct but ran into conversion errors.
ggerganov/llama.cpp#4786

[martindevans]

Did adding that extra text resolve the initial issue?

[AshD]

No. The non instruct Mistral 7B v0.1 gave me slightly better matches (0.31 relevance scores) compared to the Mistral 7B Instruct v0.2 (0.2 relevance scores).

Reading stuff on this issue elsewhere, it looks like the way to go is to use a model specifically designed for embedding but I could not find a GGUF version of an embedding model and when I tried to convert it them (the top ones from the MTEB space), they failed with model architecture not supported.

If anyone has a GGUF version of a model that works well for embedding, would appreciate if you point me to it.

Thanks,
Ash

[AshD]

I opened one yesterday ggerganov/llama.cpp#4786

Hopefully, someone will respond.

[AshD]

Will do

[AshD]

Did some more research and I found that lama.cpp does not support embedding models as of now.

This project does https://github.com/xyzhang626/embeddings.cpp

I will try it out tomorrow.