JSONDecodeError("Expecting value", s, err.value) from None about distributed-llama

[DifferentialityDevelopment]

It's busy uploading, you can find it here:
https://huggingface.co/Azamorn/Meta-Llama-3-8B-Instruct-Distributed

[DifferentialityDevelopment]

You need to point it at the original folder of Meta-Llama-3-8B-Instruct
not the one with safetensors but with the pth files, I already converted mine earlier today.
I'll upload it to huggingface for you.

[unclemusclez]

i used download.sh and the key they provide. that was the error i got. I would like to be able to convert for the future reference.
Tried this on 3 different machines.

Thank you for the converted model. I will post my results with 8x RPi 3b

[unclemusclez]

Model name: Meta-Llama-3-8B-Instruct
Target float type: q40
Target file: dllama_meta-llama-3-8b-instruct_q40.bin

I assumed this script would work for all of the LLama3 models, but it does not. Any chance we could get some instruct and or codellama conversion scripts?

[b4rtaz]

Could you try to run convert-llama.py from the directory where you have that script?

distributed-llama/converter % python convert-llama.py <modelPath> <targetFloatType>

Btw: in your logs I see JSONDecodeError. Are you pointing a correct directory?

[unclemusclez]

Yes.

The script DOES NOT work with Instruct Models or CodeLLama Models.

However, It DOES WORK for 8B, non-instruct, non-code models.

[DifferentialityDevelopment]

Could you give more information, what repo, what folder you pointing it at etc?

I should probably say it again, it doesn't work with safetensor files, but with .pth files.
Ideally the script would work with safetensor models as well but not right now.

[unclemusclez]

i am downloading models directly from the meta repository. In order to download these models, you need a key mailed to you. I run the download.sh script as provided. when executing the converter script on models that are designed for Instruct/Code, the script fails.

https://i.imgur.com/XjvNKLl.png

When executing the Tokenizer script for instruct models:

https://i.imgur.com/Q4akLbO.png

[DifferentialityDevelopment]

I downloaded them straight from huggingface
https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct
then I used the files in the original folder in the model repo.

[unclemusclez]

i'll try this maybe i downloaded something wrong