Tokenizer class LLaMATokenizer does not exist or is not currently imported. about gptq-for-llama

[BEpresent]

For anyone experiencing this problem, it could be related to the entry inside the tokenizer_config.json as they changed this recently: (LlamaTokenizer not LLaMaTokenizer ).

"tokenizer_class": "LlamaTokenizer",

Changing this solved it for me.

[DanielWe2]

I had the same issue. In my case the pip package sentencepiece was missing preventing some imports in the LLaMa modules of transformers.

[clxyder]

For anyone experiencing this problem, it could be related to the entry inside the tokenizer_config.json as they changed this recently: (LlamaTokenizer not LLaMaTokenizer ).

"tokenizer_class": "LlamaTokenizer",

Changing this solved it for me.

I can confirm this is the correct solution to the problem.

huggingface/transformers#22222 (comment)

[qwopqwop200]

It is possible that the checkpoint is out of date.
Try converting again.

[C0rn3j]

Do you mean by running python setup_cuda.py install again?

Because that's what downloaded the checkpoints in the first place

[brandonrobertz]

Torch version is too new. Try 1.X

[C0rn3j]

I am on CUDA 12.1, which is not supported by torch 1.x.

[brandonrobertz]

I had to downgrade my CUDA stack down to 11.7 and use torch 1.13. Best of luck!

[brandonrobertz]

I had the same issue. In my case the pip package sentencepiece was missing preventing some imports in the LLaMa modules of transformers.

Interesting. sentencepiece is a dependency of transformers, so a proper and successful install of transformers should result in sentencepiece being installed as well.

[C0rn3j]

@brandonrobertz That is not correct - https://github.com/pytorch/pytorch/blob/master/requirements.txt

@DanielWe2 Yep, that was it for me too, thanks for finding it out!

[brandonrobertz]

@brandonrobertz That is not correct - https://github.com/pytorch/pytorch/blob/master/requirements.txt

That's the wrong library. It's a dep of transformers, not torch:

https://github.com/huggingface/transformers/blob/main/setup.py#L162

Since you need to use the fork specified in the README (-e git+https://github.com/zphang/transformers@660dd6e2bbc9255aacd0e60084cf15df1b6ae00d#egg=transformers), this is a common failure mode for people.

[Azeirah]

I'm still not sure how to fix this. I'm getting this import exception as well.

(gptq)$ pip list
Package                  Version
------------------------ -----------
accelerate               0.17.1
aiofiles                 23.1.0
aiohttp                  3.8.4
aiosignal                1.3.1
altair                   4.2.2
anyio                    3.6.2
async-timeout            4.0.2
attrs                    22.2.0
bitsandbytes             0.37.1
brotlipy                 0.7.0
certifi                  2022.12.7
cffi                     1.15.1
charset-normalizer       2.0.4
click                    8.1.3
contourpy                1.0.7
cryptography             39.0.1
cycler                   0.11.0
entrypoints              0.4
fastapi                  0.94.1
ffmpy                    0.3.0
filelock                 3.9.0
flexgen                  0.1.7
flit_core                3.6.0
fonttools                4.39.2
frozenlist               1.3.3
fsspec                   2023.3.0
gmpy2                    2.1.2
gradio                   3.18.0
h11                      0.14.0
httpcore                 0.16.3
httpx                    0.23.3
huggingface-hub          0.13.2
idna                     3.4
importlib-metadata       6.0.0
importlib-resources      5.12.0
Jinja2                   3.1.2
jsonschema               4.17.3
kiwisolver               1.4.4
linkify-it-py            2.0.0
Markdown                 3.4.1
markdown-it-py           2.2.0
MarkupSafe               2.1.1
matplotlib               3.7.1
mdit-py-plugins          0.3.5
mdurl                    0.1.2
mkl-fft                  1.3.1
mkl-random               1.2.2
mkl-service              2.4.0
mpmath                   1.2.1
multidict                6.0.4
networkx                 2.8.4
numpy                    1.23.5
nvidia-cublas-cu11       11.10.3.66
nvidia-cuda-nvrtc-cu11   11.7.99
nvidia-cuda-runtime-cu11 11.7.99
nvidia-cudnn-cu11        8.5.0.96
orjson                   3.8.7
packaging                23.0
pandas                   1.5.3
peft                     0.2.0
Pillow                   9.4.0
pip                      23.0.1
psutil                   5.9.4
PuLP                     2.7.0
pycparser                2.21
pycryptodome             3.17
pydantic                 1.10.6
pydub                    0.25.1
pyOpenSSL                23.0.0
pyparsing                3.0.9
pyrsistent               0.19.3
PySocks                  1.7.1
python-dateutil          2.8.2
python-multipart         0.0.6
pytz                     2022.7.1
PyYAML                   6.0
quant-cuda               0.0.0
regex                    2022.10.31
requests                 2.28.1
rfc3986                  1.5.0
rwkv                     0.4.2
safetensors              0.3.0
sentencepiece            0.1.94
setuptools               65.6.3
six                      1.16.0
sniffio                  1.3.0
starlette                0.26.1
sympy                    1.11.1
tokenizers               0.13.2
toolz                    0.12.0
torch                    1.13.1
torchaudio               2.0.0
torchvision              0.15.0
tqdm                     4.65.0
transformers             4.28.0.dev0
triton                   2.0.0
typing_extensions        4.4.0
uc-micro-py              1.0.1
urllib3                  1.26.14
uvicorn                  0.21.1
websockets               10.4
wheel                    0.38.4
yarl                     1.8.2
zipp                     3.15.0

(gptq) /LLaMA/text-generation-webui$ tree models
├── llama-7b
│   ├── checklist.chk
│   ├── consolidated.00.pth
│   └── params.json
├── llama-7b-4bit.pt
├── llama-7b-hf
│   ├── config.json
│   ├── generation_config.json
│   ├── pytorch_model.bin.index.json
│   ├── special_tokens_map.json
│   ├── tokenizer_config.json
│   └── tokenizer.model
└── place-your-models-here.txt

(gptq) $ python server.py --gptq-bits 4 --model llama-7b-hf

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================
/home/lb/miniconda3/envs/gptq/lib/python3.9/site-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: /home/lb/miniconda3/envs/gptq did not contain libcudart.so as expected! Searching further paths...
  warn(msg)
/home/lb/miniconda3/envs/gptq/lib/python3.9/site-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: /usr/lib/x86_64-linux-gnu:/usr/lib did not contain libcudart.so as expected! Searching further paths...
  warn(msg)
/home/lb/miniconda3/envs/gptq/lib/python3.9/site-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('@/tmp/.ICE-unix/2018,unix/lb-desktop'), PosixPath('local/lb-desktop')}
  warn(msg)
/home/lb/miniconda3/envs/gptq/lib/python3.9/site-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/etc/xdg/xdg-ubuntu')}
  warn(msg)
/home/lb/miniconda3/envs/gptq/lib/python3.9/site-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/usr/share/gconf/ubuntu.mandatory.path')}
  warn(msg)
/home/lb/miniconda3/envs/gptq/lib/python3.9/site-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('0'), PosixPath('1')}
  warn(msg)
/home/lb/miniconda3/envs/gptq/lib/python3.9/site-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/org/gnome/Terminal/screen/446d1daa_a336_4437_b62c_13553df3323e')}
  warn(msg)
/home/lb/miniconda3/envs/gptq/lib/python3.9/site-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/usr/share/gconf/ubuntu.default.path')}
  warn(msg)
/home/lb/miniconda3/envs/gptq/lib/python3.9/site-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//debuginfod.ubuntu.com '), PosixPath('https')}
  warn(msg)
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching /usr/local/cuda/lib64...
CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so
CUDA SETUP: Highest compute capability among GPUs detected: 6.1
CUDA SETUP: Detected CUDA version 117
/home/lb/miniconda3/envs/gptq/lib/python3.9/site-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: WARNING: Compute capability < 7.5 detected! Only slow 8-bit matmul is supported for your GPU!
  warn(msg)
CUDA SETUP: Loading binary /home/lb/miniconda3/envs/gptq/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda117_nocublaslt.so...
Loading llama-7b-hf...
CUDA extension not installed.
Loading model ...
Done.
Traceback (most recent call last):
  File "/home/lb/Downloads/LLaMA/text-generation-webui/server.py", line 236, in <module>
    shared.model, shared.tokenizer = load_model(shared.model_name)
  File "/home/lb/Downloads/LLaMA/text-generation-webui/modules/models.py", line 158, in load_model
    tokenizer = AutoTokenizer.from_pretrained(Path(f"models/{shared.model_name}/"))
  File "/home/lb/miniconda3/envs/gptq/lib/python3.9/site-packages/transformers/models/auto/tokenization_auto.py", line 677, in from_pretrained
    raise ValueError(
ValueError: Tokenizer class LLaMATokenizer does not exist or is not currently imported

I had to downgrade my CUDA stack down to 11.7 and use torch 1.13. Best of luck!

I am using CUDA version 11.7 as well, with torch 1.13.1

@brandonrobertz That is not correct - https://github.com/pytorch/pytorch/blob/master/requirements.txt

That's the wrong library. It's a dep of transformers, not torch:

https://github.com/huggingface/transformers/blob/main/setup.py#L162

Since you need to use the fork specified in the README (-e git+https://github.com/zphang/transformers@660dd6e2bbc9255aacd0e60084cf15df1b6ae00d#egg=transformers), this is a common failure mode for people.

My sentencepiece is installed as version 0.1.94. Which satisfies the constraint in the link.