Comments (12)
For anyone experiencing this problem, it could be related to the entry inside the tokenizer_config.json
as they changed this recently: (LlamaTokenizer
not LLaMaTokenizer
).
"tokenizer_class": "LlamaTokenizer",
Changing this solved it for me.
from gptq-for-llama.
I had the same issue. In my case the pip package sentencepiece was missing preventing some imports in the LLaMa modules of transformers.
from gptq-for-llama.
For anyone experiencing this problem, it could be related to the entry inside the
tokenizer_config.json
as they changed this recently: (LlamaTokenizer
notLLaMaTokenizer
)."tokenizer_class": "LlamaTokenizer",
Changing this solved it for me.
I can confirm this is the correct solution to the problem.
huggingface/transformers#22222 (comment)
from gptq-for-llama.
It is possible that the checkpoint is out of date.
Try converting again.
from gptq-for-llama.
Do you mean by running python setup_cuda.py install
again?
Because that's what downloaded the checkpoints in the first place
from gptq-for-llama.
Torch version is too new. Try 1.X
from gptq-for-llama.
I am on CUDA 12.1, which is not supported by torch 1.x.
from gptq-for-llama.
I had to downgrade my CUDA stack down to 11.7 and use torch 1.13. Best of luck!
from gptq-for-llama.
I had the same issue. In my case the pip package sentencepiece was missing preventing some imports in the LLaMa modules of transformers.
Interesting. sentencepiece is a dependency of transformers, so a proper and successful install of transformers should result in sentencepiece being installed as well.
from gptq-for-llama.
@brandonrobertz That is not correct - https://github.com/pytorch/pytorch/blob/master/requirements.txt
@DanielWe2 Yep, that was it for me too, thanks for finding it out!
from gptq-for-llama.
@brandonrobertz That is not correct - https://github.com/pytorch/pytorch/blob/master/requirements.txt
That's the wrong library. It's a dep of transformers, not torch:
https://github.com/huggingface/transformers/blob/main/setup.py#L162
Since you need to use the fork specified in the README (-e git+https://github.com/zphang/transformers@660dd6e2bbc9255aacd0e60084cf15df1b6ae00d#egg=transformers
), this is a common failure mode for people.
from gptq-for-llama.
I'm still not sure how to fix this. I'm getting this import exception as well.
(gptq)$ pip list
Package Version
------------------------ -----------
accelerate 0.17.1
aiofiles 23.1.0
aiohttp 3.8.4
aiosignal 1.3.1
altair 4.2.2
anyio 3.6.2
async-timeout 4.0.2
attrs 22.2.0
bitsandbytes 0.37.1
brotlipy 0.7.0
certifi 2022.12.7
cffi 1.15.1
charset-normalizer 2.0.4
click 8.1.3
contourpy 1.0.7
cryptography 39.0.1
cycler 0.11.0
entrypoints 0.4
fastapi 0.94.1
ffmpy 0.3.0
filelock 3.9.0
flexgen 0.1.7
flit_core 3.6.0
fonttools 4.39.2
frozenlist 1.3.3
fsspec 2023.3.0
gmpy2 2.1.2
gradio 3.18.0
h11 0.14.0
httpcore 0.16.3
httpx 0.23.3
huggingface-hub 0.13.2
idna 3.4
importlib-metadata 6.0.0
importlib-resources 5.12.0
Jinja2 3.1.2
jsonschema 4.17.3
kiwisolver 1.4.4
linkify-it-py 2.0.0
Markdown 3.4.1
markdown-it-py 2.2.0
MarkupSafe 2.1.1
matplotlib 3.7.1
mdit-py-plugins 0.3.5
mdurl 0.1.2
mkl-fft 1.3.1
mkl-random 1.2.2
mkl-service 2.4.0
mpmath 1.2.1
multidict 6.0.4
networkx 2.8.4
numpy 1.23.5
nvidia-cublas-cu11 11.10.3.66
nvidia-cuda-nvrtc-cu11 11.7.99
nvidia-cuda-runtime-cu11 11.7.99
nvidia-cudnn-cu11 8.5.0.96
orjson 3.8.7
packaging 23.0
pandas 1.5.3
peft 0.2.0
Pillow 9.4.0
pip 23.0.1
psutil 5.9.4
PuLP 2.7.0
pycparser 2.21
pycryptodome 3.17
pydantic 1.10.6
pydub 0.25.1
pyOpenSSL 23.0.0
pyparsing 3.0.9
pyrsistent 0.19.3
PySocks 1.7.1
python-dateutil 2.8.2
python-multipart 0.0.6
pytz 2022.7.1
PyYAML 6.0
quant-cuda 0.0.0
regex 2022.10.31
requests 2.28.1
rfc3986 1.5.0
rwkv 0.4.2
safetensors 0.3.0
sentencepiece 0.1.94
setuptools 65.6.3
six 1.16.0
sniffio 1.3.0
starlette 0.26.1
sympy 1.11.1
tokenizers 0.13.2
toolz 0.12.0
torch 1.13.1
torchaudio 2.0.0
torchvision 0.15.0
tqdm 4.65.0
transformers 4.28.0.dev0
triton 2.0.0
typing_extensions 4.4.0
uc-micro-py 1.0.1
urllib3 1.26.14
uvicorn 0.21.1
websockets 10.4
wheel 0.38.4
yarl 1.8.2
zipp 3.15.0
(gptq) /LLaMA/text-generation-webui$ tree models
├── llama-7b
│ ├── checklist.chk
│ ├── consolidated.00.pth
│ └── params.json
├── llama-7b-4bit.pt
├── llama-7b-hf
│ ├── config.json
│ ├── generation_config.json
│ ├── pytorch_model.bin.index.json
│ ├── special_tokens_map.json
│ ├── tokenizer_config.json
│ └── tokenizer.model
└── place-your-models-here.txt
(gptq) $ python server.py --gptq-bits 4 --model llama-7b-hf
===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================
/home/lb/miniconda3/envs/gptq/lib/python3.9/site-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: /home/lb/miniconda3/envs/gptq did not contain libcudart.so as expected! Searching further paths...
warn(msg)
/home/lb/miniconda3/envs/gptq/lib/python3.9/site-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: /usr/lib/x86_64-linux-gnu:/usr/lib did not contain libcudart.so as expected! Searching further paths...
warn(msg)
/home/lb/miniconda3/envs/gptq/lib/python3.9/site-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('@/tmp/.ICE-unix/2018,unix/lb-desktop'), PosixPath('local/lb-desktop')}
warn(msg)
/home/lb/miniconda3/envs/gptq/lib/python3.9/site-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/etc/xdg/xdg-ubuntu')}
warn(msg)
/home/lb/miniconda3/envs/gptq/lib/python3.9/site-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/usr/share/gconf/ubuntu.mandatory.path')}
warn(msg)
/home/lb/miniconda3/envs/gptq/lib/python3.9/site-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('0'), PosixPath('1')}
warn(msg)
/home/lb/miniconda3/envs/gptq/lib/python3.9/site-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/org/gnome/Terminal/screen/446d1daa_a336_4437_b62c_13553df3323e')}
warn(msg)
/home/lb/miniconda3/envs/gptq/lib/python3.9/site-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/usr/share/gconf/ubuntu.default.path')}
warn(msg)
/home/lb/miniconda3/envs/gptq/lib/python3.9/site-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//debuginfod.ubuntu.com '), PosixPath('https')}
warn(msg)
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching /usr/local/cuda/lib64...
CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so
CUDA SETUP: Highest compute capability among GPUs detected: 6.1
CUDA SETUP: Detected CUDA version 117
/home/lb/miniconda3/envs/gptq/lib/python3.9/site-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: WARNING: Compute capability < 7.5 detected! Only slow 8-bit matmul is supported for your GPU!
warn(msg)
CUDA SETUP: Loading binary /home/lb/miniconda3/envs/gptq/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda117_nocublaslt.so...
Loading llama-7b-hf...
CUDA extension not installed.
Loading model ...
Done.
Traceback (most recent call last):
File "/home/lb/Downloads/LLaMA/text-generation-webui/server.py", line 236, in <module>
shared.model, shared.tokenizer = load_model(shared.model_name)
File "/home/lb/Downloads/LLaMA/text-generation-webui/modules/models.py", line 158, in load_model
tokenizer = AutoTokenizer.from_pretrained(Path(f"models/{shared.model_name}/"))
File "/home/lb/miniconda3/envs/gptq/lib/python3.9/site-packages/transformers/models/auto/tokenization_auto.py", line 677, in from_pretrained
raise ValueError(
ValueError: Tokenizer class LLaMATokenizer does not exist or is not currently imported
I had to downgrade my CUDA stack down to 11.7 and use torch 1.13. Best of luck!
I am using CUDA version 11.7 as well, with torch 1.13.1
@brandonrobertz That is not correct - https://github.com/pytorch/pytorch/blob/master/requirements.txt
That's the wrong library. It's a dep of transformers, not torch:
https://github.com/huggingface/transformers/blob/main/setup.py#L162
Since you need to use the fork specified in the README (
-e git+https://github.com/zphang/transformers@660dd6e2bbc9255aacd0e60084cf15df1b6ae00d#egg=transformers
), this is a common failure mode for people.
My sentencepiece is installed as version 0.1.94. Which satisfies the constraint in the link.
from gptq-for-llama.
Related Issues (20)
- llama_inference 4bits error
- inference with the saved model error: AttributeError: module 'torch.backends.cuda' has no attribute 'sdp_kernel' HOT 2
- Could not obtain official perplexity using bloom_eval()
- can it support openllama model?
- Issue with GPTQ HOT 1
- High PPL when groupsize != -1 for OPT model after replace linear layer with quantlinear. HOT 1
- Help: Quantized llama-7b model with custom prompt format produces only gibberish HOT 1
- Why does the model quantization prompt KILLED at the end? HOT 2
- Would GPTQ be able to support LLaMa2? HOT 1
- Can i quantize HF version of llama model
- Transformers broke again (AttributeError: 'GPTQ' object has no attribute 'inp1') HOT 1
- LoRa and diff with bitsandbytes
- neox.py needs to add "import math"
- error: block with no terminator, has llvm.cond_br %5624, ^bb2, ^bb3
- Support Mistral.
- Syntax changed in triton.testing.do_bench() causing error when running llama_inference.py
- datasets.utils.info_utils.ExpectedMoreSplits: {'validation'} HOT 1
- Error when load GPTQ model
- GPTQ vs bitsandbytes
- add support for minicpm
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from gptq-for-llama.