Comments (6)
Safetensors were only just implemented as they have not been released on PyPi yet. You must install from main to use this feature until the next release (hopefully soon).
from autoawq.
Did git clone https://github.com/casper-hansen/AutoAWQ && cd AutoAWQ && pip install .
.
Now I'm greeted with
No module named 'awq.modules.fused
Runner failed with exception: ModuleNotFoundError("No module named 'awq.modules.fused'")
File "/pkg/modal/_container_entrypoint.py", line 484, in call_function_async
imp_fun.obj.__enter__()
File "/root/pygmalion-13b-q4.py", line 72, in __enter__
from awq import AutoAWQForCausalLM
File "/usr/local/lib/python3.10/dist-packages/awq/__init__.py", line 2, in <module>
from awq.models.auto import AutoAWQForCausalLM
File "/usr/local/lib/python3.10/dist-packages/awq/models/__init__.py", line 1, in <module>
from .mpt import MptAWQForCausalLM
File "/usr/local/lib/python3.10/dist-packages/awq/models/mpt.py", line 70, in <module>
from awq.modules.fused.block import MPTBlock
ModuleNotFoundError: No module named 'awq.modules.fused'
I'm inspired by https://huggingface.co/TheBloke/Pygmalion-2-13B-AWQ#install-the-necessary-packages
Will try rebuilding from source with -e
parameter.
As far as I know, it was based on Llama 2 - not MPT model
EDIT:
Failed too
Cloning into 'AutoAWQ'...
Obtaining file:///AutoAWQ
Preparing metadata (setup.py): started
Preparing metadata (setup.py): finished with status 'error'
error: subprocess-exited-with-error
× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [8 lines of output]
Traceback (most recent call last):
File "<string>", line 2, in <module>
File "<pip-setuptools-caller>", line 34, in <module>
File "/AutoAWQ/setup.py", line 96, in <module>
arch_flags = get_compute_capabilities()
File "/AutoAWQ/setup.py", line 81, in get_compute_capabilities
raise RuntimeError("GPUs with compute capability less than 8.0 are not supported.")
RuntimeError: GPUs with compute capability less than 8.0 are not supported.
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed
× Encountered error while generating package metadata.
╰─> See above for output.
note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
Terminating task due to error: failed to run builder command "git clone https://github.com/casper-hansen/AutoAWQ && cd AutoAWQ && pip install -e ."
I'll stash until you release 0.0.3 :)
from autoawq.
It looks like the GPU you are using is not compatible and then your installation fails. Hoping to release soon when #53 is merged
from autoawq.
I fixed the error below. The module was not found because it was missing the __init__.py
file. Releasing v0.1.0 on PyPi today.
Runner failed with exception: ModuleNotFoundError("No module named 'awq.modules.fused'")
File "/pkg/modal/_container_entrypoint.py", line 484, in call_function_async
imp_fun.obj.__enter__()
File "/root/pygmalion-13b-q4.py", line 72, in __enter__
from awq import AutoAWQForCausalLM
File "/usr/local/lib/python3.10/dist-packages/awq/__init__.py", line 2, in <module>
from awq.models.auto import AutoAWQForCausalLM
File "/usr/local/lib/python3.10/dist-packages/awq/models/__init__.py", line 1, in <module>
from .mpt import MptAWQForCausalLM
File "/usr/local/lib/python3.10/dist-packages/awq/models/mpt.py", line 70, in <module>
from awq.modules.fused.block import MPTBlock
ModuleNotFoundError: No module named 'awq.modules.fused'
from autoawq.
pip install autoawq==0.1.0
from autoawq.
Got it working! Really excited to measure the speed improvements coming from AutoGPTQ. Thank you for getting this road block out of the way, Casper.
from autoawq.
Related Issues (20)
- when quantize qwen2 by autoawq, it not works successful. HOT 2
- Support Qwen2-57B-A14B?
- awqint4 to gguf ,ModuleNotFoundError: No module named 'awq.apply_awq' HOT 4
- Is there an example on how to quantize with multiple GPUs? Is it possible to quantize Llama 3 70B with 2x3090 24GB?
- I wanted to add a pull request, but it was closed immediately, prompting that the base branch is protected. HOT 2
- ConnectionError: Couldn't reach 'mit-han-lab/pile-val-backup' on the Hub (ConnectTimeout) HOT 11
- [Performance degrade]phi-3-medium-128k-instruct after awq quantized, then output repetitively HOT 2
- Support Qwen2 72 Awq quantization? HOT 2
- RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:7 and cuda:0! (when checking argument for argument index in method wrapper_CUDA__index_select)
- Unable to install with `poetry`
- Is it possible to quantize MoE models?
- Multi-GPU quantization randomly loads all host GPUs HOT 1
- Same AWQ model behaves differently on two similar machines
- deepseek-coder-v2-instruction-awq HOT 8
- Version on PyPi doesn't support Python 3.12
- Performance with PyTorch 2.3.x HOT 4
- Calibration Dataset: how to avoid computing loss on instructions?
- I encountered the following problem during the KL assessment
- Lora Adapters Support
- Gemma2 Support HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from autoawq.