Comments (20)
We now support 4-bit quantization! See README for more details.
from cogvlm.
But --fp16
will show another error message and the process crush:
Floating point exception (core dumped)
from cogvlm.
It works for two 3090 with model parallel.
from cogvlm.
It should work. If something went wrong, feel free to post here.
from cogvlm.
It seems that quantize part of the model will show error like this during inference:
error message expected scalar type BFloat16 but found Half
from cogvlm.
Yeah I got it. This is the bug of cpm_kernel, which we cannot control actually... You can avoid this by changing --bf16
to --fp16
when running code.
from cogvlm.
emm... Could you post more detailed error info?
from cogvlm.
This is all the error info from my terminal:
$ python web_demo.py --from_pretrained cogvlm-chat --version chat --english --fp16 --quant 8
[2023-10-11 10:58:42,487] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2023-10-11 10:58:47,433] [INFO] building CogVLMModel model ...
[2023-10-11 10:58:47,437] [INFO] [RANK 0] > initializing model parallel with size 1
[2023-10-11 10:58:47,438] [INFO] [RANK 0] You are using model-only mode.
For torch.distributed users or loading model parallel models, set environment variables RANK, WORLD_SIZE and LOCAL_RANK.
[2023-10-11 10:59:00,976] [INFO] [RANK 0] > number of parameters on model parallel rank 0: 17639685376
[2023-10-11 10:59:08,248] [INFO] [RANK 0] global rank 0 is loading checkpoint /output/sat/cogvlm-chat/1/mp_rank_00_model_states.pt
[2023-10-11 11:00:53,805] [INFO] [RANK 0] > successfully loaded /output/sat/cogvlm-chat/1/mp_rank_00_model_states.pt
[2023-10-11 11:00:55,647] [INFO] [RANK 0] > Quantizing model weight to 8 bits
[2023-10-11 11:00:55,699] [INFO] [RANK 0] > Quantized 5033164800 parameters in total.
web_demo.py:168: GradioDeprecationWarning: 'scale' value should be an integer. Using 4.5 will cause issues.
with gr.Column(scale=4.5):
web_demo.py:182: GradioDeprecationWarning: 'scale' value should be an integer. Using 5.5 will cause issues.
with gr.Column(scale=5.5):
web_demo.py:183: GradioDeprecationWarning: The `style` method is deprecated. Please set these arguments in the constructor instead.
result_text = gr.components.Chatbot(label='Multi-round conversation History', value=[("", "Hi, What do you want to know about this image?")]).style(height=550)
3.47.1
3.47.1
Running on local URL: http://0.0.0.0:8080
To create a public link, set `share=True` in `launch()`.
history []
error message 'NoneType' object has no attribute 'read'
history []
Floating point exception (core dumped)
from cogvlm.
I tested... It's also because of cpm_kernels. They do not support some operations in our model... Therefore, quantization with cpm_kernels is not supported for now.
from cogvlm.
Ok, and only quantize the lm part seems not make a lot memory usage shrink...so it is acceptable...
Here is a screenshot using bf16
:
![screenshot](https://private-user-images.githubusercontent.com/661860/274280968-a16f5fde-85d3-4fa9-b106-51a7a7f2c458.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjM4ODY3NjAsIm5iZiI6MTcyMzg4NjQ2MCwicGF0aCI6Ii82NjE4NjAvMjc0MjgwOTY4LWExNmY1ZmRlLTg1ZDMtNGZhOS1iMTA2LTUxYTdhN2YyYzQ1OC5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODE3JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgxN1QwOTIxMDBaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT00YjIwZGM4Yzk5YWYzMTQ4N2I3MzhkZTNhZGNjMDYwZDk3ZDA0NWQwYWMyZjEyZTk4NWU4MmZmOTNhMDE4YzJhJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.mAlzfZD1kUJSTJHeds-2WYV5tkqi-8-Ph0wQPxEXetk)
This is really a huge memory usage...is it possible to make it work with 4090 in the future?
from cogvlm.
I tested... It's also because of cpm_kernels. They do not support some operations in our model... Therefore, quantization with cpm_kernels is not supported for now.
So may I just remove quantization related code right now? Or waiting for some progress?
from cogvlm.
@aisensiy Try add a line of code to web_demo.py:
from cogvlm.
Yes, I think you can remove the quantization related in this repo. If quantization is necessary for you, maybe you can try bitsandbytes to quantize our model.
from cogvlm.
Yes, I think you can remove the quantization related in this repo. If quantization is necessary for you, maybe you can try bitsandbytes to quantize our model.
You mean quantizing the full model is possible? (I know litte about this stuff)
from cogvlm.
Yes, but it depends on the cuda kernel support. cpm_kernels misses some implementation for bf16 and slicing fp16. I'm not sure whether bitsandbytes works.
Theoretically, quantization everything is possible. But practically, some packages may have bug.
from cogvlm.
It works for two 3090 with model parallel.
How to set?
from cogvlm.
torchrun --standalone --nnodes=1 --nproc-per-node=2 cli_demo.py --from_pretrained cogvlm-chat --version chat --english --bf16
or
torchrun --standalone --nnodes=1 --nproc-per-node=2 web_demo.py --from_pretrained cogvlm-chat --version chat --english --bf16
from cogvlm.
torchrun --standalone --nnodes=1 --nproc-per-node=2 cli_demo.py --from_pretrained cogvlm-chat --version chat --english --bf16
or
torchrun --standalone --nnodes=1 --nproc-per-node=2 web_demo.py --from_pretrained cogvlm-chat --version chat --english --bf16
torchrun: error: unrecognized arguments: --nproc-per-node=2
from cogvlm.
torchrun --standalone --nnodes=1 --nproc-per-node=2 cli_demo.py --from_pretrained cogvlm-chat --version chat --english --bf16
or
torchrun --standalone --nnodes=1 --nproc-per-node=2 web_demo.py --from_pretrained cogvlm-chat --version chat --english --bf16
torchrun: error: unrecognized arguments: --nproc-per-node=2
It is caused by the PyTorch version
from cogvlm.
Yes, I think you can remove the quantization related in this repo. If quantization is necessary for you, maybe you can try bitsandbytes to quantize our model.
while I wait for quantization support, would like to try bitsandbytes
Is this correct as I read from bitsandbytes?
in cogvlm_model.py change the GLU class linear layers to
class GLU(nn.Module):
def __init__(self, args, in_features):
super().__init__()
# self.linear_proj = nn.Linear(in_features, args.hidden_size, bias=False)
# self.norm1 = nn.LayerNorm(args.hidden_size)
# self.act1 = nn.GELU()
# self.act2 = nn.functional.silu
# self.dense_h_to_4h = nn.Linear(args.hidden_size, args.inner_hidden_size, bias=False)
# self.gate_proj = nn.Linear(args.hidden_size, args.inner_hidden_size, bias=False)
# self.dense_4h_to_h = nn.Linear(args.inner_hidden_size, args.hidden_size, bias=False)
self.linear_proj = bnb.nn.Linear8bitLt(in_features, args.hidden_size, bias=False, has_fp16_weights=False, threshold=6.0)
self.norm1 = nn.LayerNorm(args.hidden_size)
self.act1 = nn.GELU()
self.act2 = nn.functional.silu
self.dense_h_to_4h = bnb.nn.Linear8bitLt(args.hidden_size, args.inner_hidden_size, bias=False, has_fp16_weights=False, threshold=6.0)
self.gate_proj = bnb.nn.Linear8bitLt(args.hidden_size, args.inner_hidden_size, bias=False, has_fp16_weights=False, threshold=6.0)
self.dense_4h_to_h = bnb.nn.Linear8bitLt(args.inner_hidden_size, args.hidden_size, bias=False, has_fp16_weights=False, threshold=6.0)
def forward(self, x):
x = self.linear_proj(x)
x = self.act1(self.norm1(x))
x = self.act2(self.gate_proj(x)) * self.dense_h_to_4h(x)
x = self.dense_4h_to_h(x)
return x
Interestingly, when using has_fp16_weights=False, not only the quality of caption deteriorates alot, but also the time taken to caption images increases. has_fp16_weights=True takes almost same time as normal nn.Linear layer.
from cogvlm.
Related Issues (20)
- openai_api.py调用的模型类型 HOT 1
- 加载报错
- vllm适配
- 请问怎么用transformer来进行推理? HOT 2
- A bug while running cli_demo_hf.py HOT 2
- 预训练阶段段loss下降后升高了,模型训崩了 HOT 2
- 类型问题 HOT 1
- Did you fine-tune VE and MLP Adaptor at both pre-training stage and SFT stage?
- What's the difference between two CogVLM in Table 2 and Table 2
- a trouble in finetune chatglm: ZeroDivisionError: integer division or modulo by zero HOT 1
- How to evaluate the performance? HOT 1
- 拉了一个多模态大模型技术交流群,大家可以加入进来进行技术交流
- run evaluate_cogagent_demo.py error from sat import mpu, get_args, get_tokenizer error
- Reproduce results on visual7w. HOT 2
- 無法複現飛書上的示例
- When running the code of the module finetune_demo on the windosw11 system, an error will be reported
- AttributeError: 'str' object has no attribute 'shape' HOT 1
- Better set a specific version of transformers lib HOT 1
- Why is it that running inference on two cards is slower than running on a single card?
- cogvlm_grounding 并行调用和串行结果存在一定差异
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cogvlm.