milvlg / imp Goto Github PK
View Code? Open in Web Editor NEWa family of highly capabale yet efficient large multimodal models
License: Apache License 2.0
imp's People
Contributors
Stargazers
Watchers
Forkers
paradoxzw bruceisme tensoic-ai adarshxs beachwang m3dade dive-into-papers sanowl emiya-syw untergang1 saurabhlalsaxena cooleel nahidalam ricar0imp's Issues
Quantized model latency issue.
Hi, I am trying to see if I can load a quantized model of this.
When I load in 4-bit, the model size is smaller but the latency significantly increases.
Not sure if there needs to be any changes to be done to support quantization.
Please, let me know.
I can also help in creating a MR to make the quantized model better.
Thanks
The training environment
Hi,
I try to reproduction the training of Imp. But there is some problem with the training environment as following shows.
My transformers is 4.31.0 as requirement.
finetune_lora_custom.sh
I only changed
IMP_MODEL='./checkpoints/imp-v1-3b'
--data_path
--image_folder
but have this infomation in my terminal
You are using a model of type imp to instantiate a model of type llava. This is not supported for all configurations of models and can yield errors.
You are using a model of type imp to instantiate a model of type llava. This is not supported for all configurations of models and can yield errors.
You are using a model of type imp to instantiate a model of type llava. This is not supported for all configurations of models and can yield errors.
You are using a model of type imp to instantiate a model of type llava. This is not supported for all configurations of models and can yield errors.
Downloading config.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 576/576 [00:00<00:00, 1.89MB/s]
[2024-02-22 16:33:49,885] [WARNING] [partition_parameters.py:836:_post_init_method] param `probe` in SiglipMultiheadAttentionPoolingHead not on GPU so was not broadcasted from rank 0
[2024-02-22 16:33:53,686] [INFO] [partition_parameters.py:453:__exit__] finished initializing model with 7.77B parameters
Traceback (most recent call last):
File "/data1/*** /imp/imp_llava/train/train_mem.py", line 15, in <module>
train()
File "/data1/***/imp/./imp_llava/train/train.py", line 827, in train
model = LlavaLlamaForCausalLM.from_pretrained(
File "/data1/***/site-packages/transformers/modeling_utils.py", line 2903, in from_pretrained
) = cls._load_pretrained_model(
File "/data1/***/site-packages/transformers/modeling_utils.py", line 3125, in _load_pretrained_model
model.apply(model._initialize_weights)
File "/data1/***/site-packages/torch/nn/modules/module.py", line 884, in apply
module.apply(fn)
File "/data1/***/site-packages/torch/nn/modules/module.py", line 884, in apply
module.apply(fn)
File "/data1/***/site-packages/torch/nn/modules/module.py", line 885, in apply
fn(self)
File "/data1/***/site-packages/transformers/modeling_utils.py", line 1261, in _initialize_weights
self._init_weights(module)
File "/data1/***/site-packages/transformers/models/llama/modeling_llama.py", line 472, in _init_weights
module.weight.data[module.padding_idx].zero_()
IndexError: index 50256 is out of bounds for dimension 0 with size 0
[2024-02-22 16:33:55,511] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 2275311
[2024-02-22 16:33:55,524] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 2275312
[2024-02-22 16:33:55,535] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 2275313
[2024-02-22 16:33:55,545] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 2275314
demo链接错啦，把逗号改成.
Model Evaluation
Hi,
I have trained imp with lora. However, it does not process the reference when I run the evaluation scripts.
Following is the output when I eval pope.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.