Comments (4)
目前已经支持了,详细情况参考 #326 ,此外还有个问题需要注意一下:
- transformer类的模型,例如你这边使用的Whisper,模型定义大部分是通过config在init时就固定了,剪枝后模型的部分维度发生变化,你需要手动改一下这部分维度的定义。例如在你提供的WhisperAttention中进行剪枝,需要减少num_heads和embed_dim,剪枝后需要手动进行修改。
pruner = OneShotChannelPruner(model, torch.ones(1, 80, 3000), args.config)
# (Optional) A new config file with layer-level sparsity will be generated inplace
# If you want to customize those generated content, you may do that before calling `.prune`
pruner.generate_config(args.config)
# Get the pruned untrained model
pruner.prune()
# Manually modify related predefined dimension variables
for idx_l, layer in enumerate(model.audio_encoder.model.layers):
layer.self_attn.num_heads = layer.self_attn.q_proj.out_features // layer.self_attn.head_dim
layer.self_attn.embed_dim = layer.self_attn.num_heads * layer.self_attn.head_dim
model(torch.ones(1, 80, 3000))
exit()
from tinyneuralnetwork.
目前已经支持了,详细情况参考 #326 ,此外还有个问题需要注意一下:
- transformer类的模型,例如你这边使用的Whisper,模型定义大部分是通过config在init时就固定了,剪枝后模型的部分维度发生变化,你需要手动改一下这部分维度的定义。例如在你提供的WhisperAttention中进行剪枝,需要减少num_heads和embed_dim,剪枝后需要手动进行修改。
pruner = OneShotChannelPruner(model, torch.ones(1, 80, 3000), args.config) # (Optional) A new config file with layer-level sparsity will be generated inplace # If you want to customize those generated content, you may do that before calling `.prune` pruner.generate_config(args.config) # Get the pruned untrained model pruner.prune() # Manually modify related predefined dimension variables for idx_l, layer in enumerate(model.audio_encoder.model.layers): layer.self_attn.num_heads = layer.self_attn.q_proj.out_features // layer.self_attn.head_dim layer.self_attn.embed_dim = layer.self_attn.num_heads * layer.self_attn.head_dim model(torch.ones(1, 80, 3000)) exit()
确实能跑通了,谢谢!请问transformer剪枝的时候剪掉的是注意力头吗
from tinyneuralnetwork.
一般三个维度剪,Attention_head,MLP_intermediate_size,hidden_size。目前是屏蔽了hidden_size维度,剪前两者。你可以把模型print看一下对应维度的减少。
from tinyneuralnetwork.
一般三个维度剪,Attention_head,MLP_intermediate_size,hidden_size。目前是屏蔽了hidden_size维度,剪前两者。你可以把模型print看一下对应维度的减少。
明白了,多谢多谢
from tinyneuralnetwork.
Related Issues (20)
- Does tinynn support following int16 quantization? HOT 1
- jit.trace succeed but tinynn tracer failed HOT 1
- It became larger after converting to tflite model HOT 4
- how to do Post-training integer quantization with int16 activation HOT 4
- unnecessary float() variables cause quantization to fail. HOT 7
- aten::index nodes take multiple indices in PyTorch model but cause an error when trying to convert to TFLite HOT 1
- aten::repeat_interleave is considered an unsupported Tensor and causing an error when trying to convert to TFLite HOT 2
- convert model error HOT 5
- 请问下 转tflite 模型能支持签名吗? HOT 9
- [PTQ Converter] 'Linear+relu' module conversion failed HOT 9
- [quantizer] activation nodes that was used multiple times will not work with OP fusion
- [converter] aten::pad with mode="circular"
- [converter] batchnorm + conv fusion
- [Converter] TFLite size is larger than expected HOT 7
- ONNX进行QAT报错 HOT 3
- 关于模型中使用了nn.Parameter()会导致转TFlite会失败的问题 HOT 5
- Huge loss when applying QAT on YOLOv8n HOT 40
- ViT PTQ Error HOT 2
- QAT problem on YOLOv9 HOT 7
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tinyneuralnetwork.