Comments (1)
-
For BERT, we set batch_size = 6, Seqence_len = 512, Embed_dim = 1024, Heads=16, MLP_ratio=4
-
For ViT, we set batch_size = 48, Seqence_len = 64, Embed_dim = 1024, Heads=16, MLP_ratio=4
- For QKV and Projection layer M = (batch_size * Seqence_len), N = Embed_dim, K = Embed_dim.
- For MLP layers, N or K should further multiply with "MLP_ratio".
- For batch dot or multi-head attention, M = Seqence_len, N (K) = Embed_dim//Heads, K (N) = Seqence_len; batch dot size = batch_size *Heads.
Hope this will help!
from charm.
Related Issues (14)
- Some questions about the paper and code of CHARM HOT 3
- How to profile DDR bandwidth? HOT 1
- ERROR: [v++ 60-602] Source file does not exist: hw.xclbin HOT 2
- [AIE-ROUTER-3] Demand of 200 at node AIE_INTF_B3_CORE_X24Y0/AIE_PL_TO_AIE_0 exceeds HOT 1
- wrong output of aie simulation of int8 mm_kernel0 HOT 2
- About pipeline view in paper DAC'23 HOT 1
- how to run sd card image on qemu ? please provide some insights for testing the images built after make all command on qemu emulation HOT 4
- Getting Error after launching launch_hw_emu.sh HOT 3
- Not able to run launch_hw_emu.sh . It is giving some jupyter notebook not found issue please provide appropriate steps to successfully launch it and test it HOT 1
- How can i run and test simulation for different examples such as BERT and all ? Is it fixed bert application like if i need to run it for different bert version having different matrix size how can i run it ?
- I have to run general application lets say any cnn applications ? What changes i need to do ? Can you please give me overview how can i use existing charm code to do that !
- Vkc190 Simulator? HOT 1
- non-GEMM operations HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from charm.