Comments (4)
we aim to stabilize the spectral-wise self-attention map produced by Q and K. For V, we apply mask-attention and use LN to process it in the subsequent layers. I can also try to apply an LN to process V in the MS-MSA module.
from mst.
Hi,
Thanks for your interests. Another layer normalization is in Line 150-151 of MS_MSA module in MST.py, check this
from mst.
Thanks for your quick reply and i hope that won't bother you.
So the normalize is only applied on Q and K in computing attention matrix, leaving V not normed. That's different from the LN in common transformer-based models which applies to the whole feature map. I wonder what the reason this design is for?
from mst.
okay i got it. appreciate your great work :)
from mst.
Related Issues (20)
- mask 3d shift s HOT 1
- 关于真实实验 HOT 3
- 请问如果想要直接使用该功能,直接按照验证步骤,并在此之前把我的RGB转化成mat数据即可吗 HOT 2
- 请问real训练过程为何掩码也进行随机裁剪 HOT 2
- 关于参数的疑问 HOT 3
- 数据集中RGB图像 HOT 2
- Question about the ADMM-Net HOT 2
- real-test.py HOT 3
- Real-architecture HOT 2
- Wrong paper links presented in README HOT 1
- 测试数据集转为RGB图像 HOT 1
- TwIST、GAP-TV、DeSCI的可视化结果 HOT 1
- 这里乘2是什么意思 HOT 1
- 训练MST方法之前所提出的模型时的问题。 HOT 11
- DAUHST的flops计算 HOT 1
- 关于test_s HOT 1
- 关于归一化问题 HOT 1
- 关于一个细节的问题 HOT 4
- 一轮训练时间 HOT 5
- mask_3d_shift和mask的关系 HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mst.