Comments (4)
Hi,
Thanks for the good words!
The feature map needs to correspond to a non-negative similarity score. That means any <φ(x), φ(y)> >= 0 for all x, y would be usable. Section 3.2.1 in the paper has a few more details and discussion regarding that matter.
Cheers,
Angelos
from fast-transformers.
I use function elu(x)+1 in the model,I found that the convergence was very slow ,Whether choosing another mapping function can accelerate the convergence? Which mapping function is better?
Thanks for your help.
from fast-transformers.
I would say this is an open research problem. In some cases we observed the same (see supplementary section B) in others not. Increasing the dimensions of the query and key (or the number of layers) would make the model slower but increase its representation capacity which might improve the convergence.
For now it depends on the problem, problems that require sparse attention patterns might be harder to learn using this feature map.
from fast-transformers.
I am closing this issue since it is not a bug or a new feature.
Feel free to reopen it if you have more questions.
from fast-transformers.
Related Issues (20)
- Parallel complexity of Linear Attention is O(N)? HOT 1
- how causal mask constructed in training batch model with linear causal attention?
- Runtime error on causal_product_cpu on GCC/G++ 11 HOT 3
- Casual attention is cheating by looking in the future HOT 1
- Installing error on linux HOT 4
- Any decoder example? HOT 1
- Can't officially save Linear Attention model HOT 2
- can offer built code for linus?
- Speed of recurrent model HOT 2
- Cuda version HOT 1
- Example for NLP
- Understanding how to define key, query and value for the cross attention calculation
- Detailed implementation of `clustered_sparse_dot_product`
- Windows installation - building wheel error HOT 27
- Got different result for the same batch HOT 1
- Error about `causal_product_cpu.cpython-38-darwin.so` on Mac HOT 2
- `.causal_product_cuda` missing in pip installed version on linux HOT 4
- Provenance of algorithms
- ImportError HOT 1
- ERROR: Could not build wheels for pytorch-fast-transformers, which is required to install pyproject.toml-based projects HOT 10
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fast-transformers.