Comments (5)
Hi,
Since the attention is never explicitly computed in any "linear" model it is not possible to use a simple relative positional encoding because it would require a different value to be added per query, making the attention computation quadratic again.
If you can come up with a factorized relative positional embedding (or if one exists already I don't know of one), then you could add it to both linear and causal-linear.
By factorized, I mean something like the following:
Let E_i be the embedding for position i then in normal positional encoding you can have something of the following form.
Q'_i = Q_i + E_i
K'_ij = K_j + E_{j-i}
which gives a dot product <Q_i, K_j> + <E_i, E_{j-i}>. That second term of the dot product is what needs to be modeled in a way such that i is not present in the right hand side of the dot product. At first glance I do not see a solution.
Cheers,
Angelos
from fast-transformers.
That is a bummer, I loved the performance of the lib, and the RNN trick is amazing.
@angeloskath Do you recommend a Positional Encoding that can work with fast-transformers for the Named-entity recognition in (VERY long) texts?
I tried Axial (https://github.com/lucidrains/axial-positional-embedding), sinusoidal and Embedding.
I got terrible results with Axial, and so-so with sinusoidal and Embedding. If you can give me any pointers, I will be very happy
from fast-transformers.
I can't say that I know of an absolute position embedding that would work better for long sequences. Probably a good area for research.
from fast-transformers.
I am closing the issue but feel free to reopen it or open a new one, especially if you find a good way to do relative positional encoding with kernelized attention.
Thanks,
Angelos
from fast-transformers.
For those interested, the proposed method in Transformers with convolutional context for ASR
might be one convenient way to mimic the effects of relative positional embedding.
For ASR, this works quite well and as reported in several papers, outperforms fixed sinusoidal embeddings (with softmax transformer).
Thanks,
Apoorv
from fast-transformers.
Related Issues (20)
- CUDA error: CUBLAS_STATUS_INVALID_VALUE HOT 1
- causal-linear do not use attn_mask ? HOT 1
- Windows installation - Building wheel HOT 3
- Training Language Model HOT 2
- Parallel complexity of Linear Attention is O(N)? HOT 1
- how causal mask constructed in training batch model with linear causal attention?
- Runtime error on causal_product_cpu on GCC/G++ 11 HOT 3
- Casual attention is cheating by looking in the future HOT 1
- Installing error on linux HOT 4
- Any decoder example? HOT 1
- Can't officially save Linear Attention model HOT 2
- can offer built code for linus?
- Speed of recurrent model HOT 2
- Cuda version HOT 1
- Example for NLP
- Understanding how to define key, query and value for the cross attention calculation
- Detailed implementation of `clustered_sparse_dot_product`
- Windows installation - building wheel error HOT 27
- Got different result for the same batch HOT 1
- Error about `causal_product_cpu.cpython-38-darwin.so` on Mac HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fast-transformers.