Comments (4)
Hi,
Thank you for your interest in our work. Yes, we will release a separate repo with our experiments. Though for the above code snippet, I can see that the position embeddings are missing. Try adding those to see if the loss converges.
Thanks,
Apoorv
from fast-transformers.
Ah yes of course, can you explain how you handled the positional embeddings for ASR?
from fast-transformers.
We use fixed sinusoidal positional embeddings that are concatenated to the input speech features.
The concatenated features are then linearly projected and given as input to the transformer.
Following is the PositionalEmbedding class that takes as input speech features of shape N x T x E and returns concatenated features. The output has the shape N x T x (E + d_model)
class PositionalEmbedding(nn.Module):
def __init__(self, d_model, dropout=0.0, max_len=5000):
super(PositionalEmbedding, self).__init__()
self.dropout = nn.Dropout(p=dropout)
self.d_model = d_model
pe = torch.zeros(max_len, d_model)
position = torch.arange(0, max_len, dtype=torch.float).unsqueeze(1)
div_term = torch.exp(torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model))
pe[:, 0::2] = torch.sin(position * div_term)
pe[:, 1::2] = torch.cos(position * div_term)
pe = pe.unsqueeze(0)
self.register_buffer('pe', pe)
def forward(self, x):
pos_embedding = self.pe[:, :x.size(1), :]
pos_embedding = torch.repeat_interleave(pos_embedding, x.shape[0], dim=0)
x = torch.cat([x, pos_embedding], dim=2)
return self.dropout(x)
from fast-transformers.
@iiSeymour Chris I am closing the issue, feel free to reopen it or submit a new one if you are still experiencing problems.
Cheers,
Angelos
from fast-transformers.
Related Issues (20)
- Runtime error on causal_product_cpu on GCC/G++ 11 HOT 3
- Casual attention is cheating by looking in the future HOT 1
- Installing error on linux HOT 4
- Any decoder example? HOT 1
- Can't officially save Linear Attention model HOT 2
- can offer built code for linus?
- Speed of recurrent model HOT 2
- Cuda version HOT 1
- Example for NLP
- Understanding how to define key, query and value for the cross attention calculation
- Detailed implementation of `clustered_sparse_dot_product`
- Windows installation - building wheel error HOT 27
- Got different result for the same batch HOT 1
- Error about `causal_product_cpu.cpython-38-darwin.so` on Mac HOT 2
- `.causal_product_cuda` missing in pip installed version on linux HOT 4
- Provenance of algorithms
- ImportError HOT 1
- ERROR: Could not build wheels for pytorch-fast-transformers, which is required to install pyproject.toml-based projects HOT 10
- [WinError 2] The system cannot find the file specified: build_ext HOT 1
- Speed of linear attention slower than the attention implemented in pytorch
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fast-transformers.