Experimental Code - CTC Loss issue. about fast-transformers

[apoorv2904]

Hi,

Thank you for your interest in our work. Yes, we will release a separate repo with our experiments. Though for the above code snippet, I can see that the position embeddings are missing. Try adding those to see if the loss converges.

Thanks,
Apoorv

[iiSeymour]

Ah yes of course, can you explain how you handled the positional embeddings for ASR?

[apoorv2904]

We use fixed sinusoidal positional embeddings that are concatenated to the input speech features.
The concatenated features are then linearly projected and given as input to the transformer.

Following is the PositionalEmbedding class that takes as input speech features of shape N x T x E and returns concatenated features. The output has the shape N x T x (E + d_model)

class PositionalEmbedding(nn.Module):

    def __init__(self, d_model, dropout=0.0, max_len=5000):
        super(PositionalEmbedding, self).__init__()
        self.dropout = nn.Dropout(p=dropout)

        self.d_model = d_model
        pe = torch.zeros(max_len, d_model)
        position = torch.arange(0, max_len, dtype=torch.float).unsqueeze(1)
        div_term = torch.exp(torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model))
        pe[:, 0::2] = torch.sin(position * div_term)
        pe[:, 1::2] = torch.cos(position * div_term)
        pe = pe.unsqueeze(0)
        self.register_buffer('pe', pe) 

    def forward(self, x): 
        pos_embedding =  self.pe[:, :x.size(1), :]
        pos_embedding = torch.repeat_interleave(pos_embedding, x.shape[0], dim=0)
        x =  torch.cat([x, pos_embedding], dim=2)
        return self.dropout(x)

[angeloskath]

@iiSeymour Chris I am closing the issue, feel free to reopen it or submit a new one if you are still experiencing problems.

Cheers,
Angelos