a question about PositionalEmbedding module about behavior-sequence-transformer-pytorch

[watchwater]

Hello, thanks for your patient and detailed explanation, I will read code again with the help of this explanation.

[jiwidi]

Hi! Thanks for the question, let me try answer it

Is exactly just returning the weight of the positional embedding, weight is the actual value of the embeddings.

Since we are using a fixed sequence length and we are creating positional embeddings for positions==max_len we will want the full weight of the positional embedding table.

So for max_len=10 the positional embedding table will have 10 entries right?? We want to access those 10 entries and sum the value of those entries to the already existing sequence items (we are doing sum of positional embedding here, not concat)

[shamanez]

    
    def forward(self, batch):
        transfomer_features, user_features, target_movie_rating = self.encode_input(batch)
        transfomer_features = self.positional_embedding(transfomer_features)
        transformer_output = self.transfomerlayer(transfomer_features)
        transformer_output = torch.flatten(transformer_output,start_dim=1)

According to the above function, you only input the positional encoding to the transformer layer. Because the output from self.positional_embedding is actually an embedding lookup. I think you need an extra step to add transfomer_features, with positional encodings before sending it through the transformer layer.