Explanation of the input parameters of the forward function of the model about mmt

[gabeur]

Thanks for your interest in our work.

• features_t encode the time in the video at which each feature was extracted. As described in the section 3.1 of the paper, we use temporal embeddings to provide temporal information about the time in the video when each feature was extracted. The temporal information of each feature is discretized here and then embedded here.
• features_ind encode if the feature is valid (1) or if it is padding (0). It is then used here to set the attention mask so that padding tokens are not attended.

[code10086web]

Thank you very much for your reply. By any chance, Is it possible for you to explain the meaning of the parameters of the vid_bert function on line 577 of file model/model.py? especially the parameter input_ids.

[gabeur]

• input_ids were used initially to encode the different token classes but it became useless with the temporal embeddings. You will find in the forward function that it is only used to infer the shape of the missing token_type_ids.
• attention_mask encode the tokens that should be attended (valid tokens) in contrast to the padding tokens that should not be attended.
• token_type_ids encode the expert used to extract the feature (called "Expert embeddings" in part 3.1 of the paper).
• position_ids encode the time in the video at which each feature was extracted (called "Temporal embeddings" in part 3.1 of the paper)
• features are the expert features (called "Features" in part 3.1 of the paper)