pmt-aaai23's Introduction

PMT-AAAI23

Efficient End-to-End Video-Question Answering with Pyramidal Multimodal Transformer - AAAI23

PMT

This is the PyTorch Implementation of our paper "[Efficient End-to-End Video-Question Answering with Pyramidal Multimodal Transformer]". (accepted by AAAI’23)

Data Preparation

Download the dataset
MSVD-QA: link
MSRVTT-QA: link
TGIF-QA: link
ActivityNet-QA: link Youtube2Text-QA: please ref link For the text-to-video retrieval task in our ablation study, pleade ref link
Word Glove Embedding and Video Frames extraction
1. To extract questions or answers Glove Embedding, please ref here.
  Take the action task in TGIF-QA dataset as an example, we have features at the path /inputdata: TGIF/word/Action/TGIF_Action_train_questions.pt TGIF/word/Action/TGIF_Action_test_questions.pt TGIF/word/Action/TGIF_Action_vocab.json
2. To extract video frames, we use the skvideo.io module to eatract the images and then transfer it to .npz format. for Action task in the TGIF-QA dataset as example, we have .npz files at the path /inputdata: TGIF/video/Action/tumblr_no00ddSlG31t34v14o1_250.npz TGIF/video/Action/tumblr_nd24xaX8d11qkb1azo1_250.npz ... TGIF/video/Action/tumblr_no00ddSlG31t34v14o1_250.npz TGIF/video/Action/tumblr_nd24xaX8d11qkb1azo1_250.npz ...

Reference

@article{peng2022PMT,
     title={Efficient End-to-End Video-Question Answering with Pyramidal Multimodal Transformer},
     author={Peng Min, Wang Chongyang, Shi Yu, Zhou Xiang-Dong},
     journal={Proceedings of the 37th AAAI Conference on Artificial Intelligence (AAAI)},
     year={2023}}

Recommend Projects