biubiuisacat / video-mamba-suite

Video Mamba Suite: State Space Model as a Versatile Alternative for Video Understanding

Guo Chen, Yifei Huang, Jilan Xu, Baoqi Pei, Zhe Chen, Zhiqi Li, Jihao Wang, Kunchang Li, Tong Lu and Limin Wang.

Introduction

Understanding videos is one of the fundamental directions in computer vision research, with extensive efforts dedicated to exploring various architectures such as RNN, 3D CNN, and Transformers. The newly proposed architecture of state space model, e.g, Mamba, shows promising traits to extend its success in long sequence modeling to video modeling. To assess whether Mamba can be a viable alternative to Transformers in the video understanding domain, in this work, we conduct a comprehensive set of studies, probing different roles Mamba can play in modeling videos, while investigating diverse tasks where Mamba could exhibit superiority. We categorize Mamba into four roles for modeling videos, deriving a Video Mamba Suite composed of 14 models/modules, and evaluating them on 12 video understanding tasks. Our extensive experiments reveal the strong potential of Mamba on both video-only and video-language tasks while showing promising efficiency-performance trade-offs. We hope this work could provide valuable data points and insights for future research on video understanding.

📢 NOTICE

The code is being sorted out and will be released gradually in the next few days.

📢 News

(2024/03/15) 🔄The repository is public.

(2024/03/12) 🔄The repository is created.

Preliminary Installation

1. Install the preliminary requirements.

# create environment
conda create -n video-mamba-suite python=3.9
pip install torch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 --index-url https://download.pytorch.org/whl/cu118

# install requirements
conda activate video-mamba-suite
pip install requirement.txt

# install mamba
cd causal-conv1d
python setup.py develop
cd ..
cd mamba
python setup.py develop
cd ..

2. For each tasks, enter their folders to follow installation instructions.

3. If requirement.txt is missing some libraries, please propose an issue as soon as possible.

Mamba for Video Temporal Modeling

Supported tasks:

Temporal Action Localization

Temporal Action Segmentation

Video dense captioning

Video paragraph captioning

Action Anticipation

Mamba for Cross-modal Interaction

Supported tasks:

Video Temporal Grounding

Highlight Detection

Mamba as Video Temporal Adapter

Supported tasks:

Multi-instance Retrieval

Action Recognition

Long-form Video Question-Answer

Mamba for Spatial-temporal Modeling

Supported tasks:

Multi-instance Retrieval

Action Recognition

Long-form Video Question-Answer

Related dataset resources:

THUMOS-14	ActivityNet	HACS Segment	FineAction	GTEA
YouCook2	Breakfast	FineAction	Epic-kitchen-100	Ego4D
EgoSchema	QvHighlight	Charade-STA

Cite

If you find this repository useful, please use the following BibTeX entry for citation.

@misc{2024videomambasuite,
      title={Video Mamba Suite: State Space Model as a Versatile Alternative for Video Understanding}, 
      author={Guo Chen, Yifei Huang, Jilan Xu, Baoqi Pei, Zhe Chen, Zhiqi Li, Jiahao Wang, Kunchang Li, Tong Lu, Limin Wang},
      year={2024},
      eprint={2403.09626},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

License

This project is released under the MIT License

Acknowledgement

This repository is built based on ActionFormer, UniVTG, ASFormer, PDVC, Testra, MAT, AVION, InternVideo, EgoSchema, ViM and Mamba repository.