ziplab / longvlm

LongVLM: Efficient Long Video Understanding via Large Language Models

[ECCV 2024] This is the official repository for our paper: LongVLM: Efficient Long Video Understanding via Large Language Models by Yuetian Weng, Mingfei Han, Haoyu He, Xiaojun Chang and Bohan Zhuang.

Architecture

Installation

conda create -n longvlm python==3.9
pip install torch==1.10.1+cu111 torchvision==0.11.2+cu111 torchaudio==0.10.1
pip install -r requirements.txt
export PYTHONPATH="./:$PYTHONPATH"

### For Training, install flash-attention
pip install ninja
git clone https://github.com/HazyResearch/flash-attention.git
cd flash-attention
git checkout v1.0.7
python setup.py install

Pretrained model

Download pretrained base model and delta weights from Huggingface.

Apply delta weights to the base model:

python scripts/apply_delta.py \
    --base YOUR_PATH_TO_LLAMA_7B \
    --target YOUR_PATH_TO_LLAVA_7B \
    --delta YOUR_PATH_TO_LLAVA_DELTA_7B

Data preparation

Download ActivityNet dataset. Dataset architecture:

datasets
|___anet
|   |___v1-2/
|   |   |___train/
|   |   |   |___xxxx.mp4
|   |   |   |___xxxx.mp4
|   |   |___val/
|   |   |   |___xxxx.mp4
|   |   |   |___xxxx.mp4
|   |___v1-3/
|   |   |___train_val/
|   |   |   |___xxxx.mp4
|   |   |   |___xxxx.mp4
|   |___anet_benchmark_video_id.txt
|   |___video_list_v1_2_val.txt
|   |___video_list_v1_2_train.txt
|   |___video_list_v1_3_trainval.txt

Extract visual features using CLIP-VIT/L-14:

python scripts/save_features.py \
    --video_dir_path YOUR_PATH_TO_VIDEO_FOLDER \
    --list_file YOUR_PATH_TO_VIDEO_ID_FILE \
    --clip_feat_path_local YOUR_PATH_TO_SAVE_VISUAL_FEATURE_L \
    --clip_feat_path_mem YOUR_PATH_TO_SAVE_VISUAL_FEATURE_G

Train and Evaluation

sh run.sh

Examples

Acknowledgement

This work is built upon VideoChatGPT. Thanks to their awesome work.

longvlm's Issues

Unable to Reproduce videochatgpt Benchmark Results

Hello,

Thank you for your open-source contribution. I have trained a model using the code you provided. However, I am seeing different results on the videochatgpt benchmark compared to what is reported in the paper. My scores across the five metrics are 2.72, 2.47, 3.11, 2.29, and 2.71, with an average of 2.66, which is different from the reported average of 2.89.
Considering that different versions of ChatGPT might affect the outcomes, could you please provide a pretrained model for testing? This would help verify the results. Thank you for your assistance.

dependency error

When i run pip install -r requirements.txt, error occurs as blow:
ERROR: Cannot install -r requirements.txt (line 205) and nvidia-nccl-cu12==2.18.1 because these package versions have conflicting dependencies.

The conflict is caused by:
The user requested nvidia-nccl-cu12==2.18.1
torch 2.2.2 depends on nvidia-nccl-cu12==2.19.3; platform_system == "Linux" and platform_machine == "x86_64"

To fix this you could try to:

1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

Could you please help me?

Thanks!

When will the LongVLM code be released?

LongVLM seems like an amazing leap forward in VLM space. And I am looking forward to the release of the video model!

question about evaluation and chatgpt version?

Thank you for the open source. May I ask when your experiment took place? Since different versions of ChatGPT can lead to different evaluation results, how did you compare it with Video-ChatGPT?

Coder release

Hi~ Thanks for the amazing work! Could I know when the code will be released?