shuxjweb / uni-perceiver

Uni-Perceiver

This repository contains training (pre-training, fine-tuning, prompt-tuning), evaluation code and pretrained models for the following papers:

Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks, CVPR 2022.

Uni-Perceiver-MoE: Learning Sparse Generalist Models with Conditional MoEs.

Introduction

Uni-Perceiver is a generalist model (generic perception model) that can process a variety of modalities and tasks with unified modeling and shared parameters. Different perception tasks are modeled as the same formulation, that is, finding the maximum likelihood target for each input through the similarity of their representations. Meanwhile, Uni-Perceiver is pre-trained on several uni-modal and multi-modal tasks, and evaluated on a variety of downstream tasks, including novel tasks that did not appear in the pre-training stage. Thanks to the unified formulation, it shows the ability of zero-shot inference on novel tasks, and shows promising performance close to or on par with SOTA results by prompt tuning or finetuning.

In Uni-Perceiver-MoE, we found that the interference among different tasks and modalities can lead to performance degradation of generalist models on some tasks compared with task-specialized models. We introduce the Conditional Mixture-of-Experts (Conditional MoEs) to mitigate such interference. By incorporating the proposed Conditional MoEs, Uni-Perceiver-MoE can effectively mitigate the interference across tasks and modalities, and achieves state-of-the-art results on a series of downstream tasks via prompt tuning on 1% of downstream data. Moreover, the introduction of Conditional MoEs still holds the generalization ability of generalist models to conduct zero-shot inference on new tasks,

Main Results and Pretrained Models

Base Models

Task	Image Classification	Image Caption		Image Retrieval				Video Classification	Video Caption	Video Retrieval
Dataset	ImageNet-1k	MSCOCO	Flickr30k	MSCOCO		Flickr30k		Kinetics-400	MSVD	MSVD
Split	ILSVRC 2012 val	Karpathy test	test	Karpathy test		test		test-dev	val	val	val
Metric	Acc@1	BLEU-4	BLEU-4	R@1 i2t	R@1 t2i	R@1 i2t	R@1 t2i	Acc@1	BLEU-4	R@1 v2t	R@1 t2v
Uni-PerceiverBASE w/o Tuning	79.2	32.0	14.7	64.9	50.7	82.3	71.1	74.5	22.6	50.3	38.7
Uni-PerceiverBASE PT (1%)	80.9	35.5	30.2	68.4	51.9	91.0	76.0	74.8	59.5	62.7	43.8
Uni-PerceiverBASE FT (100%)	84.0	36.4	31.2	69.8	53.9	92.7	77.5	77.7	63.3	62.8	45.8
Uni-Perceiver-MoEBASE w/o Tuning	80.3	33.2	15.9	64.6	51.6	82.1	75.8	76.8	23.4	52.8	40.0
Uni-Perceiver-MoEBASE PT (1%)	82.0	36.8	30.7	68.9	52.6	91.3	78.5	77.2	60.0	65.6	45.3
Uni-Perceiver-MoEBASE FT (100%)	84.5	37.3	32.4	70.5	54.1	93.6	79.8	79.3	65.4	65.0	47.8

Large Models

Task	Image Classification	Image Caption		Image Retrieval				Video Classification	Video Caption	Video Retrieval
Dataset	ImageNet-1k	MSCOCO	Flickr30k	MSCOCO		Flickr30k		Kinetics-400	MSVD	MSVD
Split	ILSVRC 2012 val	Karpathy test	test	Karpathy test		test		test-dev	val	val	val
Metric	Acc@1	BLEU-4	BLEU-4	R@1 i2t	R@1 t2i	R@1 i2t	R@1 t2i	Acc@1	BLEU-4	R@1 v2t	R@1 t2v
Uni-PerceiverLARGE w/o Tuning	82.7	35.3	15.1	67.8	54.1	83.7	74.2	79.5	24.7	45.4	34.2
Uni-PerceiverLARGE PT (1%)	84.2	38.6	32.9	73.3	56.2	92.1	80.0	80.0	67.2	65.5	48.6
Uni-PerceiverLARGE FT (100%)	86.2	39.2	35.5	74.4	57.9	94.7	82.1	81.9	68.3	65.2	50.8
Uni-Perceiver-MoELARGE w/o Tuning	83.4	35.5	15.8	67.9	55.3	83.6	75.9	82.1	24.6	45.7	41.9
Uni-Perceiver-MoELARGE PT (1%)	84.9	39.3	33.7	73.3	57.1	92.4	80.6	83.0	67.6	66.4	50.3
Uni-Perceiver-MoELARGE FT (100%)	86.4	40.5	36.2	74.7	58.3	94.1	83.7	84.2	68.9	67.6	52.3

• The numbers are slightly better than the original paper of Uni-Perceiver, which are from the reproduced version of Uni-Perceiver used as the baseline of Uni-Perceiver-MoE.
• The image resolution for all tasks is 224x224.
• See OtherResults.md for results on more tasks and datasets.

Usage

Requirements

• Linux, CUDA>=10.1, GCC>=5.4

• Python >=3.7

• pytorch >= 1.8.0

• JAVA >= 1.8 (for caption task evaluation)

Installation

git clone https://github.com/fundamentalvision/Uni-Perceiver
cd Uni-Perceiver
pip install -r requirements.txt

Data

See prepare_data.md.

Pre-trained Model Weights

See checkpoints.md.

Pre-training

See pretraining.md.

Fine-tuning

See finetuning.md.

Prompt-tuning

See prompt_tuning.md.

Inference

See inference.md.

TODO

• release more pretrained models

  • Uni-Perceiver Tiny model
  • Uni-Perceiver Small model
  • Uni-Perceiver Huge model

• support more datasets and tasks

License

Uni-Perceiver is licensed under the Apache-2.0 License.

Citing Uni-Perceiver

If you find Uni-Perceiver useful in your research, please consider giving a star ⭐ and citing:

 @article{zhu2021uni,
  title={Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks},
  author={Zhu, Xizhou and Zhu, Jinguo and Li, Hao and Wu, Xiaoshi and Wang, Xiaogang and Li, Hongsheng and Wang, Xiaohua and Dai, Jifeng},
  journal={arXiv preprint arXiv:2112.01522},
  year={2021}

}

@article{zhu2022uni,
  title={Uni-Perceiver-MoE: Learning Sparse Generalist Models with Conditional MoEs},
  author={Zhu, Jinguo and Zhu, Xizhou and Wang, Wenhai and Wang, Xiaohua and Li, Hongsheng and Wang, Xiaogang and Dai, Jifeng},
  journal={arXiv preprint arXiv:2206.04674},
  year={2022}
}

Acknowledgements

Many thanks to following codes that help us a lot in building this codebase:

• Detectron2
• X-modaler
• deit
• VL-BERT
• TimeSformer
• CLIP