run-lin / journeydb

JourneyDB

[Project Page] [Paper] [Code] [HuggingFace] [OpenDataLab]

Dataset Description

Summary

JourneyDB is a large-scale generated image understanding dataset that contains 4,429,295 high-resolution Midjourney images, annotated with corresponding text prompt, image caption and visual question answering.

Supported Tasks

JourneyDB supports 4 downstream tasks, i.e. Prompt Inversion, Style Retrieval, Image Caption, and Visual Question Answering. We evaluate many existing methods on these tasks and provide a comprehensive benchmark. Please see our Paper for more details.

Dataset Details

Data Collection

For each image instance, we acquire the corresponding text prompts used to generate the images with Midjourney. Furhtermore, we employ the GPT3.5 to generate the caption and VAQ groundtruth.

Data Instances

We provide several examples to show the contents of each instance of the dataset.

Data Splits

We provide detailed statistics for each split subset in the following table. We randomly split the whole dataset into roughly 20 : 1 to obtain the training and validation set. The training set contains 4,189,737 labeled images and 1,385,317 labeled prompts. The validation set contains 235,156 images and 82,093 prompts. And we additionally sample a testing set for manual filtering. The testing set contains 5,402 images and 5,171 prompts.

	Image	Prompt	Labeled Image	Labeled Prompt	Style QA	Content QA
Training Set	4,453,193	1,643,375	4,189,737	1,385,317	7,056,394	8,775,971
Validation Set	234,156	82,093	234,156	82,093	311,569	374,310
Testing Set	5,402	5,171	5,402	5,171	10,040	11,369
Total	4,692,751	1,730,639	4,429,295	1,472,581	7,378,003	9,161,650

Acquirements

Downloads

Please fill in the form to acquire the download link.

License

The JourneyDB dataset is available under the customized Terms of Usage.

Citation

@misc{pan2023journeydb,
      title={JourneyDB: A Benchmark for Generative Image Understanding}, 
      author={Junting Pan and Keqiang Sun and Yuying Ge and Hao Li and Haodong Duan and Xiaoshi Wu and Renrui Zhang and Aojun Zhou and Zipeng Qin and Yi Wang and Jifeng Dai and Yu Qiao and Hongsheng Li},
      year={2023},
      eprint={2307.00716},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

Contributions

Junting Pan*, Keqiang Sun*, Yuying Ge, Hao Li, Haodong Duan, Xiaoshi Wu, Renrui Zhang, Aojun Zhou, Zipeng Qin, Yi Wang, Jifeng Dai, Yu Qiao, Hongsheng Li⁺

(* equal contribution, ⁺ corresponding author)

Contact

If you have any problem or suggestion, please feel free to open an issue or send emails to the contributors.

TODO List

• Project Page
• Technical Report
• Release Dataset
• Release Dataset Tools
• Release Evaluation Scripts
• Release Pretrained Model
• Stable Diffusion Version