haohanwang / imagenet-sketch

ImageNet-Sketch data set for evaluating model's ability in learning (out-of-domain) semantics at ImageNet scale

ImageNet-Sketch

Description

ImageNet-Sketch data set consists of 50000 images, 50 images for each of the 1000 ImageNet classes. We construct the data set with Google Image queries "sketch of __", where __ is the standard class name. We only search within the "black and white" color scheme. We initially query 100 images for every class, and then manually clean the pulled images by deleting the irrelevant images and images that are for similar but different classes. For some classes, there are less than 50 images after manually cleaning, and then we augment the data set by flipping and rotating the images.

This github repository consists of the scripts we used to conduct query and clean images.

Download the Data

• Hugging Face dataset
  • one can use the data with

    from datasets import load_dataset
    dataset = load_dataset("imagenet_sketch")  
    

  • Thanks to Nathan Raw for setting up
• Links
  • from Google Drive
  • from Kaggle
• Information
  • zip file is 7.8 GB
  • extracted files will be 8.4 GB

ImageNet-Sketch Leaderboard

Method	Reference	Backbone	From Scratch*	Top1-Acc.	Top5-Acc.
Texture Debiased Augmentation	Hermann et al. (NeurIPS 2020)	ResNet50	Y	30.9%	51.4%
Anisotropic diffusion	Mishra et al.	ResNet50	Y	24.49%	41.81%
Random Convolutions	Xu et al.	AlexNet	Y	18.09%	35.40%
RSC	Huang et al. (ECCV 2020)	AlexNet	Y	16.12%	30.78%
PAR	Wang et al. (NeurIPS 2019)	AlexNet	N	13.06%	26.27%
AlexNet Baseline		AlexNet	N/A	12.04%	24.80%

*This column indicates whether the model is trained from the scratch or built and fine-tuned based a pretrained backbone model.

Analysis

• An interesting analysis is performed at Kaggle, evaluating a large number of CNNs with our data set
  • Can pretrained ImageNet models generalize to sketches
• A comprehensive evaluation of popular vision models on ImageNet-Sketch
  • Pytorch Image Models' performances on ImageNet Sketch (300+ models)

Reference

The data set is introduced together with the following paper in NeurIPS 2019, so if you find this data set helpful, please consider citing it:

Learning Robust Global Representations by Penalizing Local Predictive Power

@inproceedings{wang2019learning,
        title={Learning Robust Global Representations by Penalizing Local Predictive Power},
        author={Wang, Haohan and Ge, Songwei and Lipton, Zachary and Xing, Eric P},
        booktitle={Advances in Neural Information Processing Systems},
        pages={10506--10518},
        year={2019}
}

imagenet-sketch's Issues

Direct test of ResNet50 with pytorch official released weights can achieve ~24% top1 accuracy

Has anyone got similar results?

Missing License

Can you please add a licensing to it.

Add dataset to Hugging Face Hub

Hey there, I'd really like to add this dataset to the Hugging Face datasets package. Unfortunately, Google Drive links have rate limits, so we would rather the data is coming from somewhere else.

I've unofficially uploaded the dataset to the Hugging Face Hub here for now so I can include it in my PR to add this to datasets. What would be great is if you did the same under your own account so we could make it a bit more official 🤗 . I can even actually transfer my repo's ownership to you, so all I'd need is your account name.

If you have any issue with us uploading the data to the Hub, please let me know too and I'll take it down!

some images are simply black

Thanks for providing this nice dataset!

I just wanted to point out that some images are simply black and don't show any sketch at all. For instance, in category n03085013, the images sketch_3.JPEG and sketch_33.JPEG are entirely black. This also occurs for other categories.

Number of images

The description says that there are 50000 images. However, it seems like there are 50889. Some synsets contain sketches numbered 0-50 whereas others contain sketches numbered 0-49.