yaoyao-liu / tiered-imagenet-tools

Tools for generating tieredImageNet dataset and processing batches

Tools for tieredImageNet Dataset

This repo provides python source codes for creating tieredImageNet dataset from ImageNet and the utils for generating batches during training. This repo is related to our work on few-shot learning: Meta-Transfer Learning.

Summary

• About tieredImageNet
• Requirements
• Usage
• Performance
• Acknowledgement

About tieredImageNet

The tieredImageNet dataset is a larger subset of ILSVRC-12 with 608 classes (779,165 images) grouped into 34 higher-level nodes in the ImageNet human-curated hierarchy. This set of nodes is partitioned into 20, 6, and 8 disjoint sets of training, validation, and testing nodes, and the corresponding classes form the respective meta-sets. As argued in Ren et al. (2018), this split near the root of the ImageNet hierarchy results in a more challenging, yet realistic regime with test classes that are less similar to training classes.

Requirements

• Python 2.7 or 3.x
• numpy
• tqdm
• opencv-python
• Pillow

Usage

First, you need to download the image source files from ImageNet website. If you already have it, you may use it directly.

Filename: ILSVRC2012_img_train.tar
Size: 138 GB
MD5: 1d675b47d978889d74fa0da5fadfb00e

Then clone the repo:

git clone https://github.com:y2l/tiered-imagenet-tools.git
cd tiered-imagenet-tools

To generate tieredImageNet dataset from tar file:

python tiered_imagenet_generator.py --tar_dir [your_path_of_the_ILSVRC2012_img_train.tar]

To generate tieredImageNet dataset from untarred folder:

python tiered_imagenet_generator.py --imagenet_dir [your_path_of_imagenet_folder]

If you want to resize the images to the specified resolution:

python tiered_imagenet_generator.py --tar_dir [your_path_of_the_ILSVRC2012_img_train.tar] --image_resize 100

P.S. In default settings, the images will be resized to 84 × 84.

If you don't want to resize the images, you may set --image_resize 0.

To use the TieredImageNetDataLoader class:

from tiered_imagenet_dataloader import TieredImageNetDataLoader

dataloader = TieredImageNetDataLoader(shot_num=5, way_num=5, episode_test_sample_num=15)

dataloader.generate_data_list(phase='train')
dataloader.generate_data_list(phase='val')
dataloader.generate_data_list(phase='test')

dataloader.load_list(phase='all')

for idx in range(total_train_step):
    episode_train_img, episode_train_label, episode_test_img, episode_test_label = \
        dataloader.get_batch(phase='train', idx=idx)
    ...

Acknowledgement

This repo uses the source code from the following repos:

Model-Agnostic Meta-Learning

Optimization as a Model for Few-Shot Learning

tiered-imagenet-tools's Issues

Loader bug

Could you confirm that in the following line, self.shot_num should be self.way_num？It is not consistent with mini-imagenet-tools and gives error.
https://github.com/yaoyao-liu/tiered-imagenet-tools/blob/master/tiered_imagenet_dataloader.py#L189

I ran the mini_imagenet_generator.py and only the floder tiered_imagenet generated

Thank you for sharing such great work. I'm new to it and it helps me a lot. I ran the tiered_imagenet_generator.py and only the floder tiered_imagenet generated. This folder contains trian, val, and test. Further, the processed_images directory is not created. Can I sample images directly from each of the three folders in tiered_imagenet directory to generate the NwayKshot task for train,val,test ? In my code, the MiniImagenet dataset also takes this form.

category and class

can you explain to me the correspondence between the folder and the calss category, I read the content under the tiered_imagenet_split folder, but I don’t quite understand

Looks like your code doesn't extract the sub tars.

Hi, thanks for sharing.
But I meet a problem when using your code.
When I run the following command: python tiered_imagenet_generator.py --tar_dir [your_path_of_the_ILSVRC2012_img_train.tar], I got many lines of 0it [00:00, ?it/s]. After checking your code, I think it does not extract the sub tars automatically:

I tested it on both Windows and Linux (Ubuntu 20.04).
In the end, I think you should mention that users should manually extract the sub tars in the repo's readme.

Thanks.

Which split file should we use to match the exact split proposed by Ren et.al 2018

I did some experiments, it seems that the dataset provided by https://github.com/renmengye/few-shot-ssl-public might be different from the split provided this repository, which could give miss leading results. Can the auther check which split file should we use to generate exactly the same split as Ren et.al 2018, thanks!

Does not process images

Running the following command:
python tiered_imagenet_generator.py --tar_dir [your_path_of_the_ILSVRC2012_img_train.tar]

creates the directory structure: tiered-imagenet/train/n**/ but no files.
Further, the processed_images directory is not created.

I tried the same command with the mini-imagenet-tools repo and it works correctly, so I'm assuming that my tar-file is fine.