eambutu / snail-pytorch

Implementation of "A Simple Neural Attentive Meta-Learner" (SNAIL, https://arxiv.org/pdf/1707.03141.pdf) in PyTorch

A Simple Neural Attentive Meta-Learner (SNAIL) in PyTorch

An implementation of Simple Neural Attentive Meta-Learner (SNAIL) (paper) in PyTorch.

Much of the boiler plate code for setting up datasets and what not came from a PyTorch implementation of Prototypical Networks.

Mini-Imagenet Dataset

Follow the instructions here: https://github.com/renmengye/few-shot-ssl-public to download the mini-imagenet dataset.

Performance

Below are the following attempts to reproduce the results in the reference paper:

Omniglot:

Model	1-shot (5-way Acc.)	5-shot (5-way Acc.)	1 -shot (20-way Acc.)	5-shot (20-way Acc.)
Reference Paper	99.07%	99.78%	97.64%	99.36%
This repo	98.31%*	99.26%**	93.75%°	97.88%°°

* achieved running python train.py --exp omniglot_5way_1shot --cuda

* achieved running python train.py --exp omniglot_5way_5shot --num_samples 5 --cuda

* achieved running python train.py --exp omniglot_20way_1shot --num_cls 20 --cuda

* achieved running python train.py --exp omniglot_20way_5shot --num_cls 20 --num_samples 5 --cuda

Mini-Imagenet:

In progress. Writing the code for the experiments should be done soon but the main bottleneck in these experiments for me is compute, if someone would be willing to run and report numbers that would be much appreciated.

RL:

In progress.

snail-pytorch's Issues

TC blocks calculated in a wrong way?

I guess the code in the file blocks.py

int(math.ceil(math.log(seq_length)))

is wrong? Because math.log(x) return the natural logarithm of x, and I suppose we want a base-2 logarithm of x here?

random uniform choosing of k in k-shot in training

In the original paper, "the number of shots K was chosen uniformly at random from 1 to 5." So the forward function of SNAIL should take an arbitrary K instead of defined at init, and training should follow the uniform random choosing as well.

Thanks!

Some discrepancies from the paper

For the Mini-ImageNet ResNet-based encoder, I believe the authors use mean pooling instead of max pooling, as done by this implementation. I am currently unable to reproduce the mini-imagenet results with your codebase (even train accuracy does not go up), and am hunting for discrepancies :)

rl version

Hi
have you finished the RL part? thanks in advance!!