merhs / sasa-pytorch

Unofficial implementation of Stand-Alone Self-Attention in Vision Models (obsolete)

PyTorch Implementation of Stand-Alone Self-Attention in Vision Models

This is NOT an official implementation. Please let me know whether this implementation contains any misreadings of the original paper.

Prerequisites

• Python +3.6
• pytorch +1.1.0
• scipy
• Pillow
• torchvision

Benchmark (WIP)

Trained with ImageNet. (WIP: CIFAR-10, CIFAR-100)

Backbone network and parameters are based on the official torchvision ResNet and trainer example.

Trained up to 90 epochs / batch 64 on a single NVIDIA 1080Ti GPU, with SGD optimizer with a learning rate of 0.1 which is linearly warmed up for 10 epochs followed by cosine decay. (according to the SASA paper)

sasa-pytorch's Issues

Excessive GPU Consumption

probably matmul or view something...

should try einsum

Why is this marked obselete?

Thanks so much for writing this implementation @MerHS. Why is it marked obsolete in the description?

How should we implement Attention stem?

currently Full-Attention network uses 2x more GPU & time.

The resulting p^m_ab are shared across the 4 attention heads for the mixture stem layer.

I cannot understand this line. ResNet50 uses Conv2d(3, 64, (7, 7)) as stem layer, so it cannot use 4 seperable convolution. (3 != 0 (mod 4))

Also, what exact value is the mixture count m?

Are there any results for this code?

Anybody please let me know if there are any results for IMAGENET, CIFAR10 or CIFAR100

Sum is over both channels and groups

Hi, according to the paper last sentence on pg 3 I think the sum in row 82 should only be over fc/g rather than all of fc. That should give size b,g,fh,fw,kh,kw in row 84.

As it currently stands, I think you effectively use groups=1 and the spatial kernel is the same for all channels. When increasing the number of groups, I think the kernel should vary between some channels (the groups).

SASA-pytorch/model/sa_layer.py

Line 82 in 7d11385

vx = (win_q.unsqueeze(4).unsqueeze(4) * win_k).sum(dim=1) # b, fh, fw, kh, kw