shi-labs / compact-transformers Goto Github PK

if we have set of image folders how to train on this type of datasets ?

Dear @stevenwalton,

I am trying to find the code block of Algorithm 1 CCT ConvBlock(PyTorch-style).
Is this code available on paper?

Regards,
Khawar

Any plans on releasing the weights for the baseline ViT on imagenet?

How many image reductions should I have?
Is this right?

model = cct_7(img_size=im_size,
              num_classes=classes,
              positional_embedding='learnable',
              n_conv_layers=2,
              kernel_size=7,
              stride=2,
              padding=3,
              pooling_kernel_size=3,
              pooling_stride=2,
              pooling_padding=1)

look at the Table1 in paper：

I understand the CCT model is more parameters and MAC than ResNet164 but with same Acc

I am working on applying cct_7 for another imgsize=128 example but my main problem is the CUDA Memory usage. Do you have any intuitions on parameters I could modify to reduce CUDA memory usage? Thanks!

@alihassanijr @stevenwalton @honghuis @Walleclipse @itsShnik
Hi there,

do you have any suggestions and thoughts regarding reducing the computational cost of DINO.
can dino benefit from cct.

Demo of Dino

Thanks for this interesting work! I am just wondering if you can provide the command line for training the cifar-100 dataset?

Hi,

Thank you for the wonderful paper.
I have trained the CCT_7/3x1 by default and however, the result is differ from the paper.
CIFAR10: 93.67
CIFAR100: 73.15
I've tried it many times and It never reaches 94.72% for CIFAR10 and 76.67% for CIFAR100.
Could you please help me what makes it differ?

Thanks.

Hi, I was wondering if I could use your finetuned model on my custom data! I already managed to run the Keras tutorial on my data, but I started from scratch! Is is possible to fine tune it? should I simply load your pretrained files?

Thank you in advance

Hi,

i am a little confuse about the output of the CCT. If I have a classification task with n possible classes, are the outputs the logits of each class? Thus to find the respective probabilities i have to apply a softmax function, or are the outputs the probabilities?

Thanks in advance

Hello I have made a different file custom_vit.py under src folder and I have created my model.
However I am getting the following error

Traceback (most recent call last):
File "/home/iliask/PycharmProjects/Compact-Transformers/train.py", line 806, in
main()
File "/home/iliask/PycharmProjects/Compact-Transformers/train.py", line 357, in main
model = create_model(
File "/home/iliask/miniconda3/envs/SLVTP/lib/python3.9/site-packages/timm/models/factory.py", line 79, in create_model
raise RuntimeError('Unknown model (%s)' % model_name)
RuntimeError: Unknown model (custom_vit_2_4_32)

Hi, this work is awesome.
I just have one little question. The paper says the total batch size is 128 for CIFAR's and 4 GPU's were used in parallel.
That doesn't mean the total batch size is 128 * 4 = 512, does it?
DDP is for Imagenet, and non-distributed is for CIFAR, am I correct?

Thanks a ton :)

Hi,

if i want to use the model cct_14_7x2_224 pretrained on the Imagenet dataset, do i have to right normalize the images using the mean and standard dev of imagenet dataset, respectively (0.485, 0.456, 0.406) (0.229, 0.224, 0.225). Rigth? Before this normalization, do i have to make the rescaling of the pixel values from [0:255] to [0:1] ?

Thanks

Hi @stevenwalton

What is the difference between your "TransformerEncoderLayer" and original Vit paper "Transformer" class?

Orginal VIT

class Transformer(nn.Module):
    def __init__(self, dim, depth, heads, dim_head, mlp_dim, dropout):
        super().__init__()
        self.layers = nn.ModuleList([])  # There are using Residual
        for _ in range(depth):
            self.layers.append(nn.ModuleList([
                Residual(PreNorm(dim, Attention(dim, heads=heads, dim_head=dim_head, dropout=dropout))),
                # Here they implemented Residual
                Residual(PreNorm(dim, FeedForward(dim, mlp_dim, dropout=dropout)))
            ]))

    def forward(self, x, mask=None):
        for attn, ff in self.layers:
            x = attn(x, mask=mask)  # Chnage in this part
            # embed()
            x = ff(x)
        return x

Your Transformer

class TransformerEncoderLayer(nn.Module):
    """
    Inspired by torch.nn.TransformerEncoderLayer and
    rwightman's timm package.
    """

    def __init__(self, d_model, nhead, dim_feedforward=2048, dropout=0.1,
                 attention_dropout=0.1, drop_path_rate=0.1):
        super(TransformerEncoderLayer, self).__init__()
        self.pre_norm = nn.LayerNorm(d_model)
        self.self_attn = Attention(dim=d_model, num_heads=nhead,
                                   attention_dropout=attention_dropout, projection_dropout=dropout)

        self.linear1 = nn.Linear(d_model, dim_feedforward)
        self.dropout1 = nn.Dropout(dropout)
        self.norm1 = nn.LayerNorm(d_model)
        self.linear2 = nn.Linear(dim_feedforward, d_model)
        self.dropout2 = nn.Dropout(dropout)

        self.drop_path = DropPath(drop_path_rate) if drop_path_rate > 0 else nn.Identity()

        self.activation = F.gelu

    def forward(self, src: torch.Tensor, mask=None, *args, **kwargs) -> torch.Tensor:
        src = src + self.drop_path(self.self_attn(self.pre_norm(src)))
        src = self.norm1(src)
        src2 = self.linear2(self.dropout1(self.activation(self.linear1(src))))
        src = src + self.drop_path(self.dropout2(src2))
        return src

Actually, I made some modifications in the original VIT and I want to add that modified part in your Transformer Encoder layer but your's code is different in terms of TransformerEncoderLayer.

Hi, It's a nice work. But I still feel little bit confused about some points.
(1) To make the ViT still effective for small data, CCT and CVT modify the tokenization part. And CVT applies patching while CCT uses the convolution. Is my understanding correct?
(2) In the original Vit paper, they use Hybrid Vit to do the investigation and comparison. So, your method still belongs to Hybrid Vit and CCT is more similar to Hybrid Vit, right? Did you compare yours with the original Hybrid Vit? It seems that I didn't see it in your paper. BTW, I just want to make sure that the convolution (ResNet) of the original Hybrid Vit is also training along with Vit ,right?

Thanks for help!

Hi, can you share the HP that your team trained it on RandAug with Timm? I have followed the same steps but still stuck on 76.9% and couldn't go any further (trained for 300 epochs should reach approximately 80%). Thanks for your help.

Hi!
I started training your model on my data, and after some reserach I did not manage to understand how the validation set should be made in terms of directories! I know it's timm library, but I was wondering if you could help me.

My training data il like this: -/train/0/class1/ .....jpeg
class2/ .....
class3/....
class4/....

How and where should I position my validation folder?

Dear @stevenwalton,

I would like to use the configuration of the best model directly from the CCT Transformer class not calling the model function. I want to directly use the CCT Transformer class like VIT. Like below

https://github.com/lucidrains/vit-pytorch Please check the VIT model

v = ViT(
    image_size = 256,
    patch_size = 32,
    num_classes = 1000,
    dim = 1024,
    depth = 6,
    heads = 16,
    mlp_dim = 2048,
    dropout = 0.1,
    emb_dropout = 0.1
)

For CCT

"CCT": CCT (
            # GPU_ID=GPU_ID,
            # img_size=112,
            # num_classes=NUM_CLASS,
            positional_embedding='positional_embedding',
            n_conv_layers=2,
            kernel_size=3,
            patch_size=8
        ),

Hi, Steven Walton
Thank you for sharing your amazing work!

If feasible, I would like to ask a question.
In the paper I found that "ViT, by default, uses a dropout for the MLP heads only with a probability of 0.1, and no attention dropout.Conversely, we do not use MLP dropout and only use attention dropout (p = 0.1)." However, in utils/transformers.py I found:

src_temp = self.activation(self.linear1(src))
src2 = self.linear2(self.dropout1(src_temp))
src = src + self.drop_path(self.dropout2(src2))

I wonder whether my understanding is wrong. I’d really appreciate it if you can let me know.

Hello, I observed that your code loads image files from folders that are split in train/test. Did you convert the original files of CIFAR10/100 into jpg files ? I use this repo https://github.com/knjcode/cifar2png for the datasets.
Thanks in advance

This is very exciting! Thank you! I'm interested in exploring this with NLP. Unfortunately, I'm running into some issues that seem to be related to expected tensor sizes when running one of your models. Do you have any extra NLP-related materials/training scripts that you can share?

Thank you for sharing this amazing work. I am currently attempting to apply your ideas to a specific problem with bigger images sized 128x128. Do you have any recommendations on how to improve the performances of your network on bigger images?

I saw the results on the ImageNet dataset have been published. Is this result pre-trained with additional datasets? What are the training parameters? Epoch, batch size?

Hi, I've read your excellent paper a few times now but I'm struggling on the intuition on why the sequence pooling approach should be substantially different or superior to a standard class token?

We've known for some time that the class token representations do not give good embedding representations, which is why it's common to pool the embedding representations through mean pooling - such as in Sentence Transformers.

But the use of another set of weights to compute attention weights for creating a pooled representation doesn't sound that different to what a class token should be doing.

A class token should in theory be able to pool representations from the previous layer.

What's the intuition, and what is different in the math or the architecture that would make this a more effective architecture, and lead to much efficient model sizes?

Been scratching my head about it for a number of days and thought I'd ask?

Thanks again for this great paper and library, and for enabling powerful transformers with a lot less parameters!

Hi, thank you for this very clean open-source implementation!

I've been testing out some modifications and noticed, even without the modifications, I'm not quite achieving the same accuracies as you report on CIFAR-10. I wondered if you had any suggestions about things that I might have missed.

Specifically, I tried to reproduce your results using one of your configs with the command python train.py -c configs/pretrained/cct_7-3x1_cifar10_300epochs.yml --model cct_7_3x1_32 datasets/CIFAR-10-images/ --log-wandb. I copied the dataset from this github repo. I couldn't find details on whether you use a train/validation split so then trained on all 50000 training images (i.e. with no validation set) and tested on all 10000 test images. I used your validate function for computing the test accuracy (by renaming the test image folder so that it is loaded as a validation set). After the full 300 epochs, I obtained 93.21% test accuracy, rather than the 96.53% that I think you report for this config in the README. Please let me know if there's anything I should do differently to obtain these results - perhaps using a different train/test split, computing the test loss in a different way, turning on EMA averaging, or if there's anything else that I might have missed.

I also tried computing test statistics in the same way after loading your pretrained cct_7_3x1_32 checkpoint (instead of training it) and got a lower test accuracy of 91.67%. So this makes me think that the issue is likely related to testing rather than training.

Thanks!

Firstly, Thank for your great work.
I trained your CCT6 on google colab and the result is

[Epoch 200][Train][100] 	 Loss: 5.9658e-01 	 Top-1  95.77
[Epoch 200][Train][110] 	 Loss: 5.9784e-01 	 Top-1  95.70
[Epoch 200][Train][120] 	 Loss: 5.9848e-01 	 Top-1  95.67
[Epoch 200][Train][130] 	 Loss: 5.9910e-01 	 Top-1  95.61
[Epoch 200][Train][140] 	 Loss: 6.0073e-01 	 Top-1  95.55
[Epoch 200][Train][150] 	 Loss: 6.0226e-01 	 Top-1  95.46
[Epoch 200][Train][160] 	 Loss: 6.0283e-01 	 Top-1  95.46
[Epoch 200][Train][170] 	 Loss: 6.0209e-01 	 Top-1  95.50
[Epoch 200][Train][180] 	 Loss: 6.0230e-01 	 Top-1  95.50
[Epoch 200][Train][190] 	 Loss: 6.0174e-01 	 Top-1  95.52
[Epoch 200][Train][200] 	 Loss: 6.0133e-01 	 Top-1  95.53
[Epoch 200][Train][210] 	 Loss: 6.0177e-01 	 Top-1  95.50
[Epoch 200][Train][220] 	 Loss: 6.0121e-01 	 Top-1  95.51
[Epoch 200][Train][230] 	 Loss: 6.0156e-01 	 Top-1  95.49
[Epoch 200][Train][240] 	 Loss: 6.0092e-01 	 Top-1  95.52
[Epoch 200][Train][250] 	 Loss: 6.0064e-01 	 Top-1  95.54
[Epoch 200][Train][260] 	 Loss: 6.0068e-01 	 Top-1  95.54
[Epoch 200][Train][270] 	 Loss: 6.0083e-01 	 Top-1  95.52
[Epoch 200][Train][280] 	 Loss: 6.0039e-01 	 Top-1  95.54
[Epoch 200][Train][290] 	 Loss: 6.0092e-01 	 Top-1  95.52
[Epoch 200][Train][300] 	 Loss: 6.0109e-01 	 Top-1  95.52
[Epoch 200][Train][310] 	 Loss: 6.0116e-01 	 Top-1  95.53
[Epoch 200][Train][320] 	 Loss: 6.0130e-01 	 Top-1  95.52
[Epoch 200][Train][330] 	 Loss: 6.0153e-01 	 Top-1  95.51
[Epoch 200][Train][340] 	 Loss: 6.0245e-01 	 Top-1  95.46
[Epoch 200][Train][350] 	 Loss: 6.0211e-01 	 Top-1  95.47
[Epoch 200][Train][360] 	 Loss: 6.0238e-01 	 Top-1  95.47
[Epoch 200][Train][370] 	 Loss: 6.0208e-01 	 Top-1  95.49
[Epoch 200][Train][380] 	 Loss: 6.0174e-01 	 Top-1  95.51
[Epoch 200][Train][390] 	 Loss: 6.0167e-01 	 Top-1  95.51
[Epoch 200][Eval][0] 	 Loss: 6.7492e-01 	 Top-1  93.75
[Epoch 200][Eval][10] 	 Loss: 6.7268e-01 	 Top-1  93.39
[Epoch 200][Eval][20] 	 Loss: 6.7867e-01 	 Top-1  93.42
[Epoch 200][Eval][30] 	 Loss: 6.7773e-01 	 Top-1  93.42
[Epoch 200][Eval][40] 	 Loss: 6.8292e-01 	 Top-1  93.43
[Epoch 200][Eval][50] 	 Loss: 6.7983e-01 	 Top-1  93.41
[Epoch 200][Eval][60] 	 Loss: 6.8163e-01 	 Top-1  93.37
[Epoch 200][Eval][70] 	 Loss: 6.7865e-01 	 Top-1  93.43
[Epoch 200] 	 	 Top-1  93.44 	 	 Time: 149.51
Script finished in 149.51 minutes, best top-1: 93.48, final top-1: 93.44

The accuracy on training set is always higher than on validation set. The model is saturated at about 91% accuracy on validation set. Is it acceptable? or the model losses its generalization or something overfitting.

First of all, thanks for your amazing work!

And it seems that your TransformerEncoderLayer implementation is a bit different from the 'mainstream' implementations, because you create your residual link after the LayerNorm procedure:

However, from the original paper of ViT and many other implementations, the residual link is created before the LayerNorm procedure:

src = src + self.drop_path(self.self_attn(self.pre_norm(src)))
src2 = self.norm1(src)
src2 = self.linear2(self.dropout1(self.activation(self.linear1(src2))))
src = src + self.drop_path(self.dropout2(src2))

I'm just wondering whether this is on purpose or some kind of 'typo'? Thanks in advance!

Hi, sorry to trouble you again. I have successfully your results on cifar10, and I plan to reproduce the results on ImageNet.
I follow your configs for imagenet and my result is lower than yours about 2%.

I would like to know the interpolation's type used in your paper for imagenet.
As most paper's only use Bicubic, and your paper use "random" for imagenet.

Thanks very much!

Great work
How can I use custom dataset on your model.

Hello. First, thanks for sharing your impressive work.

Since I want to train CCT models for CIFAR-100 for reproduction,
I'd like to ask you about how to split the dataset for train/val/test.

Thanks.

Hi,

I'm trying to figure out how to train your model to achieve the SOTA accuracy you report on Flowers102. It seems like using finetuned/cct_14-7x2_flowers102.yml will download the model with the 99.76% test accuracy you report, but I can't find any config files which actually train this model from scratch (or from e.g. an ImageNet checkpoint if you use that). Do you mind pointing me to any config files for this that I might have missed, or else to a description of the training procedure for your SOTA Flowers102 model so that I can try to reproduce it?

Thanks for your help,
Will

Hi, Firstly, Thank for your great work.

I am trying to use pretrained model on flowers-102 and the accuracy I could achieve is around 8%. Could you provide some details for pretrained model on flowers102? Btw, other pretrained model is working in my code. My setting for flowers102

initialization of net

elif 'cct_7' in arch:
    model = getattr(models, arch)(img_size=224,
                                    num_classes=num_classes,
                                    positional_embedding='sine',
                                    n_conv_layers=2,
                                    kernel_size=7) 

elif 'cct_14' in arch:
    model = getattr(models, arch)(img_size=384,
                                    num_classes=num_classes,
                                    positional_embedding='learnable',
                                    n_conv_layers=2,
                                    kernel_size=7)

dataloader

elif args.data == 'flowers102':
  traindir = os.path.join(args.data_root, 'flower_data/train')
  valdir = os.path.join(args.data_root, 'flower_data/valid')
  testdir = os.path.join(args.data_root, 'flower_data/test')
  normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                   std=[0.229, 0.224, 0.225])

  train_set = datasets.ImageFolder(traindir, transforms.Compose([
      transforms.RandomResizedCrop(224),
      transforms.RandomHorizontalFlip(),
      transforms.ToTensor(),
      normalize
  ]))
  
  val_set = datasets.ImageFolder(valdir, transforms.Compose([
      transforms.Resize(256),
      transforms.CenterCrop(224),
      transforms.ToTensor(),
      normalize
  ]))

Thank you

Hello, thanks very much for your work。
can you provide how to calculate the MAC of the computer vision models? e.g., CCT, VIT,..

In lines 188, 223 and 266 of "cct.py", the”stochastic_depth“ should be "stochastic_depth_rate". So is the stochastic_depth_rate always 0.1?

Looks like it got changed over in the Version 1.1 commit

Thank you for your dedication.
You mentioned that changing images sizes will affect the sequence length. Can CCT take inputs with various image sizes? For example, if I've trained CCT with image_size = 256x256. Can I test this CCT on images with size 512x512?

Hi,
first of all thanks for your work. I would like to use the pertained model cct_14_7x2_224, however it seems unavailable right now. Could you please put it back online?
Thank you

Hi, thank you for sharing your code. I have found an unclear issue with CVT. In your paper, you say that CVT only uses a linear layer as a projection layer and there is no Convolution, but it is inconsistent in your code. Indeed, CVT and VIT-lite used a conv block (Conv2D-ReLU-MaxPool2D) before applying these maps into Transformer's encoders.

Can you explain it little more?

Hi @alihassanijr,

Many thanks for your super interesting work, and sharing the elegant code with the community.

I am able to replicate your CIFAR-10 and CIFAR-100 results perfectly. But, there is a large gap when it comes to the Flowers dataset.

After running the following command:

python train.py -c configs/datasets/flowers102.yml --model cct_7_7x2_224_sine ./data/flowers102 --log-wandb

I am able to get only 62% accuracy. Please find the wandb report here. I am attaching the logs too:
output.log

The only change that I made to the code was to use the PyTorch dataloaders:

from torchvision.datasets import Flowers102
dataset_train = Flowers102(root=args.data_dir, split="train", download=True)
dataset_eval = Flowers102(root=args.data_dir, split="test", download=True)

I am sure that this might be some minor configuration issue for Flowers Dataset, as I am able to replicate the results on CIFAR-10 and CIFAR-100.

Thanks again, and it would be very kind of you if you could help me.

Thanks,
Joseph

Hello,

Love your work, it's coming in handy for deep learning on my NLP-like nucleotide data. In my use-case, I regularly pad sequences and in order to account for this padding, I would like to mask these "tokens" during the forward pass. However, when employing the masking tensor in the 'text' directory, I believe TextTokenizer.forward() should return the following (or similar):

return x, self.forward_mask(mask) if mask is not None else (x,mask)

By default, we return the argument mask rather than the tensor with the correct shape for the downstream mask required by the masked attention. This is OK for the embedding step as the forward step returns the same tensor (perhaps a bit of optimization could be of use here...), but this behavior causes errors in the tokenizer case.

I can make a PR if desired.

Again, thanks for your efforts. I have a few other questions but I'll save them for other issues.

Traceback (most recent call last): File "main.py", line 251, in <module> main() File "main.py", line 105, in main patch_size=args.patch_size) File "C:\Users\彭张智computational AI\Desktop\Compact-Transformers-main\src\cct.py", line 326, in cct_7 *args, **kwargs) File "C:\Users\彭张智computational AI\Desktop\Compact-Transformers-main\src\cct.py", line 286, in _cct *args, **kwargs) File "C:\Users\彭张智computational AI\Desktop\Compact-Transformers-main\src\cct.py", line 267, in __init__ *args, **kwargs) File "C:\Users\彭张智computational AI\Desktop\Compact-Transformers-main\src\cct.py", line 98, in __init__ nn.init.trunc_normal_(self.positional_emb, std=0.2) AttributeError: module 'torch.nn.init' has no attribute 'trunc_normal_'

Hello, I got the error "AttributeError: 'TransformerClassifier' object has no attribute 'num_tokens'" when I use cct_7_7x2_224_sine and set pretrained=True

Hi,

I was reading the medium post, https://medium.com/pytorch/training-compact-transformers-from-scratch-in-30-minutes-with-pytorch-ff5c21668ed5.

Once screenshot shows that

Could you also share the value of embed_dim.

With default value of 768, it doesn't seem likely that 0.2M parameters are correct for the model in the first row.

Best,
Tianyu.

Hi, sorry to bother you. I have a question about Vit-lite model. In Vit-lite, the convolution layer in Tokenizer should be 0. But I found the conv_layers is defined as 1 in Vit-lite. So when I use Vit-lite, is the Command like this? python3 main.py --dataset cifar10 --epochs 300 --lr 0.001 --model vit_lite_7 --patch-size 4 --conv-size 3 --conv-layers 0 --warmup 10 --batch-size 128 ./cifar10

thank you for your nice work, but i can't find pretrain model, could you tell me where i can find it.
best wish for you

Hello,
have you tried to train this model on ImageNet?
I get only 45% accuracy with the same training hyperparameters as cct_14_7x2_224
thanks,
Ilias

Hi, Ali Hassani
Thank you for sharing your amazing work!

If feasible, I would like to ask a question.
I wonder what training settings you use for CIFAR100?
Because when I use the default setting with resnet18, my best result is 58% differ from the paper which is 63.41%.
I have tried it twice and tweak the warm up to 5 or 10 epochs.
In addition, CCT-2 result is 65.74% in the paper is 66.90%.

Resnet18 pytorch result:
Accu Epoch 190 | 57.7
Accu Epoch 191 | 57.85
Accu Epoch 192 | 57.87
Accu Epoch 193 | 57.92
Accu Epoch 194 | 57.61
Accu Epoch 195 | 57.97
Accu Epoch 196 | 57.78
Accu Epoch 197 | 57.92
Accu Epoch 198 | 58.02
Accu Epoch 199 | 57.94
Accu Epoch 200 | 57.84

Thanks

I want to reproduce your paper, however I find the weight decay and learning rate of cifar10 in *.yaml is different from paper's said, could your please tell me which parameter should I use?