Difference between the paper and the code about compact-transformers

[stevenwalton]

Hey Haelles,

If you look at src/cct:231 you'll see that we hard code CCT (and CVT) to have a dropout of 0. Though ViTLite line 161 has a dropout of 0.1.

The lines that you are looking at (src/transformers.py:62) are called by all the models. They all use the same Transformer Encoder structure. But CCT and CVT will not apply dropout and you can safely ignore that for those models, unless you want to do some hyper parameter testing yourself.

I can see the ambiguity in how we wrote it. We'll make sure this is more clear in the next version of the paper. Thank you!

[Haelles]

Hey Haelles,

If you look at src/cct:231 you'll see that we hard code CCT (and CVT) to have a dropout of 0. Though ViTLite line 161 has a dropout of 0.1.

The lines that you are looking at (src/transformers.py:62) are called by all the models. They all use the same Transformer Encoder structure. But CCT and CVT will not apply dropout and you can safely ignore that for those models, unless you want to do some hyper parameter testing yourself.

I can see the ambiguity in how we wrote it. We'll make sure this is more clear in the next version of the paper. Thank you!

Thank you for your reply.
If I may ask again, I added print(self.dropout1) at src/transformers.py:62 and got Dropout(p=0.1, inplace=False). I think there is something wrong here src/cct.py:109, the value of parameter "dropout" should be "dropout" rather than "dropout_rate". The value of "dropout_rate" is 0.1 here.
I used python main.py --model cct_7 --conv-size 3 --conv-layers 1 path/to/cifar10
Really appreciate your help!

[alihassanijr]

Hello @Haelles

Thank you for bringing this to our attention. We noticed there was a small key mismatch in this version, and we just fixed it in the latest commit (39b45de).

[Haelles]

Thanks for your response!