Models implementation: number of channels about simkd

[DefangChen]

Do you mean the default implementation in https://github.com/pytorch/vision/blob/main/torchvision/models/resnet.py ?
Actually, the torchvision implementation is designed for ImageNet dataset.
We follow the original implementation of ResNet paper and previous compared knowledge distillation methods to adopt different architectures for CIFAR and ImageNet.

[antheva]

Which KD method repository / implementation are you referring to exactly? I knew that the first layers of ResNet need to be changed for CIFAR100 but I didn't know the number of channels require an adjustment as well.

[DefangChen]

A typical implementation can be found at https://github.com/HobbitLong/RepDistiller.
In addition to the first layer, ResNet's default setting for CIFAR-100 changes the channel numbers based on Kaiming's cvpr paper.
I guess If you use the ImageNet architecture, it should also work fine. Keep in mind that we heavily use the Wide ResNet architecture to increase the channel numbers, such as ResNet-32x4.

[antheva]

Thank you for your quick and clear answers.
Just one more question: why didn't you extensively evaluate your method on ImageNet too? If I'm not wrong there's only a couple of experiments in the paper and supplementary materials.

[DefangChen]

The reason is simply the lack of adequate computational resources. At that time, I only have access to 4xA40 for a limited period of time. However, I strongly believe that thoroughly evaluating existing methods (reproducing and tuning them) on ImageNet is very valuable and can potentially lead to promising research ideas.

[antheva]

Thanks!