The role of projector layer about simkd

[DefangChen]

Generally, feature distributions of the teacher and student model are statistically different and cannot be directly compared in practice even their dimensions are equal. We thus follow the setting of previous KD methods (e.g., FitNet, CRD, SRRL, SemCKD) and retain this design.

Experiments show that without reusing the teacher classifier, the modified student model (containing extra projector layer) cannot achieve good results by training with regular cross-entropy loss or KD loss. This indicates the projector layer itself do not sufficiently account for the SimKD performance.

[ZX-lang]

Generally, feature distributions of the teacher and student model are statistically different and cannot be directly compared in practice even their dimensions are equal. We thus follow the setting of previous KD methods (e.g., FitNet, CRD, SRRL, SemCKD) and retain this design.

Experiments show that without reusing the teacher classifier, the modified student model (containing extra projector layer) cannot achieve good results by training with regular cross-entropy loss or KD loss. This indicates the projector layer itself do not sufficiently account for the SimKD performance.

Thank you for your reply. So how to find a method to remove the projector is still a difficult problem.

[DefangChen]

Exactly. I believe it is of great importance to the KD research.

[ZX-lang]

Exactly. I believe it is of great importance to the KD research.

Thank you. I will try some other way to solve this problem.

[ZX-lang]

Exactly. I believe it is of great importance to the KD research.

When I use wrn40_2 as teacher and wrn40_1 as a student, I get a result that is not as good as the paper shows, could you please release the implementation of wrn

[DefangChen]

Note that the implementation of wrn in CRD is somewhat problematic. It actually contains 38 layers rather than 40 layers. We have discussed this issue in line 18-24 of resnet.py. (Due to the existence of this issue, I recommend to use resnet-depth x factor rather than wrn-depth x factor.)

In all our implementations, we actually use resnet.py to obtain resnet-38x2 using the scripts and rename it as wrn40_2 for consistency. (The pretrained resnet-38x2 model has been provided in GoogleDrive.)

If you use a pre-trained teacher model with lower accuracy, it may lead to lower accuracy for all KD methods.

[ZX-lang]

Note that the implementation of wrn in CRD is somewhat problematic. It actually contains 38 layers rather than 40 layers. We have discussed this issue in line 18-24 of resnet.py. (Due to the existence of this issue, I recommend to use resnet-depth x factor rather than wrn-depth x factor.)

In all our implementations, we actually use resnet.py to obtain resnet-38x2 using the scripts and rename it as wrn40_2 for consistency. (The pretrained resnet-38x2 model has been provided in GoogleDrive.)

If you use a pre-trained teacher model with lower accuracy, it may lead to lower accuracy for all KD methods.

Thank you. I will try it