Comments (5)
Removing classification loss is training now.
Removing bbox regression loss causes very slight performance drop whe main LD is used (if add VLR LD, never try yet, but probably help).
Removing DFL will improve AP slightly when LD is used (this phenomenon was observed many times).
from ld.
Removing classification loss is training now.
Thanks for your reply. But I can't quite understand this sentence. Please let me express my questions more clearly.
Currently the implementation of LDHead.loss_single
indicates the overall loss is composed of 4 parts: loss_cls
, loss_bbox
, loss_dfl
and loss_ld
. And the first 3 parts require ground truth annotations to be computed, while loss_ld
only requires soft_targets
which is computed by the teacher model. So:
-
Does the current implementation of loss computation rely on ground truth annotations? Because according to my understanding, there are cases in knowledge distillation where ground truth annotations are unavailable.
-
If no, how does the current implementation work without ground truth? Can I only use
loss_ld
as the overall loss and droploss_cls
,loss_bbox
andloss_dfl
? According to your reply,loss_bbox
andloss_dfl
only affect mAP slightly, so what aboutloss_cls
?
from ld.
I'm trying an experiment without cls_loss.
BTW, why does KD method remove GT annotation? Is there any literature?
You can disable cls_loss, bbox_loss and DFL of course, however, the label assignment still leverages the GT information (i.e., decide where to distill). If you remove these three losses and you use the full map locations to distill, then no GT information will be used. But notice that even if you do so, the teacher detector was trained with GT annotation.
from ld.
2022-02-28 00:19:13,754 - mmdet - INFO - Epoch [12][7050/7330] lr: 1.000e-04, eta: 0:02:22, time: 0.507, data_time: 0.009, memory: 3824, loss_cls: 0.0000, loss_bbox: 0.3418, loss_dfl: 0.0000, loss_ld: 0.0000, loss_ld_neg: 0.0000, loss_cls_kd: 0.1564, loss_cls_kd_neg: 0.0000, loss_gibox_im: 0.0000, loss_im: 0.0000, loss_im_neg: 0.0000, loss: 0.4982
2022-02-28 00:19:39,186 - mmdet - INFO - Epoch [12][7100/7330] lr: 1.000e-04, eta: 0:01:57, time: 0.509, data_time: 0.009, memory: 3824, loss_cls: 0.0000, loss_bbox: 0.3358, loss_dfl: 0.0000, loss_ld: 0.0000, loss_ld_neg: 0.0000, loss_cls_kd: 0.1495, loss_cls_kd_neg: 0.0000, loss_gibox_im: 0.0000, loss_im: 0.0000, loss_im_neg: 0.0000, loss: 0.4852
2022-02-28 00:20:04,561 - mmdet - INFO - Epoch [12][7150/7330] lr: 1.000e-04, eta: 0:01:31, time: 0.508, data_time: 0.009, memory: 3824, loss_cls: 0.0000, loss_bbox: 0.3399, loss_dfl: 0.0000, loss_ld: 0.0000, loss_ld_neg: 0.0000, loss_cls_kd: 0.1526, loss_cls_kd_neg: 0.0000, loss_gibox_im: 0.0000, loss_im: 0.0000, loss_im_neg: 0.0000, loss: 0.4925
2022-02-28 00:20:29,878 - mmdet - INFO - Epoch [12][7200/7330] lr: 1.000e-04, eta: 0:01:06, time: 0.507, data_time: 0.009, memory: 3824, loss_cls: 0.0000, loss_bbox: 0.3240, loss_dfl: 0.0000, loss_ld: 0.0000, loss_ld_neg: 0.0000, loss_cls_kd: 0.1539, loss_cls_kd_neg: 0.0000, loss_gibox_im: 0.0000, loss_im: 0.0000, loss_im_neg: 0.0000, loss: 0.4779
2022-02-28 00:20:55,356 - mmdet - INFO - Epoch [12][7250/7330] lr: 1.000e-04, eta: 0:00:40, time: 0.510, data_time: 0.008, memory: 3824, loss_cls: 0.0000, loss_bbox: 0.3488, loss_dfl: 0.0000, loss_ld: 0.0000, loss_ld_neg: 0.0000, loss_cls_kd: 0.1390, loss_cls_kd_neg: 0.0000, loss_gibox_im: 0.0000, loss_im: 0.0000, loss_im_neg: 0.0000, loss: 0.4879
2022-02-28 00:21:20,694 - mmdet - INFO - Epoch [12][7300/7330] lr: 1.000e-04, eta: 0:00:15, time: 0.507, data_time: 0.008, memory: 3824, loss_cls: 0.0000, loss_bbox: 0.3350, loss_dfl: 0.0000, loss_ld: 0.0000, loss_ld_neg: 0.0000, loss_cls_kd: 0.1496, loss_cls_kd_neg: 0.0000, loss_gibox_im: 0.0000, loss_im: 0.0000, loss_im_neg: 0.0000, loss: 0.4845
2022-02-28 00:21:45,821 - mmdet - INFO - Saving checkpoint at 12 epochs
2022-02-28 00:24:14,925 - mmdet - INFO - Evaluating bbox...
2022-02-28 00:25:27,692 - mmdet - INFO - Exp name: im_r101_r50_coco_1x.py
2022-02-28 00:25:27,693 - mmdet - INFO - Epoch(val) [12][7330] bbox_mAP: 0.3270, bbox_mAP_50: 0.4820, bbox_mAP_75: 0.3520, bbox_mAP_s: 0.1940, bbox_mAP_m: 0.3680, bbox_mAP_l: 0.3980, bbox_mAP_copypaste: 0.327 0.482 0.352 0.194 0.368 0.398
The above training settings are bbox_loss on positive locations, and cls KD on full map locations.
Removing cls_loss causes significant AP drops (7.4 points).
from ld.
All my doubts are cleared. Thanks a lot!
from ld.
Related Issues (20)
- I reimplement your model (Teacher: R101, Student: R50, Detection head: Foveabox) and got RuntimeError: shape '[-1, 68]' is invalid for input of size 115200 HOT 5
- Welcome update to OpenMMLab 2.0 HOT 2
- 请问下代码里的教师模型是跟着学生模型一起训练的吗 HOT 2
- If I set output_feature = False , Code don't work HOT 1
- 参考install.md 里面配置的环境无法找到cuda HOT 13
- 简易版本复现问题 HOT 5
- 将DKD与LD结合的问题 HOT 8
- Implement LD on VisDrone dataset HOT 5
- 学生网络与教师网络性能差距较大
- mmpycocotools HOT 1
- 请问下是否可以迁移到centernet呢 HOT 2
- 知识蒸馏 HOT 10
- 请问是使用 ldv2/ld_r50_gflv2_r101_fpn_1x.py 这个文件来复现 Temperature τ in LD 这个实验吗? HOT 1
- kd_loss implementation issue HOT 9
- 关于数据集minicoco HOT 4
- About Adaptive Layer HOT 2
- Can we pass different set of image to teacher and student. HOT 6
- samples_per_gpu HOT 1
- loadcheckpoint出现问题 HOT 2
- Knowledge distillation related
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ld.