flhonker / awesome-knowledge-distillation Goto Github PK

Thank u for your contribution.
Cloud i create a new branch that add a category “kd in rec”?
thank u again!

Hi, Thank you for listing our paper "Knowledge Distillation Beyond Model Compression.", we really appreciate your effect.

I would like to request correcting the authorship of the paper, it should be Sarfraz et. al.

Also, our work isn't related to pruning and quantization, I believe a more relevant position for it would be in "beyond"

Task-Oriented Feature Distillation
https://papers.nips.cc/paper/2020/file/a96b65a721e561e1e3de768ac819ffbb-Paper.pdf
https://github.com/ArchipLab-LinfengZhang/Task-Oriented-Feature-Distillation

Hi @FLHonker
I notice that there is no paper in 2024 listed yet. Could you please add a relevant paper from CVPR 2024 Highlight. The information is given below:

Title: Logit Standardization in Knowledge Distillation
Paper: https://arxiv.org/abs/2403.01427
Github: https://github.com/sunshangquan/logit-standardization-KD
Supplements: https://sunsean21.github.io/resources/cvpr2024_supp.pdf

The paper discusses the possibility of assigning temperatures distinctly between teacher/student and dynamically across samples. It then proposes a weighted Z-score logit standardization as a plug-and-play preprocess, capable of boosting the existing logit-based KD methods.
Thank you for your attention.

Would you mind adding this paper to the "Applications of KD" section?

How many Observations are Enough? Knowledge Distillation for Trajectory Forecasting. Monti, Alessio et al. CVPR 2022

感谢您的分享，
就提个建议，如果有项目链接就很更好了

Hi,

My name is Marco Mistretta, and I am a researcher at MICC (Florence, Italy).
Thank you for creating this repository! It has been an invaluable resource for me.

Could you please add our ECCV24 paper, Improving Zero-shot Generalization of Learned Prompts via Unsupervised Knowledge Distillation?
It's the first time that unsupervised knowledge distillation has been applied to prompt learning in the few-shot setting!

Thank you for your time and consideration!

TITLE: Improving Zero-shot Generalization of Learned Prompts via Unsupervised Knowledge Distillation
CONFERENCE: ECCV24
PAPER: https://arxiv.org/abs/2407.03056
CODE: https://github.com/miccunifi/KDPL

Thanks for your work!

In data-free KD section, the paper: "DAFL:Data-Free Learning of Student Networks. ICCV 2019" have released codes in: https://github.com/huawei-noah/Data-Efficient-Model-Compression/tree/master/DAFL

Could you update it? Thanks!

Could you help to add this paper belonging to structured distillation I think:

Yu, Lu, Vacit Oguz Yazici, Xialei Liu, Joost van de Weijer, Yongmei Cheng, and Arnau Ramisa. "Learning Metrics from Teachers: Compact Networks for Image Embedding." In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2907-2916. 2019.

Paper: https://www.bmvc2021-virtualconference.com/assets/papers/0259.pdf
Code: https://github.com/ZHANGHeng19931123/MutualGuide

I find there are lacks of papers in 2022?
Continue to collect the related research published in 2022?

您好，感谢分享，感觉目前与推荐相关的paper较少，所以想问问您，是因为这部分内容比较新，还是说有比较严重的问题难以开展研究？