Comments (5)
Glad that I have addressed your concerns.
The equation in the paper seems wrong.
It should be
Thanks for your question!
from weighted-soft-label-distillation.
Hi @LcDog , thanks for your quick reply!
I have double-checked the derivation of Heskes's paper and gradually understood the decomposition for the expected error defined on the KL Div. Unfortunately, I still did not get the idea of how E(y) and E(\bar{y}) are all equals to 1
leads to
I am really sorry for bothering you again :(
As far as I can tell,
I am not sure how to use E(y) and E(\bar{y}) are all equals to 1
in the above equation. It seems that
Many thanks!
from weighted-soft-label-distillation.
Cool! The derivation is very clear!
The notation is ok but I found E[y]=1 and E[\overline y]=1
quite confusing, which makes me wonder is there any
tricks in Equation(2). 😅
By the way, could you further explain how to get the equation in your first comment?
As for
I just quoted the conclusion in the ICLR paper "The derivation of the variance term is based on the facts that $\frac{\log \overline{y}{ce}}{\mathbb{E}\mathcal{D}[\log \hat{y}_{ce}]}$ is a constant and ...".
Thanks for your responses!
from weighted-soft-label-distillation.
Hi @HolmesShuan , thanks for your question!
The reason is that E(y) and E(\bar{y}) are all equals to 1. Please refer to the equation above Eqn 4 from Heskes's paper: an extra \bar{p}(y) in the second row.
This follows the definition used in Eqn 2 from Heskes's paper. In their paper, it is defined \int dy a(y) = 1. (constraints in Eqn 2)
In our notations, y becomes x and a() becomes y=t(). We originally kept all notations the same as Heskes's paper, but a reviewer thinks the notations are not appropriate thus we changed all notations.
Sorry for the confusion! Please let me know if there are any further questions.
from weighted-soft-label-distillation.
No problem! You are very welcome to ask anything~
No need to decompose the two terms. We first have
then
so
similar we have
By the way, could you further explain how to get the equation in your first comment?
Seems like that our notations are somewhat misleading
from weighted-soft-label-distillation.
Related Issues (8)
- 缺少dataset文件 HOT 1
- The pretrained teacher and hyper-parameters on CIFAR-100 HOT 8
- Hello, I have a question about training CIFAR-100 HOT 3
- Hi, I cannot reproduce your reported performance on CIFAR-100. HOT 6
- Assumption 1: a gap between "KD helps calibrate" and "KD reduces variance". HOT 4
- Hyper-parameters settings? HOT 1
- KD Loss keeps raising during training
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from weighted-soft-label-distillation.