Hi, I have read your excellent work several times. The bias-variance idea is very inte

Hi <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="

Cool! The derivation is very clear! It seems that <a href="https

Hi <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="

Minor questions on Eq.(2) about weighted-soft-label-distillation HOT 5 CLOSED

HolmesShuan commented on August 17, 2024 1

Minor questions on Eq.(2)

from weighted-soft-label-distillation.

Comments (5)

lsongx commented on August 17, 2024 2

Glad that I have addressed your concerns.

The equation in the paper seems wrong.
It should be $\mathbb{E}_D[\log\frac{\overline{\mathrm y}}{\hat{\mathrm y}}]=\log Z$

Thanks for your question!

from weighted-soft-label-distillation.

HolmesShuan commented on August 17, 2024 1

Hi @LcDog , thanks for your quick reply!

I have double-checked the derivation of Heskes's paper and gradually understood the decomposition for the expected error defined on the KL Div. Unfortunately, I still did not get the idea of how E(y) and E(\bar{y}) are all equals to 1 leads to
$\mathbb{E}_D[\mathbb{E}_\mathrm{x}[\mathrm{y}\log\frac{\overline{\mathrm{y}}_{ce}}{\hat{y}_{ce}}]]]=\mathbb{E}_D[D_{KL}(\overline{\mathrm{y}}_{ce},\hat{\mathrm{y}}_{ce})]?$

I am really sorry for bothering you again :(

As far as I can tell,
$\mathbb{E}_D[\mathbb{E}_x[\mathrm y\log\frac{\overline{\mathrm y}}{\hat{\mathrm y}}]]=\mathbb{E}_D[\mathbb{E}_x[\mathrm y\log\overline{\mathrm y}-\mathrm y\log\hat{\mathrm y}]]=\mathbb{E}_x[\mathrm y\log\overline{\mathrm y}]-\mathbb{E}_D[\mathbb{E}_x[\mathrm y\log\hat{\mathrm y}]]$
I am not sure how to use E(y) and E(\bar{y}) are all equals to 1 in the above equation. It seems that
$\mathbb{E}_x[f(x)g(x)]\neq\mathbb{E}_x[f(x)]\mathbb{E}_x[g(x)]$

Many thanks!

from weighted-soft-label-distillation.

HolmesShuan commented on August 17, 2024 1

Cool! The derivation is very clear!

It seems that
$\mathbb{E}_D[\log\frac{\overline{\mathrm y}}{\hat{\mathrm y}}]]=-\log\mathrm Z?$
according to
$\overline{\mathrm y}=\frac{1}{\mathrm Z}\exp(\mathbb{E}_D[\log \hat{\mathrm y}])$

The notation is ok but I found E[y]=1 and E[\overline y]=1 quite confusing, which makes me wonder is there any
$\mathbb{E}[*]=\mathbb{E}[*]\cdot1=\mathbb{E}[*]\cdot\mathbb{E}[\mathrm y]=\mathbb{E}[*]\cdot\mathbb{E}[\overline{\mathrm y}]...$
tricks in Equation(2). 😅

By the way, could you further explain how to get the equation in your first comment?

As for
$\frac{\log \overline{\mathrm y}_{ce}}{\mathbb{E}_D[\log \hat{y}_{ce}]}=constant$
I just quoted the conclusion in the ICLR paper "The derivation of the variance term is based on the facts that $\frac{\log \overline{y}{ce}}{\mathbb{E}\mathcal{D}[\log \hat{y}_{ce}]}$ is a constant and ...".

Thanks for your responses!

from weighted-soft-label-distillation.

lsongx commented on August 17, 2024

Hi @HolmesShuan , thanks for your question!
The reason is that E(y) and E(\bar{y}) are all equals to 1. Please refer to the equation above Eqn 4 from Heskes's paper: an extra \bar{p}(y) in the second row.
This follows the definition used in Eqn 2 from Heskes's paper. In their paper, it is defined \int dy a(y) = 1. (constraints in Eqn 2)
In our notations, y becomes x and a() becomes y=t(). We originally kept all notations the same as Heskes's paper, but a reviewer thinks the notations are not appropriate thus we changed all notations.
Sorry for the confusion! Please let me know if there are any further questions.

from weighted-soft-label-distillation.

lsongx commented on August 17, 2024

No problem! You are very welcome to ask anything~

No need to decompose the two terms. We first have
$\mathbb{E}_D[\log\frac{\overline{\mathrm y}}{\hat{\mathrm y}}]=\log Z$
then
$\int y\mathbb{E}_D[\log\frac{\overline{\mathrm y}}{\hat{\mathrm y}}] \mathrm{d}x=\int y\log Z \mathrm{d}x$
so
$\mathbb{E}_D[\int y\log\frac{\overline{\mathrm y}}{\hat{\mathrm y}} \mathrm{d}x]=\int y\log Z \mathrm{d}x$
similar we have
$\mathbb{E}_D[\int \bar{y}\log\frac{\overline{\mathrm y}}{\hat{\mathrm y}} \mathrm{d}x]=\int \bar{y}\log Z \mathrm{d}x$

By the way, could you further explain how to get the equation in your first comment?
$\frac{\log \overline{\mathrm y}_{ce}}{\mathbb{E}_D[\log \hat{y}_{ce}]}=constant$
Seems like that our notations are somewhat misleading

from weighted-soft-label-distillation.

Minor questions on Eq.(2) about weighted-soft-label-distillation HOT 5 CLOSED

Comments (5)

Related Issues (8)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent

Jobs