the confidence of ece metric about emcmc

[lblaoke]

Thanks for pointing out your confusion! The evaluation code has been updated. We now use AUROC & AUPR for OOD detection, and AUROC & ECE for confidence calibration. The entropy-based uncertainty scores are normalized to [0, 1]. The paper will also be updated soon.

Just a reminder that the entropy-based uncertainty is only used for OOD detection. We use standard softmax probabilities for confidence calibration.

[lblaoke]

Setting confidence to be 1 - entropy is just for evaluation purpose, since we aim to evaluate the Bayesian predictive uncertainty (computed by entropy). You can even set confidence to be -entropy. Only the relative uncertainty values are meaningful in the ECE metric

[J-zin]

thanks for your super quick reply.

I am still confused.

Think about this case: we have entropy > 2 for all predictions, then 1 - entropy would be negative, and we have ECE= 0. Does it mean the model is well calibrated?

[lblaoke]

Yes, the example is right, but it won't happen to a trained model. The reason why I use 1 - entropy rather than 1 - entropy/log(N) is: The real entropy values obtained are all very close to 0, and if I normalize it to the range [0, 1], there will not be any small confidence score. That's bad for ECE computation

[J-zin]

Thanks Bolian, it makes more sense to me.

One more thing, I find one incorrect claim in Lemma 1. It is true that the marginal distribution of $\theta_a$ coincides with the original posterior $p(\theta_a | D)$. However, it seems that the marginal of $\theta$ is not the same as the original posterior $p(\theta | D)$, because you have $2 \theta^T \theta_a$ inside the exponential, and $\theta_a$ couldn't be marginalised out.

[lblaoke]

You may check the proof of Lemma 1. The current ArXiv version has a small typo, but it will help you understand this claim.

The point is: the integral is only w.r.t. $\theta_a$.

[J-zin]

my bad, ur right. There is a small typo in eq.21, but the idea is correct. The main trick is $\int \exp (-\frac{1}{2\sigma} || x - \mu ||^2) dx = (2\pi \sigma)^{d/2}$. Sorry for that. Nice paper again!