how to get uncertainty? about pytorch-classification-uncertainty

[Cogito2012]

@flashcp I think you're misunderstanding the 'b_k'.

In this paper, b_k is belief mass and b_k = e_k / S = (\alpha_k - 1) / S. However, the category probability p_k=\alpha_k / S.

For the Master Yoda example, the evidence e_k = 0 for each of K categories so that b_k=0 and \alpha_k = 1. Therefore, the uncertainty u = 1 - sum_k(b_k) = 1 and the class prob p_k = 1 / K.

[takuya-takeuchi]

Hi, everyone.
I applied this method to mobilenetv2.
Probability is not bad but uncertainty is not good.

As @Cogito2012 said, u + sum_k(b_k) could be close to 1.0.
I understand probability and uncertainly are inverse proportional.

However, result was not.
This image is generated from mnist validation dataset.

Metrics
       accuracy: 0.9936
      precision: 0.9936327472490634
         recall: 0.9936
             f1: 0.9936039409292189

              precision    recall  f1-score   support

           0    0.97898   0.99796   0.98838       980
           1    0.99648   0.99736   0.99692      1135
           2    0.99612   0.99612   0.99612      1032
           3    0.99604   0.99505   0.99554      1010
           4    0.99287   0.99287   0.99287       982
           5    0.99551   0.99439   0.99495       892
           6    0.99895   0.98956   0.99423       958
           7    0.99220   0.99027   0.99124      1028
           8    0.99487   0.99487   0.99487       974
           9    0.99401   0.98712   0.99055      1009

    accuracy                        0.99360     10000
   macro avg    0.99360   0.99356   0.99357     10000
weighted avg    0.99363   0.99360   0.99360     10000

I can not see scatter graph about distribution of probability and uncertainty from Murat Sensoy's original paper.
Is there someone who reproduced like the above result?

Thanks a lot.

[Cogito2012]

@takuya-takeuchi From your figure, it seems that you ploted the uncertainty (u) w.r.t. the maximum probability of all classes (e.g., max{p_1, p_2, ..., p_K}, because most of the dots are close to p=1.0. However, according to the equations in this EDL paper, I don't think there will be a relationship between u and max{p_1, p_2, ..., p_K}.

[flashcp]

@flashcp I think you're misunderstanding the 'b_k'.

In this paper, b_k is belief mass and b_k = e_k / S = (\alpha_k - 1) / S. However, the category probability p_k=\alpha_k / S.

For the Master Yoda example, the evidence e_k = 0 for each of K categories so that b_k=0 and \alpha_k = 1. Therefore, the uncertainty u = 1 - sum_k(b_k) = 1 and the class prob p_k = 1 / K.

you are right, thanks for you reply