Behaviors of AUROC and Average Precision are inconsistent when all labels are equal about torchmetrics

[SkafteNicki]

Hi @weihua916, thanks for raising this issue.
I created PR #2507 that is intended to close this issue. The intention behind our implementations are to match sklearn pretty close. By this I mean:

• Averageprecision when all labels are 1 in sklearn returns a score of 1, which we are also doing
• Averageprecision when all labels are 0 in sklearn returns a score of -0.0, where our implementation returns nan. That is not the intention and this will be fixed in PR #2507 to raise a user warning and return -0.0 similar to sklearn.
• AUROC in sklearn completely fails in both the case when all labels are 1 and all labels are 0. We instead have chosen to raise user warnings that scores in both cases are essentially undefined and return the arbitrary score of 0. The reason for this is that other users have requested that metrics do not crash there code during training, which will also happens if the scores return nan. We therefore have chosen to go with a real, but arbitrary score.

[weihua916]

Thank you for addressing the issue! For AUROC, I personally still believe nan is better, since it's easy to convert nan to 0 outside of torch-metrics. Currently, the arbitrary AUROC score of 0 may be confused with the actual score of 0.

[SkafteNicki]

@weihua916 I do not necessarily disagree with you on that auroc should return nan and not 0, however we had overwhelmingly feedback when the metric was introduced in the beginning that this was to be preferred.

[weihua916]

Understood. Thanks for your consideration!