Question about F1 score for Cifar-10 about edgeml

[SachinG007]

Hi,
This depends upon what you used as a threshold score for F1 score computation.
Can you please share more details, so that we can help you out?

Thanks

[Yuchong-Geng]

Thanks for the response.

So I am trying to run the cifar experiment as follow:

python3 main_cifar.py --lamda 1 --radius 8 --lr 0.001 --gamma 1 --ascent_step_size 0.001 --batch_size 256 --epochs 100 --optim 0 --normal_class 0

And instead of using the default "AUC" metric, I am using the F1 score that is defined in the DROCCTrainer.

thresh = np.percentile(scores, 20)
y_pred = np.where(scores >= thresh, 1, 0)
prec, recall, test_metric, _ = precision_recall_fscore_support(
labels, y_pred, average="binary") 

[SachinG007]

Hi,
We kept this threshold for F1 score mainly for tabular experiment results (since previous work in that domain considers F1 score as a metric at a specific threshold). For images, we would suggest you look at AUC scores only since it does not consider just one specific threshold but gives an overall picture from various thresholds. Moreover, this is in line with the previous works on images.

Thanks

[SachinG007]

Hi @Yuchong-Geng ,

Can I close the issue now?

[Yuchong-Geng]

Sure. Thanks for the help!