Elastic aggregation and Non-iid data by using dirichlet distribution scenario about fl-bench

[KarhouTam]

Hi, @minmincute912. Thanks for your attention to FL-bench first.

alpha == 0.1 or alpha == 0.5, the output performance always lower than the performance of the method which has alpha == 100.

The fact is the label heterogeneity is negative correlated to alpha. You can evaluate it by split the same dataset using the same arguments except -a and check the class_distribution.pdf that auto generated in data/<dataset> folder. And you will find that when alpha becomes greater, the label heterogeneity decreasing. Back to the results of elastic aggregation. I think is normal that a traditional FL method performs better when label heterogeneity weaker.

Especially, i'm very interested in the new method "Elastic Aggregation" but i see this method do not show better performance than FedAvg as the paper mentioned.

First of all, I don't know whether you experimental settings are the same as the original paper's. And second, acutally I don't responsible for answering the algorithm performance questions, except you find it is due to my wrong code. About Elastic Aggregation, because its author doesn't publish the source code, it's hard to identically reproduce the experiment environment for me. I just simply reproduce the algorithm by its pseudo codes and that's it. If you find something wrong in my code, please let me know (or open a PR for contribution to this project).

details on accuracy and loss after fine-tuning, i just see it's always equal 0.00%

Because you didn't set the finetune_epoch and it is default to 0, which means no local fine-tuning after model testing. So obviously the results about that is 0.00%, which doesn't means 0 loss or 0 accuracy. they mean NONE.

In the end, thanks for your attention and appreciation to me again. Wish you have a good time with FL-bench. 😉

[minmincute912]

Sorry, i typed wrong something, at prolem 1 i just mentioned, i find that smaller alpha like 0.1 always gives better results than larger alpha like 100, not lower @@, i don't know why, can u check this!! Thanh you very much

[minmincute912]

when alpha==100

alpha==0.1

i see when alpha == 100, the performance approximately about 50% but when alpha == 0.1, the performance can observe that around higher than

[KarhouTam]

First of all, the FL-bench learning curve plots actually don't have that much reference value. It's stats only relate to participated clients at each round (10 clients if client num is 100 and join_ratio = 0.1). And it is probably different from the curve in the paper, which calculation may include all clients.

Let me claim that FL-bench's curves mainly provide reference to whether the algorithm collapsing during training. In the case of traditional FL algorithms, like Elastic Aggregation, the test_before one, which means results computed from the global model *before local client training` is the one you should refer to. Unless the algorithm involves local fine-tuning or something about meta-learning (Per-FedAvg), the orange one has less importance.

All learning curves in traditional FL algorithm papers are basically the blue one. And in this situation, Elastic Aggregation runs better with -a 100 than -a 0.1.

BTW, you should mainly consider this value as the final performance of a algorithm, which involves all clients and fair enough (compare to those curve, which involves part of clients).

[KarhouTam]

This issue is closed due to long time no response.