hello, when i change 'split' to 'user', it can't run anymore about fl-bench

[LyaCJ]

I got that, Thank you very much!!

[KarhouTam]

I'm the maintainer, so that's what I need to do. 🤗

[KarhouTam]

Hello, LyaCJ.
Regarding your 1st question, this is because when the split is user, the whole set of clients is split into [train client, test client]. This means that there is no testset in train client and no trainset in testclient. For the bug you encountered, I will fix the code ASAP.

For your 2nd question, the trainset and testset need to be in the same distribution, but with no intersection. In my code, FL methods won't train models with test data while training.

For your confusion about accuracy, e.g., 13.65% -> 79.73%, 13.65% is the average accuracy of evaluation over sampled clients' testsets before local training, while 79.73% is the accuracy after local fine-tuning.

Note that each client's testset only contains data sampled from the same data distribution as the trainset.

e.g.

If client A's train data are from classes [1,2,3,4], the test data client A holds are also only from [1,2,3,4].

Testsets in each client are small, and because of your Non-IID data partitioning setup, each client model can fit very well into their data distribution. As a result, client models perform well on their private testset after local fine-tuning (especially if the test set is small) is reasonable.

[KarhouTam]

Glad to solve your issue.
If you still wanna run code over user split, make sure to keep your eye on my work and pull the latest code after I fix the bug. 😏
Thanks for your attention.

[LyaCJ]

ok!!!

[KarhouTam]

Hi, @LyaCJ.
I have already fixed the bug, you can pull the latest now!

[LyaCJ]

Wow, so nice of you!!! Thank you very much!