is the trainset and testset same in the same client? about fl-bench

[LyaCJ]

thank you very much!!! i will check the code and try to improve the acc.

[LyaCJ]

fedavg model, by the way. and i think acc is 50% is lower than i thought it would be. does that the alpha impact two much?

[KarhouTam]

Hi, LyaCJ. Thanks for your attention to this benchmark first.
For your 1-st question, here's the code for splitting the trainset and testset (in data/utils/run.py):

        for client_id, idx in enumerate(partition["data_indices"]):
            if args.split == "sample":
                num_train_samples = int(len(idx) * args.fraction)
                np.random.shuffle(idx)
                idx_train, idx_test = idx[:num_train_samples], idx[num_train_samples:]
                partition["data_indices"][client_id] = {
                    "train": idx_train,
                    "test": idx_test,
                }
            else:
                if client_id in clients_4_train:
                    partition["data_indices"][client_id] = {"train": idx, "test": []}
                else:
                    partition["data_indices"][client_id] = {"train": [], "test": idx}

It can be seen that the trainset and testset are split according to args.fraction and with no intersection.

For your 2-nd question, have you checked the all_stats.json in data/${dataset_you_used} with the target client ID (through debugging to get)? all_stats.json lists all label distributions around clients.

[KarhouTam]

fedavg model, by the way. and i think acc is 50% is lower than i thought it would be. does that the alpha impact two much?

Of course the $\alpha$ in dirichlet affects a lot. My dirichlet partition code should have no problem. About your 50% acc confusion, cuz' there are so many variables that can affect the final result. For your current information, I can't give you a clear explanation.