There is no VALIDATION set for FEMNIST LEAF - help wanted about fl-bench

[KarhouTam]

Thanks for pointing it out. You can manually add "val": [], in

FL-bench/data/utils/process.py

Lines 193 to 197 in a7a33ea

	partition["separation"] = {
	"train": clients_4_train,
	"test": clients_4_test,
	"total": client_cnt,
	}

Or modify self.val_clients = partition["separation"].get("val", []) in FedAvgServer.__init__() in src/server/fedavg.py

This bug will be fixed soon.

[KarhouTam]

I'm currently busy on injecting parallelism into FL-bench, which will vastly improve efficiency. Please stay tuned.

[KarhouTam]

The bug is fixed and now FL-bench can train in parallel.

[KarhouTam]

First, FL-bench does not support adversaries.

About the model, You need to find out the detailed architecture yourself. If it's already integrated in FL-bench, just use it by

# your config yaml file
common:
  model: model_name

About the hyperparameters, you need to decide the values and set them by

# your config yaml file
common:
  ...
ditto:
  lamda: ...
  ...

About other FL training hyperparameters, check README for the settings FL-bench supports and set their value by yourself.

Sorry, I can't teach you the exact way to reproduce the results in your screenshot. I can only support you in using FL-bench.
Good luck, @tejalgoyal2.

[tejalgoyal2]

Thank you for your instant response.
I will try this.

Thank you again, waiting for the bug fix :)

[tejalgoyal2]

Hello, thank you for the assistance. Also, can you advise me on what attributes I need to change and how can I change them to reproduce the DITTO paper results?

Thank you :)

[KarhouTam]

If you are asking about the parallelism, the details are shown on the README.
If you are asking how to reproduce ditto, please refer to the paper for the hyperparameters. About the way to set FL method's hyperparameters is also shown on the README.

[tejalgoyal2]

Yes the second one, I was having difficulty in matching the parameters DITTO used and the parameters mentioned in the ReadME.

Like for example if I want to reproduce results similar to this table:

What inputs I will need to change. Thank you so much for helping :)