Comments (7)
I have the same problems and I also have results around 50 % for the small sent140 dataset. Could you provide accuracy values incl. used parameters/data set sizes etc for the three datasets, @scaldas? Many thanks.
from leaf.
@Enehta I was able to reproduce sent140 experiments with the following settings:
- Dataset preprocessing:
leaf/data/sent140/ $> ./preprocess.sh --sf 0.5 -t sample --tf 0.8 -k 3 --spltseed 1549775860
- Model execution:
leaf/models $> python3 main.py -dataset sent140 -model stacked_lstm -lr 0.0003 --clients-per-round 2 --num-rounds 10
For 2 clients, 10 rounds and k=100, the model converges to mean accuracy of ~60% (10th percentile: 25%, 90th percentile: 100%). Can you try this and see if it works out?
from leaf.
@slowbull @Enehta for the FEMNIST experiments, I have a script in an open PR that should be able to run the experiments (https://github.com/TalwalkarLab/leaf/blob/5aab2848844c82cce82769be1fd4bd14c891c920/femnist.sh). The results generated are qualitatively similar to those reported in the paper. Can you try this and let me know if it works for you?
from leaf.
@gokart23 I tried both scripts and have not finished all configurations yet, but the FEMNIST script converges around 80 % and the sent140 max. at 60 %. Does that fit to your best results?
from leaf.
@gokart23 You are creating the data iid, but what is with niid? have you reached good accuracy values there?
from leaf.
@Enehta: @gokart23 just uploaded some scripts that get a good accuracy with non-iid data (see 7de6548).
from leaf.
@Enehta Have you reproduced the result of FEMNIST in non-i.i.d. in the original paper?
I also encountered problems with low accuracy!
from leaf.
Related Issues (20)
- A question on stacked_lstm model for sent140
- Experiment on sent140 is not generating the result mentioned in the paper
- The Reddit (small) splits seem to have the same data for training, validation and test sets HOT 4
- Download dataset too slow.
- download the femnist data HOT 1
- tf.layers.dense logit values are not correct in synthetic log_reg.py model
- federated learning anomaly detection HOT 2
- PyTorch Version? HOT 1
- Is F-EMNIST class-balanced?
- Fedprox
- local differential privacy
- incorrect model for CelebA
- The statistics for FEMNIST seems to be inaccurate? HOT 2
- Preprocessing of sent140
- TypeError: 'tuple' object does not support item assignment HOT 2
- accuracy
- If i want to use cifar10 how can i do it ? HOT 1
- split data raise error HOT 1
- Completed porting all shell commands in LEAF into python code!
- [Suggestion] Change Image.ANTIALIAS to Image.LANCZOS in FEMNIST/data_to_json.py
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from leaf.