kohpangwei / group_dro Goto Github PK
View Code? Open in Web Editor NEWDistributionally robust neural networks for group shifts
License: MIT License
Distributionally robust neural networks for group shifts
License: MIT License
Is there other places where I can download it?
Thanks!
With the command in repo for MNLI, I am not able to reproduce your results.
Could you send me the command which reproduce your best results on CUB and CeleBA?
Especially for that with Large Weight Decay and Early stop
Thanks a lot.
Hi,
Could you provide these files?
list_attr_celeba.csv
list_eval_partition.csv
It would be easier to just use what you already have, instead of converting them on my own.
Edit:
For now, I will be using these:
https://raw.githubusercontent.com/togheppi/cDCGAN/master/list_attr_celeba.csv
https://raw.githubusercontent.com/Golbstein/keras-face-recognition/master/list_eval_partition.csv
You could just close this issue.
It seems that there is no l2-penalities implementation in this code. Should I implement it myself to reproduce the results in the paper?
Hi
Could you provide the command to re-generate your cached_mnli_files. Also, Is this possible to have the code working from the raw text data of MNLI. thanks.
First, thank you for putting in the effort to document your experiments so clearly! Unfortunately, it seems that the download links in the CodaLab worksheet are broken. Is there a way to fix this?
How to reproduce:
Error: 404 Not Found
Sorry, the requested URL'https://worksheets.codalab.org/rest/bundles/0x227a9d64524a46e29e34177b8073cb44/contents/blob/'
caused an error:
Path '' in bundle 0x227a9d64524a46e29e34177b8073cb44 not found
Following up on previous question #7
Please can you clarify whether we need to sample only one group at each iteration or it is OK to have multiple groups in a batch? In your algorithm, it seems to say that we need to sample only one group at each iteration, but this doesn't seem to be the case in the code.
Additionally, please can you comment on the following remark from this paper https://arxiv.org/abs/2010.12230 ?
Is this remark justified?
Thanks for the good work,
Can you please tell me how to obtain three datasets' ERM baseline, e.g., the commands?
Hi,
I cannot open the link that you provide (https://nlp.stanford.edu/data/dro/waterbird_complete95_forest2water2.tar.gz) for waterbird dataset. Could you create a new link to access the dataset? Thanks a lot!
Hi @kohpangwei and @ssagawa ,
Your paper mentioned that "All benchmark models were evaluated at the best early stopping epoch (as measured by robust validation accuracy)." However, your code
Lines 205 to 209 in ca58872
Did you use the model selection rule (i) for ERM in your paper's experiments (e.g., Table 3)? I'm trying to reproduce your results, but I'm not sure if your results on ERM are from (i) or not.
The link to the Codalab worksheet is broken. The page shows "Not found: '/worksheets/0x621811fe446b49bb818293bae2ef88c0'." Could you please update it? Thank you!
Dear authors, Thank you for sharing a well polished codebase!
For the results in table 1, I have noticed that DRO method is always run with "reweight_groups" flag set to "True", whereas the same flag is "False" for the ERM algorithm [1]. As per the code, the "reweight_groups" flag performs a weighted random sampling guaranteeing an equal count of each group in any given batch. On the other hand, the ERM algorithm receives a smaller count of the minority sample as there is no weighted random sampling. Such a difference in implementation between ERM and GroupDRO suggests for an unfair comparison between the two methods.
Surely, as pointed out in the comment [2], the loss function could be considered unaffected by the "reweight_groups" flag as the DRO method uses the mean of per-group losses. However, the empirical estimate of these means in a given batch would be highly noisy when the sample count of the minority group is very small. This makes me wonder (and I hope it's okay for me to ask), that the gains reported in the paper are attributed solely to the use of weighted random sampling procedure rather than DRO update rule? Please clarify
Do you have any comparisons of the DRO algorithm with "reweight_groups" flag set to "False"? How does ERM with "reweight_flag=True" compare to ERM with "reweight_flag=False"?
Thank you
[1] https://worksheets.codalab.org/worksheets/0x621811fe446b49bb818293bae2ef88c0
[2]
Line 56 in f7eae92
Hi
With the command in repo for MNLI, I am not able to reproduce your results.
Could you send me the command which reproduce your best results on MNLI?
Basically seems some argument are missing from that command. thanks.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.