alinlab / lff

Learning from Failure: Training Debiased Classifier from Biased Classifier (NeurIPS 2020)

Learning from Failure: Training Debiased Classifier from Biased Classifier

Official PyTorch implementation of "Learning from Failure: Training Debiased Classifier from Biased Classifier" (NeurIPS 2020) by Junhyun Nam et al.

Install dependencies

pip install -r requirements.txt

Make dataset

python make_dataset.py with server_user make_target=colored_mnist

Training

python train.py with server_user colored_mnist skewed3 severity4

Citation

@inproceedings{nam2020learning,
  title={Learning from Failure: Training Debiased Classifier from Biased Classifier},
  author={Junhyun Nam and Hyuntak Cha and Sungsoo Ahn and Jaeho Lee and Jinwoo Shin},
  booktitle={Advances in Neural Information Processing Systems},
  year={2020}
}

lff's Issues

can't download the celeba dataset.

I can't download the celeba dataset. It always shows that I can't connect to the server or there's no server, no response, etc. Does anyone know how to solve it? What mistake did I make

Unable to replicate numbers on CelebA

Hi,

After trying to run the code for CelebA (with ResNet18), I get the following results:

1. valid_attrwise_accs_b
   tensor([[0.9578, 0.9966], [0.8382, 0.2747]])
2. valid_attrwise_accs_d
   tensor([[0.0300, 0.0205], [0.6841, 0.9231]])

Could you please specify the exact way to replicate the results in the paper?

Thank you!

Experiments on BAR

Could you provide the detailed settings of baseline/rebias/yours on the BAR dataset to get the reported performance

Incorrect accuracy computation

In train.py the valid_accs_b and valid_accs_d are computated as a mean of the mean attribute-wise accuracies. This will result in incorrect results if the class and bias labels are not equally distributed.

Cannot reproduce results

Hi,

I ran the experiments for colored MNIST three times and got the following results:
python train.py with server_user colored_mnist skewed3 severity4
valid_accs_d aligned 0.5823 valid_accs_d skewed 0.7024
valid_accs_d aligned 0.6067 valid_accs_d skewed 0.6851
valid_accs_d aligned 0.6767 valid_accs_d skewed 0.6791

avg: 62.19 and 68.88

Same issue for CIFAR
python train.py with server_user corrupted_cifar10 type0 skewed3 severity4
valid_accs_d aligned 0.5980 valid_accs_d skewed 0.3731
valid_accs_d aligned 0.5760 valid_accs_d skewed 0.3493

avg: 58.7 36.12

Am I making any mistake or are you using any other set of hyperparameters?

GCE implementation

Why is GCE implemented as

F.cross_entropy(logits, targets, reduction='none') * (Yg.squeeze().detach()**self.q)*self.q

instead of (1 - Yg.squeeze().detach()**self.q) / self.q ?