google-research / augmix

AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty

Google Research

This repository contains code released by Google Research.

All datasets in this repository are released under the CC BY 4.0 International license, which can be found here: https://creativecommons.org/licenses/by/4.0/legalcode. All source files in this repository are released under the Apache 2.0 license, the text of which can be found in the LICENSE file.

---

Because the repo is large, we recommend you download only the subdirectory of interest:

SUBDIR=foo
svn export https://github.com/google-research/google-research/trunk/$SUBDIR

If you'd like to submit a pull request, you'll need to clone the repository; we recommend making a shallow clone (without history).

git clone [email protected]:google-research/google-research.git --depth=1

---

Disclaimer: This is not an official Google product.

Updated in 2023.

augmix's Issues

Can you share any acc information per epoch?

Hi Author I am a college student studying deep learning in Korea. I read your paper very interesting. But I am running AugMix on imagenet and I have to wait 6 days with my GPU machine. So I'm wondering if learning is going well. To find out the trend in the middle, can you share any acc information per epoch?

`depth` is constant each `width`

Hello, thank you for releasing a implementation.

In my understanding, we change number of op as depth each width , however this code should be same depth each width here.

augmix/augment_and_mix.py

Line 61 in 7c84885

depth = depth if depth > 0 else np.random.randint(1, 4)

So, we change code like below, right?

depth = np.random.randint(1, depth + 1 if depth > 0 else 4)

Best, regard.

Nan loss for ResNext backbone trained on cifar 100

Thank you for your work. While trying your code for the Resnext backbone on cifar100, I get nan values for the training loss. As mentioned in the published paper, I use the initial learning rate of 0.1 for SGD with cosine scheduling.

Little difference between with/without JSD loss

Thanks for your impressive contribution so much. I have one question when training this model.
When I set the no_jsd to True/False (cifar 100) using python cifar.py to get table 4, I got a 36.073(published 39.8) for no_jsd=True and 35.895(published 35.9). Do you have any idea why I got such a huge difference when setting the no_jsd to True?

CIFAR-10/ImageNet-P code

Hi, thank you for releasing the official implementation.

Do you plan to make public the codebase of CIFAR-10-P and ImageNet-P? Or are there official implementation for that elsewhere?

Thank you in advance.

I found an error

command for imagenet evaluation

Hi, do you have a command for imagenet evaluation on clean imagenet dataset and corrupted dataset? I have downloaded the pretrained weight but found it difficult to reproduce the result.

Thank you!

Reproduce of Baseline

no issue.

torch version 1.2.0 not compatible with GPU version

Can the torch version provided in the requirement be changed to the version?

KL divergenve

augmix/cifar.py

Line 233 in 9b9824c

p_mixture = torch.clamp((p_clean + p_aug1 + p_aug2) / 3., 1e-7, 1).log()

Hi.
KL divergence is indeed asymmetry distribution.
The code looks to minimize the mixture of distribution to p_original, p_aug1, p_aug2

The CIFAR-10-C dataset are not normalized

I found that clean training and testing images are normalized, but why the test images in CIFAR-10-C are not normalized? Will it influence the performance?

test_data.data = np.load(base_path + corruption + '.npy')
test_data.targets = torch.LongTensor(np.load(base_path + 'labels.npy'))
test_loader = torch.utils.data.DataLoader(
        test_data,
        batch_size=args.eval_batch_size,
        shuffle=False,
        num_workers=args.num_workers,
        pin_memory=True)

There is a big gap between the results of the code and the results in the paper on CIFAR10-C

I run this script several times to test the performance of proposed method on CIFAR10-C and CIFAR100-C:
python cifar.py -m resnext -e 200

10.9% error rate of CIFAR10-C is shown in the paper, however I got about 29% error rate.
Then, I tried different models, but there is still a big gap between the results of this code and the results in the paper on CIFAR10-C.

But the results on the CIFAR100-C are close to the paper.
BTW, I did not modify the code.

I would like to know:

1. Is the hyperparameter setting of cifar10 incorrectly given?
2. Are there more implementation details？

or could you please give some explanation?

THANKS A LOT!

Corruption acc. of a Resnet50 trained on cifar10 with augmix

Hi there, I trained a Resnet50 on CIFAR10 using the cifar.py script in this repository.

The clean acc. was about 95% but corruption accuracy was less than reported in the original paper. I have pasted below accuracies for CIFAR10-C at Severity 5. The mean corruption acc. across all corruptions and severity levels was 81%. I understand the architecture used in the original work was different, so is this an expected corruption acc. variation with the architecture?

Thank you.

Corruption severity: 5
gaussian_noise 0.6097
shot_noise 0.6647
impulse_noise 0.6747
speckle_noise 0.6906
defocus_blur 0.79
glass_blur 0.5383
motion_blur 0.7412
zoom_blur 0.7565
gaussian_blur 0.738
snow 0.7785
frost 0.7448
fog 0.6972
brightness 0.8831
spatter 0.8641
contrast 0.454
elastic_transform 0.6711
pixelate 0.5521
jpeg_compression 0.753
saturate 0.8935

A question in "third_party/WideResNet_pytorch/wideresnet.py out = self.relu2(self.bn2(self.conv1(out)))

The 42nd row in the code : out = self.relu2(self.bn2(self.conv1(out))), is it supposed to be out = self.relu2(self.bn2(self.conv1(x)))?

none

nothing

What about the performance on regular classification problems?

Can Augmix boost training on regular classification problems as Mixup or Cutmix do?

Use combined with Object Detection

Is it possible a ease-to-use/integration with object detection model like DETR ? I mean, if AugMix works only over the image so the bounding boxes wont be touched and no extra manipulations gonna be required for bounding boxes. Could you give me any hint, please.

augmentations used in augmix

Hi,

I have a couple of questions about the augmentations used in augmix -

1. The AugMix paper mentions that contrast augmentations were removed from augmix as that would overlap with one of the tested corruptions (Contrast) - but I see that AutoContrast is still used in the code: https://github.com/google-research/augmix/blob/master/augmentations.py#L141

2. I am curious how or why the augmentations in augmix impact performance on these corruptions as the connection between them is not immediately clear. Do you have a take on this, perhaps through an ablation study of the augmentations in augmix?

Thank you.

Repeated Evaluation in ImageNet

augmix/imagenet.py

Lines 485 to 489 in 7c84885

	corruption_accs = test_c(net, test_transform)
	for c in CORRUPTIONS:
	print('\t'.join(map(str, [c] + corruption_accs[c])))
	print('mCE (normalized by AlexNet):', compute_mce(corruption_accs))

in imagenet.py,
the evaluation of robustness is unnecessarily repeated in each iteration.

mean and std for every channel is 0.5 instead of (0.5071, 0.4867. 0.4408) as mean  and (0.2657, 0.2565, 0.2761) as std

Hi,
For normalizing cifar100 dataset, you used (0.5, 0.,5 0.,5) for mean and std instead of (0.5071, 0.4867. 0.4408) as mean and (0.2657, 0.2565, 0.2761) as std which is more commonly used. Is this by design? and if so, could you please explain in brief, as to why?

Jensen–Shannon divergence

Hello,

I'm trying to understand Jensen–Shannon divergence, I still don't understand the math behind it, but someone asked me to investigate about it and Augmix because of this paragraph:

Alternatively, we can view each set as an empirical distribution and measure the distance between
them using Kullback-Leibler (KL) or Jensen-Shannon (JS) divergence. The challenge for learning
with KL or JS divergence is that no useful gradient is provided when the two empirical distributions
have disjoint supports or have a non-empty intersection contained in a set of measure zero.

from here: https://arxiv.org/pdf/1907.10764.pdf

Is this problem presented in Augmix?

Testing with the best model

I found that cifar.py code does not test the performance with the best model.

augmix/cifar.py

Line 431 in 7c84885

test_c_acc = test_c(net, test_data, base_c_path)

In addition, why the validation set of CIFAR is not used in the model selection ??

About the loss function

Hello, I have read your paper and it's really impressive work. I also have a question about the loss function when I read the AUGMIX Pseudocode part. The question is why you didn't use the x_{augmix1} and x_{augmix2} as the input of the loss function L? It seems that you did not use data augmentation when training the model, augmix_data only works in the regularization term of Jensen-Shannon.

ImageNet hparams

I'm having trouble achieving decent ImageNet results with the mixing + JSD loss. Are the hparams in the imagenet.py script what was used for paper results, was the same code used for the paper?

Any details on hparams for the paper results for ImageNet would be appreciated. Are these correct?

• Epochs = 90 or 180?
• JSD loss lambda = 12
• batch size =256, that represents an effective batch size of 256*3=768, which is large for ResNet50 at FP32 and suggests 4+ GPU?
• LR = 0.1
• AugMix severity = 1
• AugMix prob coeff = 0.1

Thanks

Issue loading pretrained weights

When I download the checkpoint.pth.tar file for the pretrained ImageNet model I get the error
tar: Error opening archive: Unrecognized archive format, would it be possible to release the weights in another format (like a zip perhaps?) Unfortunately I don't have access to the ImageNet dataset so I can't retrain it myself