roatienza / straug

Image transformations designed for Scene Text Recognition (STR) data augmentation. Published at ICCV 2021 Workshop on Interactive Labeling and Data Augmentation for Vision.

Data Augmentation for Scene Text Recognition

(Pronounced as "strog")

Paper

• ICCV 2021 Workshop
• Arxiv

Why it matters?

Scene Text Recognition (STR) requires data augmentation functions that are different from object recognition. STRAug is data augmentation designed for STR. It offers 36 data augmentation functions that are sorted into 8 groups. Each function supports 3 levels or magnitudes of severity or intensity.

Given a source image:

it can be transformed as follows:

1. warp.py - to generate Curve, Distort, Stretch (or Elastic) deformations

Curve	Distort	Stretch

2. geometry.py - to generate Perspective, Rotation, Shrink deformations

Perspective	Rotation	Shrink

3. pattern.py - to create different grids: Grid, VGrid, HGrid, RectGrid, EllipseGrid

Grid	VGrid	HGrid	RectGrid	EllipseGrid

4. blur.py - to generate synthetic blur: GaussianBlur, DefocusBlur, MotionBlur, GlassBlur, ZoomBlur

GaussianBlur	DefocusBlur	MotionBlur	GlassBlur	ZoomBlur

5. noise.py - to add noise: GaussianNoise, ShotNoise, ImpulseNoise, SpeckleNoise

GaussianNoise	ShotNoise	ImpulseNoise	SpeckleNoise

6. weather.py - to simulate certain weather conditions: Fog, Snow, Frost, Rain, Shadow

Fog	Snow	Frost	Rain	Shadow

7. camera.py - to simulate camera sensor tuning and image compression/resizing: Contrast, Brightness, JpegCompression, Pixelate

Contrast	Brightness	JpegCompression	Pixelate

8. process.py - all other image processing issues: Posterize, Solarize, Invert, Equalize, AutoContrast, Sharpness, Color

Posterize	Solarize	Invert	Equalize

AutoContrast	Sharpness	Color

Pip install

pip3 install straug

How to use

Command line (e.g. input image is nokia.png):

>>> from straug.warp import Curve
>>> from PIL import Image
>>> img = Image.open("nokia.png")
>>> img = Curve()(img, mag=3)
>>> img.save("curved_nokia.png")

Python script (see test.py):

python3 test.py --image=<target image>

For example:

python3 test.py --image=images/telekom.png

The corrupted images are in results directory.

If you want to randomly apply only the desired augmentation types among multiple augmentations, see test_random_aug.py

Reference

• Image corruptions (eg blur, noise, camera effects, fog, frost, etc) are based on the work of Hendrycks et al.

Citation

If you find this work useful, please cite:

@inproceedings{atienza2021data,
  title={Data Augmentation for Scene Text Recognition},
  author={Atienza, Rowel},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
  pages={1561--1570},
  year={2021}
}

straug's Issues

(Request) Add a parameter for borderValue using in warp (TPS transforms)

As per title, it would be much convinient if you provide a seperate parameter for borderValue using in cv2 warpImage method. The default value is 0, i.e. the black fill color.

Questions about paper

First of all, thank you for your great work.
I read the paper and met 2 problem：
why there is no CRNN line in figure 13？

And what's the N corresponding to table 5？Is it the best result in a grid search?

How to deal with underfitting?

Hello, I am a fresh researcher and recently I noticed your code which is very useful to solve my problem to some extent. My project is also scene text recognition while the dataset is much more irregular. I think your paper give me a constructive guidance. However, there is still some problems that when the N(number of functions in each channel) is going to be larger, maybe 3 or 4, the model preforms hardly to be fitted. the accuracy of training set is always surrounding about 90%. For more, if I add a preprocessing method of random cut, the accuracy is always surrounding about 80%. Could you give me some suggestions to deal with such problems? Thanks.

CRNN Data enhancement

Can you provide some methods for crnn data enhancement？

speed is very low

The speed is very low, how to deal with it?

how to glared image

@roatienza thank for great repo.
I want to aug image like that:

and how to aug. I think it is: "glared image"
Thank for your help.

question about training speed

thanks for your excellent job! it seems that the training is very slow when i use the straug(6x times slower than that without straug). What about the real speed when you test? The following is my aug-code.

class RecStraugRandAug(object):
    def __init__(self, num_aug=2, prob=0.5, **kwargs):
        super().__init__()
        self.num_aug = num_aug
        self.prob = prob
        try:
            from straug.blur import GaussianBlur, DefocusBlur, MotionBlur, GlassBlur
            from straug.camera import Contrast, Brightness, JpegCompression, Pixelate
            from straug.geometry import Perspective, Shrink, Rotate
            from straug.noise import GaussianNoise, ShotNoise, ImpulseNoise, SpeckleNoise
            from straug.pattern import Grid, VGrid, HGrid, RectGrid, EllipseGrid
            from straug.process import Posterize, Solarize, Invert, Equalize, AutoContrast, Sharpness, Color
            from straug.warp import Stretch, Distort, Curve
            from straug.weather import Fog, Snow, Frost, Rain, Shadow
            self.augs = [
                [GaussianBlur(), DefocusBlur(), MotionBlur(), GlassBlur()],
                [Contrast(), Brightness(), JpegCompression(), Pixelate()],
                [Perspective(), Shrink(), Rotate()],
                [GaussianNoise(), ShotNoise(), ImpulseNoise(), SpeckleNoise()],
                [Grid(), VGrid(), HGrid(), RectGrid(), EllipseGrid()],
                [Posterize(), Solarize(), Invert(), Equalize(), AutoContrast(), Sharpness(), Color()],
                [Stretch(), Distort(), Curve()],
                [Fog(), Snow(), Frost(), Rain(), Shadow()],
            ]
        except Exception as ex:
            print(f"exception: {ex}, you can install straug using `pip install straug`")
            exit(-1)
    
    def __call__(self, data):
        img = Image.fromarray(data["image"])
        for idx in range(self.num_aug):
            aug_type_idx = np.random.randint(0, len(self.augs))
            aug_idx = np.random.randint(0, len(self.augs[aug_type_idx]))
            img = self.augs[aug_type_idx][aug_idx](img, mag=random.randint(-1,2), prob=self.prob)
        data["image"] = np.array(img)
        return data

enhance a dataset

Thanks for your work! How to perform batch operations? I want to enhance a dataset.

Gaussian blur kernel size for small images

Currently, the kernel size is fixed to 31x31 (

straug/blur.py

Line 38 in 43f9ca9

kernel = (31, 31)

)

This causes an error internally in the call to reflection_pad2d():
RuntimeError: Argument #4: Padding size should be less than the corresponding input dimension, but got: padding (15, 15) at dimension 3 of input 4

if one of the image's dimensions is less than the kernel size.

Should the kernel size be a percentage of the image's dimensions instead, e.g. 30-50% of the smaller dimension?

RandAugment

hi thanks for your work

are you implement the RandAugment?

in your paper:

geometry = [Rotate(), Perspective(), Shrink()]
noise = [GaussianNoise()]
blur = [MotionBlur()]
augmentations = [geometry, noise, blur]
img = RandAugment(img, augmentations, N=3)