Using Augmentor with TensorFlow about augmentor

[mdbloice]

Hi there, as I do not have TensorFlow currently installed on my main dev machine, I can't really add this functionality at present. However, it is something I definitely want to add, so I will take a look at doing this some time soon. I can't make any promises regarding when this might be unfortunately. That said, I'd be happy to take a look at a PR you might want to make!

[devforfu]

@benjaminbarbe If I have correctly understood your question, I guess that there is no too much difference in case of tensorflow. At least, when you're writing a training loop by hand, like in this snippet (probably has typos, but I guess you've got the idea):

from Augmentor.Operations import RotateRange
from PIL import Image

op = RotateRange(probability=0.5, max_left_rotation=30, max_right_rotation=30)

with tf.Session() as session:
    tf.global_variables_initialized().run()
    for epoch in range(n_epochs):
        for x_batch, y_batch in training_batches:
            pil_images = [Image.fromarray(x) for x in x_batch]
            augmented = op.perform_operation(pil_images)
            arrays = [np.asarray(img, dtype=float) for img in augmented]
            sess.run([train_op], feed_dict={x: x_batch, y: y_batch})

Probably there are some helpful utils in the library to make this process more simple. Though if you want to augment images right on GPU, then you need something more involved.

[johndpope]

you may like miniconda to setup tensorflow with cpu / very simple to get up and running.
https://gist.github.com/johndpope/187b0dd996d16152ace2f842d43e3990

[benjaminbarbe]

Hello @mdbloice

Do you have some hints to get started?

[alshedivat]

Tensorflow has the standard data processing API. I'm interested to interface Augmentor with the API.

As I can see, Augmentor is implemented entirely in Python. It should be possible to use tf.py_func to wrap Augmentor's functionality and provide data mappers that can be used with tf.data.Dataset.

@mdbloice, I'm not very familiar with Augmentor, so what would be a good point to start? Also, can Augmentor process images on a GPU?

[johndpope]

I came across this repo the other week - https://github.com/tueimage/SE2CNN
it will build a roto-translation covariant convolutional network with tensorflow. (so you don't need to add extra images to training augmentation - the neural net will allow invariants in rotation / translation) https://arxiv.org/pdf/1804.03393

[johndpope]

fyi - Jonas Adler @adler-j has created a tensorflow example for deforming images with his own function. There's a tensorflow sample. Should be able to cherry pick / glue tensorflow code https://github.com/adler-j/tfdeform

[mdbloice]

Hi @alshedivat, sorry for the delay in replying, work duties have kept me from doing any work on Augmentor for the past few weeks. Actually, what I am working on next is proper support for TensorFlow. Right now, using Augmentor with Keras should work, and PyTorch should also work. Next step is to get Augmentor working with TF properly without workarounds. Expect it in the next release (although I cannot say for sure when that might be, unfortunately).

[alshedivat]

@mdbloice, cool, thanks for your reply! Do you have a roadmap for TF support? I'm pretty familiar with TF and it's Dataset API, so happy to contribute if that accelerates development.

Also, it looks like Augmentor doesn't use opencv, does it? Another image augmentation library shows that using highly optimized opencv gives best throughput.

[mdbloice]

HI @alshedivat, no I do not really have a roadmap right now for TF support, I am only able to work on Augmentor when time allows. And it's not really predictable when that might be, so I cannot say for sure when I might be able to work on new features or bugs unfortunately!

As for OpenCV, no I do not use it because it is a rather large dependency that I wanted to avoid. The library you linked to looks good though! I think with multi-threading the throughput in Augmentor is pretty good.

As for contributing, that would be great - I presume TF expects a generator of some kind, and if you check out the code for the keras_generator() function, you'll see how it works:

Augmentor/Augmentor/Pipeline.py

Lines 456 to 543 in 5a8f6d6

	def keras_generator(self, batch_size, scaled=True, image_data_format="channels_last"):
	"""
	Returns an image generator that will sample from the current pipeline
	indefinitely, as long as it is called.
	.. warning::
	This function returns images from the current pipeline
	**with replacement**.
	You must configure the generator to provide data in the same
	format that Keras is configured for. You can use the functions
	:func:`keras.backend.image_data_format()` and
	:func:`keras.backend.set_image_data_format()` to get and set
	Keras' image format at runtime.
	.. code-block:: python
	>>> from keras import backend as K
	>>> K.image_data_format()
	'channels_first'
	>>> K.set_image_data_format('channels_last')
	>>> K.image_data_format()
	'channels_last'
	By default, Augmentor uses ``'channels_last'``.
	:param batch_size: The number of images to return per batch.
	:type batch_size: Integer
	:param scaled: True (default) if pixels are to be converted
	to float32 values between 0 and 1, or False if pixels should be
	integer values between 0-255.
	:type scaled: Boolean
	:param image_data_format: Either ``'channels_last'`` (default) or
	``'channels_first'``.
	:type image_data_format: String
	:return: An image generator.
	"""
	if image_data_format not in ["channels_first", "channels_last"]:
	warnings.warn("To work with Keras, must be one of channels_first or channels_last.")
	while True:
	# Randomly select 25 images for augmentation and yield the
	# augmented images.
	# X = np.array([])
	# y = np.array([])
	# The correct thing to do here is to pre-allocate
	# batch = np.ndarray((batch_size, 28, 28, 1))
	X = []
	y = []
	for i in range(batch_size):
	# Pre-allocate
	# batch[i:i+28]
	# Select random image, get image array and label
	random_image_index = random.randint(0, len(self.augmentor_images)-1)
	numpy_array = np.asarray(self._execute(self.augmentor_images[random_image_index], save_to_disk=False))
	label = self.augmentor_images[random_image_index].categorical_label
	# Reshape
	w = numpy_array.shape[0]
	h = numpy_array.shape[1]
	if np.ndim(numpy_array) == 2:
	l = 1
	else:
	l = np.shape(numpy_array)[2]
	if image_data_format == "channels_last":
	numpy_array = numpy_array.reshape(w, h, l)
	elif image_data_format == "channels_first":
	numpy_array = numpy_array.reshape(l, w, h)
	X.append(numpy_array)
	y.append(label)
	X = np.asarray(X)
	y = np.asarray(y)
	if scaled:
	X = X.astype('float32')
	X /= 255. # PR #126
	yield (X, y)

If you can write a function like tf_generator or something similar that would output images in the format that is expected by TensorFlow, then I think that would be all there is to it! Let me know what you think.