Hello from Imgo! This is a blog series about my largest and most complex project to date, a home-made library, distributed through PyPi, whose aim is to streamline and facilitate various aspects of the data processing phase of image classification projects. In this series, I hope to document the ups and downs of building a library from scratch.
Recap
In the previous post, I discussed the tweaks and changes that I had implemented in both the uptools and augtools modules. These principally revolved around streamlining the code pipeline in order to remove significant bottlenecks. I have since released imgo 2.1 which deploys these changes, and is mostly ready for my intended usage, but still has room for improvement in some areas.
Progress Update
Since the last post, I have mainly been working with augtools in order to get it ready for its ultimate purpose: generating additional images for the DeepShroom dataset.
The first point to mention is that I noticed how significantly the original image size affects the quality of the augmentation. This can be seen in the below image, which in its original form has dimensions of 82 by 91 pixels. When augmented, the result is generally a blurry mess that is not particularly useful:
However, by creating an automatic threshold rescaling function, I was able to upscale all of the images in the dataset with equivalent square dimensions less than 300 pixels and rescale them up to a 300 by 300 square image. The resulting image fed through the same augmenter looks like this:
Because many of the transformations operate pixel by pixel, it makes a big difference how many pixels there are to work with. So even if the original image is unchanged in terms of resolution, the augmented images are much more discernible in terms of patters and borders. This is of course very useful for the image classification task (still down the pipeline).
I have also been experimenting with usable parameters for the augmenter itself, and I am gradually honing in on a good set of parameters to use. Compare, for example, the scale used for the Piecewise Affine transformation below:
It is clear from the above that a lower bound of 0.01 is too low as it does not seem to do anything, and that an upper bound of 0.25 is too high, as it distorts the images unrecognisably.
Changing the range from [0.05,0.09] instead, I am able to generate some more realistic images which are very different from the original, as we can see:
I have iterated this process for all of the transformations, and have come up with the following parameter values:
augtools.Augmenter(pwa_scale=(0.06,0.085),
clip_limit=(1,5),
b_sev=3,
x_scale=(0.7,1.3),
y_scale=(0.8,1.15),
edge_mode="reflect",
g_sev=1,
e_sev=3,
shear_range=(-35,35),
rotate_range=(-35,35),
x_shift=(-0.05,0.05),
y_shift=(-0.1,0.1),
dropout_pair=(0.15,0.85),
h_flip=0.5,
v_flip=0.5,
contrast=True,
sharpness=True,
randomize_batch=True)A few things are worth noting here. First of all, many of the values are significantly lower (or less severe) than I had originally thought they would be. This is because the distortion effect compounds rather dramatically as the number of transformations increases. This is especially true for the transformations that create blank pixels, such as the rotate, shear, scale, and shift functions. All of these create additional pixels that need to be filled in some way, and when these pixels are subjected to augmentation transformations, the results are often extremely noisy.
This leads me on to the second noteworthy point. In its current live version, imgo.augtools does not let the user choose how to fill in blank pixels created by transformations. It automatically implements the "edge" method from Imgaug, which makes the images rather noisy (especially when further transformations occur after the pixels have been filled). I have therefore implemented the option (as can be noticed from the arguments in the constructor above) to change the edge fill mode. Imgo now includes Imgaug's "constant" and "reflect" methods, which are illustrated below:
The combination of these changes and the parameter optimization results in the following:
I still do not think that these augmented images are fit for purpose, but they are getting closer.
In Development
An use-case issue I have noticed with augtools is that, as in the above illustration, there is no way to obtain sufficiently reproducible results because the augment_flow method always applies the augmentations in a random order. This is good for general augmentation tasks, but if the task is to generate new images it would be useful to be able to specify the order in which the transformations are applied, because it can lead to drastically different results.
I am therefore currently working on a simple, user-friendly, and pythonic way to specify the order of the functions in the augmentation run. Ideally, this will be given by the order in which the parameters are given in the constructor of the augmentation object. Watch this space!
