Hello from Imgo! This is a blog series about my largest and most complex project to date, a home-made library, distributed through PyPi, whose aim is to streamline and facilitate various aspects of the data processing phase of image classification projects. In this series, I hope to document the ups and downs of building a library from scratch.
The Story So Far
In this series, I have been documenting my journey through all the phases of the development of a Python library. From initial deployment, to scaling, debugging, and streamlining, I have been working towards the goal of building a powerful and efficient library to help with computer vision projects.
In the last post, I discussed the issues I had with scaling the project and working with large datasets. I had found several bottlenecks and inefficiencies in the source code, and was able to make some headway in resolving them.
Progress Report
After making some additional tweaks, I am pleased to announce that imgo Version 2 is now live and available for download via Pip!
There have not been that many changes in the code since the last version. Apart from some pythonification, the main changes have been as follows:
First, I have removed the option to save datasets as arrays to disk in multiple formats, and restricted the save format to h5. This is because of the vastly superior performance in terms of save and load time, as well as resulting file size, discussed in the previous post.
Secondly, I have decided to implement mandatory image rescaling when initializing an Image_Dataset object. This is because saving in h5 format requires the arrays to be of consistent dimensions across all axes, and since I had decided to remove the option to save as numpy arrays, leaving the user open to initialize the dataset without being able to save it seemed like a user-unfriendly feature. There is, in the end, a tradeoff between flexibility and efficiency in this case, and because most image classification projects will involve tasks that require consistent image dimensions, it seemed like an obvious choice.
Current Working Results
As I mentioned in the previous post, the real-world test case for this library is my mushroom image database, comprising 10,000 images or so across 20 classes. I was able to use the augtools module to remove the class imbalance and effectively double the dataset size. I look forward now to honing the augmentation parameters in order to generate more organic-looking images, and to train a classifier on the newly expanded dataset.
Below is a sample of the augmenter on the Boat/Car/Helicopter dataset included in the imgo repo:
Performance in the sister library, uptools, has also improved through the streamlining of calculations documented in the previous post. The full mushroom set (including the augmented images) takes around 3 minutes to initialize as an Image_Dataset object at an image scale of 200x200 pixels without any statistical reduction of pixel values, approximately 4 minutes and 45 seconds if normalizing. Standardization takes much longer, in the order of 10 minutes in total (including initialization and splitting time). It therefore still takes a reasonably long time, but since these operations (including the normalization or standardization) now only occur once, it no longer seems to such a problem as to eliminate the module’s usefulness entirely.
And so there we have it! Imgo is live and ready to be tested by friendly strangers. My next step is to identify optimal parameter ranges for the augtools augmenter to generate usable mushroom images.
In the meantime, you are invited to play around with imgo! Install the latest version by running pip install imgo and have a look at the demos in the GitHub repo. Any feedback is greatly appreciated, and I can be reached at info@elbydata.com.
Happy machine learning!
