guorbit / utilities

This repo is intended to be used for the guorbit software team as a utility library. See documentation below

Note before installation: None of these commands have been properly tested. Make sure you installed the package in a virtual environment.

Installation

For installing the utilities repo as a package use the following commands in the terminal: Note: you need to have a working ssh key to access github from your current machine. you need to have wheel installed.

pip wheel . --no-deps --wheel-dir=dist

After build run the following command to install, the built package.

pip install .

Preparing for development

Prerequisites

It is recommended to use a virtual environment for development. To create a virtual environment, with python 3.10 as of now. To create a virtual environment run the following command in the terminal with anaconda:

conda create -n <env_name> python=3.10

Then activate the virtual environment by running the following command:

conda activate <env_name>

The rest of the dependencies are in the pyproject.toml file. To install those dependencies run the following command in the terminal:

pip install .[dev]

You might potentially have to build the project as well. If yes in that case run the following command:

pip wheel . --no-deps --wheel-dir=dist

Additionally, it is recommended to use certain extensions for development(in case you are using vs code). These extensions are listed as recommended in the utilities.code-workspace file. To install these extensions, open the utilities.code-workspace file in vs code and click on the install button when prompted. This also enables other checking tools like linting and type checking.

Project branching strategy

The project is using a staging branching combined with feature branching strategy. The main branch is the production branch. The staging branch is the staging branch meant for advanced longer testing. The development branch is the development branch with experimental features and less reliability. The feature branches are used for new feature development. The feature branches are branched off from the development branch. The development branch is branched off from the staging branch. At release the development branch is merged into the staging branch. The staging branch is merged into the main branch, if the branches pass sucsessfully the CI pipeline. - Automated build testing is still under considerationd

The following diagram shows the branching strategy:

Main ------------------------------------------> 
                                             ^  
                                            /
Staging -----------------------------------/--->
                                        ^
                                       /
Development --------------------------/-------->
                \                    /
                 \-Feature branches-/

Testing

To run the tests run the following command in the terminal:

pytest

If the above command doesn't work, run the following command:

python -m pytest

In case you want to run a coverage test, run the following command:

python -m pytest --cov --cov-report term

Linting

To run the linting tests run the following command in the terminal (this also loads in the linting configuration stored in the project root):

pylint --rcfile=.pylintrc utilities

Future plans (Ever expanding list)

• Docker build and deployment for the package, to docker hub.
• PyPi build and deployment for the package.
• Additional CI pipeline for staging environment.
• Mutation testing.
• Tensorboard building package
• Scalable ML data reading pipeline
• Model conversion pipelines
• Image manupulation tools
• Image augmentation tools

Submit a suggestion for a new feature by creating an issue, or if you already have it done by creating a pull request to development.

utilities's Issues

Flowreader exp incorrect indexing

The indexing inside the Flowreader exp, incorrectly calculates the validity index such that rarely updating data

Migrate setup.py to pyproject.toml

Migrate setup.py to pyproject.toml. This will allow a more unified packaging experience. It also provides greater management over version variables and things like that.

• #54
• #55
• #56
• #57
• #58

Migrate sphinx to use package version instead of hardcoded data

Image augmentation pipeline, applies random flips incorrectly

Happens due to the mask being flattened, such the same random transforms don't apply to the masks what apply to an image.

confusion matrix (per epoch/entire training)

MongoDB

master environment

Environment containing main code, this is where the main package is deployed after approvals from staging. Should require more approvals than staging

Set up branch protection etc for land-class repo

The new flowreader stores some information as a class variable rather than instance variable

Add function to convert model into .pb model for usage in tf lite

Migrate setup.py to pyproject.toml

merge cloudbase-iteration-too repo into utilities

Add version and similar data to package directory

Local storage loader

Create new implementation of flow from directory

Change from the current TensorFlow hacky implementation of flow from directory to new implementation, for better customizability

Image cutter crashes if image can be cut perfectly

Enable usage of data loader classes

In the experimental flow reader enable the usage of classes implementing the data loader interface providing flexibility over the type of storage the data is read from

• #33
• #34

staging environment

Staging environment is the prerelease version of the library, passing the environment deploys a nightly version of the library, such that it is updated much faster than the normal version.
Additional testing required, with more realistic scenarios for testing actual functionality.

Functions sent to augmentation pipeline proved inconsistent

Some functions such as tf.image.random_flip didn't prove consistent such that it required a custom implementation

flowreader_exp incorrect dataset size calculation

uses batch size to determine the dataset size instead of minibatch

merge software_template and utilities repo

Add mini batch functionality

Add mini batch functionality to the flowreader.py API, such that custom training functions can set the desired minibatch size for training

Update dev dependencies to match all required dev dependencies

Migrate docker file from software-base to utilities

Migrate docker file to repo.
Will be used for future docker builds

Add tensorboard class using module based system

The class would allow the generation of a tensorboard instance with user specifiable features

• #44
• #45
• #46

Add image preprocessing pipeline to utilities repo

MySQL

SQLite

development environement

The environment where all the under-development code is deployed to. Runs the standard lightweight tests to accelerate testing time

Flow reader, sends wrong reshape size to the augmentation pipeline

Results in crash

• fix problem in augmentation pipeline
• fix tests to past on the updated augmentation pipeline

Setup CI/CD for utilities to auto push on PyPi

This also requires, correct branching strategy.
Proposed strategy:

• Feature branching around the master branch
• Additional branch for deployment, as a staging branch

Sphinx documentation for utilities repo

Database system loader

Merge all utilities under utilities repo

Add documentation for reading strategies

Add documentation to reading strategies that can be auto deployed from github

Enable async data loading

In the experimental flow reader add option to async load the next batch providing more seamless training

prediction over validation dataset (can the number of features shown in figure depending on implementation)

Add data reader heads

Would provide custom classes for reading specific datatypes. This provides two benefits:

1. More maintainable code as it is abstracted out
2. Way more flexible reader function

Set up branch protection etc for utilities repo

Database logger class

Would provide an easy implementation of connecting to a database depending on choice

• #48
• #49
• #50
• #51

Setup environments for separate staging branches

environments to setup:

• #62
• #63
• #64

this also requires specific rules and tests to run on the separate branches

Clip data after augmentation

After the augmentation is done values have to be clipped such that no data is over 1 or below 0

Issue with loading batch into datareader

Describe the bug
Flow generator experimental can't load partial batches (eg.: when its at the end of the dataset)

To Reproduce
Steps to reproduce the behavior:
set a batch size that is not divider of the dataset size, then start a training

Expected behavior
Load a partial batch

Please complete the following information:

• TF 2.13
• UTILS (2.0)

Additional context
Add any other context about the problem here.

Migrate build to use package version instead of hardcoded data

Remove unnecessary reshape operations to the end. Reduces complexity

Describe the bug
There are a huge number of unnecessary flattening and unflattening operation in the flowgenerator_exp, which could be reduced to only one reducing complexity and possibly computational time

To Reproduce
Just look at source code

Expected behavior
None

Screenshots
No Screenshots

Please complete the following information:

• Library version [e.g. 0.1]

Add functionality to calculate class distribution stats on per image bases

Add functionality to calculate class distribution stats on per image bases. Required for a possible fix on class imbalance on segmentation tasks

Add segmentation image data generator to utilities repo

PostgreSQL

logging hyperparameter settings if possible (if not no problem as it is logged in db as well)

Add explanation of purpose and functionality of all tools in this repo