finlaymorrison / bunjilearn

Machine learning library written from scratch in C++

BunjiLearn

---

BunjiLearn Logging Level Reference

Standard Logging Level	BunjiLearn Logging Level	Log Colour
BUNJI_DEBUG	BUNJI_PURR	BLUE
BUNJI_INFO	BUNJI_MEOW	WHITE
BUNJI_WARNING	BUNJI_HISS	YELLOW
BUNJI_ERROR	BUNJI_BITE	RED

bunjilearn's Issues

Clean up neural network operators

Activations currently only consider the first element of the deepest dimension in calculating the output and gradient, these should be generalized to calculate the activations and gradients of any input in any of the 3 dimensions. Loops also currently make use of int, however std::size_t should be used instead.

Implement the crossentropy loss function

Implement the crossentropy loss function as defined by ml-cheatsheet.

Convert training from full-batch to mini-batch

Currently the Trainer::fit function only implements full-batch gradient descent. This function should take in an integer which defines the batch size to use when training. This batch size can be set to the dataset size to give full-batch gradient descent. This change will open up the possibility of making the optimiser configurable in a later change.

Make all neural net operator objects derive from single class

All neural net operator objects, networks, layers, and activations, should derive from a base class. From this base class the network and layers will be derived, and activations will then be derived from the layer class. The network should then use a vector of this abstract trainable object class instead of layers.

This will allow for models to scale better as you can assemble multiple networks into a single, larger, network.

Restructure project directory into several subdirectories

Add a subdirectory for each of the main components of the project, e.g:

• Layers
• Loss
• Dataset
• Framework

Each subdirectory should come complete with a CMake file which will expose the relevant directories to a calling CMake file.

Implement layer build virtual function

Layers should have a function for initializing their parameters which can be called after their contruction. There should be constructors available for layers that can also initailize the parameters; however, this functionality should be implemented in the build function.

There should also be functionality to pass in unbuilt layers into a network object in the add_layer function, where they will then be built, given that the previous layer output shape is known.

Derive the loss subclass from a metric class and add in an accuracy metric

There should be a parent abstract class 'Metric' which defines a pure virtual function for obtaining a value from a metric. The Loss class should derive from the 'Metric' class which then declares another pure virtual function for obtaining the derivative of the loss function.

Effectively, differentiable metrics will be implemented as a loss function which can still be used as a model metric but can also be a training loss function.

Arrange sections of library into namespaces.

The entirety of the library should be wrapped within a namespace, and each part of the library should be wrapped within a sub-namespace. e.g:

bunji::layer::Dense();
bunji::metric::Crossentropy();
bunji::model::Trainer();

Create a class for handling console output

A class for handling different types of console output should be created which provides functionality for supressing different output types through CMake definitions depending on what is required in a build.

Make trainer use testing and validation dataset partitions

During training the given metrics should be evaluated against both the training and validation datasets. An additional function should also be provided on the Trainer class which evaluates the performance of the model against the metrics on the testing dataset.

Implement console output to provide training feedback

The training loop function should implement a progress bar which gives information on how far through the dataset the current epoch of training is using a progress bar. Additionally, there should be a way to specify the metrics of interest to be trained, such as loss and accuracy.

Add helper functions for working with tensors

Functions should be added to make working with tensors more clean, particularly initialization to a specific size.

Implement a larger variety of layers

The current layer set is rather limiting. The following layers should be implemented.

• Pooling (Max, Average)
• Convolutional
• Dropout
• Batch Normalisation

Add an option to attach an activation directly to a layer

An activation should be able to be passed into the contructor of any layer. This will help make model definitions less verbose and will improve readability of code using this library. This should be optional and should default to no activation function.

Add python API.

A Python api should be implemented with PyBind11 which exposes the high-level functionality of the library. An interface to construct all layers and metrics should be made, as well as an interface for interacting with the trainer. The dataset will require some functions to exposed also.

Check condition in line

BunjiLearn/bunjilearn/model/trainer.cpp

Line 45 in 5c5c757

if (training && ((i+1) % 32 == 0))

The constant value of 32 may need to be the parameter 'batch_size'

Implement iterators on the Tensor class and make use of STL algorithms.

Iterators should be provided on the Tensor, Tensor<>::View, and Tensor<>::ConstView classes. These iterators should be compatible for use with STL algorithms. Implement the current functionality using STL algorithms wherever possible.