The purpose of this repository is to serve as a practical material for teaching students the fundamentals of neural network structure and design.
At the moment there are two main components to the repository:
Contains an implementation of a basic neural network library supporting both forward and backward propagation. The library is inspired by PyTorch -- a popular ML framework and can be treated as a very simplified version of it. All operations are essentially performed on NumPy arrays.
For education purposes some methods implementations are removed and students are tasked to implement those methods themselves. This way the package is only a template of an ML framework. Implementing the missing logic should be a valuable exersice for the students. On the other hand, the logic that is kept should ease the burden of implementing everything by themselves and focus students only on the core components responsible for neural network inference and training.
nn_lib.math_fns
implements the expected behaviour of every supported mathematical function during both forward (value) and backward (gradient) passesnn_lib.tests
contains rich test base target at checking the correctness of students' implementationsnn_lib.tensor
is the core component ofnn_lib
, implements application of math operations on Tensors, and gradient propagation and accumulationnn_lib.mdl
contains an interface of a Module class (similar totorch.nn.Module
) and some implementations of itnn_lib.optim
contains an interface for an NN optimizer and a Stochastic Gradient Descent (SGD) optimizer as the simplest version of itnn_lib.data
contains data processing -related components such as Dataset or Dataloader
An example usage of nn_lib
package for the purpose of training a small Multi-Layer
Perceptron (MLP) neural network on a toy dataset of 2D points for binary
classification task. Again some methods implementations are removed to be implemented
by students as an exercise.
The example describes a binary MLP NN model (toy_mlp.binary_mlp_classifier
),
a synthetically generated 2D toy dataset (toy_mlp.toy_dataset
), a class for
training and validating a model (toy_mlp.model_trainer
) and the main execution
script (toy_mlp.train_toy_mlp
) that demonstrates a regular pipeline of solving a
task using machine learning approach.
- Clone this repository
- Create a new private repository for yourself in GitHub
- Set up two remotes: the first one for this repo, the second one for your repo
- Invite me to the repository, I will review your progress
Methods marked with a comment TODO: implement me as an exercise
are for you to
implement. Most of the to-implement functionality is covered by tests inside
nn_lib.tests
directory.
Please note that all the tests should be correct as there exists an implementation that passes all of them. So do not edit the tests unless you are totally sure and can prove that there is a bug there.
At the end all the test must pass, but the recommended order of implementation is the following:
.forward()
methods for classes insidenn_lib.math_fns
(test_tensor_forward.py
).backward()
methods for classes insidenn_lib.math_fns
(test_tensor_backward.py
)- modules functionality inside
nn_lib.mdl
(test_modules.py
) - optimizers functionality inside
nn_lib.optim
(test_optim.py
) - MLP neural network methods inside
toy_mlp.binary_mlp_classifier.py
- training-related methods inside
toy_mlp.model_trainer.py
If everything is implemented correctly the toy MLP example should be able to be trained
successfully reaching 95+ % validation accuracy (toy_mlp.train_toy_mlp.py
) on all three
of toy datasets.
After (1) completion of this part, (2) review of your changes by me and (3) 1:1 discussion for checking the understanding, you will receive X points (to be decided).
python3 -m venv env
source env/bin/activate
python3 -m pip install -r requirements.txt
python -m venv env
.\env\Scripts\activate.bat
python -m pip install -r requirements.txt
deactivate
pip freeze > requirements.txt