rodmarkun / tinny

Simple neural networks built from scratch using just raw Python and Numpy

TiNNy: Simple neural networks built from scratch with just Python and Numpy

TiNNy is a minimalist project aimed at demonstrating the power and potential of neural networks using nothing but Python and Numpy. This project is designed for educational purposes, allowing users to understand the fundamentals of neural network architectures, forward propagation, backpropagation, and training processes in a hands-on manner.

Dive Into Demos

TiNNy comes equipped with two sample Python notebooks. These demos are a great way to see TiNNy in action:

• demo_classification.ipynb: TiNNy classification project using the MNIST handwritten digit dataset and a Heart attack prediction dataset.
• demo_regression.ipynb: TiNNy regression project using the Boston housing price and the Diamond price prediction datasets.

Math explained

We begin by having an input of features X and an input of labels y, as neural networks are part of supervised learning. The goal is to learn a function that maps inputs to outputs, minimizing the error between the predicted outputs and the actual labels. This process involves several steps, prominently including the forward pass, backpropagation, and parameter update. Here's how each step works mathematically:

Forward

In the forward pass, the network applies a series of transformations to the input data to compute the output predictions. This involves computing the linear combination of inputs and weights, adding a bias, and then applying an activation function. The process for layer i can be visualized as:

Here, L^i represents the linear combination at layer i, w^i and b^i are the weights and bias for layer i, respectively, and A^(i-1) is the activation from the previous layer or the input data for the first layer. A^i is the output of the activation function applied to L^i, moving the data forward through the network.

Backpropagation

Backpropagation is the process of adjusting the network's weights and biases to minimize the error between the actual labels and the predictions. It involves two key steps: computing the gradients for the output layer and then for the dense (hidden) layers.

Backpropagation in Output Layer

The gradient of the loss with respect to the output layer's weights, biases, and activations are computed as follows:

dL^k denotes the derivative of the loss with respect to the activations of the last layer, dW^k and db^k are the gradients of the loss with respect to the weights and biases of the last layer, respectively. A^k is the activation of the last layer, and y is the true labels.

Backpropagation in Dense Layers

For each of the preceding layers, the gradients are computed taking into account the derivative of the activation function to propagate the error backward through the network:

Here, g'(L^i) represents the derivative of the activation function applied at layer i, allowing the gradient of the loss to be propagated back through the network.

Update Parameters

Finally, the parameters of the network are updated using the gradients computed during backpropagation. This step adjusts the weights and biases in the direction that minimally reduces the error:

W^i and b^i are updated by subtracting a fraction of the gradients dW^i and db^i, scaled by a learning rate alpha. This iterative process of forward pass, backpropagation, and parameter update continues until the model's performance reaches a satisfactory level.

Getting Started with TiNNy

Start using TiNNy by setting up the project on your personal computer. Follow these simple steps to get started:

First, clone the repository to your local machine using Git. Open your terminal and run the following commands:

git clone https://github.com/rodmarkun/tiNNy
cd tiNNy

TiNNy requires certain Python packages to function properly. Ensure all dependencies are installed by executing the following command in the terminal:

pip install -r ./requirements.txt

With the installation complete, you're ready to incorporate TiNNy into your Python scripts or Jupyter notebooks. Simply import the library using:

import tinny

You are now equipped to explore the features and capabilities of TiNNy. Happy coding!