rorybyrne / neuralnotwork

A Neural Network implemented from scratch

NeuralNotWork

A Neural Network implemented from scratch

TODO

Fix backpropagation Re-add bias Tests Add a wiki

neuralnotwork's Issues

Re-implement bias

My old implementation of bias proved to be a nightmare when trying to implement backpropagation. This is partly because of what is described in #8.

Time for attempt number 2 at biases.

Network model design

We can discuss here different ways to design the model of the network itself. For example, should a layer be represented as an object, or merely a list of lists of neurons?

Right now, the Network object holds the layers as a List of tuples, where each tuple looks like this: (W, [n_0, n_1, ..., n_k]), where k is the number of neurons in the layer and W is the weight matrix for the layer.

The main problem at the moment is where to store the weights. The model described above it frustrating because individual neurons need three things to compute its output:

• His index in his own layer - i.e. his value k
• The index of the neuron he is currently giving his output to - its value k, which we will call j here
• The ability to access the weight matrix for value W_kj

I am firmly against the idea of storing the weight matrix in every neuron, which means that a neuron must be able to ask its parent for the appropriate value from the weight matrix when it's computing. Any other ideas?

---

One way of doing this is to have a member variable inside the neuron for its own index k, its layers index in the overall network, and a network_shape variable which describes the size of each layer - i.e. [6, 3, 3]. The Neuron also has a reference to the Network, allowing it to ask the Network for the appropriate W_ij.

I'll update this as I think of other ideas.

Start writing tests

It's about time...

Fix Backpropagation

Backprop isn't implemented fully correctly at the moment.

Add variables to layers which describe their expected inputs an outputs

I should mathematically deduce the expectations of each layer in terms of input and output shapes. Each layer should automatically know the exact shape of all the matrices and vectors associated with it.

Implement activations as layers

I spotted an implementation of activation functions as layers here. It might simplify the gradient juggling during backpropagation - each layer performs one gradient multiply and passes it on/updates.