linbinyang / yannpp

Deep Neural Network from scratch in C++ for learning purposes

yannpp

This is an educational effort to help understand how deep neural networks work.

In order to achieve this goal I prepared a small number of selected educational materials and heavily documented pure C++ implementation of CNN that classifies MNIST digits.

Understand

In order to fully understand what is going on, I would recommend doing following:

• read great Michael Nielsen's online book to understand all the basics and do the exercises (at least derivation of BP1-BP4)
• read "Backpropagation In Convolutional Neural Networks" pdf in the docs/ to understand how to prove backpropagation equations for convolutional layers
• read "A guide to convolution arithmetic" pdf in docs/ to understand what is padding and how to convolve input and filter

After this you will be able to understand code in the repo.

Get in

C++ code in the repo is simple enough to work in Windows/Mac/Linux. You can use CMake to compile it (check out .travis.yml or appveyor.yml to see how it's done in Linux or Windows).

In order to use MNIST data you will need to unzip archives in the data/ directory first. Also compiled executable accepts path to this data/ directory as first command line argument.

See

Main learning loop (as defined in network2_t::backpropagate()) looks like this:

// feedforward input
for (size_t i = 0; i < layers_size; i++) {
    input = layers_[i]->feedforward(input);
}

// backpropagate error
array3d_t error(result);
for (size_t i = layers_size; i-- > 0;) {
    error = layers_[i]->backpropagate(error);
}

Because of this simplicity most interesting things are located in src/layers/ directory that contains implementations of those feedforward() and backpropagate() methods for each layer.

This codebase contains it's own greatly simplified ndarray as in Numpy and it's called array3d_t. Most useful feature of the array is the ability to slice parts of it's data as subarrays.

network1_t as used in examples/mnist_simple.cpp is all-in-one implementation of network with fully-connected layers while network2_t is more "abstract" implementation that uses arbitrary layers in other examples.

Do

Codebase should encourage you to experiment. For example, examples/mnist_deeplearning.cpp file specifically contains lots of experimental code (e.g. reducing size of the input to be able to experiement with network topology, commented layers in the network itself etc.) that can show you how to experiment. Experimentation is required to select hyperparameters, to see if your network converges etc.

Cope

Feel free to say thank you if it was useful. Also this code (as any other) may contain bugs or other problems - all contributions are highly welcome.

• Fork yannpp repository on GitHub
• Clone your fork locally
• Configure the upstream repo (git remote add upstream [email protected]:ribtoks/yannpp.git)
• Create local branch (git checkout -b your_feature)
• Work on your feature
• Push the branch to GitHub (git push origin your_feature)
• Send a pull request on GitHub

Get out

There are many other similar efforts on GitHub. Their common problems are: code that is hard to read or code with too much magic inside (mainly related to python). Here's a short list with similar efforts with very easy code to understand:

• Nympy-CNN
• zeta-learn
• Machine Learning Numpy