antahiap / debugging-dl-models

Material for the class on debugging deep learning models at Data Science Retreat.

Useful resources for debugging:

• Debug a deep learning network: A nicely written blog post on how to debug a deep learning model.
• Recipe for training neural networks: An Andrej Karpathy's blog post on how to train neural nets.
• Troubleshooting deep learning models: A great video to watch where many debugging steps are summarized in a comprehensive way.
• Machine learning yearning: A practical manual written by Andrew Ng, which gives a full overview how one should structure a deep learning project.
• Bayesian optimization: Bayesian optimization for hyperparameter search.
• Dive into deep learning: An amazing website with theory of deep learning, code examples, exercises. It starts from basics and covers the most advanced topics in DL.
• Deep learning: The Bible of deep learning written by Ian Goodfellow and Yoshua Bengio and Aaron Courville. If you wanna go deep, this book is a must to read.

Some other resources, explaining notebook materials:

• Understanding LSTM networks: A clear written post on LSTMs. Good for a quick overview and recalling some basics.
• Batch normalization explained: A blog post explaining batch normalization in detail.
• See-RNN package: A package that helps to visualize RNNs. Check it out if you want to dig into RNNs.
• Gradient clipping: An article on gradient clipping in RNNs.
• Axiomatic attribution for deep networks: An article about Integrated Gradients, which is a useful tool for debugging neural nets.
• Attribution baselines: A really well written blog post on importance of choosing a good baseline for Integrated Gradients.
• Teachin Large Language Models to Self-Debug: A paper about Rubber Duck self-debugging with LLMs.
• LangChain Docs: LangChain documentation page.

Prerequisites:

• Clone the repo:

git clone https://github.com/stanislav-chekmenev/debugging-dl-models

• Please, create a new virtual environment with Python=3.10. Feel free to use any of your choice. I prefer Virtualenv.

Conda:

conda create --name <name> python=3.10
conda activate <name>

Virtualenv: For Ubuntu 20.04:

sudo apt install virtualenv
python3 -m venv <path/to/venv>
source <path/to/venv>/bin/activate

• Upgrade pip, it might be of an old version

pip install pip --upgrade

• Install requirements:

pip install -r requirements.txt

• Install PyCharam Community Edition:

  • Follow this manual for the installation details.
  • If you are on Linux Ubuntu starting from 16.04, then please use the following command:

   sudo snap install pycharm-community --classic

• Create a new project and virtual environment in Pycharm: There are several options how to do it. This is one of them.

  • Open a new terminal window and run the following command to start Pycharm:

   pycharm-community &

  • You will see a Welcome screen, click New Project. If you already use Pycharm and see a project open, choose File | New Project.
  • In the location field type in the location where you cloned the repo to and choose pycharm-debugging directory.
  • Tick the box that is called "Previously configured interpreter" and choose the virtual environment that you created for this class, either conda or Virtualenv.
  • Deselect the Create a main.py welcome script checkbox. And click Create.

That should be sufficient to run everything. Thank you!

Class material:

• Colab 0: Introduction to debugging deep learning models
• Colab 1: Implementation recipe of a bug free model
• Colab 2: Most common bugs
• Colab 3: Rubber duck debugging with LLMs