matthew-trachtenberg / learnanalytics-deeplearning-azure

Learning Materials for Deep Learning on Azure

Deep Learning on Azure

This repository contains materials to help you learn about Deep Learning with the Microsoft Cognitive Toolkit (CNTK) and Microsoft Azure. Students can find slides, tutorial notebooks, and scripts covering a variety of deep learning fundamentals and applications. These course assets will teach you how to implement convolutional networks, recurrent networks, and generative models and apply them to problems in computer vision, natural language processing, and reinforcement learning. The course materials will pay particular attention on how to implement these algorithms most effectively using the resources provided by the Azure infrastructure, and best practices when working with CNTK.

Part I - Fundamentals and Azure for Machine Learning

1. Pretensions to Thinking and Learning - Overview of Machine Learning
2. A Minimal Introduction to AI, Representation Learning, and Deep Learning
3. Deploying and Accessing the Linux Data Science Virtual Machine
4. Computational Graphs, Symbolic Differentation, and Auto-Differentiation
5. Overview of the Microsoft Cognitive Toolkit (CNTK) and Other Deep Learning Frameworks
6. Activation Functions and Network Architectures
7. Representational Power and Capacity

Part II - Optimization

1. Backpropagation and Stochastic Optimization for Training Neural Networks
2. Momentum and Acceleration Methods
3. Regularization, Normalization, and Dropout
4. Distributed Training and Evaluation with Azure Batch AI
5. Practical Bayesian Optimization for Hyperparameter Search
6. Evolutionary Strategies for Parameter Search

Part III - Convolutional Neural Networks

1. Scaling Networks to Images
2. Receptive Fields, Spatial Arrangements, Strides and Filters
3. Dilated Convolutions and Pooling
4. Skip Connections and Residual Networks

Part IV - Recurrent Networks

1. Dense Word Vector Representations
2. Comparison of word2Vec, GloVe, and fasttext
3. Recurrent Neural Networks and Language Models
4. GRUs, LSTMs, and Recursive Architectures
5. Vanishing and Exploding Gradients
6. Memory and Attention

Part V - Reinforcement Learning

1. Optimal Control and Planning
2. Policy Gradients
3. Q-learning
4. Actor-Critic Methods
5. Evolutionary Strategies as an Alternative to Policy Methods

Part VI - Generative Models

1. Visualizing and Understanding Neural Networks with Saliency Maps
2. Adversarial Attacks on Neural Networks
3. Metrics on Distributions for Implicit Generative Models
4. Generative Adversarial Networks
5. Belief Nets and Change of Variable Models
6. Approximate Methods using the Variational Autoencoder

Part VII - Operationalization Methods

1. HDInsight, pyspark and mmlspark
2. Azure Batch Shipyard / Azure Batch Training
3. Azure container services
4. SQL Server 2017
5. The embedding learning library and web applications

Useful Resources

Online Courses

• Deep Learning Explained, edX 2017
  • Online MOOC that covers the fundamentals of Deep Learning with the Microsoft Cognitive Toolkit
  • Consists of 7 modules
  • Released in June 2017
• deeplearning.ai - Coursera Specialization Taught by Andrew Ng
  • Specialization consisting of 5 MOOCs on Deep Learning taught by Andrew Ng
  • Taught using TensorFlow
• fastAI
  • 2 Deep Learning courses taught by Jeremy Howard and Rachel Thomas at USF
• CS231n - Convolutional Networks for Visual Recognition
• CS224n - Natural Language Processing with Deep Learning

Online Books and Blogs

• Neural Networks and Deep Learning - Michael Nielsen
• Deep Learning - Ian Goodfellow, Youshua Bengio & Aaron Courville
• Chris Olah's Blog
• Distill Publications
• Andrej Karpathy's Blog
• Denny Britz's Blog
• Edwin Chen's Blog
• Off the Convex Path - Join blog with Sanjeev Arora, Moritz Hardt & Nisheeth Vishnoi
• BAIR - Berkeley AI Research Blog