ttungl / deep-learning

Implemented the deep learning techniques using Google Tensorflow that cover deep neural networks with a fully connected network using SGD and ReLUs; Regularization with a multi-layer neural network using ReLUs, L2-regularization, and dropout, to prevent overfitting; Convolutional Neural Networks (CNNs) with learning rate decay and dropout; and Recurrent Neural Networks (RNNs) for text and sequences with Long Short-Term Memory (LSTM) networks.

Deep Learning

This is the online course on Udacity.

Instructor: Dr. Vincent Vanhoucke from Google Brain.

Notice: The original assignments can be found at this link.

* Logistic Regression, Stochastic Optimization, and Data & parameters tuning.

• notMNIST - Deep Learning with Tensorflow.

* Deep Neural Networks

• Implemented a fully connected network using SGD and ReLUs with Tensorflow

* Regularization

• Implemented a multi-layer neural network using ReLUs, L2-regularization, and dropout, to prevent overfitting.

* Convolutional Networks

• Reading:
  • C. Olah, Conv Nets: A Modular Perspective.
  • C. Olah, Understanding Convolutions.
  • C. Olah, Calculus on Computational Graphs: Backpropagation.
  • V. Dumoulin and F. Visin, A guide to convolution arithmetic for deep learning. (arXiv, March 2016).
• Implemented a convolutional neural network.

* Deep Models for Text and Sequences

[Embeddings]

• Reading:
  • T. Mikolov et al., Efficient Estimation of Word Representations in Vector Space. (arXiv, Sep 2013).
  • T. Mikolov et al., Distributed Representations of Words and Phrases and their Compositionality. (arXiv, Oct 2013). Subsampling and Negative sampling in Word2Vec.
• Implemented a Word2Vec with the skip-gram model and CBOW model in different datasets of Wikipedia text as follows:
  • With text8.zip dataset.
  • With enwik8.zip dataset.

[Recurrent Neural Networks]

• Reading:
  • Understanding LSTM networks
  • Attention and Augmented Recurrent Neural Networks
  • H. Sak et al., Long Short-Term Memory Based Recurrent Neural Network Architectures for Large Vocabulary Speech Recognition. (arXiv, Feb 2014).
• LSTM: Long Short-Term Memory
  • Note: Backprop for RNNs example can be found at these links [1] and [2].
  • char-LSTM implementation with numpy

[Resources]

• Deep Learning with Python, Nov 2017 by François Chollet