dpressel / dliss-tutorial

Tutorial for International Summer School on Deep Learning, 2019

dliss-tutorial

Tutorial for International Summer School on Deep Learning, 2019 in Gdansk, Poland

Sections

Overview Talk

https://docs.google.com/presentation/d/1DJI1yX4U5IgApGwavt0AmOCLWwso7ou1Un93sMuAWmA/

Tutorial

There are currently 3 hands-on sections to this tutorial.

• The first section covers pre-trained word embeddings (colab)

• The second section covers pre-trained contextual emeddings (colab)

• The third section covers fine-tuning a pre-trained model (colab)

Updates

• April 2022 If you are interested in learning how to build different Transformer architectures from the ground up, I have a new set of tutorials with in-depth details and full implementations of several popular Transformer models. They show how to build models step by step, how to pretrain them, and how to use them for downstream tasks. There is an accompanying Python package that contains all of the tutorial pieces put together

• July 2020 I have posted a set of Colab tutorials using MEAD which is referenced in these tutorials. This new set of notebooks covers similar material, including transfer learning for classification and taggers, as well as training Transformer-based models from scratch using the MEAD API with TPUs. MEAD makes it easy to train lots of powerful models for NLP using a simple YAML configuration and makes it easy to extend the code with new models while comparing against strong baselines!

dliss-tutorial's Issues

Detach the hidden states

Dear Daniel Pressel,

I have found your tutorial for the International Summer School on Deep Learning very elucidating (I was not there, though). However, there is one concept that is not clear to me. For example, when you say in the Second Section (Contextualized Embeddings) in the colab:

At every step of training, we will detach our hidden states, preventing full backpropagation, but we will initialize the new batch from our old hidden state

Could you further elaborate on why detaching the hidden states (both h, and c) is a necessary step? Or could you point to a reference where this is explained?

Thanks!