jackbandy / deep_learning_ulmfit

Group project for deep learning, replication for "Universal Language Model Fine-tuning for Text Classification" https://arxiv.org/pdf/1801.06146.pdf

deep_learning_ulmfit

Group project for deep learning, replication for "Universal Language Model Fine-tuning for Text Classification" https://arxiv.org/pdf/1801.06146.pdf

Main Goals

• Get (some of) the datasets
• Separate "library" building blocks from tests and experiment code
• Stand up the eval pipeline
• Get bad scores on an untrained model
• Get scores on a pretrained model
• Reproduce last row of table 3 with AG
• Table 4,5,6,7 "would be cool"

Milestones

• Gather resources and set up repo (Week of April 20)
• Milestone 2 (Week of April 27)
• Milestone 3 (Week of May 4)
• Content Ready by May 10
• Milestone 4 (Week of May 11)
  • Class Presentation on May 13th
• Milestone 5 (Week of May 18)
• Milestone 6 (Week of May 25)
• Milestone 7 (Week of June 1)
• Final
  • Create one self-contained notebook
  • Organize notebook according to "tricks" in the paper
  • Add code for fine-tuning on custom data

Presentation

• Intro and related work (Victor)
• General domain LM pretraining (Unnati)
• Target task LM fine-tuning (Victor) —> discriminative fine-tuning, slanted triangular learning rates
• Target task classifier fine-tuning” (Jack) —> concat pooling, gradual unfreezing
• Experiments + Results (Jack)
  • sentiment analysis
  • question classification
  • topic classification
• Analysis
  • Low shot learning & impact of pretraining (Unnati)
  • impact of LM fine-tuning (Victor)
  • impact of classifier fine-tuning (Jack)
  • classifier fine-tuning behavior & impact of bidirectionality (Unnati)
• Discussion & future work & final remarks (Victor)

Resources

• FastAI's ULMFiT website: http://nlp.fast.ai/category/classification.html
• Video tutorial by one of the authors: https://www.youtube.com/watch?v=vnOpEwmtFJ8&feature=youtu.be&t=4511
• Scripts for IMDB tasks in the paper/video: https://github.com/fastai/fastai/tree/master/courses/dl2/imdb_scripts

Reproducibility info:

Feature	Value
Year Published	2018
Year First Attempted	2018(?)
Venue Type	Conference
Rigor vs Empirical*	Empirical
Has Appendix	No
Looks Intimidating	Nah
Readability*	Good
Algorithm Difficulty*	n/a
Pseudo Code*	No
Primary Topic*	Text Classification
Exemplar Problem	Not really
Compute Specified	No
Hyperparameters Specified*	Some
Compute Needed*	?
Authors Reply*	Yes
Code Available	Yes
Pages	9 (12 with ref)
Publication Venue	ACL
Number of References	~50
Number Equations*	3
Number Proofs	0