NLP702-2024

NLP702 Deep Learning for Natural Language Processing

Catalog Description: This course provides a methodological and an in-depth background on key core Natural Language Processing areas based on deep learning. It builds upon fundamental concepts in Natural Language Processing integrating advances on large language models (LLMs). It assumes familiarity with mathematical and machine learning concepts and programming.
Goal: This graduate-level course aims to instill a deeper and thorough understanding of advanced Natural Language Processing methods based on deep learning, to equip students with capabilities of researching, developing, and implementing these methods.
The course covers the following key core areas: (I) Fundamentals of LLMs, (II) LLM Efficiency and Safety, (III) Multilinguality, Machine Translation, and Multimodality

This advanced course will use research papers. Students may find the following textbooks relevant:

(1) Tunstall, L., von Werra, L., & Wolf, T. (2022). Natural Language Processing with Transformers. O'Reilly Media, Inc., ISBN 978-1098136796.

(2) Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. 2016, MIT Press. ISBN: 9780262035613.
Relevant research papers, technical reports, and surveys for each topic, where needed, will be provided to students. In addition, the following textbooks may be useful:

(1) Chris Manning et al, Foundation of statistical natural language processing, 1999, MIT Press, ISBN: 0262133601.

(2) Dan Jurafsky and James H. Martin, Speech and Language Processing (3rd edition, draft) https://web.stanford.edu/~jurafsky/slp3/

(3) Aurélien Géron. Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems (2nd Edition). 2019, O'Reilly Media, ISBN 9781492032649

(4)François Chollet. Deep Learning with Python. 2017, Manning Publications Co. , ISBN 9781617294433.

(5)Yue Zhang, Zhiyang Teng. Natural Language Processing: A Machine Learning Perspective, 2021, Cambridge University Press, ISBN 9781108420211.

(6) Jacob Eisenstein. Introduction to Natural Language Processing. 2019, MIT Press, ISBN 9780262042840.

Teaching Week	Topic	Leture	Lab
1	Course Overview & LLMS in NLP	Self-supervised learning; Attention and transformer variations; Masked language models: encoder-only and encoder-decoder	NumPy; PyTorch
2	Causal LMs (CLMs)	Generative pretraining; Use of CLMs, EEval of CLMs	Transformer architecture; Pretraining of GPT2; Calculating perplexity
3	Instruction Tuning	Instruction Finetuning and Evaluation	Instruction tuning
4	LLM Prompting	Model Prompting	Encoder-only model prompting; Pmompt engineering