jaeyoung0909 / probprog19

CS423 Probabilistic Programming, Spring 2019, KAIST

This is a webpage of the course "CS423 Probabilistic Programming", which is offered at the KAIST CS department in the spring of 2019. The webpage will contain links to lecture slides and other course-related materials.

Probabilistic programming refers to the idea of developing a programming language for writing and reasoning about probabilistic models from machine learning and statistics. Such a language comes with the implementation of several generic inference algorithms that answer various queries about the models written in the language, such as posterior inference and marginalisation. By providing these algorithms, a probabilistic programming language enables data scientists to focus on designing good models based on their domain knowledge, instead of building effective inference engines for their models, a task that typically requires expertise in machine learning, statistics and systems. Even experts in machine learning and statistics may get benefited from such a probabilistic programming system because using the system they can easily explore highly advanced models.

This course has two goals. The first is to help students to be a good user of an expressive probabilistic programming language. Throughout the course, we will use a particular language, called Anglican, but we will emphasise general principles that apply to a wide range of existing probabilistic programming systems. The second goal is to expose the students to recent exciting results in probabilistic programming, which come from machine learning, statistics, programming languages, and probability theory.

1. Important Announcements

[May 27] Homework 4 - Solve only the first question.

It is likely that we won't have time to cover the theory of Quasi-Borel spaces in the remaining lectures. So, skip the second question in Homework 4, which is about product and function-space constructions for Quasi-Borel spaces.

[May 27] Project presentation and project report.

We will have project presentation on the 3rd and the 5th of June. Each group will give a 15-minute talk and have a 5-min Q&A session. Here is the order of presentation.

3 June (Monday)

1. Seungmin Ahn, Woogeun Bae, Wonjoon Jang, and Hyungseung Whang.
2. Youngin Lee, Vu Phuong Thao, Magnus Myrmo Osberg, and Hyeonguk Ryu.
3. Changmin Lee, An Guoyuan, Minsu Kim, and Ahcheong Lee.
4. Seongwoong Jo, Dohyun Kim, Minji Lee, and Dongpyeong Seo.

5 June (Wednesday)

1. Yoonseo Kim, Haebeom Lee, and Yuna Seol.
2. Seunghyeob Lee.
3. Sanguk Park.

Project report. Due: 2:00pm on 7 June (Friday).

All groups including those who do track B should submit reports on what they did and what they learnt. One report is enough for each group. The reports should have at most two pages. As usual, you can email your report to TAs and me, or submit it in the homework submission box.

[May 19] No lecture on 22 May.

We will not have a class on the 22nd of May (Wednesday). This is because I am involved in running the interviews for the graduate admission for KAIST CS.

[May 12] Homework4 is out.

The deadline is 6:00pm on 31 May 2019 (Friday). Submit your solutions to the TAs by putting them in the homework submission box on the third floor of the E3-1 building. If you type up your solutions, you can email them to TAs. In that case, email them to both Mr Kwonsoo Chae ([email protected]) and Mr Hyunsu Kim ([email protected]).

[April 30] No lecture on 6 May.

The 6th of May is a national holiday. We won't have a lecture on that day.

[April 28] Homework3 is out.

The deadline is 6:00pm on 10 May 2019 (Friday). Submit your solutions to the TAs by putting them in the homework submission box on the third floor of the E3-1 building. If you type up your solutions, you can email them to TAs. In that case, email them to both Mr Kwonsoo Chae ([email protected]) and Mr Hyunsu Kim ([email protected]).

[March 31] Homework2 is out.

The deadline is 6:00pm on 12 April 2019 (Friday). Submit your solutions to the TAs by putting them in the homework submission box on the third floor of the E3-1 building. If you type up your solutions, you can email them to TAs. In that case, email them to both Mr Kwonsoo Chae ([email protected]) and Mr Hyunsu Kim ([email protected]).

[March 28] Project pitch on 3 April.

On 3 April, each project group will have to give a short presentation on its project. The presentation will have to be mainly about topic, motivation, plan and preliminary findings. Groups on track A will have to prepare 10-minute presentations, which should include time for the Q&A session. Groups on track B will have to prepare 5-minute presentations, which also should include time for the Q&A session. The presentations of the groups on track B may be slightly different and focus on giving previews on the topics that they will teach. The order of presentation is as follows:

1. Seungmin Ahn, Woogeun Bae, Wonjoon Jang, and Hyungseung Whang.
2. Youngin Lee, Vu Phuong Thao, Magnus Myrmo Osberg, and Hyeonguk Ryu.
3. Changmin Lee, An Guoyuan, Minsu Kim, and Ahcheong Lee.
4. Seongwoong Jo, Dohyun Kim, Minji Lee, and Dongpyeong Seo.
5. Yoonseo Kim, Haebeom Lee, and Yuna Seol.
6. Seunghyeob Lee.
7. Sanguk Park.
8. Jaeseong Lee, Jinwon Lee, and Soyoon Oh.
9. Yongsu Baek, Junho Han, Jihoon Ko, and Seungwoo Lee.

[March 23] Penality scheme for late homework submissions.

Here is our scheme for handling late homework submissions. It assumes that the total marks are 100.

1. <= One day late: -10
2. <= Two days late: -20
3. <= Three days late: -30
4. <= Four days late: -40
5. More than four days late: -100.

[March 9] Homework1 is out.

The deadline is 6:00pm on 22 March 2019 (Friday). Submit your solutions to the TAs by putting them in the homework submission box on the third floor of the E3-1 building. If you type up your solutions, you can email them to TAs. In that case, email them to both Mr Kwonsoo Chae ([email protected]) and Mr Hyunsu Kim ([email protected]).

[February 27] TA office hours

Kwonsoo and Hyunsu will have TA office hours from 3:00pm to 5:00pm on every Tuesday at the room 3415 in the E3-1 building starting from 5 March.

[February 25] Homework0 is out.

You don't have to submit your answer. But we strongly recommend you to try it. This homework will teach you how to run Anglican.

[February 25] Project group.

The group project is an important part of this course. Find your project partners by 13 March 2019, and inform all of Hongseok, Hyunsu and Kwonsoo by email. Each group should consist of 3-4 students. Finally, if your group wants to go for Track B, contact Hongseok as early as possible by email.

2. Logistics

Evaluation

• Final exam (40%). Project (40%). Homework (20%).

Teaching Staffs

• Lecturer: Prof Hongseok Yang (email: [email protected], office hours: 6:00pm - 7:00pm on Monday at the room 3403 in the E3-1 building)
• TA1: Kwonsoo Chae (email: [email protected], office hours: 3:00pm - 5:00pm on every other Tuesday starting from 5 March at 3415 in E3-1)
• TA2: Hyunsu Kim (email: [email protected], office hours: 3:00pm - 5:00pm on every other Tuesday starting from 12 March at 3415 in E3-1)

Place and Time

• Place: room 111 in the N1 building
• Time: 1:00pm - 2:15pm on Monday and Wednesday from 25 February 2019 until 10 June 2019.
• Final exam: 1:00pm - 3:00pm on 10 June 2019 (Monday) at the room 111 in the N1 building.

Online Discussion

• We will use KLMS.

3. Homework

Submit your solutions by putting them in the homework submission box in the third floor of the E3-1 building. If you type up your solutions, you can email them to TAs. In that case, email them to both Mr Kwonsoo Chae ([email protected]) and Mr Hyunsu Kim ([email protected]).

• Homework0 - Don't submit.
• Homework1 - Deadline: 6:00pm on 22 March 2019.
• Homework2 - Deadline: 6:00pm on 12 April 2019.
• Homework3 - Deadline: 6:00pm on 10 May 2019.
• Homework4 - Deadline: 6:00pm on 31 May 2019.

4. Tentative Plan

• 02/25 (Mon) - Introduction. Slides. Homework0.
• 02/27 (Wed) - Basics of Clojure and Tiny Bit of Anglican. Slides. Gorilla worksheet. Programs.
• 03/04 (Mon) - Basics of Clojure and Tiny Bit of Anglican.
• 03/06 (Wed) - Posterior Inference, Basics of Anglican, and Importance Sampling. Slides.
• 03/11 (Mon) - Posterior Inference, Basics of Anglican, and Importance Sampling.
• 03/13 (Wed) - Generative Modelling with Anglican. Slides. Gorilla worksheet for 2D physics. Gorilla worksheet for program induction. (Note: In order to run the 2D physics example, you will have to copy bounce.clj to the anglican-user/src directory and replace anglican-user/project.clj by this file).
• 03/18 (Mon) - Generative Modelling with Anglican.
• 03/20 (Wed) - Markov Chain Monte Carlo. Slides.
• 03/25 (Mon) - Markov Chain Monte Carlo.
• 03/27 (Wed) - Markov Chain Monte Carlo.
• 04/01 (Mon) - Implementing Inference Algorithms for Probabilistic Programs. Slides. Note. Handwritten Notes: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10.
• 04/03 (Wed) - Group Project Pitch.
• 04/08 (Mon) - Implementing Inference Algorithms for Probabilistic Programs.
• 04/10 (Wed) - Implementing Inference Algorithms for Probabilistic Programs.
• 04/15 (Mon), 04/17 (Wed) - NO LECTURES. Midterm Exam.
• 04/22 (Mon) - Implementing Inference Algorithms for Probabilistic Programs.
• 04/24 (Wed) - Stochastic Variational Inference. Slides. Handwritten Notes: 1, 2, 3. (Corrections for the third note.)
• 04/29 (Mon) - Stochastic Variational Inference.
• 05/01 (Wed) - Amortised Inference. Slides.
• 05/06 (Mon) - NO LECTURE. Children's Day.
• 05/08 (Wed) - Reparameterisation Gradient, Model Learning via ELBO, IWAE and Doubly-Reparameterised Gradient Estimator. Handwritten Notes: 1, 2, 3, 4. Incomprehensible Handwritten Notes on Doubly-Reparameterised Gradient Estimator: 1, 2, 3.
• 05/13 (Mon) - Group presentation 1: Causality. Students' slides.
• 05/15 (Wed) - Group presentation 2: Pyro and its internals. Students' slides.
• 05/20 (Mon) - Denotational Semantics of Probabilistic Programs. Slides. Note.
• 05/22 (Wed) - NO LECTURE. Graduate Admission.
• 05/27 (Mon) - Denotational Semantics of Probabilistic Programs.
• 05/29 (Wed) - Denotational Semantics of Probabilistic Programs.
• 06/03 (Mon) - Student Presentation
• 06/05 (Wed) - Student Presentation
• 06/10 (Mon), 06/12 (Wed) - NO LECTURES. Final Exam.

5. Studying Materials

Studying the lecture slides and notes and the homework exercises of the course is likely to be the most time-efficient way to catch up with this course. Also, at each lecture, we will give students pointers to the related papers. If a student does not understand a certain concept, we encourage him or her to look it up in the Internet. We typically do this when we encounter a similar problem. In our case, Wikipedia, lecture notes or survey articles have helped us the most.

The next best option is to read the following draft book on probabilistic programming:

1. "An Introduction to Probabilistic Programming" by Jan-Willem van de Meent, Brooks Paige, Hongseok Yang and Frank Wood. Reading this book will give a broader view on probabilistic programming and much deeper understanding into its inference algorithms and their implementations.

If a student feels that she or he lacks background knowledge on machine learning, we recommend him or her to have a look at the following online materials.

2. The online book "Probabilistic Programming and Bayesian Methods for Hackers" describes Bayesian Machine Learning using a probabilistic programming system called PyMC. Hongseok found this book easy to follow and good at explaining basics and intuitions.

3. A more standard reference on machine learning is Bishop's book "Pattern Recognition and Machine Learning".

Two good ways to understand probabilistic programming are to try a wide range of examples and to understand common implementation techniques for probabilistic programming languages. The following documents provide such examples or explain those techniques.

4. Anglican website. In particular, students will learn a lot by trying examples in the site.

5. Forestdb.org is a great source of interesting probabilistic programs.

6. Edward tutorial website and Pyro example website. Edward and Pyro are so called deep probabilistic programming languages that attempt to combine deep learning and probabilistic programming. These web pages contain interesting examples that one can try using these languages.

7. Goodman and Stuhlmuller's book "The Design and Implementation of Probabilistic Programming Languages". This web-based book describes the implementation of WebPPL, a probabilistic programming language on top of JavaScript. Many techniques in the book are general and apply to other probabilistic programming languages.

Frank Wood taught a graduate-level course on probabilistic programming at UBC. It emphasises on actually building efficient runtimes for probabilistic programming languages. Here is a link to his course.

8. Wood's graduate-level course at UBC.

6. Group Project

A group project is a crucial part of this course. 3-4 students will form a project group, and they will carry out a project in Track A or in Track B:

1. Track A: A group develops an interesting application of Anglican or other probabilistic programming languages. The members of the group may attempt to find an efficient encoding of a highly complex probabilistic model (such as sequence memoizer) in Anglican, or they may develop a new probabilistic model for a complex data set and analyse the data set, or they may try to find a novel use of probabilistic programming for solving well-known existing problems (such as figuring out secret key in some security protocol).

2. Track B: At most two groups will be on this track. The goal of a group in this case is to study an advanced research topic on probabilistic programming or machine learning, to gain deep understanding about it, and to help fellow students acquire the same level of understanding. Specifically, a group performs an in-depth study on one of two advanced hot topics, causality and Pyro. Then, the group in this track has to teach what it learnt to other students in the course. By teaching, we mean (i) a presentation on the studied topic and (ii) a preparation of reading material and exercise problems. Also, the group has to submit a report on what they learnt. We recommend the report to be 2-to-4 pages long, although it can be longer if needed. Further information about causality and Pyro is given at the end of this webpage.

Group (Track A)

1. Seungmin Ahn, Woogeun Bae, Wonjoon Jang, and Hyungseung Whang.
2. Youngin Lee, Vu Phuong Thao, Magnus Myrmo Osberg, and Hyeonguk Ryu.
3. Changmin Lee, An Guoyuan, Minsu Kim, and Ahcheong Lee.
4. Seongwoong Jo, Dohyun Kim, Minji Lee, and Dongpyeong Seo.
5. Yoonseo Kim, Haebeom Lee, and Yuna Seol.
6. Seunghyeob Lee.
7. Sanguk Park.

Group (Track B)

1. Jaeseong Lee, Jinwon Lee, and Soyoon Oh - Pyro and its internals.
2. Yongsu Baek, Junho Han, Jihoon Ko, and Seungwoo Lee - Causality.

Concrete Tasks

1. [Deadline: midnight on 13 March (Wed)] Form a group and inform us by email (Hongseok, Hyunsu and Kwonsoo).
2. [3 April (Wed)] Presentation of each group about its topic.
3. [13 May (Mon)] Presentation on causality by a group on track B.
4. [15 May (Wed)] Presentation on Pyro and its internals by a group on track B.
5. [3 June (Mon), 5 June (Wed)] Presentation of group projects.
6. [Deadline: midnight on June 7 (Fri)] Submit a report on the project. We recommend each report to be 2-to-4 pages long, although it is fine to write a longer report if needed.

Two Topics in Track B

1. Causality

By causality, we mean attempts to discover the cause-and-effect relationships among various factors (or variables) from data (and a given set of assumptions), reason about the consequence of an intervention on such relationships (such as medical treatment), and infer useful information about counterfactuals, i.e., situations that did not happened in reality but could have happened. It is one of the emerging topics in machine learning, with a lot of blessings from leading figures from the field. In particular, bringing deep learning and causality together is one of the active research topics nowadays.

So far causality has not been studied seriously by researchers in probabilistic programming, although they two may be closely related. For instance, results on causality are often explained in terms of data-generating processes, and probabilistic programming encourages scientists to write such processes as programs. An ultimate goal of this group project is to fill in this gap slightly, find out a way of bringing causality and probabilistic programming together, and teach fellow students about what they found and learnt. Of course, this is a challenging goal, and it is ok to fail. We mentioned the goal, mainly to students in this project to navigate the big topic of causality better.

Here are a few references on causality that we recommend.

1. Ferenc Huszar's blog articles are very good, and introduce causality gently and intuitively. (article1, article2, article3). It is also worth reading comments of these articles. In particular, one of the comments of the first article lists recent papers on deep learning and causality.

2. Judea Pearl has been working on causality for many years. His interview gives you a glimpse into why he is so passionate about this topic. According to what we hear, his recent general-science book "The Book of Why" is very readable and provides good informal overview on causality. Pearl's book "Causality: Models, Reasoning and Inference" is the standard reference. Finally, it may be worth reading Pearl's recent article on causality in CACM.

2. Pyro and its internals

Pyro is a probabilistic programming language from Uber, which attempts to support the combined use of deep learning and probabilistic programming. It is still being developed. Its syntax and implementations are not finalised yet, and change constantly. But it is one of the most exciting developments in probabilistic programming.

Pyro contains novel language primitives, such as plate, random module and tensor shape for distribution object, which can help programmers or data scientists to express or specify their intentions more clearly. Also, its implementation uses novel interesting techniques, such as poutine, automatic guide generation, and selective marginalisation of discrete random variables using the tensor broadcasting mechanism. These techniques enable efficient probabilistic inference of Pyro programs. Finally, there are cool Pyro examples, such as Attend-Infer-Repeat, that use neural networks and probabilistic programming in a non-trivial way.

The goal of this project is to study these apsects of Pyro (i.e., language design, implementation techniques, and cool examples and libraries), and explain them to the other students.

1. The main reference for the project is Pyro's webpage. In particular, we suggest students in the project to study the section on examples very closely. It contain information about examples, languages, implementation techniques, and other interesting libraries, such as automatic guide generation. The section even contains information about how to implement a baby version of Pyro.

2. Pyro and Edward from Google share several language design decisions and implementation techniques. Looking at the following papers on Edward may help students to understand these shared aspects, and also general principles for designing and implementing such deep probabilistic programming languages. However, we advise students to prioritise the study of Pyro's webpage over reading these papers.

   1. "Simple, distributed, and accelerated probabilistic programming" by Tran et al. at NeurIPS'18.
   2. "TensorFlow Distributions" by Dillon et al.