disegnovitruviano / intro-synthetic-media

Introduction to Synthetic Media Class at ITP/NYU.

Introduction to Synthetic Media

• 7 weeks, 1/31/2020 - 3/13/2020, Friday 9 am - 11:30 am
• 370 Jay Street, Room 407, ITP/NYU
• Office Hours
• All class dates

COURSE DESCRIPTION

“Technology should not aim to replace the humans, rather amplify human capabilities.” —-Douglas Engelbart

Generative machine learning models open new possibilities for creating images, videos, and text. This class explores the idea of how artists, designers and creators can use machine learning in their own design process. The goal of this class is to learn and understand some common machine learning techniques and use them to generate creative outputs.

Students will learn to use pre-trained models, and train their own models in the cloud using Runway. For each week, we will discuss the history, theory, datasets, application of the machine learning models, and build experiments based on the model. In addition to Runway, we will be using JavaScript libraries like the p5.js, ml5.js, and TensorFlow.js, and applications like Photoshop, Unity and Figma.

This course is aimed at people working in creative disciplines who wish to learn about machine intelligence and how to apply it in their own fields. It is not aimed at scientists or engineers who are seeking a rigorous technical course on machine learning. This course is more practical than it is theoretical; we are interested less in proving theorems and equations, and more into hacking existing tools for making machines that do interesting things.

Prospective students are expected to have taken an ICM (Introduction to Computational Media) course, or have equivalent programming experience with Python or JavaScript.

OUTLINE:

• Week 1 Introduction
  • Intro to ML
  • Deep learning
  • Neural networks
• Week 2 ML for designers
  • Unity, figma, photoshop
  • Runway design plugins
• Week 3 Image generation
  • GAN
  • bigGAN
  • StyleGAN
  • How to train your own styleGAN
• Week 4 Text Generation
  • LSTM
  • GPT-2
  • Training GPT-2
• Week 5 Media Translation
  • Image-to-Image Translation: Style transfer models, pix2pix, cycleGAN
  • Text2Image Translation: im2txt, attnGAN
• Week 6 Semantic Image Segmentation/Synthesis
  • SPADE-COCO
  • DeepLab
• Week 7 Final project presentation

Links:

• Runway ML: http://runwayml.com
• GAN Lab: https://poloclub.github.io/ganlab/
• NIPS 2016 Tutorial: Generative Adversarial Networks: https://arxiv.org/pdf/1701.00160.pdf, http://www.iangoodfellow.com/slides/2016-12-04-NIPS.pdf
• Runway + p5.js: https://github.com/runwayml/p5js

A list of ML models we will be covering:

Image generation:

• StylanGAN: https://github.com/NVlabs/stylegan
• BigGAN: https://github.com/ajbrock/BigGAN-PyTorch

Style Transfer

• Fast-style-transfer: https://github.com/lengstrom/fast-style-transfer
• Arbitrary-Image-Stylization: https://github.com/tensorflow/magenta/tree/master/magenta/models/arbitrary_image_stylization

Semantic Image Segmentation/Synthesis

• Deeplab: https://github.com/tensorflow/models/tree/master/research/deeplab
• Sapde-coco: https://github.com/NVlabs/SPADE

Image-to-Image Translation:

• pix2pix: https://phillipi.github.io/pix2pix/
• pix2pixHD: https://github.com/NVIDIA/pix2pixHD

Text Generation

• LSTM
• gpt-2: https://github.com/openai/gpt-2

EQUIPMENT

You will need a modern laptop (4 years old or younger is a good rule of thumb). Most required software is freely available.

Policies

Evaluation

You are required to attend all class meetings and submit all weekly assignments and a final project.

Grading (pass/fail) will be based on a combination of factors:

• Attendance, participation in class discussion, and engagement in other students' projects (40%)
• Assignments (40%)
• Final Project (20%)

Please see ITP's statement on Pass/Fail which states that a "Pass" is equivalent to an "A" or a "B" while anything less would be considered a "Fail".

Attendance is mandatory. Please inform your teacher via email if you are going to miss a class. Two unexcused absences is cause for failing the class. (An unexcused lateness of 10 minutes or more is equivalent to 1/2 an absence.)

This class will be participatory, you are expected to participate in discussions and give feedback to other students both in class and participate with their projects. This (along with attendance) is 40% of your grade.

Class will culminate with final projects. You are expected to push your abilities to produce something that utilizes what you have learned in the class that is useful in some manner to yourself or the world. This will comprise 20% of your grade.

Statement of Academic Integrity

Plagiarism is presenting someone else’s work as though it were your own. More specifically, plagiarism is to present as your own: A sequence of words quoted without quotation marks from another writer or a paraphrased passage from another writer’s work or facts, ideas or images composed by someone else.

USE OF FREE AND OPEN SOURCE CODE EXAMPLES

(The following is adapted from Golan Levin’s Interactivity and Computation Course (Fall 2018) at Carnegie Mellon University.)

You must cite the source of any code you use

with the exception of examples specifically provided by the professor that are demonstrated in the videos for this course. Please note the following additional expectations and guidelines:

Check the License.

When using others' code, pay attention to the license under which it has been released, and be certain to fulfill the terms and requirements of those licenses. Descriptions of common licenses, and their requirements, can be found at choosealicense.com. Some licenses may require permission. If you are confused or aren’t sure how to credit code, ask one of the course instructors and make your best good faith effort. Not properly citing code sources is grounds for a 0 on an assignment.

Use Libraries.

The use of general, repurposable libraries is strongly encouraged. The people who developed and contributed these components to the community worked hard, often for no pay; acknowledge them by citing their name and linking to their repository.

Be Careful.

It sometimes happens that an artist places the entire source code for their sketch or artwork online, as a resource from which others can learn. Assignments professors give in new-media arts courses are often similar (e.g. "Clock"); you may also discover the work of a student in some other class or school, who has posted code for a project which responds to a similar assignment. You should probably avoid this code. At the very least, you should be careful about approaching such code for possible re-use. If it is necessary to do so, it is best to extract components that solve a specific technical problem, rather than those parts which operate to create a poetic experience. Your challenge, if and/or when you work with others' code, is to make it your own. It should be clear that downloading an artwork from someone's GitHub and simply changing the colors would be disgracefully lazy. And doing so without proper citation would be outright plagiarism.

Statement of Principle

The core of the educational experience at the Tisch School of the Arts is the creation of original academic and artistic work by students for the critical review of faculty members. It is therefore of the utmost importance that students at all times provide their instructors with an accurate sense of their current abilities and knowledge in order to receive appropriate constructive criticism and advice. Any attempt to evade that essential, transparent transaction between instructor and student through plagiarism or cheating is educationally self-defeating and a grave violation of Tisch School of the Arts community standards. For all the details on plagiarism, please refer to page 10 of the Tisch School of the Arts, Policies and Procedures Handbook, which can be found online at: http://students.tisch.nyu.edu/page/home.html

Statement on Accessibility

Please feel free to make suggestions to your instructor about ways in which this class could become more accessible to you. Academic accommodations are available for students with documented disabilities. Please contact the Moses Center for Students with Disabilities at 212 998-4980 for further information.

Statement on Counseling and Wellness

Your health and safety are a priority at NYU. If you experience any health or mental health issues during this course, we encourage you to utilize the support services of the 24/7 NYU Wellness Exchange 212-443-9999. Also, all students who may require an academic accommodation due to a qualified disability, physical or mental, please register with the Moses Center 212-998-4980. Please let your instructor know if you need help connecting to these resources.

Statement on use of Electronic Devices

Laptops will be an essential part of the course and may be used in class during workshops and for taking notes in lecture. Laptops must be closed during class discussions and student presentations. Phone use in class is strictly prohibited unless directly related to a presentation of your own work or if you are asked to do so as part of the curriculum.