facebookresearch / luckmatters

Understanding Training Dynamics of Deep ReLU Networks

This repo contains the code of the following 6 papers:

Analysis of Self-supervised learning (./ssl)

Understanding the Role of Nonlinearity in Training Dynamics of Contrastive Learning

Yuandong Tian

arXiv

Understanding Deep Contrastive Learning via Coordinate-wise Optimization

Yuandong Tian

NeurIPS'22 Oral

Understanding self-supervised Learning Dynamics without Contrastive Pairs

Yuandong Tian, Xinlei Chen, Surya Ganguli

ICML 2021 link, Outstanding Paper Honorable Mention

Understanding Self-supervised Learning with Dual Deep Networks

Yuandong Tian, Lantao Yu, Xinlei Chen, Surya Ganguli

arXiv link

Teacher-student setting in supervised learning

Student Specialization in Deep ReLU Networks With Finite Width and Input Dimension (./student_specialization)

Yuandong Tian

ICML 2020 link

Luck Matters: Luck Matters: Understanding Training Dynamics of Deep ReLU Networks (./luckmatter)

Yuandong Tian, Tina Jiang, Qucheng Gong, Ari Morcos

arxiv link