jwkanggist / self-supervised-learning-narratives-1

거꾸로 읽는 self-supervised learning 파트 1

[스터디 종료] 거꾸로 읽는 self-supervised-learning 파트1

• 정말 세상에 좋은 모델이 너무 많고, 유명한 논문도 너무 많습니다.
• 우리는 현시점에서 지난 2년간 정말 역사적으로 의미가 있었던 self-supervised learning 논문을 다시 봅니다.
• 해당 논문에서 제시하는 메소드의 특징 그리고 역사적으로 평가되는 이유에 대해서 즐겁게 토론하는 시간을 가집니다.
• 우리가 매주 모이는 zoom link

기간

• 2022년 4월 16일 ~ 2022년 06월 04일
• 오전 10시30분 ~ 오후 12시

발표 논문 및 순서

google sheet

Representation learning for clustering

	paper title	논문 발표	코드 분석
1주차	DEC: Unsupervised Deep Embedding for Clustering Analysis	강재욱	강재욱
2주차	DAC: Deep Adaptive Image Clustering (논문리뷰영상), (코드리뷰영상)	김건	김세용
3주차	DeepClustering: Deep Clustering for Unsupervised Learning of Visual Features (논문리뷰영상), (코드리뷰영상)	김효실	김희은

Representation learning 코어 기술

	paper title	논문 발표	코드 분석
4주차	VQ-VAE: Neural Discrete Representation Learning (논문리뷰영상)	김기범	황중원
5주차	CPC v1: Representation Learning with Contrastive Predictive Coding (논문리뷰영상), (코드리뷰영상)	정두해	김세용

Instance-level contrastive learning

	paper title	논문 발표	코드 분석
6주차	MoCo v1: Momentum Contrast for Unsupervised Visual Representation Learning (논문리뷰영상)	오진우
7주차	SwAV: Unsupervised Learning of Visual Features by Contrasting Cluster Assignments (논문리뷰영상)	박성훈

Self-supervised learning withtout negative examples

	paper title	논문 발표	코드 분석
8주차	SimCLR: A Simple Framework for Contrastive Learning of Visual Representations (논문리뷰영상), (코드리뷰영상)	이영수	이동형
9주차	warp-up	강재욱

관련 링크

• 모두의 연구소 풀잎스쿨 페이지

self-supervised-learning-narratives-1's Issues

[5주차] Representation Learning with Contrastive Predictive Coding 코드 분석

코드 링크

https://github.com/rest1h/Contrastive-Predictive-Coding

논문 링크

https://arxiv.org/pdf/1807.03748.pdf

우선 튜토리얼만 업로드 했습니다.
원본 코드를 재현 할 수 있도록 수정한 부분을 내일까지 업데이트 할 예정입니다.

영상 링크

https://youtu.be/6_XlRoJWU_U

[5주차] Representation Learning with Contrastive Predictive Coding

Keywords

CPC, InfoNCE, Predictive Coding, Negative Sampling

TL;DR

• Predictive coding과 NCE를 활용하여 여러 도메인에 적용될 수 있는 unsupervised 학습 방법 제시
• 기존 binary 비교 방식의 NCE에서 확장하여 한 입력 데이터 내에서 복수의 negative sample들과 하나의 positive sample을 비교하는 방식의 InfoNCE 제시
• 정보 이론 관점에서 negative sampling에 대한 수학적 분석

Abstract

While supervised learning has enabled great progress in many applications, unsupervised learning has not seen such widespread adoption, and remains an important and challenging endeavor for artificial intelligence. In this work, we propose a universal unsupervised learning approach to extract useful representations from high-dimensional data, which we call Contrastive Predictive Coding. The key insight of our model is to learn such representations by predicting the future in latent space by using powerful autoregressive models. We use a probabilistic contrastive loss which induces the latent space to capture information that is maximally useful to predict future samples. It also makes the model tractable by using negative sampling. While most prior work has focused on evaluating representations for a particular modality, we demonstrate that our approach is able to learn useful representations achieving strong performance on four distinct domains: speech, images, text and reinforcement learning in 3D environments.

Paper link

https://arxiv.org/abs/1807.03748

Presentation link

https://docs.google.com/presentation/d/1QDXmJL5YvycXf8vL-OYP61FKVTUecNqk/edit?usp=sharing&ouid=114847754426815005538&rtpof=true&sd=true

video link

https://youtu.be/vgzDpgxDVGQ

[2주차] DAC: Deep Adaptive Image Clustering

Keywords

pairwise binary classification, clustering

TL;DR

Abstract

Image clustering is a crucial but challenging task in machine learning and computer vision. Existing methods often ignore the combination between feature learning and clustering. To tackle this problem, we propose Deep Adaptive Clustering (DAC) that recasts the clustering problem into a binary pairwise-classification framework to judge whether pairs of images belong to the same clusters. In DAC, the similarities are calculated as the cosine distance between label features of images which are generated by a deep convolutional network (ConvNet). By introducing a constraint
into DAC, the learned label features tend to be one-hot vectors that can be utilized for clustering images. The main challenge is that the ground-truth similarities are unknown in image clustering. We handle this issue by presenting an alternating iterative Adaptive Learning algorithm where each iteration alternately selects labeled samples and trains
the ConvNet. Conclusively, images are automatically clustered based on the label features. Experimental results show that DAC achieves state-of-the-art performance on five popular datasets, e.g., yielding 97.75% clustering accuracy on MNIST, 52.18% on CIFAR-10 and 46.99% on STL-10.

Paper link

https://openaccess.thecvf.com/content_ICCV_2017/papers/Chang_Deep_Adaptive_Image_ICCV_2017_paper.pdf

Presentation link

video link

[2주차] DAC 코드 리뷰

코드 링크를 공유 드립니다.

r 값을 None으로 처리하는 부분에 대해서 혹시 좋은 아이디어 있으시면 알려주시면 감사하겠습니다~!

구현 중인 코드

https://github.com/rest1h/self-supervised-learning-narratives-1/tree/main/week1_DAC

저자분 코드

https://github.com/vector-1127/DAC

참고한 깃헙

https://github.com/JiaxinZhuang/DAC.Pytorch
https://github.com/graykode/DAC
https://github.com/GuHongyang/DAC-pytorch/blob/master/main.py

유튜브 링크

https://youtu.be/Yp9GgiBKzts

[6주차] Momentum Contrast for UnsupervisedVisual Representation Learning

Keywords

Contrastive Learning, Dynamic Dictionary, Momentum Update, InfoNCE Loss, Instance Discrimination Task

TL;DR

MoCo는 queue 기반의 dictionary를 구축하여 large-size negative pair를 만들어 주고 momentum update을 이용해 key representation의 consistency를 최대화 하여 contrastive learning 성능을 높인 모델이다.

Abstract

We present Momentum Contrast (MoCo) for unsupervised visual representation learning. From a perspective on contrastive learning as dictionary look-up, we build a dynamic dictionary with a queue and a moving-averaged encoder. This enables building a large and consistent dicionary on-the-fly that facilitates contrastive unsupervised learning. MoCo provides competitive results under the common linear protocol on ImageNet classification. More importantly, the representations learned by MoCo transfer well to downstream tasks. MoCo can outperform its supervised pre-training counterpart in 7 detection/segmentation tasks on PASCAL VOC, COCO, and other datasets, some- times surpassing it by large margins. This suggests that the gap between unsupervised and supervised representation learning has been largely closed in many vision tasks.

Paper link

http://openaccess.thecvf.com/content_CVPR_2020/html/He_Momentum_Contrast_for_Unsupervised_Visual_Representation_Learning_CVPR_2020_paper.html

Presentation link

https://drive.google.com/file/d/1t6vWM37Walz2S2Hojche_RG_A8KvN1Mx/view?usp=sharing

video link

https://youtu.be/S-FsnPxAEFA

[7주차]SwAV: Unsupervised Learning of Visual Features by Contrasting Cluster Assignments

Keywords

Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
Sinkhorn distance, transportation polytope

Abstract

Augmentation 을 하여, qs-->vt, qt-->vs 로 추정하기.
이 과정에서 transportation polytope 로 q, probability matrix 를 최적화시키려는 시도가 있었음.

Paper link

논문 원본 링크: https://proceedings.neurips.cc/paper/2020/file/70feb62b69f16e0238f741fab228fec2-Paper.pdf

Presentation link

https://docs.google.com/presentation/d/1ycPIdLBBzMWptNa37Xzh4a52hTOY4EF4/edit?usp=drive_web&ouid=109708923245527629092&rtpof=true

video link

https://youtu.be/6u_yt_Ql3qs

개인적으로 하고 싶은 말(박성훈님)

다양한 인용 논문을 통해 이해하려고 시도했으나 쉽지 않았음.
어떤 contribute 라도 감사합니다.

이메일은 [email protected]으로 보내주세요!

[220430] 공지사항

• 지난주 이슈 올리기 : @Lightning2
• 지난주 영상 먼저 여기 분들께 공개하겠습니다.
• 자료화면 더 키우기
• 유투브 채널에 뷰어 권한으로 초대 드림

오리엔테이션

• 해당 클래스의 목적
• 구성원 소개 (이름, 하시는일, 이유)
  • 구성원 정보 채우기
• 진행 방식 소개
  • 전체 논문 리스트 리뷰
  • 발표자 모집 (첫번째 클래스가 끝날때 까지 시트 채우기)
    • 녹화 진행 여부 동의
      • 녹화자와 진행자의 화면 고정으로 진행
      • 진행자의 화면고정은 질문시에만 진행
    • 발표자료 및 레포 공개 동의
  • 진행 방식
    • 이슈등록 기반의 방식
    • 퍼실과 대담형식의 발표
      • 논문 읽기
      • 코드 리뷰
    • 질문 및 토론
      • 채팅과 이슈기능의 적극적 활용

출석 및 사전학습 체크는 따로 안하겠습니다. :)

• https://modulabs.co.kr/mypage-nav/#mlfmyp_open_flip_facil_detail_view?237910

[1주차] DEC: Unsupervised Deep Embedding for Clustering Analysis

Keywords

• DEC, clustering

TL;DR

• 단순히 feature 간의 거리를 이용하여 clustering을 하는 것이 아니라, 데이터의 feature representations와 cluster assignments를 Neural Network를 통해 동시에 학습할 수 있는가?
  • YES
    • target likelihood distribution을 student-t로 모델링하고
    • backprob을 통해서 student-t의 mean과 encoder 파라미터를 동시에 업데이트

Abstract

Clustering is central to many data-driven application domains and has been studied extensively in terms of distance functions and grouping algorithms. Relatively little work has focused on learning representations for clustering. In this paper, we propose Deep Embedded Clustering (DEC), a method that simultaneously learns feature representations and cluster assignments using deep neural networks. DEC learns a mapping from the data space to a lower-dimensional feature space in which it iteratively optimizes a clustering objective. Our experimental evaluations on image and text corpora show significant improvement over state-of-the-art methods.

Paper link

https://proceedings.mlr.press/v48/xieb16.pdf

Presentation link

https://www.notion.so/jwkangmarco/Unsupervised-Deep-Embedding-for-Clustering-Analysis-398f1d4910b5446981fd054395798fcd

video link

[3주차] Deep Clustering for Unsupervised Learning of Visual Features

Keywords

unsupervised learning, clustering

TL;DR

Abstract

Clustering is a class of unsupervised learning methods that has been extensively applied and studied in computer vision. Little work has been done to adapt it to the end-to-end training of visual features on large scale datasets.
In this work, we present DeepCluster, a clustering method that jointly learns the parameters of a neural network and the cluster assignments of the resulting features. DeepCluster iteratively groups the features with a standard clustering algorithm, kmeans, and uses the subsequent assignments as supervision to update the weights of the network. We apply DeepCluster to the unsupervised training of convolutional neural networks on large datasets like ImageNet and YFCC100M. The resulting model outperforms the current state of the art by a significant margin on all the standard benchmarks.

Paper link

https://arxiv.org/pdf/1807.05520.pdf

Presentation link

https://drive.google.com/file/d/14pCf05Y6Q5vHUNORciUMYVxhOm6EP_Fi/view?usp=sharing

video link

https://youtu.be/CwdevCWglJk

[3주자] Deep Clustering 코드 분석

[4주차] VQ-VAE 코드 분석

구현 중인 코드

https://github.com/HwangJohn/vq-vae-test

저자분 코드

• v1: https://github.com/deepmind/sonnet/blob/v1/sonnet/examples/vqvae_example.ipynb
• v2: https://github.com/deepmind/sonnet/blob/v2/examples/vqvae_example.ipynb

참고한 깃헙

• https://github.com/zalandoresearch/pytorch-vq-vae
• https://avdnoord.github.io/homepage/
• https://github.com/deepmind/sonnet/blob/v2/examples/vqvae_example.ipynb

참고하면 좋은자료

• https://github.com/ritheshkumar95/pytorch-vqvae/blob/master/final_project.pdf

유튜브 링크

• https://www.youtube.com/watch?v=VZFVUrYcig0

[220423] 공지사항

• 진행방식
  • 1. 이슈 등록 (적어도 금요일 저녁)
  • 2. 발표자는 카메라 온 (모두 카메라온 장려)
  • 3. 리액션!
  • 4. 줌 이름은 본명 부탁드립니다.
  • 5. 스케쥴 조정 (한해연님)
• 발표 진행
  • 자기소개
  • 화면 띄우기
  • 진행자가 중간중간 정리할수 잇음

우리는 풀잎스쿨

• 논문은 같이 읽어와야 재밌다!
• SNS 공유는 녹화영상 검수 후 공유 예정

test

[4주차] Neural Discrete Representation Learning(VQ-VAE)

Keywords

VAE, Autoencoder, unsupervised learning

TL;DR

Vector QuantisedVariational AutoEncoder (VQ-VAE)를 소개한 논문.
이산 잠재 표현(discrete latent representation)을 학습하기 의해 vector quantization의 아이디어를 사용했다.

Abstract

Learning useful representations without supervision remains a key challenge in machine learning. In this paper, we propose a simple yet powerful generative model that learns such discrete representations. Our model, the Vector QuantisedVariational AutoEncoder (VQ-VAE), differs from VAEs in two key ways: the encoder network outputs discrete, rather than continuous, codes; and the prior is learnt rather than static. In order to learn a discrete latent representation, we incorporate ideas from vector quantisation (VQ). Using the VQ method allows the model to circumvent issues of “posterior collapse” - where the latents are ignored when they are paired with a powerful autoregressive decoder - typically observed in the VAE framework. Pairing these representations with an autoregressive prior, the model can generate high quality images, videos, and speech as well as doing high quality speaker conversion and unsupervised learning of phonemes, providing further evidence of the utility of the learnt representations.

Paper link

https://arxiv.org/abs/1711.00937

Presentation link

https://github.com/isingmodel/TIL/blob/main/2022/05_07_Neural_Discrete_Representation_Learning/Neural%20Discrete%20Representation%20Learning.pdf

video link

https://youtu.be/tF1WSN-11PQ