xuanyuzhou98 / i-revnet-android Goto Github PK

Preliminary milstones:
#milestone 1: +batch norm + train cifar and check accuracy
#milestone 2: train imagenet, 50 accuracy, 1 week training time, 50 + accuracy

Potential directions for applications:

1. text completion, autoreply - finetune(benchmark? energy? total flops?)
2. customized image filter(style transfer), editing, tagging - maybe generative?
3. audio style transfer
4. improve camera itself

To do:

1. image tagging - pick small datasets, do image classification, image style transfer
2. audio - review wavenet paper, audio style transfer

deeplearning4j has a package for mnist iterator, but that works when we are allowed to create a folder in home directory, which is not allowed in android device. We need to figure out where to put our mnist dataset on device.

ODT Todo:

1. Train in GPU with batch norms and then delete batch norm to finetune, check accuracy. (@tianrengao )
2. Implement model weights loading from PyTorch (@xuanyuzhou98 )
3. 2, 2, 2. Train on android phone, reduce 200 to 20. Find a quick training recipe, better optimizer? (@floraxue )
4. Check 18 18 18, default gc frequency vs "decreased "gc frequency (@floraxue @tianrengao )

NLP Todo:

1. Engineering Todo:

• 把waveglow的layers一层层转写到deeplearning4j
• 在gpu上复现waveglow (@BohanZhai )

2. Research Todo:

• 把sample efficient 和 waveglow merge起来
• Design sample efficient training pipeline
• Hopefully, we can directly delete text

Applications to leverage on-device training:

Speech (recognition and synthesis, adaptation:
Speech recognition: https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/0007893.pdf
TTS:
https://openreview.net/pdf?id=rkzjUoAcFX
https://arxiv.org/pdf/1810.07217.pdf
http://people.csail.mit.edu/wnhsu/assets/pdf/is17_learning_v2_43.pdf
Dataset: https://ai.google/tools/datasets/libri-tts
Code: https://pytorch.org/hub/nvidia_deeplearningexamples_tacotron2/

Meta learning, few shot learning, domain adaptation:
Few shot image classification: https://openreview.net/pdf?id=HkxLXnAcFQ
Meta learning for supervised domain adaptation: https://arxiv.org/pdf/1711.02536.pdf
Learning to segment everything: https://arxiv.org/abs/1711.10370
Online adaptation for depth estimation: https://arxiv.org/pdf/1904.08462.pdf

Meta learning for RL:
Model based online adaptation: https://arxiv.org/pdf/1803.11347.pdf
https://arxiv.org/pdf/1812.07671.pdf

Continual learning:https://arxiv.org/pdf/1802.07569.pdf

Adaptive CV:
Problem formulation: Learn a model that can adapt to new classes/data distribution/tasks with a few target data
Data distribution (domain adaptation): Sim2Real, AmazonWeb -> Real, and so on
Few-shot: expanding to 100 shots. The goal is to fully close the performance gap with supervised learning, while reducing the amount of data and computation needed.
Milestones:
Literature search
Problem formulation:
Application scenarios
Determine how many shots
How to get source/target domain
How to benchmark
How to prepare data
Etc.
Algorithm development:
Given, say, 100 samples of target data, are we able to obtain the same accuracy as fully supervised learning? How many target data is needed?
Do existing few shot learning / domain adaptation algorithms solve this problem?
Baseline, baseline++, MAML, etc.
If they do not work well, can we propose new algorithms to solve this problem?
Move to mobile
People interested:
Xuanyu, Tianyuan

Adaptive TTS:
Problem: given a few samples of speech from a target person, transfer a TTS model to synthesize voices close to the target person
Milestones:
Literature search, identify a framework to start from
Reduce the model such that we can run inference on a mobile
Make the model to reversible so we can leverage our on-device training
Move to mobile
People:
Flora, Tianren, Bohan

Done:

• Deploy iris model to real device (@xuanyuzhou98 , 3/28)
• Write and train a convolutional NN for MNIST(@floraxue @xuanyuzhou98 , 3/28)
• Write iRev Forward Pass(@BohanZhai @tianrengao @xuanyuzhou98 , 3/28)
• Flop Counting (@floraxue @tianrengao )

Todo

• Write all the inverse function to calculate activations (@floraxue @xuanyuzhou98 @BohanZhai )
• Manually apply gradients to the coefficients (@xuanyuzhou98 @tianrengao )
• Draw the constraints graph for reversible and nonreversible(@ALL)
• Prepare for presentation