madaanpulkit / sageref

SageRef: Single Image Reflection Removal

SageRef

SageRef: Single Image Reflection Removal

• SageRef
• Usage
• Datasets Used
• Evaluation
  • Predictions
  • Performance Metrics
  • Models used for comparisons

Usage

$ python main.py --help
usage: main.py [-h] --gpu GPU --mode {train,eval,predict} [--epochs EPOCHS] [--latent_dim LATENT_DIM] [--out_dir OUT_DIR]
               [--data_dir DATA_DIR] [--split_dir SPLIT_DIR] [--batch_size BATCH_SIZE] [--learning_rate LEARNING_RATE]

run the relection removal experiment

optional arguments:
  -h, --help            show this help message and exit
  --gpu GPU             gpu id
  --mode {train,eval,predict}
                        mode: [train, eval, predict]
  --epochs EPOCHS       number of training epochs
  --latent_dim LATENT_DIM
                        latent space feature dimensions
  --out_dir OUT_DIR     output directory
  --data_dir DATA_DIR   data directory
  --split_dir SPLIT_DIR
                        data directory
  --batch_size BATCH_SIZE
                        batch size for training
  --learning_rate LEARNING_RATE
                        learning rate

Datasets Used

• SIR2 Benchmark Dataset
• CeilNet
• Dataset by Zhang et al

Evaluation

Predictions

# run the following code after defining img_path and module
from src.utils import predict
img = Image.open(img_path)
predict(module, img)

Performance Metrics

• PSNR (Peak Signal-to-Noise Ratio): torchmetrics.PeakSignalNoiseRatio
• SSIM (Structural Similarity Index): torchmetrics.StructuralSimilarityIndexMeasure
• LPIPS (Learned Perceptual Image Patch Similarity): torchmetrics.image.lpip.LearnedPerceptualImagePatchSimilarity

Models used for comparisons

• "Robust Reflection Removal with Reflection-free Flash-only Cues" (CVPR 2021) (pdf, github)
• "Trash or Treasure? An Interactive Dual-Stream Strategy for Single Image Reflection Separation" (NeurIPS 2021) (pdf, github)
• "Single Image Reflection Removal Exploiting Misaligned Training Data and Network Enhancements" (CVPR 2019) (pdf,github)
• "Single Image Reflection Removal with Perceptual Losses" (CVPR 2018) (pdf, github)

Acknowledgement

The network architecture and the codebase is based on Phillip Lippe's UvA Deep Learning Tutorial 9: Deep Autoencoders

sageref's Issues

Create a dataset for evaluation

This dataset should have a mix of the following:

• items that have been held out from dataset used for training
• items that are from a completely different dataset (yet relevant to our problem)
• some synthetically created images to test any remaining cases (edge cases, anomalies, etc.)

Ideally, we should have at least one dataset that is the standard in the industry for evaluating solutions similar to ours. It will help in comparison with other papers.

Deciding on broad domains for ownership

We should be deciding on broad categories to own and be responsible for, we'll still be collaborating on most of the work but ownership makes it easy to track work.

Major tasks I think are:

1. Data - @ARaps1
2. Model - @madaanpulkit
3. Evaluation - @nrishabh
4. Experiment Design & Analysis - @parthvinm
5. Demo - @parthvinm

Find a model for comparison

Find a model that has already been tested on same dataset

Create a data module using pytorch lightning

Requirements:

• Makes data accessible
• Allows for customization of the splits between training, validation and testing sets.

Fix LPIPS

LPIPS is throwing some error apparently. So, deal with that.

Adapt model to sageref usecase

nets.py and model.py currently are as is of the source, adapt them to sageref usecase

Restructure data to house inside splits file

migrate to LightningCLI

migrate from argparse to LightningCLI, to make everything cohesive and coherent

Add metrics to model evaluation

Modify lightening model object to include calculation of the finalized evaluation metrics using built-in functions

Update the PL Data Module to not include splits and transfer splits to the utils

Add Issues for ownerships

Just edit this issue and tick your respective checkbox if you are done making issues for your ownership

• Data - @ARaps1
• Model - @madaanpulkit
• Evaluation - @nrishabh
• Experiment Design & Analysis - @parthvinm
• Demo - @parthvinm

NOTE: put sprint-1 as the milestone for these issues

Add checkpoint callback

Checkpoint with

• fixed frequency
• best val metric model
• latest model (for resume functionality)

Discord Webhook Test

A test issue to see webhook working

Linter config

@ARaps1 can you point me to the source of the linter config code used?
What linter does it use?
And does this only check or does it make changes to the code as well?

The linter is failing on setup.

Create entry point file

We need a main.py file on the top level of the repo dir structure, which takes in all the args and also task to perform (train, eval, infer). This will then call files from src for the configuration provided.

Add args for experiment hparams

Use argparse for now to add hparams for experiments

Modify code to include SIR Benchmanrk Dataset

Missing `requirements.txt`

Add requirements.txt for reproducible setup

Network accepts `32 x 32` images

Change the network to accept higher res images, since we have access to those.

Finalize a couple of standard models to compare against (Benchmark Models)

Do we want to do this? I feel like this should be less of a priority for class project but important if/when we publish our work as a paper or something.

Finalize evaluation metrics and methods

Explore publications to stake out different methods of evaluating the model. Ideal case scenario would be finding a couple of metrics that are standard in the field.

Fix low SSIM

The SSIM is in the order of e-2. Find out why and fix that.

add train mode to `main.py`

Download Dataset and Place in Google Drive and Allow Downloads

We want to keep the dataset in a spot where is accessible for all. This would allow others to download the dataset.

Experimental Setup

Add the following to Readme.md once finalized:

• Dataset
• Evaluation metrics
• Algorithms/Models