cvrl / sinc-rppg

SiNC-rPPG

Non-Contrastive Unsupervised Learning of Physiological Signals from Video

Highlight paper in Conference on Computer Vision and Pattern Recognition (CVPR) 2023

Paper | Video

Figure 1: Overview of the SiNC framework for rPPG compared with traditional supervised and unsupervised learning. Supervised and contrastive losses use distance metrics to the ground truth or other samples. Our framework applies the loss directly to the prediction by shaping the frequency spectrum, and encouraging variance over a batch of inputs. Power outside of the bandlimits is penalized to learn invariances to irrelevant frequencies. Power within the bandlimits is encouraged to be sparsely distributed near the peak frequency.

Contents

• Preprocessing code for the PURE dataset is in src/preprocessing/PURE
• Training code is in src/train.py
• Testing code is in src/test.py
• Experiment config file is in src/args.py
• Loss functions are in src/utils/losses.py
• Model architectures are in src/models/
• Dataloaders are in src/datasets/
• TODO: preprocessing code for UBFC-rPPG, DDPM, and HKBU-MARs.

Installation

Install dependencies with python3:

pip install -r requirements.txt

To Run

1.) To prepare the data for training, download PURE and follow the steps in src/preprocessing/PURE

2.) Train several models with:

./scripts/train_PURE.sh

3.) Test the models with:

./scripts/test_PURE.sh

Notes

When new dataloaders are added, make sure to add them to src/datasets/utils.py so they can be selected from a corresponding command-line argument. You can run cross-dataset experiments by adding new datasets to line 30 in src/test.py.

Citation

If you use any part of our code or data, please cite our paper.

@inproceedings{speth2023sinc,
  title={Non-Contrastive Unsupervised Learning of Physiological Signals from Video},
  author={Speth, Jeremy and Vance, Nathan and Flynn, Patrick and Czajka, Adam},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2023},
}

sinc-rppg's Issues

Where is the SiNC model

Thank you very much for your work. I would like to ask where the SiNC model is, as I only found PhysNet and RPNet in the downloaded code. Additionally, I would like to inquire about the approximate FLOPs of this model.

The results of the PURE test are different from those in the paper

After training and testing on PURE, the test results are significantly different from those in the paper，The results I got are shown below，I got the result step by step according to the readme, but I don't know where I made a mistake. Could you explain why?

pure_testing
ME, MAE, RMSE, r
-1.00 $\pm$ 1.26 & 1.75 $\pm$ 1.48 & 5.82 $\pm$ 3.01 & 0.96 $\pm$ 0.03

Is this really non-contrastive learning?

Hi,

Impressive work, it looks so simple.

In my view, the core of the work consists of three types of loss: Bandwidth, Sparsity, and Variance, which are referred to as IPR, SNR, and EMD in the code. Both IPR and SNR are weakly supervised terms based on prior knowledge. To prevent model collapse into trivial solutions, an EMD term is introduced.

EMD essentially constructs negative pairs by forcing the model to output a uniformly distributed psd within a batch. This effectively compares samples within a batch to ensure their distances are maximized as much as possible. Consider this scenario: if the batch size is 1, then EMD cannot function because it's impossible to construct negative pairs.

EMD does not completely solve the problem of model collapse into trivial solutions; it cannot guarantee that psd distributions produced by models positively correlate with ground truth labels—sometimes models may just randomly distribute outputs within bandwidth.

Based on my replication results, convergence is not very stable; sometimes good outcomes can be achieved while other times they're quite poor.

Looking forward to more code releases from the author!

Sorry, where I can find the .csv file needed in the make_matadata.py

Sorry, where I can find the .csv file needed in the make_matadata.py

Experiment on UBFC cannot work

The UBFC.py is not finished.

def __getitem__(self, idx):
    raise NotImplementedError

After preprocessing the UBFC-rPPG Dataset, run train.py with UBFC, there come errors as below:

File "../SiNC/src/datasets/UBFC.py", line 95, in set_augmentations
raise NotImplementedError
NotImplementedError

Could you mind to share the pretrained model weight on PURE?

Thanks for sharing the code of this work, its really amazing! I want to test it on the real world including head movement, but I dont have the permition to access PURE Dataset now. Would you mind to share the model weight on PURE dataset?

Thanks for sharing