tudelft / event_flow

Self-Supervised Learning of Event-based Optical Flow with Spiking Neural Networks

Self-Supervised Learning of Event-based Optical Flow with Spiking Neural Networks

Work accepted at NeurIPS'21 [paper, video].

If you use this code in an academic context, please cite our work:

@article{hagenaarsparedesvalles2021ssl,
  title={Self-Supervised Learning of Event-Based Optical Flow with Spiking Neural Networks},
  author={Hagenaars, Jesse and Paredes-Vall\'es, Federico and de Croon, Guido},
  journal={Advances in Neural Information Processing Systems},
  volume={34},
  year={2021}
}

This code allows for the reproduction of the experiments leading to the results in Section 4.1.

Usage

This project uses Python >= 3.7.3 and we strongly recommend the use of virtual environments. If you don't have an environment manager yet, we recommend pyenv. It can be installed via:

curl https://pyenv.run | bash

Make sure your ~/.bashrc file contains the following:

export PATH="$HOME/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

After that, restart your terminal and run:

pyenv update

To set up your environment with pyenv first install the required python distribution and make sure the installation is successful (i.e., no errors nor warnings):

pyenv install -v 3.7.3

Once this is done, set up the environment and install the required libraries:

pyenv virtualenv 3.7.3 event_flow
pyenv activate event_flow

pip install --upgrade pip==20.0.2

cd event_flow/
pip install -r requirements.txt

Download datasets

In this work, we use multiple datasets:

• event_flow/datasets/data/training: UZH-FPV Drone Racing Dataset (Delmerico, ICRA'19)
• event_flow/datasets/data/MVSEC: Multi View Stereo Event Camera Dataset (Zhu, RA-L'18)
• event_flow/datasets/data/ECD: Event-Camera Dataset (Mueggler, IJRR'17)
• event_flow/datasets/data/HQF: High Quality Frames (Stoffregen and Scheerlinck, ECCV'20)

These datasets can be downloaded in the expected HDF5 data format from here, and are expected at event_flow/datasets/data/ (as shown above).

Download size: 19.4 GB. Uncompressed size: 94 GB.

Details about the structure of these files can be found in event_flow/datasets/tools/.

Download models

The pretrained models can be downloaded from here, and are expected at event_flow/mlruns/.

In this project we use MLflow to keep track of the experiments. To visualize the models that are available, alongside other useful details and evaluation metrics, run the following from the home directory of the project:

mlflow ui

and access http://127.0.0.1:5000 from your browser of choice.

Inference

To estimate optical flow from event sequences from the MVSEC dataset and compute the average endpoint error and percentage of outliers, run:

python eval_flow.py <model_name> --config configs/eval_MVSEC.yml

# for example:
python eval_flow.py LIFFireNet --config configs/eval_MVSEC.yml

where <model_name> is the name of MLflow run to be evaluated. Note that, if a run does not have a name (this would be the case for your own trained models), you can evaluated it through its run ID (also visible through MLflow).

To estimate optical flow from event sequences from the ECD or HQF datasets, run:

python eval_flow.py <model_name> --config configs/eval_ECD.yml
python eval_flow.py <model_name> --config configs/eval_HQF.yml

# for example:
python eval_flow.py LIFFireNet --config configs/eval_ECD.yml

Note that the ECD and HQF datasets lack ground truth optical flow data. Therefore, we evaluate the quality of the estimated event-based optical flow via the self-supervised FWL (Stoffregen and Scheerlinck, ECCV'20) and RSAT (ours, Appendix C) metrics.

Results from these evaluations are stored as MLflow artifacts.

In configs/, you can find the configuration files associated to these scripts and vary the inference settings (e.g., number of input events, activate/deactivate visualization).

Training

Run:

python train_flow.py --config configs/train_ANN.yml
python train_flow.py --config configs/train_SNN.yml

to train an traditional artificial neural network (ANN, default: FireNet) or a spiking neural network (SNN, default: LIF-FireNet), respectively. In configs/, you can find the aforementioned configuration files and vary the training settings (e.g., model, number of input events, activate/deactivate visualization). For other models available, see models/model.py.

Note that we used a batch size of 8 in our experiments. Depending on your computational resources, you may need to lower this number.

During and after the training, information about your run can be visualized through MLflow.

Uninstalling pyenv

Once you finish using our code, you can uninstall pyenv from your system by:

1. Removing the pyenv configuration lines from your ~/.bashrc.
2. Removing its root directory. This will delete all Python versions that were installed under the $HOME/.pyenv/versions/ directory:

rm -rf $HOME/.pyenv/

event_flow's Issues

"SpikingRecEVFlowNet" encoder and decoder framework for image reconstruction using event camera

@Huizerd ,

I was using your module "SpikingRecEVFlowNet" as a a network to reconstruct images from event camera rather than optical flow and according modified the input and output channels compatible to image reconstruction. After making the environmental setup. I landed with bad image reconstruction results

Left is the target image and right is the predicted image. Iam using supervised technique rather than self-supervised framework using temporal consistency and LIPS loss function.

Can you please suggest in your module "SpikingRecEVFlowNet" encoder and decoder framework what changes needs to be incorporated apart from input and output channels.
Requesting you kindly to help on this regard

confused about the implement of gradient of arctan x

event_flow/models/spiking_util.py

Line 42 in 7d90c9d

sg = 1 / (1 + width * x.abs()) ** 2

The derivative of arctan x is 1/(1 + x**2), but in your code it's 1 / (1 + width * x.abs()) ** 2.Is there anything wrong?
I appreciate your quick response.

About time-steps

Hi, don't know if you are still there~ May I ask the applied time-steps of trained SNN, according to the code and paper, I found it to be one defaultly. Have you tried to add up the time-steps? Thanks!

mlflow.exceptions.MlflowException: Run 'LIFFireNet' not found

Kindly let us know how to solve this issue after running
python eval_flow.py LIFFireNet --config configs/eval_ECD.yml
Kindly help

Inquiry Regarding GPU Usage and Parallel Computation

Dear authors,

It's me again. I'm curious about the GPU(s) you utilized for neural network training in your implementation.
And, could you provide experiment detail about the training time?
Additionally, did you consider employing multiple GPUs for parallel processing?

Thank you for your time and assistance.

Question about the data augmentation

Dear authors,

Firstly, I'd like to express my gratitude for your contributions to event-based optical flow research; it's truly remarkable work. I have been delving into your code, specifically the self-supervised event-based optical flow codebase, and I have a small question.

In dataloader/h5.py line 279 ,see below

# data augmentation
xs, ys, ps = self.augment_events(xs, ys, ps, batch)

The function augment_events() in dataloader/base.py line 88, see below

    def augment_events(self, xs, ys, ps, batch):
        """
        Augment event sequence with horizontal, vertical, and polarity flips.
        :return xs: [N] tensor with event x location
        :return ys: [N] tensor with event y location
        :return ps: [N] tensor with event polarity ([-1, 1])
        :param batch: batch index
        :return xs: [N] tensor with augmented event x location
        :return ys: [N] tensor with augmented event y location
        :return ps: [N] tensor with augmented event polarity ([-1, 1])
        """

        for i, mechanism in enumerate(self.config["loader"]["augment"]):

            if mechanism == "Horizontal":
                if self.batch_augmentation["Horizontal"][batch]:
                    xs = self.config["loader"]["resolution"][1] - 1 - xs

            elif mechanism == "Vertical":
                if self.batch_augmentation["Vertical"][batch]:
                    ys = self.config["loader"]["resolution"][0] - 1 - ys

            elif mechanism == "Polarity":
                if self.batch_augmentation["Polarity"][batch]:
                    ps *= -1

        return xs, ys, ps

To the best of my knowledge, when performing data augmentation, the dataset should ideally include both the original data and the augmented data. For instance, if we consider horizontal augmentation, both the original dataset and the horizontally augmented dataset should be present.

However, based on the provided code, it appears that only the original data is being utilized for horizontal, vertical, and polarity transformations. The dataset does not appear to include the augmented versions.
Can you explain it? Thanks in advance.

Question about the idea of average timestamp image

Dear authors, thank you for the great work. I am fairly new to optical flow research and I'm having some tough time understanding the average timestamp image loss in the paper. Can you further explain it? Thanks in advance.

To be specific, I can understand how events are warped to t_ref, optical flow and interpolations,

but I failed to understand the meaning of fw_iwe_pos_ts from here: fw_iwe_pos_ts = interpolate(fw_idx.long(), fw_weights * ts_list, self.res, polarity_mask=pol_mask[:, :, 0:1])

and I failed to get how the image create from fw_iwe_pos_ts /= fw_iwe_pos + 1e-9 can produce a loss to direct learning of optical flow estimation.

Confused about the evaluation result of my trained model

I trained a spiking neural network with the command (python train_flow.py --configs/train_SNN.yml). Then, to test my trained model, I run the command (python eval_flow.py <model_name> --config configs/eval_MVSEC.yml), where for I use my own trained model ID.
However, the estimation results of my trained model are not as good as the pre-trained model. How do I reproduce the results of the pre-trained model?
my trained model:

pretrained model:

I appreciate your quick response.^.^