[TOC]
The work is based on the paper Learning Monocular Dense Depth from Events 1 as a reproduction project. Instead of using MVSEC dataset as in the original paper, we applied the model to a new dataset DSEC2.
As an extra experiment, we also tried different loss function using Structural similarity (SSIM) loss, which yields lower metric error.
An event camera, also known as a neuromorphic camera3, silicon retina4 or dynamic vision sensor5, is an imaging sensor that responds to local changes in brightness. Conventional cameras mainly use a shutter to capture images. However, event cameras mainly capture images with each pixels operating independently and asynchronously. It will only report changes of intensity at the pixel level.
Event cameras have a rather high dynamic range, mostly about 120dB6, which is about four times of the human eyes'. Event cameras are thus sensors that can provide high-quality visual information even in challenging high-speed scenarios and high dynamic range environments, enabling new application domains for vision-based algorithms. Recently, event cameras have received great interest in object recognition, autonomous vehicles, and robotics7.
Fig. 1 Event camera output verses standard camera outputDepth prediction is the visual ability to perceive the world in three dimensions (3D) and the distance of an object. It has a rather important role in robotics and the automotive industry.
However, while event cameras have appealing properties, they also present unique challenges. Usually, the boundaries of the scene are not clearly prominent due to its working principle. Therefore, it needs to be filled in low contrast regions where no events are triggered, which makes the prediction much challenging.
Early works on event-based depth estimation used multi-view stereo and later Simultaneous Localization and Mapping (SLAM) to build a representation of the environment and therefore derivate metric depth. But these methods either rely on the expensive sensors or use the scale given by available camera poses. Purely vision-based approaches have investigated depth estimation using stereo event cameras, where they rely on maximizing temporal (as opposed to photometric) consistency between a pair of event camera streams to perform disparity and depth estimation.
In the work 1, they mainly focus on dense, monocular, and metric depth estimation using an event camera, which addresses the aforementioned limitations. This is the first paper that dense monocular depth is predicted using only events.
The output of an ideal event camera is a stream of asynchronous events. A event
Due to the sparse and asynchronous property of event stream, it's hard to feed into the neural network. Therefore, pre-processing event stream to other dense tensor-like representations while preserving its time information should be considered. In this paper, the authors use a novel representation named as spatio-temporal voxel grid. First, the input event stream is grouped into several fixed-size non-overlapping sliding windows based on their time
Then the event stream can be converted to a
As shown in the figure above, the network we produced has a recurrent, fully convolutional structure. It can simply be divided as a header, an encoder, residual blocks, a decoder and a predictor.
This network structure draws on UNet, which has proven its potential on Frame-based Depth Estimation tasks. The skip connections between encoder and decoder make sure features are learned for contracting the image can be used for reconstruction. Therefore, it guarantees the performance for pixel-by-pixel prediction. Due to the flow mechanism of event data, the input voxel grid shares some information with other grids, therefore, LSTM is used to discover temporal features.
The header of this network is a 2D convolutional layer followed by Batch Normalization. The kernel size is set to be 5. The activation function is ReLU. It takes 15x640x480 event volumn as input. The encoder consists of three similar layers with different channel size. Each layer has a 2D convolutional layer and a ConvLSTM layer, which has a LSTM structure with a convolultional gate. The kernel size of convolutional layer is 5, and that of ConvLSTM is selected to be 3. After the encoder is 2 cascade residual layers with kernel size 3. In the residual layer there are 2 convolutional network with Batch Normalization. The activation function is ReLU. Summation is applied over the skip connection. The decoder has three similar layers with different output channel size. Each layer consists of an upsampling convolution and a normal convolution with kernel size 5. Finally, the network use a predictor to output, which is a depth-wise convolution with kernel size 1. This network applies summation over all the skip connections. States from the ConvLSTM will be used for the next event volumn.
The network is trained to output log depth maps
which have the advantage of representing large depth variations in a compact range, facilitating learning.
$$ \hat{\mathcal{D}}{m, k}=\mathcal{D}{\max } \exp \left(-\alpha\left(1-\hat{\mathcal{D}}_{k}\right)\right) $$
where $\mathcal{D}{\max }$ is the maximum depth range of prediction. $α$ is a parameter chosen, such that a log depth value of 0 maps to minimum observed depth. We take $\mathcal{D}{\max }=80 m$ and
Therefore, the ground truth disparity is converted firstly to depth
$$ \mathcal{D}m=\frac{T{x} f}{-\left[d-\left(c_{x}-c_{x}^{\prime}\right)\right]} $$
where
$$ \mathcal{D}_{log}=\frac{log(\frac{\mathcal{D}m}{\mathcal{D}{\max }})}{\alpha}+1 $$
Here we use
The network is trained to minimize the scale-invariant loss $\mathcal{L}{k, \text{si}}$ and multi-scale scale-invariant gradient matching loss $\mathcal{L}{k, \text{grad}}$ at each time step
$$ \mathcal{L}{\text{tot}}=\sum{k=0}^{L-1} \mathcal{L}{k, \text{si}}+\lambda \mathcal{L}{k, \text{grad}} $$
where
$$ \mathcal{L}{k, \text{si}}=\frac{1}{n} \sum{\mathbf{u}}\left(\mathcal{R}{k}(\mathbf{u})\right)^{2}-\frac{1}{n^{2}}\left(\sum{\mathbf{u}} \mathcal{R}_{k}(\mathbf{u})\right)^{2} $$
where
The multi-scale scale-invariant gradient matching loss
$$ \mathcal{L}{k, \operatorname{grad}}=\frac{1}{n} \sum{s} \sum_{\mathbf{u}}\left|\nabla_{x} \mathcal{R}{k}^{s}(\mathbf{u})\right|+\left|\nabla{y} \mathcal{R}_{k}^{s}(\mathbf{u})\right| $$
where
We have chosen the same hyper-parameters as in the paper1. There are totally 4 scales for
The training dataset is different from what reported in the paper. In the paper they trained their network on DENSE and MVSEC^[https://daniilidis-group.github.io/mvsec/]. DENSE data set is a synthetic dataset recorded from a simulation environment.
Our training dataset only includes subfiles from DSEC dataset 2, namely zurich_city_00_b
and interlaken_00_c
. DSEC offers data from a wide-baseline stereo setup of two color frame cameras and two high-resolution monochrome event cameras. In this project, we only use the data from the left event camera as training data and the disparity as ground truth.
A significant drawback of MVSEC for stereo algorithms is the small resolution of the DAVIS346 cameras (346 x 260 pixels, 0.1 Megapixels) and the small baseline of 10 centimeters on top of that. On the other hand, DSEC dataset provides the resolution of 0.3 Megapixels (640×480) and a larger baseline of 60 centimeters on top of that. The depth error is approximately inversely proportional to the product of the baseline and focal length. Therefore, such improvements can provide us with more accurate data.
Fig. 3 Event data and image data from DSEC datasetDue to the limitation on computational resources, we only use a batch size of 4.
Same as in the paper, we train with a learning rate of
But because of the time limitation of Colab, the training process is not continous. We have to wait 12 hours for the Colab to be available. As shown in the following loss plots, the loss curves are not continous because of this problem.
Fig. 4 Training loss Fig. 5 Validation lossBesides the loss terms introduced previously, we also try adding SSIM loss as another term. SSIM is a perception-based model that considers image degradation as perceived change in structural information, while also incorporating important perceptual phenomena, including both luminance masking and contrast masking terms9.
The measure between two windows
-
$\mu_{x}$ the average of$x$ ; -
$\mu_{y}$ the average of$y$ ; -
$\sigma_{x}^{2}$ the variance of$x$ ; -
$\sigma_{y}^{2}$ the variance of$y$ ; -
$\sigma_{x y}$ the covariance of$x$ and$y$ ; -
$c_{1}=0.01^2, c_{2}=0.03^2$ as hyper-parameters.
SSIM generally considers the similarity between 2 images and outputs the similarity index in range
Hence the loss function becomes
$$ \mathcal{L}{\text{with_ssim}}=\sum{k=0}^{L-1} \mathcal{L}{k, \text{si}}+\lambda{grad} \mathcal{L}{k, \text{grad}}+\lambda{ssim} (1-\mathcal{SSIM}k) $$ where $\lambda{ssim}$ is the SSIM loss weight, setted as 0.05.
SSIM loss term further promotes the similarity between predicted depth map and ground truth depth map. Similarly, we calculate this loss only on valid pixels by making invalid pixels value zero during loss calculation, in both ground truth label and prediction output. The SSIM loss implementation was built upon pytorch_ssim
^[https://github.com/Po-Hsun-Su/pytorch-ssim].
Due to the training code is missing in the official repository, we managed to build this neural network from scratch to reproduce the results the reported. Our repository can be found here
The results are visualizaed in inverse depth. Higher intensity denotes lower depth (closer distance).
Lower value indicates better result for all metrics.
Metric | Total Mean absolute error(MAE) | Total Mean square error(MSE) |
---|---|---|
Performance | 16.84 | 380.07 |
Model types | Event based | Event based | Frame based |
---|---|---|---|
Metrics | Paper model1 | Our model | MonoDepth12 |
MAE in 10m | 1.85 | 5.32 | 3.44 |
MAE in 20m | 2.64 | 8.94 | 7.02 |
MAE in 30m | 3.13 | 12.04 | 10.03 |
In the paper, they trained 300 epoches (127800 iterations) on the real world dataset consist of 8523 samples. Due to limited computation resources, we trained 50 epoches on a smaller dataset with 225 samples only. Each sample is a 50ms time window consist of 5,000 to 500,000 events. Though produced reasonable results, our model still underperforms the baseline models.
We trained the model again with SSIM loss term. After only 7 epochs, we already observe lower metric errors. For comparison, we also overlay the orignal training metric curves without SSIM loss. The metric plots for the first 7 epochs are shown as follow. As you can see, the model with SSIM loss (dark blue) performs better than original model (light blue), which Mean Average Error for all stages decreases more slowly.
Fig. 8 MAE in 10m Fig. 9 MAE in 20m Fig. 10 MAE in 30m Fig. 11 Total MAE Fig. 12 Total MSEWith SSIM loss term, the metric errors descend to low values after training for only 7 epochs.
Model | MAE | MSE | MAE in 10m | MAE in 20m | MAE in 30m |
---|---|---|---|---|---|
Model trained with SSIM loss | 13.84 | 339.28 | 3.22 | 4.83 | 7.85 |
Original model | 20.43 | 616.21 | 5.08 | 9.09 | 13.43 |
Due to time limit, we did not train the model with SSIM loss again for 50 epochs. But the model metrics are already better than the previous results. This implies the SSIM loss could not only be used for training on image data from frame-based camera, but also be applied to event data.
- Suffering from limited computational resources, the network is trained on relatively small amount of data. The total number of images in the training set is 900 (225 batch samples). This is one of the main reason why our model underforms their original model.
- One way to boost the training process is to convert all the ground truth disparity data to log depth data in advance, and store it locally. Instead of converting the disparity data during training and validation, using the pre-computed log depth data directly can save computations.
- Although our model use sparse event streams as input data, the sparse depth images as the ground truth, by carefully selecting losses, we finally obtain dense depth estimations.
- The original paper uses synthetic data from simulation and performs data augumenation by random cropping and horizontal flip of the training samples. This could also be one the reason why our model underperforms theirs, since we only use real events from the dataset.
- The model captures the large scale features (e.g. depth on ground and buildings) very well. But the small scale features are not well captured yet. As shown in the following picture, the pedestrain's outline is not clear. This could be one of the main sources of error.
- We did not train enough with SSIM loss due to time limit. This could be a problem as we do not know the final performance of the model after sufficient training. Thus, from the metrics for the first 7 epochs, we could only suggest SSIM loss could potentially be useful in training depth estimation model on event data.
We have successfully reproduce the paper Learning Monocular Dense Depth from Events, enabling depth prediction using monocular event camera data. Instead of using the MVSEC dataset, we applied the model to another dataset, namely DSEC dataset.
Though not as good as shown in the paper, the model still achieves fairly good result in the evaluation. We also fill the blank of training code which is not given in the original code website.
Finally, we tried adding SSIM loss term in training, which yields lower metric errors. Thus, SSIM loss could potentially be a useful loss term to boost training and to improve performance.
Footnotes
-
J. Hidalgo-Carrió, D. Gehrig and D. Scaramuzza, "Learning Monocular Dense Depth from Events," 2020 International Conference on 3D Vision (3DV), 2020, pp. 534-542, doi: 10.1109/3DV50981.2020.00063. ↩ ↩2 ↩3 ↩4
-
Gehrig, M., Aarents, W., Gehrig, D., and Scaramuzza, D., “DSEC: A Stereo Event Camera Dataset for Driving Scenarios”, arXiv e-prints, 2021. ↩ ↩2
-
Li, Hongmin; Liu, Hanchao; Ji, Xiangyang; Li, Guoqi; Shi, Luping (2017). "CIFAR10-DVS: An Event-Stream Dataset for Object Classification". Frontiers in Neuroscience. 11: 309. doi:10.3389/fnins.2017.00309. ↩
-
Sarmadi, Hamid; Muñoz-Salinas, Rafael; Olivares-Mendez, Miguel A.; Medina-Carnicer, Rafael (2021). "Detection of Binary Square Fiducial Markers Using an Event Camera". IEEE Access. 9: 27813–27826. arXiv:2012.06516. doi:10.1109/ACCESS.2021.3058423 ↩
-
Liu, Min; Delbruck, Tobi (May 2017). "Block-matching optical flow for dynamic vision sensors: Algorithm and FPGA implementation". 2017 IEEE International Symposium on Circuits and Systems (ISCAS). pp. 1–4. arXiv:1706.05415. doi:10.1109/ISCAS.2017.8050295. ↩
-
Longinotti, Luca. "Product Specifications". iniVation. Retrieved 2019-04-22. ↩
-
Hambling, David. "AI vision could be improved with sensors that mimic human eyes". New Scientist. Retrieved 2021-10-28. ↩
-
Eigen, D., Puhrsch, C., and Fergus, R., “Depth Map Prediction from a Single Image using a Multi-Scale Deep Network”, arXiv e-prints, 2014. ↩
-
Wikipedia contributors. "Structural similarity." Wikipedia, The Free Encyclopedia. Wikipedia, The Free Encyclopedia, 8 May. 2022. Web. 15 Jun. 2022. ↩
-
Z. Wang, E. P. Simoncelli and A. C. Bovik, "Multiscale structural similarity for image quality assessment," The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003, 2003, pp. 1398-1402 Vol.2, doi: 10.1109/ACSSC.2003.1292216. ↩
-
Sagar, A., “Monocular Depth Estimation Using Multi Scale Neural Network And Feature Fusion”, arXiv e-prints, 2020. ↩
-
Godard, C., Mac Aodha, O., and Brostow, G. J., “Unsupervised Monocular Depth Estimation with Left-Right Consistency”, arXiv e-prints, 2016. ↩