nkdnnlr / event-based-camera-simultaneous-mosaicing-and-tracking

Event Based Camera Project: Simultaneous Mosaicing and Tracking

Authors: Céline Nauer, Joël Bachmann, Nik Dennler

We attempt to implement the methods proposed by Kim et. al (2014) to simultaneously map the environment and track the camera pose in SO(3). For the mapping part, we are heavily guided by the MATLAB code written by G. Gallego of RPG. For the tracking algorithm, we implemented the theoretical framework of a particle filter for event cameras proposed by Kim et. al (2014).

This project is a part of the lecture 3D Vision by M. Pollefeys at ETHZ.

References:

1. H. Kim, A. Handa, R. Benosman, S.-H. Ieng, A.J. Davison, Simultaneous Mosaicing and Tracking with an Event Camera. British Machine Vision Conference, 2014.

2. H. Rebecq, T. Horstschaefer, G. Gallego, D. Scaramuzza, EVO: A Geometric Approach to Event-based 6-DOF Parallel Tracking and Mapping in Real-time. IEEE Robotics and Automation Letters (RA-L), Vol. 2, Issue 2, pp. 593-600, Apr. 2017.

3. G. Gallego, C. Forster, E. Mueggler, D. Scaramuzza, Event-based Camera Pose Tracking using a Generative Event Model. arXiv:1510.01972, 2015.

event-based-camera-simultaneous-mosaicing-and-tracking's Issues

Size out of bound

File "/Users/celine/Library/Mobile Documents/comappleCloudDocs/NSC/3D Vision/Event-Based-Camera-Simultaneous-Mosaicing-and-Tracking/Tracking_Particle_Filter/tracking.py", line 408, in
particles['logintensity_t'] = particles.apply(lambda row: intensity_map[int(row.v), int(row.u)], axis=1)
IndexError: ('index 1024 is out of bounds for axis 0 with size 1024', 'occurred at index 0')

Find bottleneck and thus make code run faster

Time every function and see which ones use the most time. Check if those functions can be rewritten to gain performance.

Particle converges on point on trajectory, why?

replace as_matrix with .values

/Users/celine/Library/Mobile Documents/comappleCloudDocs/NSC/3D Vision/Event-Based-Camera-Simultaneous-Mosaicing-and-Tracking/Tracking_Particle_Filter/tracking.py:467: FutureWarning: Method .as_matrix will be removed in a future version. Use .values instead.
rotmats[i] = sp.logm(particles['Rotation'].as_matrix()[i])

what does the tracker actually do?

Implement motion estimation

Event based camera tracking in SO(3) based on particle filtering

error message appears when mod(nr of events / batch size) = 0

Traceback (most recent call last):
File "/anaconda3/envs/project3Dvision/lib/python3.6/site-packages/pandas/core/indexes/base.py", line 2657, in get_loc
return self._engine.get_loc(key)
File "pandas/_libs/index.pyx", line 108, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/index.pyx", line 132, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/hashtable_class_helper.pxi", line 987, in pandas._libs.hashtable.Int64HashTable.get_item
File "pandas/_libs/hashtable_class_helper.pxi", line 993, in pandas._libs.hashtable.Int64HashTable.get_item
KeyError: 10000

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/celine/Library/Mobile Documents/comappleCloudDocs/NSC/3D Vision/Event-Based-Camera-Simultaneous-Mosaicing-and-Tracking/Tracking_Particle_Filter/tracking.py", line 637, in
run()
File "/Users/celine/Library/Mobile Documents/comappleCloudDocs/NSC/3D Vision/Event-Based-Camera-Simultaneous-Mosaicing-and-Tracking/Tracking_Particle_Filter/tracking.py", line 612, in run
t_batch = events.loc[event_nr]['t']
File "/anaconda3/envs/project3Dvision/lib/python3.6/site-packages/pandas/core/indexing.py", line 1500, in getitem
return self._getitem_axis(maybe_callable, axis=axis)
File "/anaconda3/envs/project3Dvision/lib/python3.6/site-packages/pandas/core/indexing.py", line 1913, in _getitem_axis
return self._get_label(key, axis=axis)
File "/anaconda3/envs/project3Dvision/lib/python3.6/site-packages/pandas/core/indexing.py", line 141, in _get_label
return self.obj._xs(label, axis=axis)
File "/anaconda3/envs/project3Dvision/lib/python3.6/site-packages/pandas/core/generic.py", line 3585, in xs
loc = self.index.get_loc(key)
File "/anaconda3/envs/project3Dvision/lib/python3.6/site-packages/pandas/core/indexes/base.py", line 2659, in get_loc
return self._engine.get_loc(self._maybe_cast_indexer(key))
File "pandas/_libs/index.pyx", line 108, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/index.pyx", line 132, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/hashtable_class_helper.pxi", line 987, in pandas._libs.hashtable.Int64HashTable.get_item
File "pandas/_libs/hashtable_class_helper.pxi", line 993, in pandas._libs.hashtable.Int64HashTable.get_item
KeyError: 10000

Implement mosaicing part

Implement mosaicing analogously to Gallego et. al. (https://github.com/uzh-rpg/rpg_image_reconstruction_from_events). Poses are assumed to be given.