Stereo reconstruction is the concept of realization of depth from a pair of stereo images.
Stereo images are two images of the same object such that, they have parallel or slightly verged image planes.
Stereo reconstruction is used primarily in entertainment and/or 3D modelling. We are trying to obtain a usable point cloud from a pair of stereo images.
Currently the project includes disparity map and pointcloud obtained from one dataset.
Note : Code can be obtained from the "Codes" directory.
What are camera parameters and the intrinsic matrix?
The above matrix is called the intrinsic camera matrix which provides us with the camera parameters, where
fx is the focal length of the camera, fy is given as fy = a fx, where a is the aspect ratio and Cx and Cy are the offsets.
Relation between the depth and disparity :
Applying similarity of triangles on the above diagram, we obtain the relation as
Disparity = x - x' = Bf/Z
Where Z refers to the z coordinate or depth of the image. And f is the focal length.
Projections and transformations :
- Translation
- Euclidean transformation
- Similarity transformation
- Affine transformation and
- Projective transformation
To know more about the transformations, you can check out https://youtu.be/91eroryYZno
Perpective transformation matrix :
The above matrix is called the perspective projection matrix, which is a necessary parameter in the reprojectImageTo3D function of openCV.
Where Cx and Cy are offsets, f is the focal length of camera, Tx is the baseline, Cx-Cx' is the difference of offsets.
Disparity map : Disparity map refers to a mapping of the apparent shift or pixel difference of points in the two images.
Perspective transformation matrix : It is the matrix that is used to obtain the point in the image plane from a point in 3D. It takes in parameters of the camera and the baseline into account.
Point cloud : Point cloud is a set of all the points in any object.
- Creating a disparity map from the two stereo images.
- Obtain the perpective projection matrix from camera parameters.
- Obtain the pointcloud from the disparity map using the reprojectImageTo3D function of openCV.
- Visualise the pointcloud.(through meshlab in this case)
Coding was entirely done in python and the following libraries were used :
- The Numpy library
- The Matplotlib library
- The OpenCV library
- The plyfile library
Left image :
Right image :
The above is the result of the algorithm to obtain the disaprity map.
The above is the visualization of the pointcloud obtained.
Credits : The explanation on stereo geometry was referenced from OpenCV tutorials. https://docs.opencv.org/4.5.2/dd/d53/tutorial_py_depthmap.html