nbarba / py3drec

3D modeling from uncalibrated images

py3DRec

This repository contains a revised implementation of the structure from motion algorithm used in Barbalios et al., "3D Human Face Modelling From Uncalibrated Images Using Spline Based Deformation", VISSAP 2008. Given at least three views of an object, and identified feature points across these views, the algorithm generates a 3D reconstruction of the object.

Overview

The algorithm takes a series of features (i.e. 2D coordinates), tracked through a sequence of images (i.e. taken with a common handheld camera), and returns the 3D coordinates of these features in the metric space. To make this happen, the algorithm consists of the following steps:

• Selecting two views (images), e.g. i-th and j-th view, the fundamental matrix and the epipoles are computed from the corresponding 2D features of the i-th and j-th view. For best results, the fartherst views in the sequence are selected.
• Estimating the projection matrices for the i-th and j-th view. To do so, the i-th view is assumed to be aligned with the world frame, and the projection matrix for the j-th view can be deduced using the fundamental matrix, the epipole and the reference frame of the reconstruction.
• Triangulation of the 2D features of the i-th and j-th views to get an initial estimate of the 3D point coordinates of the 2D features.
• Estimating the projection matrices for all the remaining views, using the 3D points we got from triangulation.
• Bundle adjustment, i.e. an overall optimization to refine the 3D points and the projection matrices by minimizing the reprojection error of the 3D points back to each view.
• Self-calibration to estimate the camera intrisic parameters for each view, and transform the 3D point coordinates from projective to metric space.
• Delaunay triangulation of the 3D points, to get a 3D structure.

Note: The algorithm used in the above referenced paper included an additional step, where the 3D points were used to deform a generic face model.

Prerequisites

The following python packages are required to run the projet:

• NumPy
• pandas
• SciPy
• numpy-stl
• pillow

Usage

Code Structure

The code consists of four classes, each one designed for a specific task:

• ImageSequence: holds all the image sequence related stuff, like sequence length, width and height of each image, 2D feature coordinates across all images. It also contains a method (i.e. show()), to visualize the image sequence with the 2D features highlighted.
• EpipolarGeometry: implements epipolar geometry related operations, such as computing the fundamental matrix, finding the epipoles, triangulation, computing homographies, and the reprojection error for one or multiple views.
• UncalibratedReconstruction: the main class that implements the 3D modeling algorithm. Constructor arguments include:
  • sequence length: the length of the image sequence
  • width: the width of images in the sequence
  • length: the length of the images in the sequence
  • triang_method: triangulation method (0: standard triangulation, 1: polynomial triangulation)
  • opt_triang: optimize triangulation result (i.e. 3D points)
  • opt_f: optimize fundamental matrix estimation
  • self_foc: for self-calibration, defines the type of focal length expected across views (0: fixed, 1: varying)
• RecModel: holds the result of the reconstruction, i.e. projection matrices, rotation matrices, translation vectors, camera intrisic parameters vectors and the 3D structure point coordinates. It also contains a method, named export_stl_file, that performs Delaunay triangulation and saves the result in stl format.

Example

A toy-example is provided in the examples folder. It is a sequence of pictures of a poster on a wall (3 images). The 2D features were automatically selected using the SIFT algorithm, and are given in a txt file that has the following format: | x coordinates | y coordinates | feature id | image id |.

To run the example, use the following command:

python uncalibrated_rec.py --input_file=./example/features_poster.txt --show

The first argument (i.e. --input_file) defines the txt file with the features, the second flag (--show) displays the image sequence together with the 2D features (remove this argument to not show the sequence).

After the algorithm is executed, two files should be generated:

• rec_model_cloud.txt : contains the 3D homogeneous coordinates of the reconstructed 3D model.
• reconstructed_model.stl: an stl file of the reconstructed 3D model

License

This project is licensed under the MIT License - see the LICENSE.md file for details

py3drec's Issues

Feature extraction questions (not the code problem)

First, thank you for your hard work to implement such interesting research.

I would like to ask a question about the feature extraction using SIFT algorithm. May you give some clues or suggestions to get the similar feature you extract? Because when I used your example dataset to extract features using OpenCV, I can't get the direct or same value similar to your feature extraction in example/features_poster.txt. If you think this is inappropriate, you can ignore this problem. Thank you, XD

TypeError: unsupported operand type(s) for +: 'method' and 'method'

Upon executing the given command in the README.md:

python uncalibrated_rec.py --input_file=./example/features_poster.txt --show

I receive:

-------------------------------
 Uncalibrated 3D Reconstruction 

--------------------------------
Traceback (most recent call last):
  File "uncalibrated_rec.py", line 495, in <module>
    main(input_file=args.input_file, show=args.show)
  File "uncalibrated_rec.py", line 431, in main
    rec_engine = UncalibratedReconstruction(sequence.length, sequence.width, sequence.height)
  File "uncalibrated_rec.py", line 43, in __init__
    self._mm = (width + height) / 2
TypeError: unsupported operand type(s) for +: 'method' and 'method'

Any recommendations? (I use Python 3.8)

Is the keypoints stored after one to one matching?

@nbarba Is the key points in example/feature_poster.txt stored after the matching procedure or raw key points as returned by the keypoint detector. I was planning to try with a different set of images. Bit confused about the process.

AttributeError: can't set attribute

I was trying to run the example code and encountered with following error. I haven't made any significant changes in the file.

Traceback (most recent call last):
File "uncalibrated_rec.py", line 472, in
main(input_file=args.input_file,show=args.show)
File "uncalibrated_rec.py", line 403, in main
sequence=ImageSequence(input_file);
File "/home/mk/Documents/3D Pose/py3DRec-master/image_sequence.py", line 10, in init
self.load_features(filename)
File "/home/mk/Documents/3D Pose/py3DRec-master/image_sequence.py", line 44, in load_features
self.feat_2d=np.zeros(shape=[self._length,4,self._number_of_features])
AttributeError: can't set attribute

Error while running the code

When i am running uncalibrated_rec.py, i get this error:

Traceback (most recent call last):
File "uncalibrated_rec.py", line 472, in
main(input_file=args.input_file,show=args.show)
File "uncalibrated_rec.py", line 406, in main
rec_engine=UncalibratedReconstruction(sequence.length,sequence.width,sequence.height)
AttributeError: 'ImageSequence' object has no attribute 'length'

what are argumets parameters used for sift function

hi,

i am not able to reproduce the same result for poster example using my sift function, can you please tell me the argument parameters used for sift function like,
nfeatures
nOctaveLayers
contrastThreshold
edgeThreshold
sigma

"> Saving model ..." The following error occurs during processing: What is the problem?

Traceback (most recent call last):
File "uncalibrated_rec.py", line 472, in
main(input_file=args.input_file,show=args.show)
File "uncalibrated_rec.py", line 451, in main
recModel.P=P
File "D:\3DS\py3DRec-master\model.py", line 33, in P
self._R, self._t = extract_rotation_translation(P)
NameError: name 'extract_rotation_translation' is not defined

SIFT algorithm for three images

Many thanks for your awesome python code. I have an issue regarding extracting image features utilizing SIFT algorithm to create the "features.txt "file for the further processing in your code. I would be grateful if you might let mew know how can I create this file with SIFT algorithm for my own three images. I used matlab but it was not as optimized as your own features for the same pictures in your py3DRec Repository.
Thank you in advance.