lole-elol / tum_cv_challenge_ss23

TUM Computer Vision Challenge SS2023

Usage

The GUI can be started by running main.m. This will open a window where you can choose images of a room. The program will then reconstruct a 3D Model of this room.

More information about the user interface can be found in +gui/README.md

Project structure

+benchmark/
+gui/
  createGui.m
  +cb/
+logic/
  +pointcloud/
  +reconstruct3D/
+plotting/
+util/
config/
README.txt
main.m
pipeline.m

The program started using main.m. All GUI related files are located in the subfolder +gui/. The program logic is located in +logic/.

+util/ includes utility scripts for example for working with the demo data. The code in +plotting/ is for plotting related functions (usually for debugging)

+benchmark/ contains a complete toolset for benchmarking 3d reconstruction and model detection. More information can be found in +benchmark/README.md

Toolboxes:

(command: ver) MATLAB Version 9.14 (R2023a) Computer Vision Toolbox Version 10.4 (R2023a) Image Processing Toolbox Version 11.7 (R2023a) Lidar Toolbox Version 2.3 (R2023a) Medical Imaging Toolbox Version 1.1 (R2023a) Signal Processing Toolbox Version 9.2 (R2023a) Statistics and Machine Learning Toolbox Version 12.5 (R2023a)

Data

Data to large to be transmitted/handled by git is accessible over lrz.de.

• Test Data: Demo Images and Pointclouds. Download this data into the root of the project
• Benchmark Data: Results from Benchmarks. Download this into +benchmark/demoData/ to analyse the results

Coding conventions

Code Documentation

Each function should include a comment explaining what it does and defining the input an output parameters. The comment should look as follows:

% FUNCTIONNAME - short description of your function
%   Optional longer description of your function
% Inputs:
%   inputParam: Description of the input parameter
%   optionalParam = "default value": Description of the optional input parameter
% Outputs:
%   outputParam: Description of the output parameter

Naming conventions

Variables

• Use camelCase for variables.

Functions

• Use camelCase for functions.
• No indention inside a function. Only indent if/for/while statements.
• Use Name=Value for function arguments instead of "Name", "Value". E.g.:

function outputVar = helloWorld(inputVar)
% HELLOWORLD displays hello world and checks if 1 is actually 1
% Inputs:
%   inputVar: some input that is irrelebant
% Outputs:
%   outputVar: some output that is always 1
disp("hello world!");
if 1 ~= 1
  disp("Everything is a lie");
  status = callInCaseOfEmergency(destroyWorld=true)
end
outputVar = 1;
end

tum_cv_challenge_ss23's Issues

Deal with error "no essential matrix" somehow

• skip image and try next one
• try again max x times

Camera poses in multiview point in wrong direction for some reason

I don't understand why the last cameras start to point in a different direction… This needs to be checked, eventhough the 3d reconstruction seems to be fine

Plane detection

Detect and fit planes in the point cloud in order to detect walls, flor and ceiling

Use camera positions to more accuratly compute camera rotations

https://www.moodle.tum.de/mod/forum/discuss.php?d=438606

We could basically use the image positions now, to know which images are close to each other.
This should be

• integrated in the Epipolar geometry (only compute R)
• Instead of presorting, use x closest cameras to do reconstruction. So we loop through all the cameras and for each camera we compare them to the closest ones

Create a program to create a 3D reconstruction with 2 pictures

Description: Input 2 images with different sizes and calibration matrices. Output: 3D points of the features.

• Check functions of the CV toolbox to do the task
• Implement what we did in the homework, but faster and easier with the toolbox
• Only a prototype is enough, details should come later

Is filtering of the negative points in the z direction necessary?

@danielsanjosepro correct me if I am wrong, but in the function getTriangulatedPointsMultiView both outliers, and negative points in z direction are filtered. When plotting, the y axis is the height and the xz plane represents the ground. But if the points are filtered in the z direction, we lose points, am I right?

This is the change in the code in the function getTriangulatedPointsMultiView:

valid_index = valid_index & (reprojectionErrors < max_reprojection_error);

If the negative points in the z direction are not filtered, I get such a result. This is a sample with 30 images from the delivery area.

To achieve this result I am applying preprocessing, which I will push tomorrow and aggresive denosing would still be helpful, but just by the way.

Create Input Image GUI

• GUI that should be able to load n images with different formats
• The formats should be modified if possible, otherwise don't accept them
• Some feedback to show the user which pictures have been uploaded

Match featureless points

a completely flat wall has no features, so we can match points in flat walls between pictures to generate a more dense point cloud reconstruction.
-> Do something with those areas to generate points in planes

Create a program to create a 3D reconstruction with `n` pictures

• Use results of #3 with multiple pictures

Find relevant ressources and save them

Scale world points properly to compute eucledian distances

For some reason the reconstruction has wrong units now. I haven't looked at it in detail. Just adding this issue to do it soon.

Feature Calibrate Camera before/after uploading some pictures

• Somehow display a checkerboard in the app and ask the user to take 3 pictures of the checkerboard in different angles
• if we don't do this, we can not use it for unknown pictures.

Dense Point Cloud generation

Generate more points and use epipolar geometry to get better feature matching after finishing the 3D reconstrucion

Do loop closure to get better camera positions

https://de.mathworks.com/help/vision/ug/monocular-visual-simultaneous-localization-and-mapping.html (also on notion)

Doing a loop closure, we can adapt the poses of the cameras and avoid drift. If we relate image 1 to image 26 for example.
Having some sort of potential closure neighbors would be nice. The sorting of the images should be able to give us potential closures @lufixSch @lole-elol

Precompute features of all the images and then the geometry dynamically

If we fail to compute the E matrix, use next nearest possible neighbor

Add preprocessing steps for better 3D reconstruction results

• Gaussian filter to eliminate noise
• Color filtering
• Canny Edge detector for better feature pairing
• Hough Transform to find lines in the picture and use them for a better reconstruction

Coloring of the point cloud for multiview reconstruction

Create a function that uses the images and the tracks to color the 3d point cloud. This is optional but I think the visual aid helps.

Resizing images lead to a sheared 3d reconstruction

Resizing images shears reconstruction since camera parameters are wrong. Parameters should be scaled as well if that even works.
Delivery area with 2 pictures:

Delivery area with 2 pictures scaled by 0.5:

GUI to plot 3D points properly

• Basically a simple view of plotting the reconstruction points
• If possible distances should be measurable as well

Try to find out why matlab sometimes crash on ubuntu OS

What sometimes happens:

Does it happen to you as well?

3d Model Detection Pipeline

Create modelDetection pipeline which takes a point cloud and returns all useful 3d models for displaying the room

Add all 3d reconstruct scripts in `+logic` in a subfolder to bring some order

Save hyperparameters in files (json or so) to be able to load them and make different test

Find a proper solution to save all the model parameters to make different tests and have a way to know which parameters worked fine.
Maybe make it depend on the task (preprocessing, 3d reconstruction or cuboid fitting) so that we can combine multiple results easier with each other

Preprocess Point cloud

Filter outliers and cluster point cloud in order to find useful features

• remove outliers (zscore)
• cluster point cloud
• remove clusters with few points (alternative way to remove outliers)