manurare / 360monodepth

Code release for 360monodepth. With our framework we achieve monocular depth estimation for high resolution 360° images based on aligning and blending perspective depth maps.

360MonoDepth

Paper | Project Page

This is the code for 360MonoDepth: High-Resolution 360° Monocular Depth Estimation

Manuel Rey-Area*, Mingze Yuan*, Christian Richardt
University of Bath
*denotes equal contribution
CVPR 2022

Setup

Tested with Python >= 3.8

Dependencies for C++ code:

• Ceres 2.0.0
• Eigen 3.3.9
• Glog 0.5.0
• Gflags 2.2.2
• GTest 1.10.0
• OpenCV 4.2.0
• Boost 1.75.0
• pybind11 2.8.1

Dependencies for python are in code/python/requirements.txt

With Docker

We recommend Docker to run 360MonoDepth to avoid problems with dependencies.

docker build -t 360monodepth .
docker run -it --runtime=nvidia -e NVIDIA_VISIBLE_DEVICES=0 360monodepth sh -c "cd /monodepth/code/python/src; python3 main.py --expname test_experiment --blending_method all --grid_size 8x7"

Without Docker

1. We need to create a conda environment with python 3.8 and build the C++ targets

conda create -n 360monodepth python=3.8
conda activate 360monodepth
pip install -r code/python/requirements.txt

2. Build pybind11 in code/cpp/3rd_party first (or apt-get install python3-pybind11). Then, modify cmakeconfig and code/cpp/python/setup.py to add own paths to libraries/includes

cd code/cpp
mkdir build && cd build
cmake -DCMAKE_BUILD_TYPE=Release ../
make -j8

3. Copy all the dependent DLL/so files to code/cpp/python/[dll,so]. For example, in Linux code/cpp/python/so should contain the following dynamic libraries: libamd.so.2, libcholmod.so.3, libglog.so, libm.so.6, libsuitesparseconfig.so.5, libblas.so.3, libcolamd.so.2, libglog.so.0, libopencv_core.so.4.2, libtbb.so.2, libcamd.so.2, libcxsparse.so.3, libgomp.so.1, libopencv_imgproc.so.4.2, libccolamd.so.2, libgflags.so.2.2, liblapack.so.3, libquadmath.so.0, libceres.so.2, libgfortran.so.5, libmetis.so.5, libspqr.so.2

cd code/cpp/python
python setup.py build
python setup.py bdist_wheel
pip install code/cpp/python/dist/instaOmniDepth-0.1.0-cp38-cp38-linux_x86_64.whl

OPTIONAL: To add support for BoostingMonocularDepth

git submodule update --init

And download the required weights as indicated in their README.

Running code

Always execute this command per new instance of shell.

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/path/to/conda/envs/360monodepth/lib/python3.8/site-packages/instaOmniDepth

Now, we can run the code!

cd code/python/src
python main.py --expname test_experiment --blending_method all --grid_size 8x7

Data

Data must be provided with the --data flag and must be a .txt file with the following structure:

/path/to/dataset/filename00_rgb.[png,jpg] /path/to/dataset/filename00_depth.dpt
/path/to/dataset/filename01_rgb.[png,jpg] /path/to/dataset/filename01_depth.dpt
		.				          .
		.				          .
		.				          .

An example can be found at data/erp_00_data.txt. In case of using data without GT, None should be written in the second column.

Citation

@inproceedings{reyarea2021360monodepth,
	title={{360MonoDepth}: High-Resolution 360{\deg} Monocular Depth Estimation},
	author={Manuel Rey-Area and Mingze Yuan and Christian Richardt},
	booktitle={CVPR},
	year={2022}}

360monodepth's Issues

Fine-tune/train model on custom dataset

Hi,
I want to fine-tune or train the model on the custom dataset.

Could you please give some guidance?

Thanks,
Hieu

About Dpt File

1. Is the [0001_depth.dpt] file created at runtime or should it be provided as input?

2. Could anyone provide information about the .dpt file used here?

3. Should the .dpt file be provided as input along with the 360 image, or is the 360 image alone sufficient?

your insights are greatly appreciated

Increase number of images step 1

Hi @manurare, currently the model takes 20 images from the faces of an icosahedron.
Could you recommend me ways in which I can increase the number of images (close to 100 or 200) and the code to do the same?

I am aware that project_icosahedron.py implements the required projections. More details will be super helpful.
Thanks!

Question about the depth image in Replica360 dataset

Hi 360monodepth team, awesome work!
I follow the method to produce my customed replica360 dataset as mentioned in your work. The panoramic rgb image looks ok, but the depth image is likely been truncated cause of inappropriate depth scale or depth image format.
Specificly, I use matryodshka-replica360 code to render panoramic depth images, I find the depth image can be naturally unprojected to a unit sphere, the we can get the hole room pointcloud at once.

./ReplicaSDK/ReplicaRendererDataset /media/ziqianbai/BACKPACK_DATA1/Replica_all/replica_v1/room_0/mesh.ply /media/ziqianbai/BACKPACK_DATA1/Replica_all/replica_v1/room_0/textures/ /media/ziqianbai/BACKPACK_DATA1/Replica_all/replica_v1/room_0/glass.sur ../glob/train/room_0_6dof.txt y /media/ziqianbai/BACKPACK_DATA1/Replica_all/replica_for_panonerf/room_0/ 1024 512 ../glob/pro2pos.txt

But the pointcloud seems imprecise, the depth is likely been truncated cause of inappropriate depth scale or depth image format:

The code I used to unproject the equirect_depth_image :

def test_spherical_depth():
    def get_unit_spherical_map():
        h = 512
        w = 1024
        Theta = np.arange(h).reshape(h, 1) * np.pi / h + np.pi / h / 2
        Theta = np.repeat(Theta, w, axis=1)
        Phi = np.arange(w).reshape(1, w) * 2 * np.pi / w + np.pi / w - np.pi
        Phi = -np.repeat(Phi, h, axis=0)

        X = np.expand_dims(np.sin(Theta) * np.sin(Phi),2)
        Y =  np.expand_dims(np.cos(Theta),2)
        Z = np.expand_dims(np.sin(Theta) * np.cos(Phi),2)
        unit_map = np.concatenate([X,Z,Y],axis=2)

        return unit_map

    depth_img_filepath = '/media/ziqianbai/BACKPACK_DATA1/Replica_all/replica_for_panonerf/room_0/room_0_0000_pos12.png'
    raw_depth_img = Image.open(depth_img_filepath)
    depth_img = ImageOps.grayscale(raw_depth_img)
    depth_img = np.asarray(depth_img)
    # depth_img = np.asarray(Image.open(depth_img_filepath))
    depth_img=np.expand_dims((depth_img*16.0),axis=2)
    pointcloud = depth_img * get_unit_spherical_map()

    o3d_pointcloud = o3d.geometry.PointCloud()
    o3d_pointcloud.points = o3d.utility.Vector3dVector(pointcloud.reshape(-1,3))
    o3d.io.write_point_cloud('/media/ziqianbai/BACKPACK_DATA1/Replica_all/replica_for_panonerf/room_0/room_0_0000_pcl_2.ply', o3d_pointcloud)

I'm sure you probably have similar problems. Could u help me figure out the reason the resulting phenomenon? Thanks in advance! @cr333 @manurare

How to have access to the .dpt depth files?

Hello,

Thanks to the very nice 360 RGBD datasets! I'm struggling with using the .dpt files...Don't know how to open it and get the depth numbers.

Could you give an example code for reading in the .dpt file?

Thank you.

Docker Environment problem

Hi, nice to see your interesting work

I have problems to run your docker environment on Ubuntu 20.04 and Windows 10

Can you help to test whether the docker environment is valid or not

Thanks a lot !

Unknown runtime specified nvidia.

Hello,

The result of your program looks amazing, I am trying to run the code myself to take a live look but when I try to run the code after the docker builds successfully it provides me this error:
docker: Error response from daemon: Unknown runtime specified nvidia.

I then modified the docker run command to use the --gpus all instead of the --runtime=nvidia command, this got me a little further however I still get a NVIDIA GeForce RTX 3070 with CUDA capability sm_86 is not compatible with the current PyTorch installation error.

Am I doing something wrong, do you have any advice?

Thank you.

How to give our own input data?

I have a 360 image and i don't have .dpt file
So how to give the input
Please guide me

Convert disparity to depth maps

Hi,
thanks for sharing the code.

Based on my understanding, the output erp_dispmap_blend of the depth_estimation() function is the disparity maps not the depth map.

If so, how can I convert disparity to depth and store them as pfm.

Thanks.

python module not installed

Hi, and thanks for this code! I have it built, and when running main.py I see this error:

2024-04-02 17:55:43,282 - utility.depthmap_align - ERROR - depthmapAlign python module do not install, please build and install it reference the readme.

Can you please point me at the instructions for this?

I have built in visual studio successfully.

EigenSolvers import error

While running python main.py --expname test_experiment --blending_method all --grid_size 8x7
I'm getting ImportError: /home/dbot/Desktop/360monodepth/360monodepth/code/cpp/lib/EigenSolvers.cpython-38-x86_64-linux-gnu.so: undefined symbol: _ZN6google10LogMessage6streamEv

Please help me with this

point_clouds

Thank you for your great work.when I use depth map and rgb to genetate point clouds,in one scene,there are many points,how I process all point clouds to genetate a Integral point clound.In the website, I see a example, x = np.array([0, 0, 0)
x_prime = r @ x + c,but the final point cloud is chaotic.

Missing directory

Hello,

When running Docker I run into this error:

[12/12] RUN cd /monodepth/python/src/test && python3 ./test_depthmapAlign_module.py --task 1:
#16 0.278 /bin/sh: 1: cd: can't cd to /monodepth/python/src/test

I'm not well versed in Docker, but seems like it's referencing a directory that's not there. What can I do?

Find parameters for disparity2depth

Hi,

I see that the function disparity2depth() has 2 parameters baseline and focal.

Do you know how to find these two parameters?

Since I use only a single 360 camera with Lidar to capture depths, there is no baseline between the two cameras.

Could you please suggest some approach to determine the baseline and focal?

How to have access to the depth map

Hello,

Thank you very much for this nice library. Could you please let me know how I can access the output depth map? When I run your code on an unseen equirectangular RGB image, I get five images as follows:
000_360monodepth_midas2_frustum.png, 000_360monodepth_midas2_mean.png, 000_360monodepth_midas2_nn.png, 000_360monodepth_midas2_poisson.png, 000_360monodepth_midas2_radial.png

However, I would like to generate the point cloud from the output of your program to see how the depth prediction works visually.

Scale consistency

Hi I m gonna make a test with your nice library, but a quick question about scale, if I would provide 2 images of similar location would I get a consistent scale in both results? Is there anyway to get into the right scale or just like midas it s an avarage scale ?

Thanks

Crash when transposing A matrix

Hi, the main.py python script crashes at this line part of compute_linear_system_matrices() as part of the poisson blending process.

    self.eigen_solver.A = self.A.transpose().dot(self.A)

Any idea how should I debug or solve it?

Rotation for each tangent image

Is there an easy way to get a rotation matrix for each tangent image? i.e. the rotation matrix between the tangent image and the panorama's coordinate system.

Using ZoeDepth produce bad blending

I have simplify and extent this repository in this fork so I can easily use it as a pipeline to extract depth from 360 photos taken with a Ricoh camera. Can't made poisson blending to work (#18) but I had success using frustum blending.
Here the results of the test image on midas2 and midas3 models


When extending the models to support ZoeDepth motivated by their spatial consistency the results seams to have blending issues.

Wonder if the approach I'm using is the correct, and if there is something else I should be doing?

ImportError for EigenSolvers

Thank you for this amazing work. I successfully installed everything with the help of Dockerfile. However, when I ran the main.py code, I got following error everytime. I tried recompiling everything but still got same error.

root@55d28786ec5a:/monodepth/code/python/src# python3 main.py 
Adding '/monodepth/code/python/src' to sys.path
2023-02-10 08:01:56,106 - utility.depthmap_align - INFO - depthmapAlign python module installed!
Traceback (most recent call last):
  File "/monodepth/code/python/src/main.py", line 17, in <module>
    from utility import blending
  File "/monodepth/code/python/src/utility/blending.py", line 6, in <module>
    from EigenSolvers import LinearSolver
ImportError: /monodepth/code/cpp/lib/EigenSolvers.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN6google10LogMessage6streamEv
root@55d28786ec5a:/monodepth/code/python/src#

Can you help me with this? Thank you.

what is the 'configuration' file, in 'import configuration as config'

hi,
when i study the 360monodepth source code, I can not find the 'configuration' file or can not install the module 'configuration' by 'pip install configuration'.
need you help .
thankyou

最终得到的 DPT深度数据，怎么样转换成一个全景图片呢？

Struggeling with docker

Hi, I really hoped I could get it to work on my own, but I've been struggling. I tried installing it with wsl2 & docker desktop, but that didn't work. I tried building it on windows and that after hours 'make' failed. I actually almost got it with wsl2 ubuntu, but there were some weird warnings when makeing the build.

Anyways, i thought I finally got it with the docker build in wsl2 ubuntu. And it seemed to have built it correctly, it didn't seem to throw any errors. But trying to run it with the included test data I am getting the error I included.

I would appreciate it if anyone knew how to fix my issue.

`(360monodepth) vanessa@Vanessa-Workstation:/mnt/d/Uni/Master/linux/docker/360monodepth$ docker run -it --runtime=nvidia -e NVIDIA_VISIBLE_DEVICES=0 360monodepth sh -c "cd /monodepth/code/python/src; python3 main.py --expname test_experiment --blending_method all --grid_size 8x7"

==========
== CUDA ==

CUDA Version 11.4.3

Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.

Adding '/monodepth/code/python/src' to sys.path
2024-04-18 15:34:29,861 - utility.depthmap_align - INFO - depthmapAlign python module installed!
----------------- Options ---------------
available_steps: [1, 2, 3, 4] [default: None]
blending_method: all [default: poisson]
coeff_fixed_face_index: -1 [default: None]
data_fns: ../../../data/erp_00_data.txt [default: None]
dataset_matterport_blur_area_height: 0 [default: None]
dataset_matterport_blurarea_shape: circle [default: None]
dataset_matterport_hexagon_mask_enable: False [default: None]
debug_enable: False [default: None]
debug_output_dir: None
dispalign_align_coeff_grid_height: 7 [default: None]
dispalign_align_coeff_grid_width: 8 [default: None]
dispalign_ceres_max_linear_solver_iterations: 10 [default: None]
dispalign_corr_thread_number: 10 [default: None]
dispalign_debug_enable: False [default: None]
dispalign_iter_num: 100 [default: None]
dispalign_method: group [default: None]
dispalign_output_dir: None
dispalign_pixelcorr_downsample_ratio: 0.001 [default: None]
dispalign_pyramid_layer_number: 1 [default: None]
dispalign_weight_project: 1.0 [default: None]
dispalign_weight_scale: 0.007 [default: None]
dispalign_weight_smooth: 40.0 [default: None]
expname: test_experiment [default: monodepth]
grid_search: False
multi_res_grid: False [default: None]
parser: ArgumentParser(prog='main.py', usage=None, description=None, formatter_class=<class 'argparse.HelpFormatter'>, conflict_handler='error', add_help=True) [default: None]
persp_monodepth: midas2
rm_debug_folder: True
sample_size: 0
subimage_available_list: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19] [default: None]
subimage_padding_size: 0.3 [default: None]
subimage_tangent_image_width: 400 [default: None]
test: False [default: None]
----------------- Options End -------------------
Set filename prefix: 0001_rgb
Set file root dir: ../../../data/erp_00/debug/
2024-04-18 15:34:29,883 - main - INFO - 1) load ERP image & project to 20 face images
2024-04-18 15:34:29,883 - utility.projection_icosahedron - DEBUG - Generate rectangle tangent image.
2024-04-18 15:34:29,883 - utility.projection_icosahedron - INFO - project ERP image 3 channels RGB map
2024-04-18 15:34:29,883 - utility.projection_icosahedron - DEBUG - generate the tangent image 0
Traceback (most recent call last):
File "main.py", line 557, in
monodepth_360(opt)
File "main.py", line 486, in monodepth_360
estimated_depthmap, times = depthmap_estimation(erp_rgb_image_data, fnc, opt, blend_it, iter)
File "main.py", line 170, in depthmap_estimation
subimage_rgb_list, _, points_gnomocoord = proj_ico.erp2ico_image(erp_rgb_image_data,
File "/monodepth/code/python/src/utility/projection_icosahedron.py", line 270, in erp2ico_image
triangle_param = get_icosahedron_parameters(triangle_index, padding_size)
File "/monodepth/code/python/src/utility/projection_icosahedron.py", line 149, in get_icosahedron_parameters
triangle_points_tangent_pading = polygon.enlarge_polygon(triangle_points_tangent, padding_size)
File "/monodepth/code/python/src/utility/polygon.py", line 88, in enlarge_polygon
v1 = np.array([old_points[j][0] - old_points[i][0], old_points[j][1] - old_points[i][1]], np.float)
File "/usr/local/lib/python3.8/dist-packages/numpy/init.py", line 305, in getattr
raise AttributeError(former_attrs[attr])
AttributeError: module 'numpy' has no attribute 'float'.
np.float was a deprecated alias for the builtin float. To avoid this error in existing code, use float by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use np.float64 here.
The aliases was originally deprecated in NumPy 1.20; for more details and guidance see the original release note at:
https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations`

Killed while running with different input

When i give different input without GT and None in second coloumn of the .txt file. It gives a killed command in terminal
Please help me with this.