cyang0515 / noncuboidroom Goto Github PK
View Code? Open in Web Editor NEWLearning to Reconstruct 3D Non-Cuboid Room Layout from a Single RGB Image
License: MIT License
Learning to Reconstruct 3D Non-Cuboid Room Layout from a Single RGB Image
License: MIT License
Hi, I am trying to create a visualizer that allows me to visualize the room layout. I am using a custom image for inference.
The results look good, when i check the segmentation overlayed on my image. The main issue I have is when I try to use open 3d to plot, it looks like the depth map is curved. Do you have any explanation for this?
This is my Open3d visualization of the polygons:
The main issue is the curvature, and that is due to the depth map extracted by the network. Any idea of how to solve it?
Thanks
Hello.
I really enjoyed reading your paper and am so excited to test your code.
Here are some outputs that used your pretrained model, both Structure3D and NYU303.
I am having a hard time getting the right output as explained in the paper.
It was happening for both pretrained models.
Could you please share some more explanations on how to use the pretrained model you provided?
And it would be great if you can share some sample images you used for the inference.
Thank you.
Hello guys, I really appreciate the work and I was finally able to reconstruct and visualize my room from a 2d picture.
I was wandering if you have any directions/ideas on how we could combine different predictions from different images of the same room, to obtain a complete layout reconstruction of the full room. The idea would be to we take 3 or 4 partially overlapping pictures that cover the whole room and then having a full 3D reconstruction of the room, maybe by using the camera intrinsics and extrinsics and the by joining the extracted planes. This is just an idea but I wanted to know if you thought about it and if you think is something feasible or not. Thanks a lot.
Thanks for your good job. Is there any code about generating line segment data?
I tried to download the structured 3D dataset but it when I try to extract the zip file it says it is corrupted. I tried several times but the result is always the same. Anyone had the same problem?
I would know what are the steps to run an inference on custom data? More precisely, if we run the command specified on the readme, it will use the custom dataset, what are the things to modify on the dataset and on the cfg.yaml so we can run the model on custom data?
Hello, Thanks for the impressive work of you and your team!
The output of your network are planes, lines and plane parameters, and there is no semantic infomation included in the output , so i`m very confuse about the computing method of metric PE, expecting for your reply.
Thanks again.
I was reviewing the code, i noticed that dt_params3d_pixelwise is not used in the code. Line 326 in test.py
# post process on output feature map size, and extract planes, lines, plane params instance and plane params pixelwise dt_planes, dt_lines, dt_params3d_instance, dt_params3d_pixelwise = post_process(x, Mnms=1)
In the function ConvertLayout in reconstruction.py
there is a parameter called pixelwise which is set to None. If i try to assign dt_params3d_pixelwise
to the pixelwise parameter function of Convert Layout i get an error.
# opt results seg, depth, img, polys = ConvertLayout( inputs['img'][i], ups, downs, attribution, K=inputs['intri'][i].cpu().numpy(), pwalls=params_layout, pfloor=pfloor, pceiling=pceiling, ixy1map=inputs['ixy1map'][i].cpu().numpy(), valid=inputs['iseg'][i].cpu().numpy(), oxy1map=inputs['oxy1map'][i].cpu().numpy(), pixelwise=None )
So my questions are:
Thanks
Hi
I don't really understand the process of optimizing plane parameter with detected line. In the paper, the optimize function seems to let the line intersected by two walls to fit the detected one.
NonCuboidRoom/models/reconstruction.py
Line 309 in ba14669
NonCuboidRoom/models/reconstruction.py
Line 318 in ba14669
NonCuboidRoom/models/reconstruction.py
Line 310 in ba14669
This is great work! Very impressive.
I'm able to visualize the planes in the image by enabling this
But how do we generate the 3D model as shown in the paper?
Thanks
I am doing some inference using the Structure3D_pretrained.pt model which is downloaded from this repo, and the custom image is come from the InteriorNet dataset, which is introduced by the KuJiaLe, too.
The image size is (480, 640), and the camera intrinsic is [[600, 0, 320], [0, 600, 240], [0, 0, 1].
Part of the result output by the model seems to be reasonable, but other result is hard to accept.
So should i change the intrinsic matrix ? or how should i modify the hyperparameter setting ?
Any suggestion will be grateful.
In the following pictures, red is the GT edge and green is predicted by the model:
Thanks for your excellent work!
Many thanks for making this dataset available!
I have downloaded the Structured3D_panorama_00.zip ... Structured3D_panorama_17.zip from the Azure cloud and found that all except 14,15,16,17 have some files corrupted inside the zip archives. I have downloaded the archives on multiple university and my private internet connections and used different applications to unzip the files (unzip, 7z in Ubuntu 22.04 and with Windows 10 archive manager). In all cases, the same errors were encountered. For example, unzipping the Structured3D_panorama_13.zip results in the following errors:
unzip -q Structured3D_panorama_13.zip -d /media/jiri/Pluto/datasets/Structured3D/Structured3D_panorama/13/
error: invalid compressed data to inflate /media/jiri/Pluto/datasets/Structured3D/Structured3D_panorama/13/Structured3D/scene_02601/2D_rendering/843/panorama/simple/normal.png
/media/jiri/Pluto/datasets/Structured3D/Structured3D_panorama/13/Structured3D/scene_02608/2D_rendering/345/panorama/full/rgb_coldlight.png bad CRC 406b2c0b (should be 53007390)
file #1444: bad zipfile offset (local header sig): 536390646
/media/jiri/Pluto/datasets/Structured3D/Structured3D_panorama/13/Structured3D/scene_02609/2D_rendering/1235/panorama/simple/rgb_warmlight.png bad CRC fbc88119 (should be 35d6a845)
file #1486: bad zipfile offset (local header sig): 553273189
file #1487: bad zipfile offset (local header sig): 553279282
file #1488: bad zipfile offset (local header sig): 553279355
file #1489: bad zipfile offset (local header sig): 553279437
file #1490: bad zipfile offset (local header sig): 553279568
file #1491: bad zipfile offset (local header sig): 553279656
file #1492: bad zipfile offset (local header sig): 553519416
/media/jiri/Pluto/datasets/Structured3D/Structured3D_panorama/13/Structured3D/scene_02609/2D_rendering/4681/panorama/empty/rgb_coldlight.png bad CRC c4090c4d (should be 02b78ee6)
file #1579: bad zipfile offset (local header sig): 586400993
and with 7z
7z x -o/media/jiri/Pluto/datasets/Structured3D/Structured3D_panorama/13/ Structured3D_panorama_13.zip
7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=en_GB.UTF-8,Utf16=on,HugeFiles=on,64 bits,8 CPUs Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz (306F2),ASM,AES-NI)
Scanning the drive for archives:
1 file, 12381936537 bytes (12 GiB)
Extracting archive: Structured3D_panorama_13.zip
--
Path = Structured3D_panorama_13.zip
Type = zip
Physical Size = 12381936537
64-bit = +
ERROR: Data Error : Structured3D/scene_02601/2D_rendering/843/panorama/simple/normal.png
ERROR: CRC Failed : Structured3D/scene_02608/2D_rendering/345/panorama/full/rgb_coldlight.png
ERROR: Headers Error : Structured3D/scene_02608/2D_rendering/345/panorama/full/rgb_rawlight.png
ERROR: CRC Failed : Structured3D/scene_02609/2D_rendering/1235/panorama/simple/rgb_warmlight.png
ERROR: Headers Error : Structured3D/scene_02609/2D_rendering/1235/panorama/simple/semantic.png
ERROR: Headers Error : Structured3D/scene_02609/2D_rendering/1419
ERROR: Headers Error : Structured3D/scene_02609/2D_rendering/1419/panorama
ERROR: Headers Error : Structured3D/scene_02609/2D_rendering/1419/panorama/camera_xyz.txt
ERROR: Headers Error : Structured3D/scene_02609/2D_rendering/1419/panorama/empty
ERROR: Headers Error : Structured3D/scene_02609/2D_rendering/1419/panorama/empty/albedo.png
ERROR: Headers Error : Structured3D/scene_02609/2D_rendering/1419/panorama/empty/depth.png
ERROR: CRC Failed : Structured3D/scene_02609/2D_rendering/4681/panorama/empty/rgb_coldlight.png
ERROR: Headers Error : Structured3D/scene_02609/2D_rendering/4681/panorama/empty/rgb_rawlight.png
Sub items Errors: 13
Archives with Errors: 1
Sub items Errors: 13
Could you please verify that these files are fine in the Azure cloud? If yes, then perhaps the files are corrupted only on the Azure mirror in the UK, which would be a very serious fault in Azure data management.
Kind Regards
Jiri
may I ask about the training time and the gpu device you used while training?
When I run this command python test.py --pretrained ./pretrained/NYU303_pretrained.pt --data NYU303
I get this error output.log
I tried to use the Structured 3D pretrained model on some custom images taken from my phone and online.
I noticed that the prediction are not very correct and I wanted to know if there is any preprocessing step that needs to be done on the images. Here are some results I obtained:
Any help would be appreciated
Hi author,
I have a question regarding the network design.
From the planar detection (Section 3.1), you indicated that "Each channel of the center likelihood map C represents different categories"., and looks like that you attempt to solve planar detection + classification of wall/ceiling/floor together via center likelihood map.
This make me confused. As the channel for offset is still HW2 instead of HW6, where it looks like that wall/ceiling/floor share the same offset. I am not quite understand the design here as normally, the offset for wall/ceiling/floor may not be the same. Could you make some comments for my question? and will it make more sense to decouple them to keypoint detection + classification instead?
Thank you if you can consider answer my question.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.