daeyun / scene3d

3D indoor scene surface prediction from RGB

Installation

Clone the repository with the --recurse-submodules option to download third party dependencies.

git clone --recurse-submodules [email protected]:daeyun/scene3d.git

Setting up Python virtual environment

Download the Python 3.6 version of Anaconda.

From the repository's root directory, run the following command to create a new Conda environment and install and dependencies. scene3d can be replaced with any name you want.

conda env create --file python/environment.yml --name scene3d

More info on managing Conda environments can be found here.

Build instructions for C++

cd ./cpp/third_party/repos/mve
make -j12
# after g++-7 is installed:
./cpp/third_party/build_scripts/SUNCGtoolbox.sh
./cpp/third_party/build_scripts/assimp.sh
./cpp/third_party/build_scripts/c-blosc.sh

./cpp/scripts/build_all.sh

Compiled binaries can be found in ./cpp/cmake-build-release/apps.

For example,

./cpp/cmake-build-release/apps/render --help

Development

Project structure

Python

• The project's library is in python/scene3d.
• Work-in-progress scripts are in python/scratch.
• Web UI code: python/web.

scene3d's Issues

Better visualization API

I'm using the GitHub issue tracking system as a to-do list. Sorry if you receive a lot of notifications.

Currently I don't have a way to generate and visualize a small set of or individual examples. Currently I'm running a script that generates examples 0 to N (depending on when I stop the script). So in order to make debugging and visualization easier, I need to refactor the python interface do that it can run it on arbitrary examples.

Bounding box heuristics

I'm using the GitHub issue tracking system as a to-do list. Sorry if you receive a lot of notifications.

This issue has to do with improving the bounding box heuristics for overhead viewpoint proposal.

1. The current approach fails on some images, around 1 out of 10 examples. I need to fix this before generating a full dataset.

2. Need to constrain the zoom factor. When there's one small object present in the scene (e.g. the bench example), it currently focuses on that and ignores the rest of the scene. I want to improve this by first proposing multiple bounding boxes (e.g. with and without the background) and merging them into a single bounding box.

To-do list

This is mostly to help myself keep track of things. Hopefully the notifications aren't too spammy, if you receive them.

• Find another ray-tracing library. NanoRT's doesn't have multi-hit ray traversal fully implemented. Update: Made a small modification to NanoRT to make it work; not optimal performance, but it works.
• Fix bug in finding plants and people. Model ids weren't enough for some reason.
• Make depth rendering multi-threaded. Currently depth rendering takes way too long. Done: speed-up is around 7, using 11 threads. Still takes 36 hours on a single machine.

zhe

• added human in the scene with a suitable pose, checking collision and supporting surface

calculating fov y

try to use (preset x_fov)
y_fov = arctan(tan(x_fov) * image_h/image_w). ----- (1)

but always get error and warning like:
[warning] xf: 0.5534, yf: 0.33445, width: 1920, height: 1080, xl: 1553.92, yl: 1553.94
terminate called after throwing an instance of 'std::runtime_error'
what(): Inconsistent distance to image plane.
Aborted (core dumped)

Suggested using calculating from https://github.com/daeyun/scene3d/blob/cdf912855aec230d5fc8f0eaa59f63909c3a9621/cpp/lib/multi_layer_depth_renderer.h
// Distance to the image plane according to the x fov.
double xl = 0.5 * width_ / std::tan(xf);
// Distance to the image plane according to the y fov.
double yl = 0.5 * height_ / std::tan(yf);
to directly making it consistent,

And also found the formula (1) is different with what I read from wiki
https://en.wikipedia.org/wiki/Field_of_view_in_video_games

Improving the depth rendering API, ignoring some object categories

I'm using the GitHub issue tracking system as a to-do list. Sorry if you receive a lot of notifications.

1. I need to refactor the ray tracing and depth rendering code so that it's easier to control which object categories or instances are rendered (or ignored). Currently it's hard to do things like "ignore chandeliers" without manually hardcoding the model IDs.

2. (1) would help with implementing the "first exit of the same object instance" layer. The code is too convoluted and it's slowing down development.