francescopittaluga / invsfm

InvSFM: Revealing Scenes by Inverting Structure from Motion Reconstructions [CVPR 2019]

InvSFM: Revealing Scenes by Inverting Structure from Motion Reconstructions

Synthesizing Imagery from a SFM Point Cloud: From left to right--Top view of a SfM reconstruction of an indoor scene; 3D points projected into a viewpoint associated with a source image; the image reconstructed using our technique; and the source image.

This repository contains a reference implementation of the algorithms described in the CVPR 2019 paper Revealing Scenes by Inverting Structutre from Motion Reconstructions. This paper was selected as a Best Paper Finalist at CVPR 2019. For more details about the project, please visit the main project page.

If you use this code/model for your research, please cite the following paper:

@inproceedings{pittaluga2019revealing,
  title={Revealing scenes by inverting structure from motion reconstructions},
  author={Pittaluga, Francesco and Koppal, Sanjeev J and Bing Kang, Sing and Sinha, Sudipta N},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
  pages={145--154},
  year={2019}
}

Installation Guide

The code was tested with Tensorflow 1.10, Ubuntu 16, NVIDIA TitanX / NVIDIA 1080ti.

Step 1: Install dependencies

See requirements.txt. The training code depends only on tensorflow. The demos additionally depend on Pillow and scikit-image.

Step 2: Download the pre-trained model weights

Run $ bash download_wts.sh to programatically download and untar wts.tar.gz (1.24G). Alternatively, manually download wts.tar.gz from here and untar it in the root directory of the repo.

Step 3: Download the demo data

Run $ bash download_data.sh to programatically download and untar data.tar.gz (11G). Alternatively, manually download data.tar.gz from here and untar it in the root directory of the repo.

Step 4: Run the demos

$ python demo_5k.py 
$ python demo_colmap.py

Note: Run $ python demo_5k.py --help and $ python demo_colmap.py --help to see the various demo options available.

Step 5: Run the training scripts

$ python train_visib.py 
$ python train_coarse.py 
$ python train_refine.py 

Note: Run $ python train_*.py --help to see the various training options available.

invsfm's Issues

InvalidArgumentError (see above for traceback): segment_ids[2509] = 87841 is out of range [0, 87841)

Has anyone encountered this problem?
Python==3.6
tf==1.10

And if I want to input my own image, what should I do?
Sorry, I am very new; If anyone can help me solve this problem, I would be very grateful.

Ask for the requirements

Hi
Could you please provide the tensorflow version?

Thank you so much

Question about how to generate 2D feature map for training VisibNet

Thanks for your great work! After reading your paper, I have some questions regarding how to generate a 2D feature map for training VisibNet.
When 3D point cloud is projected into a 2D feature map, I believe some 3D points will gather together on the 2D map, i.e., many 3D points--->one 2D point, some of the 3D points are on the foreground and they can be visible, while some are on the background and they cannot be seen. Under the circumstances, how to label the 2D point with "visible" or "occluded"?

Training data generation

Hello, is it possible to release code, which capable to generate train data from original datasets? And asking in advance: how hard to adapt it to my custom dataset? Thanks

Question with datasets for training VisibNet

Thanks for your great work!
After reading your paper and codes, I am trying to train sub-networks and I get some questions regarding how to train VisibNet.
While VisibNet is being trained, I believe datasets such as NYU or MegaDepth, with being dense reconstructed in COLMAP(called VisibDense in the paper), are necessary.
Before preparing dense-datasets, I made three dense scenes with COLMAP and realised that it is an insane work to do.
Again, I coudn't stop respecting you to do such tough works.

Back to the point, I am wondering if you can share dense datasets.
If it is not possible, can you tell me how long it took to get whole post-VisibDense datasets used in training VisibNet?

Training time

Hello, first of all, your project looks great :) I have question about time training. How much time each training stage(visib, coarse, refine) takes?

Unable to use newly trained models

I am trying to train the coarsenet to see if it can learn to produce images from very sparse point clouds. Here is my series of steps:

1. I load the pretrained coarsent model ('wts/depth_sift_rgb/coarsenet.model.npz') and a suitable annotations file (containing very few examples).
2. Providing a low percentage of points to be used (between 5 and 10), I let it run for some iterations (around 10000) without changing any other default hyper-parameters.
3. To check the results, I run the demo_5k.ply, after changing the coarsenet model to one obtained from above. First, I run into the following error, which I think is because the saved weights do not have the mean and variances (for eg 'ec0_mn', 'ec0_vr') saved for each layer, as provided in the pre-trained weights. If anyone has successfully directly used modified weights, it wold be really helpful.
   
   I try to circumvent the above error by explicitly loading the corsenet model with batchnorm set to 'set' and then doing the following:

C.load(sess, cnet_input)
for k in C.bn_outs:
    C.weights[k] = C.bn_outs[k]

Running with these weights gives bogus results (uniformly grey images as shown below). I wonder how the weights have shifted so much in just a few iterations starting from pre-trained weights which give great results.

I tried with the example annotation file as well and without the sparsity constraints but it led to similar results. So basically, if I sart from pretrained weights, the model seems to just degrade very quickly even using the same training data. think I might be doing something wrong as these results are strange. Has anyone been successful in training over additional data starting from pre-trained weights ?