Geometry-Free View Synthesis: Transformers and no 3D Priors
Geometry-Free View Synthesis: Transformers and no 3D Priors
Robin Rombach*,
Patrick Esser*,
Björn Ommer
* equal contribution
Interactive Scene Exploration Results
RealEstate10K:
Videos: short (2min) / long (12min)
ACID:
Videos: short (2min) / long (9min)
Demo
Installation
The demo requires building a PyTorch extension. If you have a sane development environment with PyTorch, g++ and nvcc, you can simply
pip install git+https://github.com/CompVis/geometry-free-view-synthesis#egg=geometry-free-view-synthesis
If you run into problems and have a GPU with compute capability below 8, you can also use the provided conda environment:
git clone https://github.com/CompVis/geometry-free-view-synthesis
conda env create -f geometry-free-view-synthesis/environment.yaml
conda activate geofree
pip install geometry-free-view-synthesis/
Running
After installation, running
braindance.py
will start the demo on a sample scene.
Explore the scene interactively using the WASD
keys to move and arrow keys
to
look around. Once positioned, hit the space bar
to render the novel view with
GeoGPT.
You can move again with WASD keys. Mouse control can be activated with the m
key. Run braindance.py <folder to select image from/path to image>
to run the
demo on your own images. By default, it uses the re-impl-nodepth
(trained on
RealEstate without explicit transformation and no depth input) which can be
changed with the --model
flag. The corresponding checkpoints will be
downloaded the first time they are required. Specify an output path using
--video path/to/vid.mp4
to record a video.
> braindance.py -h
usage: braindance.py [-h] [--model {re_impl_nodepth,re_impl_depth}] [--video [VIDEO]] [path]
What's up, BD-maniacs?
key(s) action
=====================================
wasd move around
arrows look around
m enable looking with mouse
space render with transformer
q quit
positional arguments:
path path to image or directory from which to select image. Default example is used if not specified.
optional arguments:
-h, --help show this help message and exit
--model {re_impl_nodepth,re_impl_depth}
pretrained model to use.
--video [VIDEO] path to write video recording to. (no recording if unspecified).
BibTeX
@misc{rombach2021geometryfree,
title={Geometry-Free View Synthesis: Transformers and no 3D Priors},
author={Robin Rombach and Patrick Esser and Björn Ommer},
year={2021},
eprint={2104.07652},
archivePrefix={arXiv},
primaryClass={cs.CV}
}