s0rrymaker77777 / total-recon

Total-Recon: Deformable Scene Reconstruction for Embodied View Synthesis

Total-Recon: Deformable Scene Reconstruction for Embodied View Synthesis

Project | Paper

This is the official PyTorch implementation of "Total-Recon: Deformable Scene Reconstruction for Embodied View Synthesis".

Chonghyuk Song, Gengshan Yang, Kangle Deng, Jun-Yan Zhu, Deva Ramanan
Carnegie Mellon University
ICCV 2023

cover_figure.mp4

Given a long video of deformable objects captured by a handheld RGBD sensor, Total-Recon renders the scene from novel camera trajectories derived from in-scene motion of actors: (1) egocentric cameras that simulate the point-of-view of a target actor (such as the pet) and (2) 3rd-person (or pet) cameras that follow the actor from behind. Our method also enables (3) 3D video filters that attach virtual 3D assets to the actor. Total-Recon achieves this by reconstructing the geometry, appearance, and root-body and articulated motion of each deformable object in the scene as well as the background.

Timeline

We plan to release our code in the following 4 stages:

• Inference and Evaluation code for 4 select sequences of our stereo RGBD dataset
• Raw data for all sequences of our dataset
• Training code for all sequences of our dataset
• Data preprocessing code for user-provided RGBD videos

Getting Started

Dependencies

(1) Clone repo (including submodules):

git clone https://github.com/andrewsonga/Total-Recon.git --recursive

# This step is REQUIRED for all subsequent steps!
cd Total-Recon

(2) Install conda env:

conda env create -f misc/totalrecon-cu113.yml
conda activate totalrecon-cu113

(3) Install submodules:

pip install -e third_party/pytorch3d
pip install -e third_party/kmeans_pytorch
python -m pip install detectron2 -f \
  https://dl.fbaipublicfiles.com/detectron2/wheels/cu113/torch1.10/index.html

(4) Install ffmpeg:

apt-get install ffmpeg

Data

We provide raw and preprocessed data for the "human-dog", "human-cat", "human2" and "dog1 (v1)" sequences for now, but we will release the raw data for all 11 sequences of our stereo RGBD dataset very soon.

(1) Download raw data and preprocessed data, and untar them.

bash download_rawdata.sh
# the zipped preprocessed data will take up around a total of 90Gb
bash download_preprocessed.sh

# untar raw data
tar -xzvf rawdata_forrelease.tar.gz

# untar preprocess data (approrpriately rename `filename`)
# the unzipped preprocessed data will take up around a total of 100Gb
filename=database_humandog.tar.gz
tar -xzvf $filename

(2) Appropriately place the downloaded data with the following scripts:

# place raw data under raw/
# argv[1]: The directory inside Total-Recon where the downloaded raw data is stored

src_dir=rawdata_forrelease
bash place_rawdata.sh $src_dir

# place preprocessed data under database/ (approrpriately rename `src_dir`)
# argv[1]: The directory inside Total-Recon where the downloaded preprocessed data is stored

src_dir=database_humandog   
tgt_dir=database            
bash place_preprocessed.sh $src_dir $tgt_dir

(3) Download the pre-trained VCN optical flow model for data preprocessing (instructions are taken from BANMo):

mkdir lasr_vcn
wget https://www.dropbox.com/s/bgsodsnnbxdoza3/vcn_rob.pth -O ./lasr_vcn/vcn_rob.pth

(4) Preprocess raw data (takes around a few hours; don't run if you have already downloaded preprocessed data)

Multi-foreground-object sequences (e.g. humandog):

# argv[1]: Sequence name that points to folders under `raw/` (minus the suffix -leftcam or -rightcam).

bash preprocess_rawdata_multiobj.sh humandog-stereo

Single-foreground-object sequences (e.g. cat2):

# argv[1]: Sequence name that points to folders under `raw/` (minus the suffix -leftcam or -rightcam).
# argv[2]: human or not, where `y` denotes human and  `n` denotes quadreped.

bash preprocess_rawdata_singleobj.sh cat2-stereo n

Pre-trained Models

(1) Download the pre-trained models, and untar them.

bash download_models.sh

tar -xzvf pretrained_models_forrelease.tar.gz

(2) Appropriately place the downloaded pretrained models with the following script:

# Place the pre-trained models under logdir/
# argv[1]: The directory inside Total-Recon where the downloaded preprocessed data is stored

src_dir=pretrained_models_forrelease
bash place_models.sh $src_dir

3D Assets

To run the 3D video filter and to be able to visualize flying embodied-view cameras, purchase and download 3D models in .obj format for 1) the unicorn horn, and 2) the Canon camera.

Rename the .obj file for the camera mesh to camera.obj, then place the file camera.obj and unzipped folder UnicornHorn_OBJ inside mesh_material.

Inference

Mesh and Root-body Pose Extraction

Before inference or evaluation can be done, please extract the object-level meshes and root-body poses from the trained model:

# argv[1]: gpu number (0, 1, 2, ...)
# argv[2]: folder name of the trained model inside logdir/

seqname=humandog-stereo000-leftcam-jointft    # (appropriately rename `seqname`)
bash extract_fgbg.sh $gpu $seqname

Egocentric View Synthesis

nvs-fpsview-winput-trimmed.mp4

(takes around a few hours) The rendered videos will be saved as nvs-fpsview-*.mp4 inside logdir/$seqname/

bash scripts/render_nvs_fgbg_fps.sh $gpu $seqname $add_args

per-sequence arguments (add_args)

1. Human-dog

seqname=humandog-stereo000-leftcam-jointft

add_args='--fg_obj_index 1 --asset_obj_index 1 --fg_normalbase_vertex_index 96800 --fg_downdir_vertex_index 1874 --asset_scale 0.003 --firstpersoncam_offset_z 0.05 --firstpersoncam_offsetabt_xaxis 15 --firstpersoncam_offsetabt_zaxis 0 --asset_offset_z -0.05 --scale_fps 0.50'

2. Human-cat

seqname=humancat-stereo000-leftcam-jointft

add_args='--fg_obj_index 0 --asset_obj_index 0 --fg_normalbase_vertex_index 170450 --fg_downdir_vertex_index 51716 --asset_scale 0.003 --firstpersoncam_offset_z 0 --firstpersoncam_offsetabt_xaxis 0 --firstpersoncam_offsetabt_zaxis 0 --fix_frame 50 --scale_fps 0.75'

3. Dog1 (v1)

seqname=dog1-stereo000-leftcam-jointft

add_args='--fg_obj_index 0 --asset_obj_index 0 --fg_normalbase_vertex_index 159244 --fg_downdir_vertex_index 93456 --asset_scale 0.003 --firstpersoncam_offset_z 0.05 --firstpersoncam_offsetabt_xaxis 35 --firstpersoncam_offsetabt_yaxis 30 --firstpersoncam_offsetabt_zaxis 20 --asset_offset_z -0.05 --scale_fps 0.75'

4. Human 2

seqname=human2-stereo000-leftcam-jointft

add_args='--fg_obj_index 0 --asset_obj_index 0 --fg_normalbase_vertex_index 114756 --fg_downdir_vertex_index 114499 --asset_scale 0.003 --firstpersoncam_offset_z 0.05 --firstpersoncam_offsetabt_xaxis 15 --firstpersoncam_offsetabt_yaxis 20 --firstpersoncam_offsetabt_zaxis 0 --asset_offset_z -0.05 --scale_fps 0.75'

3rd-Person-Follow (3rd-Pet-Follow) View Synthesis

nvs-tpsview-winput-trimmed.mp4

(takes around a few hours) The rendered videos will be saved as nvs-tpsview-*.mp4 inside logdir/$seqname/

bash scripts/render_nvs_fgbg_tps.sh $gpu $seqname $add_args

per-sequence arguments (add_args)

1. Human-dog

seqname=humandog-stereo000-leftcam-jointft

add_args='--fg_obj_index 1 --asset_obj_index 1 --thirdpersoncam_fgmeshcenter_elevate_y 0 --thirdpersoncam_offset_x 0 --thirdpersoncam_offset_y 0.25 --thirdpersoncam_offset_z -0.80 --thirdpersoncam_offsetabt_zaxis 0 --asset_scale 0.003 --scale_tps 0.70'

2. Human-cat

seqname=humancat-stereo000-leftcam-jointft

add_args='--fg_obj_index 0 --asset_obj_index 0 --thirdpersoncam_fgmeshcenter_elevate_y 0.70 --thirdpersoncam_offset_y 0.16 --thirdpersoncam_offset_z -0.40 --asset_scale 0.003 --thirdpersoncam_offsetabt_zaxis 0 --thirdpersoncam_offsetabt_yaxis -10 --fix_frame 50 --scale_tps 0.70'

3. Dog1 (v1)

seqname=dog1-stereo000-leftcam-jointft

add_args='--thirdpersoncam_offset_x 0.05 --thirdpersoncam_fgmeshcenter_elevate_y 0.30 --thirdpersoncam_offset_y 0.50 --thirdpersoncam_offset_z -0.75 --thirdpersoncam_offsetabt_zaxis 20 --thirdpersoncam_offsetabt_xaxis 0 --asset_scale 0.003 --scale_tps 0.70'

4. Human 2

seqname=human2-stereo000-leftcam-jointft

add_args='--thirdpersoncam_offset_x -0.05 --thirdpersoncam_fgmeshcenter_elevate_y 0.80 --thirdpersoncam_offset_y 0.05 --thirdpersoncam_offset_z -0.40 --thirdpersoncam_offsetabt_zaxis 0 --thirdpersoncam_offsetabt_xaxis 0 --asset_scale 0.003 --scale_tps 0.70'

Bird's-Eye View Synthesis

nvs-bev-winput-trimmed.mp4

(takes around a few hours) The rendered videos will be saved as nvs-bev-*.mp4 inside logdir/$seqname/

bash scripts/render_nvs_fgbg_bev.sh $gpu $seqname $add_args

per-sequence arguments (add_args)

1. Human-dog

seqname=humandog-stereo000-leftcam-jointft

add_args='--fg_obj_index 0 --fix_frame 65 --topdowncam_offset_x 0.10 --topdowncam_offset_y 0.60 --topdowncam_offset_z -0.05 --topdowncam_offsetabt_zaxis -15'

2. Human-cat

seqname=humancat-stereo000-leftcam-jointft

add_args='--fg_obj_index 0 --fix_frame 157 --topdowncam_offset_x 0.01 --topdowncam_offset_y 0.60 --topdowncam_offset_z 0 --topdowncam_offsetabt_zaxis 157'

3. Dog1 (v1)

seqname=dog1-stereo000-leftcam-jointft

add_args='--fg_obj_index 0 --fix_frame 18 --topdowncam_offset_x -0.0 --topdowncam_offset_y 0.90 --topdowncam_offset_z 0.30 --topdowncam_offsetabt_zaxis 10 --topdowncam_offsetabt_yaxis 0'

4. Human 2

seqname=human2-stereo000-leftcam-jointft

add_args='--fg_obj_index 0 --fix_frame 40 --topdowncam_offset_x -0.070 --topdowncam_offset_y 0.28 --topdowncam_offset_z -0.03 --topdowncam_offsetabt_zaxis 30 --topdowncam_offsetabt_yaxis 0'

Render 6-DoF Root-body Trajectory (Viewed from Bird's Eye View)

nvs-rootbodytraj-winput-trimmed.mp4

(takes around an hour) The rendered video will be saved as nvs-bev-traj-rootbody-*.mp4 inside logdir/$seqname/

bash scripts/render_traj.sh $gpu $seqname --render_rootbody --render_traj_bev $add_args

per-sequence arguments (add_args)

1. Human-dog

seqname=humandog-stereo000-leftcam-jointft

add_args='--fg_obj_index 0 --rootbody_obj_index 1 --fix_frame 65 --topdowncam_offset_x 0.10 --topdowncam_offset_y 0.60 --topdowncam_offset_z -0.05 --topdowncam_offsetabt_zaxis -15'

2. Human-cat

seqname=humancat-stereo000-leftcam-jointft

add_args='--fg_obj_index 0 --rootbody_obj_index 0 --fix_frame 157 --topdowncam_offset_x 0.01 --topdowncam_offset_y 0.60 --topdowncam_offset_z 0 --topdowncam_offsetabt_zaxis 157'

3. Dog1 (v1)

seqname=dog1-stereo000-leftcam-jointft

add_args='--fg_obj_index 0 --rootbody_obj_index 0 --fix_frame 18 --topdowncam_offset_x -0.0 --topdowncam_offset_y 0.90 --topdowncam_offset_z 0.30 --topdowncam_offsetabt_zaxis 10 --topdowncam_offsetabt_yaxis 0'

4. Human 2

seqname=human2-stereo000-leftcam-jointft

add_args='--fg_obj_index 0 --rootbody_obj_index 0 --fix_frame 40 --topdowncam_offset_x -0.070 --topdowncam_offset_y 0.28 --topdowncam_offset_z -0.03 --topdowncam_offsetabt_zaxis 30 --topdowncam_offsetabt_yaxis 0'

Render 6-DoF Egocentric Camera Trajectory (Viewed from Stereo View)

nvs-fpscamtraj-winput-trimmed.mp4

(takes around an hour) The rendered video will be saved as nvs-stereoview-traj-fpscam-*.mp4 inside logdir/$seqname/

bash scripts/render_traj.sh $gpu $seqname --render_fpscam --render_traj_stereoview $add_args

per-sequence arguments (add_args)

1. Human-dog

seqname=humandog-stereo000-leftcam-jointft

add_args='--rootbody_obj_index 1 --fg_obj_index 1 --asset_obj_index 1 --fg_normalbase_vertex_index 96800  --fg_downdir_vertex_index 1874 --firstpersoncam_offset_z 0 --firstpersoncam_offsetabt_xaxis 15 --firstpersoncam_offsetabt_zaxis 0 --asset_offset_z 0'

2. Human-cat

seqname=humancat-stereo000-leftcam-jointft

add_args='--fg_obj_index 0 --asset_obj_index 0 --fg_normalbase_vertex_index 170450 --fg_downdir_vertex_index 51716 --firstpersoncam_offset_z 0 --firstpersoncam_offsetabt_xaxis 0 --firstpersoncam_offsetabt_zaxis 0 --fix_frame 50'

3. Dog1 (v1)

seqname=dog1-stereo000-leftcam-jointft

add_args='--fg_obj_index 0 --asset_obj_index 0 --fg_normalbase_vertex_index 159244 --fg_downdir_vertex_index 93456 --firstpersoncam_offset_z 0 --firstpersoncam_offsetabt_xaxis 35 --firstpersoncam_offsetabt_yaxis 30 --firstpersoncam_offsetabt_zaxis 20 --fix_frame 18 --topdowncam_offset_y 0.0 --topdowncam_offset_z 0 --topdowncam_offset_x 0 --topdowncam_offsetabt_yaxis 0 --topdowncam_offsetabt_xaxis 0'

4. Human 2

seqname=human2-stereo000-leftcam-jointft

add_args='--fg_obj_index 0 --asset_obj_index 0 --fg_normalbase_vertex_index 114756 --fg_downdir_vertex_index 114499 --firstpersoncam_offset_z 0.05 --firstpersoncam_offsetabt_xaxis 15 --firstpersoncam_offsetabt_yaxis 0 --firstpersoncam_offsetabt_zaxis 10 --asset_offset_z -0.05 --fix_frame 0 --topdowncam_offset_y 0.1 --topdowncam_offset_z 0.2 --topdowncam_offset_x 0 --topdowncam_offsetabt_yaxis -10 --topdowncam_offsetabt_xaxis -80'

Render 6-DoF 3rd-Person-Follow Camera Trajectory (Viewed from Stereo View)

nvs-tpscamtraj-winput-trimmed.mp4

(takes around an hour) The rendered video will be saved as nvs-stereoview-traj-tpscam-*.mp4 inside logdir/$seqname/

bash scripts/render_traj.sh $gpu $seqname --render_tpscam --render_traj_stereoview $add_args

per-sequence arguments (add_args)

1. Human-dog

seqname=humandog-stereo000-leftcam-jointft

add_args='--rootbody_obj_index 1 --fg_obj_index 1 --asset_obj_index 1 --thirdpersoncam_fgmeshcenter_elevate_y 0 --thirdpersoncam_offset_x 0 --thirdpersoncam_offset_y 0.25 --thirdpersoncam_offset_z -0.80 --thirdpersoncam_offsetabt_zaxis 0'

2. Human-cat

seqname=humancat-stereo000-leftcam-jointft

add_args='--fg_obj_index 0 --asset_obj_index 0 --thirdpersoncam_fgmeshcenter_elevate_y 0.70 --thirdpersoncam_offset_y 0.16 --thirdpersoncam_offset_z -0.40 --thirdpersoncam_offsetabt_zaxis 0 --thirdpersoncam_offsetabt_yaxis -10 --fix_frame 50'

3. Dog1 (v1)

seqname=dog1-stereo000-leftcam-jointft

add_args='--fg_obj_index 0 --asset_obj_index 0 --thirdpersoncam_offset_x 0.05 --thirdpersoncam_fgmeshcenter_elevate_y 0.30 --thirdpersoncam_offset_y 0.50 --thirdpersoncam_offset_z -0.75 --thirdpersoncam_offsetabt_zaxis 20 --thirdpersoncam_offsetabt_xaxis 0 --fix_frame 18 --topdowncam_offset_y 0.0 --topdowncam_offset_z 0 --topdowncam_offset_x 0 --topdowncam_offsetabt_yaxis 0 --topdowncam_offsetabt_xaxis 0'

4. Human 2

seqname=human2-stereo000-leftcam-jointft

add_args='--fg_obj_index 0 --asset_obj_index 0 --thirdpersoncam_offset_x -0.05 --thirdpersoncam_fgmeshcenter_elevate_y 0.80 --thirdpersoncam_offset_y 0.05 --thirdpersoncam_offset_z -0.40 --thirdpersoncam_offsetabt_zaxis 0 --thirdpersoncam_offsetabt_xaxis 0 --fix_frame 0 --topdowncam_offset_y 0.1 --topdowncam_offset_z 0.2 --topdowncam_offset_x 0 --topdowncam_offsetabt_yaxis -10 --topdowncam_offsetabt_xaxis -80'

Render Meshes for Reconstructed Objects, Egocentric Camera (Blue), and 3rd-Person-Follow Camera (Yellow)

nvs-povcams-mesh-winput-trimmed.mp4

(takes around an hour) The rendered video will be saved as nvs-embodied-cams-mesh.mp4 inside logdir/$seqname/

bash scripts/render_embodied_cams.sh $gpu $seqname $add_args

per-sequence arguments (add_args)

1. Human-dog

seqname=humandog-stereo000-leftcam-jointft

add_args='--fg_obj_index 1 --asset_obj_index 1 --fg_normalbase_vertex_index 96800  --fg_downdir_vertex_index 1874 --asset_scale 0.003  --render_cam_stereoview --firstpersoncam_offset_z 0.05 --firstpersoncam_offsetabt_xaxis 15 --asset_offset_z -0.05 --thirdpersoncam_offset_y 0.25 --thirdpersoncam_offset_z -0.80 --scale_fps 1.0 --scale_tps 1.0'

2. Human-cat

seqname=humancat-stereo000-leftcam-jointft

add_args='--fg_obj_index 0 --asset_obj_index 0 --fg_normalbase_vertex_index 170450 --fg_downdir_vertex_index 51716 --asset_scale 0.003 --render_cam_stereoview --firstpersoncam_offset_z 0 --firstpersoncam_offsetabt_xaxis 0 --firstpersoncam_offsetabt_zaxis 0 --thirdpersoncam_fgmeshcenter_elevate_y 0.70 --thirdpersoncam_offset_y 0.16 --thirdpersoncam_offset_z -0.40 --thirdpersoncam_offsetabt_zaxis 0 --scale_fps 1.0 --scale_tps 1.0'

3. Dog1 (v1)

seqname=dog1-stereo000-leftcam-jointft

add_args='--fg_obj_index 0 --asset_obj_index 0 --fg_normalbase_vertex_index 159244 --fg_downdir_vertex_index 93456 --asset_scale 0.004 --render_cam_stereoview --firstpersoncam_offset_z 0 --firstpersoncam_offsetabt_xaxis 35 --firstpersoncam_offsetabt_yaxis 30 --firstpersoncam_offsetabt_zaxis 20 --scale_fps 1.0 --thirdpersoncam_offset_x 0.05 --thirdpersoncam_fgmeshcenter_elevate_y 0.30 --thirdpersoncam_offset_y 0.50 --thirdpersoncam_offset_z -0.75 --thirdpersoncam_offsetabt_zaxis 20 --thirdpersoncam_offsetabt_xaxis 0 --scale_tps 1.0 --fix_frame 18 --topdowncam_offset_y 0.0 --topdowncam_offset_z 0 --topdowncam_offset_x 0 --topdowncam_offsetabt_yaxis 0 --topdowncam_offsetabt_xaxis 0'

4. Human 2

seqname=human2-stereo000-leftcam-jointft

add_args='--fg_obj_index 0 --asset_obj_index 0 --fg_normalbase_vertex_index 114756 --fg_downdir_vertex_index 114499 --asset_scale 0.004 --render_cam_stereoview --firstpersoncam_offset_z 0.05 --firstpersoncam_offsetabt_xaxis 15 --firstpersoncam_offsetabt_yaxis 0 --firstpersoncam_offsetabt_zaxis 10 --asset_offset_z -0.05 --scale_fps 1.0 --thirdpersoncam_offset_x -0.05 --thirdpersoncam_fgmeshcenter_elevate_y 0.80 --thirdpersoncam_offset_y 0.05 --thirdpersoncam_offset_z -0.40 --thirdpersoncam_offsetabt_zaxis 0 --thirdpersoncam_offsetabt_xaxis 0 --scale_tps 1.0 --fix_frame 0 --topdowncam_offset_y 0.1 --topdowncam_offset_z 0.2 --topdowncam_offset_x 0 --topdowncam_offsetabt_yaxis -10 --topdowncam_offsetabt_xaxis -80'

Render 3D Video Filters

nvs-3dfilter-winput-trimmed.mp4

(takes around a few hours) The rendered video will be saved as nvs-inputview-rgb_with_asset.mp4 inside logdir/$seqname/

bash scripts/render_nvs_fgbg_3dfilter.sh $gpu $seqname $add_args

per-sequence arguments (add_args)

1. Human-dog

seqname=humandog-stereo000-leftcam-jointft

add_args='--fg_obj_index 1 --asset_obj_index 1 --fg_normalbase_vertex_index 96800 --fg_downdir_vertex_index 1874 --asset_scale 0.0006 --input_view --noevaluate'

2. Human-cat

seqname=humancat-stereo000-leftcam-jointft

add_args='--fg_obj_index 0 --input_view --asset_obj_index 0 --fg_normalbase_vertex_index 170450 --fg_downdir_vertex_index 51716 --asset_scale 0.0006 --input_view --noevaluate'

3. Dog1 (v1)

seqname=dog1-stereo000-leftcam-jointft

add_args='--fg_obj_index 0 --input_view --asset_obj_index 0 --fg_normalbase_vertex_index 159244 --fg_downdir_vertex_index 93456 --asset_scale 0.0006 --input_view --asset_offsetabt_xaxis -25 --asset_offsetabt_yaxis 35 --noevaluate'

4. Human 2

seqname=human2-stereo000-leftcam-jointft

add_args='--fg_obj_index 0 --input_view --asset_obj_index 0 --fg_normalbase_vertex_index 114756 --fg_downdir_vertex_index 114499 --asset_scale 0.0005 --input_view --asset_offsetabt_yaxis 10 --noevaluate'

Evaluation

Stereo View Synthesis (train on left camera, evaluate on right camera)

nvs-stereoview-all-trimmed.mp4

(takes around a few hours) The rendered videos will be saved as nvs-stereoview-*.mp4 inside logdir/$seqname/

bash scripts/render_nvs_fgbg_stereoview.sh $gpu $seqname
python print_metrics.py --seqname $seqname --view stereoview

Train View Synthesis (train and evaluate on left camera)

nvs-inputview-all-trimmed.mp4

(takes around a few hours) The rendered videos will be saved as nvs-inputview-*.mp4 inside logdir/$seqname/

bash scripts/render_nvs_fgbg_inputview.sh $gpu $seqname
python print_metrics.py --seqname $seqname --view inputview

---

Citation

If you find this repository useful for your research, please cite the following work.

@article{song2023totalrecon,
  title={Total-Recon: Deformable Scene Reconstruction for Embodied View Synthesis},
  author={Song, Chonghyuk and Yang, Gengshan and 
          Deng, Kangle and Zhu, Jun-Yan and Ramanan, Deva},
  journal={arXiv},
  year={2023}
}

---

Acknowledgements

We thank Nathaniel Chodosh, Jeff Tan, George Cazenavette, and Jason Zhang for proofreading our paper and Songwei Ge for reviewing our code. We also thank Sheng-Yu Wang, Daohan (Fred) Lu, Tamaki Kojima, Krishna Wadhwani, Takuya Narihira, and Tatsuo Fujiwara as well for providing valuable feedback. This work is supported in part by the Sony Corporation, Cisco Systems, Inc., and the CMU Argo AI Center for Autonomous Vehicle Research. This codebase is heavily based on BANMo and also uses evaluation code from NSFF.