zju3dv / instant-nvr

[CVPR 2023] Code for "Learning Neural Volumetric Representations of Dynamic Humans in Minutes"

Learning Neural Volumetric Representations of Dynamic Humans in Minutes

Project Page | Video | Paper | Data

Learning Neural Volumetric Representations of Dynamic Humans in Minutes

Chen Geng*, Sida Peng*, Zhen Xu*, Hujun Bao, Xiaowei Zhou (* denotes equal contribution)

CVPR 2023

Installation

See here.

Reproducing results in the paper

We provide two scripts to help reproduce the results shown in the paper.

After installing the environment and the dataset, for evaluation on the ZJU-MoCap dataset, run:

sh scripts/eval_zjumocap.sh

For evaluation on the MonoCap dataset, run:

sh scripts/eval_monocap.sh

Evaluation on ZJU-MoCap

Let's take "377" as an example.

Training on ZJU-MoCap can be done by running.

export name=377
python train_net.py --cfg_file configs/inb/inb_${name}.yaml exp_name inb_${name} gpus ${GPUS} 

Evaluation can be done by running:

export name=377
python run.py --type evaluate --cfg_file configs/inb/inb_${name}.yaml exp_name inb_${name} gpus ${GPUS}

Evaluation on MonoCap

Let's take "lan" as an example.

Training on Monocap can be done by running:

export name=lan
python train_net.py --cfg_file configs/inb/inb_${name}.yaml exp_name inb_${name} gpus ${GPUS} 

Evaluation can be done by running:

export name=lan
python run.py --type evaluate --cfg_file configs/inb/inb_${name}.yaml exp_name inb_${name} gpus ${GPUS}

TODO List

This repository currently serves as the release of the technical paper's implementation and will undergo future updates (planned below) to enhance user-friendliness. We warmly welcome and appreciate any contributions.

• Instruction on running on custom datasets (Kudos to @tian42chen!!)
• Add support for further acceleration using CUDA
• Add a Google Colab notebook demo

Bibtex

If you find the repo useful for your research, please consider citing our paper:

@inproceedings{instant_nvr,
    title={Learning Neural Volumetric Representations of Dynamic Humans in Minutes},
    author={Chen Geng and Sida Peng and Zhen Xu and Hujun Bao and Xiaowei Zhou},
    booktitle={CVPR},
    year={2023}
}

instant-nvr's Issues

Camera coordinate

Hi,I'd like to know the which coordinate is camera extrinsics defining in,SMPL or world?

Missing files in zju-mocap dataset

Thanks for your open source! When I run on 377 sample, I got this problem.

I did not find the bigpose_uv.npy file in my zju-mocap dataset. In addition, I also noticed that the dierctory of the dataset in this code is also different from the previous projects using zjumocap. (For example, the previous subject folder was named "Coreview_377" instead of "my_377", "joints.npy"'s path in this code is "smpl_lbs/" rather than "lbs/" in my zjumocap dataset. ) The following is part of my zjumocap directory.

I downloaded the zju mocap dataset in October 2022. Does this mean that the dataset has been updated to a new version during this period? Should I re-acquire zjumocap to solve this problem?

Difference between bigpose_vertices and tvertices

Hi, i notice that when load for the v-shaped, the config states that we should load the bigpose_vertices instead of tvertices.
I am curious about the difference between this two file.
Can you share about the insight of this setting?

Thanks!

Question about smpl-meta

Thank you for your outstanding work!
I encountered the following error while running the script:
File "/liuxianguo1/LUOZD/instant-nvr-master/instant-nvr-master/lib/datasets/h36m/tpose_dataset.py", line 98, in load_smpl
faces = np.load(os.path.join(smpl_meta_root, 'faces.npy')).astype(np.int64)
File "/opt/conda/envs/instant-nvr/lib/python3.9/site-packages/numpy/lib/npyio.py", line 427, in load
fid = stack.enter_context(open(os_fspath(file), "rb"))
FileNotFoundError: [Errno 2] No such file or directory: 'data/smpl-meta/faces.npy'
Could you please advise on how to obtain this file faces.npy'? Thanks again!

questions about zju-mocap

Thank you for your wonderful open source work!
I would like to ask some questions about zju-mocap. Why were the original 313,315 and 390 subjects dropped? 390 subject wears loose clothes with a lot of folds, which I think is of significance for evaluation. I would like to ask why 313,315,390 subjects were not tested in instant-nvr paper. Will these subjects be used in zju-mocap later?

where is bigpose_uv.npy

Hi, I cannot find the bigpose_uv.npy in the zju_mocap dataset.

missing bigpose_uv.npy in ZJU-Mocap dataset

instant-nvr/lib/datasets/h36m/tpose_dataset.py

Line 374 in be56450

tuv = np.load(os.path.join(self.data_root, 'bigpose_uv.npy'))

About codes

Hi，
When reviewing the code, I found a portion that I don't quite understand. In "inb_part_network_multiassign.py" line 113"tpose = init_bigpose + resd".Isn't 'init_bigpose' here the pose under the initial T-pose, I'm a bit confused.

Misalignment Between the Dataset and the Code

Hi, Thank you for your great work! Some questions confuse me.
There's a lot of misalignment between the dataset and the code (tpose_dataset.py), like there is no bigpose_uv.npy in the CoreView_377 dataset, but in your code：
tuv = np.load(os.path.join(self.data_root, 'bigpose_uv.npy'))

About point sampling by bigpose_uv.npy

Hi, thanks for the sharing code and dataset.

I am curious how uv map coordinate is set, as far as i know, uv coordinate is pre-defined mapping relationship between vertex and 2D plane coordinate.

In your code, i think the uv map file is shape of 3D volume, you sample 2D coordinate with 3D trilinear interpolation.

(1) I want to know how you created file from SMPL vertex.

(2) From below code,

def pts_sample_uv(pts, uv, bounds, mode='bilinear'):
    """sample uv coords for points
    pts: n_batch, n_points, 3
    bw: n_batch, d, h, w, 2
    bounds: n_batch, 2, 3
    """

    pts = pts.clone()
    # interpolate blend weights
    min_xyz = bounds[:, 0]
    max_xyz = bounds[:, 1]
    bounds = max_xyz[:, None] - min_xyz[:, None]
    grid_coords = (pts - min_xyz[:, None]) / bounds
    grid_coords = grid_coords * 2 - 1
    # convert xyz to zyx, since the blend weight is indexed by xyz
    grid_coords = grid_coords.flip(-1)

    # the blend weight is indexed by xyz
    uv = uv.permute(0, 4, 1, 2, 3)
    grid_coords = grid_coords[:, None, None]
    uv = F.grid_sample(uv,
                       grid_coords,
                       mode=mode,
                       padding_mode='border',
                       align_corners=True)
    uv = uv[:, :, 0, 0]

    return uv

Specifically, for this code,

# convert xyz to zyx, since the blend weight is indexed by xyz
grid_coords = grid_coords.flip(-1)

I still confused why we need to flip last dimension.
Is uv map volume is defined (Z,Y,X,2) ??
I am confused about if volume locations are created with coordinate of vertex, I think the Z's value would be smallest one.
However, the actual shape of the uv volume tensor shape is (72,68,18,2), indeed looks like third-dimension(18) would be Z-dimension coordinate.

May be my elementary misunderstanding, thanks!

Can't find MonoCap Dataset

I couldn't find the relevant download link in the link(VCAI: VCAI-ASSETS (mpg.de)) you provided for MonoCap Dataset. Could you please provide a more detailed link?

Someing wrong when training on Monocap dataset

An error occurred when I used the link to email to download the monocap dataset and used it to run instant-nvr.

Is there something wrong with the data but the Dc.Sida Peng already told me the dataset is already preprocessed?

Question about bweights (pbw)

Hi, i want to transfer the zju-mocap from opencv to opengl coordinates.
And follow A-nerf, I need to perform the transformation matrix on RH and TH.
However, I notive that in instant-nvr, we also use the blending weight, So i am curious that Do I need to perform the coordinates
transformation on this weight ?
I guess the answer is no since it is in the smpl coodinates

question about refined_ZJU_MoCap

Hi,I'd like to know the detailed description of refined_ZJU_MoCap,especially about the "colmap" ,"params","annots.npy","schp" and the "smpl_meta" data,thank you so much.

How Can I render avatar in static tpose

Thanks for your amazing work! I would like to know if I can render avatar in tpose, is there a script for that?

and the bullet-type render is not working. it seems like a "render_utils" file is missing

  File "/home/zilongwang/Research/Diffusion_Human/instant-nvr/lib/datasets/h36m/tpose_novel_view_dataset.py", line 12, in <module>
    from lib.utils import render_utils
ImportError: cannot import name 'render_utils' from 'lib.utils' (unknown location)

hope you can take a look at this part

How to estimate SMPL parameters like in the datasets

Hello,

I've been working with the datasets provided in this repository (ZJU_Mocap and MonoCap) and encountered an issue when integrating custom data. The root of the problem lies in the difference between the SMPL parameters estimated by us and those provided with your datasets.

Could you share the method or tools used to estimate your dataset's SMPL parameters?

Plan about instruction on running on custom datasets

Thank you for your exceptional work. I greatly appreciate your efforts.

I would like to compare your method using custom datasets, but there are several things that I am unsure of how to obtain.

Could you kindly provide a brief instruction on how to handle custom datasets? I would be grateful for your assistance.

Unable to find bigpose_uv.npy file in the ZJU-Mocap dataset

Hi,
Thanks for the amazing work! I am trying to run the code base on ZJU-MoCap dataset, but in the dataset I downloaded, there are no 'bigpose_uv.npy' and "head_uv.obj" files. Did you create them separately? Or there's something wrong with the dataset I donwloaded?

How to Generate Human Masks for Custom Datasets

Hi,

Thank you for the instructions on using custom datasets (#6). However, the instructions do not include how to generate human masks like the 'mask' and 'schp' folders in the MonoCap and ZJU-Mocap-Refine datasets.

Could you provide a simple guide on how to create human masks for custom datasets?

normal in the ZJU-mocap dataset

There are only a few datasets published in the readme:
└── zju-mocap
├── my_377
├── my_386
├── my_387
├── my_392
├── my_393
└── my_394
313, 315, 390 is missing,mainly the lack of normal among them. how can I make it up

Question about transform view direction

Hello,

Thank you for the excellent work you've shared here! I have a query concerning the necessity of transforming directions from the pose space to the T pose. May I ask what's the purpose of this step? Are there any potential drawbacks if we choose to omit this transformation process? Since I think optimizing two pose spaces with different view directions can reach similar results.

Thank you in advance for your time and insights.
Best regards.