hustvl / gneuvox

GNeuVox: Generalizable Neural Voxels for Fast Human Radiance Fields

GNeuVox: Generalizable Neural Voxels for Fast Human Radiance Fields

Project Page | Arxiv Paper | Video

Generalizable Neural Voxels for Fast Human Radiance Fields
Taoran Yi^1*, Jiemin Fang^2,1*, Xinggang Wang^1✉, Wenyu Liu^1
1School of EIC, HUST   ²Institute of AI, HUST  
^* denotes equal contributions.

In this paper, we propose a rendering framework that can learn moving human body structures extremely quickly from a monocular video, named as GNeuVox. The framework is built by integrating both neural fields and neural voxels. Especially, a set of generalizable neural voxels are constructed. With pretrained on various human bodies, these general voxels represent a basic skeleton and can provide strong geometric priors. For the fine-tuning process, individual voxels are constructed for learning differential textures, complementary to general voxels. Thus learning a novel body can be further accelerated, taking only a few minutes. Our method shows significantly higher training efficiency compared with previous methods, while maintaining similar rendering quality.

Prerequisite

Configure Environment

Create and activate a virtual environment.

conda create --name gneuvox python=3.7
conda activate gneuvox

Install the required packages.

pip install -r requirements.txt

Download SMPL Model

Download the gender neutral SMPL model from here, and unpack mpips_smplify_public_v2.zip.

Copy the smpl model.

SMPL_DIR=/path/to/smpl
MODEL_DIR=$SMPL_DIR/smplify_public/code/models
cp $MODEL_DIR/basicModel_neutral_lbs_10_207_0_v1.0.0.pkl third_parties/smpl/models

Follow this page to remove Chumpy objects from the SMPL model.

Data Preparation

Prepare The ZJU-Mocap Dataset

First, download ZJU-Mocap dataset from here.

Second, modify the yaml file of subject 377 at tools/prepare_zju_mocap/377.yaml (e.g., subject 377).

Finally, run the data preprocessing script.

cd tools/prepare_zju_mocap
python prepare_dataset.py --cfg 377.yaml
python prepare_dataset_eval.py --cfg 377_eval.yaml
cd ../../

Prepare The People Snapshot Dataset

First, you have to first read and agree the license terms here. Then you can download the data we have processed with EasyMocap from google drive and baidu drive.

Prepare A Monocular Self-rotating Video

Use EasyMocap to process the video and run the data preprocessing script.

cd tools/prepare_easy_mocap
python prepare_datase_easy_mocap.py --cfg easy_mocap.yaml
cd ../../

Training

python train.py --cfg configs/scene/zju_mocap/377.yaml

You can download the pretrained models from google drive and baidu drive. Put the pretrained models in experiments/$task/$experiment. Then use _C.resume = True in configs/config.py to load the pretrained models.

Evaluation

You can use the following script to evaluate the model.

python eval.py --cfg configs/scene/zju_mocap_eval/377.yaml 

Render output

Render the frame input (i.e., training sequence).

python run.py --type movement \
   --cfg configs/scene/zju_mocap_eval/377.yaml   

Run free-viewpoint rendering on a particular frame (e.g., frame 128).

python run.py --type freeview \
    --cfg configs/scene/zju_mocap_eval/377.yaml   \
    freeview.frame_idx 128

Render the learned canonical appearance (T-pose).

python run.py --type tpose \
    --cfg configs/scene/zju_mocap_eval/377.yaml   

Main results

The ZJU-Mocap Dataset

Method	Pretrain Dataset	Perscene Iterations	Time	PSNR	SSIM	LPIPS (x10-2)
Neural Body	✗	-	-	29.08	0.9616	5.229
HumanNeRF	✗	400k	∼ hours	30.24	0.9680	3.173
GNeuVox	✗	10k	50 mins	30.26	0.9678	3.450
GNeuVox	ZJU-MoCap	1k	5 mins	30.26	0.9682	3.420
GNeuVox	Human3.6M	3k	15 mins	30.11	0.9677	3.399

Acknowledgements

Our code is partially based on HumanNeRF and DirectVoxGO. Thanks for their awesome works.

Citation

If you find this repository/work helpful in your research, welcome to cite the paper and give a ⭐.

@article{gneuvox,
        title={Generalizable Neural Voxels for Fast Human Radiance Fields},
        author={Taoran Yi and Jiemin Fang and Xinggang Wang and Wenyu Liu},
        journal={arxiv:2303.15387},
        year={2023}
        }

gneuvox's Issues

Rendering Camera angle issue for People Snapshot

Hello @taoranyi,

I have trained all my models from scratch. For people-snapshot, The rendering result is good but the camera angle is not accurate.
for z-axis

for y-axis

I have processed the file from EasyMocap "demo for motion capture",
I am not understanding what could be the error.

I checked your cameras.pkl, it is the same as mine. Have you faced any issues like this?

Producing Empty Images Issue during Training.

@taoranyi Hello,

I am actually facing the issue during training - Producing Empty Images. It's a new dataset . Do you potentially know the cause for this? It seems a value mismatch between the camera rays on the image, BUT how can I debug this step-wise?

Thank you in advanced !!

[Dataset Path] dataset/genebody/barry
 -- Total Frames: 16
************************************
genebody_barry_train
core.data.human_nerf.train
[Dataset Path] dataset/genebody/barry
 -- Total Frames: 600
Epoch: 1 [Iter 20, 19/600 (3%), 10.47 sec] Loss: 0.8744 [lpips: 0.8236 mse: 0.0508 ]
Epoch: 1 [Iter 40, 39/600 (6%), 8.81 sec] Loss: 0.2724 [lpips: 0.2506 mse: 0.0218 ]
Epoch: 1 [Iter 60, 59/600 (10%), 8.90 sec] Loss: 0.4936 [lpips: 0.4518 mse: 0.0418 ]
Epoch: 1 [Iter 80, 79/600 (13%), 8.84 sec] Loss: 0.5618 [lpips: 0.5370 mse: 0.0248 ]
Epoch: 1 [Iter 100, 99/600 (16%), 8.84 sec] Loss: 0.2704 [lpips: 0.2668 mse: 0.0036 ]
Evaluate Progress Images ...
  0%|                                                                                                                                                                                          | 0/16 [00:50<?, ?it/s]
Produce empty images; reload the init model.
Load checkpoint from experiments/human_nerf/genebody/barry/single_gpu/init.tar ...
Epoch: 1 [Iter 20, 119/600 (20%), 59.33 sec] Loss: 0.3576 [lpips: 0.3395 mse: 0.0180 ]

Pre-training

Hi, The repository doesn't have code for pre-training, only fine tuning code is released. Would you have some pointers on how to implement the pre-training pipeline with a custom dataset?

How to prepare a monocular dataset from easymocap?

I want to know the basic format of the data set, because it is not clearly written in the framework of easymocap. Thanks a lot.

H36m training requirements -

Hello @taoranyi,

I appreciate your work here. I also planning to train humannerf on h36m dataset, atleast for 1 subject.
Could you please tell me, how can we get the h36m smpl parameters and the necessary mask

I tried to use the Easymocap file, since h36m provides the 3d groundtruth https://github.com/zju3dv/EasyMocap/blob/master/apps/demo/smpl_from_keypoints.py (Is this the correct way?)

So I got the smpl parameters like this as an example: I used the h36m Positions_3D_mono files (are these correct files)

  {  "id": 1611,
    "Rh": [
      [-1.109, 4.132, -2.318]
    ],
    "Th": [
      [-246.794, 285.948, 657.016]
    ],
    "poses": [
      [0.000, 0.000, 0.000, 0.917, -1.202, 1.189, 0.836, 1.005, 0.382, 2.144, -0.020, -0.395, -0.627, -1.443, 0.152, ...]
    ],
    "shapes": [
      [11905.026, -2553.326, 698.123, -111.642, 616.076, 136.446, -253.905, -107.915, 117.896, 356.510]
    ]
}

then I generated the mask using SAM -

I used the prepare_dataset.py for the humannerf wild/monocular since I am using a single camera from h36m.
after all that, I got the data loader error, when I placed num_workers to 0,

The near, far list are empty which are generated from the "joints". The bbox values are vast:
min x: -247
max x: +256

So, I need little help on what I could be doing wrong here, is it the masking issue or smpl issue or something? Really need your help, since you already trained on h36m.

Thank you once again. I hope you can help me out

Help for Easymocap Error - People-Snapshot Dataset

@taoranyi, Hello

Since it is mentioned in the repo, that the People-snapshot dataset was processed using easymocap "Demo motion capture on monocular videos". I tried to do the same but for some reason, I encountered dataloader issue.

Steps I followed:

1. Extracted the images using "extract_images.py" and stored it in " /home/ndip/EasyMocap/images/01/ folder" and 2dkeypoints in /home/ndip/EasyMocap/keypoints/01/
2. Running the following command - python apps/demo/mocap.py /home/ndip/EasyMocap/ --work internet --subs 01
3. I have placed the SPIN, MANO and SMPL-H in the respective folders
4. I did not pass the intrinsic and extrinisic yml files - so the files were generated automatic
5. There is an output folder named - output-smpl-3d which has 3 yml files

Why is the "exp" always smpl-3d, is there a way to change the name? Since there are many modes available.

It will be helpful if you can provide any information, how you processed the Snapshot dataset using Easymocap

Processing data with Easymocap doesn't produce masks, but the processing script requires masks.

Is there a way to generate masks directly from the easymocap processing code?

How to train the pretrained model?

Thanks for the great work~ I wonder how to train the pretrained model?

About the experiment using The People Snapshot Dataset

Thank you for the excellent work!

When experimenting with The People Snapshot Dataset, there were some bugs when loading the data, which should be related to the dataset format. I don't know whether I did not find the right way to do it, but after making some simple changes to the code, it can run successfully.

GNeuVox/core/data/dataset/train.py

Lines 169 to 172 in 6893782

	keyword = 'frame_'+ self.now_subject
	print('keyword',keyword)
	img_paths = list_files(os.path.join(self.dataset_path, 'images'),
	exts=['.png'],keyword = keyword)

img_paths = list_files(os.path.join(self.dataset_path, 'images'), exts=['.png'])

GNeuVox/core/data/dataset/train.py

Lines 376 to 377 in 6893782

	subject = frame_name_split_list[1]
	time_int = int(frame_name_split_list[2])

subject = self.now_subject
time_int = int(frame_name)

GNeuVox/core/data/dataset/eval_cam.py

Line 365 in 6893782

time_int = int(frame_name_split_list[2])

time_int = int(frame_name)

mesh

Can you provide a file for generating mesh？