Mingwu Zheng, Haiyu Zhang, Hongyu Yang, Di Huang
Official code for CVPR 2023 paper NeuFace: Realistic 3D Neural Face Rendering from Multi-view Images.
The paper presents a novel 3D face rendering model, namely NeuFace, to learn accurate and physically-meaningful underlying 3D representa tions by neural rendering techniques. NeuFace naturally incorporates the neural BRDFs into physically based rendering, capturing facial geometry and complex appearance properties in a collaborative manner. Additionally, NeuFace has a decent generalization ability to common objects.
The code is compatible with python 3.6.13 and pytorch 1.9.1.
To create an anaconda environment named neuface with the required dependencies, run:
conda create -n neuface python==3.6.13
conda activate neuface
pip install -r requirement.txt
For human face, We use data from FaceScape Dataset to evaluate our model. The detailed 3D mesh is used to generate a mask of each image's face area. For more information about the data preprocessing of 3D mesh, refer to ImFace. The ImFace model can be download from pretrained-model.
For common objects, the DTU dataset is used for model evaluation.
To train NeuFace on Facescape dataset, run:
python scripts/train_pl.py
Make sure that the variables in your config file are correct. Results can be found in {out_dir}/{expname}.
The trained model can be downloaded from (FaceScape's author allows to release the trained model):
| Trained Model | Description |
|---|---|
| NeuFace_1_id_2_exp | train on 1 id with 2 exp (smile) of Facescape dataset |
If you want to use our trained model, please place the downloaded file in exp_pl/ckpt/{trained_model}.
To evaluate the novel view metrics, run:
python scripts/eval_pl.py --ckpt [ckpt_path] --out_dir [our_dir]
Results can be found in {our_dir}/test/{expname}.
To train NeuFace on DTU dataset, run:
cd common_object
python training/exp_runner.py --conf ./confs/dtu_fixed_cameras.conf --scan_id [scan_id] --gpu [GPU_ID]
Make sure that [dataset.data_dir] in your config file is correct. The results can be found in common_object/exps/{train.expname}/{timestamp}. The trained model can be downloaded from:
| Trained Model | Description |
|---|---|
| NeuFace_DTU_65 | train on 65 scan of DTU dataset |
| NeuFace_DTU_110 | train on 110 scan of DTU dataset |
| NeuFace_DTU_118 | train on 118 scan of DTU dataset |
If you want to use our trained model, please place the downloaded file in common_object/exps/{trained_model}.
To evaluate the training view metrics, run:
cd common_object
python evaluation/eval.py --conf ./confs/dtu_fixed_cameras.conf --scan_id [SCAN_ID] --eval_rendering --gpu [GPU_INDEX]
Results can be found in common_object/evals/{train.expname}/rendering.
If you find our work useful in your research, please consider citing:
@inproceedings{zheng2023neuface,
title={NeuFace: Realistic 3D Neural Face Rendering from Multi-view Images},
author={Zheng, Mingwu and Zhang Haiyu and Yang, Hongyu and Huang, Di},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
year={2023}
}
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent