Please note that the main part of the code has been released, though we are still testing it to fix possible glitches. Thank you.
Python and C++ code for realistic 3D face modeling from single image using our shape and detail regression networks published in CVPR 2018 [1] (follow the link to our PDF which has many, many more reconstruction results.)
This page contains end-to-end demo code that estimates the 3D facial shape with realistic details directly from an unconstrained 2D face image. For a given input image, it produces standard ply files of the 3D face shape. It accompanies the deep networks described in our paper [1] and [2]. The occlusion recovery code, however, will be published in a future release. We also include demo code and data presented in [1].
Before compiling the code, please, make sure to have all the required data in the following specific folder:
- Download our Bump-CNN and move the CNN model (1 file:
ckpt_109_grad.pth.tar) into theCNNfolder - Download our PyTorch CNN model and move the CNN model (3 files:
shape_model.pth,shape_model.py,shape_mean.npz) into theCNNfolder - Download the Basel Face Model and move
01_MorphableModel.matinto the3DMM_modelfolder - Acquire 3DDFA Expression Model, run its code to generate
Model_Expression.matand move this file the3DMM_modelfolder - Go into
3DMM_modelfolder. Run the scriptpython trimBaselFace.py. This should output 2 filesBaselFaceModel_mod.matandBaselFaceModel_mod.h5. - Download dlib face prediction model and move the
.datfile into thedlib_modelfolder.
Note that we modified the model files from the 3DMM-CNN paper. Therefore, if you generated these files before, you need to re-create them for this code.
There are 2 options below to compile our code:
- Install Docker CE
- With Linux, manage Docker as non-root user
- Install nvidia-docker
- Build docker image:
docker build -t extreme-3dmm-docker .
The steps below have been tested on Ubuntu Linux only:
- Install Python2.7
- Install the required third-party packages:
sudo apt-get install -y libhdf5-serial-dev libboost-all-dev cmake libosmesa6-dev freeglut3-dev
- Install Dlib C++ library. Sample code to comiple Dlib:
wget http://dlib.net/files/dlib-19.6.tar.bz2
tar xvf dlib-19.6.tar.bz2
cd dlib-19.6/
mkdir build
cd build
cmake ..
cmake --build . --config Release
sudo make install
cd ..
- Install PyTorch
- Install other required third-party Python packages:
pip install opencv-python torchvision scikit-image cvbase pandas mmdnn dlib
- Config Dlib and HDF5 path in CMakefiles.txt, if needed
- Build C++ code
mkdir build;
cd build; \
cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=../demoCode ..;
make;
make install;
cd ..
This code should generate TestBump in demoCode folder
If you compile our code with Docker, you need to start a Docker container to run our code. You also need to set up a shared folder to transfer input/output data between the host computer and the container.
- Prepare the shared folder on the host computer. For example,
/home/ubuntu/shared - Copy input data (if needed) to the shared folder
- Start container:
nvidia-docker run --rm -ti --ipc=host --privileged -v /home/ubuntu/shared:/shared extreme-3dmm-docker bash
Now folder /home/ubuntu/shared on your host computer will be mounted to folder /shared inside the container
- Go into
demoCodefolder. The demo script can be used from the command line with the following syntax:
$ Usage: python testBatchModel.py <inputList> <outputDir>where the parameters are the following:
<inputList>is a text file containing the paths to each of the input images, one in each line.<outputDir>is the path to the output directory, where ply files are stored.
An example for <inputList> is demoCode/testImages.txt
../data/test/03f245cb652c103e1928b1b27028fadd--smith-glasses-too-faced.jpg ../data/test/20140420_011855_News1-Apr-25.jpg ....
The output 3D models will be <outputDir>/<imageName>_<postfix>.ply with <postfix> = <modelType>_<poseType>. <modelType> can be "foundation", "withBump" (before soft-symmetry),"sparseFull" (soft-symmetry on the sparse mesh), and "final". <poseType> can be "frontal" or "aligned" (based on the estimated pose).
The final 3D shape has <postfix> as "final_frontal". You can config the output models in code before compiling.
The PLY files can be displayed using standard off-the-shelf 3D (ply file) visualization software such as MeshLab.
Sample command:
python testBatchModel.py testImages.txt /sharedNote that our occlusion recovery code is not included in this release.
- Go into
demoCodefolder. The demo script can be used from the command line with the following syntax:
$ Usage: ./testPaperResults.shBefore exiting the docker container, remember to save your output data to the shared folder.
If you find this work useful, please cite our paper [1] with the following bibtex:
@inproceedings{tran2017extreme,
title={Extreme {3D} Face Reconstruction: Seeing Through Occlusions},
author={Tran, Anh Tuan and Hassner, Tal and Masi, Iacopo and Paz, Eran and Nirkin, Yuval and Medioni, G\'{e}rard},
booktitle={IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)},
year=2018
}[1] A. Tran, T. Hassner, I. Masi, E. Paz, Y. Nirkin, G. Medioni, "Extreme 3D Face Reconstruction: Seeing Through Occlusions", IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, June 2018
[2] A. Tran, T. Hassner, I. Masi, G. Medioni, "Regressing Robust and Discriminative 3D Morphable Models with a very Deep Neural Network", CVPR 2017
- Dec. 2018, Convert to Dockerfile
- Dec. 2017, First Release
Please, see the LICENSE here
If you have any questions, drop an email to [email protected] , [email protected] and [email protected] or leave a message below with GitHub (log-in is needed).
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent