patrikhuber / 4dface

Real-time 3D face tracking and reconstruction from 2D video

4dface: Real-time 3D face tracking and reconstruction from 2D video

This is a demo app showing face tracking and 3D Morphable Model fitting on live webcams and videos. It builds upon the 3D face model library eos and the landmark detection and optimisation library superviseddescent.

(caveat: due to recent additions, "real time" at the moment means around 5 fps.)

Note: This is a research demo from 2015/2016. 4dface has since been commercialised and an improved, up-to-date version is available on www.4dface.io. This repository also contains an older version of the morphable face model fitting library eos. eos is still in active development, head over to github.com/patrikhuber/eos for the latest version.

Build & run

1. Clone with submodules: git clone --recursive git://github.com/patrikhuber/4dface.git, or, if you've already cloned it, get the submodules with git submodule update --init --recursive inside the 4dface directory.

2. Make sure you've got boost (>=1.54.0 should do), OpenCV (>=3.0), Eigen (>=3.3.0) and a recent compiler (>=gcc-5, >=clang-4, >=VS2017) installed. For Ubuntu 14.04 and newer, this will do the trick:

   sudo add-apt-repository ppa:ubuntu-toolchain-r/test
   sudo apt-get update
   sudo apt-get install gcc-7 g++-7 libboost-all-dev libeigen3-dev libopencv-dev opencv-data
   

   For Windows, we recommend vcpkg to install the Boost, OpenCV and Eigen dependencies.

3. Build the app: Run from outside the source directory:

   1. mkdir build && cd build

   2. cmake -DCMAKE_INSTALL_PREFIX=../install -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER=gcc-7 -DCMAKE_CXX_COMPILER=g++-7 -DOpenCV_haarcascades_DIR=/usr/share/opencv/haarcascades/ ../4dface/

   On Windows, add -G "Visual Studio 15 2017 Win64". Also, you will probably need to add -C ../4dface/initial_cache.cmake as first argument - copy the file from initial_cache.cmake.template and adjust the paths. Or, with vcpkg, use -DCMAKE_TOOLCHAIN_FILE=....

   If you get an error about OpenCV_haarcascades_DIR, adjust -DOpenCV_haarcascades_DIR to point to the directory of haarcascade_frontalface_alt2.xml from OpenCV.

4. Type make or build in Visual Studio.

5. Type make install, or run the INSTALL target in Visual Studio, to copy all required files into a share/ directory next to the executable.

Then just double-click the 4dface app from the install-directory or run with 4dface -i videofile to run on a video.

Keyboard shortcuts

When running the 4dface app, q quits, r resets the tracking, and s saves an OBJ of the current model to the hard disk (into the directory where it was run from).

Working with the libraries

If you're interested in working with the libraries, we recommend to clone and build them separately. They come with their own CMake project files and have their own GitHub issues pages.

• eos: A lightweight header-only 3D Morphable Face Model fitting library in modern C++11/14
• superviseddescent: A C++11 implementation of the supervised descent optimisation method

License & contributions

This code is licensed under the Apache License, Version 2.0. The subprojects are also licensed under the Apache License, Version 2.0, except for the 3D morphable face model, which is free for use for non-commercial purposes - for commercial purposes, contact the Centre for Vision, Speech and Signal Processing.

Contributions are very welcome! (best in the form of pull requests.) Please use Github issues for any bug reports, ideas, and discussions.

If you use this code in your own work, please cite one (or both) of the following papers:

• Fitting 3D Morphable Models using Local Features, P. Huber, Z. Feng, W. Christmas, J. Kittler, M. Rätsch, IEEE International Conference on Image Processing (ICIP) 2015, Québec City, Canada [PDF].

• A Multiresolution 3D Morphable Face Model and Fitting Framework, P. Huber, G. Hu, R. Tena, P. Mortazavian, W. Koppen, W. Christmas, M. Rätsch, J. Kittler, International Conference on Computer Vision Theory and Applications (VISAPP) 2016, Rome, Italy [PDF].

4dface's Issues

DrawFace on Video

Is there a way to instead of drawing the grid, draw the 3d model in that position?

Thanks.

Add more PCA coefficients in 4D face to obtain a more accurate morphable model

Hi Patrik,

Thank you very much for your release of this genius application. It works prefectly to construct 3D face model in the real time.

But I have a problem when I want to construct a 3D model from different videos of a large variety of faces , il seems that the constrcuted models always look like the moyen face, which means that the deformation of the morphable model is not so obvious, then I check the shape coefficients which has just 5 elements (Perhaps the number of shape coefficients is not abundant), how can I add more PCA coefficients by detail if I want to get a more accurate model which doesn't look like the moyen face that you have offered ??????

Waiting for your answers.

Best regards

Texture.cpp: use COLOR_BGR2BGRA instead of CV_BGR2BGRA

I was compiling 4dface and figured out that while compiling if you change to COLOR_BGR2BGRA, we can compile using latest version of opencv.

best

the speed of the method

Hi, I study face tracking recently, and I find your code. I want to use your code on the phone, so the speed is important for me, but the time of my study is limited, so I want to know the fps of process on Andriod or IOS. Would you mind tell me?
Hope your reply. Thank you very much.

fit_shape_model

fit_shape_model can not be founded;where is it?

error: ‘const class Eigen::ArrayWrapper<const Eigen::Matrix<float, -1, 1> >’ has no member named ‘rsqrt’

hello and thank you for the code
I'm trying to compile on KDE Neon (Ubuntu 16.04)
and got the following error :

make
Scanning dependencies of target 4dface
[ 50%] Building CXX object CMakeFiles/4dface.dir/apps/4dface.cpp.o
In file included from /shared/foss/4dface/external/eos/include/eos/morphablemodel/MorphableModel.hpp:26:0,
from /shared/foss/4dface/apps/helpers.hpp:27,
from /shared/foss/4dface/apps/4dface.cpp:20:
/shared/foss/4dface/external/eos/include/eos/morphablemodel/PcaModel.hpp: In function ‘Eigen::MatrixXf eos::morphablemodel::normalise_pca_basis(const MatrixXf&, const VectorXf&)’:
/shared/foss/4dface/external/eos/include/eos/morphablemodel/PcaModel.hpp:399:78: error: ‘const class Eigen::ArrayWrapper<const Eigen::Matrix<float, -1, 1> >’ has no member named ‘rsqrt’
const Eigen::VectorXf one_over_sqrt_of_eigenvalues = eigenvalues.array().rsqrt();
^
CMakeFiles/4dface.dir/build.make:62: recipe for target 'CMakeFiles/4dface.dir/apps/4dface.cpp.o' failed
make[2]: *** [CMakeFiles/4dface.dir/apps/4dface.cpp.o] Error 1
CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/4dface.dir/all' failed
make[1]: *** [CMakeFiles/4dface.dir/all] Error 2
Makefile:149: recipe for target 'all' failed
make: *** [all] Error 2

this is the output of cmake :

cmake -DCMAKE_INSTALL_PREFIX=../4dface_install -DCMAKE_BUILD_TYPE=Release -DOpenCV_haarcascades_DIR=/usr/share/opencv/haarcascades/ ../4dface/
-- The C compiler identification is GNU 5.4.0
-- The CXX compiler identification is GNU 5.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found OpenCV: /usr (found suitable version "3.2.0", minimum required is "3") found components: core imgcodecs imgproc highgui videoio objdetect
-- OpenCV include dir found at /usr/include;/usr/include/opencv
-- OpenCV library dir found at
-- Boost version: 1.58.0
-- Found the following Boost libraries:
-- system
-- filesystem
-- program_options
-- Boost found at /usr/include
-- Found Eigen3: /usr/local/include/eigen3 (Required is at least version "2.91.0")
-- Eigen3 found: TRUE
-- Eigen3 include dir found at /usr/local/include/eigen3
-- Eigen3 version: 3.2.10
-- Downloading face_landmarks_model_rcr_68.bin (84.6 MB)...
-- [download 100% complete]
....
-- face_landmarks_model_rcr_68.bin successfully downloaded.
-- Configuring done
-- Generating done
-- Build files have been written to: /shared/foss/4dface_build

No warning, No error, No make...

Hi, I have been stuck here for many days...
I follow the instruction but after I do the cmake ....., no make file showed. so I cannot do the make. I'm new to related things so I have hard time figuring out what's wrong. Please help me, thank you.
the following are the command line and [result.]
C:\ZhouJunfeng\project\4dface\build>cmake -C ../4dface/initial_cache.cmake -DCMAKE_INSTALL_PREFIX=../install -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER=gcc-7 -DCMAKE_CXX_COMPILER=g++-7 -DOpenCV_haarcascades_DIR=C:\\ZhouJunfeng\\project\\vcpkg\\vcpkg\\packages\\opencv_x64-windows\\share\\opencv\\haarcascades\\ ../4dface/ -G "Visual Studio 15 2017" loading initial cache file ../4dface/initial_cache.cmake -- The C compiler identification is MSVC 19.16.27027.1 -- The CXX compiler identification is MSVC 19.16.27027.1 -- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2017/Community/VC/Tools/MSVC/14.16.27023/bin/Hostx86/x86/cl.exe -- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2017/Community/VC/Tools/MSVC/14.16.27023/bin/Hostx86/x86/cl.exe -- works -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Detecting C compile features -- Detecting C compile features - done -- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2017/Community/VC/Tools/MSVC/14.16.27023/bin/Hostx86/x86/cl.exe -- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2017/Community/VC/Tools/MSVC/14.16.27023/bin/Hostx86/x86/cl.exe -- works -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Detecting CXX compile features -- Detecting CXX compile features - done -- Found OpenCV: C:/ZhouJunfeng/project/vcpkg/vcpkg/installed/x64-windows (found suitable version "3.4.3", minimum required is "3") found components: core imgcodecs imgproc highgui videoio objdetect -- OpenCV include dir found at C:/ZhouJunfeng/project/vcpkg/vcpkg/installed/x64-windows/include;C:/ZhouJunfeng/project/vcpkg/vcpkg/installed/x64-windows/include/opencv -- OpenCV library dir found at C:\ZhouJunfeng\project\vcpkg\vcpkg\packages\opencv_x64-windows\lib -- Boost version: 1.59.0 -- Found the following Boost libraries: -- system -- filesystem -- program_options -- Boost found at C:/tools/boost_1_59_0 -- Eigen3 found: TRUE -- Eigen3 include dir found at C:\ZhouJunfeng\project\vcpkg\vcpkg\installed\x64-windows\include\eigen3 -- Eigen3 version: 3.3.7 -- Configuring done -- Generating done -- Build files have been written to: C:/ZhouJunfeng/project/4dface/build

Using texture images from mtl file

Hi

I want to use texture images from mtl file instead of using isomap function image, could you give me any hint, please?

how to read the NoseTri.bin.bin .EyeballsTri.bin...

Hi, how to get the data in regions file ,can you give me the code ? @patrikhuber

How can I get the HD expression_blendshapes_XXX.bin

Dear Patrik,

Thanks for your code, which is very helpful for my research. I want to try the HD morphable model. I have downloaded the HD "sfm_shape_16759.bin" and "sfm_shape_29587.bin", but I don't find where to download the corresponding "expresssion_blendshapes_167559.bin" and "expresssion_blendshapes_29587.bin" files, as well as "ibug2did.txt" files? I wonder whether you provide these files for test?

Best Regards
Linwei

Error when using the BFM model

After using the conversion scripts available (convert_bfm2009_to_raw_binary.m ; bfm-binary-to-cereal ),
I got this message
Loading shape PCA basis matrix with 160470 rows and 199 cols.
Loading colour PCA basis matrix with 160470 rows and 199 cols.
Warning: PCA shape model's data dimension is different from the number of textur
e coordinates given. The converted model is still saved, but does most likely no
t work correctly for texturing.

But I was wondering about the warning is there any problem in the conversion process. Because when I used the generated file in 4dface project I got this error:

OpenCV Error: Assertion failed (s >= 0) in cv::setSize, file C:\opencv\github-2.
4\opencv\modules\core\src\matrix.cpp, line 116
OpenCV Error: Insufficient memory (Failed to allocate 128508788 bytes) in cv::Ou
tOfMemoryError, file C:\opencv\github-2.4\opencv\modules\core\src\alloc.cpp, lin
e 52

dlib landmarks failed under big face angle towards camera

Hi @patrikhuber
I noticed the shape predictor cannot output landmark failed under big face angle towards camera. Does it work as intended? If so, is there any way to fix it?
PS I used mtcnn(https://github.com/jjjjohnson/MTCNN_face_detection_alignment) as face detector and dlib.shape_predictor to output the landmarks.

Thanks a lot!

Make Install Error

Hi,

I'm having some errors with the make install step, I've followed the readme steps. This is the error I'm getting.

nathan@nathan-HP-ENVY-x360-Convertible:~/build2$ make install
[100%] Built target 4dface
Install the project...
-- Install configuration: "Release"
-- Installing: /usr/local/bin/4dface
CMake Error at cmake_install.cmake:47 (file):
file INSTALL cannot copy file "/home/nathan/build2/4dface" to
"/usr/local/bin/4dface".

Makefile:85: recipe for target 'install' failed
make: *** [install] Error 1

Any help would be appreciated and great work with all your projects.

How to implement merged isomap and visualize mesh in Python

Hi @patrikhuber,
I checked the help(eos) and cannot find any info about merging isomap. And it is in the helpers.hpp for cpp version. Is is something I should write as the helper function?

Thanks a lot!
Junjie

Compilation error: Opencv undefined reference

I follow the build&run process exactly on Ubuntu 16.04 and encounter the following error when execute the command in build folder:

make

Appreciate any help.

[ 50%] Building CXX object CMakeFiles/4dface.dir/apps/4dface.cpp.o
[100%] Linking CXX executable 4dface
CMakeFiles/4dface.dir/apps/4dface.cpp.o: In function `cv::String::String(char const*) [clone .constprop.2718]':
4dface.cpp:(.text+0x1c69): undefined reference to `cv::String::allocate(unsigned long)'
CMakeFiles/4dface.dir/apps/4dface.cpp.o: In function `cv::Mat::Mat(int, int, int, void*, unsigned long) [clone .constprop.2716]':
4dface.cpp:(.text+0x1d6a): undefined reference to `cv::String::allocate(unsigned long)'
4dface.cpp:(.text+0x1db7): undefined reference to `cv::error(int, cv::String const&, char const*, char const*, int)'
4dface.cpp:(.text+0x1dbf): undefined reference to `cv::String::deallocate()'
4dface.cpp:(.text+0x1dec): undefined reference to `cv::String::deallocate()'
CMakeFiles/4dface.dir/apps/4dface.cpp.o: In function `draw_wireframe(cv::Mat, eos::render::Mesh const&, glm::tmat4x4<float, (glm::precision)0>, glm::tmat4x4<float, (glm::precision)0>, glm::tvec4<float, (glm::precision)0>, cv::Scalar_<double>)':
4dface.cpp:(.text+0x29e0): undefined reference to `cv::line(cv::_InputOutputArray const&, cv::Point_<int>, cv::Point_<int>, cv::Scalar_<double> const&, int, int, int)'
4dface.cpp:(.text+0x2a84): undefined reference to `cv::line(cv::_InputOutputArray const&, cv::Point_<int>, cv::Point_<int>, cv::Scalar_<double> const&, int, int, int)'
4dface.cpp:(.text+0x2b05): undefined reference to `cv::line(cv::_InputOutputArray const&, cv::Point_<int>, cv::Point_<int>, cv::Scalar_<double> const&, int, int, int)'
CMakeFiles/4dface.dir/apps/4dface.cpp.o: In function `rcr::draw_landmarks(cv::Mat, cv::Mat, cv::Scalar_<double>)':
4dface.cpp:(.text+0x2f59): undefined reference to `cv::circle(cv::_InputOutputArray const&, cv::Point_<int>, int, cv::Scalar_<double> const&, int, int, int)'
CMakeFiles/4dface.dir/apps/4dface.cpp.o: In function `draw_axes_topright(float, float, float, cv::Mat)':
4dface.cpp:(.text+0x6559): undefined reference to `cv::line(cv::_InputOutputArray const&, cv::Point_<int>, cv::Point_<int>, cv::Scalar_<double> const&, int, int, int)'
4dface.cpp:(.text+0x65f8): undefined reference to `cv::line(cv::_InputOutputArray const&, cv::Point_<int>, cv::Point_<int>, cv::Scalar_<double> const&, int, int, int)'
4dface.cpp:(.text+0x6694): undefined reference to `cv::line(cv::_InputOutputArray const&, cv::Point_<int>, cv::Point_<int>, cv::Scalar_<double> const&, int, int, int)'
CMakeFiles/4dface.dir/apps/4dface.cpp.o: In function `cv::String::String(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)':
4dface.cpp:(.text._ZN2cv6StringC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE[_ZN2cv6StringC5ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE]+0x2f): undefined reference to `cv::String::allocate(unsigned long)'
CMakeFiles/4dface.dir/apps/4dface.cpp.o: In function `_ZNK2cv3MatcvNS_3VecIT_XT0_EEEIfLi4EEEv':
4dface.cpp:(.text._ZNK2cv3MatcvNS_3VecIT_XT0_EEEIfLi4EEEv[_ZNK2cv3MatcvNS_3VecIT_XT0_EEEIfLi4EEEv]+0x63): undefined reference to `cv::String::allocate(unsigned long)'
4dface.cpp:(.text._ZNK2cv3MatcvNS_3VecIT_XT0_EEEIfLi4EEEv[_ZNK2cv3MatcvNS_3VecIT_XT0_EEEIfLi4EEEv]+0x119): undefined reference to `cv::error(int, cv::String const&, char const*, char const*, int)'
4dface.cpp:(.text._ZNK2cv3MatcvNS_3VecIT_XT0_EEEIfLi4EEEv[_ZNK2cv3MatcvNS_3VecIT_XT0_EEEIfLi4EEEv]+0x121): undefined reference to `cv::String::deallocate()'
4dface.cpp:(.text._ZNK2cv3MatcvNS_3VecIT_XT0_EEEIfLi4EEEv[_ZNK2cv3MatcvNS_3VecIT_XT0_EEEIfLi4EEEv]+0x240): undefined reference to `cv::String::deallocate()'
CMakeFiles/4dface.dir/apps/4dface.cpp.o: In function `_ZNK2cv3MatcvNS_3VecIT_XT0_EEEIfLi3EEEv':
4dface.cpp:(.text._ZNK2cv3MatcvNS_3VecIT_XT0_EEEIfLi3EEEv[_ZNK2cv3MatcvNS_3VecIT_XT0_EEEIfLi3EEEv]+0x63): undefined reference to `cv::String::allocate(unsigned long)'
4dface.cpp:(.text._ZNK2cv3MatcvNS_3VecIT_XT0_EEEIfLi3EEEv[_ZNK2cv3MatcvNS_3VecIT_XT0_EEEIfLi3EEEv]+0x119): undefined reference to `cv::error(int, cv::String const&, char const*, char const*, int)'
4dface.cpp:(.text._ZNK2cv3MatcvNS_3VecIT_XT0_EEEIfLi3EEEv[_ZNK2cv3MatcvNS_3VecIT_XT0_EEEIfLi3EEEv]+0x121): undefined reference to `cv::String::deallocate()'
4dface.cpp:(.text._ZNK2cv3MatcvNS_3VecIT_XT0_EEEIfLi3EEEv[_ZNK2cv3MatcvNS_3VecIT_XT0_EEEIfLi3EEEv]+0x260): undefined reference to `cv::String::deallocate()'
CMakeFiles/4dface.dir/apps/4dface.cpp.o: In function `_ZNK2cv3MatcvNS_3VecIT_XT0_EEEIfLi2EEEv':
4dface.cpp:(.text._ZNK2cv3MatcvNS_3VecIT_XT0_EEEIfLi2EEEv[_ZNK2cv3MatcvNS_3VecIT_XT0_EEEIfLi2EEEv]+0x63): undefined reference to `cv::String::allocate(unsigned long)'
4dface.cpp:(.text._ZNK2cv3MatcvNS_3VecIT_XT0_EEEIfLi2EEEv[_ZNK2cv3MatcvNS_3VecIT_XT0_EEEIfLi2EEEv]+0x119): undefined reference to `cv::error(int, cv::String const&, char const*, char const*, int)'
4dface.cpp:(.text._ZNK2cv3MatcvNS_3VecIT_XT0_EEEIfLi2EEEv[_ZNK2cv3MatcvNS_3VecIT_XT0_EEEIfLi2EEEv]+0x121): undefined reference to `cv::String::deallocate()'
4dface.cpp:(.text._ZNK2cv3MatcvNS_3VecIT_XT0_EEEIfLi2EEEv[_ZNK2cv3MatcvNS_3VecIT_XT0_EEEIfLi2EEEv]+0x22c): undefined reference to `cv::String::deallocate()'
CMakeFiles/4dface.dir/apps/4dface.cpp.o: In function `rcr::HogTransform::operator()(cv::Mat, unsigned long, int)':
4dface.cpp:(.text._ZN3rcr12HogTransformclEN2cv3MatEmi[_ZN3rcr12HogTransformclEN2cv3MatEmi]+0x19d1): undefined reference to `cv::String::allocate(unsigned long)'
4dface.cpp:(.text._ZN3rcr12HogTransformclEN2cv3MatEmi[_ZN3rcr12HogTransformclEN2cv3MatEmi]+0x1a22): undefined reference to `cv::error(int, cv::String const&, char const*, char const*, int)'
4dface.cpp:(.text._ZN3rcr12HogTransformclEN2cv3MatEmi[_ZN3rcr12HogTransformclEN2cv3MatEmi]+0x1a2e): undefined reference to `cv::String::deallocate()'
4dface.cpp:(.text._ZN3rcr12HogTransformclEN2cv3MatEmi[_ZN3rcr12HogTransformclEN2cv3MatEmi]+0x1dc4): undefined reference to `cv::String::deallocate()'
CMakeFiles/4dface.dir/apps/4dface.cpp.o: In function `main':
4dface.cpp:(.text.startup+0xa88): undefined reference to `cv::CascadeClassifier::load(cv::String const&)'
4dface.cpp:(.text.startup+0xa96): undefined reference to `cv::String::deallocate()'
4dface.cpp:(.text.startup+0xcd6): undefined reference to `cv::VideoCapture::open(cv::String const&)'
4dface.cpp:(.text.startup+0xce2): undefined reference to `cv::String::deallocate()'
4dface.cpp:(.text.startup+0xe9e): undefined reference to `cv::namedWindow(cv::String const&, int)'
4dface.cpp:(.text.startup+0xeaa): undefined reference to `cv::String::deallocate()'
4dface.cpp:(.text.startup+0xed1): undefined reference to `cv::String::allocate(unsigned long)'
4dface.cpp:(.text.startup+0xeee): undefined reference to `cv::namedWindow(cv::String const&, int)'
4dface.cpp:(.text.startup+0xefa): undefined reference to `cv::String::deallocate()'
4dface.cpp:(.text.startup+0x1390): undefined reference to `cv::CascadeClassifier::detectMultiScale(cv::_InputArray const&, std::vector<cv::Rect_<int>, std::allocator<cv::Rect_<int> > >&, double, int, int, cv::Size_<int>, cv::Size_<int>)'
4dface.cpp:(.text.startup+0x2a73): undefined reference to `cv::String::allocate(unsigned long)'
4dface.cpp:(.text.startup+0x2a95): undefined reference to `cv::imshow(cv::String const&, cv::_InputArray const&)'
4dface.cpp:(.text.startup+0x2aa1): undefined reference to `cv::String::deallocate()'
4dface.cpp:(.text.startup+0x2af4): undefined reference to `cv::String::allocate(unsigned long)'
4dface.cpp:(.text.startup+0x2b11): undefined reference to `cv::imshow(cv::String const&, cv::_InputArray const&)'
4dface.cpp:(.text.startup+0x2b1d): undefined reference to `cv::String::deallocate()'
4dface.cpp:(.text.startup+0x3295): undefined reference to `cv::String::allocate(unsigned long)'
4dface.cpp:(.text.startup+0x32dc): undefined reference to `cv::imwrite(cv::String const&, cv::_InputArray const&, std::vector<int, std::allocator<int> > const&)'
4dface.cpp:(.text.startup+0x32e8): undefined reference to `cv::String::deallocate()'
4dface.cpp:(.text.startup+0x3477): undefined reference to `cv::imshow(cv::String const&, cv::_InputArray const&)'
4dface.cpp:(.text.startup+0x3483): undefined reference to `cv::String::deallocate()'
4dface.cpp:(.text.startup+0x39f7): undefined reference to `cv::String::deallocate()'
4dface.cpp:(.text.startup+0x3a21): undefined reference to `cv::String::deallocate()'
4dface.cpp:(.text.startup+0x3b86): undefined reference to `cv::String::deallocate()'
4dface.cpp:(.text.startup+0x3d02): undefined reference to `cv::String::deallocate()'
CMakeFiles/4dface.dir/apps/4dface.cpp.o:4dface.cpp:(.text.startup+0x3d16): more undefined references to `cv::String::deallocate()' follow
collect2: error: ld returned 1 exit status
CMakeFiles/4dface.dir/build.make:101: recipe for target '4dface' failed
make[2]: *** [4dface] Error 1
CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/4dface.dir/all' failed
make[1]: *** [CMakeFiles/4dface.dir/all] Error 2
Makefile:149: recipe for target 'all' failed
make: *** [all] Error 2

Is it possible to use my own face model?

Hi,
thank you for your amazing project. However, I want to know if I can use my own face model? I noticed that now your project is using a sfm_shape_3448.bin file made by cereal. However, my face model is an obj file. I wish to know how to incorporate my own face model into your project? What jobs are needed to be done? Is your code compatible with different face model?

Thanks!

Does 4dface fit only to landmarks or edges as well

Hi,

Thank you for making 4dface available.
I notice that 4dface depends on eos under the hood, and eos has been modified so that it fits to both landmarks and edges, based on "Fitting a 3D Morphable Model to Edges: A Comparison Between Hard and Soft Correspondences" .
So my question was, is 4dface currently fitting to both, or just landmarks (as in simple fit of eos) ?

Thank you, and apologies in advance if this isn't the right place to post this.

Gropuing mesh and texture

Hi

my model contains mesh that grouped into submeshes, so I have texture for each mesh, how can I bind these textures to their corresponding meshes?

Possibility of choosing quality versus speed trade-offs to decrease frame processing times?

Hi

First of all, thanks for the great work you are doing with 4dface and all the other related libraries.

I ran the latest pre-built release and it works correctly, however the example runs at about 5 FPS.

I have the following question: what would be the best ways / parameters that could be tweaked to decrease processing times, at the expense of losing some of the accuracy?

For example:

• would input image down-scaling help reduce times?
• could a simpler face mesh (with less vertices) help decrease mesh fitting times?

These the specs of the machine I'm using:

• Microsoft Windows 10 Pro
• NVIDIA GeForce GTX 1080
• Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz, 4001 Mhz, 4 main processors, 8 logic processors
• 32.0 GB of RAM
• Logitech C930e Webcam (USB 2.0, max FPS seems to be 30)

Regards

Undefined symbols for architecture x86_64: Mac Sierra + Homebrew

I have tried to compile on Mac OS X Sierra, using Homebrew for the dependencies, and failed.

The cmake command:

cmake -DCMAKE_INSTALL_PREFIX=../install -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER=gcc-7 -DCMAKE_CXX_COMPILER=g++-7 -DOpenCV_haarcascades_DIR=/usr/local/Cellar/opencv/3.4.1_2/share/OpenCV/haarcascades ../

I placed the "build" dir inside the "4dface" dir.
And this is the failure message.

$ make
Scanning dependencies of target 4dface
[ 50%] Building CXX object CMakeFiles/4dface.dir/apps/4dface.cpp.o
[100%] Linking CXX executable 4dface
Undefined symbols for architecture x86_64:
  "cv::CascadeClassifier::detectMultiScale(cv::_InputArray const&, std::vector<cv::Rect_<int>, std::allocator<cv::Rect_<int> > >&, double, int, int, cv::Size_<int>, cv::Size_<int>)", referenced from:
      _main in 4dface.cpp.o
  "cv::imwrite(cv::String const&, cv::_InputArray const&, std::vector<int, std::allocator<int> > const&)", referenced from:
      _main in 4dface.cpp.o
  "boost::program_options::to_internal(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)", referenced from:
      boost::program_options::basic_command_line_parser<char>::basic_command_line_parser(int, char const* const*) in 4dface.cpp.o
  "boost::program_options::validation_error::get_template[abi:cxx11](boost::program_options::validation_error::kind_t)", referenced from:
      boost::program_options::validation_error::validation_error(boost::program_options::validation_error::kind_t, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int) in 4dface.cpp.o
  "boost::program_options::options_description::options_description(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int, unsigned int)", referenced from:
      _main in 4dface.cpp.o
  "boost::program_options::invalid_option_value::invalid_option_value(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)", referenced from:
      void boost::program_options::validate<boost::filesystem::path, char>(boost::any&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, boost::filesystem::path*, long) in 4dface.cpp.o
  "boost::program_options::error_with_option_name::error_with_option_name(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)", referenced from:
      boost::program_options::validation_error::validation_error(boost::program_options::validation_error::kind_t, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int) in 4dface.cpp.o
  "boost::program_options::arg[abi:cxx11]", referenced from:
      boost::program_options::typed_value<boost::filesystem::path, char>::name[abi:cxx11]() const in 4dface.cpp.o
  "boost::program_options::detail::cmdline::cmdline(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&)", referenced from:
      boost::program_options::basic_command_line_parser<char>::basic_command_line_parser(int, char const* const*) in 4dface.cpp.o
  "boost::program_options::operator<<(std::basic_ostream<char, std::char_traits<char> >&, boost::program_options::options_description const&)", referenced from:
      _main in 4dface.cpp.o
  "boost::program_options::error_with_option_name::substitute_placeholders(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const", referenced from:
      vtable for boost::exception_detail::error_info_injector<boost::program_options::invalid_option_value> in 4dface.cpp.o
      vtable for boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::program_options::invalid_option_value> > in 4dface.cpp.o
      vtable for boost::exception_detail::error_info_injector<boost::program_options::validation_error> in 4dface.cpp.o
      vtable for boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::program_options::validation_error> > in 4dface.cpp.o
      vtable for boost::program_options::validation_error in 4dface.cpp.o
      vtable for boost::program_options::invalid_option_value in 4dface.cpp.o
  "boost::program_options::value_semantic_codecvt_helper<char>::parse(boost::any&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, bool) const", referenced from:
      vtable for boost::program_options::typed_value<boost::filesystem::path, char> in 4dface.cpp.o
ld: symbol(s) not found for architecture x86_64
collect2: error: ld returned 1 exit status
make[2]: *** [4dface] Error 1
make[1]: *** [CMakeFiles/4dface.dir/all] Error 2
make: *** [all] Error 2

eos v0.12.1

Hi,

With the eos update v0.12.1, does 4dface work?

Regards,

Andoni.

Face Fine Details Fitting

I noticed that the fitting module of eos only contains code of fitting landmarks (related to function fitting::fit_shape_and_pose). The result 3D model is smooth and do not have much details. Is there any way to improve the details (i.e wrinkles) of the result model?
I thought that the detail missing is caused by the low resolution of morphable model (3448 vertices model). But even I used the higher resolution model, I do not get the expected details.

Make Error on Mac

Hi,
Thanks for great work man,,, I'm trying to run it on mac,

cmake -DCMAKE_INSTALL_PREFIX=../install -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER=/opt/local/bin/gcc-mp-5 -DCMAKE_CXX_COMPILER=/opt/local/bin/g++-mp-5 -DOpenCV_haarcascades_DIR=/opt/local/share/OpenCV/haarcascades/ ../4dface/

is ok and done successfully but when i want to make the program i got

Scanning dependencies of target 4dface
[ 50%] Building CXX object CMakeFiles/4dface.dir/apps/4dface.cpp.o
[100%] Linking CXX executable 4dface
Undefined symbols for architecture x86_64:
  "cv::CascadeClassifier::detectMultiScale(cv::_InputArray const&, std::vector<cv::Rect_<int>, std::allocator<cv::Rect_<int> > >&, double, int, int, cv::Size_<int>, cv::Size_<int>)", referenced from:
      _main in 4dface.cpp.o
  "cv::imwrite(cv::String const&, cv::_InputArray const&, std::vector<int, std::allocator<int> > const&)", referenced from:
      _main in 4dface.cpp.o
  "boost::program_options::to_internal(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)", referenced from:
      boost::program_options::basic_command_line_parser<char>::basic_command_line_parser(int, char const* const*) in 4dface.cpp.o
  "boost::program_options::validation_error::get_template[abi:cxx11](boost::program_options::validation_error::kind_t)", referenced from:
      boost::program_options::validation_error::validation_error(boost::program_options::validation_error::kind_t, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int) in 4dface.cpp.o
  "boost::program_options::options_description::options_description(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int, unsigned int)", referenced from:
      _main in 4dface.cpp.o
  "boost::program_options::invalid_option_value::invalid_option_value(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)", referenced from:
      void boost::program_options::validate<boost::filesystem::path, char>(boost::any&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, boost::filesystem::path*, long) in 4dface.cpp.o
  "boost::program_options::error_with_option_name::error_with_option_name(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)", referenced from:
      boost::program_options::validation_error::validation_error(boost::program_options::validation_error::kind_t, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int) in 4dface.cpp.o
  "boost::program_options::arg[abi:cxx11]", referenced from:
      boost::program_options::typed_value<boost::filesystem::path, char>::name() const in 4dface.cpp.o
  "boost::program_options::detail::cmdline::cmdline(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&)", referenced from:
      boost::program_options::basic_command_line_parser<char>::basic_command_line_parser(int, char const* const*) in 4dface.cpp.o
  "boost::program_options::operator<<(std::basic_ostream<char, std::char_traits<char> >&, boost::program_options::options_description const&)", referenced from:
      _main in 4dface.cpp.o
  "boost::program_options::error_with_option_name::substitute_placeholders(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const", referenced from:
      vtable for boost::exception_detail::error_info_injector<boost::program_options::invalid_option_value> in 4dface.cpp.o
      vtable for boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::program_options::invalid_option_value> > in 4dface.cpp.o
      vtable for boost::exception_detail::error_info_injector<boost::program_options::validation_error> in 4dface.cpp.o
      vtable for boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::program_options::validation_error> > in 4dface.cpp.o
      vtable for boost::program_options::validation_error in 4dface.cpp.o
      vtable for boost::program_options::invalid_option_value in 4dface.cpp.o
  "boost::program_options::value_semantic_codecvt_helper<char>::parse(boost::any&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, bool) const", referenced from:
      vtable for boost::program_options::typed_value<boost::filesystem::path, char> in 4dface.cpp.o
ld: symbol(s) not found for architecture x86_64
collect2: error: ld returned 1 exit status
make[2]: *** [4dface] Error 1
make[1]: *** [CMakeFiles/4dface.dir/all] Error 2
make: *** [all] Error 2

btw, my packages are updated,,
any idea abt this error?
Thanks in advance,,,

An error when run 4dface app

I did the steps as you say in readme.md , and my system is ubuntu 14.04 , but when I run the 4dface app ,it occurs ## "terminate called after throwing an instance of 'std::runtime_error' what(): Error opening given file: ../share/sfm_shape_3448.bin
已放弃 (核心已转储)"

How can I solve the problem ?

please help me, thank you

std::tie(mesh, rendering_params) = fitting::fit_shape_and_pose( ..........), you have fitted the pose, shape model, and expression blendshapes to landmarks in an iterative way using this function, and obtained a merged mesh, but why you merge the isomap and shape_coefficients again using the following two functions:
const Mat merged_isomap = isomap_averaging.add_and_merge(isomap);
shape_coefficients = pca_shape_merging.add_and_merge(shape_coefficients);

Eigen compilation errors on Ubuntu 16.10

Got a bunch of errors trying to compile. Any ideas?

[ 50%] Building CXX object CMakeFiles/4dface.dir/apps/4dface.cpp.o
In file included from /usr/include/eigen3/Eigen/Core:387:0,
from /home/vess/code/3d/experiments/4dface/4dface/external/eos/include/eos/fitting/blendshape_fitting.hpp:27,
from /home/vess/code/3d/experiments/4dface/4dface/external/eos/include/eos/fitting/fitting.hpp:32,
from /home/vess/code/3d/experiments/4dface/4dface/apps/4dface.cpp:24:
/usr/include/eigen3/Eigen/src/Core/AssignEvaluator.h: In instantiation of ‘void Eigen::internal::call_assignment_no_alias(Dst&, const Src&, const Func&) [with Dst = Eigen::Matrix<float, -1, -1, 1>; Src = Eigen::Inverse<Eigen::ColPivHouseholderQR<Eigen::Matrix<float, -1, -1, 1> > >; Func = Eigen::internal::assign_op<float, float>]’:
/usr/include/eigen3/Eigen/src/Core/PlainObjectBase.h:721:41: required from ‘Derived& Eigen::PlainObjectBase::_set_noalias(const Eigen::DenseBase&) [with OtherDerived = Eigen::Inverse<Eigen::ColPivHouseholderQR<Eigen::Matrix<float, -1, -1, 1> > >; Derived = Eigen::Matrix<float, -1, -1, 1>]’
/usr/include/eigen3/Eigen/src/Core/PlainObjectBase.h:531:19: required from ‘Eigen::PlainObjectBase::PlainObjectBase(const Eigen::DenseBase&) [with OtherDerived = Eigen::Inverse<Eigen::ColPivHouseholderQR<Eigen::Matrix<float, -1, -1, 1> > >; Derived = Eigen::Matrix<float, -1, -1, 1>]’
/usr/include/eigen3/Eigen/src/Core/Matrix.h:379:29: required from ‘Eigen::Matrix<_Scalar, _Rows, _Cols, _Options, _MaxRows, _MaxCols>::Matrix(const Eigen::EigenBase&) [with OtherDerived = Eigen::Inverse<Eigen::ColPivHouseholderQR<Eigen::Matrix<float, -1, -1, 1> > >; _Scalar = float; int _Rows = -1; int _Cols = -1; int _Options = 1; int _MaxRows = -1; int _MaxCols = -1]’
/home/vess/code/3d/experiments/4dface/4dface/external/superviseddescent/include/superviseddescent/regressors.hpp:294:53: required from here
/usr/include/eigen3/Eigen/src/Core/AssignEvaluator.h:785:49: error: ambiguous template instantiation for ‘struct Eigen::internal::Assignment<Eigen::Matrix<float, -1, -1, 1>, Eigen::Inverse<Eigen::ColPivHouseholderQR<Eigen::Matrix<float, -1, -1, 1> > >, Eigen::internal::assign_op<float, float>, Eigen::internal::Dense2Dense, void>’
Assignment<ActualDstTypeCleaned,Src,Func>::run(actualDst, src, func);
^
/usr/include/eigen3/Eigen/src/Core/AssignEvaluator.h:823:8: note: candidates are: template<class DstXprType, class SrcXprType, class Functor, class Weak> struct Eigen::internal::Assignment<DstXprType, SrcXprType, Functor, Eigen::internal::Dense2Dense, Weak> [with DstXprType = Eigen::Matrix<float, -1, -1, 1>; SrcXprType = Eigen::Inverse<Eigen::ColPivHouseholderQR<Eigen::Matrix<float, -1, -1, 1> > >; Functor = Eigen::internal::assign_op<float, float>; Weak = void]
struct Assignment<DstXprType, SrcXprType, Functor, Dense2Dense, Weak>
^
In file included from /usr/include/eigen3/Eigen/LU:35:0,
from /usr/include/eigen3/Eigen/Dense:2,
from /home/vess/code/3d/experiments/4dface/4dface/external/eos/include/eos/fitting/closest_edge_fitting.hpp:37,
from /home/vess/code/3d/experiments/4dface/4dface/external/eos/include/eos/fitting/fitting.hpp:34,
from /home/vess/code/3d/experiments/4dface/4dface/apps/4dface.cpp:24:
/usr/include/eigen3/Eigen/src/LU/InverseImpl.h:290:8: note: template<class DstXprType, class XprType> struct Eigen::internal::Assignment<DstXprType, Eigen::Inverse, Eigen::internal::assign_op<typename DstXprType::Scalar, typename SrcXprType::Scalar>, Eigen::internal::Dense2Dense> [with DstXprType = Eigen::Matrix<float, -1, -1, 1>; XprType = Eigen::ColPivHouseholderQR<Eigen::Matrix<float, -1, -1, 1> >]
struct Assignment<DstXprType, Inverse, internal::assign_op<typename Ds
^
In file included from /usr/include/eigen3/Eigen/QR:36:0,
from /usr/include/eigen3/Eigen/Dense:4,
from /home/vess/code/3d/experiments/4dface/4dface/external/eos/include/eos/fitting/closest_edge_fitting.hpp:37,
from /home/vess/code/3d/experiments/4dface/4dface/external/eos/include/eos/fitting/fitting.hpp:34,
from /home/vess/code/3d/experiments/4dface/4dface/apps/4dface.cpp:24:
/usr/include/eigen3/Eigen/src/QR/ColPivHouseholderQR.h:620:8: note: template<class DstXprType, class MatrixType, class Scalar> struct Eigen::internal::Assignment<DstXprType, Eigen::Inverse<Eigen::ColPivHouseholderQR >, Eigen::internal::assign_op<Scalar, Scalar>, Eigen::internal::Dense2Dense> [with DstXprType = Eigen::Matrix<float, -1, -1, 1>; MatrixType = Eigen::Matrix<float, -1, -1, 1>; Scalar = float]
struct Assignment<DstXprType, Inverse<ColPivHouseholderQR >, intern
^
In file included from /usr/include/eigen3/Eigen/Core:387:0,
from /home/vess/code/3d/experiments/4dface/4dface/external/eos/include/eos/fitting/blendshape_fitting.hpp:27,
from /home/vess/code/3d/experiments/4dface/4dface/external/eos/include/eos/fitting/fitting.hpp:32,
from /home/vess/code/3d/experiments/4dface/4dface/apps/4dface.cpp:24:
/usr/include/eigen3/Eigen/src/Core/AssignEvaluator.h:785:49: error: incomplete type ‘Eigen::internal::Assignment<Eigen::Matrix<float, -1, -1, 1>, Eigen::Inverse<Eigen::ColPivHouseholderQR<Eigen::Matrix<float, -1, -1, 1> > >, Eigen::internal::assign_op<float, float>, Eigen::internal::Dense2Dense, void>’ used in nested name specifier
Assignment<ActualDstTypeCleaned,Src,Func>::run(actualDst, src, func);
^
CMakeFiles/4dface.dir/build.make:62: recipe for target 'CMakeFiles/4dface.dir/apps/4dface.cpp.o' failed
make[2]: *** [CMakeFiles/4dface.dir/apps/4dface.cpp.o] Error 1
CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/4dface.dir/all' failed
make[1]: *** [CMakeFiles/4dface.dir/all] Error 2
Makefile:149: recipe for target 'all' failed
make: *** [all] Error 2

Error in running the INSTALL target

Hi Patrik

Thanks for the code

I have this error message when I try to build the install project in /visual Studio 2015.

Unhandled exception at 0x000007FEFCCEA06D in 4dface.exe: Microsoft C++ exception: cereal::Exception at memory location 0x00000000001C0F40.

Face-detection performance

Hello Patrik Huber,

Thank you for sharing your work.
I just compiled 4dface and applied it on a few videos.
But it seems that it has some performance issue for detecting a face.
Can I get some instruction for improving?

Easy way to adjust frame size?

Happy to say I've also gotten 4dface running on a mac using Homebrew.

Is there an easy way to adjust the frame size the tracking runs on? Would performance improve beyond 5fps after reducing the frame size of the webcam video?

Thanks in advance!

Blendshapes of the 16k vertex model produce faulty shapes

Thank you for your effort in this repo! I've learnt a lot from it.

I was wondering if the blendshape is working as intended. In addition to writing a neutral face to an .obj file, I also wanted to write a face with expression to another file.
// neutral_expression and merged_mesh are defined in 4dface.cpp render::write_textured_obj(neutral_expression, "neutral.obj"); render::write_textured_obj(merged_mesh, "expression.obj");

The neutral face looks great, but not for the one with expression. I think it has something to do with the blendshape, but I'm not sure.

How to change the shape&expression model in 4dface?

I've tried to replace shape model, expression model and contour correspondence with the surrey 29587 version, but it aborted always, I wonder how could I use a higher resolution model to fit photos?

Model and blendshapes dimension mismatch

Hi Patrik,

Thanks for your code. I'm trying to run 4dface.exe with the binaries "sfm_1724.bin", "sfm_16759.bin" and "sfm_29587.bin" but I have this error message:

4dface -m "../share/sfm_29587.bin"

OpenCV Error: Assertion failed (C.type() == type && (((flags&GEMM_3_T) == 0 && C.rows == d_size.height && C.cols == d_size.width) || ((flags&GEMM_3_T) != 0 && C.rows == d_size.width && C.cols == d_size.height))) in cv::gemm, file C:\buildslave64\win64_amdocl\master_PackSlave-win64-vc14-shared\opencv\modules\core\src\matmul.cpp, line 923

At first, I thought the problem occurs because I was using "sfm_xxx.bin" (I don't have "sfm_shape_xxxx.bin") but the program works with "sfm_3448.bin".
I don't know if this error can occur because I don't have "expression_blendshapes_xxxx.bin".

Regards,

Andoni.

Update travis and AppVeyor configs

travis and AppVeyor need to be updated for OpenCV 3 and to work with the new eos version.
4dface builds just fine though, the build failures are just because travis and AppVeyor haven't been updated yet.

Publish pre-built executable

pre-built executable (coming soon)

I would love to try it out! The build steps are nice but a little daunting because my dev environment/knowledge is very far from being able to built this. Could you please publish the binaries?

How can I use my own set of 2D landmarks in eos

Dear Patrik,

Thanks for your great working, which is very helpful for my research.
I want to try use another set of 2D landmarks(72 landmarks), which I manually labeled the corresponding vertex on the PCA model, and change ibug2did.txt for remapping. I found that the results of the testing is wrong. How can I use my own set of 2D landmarks in eos?

Best Regards
Che #

Build instructions for Mac OS X (using macports)

Just have built it on Mac using macports.

If you'd like to update the doc here are the instructions.

Step 2.
sudo port install gcc5 boost eigen3 cmake

(E.g. mind different names and no need for g++, opencv-data)

Step 3.

cmake -DCMAKE_INSTALL_PREFIX=../install -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++ -DOpenCV_haarcascades_DIR=/opt/local/share/OpenCV/haarcascades/ ../4dface/

(Mind proper reference to compiler and library).

I would also mention that w/o command line parameters it uses webcam.

uv-mapping of a custom head model

Hi

I am using full head model. So I do not need to use the extracted texture from the video frame. When I passed the texture image to create_mipmapped_texture function I got a distorted texture, could explain to me how can I solve this problem? Do I use the correct function?

The result that I got is showed in the figure.

Regards

Moving the face in rendering window

Hi

I hope you are doing well.

I have a question about that why the face is moving when the speaker starts to talk, I mean the rendered face is moving not just the lips. Could you answer me, please?

Regards

Don't have "../4Dface/data/face_landmarks_model_rcr_68.bin"

There's no face_landmarks_model_rcr_68.bin in the 4dface document,I try face_landmarks_model_rcr_22.bin,it runs sucessfully,but the landmarks only have 22,Where can I get the face_landmarks_model_rcr_68.bin?Thank you very much.

Model fitting to videos

Quick question on how things work - it seems that reset the landmarks every frame. So does the fitting improve with multiple frames? Does the PCA get refined? Or is it each frame for itself?

rendering profile face

hi, patrik, how can I render face with different pose.
I tried yaw pitch roll in RenderingParameters, but no lucky.

Shortcuts in the 4dface app

Good morning,
First I would like to thank you for making such a cool software available as open source! I am running it on ubuntu throught the console command "cd install/bin" and "./4dface -i test.mp4", test.mp4 being a video file present in install/bin. I cannot run the application by double clicking on it in the folder install/bin directly, do you know if it is normal?
It opens two windows: my video with a green mesh on top of the face and a small render window where the 3d imaging of the face appears. My issue is I am stuck here with the functionalities: i cannot use the shortcuts q, r ans especially s in order to save the 3d model (mesh+texture). Could you please explain how to export the 3d model? Is there a command i could write in the console?

Thanks a lot for your work,
Quentin

Binding obj and texture

Hi, I'm new to computer vision. Just a quick question, how to display the reconstructed face using the saved obj and png files? Is there any sample code?

error when run 4dface

Error reading the RCR model "../share/face_landmarks_model_rcr_68.bin": Failed to read 14797344 bytes from input stream! Read 3905020

would you please provide some detail describes about how to fit 3DMM to frames in an iterative way? Thank you

every video outputs the same 3d face shape

I have tried 4dface on several videos. All of them when I press 's', outputs the extactly same 3d face model, with different textures. Is that normal? Or I did something wrong?

./4dface -i test.mov

---

ok, I see the problem. nose shapes are not quite right on eastern faces. like small nose faces.

Could not find Eigen3 with Visual studio 2017

I have done all but still getting this error

Severity Code Description Project File Line Suppression State
Error CMake Error at C:/Program Files (x86)/Microsoft Visual Studio/2017/Enterprise/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.8/Modules/FindPackageHandleStandardArgs.cmake:137 (message):
Could NOT find Eigen3 (missing: EIGEN3_INCLUDE_DIR EIGEN3_VERSION_OK)
(Required is at least version "2.91.0") C:/Program Files (x86)/Microsoft Visual Studio/2017/Enterprise/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.8/Modules/FindPackageHandleStandardArgs.cmake 137

on this place

macro(_FPHSA_FAILURE_MESSAGE _msg)
if (${_NAME}_FIND_REQUIRED)
message(FATAL_ERROR "${_msg}")
else ()
if (NOT ${_NAME}_FIND_QUIETLY)
message(STATUS "${_msg}")
endif ()
endif ()
endmacro()

Please help to solve, thank you

render::write_textured_obj does not work when the current directory path contains space

Path like C:\Program Files\4dface\ won't work

Running from a directory with no space in the path solves the problem.

How to retrieve the head pose estimation / Euler angles

The eos library provides a linear scaled orthographic projection estimation of the camera pose. Is there also the possibilty to retrieve the current head pose? Or alternatively translate the camera pose into the head pose?
To be specific the head pose in terms of the euler angles or rotation with its centroid for example on the nose tip and directed towards the "camera".

toml module is missing

I created symbolic links for the external modules eos and
When I build, it still has errors:

~/dev/4dface/build$ make
[ 50%] Building CXX object CMakeFiles/4dface.dir/apps/4dface.cpp.o
In file included from /home/alex/dev/4dface/4dface-master/apps/helpers.hpp:26:0,
from /home/alex/dev/4dface/4dface-master/apps/4dface.cpp:20:
/home/alex/dev/4dface/4dface-master/external/eos/include/eos/core/LandmarkMapper.hpp:25:20: fatal error: toml.hpp: No such file or directory
compilation terminated.