hansonrobotics / evenet

A TensorFlow implementation of DeepMind's WaveNet paper for generation of facial animations.

EveNet: Expression of Emotion and Visimes Network

This is a fork of ibab's excellent implementation of WaveNet. Here we are implementing changes for the generation of facial animations.

Experiments

Network with softmax output layer

Authors of wavenet stated that modeling the conditional distribution:

with softmax distribution tends to work better than some of previous modeling, such as mixture density network or mixture of conditional Gaussian scale mixtures.

This approach models each shape key values of each frame to softmax probability distribution of 11 classes. The probability values are calculated with the method explained here.

Here is github branch for this Experiment.

Shape key sampling

This approach is variant of approach adopted in pixelcnn, to extend the dependencies among pixels to color channels. In pixelrnn paper the author stated the joint probability for pixel sampling is

,

Where, is pixel.
They also extend this pixel dependencies to color channels as follows:-

If we use indexes for color channels(assign 1 to R index, 2 to G index and 3 to B index), the joint probability p(x) now can be written as

This type of joint probability can be applied to shape key sampling. Now the above joint probability can be rewritten for shape key sampling as

where S is number of samples, N is number of shape keys in single frame, is frame and is jth shape key of ith frame.

Code implementation for this experiment is found on github branch.

Requirements

TensorFlow needs to be installed before running the training script. Code is tested on TensorFlow version 1.0.1 for Python 2.7 and Python 3.5.

In addition, librosa must be installed for reading and writing audio.

To install the required python packages, run

pip install -r requirements.txt

For GPU support, use

pip install -r requirements_gpu.txt

Running tests

Install the test requirements

pip install -r requirements_test.txt

Run the test suite

./ci/test.sh

Related projects

• tensorflow-wavenet, the WaveNet implementation this is based on.

evenet's Issues

Docker Container of public version of the Hanson Robotics software

• Separate the proprietary from public portions of the code
• Create test cases to validate container
• Deploy onto Docker Hub

Quantify the accuracy of the model output

Explore approaches to quantify the accuracy of the model output.

• Define a a standard panel of users that will provide feedback
• Define scoring method
• Validate score with existing example output

Create Training Dataset

• Identify Director, actors and studio
• Develop scripts for actors
• Obtain cameras
• Film 3D movies of these different emotions
• Edit footage
• Label movies

Evaluate alternative wavenet architectures

These alternatives should be considered using 2 criteria:

• accuracy
• training and inference times

• Research Approaches
• Document Pros and Cons
• Build Prototype of selected architectures

Create a list of issues with current EveNet model

• Define set of animations to use for the evaluation
• Score each animation using method defined in this issue
• Create a prioritised list of issues