hay / facetool

Command line utility to manipulate faces in videos and images

facetool.py

Command line utility to manipulate, recognize and detect faces in videos and images, written in Python 3

This utility allows you to do all kinds of operations on face images and videos, straight from the command line. Think of it as ffmpeg for faces.

Some things facetool can do:

• Face swapping
• Face recognition
• Create an average face
• Counting faces
• Cropping faces (even multiple ones from a single image)
• Classifying faces based on age and gender
• Extracting and combining frames from and to videos

Installation

In theory, this tool should work on any system that can run the required libraries (dlib, opencv and a couple of other ones). In practice i've only tested on macOS 10.14 and Ubuntu Linux.

General instructions

I highly recommend using poetry for installing the dependencies. You'll also need a working version of Python 3.6 or higher.

Clone the repo

git clone https://github.com/hay/facetool

Then run either

poetry install

Or

pip3 install dlib numpy opencv-python ffmpeg-python scikit-image tqdm imutils tensorflow pandas face-recognition dataknead sklearn

This can easily take an hour, so don't worry if it takes a while! After installing the libraries (either with or without poetry) try running the script by typing ./facetool.py.

If that all works you can try some of examples.

Ubuntu Linux (18.04)

If you're having problems running poetry you might need to symlink python3 to python

sudo ln -s /usr/bin/python3 /usr/bin/python

You might need a couple of dependencies

apt-get install python3-venv python3-dev cmake make build-essential libsm6 libxrender-dev ffmpeg

If you're getting

ModuleNotFoundError: No module named 'distutils.util'

When using poetry try

sudo apt-get install python3-distutils --reinstall

macOS

Install the dependencies using brew:

    brew install cmake ffmpeg

Examples

Face swapping on images and video files

Put the features of face.jpg on head.jpg and save the result as swap.jpg

facetool.py swap -i face.jpg -t head.jpg -o swap.jpg

Put the features of face.jpg on a video file called head.mp4 and save as swap.mp4

facetool.py swap -i face.jpg -t head.mp4 -o swap.mp4

Put the features of a video file called face.mp4 on another video file called head.mp4 and save as swap.mp4. The audio of face.mp4 will be used for the output.

facetool.py swap -i face.mp4 -t head.mp4 -o swap.mp4

Take one 'head' image called head.jpg and generate a new faceswap for every file in a directory called dir-to-face.

facetool.py swap -i faces -t head.jpg -o dir-to-face

The other way around: apply the face of face.jpg to a directory of heads and output to a directory called face-to-dir

facetool.py swap -i face.jpg -t heads -o face-to-dir

When having multiple faces in an input image, select the one you want to use (in the example there are three faces and we want the second face)

facetool.py swap -i face.jpg -t head.jpg -o swap.jpg -so x,0,x

Put one face on all faces in a target image

facetool.py swap -i face.jpg -t group.jpg -o swap.jpg -so 0 -sr

Swap two faces in a single image (given that the image has only two faces)

facetool.py swap -i group.jpg -t group.jpg -o group-swapped.jpg -so 1,0

Only swap nose and mouth

facetool.py swap -i face.jpg -t head.jpg -o swap.jpg --no-eyesbrows

Only swap eyes and brows

facetool.py swap -i face.jpg -t head.jpg -o swap.jpg --no-nosemouth

Only swap mouth

facetool.py swap -i face.jpg -t head.jpg -o swap.jpg --only-mouth

Face recognition and clustering

Calculate the distance between 'alice.jpg' and a folder full of faces

facetool.py distance -i alice.jpg -t faces

The same, but then print percentages instead of distance

facetool.py distance -i alice.jpg -t faces --as-percentage

Save the results to a CSV file instead of printing to the command line

facetool.py distance -i alice.jpg -t faces --as-percentage -of csv -o results.csv

Encode a path of images and save the encodings to a file called encodings.json

facetool.py encode -i faces -o encodings.json

Use the encodings.json file instead of recalculating all the encodings. Note the -m (model) switch.

facetool.py distance -i alice.jpg -m encodings.json

Cluster all faces in a set of images and output a json file

facetool.py cluster -i faces -o cluster.json

The same, but move the clustered images to a new set of directories

facetool.py cluster -i faces -o clustered-images

Use an existing .json file (as generated by the encode command)

facetool.py cluster -i encodings.json -o cluster.json

Face averaging

Create an 'average.jpg' face from a folder of faces

facetool.py average -i faces -o average.jpg

Do the same thing from a single image that has multiple faces

facetool.py average -i group.jpg -o average.jpg

You can also average all faces in a video, which will give an interesting, ephemeral effect

facetool.py average -i face.mp4 -o average.jpg

Save 'original' and 'transformed' averaged faces as well

facetool.py average -i faces -o average.jpg --save-originals --save-warped

Classifying age and gender

Get the age and gender of a single image and print to console

facetool.py classify -i face.jpg

Get the age and gender of all images in a folder and write to a csv file

facetool.py classify -i faces/ -of csv -o classified.csv

Face detection, position and cropping

Count the number of faces in face.jpg

facetool.py count -i face.jpg

Count the number of faces in all images in directory faces

facetool.py count -i faces

Show the bounding box of all faces in face.jpg

facetool.py locate -i face.jpg

Create a new image called face-box.jpg that draws bounding boxes around all faces in face.jpg

facetool.py locate -i face.jpg -o face.box.jpg

Draw bounding boxes on all faces for all images in directory faces and save to locations

facetool.py locate -i faces -o locations

Show the poses of all faces in face.jpg

facetool.py pose -i face.jpg

Create a new image called face-pose.jpg that shows the shapes and poses of face.jpg

facetool.py pose -i face.jpg -o face-pose.jpg

Crop all faces from face.jpg and save to new files in the directory cropped. This will also work with a single image with multiple faces.

facetool.py crop -i face.jpg -o cropped

Media utilites

Convert a movie file called movie.mp4 to a set of JPG files in a directory called frames (used for video swapping)

facetool.py extractframes -i movie.mp4 -o frames

Convert a set of JPG files from the directory frames to a movie file called movie.mp4 (used for video swapping)

facetool.py combineframes -i frames -o movie.mp4

Combine a WAV file with an existing movie and save to a new one

facetool.py combineaudio -i movie.mp4 -ai sound.wav -o movie-sound.mp4

Return metadata about an image or video file in JSON format

facetool.py probe -i movie.mp4

Troubleshooting

• Before opening an issue, try running your command with the -v (verbose) switch, because this will give you more debug information. When using -vv (extra verbose) facetool will abort the program on exceptions.
• Note that, by default, facetool doesn't stop at errors.

All options

You'll get this output when running facetool.py -h.

usage: facetool [-h] -i INPUT [-o OUTPUT] [-t TARGET] [-ai AUDIO_INPUT]
                [--as-percentage] [-bl BLUR] [-dd DATA_DIRECTORY] [-f]
                [-fr FRAMERATE] [-fa FEATHER] [-if] [-ih IMAGE_HEIGHT]
                [-iw IMAGE_WIDTH] [-kt] [-m MODEL] [--no-audio] [-nocc]
                [--no-eyesbrows] [--no-nosemouth] [--no-threading]
                [--only-mouth] [-of {default,csv,json}] [-pp PREDICTOR_PATH]
                [--profile] [-q] [-s] [--save-originals] [--save-warped]
                [--swap-method {faceswap,faceswap3d}] [-so SWAP_ORDER]
                [-sp SAMPLE_PERCENTAGE] [-sr] [--temp-dir TEMP_DIR] [-v] [-vv]
                [--warp-3d]
                [{average,classify,cluster,combineaudio,combineframes,count,distance,crop,encode,extractframes,landmarks,locate,pose,probe,sample,swap}]

Manipulate faces in videos and images

positional arguments:
  {average,classify,cluster,combineaudio,combineframes,count,distance,crop,encode,extractframes,landmarks,locate,pose,probe,sample,swap}

optional arguments:
  -h, --help            show this help message and exit
  -i INPUT, --input INPUT
                        Input file or folder, 'face' when swapping
  -o OUTPUT, --output OUTPUT
                        Output file or folder
  -t TARGET, --target TARGET
                        'Head' when swapping
  -ai AUDIO_INPUT, --audio-input AUDIO_INPUT
                        Add a separate audio file with the end result movie
  --as-percentage       Show face distances as percentages
  -bl BLUR, --blur BLUR
                        Amount of blur to use during colour correction
  -dd DATA_DIRECTORY, --data-directory DATA_DIRECTORY
                        Directory where the data files are located
  -f, --force           Force commands and ignore warnings, like with sample
  -fr FRAMERATE, --framerate FRAMERATE
  -fa FEATHER, --feather FEATHER
                        Softness of edges on a swapped face
  -if, --ignore-nofaces
                        When having no faces to swap, keep the original input
                        image
  -ih IMAGE_HEIGHT, --image-height IMAGE_HEIGHT
                        Height of output image / height
  -iw IMAGE_WIDTH, --image-width IMAGE_WIDTH
                        Width of output image / video
  -kt, --keep-temp      Keep temporary files (used with video swapping)
  -m MODEL, --model MODEL
                        Use a precalculated model (for calculating distances)
  --no-audio
  -nocc, --no-colour-correct
                        Don't colour correct
  --no-eyesbrows
  --no-nosemouth
  --no-threading        Don't use multithreading
  --only-mouth
  -of {default,csv,json}, --output-format {default,csv,json}
                        Specify output format
  -pp PREDICTOR_PATH, --predictor-path PREDICTOR_PATH
  --profile             Show profiler information
  -q, --quiet           Don't print output to the console
  -s, --swap            Swap input and target
  --save-originals      Save original images when averaging faces
  --save-warped         Save warped images when averaging faces
  --swap-method {faceswap,faceswap3d}
                        Swap method for faceswap (options are: ['faceswap',
                        'faceswap3d']
  -so SWAP_ORDER, --swap-order SWAP_ORDER
                        Comma-separated list with order of faceswaps on
                        target, implies a multiswap
  -sp SAMPLE_PERCENTAGE, --sample-percentage SAMPLE_PERCENTAGE
                        Percentage of files in a directory to randomly remove
                        (used for the sample command)
  -sr, --swap-order-repeat
                        When using --swap-order and there are not enough
                        target faces, repeat the sequence
  --temp-dir TEMP_DIR   Define the directory where temporary files should be
                        placed
  -v, --verbose         Show debug information
  -vv, --extra-verbose  Show debug information AND raise / abort on exceptions
  --warp-3d             Swap faces and morph to coordinates of target face

Limitations / TODO

• More advanced swapping methods like 'deepfake' are not supported.
• Even though you could use the library in your own scripts (instead of using the command line utility), this isn't very well supported yet.
• No multithreading / processor support.
• Face swapping is limited to one face.
• Operations on videos will remove the audio.

Testing

facetool doesn't have a proper test suite yet, but you could try running test-all.py in the test directory to try a couple of common examples.

License

Licensed under the MIT license.

Credits

Written by Hay Kranen.

This utility is built around well-known libraries like dlib, face_recognition and opencv.

Faceswapping

Faceswapping algorithm by Matthew Earl, licensed under the MIT license.

Copyright (c) 2015 Matthew Earl

Age and gender classifier

Age and gender classifier by Boyuan Jiang, licensed under the MIT license.

Copyright (c) 2017 Boyuan

facetool's Issues

Temporary directories should have a unique name to avoid deletion of files

Currently, when swapping videos, output of a swap goes to directories like out-tmp and head-tmp. However, when running multiple instances of facetool at the same time, the new instance will delete the other folders. So, temporary directories should have a unique name.

When classifying, some images tend to dump errors like this:

ValueError: Cannot feed value of shape (1, 1604, 1598, 3) for Tensor 'input_image:0', which has shape '(?, 160, 160, 3)'

Very few actually work. Many produce no stdout at all.
Using tensorflow 1.13.1, on Python 3.6.

Support for Apple Silicon / M1

Unfortunately facetool doesn't build on Apple Silicon because numpy doesn't compile on Apple Silicon. The appropriate numpy issue can be found here.