somuchtome / faceswap

Face swap via diffusion models [Lora+IP-Adapter+Controlnet+text embedding optimization]

Face Swap via Diffusion Model

PyTorch implementation of Face Swap via Diffusion Model. Technical Report:https://arxiv.org/abs/2403.01108

Environment setup

cd Faceswap
conda env create -f environment.yaml 
conda activate swap
cd ldm
pip install -e ".[torch]"

Note

the diffusers have been modified and we define new pipe "StableDiffusionControlNetFaceInpaintPipeline" to support IP-adapter and our text embedding optimization which is different from huggingface-released.

Download Pretrained Weights

The weights required for FaceParser can be downloaded from link. The weights required for ArcFace can be downloaded from link.

mkdir checkpoints
mv arcface_checkpoint.tar checkpoints/ 
mv FaceParser.pth checkpoints/ 

Directories structure

The dataset and checkpoints should be placed in the following structures below

Faceswap
├── checkpoints
    ├── arcface_checkpoint.tar
    ├── FaceParser.pth
├── data
    └── src
        ├── 001.png
        └── ...
    └── targ
        ├── 001.png
        └── ...

Face swap

Place source and target images in data/src, and data/targ. Then run the following.

sh swap.sh

Face restoration postprocess

# git clone this repository
git clone https://github.com/sczhou/CodeFormer.git
cd CodeFormer
python basicsr/setup.py develop

Quick Inference

Download Pre-trained Models:

Download the facelib and dlib pretrained models from [Releases | Google Drive | OneDrive] to the weights/facelib folder. You can manually download the pretrained models OR download by running the following command:

python scripts/download_pretrained_models.py facelib
python scripts/download_pretrained_models.py dlib (only for dlib face detector)

Download the CodeFormer pretrained models from [Releases | Google Drive | OneDrive] to the weights/CodeFormer folder. You can manually download the pretrained models OR download by running the following command:

python scripts/download_pretrained_models.py CodeFormer

🧑🏻 Face Restoration (cropped and aligned face)

# For cropped and aligned faces (512x512)
python inference_codeformer.py -w 0.5 --has_aligned --input_path [image folder]|[image path]

Acknowledgments

This code borrows heavily from DiffFace, Diffusers, Codeformer and Lora.

faceswap's Issues

Test results

Great work! Could you please share some test results? e.g. source, target, output?

The weights required for FaceParser can't get？

3q