georgefwt / face-landmark-controlnet

Proof of concept for control landmarks in diffusion models!

Face landmark ControlNet

ControlNet with Face landmark

I trained using ControlNet, which was proposed by lllyasviel, on a face dataset. By using facial landmarks as a condition, finer face control can be achieved.

Currently, I’m using Stable Diffusion 1.5 as the base model and dlib as the face landmark detector (those with the capability can replace it with a better one). Find the checkpoint here https://huggingface.co/georgefen/Face-Landmark-ControlNet.

Create conda environment:

conda env create -f environment.yaml
conda activate control
wget http://dlib.net/files/shape_predictor_68_face_landmarks.dat.bz2
bzip2 -d shape_predictor_68_face_landmarks.dat.bz2

Testing it by:

python gradio_landmark2image.py

Generate face with the identical poses and expression

To create a new face, input an image and extract the facial landmarks from it. These landmarks will be used as a reference to redraw the face while ensuring that the original features are retained.

Control the facial expressions and poses of generated images

For the images we generated, we have the prompt and random seed used to generate them. While keeping the prompt and random seed, we can also edit the landmarks to modify the facial expressions and postures of the generated results.

Credits

Thanks to lllyasviel for his amazing work on https://github.com/lllyasviel/ControlNet!, this is totaly based on his work.

This is just a proof of concept and should not be applied for any risky purposes.

face-landmark-controlnet's Issues

Could you please publish the train code?

Thank you for the awesome work you did!

I was wondering if you kindly publish the train code for the custom ControlNet conditioning you made.

Whether it is possible to achieve the state of generating pictures with the features of the original picture on the face

Does the current repo control only the expressions or can it also control the facial features of the generated image, i.e. more like the original image itself.
I also noticed that there is a preliminary repo that is trying to control the facial features, is there any reference.

https://github.com/wangqixun/image-controlnet

How to use it in sd-webui-controlnet extension?

Yes, I downloaded the 5gb model and placed it in models\controlnet, it can be loaded within the webui extension, but it seems that its guidiance strength is not enough, and ofc, without a preprocessor. How do I achieve the effects given, and what file should I put to get the annotator?

MiraclePlus Potential Entrepreneurs Seeking

I am Haowen Lei from MiraclePlus, focusing on AIGC. I have read your project "Face-Landmark-ControlNet" and am anxious to establish contact with you. If you are willing to communicate, please add my WeChat： Lion_LeiHw

About pth

hi, where can i find 'control_sd15_landmarks.pth'