xliu443 / stable-diffusion-prompt-inpainting

This project helps you do prompt-based inpainting without having to paint the mask - using Stable Diffusion and Clipseg

stable-diffusion-prompt-inpainting

This project helps you do prompt-based inpainting without having to paint the mask - using Stable Diffusion and Clipseg

It's currently a notebook based project but we can convert it into a Gradio Web UI.

It takes 3 mandatory inputs.

1. Input Image URL
2. Prompt of the part in the input image that you want to replace
3. Output Prompt

There are certain parameters that you can tune

1. Mask Precision
2. Stable Diffusion Generation Strength

Step-by-Step Tutorial - https://www.youtube.com/watch?v=bv1Wur3DvZ0

Inital Req:

• If you are using Stable Diffusion from Hugging Face 🤗 for the first time, You need to accept ToS on the Model Page and Get your Token from your user profile

Requires:

• GPU (runs fine on Google Colab Tesla T4)

Credit:

• Clipseg Model Weights - https://github.com/timojl/clipseg
• Stable Diffusion In-painting - https://www.youtube.com/watch?v=N913hReVxMM

Sample Work: