The official implementation of paper "BrushNet: A Plug-and-Play Image Inpainting Model with Decomposed Dual-Branch Diffusion"
Home Page: https://tencentarc.github.io/BrushNet/
License: Other
Hi,
Thank you for releasing the 1.5 model for segmented inpainting. It's working pretty great but it's not working as well for masks that don't follow segmentations. Do you plan on releasing the random brush model as well ?
Thanks
No response
Hi, thanks for your code. I have a question, if I want to remove an object (for example I want to remove this blueberry), how should I write the prompt? If I wrote "remove the blueberry" it might not work well.
Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...].
I noticed you preprocessed the images and generated several features for each image, could you provide the preprocessing codes? Thanks a lot!
Describe the solution you'd like.
A clear and concise description of what you want to happen.
Describe alternatives you've considered.
A clear and concise description of any alternative solutions or features you've considered.
Additional context.
Add any other context or screenshots about the feature request here.
In section 4.3 of the paper, I saw that you mentioned blend operation, but I didn’t see them in the code?
No response
Code
I've noticed a detail that during training, the mask uses INTER_CUBIC and the latent dimension of the mask has continuous values between 0 and 1. However, during inference, it always uses discrete values of 0 and 1. Is there a problem with this?
Running examples/brushnet/app_brushnet.py fails with this error message:
File "/media/iwoolf/tenT/BrushNet/./examples/brushnet/app_brushnet.py", line 10, in
from diffusers import StableDiffusionBrushNetPipeline, BrushNetModel, UniPCMultistepScheduler
ImportError: cannot import name 'StableDiffusionBrushNetPipeline' from 'diffusers' (/media/iwoolf/BigDrive/anaconda3/envs/Brushnet/lib/python3.9/site-packages/diffusers/init.py)
I have tried uninstalling and re-installing diffusers, but it made no difference.
I'm running Ubuntu 22.04.4LTS
python examples/brushnet/app_brushnet.py
/media/iwoolf/BigDrive/anaconda3/envs/Brushnet/lib/python3.9/site-packages/transformers/utils/generic.py:485: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
_torch_pytree._register_pytree_node(
/media/iwoolf/BigDrive/anaconda3/envs/Brushnet/lib/python3.9/site-packages/transformers/utils/generic.py:342: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
_torch_pytree._register_pytree_node(
/media/iwoolf/BigDrive/anaconda3/envs/Brushnet/lib/python3.9/site-packages/transformers/utils/generic.py:342: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
_torch_pytree._register_pytree_node(
Traceback (most recent call last):
File "/media/iwoolf/tenT/BrushNet/examples/brushnet/app_brushnet.py", line 10, in
from diffusers import StableDiffusionBrushNetPipeline, BrushNetModel, UniPCMultistepScheduler
ImportError: cannot import name 'StableDiffusionBrushNetPipeline' from 'diffusers' (/media/iwoolf/BigDrive/anaconda3/envs/Brushnet/lib/python3.9/site-packages/diffusers/init.py)
Instructions, please?diffusers version: 0.27.2hello! Thanks for your excellent work. I want to ask when to release the checkpoint? Thanks a lot :-)
No response
When I visit the HuggingFace demo page here: https://huggingface.co/spaces/TencentARC/BrushNet
It fails to load with the saying Runtime error Memory limit exceeded (46Gi).
Simply visit: https://huggingface.co/spaces/TencentARC/BrushNet
===== Application Startup at 2024-04-02 14:01:10 =====
Installing correct gradio version...
Found existing installation: gradio 3.50.2
Uninstalling gradio-3.50.2:
Successfully uninstalled gradio-3.50.2
Collecting gradio==3.50.0
Downloading gradio-3.50.0-py3-none-any.whl (20.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 20.3/20.3 MB 100.7 MB/s eta 0:00:00
Requirement already satisfied: websockets<12.0,>=10.0 in /home/user/.pyenv/versions/3.9.19/lib/python3.9/site-packages (from gradio==3.50.0) (11.0.3)
Requirement already satisfied: packaging in /home/user/.pyenv/versions/3.9.19/lib/python3.9/site-packages (from gradio==3.50.0) (24.0)
Requirement already satisfied: pyyaml<7.0,>=5.0 in /home/user/.pyenv/versions/3.9.19/lib/python3.9/site-packages (from gradio==3.50.0) (6.0.1)
Requirement already satisfied: aiofiles<24.0,>=22.0 in /home/user/.pyenv/versions/3.9.19/lib/python3.9/site-packages (from gradio==3.50.0) (23.2.1)
Requirement already satisfied: jinja2<4.0 in /home/user/.pyenv/versions/3.9.19/lib/python3.9/site-packages (from gradio==3.50.0) (3.1.3)
Requirement already satisfied: pydantic!=1.8,!=1.8.1,!=2.0.0,!=2.0.1,<3.0.0,>=1.7.4 in /home/user/.pyenv/versions/3.9.19/lib/python3.9/site-packages (from gradio==3.50.0) (1.10.14)
Requirement already satisfied: ffmpy in /home/user/.pyenv/versions/3.9.19/lib/python3.9/site-packages (from gradio==3.50.0) (0.3.2)
Requirement already satisfied: orjson~=3.0 in /home/user/.pyenv/versions/3.9.19/lib/python3.9/site-packages (from gradio==3.50.0) (3.10.0)
Requirement already satisfied: matplotlib~=3.0 in /home/user/.pyenv/versions/3.9.19/lib/python3.9/site-packages (from gradio==3.50.0) (3.8.3)
Requirement already satisfied: python-multipart in /home/user/.pyenv/versions/3.9.19/lib/python3.9/site-packages (from gradio==3.50.0) (0.0.9)
Requirement already satisfied: gradio-client==0.6.1 in /home/user/.pyenv/versions/3.9.19/lib/python3.9/site-packages (from gradio==3.50.0) (0.6.1)
Requirement already satisfied: semantic-version~=2.0 in /home/user/.pyenv/versions/3.9.19/lib/python3.9/site-packages (from gradio==3.50.0) (2.10.0)
Requirement already satisfied: pandas<3.0,>=1.0 in /home/user/.pyenv/versions/3.9.19/lib/python3.9/site-packages (from gradio==3.50.0) (2.2.1)
Requirement already satisfied: huggingface-hub>=0.14.0 in /home/user/.pyenv/versions/3.9.19/lib/python3.9/site-packages (from gradio==3.50.0) (0.22.1)
Requirement already satisfied: httpx in /home/user/.pyenv/versions/3.9.19/lib/python3.9/site-packages (from gradio==3.50.0) (0.27.0)
Requirement already satisfied: altair<6.0,>=4.2.0 in /home/user/.pyenv/versions/3.9.19/lib/python3.9/site-packages (from gradio==3.50.0) (5.2.0)
Requirement already satisfied: pillow<11.0,>=8.0 in /home/user/.pyenv/versions/3.9.19/lib/python3.9/site-packages (from gradio==3.50.0) (9.5.0)
Requirement already satisfied: typing-extensions~=4.0 in /home/user/.pyenv/versions/3.9.19/lib/python3.9/site-packages (from gradio==3.50.0) (4.10.0)
Requirement already satisfied: uvicorn>=0.14.0 in /home/user/.pyenv/versions/3.9.19/lib/python3.9/site-packages (from gradio==3.50.0) (0.29.0)
Requirement already satisfied: pydub in /home/user/.pyenv/versions/3.9.19/lib/python3.9/site-packages (from gradio==3.50.0) (0.25.1)
Requirement already satisfied: numpy~=1.0 in /home/user/.pyenv/versions/3.9.19/lib/python3.9/site-packages (from gradio==3.50.0) (1.26.4)
Requirement already satisfied: importlib-resources<7.0,>=1.3 in /home/user/.pyenv/versions/3.9.19/lib/python3.9/site-packages (from gradio==3.50.0) (6.4.0)
Requirement already satisfied: fastapi in /home/user/.pyenv/versions/3.9.19/lib/python3.9/site-packages (from gradio==3.50.0) (0.110.0)
Requirement already satisfied: markupsafe~=2.0 in /home/user/.pyenv/versions/3.9.19/lib/python3.9/site-packages (from gradio==3.50.0) (2.1.5)
Requirement already satisfied: requests~=2.0 in /home/user/.pyenv/versions/3.9.19/lib/python3.9/site-packages (from gradio==3.50.0) (2.31.0)
Requirement already satisfied: fsspec in /home/user/.pyenv/versions/3.9.19/lib/python3.9/site-packages (from gradio-client==0.6.1->gradio==3.50.0) (2024.2.0)
Requirement already satisfied: toolz in /home/user/.pyenv/versions/3.9.19/lib/python3.9/site-packages (from altair<6.0,>=4.2.0->gradio==3.50.0) (0.12.1)
Requirement already satisfied: jsonschema>=3.0 in /home/user/.pyenv/versions/3.9.19/lib/python3.9/site-packages (from altair<6.0,>=4.2.0->gradio==3.50.0) (4.21.1)
Requirement already satisfied: filelock in /home/user/.pyenv/versions/3.9.19/lib/python3.9/site-packages (from huggingface-hub>=0.14.0->gradio==3.50.0) (3.13.3)
Requirement already satisfied: tqdm>=4.42.1 in /home/user/.pyenv/versions/3.9.19/lib/python3.9/site-packages (from huggingface-hub>=0.14.0->gradio==3.50.0) (4.66.2)
Requirement already satisfied: zipp>=3.1.0 in /home/user/.pyenv/versions/3.9.19/lib/python3.9/site-packages (from importlib-resources<7.0,>=1.3->gradio==3.50.0) (3.18.1)
Requirement already satisfied: kiwisolver>=1.3.1 in /home/user/.pyenv/versions/3.9.19/lib/python3.9/site-packages (from matplotlib~=3.0->gradio==3.50.0) (1.4.5)
Requirement already satisfied: contourpy>=1.0.1 in /home/user/.pyenv/versions/3.9.19/lib/python3.9/site-packages (from matplotlib~=3.0->gradio==3.50.0) (1.2.0)
Requirement already satisfied: python-dateutil>=2.7 in /home/user/.pyenv/versions/3.9.19/lib/python3.9/site-packages (from matplotlib~=3.0->gradio==3.50.0) (2.9.0.post0)
Requirement already satisfied: fonttools>=4.22.0 in /home/user/.pyenv/versions/3.9.19/lib/python3.9/site-packages (from matplotlib~=3.0->gradio==3.50.0) (4.50.0)
Requirement already satisfied: cycler>=0.10 in /home/user/.pyenv/versions/3.9.19/lib/python3.9/site-packages (from matplotlib~=3.0->gradio==3.50.0) (0.12.1)
Requirement already satisfied: pyparsing>=2.3.1 in /home/user/.pyenv/versions/3.9.19/lib/python3.9/site-packages (from matplotlib~=3.0->gradio==3.50.0) (3.1.2)
Requirement already satisfied: tzdata>=2022.7 in /home/user/.pyenv/versions/3.9.19/lib/python3.9/site-packages (from pandas<3.0,>=1.0->gradio==3.50.0) (2024.1)
Requirement already satisfied: pytz>=2020.1 in /home/user/.pyenv/versions/3.9.19/lib/python3.9/site-packages (from pandas<3.0,>=1.0->gradio==3.50.0) (2024.1)
Requirement already satisfied: idna<4,>=2.5 in /home/user/.pyenv/versions/3.9.19/lib/python3.9/site-packages (from requests~=2.0->gradio==3.50.0) (3.6)
Requirement already satisfied: charset-normalizer<4,>=2 in /home/user/.pyenv/versions/3.9.19/lib/python3.9/site-packages (from requests~=2.0->gradio==3.50.0) (3.3.2)
Requirement already satisfied: certifi>=2017.4.17 in /home/user/.pyenv/versions/3.9.19/lib/python3.9/site-packages (from requests~=2.0->gradio==3.50.0) (2024.2.2)
Requirement already satisfied: urllib3<3,>=1.21.1 in /home/user/.pyenv/versions/3.9.19/lib/python3.9/site-packages (from requests~=2.0->gradio==3.50.0) (2.2.1)
Requirement already satisfied: click>=7.0 in /home/user/.pyenv/versions/3.9.19/lib/python3.9/site-packages (from uvicorn>=0.14.0->gradio==3.50.0) (8.0.4)
Requirement already satisfied: h11>=0.8 in /home/user/.pyenv/versions/3.9.19/lib/python3.9/site-packages (from uvicorn>=0.14.0->gradio==3.50.0) (0.14.0)
Requirement already satisfied: starlette<0.37.0,>=0.36.3 in /home/user/.pyenv/versions/3.9.19/lib/python3.9/site-packages (from fastapi->gradio==3.50.0) (0.36.3)
Requirement already satisfied: sniffio in /home/user/.pyenv/versions/3.9.19/lib/python3.9/site-packages (from httpx->gradio==3.50.0) (1.3.1)
Requirement already satisfied: httpcore==1.* in /home/user/.pyenv/versions/3.9.19/lib/python3.9/site-packages (from httpx->gradio==3.50.0) (1.0.5)
Requirement already satisfied: anyio in /home/user/.pyenv/versions/3.9.19/lib/python3.9/site-packages (from httpx->gradio==3.50.0) (4.3.0)
Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /home/user/.pyenv/versions/3.9.19/lib/python3.9/site-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio==3.50.0) (2023.12.1)
Requirement already satisfied: referencing>=0.28.4 in /home/user/.pyenv/versions/3.9.19/lib/python3.9/site-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio==3.50.0) (0.34.0)
Requirement already satisfied: rpds-py>=0.7.1 in /home/user/.pyenv/versions/3.9.19/lib/python3.9/site-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio==3.50.0) (0.18.0)
Requirement already satisfied: attrs>=22.2.0 in /home/user/.pyenv/versions/3.9.19/lib/python3.9/site-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio==3.50.0) (23.2.0)
Requirement already satisfied: six>=1.5 in /home/user/.pyenv/versions/3.9.19/lib/python3.9/site-packages (from python-dateutil>=2.7->matplotlib~=3.0->gradio==3.50.0) (1.16.0)
Requirement already satisfied: exceptiongroup>=1.0.2 in /home/user/.pyenv/versions/3.9.19/lib/python3.9/site-packages (from anyio->httpx->gradio==3.50.0) (1.2.0)
Installing collected packages: gradio
Successfully installed gradio-3.50.0
[notice] A new release of pip available: 22.3.1 -> 24.0
[notice] To update, run: pip install --upgrade pip
Installing Finished!I ran this on the web
No response
thanks for your great work. Do you have a plan to adjust the brushnet to sd-webui-automatic1111111 ?
I downloaded the weights you guys provided, but there seems to be some issues with the weights
ValueError: Cannot load <class 'diffusers.models.brushnet.BrushNetModel'> from /model/BrushNet/unet because the following keys are missing:
brushnet_up_blocks.11.weight, brushnet_up_blocks.4.bias, brushnet_up_blocks.10.bias, brushnet_down_blocks.9.bias, brushnet_up_blocks.6.weight, brushnet_down_blocks.3.bias, brushnet_down_blocks.5.bias, brushnet_up_blocks.7.bias, brushnet_up_blocks.3.bias, brushnet_up_blocks.12.weight, brushnet_down_blocks.8.weight, brushnet_up_blocks.13.weight, brushnet_down_blocks.6.bias, brushnet_up_blocks.9.bias, brushnet_down_blocks.11.bias, brushnet_down_blocks.5.weight, brushnet_down_blocks.4.bias, brushnet_mid_block.bias, brushnet_down_blocks.2.weight, brushnet_down_blocks.3.weight, brushnet_down_blocks.1.bias, brushnet_down_blocks.7.bias, brushnet_down_blocks.0.bias, brushnet_up_blocks.12.bias, brushnet_up_blocks.7.weight, brushnet_up_blocks.8.bias, brushnet_up_blocks.2.bias, brushnet_up_blocks.14.weight, brushnet_up_blocks.0.weight, brushnet_up_blocks.11.bias, brushnet_up_blocks.3.weight, brushnet_up_blocks.1.bias, brushnet_up_blocks.6.bias, brushnet_up_blocks.4.weight, brushnet_down_blocks.4.weight, brushnet_up_blocks.0.bias, brushnet_down_blocks.0.weight, brushnet_up_blocks.2.weight, brushnet_up_blocks.10.weight, brushnet_mid_block.weight, brushnet_up_blocks.13.bias, brushnet_up_blocks.1.weight, brushnet_up_blocks.14.bias, brushnet_down_blocks.7.weight, brushnet_up_blocks.8.weight, brushnet_up_blocks.9.weight, brushnet_down_blocks.1.weight, brushnet_down_blocks.10.weight, conv_in_condition.bias, brushnet_down_blocks.8.bias, conv_in_condition.weight, brushnet_up_blocks.5.bias, brushnet_down_blocks.2.bias, brushnet_down_blocks.6.weight, brushnet_down_blocks.11.weight, brushnet_down_blocks.10.bias, brushnet_down_blocks.9.weight, brushnet_up_blocks.5.weight.
Please make sure to pass low_cpu_mem_usage=False and device_map=None if you want to randomly initialize those weights or else make sure your checkpoint file is correct.
realisticVisionV60B1_v51VAE
No response
diffusers version: 0.27.0No response
When I tried the huggingface demo, I found that the shape of the object was always very close to the shape of the mask, and it was very strange, why
No response
No response
There is an option like strength, what I want is for the masked part to take the color of the original image.
Can this network be trained with SDXL pretrained models?
Hi, I am trying to inpaint a person in specific pose. Is it possible to use controlnet with this repo?
No response
Your work is truly good job!
Btw, when will the SDXL weights released? Hope to try it!
can I using brushnet with ip adapter ?
这是一个很棒的框架，可以应用在很多方面！
不过在论文中的实现细节上，我看到是用来8块V100GPU，请问使用的GPU显存是多大呢？如果只有24G显存是否能够训练？
Dear developer,
Thank you for your wonderful work.
The paper seems to only compare 4-channel controlnet-inpainting.
Have you ever compared the 9-channel controlnet-inpainting model?
What I mean is that the input of UNet is 9-channel.
Best wishes.
It is recommended to upload the ckpt to Hugging Face, as there are always download failures with Google Drive.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.