exx8 / differential-diffusion Goto Github PK

Amazing work! I've tried using several SDXL text2img models for img2img or inpaint tasks and have had good results. However, when I attempted to use the SDXL turbo model for these tasks, it failed. The generated content had nothing to do with my original image and it seemed like the model was still trying to perform the text2img task. I'm curious about what's going on. Do you have any suggestions?

Thank you for your work and open-source code! I would like to ask a question, Can Differential Diffusion help improve the inpainting model? e.g runwayml/stable-diffusion-inpainting

hello，thanks for your share，i have a question about the detail， what is the difference between differential-diffusion and diffusers inpainting pipline？just the mask type， change from int to float？

It becomes very convenient if you can use it，THX~

Is it possible to clone the project and use it alone?

First, thanks for your team's hard work. I am wonder that will your team support stable cascade in future？ or others better diffuse model, such as playground v2.5 and stable diffusion 3.

Optimize differential diffusion for most consumer GPUs
My GPU: NVIDIA GeForce GTX 1060 6GB
slow ass steps: 15/40 [14:33<24:19, 58.37s/it]

It would be great to include out-of-the-box support for SDXL-Lightning

Demos of SDXL-Lightning available here:

• https://fastsdxl.ai/
• https://huggingface.co/spaces/ByteDance/SDXL-Lightning

I used differential diffusion to do some experiment, and I turned out wonderful! But I am wondering if differential diffusion is able to fuse two images. I have a background image containing a object and background, a foreground image containing an object with transparent background and a mask of foreground. How can I take use of diffienertial diffusion to do this task? I noticed there are two parameters named original_image and image, I used them to do some testing and it didn't work out. Did I do it in a wrong way? Whether is diffierential diffusion able to accomplish this task?

couldn't find any valid examples in these two links, nor did see clear instructions for using comfyUI for your project
comfyanonymous/ComfyUI#2876

https://github.com/vladmandic/automatic

can use this with sd 1.5?

From reviewing the paper, we wonder if the mask may have been reversed by mistake.

In the denoising loop, $z_{mix}$ is combining the result of the previous denoising step $z_{t+1}$, with $z_t^ \prime$ which has just been obtained from $z_{init}$ with the current noise level. It would seem that the "injected" part is $z_t^ \prime$, because it is initialized from $z_{init}$ with current noise level, thus skipping all the previous denoising steps and being "injected" later.

This seems consistent with the fact that the mask is getting bigger and bigger ($t$ is becoming smaller and smaller), and that the region that is injected becomes correspondingly smaller (including only the parts with smallest amount of change allowed).

However in the paper (Figure 3) it states the opposite "Top: $z_{𝑡+1}$ ⊙ 𝑚𝑎𝑠𝑘, which corresponds to the injected fragments at each time-step. bottom: 𝑧'_𝑡 ⊙ (1 − 𝑚𝑎𝑠𝑘) which corresponds to the fragments which are not injected at this time-step."

Can you confirm or dismiss whether there is an error in the paper? Or are we misunderstanding by claiming that the injected part is $z_t^ \prime$ ? If we are mistaken, do you know what we may be missing?

Thanks so much in advance for your help and clarification.

is this working with1.5 inpainting or sdxl inpainting?

ERROR: Could not find a version that satisfies the requirement nvidia-cudnn-cu12==8.9.2.26 (from versions: 0.0.1.dev5, 8.9.4.25, 8.9.5.29, 8.9.6.50, 8.9.7.29)
ERROR: No matching distribution found for nvidia-cudnn-cu12==8.9.2.26

This has a lot of potential if used together with ip adapter and controlnet
However, the current pipeline is very old and does not support the new features.
can you please update and add pipelines for ip adapter and controlnet

When running the script for SDXL, I get the following error:
The config attributes {'force_upcast': True} were passed to AutoencoderKL, but are not expected and will be ignored. Please verify your config.json configuration file.
Traceback (most recent call last):
File "G:\ComfyUI_windows_portable\differential-diffusion\SDXL\run.py", line 49, in
edited_images = base(prompt=prompt, original_image=image, image=image, strength=1, guidance_scale=17.5,
File "G:\ComfyUI_windows_portable\differential-diffusion\venv\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "G:\ComfyUI_windows_portable\differential-diffusion\SDXL\diff_pipe.py", line 853, in call
device = self._execution_device
File "G:\ComfyUI_windows_portable\differential-diffusion\venv\lib\site-packages\diffusers\configuration_utils.py", line 137, in getattr
raise AttributeError(f"'{type(self).name}' object has no attribute '{name}'")
AttributeError: 'StableDiffusionXLDiffImg2ImgPipeline' object has no attribute '_execution_device'

I only use sdxl base model to run sdxl/run.py, but got the result below:

anyone who can help will be appreciated.

added implementation to sdnext via commit vladmandic/automatic@0239435

do you have plans to add sd15 pipeline?

Hi, sorry to disturb..
Do you plan to make it available for Forge WebUI?