lkwq007 / stablediffusion-infinity

Outpainting with Stable Diffusion on an infinite canvas

stablediffusion-infinity

Outpainting with Stable Diffusion on an infinite canvas.

Girl.with.a.Pearl.Earring.mp4

Status

Powered by Stable Diffusion inpainting model, this project now works well. However, the quality of results is still not guaranteed. You may need to do prompt engineering, change the size of the selection, reduce the size of the outpainting region to get better outpainting results.

The project now becomes a web app based on PyScript and Gradio. For Jupyter Notebook version, please check out the ipycanvas branch.

Pull requests are welcome for better UI control, ideas to achieve better results, or any other improvements.

Update: the project add photometric correction to suppress seams, to use this feature, you need to install fpie: pip install fpie (Linux/MacOS only)

Docs

Get Started

• Setup for Windows: setup_guide
• Setup for Linux: setup_guide
• Setup for MacOS: setup_guide
• Running with Docker on Windows or Linux with NVIDIA GPU: run_with_docker
• Usages: usage

FAQs

• The result is a black square:
  • False positive rate of safety checker is relatively high, you may disable the safety_checker
  • Some GPUs might not work with fp16: python app.py --fp32 --lowvram
• What is the init_mode
  • init_mode indicates how to fill the empty/masked region, usually patch_match is better than others
• Why not use postMessage for iframe interaction
  • The iframe and the gradio are in the same origin. For postMessage version, check out gradio-space version

Known issues

• The canvas is implemented with NumPy + PyScript (the project was originally implemented with ipycanvas inside a jupyter notebook), which is relatively inefficient compared with pure frontend solutions.
• By design, the canvas is infinite. However, the canvas size is finite in practice. Your RAM and browser limit the canvas size. The canvas might crash or behave strangely when zoomed out by a certain scale.
• The canvas requires internet: You can deploy and serve PyScript, Pyodide, and other JS/CSS assets with a local HTTP server and modify index.html accordingly.
• Photometric correction might not work (taichi does not support the multithreading environment). A dirty hack (quite unreliable) is implemented to move related computation inside a subprocess.
• Stable Diffusion inpainting model is much slower when selection size is larger than 512x512

Credit

The code of perlin2d.py is from https://stackoverflow.com/questions/42147776/producing-2d-perlin-noise-with-numpy/42154921#42154921 and is not included in the scope of LICENSE used in this repo.

The submodule glid_3_xl_stable is based on https://github.com/Jack000/glid-3-xl-stable

The submodule PyPatchMatch is based on https://github.com/vacancy/PyPatchMatch

The code of postprocess.py and process.py is modified based on https://github.com/Trinkle23897/Fast-Poisson-Image-Editing

The code of convert_checkpoint.py is modified based on https://github.com/huggingface/diffusers/blob/main/scripts/convert_original_stable_diffusion_to_diffusers.py

The submodule sd_grpcserver and handleImageAdjustment() in utils.py are based on https://github.com/hafriedlander/stable-diffusion-grpcserver and https://github.com/parlance-zz/g-diffuser-bot

w2ui.min.js and w2ui.min.css is from https://github.com/vitmalina/w2ui. fabric.min.js is a custom build of https://github.com/fabricjs/fabric.js

interrogate.py is based on https://github.com/pharmapsychotic/clip-interrogator v1, the submodule blip_model is based on https://github.com/salesforce/BLIP

stablediffusion-infinity's Issues

hugging face token problems

when pasting my token in the huggingface token field I get the following message:

"There was a specific connection error when trying to load CompVis/stable-diffusion-v1-4:
<class 'requests.exceptions.HTTPError'> (Request ID: )"

alternitavely I have seen a fix requiring the insertion of "use_auth_token=''" into the ipnyb file,
could you tell me where and how exactly to insert that there? thanks in advance

Legacy-install-failure installing fpie package

While setting up the environment file I had some problems with the fpie package. I already updated the wheel and installed the cmake package manually.

• OS: Ubuntu debian 18.04.6 LTS
• CUDA 11.4
• Nvidia RTX A5000 and Nvidia GeForce RTX 3090
• Browser firefox

Different results on different platform (resizing)

on google colab the image that results from outpainting looks totally normal, but when running the local Windows notebook everything gets messed up.

The only difference between the two environments is google Colab has "Resize SD Input to 512x512" while my windows machine does not. Resizing to 512 causes out of memory error:

RuntimeError: CUDA out of memory. Tried to allocate 512.00 MiB (GPU 0; 4.00 GiB total capacity; 2.65 GiB already allocated; 0 bytes free; 3.19 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

This is what it looks like on my local machine. Steps = 50, guidance = 7.5, strength = 0.75, edge pad, resize disabled:

Google Colab, same settings as above but with resize enabled:

Is there a way to either use less memory when resizing or get a better outpaint by changing the rest of the settings to accommodate? I am willing to wait a long time to get a better result, I am in no rush.

Issue with colors

Selected area of the image has a magenta tint after outpainting.

Colab is missing libraries

Utils and canvas are missing when I try to run the notebook. Tried to in stall with !pip install utils canvas
it still fails, probably the wron canvas library?

ImportError                               Traceback (most recent call last)
[<ipython-input-4-a4748a0d1586>](https://localhost:8080/#) in <module>
      8 
      9 from utils import *
---> 10 from canvas import InfCanvas
     11 
     12 import numpy as np

ImportError: cannot import name 'InfCanvas' from 'canvas' (/usr/local/lib/python3.7/dist-packages/canvas/__init__.py)

Error: No module named 'skimage'

(sd-inf) C:\Users\ZeroCool22\Desktop\stablediffusion-infinity\stablediffusion-infinity>python app.py
Traceback (most recent call last):
  File "C:\Users\ZeroCool22\Desktop\stablediffusion-infinity\stablediffusion-infinity\app.py", line 14, in <module>
    import skimage
ModuleNotFoundError: No module named 'skimage'

[Feature Request] Local GPU with Gradio

Is your feature request related to a problem? Please describe.
Currently we need to use Huggingface, but I would love to use my own GPU

Describe the solution you'd like
A way of locally using Outpainting.

Describe alternatives you've considered
Not applicable I believe

Additional context
Being held by Huggingface is not that great

Thanks for the lib! It's amazing.

cudatoolkit missing

Environment method fails with

Solving environment: failed
ResolvePackageNotFound:
  - nettle==3.7.3=hbbd107a_1
  - {long list of unfound packages}

The manual install fails because cudatoolkit is missing:

PackagesNotFoundError: The following packages are not available from current channels:
  - cudatoolkit=11.3

MacBook Pro M1 14" - MacOS 12.5.1

How run it locally with our own GPU?

git clone --recurse-submodules https://github.com/lkwq007/stablediffusion-infinity
cd stablediffusion-infinity

run jupyter notebook or jupyter lab here

Don't know what to do in the last step to run it Locally:

run jupyter notebook or jupyter lab here

[Bug] Error when using Photometric Correction on Windows install

Everything was installed correctly there were no errors. Initially, it was working great then I toggled Photometric Correction and the prompt box was replaced by Error in red and un selectable. The only way to resume was to uncheck Photometric Correction and re-run the outpaint. Command line reported the following:

Traceback (most recent call last):
File "C:\Users\mike4.conda\envs\sd-inf\lib\site-packages\gradio\routes.py", line 275, in run_predict
output = await app.blocks.process_api(
File "C:\Users\mike4.conda\envs\sd-inf\lib\site-packages\gradio\blocks.py", line 760, in process_api
result = await self.call_function(fn_index, inputs, iterator)
File "C:\Users\mike4.conda\envs\sd-inf\lib\site-packages\gradio\blocks.py", line 671, in call_function
prediction = await anyio.to_thread.run_sync(
File "C:\Users\mike.conda\envs\sd-inf\lib\site-packages\anyio\to_thread.py", line 31, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "C:\Users\mike4.conda\envs\sd-inf\lib\site-packages\anyio_backends_asyncio.py", line 937, in run_sync_in_worker_thread
return await future
File "C:\Users\mike4.conda\envs\sd-inf\lib\site-packages\anyio_backends_asyncio.py", line 867, in run
result = context.run(func, *args)
File "C:\Users\mike4\stablediffusion-infinity\app.py", line 263, in run_outpaint
image = Image.fromarray(image)
File "C:\Users\mike4.conda\envs\sd-inf\lib\site-packages\PIL\Image.py", line 2974, in fromarray
return frombuffer(mode, size, obj, "raw", rawmode, 0, 1)
File "C:\Users\mike4.conda\envs\sd-inf\lib\site-packages\PIL\Image.py", line 2901, in frombuffer
return frombytes(mode, size, data, decoder_name, args)
File "C:\Users\mike4.conda\envs\sd-inf\lib\site-packages\PIL\Image.py", line 2843, in frombytes
im.frombytes(data, decoder_name, args)
File "C:\Users\mike4.conda\envs\sd-inf\lib\site-packages\PIL\Image.py", line 795, in frombytes
s = d.decode(data)
TypeError: a bytes-like object is required, not 'Image'

I'm using Windows 11 in Chrome browser, Nvidia 2070 super.

I'm pretty new to Github though I haven't had any serious problems yet with installations.

No module named 'fpie'

I've been testing the previous version of this for a few days now. I just did a git pull and an now getting a new error.

(sd-inf) D:\stablediffusion-infinity>python app.py
patch_match compiling failed
No module named 'fpie'
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.

I can't seem to get fpie to install with pip install fpie or with the -I option.

And also, I can't find any place to add the share=True either in the app.py or on the command line.

Any update on getting patch_match to work on Windows?

Thanks,
Tom

Trying to build the docker container but debian archives are not responsive. Possible to route / change Sources ?

[2/6] RUN apt-get update && apt install -y fonts-dejavu-core build-essential libopencv-dev && apt-get clean: #0 0.317 + apt-get update #0 0.414 Get:1 http://security.debian.org/debian-security bullseye-security InRelease [48.4 kB] #0 0.414 Get:2 http://deb.debian.org/debian bullseye InRelease [116 kB] #0 30.44 Err:1 http://security.debian.org/debian-security bullseye-security InRelease #0 30.44 Connection timed out [IP: 151.101.194.132 80] #0 30.44 Err:2 http://deb.debian.org/debian bullseye InRelease #0 30.44 Connection timed out [IP: 199.232.174.132 80] #0 30.51 Get:3 http://deb.debian.org/debian bullseye-updates InRelease [44.1 kB] #0 60.53 Err:3 http://deb.debian.org/debian bullseye-updates InRelease #0 60.53 Connection timed out [IP: 199.232.174.132 80] #0 60.54 Reading package lists... #0 60.55 W: Failed to fetch http://deb.debian.org/debian/dists/bullseye/InRelease Connection timed out [IP: 199.232.174.132 80] #0 60.55 W: Failed to fetch http://security.debian.org/debian-security/dists/bullseye-security/InRelease Connection timed out [IP: 151.101.194.132 80] #0 60.55 W: Failed to fetch http://deb.debian.org/debian/dists/bullseye-updates/InRelease Connection timed out [IP: 199.232.174.132 80] #0 60.55 W: Some index files failed to download. They have been ignored, or old ones used instead. #0 60.55 + apt install -y fonts-dejavu-core build-essential libopencv-dev #0 60.56 #0 60.56 WARNING: apt does not have a stable CLI interface. Use with caution in scripts. #0 60.56 #0 60.56 Reading package lists... #0 60.57 Building dependency tree... #0 60.57 Reading state information... #0 60.58 E: Unable to locate package fonts-dejavu-core #0 60.58 E: Unable to locate package build-essential #0 60.58 E: Unable to locate package libopencv-dev

This is what i found duckducking:

https://deb.debian.org/

If you hit the server behind deb.debian.org directly, either because you use an older apt or because you use a HTTP proxy that does not support SRV records, your requests will get HTTP redirected to one of the CDN instances. If you want to avoid the redirects, you can pick one instance directly. For instance, this also works in your sources.list:

deb http://cdn-fastly.deb.debian.org/debian stable main
deb http://cdn-fastly.deb.debian.org/debian-security stable/updates main

how to disable nsfw filter

I want to draw things that are not nsfw but it detects me as nsfw and draws me a black square.

File not found error on windows after paste huggingface login token

I've checked this guide #12 but the error is different

        To login, `huggingface_hub` now requires a token generated from https://huggingface.co/settings/tokens .

Token:
Traceback (most recent call last):
  File "B:\tools\anaconda3\envs\sd-inf\Scripts\huggingface-cli-script.py", line 9, in <module>
    sys.exit(main())
  File "B:\tools\anaconda3\envs\sd-inf\lib\site-packages\huggingface_hub\commands\huggingface_cli.py", line 41, in main
    service.run()
  File "B:\tools\anaconda3\envs\sd-inf\lib\site-packages\huggingface_hub\commands\user.py", line 176, in run
    _login(self._api, token=token)
  File "B:\tools\anaconda3\envs\sd-inf\lib\site-packages\huggingface_hub\commands\user.py", line 344, in _login
    hf_api.set_access_token(token)
  File "B:\tools\anaconda3\envs\sd-inf\lib\site-packages\huggingface_hub\hf_api.py", line 705, in set_access_token
    write_to_credential_store(USERNAME_PLACEHOLDER, access_token)
  File "B:\tools\anaconda3\envs\sd-inf\lib\site-packages\huggingface_hub\hf_api.py", line 528, in write_to_credential_store
    with subprocess.Popen(
  File "B:\tools\anaconda3\envs\sd-inf\lib\subprocess.py", line 966, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "B:\tools\anaconda3\envs\sd-inf\lib\subprocess.py", line 1435, in _execute_child
    hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
FileNotFoundError: [WinError 2] The system cannot find the file specified

[Bug] doesn't work because of the addition of negative prompt

it worked until yesterday but doesn't work anymore because of the Negative Prompt
I haven't seen Negative Prompt thing until yesterday but it poped up and it returns error about it

 File "/usr/local/lib/python3.7/dist-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
TypeError: __call__() got an unexpected keyword argument 'negative_prompt'

fatal: destination path 'stablediffusion-infinity' already exists and is not an empty directory.

I tried to run the colab. In the first time I got "Session crashed" error.

After restarting I got..

fatal: destination path 'stablediffusion-infinity' already exists and is not an empty directory.

How to fix this? Please help

patch_match compiling failed

I already execute both commands (running locally with CONDA PROMPT):

**pip install opencv-python

conda install -c conda-forge opencv**

Still giving that error.

Suggestion

Thanks for the great work and sharing it on Colab.

I hit PIL Resampling not found error.

import PIL.Image
if not hasattr(PIL.Image, 'Resampling'):  # Pillow<9.0
  PIL.Image.Resampling = PIL.Image
# Now PIL.Image.Resampling.BICUBIC is always recognized.

Adding this helped before the last line. Just leaving it here in case if anyone hits the same error.

login pass ?

using colab :

any idea ?

Where to put previously downloaded Stable Diffusion model?

Where should someone put a previously downloaded Stable Diffusion model, and what should the model file be named?

Looking at def get_model() it's not immediately obvious where it would get put.

If any code modifications are needed to bypass the need for accessing HuggingFace (where the model has previously been downloaded from HuggingFace), what would be the recommendation?

Documented issues from troubleshooting on Windows

I helped 4 separate people in the last 5 hours set this software up on Windows, here's what issues I've run into and the solutions. I apologize for this being condensed into a single issue, but I'm low on time:

Problem 1: Tokens
Many people have been having issues using huggingface cli login to register their token for usage with this software. The workaround to this is to use use_auth_token='tokenvalue' (where tokenvalue is the token) rather than use_auth_token=True. This shouldn't be the default, but it should be documented as a potential workaround.

Problem 2: AttributeError: module 'PIL.Image' has no attribute 'Resampling'
This issue only occurred for one person. No amount of re-running existing cells fixes it, and all dependencies are installed.

Problem 3: OSError: Windows requires Developer Mode to be activated, or to run Python as an administrator, in order to create symlinks.
This doesn't seem to be an issue with this software as much as it is with permissions in general, but it's still an issue that's odd. It didn't occur for 3 out of the 4 people I was helping. The person who ran into this issue had an administrator account, and was running it under the same conditions as the other users. Developer mode did fix it.

Problem 4: OpenCV
Installing opencv with conda seems to prevent errors with patch_match. This is the command I've been recommending and it has successfully prevented it: conda install -c conda-forge opencv.

Concerns:
No such comm target registered: jupyter.widget.control
Full error:

[IPKernelApp] ERROR | No such comm target registered: jupyter.widget.control
[IPKernelApp] WARNING | No such comm: a5f38cb6-3d6d-46b5-8493-fea3b4bc45fb

There's no clear reason as to why this appears, but it shows up in the log. Functionality does not appear to be inhibited.

Missing LICENSE

I see you have no LICENSE file for this project. The default is copyright.

I would suggest releasing the code under the GPL-3.0-or-later or AGPL-3.0-or-later license so that others are encouraged to contribute changes back to your project.

The result is a black square

I wanna draw something,but when I run the app,the result is a black square。
And no suggestions In Terminal,so I didn't know how to solve the problem.

[Bug, Crash]

XXX

Setup failed - Invalid token!

Using either a read or write token, during setup it always states "Invalid token!"?

(suggestion) It is possible to compile patchmatch on windows without conda

you can build PatchMatch into a DLL on windows w/ visual studio 2022 and modify the patch_match.py to allow it to work on any windows x64 machine

Windows version, ModuleNotFoundError: No module named 'cv2'

After following the instructions and opening the project in Jupyter Lab, I attempt to open the ipynb file for Windows where the second code block is highlighted red with a module not found error. There are two tracebacks; the first is for "from utils import *" and the second is "import cv2". The error at the bottom of the code block states, "No module named 'cv2'.

I'm more than happy to supply whatever information or data you require. Thank you in advance.

Docker-Missing entrypoint.sh

Hello.

I am brand new to Docker, and this is my first attempt at using it.

I am on Windows 10, admin command prompt.

I have run the commands as shown under the install directions. The first two command work, however, I am getting the following error when I try to 'compose up':

C:\stable-diffusion\stablediffusion-infinity>docker-compose up
[+] Running 4/4
 - Network stablediffusion-infinity_default          Created                                                       0.1s
 - Volume "stablediffusion-infinity_cond_env"        Created                                                       0.0s
 - Volume "stablediffusion-infinity_user_home"       Created                                                       0.0s
 - Container stablediffusion-infinity-sd-infinity-1  Created                                                       0.2s
Attaching to stablediffusion-infinity-sd-infinity-1
stablediffusion-infinity-sd-infinity-1  | exec /home/user/app/docker/entrypoint.sh: no such file or directory
stablediffusion-infinity-sd-infinity-1 exited with code 1

What information can I give to help troubleshoot? Since I am on Win10, I am having issues locating the directories in the WSL system. To be honest, I am not even sure I am using the right words to describe the above sentence.

Any help is welcome.

ImportError: cannot import name 'get_hf_file_metadata' from 'huggingface_hub'

(sd-inf) zerocool@DESKTOP-IFR8E96:~/github/stablediffusion-infinity$ python app.py
Traceback (most recent call last):
  File "/home/zerocool/github/stablediffusion-infinity/app.py", line 9, in <module>
    import diffusers
  File "/home/zerocool/anaconda3/envs/sd-inf/lib/python3.10/site-packages/diffusers/__init__.py", line 32, in <module>
    from .pipelines import DDIMPipeline, DDPMPipeline, KarrasVePipeline, LDMPipeline, PNDMPipeline, ScoreSdeVePipeline
  File "/home/zerocool/anaconda3/envs/sd-inf/lib/python3.10/site-packages/diffusers/pipelines/__init__.py", line 15, in <module>
    from .latent_diffusion import LDMTextToImagePipeline
  File "/home/zerocool/anaconda3/envs/sd-inf/lib/python3.10/site-packages/diffusers/pipelines/latent_diffusion/__init__.py", line 6, in <module>
    from .pipeline_latent_diffusion import LDMBertModel, LDMTextToImagePipeline
  File "/home/zerocool/anaconda3/envs/sd-inf/lib/python3.10/site-packages/diffusers/pipelines/latent_diffusion/pipeline_latent_diffusion.py", line 8, in <module>
    from transformers.activations import ACT2FN
  File "/home/zerocool/anaconda3/envs/sd-inf/lib/python3.10/site-packages/transformers/__init__.py", line 30, in <module>
    from . import dependency_versions_check
  File "/home/zerocool/anaconda3/envs/sd-inf/lib/python3.10/site-packages/transformers/dependency_versions_check.py", line 17, in <module>
    from .utils.versions import require_version, require_version_core
  File "/home/zerocool/anaconda3/envs/sd-inf/lib/python3.10/site-packages/transformers/utils/__init__.py", line 48, in <module>
    from .hub import (
  File "/home/zerocool/anaconda3/envs/sd-inf/lib/python3.10/site-packages/transformers/utils/hub.py", line 32, in <module>
    from huggingface_hub import (
ImportError: cannot import name 'get_hf_file_metadata' from 'huggingface_hub' (/home/zerocool/anaconda3/envs/sd-inf/lib/python3.10/site-packages/huggingface_hub/__init__.py)
(sd-inf) zerocool@DESKTOP-IFR8E96:~/github/stablediffusion-infinity$

Canvas not showing up

Using google notebook. Everything works fine and can export images, but I don't have a canvas and can't actually see them. Can't move around the box and can't see what images I upload. It worked the first time I tried to use it although I think I didn't run every piece of code when I did that so the prompts wouldn't work.

Package opencv was not found in the pkg-config search path.

When running:

jupyter lab

Then going to:

http://localhost:8888/lab

Then finally:

stablediffusion_infinity.ipynb

The following happens on the "import libs" step:

Package opencv was not found in the pkg-config search path.
Perhaps you should add the directory containing `opencv.pc'
to the PKG_CONFIG_PATH environment variable
Package 'opencv', required by 'virtual:world', not found
Package opencv was not found in the pkg-config search path.
Perhaps you should add the directory containing `opencv.pc'
to the PKG_CONFIG_PATH environment variable
Package 'opencv', required by 'virtual:world', not found
Package opencv was not found in the pkg-config search path.
Perhaps you should add the directory containing `opencv.pc'
to the PKG_CONFIG_PATH environment variable
Package 'opencv', required by 'virtual:world', not found

It is a fresh install and might be reproducible under a similar (Arch Linux) environment.

inpaint is not defined

Hello, attempting to use this software in a local environment. Installed per the readme without issues. Launched into the JupyterLab Notebook. Ran all the cells until the end when the GUI appeared. Uploaded a picture, moved the frame outside the image and clicked "Outpainting" and the following error was thrown.

NameError Traceback (most recent call last)
Cell In [6], line 22, in run_outpaint(btn)
20 # mask_image=mask_image.filter(ImageFilter.GaussianBlur(radius = 8))
21 with autocast("cuda"):
---> 22 images = inpaint(
23 prompt=base.text_input.value,
24 init_image=init_image.resize(
25 (process_size, process_size), resample=Image.Resampling.LANCZOS
26 ),
27 mask_image=mask_image.resize((process_size, process_size)),
28 strength=strength_slider.value,
29 num_inference_steps=step_input.value,
30 guidance_scale=guidance_input.value,
31 )["sample"]
32 else:
33 with autocast("cuda"):

NameError: name 'inpaint' is not defined

Request: Keyboard shortcuts

Hi! What a fine tool you have created!

What about the following idea? In order to use the tool more efficient, it could make sense to introduce keyboard shortcuts to access the most important functionalities.

Keep up the great work!

-Tristan

Error response from daemon: could not select device driver "nvidia" with capabilities: [[gpu]]

Hello,

I have an nvidia RTX 3060 12gb card with proprietary nvidia drivers installed, but I am getting the following issue:

I first run:

docker-compose build
docker-compose up

Then I am getting the following output:

Attaching to stablediffusion-infinity-sd-infinity-1
Error response from daemon: could not select device driver "nvidia" with capabilities: [[gpu]]

Functbl not defined

I've gotten to the point where I can generate the initial image but all inpainting is blocked by functbl and I can't even guess at what it's supposed to be referencing.

NameError                                 Traceback (most recent call last)
[<ipython-input-16-b4e8caceb637>](https://localhost:8080/#) in run_outpaint(btn)
     18         process_size = 512 if resize_check.value else base.selection_size
     19         if mask.sum() > 0:
---> 20             img, mask = functbl[base.fill_button.value](img, mask)
     21             init_image = Image.fromarray(img)
     22             mask = 255 - mask

NameError: name 'functbl' is not defined

Smaller environment.yml?

Looks like the environment.yml might've been generated by freezing a global system environment, since it includes a ton of things including stuff like alsa-lib and a bunch of X11 stuff.

Could we get only the actual deps of the script? I tried to go through it manually but there's too many to try it one by one.

Tooltips for buttons?

Especially icon buttons. I would love to start with a vague idea of the difference between "curly arrow" and "bendy arrow"!

(side note - are there docs I've overlooked? tooltips would be better but "guessing" comes a distant third in my onboarding preferences ;-) )

Problem with installing libffi& from gradio module

Running the app.py script gave the error ImportError: libffi.so.7: cannot open shared object file: No such file or directory. This comes from the gradio package. I already rebuild and reinstalled the libffi and gradio package but it comes down to the same error.

• OS: Ubuntu debian 18.04.6 LTS
• CUDA 11.4
• Nvidia RTX A5000 and Nvidia GeForce RTX 3090
• Browser firefox

Mac version, no package 'opencv' found

the error I'm getting when running code on MacOS12.6:
Compiling and loading c extensions from "/Users/x/stablediffusion-infinity/PyPatchMatch".
rm -rf build/obj libpatchmatch.so
Package opencv was not found in the pkg-config search path.
Perhaps you should add the directory containing opencv.pc' to the PKG_CONFIG_PATH environment variable No package 'opencv' found build/obj build/obj/csrc [dep] csrc//masked_image.cpp ... In file included from csrc//masked_image.cpp:1: csrc/masked_image.h:3:10: fatal error: 'opencv2/core.hpp' file not found #include <opencv2/core.hpp> ^~~~~~~~~~~~~~~~~~ 1 error generated. Package opencv was not found in the pkg-config search path. Perhaps you should add the directory containing opencv.pc'
to the PKG_CONFIG_PATH environment variable
No package 'opencv' found
[dep] csrc//inpaint.cpp ...
csrc//inpaint.cpp:3:10: fatal error: 'opencv2/imgcodecs.hpp' file not found
#include <opencv2/imgcodecs.hpp>
^~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
Package opencv was not found in the pkg-config search path.
Perhaps you should add the directory containing opencv.pc' to the PKG_CONFIG_PATH environment variable No package 'opencv' found [dep] csrc//nnf.cpp ... In file included from csrc//nnf.cpp:5: csrc/masked_image.h:3:10: fatal error: 'opencv2/core.hpp' file not found #include <opencv2/core.hpp> ^~~~~~~~~~~~~~~~~~ 1 error generated. Package opencv was not found in the pkg-config search path. Perhaps you should add the directory containing opencv.pc'
to the PKG_CONFIG_PATH environment variable
No package 'opencv' found
[dep] csrc//pyinterface.cpp ...
In file included from csrc//pyinterface.cpp:1:
csrc/pyinterface.h:1:10: fatal error: 'opencv2/core.hpp' file not found
#include <opencv2/core.hpp>
^~~~~~~~~~~~~~~~~~
1 error generated.
Package opencv was not found in the pkg-config search path.
Perhaps you should add the directory containing `opencv.pc'
to the PKG_CONFIG_PATH environment variable
No package 'opencv' found
[CC] csrc//pyinterface.cpp ...
In file included from csrc//pyinterface.cpp:1:
csrc/pyinterface.h:1:10: fatal error: 'opencv2/core.hpp' file not found
#include <opencv2/core.hpp>
^~~~~~~~~~~~~~~~~~
1 error generated.
make: *** [build/obj/csrc//pyinterface.o] Error 1
patch_match compiling failed

No an issue, can i use this with my own model?

In other colabs I only have to load my model but I'm not sure how to do it here

ModuleNotFoundError: No module named 'transformers.activations'

Hi,

I created a conda environment and installed all the requirements successfully. But, I get this error message:

    from transformers.activations import ACT2FN
ModuleNotFoundError: No module named 'transformers.activations'

I use macOS Monterey with m1 processor.

Bad result

I am running a local notebook and am getting very bad false results.
Example
Example

patch_match compiling failed

I have installed opencv via pip install opencv-python but seem to be encountering a patch_match issue. Not sure whats causing it. Cheers.

(sd-inf) C:\Users\wstob\stablediffusion-infinity>python app.py
patch_match compiling failed
Traceback (most recent call last):
File "C:\Users\wstob\stablediffusion-infinity\app.py", line 221, in
frame = gr.HTML(test(2), visible=False)
File "C:\Users\wstob\stablediffusion-infinity\app.py", line 31, in test
x = load_html()
File "C:\Users\wstob\stablediffusion-infinity\app.py", line 23, in load_html
canvaspy = f.read()
File "C:\Users\wstob\anaconda3\envs\sd-inf\lib\encodings\cp1252.py", line 23, in decode
return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x8f in position 347: character maps to

Suggestions

Very much lags when moving the area. Not usable without erasing the area. Undoing the previous action would be nice. The colab needs to move the work area to a new tab, if possible.

error: [Errno 2] No such file or directory: 'cmake'. (Ubuntu)

 Running setup.py install for fpie: started
  Running setup.py install for fpie: finished with status 'error'

Pip subprocess error:
  error: subprocess-exited-with-error

  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [17 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-cpython-310
      creating build/lib.linux-x86_64-cpython-310/fpie
      copying fpie/numba_solver.py -> build/lib.linux-x86_64-cpython-310/fpie
      copying fpie/gui.py -> build/lib.linux-x86_64-cpython-310/fpie
      copying fpie/np_solver.py -> build/lib.linux-x86_64-cpython-310/fpie
      copying fpie/cli.py -> build/lib.linux-x86_64-cpython-310/fpie
      copying fpie/taichi_solver.py -> build/lib.linux-x86_64-cpython-310/fpie
      copying fpie/process.py -> build/lib.linux-x86_64-cpython-310/fpie
      copying fpie/args.py -> build/lib.linux-x86_64-cpython-310/fpie
      copying fpie/io.py -> build/lib.linux-x86_64-cpython-310/fpie
      copying fpie/__init__.py -> build/lib.linux-x86_64-cpython-310/fpie
      running build_ext
      error: [Errno 2] No such file or directory: 'cmake'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for fpie
  error: subprocess-exited-with-error

  × Running setup.py install for fpie did not run successfully.
  │ exit code: 1
  ╰─> [19 lines of output]
      running install
      /home/zerocool/anaconda3/envs/sd-inf/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        warnings.warn(
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-3.10
      creating build/lib.linux-x86_64-3.10/fpie
      copying fpie/numba_solver.py -> build/lib.linux-x86_64-3.10/fpie
      copying fpie/gui.py -> build/lib.linux-x86_64-3.10/fpie
      copying fpie/np_solver.py -> build/lib.linux-x86_64-3.10/fpie
      copying fpie/cli.py -> build/lib.linux-x86_64-3.10/fpie
      copying fpie/taichi_solver.py -> build/lib.linux-x86_64-3.10/fpie
      copying fpie/process.py -> build/lib.linux-x86_64-3.10/fpie
      copying fpie/args.py -> build/lib.linux-x86_64-3.10/fpie
      copying fpie/io.py -> build/lib.linux-x86_64-3.10/fpie
      copying fpie/__init__.py -> build/lib.linux-x86_64-3.10/fpie
      running build_ext
      error: [Errno 2] No such file or directory: 'cmake'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> fpie

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

failed

CondaEnvException: Pip failed

(base) zerocool@DESKTOP-IFR8E96:~/github/stablediffusion-infinity$

Should i run: sudo snap install cmake “–classic” ?

-WSL2 + Ubuntu.

ModuleNotFoundError: No module named 'utils'

when I run "import libs" box
i keep getting this error

ModuleNotFoundError: No module named 'utils'

[Feature Request] A small list of possible quality of life improvements

Firstly, thank you for your time and energy in creating this amazing SD addon, it's really amazing.
Secondly, the following features I feel would be a great asset if added to your existing app and offering them for your kind consideration, please and thank you again.

1: Possible solution to lagging canvas: add scrolling bars to move around canvas. Would also be useful for creating clean borders.
2: Zoom in/out of canvas feature in aid of navigating and/or navigation thumbnail like photoshop.
3: A clear canvas button, instead of having to refresh the browser address.
4: A save default settings button.
5: Tool changer, size and shape. Perhaps even a block erase in the way of making the current render curser clear.
6: (I'm sure it's in the works but I wanted to add it to this list anyway.) Photometric Correction for Windows support, please.

Much appreciated.

M.

Laggy canvas fix ?

I wonder why the canvas is so laggy. It's so laggy i just gave up.

This process is ultra simple and shouldn't rely on any GPU.
There's no strong calculations here, it's just moving an image on a canvas, any browser handle this.

This process shouldn't be handled by the server, only the browser client.
Pretty sure there's a workaround by letting the browser handling the canvas process, then transmitting the current position of the image/selected area to SD when needed.

PNG with transparent pixels

When I upload a PNG with transparent pixels, they are treated as black pixels.

What logs should I look at ?

Where ?
Watch the log for the url to open in your browser. Choose the one that starts with http://127.0.0.1:8888/