camenduru / controlnet-colab

🐣 Please follow me for new updates https://twitter.com/camenduru
🔥 Please join our discord server https://discord.gg/k5BwmmvJJU
🥳 Please join my patreon community https://patreon.com/camenduru

🚨 Stable Diffusion WebUI Colabs With ControlNet

https://github.com/camenduru/stable-diffusion-webui-colab

🚦 WIP 🚦

🚨 Only 1.5 base models may work right now

🦒 Colab

Colab Page	Model Page
stable_diffusion_controlnet_colab	https://huggingface.co/CompVis/stable-diffusion-v-1-4-original
waifu_diffusion_controlnet_colab	https://huggingface.co/hakurei/waifu-diffusion-v1-3
🚧 stable_diffusion_inpainting_controlnet_colab	https://huggingface.co/runwayml/stable-diffusion-inpainting
stable_diffusion_1_5_controlnet_colab	https://huggingface.co/runwayml/stable-diffusion-v1-5
mo_di_diffusion_controlnet_colab (Use the tokens modern disney style in your prompts for the effect.)	https://huggingface.co/nitrosocke/mo-di-diffusion
arcane_diffusion_3_controlnet_colab (Use the tokens arcane style in your prompts for the effect.)	https://huggingface.co/nitrosocke/Arcane-Diffusion
cyberpunk_anime_diffusion_controlnet_colab (Use the tokens dgs illustration style in your prompts for the effect.)	https://huggingface.co/DGSpitzer/Cyberpunk-Anime-Diffusion
midjourney_v4_diffusion_controlnet_colab (Use the tokens mdjrny-v4 style in your prompts for the effect.)	https://huggingface.co/prompthero/midjourney-v4-diffusion
papercut_diffusion_controlnet_colab (Use the tokens PaperCut in your prompts for the effect.)	https://huggingface.co/Fictiverse/Stable_Diffusion_PaperCut_Model
samdoesart_diffusion_controlnet_colab (Use the tokens SamDoesArt in your prompts for the effect.)	https://huggingface.co/Sandro-Halpo/SamDoesArt-V3
anything_3_controlnet_colab	https://huggingface.co/ckpt/anything-v3.0
woolitize_controlnet_colab (Use the tokens woolitize style in your prompts for the effect.)	https://huggingface.co/plasmo/woolitize
inkpunk_controlnet_colab (Use the tokens nvinkpunk style in your prompts for the effect.)	https://huggingface.co/Envvi/Inkpunk-Diffusion
rando_mix_3_controlnet_colab	https://huggingface.co/sentet1c/RandoMix3 (Thanks to Sint for the suggestion ❤)
f222_controlnet_colab	https://huggingface.co/lilpotat/f2
berry_mix_1_5_controlnet_colab	https://huggingface.co/lilpotat/rbm
🚧 stable_diffusion_v2_1_controlnet_colab	https://huggingface.co/stabilityai/stable-diffusion-2-1
🚧 stable_diffusion_v2_1_base_controlnet_colab	https://huggingface.co/stabilityai/stable-diffusion-2-1-base
🚧 stable_diffusion_v2_controlnet_colab	https://huggingface.co/stabilityai/stable-diffusion-2
🚧 stable_diffusion_v2_base_controlnet_colab	https://huggingface.co/stabilityai/stable-diffusion-2-base
🚧 stable_diffusion_v2_depth_controlnet_colab	https://huggingface.co/stabilityai/stable-diffusion-2-depth
🚧 stable_diffusion_v2_inpainting_controlnet_colab	https://huggingface.co/stabilityai/stable-diffusion-2-inpainting
hassan_blend_1_4_controlnet_colab	https://huggingface.co/hassanblend/hassanblend1.4 (Thanks to maximaljahn for the suggestion ❤)
yiffy_e18_controlnet_colab	https://huggingface.co/Doubleyobro/yiffy-e18
pony_diffusion_2_controlnet_colab	https://huggingface.co/AstraliteHeart/pony-diffusion-v2
elysium_anime_2_controlnet_colab	https://huggingface.co/hesw23168/SD-Elysium-Model
eimis_anime_diffusion_1_controlnet_colab	https://huggingface.co/eimiss/EimisAnimeDiffusion_1.0v (Thanks to Tjbr0ss2 for the suggestion ❤)
dreamlike_diffusion_1_controlnet_colab (Use the tokens dreamlikeart style in your prompts for the effect.)	https://huggingface.co/dreamlike-art/dreamlike-diffusion-1.0
van_gogh_diffusion_controlnet_colab (Use the tokens lvngvncnt style in your prompts for the effect.)	https://huggingface.co/dallinmackay/Van-Gogh-diffusion
analog_diffusion_controlnet_colab (Use the tokens analog style in your prompts for the effect.)	https://huggingface.co/wavymulder/Analog-Diffusion
monstermash_any_3_controlnet_colab	https://huggingface.co/N75242/monstermash_anyv3
yohan_diffusion_controlnet_colab	https://huggingface.co/andite/yohan-diffusion (Thanks to EBIX_benis for the suggestion ❤)
openjourney_v2_diffusion_controlnet_colab	https://huggingface.co/prompthero/openjourney-v2
vintedois_diffusion_0_1_controlnet_colab (Use the tokens estilovintedois in your prompts for the effect.)	https://huggingface.co/22h/vintedois-diffusion-v0-1
knollingcase_controlnet_colab (Use the tokens knollingcase in your prompts for the effect.)	https://huggingface.co/Aybeeceedee/knollingcase
comic_diffusion_v2_controlnet_colab (Use the tokens charliebo artstyle holliemengert artstyle marioalberti artstyle pepelarraz artstyle andreasrocha artstyle jamesdaly artstyle in your prompts for the effect.)	https://huggingface.co/ogkalu/Comic-Diffusion
complex_lineart_controlnet_colab (Use the tokens ComplexLA style in your prompts for the effect.)	https://huggingface.co/Conflictx/Complex-Lineart
synthwave_punk_v2_controlnet_colab (Use the tokens snthwve style nvinkpunk in your prompts for the effect.)	https://huggingface.co/ItsJayQz/SynthwavePunk-v2
illustration_diffusion_controlnet_colab (Use the tokens holliemengert artstyle in your prompts for the effect.)	https://huggingface.co/ogkalu/Illustration-Diffusion
seek_art_mega_v1_controlnet_colab	https://huggingface.co/coreco/seek.art_MEGA
popup_book_diffusion_controlnet_colab (Use the tokens popupBook in your prompts for the effect.)	https://huggingface.co/RayHell/popupBook-diffusion
riffusion_v1_controlnet_colab (Use with Script -> Riffusion Audio Generator)	https://huggingface.co/riffusion/riffusion-model-v1 (Thanks to Neverstorm for the suggestion ❤)
🚧 waifu_diffusion_v1_4_controlnet_colab	https://huggingface.co/hakurei/waifu-diffusion-v1-4
protogen_v2_2_controlnet_colab	https://huggingface.co/ckpt/Protogen_V2.2
elldreths_lucid_mix_v10_controlnet_colab	https://huggingface.co/ckpt/elldrethSLucidMix_v10
all_in_one_pixel_model_controlnet_colab (Use the tokens pixelsprite 16bitscene in your prompts for the effect.)	https://huggingface.co/PublicPrompts/All-In-One-Pixel-Model
cafe_instagram_v6_controlnet_colab	https://huggingface.co/cafeai/cafe-instagram-sd-1-5-v6
cinematic_diffusion_controlnet_colab (Use the tokens syberart in your prompts for the effect.)	https://huggingface.co/Basunat/Cinematic-Diffusion
food_crit_controlnet_colab (Use the tokens food_crit in your prompts for the effect.)	https://huggingface.co/plasmo/food-crit
simpmaker_3k_controlnet_colab	https://huggingface.co/AmethystVera/SimpMaker-3K1
🚧 duchaitenaiart_v1_1_controlnet_colab	https://huggingface.co/DucHaiten/DucHaitenAIart (Thanks to DucHaiten for the suggestion ❤)
uber_realistic_v1_controlnet_colab	https://huggingface.co/lilpotat/urp (Thanks to cdefghijkl for the suggestion ❤)
izumi_v1_controlnet_colab	https://huggingface.co/lilpotat/iz
yiffy_mix_controlnet_colab	https://civitai.com/models/3671/yiffymix
babes_blend_controlnet_colab	https://civitai.com/models/2220/babes
samdoessexy_blend_controlnet_colab	https://civitai.com/models/1180/samdoessexy-blend
comics_blend_controlnet_colab (Use the tokens ComplexLA style nvinkpunk marioalberti artstyle ghibli style in your prompts for the effect.)	https://civitai.com/models/1491/comicsblend
dream_like_sam_kuvshino_controlnet_colab	https://civitai.com/models/1473/dreamlikesamkuvshinov
8528_diffusion_controlnet_colab	https://huggingface.co/852wa/8528-diffusion (Thanks to 黄色いバラ for the suggestion ❤)
hasdx_controlnet_colab	https://civitai.com/models/3758/hasdx (Thanks to EBIX_benis for the suggestion ❤)
abyss_orange_mix_2_controlnet_colab	https://huggingface.co/WarriorMama777/OrangeMixs
dreamlike_photoreal_2_controlnet_colab	https://huggingface.co/dreamlike-art/dreamlike-photoreal-2.0
photorealistic_fuen_v1_controlnet_colab	https://huggingface.co/claudfuen/photorealistic-fuen-v1
🚧 cool_japan_diffusion_2_1_controlnet_colab	https://huggingface.co/aipicasso/cool-japan-diffusion-2-1-0
anything_gape_controlnet_colab	https://huggingface.co/Inzamam567/AnyGape (Thanks to soliwoli for the suggestion ❤)
anything_4_controlnet_colab	https://huggingface.co/andite/anything-v4.0
counterfeit_v2_0_controlnet_colab	https://huggingface.co/gsdf/Counterfeit-V2.0 (Thanks to 黄色いバラ for the suggestion ❤)
grapefruit_controlnet_colab	https://huggingface.co/ckpt/grapefruit
open_anime_journey_controlnet_colab	https://huggingface.co/MehjourneyClosedAI/OpenAnimeJourney (Thanks to 黄色いバラ for the suggestion ❤)
a_certainty_controlnet_colab	https://huggingface.co/JosephusCheung/ACertainty
a_certain_model_controlnet_colab	https://huggingface.co/JosephusCheung/ACertainModel
a_certain_thing_controlnet_colab	https://huggingface.co/JosephusCheung/ACertainThing
🚧 7th_layer_controlnet_colab	https://huggingface.co/syaimu/7th_Layer (Thanks to Cherryblossomoflove for the suggestion ❤)
🚧 rpg_v4_controlnet_colab	https://huggingface.co/Anashel/rpg (Thanks to brunocalado for the suggestion ❤)
🚧 waifu_diffusion_v1_5_controlnet_colab	https://huggingface.co/waifu-diffusion/wd-1-5-beta
🚧 protogen_nova_controlnet_colab	https://huggingface.co/darkstorm2150/Protogen_Nova_Official_Release (Thanks to alvaromaltrain for the suggestion ❤)
🚧 protogen_x3_4_controlnet_colab	https://huggingface.co/darkstorm2150/Protogen_x3.4_Official_Release (Thanks to alvaromaltrain for the suggestion ❤)

Main Repo

https://github.com/lllyasviel/ControlNet thanks to https://twitter.com/lvminzhang ❤

Paper

https://github.com/lllyasviel/ControlNet/raw/main/github_page/control.pdf

Modified Code

https://huggingface.co/spaces/hysts/ControlNet-with-other-models thanks to https://twitter.com/hysts12321 ❤

Extracted ControlNet weights

https://huggingface.co/kohya-ss/ControlNet-diff-modules/tree/main thanks to https://twitter.com/kohya_tech ❤
https://huggingface.co/webui/ControlNet-modules-safetensors/tree/main thanks to https://github.com/ClashSAN ❤

License

https://github.com/lllyasviel/ControlNet/blob/main/LICENSE
https://huggingface.co/spaces/hysts/ControlNet-with-other-models/blob/main/LICENSE

controlnet-colab's Issues

Where?

Hi, I'm trying to find a way to convert a bunch of images into a ControlNet SoftEdge/HED format. Does anyone know if there is a Colab notebook for that?

=>

Any chance of adding a batch mode, such as the one in Automatic1111

Hey, is there any chance of adding a batch mode into the ControlNet options, so that it'd be possible to edit multiple images on one go? Looking for alternatives since Colab stopped supporting WebUI, and this notebooks may be a solution!

Thanks for all the work

ControlNet with Anime

Hello!

Amazing! The author just posted a tutorial to use ControlNet with Anime [AnythingV3] Link

I'm having a problem:

I run on Colab, the computer does not have enough VRAM to merge model.
Need at least 21.11 Gb RAM to load 3 model.
Can someone share with me the model after the merge "control_any3.pth"?
I need "Canny edge" model.

Thank you very much!

output

Please add the super merger extension.

I need this extension to merge lora. I was able to install it but there was a problem with lora, lora which I downloaded via civit ai, but in the lora tab the model is so it is not there.

Can't use transfer control to other models colab

Hello, i'm having these errors when i try to use the colab in colab free plan:

The system ram goes really up and it stops:

I'm using control_sd15_openpose

Support for 2.x SD models

Hello and thanks for your contribution!
Can we use this colab notebook on 2.x Stable Diffusion models? The original repository doesn't seem to work yet on these models

RuntimeError: Error(s) in loading state_dict for ControlLDM

stable_diffusion_inpainting_controlnet_colab.ipynb doesn't work.

【bug】loading error about waifu_diffusion_v1_4 model

Loaded model config from [ControlNet/models/cldm_v15.yaml]
Loaded state_dict from [models/wd-1-4-anime_e1.ckpt]
Traceback (most recent call last):
  File "app.py", line 52, in <module>
    model = Model()
  File "/content/ControlNet-with-other-models/model.py", line 67, in __init__
    self.set_base_model(DEFAULT_BASE_MODEL_REPO,
  File "/content/ControlNet-with-other-models/model.py", line 75, in set_base_model
    self.load_base_model(base_model_url)
  File "/content/ControlNet-with-other-models/model.py", line 89, in load_base_model
    self.model.load_state_dict(load_state_dict(model_path,
  File "/content/anaconda3/envs/control/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1671, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for ControlLDM:
	size mismatch for model.diffusion_model.input_blocks.1.1.proj_in.weight: copying a param with shape torch.Size([320, 320]) from checkpoint, the shape in current model is torch.Size([320, 320, 1, 1]).
	size mismatch for model.diffusion_model.input_blocks.1.1.transformer_blocks.0.attn2.to_k.weight: copying a param with shape torch.Size([320, 1024]) from checkpoint, the shape in current model is torch.Size([320, 768]).
	size mismatch for model.diffusion_model.input_blocks.1.1.transformer_blocks.0.attn2.to_v.weight: copying a param with shape torch.Size([320, 1024]) from checkpoint, the shape in current model is torch.Size([320, 768]).