samedii / perceptor

Modular image generation library

perceptor

Modular image generation library.

Install

poetry add perceptor

Or, for the old timers:

pip install perceptor

CUDA 11.3 (Support RTX cards)

poetry run pip uninstall torch torchvision -y && poetry run pip install torch==1.12.1 torchvision==0.13.1 --extra-index-url https://download.pytorch.org/whl/cu113

Interface

Shortlist of available features. See the API reference for more information.

perceptor
  .drawers
    .BruteDiffusion
    .DeepImagePrior
    .Raw
    .RuDALLE
    .StyleGANXL
  .models
    .AdaBinsDepth
    .DeepImagePrior
    .VelocityDiffusion (yfcc_2, yfcc_1, cc12m_1_cfg, wikiart)
    .latent_diffusion
      .Text2Image
      .Face
      .SuperResolution
    .GuidedDiffusion (openai, pixelart)
    .MidasDepth
    .MonsterDiffusion (all, tinyhero)
    .StyleGANXL
    .RuDALLE
    .StableDiffusion
    .SuperResolution
  .losses
    # Text-image similarity
    .BLIP
    .CLIP
    .CLOOB
    .LiT
    .GlideCLIP
    .OpenCLIP
    .OWLViT
    .RuCLIP
    .SLIP
    # Other
    .AestheticVisualAssessment
    .LPIPS
    .Memorability
    .MidasDepth
    .SimulacraAesthetic
    .Smoothness
    .StyleTransfer
  .transforms
    .clamp_with_grad
    .resize
    .SuperResolution
  .utils
    .gradient_checkpoint

perceptor's Issues

StyleGAN XL 1024x1024 model

256x256 model already added. Should add 512 and 1024 checkpoints for imagenet and ffhq.

https://github.com/autonomousvision/stylegan_xl

Latent diffusion inpainting model

Three other models from this repo have already been added (super resolution, text2image, and text2image finetuned on no watermarks)
https://github.com/CompVis/latent-diffusion

Background removal models

https://github.com/xuebinqin/U-2-Net
https://github.com/PeterL1n/RobustVideoMatting
https://colab.research.google.com/drive/1bLn5FOR6YSTl0ca3YL3RcDdI2rcvF9Ll

Dynamic thresholding

Compare thresholding implementation with majesty diffusion
https://github.com/multimodalart/majesty-diffusion

Pulp sci-fi diffusion model

https://github.com/KaliYuga-ai/Pulp-Sci-Fi-Diffusion/blob/main/Pulp_Sci_Fi_Diffusion_v1_0.ipynb

Simulacra aesthetic model

https://github.com/crowsonkb/simulacra-aesthetic-models

OpenCLIP ViT-B/16+ 240x240 model

Other checkpoints are already added but a larger model was trained with a larger resolution too
https://github.com/mlfoundations/open_clip

Migrate to transformers implementation of CLIP

https://github.com/huggingface/transformers

Allows returning last hidden layer that is used in stable diffusion.

Diff2x model

https://github.com/peterwilli/Diff2X

CLIPSeg

https://github.com/timojl/clipseg

Composable stable diffusion

https://energy-based-model.github.io/Compositional-Visual-Generation-with-Composable-Diffusion-Models/

https://github.com/energy-based-model/Compositional-Visual-Generation-with-Composable-Diffusion-Models-PyTorch

https://huggingface.co/spaces/Shuang59/Composable-Diffusion

Panini face restoration model

Port pytorch model https://github.com/jianzhangcs/panini

Another aesthetic predictor

https://github.com/christophschuhmann/improved-aesthetic-predictor

UViM model

https://paperswithcode.com/paper/uvim-a-unified-modeling-approach-for-vision
https://github.com/google-research/big_vision

Do not load stable diffusion via pipeline

Watercolor diffusion model

https://github.com/KaliYuga-ai/Pixel-Art-Diffusion/blob/main/Watercolor_Diffusion_v1_0.ipynb

StyleGAN Human models

https://stylegan-human.github.io/

spherical loss for blip

I noticed the use of the spherical loss in the BLIP files. What is the gain you get from this loss?

Handpainted CG diffusion model

https://github.com/FeiArt-Ai/Handpainted-CG-Diffusion/tree/474669d3fa42b0b9666e75eb716bb4efbef8caa9

Let velocity_diffusion.schedule_ts take timesteps as arguments instead of sigma

Investigate if decorator.py or wrapt help intellisense with cache decorator

https://github.com/micheles/decorator/

https://github.com/GrahamDumpleton/wrapt

Deep image prior drawer

https://github.com/crowsonkb/deep-image-prior

Stable diffusion model

https://github.com/pesser/stable-diffusion

GroupViT model

https://huggingface.co/docs/transformers/model_doc/groupvit

Conditioned prior model

https://huggingface.co/nousr/conditioned-prior
https://github.com/laion-ai/deep-image-diffusion-prior

Allows converting text embeddings to image embeddings

FLAVA model

https://huggingface.co/docs/transformers/model_doc/flava

from PIL import Image
import requests
from transformers import FlavaProcessor, FlavaModel

model = FlavaModel.from_pretrained("facebook/flava-full")
processor = FlavaProcessor.from_pretrained("facebook/flava-full")

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

inputs = processor(text=["a photo of a cat"], images=image, return_tensors="pt", padding=True)

outputs = model(**inputs)
logits_per_image = outputs.contrastive_logits_per_image  # this is the image-text similarity score
probs = logits_per_image.softmax(dim=1)  # we can take the softmax to get the label probabilities

CLIP style transfer

https://github.com/cyclomon/CLIPstyler

LAION aesthetic predictor models

https://github.com/LAION-AI/aesthetic-predictor

Multilingual CLIP model

https://huggingface.co/M-CLIP/XLM-Roberta-Large-Vit-B-16Plus

DeFILIP perceptor

https://github.com/Sense-GVT/DeCLIP

FILIP and DeCLIP may as well be added at the same time.

Prompt-to-prompt with stable diffusion / cross attention control

https://github.com/google/prompt-to-prompt

Stable diffusion inpainting model

CLIP interrogator

ViT-L aesthetic embeddings

Used here
https://github.com/multimodalart/majesty-diffusion

Add guided diffusion portrait model

alembics/disco-diffusion@d6d68b0#commitcomment-79429271
https://huggingface.co/felipe3dartist/portrait_generator_v1.5/blob/main/ema_0.9999_165000.pt

CLOOB perceptor

Choose between
https://github.com/crowsonkb/cloob-training
and
https://github.com/ml-jku/cloob

Optimize stable diffusion

Might be merged into diffusers
huggingface/diffusers#532

https://github.com/neonsecret/stable-diffusion

https://github.com/TheLastBen/fast-stable-diffusion

Only inference x2.4
https://github.com/facebookincubator/AITemplate/tree/main/examples/05_stable_diffusion

OWL-ViT perceptor

https://huggingface.co/docs/transformers/model_doc/owlvit

Openclip ViT-L/14 perceptor

Added in v1.3.0
https://github.com/mlfoundations/open_clip

Waifu stable diffusion

https://huggingface.co/hakurei/waifu-diffusion/tree/main

Use predicted noise as base when taking steps in latent diffusion

We get numerical errors when alpha/sigma is close to zero otherwise

Additionally

• Refactor and rename eps to "predicted noise"
• Refactor to use "prediction" class as output

Noised CLIP

Add CLIP model from here
https://github.com/openai/glide-text2im

that has been trained to handle noisy images.

LiT perceptor

Port models from this jax repo https://github.com/google-research/vision_transformer#lit-models

Pixel art diffusion model

https://github.com/KaliYuga-ai/Pixel-Art-Diffusion

Add diffusers face model

https://github.com/huggingface/diffusers

from diffusers import DDPMPipeline, DDIMPipeline, PNDMPipeline

model_id = "google/ddpm-celebahq-256"

# load model and scheduler
ddpm = DDPMPipeline.from_pretrained(model_id) # you can replace DDPMPipeline with DDIMPipeline or PNDMPipeline for faster inference

Replace stable diffusion decoder

https://huggingface.co/stabilityai/sd-vae-ft-mse

Text off vectors are not included in package

FileNotFoundError: [Errno 2] No such file or directory: 'perceptor/losses/clip/vectors/textoff.json'

LANCZOS default when downsampling

LANCZOS is supposed to be good for downsampling

Improved VQ diffusion model

https://github.com/cientgu/VQ-Diffusion/tree/Improved_VQ-Diffusion

CLOOB latent diffusion models

https://github.com/JD-P/cloob-latent-diffusion