jackzhousz / pair-diffusion

PAIR-Diffusion: Object-Level Image Editing with Structure-and-Appearance Paired Diffusion Models, 2023

PAIR-Diffusion

This is the official repo for the work:

PAIR-Diffusion: Object-Level Image Editing with Structure-and-Appearance Paired Diffusion Models
Vidit Goel^1*, Elia Peruzzo^1,2*, Yifan Jiang³, Dejia Xu³, Nicu Sebe², Trevor Darrell⁴, Zhangyang Wang^1,3 and Humphrey Shi ^1,5,6

In association with Picsart AI Research (PAIR)¹, UTrento², UT Austin³, UC Berkeley⁴, UOregon⁵, UIUC⁶

*equal contribution

We built Structure and Appearance Paired (PAIR) Diffusion that allows reference image-guided appearance manipulation and structure editing of an image at an object level. Describing object appearances using text can be challenging and ambiguous, PAIR Diffusion enables a user to control the appearance of an object using images.

Try our demo at

[arXiv][Video]

News

• [04/09/2023] Inference code released
• [04/07/2023] Demo relased on 🤗Huggingface space!
• [03/30/2023] Paper released on arXiv

Results

Given below are results for appearace editing using our method on SDv1.5

Object Level Image Editing

Stable Diffusion Results

• Applying our method to Stable Diffusion allows reference image based editing in the wild

• We can edit both structure and appearance of the objects independently.

Unconditional Diffusion Models

• PAIR diffusion also works with unconditional diffusion models, we show results on LSUN churches, bedrooms and CelebA-HQ. We can perform edits using both in-domain and out-of-domain reference images.

Requirements

Setup the conda environment using the command below. We use Oneformer to get segmentation maps during inference, please setup environment for Oneformer following the repo

conda env create -f environment.yaml
conda activate pair-diff

Inference

To run the model launch the gradio demo using the command below. It will download the required models as well.

python gradio_app.py

Pretrained Models

We applied PAIR Diffusion on SDv1.5 and uses COCO-Stuff dataset for finetuning the model. The model card can be downloaded from here

BibTeX

If you use our work in your research, please cite our publication:

@article{goel2023pair,
      title={PAIR-Diffusion: Object-Level Image Editing with Structure-and-Appearance Paired Diffusion Models},
      author={Goel, Vidit and Peruzzo, Elia and Jiang, Yifan and Xu, Dejia and Sebe, Nicu and Darrell, Trevor and 
      Wang, Zhangyang and Shi, Humphrey},
      journal={arXiv preprint arXiv:2303.17546},
      year={2023} 
      }