kwaivgi / i2v-adapter

I2V-Adapter: A General Image-to-Video Adapter for Diffusion Models

I2V-Adapter: A General Image-to-Video Adapter for Diffusion Models

🔆 Introduction

This is a PyTorch/GPU implementation of the paper I2V-Adapter: A General Image-to-Video Adapter for Diffusion Models

Setup repository and conda environment

git clone https://github.com/KwaiVGI/I2V-Adapter.git
cd I2V-Adapter

conda env create -f environment.yaml
conda activate i2v_adapter

Download pretrained models

git lfs install
git clone https://huggingface.co/runwayml/stable-diffusion-v1-5 pretrained_models/stable-diffusion-v1-5
git clone https://huggingface.co/space-xun/i2v_adapter pretrained_models/I2V-Adapter
git clone https://huggingface.co/h94/IP-Adapter pretrained_models/IP-Adapter
mv pretrained_models/I2V-Adapter/animatediff_v15_v1_ipplus.pth pretrained_models/stable-diffusion-v1-5/unet/

Inference

bash infer.sh

BibTeX

@inproceedings{guo2023i2v,
author = {Xun Guo, Mingwu Zheng, Liang Hou, Yuan Gao, Yufan Deng, Pengfei Wan, Di Zhang, Yufan Liu, Weiming Hu, Zhengjun Zha, Haibin Huang, Chongyang Ma},
title = {I2V-Adapter: A General Image-to-Video Adapter for Diffusion Models},
year = {2024},
booktitle = {SIGGRAPH 2024 Conference Papers}
}

🤗 Acknowledgements

Codebase builds on Diffusers and AnimateDiff.

i2v-adapter's Issues

is the I2V Adapter code not released yet?

Excuse me, is the I2V Adapter code not released yet? I only see the IP Adapter code

SDXL I2V-Adapter weights

Thank you for releasing the code and model for SD1.5!

In the paper, it is clarified that there was were two I2V-Adapters trained - one for SD1.5 for SD1.5 AnimateDiff models, and one for SDXL for an SDXL AnimateDiff models (I'm assuming this to be the AnimateDiff beta release of the SDXL model). It also mentions that for the majority of the examples provided, it was the SDXL I2V-Adapter used.

Is there a timeline for when the SDXL I2V-Adapter weights will be released? The diffusers code can wait, but it would be great if the SDXL I2V weights could be released as soon as possible, as I'd be able to make them work the same as the SD1.5 weights within ComfyUI.

Thanks!

做得很好，可否兼容一下高版本的diffusers，现在的0.11.1版本太低了。可否原生支持一下ComfyUI，融入ComfyUI中使用。

如何运行ControlNet

看到论文中有关于图片复制ControlNet的内容，但示例中没有这部分，请问如何运行呀

Frame Similarity Prior variables

Hello, thank you for making this public, may I ask you where in the code could I find these two variables - t0=0.8, p=1.0?

min VRAM?

I have a 24G GPU, but it encountered CUDA out of memory?

What is the minimum VRAM requirement?