tooncrafter / tooncrafter

a research paper for generative cartoon interpolation

ToonCrafter: Generative Cartoon Interpolation

🔆 Introduction

⚠️ Please check our disclaimer first.

🤗 ToonCrafter can interpolate two cartoon images by leveraging the pre-trained image-to-video diffusion priors. Please check our project page and paper for more information.

1.1 Showcases (512x320)

Input starting frame	Input ending frame	Generated video

1.2 Sparse sketch guidance

Input starting frame	Input ending frame	Input sketch guidance	Generated video

2. Applications

2.1 Cartoon Sketch Interpolation (see project page for more details)

Input starting frame	Input ending frame	Generated video

2.2 Reference-based Sketch Colorization

Input sketch	Input reference	Colorization results

📝 Changelog

• Add sketch control and colorization function.
• [2024.05.29]: 🔥🔥 Release code and model weights.
• [2024.05.28]: Launch the project page and update the arXiv preprint.

🧰 Models

Model	Resolution	GPU Mem. & Inference Time (A100, ddim 50steps)	Checkpoint
ToonCrafter_512	320x512	TBD (perframe_ae=True)	Hugging Face

Currently, our ToonCrafter can support generating videos of up to 16 frames with a resolution of 512x320. The inference time can be reduced by using fewer DDIM steps.

⚙️ Setup

Install Environment via Anaconda (Recommended)

conda create -n tooncrafter python=3.8.5
conda activate tooncrafter
pip install -r requirements.txt

💫 Inference

1. Command line

Download pretrained ToonCrafter_512 and put the model.ckpt in checkpoints/tooncrafter_512_interp_v1/model.ckpt.

  sh scripts/run.sh

2. Local Gradio demo

Download the pretrained model and put it in the corresponding directory according to the previous guidelines.

  python gradio_app.py 

📢 Disclaimer

Calm down. Our framework opens up the era of generative cartoon interpolation, but due to the variaity of generative video prior, the success rate is not guaranteed.

⚠️This is an open-source research exploration, instead of commercial products. It can't meet all your expectations.

This project strives to impact the domain of AI-driven video generation positively. Users are granted the freedom to create videos using this tool, but they are expected to comply with local laws and utilize it responsibly. The developers do not assume any responsibility for potential misuse by users.

---

tooncrafter's Issues

Is it possible to generate clips unconditionally?

Ai anime

Security Concerns with a Windows Fork of This Project

Hi,

I'm looking into using your project and I noticed a Windows-specific fork ( https://github.com/sdbds/ToonCrafter-for-windows ) that alters the installation process by setting the PowerShell execution policy to Unrestricted. This raises some security red flags for me.

I’d like to ask:

• Are you aware of this fork and the changes made?
• Do you think these modifications, especially setting the execution policy to Unrestricted, are safe?
• Do you have any plans to officially support Windows while ensuring security best practices, or do you endorse this fork?

Your insights would really help me and other potential users decide whether to use this fork or wait for an official update.

Thanks for your work on this project and looking forward to your thoughts!

Developed a full version of ToonCrafter

Have you developed a full version of ToonCrafter yet? If you have already developed full version, we are very interested in understanding more about its capabilities and features

we need Color reference

EU law sourcing datasets summary

Due to the recent EU laws being passed, I'd like to know what the dataset is that it's trained on. The applications of this is limited in my region if I cannot point out the datasets of the tool.

It`s possible is use as video interpolation x2 fps to anime

Is it possible to use it in the mode of generating only one fps. What would x2 do to fps anime?

Recommended VRAM?

I see that model's size is 10Gb. How much vram it needs on inference, 16gb is enough?

What about a gui version?

Could u please provide a gui exe windows version of ToonCrafter in release section?

PLEASE RELEASE THE CODE ON HOW TO USE THE SKETCH MODELS AND COLOR REFEREBNCE SKETCH INTERPOLATION PLEASE

sketch_encoder.ckpt

strong！how to use this model

Please help!Error occurred when executing DownloadAndLoadDynamiCrafterModel:local variable 'dcmodel' referenced before assignment

Error occurred when executing DownloadAndLoadDynamiCrafterModel:

local variable 'dcmodel' referenced before assignment

File "D:\Program Files\ComfyUI-aki-v1.3\execution.py", line 152, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "D:\Program Files\ComfyUI-aki-v1.3\execution.py", line 82, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "D:\Program Files\ComfyUI-aki-v1.3\execution.py", line 75, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "D:\Program Files\ComfyUI-aki-v1.3\custom_nodes\ComfyUI-DynamiCrafterWrapper\nodes.py", line 146, in loadmodel
return (dcmodel,)

About ComfyUI implement

@Doubiiu Hello ToonCrafter Team,

We are a team called AIGODLIKE, a group is dedicated to the study of animation and cg technology, and we are excited to share to users that we have discovered your innovative animation technology, ToonCrafter.

We are currently working on implementing ToonCrafter into ComfyUI, a node-based powerful and modular stable diffusion GUI.

We believe that integrating ToonCrafter into ComfyUI will greatly enhance animation capabilities and provide significant benefits to animation industry.

Please feel free to reach out if you have any questions or if there are any specific guidelines or licences we should follow during the integration process. We look forward to your feedback and are excited about the possibilities of this collaboration will bring.

Thank you for your time and consideration.

No module named 'antlr4'

hello just have this erros when i try to run "python gradio_app.py":
(env) Z:\ia\toonmaker\ToonCrafter>python gradio_app.py
Traceback (most recent call last):
File "Z:\ia\toonmaker\ToonCrafter\gradio_app.py", line 4, in
from scripts.gradio.i2v_test_application import Image2Video
File "Z:\ia\toonmaker\ToonCrafter\scripts\gradio\i2v_test_application.py", line 3, in
from omegaconf import OmegaConf
File "Z:\ia\toonmaker\ToonCrafter\env\lib\site-packages\omegaconf_init_.py", line 1, in
from .base import Container, DictKeyType, Node, SCMode
File "Z:\ia\toonmaker\ToonCrafter\env\lib\site-packages\omegaconf\base.py", line 9, in
from antlr4 import ParserRuleContext
ModuleNotFoundError: No module named 'antlr4'

i already fixed the issue of decord : ModuleNotFoundError: No module named 'decord' with pip install decord
but this time i can not do in and it is not in requierement.txt
(just folowed the instructions from this guy:
https://www.youtube.com/watch?v=hc5nF6rGa68&t=183s 1:46-3:08)

Issues installing. Error message.

Hello, I am trying to install ToonCrafter using the install.ps1 file, how can I fix this?

I am getting these errors:

Collecting torch==2.0.1
Downloading https://download.pytorch.org/whl/cu118/torch-2.0.1%2Bcu118-cp310-cp310-win_amd64.whl (2619.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.6/2.6 GB 1.9 MB/s eta 0:00:00
ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: 'C:\Users\Nebula PC\AppData\Local\Temp\pip-build-tracker-_6miaosx\4efdef60987dfb791b1ce7a28586a1374a8c3154277ce13ec300c376'

check models...

Directory: C:\Users\Nebula PC\Desktop\Tooncrafter\ToonCrafter-for-windows

Mode LastWriteTime Length Name

---

d----- 2/06/2024 5:34 PM checkpoints
Downloading tooncrafter_512_interp_v1 model...
huggingface-cli : The term 'huggingface-cli' is not recognized as the name of a cmdlet, function, script file, or
operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try
again.
At C:\Users\Nebula PC\Desktop\Tooncrafter\ToonCrafter-for-windows\install.ps1:24 char:5

• huggingface-cli download --resume-download Doubiiu/ToonCrafter -- ...
  

• ~~~~~~~~~~~~~~~
  

  • CategoryInfo : ObjectNotFound: (huggingface-cli:String) [], CommandNotFoundException
  • FullyQualifiedErrorId : CommandNotFoundException

are there plans to release the Reference-based Sketch guidance?

M1 Mac launch (froze on AE working on z of shape)

1. I installed all packages specific to ARM,
2. skipped xformrs because of CUDA
3. when running it stucks on AE working on z of shape (1, 4, 32, 32) = 4096 dimensions. (main checkpoint is already downloaded into checkpoints/tooncrafter_512_interp_v1)

seems like it's stuck somewhere on HF downloads. lib/python3.8/site-packages/huggingface_hub/file_download.py", line 1221, in hf_hub_download kind of stuff.

Can I train tooncrafter checkpoint locally?

Hey. First of all - great job on this!
I feel like this tool has so much potential, but, like Stable Diffusion - it feels like most of it is hidden upon release.
Any plans on making documentation/tools for community to train their own checkpoints, or maybe loras/etc?
I'd be really interested in that.

If not though - still thank you for this awesome tool and special thanks for making it opensourced!

Torch not compiled with CUDA enabled

I am trying to test the tool but after running the installation process it does not work when I click on generate.
In the console an exception is raised on cuda initialization (Torch not compiled with CUDA enabled)
My card is a 3080 with 10GB.
Below I attach the full log after running gradio app

(tooncrafter) C:\Users\user\dev\ToonCrafter>python gradio_app.py
WARNING[XFORMERS]: xFormers can't load C++/CUDA extensions. xFormers was built for:
    PyTorch 2.0.0+cu118 with CUDA 1108 (you have 2.0.0+cpu)
    Python  3.8.10 (you have 3.8.5)
  Please reinstall xformers (see https://github.com/facebookresearch/xformers#installing-xformers)
  Memory-efficient attention, SwiGLU, sparse and more won't be available.
  Set XFORMERS_MORE_DETAILS=1 for more details
AE working on z of shape (1, 4, 32, 32) = 4096 dimensions.
checkpoints/tooncrafter_512_interp_v1/model.ckpt
>>> model checkpoint loaded.
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Global seed set to 123
start: talking man 2024-06-07 23:46:38
Traceback (most recent call last):
  File "C:\Users\user\miniconda3\envs\tooncrafter\lib\site-packages\gradio\queueing.py", line 532, in process_events
    response = await route_utils.call_process_api(
  File "C:\Users\user\miniconda3\envs\tooncrafter\lib\site-packages\gradio\route_utils.py", line 276, in call_process_api
    output = await app.get_blocks().process_api(
  File "C:\Users\user\miniconda3\envs\tooncrafter\lib\site-packages\gradio\blocks.py", line 1923, in process_api
    result = await self.call_function(
  File "C:\Users\user\miniconda3\envs\tooncrafter\lib\site-packages\gradio\blocks.py", line 1509, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "C:\Users\user\miniconda3\envs\tooncrafter\lib\site-packages\anyio\to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
  File "C:\Users\user\miniconda3\envs\tooncrafter\lib\site-packages\anyio\_backends\_asyncio.py", line 2177, in run_sync_in_worker_thread
    return await future
  File "C:\Users\user\miniconda3\envs\tooncrafter\lib\site-packages\anyio\_backends\_asyncio.py", line 859, in run
    result = context.run(func, *args)
  File "C:\Users\user\miniconda3\envs\tooncrafter\lib\site-packages\gradio\utils.py", line 832, in wrapper
    response = f(*args, **kwargs)
  File "C:\Users\user\dev\ToonCrafter\scripts\gradio\i2v_test_application.py", line 51, in get_image
    model = model.cuda()
  File "C:\Users\user\miniconda3\envs\tooncrafter\lib\site-packages\lightning_fabric\utilities\device_dtype_mixin.py", line 69, in cuda
    device = torch.device("cuda", torch.cuda.current_device())
  File "C:\Users\user\miniconda3\envs\tooncrafter\lib\site-packages\torch\cuda\__init__.py", line 674, in current_device
    _lazy_init()
  File "C:\Users\user\miniconda3\envs\tooncrafter\lib\site-packages\torch\cuda\__init__.py", line 239, in _lazy_init
    raise AssertionError("Torch not compiled with CUDA enabled")
AssertionError: Torch not compiled with CUDA enabled

How do I add --listen flag?

Trying to run this on Runpod as my 3090 just keeps getting Out of memory. However runpod requires this to have the --listen flag. I tried running gradio_app.py with --listen but I get the error

usage: gradio_app.py [-h]
gradio_app.py: error: unrecognized arguments: --listen
root@502fd222ada8:/workspace/ToonCrafter# 

I also tried to figure out where to add it in the python script but I can't so raising it as an issue.

Dataset

I saw that you collected 270K cartoon video clips in this paper, will you open this dataset in the future?

Add ComfyUI support

Hope to add support for ComfyUI

fodase

ModuleNotFoundError: No module named 'decord'

I have no idea how to solve this. this happened after trying to run python gradio_app.py

H:\AI\ToonCrafter> python gradio_app.py
Traceback (most recent call last):
File "H:\AI\ToonCrafter\gradio_app.py", line 4, in
from scripts.gradio.i2v_test_application import Image2Video
File "H:\AI\ToonCrafter\scripts\gradio\i2v_test_application.py", line 5, in
from scripts.evaluation.funcs import load_model_checkpoint, save_videos, batch_ddim_sampling, get_latent_z
File "H:\AI\ToonCrafter\scripts\evaluation\funcs.py", line 4, in
from decord import VideoReader, cpu
ModuleNotFoundError: No module named 'decord'

H:\AI\ToonCrafter>

Which config controls temporal layer freezing?

Hi,

Thanks for your great work. I was analyzing the code, and was curious where the temporal layers in the current training setup are frozen (my understanding is that they are frozen in training to ensure the motion prior is not lost) ? I don't see any location where the temporal layers are being frozen.

Do let me know which line of code i am missing.

Thanks!
Adi

🙇‍♂️ Thank you the legends who developed and open-sourced this, which puts that AnimateAnyone repo with 14000+ stars and zero code to shame.

Hardware requirements

Hi there,

What are the hardware requirements to run this on a local machine?

It's different from the official example

This is what I ran：

Japan_v2_2_062266_s2_frame1_sample0.mp4

Official version：

Directly running sh scripts/run.sh without changing settings

I create colab version for ToonCrafter

see it ToonCrafter

How to use API remote call, is there any documentation?

How to use API remote call, is there any documentation?

Training Compute needed

Hi!

I was curious how many A100/H100 GPU hours training took on the entire training corpus?

Thanks!
Adi

How do you add the sketch to the reffrence?

I'm trying to see how you color black and white animations. How do you add the sketch reference?

ImportError: libgthread-2.0.so.0: cannot open shared object file: No such file or directory

i'm trying to run this on sagemaker studio lab but since it has a limited storage i can't download the checkpoint in the usual way and need to use a little trick. When i download the moldel in that way it create a new folder inside the checkpoint one. is there a way to change where the model should be found in the codes?

Multi GPU support

Hi. I have a desktop that has 2x Tesla T4s, and it should be working because it has 32G VRAM in total, while other people reported to have a 27G VRAM usage when inferring. It should work but when inferring, only 1 gpu has being used, which caused a cuda out of memory error, what parts of code should i edit so that it can work on multiple GPUs? Thanks in advance

Decord cannot be installed on a MAC. Is there a MAC version available?

Does not generate any output

after running ToonCrafter, it just generates forever and nothing happens for almost 40 minutes.
My GPU is RTX 4070Ti.

AE working on z of shape (1, 4, 32, 32) = 4096 dimensions.
checkpoints/tooncrafter_512_interp_v1/model.ckpt
>>> model checkpoint loaded.
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Global seed set to 789
start: an anime scene 2024-06-01 11:15:03
C:\ToonCrafter-for-windows\venv\lib\site-packages\torchvision\transforms\functional.py:1603: UserWarning: The default value of the antialias parameter of all the resizing transforms (Resize(), RandomResizedCrop(), etc.) will change from None to True in v0.17, in order to be consistent across the PIL and Tensor backends. To suppress this warning, directly pass antialias=True (recommended, future default), antialias=None (current default, which means False for Tensors and True for PIL), or antialias=False (only works on Tensors - PIL will still use antialiasing). This also applies if you are using the inference transforms from the models weights: update the call to weights.transforms(antialias=True).
  warnings.warn(

Is it supposing to be taking this long to generate? (4060 ti 16gb vram)

Its taking me a long time 12 mins to generate a video. I have a 4060ti 16gb vram gpu. Any suggests to make it go faster?

Results combining two videos

Not an issue but just to share results. I'm combining two output videos made with three keyframes... A to B and then B to C.

Result:

Plate_TwoVidsCombine_01.mp4

Input:

Very nice! Thank you for sharing the model and code. ❤️

did we can train the model?

it's a great job, thanks your release. i want to know is there train code or only inference code release??

What about version 1024?

be launched?

hugging face demo error

when I used this demo show error

fire

Larger model release

Hey, thanks for a very impressive work, since you've fine-tuned from DynamicCrafter, which also has 576x1024 resolution model, should be expect a higher-res version of ToonCrafter any time soon? Or you're not planning to release it?

About training loss value

Hi, I am finetuning your model and training the spatial layer. Can you share your tensorboard file about the loss change?

Would it be possible to generate 720p or at least 480p videos?

Forgive my clumsiness, I guess it's something that's still developing. It's an amazing tool, but if it could generate slightly larger images it could be used beyond the demo.
I also see that the brightness/tone/contrast in the video changes slightly with respect to the original images.

How to implement Reference-based Sketch Colorization？

How to implement Reference-based Sketch Colorization？

Suggestion: Make a lite version that only cleans up existing human made animations

I'm not sure if this would work, but wouldn't the cleanup (reference-based sketch colorization) part of the AI be easy to isolate? It'd also be lighter on system resources probably.

Many human animators like to rough animate but cleanup is tedious. The clean-up examples also look the best out of them all.

Inputting PNG sequences would be nice too.

Few issues during the installation

Only this steps do not work on Windows:

conda create -n tooncrafter python=3.8.5
conda activate tooncrafter
pip install -r requirements.txt

• Install Visual Studio Installer -> Individual Components -> MSVC v141 - VS 2017 C++ x64/x86 build tools (v14.16)

  conda create -n tooncrafter python=3.8.5
  conda activate tooncrafter

  pip install -r requirements.txt
  

...

I am not sure of why the short brief instructions are for...

raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'") AttributeError: 'VisionTransformer' object has no attribute 'input_patchnorm'

start: smiling 2024-06-09 20:45:35
Traceback (most recent call last):
File "C:\Users\ssk90\AppData\Local\Programs\Python\Python310\lib\site-packages\gradio\routes.py", line 384, in run_predict
output = await app.get_blocks().process_api(
File "C:\Users\ssk90\AppData\Local\Programs\Python\Python310\lib\site-packages\gradio\blocks.py", line 1032, in process_api
result = await self.call_function(
File "C:\Users\ssk90\AppData\Local\Programs\Python\Python310\lib\site-packages\gradio\blocks.py", line 844, in call_function
prediction = await anyio.to_thread.run_sync(
File "C:\Users\ssk90\AppData\Local\Programs\Python\Python310\lib\site-packages\anyio\to_thread.py", line 31, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "C:\Users\ssk90\AppData\Local\Programs\Python\Python310\lib\site-packages\anyio_backends_asyncio.py", line 937, in run_sync_in_worker_thread
return await future
File "C:\Users\ssk90\AppData\Local\Programs\Python\Python310\lib\site-packages\anyio_backends_asyncio.py", line 867, in run
result = context.run(func, *args)
File "C:\SD U V5.1\ToonCrafter\ToonCrafter-fp16\scripts\gradio\i2v_test_application.py", line 94, in get_image
cond_images = model.embedder(img_tensor.unsqueeze(0)) ## blc
File "C:\Users\ssk90\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\nn\modules\module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "C:\Users\ssk90\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\nn\modules\module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
File "C:\SD U V5.1\ToonCrafter\ToonCrafter-fp16\lvdm\modules\encoders\condition.py", line 339, in forward
z = self.encode_with_vision_transformer(image)
File "C:\SD U V5.1\ToonCrafter\ToonCrafter-fp16\lvdm\modules\encoders\condition.py", line 346, in encode_with_vision_transformer
if self.model.visual.input_patchnorm:
File "C:\Users\ssk90\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\nn\modules\module.py", line 1709, in getattr
raise AttributeError(f"'{type(self).name}' object has no attribute '{name}'")
AttributeError: 'VisionTransformer' object has no attribute 'input_patchnorm'

I tried Updating torch 2.3.1+cu118, torchvision 0.18.1+cu118, open-clip-torch 2.24.0 it...but the same issues is same...😭😭