rese1f / stablevideo

[ICCV 2023] StableVideo: Text-driven Consistency-aware Diffusion Video Editing

I am a master student of Information Processing Lab at University of Washington. I am currently working on embodied agent and video understanding. Have a look at my homepage for more details.

When I am not doing research, I like photography, traveling, and singing.

---

My GPTs:

• Academic Paper Writing Assistant: For AI academic papers.
• Paper Search Engine: Expert in latest academic paper search and summary.

---

Updates:

• 03/2024: One paper accepted to ICLR 2024 workshop at LLM Agents.
• 02/2024: Two papers accepted to CVPR 2024.
• 02/2024: Invited talk at AAAI 2024 workshop at IMAGEOMICS.
• 12/2023: One paper accepted to ICASSP 2024.
• 12/2023: One paper accepted to AAAI 2024.
• 11/2023: Two papers accepted to WACV 2024 and its workshop at CV4Smalls.
• 09/2023: One paper accepted to ICCV 2023 workshop at TNGCV-DataComp.
• 09/2023: One paper accepted to IEEE T-MM.
• 08/2023: One paper accepted to BMVC 2023.
• 07/2023: Two papers accepted to ACM MM 2023.
• 07/2023: Finished my research internship at Microsoft Research Asia (MSRA), Beijing.
• 07/2023: Two papers accepted to ICCV 2023.

stablevideo's Issues

Do we need foreground target mask to obtain key frames?

I woder how to train the model?

I want to modify the ControlNet condition. But I have no idea how to train the guidance diffusion.

`dpt_hybrid-midas-501f0c75.pt` was in the wrong position in readme.md

In Currently readme.md ,the origin project structure was wrong

StableVideo
├── ...
├── ckpt
│   ├── cldm_v15.yaml
|   ├── dpt_hybrid-midas-501f0c75.pt
│   ├── control_sd15_canny.pth
│   └── control_sd15_depth.pth
├── data
│   └── car-turn
│       ├── checkpoint # NLA models are stored here
│       ├── car-turn # contains video frames
│       ├── ...
│   ├── blackswan
│   ├── ...
└── ...

the correct project structure should be like

StableVideo
├── ...
├── annotator
│   └── ckpts
│       ├── cdpt_hybrid-midas-501f0c75.pt
├── ckpt
│   ├── cldm_v15.yaml
│   ├── control_sd15_canny.pth
│   └── control_sd15_depth.pth
├── data
│   └── car-turn
│       ├── checkpoint # NLA models are stored here
│       ├── car-turn # contains video frames
│       ├── ...
│   ├── blackswan
│   ├── ...
└── ...

---

Besides, in case people need to install the cdpt_hybrid-midas-501f0c75.pt from huggingfacehub , we have to install these packages below

pip install tb-nightly
pip install basicsr 

Lot of issues making this work. Now: EOFError: Ran out of input

Hi,

I'm not a pro dev and I have to say that the readme is quite ambiguous for example how to install Controlnet, just some generic link where it doesn't tell much. The the other link to a github repo for training with your videos. Nothing really explaining of how to do it.

It would be great to appreciate your work if it could be more clear how to run it.

I'm stuck on running the python app.py. I get an error that tried to solve but nothing seems to work. This is the error message:

... Setting up MemoryEfficientCrossAttention. Query dim is 1280, context_dim is 768 and using 8 heads. Loaded model config from [ckpt/cldm_v15.yaml] Traceback (most recent call last): File "C:\Users\zaesa\StableVideo\app.py", line 358, in <module> stablevideo.load_canny_model() File "C:\Users\zaesa\StableVideo\app.py", line 51, in load_canny_model canny_model.load_state_dict(load_state_dict(canny_model_cfg, location='cuda'), strict=False) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\zaesa\StableVideo\cldm\model.py", line 18, in load_state_dict state_dict = get_state_dict(torch.load(ckpt_path, map_location=torch.device(location))) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\zaesa\anaconda3\envs\stablevideo\Lib\site-packages\torch\serialization.py", line 815, in load return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\zaesa\anaconda3\envs\stablevideo\Lib\site-packages\torch\serialization.py", line 1033, in _legacy_load magic_number = pickle_module.load(f, **pickle_load_args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ EOFError: Ran out of input

Any help will be really appreciated. And if you can please make more clear the readme. It isn't really done to help people test the code.

Thanks

AttributeError: 'torch.dtype' object has no attribute 'char'

Model has 264706 params
Model has 133122 params
Model has 416379 params
Model has 416379 params
Model has 402945 params
C:\Users\admin\anaconda3\envs\stablevideo\lib\site-packages\gradio\components\video.py:334: UserWarning: Video does not have browser-compatible container or codec. Converting to mp4
warnings.warn(
Error converting video to browser-playable format ffmpeg -y -loglevel quiet -i C:\Users\admin\AppData\Local\Temp\tmpwoitac9w data\bear\bear.mp4 exited with status 1

STDOUT:

STDERR:

Traceback (most recent call last):
File "C:\Users\admin\anaconda3\envs\stablevideo\lib\site-packages\gradio\routes.py", line 488, in run_predict
output = await app.get_blocks().process_api(
File "C:\Users\admin\anaconda3\envs\stablevideo\lib\site-packages\gradio\blocks.py", line 1435, in process_api
result = await self.call_function(
File "C:\Users\admin\anaconda3\envs\stablevideo\lib\site-packages\gradio\blocks.py", line 1107, in call_function
prediction = await anyio.to_thread.run_sync(
File "C:\Users\admin\anaconda3\envs\stablevideo\lib\site-packages\anyio\to_thread.py", line 33, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "C:\Users\admin\anaconda3\envs\stablevideo\lib\site-packages\anyio_backends_asyncio.py", line 877, in run_sync_in_worker_thread
return await future
File "C:\Users\admin\anaconda3\envs\stablevideo\lib\site-packages\anyio_backends_asyncio.py", line 807, in run
result = context.run(func, *args)
File "C:\Users\admin\anaconda3\envs\stablevideo\lib\site-packages\gradio\utils.py", line 707, in wrapper
response = f(*args, **kwargs)
File "C:\Users\admin\anaconda3\envs\stablevideo\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "F:\StableVideo\app.py", line 344, in render
imageio.mimwrite(save_name, (255 * output_video.detach().cpu()).to(torch.uint8).permute(0, 2, 3, 1))
File "C:\Users\admin\anaconda3\envs\stablevideo\lib\site-packages\imageio\v2.py", line 495, in mimwrite
return file.write(ims, is_batch=True, **kwargs)
File "C:\Users\admin\anaconda3\envs\stablevideo\lib\site-packages\imageio\plugins\tifffile_v3.py", line 244, in write
self._fh.write(image, **kwargs)
File "C:\Users\admin\anaconda3\envs\stablevideo\lib\site-packages\tifffile\tifffile.py", line 1946, in write
data, byteorder + data.dtype.char, 'C'
AttributeError: 'torch.dtype' object has no attribute 'char'
Traceback (most recent call last):
File "C:\Users\admin\anaconda3\envs\stablevideo\lib\site-packages\gradio\routes.py", line 488, in run_predict
output = await app.get_blocks().process_api(
File "C:\Users\admin\anaconda3\envs\stablevideo\lib\site-packages\gradio\blocks.py", line 1435, in process_api
result = await self.call_function(
File "C:\Users\admin\anaconda3\envs\stablevideo\lib\site-packages\gradio\blocks.py", line 1107, in call_function
prediction = await anyio.to_thread.run_sync(
File "C:\Users\admin\anaconda3\envs\stablevideo\lib\site-packages\anyio\to_thread.py", line 33, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "C:\Users\admin\anaconda3\envs\stablevideo\lib\site-packages\anyio_backends_asyncio.py", line 877, in run_sync_in_worker_thread
return await future
File "C:\Users\admin\anaconda3\envs\stablevideo\lib\site-packages\anyio_backends_asyncio.py", line 807, in run
result = context.run(func, *args)
File "C:\Users\admin\anaconda3\envs\stablevideo\lib\site-packages\gradio\utils.py", line 707, in wrapper
response = f(*args, **kwargs)
File "C:\Users\admin\anaconda3\envs\stablevideo\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "F:\StableVideo\app.py", line 344, in render
imageio.mimwrite(save_name, (255 * output_video.detach().cpu()).to(torch.uint8).permute(0, 2, 3, 1))
File "C:\Users\admin\anaconda3\envs\stablevideo\lib\site-packages\imageio\v2.py", line 495, in mimwrite
return file.write(ims, is_batch=True, **kwargs)
File "C:\Users\admin\anaconda3\envs\stablevideo\lib\site-packages\imageio\plugins\tifffile_v3.py", line 244, in write
self._fh.write(image, **kwargs)
File "C:\Users\admin\anaconda3\envs\stablevideo\lib\site-packages\tifffile\tifffile.py", line 1946, in write
data, byteorder + data.dtype.char, 'C'
AttributeError: 'torch.dtype' object has no attribute 'char'
Traceback (most recent call last):
File "C:\Users\admin\anaconda3\envs\stablevideo\lib\site-packages\gradio\routes.py", line 488, in run_predict
output = await app.get_blocks().process_api(
File "C:\Users\admin\anaconda3\envs\stablevideo\lib\site-packages\gradio\blocks.py", line 1435, in process_api
result = await self.call_function(
File "C:\Users\admin\anaconda3\envs\stablevideo\lib\site-packages\gradio\blocks.py", line 1107, in call_function
prediction = await anyio.to_thread.run_sync(
File "C:\Users\admin\anaconda3\envs\stablevideo\lib\site-packages\anyio\to_thread.py", line 33, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "C:\Users\admin\anaconda3\envs\stablevideo\lib\site-packages\anyio_backends_asyncio.py", line 877, in run_sync_in_worker_thread
return await future
File "C:\Users\admin\anaconda3\envs\stablevideo\lib\site-packages\anyio_backends_asyncio.py", line 807, in run
result = context.run(func, *args)
File "C:\Users\admin\anaconda3\envs\stablevideo\lib\site-packages\gradio\utils.py", line 707, in wrapper
response = f(*args, **kwargs)
File "C:\Users\admin\anaconda3\envs\stablevideo\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "F:\StableVideo\app.py", line 344, in render
imageio.mimwrite(save_name, (255 * output_video.detach().cpu()).to(torch.uint8).permute(0, 2, 3, 1))
File "C:\Users\admin\anaconda3\envs\stablevideo\lib\site-packages\imageio\v2.py", line 495, in mimwrite
return file.write(ims, is_batch=True, **kwargs)
File "C:\Users\admin\anaconda3\envs\stablevideo\lib\site-packages\imageio\plugins\tifffile_v3.py", line 244, in write
self._fh.write(image, **kwargs)
File "C:\Users\admin\anaconda3\envs\stablevideo\lib\site-packages\tifffile\tifffile.py", line 1946, in write
data, byteorder + data.dtype.char, 'C'
AttributeError: 'torch.dtype' object has no attribute 'char'
Traceback (most recent call last):
File "C:\Users\admin\anaconda3\envs\stablevideo\lib\site-packages\gradio\routes.py", line 488, in run_predict
output = await app.get_blocks().process_api(
File "C:\Users\admin\anaconda3\envs\stablevideo\lib\site-packages\gradio\blocks.py", line 1435, in process_api
result = await self.call_function(
File "C:\Users\admin\anaconda3\envs\stablevideo\lib\site-packages\gradio\blocks.py", line 1107, in call_function
prediction = await anyio.to_thread.run_sync(
File "C:\Users\admin\anaconda3\envs\stablevideo\lib\site-packages\anyio\to_thread.py", line 33, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "C:\Users\admin\anaconda3\envs\stablevideo\lib\site-packages\anyio_backends_asyncio.py", line 877, in run_sync_in_worker_thread
return await future
File "C:\Users\admin\anaconda3\envs\stablevideo\lib\site-packages\anyio_backends_asyncio.py", line 807, in run
result = context.run(func, *args)
File "C:\Users\admin\anaconda3\envs\stablevideo\lib\site-packages\gradio\utils.py", line 707, in wrapper
response = f(*args, **kwargs)
File "C:\Users\admin\anaconda3\envs\stablevideo\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "F:\StableVideo\app.py", line 344, in render
imageio.mimwrite(save_name, (255 * output_video.detach().cpu()).to(torch.uint8).permute(0, 2, 3, 1))
File "C:\Users\admin\anaconda3\envs\stablevideo\lib\site-packages\imageio\v2.py", line 495, in mimwrite
return file.write(ims, is_batch=True, **kwargs)
File "C:\Users\admin\anaconda3\envs\stablevideo\lib\site-packages\imageio\plugins\tifffile_v3.py", line 244, in write
self._fh.write(image, **kwargs)
File "C:\Users\admin\anaconda3\envs\stablevideo\lib\site-packages\tifffile\tifffile.py", line 1946, in write
data, byteorder + data.dtype.char, 'C'
AttributeError: 'torch.dtype' object has no attribute 'char'
Traceback (most recent call last):
File "C:\Users\admin\anaconda3\envs\stablevideo\lib\site-packages\gradio\routes.py", line 488, in run_predict
output = await app.get_blocks().process_api(
File "C:\Users\admin\anaconda3\envs\stablevideo\lib\site-packages\gradio\blocks.py", line 1435, in process_api
result = await self.call_function(
File "C:\Users\admin\anaconda3\envs\stablevideo\lib\site-packages\gradio\blocks.py", line 1107, in call_function
prediction = await anyio.to_thread.run_sync(
File "C:\Users\admin\anaconda3\envs\stablevideo\lib\site-packages\anyio\to_thread.py", line 33, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "C:\Users\admin\anaconda3\envs\stablevideo\lib\site-packages\anyio_backends_asyncio.py", line 877, in run_sync_in_worker_thread
return await future
File "C:\Users\admin\anaconda3\envs\stablevideo\lib\site-packages\anyio_backends_asyncio.py", line 807, in run
result = context.run(func, *args)
File "C:\Users\admin\anaconda3\envs\stablevideo\lib\site-packages\gradio\utils.py", line 707, in wrapper
response = f(*args, **kwargs)
File "C:\Users\admin\anaconda3\envs\stablevideo\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "F:\StableVideo\app.py", line 344, in render
imageio.mimwrite(save_name, (255 * output_video.detach().cpu()).to(torch.uint8).permute(0, 2, 3, 1))
File "C:\Users\admin\anaconda3\envs\stablevideo\lib\site-packages\imageio\v2.py", line 495, in mimwrite
return file.write(ims, is_batch=True, **kwargs)
File "C:\Users\admin\anaconda3\envs\stablevideo\lib\site-packages\imageio\plugins\tifffile_v3.py", line 244, in write
self._fh.write(image, **kwargs)
File "C:\Users\admin\anaconda3\envs\stablevideo\lib\site-packages\tifffile\tifffile.py", line 1946, in write
data, byteorder + data.dtype.char, 'C'
AttributeError: 'torch.dtype' object has no attribute 'char'

how to click the render botton without a GUI?

Since I haven't enough memory to run the app, I deploy the project into server, which is linked by SSH, without a GUI(dispatch by slurm). In the case, how I get the result just in commandline?

RuntimeError: The size of tensor a (998) must match the size of tensor b (992) at non-singleton dimension 3

when I ran the "black swan" demo, and it has some problem occurs, [hers is the error message:],how to resolve it?
Traceback (most recent call last):
File "/home/pymo/miniconda3/envs/stablevideo/lib/python3.11/site-packages/gradio/routes.py", line 488, in run_predict
output = await app.get_blocks().process_api(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/pymo/miniconda3/envs/stablevideo/lib/python3.11/site-packages/gradio/blocks.py", line 1431, in process_api
result = await self.call_function(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/pymo/miniconda3/envs/stablevideo/lib/python3.11/site-packages/gradio/blocks.py", line 1109, in call_function
prediction = await anyio.to_thread.run_sync(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/pymo/miniconda3/envs/stablevideo/lib/python3.11/site-packages/anyio/to_thread.py", line 33, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/pymo/miniconda3/envs/stablevideo/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
return await future
^^^^^^^^^^^^
File "/home/pymo/miniconda3/envs/stablevideo/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 807, in run
result = context.run(func, *args)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/pymo/miniconda3/envs/stablevideo/lib/python3.11/site-packages/gradio/utils.py", line 706, in wrapper
response = f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/home/pymo/miniconda3/envs/stablevideo/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/pymo/StableVideo/app.py", line 300, in render
f_atlas = f_atlas * (1 - mask) + f_atlas_origin * mask
~~~~~~~~~~~~~~~^~~~~~
RuntimeError: The size of tensor a (998) must match the size of tensor b (992) at non-singleton dimension 3

Errors in the CPU only version and HuggingFace demo

The CPU only version errors out on running as the code still believes it is running on a CUDA compatible device and not on the CPU.

Similarly, on opening the HuggingFace demo, I get a runtime error as memory has been exceeded with the error logs showing the same errors as those I get while running the CPU only version.

why does every video need to train a new model?

why does every video need to train a new model?

TypeError: Cannot interpret 'torch.float32' as a data type

While running the Gradio app using the default repository configuration and data provided, the TypeError: Cannot interpret 'torch.float32' as a data type arises when loading the video.
What could possibly be the reason based off the screenshot attached?

Thanks for sharing, Would it run on Colab?

I've not tried it yet but jut would love to hear if it runs on Free Colab (T4 - 16GB VRAM)

PytorchStreamReader failed reading zip archive: failed finding central directory

Loaded model config from [ckpt/cldm_v15.yaml]
Loaded state_dict from [ckpt/control_sd15_canny.pth]
/home/xufang/ProgramFiles/anaconda3/envs/stablevideo/lib/python3.11/site-packages/timm/models/_factory.py:114: UserWarning: Mapping deprecated model name vit_base_resnet50_384 to current vit_base_r50_s16_384.orig_in21k_ft_in1k.
model = create_fn(
Traceback (most recent call last):
File "/home/xufang/StableVideo/app.py", line 359, in
stablevideo.load_depth_model()
File "/home/xufang/StableVideo/app.py", line 60, in load_depth_model
self.apply_midas = MidasDetector()
^^^^^^^^^^^^^^^
File "/home/xufang/StableVideo/annotator/midas/init.py", line 11, in init
self.model = MiDaSInference(model_type="dpt_hybrid").cuda()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/xufang/StableVideo/annotator/midas/api.py", line 161, in init
model, _ = load_model(model_type)
^^^^^^^^^^^^^^^^^^^^^^
File "/home/xufang/StableVideo/annotator/midas/api.py", line 96, in load_model
model = DPTDepthModel(
^^^^^^^^^^^^^^
File "/home/xufang/StableVideo/annotator/midas/midas/dpt_depth.py", line 105, in init
self.load(path)
File "/home/xufang/StableVideo/annotator/midas/midas/base_model.py", line 11, in load
parameters = torch.load(path, map_location=torch.device('cpu'))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/xufang/ProgramFiles/anaconda3/envs/stablevideo/lib/python3.11/site-packages/torch/serialization.py", line 993, in load
with _open_zipfile_reader(opened_file) as opened_zipfile:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/xufang/ProgramFiles/anaconda3/envs/stablevideo/lib/python3.11/site-packages/torch/serialization.py", line 447, in init
super().init(torch._C.PyTorchFileReader(name_or_buffer))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: PytorchStreamReader failed reading zip archive: failed finding central directory

I encountered a problem when loading from control_sd15_depth.pth. It seems there is something wrong with the function MidasDetector(). Because when I replace it with CannyDetector() the program can run but cannot generate background editing video. how to fix it?

HTTPx Timeout issue

On running repo with the default configuration, we get the following exception. It doesn't interrupt the working yet it's raised.

How to extract video frames?

Thanks for the great work.
When editing a new video based on NLA models, we should first extract the frames within a video and save it into a video_name/frames_xxx.jpg.
I want to know how to extract video frames and if there are any fixed frame numbers for a video.
It would be better if the authors could provide the related pre-processed code.

prompt template not found

prompt_template.md is not found

ERROR: Could not find a version that satisfies the requirement triton==2.0.0 (from versions: none) ERROR: No matching distribution found for triton==2.0.0

ERROR: Could not find a version that satisfies the requirement triton==2.0.0 (from versions: none)
ERROR: No matching distribution found for triton==2.0.0

Poorly Written requirements.txt

The provided requirements.txt for the StableVideo model lacks crucial dependencies and contains incorrect version specifications, leading to installation failures. Many essential packages required for the model's proper functioning are missing, and users are forced to manually search and install them, along with downgrading certain dependencies to make the model run successfully.

Does every video need to be trained before editing?

Use a 15s 1080p video as a benchmark and a 3090 24G graphics card to train the video model about how long (amp+cpu version)?

ERROR: Could not find a version that satisfies the requirement nvidia-nccl-cu11==2.14.3 (from versions: 0.0.1.dev5) ERROR: No matching distribution found for nvidia-nccl-cu11==2.14.3

Hello, when I execute

pip install -r requirements.txt

I encountered the following error:

ERROR: Could not find a version that satisfies the requirement nvidia-nccl-cu11==2.14.3 (from versions: 0.0.1.dev5)
ERROR: No matching distribution found for nvidia-nccl-cu11==2.14.3

My environment is Windows with Miniconda. Please help me, thank you!

For custom dataset

Hello, Thank you for awesome idea.
Can I know how to train and infer about custom datasets?

The background is always black

@rese1f , Thanks for your nice work!
But I met a hating problem.
I have trained my own video using the method of NLA, but when I made video style transfer using this work, the background of foreground is always black, which results in the render video's background is also
black.
What could I do to get a surprising performance like your given demo video?

The bad vidoes are blow:

What is the minimum VRAM requirement?

Wanted the minimum VRAM required to run this software?