[CVPR 2024 Highlight] Official PyTorch implementation of CoDeF: Content Deformation Fields for Temporally Consistent Video Processing
Home Page: https://qiuyu96.github.io/CoDeF/
License: Other
I want to use my new video to train, I had prepared the environment and installed the requirement libraries. But after running the comand sh scripts/train_multi.sh, I face the problem No module named 'tinycudann', and I try to use pip/pip3/conda to install it, however it can not be found. I want to kown how to deal with it?
Hello author, your project presentation was impressive. Using your instance training set to redraw the video was great, but I trained and generated my own video with poor results. Could you please help me see what the reason is？
My training parameter settings are as follows
My picture looks like this
The content of the video is that the character is jumping and moving.
The model used after training is the last one.But the canonical imageI got is like this.
This also led to poor video effects in the subsequent rendering
I have extracted the optical flow by RAFT. Could you show me how to get the xxx.npy like those in beauty_0_flow and xxx_c.npy in beauty_0_flow_confidence?
Thank you
Thank you for doing such a great job!
I want to use controlnet for video style migration. For your example beauty_1, the step of reconstruction generates two canonical images. If I simply use the controlnet to transfer these tow images, the generated video is bad even with a same controlnet seed. May I ask which step is wrong with me？
Thank you!
Hello, may I ask how to using text guide the style?
What happens when the converted video action does not change?
Is there any supported code for this project？
/content/CoDeF/ckpts/all_sequences/scene_0/base_control
这个路径下面的图片是怎么来的?
Thanks for great work!
I notice all images in transformed video are warped by the first transformed img], aka canonical_img, which makes me little confused.
code link
I'm wondering the case like people turning around could be well handled by this field.
thanks your code, can you provide the docker image to run code?
Hello, tinycudann are really hard to install on windows, at least I failed on cuda11.8 + windows.
Would consider add vanilla pytorch in modeling so that users can use without tinycudann?
Hi，
I don't understand the meaning of this line of code. Can you help explain it？Thanks！
Line 202 in 2407bfa
by the way，is there any way that could print the model network of tinycudann？ E.g. I want to see what happened In ‘implicit_video.encoder （tcnn.Network）’
Hello! when I ran ./scripts/test_canonical.sh using My own model, the following error occurred. How can I solve this problem? Thanks~
RuntimeError: grid_sampler(): expected grid and input to have same batch size, but got input with sizes [0, 3, 1024, 1024] and grid with sizes [1, 1048576, 1, 2]
File "D:\workspace\CoDeF\train.py", line 214, in forward
results = torch.nn.functional.grid_sample(
File "C:\Users\ASUS\anaconda3\lib\site-packages\torch\nn\functional.py", line 4244, in grid_sample
return torch.grid_sampler(input, grid, mode_enum, padding_mode_enum, align_corners)
RuntimeError: grid_sampler(): expected grid and input to have same batch size, but got input with sizes [0, 3, 540, 540] and grid with sizes [1, 291600, 1, 2]
But if I config mask_dir= null， the code runs well.
Hello, this project is very good, I am very grateful to the authors for open source.
I am not a professional in artificial intelligence, but I am very interested in it.
There is a question, may I simply understand that this method is an upgraded version of the previous Ebsynth method? Maybe my cognition is relatively superficial, please forgive me.
The Ebsynth‘s site is https://ebsynth.com/
你好，
本地尝试了一些视频，效果不是很稳定，请教一下看是哪里出了问题
第一个视频，46帧，视频画面基本不变，效果还是很稳定的
https://github.com/qiuyu96/CoDeF/assets/12007227/4cceed26-5d16-4770-88b2-f35ee1bf8b9a
第二个视频，64帧，即使用了flow和masks依然效果不好，提取的canonical图片也不是很好，下边分别是提取的canonical图，基于canonical的重构结果，以及转换结果（这是没有用flow和mask的结果，用了提升也很小，参数用的默认）
https://github.com/qiuyu96/CoDeF/assets/12007227/dd128b1b-bee9-4888-8a1f-e927d5eadcf3
https://github.com/qiuyu96/CoDeF/assets/12007227/6bba232b-2f36-4d31-bd22-e5eb97ec2521
第三个视频，300帧，是第二个视频的加长版本，这次提取的canonical图片就更乱了，重构的视频水影严重，转换的视频画面也不动了（训练step用了50000）
https://github.com/qiuyu96/CoDeF/assets/12007227/2d77d41d-41b4-43b8-ae30-8404f52a27eb
总体而言，加了flow的效果会有提升，加了mask的提升不如flow，视频动作稍快一些，canonical的提取就明显的差了，请教一下如何能保持效果相对稳定一些，谢谢
跑测试的时候报错，求看下！'flows': self.all_flows[idx] if (idx<len(self.all_images)-2)&(self.all_flows is not None) else -1e5,
The index check here may be incorrect, causing the index to exceed the list range. I suggest making the following modifications:
'flows': self.all_flows[idx] if (idx < len(self.all_images) - 2) and (self.all_flows is not None) and (idx < len(self.all_flows)) else -1e5,
Does it have to be 48GB? Can my V100-32GB be used? What is the minimum GPU memory?
(codef) [root@prod-emr-gpu01 torch]# pip install git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch
Looking in indexes: http://de.mirrors.cloud.aliyuncs.com/pypi/simple/
Collecting git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch
Cloning https://github.com/NVlabs/tiny-cuda-nn/ to /tmp/pip-req-build-lyo_m2r9
Running command git clone --filter=blob:none --quiet https://github.com/NVlabs/tiny-cuda-nn/ /tmp/pip-req-build-lyo_m2r9
Resolved https://github.com/NVlabs/tiny-cuda-nn/ to commit NVlabs/tiny-cuda-nn@6f018a9
Running command git submodule update --init --recursive -q
Preparing metadata (setup.py) ... error
error: subprocess-exited-with-error
× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [22 lines of output]
/tmp/pip-req-build-lyo_m2r9/bindings/torch/setup.py:5: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
from pkg_resources import parse_version
Traceback (most recent call last):
File "", line 2, in
File "", line 34, in
File "/tmp/pip-req-build-lyo_m2r9/bindings/torch/setup.py", line 11, in
from torch.utils.cpp_extension import BuildExtension, CUDAExtension
File "/data/miniconda3/envs/codef/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 19, in
from .hipify import hipify_python
File "/data/miniconda3/envs/codef/lib/python3.9/site-packages/torch/utils/hipify/hipify_python.py", line 34, in
from .cuda_to_hip_mappings import CUDA_TO_HIP_MAPPINGS
File "/data/miniconda3/envs/codef/lib/python3.9/site-packages/torch/utils/hipify/cuda_to_hip_mappings.py", line 34, in
rocm_path = subprocess.check_output(["hipconfig", "--rocmpath"]).decode("utf-8")
File "/data/miniconda3/envs/codef/lib/python3.9/subprocess.py", line 424, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
File "/data/miniconda3/envs/codef/lib/python3.9/subprocess.py", line 505, in run
with Popen(*popenargs, **kwargs) as process:
File "/data/miniconda3/envs/codef/lib/python3.9/subprocess.py", line 951, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "/data/miniconda3/envs/codef/lib/python3.9/subprocess.py", line 1837, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
NotADirectoryError: [Errno 20] Not a directory: 'hipconfig'
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed
× Encountered error while generating package metadata.
╰─> See above for output.
note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
(codef) [root@prod-emr-gpu01 torch]# python
Python 3.9.17 | packaged by conda-forge | (main, Aug 10 2023, 07:02:31)
[GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
import torch
print(torch.version)
2.0.1+cu117
print(torch.cuda.is_available())
True
If I train my own model（my video）, do I only need one image sequence?
how to train myself videos
about base_control （by ControlNet v1.1 ues which meath: depth?Openpose?SoftEdge or another)
Hi! I'm new to this, i only used notebooks in google colab before. How can i use a code there?
您好！我已经在英伟达显卡的服务器上构建好本项目，而且也跑了./scripts/test_multi.sh和./scripts/test_canonical.sh可以看到结果与预期的效果是一致的，但是我不知道如何用我自己的视频来做这个风格转换。请问用我自己的视频来做风格转换需要哪些步骤？谢谢。
it's really a very good job, but i have some problems.I want to do a video translation of a dancing video, but the canonical image is very bad, as follows:
I wonder if my hyperparameters are set incorrectly? Or is it slightly less effective for videos with a faster rate of change?
Thanks for the project ❤️ I made a colab. 🥳 I hope you like it. https://github.com/camenduru/CoDeF-colab
Hi, I test reconstruction using this video, though reconstruction is ok, but I get strange canonical image as follows
Hi, thanks for your great work. Just curious which model you used to extract optical flow used in this system?
How to adjust the config to make the training faster if large GPU memory is available? Thanks!
After running “./scripts/test_multi.sh”，I didn't find the reconstructed videos in “results/all_sequences/{NAME}/{EXP_NAME}”.
I need help.
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
/root/miniconda3/lib/python3.10/site-packages/torch/utils/tensorboard/init.py:4: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
if not hasattr(tensorboard, "version") or LooseVersion(
/root/miniconda3/lib/python3.10/site-packages/torch/utils/tensorboard/init.py:6: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
) < LooseVersion("1.15"):
/root/miniconda3/lib/python3.10/site-packages/pytorch_lightning/utilities/data.py:103: UserWarning: Total length of DataLoader across ranks is zero. Please make sure this was your intention.
rank_zero_warn(
TEST Profiler Report
In paper, the formula of flow loss is
and the paper introduce the
However, the paper also mention
Corresponding points identified by flows with high confidence should be the same points in the canonical field
which fits my intuition, and I don't know why we subtract the optical flow term here?
In code I also don't find this optical flow term:
`# Optical flow loss.
if ret.flow_loss[0] != 0:
mk_flow_t = torch.logical_and(mk_t, flows[0].sum(dim=-1)< 3)
loss = loss + torch.nn.functional.l1_loss(
ret.flow_loss[i][0][mk_flow_t], ret.flow_loss[i][1]
[mk_flow_t]) * self.hparams.flow_loss
`
Can you please explain why this optical flow term is in the formula? Thanks.
Hola mundo!!!
In equation (5) you mention w as a function of j,k but only j appears in the equation:
So how k plays a role in it?
Hello! when I ran ./scripts/test_multi.sh using the provided data, the following error occurred. How can I solve this problem? Thanks~
You are using a CUDA device ('NVIDIA GeForce RTX 4080') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
/home/ft/.conda/envs/VIST/lib/python3.8/site-packages/torch/utils/tensorboard/__init__.py:4: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
if not hasattr(tensorboard, "__version__") or LooseVersion(
/home/ft/.conda/envs/VIST/lib/python3.8/site-packages/torch/utils/tensorboard/__init__.py:6: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
) < LooseVersion("1.15"):
Testing DataLoader 0: 1%|▏ | 1/99 [00:00<00:11, 8.50it/s]/home/ft/.conda/envs/VIST/lib/python3.8/site-packages/skvideo/io/ffmpeg.py:466: DeprecationWarning: tostring() is deprecated. Use tobytes() instead.
self._proc.stdin.write(vid.tostring())
Traceback (most recent call last):
File "/home/ft/.conda/envs/VIST/lib/python3.8/site-packages/skvideo/io/ffmpeg.py", line 466, in writeFrame
self._proc.stdin.write(vid.tostring())
BrokenPipeError: [Errno 32] Broken pipe
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "train.py", line 557, in <module>
main(hparams)
File "train.py", line 550, in main
trainer.test(system, dataloaders=system.test_dataloader())
File "/home/ft/.conda/envs/VIST/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 706, in test
return call._call_and_handle_interrupt(
File "/home/ft/.conda/envs/VIST/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 42, in _call_and_handle_interrupt
return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
File "/home/ft/.conda/envs/VIST/lib/python3.8/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 92, in launch
return function(*args, **kwargs)
File "/home/ft/.conda/envs/VIST/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 749, in _test_impl
results = self._run(model, ckpt_path=ckpt_path)
File "/home/ft/.conda/envs/VIST/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 935, in _run
results = self._run_stage()
File "/home/ft/.conda/envs/VIST/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 971, in _run_stage
return self._evaluation_loop.run()
File "/home/ft/.conda/envs/VIST/lib/python3.8/site-packages/pytorch_lightning/loops/utilities.py", line 177, in _decorator
return loop_run(self, *args, **kwargs)
File "/home/ft/.conda/envs/VIST/lib/python3.8/site-packages/pytorch_lightning/loops/evaluation_loop.py", line 115, in run
self._evaluation_step(batch, batch_idx, dataloader_idx)
File "/home/ft/.conda/envs/VIST/lib/python3.8/site-packages/pytorch_lightning/loops/evaluation_loop.py", line 375, in _evaluation_step
output = call._call_strategy_hook(trainer, hook_name, *step_kwargs.values())
File "/home/ft/.conda/envs/VIST/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 288, in _call_strategy_hook
output = fn(*args, **kwargs)
File "/home/ft/.conda/envs/VIST/lib/python3.8/site-packages/pytorch_lightning/strategies/ddp.py", line 348, in test_step
return self.model.test_step(*args, **kwargs)
File "train.py", line 489, in test_step
self.video_visualizer.add(img)
File "/home/ft/Project/CoDeF/utils/video_visualizer.py", line 95, in add
self.video.writeFrame(frame)
File "/home/ft/.conda/envs/VIST/lib/python3.8/site-packages/skvideo/io/ffmpeg.py", line 471, in writeFrame
raise IOError(msg)
OSError: [Errno 32] Broken pipe
FFMPEG COMMAND:
/home/ft/.conda/envs/VIST/bin/ffmpeg -y -f rawvideo -pix_fmt rgb24 -s 540x960 -i - -r 30.00 -s 540x960 -vcodec libx264 -crf 1 -pix_fmt yuv420p /home/ft/Project/CoDeF/results/all_sequences/scene_0/base/scene_0_base.mp4
FFMPEG STDERR OUTPUT:
Testing DataLoader 0: 1%| | 1/99 [00:00<00:47, 2.07it/s]
RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 2916000 but got size 29160000 for tensor number 1 in the list.
I just changed batchsize in yalm for 10. it happened.
./scripts/test_canonical.sh
train.py': [Errno 2] No such file or directory�
Hi, really thanks for the great work! I am new to the video processing and are confused about several places in the paper, really hope you can provide me some hints, thanks in advance :)
Really appreciate your time, looking forward to your reply.
Thanks for your work, but I don't have linux environment.
Hi, I have some question when using a custom dataset. I completed training well and got a checkpoint, but the following Broken pipe error occurred during the test. How can I solve it?
thanks for your great work!
May I ask how long video length this method can support？
May I ask if you will publish a document about training hyperparameter settings. I found that many people are asking about long videos and fast exercise training skills. I saw your reply, but we don’t know how to set them up
and I found this question link crashed, here is why
#39
pkg_resources.declare_namespace('lightning_fabric').pkg_resources.declare_namespace. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packagespkg_resources.declare_namespace('pytorch_lightning').pkg_resources.declare_namespace. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packagesYou are using a CUDA device ('NVIDIA A10') that has Tensor Cores. To properly utilize them, you should set torch.set_float32_matmul_precision('medium' | 'high') which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
/data/anaconda3/envs/codef/lib/python3.10/site-packages/torch/utils/tensorboard/init.py:4: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
if not hasattr(tensorboard, "version") or LooseVersion(
/data/anaconda3/envs/codef/lib/python3.10/site-packages/torch/utils/tensorboard/init.py:6: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
) < LooseVersion("1.15"):
Testing DataLoader 0: 1%|? | 1/99 [00:00<00:16, 5.84it/s]/data/anaconda3/envs/codef/lib/python3.10/site-packages/skvideo/io/ffmpeg.py:466: DeprecationWarning: tostring() is deprecated. Use tobytes() instead.
self._proc.stdin.write(vid.tostring())
Traceback (most recent call last):
File "/data/anaconda3/envs/codef/lib/python3.10/site-packages/skvideo/io/ffmpeg.py", line 466, in writeFrame
self._proc.stdin.write(vid.tostring())
BrokenPipeError: [Errno 32] Broken pipe
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/data/CoDeF/train.py", line 557, in
main(hparams)
File "/data/CoDeF/train.py", line 550, in main
trainer.test(system, dataloaders=system.test_dataloader())
File "/data/anaconda3/envs/codef/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 706, in test
return call._call_and_handle_interrupt(
File "/data/anaconda3/envs/codef/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 42, in _call_and_handle_interrupt
return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
File "/data/anaconda3/envs/codef/lib/python3.10/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 92, in launch
return function(*args, **kwargs)
File "/data/anaconda3/envs/codef/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 749, in _test_impl
results = self._run(model, ckpt_path=ckpt_path)
File "/data/anaconda3/envs/codef/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 935, in _run
results = self._run_stage()
File "/data/anaconda3/envs/codef/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 971, in _run_stage
return self._evaluation_loop.run()
File "/data/anaconda3/envs/codef/lib/python3.10/site-packages/pytorch_lightning/loops/utilities.py", line 177, in _decorator
return loop_run(self, *args, **kwargs)
File "/data/anaconda3/envs/codef/lib/python3.10/site-packages/pytorch_lightning/loops/evaluation_loop.py", line 115, in run
self._evaluation_step(batch, batch_idx, dataloader_idx)
File "/data/anaconda3/envs/codef/lib/python3.10/site-packages/pytorch_lightning/loops/evaluation_loop.py", line 375, in _evaluation_step
output = call._call_strategy_hook(trainer, hook_name, *step_kwargs.values())
File "/data/anaconda3/envs/codef/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 288, in _call_strategy_hook
output = fn(*args, **kwargs)
File "/data/anaconda3/envs/codef/lib/python3.10/site-packages/pytorch_lightning/strategies/ddp.py", line 348, in test_step
return self.model.test_step(*args, **kwargs)
File "/data/CoDeF/train.py", line 489, in test_step
self.video_visualizer.add(img)
File "/data/CoDeF/utils/video_visualizer.py", line 95, in add
self.video.writeFrame(frame)
File "/data/anaconda3/envs/codef/lib/python3.10/site-packages/skvideo/io/ffmpeg.py", line 471, in writeFrame
raise IOError(msg)
OSError: [Errno 32] Broken pipe
FFMPEG COMMAND:
/usr/local/bin/ffmpeg -y -f rawvideo -pix_fmt rgb24 -s 540x960 -i - -r 30.00 -s 540x960 -vcodec libx264 -crf 1 -pix_fmt yuv420p /data/CoDeF/results/all_sequences/scene_0/base/scene_0_base.mp4
FFMPEG STDERR OUTPUT:
Testing DataLoader 0: 1%| | 1/99 [00:00<00:42, 2.31it/s]
太难了，全是报错，已经装一周了还没装好
Bro Visit my iWeb Repo, i've made a custom homepage for browsers bro. star my repo please.
https://github.com/2k-Aakaash/iWeb
Hi,
I followed the steps here after training to test the reconstruction: https://github.com/qiuyu96/CoDeF#test-reconstruction-
I made sure to update the script to load the model checkpoint, but the output reconstructed frames and canonical image are entirely black with all 0's. Any idea on why this might be happening?
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.