tencentarc / motionctrl Goto Github PK

@yxgeee @xinntao @wzhouxiff @eltociear @yeliudev
Hello, Thank you for releasing great work.
I have questions about SVD Model Inference Code.
Why do you multiply np.array([3, 1, 4]) to pose?? Can you tell me it in details?

And if I want to train the model with lower resolution(1024x576 -> 512x320), how do i modify the code?
Thank you in advance

MotionCtrl+svd是在svd模型上训练的CMCM吗

First of all, great work TencentARC team! We love it!

I know that this is not the repo on ComfyUI node. https://github.com/chaojie/ComfyUI-MotionCtrl but since there I can't put in a Issue I have to ask here.

I download the example workflow and I get this error. Any ideas?

Motionctrl Sample Simple:

• Failed to convert an input value to a INT value: context_overlap, https://chaojie.github.io/ComfyUI-MotionCtrl/tools/draw.html, invalid literal for int() with base 10: 'https://chaojie.github.io/ComfyUI-MotionCtrl/tools/draw.html'

From my understanding, CMCM digests over a set of poses represented by extrinsic matrix. But then how does it deal with scale ambiguity issue on SVD case?

in demo script configs/inference/run.sh, CUDA_VISIBLE_DEVICES=7 was used causing "No CUDA GPUs are available" for those who doesn't have 8 or more gpus, please consider to remove it.

Thanks a lot for your excellent work!

Do you have the plan to released the trained model?

Looking forward to all the updates to this app

我使用的是cuda121，torch2.2.1，xformers0.0.25，然后运行gradio之后会提示：
AttributeError: module 'xformers' has no attribute 'ops'. Did you mean: 'os'?

尝试过把torch+xformers一系列都降级，但是提示CUDA与这些版本不匹配，有没有办法不降级CUDA的情况下解决这个问题呢
（其他环境需要比较新的CUDA

具体报错如下：
1
[0. 0. 1.] [0. 0. 0.]
1
[0. 0. 1.] [0. 0. 0.]
C:\Users\admin\miniconda3\envs\motionctrl\lib\site-packages\torch\nn\functional.py:5476: UserWarning:

1Torch was not compiled with flash attention. (Triggered internally at ..\aten\src\ATen\native\transformers\cuda\sdp_utils.cpp:263.)

Traceback (most recent call last):
File "C:\Users\admin\miniconda3\envs\motionctrl\lib\site-packages\gradio\routes.py", line 439, in run_predict
output = await app.get_blocks().process_api(
File "C:\Users\admin\miniconda3\envs\motionctrl\lib\site-packages\gradio\blocks.py", line 1389, in process_api
result = await self.call_function(
File "C:\Users\admin\miniconda3\envs\motionctrl\lib\site-packages\gradio\blocks.py", line 1094, in call_function
prediction = await anyio.to_thread.run_sync(
File "C:\Users\admin\AppData\Roaming\Python\Python310\site-packages\anyio\to_thread.py", line 56, in run_sync
return await get_async_backend().run_sync_in_worker_thread(
File "C:\Users\admin\AppData\Roaming\Python\Python310\site-packages\anyio_backends_asyncio.py", line 2144, in run_sync_in_worker_thread
return await future
File "C:\Users\admin\AppData\Roaming\Python\Python310\site-packages\anyio_backends_asyncio.py", line 851, in run
result = context.run(func, *args)
File "C:\Users\admin\miniconda3\envs\motionctrl\lib\site-packages\gradio\utils.py", line 704, in wrapper
response = f(*args, **kwargs)
File "D:\AIGC\MotionCtrl-svd\app.py", line 514, in model_run
video_path = motionctrl_sample(
File "D:\AIGC\MotionCtrl-svd\gradio_utils\motionctrl_cmcm_gradio.py", line 148, in motionctrl_sample
c, uc = model.conditioner.get_unconditional_conditioning(
File "D:\AIGC\MotionCtrl-svd\sgm\modules\encoders\modules.py", line 179, in get_unconditional_conditioning
c = self(batch_c, force_cond_zero_embeddings)
File "C:\Users\admin\miniconda3\envs\motionctrl\lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "C:\Users\admin\miniconda3\envs\motionctrl\lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "D:\AIGC\MotionCtrl-svd\sgm\modules\encoders\modules.py", line 132, in forward
emb_out = embedder(batch[embedder.input_key])
File "C:\Users\admin\miniconda3\envs\motionctrl\lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "C:\Users\admin\miniconda3\envs\motionctrl\lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "D:\AIGC\MotionCtrl-svd\sgm\modules\encoders\modules.py", line 1012, in forward
out = self.encoder.encode(vid[n * n_samples : (n + 1) * n_samples])
File "D:\AIGC\MotionCtrl-svd\sgm\models\autoencoder.py", line 472, in encode
z = self.encoder(x)
File "C:\Users\admin\miniconda3\envs\motionctrl\lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "C:\Users\admin\miniconda3\envs\motionctrl\lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "D:\AIGC\MotionCtrl-svd\sgm\modules\diffusionmodules\model.py", line 596, in forward
h = self.mid.attn_1(h)
File "C:\Users\admin\miniconda3\envs\motionctrl\lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
return self.call_impl(*args, **kwargs)
File "C:\Users\admin\miniconda3\envs\motionctrl\lib\site-packages\torch\nn\modules\module.py", line 1520, in call_impl
return forward_call(*args, **kwargs)
File "D:\AIGC\MotionCtrl-svd\sgm\modules\diffusionmodules\model.py", line 265, in forward
h = self.attention(h)
File "D:\AIGC\MotionCtrl-svd\sgm\modules\diffusionmodules\model.py", line 250, in attention
out = xformers.ops.memory_efficient_attention(
AttributeError: module 'xformers' has no attribute 'ops'. Did you mean: 'os'?

@wzhouxiff Thanks for the interesting work!
When I reproduce the SVD version，the results are basically noises.

Problem consistency #13

the Demo of huggingface is a Erro

There is another problem. The prompt resolution of the gradient interface is 576x1024, but the background prompt model input resolution is 512x1024，What is the correct input resolution?

Finally, we hope to have a Dockerfile file or image to enable SVD to run completely.

Hello, i want to animate clouds on static shot with MotionCtrl on SVD. What parameters to use to achive my goal?

I tried [ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] but it generate random motion, not static camera.

@wzhouxiff @xinntao

Thank you for providing the great work.
Do you have a plan to release the training code and dataset?
Thank you in advance

Hi @wzhouxiff ,
thanks for the interesting work!
I downloaded the model here https://huggingface.co/TencentARC/MotionCtrl/blob/main/motionctrl_svd.ckpt for SVD+MotionCtrl inference. All keys add up, but the results are basically noises. Do you maybe have an idea what causes this? Thanks a lot!

Hi, I found that when using the model to generate video, there is always a very serious Rolling Shutter Distortion causing the lines to bend at each frame, is there any way to solve this problem? Thx

您好，请问在readme中展示的比较大尺度的运动的json文件有在dataset中吗？
具体是哪几个？谢谢~

Hi, thanks for sharing the great work! I have a little question about the camera control.

I wonder does the camera speed value have its practical meaning? say, when setting speed=0.5, what does the value 0.5 refer to? or is it just a relative indicator? Also, could you please give me some hint how to do if I want to move the camera for a certain distance, like pan right for 20 pixels? Thanks for your help :)

It appears that the SVD version supports only camera motion without the object trajectory. Can you provide some insight for omitting it?

May I know what is the license for the motionctrl_svd.ckpt?

Hi, thanks for this incredible work you've done. I'm wondering how many cuda vram needed to train motionctrl based on svd? Thank you!

when will this great project release code?

Traceback (most recent call last):
File "D:\ComfyUI_windows_portable\ComfyUI\nodes.py", line 1800, in load_custom_node
module_spec.loader.exec_module(module)
File "", line 940, in exec_module
File "", line 241, in call_with_frames_removed
File "D:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-MotionCtrl_init.py", line 1, in
from .nodes import NODE_CLASS_MAPPINGS
File "D:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-MotionCtrl\nodes.py", line 21, in
from pytorch_lightning import seed_everything
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\pytorch_lightning_init_.py", line 35, in
from pytorch_lightning.callbacks import Callback # noqa: E402
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\pytorch_lightning\callbacks_init_.py", line 14, in
from pytorch_lightning.callbacks.batch_size_finder import BatchSizeFinder
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\pytorch_lightning\callbacks\batch_size_finder.py", line 24, in
from pytorch_lightning.callbacks.callback import Callback
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\pytorch_lightning\callbacks\callback.py", line 25, in
from pytorch_lightning.utilities.types import STEP_OUTPUT
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\pytorch_lightning\utilities\types.py", line 27, in
from torchmetrics import Metric
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torchmetrics_init_.py", line 22, in
from torchmetrics import functional # noqa: E402
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torchmetrics\functional_init_.py", line 14, in
from torchmetrics.functional.audio.deprecated import permutation_invariant_training as permutation_invariant_training
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torchmetrics\functional\audio_init.py", line 14, in
from torchmetrics.functional.audio.pit import permutation_invariant_training, pit_permutate
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torchmetrics\functional\audio\pit.py", line 22, in
from torchmetrics.utilities import rank_zero_warn
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torchmetrics\utilities_init.py", line 14, in
from torchmetrics.utilities.checks import check_forward_full_state_property
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torchmetrics\utilities\checks.py", line 25, in
from torchmetrics.metric import Metric
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torchmetrics\metric.py", line 30, in
from torchmetrics.utilities.data import (
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torchmetrics\utilities\data.py", line 22, in
from torchmetrics.utilities.imports import TORCH_GREATER_EQUAL_1_12, XLA_AVAILABLE
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torchmetrics\utilities\imports.py", line 54, in
TORCHAUDIO_GREATER_EQUAL_0_10: Optional[bool] = compare_version("torchaudio", operator.ge, "0.10.0")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\lightning_utilities\core\imports.py", line 77, in compare_version
pkg = importlib.import_module(package)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "importlib_init.py", line 126, in import_module
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torchaudio_init.py", line 1, in
from torchaudio import ( # noqa: F401
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torchaudio_extension_init.py", line 43, in
_load_lib("libtorchaudio")
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torchaudio_extension\utils.py", line 61, in load_lib
torch.ops.load_library(path)
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch_ops.py", line 852, in load_library
ctypes.CDLL(path)
File "ctypes_init.py", line 376, in init
OSError: [WinError 127] 找不到指定的程序。

pls advise how to solve this. thx

Hi, I tested the demo on huggingface. My settings are: 1. control both camera and object motion; 2. the camera poses and trajectory: motion: ['Zoom Out']. speed: 1.0.; 3. trajectories is horizon_1. the prompt is: “A panda crawling on rocks”. Other parameters are default. But I got the following video, It seems that the control is not very effective.

Please help, what is the cause of this?

Any plan to release the training code?

Is there any way to increase the video seconds? Is it currently only possible to generate 1 second videos?

Hey,

Upon examining the weights of motionctrl_svd checkpoint, found that there were no weights for omcm module.

Are you planning to release them?

Thanks!