tencentarc / motionctrl Goto Github PK

Any plan to release the training code?

Thanks a lot for your excellent work!

Do you have the plan to released the trained model?

when will this great project release code?

Please help, what is the cause of this?

First of all, great work TencentARC team! We love it!

I know that this is not the repo on ComfyUI node. https://github.com/chaojie/ComfyUI-MotionCtrl but since there I can't put in a Issue I have to ask here.

I download the example workflow and I get this error. Any ideas?

Motionctrl Sample Simple:

• Failed to convert an input value to a INT value: context_overlap, https://chaojie.github.io/ComfyUI-MotionCtrl/tools/draw.html, invalid literal for int() with base 10: 'https://chaojie.github.io/ComfyUI-MotionCtrl/tools/draw.html'

Hi, I tested the demo on huggingface. My settings are: 1. control both camera and object motion; 2. the camera poses and trajectory: motion: ['Zoom Out']. speed: 1.0.; 3. trajectories is horizon_1. the prompt is: “A panda crawling on rocks”. Other parameters are default. But I got the following video, It seems that the control is not very effective.

Hi, thanks for sharing the great work! I have a little question about the camera control.

I wonder does the camera speed value have its practical meaning? say, when setting speed=0.5, what does the value 0.5 refer to? or is it just a relative indicator? Also, could you please give me some hint how to do if I want to move the camera for a certain distance, like pan right for 20 pixels? Thanks for your help :)

Hi @wzhouxiff ,
thanks for the interesting work!
I downloaded the model here https://huggingface.co/TencentARC/MotionCtrl/blob/main/motionctrl_svd.ckpt for SVD+MotionCtrl inference. All keys add up, but the results are basically noises. Do you maybe have an idea what causes this? Thanks a lot!

Looking forward to all the updates to this app

Traceback (most recent call last):
File "D:\ComfyUI_windows_portable\ComfyUI\nodes.py", line 1800, in load_custom_node
module_spec.loader.exec_module(module)
File "", line 940, in exec_module
File "", line 241, in call_with_frames_removed
File "D:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-MotionCtrl_init.py", line 1, in
from .nodes import NODE_CLASS_MAPPINGS
File "D:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-MotionCtrl\nodes.py", line 21, in
from pytorch_lightning import seed_everything
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\pytorch_lightning_init_.py", line 35, in
from pytorch_lightning.callbacks import Callback # noqa: E402
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\pytorch_lightning\callbacks_init_.py", line 14, in
from pytorch_lightning.callbacks.batch_size_finder import BatchSizeFinder
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\pytorch_lightning\callbacks\batch_size_finder.py", line 24, in
from pytorch_lightning.callbacks.callback import Callback
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\pytorch_lightning\callbacks\callback.py", line 25, in
from pytorch_lightning.utilities.types import STEP_OUTPUT
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\pytorch_lightning\utilities\types.py", line 27, in
from torchmetrics import Metric
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torchmetrics_init_.py", line 22, in
from torchmetrics import functional # noqa: E402
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torchmetrics\functional_init_.py", line 14, in
from torchmetrics.functional.audio.deprecated import permutation_invariant_training as permutation_invariant_training
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torchmetrics\functional\audio_init.py", line 14, in
from torchmetrics.functional.audio.pit import permutation_invariant_training, pit_permutate
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torchmetrics\functional\audio\pit.py", line 22, in
from torchmetrics.utilities import rank_zero_warn
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torchmetrics\utilities_init.py", line 14, in
from torchmetrics.utilities.checks import check_forward_full_state_property
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torchmetrics\utilities\checks.py", line 25, in
from torchmetrics.metric import Metric
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torchmetrics\metric.py", line 30, in
from torchmetrics.utilities.data import (
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torchmetrics\utilities\data.py", line 22, in
from torchmetrics.utilities.imports import TORCH_GREATER_EQUAL_1_12, XLA_AVAILABLE
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torchmetrics\utilities\imports.py", line 54, in
TORCHAUDIO_GREATER_EQUAL_0_10: Optional[bool] = compare_version("torchaudio", operator.ge, "0.10.0")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\lightning_utilities\core\imports.py", line 77, in compare_version
pkg = importlib.import_module(package)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "importlib_init.py", line 126, in import_module
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torchaudio_init.py", line 1, in
from torchaudio import ( # noqa: F401
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torchaudio_extension_init.py", line 43, in
_load_lib("libtorchaudio")
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torchaudio_extension\utils.py", line 61, in load_lib
torch.ops.load_library(path)
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch_ops.py", line 852, in load_library
ctypes.CDLL(path)
File "ctypes_init.py", line 376, in init
OSError: [WinError 127] 找不到指定的程序。

pls advise how to solve this. thx

Hey,

Upon examining the weights of motionctrl_svd checkpoint, found that there were no weights for omcm module.

Are you planning to release them?

Thanks!

in demo script configs/inference/run.sh, CUDA_VISIBLE_DEVICES=7 was used causing "No CUDA GPUs are available" for those who doesn't have 8 or more gpus, please consider to remove it.

Hi, I found that when using the model to generate video, there is always a very serious Rolling Shutter Distortion causing the lines to bend at each frame, is there any way to solve this problem? Thx