hehao13 / cameractrl Goto Github PK

Do you have a script for downloading the original videos in Realestate10K? Or, after downloading videos, what should I pay attention to in order to proceed with the subsequent processing operations on the dataset?

Thank you for your sharing on awesome work.
I want to conduct experiments using SparseCtrl as shown in our teaser outputs, but your code does not offer exmple code for that. Could you provide an inference code using SparseCtrl? If so, it is really helpful us. Thank you.

File "/home/anthony/work/third/stable-diffusion-good/sd2/CameraCtrl/inference.py", line 315, in
main(args)
File "/home/anthony/work/third/stable-diffusion-good/sd2/CameraCtrl/inference.py", line 200, in main
pipeline = get_pipeline(args.ori_model_path, args.unet_subfolder, args.image_lora_rank, args.image_lora_ckpt,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/anthony/work/third/stable-diffusion-good/sd2/CameraCtrl/inference.py", line 131, in get_pipeline
unet = UNet3DConditionModelPoseCond.from_pretrained_2d(ori_model_path, subfolder=unet_subfolder,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/anthony/work/third/stable-diffusion-good/sd2/CameraCtrl/cameractrl/models/unet.py", line 805, in from_pretrained_2d
model, unused_kwargs = cls.from_config(config, return_unused_kwargs=True, **unet_additional_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/anthony/miniconda3/envs/p12/lib/python3.11/site-packages/diffusers/configuration_utils.py", line 259, in from_config
model = cls(**init_dict)
^^^^^^^^^^^^^^^^
File "/home/anthony/work/third/stable-diffusion-good/sd2/CameraCtrl/cameractrl/models/unet.py", line 894, in init
super(UNet3DConditionModelPoseCond, self).init(**kwargs)
File "/home/anthony/miniconda3/envs/p12/lib/python3.11/site-packages/diffusers/configuration_utils.py", line 653, in inner_init
init(self, *args, **init_kwargs)
File "/home/anthony/work/third/stable-diffusion-good/sd2/CameraCtrl/cameractrl/models/unet.py", line 221, in init
raise ValueError(f"unknown mid_block_type : {mid_block_type}")
ValueError: unknown mid_block_type : UNetMidBlock2DCrossAttn

The results were impressive, but detail of the object in the video is change too much, can we improve it?

Dear Authors,

Thank you for your outstanding contribution!

I was wondering if it would be possible for you to share the evaluation script that employs COLMAP, including the required parameters. Additionally, could you provide details on how you calculated the Rotational Error and Translation Error as presented in your paper? This information would greatly aid in ensuring reproducibility and would be invaluable for applying these metrics in a different yet related study.

Thanks!

What's the video length during training with a A100? Do you train the unet, cameractrl and temporal modules at the same time?

good work! Will the project support picture to video in the future?

Why HF demo shows error when I upload my own image, but the example image works fine ?