khaliloulahdieng / motionctrl

Open source project contribution

MotionCtrl: A Unified and Flexible Motion Controller for Video Generation

     

---

🔥🔥 This is an official implementation of MotionCtrl: A Unified and Flexible Motion Controller for Video Generation, which is capable of independently controlling the complex camera motion and object motion of the generated videos, with only a unified model. There are some results attained with MotionCtrl and more results are showcased in our Project Page.

---

📝 Changelog

• 20231225: Release MotionCtrl deployed on LVDM/VideoCrafter.
• 20231225: Gradio Demo Available.
• 20231228: Provide local gradio demo for convenience.
• Release MotionCtrl deployed on AnimateDiff.
• Release MotionCtrl deployed on SVD.

---

⚙️ Environment

conda create -n motionctrl python=3.10.6
conda activate motionctrl
pip install -r requirements.txt

💫 Inference

• Run local inference script

1. Download the weights of MotionCtrl motionctrl.pth and put it to ./checkpoints.
2. Go into configs/inference/run.sh and set condtype as 'camera_motion', 'object_motion', or 'both'.

• condtype=camera_motion means only control the camera motion in the generated video.
• condtype=object_motion means only control the object motion in the generated video.
• condtype=both means control the camera motion and object motion in the generated video simultaneously.

3. Running scripts: sh configs/inference/run.sh

• Run local gradio demo

  python -m app --share
  

📚 Citation

If you make use of our work, please cite our paper.

@inproceedings{wang2023motionctrl,
  title={MotionCtrl: A Unified and Flexible Motion Controller for Video Generation},
  author={Wang, Zhouxia and Yuan, Ziyang and Wang, Xintao and Chen, Tianshui and Xia, Menghan and Luo, Ping and Shan, Yin},
  booktitle={arXiv preprint arXiv:2312.03641},
  year={2023}
}

🤗 Acknowledgment

The current version of MotionCtrl is built on VideoCrafter. We appreciate the authors for sharing their awesome codebase.

❓ Contact

For any question, feel free to email [email protected] or [email protected].