sysuyklin / freedrag

Official Implementation of FreeDrag

FreeDrag: Point Tracking is Not What You Need for Interactive Point-based Image Editing

demo.mp4

Web Demo

Official implementation of FreeDrag: Point Tracking is Not What You Need for Interactive Point-based Image Editing.

• Authors: Pengyang Ling*, Lin Chen*, Pan Zhang, Huaian Chen, Yi Jin
• Institutes: University of Science and Technology of China; Shanghai AI Laboratory
• [Paper] [Project Page] [Web Demo]

This repo proposes FreeDrag, a novel interactive point-based image editing framework free of the laborious and unstable point tracking process🔥🔥🔥.

Abstract

To serve the intricate and varied demands of image editing, precise and flexible manipulation of image content is indispensable. Recently, DragGAN has achieved impressive editing results through point-based manipulation. However, we have observed that DragGAN struggles with miss tracking, where DragGAN encounters difficulty in effectively tracking the desired handle points, and ambiguous tracking, where the tracked points are situated within other regions that bear resemblance to the handle points. To deal with the above issues, we propose FreeDrag, which adopts a feature-oriented approach to free the burden on point tracking within the point-oriented methodology of DragGAN. The FreeDrag incorporates adaptive template features, line search, and fuzzy localization techniques to perform stable and efficient point-based image editing. Extensive experiments demonstrate that our method is superior to the DragGAN and enables stable point-based editing in challenging scenarios with similar structures, fine details, or under multi-point targets.

📜 News

[2023/7/31] The web demo in OpenXLab is available now.

[2023/7/28] The function of real image editing is available now.

[2023/7/15] Code of local demo is available now!💥

[2023/7/11] The paper and project page are released!

💡 Highlights

• Local demo of FreeDrag
• Web demo of FreeDrag
• Diffusion-based FreeDrag
• FreeDrag anything 3D

🛠️Usage

First clone our repository

git clone --depth=1 https://github.com/LPengYang/FreeDrag

To create a new environment, please follow the requirements of NVlabs/stylegan2-ada.

Notice: It is observed that the errors (setting up PyTorch plugin “bias_act_plugin“... Failed or “upfirdn2d_plugin“... Failed) may appear in some devices, we hope these potential solutions (1, 2, 3, 4) could be helpful in this case.

Then install the additional requirements

pip install -r requirements.txt

Then download the pre-trained models of stylegan2

bash download_models.sh

Notice： The first model (face model) could be downed very slowly in some cases. In this case, you can restart the download (works sometimes) or you can directly download it from this link, please download the correct model (ffhq-512×512) and renamed it as "faces.pkl" and manually put it in the created checkpoints file (after all the other models are downloaded).

Finally initialize the gradio platform for interactive point-based manipulation

CUDA_LAUNCH_BLOCKING=1 python FreeDrag_gradio.py

You can also upload your images and then edit them. For a high-quality image inversion, it is suggested to make sure that the resolution and style (such as layout) of the uploaded images are consistent with the generated images of corresponding model. The resolution of different model is listed as follows:

Model	face	horse	elephant	lion	dog	bicycle	giraffe	cat	car	church	metface
Resolution	512	256	512	512	1024	256	512	512	512	256	1024

❤️Acknowledgments

• DragGAN
• DragDiffusion
• StyleGAN2

License

All codes used or modified from StyleGAN2 are under the Nvidia Source Code License. The code related to the FreeDrag algorithm is only allowed for personal activity. For commercial use, please contact us.

✒️ Citation

If you find our work helpful for your research, please consider citing the following BibTeX entry.

@article{ling2023freedrag,
  title={FreeDrag: Point Tracking is Not You Need for Interactive Point-based Image Editing},
  author={Ling, Pengyang and Chen, Lin and Zhang, Pan and Chen, Huaian and Jin, Yi},
  journal={arXiv preprint arXiv:2307.04684},
  year={2023}
}