guanjz20 / stylesync_pytorch

PyTorch implementation of "StyleSync: High-Fidelity Generalized and Personalized Lip Sync in Style-based Generator"

This a PyTorch-implemented repo. Find the Paddle-based version here.

StyleSync: High-Fidelity Generalized and Personalized Lip Sync in Style-based Generator (CVPR 2023)

Jiazhi Guan*, Zhanwang Zhang*, Hang Zhou†, Tianshu Hu†, Kaisiyuan Wang, Dongliang He, Haocheng Feng, Jingtuo Liu, Errui Ding, Ziwei Liu, Jingdong Wang

Project | Paper | Video | Demo

We propose StyleSync, an effective framework that enables high-fidelity lip synchronization. We identify that a style-based generator would sufficiently enable such a charming property on both one-shot and few-shot scenarios.

Code

Inference script and model code have been released.

Run Generation

The decision to suspend the release of the model weights has been made by our team based on various considerations. At present, I also do not have a specific expected date for its release. However, if you are interested in utilizing our demo for academic purposes, such as conducting a comparison in your paper, please don't hesitate to contact me by [guanjz20 at mails dot tsinghua dot edu dot cn].

Citation

@inproceedings{guan2023stylesync,
  title = {StyleSync: High-Fidelity Generalized and Personalized Lip Sync in Style-based Generator},
  author = {Guan, Jiazhi and Zhang, Zhanwang and Zhou, Hang and HU, Tianshu and Wang, Kaisiyuan and He, Dongliang and Feng, Haocheng and Liu, Jingtuo and Ding, Errui and Liu, Ziwei and Wang, Jingdong},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year = {2023}
}

stylesync_pytorch's Issues

谁跑通了？付费求教程和pth

感谢作者提供了这么好的内容，从B站过来一直就跟踪了这个项目，这么好的项目居然没有太多人关注。
本人比较菜，习惯了复制粘贴，不知道这个项目怎么能跑起来，怎么训练？所以付费求跑通高手告诉我教程，包括怎么训练，pth文件。
Vx：qimingzi1027
再次感谢作者

Details of training syncnet

Hello :)

I have a question about the training on your own syncnet.

As far as I know, it is difficult to converge the losses of Syncnet. So, many users have left messages in the Issues tab of the Wav2Lip GitHub repository.

Could you explain the architecture or training scheme of Syncnet of this model in more detail?

Thanks! :_)

伪开源

伪开源就别标榜自己开源了，甚至都没有checkpoint