zhangqianhui / gazeanimationv2

Unsupervised High-Resolution Portrait Gaze Correction and Animation (TIP 2022)

GazeAnimationV2 - Official Pytorch Implementation (TIP)

Unsupervised High-Resolution Portrait Gaze Correction and Animation
Paper: https://arxiv.org/abs/2207.00256

Dependencies

Python=3.6
pip install -r requirements.txt

Or Using Conda

-conda create -name GazeA python=3.6
-conda install tensorflow-gpu=1.9 or higher

Other packages installed by pip.

Usage

• Clone this repo:

git clone https://github.com/zhangqianhui/GazeAnimation.git
cd GazeAnimation
git checkout GazeAnimationV2

• Download the CelebAGaze dataset

  Download the tar of CelebGaze dataset from Google Driver Linking.

  cd your_path
  tar -xvf CelebAGaze.tar

• Download the CelebHQGaze dataset

Download the tar of CelebHQGaze dataset from Google Driver Linking.

cd your_path
tar -xvf CelebHQGaze.tar

Experiment Result

Gaze Correction on CelebAHQGaze

@article{zhang2022unsupervised,
  title={Unsupervised High-Resolution Portrait Gaze Correction and Animation},
  author={Zhang, Jichao and Chen, Jingjing and Tang, Hao and Sangineto, Enver and Wu, Peng and Yan, Yan and Sebe, Nicu and Wang, Wei},
  journal={IEEE Transactions on Image Processing},
  year={2022}
}

gazeanimationv2's Issues

AttributeError: 'Gaze_GAN' object has no attribute 'D_real_gan_logits'. Could you please find that whether there is any wrong in the code?

How to apply in video gaze correction?

Hi:
Your work of gaze correction in img seems not suitable for video application due to inconsistency and flicker between frames. Do u have any idea about video gaze correction?

question for Testing by using user's data

Thank for your master work, but one question: In the test process, good result for the data you offer, but when I test my private data, the result is poor, can you kindly give me some advice?
PS: I had assign the eyes' location of my private data when testing

Question About Training Time On My Own Dataset

I have a general question about training time on my own dataset. On a single Tesla P100, how long can I expect it to train before getting acceptable results? I'm currently at 165000 iters and the eye area still looks very grainy. I don't know if this is a convergence issue or just apart of the training process. Should I train further or modify my target dataset? Thanks!

git checkout GazeAnimationV2 error

git checkout GazeAnimationV2 ，got an error：error: pathspec 'GazeAnimationV2' did not match any file(s) known to git

preformnce in

excellent work! when i test the trained model in NewGanDataset(Y) the performance is good, but i test in my own dataset,the performance drop drasticly,can you tell me the preprocess method you used. appreciate for your reply!

Loss function of the classifier

Hello, this is an interesting work about eye's inpainting. Could you please answer me a few questions?

It seems you use a different symbol for the classifier's loss in equation (3), i.e., positive for data from X and negative for data from X_wave. May I ask why?

Are you enforcing this classifier to learn not only the left-or-right classification but also the real-or-fake classification? I think it is rellay strange, because we hope this classfier to classify eyes from X_wave oppositely? (I mean, for x from X_wave, if the classifier classifies left eye as right eye, and right eye as left eye, it will get a smaller loss than classify these eyes' position correctly?)

Note: all of the classifiers mentioned above denote the left-or-right classifier.

您好

请问· 在您readme最后一部分的gif图实现是在代码的哪一部分啊？~

Key discriminator/dis_FC1/Matrix not found in checkpoint

hello，when i test on my own image dataset, i meet this error~ it seems the ckpt is not same with current architecture of network.

Could you please provide a trained model？

Could you please provide a trained model? So that we can directly run test.py. Thanks~

NotFoundError: Key D/d1/d0/biases not found in checkpoint

I'm trying to run the test with the pretrained checkpoint.

Unzipped pretrained.zip to GazeCorrection/log3_25_1/checkpoints.
When I run bash scripts/test_log20_3_25_1.sh this error is thrown:

NotFoundError (see above for traceback): Key D/d1/d0/biases not found in checkpoint
	 [[Node: save/RestoreV2 = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]