Hello, I confuse the networks for the training, I see that the train_avd_network is only with 1 gpu while the other training is with 2 gpu? is that the only difference ?

Hi,
Congratulations on such a wonderful research project and thank you for making it open source. I have a few questions on which I want to get your feedback for possible approaches.

1. Is it possible to reuse some of the computations. My use-case consists of running on a few videos but with different lip movements. All the other parts of the face consist of the same movement. Is there a way to pre-compute these facial movements and only compute the mouth movements at any given run?
2. Is it possible to amplify the lip movement? Currently, the movement scale is applied to the entire image. Is there a way to apply it only to the mouth area? I tried isolating the key points which detect the mouth, but they sometimes switch to tracking something else other than the mouth which causes the mouth to distort?
   Looking forward to hearing your thoughts on this.

Thanks!

Hello,
I'm a little confused. Your method is ultimately about processing video, which should be converted frame by frame. But why can such methods produce relatively smooth video results without considering the temporal consistency of the video?
Or why not model temporal consistency.

I am not getting better result as expected . how can train model easily . can anyone provide some blog/video about it

Hi，
I have try your demo with my own video and image, but there seems some green or gray outliers in the generated video, just like the mouth showed here，any suggestions?

Thank you for this work. However, is there any way to keep the sound (the voice for exemple) in the generated video?

Thx for the great work. I noticed that the datasets are resize into 384X384. Why don't you try the 512X512 resolution or higher resolution? Is it possible to train higher resolution?

The original network cannot be directly converted to onnx，some op are not supported, such as torch.inverse. I modify part of the network and combine 4 network into 1 onnx model, then convert to openvino. And the C++/Python demo is provided respectively. It took me about a week, and I upload to Image-Animation-Turbo-Boost. Hope it is useful.

您好！
因为不知道如何私信您 冒昧这里留言 还请包涵：

疑问：我在用同样的图片和视频（模板视频）去跑您们model demo，在hugface上得到非常糟糕的结果，比如模糊的人脸：https://huggingface.co/spaces/CVPR/Image-Animation-using-Thin-Plate-Spline-Motion-Model

但是在您们搭建的replicated demo却有非常好的输出（replicated：https://replicate.com/yoyo-nb/thin-plate-spline-motion-model/versions/382ceb8a9439737020bad407dec813e150388873760ad4a5a83a2ad01b039977）
是否您们在replicated做了一些参数微调？用了不同的数据集？或者我的操作有问题？ 因为我只是AI爱好者，没有程序开发背景，所以可能描述不准确，不知能否让您明白我的问题，期待您的答复，我的邮箱：[email protected]
祈
安

附对比视频链接：
colab和hugface：
https://drive.google.com/file/d/1zD5miR4985AhIQ_LovZ-7PbGRq3VrQGc/view?usp=share_link
https://cdn.discordapp.com/attachments/1045752990079914016/1048012069494075442/colab.mp4

replicated：
https://drive.google.com/file/d/1CphSJpPIb6XETd3hPyMQYgjyhD1hTPx1/view?usp=share_link
https://cdn.discordapp.com/attachments/1045752990079914016/1048012069229838397/REPLICATED.mp4

Hey, I am now on the final step of the installation and just need to enter the command: CUDA_VISIBLE_DEVICES=0 python demo.py --config config/vox-256.yaml --checkpoint checkpoints/vox.pth.tar --source_image ./source.jpg --driving_video ./driving.mp4

But when I paste it in the command line, it tells me: "The command "CUDA_VISIBLE_DEVICES" is either misspelled or could not be found".

What did I do wrong?

As of yesterday, getting the error attached. Please help. Many thanks

Sorry, but have to ask again.

I've made major modifications to your code so I can use it for fullframe face/head/partswap.

I use the openCV caffemodel for head tracking and cropping (expanded and squared bbox) for animating the image.
Then the animated image is masked and put back to the original fullframe video, including a automatic color correction.

For masking the full head or just parts of it I use this (modified) face-makeup repo (https://github.com/zllrunning/face-makeup.PyTorch)

I must admit I don't understand all of your code in full.
So I wonder if you use masking in one of the modules and if it would be possible to get that mask...?

Without the need to use the face-makeup parsing the faceswap code could run about 30 percent faster.

Regards
Thomas

Hello, I met some problems during the training, and would like to ask if it is convenient for you to provide the training log of TED dataset? If not, are there some LOSSS values that are roughly convergent for reference? I am looking forward to your reply？

Hi,
Thanks for your wonderful work and for making it open-source! Questions about datasets have bothered me for a long time.
Many of the provided download links for the dataset are invalid. Can you provide the complete dataset?
Thanks!

I am on Windows 10 with Anaconda setup.
I've tried Python 3.10, 3.9, and 3.7 (the latest has other issues too, but also this one).
Do you have any suggestions how to debug this?

(py39) Thin-Plate-Spline-Motion-Mode>pip install -r requirements.txt
Collecting cffi==1.14.6
  Using cached cffi-1.14.6-cp39-cp39-win_amd64.whl (180 kB)
Collecting cycler==0.10.0
  Using cached cycler-0.10.0-py2.py3-none-any.whl (6.5 kB)
Collecting decorator==5.1.0
  Using cached decorator-5.1.0-py3-none-any.whl (9.1 kB)
Collecting face-alignment==1.3.5
  Using cached face_alignment-1.3.5.tar.gz (27 kB)
  Preparing metadata (setup.py) ... done
Collecting imageio==2.9.0
  Using cached imageio-2.9.0-py3-none-any.whl (3.3 MB)
Collecting imageio-ffmpeg==0.4.5
  Using cached imageio_ffmpeg-0.4.5-py3-none-win_amd64.whl (22.6 MB)
Collecting kiwisolver==1.3.2
  Using cached kiwisolver-1.3.2-cp39-cp39-win_amd64.whl (52 kB)
Collecting matplotlib==3.4.3
  Using cached matplotlib-3.4.3-cp39-cp39-win_amd64.whl (7.1 MB)
Collecting networkx==2.6.3
  Using cached networkx-2.6.3-py3-none-any.whl (1.9 MB)
Collecting numpy==1.20.3
  Using cached numpy-1.20.3-cp39-cp39-win_amd64.whl (13.7 MB)
Collecting pandas==1.3.3
  Using cached pandas-1.3.3-cp39-cp39-win_amd64.whl (10.2 MB)
Collecting Pillow==8.3.2
  Using cached Pillow-8.3.2-cp39-cp39-win_amd64.whl (3.2 MB)
Collecting pycparser==2.20
  Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Collecting pyparsing==2.4.7
  Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting python-dateutil==2.8.2
  Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting pytz==2021.1
  Using cached pytz-2021.1-py2.py3-none-any.whl (510 kB)
Collecting PyWavelets==1.1.1
  Using cached PyWavelets-1.1.1-cp39-cp39-win_amd64.whl (4.2 MB)
Collecting PyYAML==5.4.1
  Using cached PyYAML-5.4.1-cp39-cp39-win_amd64.whl (213 kB)
Collecting scikit-image==0.18.3
  Using cached scikit_image-0.18.3-cp39-cp39-win_amd64.whl (12.2 MB)
Collecting scikit-learn==1.0
  Using cached scikit_learn-1.0-cp39-cp39-win_amd64.whl (7.2 MB)
Collecting scipy==1.7.1
  Using cached scipy-1.7.1-cp39-cp39-win_amd64.whl (33.8 MB)
Collecting six==1.16.0
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
ERROR: Could not find a version that satisfies the requirement torch==1.10.0+cu113 (from versions: 1.7.1, 1.8.0, 1.8.1, 1.9.0, 1.9.1, 1.10.0, 1.10.1, 1.10.2, 1.11.0, 1.12.0, 1.12.1, 1.13.0)
ERROR: No matching distribution found for torch==1.10.0+cu113

\Thin-Plate-Spline-Motion-Model\train.py", line 93, in train
logger.log_epoch(epoch, model_save, inp=x, out=generated)
UnboundLocalError: local variable 'x' referenced before assignment

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\NEURAL\Thin-Plate-Spline-Motion-Model\run.py", line 83, in
train(config, inpainting, kp_detector, bg_predictor, dense_motion_network, opt.checkpoint, log_dir, dataset)
File "C:\NEURAL\Thin-Plate-Spline-Motion-Model\train.py", line 93, in train
logger.log_epoch(epoch, model_save, inp=x, out=generated)
TypeError: exit() takes 1 positional argument but 4 were given

Looking to retain the alpha transparancy of my source image, if that means the output is in a string of png formats that's ok, is this possible?

Hello. First of all, thx for your work and article.
I have 2 questions related to the early stages of learning.

1. At 45k iterations model still outputs just the deformed version of images without any modifications.
2. While perceptual loss stagnates, feature warping loss grows.
   Have you encountered such features during training? Also, if you will provide images images from early stages of training and early learning curves, it would be quite helpful. Thanks in advance!

Maybe stupid question.

Is it possible to get the deformed/'animated' source_ image before using the inpaint module?

What is the output of the dense motion module.?....

out_dict['contribution_maps'] = contribution_maps
out_dict['deformation'] = deformation # Optical Flow
out_dict['occlusion_map'] = occlusion_map # Multi-resolution Occlusion Masks

out_dict['deformed_source'] = deformed_source ?

Thanks in advance

Hello, I have a question about this sentence in the 4.3 Ablations section of the paper.

The dimensions of FOMM and MRAA are K ∗ (2 + 4) and (K + 1) ∗ (2 + 4), while ours is K ∗ (6 + 5 ∗ 2) + 6.

I do not understand where these numbers come from. Could you elaborate please? Thank you.

I added this model to DeepFaceLive
https://github.com/iperov/DeepFaceLive/

it took a lot of work to export to onnx, because forward pytorch code is not graph-friendly

hi, thanks for sharing this project.
I'm using colab but it only takes 9 seconds of the video source, it's possible to change the settings to take the whole video source?
Thanks

First off, amazing work on this project!

Is it possible to make the eyes always face forward so that it appears that they are talking straight into the camera at all times?

(T-P-S-M-M) C:\Users\k\Desktop\T-P-S-M-M>python run.py --config config/vox-256.yaml
C:\Users\k\Desktop\T-P-S-M-M\run.py:38: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
config = yaml.load(f)
C:\anaconda3\envs\T-P-S-M-M\lib\site-packages\torchvision\models_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
warnings.warn(
C:\anaconda3\envs\T-P-S-M-M\lib\site-packages\torchvision\models_utils.py:223: UserWarning: Arguments other than a weight enum or None for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing weights=None.
warnings.warn(msg)
C:\anaconda3\envs\T-P-S-M-M\lib\site-packages\torch\functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at C:\cb\pytorch_1000000000000\work\aten\src\ATen\native\TensorShape.cpp:3191.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
None
Use predefined train-test split.
Training...
C:\anaconda3\envs\T-P-S-M-M\lib\site-packages\torchvision\models_utils.py:223: UserWarning: Arguments other than a weight enum or None for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing weights=VGG19_Weights.IMAGENET1K_V1. You can also use weights=VGG19_Weights.DEFAULT to get the most up-to-date weights.
warnings.warn(msg)
0%| | 0/100 [00:09<?, ?it/s]
Traceback (most recent call last):
File "C:\Users\k\Desktop\T-P-S-M-M\run.py", line 83, in
train(config, inpainting, kp_detector, bg_predictor, dense_motion_network, opt.checkpoint, log_dir, dataset)
File "C:\Users\k\Desktop\T-P-S-M-M\train.py", line 55, in train
for x in dataloader:
File "C:\anaconda3\envs\T-P-S-M-M\lib\site-packages\torch\utils\data\dataloader.py", line 628, in next
data = self._next_data()
File "C:\anaconda3\envs\T-P-S-M-M\lib\site-packages\torch\utils\data\dataloader.py", line 1333, in _next_data
return self._process_data(data)
File "C:\anaconda3\envs\T-P-S-M-M\lib\site-packages\torch\utils\data\dataloader.py", line 1359, in _process_data
data.reraise()
File "C:\anaconda3\envs\T-P-S-M-M\lib\site-packages\torch_utils.py", line 543, in reraise
raise exception
ValueError: Caught ValueError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "C:\anaconda3\envs\T-P-S-M-M\lib\site-packages\torch\utils\data_utils\worker.py", line 302, in _worker_loop
data = fetcher.fetch(index)
File "C:\anaconda3\envs\T-P-S-M-M\lib\site-packages\torch\utils\data_utils\fetch.py", line 58, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "C:\anaconda3\envs\T-P-S-M-M\lib\site-packages\torch\utils\data_utils\fetch.py", line 58, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "C:\Users\k\Desktop\T-P-S-M-M\frames_dataset.py", line 172, in getitem
return self.dataset[idx % self.dataset.len()]
File "C:\Users\k\Desktop\T-P-S-M-M\frames_dataset.py", line 109, in getitem
path = np.random.choice(glob.glob(os.path.join(self.root_dir, name + '*.mp4')))
File "mtrand.pyx", line 915, in numpy.random.mtrand.RandomState.choice
ValueError: 'a' cannot be empty unless no samples are taken

please tell me what to do

Downloading the code, resolving dependency issues, etc would eat a lot of time.
A Google Colab notebook would be really helpful to quickly check the code and implement it on the cloud.

Is it possible to increase the output resolution or quality?
Can the output resolution be increased when building or training the model?

A:\Thin-Plate-Spline-Motion-Model>set CUDA_VISIBLE_DEVICES=0 & python run.py --config config/vox-256.yaml --checkpoint checkpoints/vox.pth.tar --source_image assets/source.png --driving_video assets/driving.mp4 --find_best_frame --result_video SD_Avatar_Video.mp4
Traceback (most recent call last):
File "A:\Thin-Plate-Spline-Motion-Model\run.py", line 9, in
from frames_dataset import FramesDataset
File "A:\Thin-Plate-Spline-Motion-Model\frames_dataset.py", line 2, in
from skimage import io, img_as_float32
ModuleNotFoundError: No module named 'skimage'

How do i fix this? is this a dependency that can be added in the requirements.txt or is this something more complicated?

Hi, would you be interested in adding Thin-Plate-Spline-Motion-Model to Hugging Face? The Hub offers free hosting, and it would make your work more accessible and visible to the rest of the ML community. Models/datasets/spaces(web demos) can be added to a user account or organization similar to github.

Example from other organizations:
Keras: https://huggingface.co/keras-io
Microsoft: https://huggingface.co/microsoft
Facebook: https://huggingface.co/facebook

Example spaces with repos:
github: https://github.com/salesforce/BLIP
Spaces: https://huggingface.co/spaces/salesforce/BLIP

github: https://github.com/facebookresearch/omnivore
Spaces: https://huggingface.co/spaces/akhaliq/omnivore

and here are guides for adding spaces/models/datasets to your org

How to add a Space: https://huggingface.co/blog/gradio-spaces
how to add models: https://huggingface.co/docs/hub/adding-a-model
uploading a dataset: https://huggingface.co/docs/datasets/upload_dataset.html

Please let us know if you would be interested and if you have any questions, we can also help with the technical implementation.

I want to animate this picture by using a video that I recorded myself

So what should be the dimension of the video and the source image to animate it best

e.g. example image i want to animate

Seems to go fine right up until the end when it's trying to open the RESULTS.MP4 file. I've installed in Conda, pip, and Windows and it always end with that error and no RESULTS file is ever output.

(Thin-Plate-Spline) PS F:\Thin-Plate-Spline-Motion-Model> python demo.py --config config/vox-256.yaml --checkpoint checkpoints/vox.pth.tar --source_image ./assets/mfox256.jpg --driving_video ./assets/output.mp4 --result_video RESULTS --mode standard --find_best_frame
F:\Thin-Plate-Spline-Motion-Model\demo.py:147: DeprecationWarning: Starting with ImageIO v3 the behavior of this function will switch to that of iio.v3.imread. To keep the current behavior (and make this warning disappear) use import imageio.v2 as imageio or call imageio.v2.imread directly.
source_image = imageio.imread(opt.source_image)
C:\Users\cjay777xb.conda\envs\Thin-Plate-Spline\lib\site-packages\torchvision\models_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead.
warnings.warn(
C:\Users\cjay777xb.conda\envs\Thin-Plate-Spline\lib\site-packages\torchvision\models_utils.py:223: UserWarning: Arguments other than a weight enum or None for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing weights=None.
warnings.warn(msg)
C:\Users\cjay777xb.conda\envs\Thin-Plate-Spline\lib\site-packages\torch\functional.py:478: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at C:\cb\pytorch_1000000000000\work\aten\src\ATen\native\TensorShape.cpp:2895.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
221it [00:10, 21.03it/s]
Best frame: 115
100%|████████████████████████████████████████████████████████████████████████████████| 106/106 [00:03<00:00, 34.26it/s]
100%|████████████████████████████████████████████████████████████████████████████████| 116/116 [00:02<00:00, 43.04it/s]
Traceback (most recent call last):
File "F:\Thin-Plate-Spline-Motion-Model\demo.py", line 178, in
imageio.mimsave(opt.result_video, [img_as_ubyte(frame) for frame in predictions], fps=fps)
File "C:\Users\cjay777xb.conda\envs\Thin-Plate-Spline\lib\site-packages\imageio\v2.py", line 330, in mimwrite
with imopen(uri, "wI", **imopen_args) as file:
File "C:\Users\cjay777xb.conda\envs\Thin-Plate-Spline\lib\site-packages\imageio\core\imopen.py", line 303, in imopen
raise err_type(err_msg)
ValueError: Could not find a backend to open RESULTS`` with iomode wI`.
(Thin-Plate-Spline) PS F:\Thin-Plate-Spline-Motion-Model>

(Thin-Plate-Spline) PS F:\Thin-Plate-Spline-Motion-Model> conda list

packages in environment at C:\Users\cjay777xb.conda\envs\Thin-Plate-Spline:

Name Version Build Channel

av 9.2.0 pypi_0 pypi
blas 2.116 mkl conda-forge
blas-devel 3.9.0 16_win64_mkl conda-forge
blosc 1.21.1 h74325e0_3 conda-forge
brotli 1.0.9 h8ffe710_7 conda-forge
brotli-bin 1.0.9 h8ffe710_7 conda-forge
brotlipy 0.7.0 py39hb82d6ee_1004 conda-forge
bzip2 1.0.8 h8ffe710_4 conda-forge
ca-certificates 2019.11.28 hecc5488_0 conda-forge/label/cf202003
certifi 2022.9.24 py39haa95532_0
cfitsio 3.470 h2bbff1b_7
charls 2.2.0 h6c2663c_0
charset-normalizer 2.1.1 pyhd8ed1ab_0 conda-forge
cloudpickle 2.2.0 pyhd8ed1ab_0 conda-forge
colorama 0.4.5 pyhd8ed1ab_0 conda-forge
contourpy 1.0.5 pypi_0 pypi
cryptography 37.0.4 py39h7bc7c5c_0 conda-forge
cudatoolkit 11.6.0 hc0ea762_10 conda-forge
cycler 0.11.0 pyhd8ed1ab_0 conda-forge
cytoolz 0.12.0 py39hb82d6ee_0 conda-forge
dask-core 2022.9.2 pyhd8ed1ab_0 conda-forge
ffmpeg 4.2 h6538335_0 conda-forge/label/cf202003
fonttools 4.25.0 pyhd3eb1b0_0
freetype 2.12.1 h546665d_0 conda-forge
fsspec 2022.8.2 pyhd8ed1ab_0 conda-forge
giflib 5.2.1 h8d14728_2 conda-forge
icu 68.2 h0e60522_0 conda-forge
idna 3.4 pyhd8ed1ab_0 conda-forge
imagecodecs 2021.8.26 py39hc0a7faf_1
imageio 2.22.0 pyhfa7a67d_0 conda-forge
imageio-ffmpeg 0.4.7 pyhd8ed1ab_0 conda-forge
intel-openmp 2022.1.0 h57928b3_3787 conda-forge
jpeg 9e h8ffe710_2 conda-forge
kiwisolver 1.4.4 py39h2e07f2f_0 conda-forge
lcms2 2.12 h2a16943_0 conda-forge
lerc 3.0 h0e60522_0 conda-forge
libaec 1.0.6 h39d44d4_0 conda-forge
libblas 3.9.0 16_win64_mkl conda-forge
libbrotlicommon 1.0.9 h8ffe710_7 conda-forge
libbrotlidec 1.0.9 h8ffe710_7 conda-forge
libbrotlienc 1.0.9 h8ffe710_7 conda-forge
libcblas 3.9.0 16_win64_mkl conda-forge
libclang 11.1.0 default_h5c34c98_1 conda-forge
libdeflate 1.8 h2bbff1b_5
liblapack 3.9.0 16_win64_mkl conda-forge
liblapacke 3.9.0 16_win64_mkl conda-forge
libpng 1.6.37 h1d00b33_4 conda-forge
libtiff 4.4.0 h8a3f274_0
libuv 1.44.2 h8ffe710_0 conda-forge
libwebp-base 1.2.4 h8ffe710_0 conda-forge
libxcb 1.13 hcd874cb_1004 conda-forge
libzlib 1.2.12 h8ffe710_2 conda-forge
libzopfli 1.0.3 h0e60522_0 conda-forge
llvmlite 0.39.1 pypi_0 pypi
locket 1.0.0 pyhd8ed1ab_0 conda-forge
lz4-c 1.9.3 h8ffe710_1 conda-forge
m2w64-gcc-libgfortran 5.3.0 6 conda-forge
m2w64-gcc-libs 5.3.0 7 conda-forge
m2w64-gcc-libs-core 5.3.0 7 conda-forge
m2w64-gmp 6.1.0 2 conda-forge
m2w64-libwinpthread-git 5.0.0.4634.697f757 2 conda-forge
matplotlib 3.6.0 pypi_0 pypi
mkl 2022.1.0 h6a75c08_874 conda-forge
mkl-devel 2022.1.0 h57928b3_875 conda-forge
mkl-include 2022.1.0 h6a75c08_874 conda-forge
msys2-conda-epoch 20160418 1 conda-forge
munkres 1.1.4 pyh9f0ad1d_0 conda-forge
networkx 2.8.7 pyhd8ed1ab_0 conda-forge
numba 0.56.2 pypi_0 pypi
numpy 1.23.2 py39h1a62c8c_0 conda-forge
opencv-python 4.6.0.66 pypi_0 pypi
openjpeg 2.5.0 hc9384bd_1 conda-forge
openssl 1.1.1q h2bbff1b_0
packaging 21.3 pyhd8ed1ab_0 conda-forge
partd 1.3.0 pyhd8ed1ab_0 conda-forge
pillow 9.2.0 py39hcef8f5f_2 conda-forge
pip 22.2.2 py39haa95532_0
psutil 5.9.2 pypi_0 pypi
pthread-stubs 0.4 hcd874cb_1001 conda-forge
pyopenssl 22.0.0 pyhd8ed1ab_1 conda-forge
pyparsing 3.0.9 pyhd8ed1ab_0 conda-forge
pyqt 5.12.3 py39hb0d2dfa_4 conda-forge
pyqt5-sip 4.19.18 pypi_0 pypi
pyqtchart 5.12 pypi_0 pypi
pyqtwebengine 5.12.1 pypi_0 pypi
pysocks 1.7.1 pyh0701188_6 conda-forge
python 3.9.13 h6244533_1
python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge
python_abi 3.9 2_cp39 conda-forge
pytorch 1.12.1 py3.9_cuda11.6_cudnn8_0 pytorch
pytorch-mutex 1.0 cuda pytorch
pywavelets 1.4.1 pypi_0 pypi
pyyaml 6.0 py39hb82d6ee_4 conda-forge
qt 5.12.9 h5909a2a_4 conda-forge
requests 2.28.1 pyhd8ed1ab_1 conda-forge
scikit-image 0.19.3 py39h2e25243_1 conda-forge
scipy 1.9.1 pypi_0 pypi
setuptools 59.8.0 pypi_0 pypi
six 1.16.0 pyh6c4a22f_0 conda-forge
snappy 1.1.9 h82413e6_1 conda-forge
sqlite 3.39.3 h2bbff1b_0
tbb 2021.5.0 h2d74725_1 conda-forge
tifffile 2022.8.12 pypi_0 pypi
tk 8.6.12 h8ffe710_0 conda-forge
toolz 0.12.0 pyhd8ed1ab_0 conda-forge
torchaudio 0.12.1 py39_cu116 pytorch
torchvision 0.13.1 py39_cu116 pytorch
tornado 6.2 py39hb82d6ee_0 conda-forge
tqdm 4.64.1 pyhd8ed1ab_0 conda-forge
typing_extensions 4.3.0 pyha770c72_0 conda-forge
tzdata 2022c h04d1e81_0
urllib3 1.26.11 pyhd8ed1ab_0 conda-forge
vc 14.2 h21ff451_1
vs2015_runtime 14.27.29016 h5e58377_2
wheel 0.37.1 pyhd3eb1b0_0
win_inet_pton 1.1.0 py39hcbf5309_4 conda-forge
wincertstore 0.2 py39haa95532_2
xorg-libxau 1.0.9 hcd874cb_0 conda-forge
xorg-libxdmcp 1.1.3 hcd874cb_0 conda-forge
xz 5.2.6 h8d14728_0 conda-forge
yaml 0.2.5 h8ffe710_2 conda-forge
zfp 0.5.5 h0e60522_8 conda-forge
zlib 1.2.12 h8ffe710_2 conda-forge
zstd 1.5.2 h6255e5f_4 conda-forge
(Thin-Plate-Spline) PS F:\Thin-Plate-Spline-Motion-Model>

I am try use demo.py with assets\source.png and assets\driving.mp4. Only vox works perfectly. Other models makes ugly results.

Thanks for your nice work! I met a problem while I'm training on the TED dataset (Two 32G GPUs).

  File "Thin-Plate-Spline-Motion-Model/train.py", line 55, in train
    for x in dataloader:
  File "anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 530, in __next__
    data = self._next_data()
  File "anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1224, in _next_data
    return self._process_data(data)
  File "anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1250, in _process_data
    data.reraise()
  File "anaconda3/lib/python3.7/site-packages/torch/_utils.py", line 457, in reraise
    raise exception
RuntimeError: Caught RuntimeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
    data = fetcher.fetch(index)
  File "anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 49, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "Thin-Plate-Spline-Motion-Model/frames_dataset.py", line 172, in __getitem__
    return self.dataset[idx % self.dataset.__len__()]
  File "Thin-Plate-Spline-Motion-Model/frames_dataset.py", line 133, in __getitem__
    video_array = read_video(path, frame_shape=self.frame_shape)
  File "Thin-Plate-Spline-Motion-Model/frames_dataset.py", line 43, in read_video
    video = mimread(name)
  File "anaconda3/lib/python3.7/site-packages/imageio/core/functions.py", line 369, in mimread
    int(nbyte_limit)

RuntimeError: imageio.mimread() has read over 256000000B of image data.
Stopped to avoid memory problems. Use imageio.get_reader(), increase threshold, or memtest=False

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "run.py", line 83, in <module>
    train(config, inpainting, kp_detector, bg_predictor, dense_motion_network, opt.checkpoint, log_dir, dataset)
  File "Thin-Plate-Spline-Motion-Model/train.py", line 93, in train
    logger.log_epoch(epoch, model_save, inp=x, out=generated)
TypeError: __exit__() takes 1 positional argument but 4 were given

Thanks for your replay!

Thanks for the repo and colab. I've gotten the demo to work with vox and a portrait. I'm trying to get ted to work.

What do you think is the optimal parameters of the driving footage? For ted, vox, etc.

Edit1: I used the ted checkpoint and config and made a little bit of progress with limbs showing, but still pretty messy. Perhaps just need to match dimensions of the source image to the footage perfectly in frame 1? Cropping source image to match driving video and removing the background from the source image made a tiny bit of progress, but still bad.

Edit 2: I see your comparison and tip to use taichi for full body: #38 Will try taichi.

Would you say for ted, the driving video should be cropped at chest level and above like in your examples? Could you include the single source in the assets folder for the demo gifs you made (instead of the row of gifs)? Does the background need to be close to a solid color? I grabbed ted footage that had more objects in the background, but the person stayed stationary in the center. My source images also had a lot of background noise. In general the output just has the center pushed out where the ted talker is, and it wobbles around a bit but there's no limb or facial recognition.

When using ted, I left the vox config the same because I'm unclear of how to modify that. Would you say I need to use a ted-specific config and go thru the parameters? I'll start looking now just in case.

Thank you for this great repo.

I'm using the original FOMM for about 2 years. But your modification gives much better results.
Are you planning to modify it so we can use it for face swap / part swap like the motion co-segmentation branch of FOMM?

I've made major modifications to the motion co-segmentation repo, so that now it performs
'automatic' face tracking / cropping / face swap /colour correction and putting all together again.

It would be nice to do that with a version that gives even better results.

There are four models. please indicate the model used for each technique if you can.

vox.pth.tar
ted.pth.tar
taichi.pth.tar
mgif.pth.tar

Hi @yoyo-nb ,

Thanks for your excellent work. The video-processing provides a large test set for VoxCeleb. Do you use all samples in that test set? If not, could you please share your own test set sampled from the large test set?
Or share your test samples recorded file like .csv?

Hey @yoyo-nb!

This issue isn't really an issue and more some insight into my experiments of how one might be able to add the talking head generation back to the original frame they were trying to animate. Just thought it'd be cool to share :) This would make something like the following possible rather than just a small driving video focused on the face. As you can see, if you squint, you can kind of tell that we added the crop back to the original frame but it isn't super noticeable.

The core change I made was the way one does face alignment to be based purely on rotating the image until the eyes are level. I didn't recognize any quality degradation to your approach based on doing this but what this allows is for us to "unwarp" the original image so it is again level with the rest of the original frame.

On top of that, I also made it so that one could simply supply a driving audio clip along with a driving video clip such that we first run Wav2Lip on that, and then use the lip synced video to drive the avatar. This allows for us to make the avatar say whatever we want, even if we don't have a driving video that's saying it.

Here are some more examples:

My colleague wrote a bit about it here: https://www.sievedata.com/blog/realistic-ai-avatars
Code to deploy on Sieve or to deploy yourself: https://github.com/sieve-community/examples/tree/main/talking_head_avatars

If you think it's worthwhile I'd be happy to implement this into your repo though it might be tough given the moving parts with Wav2Lip as well.

In case you are wondering who I am, I'm Mokshith and I work at Sieve where we make it easy for people to build and run video AI pipelines. We realized that to make a lot of research useful, you end up having to combine it with many moving parts and even then -- it's really slow to run. We're just working on cool projects built on top of Sieve for fun! You can also just clone and run what I described above yourself by clicking here and signing up for free :)

Hi，

Thanks for your excellent work.

Why does the regularization method in the model use nn.InstanceNorm2d instead of BN?

Everytime I try to run the model, saving vox.pth from cloud.tsinghua.edu.cn either failes or takes several minutes. It's currently loading at 4.3 KB/s! Couldn't you please host that file here on github or somewhere else, please?!?

Thank you for releasing the code, could you add the code to crop and align the face from image and video. Is it like FFHQ landmark?

Sincerely yours.

I keep getting an error of:

---> 12 source_image = imageio.imread(source_image_path)

the source image path however is definitely defined and correct, i have uploaded the png file directly into the assets folder and even put the path variable in line 12 "./assets/still.png" rather than calling source_image_path to double check but i still get the error

i've tried setting up venv 3.9. Getting the following error.

running cuda 12
c++/cli v143
MSVC v143 - VS2022 c++ x64/x86 build tools

just ran out of ideas on how to fix.

(3.9) C:\tps>pip install -r requirements.txt
Collecting cffi==1.14.6
Using cached cffi-1.14.6.tar.gz (475 kB)
Preparing metadata (setup.py) ... done
Collecting cycler==0.10.0
Using cached cycler-0.10.0-py2.py3-none-any.whl (6.5 kB)
Collecting decorator==5.1.0
Using cached decorator-5.1.0-py3-none-any.whl (9.1 kB)
Collecting face-alignment==1.3.5
Using cached face_alignment-1.3.5.tar.gz (27 kB)
Preparing metadata (setup.py) ... done
Collecting imageio==2.9.0
Using cached imageio-2.9.0-py3-none-any.whl (3.3 MB)
Collecting imageio-ffmpeg==0.4.5
Using cached imageio_ffmpeg-0.4.5-py3-none-win_amd64.whl (22.6 MB)
Collecting kiwisolver==1.3.2
Using cached kiwisolver-1.3.2-cp310-cp310-win_amd64.whl (52 kB)
Collecting matplotlib==3.4.3
Using cached matplotlib-3.4.3.tar.gz (37.9 MB)
Preparing metadata (setup.py) ... done
Collecting networkx==2.6.3
Using cached networkx-2.6.3-py3-none-any.whl (1.9 MB)
Collecting numpy==1.20.3
Using cached numpy-1.20.3.zip (7.8 MB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... error
error: subprocess-exited-with-error

× Preparing metadata (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [256 lines of output]
setup.py:66: RuntimeWarning: NumPy 1.20.3 may not yet support Python 3.10.
warnings.warn(
Running from numpy source directory.
setup.py:485: UserWarning: Unrecognized setuptools command, proceeding with generating Cython sources and expanding templates
run_build = parse_setuppy_commands()
Processing numpy/random_bounded_integers.pxd.in
Processing numpy/random\bit_generator.pyx
Processing numpy/random\mtrand.pyx
Processing numpy/random_bounded_integers.pyx.in
Processing numpy/random_common.pyx
Processing numpy/random_generator.pyx
Processing numpy/random_mt19937.pyx
Processing numpy/random_pcg64.pyx
Processing numpy/random_philox.pyx
Processing numpy/random_sfc64.pyx
Cythonizing sources
blas_opt_info:
blas_mkl_info:
No module named 'numpy.distutils._msvccompiler' in numpy.distutils; trying from distutils
customize MSVCCompiler
libraries mkl_rt not found in ['C:\tps\3.9\lib', 'C:\']
NOT AVAILABLE

  blis_info:
    libraries blis not found in ['C:\\tps\\3.9\\lib', 'C:\\']
    NOT AVAILABLE

  openblas_info:
    libraries openblas not found in ['C:\\tps\\3.9\\lib', 'C:\\']
  get_default_fcompiler: matching types: '['gnu', 'intelv', 'absoft', 'compaqv', 'intelev', 'gnu95', 'g95', 'intelvem', 'intelem', 'flang']'
  customize GnuFCompiler
  Could not locate executable g77
  Could not locate executable f77
  customize IntelVisualFCompiler
  Could not locate executable ifort
  Could not locate executable ifl
  customize AbsoftFCompiler
  Could not locate executable f90
  customize CompaqVisualFCompiler
  Could not locate executable DF
  customize IntelItaniumVisualFCompiler
  Could not locate executable efl
  customize Gnu95FCompiler
  Could not locate executable gfortran
  Could not locate executable f95
  customize G95FCompiler
  Could not locate executable g95
  customize IntelEM64VisualFCompiler
  customize IntelEM64TFCompiler
  Could not locate executable efort
  Could not locate executable efc
  customize PGroupFlangCompiler
  Could not locate executable flang
  don't know how to compile Fortran code on platform 'nt'
    NOT AVAILABLE

  atlas_3_10_blas_threads_info:
  Setting PTATLAS=ATLAS
    libraries tatlas not found in ['C:\\tps\\3.9\\lib', 'C:\\']
    NOT AVAILABLE

  atlas_3_10_blas_info:
    libraries satlas not found in ['C:\\tps\\3.9\\lib', 'C:\\']
    NOT AVAILABLE

  atlas_blas_threads_info:
  Setting PTATLAS=ATLAS
    libraries ptf77blas,ptcblas,atlas not found in ['C:\\tps\\3.9\\lib', 'C:\\']
    NOT AVAILABLE

  atlas_blas_info:
    libraries f77blas,cblas,atlas not found in ['C:\\tps\\3.9\\lib', 'C:\\']
    NOT AVAILABLE

  C:\Users\philg\AppData\Local\Temp\pip-install-0caih3bt\numpy_db68e4a8c7564561956f2d2bc5b1659f\numpy\distutils\system_info.py:1989: UserWarning:
      Optimized (vendor) Blas libraries are not found.
      Falls back to netlib Blas library which has worse performance.
      A better performance should be easily gained by switching
      Blas library.
    if self._calc_info(blas):
  blas_info:
    libraries blas not found in ['C:\\tps\\3.9\\lib', 'C:\\']
    NOT AVAILABLE

  C:\Users\philg\AppData\Local\Temp\pip-install-0caih3bt\numpy_db68e4a8c7564561956f2d2bc5b1659f\numpy\distutils\system_info.py:1989: UserWarning:
      Blas (http://www.netlib.org/blas/) libraries not found.
      Directories to search for the libraries can be specified in the
      numpy/distutils/site.cfg file (section [blas]) or by setting
      the BLAS environment variable.
    if self._calc_info(blas):
  blas_src_info:
    NOT AVAILABLE

  C:\Users\philg\AppData\Local\Temp\pip-install-0caih3bt\numpy_db68e4a8c7564561956f2d2bc5b1659f\numpy\distutils\system_info.py:1989: UserWarning:
      Blas (http://www.netlib.org/blas/) sources not found.
      Directories to search for the sources can be specified in the
      numpy/distutils/site.cfg file (section [blas_src]) or by setting
      the BLAS_SRC environment variable.
    if self._calc_info(blas):
    NOT AVAILABLE

  non-existing path in 'numpy\\distutils': 'site.cfg'
  lapack_opt_info:
  lapack_mkl_info:
    libraries mkl_rt not found in ['C:\\tps\\3.9\\lib', 'C:\\']
    NOT AVAILABLE

  openblas_lapack_info:
    libraries openblas not found in ['C:\\tps\\3.9\\lib', 'C:\\']
    NOT AVAILABLE

  openblas_clapack_info:
    libraries openblas,lapack not found in ['C:\\tps\\3.9\\lib', 'C:\\']
    NOT AVAILABLE

  flame_info:
    libraries flame not found in ['C:\\tps\\3.9\\lib', 'C:\\']
    NOT AVAILABLE

  atlas_3_10_threads_info:
  Setting PTATLAS=ATLAS
    libraries lapack_atlas not found in C:\tps\3.9\lib
    libraries tatlas,tatlas not found in C:\tps\3.9\lib
    libraries lapack_atlas not found in C:\
    libraries tatlas,tatlas not found in C:\
  <class 'numpy.distutils.system_info.atlas_3_10_threads_info'>
    NOT AVAILABLE

  atlas_3_10_info:
    libraries lapack_atlas not found in C:\tps\3.9\lib
    libraries satlas,satlas not found in C:\tps\3.9\lib
    libraries lapack_atlas not found in C:\
    libraries satlas,satlas not found in C:\
  <class 'numpy.distutils.system_info.atlas_3_10_info'>
    NOT AVAILABLE

  atlas_threads_info:
  Setting PTATLAS=ATLAS
    libraries lapack_atlas not found in C:\tps\3.9\lib
    libraries ptf77blas,ptcblas,atlas not found in C:\tps\3.9\lib
    libraries lapack_atlas not found in C:\
    libraries ptf77blas,ptcblas,atlas not found in C:\
  <class 'numpy.distutils.system_info.atlas_threads_info'>
    NOT AVAILABLE

  atlas_info:
    libraries lapack_atlas not found in C:\tps\3.9\lib
    libraries f77blas,cblas,atlas not found in C:\tps\3.9\lib
    libraries lapack_atlas not found in C:\
    libraries f77blas,cblas,atlas not found in C:\
  <class 'numpy.distutils.system_info.atlas_info'>
    NOT AVAILABLE

  lapack_info:
    libraries lapack not found in ['C:\\tps\\3.9\\lib', 'C:\\']
    NOT AVAILABLE

  C:\Users\philg\AppData\Local\Temp\pip-install-0caih3bt\numpy_db68e4a8c7564561956f2d2bc5b1659f\numpy\distutils\system_info.py:1849: UserWarning:
      Lapack (http://www.netlib.org/lapack/) libraries not found.
      Directories to search for the libraries can be specified in the
      numpy/distutils/site.cfg file (section [lapack]) or by setting
      the LAPACK environment variable.
    return getattr(self, '_calc_info_{}'.format(name))()
  lapack_src_info:
    NOT AVAILABLE

  C:\Users\philg\AppData\Local\Temp\pip-install-0caih3bt\numpy_db68e4a8c7564561956f2d2bc5b1659f\numpy\distutils\system_info.py:1849: UserWarning:
      Lapack (http://www.netlib.org/lapack/) sources not found.
      Directories to search for the sources can be specified in the
      numpy/distutils/site.cfg file (section [lapack_src]) or by setting
      the LAPACK_SRC environment variable.
    return getattr(self, '_calc_info_{}'.format(name))()
    NOT AVAILABLE

  numpy_linalg_lapack_lite:
    FOUND:
      language = c
      define_macros = [('HAVE_BLAS_ILP64', None), ('BLAS_SYMBOL_SUFFIX', '64_')]

  C:\Users\philg\AppData\Local\Temp\pip-build-env-f6y8vqpe\overlay\Lib\site-packages\setuptools\_distutils\dist.py:275: UserWarning: Unknown distribution option: 'define_macros'
    warnings.warn(msg)
  running dist_info
  running build_src
  build_src
  building py_modules sources
  creating build
  creating build\src.win-amd64-3.10
  creating build\src.win-amd64-3.10\numpy
  creating build\src.win-amd64-3.10\numpy\distutils
  building library "npymath" sources
  Traceback (most recent call last):
    File "C:\tps\3.9\lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 353, in <module>
      main()
    File "C:\tps\3.9\lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 335, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "C:\tps\3.9\lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 149, in prepare_metadata_for_build_wheel
      return hook(metadata_directory, config_settings)
    File "C:\Users\philg\AppData\Local\Temp\pip-build-env-f6y8vqpe\overlay\Lib\site-packages\setuptools\build_meta.py", line 157, in prepare_metadata_for_build_wheel
      self.run_setup()
    File "C:\Users\philg\AppData\Local\Temp\pip-build-env-f6y8vqpe\overlay\Lib\site-packages\setuptools\build_meta.py", line 248, in run_setup
      super(_BuildMetaLegacyBackend,
    File "C:\Users\philg\AppData\Local\Temp\pip-build-env-f6y8vqpe\overlay\Lib\site-packages\setuptools\build_meta.py", line 142, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 513, in <module>
      setup_package()
    File "setup.py", line 505, in setup_package
      setup(**metadata)
    File "C:\Users\philg\AppData\Local\Temp\pip-install-0caih3bt\numpy_db68e4a8c7564561956f2d2bc5b1659f\numpy\distutils\core.py", line 169, in setup
      return old_setup(**new_attr)
    File "C:\Users\philg\AppData\Local\Temp\pip-build-env-f6y8vqpe\overlay\Lib\site-packages\setuptools\__init__.py", line 165, in setup
      return distutils.core.setup(**attrs)
    File "C:\Users\philg\AppData\Local\Temp\pip-build-env-f6y8vqpe\overlay\Lib\site-packages\setuptools\_distutils\core.py", line 148, in setup
      dist.run_commands()
    File "C:\Users\philg\AppData\Local\Temp\pip-build-env-f6y8vqpe\overlay\Lib\site-packages\setuptools\_distutils\dist.py", line 967, in run_commands
      self.run_command(cmd)
    File "C:\Users\philg\AppData\Local\Temp\pip-build-env-f6y8vqpe\overlay\Lib\site-packages\setuptools\_distutils\dist.py", line 986, in run_command
      cmd_obj.run()
    File "C:\Users\philg\AppData\Local\Temp\pip-build-env-f6y8vqpe\overlay\Lib\site-packages\setuptools\command\dist_info.py", line 31, in run
      egg_info.run()
    File "C:\Users\philg\AppData\Local\Temp\pip-install-0caih3bt\numpy_db68e4a8c7564561956f2d2bc5b1659f\numpy\distutils\command\egg_info.py", line 24, in run
      self.run_command("build_src")
    File "C:\Users\philg\AppData\Local\Temp\pip-build-env-f6y8vqpe\overlay\Lib\site-packages\setuptools\_distutils\cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "C:\Users\philg\AppData\Local\Temp\pip-build-env-f6y8vqpe\overlay\Lib\site-packages\setuptools\_distutils\dist.py", line 986, in run_command
      cmd_obj.run()
    File "C:\Users\philg\AppData\Local\Temp\pip-install-0caih3bt\numpy_db68e4a8c7564561956f2d2bc5b1659f\numpy\distutils\command\build_src.py", line 144, in run
      self.build_sources()
    File "C:\Users\philg\AppData\Local\Temp\pip-install-0caih3bt\numpy_db68e4a8c7564561956f2d2bc5b1659f\numpy\distutils\command\build_src.py", line 155, in build_sources
      self.build_library_sources(*libname_info)
    File "C:\Users\philg\AppData\Local\Temp\pip-install-0caih3bt\numpy_db68e4a8c7564561956f2d2bc5b1659f\numpy\distutils\command\build_src.py", line 288, in build_library_sources
      sources = self.generate_sources(sources, (lib_name, build_info))
    File "C:\Users\philg\AppData\Local\Temp\pip-install-0caih3bt\numpy_db68e4a8c7564561956f2d2bc5b1659f\numpy\distutils\command\build_src.py", line 378, in generate_sources
      source = func(extension, build_dir)
    File "numpy\core\setup.py", line 671, in get_mathlib_info
      st = config_cmd.try_link('int main(void) { return 0;}')
    File "C:\Users\philg\AppData\Local\Temp\pip-build-env-f6y8vqpe\overlay\Lib\site-packages\setuptools\_distutils\command\config.py", line 243, in try_link
      self._link(body, headers, include_dirs,
    File "C:\Users\philg\AppData\Local\Temp\pip-install-0caih3bt\numpy_db68e4a8c7564561956f2d2bc5b1659f\numpy\distutils\command\config.py", line 162, in _link
      return self._wrap_method(old_config._link, lang,
    File "C:\Users\philg\AppData\Local\Temp\pip-install-0caih3bt\numpy_db68e4a8c7564561956f2d2bc5b1659f\numpy\distutils\command\config.py", line 96, in _wrap_method
      ret = mth(*((self,)+args))
    File "C:\Users\philg\AppData\Local\Temp\pip-build-env-f6y8vqpe\overlay\Lib\site-packages\setuptools\_distutils\command\config.py", line 137, in _link
      (src, obj) = self._compile(body, headers, include_dirs, lang)
    File "C:\Users\philg\AppData\Local\Temp\pip-install-0caih3bt\numpy_db68e4a8c7564561956f2d2bc5b1659f\numpy\distutils\command\config.py", line 105, in _compile
      src, obj = self._wrap_method(old_config._compile, lang,
    File "C:\Users\philg\AppData\Local\Temp\pip-install-0caih3bt\numpy_db68e4a8c7564561956f2d2bc5b1659f\numpy\distutils\command\config.py", line 96, in _wrap_method
      ret = mth(*((self,)+args))
    File "C:\Users\philg\AppData\Local\Temp\pip-build-env-f6y8vqpe\overlay\Lib\site-packages\setuptools\_distutils\command\config.py", line 132, in _compile
      self.compiler.compile([src], include_dirs=include_dirs)
    File "C:\Users\philg\AppData\Local\Temp\pip-build-env-f6y8vqpe\overlay\Lib\site-packages\setuptools\_distutils\_msvccompiler.py", line 401, in compile
      self.spawn(args)
    File "C:\Users\philg\AppData\Local\Temp\pip-build-env-f6y8vqpe\overlay\Lib\site-packages\setuptools\_distutils\_msvccompiler.py", line 505, in spawn
      return super().spawn(cmd, env=env)
    File "C:\Users\philg\AppData\Local\Temp\pip-install-0caih3bt\numpy_db68e4a8c7564561956f2d2bc5b1659f\numpy\distutils\ccompiler.py", line 90, in <lambda>
      m = lambda self, *args, **kw: func(self, *args, **kw)
  TypeError: CCompiler_spawn() got an unexpected keyword argument 'env'
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

Hi,

I've set up python 3.9, ran get-pip, upgraded pip, set up my venv and I get this on install;

ERROR: Could not find a version that satisfies the requirement torch==1.10.0+cu113 (from versions: 1.7.1, 1.8.0, 1.8.1, 1.9.0, 1.9.1, 1.10.0, 1.10.1, 1.10.2, 1.11.0, 1.12.0, 1.12.1)

Dockerfile would be a good idea?

I'm rocking Ubuntu 20.04

it takes too long when use ted talk model

Would anyone be so kind as to provide a tutorial to create and train your own dataset for Thin Plate Splin Motion Model please?

hi,i want to train this project,but i dont know how to download the dataset || ||,
would you help me ?

Hi,
To uspcale the video we need to have all frames in a folder, to upscale them one by one.
We can extract all the frames with ffmpeg, but the pictures are slightly damaged by the video compression.
So it would be nice to be able to keep all the frames in better quality in a folder.
Is it possible ?

great work guys!

would be great if you can a colab notebook for this project with the pretrained model , such that user can upload a video with the action and an image so that you can transfer the motion.