Comments (6)
Hello.
The two questions can be solved with the same solution.
This work is an unsupervised method, so no prior knowledge is introduced. If you want to have better performance on human faces, you can use pre-trained face landmarks detector instead of Keypoint Detector in this work, and choose K*5 points from face landmarks and manually design which points to use as a group to compute TPS transformations. After that retrain the rest of the network modules (Dense Motion Network, Inpainting Network and also include the BG Motion Predictor if there is background motion in your training data).
For example, you can use five points selected in the upper lip region as a group to calculate the TPS transformation, and similarly, you can select 5 points from each of the lower lip, left eyebrow, left eye, right eyebrow, right eye and nose regions as a group.
And when inferring, if you only want to transfer the motion of the lip region, you can calculate the TPSs corresponding to the upper and lower lip only, while the TPSs of other regions can be calculated in advance and applied to each frame of the generated video.
from thin-plate-spline-motion-model.
Thanks for the reply! I was wondering if there is a way to do this without re-training the model? Why is retraining the dense motion network needed, as it can is trained on the task of predicting the motion from the TPS transformations, does it matter if the points used for TPS transformations come from self-supervised way or if we select them manually?
from thin-plate-spline-motion-model.
I don't think this can be done without retraining the model.
Because the current model learns by unsupervised method, the contribution of each TPS to the optical flow is not concentrated in one region like FOMM, but scattered in various locations on the face. And good motion transfer results can only be obtained when multiple TPSs cooperate. If the keypoints are designed manually without retraining the dense motion network, according to my understanding, the dense motion network will still predict the contribution maps scattered in various locations as before, resulting in a potentially poorer calculated optical flow.
If these keypoints are designed manually and the dense motion network is retrained, the contribution map of each TPS will be concentrated in one region and is meaningful (I have experimented with it). And transfer some regions of the motion can be done according to this.
from thin-plate-spline-motion-model.
Hi, Congratulations on such a wonderful research project and thank you for making it open source. I have a few questions on which I want to get your feedback for possible approaches.
1. Is it possible to reuse some of the computations. My use-case consists of running on a few videos but with different lip movements. All the other parts of the face consist of the same movement. Is there a way to pre-compute these facial movements and only compute the mouth movements at any given run? 2. Is it possible to amplify the lip movement? Currently, the movement scale is applied to the entire image. Is there a way to apply it only to the mouth area? I tried isolating the key points which detect the mouth, but they sometimes switch to tracking something else other than the mouth which causes the mouth to distort? Looking forward to hearing your thoughts on this.
Thanks!
Awesome idea - did you figure this out?
from thin-plate-spline-motion-model.
mark
from thin-plate-spline-motion-model.
mark
from thin-plate-spline-motion-model.
Related Issues (20)
- AttributeError at animation generation HOT 1
- When setting “find_best_frame = True”, an error occurs in the bottom four modules in demo.ipynb. HOT 1
- Erroring out at - from demo import load_checkpoints
- After training, ask reasoning questions with checkpoints
- Training data questions
- EOFError Ran out of input HOT 11
- Invalid loadkey HOT 3
- RuntimeError: Expected hasRecord("version") to be true, but got false. HOT 2
- FileNotFoundError: [Errno 2] No such file or directory: 'checkpoints/vox.pth.tar'
- Generating video issues with notebook
- fine tune on a single speaker? HOT 1
- How to adjust this result in coding? HOT 3
- FileNotFoundError: [Errno 2] No such file or directory: 'checkpoints/vox.pth.tar' HOT 2
- [Feature request] wav2lip for 3d models's "bones"
- Still no 512 or 768 pre trained model? HOT 21
- Visualizing keypoints
- train 512 or higher resolution HOT 3
- Image animation given only keypoint vectors. HOT 1
- Is this project still maintain ?
- I have a problem with curling HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from thin-plate-spline-motion-model.