Comments (5)
我根据代码观察,发现在提取mel时进行了裁剪,是不是每一个训练的视频数据都是裁剪成了时序长度为5的mel数据和imge数据,然后放入训练,实际训练的时候只是取了时序长度为5的视频长度部分,生成的generated["prediction"]也是5张图片,而并未使用整个视频。是这样吗? 🙏
from eat_code.
你好,这里的设置应该是为了兼容针对第一阶段的需要继续训练的情况,不会影响第二阶段的训练。a2kp的模型在emotion adaptation阶段因为加了一些EAM层,并只优化这些层,所以优化参数已经发生了改变,并不能用第一阶段的optimizer_a2kp,在这里是正常的。
from eat_code.
你好,这里的设置应该是为了兼容针对第一阶段的需要继续训练的情况,不会影响第二阶段的训练。a2kp的模型在emotion adaptation阶段因为加了一些EAM层,并只优化这些层,所以优化参数已经发生了改变,并不能用第一阶段的optimizer_a2kp,在这里是正常的。
万分感谢你的耐心解答,关于数据中的x["mel"]和x["sync_mel"]有什么区别哇,为什么shape不一样,sync_mel是[1, 1, 80, 16],mel是[1, 5, 1, 80, 11],其中5表示的应该是同步步长吧,为什么mel里面就没有
from eat_code.
mel是语音的梅尔频谱,是网络的输入之一。sync_mel是用来计算sync loss的。sync_mel的形状原因可以查看wav2vec这篇论文及其代码。这个的实现可以参考这里的定义和使用.
from eat_code.
是的,因为用了wav2vec训出来的模型,为了计算sync_loss,需要一次生成5张图片,每张图片的生成又需要临近帧的特征,所以在这里提取mel时会有两重for循环。
from eat_code.
Related Issues (20)
- RuntimeError: The size of tensor a (165) must match the size of tensor b (66) at non-singleton dimension 1 on my training dataset HOT 1
- AssertionError: Caught AssertionError in DataLoader worker process 0. HOT 2
- 关于训练收敛的问题 HOT 3
- How to edit with text and interpolate states as described in the paper? HOT 2
- Would it be complicated to add eyeblinking reference videos? HOT 1
- ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (2,) + inhomogeneous part. HOT 2
- Generated Video Background Is Not Static HOT 3
- Any way support for Chinese audio? HOT 3
- 3D Keypoint Enhancement Training HOT 2
- Details about CLIP fine-tuning and zero-shot text-guided editing HOT 11
- FileNotFoundError: Package has no location <module 'imageio_ffmpeg.binaries' (namespace)> HOT 2
- Inquiry Source Data Preparation HOT 2
- Error in saving .npy file HOT 2
- How to generate files for custom audio? HOT 4
- 关于预训练使用的权重 HOT 3
- mead数据压缩问题 HOT 1
- Inquiry Regarding Preprocessing VOX2 and MEAD Dataset for Training HOT 8
- Fail to get output from demo.py HOT 2
- Can anyone share TED-Talk dataset?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from eat_code.