garfield-kh / tm2d Goto Github PK

Hi expert,
nice work to music-text to control motion, and I have studied code and paper in which music features from librosa, but I only search npy files.
May you tell me how to extract features from librosa?

I am very honored to see such a great job, and I am trying to reproduce it. There are a question:
./tm2d_60fps/eval4bailando/ # download from google drive
Where can I download eval4bailando file? I couldn't find it on google drive.

In Table1, how to compute the FID and Div? I think the processing can not produce keypoints3D for geometric features and kinetic features.

When I run the visualization, all I can see is the skeleton model of the character. But I want to see the character model demonstrated in Project bailando. Can you help me? Thank you very much.

I find that Bailando folder does not have smpl folder, how can I download it?

Hi @Garfield-kh, thanks for the great work. I am curious about the metric you proposed and have 3 questions.

1. How to evaluate the MPD and Freezing Score (PFF and AUC) in your code?

2. What do you mean about the following description in "4.2. Evaluation on Music-text Conditioned Dance Generation" in the paper?

   We use the past 25 motion frames to predict the future 30 frames, and calculate the MPD from future frame (ft) = 10 to ft = 30, respectively.

   i.e. how do you calculate the MPD from future frame (ft) = 10 to ft = 30 and present the result in Table 2? cause I cannot map this to the definition of MPD.

3. How do you evaluate the FID for in-the-wild music in Table 1? As you mentioned in "4.3. Evaluation on Music Conditioned Dance Generation":

   This is because FACT [30] and Bailando [47] requires seed motion, however, there is no ground-truth for in-the-wild scenario.

It would be of great help if you could reply.
Thanks.

Hi @Garfield-kh , Thanks for your reply to the previous question. And now I have another question. How to get audio features from MP3 text and save them as npy files. I can't find the relevant generation process in your file. Can you tell me how this should be generated, or where the file is. Thank you very much.
The reason is when I new a mp3 file and then I want to extracrt audio feature into npy file. But I encounter a mismatch between the input matrix dimensions and the weight matrix dimensions. I think a lot of this is due to the small dimension size of my npy file. So can you help me ? Thank you very much.

haven't found environment.yaml or the requierments.txt

Hi, congrats on your great work!
In dataset.py, there is opt.tokenizer_name_audio. I wonder if you can share how you obtained tokenized audio in the aistpp dataset. Thanks!

Hi, this is a great work. But I don't look at the method that how to get the text descriptions for dance in the paper. Can you help me with this question?

HumanML3D has 22 joints (no hand joints with indices 22 and 23). AIST++ has full 24 joints. TM2D uses 24 joints, and how do you convert the 22 joints defined by the HumanML3D to the 24 joints defined by the AIST++？

Hi, TM2D is a fantastic job! I'd like to know when will the code and dataset for testing be available. Thanks!