xrenaa / music-dance-video-synthesis

[ACM MM 20 Oral] PyTorch implementation of Self-supervised Dance Video Synthesis Conditioned on Music

Hi there 👋

music-dance-video-synthesis's Issues

Cannot Get Reported Result With Default Setting

Thanks for the great work. I've tried to run the code as instructed using the clean part data. The results are far from impressive as shown in the demo video (like this). Here is the loss curve:

I wonder if I have to change some settings to make it work. Any help is appreciated!

How can I use the generated matchmaker to generate a gif of the character again?

Hello author, now I have got the generated matchmaker, but I don't know how to use the matchmaker to generate the corresponding character gifs, does the code you uploaded include this part? Or do I need to go to everyone dance now to download the code to generate the corresponding gifs?

Details about your paper?

Hi,

1. what's the difference between the clean part and the noisy part?
2. which training and test data did you use to obtain the results in your Table 1?
3. How to map the generated skeletons to the images? Are you retrain the model? If so, where are the ground truth images for training?

Thanks a lot.

Demo question

Hello,

I have few questions about demo

1. how can i use my own music for demo?

2. which get_demo.py is correct one? there are multiple of them on the root folder as well as under Demo folder

3. I get similar bug as in #3 . I added audio folder in demo folder, but i get same error for output folder now

Thank you

Cross-modal Evaluation code

Hi,

I found that you have provided the second part of the cross-modal evaluation in another github page.

I wonder if it is possible for you to further provide the followings:

1. the first part of the cross-modal evaluation. (The part that create a dictionary mapping music embedding to dance sequence)
2. For now, there is only training code on github for the second part of the cross-modal evaluation. How can I use this trained model to get the score reported in the paper? Also, can you provide the pretrain model? (the one you used for generating the result in paper)

Thanks!

Is there a code to assess FID and diversity?

Is there a code to assess FID and diversity?

When measuring fid, the features extracted by stgcn is a list, how to deal with it afterwards?

When measuring fid, the features extracted by stgcn is a list, how to deal with it afterwards?

How can I use my own dataset to train network?

Hello. First of all, thank you for the great work. I'm so impressed by this project and trying to use my own dataset to train this network. However, I have no idea about making my dataset to json file that 'train.py' can read. How should I process my dataset? Thank you.

demo question

你好，
我跑起來出現了如下的異常

另外有些疑問：
1，如果要輸入原始視頻，重新訓練該怎麼做，就是那些json文件是怎麼生成的
2，怎麼輸入音頻
3，這個項目和https://github.com/NVlabs/Dancing2Music 哪個更好呢？

FileNotFoundError: [Errno 2] No such file or directory: './dataset/clean_revised_pose_pairs.json'

when I run python3 get_demo.py --output ./output/

raise:
'FileNotFoundError: [Errno 2] No such file or directory: './dataset/clean_revised_pose_pairs.json'

where should I get this file ?

thx!

Testing error

Hi, I am trying to train and test on other datasets you provided such as Ballet and Popping. However, the training is good but I am meeting the following error when testing, can you help me to fix it? Thanks!

Traceback (most recent call last):
File "get_demo.py", line 73, in
data=DanceDataset(args.data)
File "/data0/htang/projects/Music-Dance-Video-Synthesis/dataset/lisa_dataset_test.py", line 22, in init
sub_keys=sorted(pose_dict_boy[str(key)].keys())
KeyError: '047'

How to evaluate Diversity?

data usage question

Why pose keypoints processed by this:

Music-Dance-Video-Synthesis/dataset/lisa_dataset_test.py

Line 48 in f6283c0

x_coor=(temp_pose[:,:,0]/320)-1

Is the dance video size is 320 * 180? But some point is [350.1983515423332, 296.6837658604959]

It really confused me, any help would be appreciated

Can you share the video frames as shown in Fig. 6 of your paper?

music feature extraction

Hi author,thank you for your great work!
I have read your paper and I have a question that how to extrat the music feature ? If i want to use my music as input , how should I extract the feature from music?

FileNotFoundError: [Errno 2] No such file or directory: './dataset/clean_revised_pose_pairs.json' & RuntimeError: DataLoader worker (pid(s) 9784, 7708, 11984, 11008, 3636, 15428, 10724, 1548, 2312, 12376, 1792) exited unexpectedly

Run time

python train.py

I get the following error
RuntimeError: DataLoader worker (pid(s) 9784, 7708, 11984, 11008, 3636, 15428, 10724, 1548, 2312, 12376, 1792) exited unexpectedly
Help to fix the error

&

Run time

python get_demo.py --output the_output_path

I get the following error
FileNotFoundError: [Errno 2] No such file or directory: './dataset/clean_revised_pose_pairs.json'
Help to fix the error

Thank

Demo bugs

Hi, I re-trained the model, but I have bugs in get_demo.py.

And how could I generate a gif? Thanks a lot.