haofanwang / visbeat3

Python3 Implementation for 'Visual Rhythm and Beat' SIGGRAPH 2018

visbeat3

This is a migration of visbeat from Python2 to Python3. Head over to an example notebook on Colab!

This work presents a visual analogue for musical rhythm derived from an analysis of motion in video, and shows that alignment of visual rhythm with its musical counterpart results in the appearance of dance. You can visit the official project page and demos for more infos.

Install

$ pip3 install visbeat3

Usage

import os
import time

import matplotlib
import matplotlib.pyplot as plt

import numpy as np
import visbeat3 as vb

source_video = vb.PullVideo(name='video', source_location='./data/01.mp4')

# You can also directly specify an audio
# source_audio = vb.Audio('./data/all star.mp3')
source_audio = vb.PullVideo(name='audio', source_location='./data/02.mp4')

synch_video_beat = 0
synch_audio_beat = 0
nbeats = 32

output_path = './result.mp4'

# If source_audio is from an audio file, use target=source_audio 
warped = vb.Dancify(source_video=source_video, 
                    target=source_audio.getAudio(), 
                    synch_video_beat=synch_video_beat,
                    synch_audio_beat=synch_audio_beat, 
                    force_recompute=True, 
                    warp_type='quad',
                    nbeats=nbeats, 
                    output_path=output_path)

Used By

video-bgm-generation: video background music generation.

Reference

@inproceedings{davis2018visual,
  title={Visual rhythm and beat},
  author={Davis, Abe and Agrawala, Maneesh},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops},
  pages={2532--2535},
  year={2018}
}

visbeat3's Issues

not work

os.rename(self.getJSONPath(), self.getDir('backup') + os.sep + self.AOBJECT_TYPE() + ".json");
FileExistsError: [WinError 183] 当文件已存在时，无法创建该文件。: './VisBeatAssets/VideoSources\video\VideoSource.json' -> './VisBeatAssets/VideoSources\video\Data\Backups\\VideoSource.json'

Function to get the video spectrogram

Can we extract the video spectrogram using a function directly into a numpy array?

Fix imageio-ffmpeg bug with Dancify inference

I was getting this error after Dancify inference. A series of errors were as follows:

Fatal Python error: could not acquire lock for <_io.BufferedReader name=11> at interpreter shutdown, possibly due to daemon threads

This was due to a bug with imageio-ffmpeg not being the latest version, or not being installed.

pip install imageio-ffmpeg --upgrade # If already installed

Bugs find in visbeat3/Video_CV.py

line 160: nsamples should be explicitly set as an integer -- "nsamples = int(sampling_rate*duration);"
line 244: "librosa.beat.estimate_tempo" should be changed as "librosa.beat.tempo"
line 418: histsize should be explicitly set as an integer -- "histsize = int(128/int(np.power(2,HISTOGRAM_DOWNSAMPLE_LEVELS)))"

Fix Dancify Inference

Dancify wasn't working for me, but I found the issue in my case.
There's a bug with the new update that doesn't allow Dancify to run properly. This fixes it.

Change this:

visbeat3/visbeat3/Image.py

Line 32 in 4714076

USING_OPENCV=False;

To True:

USING_OPENCV=True;

It could also be removed because OpenCV seems to be a prerequisite, but doing so may also break the install.