charliegerard / beat-pose

Play a Beat Saber clone with hand movements using Tensorflow.js & PoseNet

BeatPose

Play a Beat Saber clone with hand movements using Tensorflow.js & the MoveNet model, in the browser! 🎉

Based on the awesome work by Supermedium on the beat saver viewer.

Live demo: https://beat-pose.netlify.com/

Blog post

Demo

How to use

• Visit the live demo
• Allow access to the webcam
• Place yourself about 1-2m away from your computer in a place that has enough light.
• Move your hands to see if the recognition works
• Start playing and hit beats!

Rules

None! It's not a real game, more of an experiment 🙂.

The only "restriction" is that the left hand can only destroy the red beats and the right hand, the blue beats.

Tech stack

• A-Frame
• Tensorflow.js
• PoseNet model
• Three.js

Development

⚠️ The game was originally built with Node.js v8.9.1 which is pretty old. The install fails on more recent versions.

To run locally, clone this repository and run:

npm install
npm run start

As an alternative, you can run the index.html file located at the root with a python server like python -m HTTPServer 5000 for example.

Then open localhost (on port 9999 if you ran it with npm) in your browser and the default song should be Bohemian Rhapsody - Queen, but you can search for other songs in the bottom search bar.

Support

These kinds of side projects take me a lot of time so, if you like them, your support would be really appreciated if you want to/can 🙂.

beat-pose's Issues

PoseNet is not accurate and unstable whatever people pose

I am running PoseNet on PC and saw that whatever people pose, the model can't detect the legs of the person, does this is something that is happening only to me or is a general problem. If so is there any way I can improve the accuracy of PoseNet ?

Running on PC: Micro-Star (Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz 2.20 GHz)
-Also why its so unstable, I mean I stay in one place and points keep changing places

-Can it work better on black and white frames ?

-Should I have a dark background image?

npm install failed

npm install error occured on windows10 x64