lysdexic-audio / n4m-handpose

Wraps MediaPipe HandPose inside electron and serves the detected parts via MaxAPI.

n4m-handpose v1.0.1

Wraps MediaPipe Handpose inside electron and serves the detected parts via MaxAPI.

Based around Yuichi Yogo's great work porting Electron + Tensorflow models into Node For Max

• Handpose is being used in Arthur Parmentier's Soundpainting Project
• Are you using n4m-handpose in your project? Let me know

Steps (MaxMSP)

1. Install npm dependencies by clicking the indicated button. Since Electron's kind of big in size, this make take a while depending on your network environment. When the message object says it's 'completed' you'll never have to do this step again.
2. Click on the toggle button to launch n4m-handpose.
3. The node.script emits the detected results as a dictionary, which has been parsed into (x,y,z) lists of each finger's tip positions & palmbase
4. All landmarks are also aggregated to a list to send to Wekinator if you'd like to perform gesture recognition with the data (sending to port 11000)

Steps (MaxForLive)

• n4m-handposeM4L (Max For Live Device) AMXD coming in 1.1

Changelog

• Skeleton color can be changed using GUI
• Settings are persisted using electron-store
• Handpose model controls now available in electron window UI
• Skeleton fill and stroke colour available in electron window UI
• Handpose model control settings and skeleton fill and stroke colour save automatically and persist/reload
• Render window on or offscreen using arguments to node.script
• Hand landmarks are scaled and rendered to a jitter window (m4l + maxmsp)

n4m-handpose's Issues

this browser does not support video capture,or this device does not have a camera

Big fan of this tool, Thanks for sharing, but cannot get to recognize camera anymore.

Summary or problem
After downloading modules i start the script. No user interface is visible and I receive this error after 1 minute: "this browser does not support video capture, or this device does not have a camera" I do not get this with facetracking tool which is working fine.

When I open 'developer tools' 1 error and 3 warnings : shown in order of appearance_

Warning:

1. Platform browser has already been set. Overwriting the platform with [object Object].
   Environment.setPlatform @ C:\Users\dante\Documents\Max 8\Packages\n4m-Bundler\n4m-handpose-master\n4m-handpose-master\node_modules@tensorflow\tfjs-core\dist\environment.js:38

2. (electron) Security Warning: webFrame.executeJavaScript was called without worldSafeExecuteJavaScript enabled. This is considered unsafe. worldSafeExecuteJavaScript will be enabled by default in Electron 12.
   electron/js2c/renderer_init.js:15

3. Electron Security Warning (Insecure Content-Security-Policy) This renderer process has either no Content Security electron/js2c/renderer_init.js:135 Electron Security Warning (Insecure Content-Security-Policy) This renderer process has either no Content Security. electron/js2c/renderer_init.js:135

4. ERROR: Uncaught (in promise) DOMExecption: Could not start video source - n4m-handpose_camera.js 445

5. WARNING: DevTools failed to load SourceMap: Could not load content for file:///C:/Users/dante/Documents/Max%208/Packages/n4m-Bundler/n4m-handpose-master/n4m-handpose-master/node_modules/ajv.js.map: _This warning occurs for every module (my guess) there are like a hundred all leading to the modules folder. _

system Windows 11, Max 8.1.11

Intermittent crashes on MacOS (Mojave)

Cause uncertain. Seems to only occur when filling the max dictionary, with the gate to the dict closed crashes don't seem to occur.

Add GUI parameters / fix crashes mentionned in camera.js

Hello!

It would be really nice to get more of the GUI to change some confidence parameters and control more of the model.

In the camera.js file, I see that things were commented because of crashes. Maybe things could be taken from the 2020 example from tensorflow?

https://github.com/tensorflow/tfjs-models/blob/master/handpose/demo/index.js

Install npm

Hi,
Thanks for sharing this fantastic tool.
I want to try and work on it, but the installation is not working on my computer. (Install dependencies from npm. Wait until the box below says 'completed.')
I tried a few times and waited very long, and nothing happened. It can not download the files.
Is there another way to download the files and run the software on Max?

I am using max 8.5 on a MacBook Pro (macOS 12.5).

Greetings,
Ali

Download HandPose locally

Hello :)

Thanks to you, I am soon making a pre-release of my SP recognition tool!

Because I am going to use it during the summer in places where there is no internet, I was wondering if there was a way to download the model locally rather than fetching it from the web. I would dig into that after my master thesis defence, but perhaps you would know how to make it?

Also, just let me know if you could not solve the crashes and add gui parameters for some reason, I could try my best to gelp with that also!

Arthur