juliapoo / multihand-tracking

A python wrapper for Mediapipe's Multi-Hand Tracking

Heyo! ^-^/

I'm currently a year 1 2 university student, with interest in programming languages and mathematics. I like writing small experimental stuff and occasionally tools. Available for internships~~

You can talk to me at [email protected]

Also I now have an (uncompleted) website!

multihand-tracking's Issues

palm_detection_without_custom_op.tflite

Please tell me how to do this.. thanks in advance

Mistake in transforming 3D coordinates from cropped image to original image

Hi, thanks for the great work! I would like to point out a small mistake in MultiHandTracker3D.

In multi_hand_tracker.py, line 884-889, it's transforming 2D coordinates from cropped image back to original image with 2D similarity transform Minv. However, here the z coordinates are still in cropped image and not transformed.

            kp_orig_0 = (self._pad1(joints[:,:2]) @ Minv.T)[:,:2]
            kp_orig_0 -= pad[::-1]
            
            # Add back the 3D data
            kp_orig = joints[:,:]
            kp_orig[:,:2] = kp_orig_0[:,:2]

The z coordinates need to be scaled as well, so I did the following:

            kp_orig_0 = (self._pad1(joints[:,:2]) @ Minv.T)[:,:2]
            kp_orig_0 -= pad[::-1]

            scale = np.linalg.norm(Minv[0, :2])

            # Add back the 3D data
            kp_orig = joints[:,:]
            kp_orig[:,:2] = kp_orig_0[:,:2]
            # also scale the z coordinates
            kp_orig[:, 2] *= scale

Thanks!

How can I use hand_tracking with my webcam?

Hi @JuliaPoo thank you very much for this mediapipe wrapper for python!
I have a pretty noob question, I am new to this environment and I would like to use my webcam to use the hand tracker, is this possible? You think you could upload a mini example :( I've been trying using videocapture from OpenCV, but it didn't work for me.

Do we need a post-processing on the depth data?

Hi @JuliaPoo , many thanks for providing this Python wrapper for Google's Mediapipe. But I'm wondering if we need to do some post-processing on the depth data rather concatenate them to processed 2D landmarks for outputting directly? Since I believe the scale of raw hand 2D landmarks data is unrelated to the original image, and so we need to do some post-processing to let the hand 2D landmarks project onto the original full-size image. Should we also do that on the depth data? Because it looks not correctly if plotting the 3D hand landmarks result on a 3D plot. Thanks.

Tensorflow version

Hey Julia,
I am using tf version 1.8, python 3.6.8.
I can import tensorflow.contrib.lite without errors but I cant import tensorflow.contrib.lite.interpreter.

Which version tensorflow did you use?

what's the meaning of "self._target_triangle" and "self._target_box"

Thanks for your mediapipe wrapper for python!
I'm reading your multi_hand_tracker.py. But I still can't understand what's the meaning of "self._target_triangle" and "self._target_box".
I found they represents some point in 256256 picture, but why or how should we utilize them to transform palm detections(in 256256) to bbox in original picture. Could you please give me a more detailed explanation?