Good evening,

Thank you for publishing this project. I have found extracting landmark data from MediaPipe to be a bit confusing, so I appreciate any and all examples.

I was able to follow all of your steps, and even was able to build the project using the bazel build -c opt --define MEDIAPIPE_DISABLE_GPU=1 mediapipe/examples/desktop/hand_tracking:hand_tracking_cpu command.

But when I try to run the project using the GLOG_logtostderr=1 bazel-bin/mediapipe/examples/desktop/hand_tracking/hand_tracking_cpu --calculator_graph_config_file=mediapipe/graphs/hand_tracking/hand_tracking_desktop_live.pbtxt I run into an error.

The error is as follows:

E20201022 20:22:38.307886 58398 demo_run_graph_main.cc:163] Failed to run the graph: ValidatedGraphConfig Initialization failed.
LabelsToRenderDataCalculator: ; Must provide input stream "LABELS"lators/util/labels_to_render_data_calculator.cc:81)

So it looks like the graph is looking for an input stream called LABELS, but I haven't been able to find that as of right now. Is this something you ran into while developing this example? Perhaps you know of a quick fix that I haven't seen?

Thanks a ton! 😄

I've trying for around 3 weeks, but "transfer" the tutorial from Xcode to Visual studio is a little more complex than a imagined. Is this even posible? or Am I doing something that is just avaluable for Mac?

Hey Madeline! Thanks for this awesome example, looking forward to getting it up and running.

My question is where is example-protobuf-udp? Or am I to create it in projectGenerator and add the files as described?

I tried doing the latter but received an error that none of the "google/protobuf/..." files were found. So I updated the locations of all of them to point to the google/protobuf directory in my usr folder. This got the application to compile but only led to a blank window.

I'm not sure what I'm missing here, would be grateful for your insight on this.

Thanks again!

Hi Madeline, First thank you for taking the time to post this great example. I am trying to get it running, and have checked and double checked the changes.. it builds fine, but then I get this error trying to run it.
Any help would be much appreciated . I am running MacOs, Catalina 10.15.6. Proto is 3.6.1 .

[libprotobuf ERROR external/com_google_protobuf/src/google/protobuf/text_format.cc:309] Error parsing text-format mediapipe.CalculatorGraphConfig: 40:59: Could not find type "type.googleapis.com/mediapipe.GateCalculatorOptions" stored in google.protobuf.Any.
F20200901 22:07:08.679298 520498624 parse_text_proto.h:32] Check failed: ParseTextProto(input, &result)
*** Check failure stack trace: ***
@ 0x10f9a244f google::LogMessageFatal::~LogMessageFatal()
@ 0x10f99ede9 google::LogMessageFatal::~LogMessageFatal()
@ 0x10f4fd7bb mediapipe::ParseTextProtoOrDie<>()
@ 0x10f4fb899 RunMPPGraph()
@ 0x10f4fd815 main
zsh: abort sudo GLOG_logtostderr=1

_live.pbtxt
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1705417055.522904 30032 demo_run_graph_main.cc:50] Get calculator graph config contents: # MediaPipe graph that performs hands tracking on desktop with TensorFlow

Lite on CPU.

Used in the example in

mediapipe/examples/desktop/hand_tracking:hand_tracking_cpu.

CPU image. (ImageFrame)

input_stream: "input_video"

CPU image. (ImageFrame)

output_stream: "output_video"

Generates side packet cotaining max number of hands to detect/track.

node {
calculator: "ConstantSidePacketCalculator"
output_side_packet: "PACKET:num_hands"
node_options: {
[type.googleapis.com/mediapipe.ConstantSidePacketCalculatorOptions]: {
packet { int_value: 2 }
}
}
}

Detects/tracks hand landmarks.

node {
calculator: "HandLandmarkTrackingCpu"
input_stream: "IMAGE:input_video"
input_side_packet: "NUM_HANDS:num_hands"
output_stream: "LANDMARKS:landmarks"
output_stream: "HANDEDNESS:handedness"
output_stream: "PALM_DETECTIONS:multi_palm_detections"
output_stream: "HAND_ROIS_FROM_LANDMARKS:multi_hand_rects"
output_stream: "HAND_ROIS_FROM_PALM_DETECTIONS:multi_palm_rects"
}

Subgraph that renders annotations and overlays them on top of the input

images (see hand_renderer_cpu.pbtxt).

node {
calculator: "HandRendererSubgraph"
input_stream: "IMAGE:input_video"
input_stream: "DETECTIONS:multi_palm_detections"
input_stream: "LANDMARKS:landmarks"
input_stream: "HANDEDNESS:handedness"
input_stream: "NORM_RECTS:0:multi_palm_rects"
input_stream: "NORM_RECTS:1:multi_hand_rects"
output_stream: "IMAGE:output_video"
}

Add New Node

node {
calculator: "MyPassThroughCalculator"
input_stream: "LANDMARKS:hand_landmarks"
input_stream: "NORM_RECT:hand_rect"
input_stream: "DETECTIONS:palm_detections"
output_stream: "LANDMARKS:hand_landmarks_out"
output_stream: "NORM_RECT:hand_rect_out"
output_stream: "DETECTIONS:palm_detections_out"
}

Modify input_stream names of next node

Subgraph that renders annotations and overlays them on top of the input

images (see renderer_cpu.pbtxt).

node {
calculator: "RendererSubgraph"
input_stream: "IMAGE:input_video"
input_stream: "LANDMARKS:hand_landmarks_out"
input_stream: "NORM_RECT:hand_rect_out"
input_stream: "DETECTIONS:palm_detections_out"
output_stream: "IMAGE:output_video"
}

I0000 00:00:1705417055.523425 30032 demo_run_graph_main.cc:56] Initialize the calculator graph.
E0000 00:00:1705417055.525294 30032 demo_run_graph_main.cc:156] Failed to run the graph: ValidatedGraphConfig Initialization failed.
No registered object with name: RendererSubgraph; Unable to find Calculator "RendererSubgraph"

Hi @madelinegannon .

Thank you so much for this and for the very detailed instructions. I tried running

bazel build -c opt --define MEDIAPIPE_DISABLE_GPU=1     mediapipe/examples/desktop/hand_tracking:hand_tracking_cpu
GLOG_logtostderr=1 bazel-bin/mediapipe/examples/desktop/hand_tracking/hand_tracking_cpu     --calculator_graph_config_file=mediapipe/graphs/hand_tracking/hand_tracking_desktop_live.pbtxt

and get the following errors:

WARNING: Download from https://mirror.bazel.build/github.com/tensorflow/tensorflow/archive/805e47cea96c7e8c6fccf494d40a2392dc99fdd8.tar.gz failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException GET returned 404 Not Found
WARNING: Download from http://mirror.tensorflow.org/github.com/bazelbuild/rules_closure/archive/cf1e44edb908e9616030cc83d085989b8e6cd6df.tar.gz failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException GET returned 404 Not Found
DEBUG: Rule 'rules_cc' indicated that a canonical reproducible form can be obtained by modifying arguments sha256 = "c839ff09c038f291a6935c8fc2a5874b4c4612059ba8f54314cbf39186e108d4"
DEBUG: Repository rules_cc instantiated at:
  no stack (--record_rule_instantiation_callstack not enabled)
Repository rule http_archive defined at:
  /private/var/tmp/_bazel_frederikjuutilainen/aacdf75d08bc263b7b69433e3bb50764/external/bazel_tools/tools/build_defs/repo/http.bzl:336:16: in <toplevel>
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/protocolbuffers/protobuf/archive/v3.11.4.tar.gz failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException GET returned 404 Not Found
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/google/XNNPACK/archive/05702cf4099ad019ad1abb8ba656bfe04304f32a.zip failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException GET returned 404 Not Found
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/Maratyszcza/pthreadpool/archive/76042155a8b1e189c8f141429fd72219472c32e1.tar.gz failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException GET returned 404 Not Found
INFO: Analyzed target //mediapipe/examples/desktop/hand_tracking:hand_tracking_cpu (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //mediapipe/examples/desktop/hand_tracking:hand_tracking_cpu up-to-date:
  bazel-bin/mediapipe/examples/desktop/hand_tracking/hand_tracking_cpu
INFO: Elapsed time: 1.161s, Critical Path: 0.00s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
I0427 16:12:21.331287 274976192 demo_run_graph_main.cc:47] Get calculator graph config contents: # Add New Node
node {
  calculator: "MyPassThroughCalculator"
  input_stream: "LANDMARKS:hand_landmarks"
  input_stream: "NORM_RECT:hand_rect"
  input_stream: "DETECTIONS:palm_detections"
  output_stream: "LANDMARKS:hand_landmarks_out"
  output_stream: "NORM_RECT:hand_rect_out"
  output_stream: "DETECTIONS:palm_detections_out"
}

# Modify input_stream names of next node

# Subgraph that renders annotations and overlays them on top of the input
# images (see renderer_cpu.pbtxt).
node {
  calculator: "RendererSubgraph"
  input_stream: "IMAGE:input_video"
  input_stream: "LANDMARKS:hand_landmarks_out"
  input_stream: "NORM_RECT:hand_rect_out"
  input_stream: "DETECTIONS:palm_detections_out"
  output_stream: "IMAGE:output_video"
}
I0427 16:12:21.333701 274976192 demo_run_graph_main.cc:53] Initialize the calculator graph.
E0427 16:12:21.334813 274976192 demo_run_graph_main.cc:146] Failed to run the graph: ValidatedGraphConfig Initialization failed.
No registered object with name: MyPassThroughCalculator; Unable to find Calculator "MyPassThroughCalculator" 

From the error it seems that MyPassThroughCalculator cannot be found, but I have double-checked that I did indeed copy the file src/mediapipe/my_pass_though_calculator.cc to my Mediapipe calculators directory.

ls mediapipe/calculators/core 
                                     
BUILD                                             matrix_to_vector_calculator_test.cc
add_header_calculator.cc                          merge_calculator.cc
add_header_calculator_test.cc                     merge_calculator_test.cc
begin_end_loop_calculator_graph_test.cc           mux_calculator.cc
begin_loop_calculator.cc                          my_pass_through_calculator.cc
begin_loop_calculator.h                           packet_cloner_calculator.cc
clip_detection_vector_size_calculator.cc          packet_cloner_calculator.proto
clip_vector_size_calculator.cc                    packet_inner_join_calculator.cc
clip_vector_size_calculator.h                     packet_inner_join_calculator_test.cc
clip_vector_size_calculator.proto                 packet_resampler_calculator.cc
clip_vector_size_calculator_test.cc               packet_resampler_calculator.h
concatenate_detection_vector_calculator.cc        packet_resampler_calculator.proto
concatenate_vector_calculator.cc                  packet_resampler_calculator_test.cc
concatenate_vector_calculator.h                   packet_thinner_calculator.cc
concatenate_vector_calculator.proto               packet_thinner_calculator.proto
concatenate_vector_calculator_test.cc             packet_thinner_calculator_test.cc
constant_side_packet_calculator.cc                pass_through_calculator.cc
constant_side_packet_calculator.proto             previous_loopback_calculator.cc
constant_side_packet_calculator_test.cc           previous_loopback_calculator_test.cc
counting_source_calculator.cc                     quantize_float_vector_calculator.cc
dequantize_byte_array_calculator.cc               quantize_float_vector_calculator.proto
dequantize_byte_array_calculator.proto            quantize_float_vector_calculator_test.cc
dequantize_byte_array_calculator_test.cc          round_robin_demux_calculator.cc
end_loop_calculator.cc                            sequence_shift_calculator.cc
end_loop_calculator.h                             sequence_shift_calculator.proto
flow_limiter_calculator.cc                        sequence_shift_calculator_test.cc
flow_limiter_calculator_test.cc                   side_packet_to_stream_calculator.cc
gate_calculator.cc                                side_packet_to_stream_calculator_test.cc
gate_calculator.proto                             split_normalized_landmark_list_calculator.cc
gate_calculator_test.cc                           split_normalized_landmark_list_calculator_test.cc
immediate_mux_calculator.cc                       split_vector_calculator.cc
immediate_mux_calculator_test.cc                  split_vector_calculator.h
make_pair_calculator.cc                           split_vector_calculator.proto
matrix_multiply_calculator.cc                     split_vector_calculator_test.cc
matrix_multiply_calculator_test.cc                stream_to_side_packet_calculator.cc
matrix_subtract_calculator.cc                     stream_to_side_packet_calculator_test.cc
matrix_subtract_calculator_test.cc                string_to_int_calculator.cc
matrix_to_vector_calculator.cc

I have also followed step 3 and appended to the BUILD file.

Sorry for this, it's probably just an stupid overlook on my part. :)

All the best,
Frederik

I am a beginner, so I do not understand very well waht does this means. I am doing this in Visual Studio 2017 on Windows 10, so I do not know i some problems are because of the fact that you are doing this in a Mac. This is the Error:

PS C:\Users\orest> cd C:\Users\orest\mediapipe_repo\mediapipe
PS C:\Users\orest\mediapipe_repo\mediapipe> bazel build -c opt --define MEDIAPIPE_DISABLE_GPU=1 mediapipe/examples/
desktop/hand_tracking:hand_tracking_cpu
Starting local Bazel server and connecting to it...
INFO: Repository local_execution_config_python instantiated at:
no stack (--record_rule_instantiation_callstack not enabled)
Repository rule local_python_configure defined at:
C:/users/orest/_bazel_orest/6wjl4ovg/external/org_tensorflow/third_party/py/python_configure.bzl:275:41: in <toplevel

ERROR: An error occurred during the fetch of repository 'local_execution_config_python':
Traceback (most recent call last):
File "C:/users/orest/_bazel_orest/6wjl4ovg/external/org_tensorflow/third_party/py/python_configure.bzl", line 2
10
_get_python_lib(repository_ctx, <1 more arguments>)
File "C:/users/orest/_bazel_orest/6wjl4ovg/external/org_tensorflow/third_party/py/python_configure.bzl", line 1
30, in _get_python_lib
execute(repository_ctx, <1 more arguments>)
File "C:/users/orest/_bazel_orest/6wjl4ovg/external/org_tensorflow/third_party/remote_config/common.bzl", line
208, in execute
fail(<1 more arguments>)
Repository command failed
File "", line 1
SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 2-3: truncated \UXXXXXXXX escape
ERROR: Analysis of target '//mediapipe/examples/desktop/hand_tracking:hand_tracking_cpu' failed; build aborted: Traceba
ck (most recent call last):
File "C:/users/orest/_bazel_orest/6wjl4ovg/external/org_tensorflow/third_party/py/python_configure.bzl", line 2
10
_get_python_lib(repository_ctx, <1 more arguments>)
File "C:/users/orest/_bazel_orest/6wjl4ovg/external/org_tensorflow/third_party/py/python_configure.bzl", line 1
30, in _get_python_lib
execute(repository_ctx, <1 more arguments>)
File "C:/users/orest/_bazel_orest/6wjl4ovg/external/org_tensorflow/third_party/remote_config/common.bzl", line
208, in execute
fail(<1 more arguments>)
Repository command failed
File "", line 1
SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 2-3: truncated \UXXXXXXXX escape
INFO: Elapsed time: 11.561s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (11 packages loaded, 11 targets configured)

Hi
i can not modify a graph just like yours
can you share your code in hand_tracking_desktop_live.pbtxt
i am so new to mediapipe and really want to understand graphs and calculators

thanks :)

i compile holistictracking by the build content:
`load(
"@build_bazel_rules_apple//apple:ios.bzl",
"ios_application",
)
load(
"//mediapipe/examples/ios:bundle_id.bzl",
"BUNDLE_ID_PREFIX",
"example_provisioning",
)
load("@build_bazel_rules_apple//apple:ios.bzl", "ios_static_framework")

licenses(["notice"])

MIN_IOS_VERSION = "10.0"

alias(
name = "holistictrackinggpu",
actual = "HolisticTrackingGpuApp",
)

ios_application(
name = "HolisticTrackingGpuApp",
app_icons = ["//mediapipe/examples/ios/common:AppIcon"],
bundle_id = BUNDLE_ID_PREFIX + ".HolisticTrackingGpu",
families = [
"iphone",
"ipad",
],
infoplists = [
"//mediapipe/examples/ios/common:Info.plist",
"Info.plist",
],
minimum_os_version = MIN_IOS_VERSION,
provisioning_profile = example_provisioning(),
deps = [
":HolisticTrackingGpuAppLibrary",
"@ios_opencv//:OpencvFramework",
],
)

objc_library(
name = "HolisticTrackingGpuAppLibrary",
data = [
"//mediapipe/graphs/holistic_tracking:holistic_tracking_gpu.binarypb",
"//mediapipe/modules/face_detection:face_detection_short_range.tflite",
"//mediapipe/modules/face_landmark:face_landmark.tflite",
"//mediapipe/modules/hand_landmark:hand_landmark_full.tflite",
"//mediapipe/modules/hand_landmark:handedness.txt",
"//mediapipe/modules/holistic_landmark:hand_recrop.tflite",
"//mediapipe/modules/pose_detection:pose_detection.tflite",
"//mediapipe/modules/pose_landmark:pose_landmark_full.tflite",
],
deps = [
"//mediapipe/examples/ios/common:CommonMediaPipeAppLibrary",
] + select({
"//mediapipe:ios_i386": [],
"//mediapipe:ios_x86_64": [],
"//conditions:default": [
"//mediapipe/graphs/holistic_tracking:holistic_tracking_gpu_deps",
],
}),
)

ios_static_framework(
name = "HolisticGpu",
# If hdrs isn't set Headers and Modules folders won't get generated
hdrs = ["mediapipe_wrapper.h"],
families = [
"iphone",
"ipad",
],
minimum_os_version = MIN_IOS_VERSION,
deps = [
":HolisticTrackingGpuAppLibraryStatic",
"@ios_opencv//:OpencvFramework",
],
)
objc_library(
name = "HolisticTrackingGpuAppLibraryStatic",
#add by ccp
srcs = ["mediapipe_wrapper_impl.cc"],
hdrs = ["mediapipe_wrapper.h", "mediapipe_wrapper_impl.hpp"],
data = [
"//mediapipe/graphs/holistic_tracking:holistic_tracking_gpu.binarypb",
"//mediapipe/modules/face_detection:face_detection_short_range.tflite",
"//mediapipe/modules/face_landmark:face_landmark.tflite",
"//mediapipe/modules/hand_landmark:hand_landmark_full.tflite",
"//mediapipe/modules/hand_landmark:handedness.txt",
"//mediapipe/modules/holistic_landmark:hand_recrop.tflite",
"//mediapipe/modules/pose_detection:pose_detection.tflite",
"//mediapipe/modules/pose_landmark:pose_landmark_full.tflite",
],
deps = [
"//mediapipe/framework:calculator_framework",
"//mediapipe/framework/formats:image_frame",
"//mediapipe/framework/formats:image_frame_opencv",
"//mediapipe/framework/formats:landmark_cc_proto",
"//mediapipe/framework/port:file_helpers",
"//mediapipe/framework/port:opencv_imgproc",
"//mediapipe/framework/port:parse_text_proto",
"//mediapipe/framework/port:status",
"@com_google_absl//absl/flags:flag",
"@com_google_absl//absl/flags:parse",
"//mediapipe/framework/port:ret_check",
"//mediapipe/calculators/core:flow_limiter_calculator",
"//mediapipe/calculators/image:image_properties_calculator",
"//mediapipe/calculators/image:recolor_calculator",
"//mediapipe/calculators/util:annotation_overlay_calculator",
"//mediapipe/modules/holistic_landmark:holistic_landmark_gpu",
] + select({
"//mediapipe:ios_i386": [],
"//mediapipe:ios_x86_64": [],
"//conditions:default": [
"//mediapipe/graphs/holistic_tracking:holistic_tracking_gpu_deps",
],
}),
)
`

mediapipe_wrapper_impl file is my own file which i compile into the static framwork HolisticGpu.framework

Then i got a HolisticGpu.framework,but when i run this in ios xcode project i got the error below:
[libprotobuf ERROR external/com_google_protobuf/src/google/protobuf/text_format.cc:309] Error parsing text-format mediapipe.CalculatorGraphConfig: 36:66: Could not find type "type.googleapis.com/mediapipe.FlowLimiterCalculatorOptions" stored in google.protobuf.Any.
WARNING: Logging before InitGoogleLogging() is written to STDERR
F20211224 16:55:08.138662 56981632 parse_text_proto.h:32] Check failed: ParseTextProto(input, &result)
*** Check failure stack trace: ***

can somebody do a help?many many thanks

Hi Madeline, i am only now reading your reply to my comment in youtube.

I have this question, giving the normal example of mediapipe, how can i get off the position of the hands detected for every frame? Of course print a row white when no hands are detected.

Many thanks

Thank you for putting up the nice demo!

Just wanted to note that your calculator may not have to be a passthrough one and just be a leaf with no outputs.

Could you please tell me how to extract those position, using mediapipe?
Many thanks

mediapipe/calculators/core/my_pass_through_calculator.cc:156:18: error: 'class mediapipe::WrapperHandTracking' has no member named 'InitAsDefaultInstance'
156 | wrapper->InitAsDefaultInstance();
| ^~~~~~~~~~~~~~~~~~~~~

Error encountered after all modifications