This is a Unity (2020.3.30f1) Native Plugin to use MediaPipe (0.8.9).
The goal of this project is to port the MediaPipe API (C++) one by one to C# so that it can be called from Unity.
This approach may sacrifice performance when you need to call multiple APIs in a loop, but it gives you the flexibility to use MediaPipe instead.
With this plugin, you can
- Write MediaPipe code in C#.
- Run MediaPipe's official solution on Unity.
- Run your custom
Calculator
andCalculatorGraph
on Unity.โ ๏ธ Depending on the type of input/output, you may need to write C++ code.
Here is a Hello World! example.
Compare it with the official code!
using Mediapipe;
using UnityEngine;
public sealed class HelloWorld : MonoBehaviour
{
private const string _ConfigText = @"
input_stream: ""in""
output_stream: ""out""
node {
calculator: ""PassThroughCalculator""
input_stream: ""in""
output_stream: ""out1""
}
node {
calculator: ""PassThroughCalculator""
input_stream: ""out1""
output_stream: ""out""
}
";
private void Start()
{
var graph = new CalculatorGraph(_ConfigText);
var poller = graph.AddOutputStreamPoller<string>("out").Value();
graph.StartRun().AssertOk();
for (var i = 0; i < 10; i++)
{
graph.AddPacketToInputStream("in", new StringPacket("Hello World!", new Timestamp(i))).AssertOk();
}
graph.CloseInputStream("in");
var packet = new StringPacket();
while (poller.Next(packet))
{
Debug.Log(packet.Get());
}
graph.WaitUntilDone().AssertOk();
}
}
This repository does not contain required libraries (e.g. libmediapipe_c.so
, Google.Protobuf.dll
, etc), so you need to build them first.
For step-by-step guide, please refer to the Installation Guide in Wiki.
โ ๏ธ libraries that can be built differ depending on your environment.
โ ๏ธ GPU mode is not supported on macOS and Windows.
Editor | Linux (x86_64) | macOS (x86_64) | macOS (ARM64) | Windows (x86_64) | Android | iOS | WebGL | |
---|---|---|---|---|---|---|---|---|
Linux (AMD64) 1 | โ๏ธ | โ๏ธ | โ๏ธ | |||||
Intel Mac | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | ||||
M1 Mac 2 | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | ||||
Windows 10 (AMD64) 34 | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ |
Here is a list of solutions that you can try in the sample app.
๐ The graphs you can run are not limited to the ones in this list.
Android | iOS | Linux (GPU) | Linux (CPU) | macOS (CPU) | Windows (CPU) | WebGL | |
---|---|---|---|---|---|---|---|
Face Detection | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | |
Face Mesh | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | |
Iris | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | |
Hands | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | |
Pose | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | |
Holistic | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | |
Selfie Segmentation | |||||||
Hair Segmentation | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | |
Object Detection | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | |
Box Tracking | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | |
Instant Motion Tracking | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | |
Objectron | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | |
KNIFT |
Select Mediapipe/Samples/Scenes/Start Scene
and play.
If you've built native libraries for CPU (i.e. --desktop cpu
), select CPU
for inference mode from the Inspector Window.
Make sure that you select GPU
for inference mode before building the app, because CPU
inference mode is not supported currently.
๐ง https://github.com/homuler/MediaPipeUnityPlugin/wiki
MIT
Note that some files are distributed under other licenses.
- MediaPipe (Apache Licence 2.0
- emscripten (MIT)
- FontAwesome (LICENSE)
Footnotes
-
Tested on Arch Linux. โฉ
-
Experimental, because MediaPipe does not support M1 Mac. โฉ
-
Windows 11 will be also OK. โฉ
-
Running MediaPipe on Windows is experimental. โฉ