A GPU Accelerated, self-contained, MediaPipe Plugin for TouchDesigner that runs on Mac and PC with no installation. This project currently supports all MediaPipe vision models except Interactive Segmentation and Image Embedding.
![Screenshot 2023-08-14 at 2 16 44 PM](https://private-user-images.githubusercontent.com/6014011/260537806-8f7a9eb9-fa7a-4d9b-b541-bb9c6c4e0e88.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQzNjg0MzQsIm5iZiI6MTcyNDM2ODEzNCwicGF0aCI6Ii82MDE0MDExLzI2MDUzNzgwNi04ZjdhOWViOS1mYTdhLTRkOWItYjU0MS1iYjljNmM0ZTBlODgucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDgyMiUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA4MjJUMjMwODU0WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9NzI0YzIyZDNlM2JmMWYzYzQzNDdmMWIwMGVlOWQ2YjFhNzE5MTUxZWRiZDNiY2FhNjg0MjBiYjYzMTk1M2Q3MiZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.1OywAqCm5wjo_wMbBRpjHGz2PQfMWQmCsfBwMWj6uuw)
Download the latest release.zip from the Release Section. Open up the MediaPipe TouchDesigner.toe file. All of the components are stored inside the /toxes folder. The main component is MediaPipe.tox. All of the other components are examples of how to load and display the associated model data in TouchDesigner.
On the MediaPipe component once it's loaded you can select your webcam from the drop-down. You can turn on and off the different MediaPipe models as well as preview overlays. There're also sub-menus available for each model to customize them further.
Currently the model is limited to 720p input resolution, but we're working on higher resolutions
The plugin consists of a number of components:
This component launches a Chromium web browser to host and run all of the MediaPipe vision tasks. It has a DAT output for each vision task, as well as a TOP output showing the video feed and any overlays.
Use this to process the face detection results
Use this to process the facial landmark detection results
Use this to process the hand landmark and gesture detection results
Gesture recognition task guide
Use this to process the object detection results
Use this to process the pose landmark detection results
Use this to key out segmentation results Image segmentation guide
Note: If you're hoping to get the most accurate web-cam cutout use the MultiCam model in the MediaPipe.tox. There's also a toggle to display only the background cutout which you can enable while on the multiclass model.
Use this to identify what the image might contain Image classification guide
If you are on Windows, you can use Spout to send any TOP from TouchDesigner to MediaPipe by using a Syphon Spout Out TOP and SpoutCam.
Note there is no installer, it will run from wherever the folder is
- Enter the frame rate and resolution to match your source material in TouchDesigner
- Enter the default TouchDesigner Spout output name of
TDSyphonSpoutOut
into the Starting Sender box - Click register to create the virtual webcam
The media should appear with no frames of delay!
There is no SpoutCam equivalent on Mac, but you can use Syphon to send video to OBS, and then use the OBS Virtual Webcam output to the MediaPipe task. It's not the most elegant solution, but it works.
There are lots of interesting bits of data in the CHOP output of the MediaPipe tox:
How long the MP detector took to run in ms.
How long the overlays and segmentation colors took to draw in ms.
The frame rate of the webcam video source MediaPipe is using.
What ratio of a frame it took to process the video. So 0.1 means 0.1 of a frame was needed for MediaPipe to process. This uses the project's cookRate as a reference
The number of project frames delay MediaPipe introduces. In TouchDesigner 2022.33910 this is at least 3 frames for the web browser component. If you are using Spout or Syphon to send video to a virtual webcam, you can use this as a parameter in a cache TOP to sync your input video source back up with the MediaPipe output TOP.
Tells you if the whole process is able to keep up with the input frame rate.
The MediaPipe detection tasks are very CPU and GPU intensive, so turn off any that you aren't using
If you are on a PC, you can greatly improve all CPU-based render times within TouchDesigner, including the MediaPipe tasks by disabling HyperThreading (Intel CPUs) or Simultaneous Multi-Threading (SMT - AMD CPUs). On my laptop it was a 60-80% improvement.
Enabling/disabling HyperThreading is done in your system BIOS, so look up the instructions on how to do that for your computer. You can re-enable HyperThreading at any point if you want to go back and it is not a risky change to make to your system.
This project loads the different MediaPipe models through a web browser. All of the ML models are downloaded locally and stored inside TouchDesigner's virtual file system including the website so the component can run without an internet connection. The models run using web assembly and the data coming back from the models are piped into TouchDesigner through a local WebSocket server running inside TD. This allows the components to run as standalone .tox files with GPU acceleration on any device with no setup.
This package uses yarn with vite inside node, which gives you a few options. Firstly, you need to download and install node.js
yarn install
This installs vite and all the other dependencies required. Should only be needed the first time.
yarn dev
This launches a tiny web server on port 5173 that fires a reload to the browser every time you save a file change. Super useful for debugging the web page.
Suggested use for this:
- Run
yarn dev
via the command line, or the console window of VSCode - Load the TouchDesigner project and toggle all the settings you'll want to test with.
- Enter the
MediaPipe
COMP - Disable the
webbrowser
component by clicking the X - Copy the
current_url
DAT string - Paste the url into a new Chrome tab
- Edit the url and replace the port number with 5173, so for example replace
localhost:3001
withlocalhost:5173
Chrome will now load the page from yarn dev, while still communicating with TouchDesigner via websockets. Note that parameter changes you make in TouchDesigner will not be reflected in the web page unless you repeat the steps to copy the url.
yarn build
This builds all the web page files and puts them into the _mpdist
folder. If this folder exists, the web pages used within the MediaPipe
COMP will be served from here.
There is a build_release
COMP in the top level at the top of the toe file that handles an automated build process.
To create a release zip file:
- Open the
MediaPipe TouchDesigner.toe
file - Navigate to the layout you want to be loaded when someone opens the project for the first time.
- Press
Ctrl Alt B
to trigger a build. The project will lock up for a while during the build process. - The build script will:
- Remove and recreate a
release
folder. - Perform a
yarn install
- Perform a
yarn build
- Load all files from
_mpdist
into theMediaPipe
COMP Virtual File System - Remove all external script references on DATs
- Export all tox files to
release/toxes
- Save the toe file to
release
- Zip the contents of the
release
folder torelease.zip
- Give you a popup showing the number of errors.
- Clicking ok (or anywhere!) causes the toe file to reload from the main folder so you're back where you started.
- If there were any errors in the build, you can turn on the TouchDesigner text console and run the build again to see what the errors are.