MediaPipe4U is an UnrealEngine plugin that integrated Google Mediapipe technology for capturing human motion (include body, fingers, face) and puppeteering 3D avatar through webcam, videos, and images in realtime.
| Unreal Engine | China Site | Global Site | Update |
|---|---|---|---|
| UE 5.0 | cowtransfer | One Drive | 2023-06-28 |
| UE 5.1 | cowtransfer | One Drive | 2023-06-28 |
Because the plugin is precompiled and contains a large number of C++ link symbols and debug symbols, it will cost 10G disk space after decompression (most files are UE-generated binaries in Intermediate).
Don't need to worry about disk usage, this is just disk usage during development, after the project is packaged, the plug-in disk usage is 300M only (most files are GStreamer dynamic library and speech models).
| Android Version | Download Link | Update |
|---|---|---|
| Android 7.0 or later | Download | 2023-04-21 |
About M4U Remoting
Note
This is a commercial license exclusive feature: capturing facial expressions from android device.
Free license only supports using in UE Editor, cannot be packaged this feature.
M4U Remoting Document
| Language | China Site | Global Site | Update |
|---|---|---|---|
| Chinese Mandarin | cowtransfer | Github | 2023-05-21 |
Please clone this repository to get demo project:
The demo project does not contain plugins, you need to download the plugin and copy content to the project's plugins folder to run.
Packaging Environment Requirements
- Visual Studio 2022 17.6.3 or later
- WINSDK: 10.0.22621 or later (下载地址)
- Visual C++ Redistributable 14.36.32532 or later (下载地址)
Video Tutorials (English)
Video Tutorials (Chinese)
If you have any questiongs, please check FAQ first. The problems listed there may be also yours. If you can’t find an answer in the FAQ, please post an issue. Private message or emal may cause the question to be mised .
Since the Windows version of MediaPipe does not support GPU inference, Windows relies on the CPU to inferring human pose estimation (see MediaPipe offical site for more details).
Hardware Evnrioment
CPU： AMD 3600 CPU
RAM: 32GB
GPU: Nvidia 1660s
Evaluation
Frame Rate： 18-24 fps
CPU usage：20% (Based on DEMO project)
Follow Me
[new] 🌈MediaPlayerImageSource: 支持从 Unreal Engine 的 MediaPlayer 中捕获画面作为图像源（文档）。
[new] 上半身锁定：支持设置上半身三个轴向的锁定 (文档)。
[fix] 修复校准时初始关节带有旋转角时产生的校准错误。
[fix] 阻尼算法造成的关节扭曲矫正错误。
[fix] 半身模式时校准无法正常工作。
[new] Pose Solver: 支持开启/关闭姿势优化的具体项。
[improve] 缩减编译后的体积（磁盘空间占用从 10GB 降低到 6GB）。
[fix] 脊柱旋转模式时可能造成姿势校准不正确。
[new] 新增姿态校准功能，纠正 mediapipe 原始数据中脊柱、小腿没有伸直的问题（文档）。
[new] 新增手动校准功能，姿势校准，位置校准均支持手动校准和自动校准设置（文档）。
[new] 新增脊柱旋转模式，有助于运动中稳定脚部（文档）。
[new] 手指捕捉手掌丢失时回弹处理规则 。
[new] UE Editor 动补工具加入关节锁定、校准等功能。
[new] UE Editor 动补工具支持对视频文件快进、暂停。
[improve] mediapipe 升级到最新版（v0.10.1）。
[improve] Hand Solver：加入手腕旋转阻尼，防止跳变。
[improve] LocationSolver: Location IK 算法改进，位移计算更合理。
[improve] Ground IK: 脚步和脚踝旋转平滑。
[improve] Ground IK: 加入脚步限滑，改善脚步抖动问题。
[fix] GroundIK: 检测脚踝位置不正确。
[fix] GroundIK: 节点 FeetVerticalOffset 等几个属性设置无效。
[fix] ControlRigPose Baker 工具生成的 PoseAsset 不正确。
[new] 新增 MediaPipe4USpeech （商业授权功能，免费授权无法打包）插支持离线 TTS 和口型同步（文档）。
[new] ControlRigPose 烘焙工具，方便制作 Metahuman 的口型资产。
[fix] 移除工具箱材质，使用 C++ 动态创建，修复 UE5.0 打包材质问题。
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent