ultralight-simplepose's Introduction

Ultralight-SimplePose

Support NCNN mobile terminal deployment
Based on MXNET(>=1.5.1) GLUON(>=0.7.0) framework
Top-down strategy: The input image is the person ROI detected by the object detector
Lightweight mobile terminal human body posture key point model(COCO 17 person_keypoints)
Detector:https://github.com/dog-qiuqiu/MobileNetv2-YOLOV3

Model

Mobile inference frameworks benchmark (4*ARM_CPU)

Network	Resolution	Inference time (NCNN/Kirin 990)	FLOPS	Weight size	HeatmapAccuracy
Ultralight-Nano-SimplePose	W:192 H:256	~5.4ms	0.224BFlops	2.3MB	74.3%

COCO2017 val keypoints metrics evaluate

 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets= 20 ] = 0.518
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets= 20 ] = 0.816
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets= 20 ] = 0.558
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets= 20 ] = 0.498
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets= 20 ] = 0.549
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 20 ] = 0.563
 Average Recall     (AR) @[ IoU=0.50      | area=   all | maxDets= 20 ] = 0.837
 Average Recall     (AR) @[ IoU=0.75      | area=   all | maxDets= 20 ] = 0.607
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets= 20 ] = 0.535
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets= 20 ] = 0.604

Install

pip install mxnet-cu101 gluoncv
pip install opencv-python cython pycocotools

Install mxnet according to your own cuda version

Demo

Test picture

python img_demo.py

Test camera stream

python cam_demo

How To Train

Download the coco2017 dataset

http://images.cocodataset.org/zips/train2017.zip
http://images.cocodataset.org/annotations/annotations_trainval2017.zip
http://images.cocodataset.org/zips/val2017.zip
Unzip the downloaded dataset zip file to the coco directory
交流qq群:1062122604

Train

python train_simple_pose.py

Ncnn Deploy

Dependent library: Opencv Ncnn
Read the camera video stream test by default, if you test the picture, please modify the code

Install ncnn

$ git clone https://github.com/Tencent/ncnn.git
$ cd <ncnn-root-dir>
$ mkdir -p build
$ cd build
$ make -j4
$ make install

Run ncnn sample

$ cp -rf ncnn/build/install/include ./Ultralight-SimplePose/ncnnsample/
$ cp -rf ncnn/build/install/lib ./Ultralight-SimplePose/ncnnsample/
$ g++ -o ncnnpose ncnnpose.cpp -I include/ncnn/ lib/libncnn.a `pkg-config --libs --cflags opencv` -fopenmp
$ ./ncnnpose

Ncnn Picture test results

Android sample

https://github.com/WZTENG/YOLOv5_NCNN

Thanks

SimplePose Paper:https://arxiv.org/abs/1804.06208
https://github.com/Tencent/ncnn
https://gluon-cv.mxnet.io/

ultralight-simplepose's Issues

Speed is unstable

The detection speed is unstable, which often occurs continuously for more than 100ms. device: iphone7 .

（1）INVALID_ARGUMENT : Non-zero status code returned while running BatchNormalization node.
（2）Status Message: Invalid input scale: NumDimensions() != 3
参考：https://github.com/onnx/models/issues/156解决
原因：mxnet转化为onnx时，产生错误的模型，BatchNormalization(space=0)

Ask for license (code, models)

Thank you for your great repository!
Can you add a LICENSE file to specify the license of the source code and the pretrained models (human detection, human pose estimation)?
I want to use it in an open-source project.