mogosmart / fastdeploy

⚡️An Easy-to-use and Fast Deep Learning Model Deployment Toolkit for Cloud and Edge. Including Vision, Text, Audio and Video 20+ main stream scenarios and 150+ SOTA models with end-to-end optimization and multi-platform multi-framework support.

English | 简体中文

⚡️FastDeploy is an accessible and efficient deployment Development Toolkit. It covers 🔥critical CV、NLP、Speech AI models in the industry and provides 📦out-of-the-box deployment experience. It covers image classification, object detection, image segmentation, face detection, face recognition, human keypoint detection, OCR, semantic understanding and other tasks to meet developers' industrial deployment needs for multi-scenario, multi-hardware and multi-platform .

Image Classification	Object Detection	Semantic Segmentation	Potrait Segmentation
Image Matting	Real-Time Matting	OCR	Face Alignment
Pose Estimation	Behavior Recognition	NLP	Speech
			input:Life was like a box of chocolates, you never know what you're gonna get. output:

📣 Recent Updates

• 🔥 【Live Preview】2022.11.09 20:30～21:30，《Covering the full spectrum of cloud-side scenarios with 150+ popular models for rapid deployment》
• 🔥 【Live Preview】2022.11.10 20:30～21:30，《10+ AI hardware deployments from Rockchip, Amlogic, NXP and others, straight to industry landing》
• 🔥 【Live Preview】2022.11.10 19:00～20:00，《10+ popular models deployed in RK3588, RK3568 in action》
  • Scan the QR code below using WeChat, follow the PaddlePaddle official account and fill out the questionnaire to join the WeChat group

• 🔥 2022.10.31：Release FastDeploy release v0.5.0
  

  • 🖥️ Data Center and Cloud Deployment: Support more backend, Support more CV models
    • Support Paddle Inference TensorRT, and provide a seamless deployment experience with other inference engines include Paddle Inference、Paddle Lite、TensorRT、OpenVINO、ONNX Runtime；
    • Support Graphcore IPU through paddle Inference;
    • Support tracking model PP-Tracking and RobustVideoMatting model；
    • Support one-click model quantization to improve model inference speed by 1.5 to 2 times on CPU & GPU platform. The supported quantized model are YOLOv7, YOLOv6, YOLOv5, etc.

• 🔥 2022.10.24：Release FastDeploy release v0.4.0
  

  • 🖥️ Data Center and Cloud Deployment: end-to-end optimization, Support more CV and NLP model
    • end-to-end optimization on GPU, YOLO series model end-to-end inference speedup from 43ms to 25ms;
    • Support CV models include PP-OCRv3, PP-OCRv2, PP-TinyPose, PP-Matting, etc. and provides end-to-end deployment demos;
    • Support information extraction model is UIE, and provides end-to-end deployment demos;
    • Support TinyPose and PicoDet and TinyPosePipeline deployment.
  • 📲 Mobile and Edge Device Deployment: support new backend，support more CV model
    • Integrate Paddle Lite and provide a seamless deployment experience with other inference engines include TensorRT、OpenVINO、ONNX Runtime、Paddle Inference；
    • Support Lightweight Detection Model and classification model on Android Platform，Download to try it out.
  • Web-Side Deployment: support more CV model
    • Web deployment and Mini Program deployment New OCR and other CV models capability.

Contents

• 📖 Tutorials（click to shrink）

  • Install
    • How to Install FastDeploy Prebuilt Libraries
    • How to Build and Install FastDeploy Library on GPU Platform
    • How to Build and Install FastDeploy Library on CPU Platform
    • How to Build and Install FastDeploy Library on IPU Platform
    • How to Build and Install FastDeploy Library on Nvidia Jetson Platform
    • How to Build and Install FastDeploy Library on Android Platform
  • A Quick Start - Demos
    • Python Deployment Demo
    • C++ Deployment Demo
    • A Quick Start on Runtime Python
    • A Quick Start on Runtime C++
  • API (To be continued)
    • Python API
    • C++ API
  • Performance Optimization
    • Quantization Acceleration
  • Frequent Q&As
    • 1. How to Change Inference Backends
    • 2. How to Use FastDeploy C++ SDK on Windows Platform
    • 3. How to Use FastDeploy C++ SDK on Android Platform(To be Continued)
    • 4. Tricks of TensorRT
    • 5. How to Develop a New Model(To be Continued)
  • More FastDeploy Deployment Module
    • deployment AI Model as a Service
    • Benchmark Testing

• 🖥️ Data Center and Cloud Deployment
  • A Quick Start for Python SDK
  • A Quick Start for C++ SDK
  • Supported Data Center and Cloud Model List
• 📲 Mobile and Edge Device Deployment
  • Paddle Lite NPU Deployment
  • Supported Mobile and Edge Model List
• Web and Mini Program Deployment
  • Supported Web and Mini Program Model List
• Community
• Acknowledge
• License

🖥️ Data Center and Cloud Deployment

A Quick Start for Python SDK（click to shrink）

Installation

Prerequisites

• CUDA >= 11.2 、cuDNN >= 8.0 、 Python >= 3.6
• OS: Linux x86_64/macOS/Windows 10

Install Fastdeploy SDK with CPU&GPU support

pip install fastdeploy-gpu-python -f https://www.paddlepaddle.org.cn/whl/fastdeploy.html

Conda Installation (Recommended)

conda config --add channels conda-forge && conda install cudatoolkit=11.2 cudnn=8.2

Install Fastdeploy SDK with only CPU support

pip install fastdeploy-python -f https://www.paddlepaddle.org.cn/whl/fastdeploy.html

Python Inference Example

• Prepare models and pictures

wget https://bj.bcebos.com/paddlehub/fastdeploy/ppyoloe_crn_l_300e_coco.tgz
tar xvf ppyoloe_crn_l_300e_coco.tgz
wget https://gitee.com/paddlepaddle/PaddleDetection/raw/release/2.4/demo/000000014439.jpg

• Test inference results

# For deployment of GPU/TensorRT, please refer to examples/vision/detection/paddledetection/python
import cv2
import fastdeploy.vision as vision


model = vision.detection.PPYOLOE("ppyoloe_crn_l_300e_coco/model.pdmodel",
                                 "ppyoloe_crn_l_300e_coco/model.pdiparams",
                                 "ppyoloe_crn_l_300e_coco/infer_cfg.yml")
im = cv2.imread("000000014439.jpg")
result = model.predict(im.copy())
print(result)

vis_im = vision.vis_detection(im, result, score_threshold=0.5)
cv2.imwrite("vis_image.jpg", vis_im)

A Quick Start for C++ SDK（click to expand）

Installation

• Please refer to C++ Prebuilt Libraries Download

C++ Inference Example

• Prepare models and pictures

wget https://bj.bcebos.com/paddlehub/fastdeploy/ppyoloe_crn_l_300e_coco.tgz
tar xvf ppyoloe_crn_l_300e_coco.tgz
wget https://gitee.com/paddlepaddle/PaddleDetection/raw/release/2.4/demo/000000014439.jpg

• Test inference results

// For GPU/TensorRT deployment, please refer to examples/vision/detection/paddledetection/cpp
#include "fastdeploy/vision.h"

int main(int argc, char* argv[]) {
  namespace vision = fastdeploy::vision;
  auto model = vision::detection::PPYOLOE("ppyoloe_crn_l_300e_coco/model.pdmodel",
                                          "ppyoloe_crn_l_300e_coco/model.pdiparams",
                                          "ppyoloe_crn_l_300e_coco/infer_cfg.yml");
  auto im = cv::imread("000000014439.jpg");

  vision::DetectionResult res;
  model.Predict(&im, &res);

  auto vis_im = vision::Visualize::VisDetection(im, res, 0.5);
  cv::imwrite("vis_image.jpg", vis_im);
  return 0;
 }

For more deployment models, please refer to Vision Model Deployment Examples .

Supported Data Center and Web Model List🔥🔥🔥🔥🔥

Notes: ✅: already supported; ❔: to be supported in the future; N/A: Not Available;

Task	Model	API	Linux	Linux	Win	Win	Mac	Mac	Linux	Linux	Linux	Linux
---	---	---	X86 CPU	NVIDIA GPU	Intel CPU	NVIDIA GPU	Intel CPU	Arm CPU	AArch64 CPU	NVIDIA Jetson	Graphcore IPU	Serving
Classification	PaddleClas/ResNet50	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	✅	❔
Classification	TorchVison/ResNet	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	❔	❔
Classification	ltralytics/YOLOv5Cls	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	❔	❔
Classification	PaddleClas/PP-LCNet	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	✅	❔
Classification	PaddleClas/PP-LCNetv2	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	✅	❔
Classification	PaddleClas/EfficientNet	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	✅	❔
Classification	PaddleClas/GhostNet	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	✅	❔
Classification	PaddleClas/MobileNetV1	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	✅	❔
Classification	PaddleClas/MobileNetV2	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	✅	❔
Classification	PaddleClas/MobileNetV3	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	✅	❔
Classification	PaddleClas/ShuffleNetV2	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	✅	❔
Classification	PaddleClas/SqueeezeNetV1.1	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	✅	❔
Classification	PaddleClas/Inceptionv3	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	❔	❔
Classification	PaddleClas/PP-HGNet	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	✅	❔
Classification	PaddleClas/SwinTransformer	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	❔	❔
Detection	PaddleDetection/PP-YOLOE	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	❔	❔
Detection	PaddleDetection/PicoDet	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	❔	❔
Detection	PaddleDetection/YOLOX	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	❔	❔
Detection	PaddleDetection/YOLOv3	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	❔	❔
Detection	PaddleDetection/PP-YOLO	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	❔	❔
Detection	PaddleDetection/PP-YOLOv2	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	❔	❔
Detection	PaddleDetection/Faster-RCNN	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	❔	❔
Detection	PaddleDetection/Mask-RCNN	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	❔	❔
Detection	Megvii-BaseDetection/YOLOX	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	❔	❔
Detection	WongKinYiu/YOLOv7	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	❔	❔
Detection	WongKinYiu/YOLOv7end2end_trt	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅
Detection	WongKinYiu/YOLOv7end2end_ort_	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅
Detection	meituan/YOLOv6	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	❔	❔
Detection	ultralytics/YOLOv5	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	❔	❔
Detection	WongKinYiu/YOLOR	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	❔	❔
Detection	WongKinYiu/ScaledYOLOv4	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	❔	❔
Detection	ppogg/YOLOv5Lite	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	❔	❔
Detection	RangiLyu/NanoDetPlus	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	❔	❔
KeyPoint	PaddleDetection/TinyPose	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	❔	❔
KeyPoint	PaddleDetection/PicoDet + TinyPose	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	❔	❔
HeadPose	omasaht/headpose	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	❔	❔
Tracking	PaddleDetection/PP-Tracking	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	❔	❔
OCR	PaddleOCR/PP-OCRv2	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	❔	❔
OCR	PaddleOCR/PP-OCRv3	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	❔	❔
Segmentation	PaddleSeg/PP-LiteSeg	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	❔	❔
Segmentation	PaddleSeg/PP-HumanSegLite	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	❔	❔
Segmentation	PaddleSeg/HRNet	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	❔	❔
Segmentation	PaddleSeg/PP-HumanSegServer	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	❔	❔
Segmentation	PaddleSeg/Unet	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	❔	❔
Segmentation	PaddleSeg/Deeplabv3	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	❔	❔
FaceDetection	biubug6/RetinaFace	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	❔	❔
FaceDetection	Linzaer/UltraFace	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	❔	❔
FaceDetection	deepcam-cn/YOLOv5Face	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	❔	✅
FaceDetection	insightface/SCRFD	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	❔	❔
FaceAlign	Hsintao/PFLD	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	❔	❔
FaceRecognition	insightface/ArcFace	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	❔	❔
FaceRecognition	insightface/CosFace	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	❔	❔
FaceRecognition	insightface/PartialFC	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	❔	❔
FaceRecognition	insightface/VPL	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	❔	❔
Matting	ZHKKKe/MODNet	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	❔	❔
Matting	PeterL1n/RobustVideoMatting	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	❔	❔
Matting	PaddleSeg/PP-Matting	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	❔	❔
Matting	PaddleSeg/PP-HumanMatting	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	❔	❔
Matting	PaddleSeg/ModNet	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	❔	❔
Information Extraction	PaddleNLP/UIE	Python/C++	✅	✅	✅	✅	✅	✅	✅	✅	❔	❔
NLP	PaddleNLP/ERNIE-3.0	Python/C++	❔	❔	❔	❔	❔	❔	❔	❔	❔	✅
Speech	PaddleSpeech/PP-TTS	Python/C++	❔	❔	❔	❔	❔	❔	❔	❔	--	✅

📲 Mobile and Edge Device Deployment

Paddle Lite NPU Deployment

• Rockchip-NPU / Amlogic-NPU / NXP-NPU

Supported Mobile and Edge Model List 🔥🔥🔥🔥

Task	Model	Size (MB)	Linux	Android	iOS	Linux	Linux	Linux	Linux	TBD...
---	---	---	ARM CPU	ARM CPU	ARM CPU	Rockchip-NPU RK3568/RK3588	Rockchip-NPU RV1109/RV1126/RK1808	Amlogic-NPU A311D/S905D/C308X	NXP-NPU i.MX 8M Plus	TBD...｜
Classification	PaddleClas/PP-LCNet	11.9	✅	✅	❔	❔	--	--	--	--
Classification	PaddleClas/PP-LCNetv2	26.6	✅	✅	❔	❔	--	--	--	--
Classification	PaddleClas/EfficientNet	31.4	✅	✅	❔	❔	--	--	--	--
Classification	PaddleClas/GhostNet	20.8	✅	✅	❔	❔	--	--	--	--
Classification	PaddleClas/MobileNetV1	17	✅	✅	❔	❔	--	--	--	--
Classification	PaddleClas/MobileNetV2	14.2	✅	✅	❔	❔	--	--	--	--
Classification	PaddleClas/MobileNetV3	22	✅	✅	❔	❔	❔	❔	❔	--
Classification	PaddleClas/ShuffleNetV2	9.2	✅	✅	❔	❔	--	--	--	--
Classification	PaddleClas/SqueezeNetV1.1	5	✅	✅	❔	❔	--	--	--	--
Classification	PaddleClas/Inceptionv3	95.5	✅	✅	❔	❔	--	--	--	--
Classification	PaddleClas/PP-HGNet	59	✅	✅	❔	❔	--	--	--	--
Classification	PaddleClas/SwinTransformer_224_win7	352.7	✅	✅	❔	❔	--	--	--	--
Detection	PaddleDetection/PP-PicoDet_s_320_coco	4.1	✅	✅	❔	❔	--	--	--	--
Detection	PaddleDetection/PP-PicoDet_s_320_lcnet	4.9	✅	✅	❔	❔	✅	✅	✅	--
Detection	PaddleDetection/CenterNet	4.8	✅	❔	❔	❔	--	--	--	--
Detection	PaddleDetection/YOLOv3_MobileNetV3	94.6	✅	❔	❔	❔	--	--	--	--
Detection	PaddleDetection/PP-YOLO_tiny_650e_coco	4.4	✅	❔	❔	❔	--	--	--	--
Detection	PaddleDetection/SSD_MobileNetV1_300_120e_voc	23.3	✅	❔	❔	❔	--	--	--	--
Detection	PaddleDetection/PP-YOLO_ResNet50vd	188.5	✅	❔	❔	❔	--	--	--	--
Detection	PaddleDetection/PP-YOLOv2_ResNet50vd	218.7	✅	❔	❔	❔	--	--	--	--
Detection	PaddleDetection/PP-YOLO_crn_l_300e_coco	209.1	✅	❔	❔	❔	--	--	--	--
Detection	YOLOv5s	29.3	❔	❔	❔	❔	--	--	--	--
Face Detection	BlazeFace	1.5	❔	❔	❔	❔	--	--	--	--
Face Detection	RetinaFace	1.7	❔	❔	❔	❔	--	--	--	--
Keypoint Detection	PaddleDetection/PP-TinyPose	5.5	✅	❔	❔	❔	❔	❔	❔	--
Segmentation	PaddleSeg/PP-LiteSeg(STDC1)	32.2	✅	❔	❔	✅	--	--	--	--
Segmentation	PaddleSeg/PP-HumanSeg-Lite	0.556	✅	❔	❔	✅	--	--	--	--
Segmentation	PaddleSeg/HRNet-w18	38.7	✅	❔	❔	✅	--	--	--	--
Segmentation	PaddleSeg/PP-HumanSeg	107.2	✅	❔	❔	✅	--	--	--	--
Segmentation	PaddleSeg/Unet	53.7	✅	❔	❔	✅	--	--	--	--
Segmentation	PaddleSeg/Deeplabv3	150	❔	❔	❔	✅
OCR	PaddleOCR/PP-OCRv1	2.3+4.4	❔	❔	❔	❔	--	--	--	--
OCR	PaddleOCR/PP-OCRv2	2.3+4.4	✅	❔	❔	❔	--	--	--	--
OCR	PaddleOCR/PP-OCRv3	2.4+10.6	✅	❔	❔	❔	❔	❔	❔	--
OCR	PaddleOCR/PP-OCRv3-tiny	2.4+10.7	❔	❔	❔	❔	--	--	--	--

Web and Mini Program Deployment

Task	Model	web_demo
---	---	Paddle.js
Detection	FaceDetection	✅
Detection	ScrewDetection	✅
Segmentation	PaddleSeg/HumanSeg	✅
Object Recognition	GestureRecognition	✅
Object Recognition	ItemIdentification	✅
OCR	PaddleOCR/PP-OCRv3	✅

Community

• If you have any question or suggestion, please give us your valuable input via GitHub Issues
• Join Us👬：
  • Slack：Join our Slack community and chat with other community members about ideas
  • WeChat：join our WeChat community and chat with other community members about ideas

Acknowledge

We sincerely appreciate the open-sourced capabilities in EasyEdge as we adopt it for the SDK generation and download in this project.

License

FastDeploy is provided under the Apache-2.0.