wanhaipeng / tiny-tensorrt

A simple, efficient, easy-to-use Nvidia TensorRT wrapper for CNN, support c++ and python.

tiny-tensorrt

A simple, efficient, easy-to-use nvidia TensorRT wrapper for cnn with c++ and python api,support caffe, uff and onnx format models. you will be able to deploy your model with tiny-tensorrt in few lines of code!

Trt* net = Trt();
net->CreateEngine(onnxModel, engineFile,maxBatchSize, int mode);
net->CopyFromHostToDevice(input, inputBindIndex);
net->Forward();
net->CopyFromDeviceToHost(output, outputBindIndex)

News

Upgrade to TensorRT 8.0 API, checkout 7.x branch for use under TensorRT 7 - 2021-7-9

Better int8 calibrator api, refer to User Guide - 2021-5-24

Remove caffe and uff support, convert to onnx with tf2onnx or keras.onnx. - 2021-4-23

Want to implement your own onnx plugin and don't know where to start? - 2021-1-29

onnx plugin template

Features

• Add DLA support
• Custom plugin tutorial and well_commented sample
• Custom onnx model output node
• Engine serialization and deserialization
• INT8 support
• Python api support
• Set device

System Requirements

cuda 10.0+

TensorRT 7

For python api, python 2.x/3.x and numpy in needed

Installation

Make sure you had install dependencies list above, if you are familiar with docker, you can use official docker

# clone project and submodule
git clone --recurse-submodules -j8 https://github.com/zerollzeng/tiny-tensorrt.git

cd tiny-tensorrt

mkdir build && cd build && cmake .. && make

Then you can intergrate it into your own project with libtinytrt.so and Trt.h, for python module, you get pytrt.so

Docs

User Guide

Custom Plugin Tutorial (En-Ch)

About License

For the 3rd-party module and TensorRT, you need to follow their license

For the part I wrote, you can do anything you want