nvidia-ai-iot / deepstream_triton_model_deploy

How to deploy open source models using DeepStream and Triton Inference Server

------------------------------------------------------

This sample application is no longer maintained

------------------------------------------------------

Deploying an open source model using NVIDIA DeepStream and Triton Inference Server

This repository contains contains the the code and configuration files required to deploy sample open source models video analytics using Triton Inference Server and DeepStream SDK 5.0.

Getting Started

Prerequisites:

DeepStream SDK 5.0 or use docker image (nvcr.io/nvidia/deepstream:5.0.1-20.09-triton) for x86 and (nvcr.io/nvidia/deepstream-l4t:5.0-20.07-samples) for NVIDIA Jetson.

The following models have been deployed on DeepStream using Triton Inference Server.

For further details, please see each project's README.

TensorFlow Faster RCNN Inception V2 : README

The project shows how to deploy TensorFlow Faster RCNN Inception V2 network trained on MSCOCO dataset for object detection.

ONNX CenterFace : README

The project shows how to deploy ONNX CenterFace network for face detection and alignment.

Additional resources:

Developer blog: Building Intelligent Video Analytics Apps Using NVIDIA DeepStream 5.0

Learn more about Triton Inference Server

Post your questions or feedback in the DeepStream SDK developer forums

deepstream_triton_model_deploy's Issues

Yolov5 - deepstream and trition inference server

Can we deploy Yolov5 using deepstream and trition?
Thanks.

Jetpack 4.5 triton 2.10 - deepstream 5.1 on Jetson Xavier NX

triton 2.10 supports onnx but still got error when loading model.
Release link: [(https://github.com/triton-inference-server/server/releases)]

Input or output layers are empty

an error about loading model by using triton

hi, i use triton's docker to run your model,and my step is

1. cp centerface to model_repository/

2. because of the network, i download the model from browser and run 'python3 change_dim.py'
   and the sha256sum of my centerface.onnx is 77e394b51108381b4c4f7b4baf1c64ca9f4aba73e5e803b2636419578913b5fe

3. docker run --rm -p8000:8000 -p8001:8001 -p8002:8002 -v/path/to/model_repository:/models nvcr.io/nvidia/tritonserver:20.09-py3 tritonserver --model-repository=/models --model-control-mode=explicit --load-model centerface

and i got an error: model_repository_manager.cc:899] failed to load 'centerface' version 1: Invalid argument: model 'centerface', tensor '537': the model expects 4 dimensions (shape [1,1,-1,-1]) but the model configuration specifies 4 dimensions (shape [1,1,120,160])

i don't know what the problem is. Thank you so much!

triton can not load centerface onnx model

thank you for your code about triton in deepstream:
some error occured when i run

PROBLEM LOG:
2021-01-20 09:37:30.259459: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.2 I0120 09:37:32.215074 192 metrics.cc:164] found 1 GPUs supporting NVML metrics I0120 09:37:32.220711 192 metrics.cc:173] GPU 0: GeForce RTX 2080 Ti I0120 09:37:32.221101 192 server.cc:120] Initializing Triton Inference Server **ERROR: infer_trtis_server.cpp:617 TRTIS: failed to load model centerface, trtis_err_str:INTERNAL, err_msg:failed to load 'centerface', no version is available ERROR: infer_trtis_backend.cpp:42 failed to load model: centerface, nvinfer error:NVDSINFER_TRTIS_ERROR ERROR: infer_trtis_backend.cpp:184 failed to initialize backend while ensuring model:centerface ready, nvinfer error:NVDSINFER_TRTIS_ERROR 0:00:02.651921510 192 0x7fdf00002380 ERROR nvinferserver gstnvinferserver.cpp:362:gst_nvinfer_server_logger:<primary_gie> nvinferserver[UID 1]: Error in createNNBackend() <infer_trtis_context.cpp:223> [UID = 1]: failed to initialize trtis backend for model:centerface, nvinfer error:NVDSINFER_TRTIS_ERROR** I0120 09:37:32.374438 192 server.cc:179] Waiting for in-flight inferences to complete. I0120 09:37:32.374455 192 server.cc:194] Timeout 30: Found 0 live models and 0 in-flight requests 0:00:02.652044816 192 0x7fdf00002380 ERROR nvinferserver gstnvinferserver.cpp:362:gst_nvinfer_server_logger:<primary_gie> nvinferserver[UID 1]: Error in initialize() <infer_base_context.cpp:78> [UID = 1]: create nn-backend failed, check config file settings, nvinfer error:NVDSINFER_TRTIS_ERROR 0:00:02.652056954 192 0x7fdf00002380 WARN nvinferserver gstnvinferserver_impl.cpp:439:start:<primary_gie> error: Failed to initialize InferTrtIsContext 0:00:02.652063559 192 0x7fdf00002380 WARN nvinferserver gstnvinferserver_impl.cpp:439:start:<primary_gie> error: Config file path: /root/deepstream_triton_model_deploy/centerface/config/centerface.txt 0:00:02.652150478 192 0x7fdf00002380 WARN nvinferserver gstnvinferserver.cpp:460:gst_nvinfer_server_start:<primary_gie> error: gstnvinferserver_impl start failed ** ERROR: <main:655>: Failed to set pipeline to PAUSED Quitting ERROR from primary_gie: Failed to initialize InferTrtIsContext Debug info: gstnvinferserver_impl.cpp(439): start (): /GstPipeline:pipeline/GstBin:primary_gie_bin/GstNvInferServer:primary_gie: Config file path: /root/deepstream_triton_model_deploy/centerface/config/centerface.txt ERROR from primary_gie: gstnvinferserver_impl start failed Debug info: gstnvinferserver.cpp(460): gst_nvinfer_server_start (): /GstPipeline:pipeline/GstBin:primary_gie_bin/GstNvInferServer:primary_gie App run failed

the log only told me nvinfer error:NVDSINFER_TRTIS_ERROR, but not any other information

config:
2080ti
docker : nvcr.io/nvidia/deepstream:5.0.1-20.09-triton
https://github.com/NVIDIA-AI-IOT/deepstream_triton_model_deploy.git 0208221
onnx centernet from https://github.com/Star-Clouds/CenterFace/raw/master/models/onnx/centerface.onnx

Does nvcr.io/nvidia/deepstream:5.0.1-20.09-triton support TensorRT 7.2.2?

Does the docker image nvcr.io/nvidia/deepstream:5.0.1-20.09-triton support models optimized with TensorRT 7.2.2?

is there any example with python?

Unexpected platform and fail to load model (Solved onnx with triton not suported on Jetson)

Hello,

I am encountering an error when running
"deepstream-app -c source1_primary_detector.txt"

Regarding the model, I followed the Read.me and ran "run.sh" and have kept the model in the directory indicated in this issue: #5

I haven't made any changes to the centerface.txt file.

Thanks in advance for any help.

Here is the error.

E0208 21:41:07.509156 20068 model_repository_manager.cc:1519] unexpected platform type onnxruntime_onnx for centerface
ERROR: TRTIS: failed to load model centerface, trtis_err_str:INTERNAL, err_msg:failed to load 'centerface', no version is available
ERROR: failed to load model: centerface, nvinfer error:NVDSINFER_TRTIS_ERROR
ERROR: failed to initialize backend while ensuring model:centerface ready, nvinfer error:NVDSINFER_TRTIS_ERROR
0:00:04.427607013 20068 0x35fdeac0 ERROR nvinferserver gstnvinferserver.cpp:362:gst_nvinfer_server_logger:<primary_gie> nvinferserver[UID 1]: Error in createNNBackend() <infer_trtis_context.cpp:223> [UID = 1]: failed to initialize trtis backend for model:centerface, nvinfer error:NVDSINFER_TRTIS_ERROR
I0208 21:41:07.509727 20068 server.cc:179] Waiting for in-flight inferences to complete.
I0208 21:41:07.509778 20068 server.cc:194] Timeout 30: Found 0 live models and 0 in-flight requests
0:00:04.427822468 20068 0x35fdeac0 ERROR nvinferserver gstnvinferserver.cpp:362:gst_nvinfer_server_logger:<primary_gie> nvinferserver[UID 1]: Error in initialize() <infer_base_context.cpp:78> [UID = 1]: create nn-backend failed, check config file settings, nvinfer error:NVDSINFER_TRTIS_ERROR

Problem with running centerface model.

Hi,
I have installed tritonserver(docker image) and provided centerface model in "model_repository". Docker image successfully loaded the centerface model.

By running -
"deepstream-app -c source1_primary_detector.txt"

I am getting error-
** ERROR: <create_primary_gie_bin:120>: Failed to create 'primary_gie'
** ERROR: <create_primary_gie_bin:182>: create_primary_gie_bin failed
** ERROR: <create_pipeline:1296>: create_pipeline failed
** ERROR: main:636: Failed to create pipeline
Quitting
App run failed

Please suggest if few other steps are left or not.
Thanks

Using Centernet model with batching in Deepstream

Hi, What changes should I make to use the centernet model with nvinferserver with batch size more than 1?

object class 'GstNvInferServer' has no property named 'input-tensor-meta'

I am trying to run faster rcnn from here

But getting this error:

deepstream-app -c source1_primary_faster_rcnn_inception_v2.txt 

(deepstream-app:69): GLib-GObject-WARNING **: 16:30:49.275: g_object_set_is_valid_property: object class 'GstNvInferServer' has no property named 'input-tensor-meta'
** ERROR: <main:707>: Failed to set pipeline to PAUSED
Quitting
App run failed

Not able to run with deepstream sdk.

Hi,
Firstly thanks for the project. This is awesome.

I have few doubts-

1. When i run this with dockerised tritonserver based deepstream image and follow the steps, it runs awesome.
   But how can i made it run with batch-size > 1.

2. I tried to run with deepstream sdk and updated the config file also. There was no face detection.
   I tried to debug the heatmap dims output in custom_parser.cpp,
   Debug results-
   heatmap->inferDims[0] = 1
   heatmap->inferDims[1] = 8
   heatmap->inferDims[2] = 8
   heatmap->inferDims[3] = 0
   face detected = 0

While debugging with docker tritonserver based deepstream i got (Where i get detections)-
heatmap->inferDims[0] = 1
heatmap->inferDims[1] = 1
heatmap->inferDims[2] = 120
heatmap->inferDims[3] = 160
face size 3

What is the issue with model dims? how can i use this with deepstream sdk? is there parsing logic need to be changed?
I also provided infer-dims = 0(NCHW). in infer-config configuration. But no results.

Note-
I also tried to load the model as tensorRT code, it gave me error for model-
error is-
[TRT] binding to input 0 input.1 binding index: 0
[TRT] binding to input 0 input.1 dims (b=1 c=1 h=3 w=480) size=5760
[TRT] binding to output 0 537 binding index: 1
[TRT] binding to output 0 537 dims (b=1 c=1 h=1 w=120) size=480
[TRT] binding to output 1 538 binding index: 2
[TRT] binding to output 1 538 dims (b=1 c=1 h=2 w=120) size=960
[TRT] binding to output 2 539 binding index: 3
[TRT] binding to output 2 539 dims (b=1 c=1 h=2 w=120) size=960
[TRT] binding to output 3 540 binding index: 4
[TRT] binding to output 3 540 dims (b=1 c=1 h=10 w=120) size=4800

Please suggest how can i use this model with deepstream sdk.
Will appreciate.
Thanks.

Can I save the detected face from the centerface example? What should I do?

Face Alignment in Triton

Hi, Thanks for the amazing work!

It is mentioned in README of Centerface about performing Face Alignment using Triton here

 We are using Deepstream-5.0 with Triton Inference Server to deploy the Centerface network for face detection and alignment. 

But I could not find resources on Face Alignment in this repo.
Is a different custom parser being used for Face Alignment as well?

Additionally, How can the output crops after face alignment be obtained for further processing?

Invalid argument: Input shape axis 0 must equal 8, got shape [5,600,1024,3]

Hi, I have optimized the model faster_rcnn_inception_v2 model as TF-TRT INT8 with NMS enabled (ops placed on the CPU) using TF 1.5.2 and the script https://github.com/tensorflow/tensorrt/tree/r1.14+/tftrt/examples/object_detection. I got the below performance with nms enable vs nms disable:
TF-TRT-INT8 (nms enabled): ~96FPS
TF-TRT-INT8 (no nms): ~43 FPS

The model was optimized with batch_size=8, image_shape=[600, 600], and minimum_segment_size=50. For DS-Triton deployment the max_batch_size=8

The issue is now when deploying the model to DeepStream-Triton, I got the below error Input shape axis 0 must equal 8, got shape [5,600,1024,3] (even though the model was optimized with BS=8):

    I0112 01:06:22.313573 2643 model_repository_manager.cc:837] successfully loaded 'faster_rcnn_inception_v2' version 13
    INFO: infer_trtis_backend.cpp:206 TrtISBackend id:1 initialized model: faster_rcnn_inception_v2
    2021-01-12 01:06:36.202139: I tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:733] Building a new TensorRT engine for TRTEngineOp_0 input shapes: [[8,600,1024,3]]
    2021-01-12 01:06:36.202311: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer.so.7
    2021-01-12 01:06:36.203128: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer_plugin.so.7
    2021-01-12 01:09:20.678239: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:37] DefaultLogger Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles
    2021-01-12 01:09:20.709545: I tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:733] Building a new TensorRT engine for TRTEngineOp_1 input shapes: [[800,14,14,576]]
    2021-01-12 01:10:01.273658: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:37] DefaultLogger Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles

    Runtime commands:
            h: Print this help
            q: Quit

            p: Pause
            r: Resume


    **PERF:  FPS 0 (Avg)
    **PERF:  0.00 (0.00)
    ** INFO: <bus_callback:181>: Pipeline ready

    ** INFO: <bus_callback:167>: Pipeline running

    ERROR: infer_trtis_server.cpp:276 TRTIS: failed to get response status, trtis_err_str:INTERNAL, err_msg:2 root error(s) found.
      (0) Invalid argument: Input shape axis 0 must equal 8, got shape [5,600,1024,3]
             [[{{node Preprocessor/unstack}}]]
      (1) Invalid argument: Input shape axis 0 must equal 8, got shape [5,600,1024,3]
             [[{{node Preprocessor/unstack}}]]
             [[ExpandDims_4/_199]]
    0 successful operations.
    0 derived errors ignored.
    ERROR: infer_trtis_backend.cpp:532 TRTIS server failed to parse response with request-id:1 model:
    0:03:46.539871495  2643 0x7f0cf80022a0 WARN           nvinferserver gstnvinferserver.cpp:519:gst_nvinfer_server_push_buffer:<primary_gie> error: inference failed with unique-id:1
    ERROR from primary_gie: inference failed with unique-id:1
    Debug info: gstnvinferserver.cpp(519): gst_nvinfer_server_push_buffer (): /GstPipeline:pipeline/GstBin:primary_gie_bin/GstNvInferServer:primary_gie
    Quitting
    ERROR: infer_trtis_server.cpp:276 TRTIS: failed to get response status, trtis_err_str:INTERNAL, err_msg:2 root error(s) found.
      (0) Invalid argument: Input shape axis 0 must equal 8, got shape [5,600,1024,3]
             [[{{node Preprocessor/unstack}}]]
      (1) Invalid argument: Input shape axis 0 must equal 8, got shape [5,600,1024,3]
             [[{{node Preprocessor/unstack}}]]
             [[ExpandDims_4/_199]]
    0 successful operations.
    0 derived errors ignored.
    ERROR: infer_trtis_backend.cpp:532 TRTIS server failed to parse response with request-id:2 model:
    ERROR from qtdemux0: Internal data stream error.
    Debug info: qtdemux.c(6073): gst_qtdemux_loop (): /GstPipeline:pipeline/GstBin:multi_src_bin/GstBin:src_sub_bin0/GstURIDecodeBin:src_elem/GstDecodeBin:decodebin0/GstQTDemux:qtdemux0:
    streaming stopped, reason custom-error (-112)
    I0112 01:10:01.644682 2643 model_repository_manager.cc:708] unloading: faster_rcnn_inception_v2:13
    I0112 01:10:01.917792 2643 model_repository_manager.cc:816] successfully unloaded 'faster_rcnn_inception_v2' version 13
    I0112 01:10:01.918447 2643 server.cc:179] Waiting for in-flight inferences to complete.
    I0112 01:10:01.918460 2643 server.cc:194] Timeout 30: Found 0 live models and 0 in-flight requests
    App run failed

Some idea on how to solve the input shape issue?

Problem with deepstream-triton docker

I follow the instruction with nvcr.io/nvidia/deepstream:5.1-21.02-triton, run change dim and run the app but I got this error

2021-07-06 08:31:11.413549: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.2
I0706 08:31:12.450158 4142 metrics.cc:164] found 1 GPUs supporting NVML metrics
I0706 08:31:12.455850 4142 metrics.cc:173] GPU 0: NVIDIA Quadro P4000
I0706 08:31:12.456117 4142 server.cc:120] Initializing Triton Inference Server
I0706 08:31:12.534433 4142 server_status.cc:55] New status tracking for model 'centerface'
I0706 08:31:12.534802 4142 model_repository_manager.cc:680] loading: centerface:1
I0706 08:31:12.538328 4142 onnx_backend.cc:203] Creating instance centerface_0_0_gpu0 on GPU 0 (6.1) using model.onnx
I0706 08:31:13.018687 4142 model_repository_manager.cc:837] successfully loaded 'centerface' version 1
INFO: infer_trtis_backend.cpp:206 TrtISBackend id:1 initialized model: centerface
0:00:01.979633096 4142 0x562898ea3ef0 ERROR nvinferserver gstnvinferserver.cpp:362:gst_nvinfer_server_logger:<primary_gie> nvinferserver[UID 1]: Error in specifyBackendDims() <infer_trtis_context.cpp:124> [UID = 1]: failed to create trtis backend on model:centerface because tensor:input.1 input-dims is not correct
0:00:01.979667124 4142 0x562898ea3ef0 ERROR nvinferserver gstnvinferserver.cpp:362:gst_nvinfer_server_logger:<primary_gie> nvinferserver[UID 1]: Error in createNNBackend() <infer_trtis_context.cpp:228> [UID = 1]: failed to specify trtis backend input dims for model:centerface, nvinfer error:NVDSINFER_CONFIG_FAILED
I0706 08:31:13.020304 4142 model_repository_manager.cc:708] unloading: centerface:1
I0706 08:31:13.020987 4142 model_repository_manager.cc:816] successfully unloaded 'centerface' version 1
I0706 08:31:13.021180 4142 server.cc:179] Waiting for in-flight inferences to complete.
I0706 08:31:13.021192 4142 server.cc:194] Timeout 30: Found 0 live models and 0 in-flight requests
0:00:01.980676621 4142 0x562898ea3ef0 ERROR nvinferserver gstnvinferserver.cpp:362:gst_nvinfer_server_logger:<primary_gie> nvinferserver[UID 1]: Error in initialize() <infer_base_context.cpp:78> [UID = 1]: create nn-backend failed, check config file settings, nvinfer error:NVDSINFER_CONFIG_FAILED
0:00:01.980687041 4142 0x562898ea3ef0 WARN nvinferserver gstnvinferserver_impl.cpp:439:start:<primary_gie> error: Failed to initialize InferTrtIsContext
0:00:01.980691218 4142 0x562898ea3ef0 WARN nvinferserver gstnvinferserver_impl.cpp:439:start:<primary_gie> error: Config file path: /opt/nvidia/deepstream/deepstream-5.0/samples/trtis_model_repo/deepstream_triton_model_deploy/centerface/config/centerface.txt
0:00:01.980768517 4142 0x562898ea3ef0 WARN nvinferserver gstnvinferserver.cpp:460:gst_nvinfer_server_start:<primary_gie> error: gstnvinferserver_impl start failed
** ERROR: main:655: Failed to set pipeline to PAUSED
Quitting
ERROR from primary_gie: Failed to initialize InferTrtIsContext
Debug info: gstnvinferserver_impl.cpp(439): start (): /GstPipeline:pipeline/GstBin:primary_gie_bin/GstNvInferServer:primary_gie:
Config file path: /opt/nvidia/deepstream/deepstream-5.0/samples/trtis_model_repo/deepstream_triton_model_deploy/centerface/config/centerface.txt
ERROR from primary_gie: gstnvinferserver_impl start failed
Debug info: gstnvinferserver.cpp(460): gst_nvinfer_server_start (): /GstPipeline:pipeline/GstBin:primary_gie_bin/GstNvInferServer:primary_gie
App run failed

No Detection for Centerface Model Inference with Deepstream

Hi Nvidia Team,

I am trying to run the Centerface model Inference on my Laptop using Deepstream Container nvcr.io/nvidia/deepstream:5.1-21.02-triton, App is running Successfully, but there is no Detection in the Output File(.mp4). I am not sure what's going wrong.

I am using .so which is given in the repo. I did not compiled on my Laptop.

I am attaching the config files below:

centerface.txt
source1_primary_detector.txt

Below is config.pbtxt content:

name: "centerface"
platform: "onnxruntime_onnx"
max_batch_size: 0
input [
  {
    name: "input.1"
    data_type: TYPE_FP32
    dims: [ 1, 3, 480, 640]
    #reshape { shape: [ 1, 3, 480, 640 ] }
  }
]

output [
  {
    name: "537"
    data_type: TYPE_FP32
    dims: [ 1, 1, 120, 160 ]
    label_filename: "centerface_labels.txt"
  },
  {
    name: "538"
    data_type: TYPE_FP32
    dims: [ 1, 2, 120, 160]
    label_filename: "centerface_labels.txt"
  },

  {
    name: "539"
    data_type: TYPE_FP32
    dims: [1,  2, 120, 160]
    label_filename: "centerface_labels.txt"
  },
  {
    name: "540"
    data_type: TYPE_FP32
    dims: [1, 10 , 120, 160]
    label_filename: "centerface_labels.txt"
  }
]

instance_group {
  count: 1
  gpus: 0
  kind: KIND_GPU
}

# Enable TensorRT acceleration running in gpu instance. It might take several
# minutes during intialization to generate tensorrt online caches.

#optimization { execution_accelerators {
 # gpu_execution_accelerator : [ { name : "tensorrt" } ]
#		}}

Kindly request you to assist me in resolving this issue.

Thanks,
Darshan