Add ONNXRuntime C++ interfaces for inferencing about yolort

[itsnine]

@zhiqwang hello, I'm currently working on the same task, writing onnxruntime c++ inference code, you can see what I've done here: https://github.com/itsnine/yolov5-onnxruntime. Note that I export yolov5 model to onnx using official script.
Don't you mind if I contribute to YOLO v5 Runtime Stack project when I'm done with the code? What do you think about that?

[zhiqwang]

Hi @itsnine ,

We would be delighted if you would contribute this feature. Feel free to pin me if you encounter any problems.

The default ONNX model we export here differs from the official one in following three main ways.

• The exported ONNX graph now supports dynamic shapes, and we use (3, H, W) (for example (3, 640, 640)) as the input shape.
• We embed the post-processing (nms) into the model graph, and you can check #159 (comment) for more details about this. (Here the ONNX graph is required to be dynamic.)
• We embed the pre-processing (letterbox) into the graph as well. We require the input image to be in the RGB channel, and to be rescaled to float32 [0-1] from general uint [0-255]. The main logic we use to implement this mechanism is below. (And this plays the same role of the official letterbox, but there will be a little difference in accuracy now.)
  https://github.com/zhiqwang/yolov5-rt-stack/blob/b9c67205a61fa0e9d7e6696372c133ea0d36d9db/yolort/models/transform.py#L249-L258

Let me know your thoughts or considerations about these three options.

[zhiqwang]

Note that I export yolov5 model to onnx using official script. Don't you mind if I contribute to YOLO v5 Runtime Stack project when I'm done with the code? What do you think about that?

Hi @itsnine , Yep, this could also be an option, and I would very much welcome this feature.

[itsnine]

@zhiqwang Okay, I'll try to write c++ inference code for this type of onnx yolov5 with embedded nms and letterbox :) Actually that's awesome idea to add pre- and post-processing parts to the onnx graph. You mentioned about dynamic shape, but what about batched inference?

[itsnine]

@zhiqwang I inferenced your yolov5 ONNX model with embedded pre- and post-processing in c++, it works! I'll do PR when I'm done with the code :)

[zhiqwang]

You mentioned about dynamic shape, but what about batched inference?

Hi @itsnine , This is a good question and one that yolort has already had, I file an issue #45 about it earlier, you can check this discussion #39 for more details.