Exporting Conformer CTC to TensorRT about nemo

[Slyne]

@Slyne (myself) to check this issue.

@nirraviv89 Meanwhile could you help try tensorrt 10.0:
https://developer.nvidia.com/downloads/compute/machine-learning/tensorrt/10.0.0/tensorrt-10.0.0.6.linux.x86_64-gnu.cuda-12.4.tar.gz

The installation guide:
https://docs.nvidia.com/deeplearning/tensorrt/install-guide/index.html#installing-tar

[nirraviv89]

Thanks @Slyne this is the error I get with tensorrt 10.0:

[04/08/2024-07:16:08] [TRT] [W] Tactic Device request: 24316MB Available: 22723MB. Device memory is insufficient to use tactic.
[04/08/2024-07:16:08] [TRT] [W] UNSUPPORTED_STATE: Skipping tactic 0 due to insufficient memory on requested size of 24316 detected for tactic 0x0000000000000000.
[04/08/2024-07:16:08] [TRT] [E] 10: Could not find any implementation for node {ForeignNode[ONNXTRT_squeezeTensor_643 + /layers.0/conv/Transpose_1.../layers.1/self_attn/MatMul_1]}.
[04/08/2024-07:16:08] [TRT] [E] 10: [optimizer.cpp::computeCosts::4103] Error Code 10: Internal Error (Could not find any implementation for node {ForeignNode[ONNXTRT_squeezeTensor_643 + /layers.0/conv/Transpose_1.../layers.1/self_attn/MatMul_1]}.)

[Slyne]

Thanks @Slyne this is the error I get with tensorrt 10.0:

[04/08/2024-07:16:08] [TRT] [W] Tactic Device request: 24316MB Available: 22723MB. Device memory is insufficient to use tactic.
[04/08/2024-07:16:08] [TRT] [W] UNSUPPORTED_STATE: Skipping tactic 0 due to insufficient memory on requested size of 24316 detected for tactic 0x0000000000000000.
[04/08/2024-07:16:08] [TRT] [E] 10: Could not find any implementation for node {ForeignNode[ONNXTRT_squeezeTensor_643 + /layers.0/conv/Transpose_1.../layers.1/self_attn/MatMul_1]}.
[04/08/2024-07:16:08] [TRT] [E] 10: [optimizer.cpp::computeCosts::4103] Error Code 10: Internal Error (Could not find any implementation for node {ForeignNode[ONNXTRT_squeezeTensor_643 + /layers.0/conv/Transpose_1.../layers.1/self_attn/MatMul_1]}.)

If you only want to export conformer to Tensorrt, you may check this file to export nemo conformer to onnx:

https://github.com/Slyne/NeMo/blob/dgalvez/triton-inference-server/examples/asr/triton-inference-server/scripts/export_asr_ctc_onnx.py

Convert onnx model to tensorrt model by using trtexec:

trtexec --onnx=onnx_model/model.onnx \
        --minShapes=audio_signal:1x80x100,length:1 \
        --optShapes=audio_signal:16x80x1000,length:16 \
        --maxShapes=audio_signal:32x80x6000,length:32 \
        --fp16 \
        --saveEngine=encoder.trt

For this quantization script issue, I haven't get a chance to start it. I think I can check this issue this week. (Based on my previous experience, I didn't see perf gain by using trt + quantization.)