nvidia / nsight-training Goto Github PK

Currently, if i use nsys to collect kernel time, what i got is a summary table, like below. This summary is good, but if we have run this kernel mutilple times, can we get deatil time of each run, rather than a summary?

Hello everyone

I am working on a pytorch object tracking model to convert it to tensorrt for faster inference

When inferencing tensorrt with a single batch the model is about 2x faster, but when adding batches, it becomes SLOWER

batch of 1 inference time:
pytorch - 40ms
tensorrt - 20ms

batch of 8 inference time:
pytorch - 50ms
tensorrt - 85ms

When adding batches the inference time on pytorch pretty much doesn't increase, but inferencing time with tensorrt engine increases significantly!!!

Can anybody help, why is the speed regression happening?

I have exported the model with batch size 1 and batch size 4 to nsight systems:

1 batch inference test: https://drive.google.com/file/d/1achvISpSc1pvlV2RLfSNLxCLlRsZHcnT/view?usp=sharing
4 batch inference test: https://drive.google.com/file/d/1ZuHsO28LIlETNIcWk6lh7miv2Lovco9D/view?usp=sharing

Can anybody with experience in Nvidia NSight help me understand these graphs and compare them to know where and why the performance reggression is happening?

Thank you