Benchmarking methodology used is not quite correct about warp-rnnt

[dophist]

It was tested on 2080ti

[1ytic]

Hi @AshishSardana, thank you very much for this issue! I can't believe I made such a stupid mistake in my benchmark script. In my defence I will say that when I build this package I used NVIDIA Visual Profiler, and it showed very clean performance boost. I will definitely look at this on weekend. Could you also check - is it related some how to torch.no_grad()? Additionally, could you check the gather flag, which should slitty improve a memory issue?

[dophist]

As of 2021.Apr 24th, pytorch_binding/benchmark.py with above torch.cuda.synchronize() yields:

setup(T=150, U=40, V=28)	warp_rnnt	warprnnt_pytorch
N=1	0.40	0.73
N=16	1.31	1.82
N=32	2.17	3.58
N=64	4.02	6.36
N=128	7.64	9.42

Hopefully this will save others' time from repeating these benchmarks.

[1ytic]

@dophist thank you! Could you provide hw spec?

[1ytic]

Finally, I updated the benchmark results. Indeed the warping doesn't show superior performance if you synchronise device on each iteration. On the other hand, this sync-free implementation can be useful in complex use case. Additionally, I added gather version which shows great performance with a large vocabulary.

[1ytic]

I added one more explanation 2944982 of performance issue with NVIDIA Profiler.