AsyncGPUReadback.RequestIntoNativeArray leads to video stuttering about klakndi

[keijiro]

My suspicion is that it might be related to the KlakNDI Compute Shader for encoding.

I doubt it because the compute shader in KlakNDI only does simple format conversion. It's much less intensive compared to the heavy computation done in the NDI codec with the NDI SDK. As far as I know, the NDI codec runs on the CPU, so there is no problem if the CPU utilization is low.

[keijiro]

Could you provide the information about the hardware specifications?

[Exxenoz]

Could you provide the information about the hardware specifications?

Hi Keijiro, we are using the following Nvidia GPUs on our test systems:

• P4000
• RTX A2000
• RTX 3070

[Exxenoz]

Hi @keijiro

were you able to reproduce the issue with the attached Unity project on your system as well? Unfortunately, we are still clueless. Do you have some time in the next few weeks to take a look at it?

Thanks a lot and best regards
Andreas

[keijiro]

• Have you profiled it using Profiler?
• Does your system run smoothly when using other application to stream 16K videos via NDI?

[Exxenoz]

Hi @keijiro

• Have you profiled it using Profiler?

Yes. Regular 250ms spikes are also being displayed. According to the call stack, the delay occurs in ReadbackEntry.get_ImagePointer(). Here is the recording with the profiler: https://we.tl/t-S5CuUKfbsH

• Does your system run smoothly when using other application to stream 16K videos via NDI?

While the source material has a resolution of 16k (and is played back with Unity across three 4k screens), it is only streamed in a 4k resolution via NDI. Additionally, we are streaming three 1920x1080p videos via NDI with the same application. So, in total, there are four streams. The stuttering occurs with 1x 4k stream + 1x 2k stream or 4-5x 2k streams, affecting other applications as well, such as the VLC player, where videos also start stuttering. It actually sounds like a performance issue, but according to the Task Manager, the GPU utilization averages around 30% without jumps. When I stream my 4 screens (1x 4k + 3x 2k) using the NDI Tools, the problem does not occur.

My suspicion is that it might be related to the KlakNDI Compute Shader for encoding.

[keijiro]

How about enabling V-Sync? It seems that the frame read-back queue is bloated very badly by lots of requests.

[Exxenoz]

How about enabling V-Sync? It seems that the frame read-back queue is bloated very badly by lots of requests.

V-Sync is already enabled in the main software. Stuttering still occurs. When I activate V-Sync in the attached test project above, the spikes disappear in the profiler, but the video stuttering persists.

VLC Media Player (without stuttering): https://streamable.com/fa7jru
Unity AVPro Video with 1x 4k + 4x 2k KlakNDI Streams (V-Sync enabled): https://streamable.com/zuw8au
Profiler Snippet (V-Sync enabled): https://we.tl/t-Th3Ziol1Vm

[keijiro]

As far as seeing the profiler data, the stutter doesn't seem to occur in KlakNDI. Not sure though.

[Exxenoz]

As far as seeing the profiler data, the stutter doesn't seem to occur in KlakNDI. Not sure though.

It would be easy if the profiler would indicate a possible problem spot, but apparently it doesn't. When I deactivate the NdiSender, the video runs smoothly.

[keijiro]

What happens when deactivating the video? Does NdiSender run smoothly?

[Exxenoz]

What happens when deactivating the video? Does NdiSender run smoothly?

I have deactivated the video and added a moving circle to the scene. Both the Unity application and the NDI transmission run smoothly with 5x 2k + 1x 4k streams, which was definitely not the case with the video. Even when playing the video with the VLC Media Player, the circle movement remains smooth. It's only when I add a second 4k sender (regardless of whether the video is playing) that the circle starts to stutter slightly. I have disabled the development build and profiler for testing because the circle starts stuttering earlier when the development build is enabled. GPU usage is between 20 and 30%. CPU usage has irregular spikes of 100% on some cores.

Perhaps it is related to the single-core nature of Unity. AVPro Video and KlakNDI run on the same core. However, the usage spikes are not visible in the profiler for unknown reasons.

Unity Project: https://mega.nz/file/7dxFhCLb#iUk6DBHU5U9Ex0tQPJWKqyx3j4EdmDElHRO7EZmPRCs

[keijiro]

Most of the NDI codec runs on a background thread. I guess AVPro also uses background threads. The issue might be in how they synchronize. I'm not sure how it can be investigated.

[Exxenoz]

Most of the NDI codec runs on a background thread. I guess AVPro also uses background threads. The issue might be in how they synchronize. I'm not sure how it can be investigated.

Thanks for your time. I'll let you know once I've found a solution.

[Exxenoz]

Hi Keijiro! Unfortunately, we couldn't identify the underlying issue, but we were able to resolve the video stuttering by outsourcing KlakNDI into a separate Unity application, connected to our main application via KlakSpout. 👍