Quest 2 Crashes on Selecting NDI Source - Unity 2022 Build about klakndi

[keijiro]

How about disabling the compute shaders? You can disable them by commenting out the kernel functions in Decoder.compute.

https://github.com/keijiro/KlakNDI/blob/main/jp.keijiro.klak.ndi/Runtime/Resource/Decoder.compute

You should comment out only the body of the kernel functions -- You shouldn't comment out the function declarations.

[keijiro]

Thanks for testing it.

However, the visuals didn't appear on the Canvas.

That's the intended behavior. I just wanted to check if the compute shader is relevant.

You turned it off, and the crash disappeared. It indicates that the crash happens in the compute shader or how GLES3 handles it.

[keijiro]

Just turning on/off the "Keep Alpha" switch on the sender component is enough for testing it.

[keijiro]

Thanks for the clarification. So it satisfies the requirement.

[keijiro]

First of all, I don't have a Quest 2 device, so unfortunately, I can't investigate the issue on my side.

I haven't had a firm idea, but I guess the issue is Unity version-specific because it runs on Unity 2020.

How about updating the project to the latest version of Unity 2022.3? Unity 2022.3.4 is a bit too outdated.

[eeease]

Thanks for the reply! At your suggestion, I tried making a build with Unity 2022.3.20f1, which seems to be the latest LTS version of 2022.
The same crash occurs when I select an NDI source from the dropdown on Quest.

We know that Vulkan works but we currently have a dependency on OpenGLES3 and we're unsure why OpenGLES3 works in 2020 but not 2022.

[keijiro]

How about changing the resolution of the video stream?

I still don't have any idea about the issue, but I guess it's related to the memory alignment of something. I wonder if it works with "compute friendly" dimensions like 256x256 or 1024x1024.

[eeease]

Good suggestion!
I tried a 1024x1024 video (modified via Handbrake) and the issue still occurs.
It's worth noting that I also tried making a build from Unity 2021.3.35f1, same issue.

For NDI source, I've also been testing using the Test Patterns from the official NDI Tools Launcher via Windows. Everything - test patterns, videos from VLC, etc - works fine from Unity 2020.

[eeease]

I commented out the body of the following functions:
DecodeUYVY_sRGB
DecodeUYVY_Linear
DecodeUYVA_sRGB
DecodeUYVA_Linear

I made an Android build and attempted to receive 1) NDI Test Pattern and 2) VLC video.

The performance did not get majorly impacted when I selected a source.
However, the visuals didn't appear on the Canvas.

Did I comment out too much? (I'm not too knowledgeable of shaders.)
Hopefully that helps point in the right direction.

[keijiro]

How about adding alpha channels to the sending frames? It's relevant because there are two versions of compute shaders (UYVY - without alpha, UYVA - with alpha)

You should revert commenting-out before testing it.

[eeease]

Hey @keijiro , I'm happy to try that out but not 100% sure what you mean.
Do you want me to change something in the code? I'm not a shader programmer so I'm not sure exactly how to add alpha channels to the sending frames.

Or should I try streaming something that has alpha?

[eeease]

Got it. When I select a source with the Keep Alpha bool set to true, the same issue occurs. I will try to capture the issue without revealing any of the protected images in our app.

[eeease]

NDICanvasQuest2.mp4

Here's a video of what happens when I select a source. I tried to capture the screen-tearing that I see in the headset, but it seems every way I try to capture video, the Quest just stops processing when the NDI source is selected.

[keijiro]

By the way, I wonder if GLES3 on Quest supports unordered access views (UAV), which is vital for the compute shaders. Could you check if SystemInfo.SupportsRandomWriteOnRenderTextureFormat returns true on the device?

[eeease]

That function is returning true on the device; testing in NdiReceiver:

[eeease]

Do you have any other suggestions for things to try? Anything else you'd like me to Log?
We are trying to switching our entire app to Vulkan to be able to use NDI streaming, but we're seeing much worse performance on Vulkan and running into other issues.

[keijiro]

Those kinds of driver-level issues are quite hard to investigate without an actual device. Sadly, logging can’t help a lot.