Multiple instance initialization on Android clobbers `apiVersion`

#3903 · closed · 5 comments

MarijnS95

## Description of Bug   We have a custom application (desktop and Android) that relies on features like buffer device address and ray tracing. When the Vulkan _instance and physical device_ agree on Vulkan >= 1.2, we set our `apiVersion` to >= 1.2 and use `VkPhysicalDeviceBufferDeviceAddressFeatures` without enabling the `VK_KHR_buffer_device_address` device extension, because it was promoted to 1.2. Unfortunately **some times** our app on Android fails to start under RenderDoc, showing a log and crash like the following: ``` 09-11 22:10:36.996 19878 19934 I renderdoc: @6aa4603c0000001a@ RDOC 019878: [22:10:36] vk_device_funcs.cpp(5056) - Log - Acceleration structures enabled, ALL MEMORY WILL BE MARKED AS BDA ... 09-11 22:10:37.102 19878 19951 I renderdoc: @6aa4603d00000037@ RDOC 019878: [22:10:37] vk_resource_funcs.cpp(2251) - Error - Device address bit specified but no device address extension enabled ... 09-11 22:10:37.178 19878 19951 F libc : Fatal signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x0 in tid 19951 (RenderLoop Rend), pid 19878 (our app) ... 09-11 22:10:37.396 19963 19963 F DEBUG : #01 pc 0000000000e5fa70 /data/app/~~JF18wqHwacIm8bbJiyWyiA==/org.renderdoc.renderdoccmd.arm64-hjXlT1ceqpPPCSkO4I78hQ==/lib/arm64/libVkLayer_GLES_RenderDoc.so (WrappedVulkan::vkAllocateMemory(VkDevice_T*, VkMemoryAllocateInfo const*, VkAllocationCallbacks const*, VkDeviceMemory_T**)+6776) (BuildId: cca8fae158109ffc) ``` This never happens on desktop. AI [^1] deduced that the actual crash in `vkAllocateMemory` is an unconditional call to one of the BDA functions, because the extension wasn't initialized. [^1]: Noting that this issue was hand-written though, AI was only used to analyze some of the code and point me to the right "assumed" call sites where things are going wrong. https://github.com/baldurk/renderdoc/blob/0628f10b9a21d21f4230dbf7d925211c128238ce/renderdoc/driver/vulkan/wrappers/vk_resource_funcs.cpp#L826-L839 Not reading too much into the crash itself, I'd rather like to know why [the above error about `ext_KHR_buffer_device_address` not being initialized](https://github.com/baldurk/renderdoc/blob/0628f10b9a21d21f4230dbf7d925211c128238ce/renderdoc/driver/vulkan/wrappers/vk_resource_funcs.cpp#L2211) is **sometimes** getting thrown in the first place. Quick side-note: when manually enabling the `VK_KHR_buffer_device_address` extension, this problem only moves to the next 1.2-stabilized extension: `VK_KHR_timeline_semaphore` and us loading `vkWaitSemaphore()`. ### Implied cause On Android, thanks to Adreno logs, we see that multiple instances are being created on different API versions. On a functional run (this one was taken without RenderDoc to see the original `Engine name`): ``` 09-12 16:43:55.277 12272 12311 I AdrenoVK-0: Engine Name : android framework 09-12 16:43:55.277 12272 12311 I AdrenoVK-0: Engine Version : 0x00000000 09-12 16:43:55.277 12272 12311 I AdrenoVK-0: Api Version : 0x00401000 ... 09-12 16:43:55.288 12272 12318 I AdrenoVK-0: Engine Name : Our App 09-12 16:43:55.288 12272 12318 I AdrenoVK-0: Engine Version : 0x00001000 09-12 16:43:55.288 12272 12318 I AdrenoVK-0: Api Version : 0x00403000 ``` **Android initializes an `android framework` instance**, where `0x00401000` corresponds to Vulkan 1.1 and our app's `0x00403000` corresponds to Vulkan 1.3. On a broken run with RenderDoc capturing however, these are the (slightly stripped for brevity) capture logs: ``` renderdoc: @6aa564af00000010@ RDOC 011566: [16:41:51] vk_layer.cpp( 70) - Log - Creating internal instance to bump layer refcount vulkan : Loaded layer VK_LAYER_RENDERDOC_Capture ... AdrenoVK-0: Engine Name : RenderDoc AdrenoVK-0: Engine Version : 0x0042f000 AdrenoVK-0: Api Version : 0x00403000 vulkan : internal vkGetInstanceProcAddr called for vkCreateInstance with an instance renderdoc: @6aa564af00000011@ RDOC 011566: [16:41:51] vk_device_funcs.cpp( 975) - Log - Initialised capture layer in Vulkan instance. renderdoc: @6aa564af00000012@ RDOC 011566: [16:41:51] vk_layer.cpp( 89) - Log - Created own instance B40000734CA594D0: VK_SUCCESS ... AdrenoVK-0: Engine Name : RenderDoc AdrenoVK-0: Engine Version : 0x0042f000 AdrenoVK-0: Api Version : 0x00403000 vulkan : internal vkGetInstanceProcAddr called for vkCreateInstance with an instance renderdoc: @6aa564af00000013@ RDOC 011566: [16:41:51] core.cpp(2317) - Log - Adding Vulkan device frame capturer for 0xB40000747C987A30 renderdoc: @6aa564af00000014@ RDOC 011566: [16:41:51] vk_device_funcs.cpp( 975) - Log - Initialised capture layer in Vulkan instance. ... AdrenoVK-0: Engine Name : RenderDoc AdrenoVK-0: Engine Version : 0x0042f000 AdrenoVK-0: Api Version : 0x00401000 vulkan : internal vkGetInstanceProcAddr called for vkCreateInstance with an instance renderdoc: @6aa564af0000001a@ RDOC 011566: [16:41:51] core.cpp(2317) - Log - Adding Vulkan device frame capturer for 0xB40000747C999370 renderdoc: @6aa564af0000001b@ RDOC 011566: [16:41:51] vk_device_funcs.cpp( 975) - Log - Initialised capture layer in Vulkan instance. ``` **As you can see here, the 1.1 API instance is last!** Thanks to some AI assistance, it seems that the global instance is being **overwritten** each time the captured app calls `vkCreateInstance`, so the last instance and version wins despite every instance being wrapped individually: https://github.com/baldurk/renderdoc/blob/ce50614f0e77d007982b776cbc85b8494720459d/renderdoc/driver/vulkan/wrappers/vk_device_funcs.cpp#L809-L810 https://github.com/baldurk/renderdoc/blob/ce50614f0e77d007982b776cbc85b8494720459d/renderdoc/driver/vulkan/wrappers/vk_device_funcs.cpp#L833 https://github.com/baldurk/renderdoc/blob/ce50614f0e77d007982b776cbc85b8494720459d/renderdoc/driver/vulkan/wrappers/vk_device_funcs.cpp#L853-L855 When our app ultimately calls `vkCreateDevice()`, that 1.1 API version is copied: https://github.com/baldurk/renderdoc/blob/0628f10b9a21d21f4230dbf7d925211c128238ce/renderdoc/driver/vulkan/wrappers/vk_device_funcs.cpp#L5133-L5134 And reading device extensions, just down at these lines: https://github.com/baldurk/renderdoc/blob/0628f10b9a21d21f4230dbf7d925211c128238ce/renderdoc/driver/vulkan/wrappers/vk_device_funcs.cpp#L5156-L5157 Will falsely flag `ext_KHR_buffer_device_address` as "not supported" because we never put the name in `ppEnabledExtensionNames` (as our app implied Vulkan 1.3 support), while `record->instDevInfo->vulkanVersion` is initialized to Vulkan 1.1 but `ver` here requires Vulkan 1.2. ### How to solve? I'd love to have PR'd a solution, but I believe anything "good enough" involves supporting this multi-instance scenario (and tracking (p)device parents properly) rather than a workaround that patches through a "more accurate" Vulkan version. Happy to be proven wrong (a more simple solution exists, or this AI reduction is truly incorrect) though! Also happy to spend more AI tokens on this if it'd help (writing a repro in a test-case, or proposing/drafting an actual fix. Thanks for reading all this either way! ## Steps to reproduce     Since this situation happens specifically on Android and depends on the "random" order of instance creation, I think I can set up a small Vulkan sample that always initializes a Vulkan 1.2+ instance plus stabilized feature, then initializes a second Vulkan instance on 1.1, and ultimately create a device on the 1.2 instance while trying to use 1.2 functionality on it, like BDA. Let me know if that's needed! ## Environment  * RenderDoc version: Latest compiled from 60c83d0483fa848e5f90123739bcc1cc0e31f4f2 * Operating System: Android * Graphics API: Vulkan 1.3  ## Driveby issues flagged by myself and AI - While attempting to debug this, I first landed on https://github.com/baldurk/renderdoc/blob/0628f10b9a21d21f4230dbf7d925211c128238ce/renderdoc/driver/vulkan/wrappers/vk_device_funcs.cpp#L4432-L4437 (before knowing that's for _replay_, not _capture_!): this only checks if the captured app had enabled `vulkan12Features.bufferDeviceAddress`, but it doesn't check if it was enabled via `VkPhysicalDeviceBufferDeviceAddressFeatures` instead? - I haven't tested yet, but that'd mean our captures likely won't play back? - Same issue for the other checks, supposedly. - AI also flagged that `vulkan12Features` is _what the original captured app enabled_, versus `avail12Features` which was freshly queried what the replay device supports. - For things like https://github.com/baldurk/renderdoc/blob/ce50614f0e77d007982b776cbc85b8494720459d/renderdoc/driver/vulkan/wrappers/vk_device_funcs.cpp#L5154-L5165, it flagged that if the calling app enabled `0` extensions, the loop never executes and neither does the version check for promoted extensions to enable `vulkanVersion >= ver`...

Comments

baldurk

I think I need to ban use of LLMs generating issues because this is an unreadable vomit of words. > Since this situation happens specifically on Android and depends on the "random" order of instance creation, I think I can set up a small Vulkan sample that always initializes a Vulkan 1.2+ instance plus stabilized feature, then initializes a second Vulkan instance on 1.1, and ultimately create a device on the 1.2 instance while trying to use 1.2 functionality on it, like BDA. Let me know if that's needed! Please provide a reproducible test case so someone can investigate this from that.

MarijnS95

@baldurk I specifically stated that AI was not used in writing this issue, only to pin down the million places in the code where things appear to be going wrong (which are obviously incomprehensible by hand). I'll gladly accept this as "you're terrible at writing concise issues" though 😉. It's hard to keep it short while also demonstrating that (I think) I've done my due-diligence in boiling down and pin-pointing where things might be going wrong, rather than posting logs paired with "sometimes it works other times it doesn't". Specifically, that should enable writing a much smaller and more targeted reproducer in the first place.

baldurk

I'm sorry, that is my mistake as I made an incorrect assumption. I will be honest that I didn't read down to the small footnote at the bottom of the issue where you said that you didn't write it with LLMs. I saw you mention that you were using it for debugging and given the verbosity of writing and the description being quite hard to follow and understand what the actual problem was through tangents, quotes of code and logs, and other extra information interleaved in I made the assumption that you hadn't limited your use. Regardless if you are willing to provide a reproducible test case that would be the best way to continue.

MarijnS95

Regardless AI has completely taken the life and fun out of programming for me. <details> <summary>Historic noise about AI misleading</summary> In this instance it pointed out to me that `renderdocAppInfo` is a global static that was being modified, but also mislead me into thinking that that `m_Instance` was also global. I saw it being a struct member instead, and noted to double-check that the owning struct was stored and shared globally but never got back to check in on it. I thought about it right after stripping Android's timestamp and PID/TID out of the logs for brevity (which would have already showed the issue...) but it was already late. </details> The repro I made based on that false assumption... **didn't reproduce it**, effectively voiding all the noise above in the original report. So I went back to instrumenting `renderdocAppInfo` overwrites with logs (part Claude, part me) and to no surprise, that shows the race condition. The logs below are based on https://github.com/MarijnS95/renderdoc/compare/issue-3903-instance-timing (but the line numbers won't match, I had some more logs in there that I eventually stripped out): ``` 09-13 11:36:31.727 16806 16842 I vulkan : Loaded layer VK_LAYER_RENDERDOC_Capture 09-13 11:36:31.727 16806 16842 I renderdoc: @6aa66e9f00000011@ RDOC 016806: [11:36:31] vk_layer.cpp( 70) - Log - Creating internal instance to bump layer refcount 09-13 11:36:31.728 16806 16842 I vulkan : Loaded layer VK_LAYER_RENDERDOC_Capture 09-13 11:36:31.728 16806 16842 I renderdoc: @6aa66e9f00000012@ RDOC 016806: [11:36:31] vk_device_funcs.cpp( 823) - Error - App minor version 0 09-13 11:36:31.728 16806 16842 I renderdoc: @6aa66e9f00000013@ RDOC 016806: [11:36:31] vk_device_funcs.cpp( 840) - Log - vkCreateInstance begin: tid 491577091328 requested apiVersion 400000 ... 09-13 11:36:31.733 16806 16860 I vulkan : Loaded layer VK_LAYER_RENDERDOC_Capture 09-13 11:36:31.733 16806 16860 I renderdoc: @6aa66e9f00000016@ RDOC 016806: [11:36:31] vk_device_funcs.cpp( 840) - Log - vkCreateInstance begin: tid 490512160000 requested apiVersion 403000 ... 09-13 11:36:31.740 16806 16842 I AdrenoVK-0: Application Name : RenderDoc Capturing App 09-13 11:36:31.740 16806 16842 I AdrenoVK-0: Application Version : 0x0042f000 09-13 11:36:31.740 16806 16842 I AdrenoVK-0: Engine Name : RenderDoc 09-13 11:36:31.740 16806 16842 I AdrenoVK-0: Engine Version : 0x0042f000 09-13 11:36:31.740 16806 16842 I AdrenoVK-0: Api Version : 0x00403000 09-13 11:36:31.740 16806 16842 E vulkan : internal vkGetInstanceProcAddr called for vkCreateInstance with an instance 09-13 11:36:31.740 16806 16842 I renderdoc: @6aa66e9f00000019@ RDOC 016806: [11:36:31] vk_device_funcs.cpp( 877) - Log - vkCreateInstance end: tid 491577091328 requested 400000, recorded 403000, driver call 12.02ms <-- OVERWRITTEN 09-13 11:36:31.740 16806 16842 I renderdoc: @6aa66e9f0000001a@ RDOC 016806: [11:36:31] vk_device_funcs.cpp( 997) - Log - Initialised capture layer in Vulkan instance. 09-13 11:36:31.740 16806 16842 I renderdoc: @6aa66e9f0000001b@ RDOC 016806: [11:36:31] vk_layer.cpp( 89) - Log - Created own instance B40000734CA55870: VK_SUCCESS ... 09-13 11:36:31.741 16806 16860 I AdrenoVK-0: Application Name : RenderDoc Capturing App 09-13 11:36:31.741 16806 16860 I AdrenoVK-0: Application Version : 0x0042f000 09-13 11:36:31.741 16806 16860 I AdrenoVK-0: Engine Name : RenderDoc 09-13 11:36:31.741 16806 16860 I AdrenoVK-0: Engine Version : 0x0042f000 09-13 11:36:31.741 16806 16860 I AdrenoVK-0: Api Version : 0x00403000 09-13 11:36:31.741 16806 16842 I renderdoc: @6aa66e9f0000001d@ RDOC 016806: [11:36:31] vk_device_funcs.cpp( 840) - Log - vkCreateInstance begin: tid 491577091328 requested apiVersion 401000 09-13 11:36:31.741 16806 16860 E vulkan : internal vkGetInstanceProcAddr called for vkCreateInstance with an instance 09-13 11:36:31.741 16806 16860 I renderdoc: @6aa66e9f00000020@ RDOC 016806: [11:36:31] vk_device_funcs.cpp( 877) - Log - vkCreateInstance end: tid 490512160000 requested 403000, recorded 401000, driver call 7.42ms <-- OVERWRITTEN 09-13 11:36:31.741 16806 16860 I renderdoc: @6aa66e9f00000021@ RDOC 016806: [11:36:31] core.cpp(2317) - Log - Adding Vulkan device frame capturer for 0xB40000747C986450 09-13 11:36:31.741 16806 16860 I renderdoc: @6aa66e9f00000022@ RDOC 016806: [11:36:31] vk_device_funcs.cpp( 997) - Log - Initialised capture layer in Vulkan instance. ... 09-13 11:36:31.743 16806 16842 I AdrenoVK-0: Application Name : RenderDoc Capturing App 09-13 11:36:31.743 16806 16842 I AdrenoVK-0: Application Version : 0x0042f000 09-13 11:36:31.743 16806 16842 I AdrenoVK-0: Engine Name : RenderDoc 09-13 11:36:31.743 16806 16842 I AdrenoVK-0: Engine Version : 0x0042f000 09-13 11:36:31.743 16806 16842 I AdrenoVK-0: Api Version : 0x00401000 09-13 11:36:31.743 16806 16842 E vulkan : internal vkGetInstanceProcAddr called for vkCreateInstance with an instance 09-13 11:36:31.743 16806 16842 I renderdoc: @6aa66e9f00000029@ RDOC 016806: [11:36:31] vk_device_funcs.cpp( 877) - Log - vkCreateInstance end: tid 491577091328 requested 401000, recorded 401000, driver call 2.42ms 09-13 11:36:31.743 16806 16842 I renderdoc: @6aa66e9f0000002a@ RDOC 016806: [11:36:31] core.cpp(2317) - Log - Adding Vulkan device frame capturer for 0xB40000747C9886B0 09-13 11:36:31.743 16806 16842 I renderdoc: @6aa66e9f0000002b@ RDOC 016806: [11:36:31] vk_device_funcs.cpp( 997) - Log - Initialised capture layer in Vulkan instance. 09-13 11:36:31.743 16806 16842 I renderdoc: @6aa66e9f0000002c@ RDOC 016806: [11:36:31] vk_device_funcs.cpp(1582) - Log - physical device 0: Adreno (TM) 830 (ver 512.800 patch 0x3d) - 5143:44050001 09-13 11:36:31.744 16806 16860 I renderdoc: @6aa66e9f0000002d@ RDOC 016806: [11:36:31] vk_device_funcs.cpp(5120) - Log - Created capture device from physical device 0 09-13 11:36:31.744 16806 16842 I renderdoc: @6aa66e9f0000002e@ RDOC 016806: [11:36:31] vk_device_funcs.cpp(5020) - Warning - shaderStorageImageMultisample = false, multisampled textures will have empty contents at frame start. 09-13 11:36:31.744 16806 16860 I renderdoc: @6aa66e9f0000002f@ RDOC 016806: [11:36:31] vk_device_funcs.cpp(5157) - Error - Device create? With vulkanVersion 401000 ... 09-13 11:36:31.745 16806 16842 I renderdoc: @6aa66e9f0000003a@ RDOC 016806: [11:36:31] vk_device_funcs.cpp(5120) - Log - Created capture device from physical device 0 09-13 11:36:31.745 16806 16842 I renderdoc: @6aa66e9f0000003b@ RDOC 016806: [11:36:31] vk_device_funcs.cpp(5157) - Error - Device create? With vulkanVersion 401000 ... 09-13 11:36:31.850 16806 16880 I renderdoc: @6aa66e9f00000046@ RDOC 016806: [11:36:31] vk_resource_funcs.cpp(2251) - Error - Device address bit specified but no device address extension enabled ... --------- beginning of crash 09-13 11:36:31.932 16806 16880 F libc : Fatal signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x0 in tid 16880 (RenderLoop Rend), pid 16806 (Our app) ... 09-13 11:36:32.244 16892 16892 F DEBUG : #01 pc 0000000000e6019c /data/app/~~Sj5oYts7IZLkvMC2vH5Frw==/org.renderdoc.renderdoccmd.arm64-asJ2ED2TKJS1-ja_sOGB1Q==/lib/arm64/libVkLayer_GLES_RenderDoc.so (WrappedVulkan::vkAllocateMemory(VkDevice_T*, VkMemoryAllocateInfo const*, VkAllocationCallbacks const*, VkDeviceMemory_T**)+6776) (BuildId: 7485e456f25cc24c) ``` Multiple threads have clearly overwritten the global with the wrong version that's copied right back into `record->instDevInfo->vulkanVersion` after instance creation. Some time later, you see my `Device create? With vulkanVersion 401000` log meaning the entire device runs thinking it's Vulkan 1.1, the extensions we rely on haven't been loaded through RenderDoc, and crashes occur per the original report. Simply making the global immutable and copying it to the stack for modification - not to mention no longer having `static VkApplicationInfo renderdocAppInfo` **the vessel for `apiVersion`** - solves all these problems and makes me able to start our app reliably under RenderDoc once again 🎉

baldurk

> Regardless AI has completely taken the life and fun out of programming for me. I have good news, no-one has a gun to your head (I hope) and it's entirely possible to just continue programming without voluntarily welcoming a bunch of slop into the process.