Can go into infinite recursion when combined with libstrangle about fossilize

[smcv]

I think what is happening here is that as a result of 6916486, if VK_LAYER_fossilize_GetDeviceProcAddr() somehow sees layer->getTable()->GetDeviceProcAddr == VK_LAYER_fossilize_GetDeviceProcAddr, it will call into itself until it runs out of stack space.

Perhaps GetDeviceProcAddr needs to be exempt from the check added in 6916486, so that if pName is "vkGetDeviceProcAddr", it short-circuits to returning VK_LAYER_fossilize_GetDeviceProcAddr immediately? (As though interceptCoreDeviceCommand had been called first, as it was before 6916486 - but it would likely be easier done as a special-case.)

It's entirely possible that one of the other layers involved is doing something wrong, and libstrangle certainly has other issues - but its GetDeviceProcAddr implementation seems to be heavily based on Mesa's overlay, which I would hope is doing the fallback dance correctly. It seems to me that infinite recursion is never going to be the correct answer, so it would probably be more robust if Fossilize avoids the recursion for GetDeviceProcAddr, even if the crash is not actually Fossilize's fault.

[DadSchoorse]

If layer->getTable()->GetDeviceProcAddr is fossilize's own vkGetDeviceProcAddr you have bug somewhere, and I'm pretty sure it's not in fossilize since all it does to get that point is call down the layer chain. Your hypothesis that this is caused by pName being "vkGetDeviceProcAddr" also makes little sense. The last function before VK_LAYER_fossilize_GetDeviceProcAddr in your stack trace is loader_init_device_dispatch_table, which never calls vkGetDeviceProcAddr with "vkGetDeviceProcAddr".

As a total blind guess you could try replacing https://github.com/ValveSoftware/Fossilize/blob/master/layer/dispatch_helper.cpp#L31 with table->GetDeviceProcAddr = gpa.

It would also help to know in which order the layers get loaded.