"Unhooking" about hudhook

[veeenu]

Hi Rob! That's a very good idea and it would be very useful for the development workflow. I have a couple comments.

I would avoid IPC because it would make the library more opinionated and further enforce conventions on library users. hudhook is meant to be batteries-included to a degree but I would still want it to be as simple as possible and for it to do as few things as strictly necessary. Implementing IPC would sort of force a number of design decisions I'd rather avoid.

I think a reasonable alternative approach I'd be ok with would be to simply use some global atomic primitives, i.e. DLL_PROCESS_DETACH handler stores an enumeration state in a RwLock which is polled by the render thread at every frame. If that has a certain variant, the render thread cleans up to exit, and then sets another variant which is waited upon by the handler which works through the remainder of the cleanup process. Overall, I think this is a viable strategy for development workflow, but we still have to consider the fact that we're mutating the hooked program's global state and really have no way to reason about the soundness of our changes, e.g. if they're idempotent and can be reverted cleanly. It's not uncommon for my darksoulsiii_practice_tool to just crash and burn every now and again because just flipping bytes here and there has some outcomes I can't predict. So I'm not sure I would ever suggest using such a feature in production and would rather just have the users close the program to reset state.

ThrowawayHwnd is necessary because we don't have a way of unequivocally retrieving the current program's HWND in order to query IDXGISwapChain's vtable. Using the program's HWND tends to work most of the time during development but if by chance the window isn't focused or is minimized (that tends to very often be the case, with people alt-tabbing out of a fullscreen game to launch a tool) the HWND would not be found and the method would fail. I'm open to improvements on this (i.e. can we just deregister the class after querying the vtable?), but I think the ThrowawayHwnd should stay.

[robeady]

I would avoid IPC because it would make the library more opinionated and further enforce conventions on library users.

Agreed, it's pretty much incompatible with it being "just a library"

I think a reasonable alternative approach I'd be ok with would be to simply use some global atomic primitives, i.e. DLL_PROCESS_DETACH handler stores an enumeration state in a RwLock which is polled by the render thread at every frame. If that has a certain variant, the render thread cleans up to exit, and then sets another variant which is waited upon by the handler which works through the remainder of the cleanup process.

hmm, so the detach handler would wait for the render thread to do its cleanup and than exit. In theory, I suspect waiting for other threads is high on the list of "things not to do" from DllMain. In practice I suspect it would work, especially as a development-only feature. I agree that in a "production" setting (i.e. if you cared about your game not crashing) you would not want to utilise this feature :)

ThrowawayHwnd is necessary because we don't have a way of unequivocally retrieving the current program's HWND in order to query IDXGISwapChain's vtable

I see what you mean. Yes, the window class can be unregistered to solve the issue I think.

That reminds me of another challenge I came across... I'm using this library to inject into a game emulated by PCSX2, which has a pause/resume feature which basically closes the rendered window and makes a new one. I didn't yet figure out how to detect if this happens and repeat the injection process (after this happens, I don't think the render function is hooked any more, but maybe I missed something and that's not correct)

[veeenu]

Took me a hot minute 😎

[robeady]

Oh, nice :) at some point I hope to get back to the project I was working on and try this out