GROMACS fatal errors are not handled gracefully about gmxapi

[peterkasson]

Sounds reasonable to me. If the errors are from Gromacs directly, then I think there are other ways we can catch them if needed. But if they're being triggered by MPI calls, then having an MPI error handler seems like the best solution.

[eirrgang]

I think that in some places GMX_RELEASE_ASSERT should be replaced with exceptions or other more helpful, better logged approaches. The possibility of an unhandled MPI_Error causing exit() during assertion error handling seems like a GROMACS bug. But for our purposes we can currently just keep in mind that these hard-to-catch errors exist, avoid the initialization error situations, and add an MPI error handler for good measure. It is worth double-checking whether GROMACS already adds its own MPI error handler under other circumstances.

[peterkasson]

I'm pretty sure we've done other projects where we intercept exit(), but that's a bit more invasive and probably not optimal.

[eirrgang]

Yeah, I would tend to consider it a bug if exit() is reachable.

It will take time to make GROMACS error handling more API-friendly and in the mean time I think we just have to be careful and vigilant. I had originally intended that this issue could be a place-holder to accumulate a list of all of the places where we need to preempt GMX_RELEASE_ASSERT and similar uncatchable logic, but maybe I should just consider this issue resolved when the MPI initialization is determined to be sufficiently robust.