KServe does not catch Caikit runtime status correctly when subsprocess (tgis) have issues about caikit-tgis-serving

[Xaenalt]

This should be handled by the container splitting (this sprint)

[heyselbi]

We will ping IBM to see what the expected behavior is.

[kpouget]

I see that many text-generation process remains <defunct> when the serving runtime is running after it hit this error:

sh-5.1$ ps aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
1000850+       1  0.5  0.0 19143780 611752 ?     Ssl  09:02   0:07 python3 -m caikit.runtime.grpc_server
1000850+     401  0.0  0.0      0     0 ?        Z    09:03   0:01 [text-generation] <defunct>
1000850+     403  0.0  0.0      0     0 ?        Z    09:03   0:01 [text-generation] <defunct>
1000850+     405  0.0  0.0      0     0 ?        Z    09:03   0:00 [text-generation] <defunct>

this means that Python caikit.runtime.grpc_server process does not wait() for the text-generation. This explains why it doesn't detect that text-generation failed ...

[danielezonca]

This is the ticket for reference :)

• opendatahub-io/caikit#11

[dtrifiro]

When running caikit+tgis in this configuration, this is no longer an issue.

[kpouget]

@dtrifiro what about when running caikit+tgis in the single container? 🤔

[Xaenalt]

Single container architecture was only a stopgap to the current architecture

[lugi0]

Verified in RHOAI 2.5 RC4, now if I cause a failure while loading a model (i.e. give the wrong path to the deployment modal) the InferenceService will set the modelStatus to FailedToLoad with relevant error messages.