How do you handle application healthiness? about graceful-shutdown-spring-boot

[gesellix]

Thanks :)

The repo doesn't have an explicit health check configured in the Traefik proxy. I usually leverage the /health endpoint of Spring Boot to signal service healthiness to our proxies. In the PoC, I more or less rely on Traefik's mechnism to recognize container removal to be fast enough.

In a real world scenario you would probably have to modify your service health endpoint to return "DOWN" or "OUT_OF_SERVICE" (https://docs.spring.io/spring-boot/docs/current/reference/html/production-ready-endpoints.html#_writing_custom_healthindicators) before actually shutting down the container or service. You could hook into Spring's application lifecycle events like this (non-tested code!):

...

@Configuration
class GracefulShutdownConfig implements ApplicationListener<ContextClosedEvent> {

  public static final int HEALTH_CHECK_INTERVAL_SEC = 5
  public static final int GRACEFUL_SHUTDOWN_DELAY = HEALTH_CHECK_INTERVAL_SEC * 2 * 1000
  volatile status = Status.UP

  @Override
  void onApplicationEvent(ContextClosedEvent contextClosedEvent) {
    status = Status.OUT_OF_SERVICE
    sleep(GRACEFUL_SHUTDOWN_DELAY)
  }

  @Bean
  HealthIndicator shutDownHealth() {
    [health: { Health.status(status).build() }] as HealthIndicator
  }
}

Note: this one uses constant timeouts instead of the PoC's idea - seems like I should invest some time to combine both ideas.

[gesellix]

P.S.: the Docker Engine also acts as a simple LB, so because it performs the SIGTERM itself on the container, it also removes the container (or task) from its internal list of available instances. There have been discussions/issues, how early this should happen before actually sending the SIGTERM, which you can see in the linked issue in the README of this repo.

[marcosnils]

Thanks a lot for the input. We're using external proxies in our case so we're not affected from the swarm ipvs behavior.

After thinking about it I believe that we'll specifically remove the instance from the load balancer instead of relying on the health checks. The reason for this is because the LB health configuration might have different timeouts and retry policies depending on the application, so configuring a timeout after changing the health status results a bit complicated as we can't really find a unique value that suits most cases.

[gesellix]

Alright, that's probably the best option to disable the backends explicitly :) Depending on your needs you might consider to disable the backends first and remove them after a short timeout. We need that due to long running requests - which is the reason for this PoC.

May I ask which LB you're using in your setup?

[marcosnils]

Sure, our setup looks like this:

API gateway ----> Fabio ---> backend servers
^
Nomad + Consul

Fabio listens to the changes in consul which are produced by nomad and sets up the balancing accordingly.

I'll have to check if removing the backend server from Fabio causes it to terminate all in-flight queries or if I specifically need to add a routing rule to deny new connections before deploying. I'd prefer if the first option works as I don't need to remove the rule in the end

[gesellix]

Nice, thanks!

I'll close this issue now, but feel free to continue this little chat :)