Fail to re-index because of 503 HTTP code status and liveness watches it about helm-charts

[stevehipwell]

I'll look at updating my Jira and Confluence charts to use tcpSocket for liveness.

[stevehipwell]

Thanks for opening this issue @ppuiseux, this is a result of Jira not having K8s compatible status endpoint(s). The ideal solution would be for Jira to provided a /live and a /ready endpoint and only returned the 503 on /ready. In the meanwhile you can either customise the httpGet probe to wait longer or you can set httpGet: {} and provide a tcpSocket probe instead from the chart values. You should only have to do this for liveness as a failing readiness check will not cause a pod restart.

If you can test the tcpSocket behaviour for the liveness check I will consider adding it as default to the chart.

[ppuiseux]

Thank you @stevehipwell .
I totally agree with you about the endpoints.

I set my livenessProbe like that:

livenessProbe:
  failureThreshold: 10
  initialDelaySeconds: 360
  periodSeconds: 30
  successThreshold: 1
  httpGet: null
  tcpSocket:
    port: http
  timeoutSeconds: 1

and it works.
However, during re-indexation I have an nginx error and not the Jira Error page but for final user it is the same, he doesn't have application access.

[stevehipwell]

@ppuiseux that's because the pod isn't ready during this period. The actual issue is that Jira doesn't have a K8s compatible principal of ready, it's either conservative ready (/status is 200) or it's always ready (tcpSocket available). For me conservative ready is the most correct of the two, but it sounds like I could switch to use tcpSocket for liveness checks.

I'm pretty sure that with Nginx you could use a custom backend when the pod isn't ready to show a maintenance page.

[ppuiseux]

Thank you @stevehipwell .
I raised a ticket on Atlassian Support and I am trying to find the right channel to transmit your solution of /live and /ready endpoints which are the most appropriate ways to solve this problem in my opinion too.
I close the issue here but I keep the link of this page to give it to the appropriate person..

[stevehipwell]

Thank you @ppuiseux. What are your thoughts on using tcpSocket for liveness so at least the container doesn't restart while running maintenance?

[ppuiseux]

Sorry @stevehipwell I believed I had answer that in this message.
It works well, the application stays alive while running re-indexation.
The configuration I used is in the same message.

[stevehipwell]

@ppuiseux did you also test any other http paths for liveness such as /?

[ppuiseux]

No.
I've just tested it and I've got a 302 on /
I try this edit and I come back to you.

[stevehipwell]

@ppuiseux codes from 200 to 399 are considered a success for http probes.

[ppuiseux]

I launched two re-index and both failed at the same moment (9%) with liveness on /.
I have a third running right now with liveness on tcpSocket and is reaches 25% and it continues.

I have a clue. When I run wget http://localhost:80800/ I get a 302 which redirects me to a 503.
I think that Kubernetes follows redirect and keep 503 and not 302.

[ppuiseux]

I opened a ticket on https://jira.atlassian.com/browse/JRASERVER-72508 to ask for these two new endpoints.