Use `/status/ready` as Gateway's containers readiness probe about charts

[rainest]

These being values.yaml defaults has really bitten us here. IDK if we really ever had a good reason to make these probes configurable (if either the gateway or controller fail to come up within a standard amount of time/retries, my intuition is either we've done something wrong or you have assigned your containers seriously potato levels of resources and need to up them), but they are, so we're kinda stuck with that.

It's an open question as to whether removing configurable status would cause issues in an ideal world, but in reality the answer is "probably" because the limitations of the old endpoint mean there are a fair amount of configurations in the wild that instead check a proxy route, as a DIY version of the "does the proxy have configuration?" status check now in 3.3. Simply removing this from config surface and hard-defaulting everything <3.3 to the existing /status check would disrupt anyone using that workaround. Options are:

(1) Remove probes from configuration surface, automate in templates

Problematic for the "breaks known in-use workaround" reason above, and maybe more, if there's some small segment of users doing something entirely different for unknown reasons.

Rather than having configurable probes, templates would set them automatically. This can address an old issue where probe and listen configuration need to match, but must be updated manually.

(2) Change the values.yaml default to use the 3.3+ endpoint

Breaks the original acceptance criteria, but viable. We would need to add an upgrade note that says "if you're on <3.3, you need to explicitly set the old default in your values.yaml before upgrading the chart". Upgrades that neglect to do this would hit crashloops on the upgraded Pods, so the upgrade would fail and leave the existing release running. This is annoying, but safe.

We would need to confirm upgrade failure and rollback behavior before releasing.

(3) Remove probe configuration surface, but check if not default
Option 1, but attempt to account for configurations that were using the workaround. We remove the values.yaml block, but have templates check if the keys are non-empty. If they are set, use the value given. If not, use defaults within the template. Effectively make this a hidden piece of config surface that we recommend against for any new install.

Optionally have this check to see if values are explicitly set to the old default and override if >=3.3 to account for wholly copy-pastaed default values.yamls, but that's probably going overboard.

Ed: additional issue here is that we'd either have to do this for all the probes (there are six total) or have inconsistent UX where most of them are specified directly in values.yaml, but the proxy readiness probe isn't, unless it is (it's not in the default values.yaml but we will honor a values.yaml override if present).

---

Kong with no Ingress Controller (just using /status/ready might break it as the Admin API service will never get ready and deck won't be able to configure the Gateway, which may require special handling)

We can ignore this. If you're not using a controller you're either using a static DB-less config (which loads at startup) or a database (which also loads config at startup or fails, since Pods that can't connect to the DB self-terminate immediately). The new endpoint should immediately report true for either of these.

[mheap]

What about option 4: Leave the existing defaults, handle the new liveness probes as part of docs then make a major chart release alongside KIC 3.0 to switch the default?

[rainest]

It's doable, but doesn't change anything beyond delaying it. We'd probably get low adoption of the new behavior until then, and we don't actually know when we'll release KIC 3.0.

[czeslavo]

I think my original proposition was a variation of option 2: we keep the readiness probe definition in values.yaml and we change the endpoint to the new one (/status/ready). When the proxy's image version is < 3.3 and the readinessProbe.httpGet.path == /status/ready (a new default), we override it to the old default (/status) to make the default behavior compatible with Kong < 3.3.

• If someone had readinessProbe.httpGet.path set to anything else than the default in values.yaml (for any reason) their setup won't break (their override will be used).
• If someone had readinessProbe.httpGet.path untouched:
  • For Kong < 3.3 they'll get /status
  • For Kong >= 3.3 they'll get /status/ready.

Wouldn't that work?

[mheap]

That would be the ideal end user state, but I believe that's the template hackyness that @rainest was talking about

[rainest]

I didn't think it was possible to selectively edit fields within an object default, but apparently it is, so that approach is indeed viable. It still requires a template hack, but since it doesn't make the configuration uneven, it's better than my original option 2 (#842). We can merge that immediately without breakage.

#841 is the breaking change that just uses the new endpoint in values.yaml without any compensating behavior. Older version users would need to manually configure the old default.

#842 removes the configuration from the defaults, but will still honor configuration if present. If there is no custom configuration, it will render a preset probe compatible with the Kong and KIC versions.

#844 keeps configuration and uses the new default, but will selectively edit the path without user action if needed for old version compatibility.