Can't make calls to vault via kubectl exec (vault container not found) about operations

[larsks]

@dystewart, thanks for opening this issue.

The behavior seems intermittent; if we run this script:

i=0
while :; do
  echo -n "[$i] "
  oc --as system:admin -n vault exec -it nerc-vault-0 -- date 2>&1
  sleep 1
  let i++
done

We see as output:

[0] error: unable to upgrade connection: container not found ("vault")
[1] error: unable to upgrade connection: container not found ("vault")
[2] Fri Dec  2 23:07:03 UTC 2022
[3] error: unable to upgrade connection: container not found ("vault")
[4] error: unable to upgrade connection: container not found ("vault")
[5] Fri Dec  2 23:07:06 UTC 2022
[6] error: unable to upgrade connection: container not found ("vault")
[7] Fri Dec  2 23:07:09 UTC 2022
[8] Fri Dec  2 23:07:10 UTC 2022
[9] Fri Dec  2 23:07:12 UTC 2022
[10] error: unable to upgrade connection: container not found ("vault")
[11] Fri Dec  2 23:07:14 UTC 2022
[12] Fri Dec  2 23:07:15 UTC 2022

This behavior seems specific to the vault containers; if I pick any other pod on the system, I can't reproduce these failures.

Normally, I would suspect that the pods were restarting, but they've been running consistently with no restarts:

$ k get pod -l app.kubernetes.io/instance=nerc-vault
NAME           READY   STATUS    RESTARTS   AGE
nerc-vault-0   1/1     Running   0          15d
nerc-vault-1   1/1     Running   0          15d
nerc-vault-2   1/1     Running   0          15d

[larsks]

Redeploying the vault pods for #49 seems to have resolved this issue. Repeating the test from my previous comment, we seeL

[0] Wed Dec 14 04:30:25 UTC 2022
[1] Wed Dec 14 04:30:27 UTC 2022
[2] Wed Dec 14 04:30:28 UTC 2022
[3] Wed Dec 14 04:30:29 UTC 2022
[4] Wed Dec 14 04:30:31 UTC 2022
[5] Wed Dec 14 04:30:32 UTC 2022
[6] Wed Dec 14 04:30:33 UTC 2022
[7] Wed Dec 14 04:30:35 UTC 2022
[8] Wed Dec 14 04:30:36 UTC 2022
[9] Wed Dec 14 04:30:37 UTC 2022
[10] Wed Dec 14 04:30:39 UTC 2022
.
.
.