[Bug]: "pod not available", "Cannot extract Pod status", "WAL file not found in the recovery object store" about cloudnative-pg

[ngoeddel-openi]

I fixed the unhealthy state of the cluster by destroying the instance for now. The operator created a new pod keycloak-db-pgcluster-7 and a temporary join job and after a minute the cluster got healthy again.

However that should not be the solution to this kind of error. The operator should be able to heal the cluster if one worker node goes down for whatever reason. I still hope it is just a misconfiguration but I am quite sure I did nothing wrong here.

[ngoeddel-openi]

Before I had destroyed keycloak-db-pgcluster-5 I had a look into its filesystem, especially into the folder /var/lib/postgresql/data/pgdata/pg_wal:

postgres@keycloak-db-pgcluster-5:~/data/pgdata/pg_wal$ ls -la
total 606256
drwxrws---  3 postgres tape     4096 May 16 10:41 .
drwx------ 19 postgres tape     4096 May 16 10:41 ..
-rw-rw----  1 postgres tape 16777216 May 15 16:00 00000005000000000000001A
-rw-rw----  1 postgres tape 16777216 May 15 16:00 00000005000000000000001B
-rw-rw----  1 postgres tape 16777216 May 15 16:00 00000005000000000000001C
-rw-rw----  1 postgres tape 16777216 May 15 16:00 00000005000000000000001D
-rw-rw----  1 postgres tape      169 May 15 16:00 00000005.history
-rw-rw----  1 postgres tape 16777216 May 15 16:00 00000006000000000000001E
-rw-rw----  1 postgres tape 16777216 May 15 16:00 00000006000000000000001F
-rw-rw----  1 postgres tape 16777216 May 15 16:00 000000060000000000000020
-rw-rw----  1 postgres tape 16777216 May 15 16:00 000000060000000000000021
-rw-rw----  1 postgres tape      212 May 15 16:00 00000006.history
-rw-rw----  1 postgres tape 16777216 May 15 16:00 000000070000000000000022
-rw-rw----  1 postgres tape 16777216 May 15 16:00 000000070000000000000023
-rw-rw----  1 postgres tape 16777216 May 15 16:00 000000070000000000000024
-rw-rw----  1 postgres tape 16777216 May 15 16:00 000000070000000000000025
-rw-rw----  1 postgres tape 16777216 May 15 16:00 000000070000000000000026
-rw-rw----  1 postgres tape 16777216 May 15 16:00 000000070000000000000027
-rw-rw----  1 postgres tape 16777216 May 15 16:00 000000070000000000000028
-rw-rw----  1 postgres tape 16777216 May 15 16:00 000000070000000000000029
-rw-rw----  1 postgres tape 16777216 May 15 16:00 00000007000000000000002A
-rw-rw----  1 postgres tape 16777216 May 15 16:00 00000007000000000000002B.partial
-rw-rw----  1 postgres tape      255 May 15 16:00 00000007.history
-rw-rw----  1 postgres tape 16777216 May 15 16:00 00000008000000000000002B
-rw-rw----  1 postgres tape 16777216 May 15 16:00 00000008000000000000002C
-rw-rw----  1 postgres tape 16777216 May 15 16:00 00000008000000000000002D
-rw-rw----  1 postgres tape 16777216 May 15 16:00 00000008000000000000002E
-rw-rw----  1 postgres tape      298 May 15 16:00 00000008.history
-rw-rw----  1 postgres tape 16777216 May 15 16:00 00000009000000000000002E
-rw-rw----  1 postgres tape 16777216 May 15 16:00 00000009000000000000002F
-rw-rw----  1 postgres tape      341 May 15 16:00 00000009.history
-rw-rw----  1 postgres tape 16777216 May 15 16:00 0000000A0000000000000030.partial
-rw-rw----  1 postgres tape      384 May 15 16:00 0000000A.history
-rw-rw----  1 postgres tape 16777216 May 15 16:00 0000000B0000000000000030
-rw-rw----  1 postgres tape 16777216 May 15 16:00 0000000B0000000000000031
-rw-rw----  1 postgres tape 16777216 May 15 16:00 0000000B0000000000000032
-rw-rw----  1 postgres tape 16777216 May 15 16:00 0000000B0000000000000033
-rw-rw----  1 postgres tape      428 May 15 16:00 0000000B.history
-rw-rw----  1 postgres tape 16777216 May 15 16:00 0000000C0000000000000033
-rw-rw----  1 postgres tape 16777216 May 15 16:00 0000000C0000000000000034
-rw-rw----  1 postgres tape 16777216 May 15 16:00 0000000C0000000000000035
-rw-rw----  1 postgres tape 16777216 May 15 16:00 0000000C0000000000000036
-rw-rw----  1 postgres tape 16777216 May 15 16:00 0000000C0000000000000037
-rw-rw----  1 postgres tape      472 May 15 16:00 0000000C.history
-rw-------  1 postgres tape 16777216 May 16 10:41 0000000D0000000000000038
-rw-------  1 postgres tape 16777216 May 16 10:41 0000000D0000000000000039
-rw-rw----  1 postgres tape 16777216 May 15 16:00 0000000D000000000000003A
-rw-------  1 postgres tape      516 May 16 10:41 0000000D.history
drwxrws---  2 postgres tape     4096 May 15 16:00 archive_status

I found it kind of weird that the file 0000000D000000000000003A exists here but not in the other instances. Can this be a remnant from being the primary instance before the sudden shutdown of the worker node?
I deleted that file manually and restarted the pod but it did not solve the issue. It still was a bit weird to see that file there which does not exist in the S3 backup.