Switchover and failover get stuck in the report_lst status in certain configurations about pg_auto_failover

[xinferum]

If in an architecture with 3 datanodes, if one of them is lost, the execution of switchover or promotion is considered incorrect, then it would be good to have a mechanism for blocking such actions, issue an error to the user and not cause an attempt to change roles in order to avoid the situations described above.

[Akkowicz]

The documentation regarding regarding fault tolerance, failover behaviour and settings that you can tweak is pretty extensive and available here:
https://pg-auto-failover.readthedocs.io/en/main/fault-tolerance.html
https://pg-auto-failover.readthedocs.io/en/main/failover-state-machine.html
https://pg-auto-failover.readthedocs.io/en/main/ref/configuration.html

You probably shouldn't be switching over a healthy node manually at the same time when you're loosing replicas.
If you're removing replicas from the cluster or it's expected that they'll be out for a long while, you should probably either drop them from the cluster on the monitor side or enable maintenance for them.

It is clear that if we lose another datanode, then primary will not commit transactions, since (in our case) one synchronous replica is required:

If you drop all non-functional nodes from the cluster, except the working primary it will get into "single" state and will be available for writing.

This is an open source project, if you require extensive support, it may be a good idea to look for some company or individual that offers such paid support option. Other than that, adding safe guards to CLI commands seems like a good first issue if you want to have a crack at it and think that there's actually a problem.

[Akkowicz]

You may also want to have a look at Patroni-based solution like:
https://github.com/vitabaks/postgresql_cluster

[sgrinko]

Firstly, I want to thank Dimitri Fontaine and other developers for their contribution to the development of this solution!

If one of the nodes has become unavailable, then the pg_auto_failover (witness) of course must be ready for such a possible problem and have in its arsenal a way out of such a situation.
The hang at the report_lsn stage is not good.
I really hope that the developers of this solution will find a good way to solve this problem.

Yes, this is open source software, but not everyone can write code as good as this code.

We really hope to implement the necessary cases in the witness code to solve this problem.

[Akkowicz]

I may be a little rusty, as it was a long time since I've experienced multiple node failure in pg_auto_failover clusters, but in the last 4 years of running both production and testing clusters with high volume of transactions, I haven't noticed this behaviour.

One thing is though that I let the cluster "run itself".
The only times when I do some manual switchovers/set maintenance mode is when I update and reboot servers.