Chef Backend HA Cluster Full Manual Recovery
In a Chef Backend High Availability deployment, the etcd service is extremely sensitive and can get into a bad state across backend nodes due to disk and/or network latency. When this happens, it is common for the cluster to be unable to automatically failover/recover. In this case, you'll need to sync the followers from the leader using a full basebackup procedure, because the WAL entries have already rotated. This article covers this scenario with Chef Backend HA and provides some insight into how it can be remedied.