Chef backend servers were not accepting traffic after both the leader and follower nodes were restarted. Once the servers came back online the previous leader was re-elected as the leader and the other two nodes as followers. However all three nodes remained in the waiting_for_leader state and none of the nodes were marked as eligible.

chef-backend-ctl status output shows as below:

Service        Local Status         Time in State  Distributed Node Status
leaderl        running (pid 19323)  0d 2h 8m 22s   leader: 0; waiting: 3; follower: 0; total: 3
epmd           running (pid 19326)  0d 2h 8m 22s   status: local-only
etcd           running (pid 19329)  0d 2h 8m 21s   health: green; healthy nodes: 3/3
postgresql     running (pid 19392)  0d 2h 8m 23s   error: could not reach pgsql leader for replication stats
elasticsearch  running (pid 19401)  0d 2h 8m 22s   state: green; nodes online: 3/3

Chef-cluster-status shows 

Name             IP             GUID                              Role                PG        ES          Blocked      Eligible

***************  ************** ********************************  waiting_for_leader  leader    master      not_blocked  false

***************  ************** ********************************  waiting_for_leader  follower  not_master  not_blocked  false

***************  ************** ********************************  waiting_for_leader  follower  not_master  not_blocked  false

To identify the last elected leader of the Chef backend cluster, run the following command. This command helps determine which node was previously serving as the leader before the restart or failover occurred.

/opt/chef-backend/embedded/bin/etcdctl get /cb/leader/last_leader

Check whether the bootstrap key is present on the Chef backend server. The bootstrap key is required for the backend cluster to establish the appropriate trust and communication during initialisation and leader election.

Verify that the expected bootstrap key exists and is accessible on the node. If the key is missing the backend nodes may fail to complete the bootstrap or leader-election process and can remain in the waiting_for_leader state.

/opt/chef-backend/embedded/bin/etcdctl get /cb/control/bootstrap

Restore the missing bootstrap key by running the following command. This will recreate or restore the required bootstrap key so that chef backend nodes can establish the necessary trust and proceed with the cluster initialisation and leader-election process.

/opt/chef-backend/embedded/bin/etcdctl set /cb/control/bootstrap GUID

Note: Retrieve the GUID of the last known leader from the output of the chef-cluster-status command.

Restart the leader:

chef-backend-ctl restart

Following the successful leader election and restoration of the Chef backend cluster, environment return to a normal and stable operating state.Chef backend nodes were functioning as expected and the cluster was able to resume its normal operations without further issues.

Once the backend cluster was fully restored, Chef Client nodes should begin successfully checking in again.