What is the recommended procedure for performing operating system patching on PostgreSQL nodes in a Chef Automate High Availability (HA) cluster while minimizing service disruption and ensuring cluster health?

This runbook assumes the PostgreSQL HA cluster is healthy before maintenance begins. Do not perform OS patching on a degraded cluster. Every maintenance step must be gated by a successful pre_test_check.sh execution and confirmation that PostgreSQL resynchronization (pg_basebackup) has completed.

Prerequisites

Before beginning maintenance:

• Ensure SSH access to all PostgreSQL nodes.
• Ensure administrative (sudo) access is available.
• Schedule an approved maintenance window.
• Inform stakeholders of the expected maintenance impact:
  • Follower patching: No write unavailability and no data loss risk.
  • Leader patching: Approximately 25 seconds of write unavailability, followed by 10–20 minutes of elevated application errors while connection pools recover.
• Ensure the pre_test_check.sh script is available.

Safety Rules

The following rules should always be observed:

• Patch only one PostgreSQL node at a time.
• Never continue until the previous node has fully resynchronized and pre_test_check.sh reports all checks passing.
• Never skip the pre-flight validation.
• Always stop and start Habitat using:

sudo systemctl stop hab-sup
sudo systemctl start hab-sup

Stopping Habitat preserves service specifications and bind configuration.

• Never reboot a node while pg_basebackup is running. Interrupting an active resynchronization can corrupt the PostgreSQL data directory.

Recommended Patching Sequence

Always patch follower nodes before the leader.

Recommended order:

1. Patch Follower A
2. Validate cluster
3. Patch Follower B
4. Validate cluster
5. Patch Leader
6. Final cluster validation

Recommended maintenance strategy:

Each follower typically requires 75–150 minutes, while the leader may require 90–175 minutes depending on database size and PostgreSQL resynchronization time.

Follower Node Patching

Write impact: None

Failover: No

Data loss risk: None

1. Perform pre-flight validation

Run:

./pre_test_check.sh

Ensure all checks pass before proceeding.

Specifically verify:

• Exactly 1 primary and 2 streaming standbys
• No active pg_basebackup
• Replication lag is minimal
• No wal_sender_timeout errors

If any validation fails, stop the maintenance and resolve the issue before continuing.

2. Confirm the node is a follower

Run:

SELECT pg_is_in_recovery();

Expected output:

t

t indicates the node is operating as a standby.

3. Stop Habitat and reboot

sudo systemctl stop hab-sup
sudo reboot

After reboot, Habitat automatically starts PostgreSQL and initiates a fresh pg_basebackup from the primary.

4. Wait for PostgreSQL resynchronization

Verify that resynchronization has started:

sudo pgrep -a pg_basebackup

Example output:

12345 pg_basebackup ...

Wait until the process completes.

Run:

./pre_test_check.sh

Proceed only when:

• No pg_basebackup processes are running.
• WAL receiver state is streaming.
• Replication lag is near zero.

5. Apply OS patches

sudo dnf update -y

(Use the appropriate package manager for your operating system.)

6. Perform the second reboot

sudo reboot

A second PostgreSQL resynchronization begins automatically.

Run:

./pre_test_check.sh

Confirm:

• One primary
• Two streaming standbys
• No active pg_basebackup
• Replication lag near zero

Repeat the same process for the second follower.

Leader Node Patching

Write impact: Approximately 25 seconds during controlled failover

Failover: Yes

Data loss risk: Any WAL that has not yet replicated to a standby at the moment of failover is discarded. Ensure replication lag is minimal before initiating the controlled failover.

1. Validate cluster health

Ensure both followers have already been patched and are fully synchronized.

Run:

./pre_test_check.sh

All checks must pass before continuing.

2. Verify replication lag

Run:

SELECT client_addr,
       state,
       pg_wal_lsn_diff(sent_lsn, replay_lsn) AS lag_bytes
FROM pg_stat_replication
ORDER BY client_addr;

Expected output:

client_addr    state      lag_bytes
-----------    --------   ---------
standby-1      streaming       0
standby-2      streaming       0

Wait until replication lag is close to zero before proceeding.

3. Monitor leader election

Open a separate terminal and monitor PostgreSQL roles:

watch -n 2 '<role validation command>'

Continue monitoring until a new primary is elected.

4. Perform a controlled failover

Run:

sudo systemctl stop hab-sup

Expected behaviour:

• Approximately 25 seconds of write unavailability.
• One standby is promoted automatically.
• Temporary client connection failures may occur while applications reconnect.

5. Validate the new leader

Run:

./pre_test_check.sh

Confirm:

• Exactly one primary
• At least one streaming standby
• No split-brain detected
• chef-automate status reports healthy services

6. Reboot the former leader

sudo reboot

After reboot, PostgreSQL automatically performs a fresh pg_basebackup.

Verify:

sudo pgrep -a pg_basebackup

Wait until the process completes, then run:

./pre_test_check.sh

Do not continue until:

• No active pg_basebackup
• WAL receiver state is streaming
• Replication lag is near zero

7. Apply OS patches

sudo dnf update -y

8. Perform the second reboot

sudo reboot

After PostgreSQL completes its second resynchronization, run:

./pre_test_check.sh

Confirm:

• One primary
• Two streaming standbys
• No replication lag
• No active pg_basebackup
• chef-automate status reports all services healthy

Timing Expectations

The duration of PostgreSQL resynchronization (pg_basebackup) depends on database size, write activity, and network throughput. In production environments, allow 20–60 minutes for each resynchronization.

Rollback and Abort Criteria

Immediately stop the maintenance activity and investigate if any of the following occurs:

• No new primary is elected within 90 seconds after stopping the leader.
• Split-brain is detected (more than one primary).
• A reboot is about to occur while pg_basebackup is still running.
• The new primary reports no connected standbys for more than five minutes.
• Persistent chef-automate status failures remain after failover.
• pre_test_check.sh reports an invalid wal_sender_timeout configuration.

Do not continue until the cluster has returned to a healthy state.

Pre-Patching Validation Script

The pre_test_check.sh script validates:

• PostgreSQL replication health
• Replication lag
• Streaming status
• Split-brain detection
• Active pg_basebackup operations
• Failover readiness
• Overall cluster health

The pre_test_check.sh script is not included with Chef Automate. Licensed Chef customers can obtain the latest version by opening a support ticket with Chef Support.

Recommendation

Patch only one PostgreSQL node at a time, always validate the cluster before and after every maintenance step, and ensure PostgreSQL has fully resynchronized before proceeding. For production environments, patch follower nodes during one maintenance window and schedule leader patching during a separate maintenance window to minimize operational risk.

This document is intended as a reference runbook for patching PostgreSQL nodes in a Chef Automate HA cluster. If you require clarification or have environment-specific concerns, please contact Chef Support or your Progress account team before performing the maintenance.

The pre_test_check.sh validation script is not included with Chef Automate and is not publicly available. Licensed Chef customers may request the latest version by submitting a support ticket with Chef Support.

https://community.progress.com/s/article/How-To-Submit-A-Chef-Support-Ticket