Chef 360 Disaster Recovery Restore Fails with Admin Console Timeout and restore-db / kotsadm-rqlite OOMKilled Errors

During a Disaster Recovery (DR) restore operation in Chef 360, the restore process fails while waiting for the Admin Console to deploy (kotsadm). Investigation reveals that the restore-db init container and kotsadm-rqlite-0 pods repeatedly crash or get terminated with OOMKilled (Exit Code 137) and curl: (52) Empty reply from server (Exit Code 52). This occurs when the KOTSADM RQLite database has accumulated historical data and grown excessively large (e.g., > 1 GB). Chef 360 restore failed, Chef 360 disaster recovery, kotsadm restore-db error 52, kotsadm-rqlite OOMKilled, curl 52 empty reply from server kotsadm, unable to wait for admin console timed out, kotsadm init crashloopbackoff, rqlite database size bloat, chef-360 disaster recovery restore timeout, replicated kots rqlite restore fail, exit code 137 kotsadm-rqlite

Sep 29, 2026
Knowledge

Title

URL Name

Chef-360-Disaster-Recovery-Restore-Fails-with-Admin-Console-Timeout-and-restore-db-kotsadm-rqlite-OOMKilled-Errors

Article Number

000305751

Environment

Chef360 - HyperscaleHA

Question/Problem Description

When running ./chef-360 restore in an airgapped or standard environment, the restore wizard successfully restores the cluster state but fails during the Admin Console initialization phase with a timeout error.

Symptoms:

The restore command halts with an Admin Console deployment timeout.
The kotsadm pod remains stuck in Init:CrashLoopBackOff or PodInitializing.
The kotsadm-rqlite-0 pod flaps, repeatedly restarts, or enters CrashLoopBackOff.
Even if memory limits are temporarily increased on the deployment, the Admin Console may start in an inconsistent state where cluster node management options (such as adding nodes) are missing.

Steps to Reproduce

1/ Take a full cluster backup using standard DR procedures: ./chef-360 backup create (or trigger a scheduled backup via the Admin Console/Velero) 2/ On a target host/cluster, initiate the Disaster Recovery restore process using the generated backup: ./chef-360 restore --airgap-bundle chef-360.airgap --skip-store-validation Enter the backup storage/S3 credentials and select the backup created in Step 2. 3/ Observe the restore output: Cluster state is marked as restored (✔ Cluster state restored!). The restore process halts at: ✔ Waiting for the Admin Console to deploy: 1/2 ready ERROR: failed to restore infra from backup: unable to wait for admin console: timed out waiting for the condition 4/ Check the pod status in the kotsadm namespace to observe restore-db container failures with Exit Code 52 and kotsadm-rqlite-0 terminating with OOMKilled (Exit Code 137): kubectl get pods -n kotsadm kubectl describe pod -l app=kotsadm -n kotsadm kubectl describe pod kotsadm-rqlite-0 -n kotsadm

Clarifying Information

Error Message

CLI Output: ✔ Cluster state restored! ✔ Waiting for the Admin Console to deploy: 1/2 ready ERROR: failed to restore infra from backup: unable to wait for admin console: timed out waiting for the condition Pod Status (kubectl get pods -n kotsadm): NAME READY STATUS RESTARTS AGE kotsadm-xxxxxxxxxx-xxxxx 0/1 Init:CrashLoopBackOff 1954 7d kotsadm-rqlite-0 0/1 CrashLoopBackOff 1606 7d Pod Describe / Termination Logs (kubectl describe pod -n kotsadm): State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: OOMKilled Exit Code: 137 Container: restore-db Last State: Terminated Reason: Error Exit Code: 52 restore-db.log: % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 681.3M 0 0 100 681.3M 0 679.2M 00:01 00:01 --:--:-- 679.9M curl: (52) Empty reply from server

Defect Number

Enhancement Number

Cause

The KOTSADM RQLite database (kotsadm-rqlite) accumulates historical and stale metadata over time, causing it to swell significantly (often exceeding X GB, compared to a normal size of Y MB).

During a disaster recovery restore:

Velero restores the RQLite database volume exactly as it was captured in the backup.
The restore-db init container attempts to stream and replay this massive database dump into RQLite.
The process exhausts container memory limits, causing kotsadm-rqlite to get OOMKilled (Exit Code 137) or drop HTTP connections (curl (52) Empty reply from server), preventing the Admin Console from ever becoming healthy.

Resolution

To resolve this issue, rebuild and clean the KOTSADM RQLite database on the source/running cluster, take a fresh backup with the reduced database size, and reattempt the DR restore.

Step 1: Export Current KOTS Application Configuration

Extract the kubectl-kots CLI plugin and export your current decrypted application configuration values:

# Extract the kots binary from the running pod

kubectl -n kotsadm exec deploy/kotsadm -c kotsadm -- cat /kots > kots
chmod +x kots
sudo cp kots /usr/local/bin/kubectl-kots

# Export application configuration

kubectl kots get config --decrypt --appslug chef-360 -n kotsadm > config.yaml

Step 2: Scale Down KOTS Components

Stop the Admin Console deployment and RQLite StatefulSet:

kubectl -n kotsadm scale deployment kotsadm --replicas=0

kubectl -n kotsadm scale statefulset kotsadm-rqlite --replicas=0

Step 3: Delete the Bloated RQLite Database PVC

Delete the Persistent Volume Claim backing the oversized RQLite database:

# Check existing PVCs

kubectl -n kotsadm get pvc

# Delete the RQLite database PVC

kubectl -n kotsadm delete pvc kotsadm-rqlite-kotsadm-rqlite-0

# Verify PV cleanup

kubectl get pv | grep rqlite

Step 4: Recreate RQLite and Scale Up KOTSADM

Scale RQLite back up to initialize a clean, compact database, then scale up KOTSADM:

# Scale up RQLite

kubectl -n kotsadm scale statefulset kotsadm-rqlite --replicas=3

# Wait for RQLite to reach Ready state

kubectl -n kotsadm wait --for=condition=ready pod/kotsadm-rqlite-0 --timeout=5m -n kotsadm
# Scale up KOTSADM deployment

kubectl -n kotsadm scale deployment kotsadm --replicas=1

Step 5: Reinstall the Admin Console with Exported Configuration

Re-apply the configuration to the fresh Admin Console instance using the airgap bundle:

kubectl kots install chef-360/stable \
  --namespace kotsadm \
  --exclude-admin-console \
  --airgap-bundle chef-360.airgap \
  --config-values config.yaml \
  --license-file license.yaml \
  --no-port-forward \
  --skip-preflights

Step 6: Verify Database Size & Take a Fresh Backup

Check the newly created KOTSADM/RQLite database size (it should be reduced from ~GB to ~MB's).
Once the RQlite database is created, create the backup and validate the components inside that backup file on the admin console. You should see the reduced size.
Trigger a fresh backup via the Chef 360 Admin Console or CLI.
Re-run the Disaster Recovery restore on your target cluster:

./chef-360 restore --airgap-bundle chef-360.airgap --skip-store-validation
Verify that all pods deploy successfully and the Admin Console restores cleanly without timeouts.

Workaround

Notes

Keyword Phrase

Chef 360 restore failed, Chef 360 disaster recovery, kotsadm restore-db error 52, kotsadm-rqlite OOMKilled, curl 52 empty reply from server kotsadm, unable to wait for admin console timed out, kotsadm init crashloopbackoff, rqlite database size bloat, chef-360 disaster recovery restore timeout, replicated kots rqlite restore fail, exit code 137 kotsadm-rqlite

Last Modified Date

9/29/2026 8:04 AM