The Chef‑360 Hyperscale HA deployment failed during the Helm pre‑install phase. The deployment logs showed that the secret-key-generator-job in the common-cert-manager component timed out after 3600 seconds. Two out of three SeaweedFS volume pods failed to initialize properly, resulting in repeated restarts and blocking dependent services from starting. Only one controller node was able to provision storage successfully, leaving the cluster in a partially initialized state.

The issue occurred in an air‑gapped environment where all controller, frontend, and backend nodes had been successfully onboarded and were visible in the UI prior to the failure.

Analysis of the support bundles and live troubleshooting session identified the root cause as a failure in the SeaweedFS storage layer, which backs the internal container registry and other storage‑dependent components.

Detailed findings included:

• SeaweedFS volume servers were not synchronizing correctly across the three volume pods.
• One volume server had no valid data, while the others contained inconsistent or unsynchronized volumes. 
• This led to:
  • PVC binding failures and storage unavailability. 
  • Inability of some pods (including pre‑install/security hooks) to pull images, resulting in ImagePullBackOff states. 
  • A partially initialized cluster where only one controller node could complete storage provisioning.

Network connectivity, Kubernetes resources, and OS/firewall parameters were validated and found to be healthy. The issue was isolated to SeaweedFS volume initialization and replication behavior, likely due to a corrupted or incomplete initial state of the SeaweedFS volumes.

A joint troubleshooting and remediation session was conducted, during which the following steps were taken:

1. Data Collection and Analysis

   • Collected fresh support bundles to confirm SeaweedFS volume and PVC behavior.
   • Verified that the issue was isolated to the storage layer.

2. Cluster Reset and Cleanup

   • Performed a full cleanup of the partially failed deployment. 
   • Reset the cluster to remove any corrupted or incomplete SeaweedFS state. 

3. Re‑Deployment

   • Re‑deployed all controller nodes and re‑enabled HA once all three controllers were operational. 
   • Sequentially re‑deployed backend and frontend nodes. 

4. Storage Validation

   • Confirmed that all SeaweedFS volumes were healthy and synchronized across all three volume servers. 
   • Verified that PVCs were created and bound successfully. 
   • Ensured no pods were stuck in BackOff or ImagePullBackOff states. 

5. Post‑Deployment Validation

   • Confirmed that Chef‑360 services were running and the application was accessible and healthy. 
   • Verified that the Hyperscale HA cluster completed successfully and was stable. 

After these steps, the deployment completed successfully, and the environment was confirmed stable. 

N/A