A Chef 360 deployment fails to complete, with the RabbitMQ pod continuously restarting. Dependent services (including license-proxy and other core components) also fail to start, and the platform reports a large number of errors across unrelated components in the Troubleshoot tab.

This was verified on an implicit/embedded (k0s) non-HA Chef 360 install, but the underlying mechanism, RabbitMQ persisting credentials on its data volume independently of the Kubernetes secret, is a general RabbitMQ/StatefulSet behaviour, not specific to one install type. The same diagnostic approach should apply to BYOK and HA installs as well, though this has not been separately verified on those configurations.

RabbitMQ only applies the password from its Kubernetes secret when it initialises against a fresh data volume. If the RabbitMQ pod's persisted volume already contains data from a previous (failed) install attempt, RabbitMQ instead restores that existing internal database on every restart, including whatever credentials were baked in during that first attempt. This happens regardless of what the current secret holds, so credentials can appear correct in the secret while authentication still fails against the running pod.

This is confirmed by the following line in the RabbitMQ pod's own startup log:

Persisted data detected. Restoring...

Step 1: Rule out a missing or empty secret.

kubectl get secret rmq-password-and-cookie -n chef-360 -o jsonpath='{.data.rabbitmq-password}'

If this returns empty or the secret does not exist, that's a different (simpler) issue: the secret itself failed to regenerate, and both the secret and the persisted volume should be cleared before redeploying. If it returns a value, proceed to Step 2, an existing secret value does not confirm the credentials are actually being used by the running pod.

Step 2: Confirm the persisted-volume mismatch directly.

PASS=$(kubectl get secret rmq-password-and-cookie -n chef-360 -o jsonpath='{.data.rabbitmq-password}' | base64 -d)
kubectl exec -n chef-360 chef-360-rabbitmq-0 -- curl -s -u "user:$PASS" http://localhost:15672/api/whoami

An access-denied or unauthorised response confirms the pod is authenticating against stale, persisted credentials rather than the current secret.

Step 3: Clear the persisted volume only (leave the secret intact).

kubectl scale statefulset chef-360-rabbitmq -n chef-360 --replicas=0
kubectl delete pvc data-chef-360-rabbitmq-0 -n chef-360

Step 4: Trigger a redeploy of the current app version (e.g. via the KOTS admin console). This brings RabbitMQ back up against a fresh, empty volume, so it correctly applies the current secret's credentials on initialisation.

Step 5: Verify.

kubectl get secret rmq-password-and-cookie -n chef-360 -o jsonpath='{.data.rabbitmq-password}'

Re-run the Step 2  whoami  check against the new pod to confirm authentication now succeeds.

Validation / Outcome

After clearing the persisted volume and redeploying, RabbitMQ authenticated successfully against the current secret, and the wider deployment completed without further RabbitMQ-related errors.

Deleting the PVC discards any messages currently queued in RabbitMQ. If the install never completed successfully, there is typically nothing meaningful in that queue, but this should be confirmed before proceeding on any install that had reached a further stage of completion.

This scenario is most likely to occur when an install is retried or redeployed into a namespace that already had a prior failed attempt. Where possible, deploying into a clean namespace avoids this class of issue entirely.