• Chef 360 Prod system was down.
• Unable to login into Admin console for Chef 360
• Getting Chef 360 Prod system was down unable to login into Admin console with error 502 Error
• Hundreds/Thousands of failed/Evicted Pods.

When var disk usage reaches 85% (the default imageGCHighThresholdPercent threshold), kubelet begins evicting pods that exceed the ephemeral storage soft limit. 
Once evictions start, the node receives the node.kubernetes.io/disk-pressure:NoSchedule taint, preventing any new pods from being scheduled 
— causing a self-reinforcing cycle where evicted pods are replaced by new ones that are immediately tainted and evicted again.

Once you have logged into your Chef 360 shell please following the guidance below to clean up the disk on your chef 360 instance to free up space from the /var folder and relieve the disk pressure on the system.

Step 1: Confirm Overall /var Disk Usage

df -h /var

Step 2: Identify What Is Consuming Space by drilling down into the contents of the /var folder.

# Start at var
sudo du -sh /var/* | sort -h

# Drill down further
sudo du -sh /var/lib/* | sort -h

# Keep drilling down into the largest folders
sudo du -sh /var/lib/embedded-cluster/k0s/kubelet/pods/* | sort -h

Example output — a single pod directory consuming significantly more space than others:

63M	/var/lib/embedded-cluster/k0s/kubelet/pods/444877a8-POD-UID-5d3b8b8d5381
254M	/var/lib/embedded-cluster/k0s/kubelet/pods/4bdc27fa-POD-UID-2bd7d62c0c03
976M	/var/lib/embedded-cluster/k0s/kubelet/pods/f5cc3533-POD-UID-df5547007ccc
134G	/var/lib/embedded-cluster/k0s/kubelet/pods/64342de9-POD-UID-2313b7a368bd

We can see in the output that the pod taking up the most space is the pod with UID: 64342de9-POD-UID-2313b7a368bd

Step 3: Identify the Pod using the UID that we found (64342de9-POD-UID-2313b7a368bd - 134GB usage)

kubectl get pods -A -o json | jq -r '.items[] | select(.metadata.uid=="<PASTE-UID-HERE>") | "\(.metadata.namespace)/\(.metadata.name)"'

EXAMPLE - Use the above command with the UID of the largest pod that we found:

kubectl get pods -A -o json | jq -r '.items[] | select(.metadata.uid=="64342de9-POD-UID-2313b7a368bd") | "\(.metadata.namespace)/\(.metadata.name)"'

Expected output - NAME of pod that owns the UID.

chef-360/chef-360-minio-8579ccb478-SOME_POD

(In most cases, the largest consumer will be the MinIO pod, which stores centralized logs.)

Step 4: Retrieve MinIO Credentials (username/password) using the following commands:

Since we know that the space is being taken up by the minIO pod which is holding 134GB of space we need to login to the pod to cleanup the logs.

# GET MinIO Username
kubectl -n chef-360 get secret chef-360-minio -o jsonpath='{.data.root-user}' | base64 -d; echo

# GET MinIO Password
kubectl -n chef-360 get secret chef-360-minio -o jsonpath='{.data.root-password}' | base64 -d; echo

Expected Output:

# MinIO Username
admin

# MinIO Password
123abc321

Step 5: Access the MinIO Pod by launching the MinIO interactive shell and Set an Alias for your session.

Once you have retrieved the MinIO credentials launch the MinIO interactive shell by running the following command:

kubectl -n chef-360 exec -it deploy/chef-360-minio -- bash

Once inside the interactive shell setup the alias for your current session using the credential that we just retrieved and the following command (replace "$ROOT_USER" and "$ROOT_PASSWORD" with the username and password we just got above):

mc alias set local http://localhost:9000 "$ROOT_USER" "$ROOT_PASSWORD"

Example:

mc alias set local http://localhost:9000 "admin" "123abc321"

(this will setup an alias called 'local' inside the MinIO shell so we don't have to pass credentials every time)

Step 6: Check MinIO Disk Usage

You can check what taking up the most space on the MinIO pod using the following command:

mc du local/ --depth 2

Expected Output:

18GiB	30201 objects	audit-logs
150MiB	77994 objects	b943a0e0-1234-abcd-1234-8b6d6788d31a
OB	0 objects	bookshelf
0B	0 objects	evidence
113GiB	388852 objects	general-logs

In this case we can see that the general-logs folder is taking up the most space (113GB):
https://docs.chef.io/360/1.7/centralized_log_storage/general_logs/

Step 7: Clean Up Logs

While still inside the MinIO shell delete logs older than your defined retention window (adjust as needed based on space required):

# Retain only the last 20 days
mc rm --recursive --force --older-than 20d local/general-logs

If additional space is still required, lower the retention threshold further:

# Retain only the last 10 days
mc rm --recursive --force --older-than 10d local/general-logs

Step 8: Exit the MinIO Shell:

Run the exit command to quit the MinIO shell and to go back to the Chef 360 Shell:

exit

Step 9: Identify and Clean Up any remaining Failed/Evicted Pods:

Once the disk space has been cleaned up check for pods in a failed state cluster-wide using the following command:

kubectl get pods -A | grep -Ei 'Error|CrashLoopBackOff|ImagePullBackOff|ErrImagePull|Evicted|ContainerStatusUnknown|Init:|Pending|Unknown'

Note the name of all namespaces with failed pods. 

Delete failed pods one namespace at a time using the following command (remember to replace <POD_NAMESPACE> with the actual namespace of the failed pods):

kubectl -n <POD_NAMESPACE> get pods --no-headers | awk '$3 ~ /Error|CrashLoopBackOff|ImagePullBackOff|ErrImagePull|Evicted|ContainerStatusUnknown|Init:|Pending|Unknown/ {print $1}' | xargs -r kubectl -n <POD_NAMESPACE> delete pod

Step 10: Validate Cleanup

Re-run the failed pod check command to confirm no failures remain:

kubectl get pods -A | grep -Ei 'Error|CrashLoopBackOff|ImagePullBackOff|ErrImagePull|Evicted|ContainerStatusUnknown|Init:|Pending|Unknown'

Confirm all pods are in a Running / Completed status:

kubectl get pods -A

Step 11: Confirm Recovery

Once all pods are healthy, verify that you can access and login to the Chef 360 UI.

Log Centralization - https://docs.chef.io/360/1.7/centralized_log_storage/
General-logs - https://docs.chef.io/360/1.7/centralized_log_storage/general_logs/
Audit-Logs - https://docs.chef.io/360/1.7/centralized_log_storage/audit_logs/
Log retention for general-logs/audit-logs for MinIO - https://docs.chef.io/360/1.7/get_started/install_server/#centralized-log-storage