On the Automate UI from the infrastructure tab - when we go to client runs and if you click on a chef-client node, it says "There is no client-run data available for this node". 

1. Check "chef-automate status'" on the Automate Server via ssh to ensure that all of the services are operational.

2. Check the logs on the Automate server by using  journalctl -u chef-automate |grep -i "maximum shards"
If the errors look like the one listed below, then the system has reached its maximum number of shards.

error: “InsertInspecSummary: elastic: Error 400 (Bad Request): Validation Failed: 1: this action would add [10] total shards, but this cluster currently has [1195]/[1200] maximum shards open; [type=validation_exception]”

The reason for this is that when the data retention period is large for the amount of shards that the system is designed for, this limit is reached.

This doc https://docs.chef.io/automate/data_lifecycle/#impact-on-opensearch-shards explains what the total number of shards per node should be based on the TTL that is set for the combination of both client-run data and compliance data.

3. To address this problem, we must increase the maximum shards per node. Here we are considering TTL as 60 days, so we are increasing the total number of shards per node to 2000 in the configuration to patch as shown below

• Create a .toml file with the following config parameters:

  [opensearch.v1.sys.cluster]
  max_shards_per_node = 2000

•  In order for the above config to be applied to the Automate system this has to be patched as shown below:

  chef-automate config patch </path/to/your-file.toml>

For example, if my .toml file is named shards.toml then the patch command will be like this:
 

  chef-automate config patch shards.toml

Note: It will take some time to update the deployment configuration and apply the configuration once issue the patch command. Once it is deployed, you should see on the console that the Configuration patched successfully. While applying the patch, you may notice that a few dependent services are restarted.

Executing "chef-automate config show" on the Automate server console will show us the new value that we patched for max_shards_per_node.