Failure of Chef tools and Automate internal searches

The knife status command and all other searches as implemented today in Automate2/Chef Backend HA/Chef Infra Server will fail after 10k returned hits. Nodes are the object that is most likely to hit this limit. The limit can be changed, at the cost of a less performant overall system. When the limit is being hit, you will see output like this from Elasticsearch in the logs

[2021-09-04T03:26:42,726][DEBUG][o.e.a.s.TransportSearchAction] [somenode.com] All shards failed for phase: [query]
org.elasticsearch.search.query.QueryPhaseExecutionException: Result window is too large, from + size must be less than or equal to: [10000] but was [10479]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting.

See Resolution section for more detail

Change the max window size. This is required to work around the problem for both Elasticsearch and Opensearch and will fix the problem for systems running Elasticsearch

To increase the returned hits limit to 50k, you would normally do the following on any of the backend nodes. You will only need to run the command on one node and it will affect the whole cluster. Replace the USERNAME and PASSWORD below with the results from following this article: https://supportlink.chef.io/s/article/curl-commands-return-Unauthorized-on-chef-server

curl -u USERNAME:PASSWORD -XPUT 'localhost:9200/_settings?pretty' -H 'Content-Type: application/json' -d'
{
   "index" : {
   "max_result_window" : 50000
   }
}
'

The Bug, when running Opensearch

However, when dealing with the Chef Server's index, /chef and when running Opensearch...

The clients, knife search, tidy, status, count, and possibly others expect to receive results containing output compatible with Elasticsearch 6.8.23.

A couple possible ways to achieve this outcome in Automate or standalone Chef Server, neither of which have been implemented:

• In the code of the clients listed above, add &track_total_hits=true to all outgoing request URLs
• In the Chef Servers, whether standalone or Automate-installed, cause the erchef process to do the same as above when communicating with Opensearch on the client's behalf. This could potentially be done in nginx as well

Elastic's discussion of the general problem: https://discuss.elastic.co/t/how-to-increase-the-default-size-limit-from-10000-to-1000000-in-elasticsearch/208807
 

Bug Fix ETA

No known ETA for a fix

Known internally as CHEF-15395

Bug description: https://chef-software.ideas.aha.io/ideas/CHEFA-I-927

Notes:

The following Progress Chef systems are expected to work properly:

• Chef Backend HA servers through 3.1.1 where the backend servers are managed by Chef where Elasticsearch is installed are expected to work. This is the only supported configuration

Progress Chef Server systems installed after the following levels will experience the bug listed in this article

• Automate-installed Chef Servers as of 4.0.27 https://docs.chef.io/release_notes_automate/#4.0.27
• Automate HA containing OpenSearch as the search backend
• Standalone Chef Servers as of 15.0.9 https://docs.chef.io/release_notes_server/#15.0.9