Elasticsearch/Opensearch Overview

Elasticsearch (ES) and Opensearch (OS) are the search engines used in Chef Infra and Chef Automate products.

All communication with ES/OS is done via their respective API endpoints, which are provided by ES/OS. Customers can get information on the data held in ES/OS and ES/OS itself by running queries against these endpoints. Examples of this can be found in the "Clarifying Information" section.
 

Chef Product	Endpoint Port	Full Endpoint Base Address
Chef Infra Server	9200	localhost:9200/*
Chef Automate	10144	localhost:10144/*

To clarify the above figure, see the following two notes:

1. As Opensearch replaces Elasticsearch, the endpoint port stays the same no matter the product version.
2. Chef Infra Server's ES/OS, when Chef Infra Server is embedded within Chef Automate, must be accessed by the Chef Automate endpoint.

Elasticsearch/Opensearch Concepts

Documents

Documents are JSON objects that are stored within an Elasticsearch index and are considered the base unit of storage. In the world of relational databases, documents can be compared to a row in a table.

Shards

Shards are segments of data that are created and managed by ES/OS. Shards contain WHOLE documents. A document will not be split across multiple shards.

Index (Indices)

An index is the collection of data within ES/OS. Drawing a comparison across to traditional Relational Databases, an Index is like a single Database.
Shards to Indices have a many-to-one relationship. An Index may have one or more shards to store its documents.

Nodes

This is an individual machine that can run ES/OS and hold data in shards.

More Details on Shards

Primary Shards

These are the original or base set of shards for an index.

Replica Shards

These are copies of the primary shards. If the primary shards are lost from the cluster, then one of the replicas is promoted to be the new primary.

Shard Calculation

If you have 2 primary shards and 2 replicas there will be 6 shards, the 2 originals, and the replica of these on 2 other nodes – 2original + 2replicas of those 2original. So if there were 10 original shards and a 3 node cluster with 2 replicas, there would be 30 shards on this cluster.

 

Helpful Elasticsearch/Opensearch Queries

Make sure you switch out port 10144 and 9200 as appropriate. For more details, see the table above.

SINGLE NODE ONLY! Understand unassigned shards and yellow status

In a default single node Automate install, the default status of Elasticsearch/Opensearch will be yellow because of unassigned replica shards. This condition is not ideal and produces no benefit.

When we ask for more replicas than there are nodes available, then these replicas must be in unassigned status, they do not exist and do not hold any data. Having them present in the system produces no benefit. Removing them will cause no problems.

These can be seen with

curl -s -XGET 'localhost:10144/_cat/shards/?v'

If a customer wants to get rid of the yellow status in a single node installation, they may set the number of replicas to auto, like this. (Do not do this on a multi-node installation like A2HA Automate or Automate HA):

curl -s -XPUT 'localhost:10144/_all/_settings' -H 'Content-Type: application/json' -d' { "index.auto_expand_replicas": "0-all" } '

 

See System Health

curl -s -XGET 'localhost:10144/_cat/health'

Various parts of the system’s state can be either green, yellow, or red and this state exists at three levels: shard, index, and cluster.

• green: all shards are allocated
• yellow: all primaries are allocated, but at least one replica is not
• red: at least one primary shard is not allocated in the cluster

In a single node system (a standard Automate or Infra standalone deployment), a yellow system state is not a problem. As there is only a single node, there is no node to assign replicas to and, hence, they will stay unassigned.

List Indices

curl -s -XGET 'localhost:10144/_cat/indices?v'

or, if you would just like an ordered list...

curl -s -XGET 'localhost:10144/_cat/indices?v' | tail -n +2 | awk '{print $3}' | sort

List Shards

curl -s -XGET 'localhost:10144/_cat/shards/?v'

Get Count of Documents in Index

curl -s -XGET 'localhost:10144/index-name/_search?size=0'

Ensure you replace <index-name> with the name of an index. You can list indices using the command underneath "List Indices".