Chef Automate HA OpenSearch degraded cluster after certificate rotation

Feb 2, 2026
Knowledge

Title

URL Name

Chef-Automate-HA-OpenSearch-degraded-cluster-after-certificate-rotation

Article Number

000297495

Environment

Chef Automate High Availability (HA) with Chef-Automate/HAB managed OpenSearch

Question/Problem Description

After a certificate rotation in a Chef Automate High Availability (HA) deployment, OpenSearch nodes may fail to form a healthy cluster even though individual services appear to start correctly. This condition typically presents as cluster quorum failure and downstream Automate service degradation.

The most common cause is a mismatch between the Distinguished Names (DNs) contained in newly issued certificates and the DN allow-lists configured in the OpenSearch Security plugin.

Symptoms / Impact

OpenSearch fails to form quorum
No master node is elected
Automate services dependent on OpenSearch degrade or fail
Repeated transport-layer handshake errors on port 9300
REST APIs may return misleading "security not initialized" errors

Cause

Certificate rotation in Chef Automate HA is usually performed as an in-place renewal of expiring certificates using the same certificate authority (CA) and subject template. In this case, the Subject DN does not change and OpenSearch node trust continues to function without additional configuration changes.

This issue occurs when rotation involves a CA or certificate template change, such as:

Moving from a public CA to an internal CA
Switching between internal CAs
Modifying or standardising certificate subject templates

In these scenarios, newly issued certificates may contain a different Subject DN, even though the hostname, key usages, and trust chain remain valid.

OpenSearch performs identity-based authorisation using explicit DN allow-lists. If the Subject DN presented by a node does not exactly match the configured allow-list entries, the node is rejected at the transport layer.

Steps to Reproduce

Clarifying Information

OpenSearch uses TLS in two distinct contexts:

Transport layer (node-to-node, port 9300)

Governs cluster membership and quorum. DN allow-lists are enforced at this layer.

REST layer (client-to-node, port 9200)

Governs API access. REST endpoints may appear functional even when transport-layer trust is failing.

DN matching is exact. Any difference—such as added attributes( L= ), reordered fields, or formatting changes—prevents a match.

Error Message

Common log signatures include: Not yet initialized (you may need to run securityadmin) Illegal parameter in http or transport request found … non-node certificate (no OID or security.nodes_dn incorrect configured) handshake failed for connectToRemoteMasterNode[…:9300] master not discovered or elected yet (often repeated during cluster formation attempts) Transport SSL handshake or peer authentication failures on port 9300

Defect Number

Enhancement Number

Cause

This issue occurs when rotation involves a CA or certificate template change, such as:

Moving from a public CA to an internal CA
Switching between internal CAs
Modifying or standardising certificate subject templates

In these scenarios, newly issued certificates may contain a different Subject DN, even though the hostname, key usages, and trust chain remain valid.

Resolution

Preparation

Do not manually edit opensearch.yml files on individual nodes
Perform all OpenSearch configuration changes using Chef Automate HA configuration patching from the bastion node
Confirm the canonical Subject DN of the rotated certificates before applying configuration changes

Step 1 — Confirm the certificate Subject DN

On an OpenSearch node, inspect the certificate Subject DN in canonical form:

openssl x509 -subject -nameopt RFC2253 -noout \
  -in /hab/svc/automate-ha-opensearch/config/certificates/node1.pem

Step 2 — Compare against active OpenSearch configuration

grep "plugins.security.nodes_dn" /hab/svc/automate-ha-opensearch/config/opensearch.yml
grep "plugins.security.authcz.admin_dn" /hab/svc/automate-ha-opensearch/config/opensearch.yml

Any discrepancy confirms a DN allow-list mismatch.

Step 3 — Apply a supported OpenSearch configuration patch

Create a patch file on the bastion node, for example os-dn-fix.toml:

[plugins.security]
nodes_dn = "- CN=chef-automate.example.com,O=Example Services Ltd.,L=New York,ST=New York,C=US"

[plugins.security.authcz]
admin_dn = "- CN=osadmin-chef-automate.example.com,O=Example Services Ltd.,L=New York,ST=New York,C=US"

Apply the patch:

chef-automate config patch os-dn-fix.toml --os

Step 4 — Verify recovery

Transport handshake errors no longer appear in logs

Nodes successfully discover each other
The cluster reaches quorum and elects a master node

Workaround

Notes

Certificate renewal using the same CA and subject template does not require OpenSearch configuration changes
DN mismatches alone are sufficient to prevent cluster formation, even when certificates are otherwise valid
Errors suggesting securityadmin remediation are typically secondary symptoms of transport-layer trust failure