How to Validate a Certificate Chain Before Applying Certificates to Chef Automate, Chef Automate HA, and Chef Infra Server

Title

URL Name

How-to-Validate-a-Certificate-Chain-Before-Applying-Certificates-to-Chef-Automate-Chef-Automate-HA-and-Chef-Infra-Server

Article Number

000301157

Environment

- Products: Chef Automate, Chef Automate HA, Chef Infra Server - Operating Systems: Linux , Windows - Tools: OpenSSL 3.2.3 - Cert type: RSA certificates signed by a private/internal CA or a public CA

Question/Problem Description

After providing a new or rotated certificate bundle to a Chef system (via `chef-automate cert-rotate`, or config patching using `chef-server.rb`, or similar), one or more of the following symptoms appear:

- The service fails to start or restart after the certificate change, in particular nginx and erchef services might be down/unknow state.

- Browser or API clients report an **SSL handshake failure** or **certificate verification error**.

- Chef Infra Clients fail to connect to Chef Infra Server with a `SSL_connect` error or showing HTTP 500/502 errors.

- The `chef-automate` or `chef-server-ctl` command returns an error referencing certificate trust or chain validation.

- The service starts successfully but clients report that the certificate is **not trusted** or that the **issuer is unknown**

Steps to Reproduce

The following pattern reliably triggers certificate chain issues: 1. A new TLS certificate is obtained from an internal CA (or a public CA that uses an Intermediate CA). 2. Only the **server (leaf) certificate** file is provided to the Chef configuration — the Intermediate CA or root CA certificate is omitted from the bundle. 3. The Chef service is restarted with the new configuration. 4. Clients that do not already have the Intermediate CA/root CA in their trust store fail to build the chain and reject the connection. A second common pattern: 1. A full-chain file is assembled, but the certificates are placed in the **wrong order** (e.g., Root CA first instead of server certificate first). 2. Some TLS libraries reject an out-of-order chain even if all the individual certificates are valid.

Clarifying Information

IMPORTANT Disclaimer:

The information provided in this article is intended as general guidance to assist customers in understanding and validating TLS certificate chains prior to use with Chef products.
Chef Software and the Chef Support team are not responsible for the procurement, generation, validation, management, or lifecycle of customer TLS certificates, private keys, or certificate chains.
The responsibility for ensuring that all certificates, private keys, and certificate chain files are valid, correctly assembled, unexpired, and appropriate for the intended environment rests solely with the customer and their designated system administrators.
Chef Support may assist in identifying symptoms consistent with certificate misconfiguration and can provide general guidance on the expected certificate format and structure required by Chef products. However, Chef Support does not validate, sign, or certify customer certificates, and cannot be held responsible for any service disruption, security exposure, or data unavailability arising from the use of invalid, expired, incomplete, or incorrectly configured certificates.
Customers are strongly advised to engage their internal PKI team, certificate authority, or a qualified security professional to review and approve all certificates before applying them to any production system.

Understanding the Certificate Chain:

A TLS certificate chain (also called a certificate path or chain of trust) is a sequence of certificates that links a server's certificate back to a trusted Root Certificate Authority (CA). There are three roles:

Role	Also Called	Purpose
Root CA Certificate	Trust Anchor	Self-signed; explicitly trusted by the OS/browser trust store. Signs the Intermediate CA.
Intermediate CA Certificate	Chain Certificate / Bundle	Signed by the Root CA. Signs the server (leaf) certificate. Bridges the trust gap.
Server Certificate	Leaf Certificate / End-Entity Cert	Issued to the specific hostname (e.g.: *.example.com or automate.example.com). Presented by the server during TLS handshake.

The trust relationship flows in one direction:

Root CA -- signs --> Intermediate CA -- signs --> Server (Leaf) Certificate

During a TLS handshake, the client works **up** the chain:

1. It receives the server's leaf certificate.

2. It checks whether the leaf certificate was signed by a known CA.

3. If the signer is an Intermediate CA (not directly in the trust store), the client looks for the Intermediate CA certificate — either supplied by the server or already in the client's own trust store.

4. It then checks whether the Intermediate CA was signed by a Root CA that **is** in the trust store.

5. If the full path is verified, the handshake succeeds; if any link is missing or invalid, it fails.

What a Correct Full-Chain File Looks Like:

A correctly assembled full-chain PEM file contains the certificates concatenated in this order:

IMPORTANT NOTE: Some Chef configuration options (particularly `chef-automate cert-rotate`) expect the full chain in this order. Supplying only the leaf certificate, or reversing the order, will cause handshake failures for clients that do not already trust the Intermediate CA.

Key Fields to Check on Each Certificate:

Field	What to Verify
Subject:	The identity the certificate was issued to.
Issuer:	Must match the `Subject` of the signing certificate (the CA one level up).
Validity (Not Before / Not After):	Certificate must be currently valid; check against today's date.
Basic Constraints: CA:TRUE:	Must be present on Root and Intermediate CA certs; must be `CA:FALSE` on the leaf.
Key Usage:	Intermediate CA must include `Certificate Sign, CRL Sign`. Leaf must include `Digital Signature, Key Encipherment`.
Extended Key Usage:	Leaf cert must include `TLS Web Server Authentication` for HTTPS use.
Subject Alternative Name (SAN):	Leaf cert must list the hostname(s) the Chef system will be accessed by. Modern TLS libraries ignore `CN` and require a SAN.
Authority Key Identifier:	Must match the `Subject Key Identifier` of the signing CA — links the chain cryptographically.

Error Message

- Unable to get local issuer certificate. - OpenSSL::SSL::SSLError: certificate verify failed. - Fatal - Unknown CA. - SSL_connect errors. - upstream timed out; no live upstreams. - Chef Client run may fail with 500/502 Bad Gateway error.

Defect Number

Enhancement Number

Cause

The underlying cause is almost always one or more of the following:

Incomplete chain - The Intermediate CA certificate is missing from the bundle presented to the Chef service or to connecting clients.
Wrong order in the chain file - Certificates are concatenated in the wrong sequence.
Mismatched issuer/subject - A certificate in the chain was replaced or rotated but its corresponding CA certificate was not updated, breaking the `Issuer ↔ Subject` link.
Expired certificate - One or more certificates in the chain have passed their `Not After` date.
Private key / certificate mismatch - The private key on disk does not correspond to the certificate being deployed (the public key modulus values differ).
Root CA not trusted by clients - The Root CA certificate is not present in the OS or application trust store on Chef client nodes, so even a valid chain cannot be verified.
SAN missing the required hostname - The server certificate's Subject Alternative Name does not include the FQDN clients use to reach the Chef server.

Resolution

Follow these steps **before** applying certificates to any Chef system.

All commands use OpenSSL on the command line.

Substitute your own file paths where the example uses `rootCA/rootCA.crt`, `intermediateCA/intermediateCA.crt`, `server/server.crt`, `server/server.key`, and `fullchain.crt`.

Before proceeding verify that openssl is installed and what version is being using by running the command:

openssl version

# Expected output (version may differ):
OpenSSL 3.2.3 3 Sep 2024 (Library: OpenSSL 3.2.3 3 Sep 2024)

Step 1 — Inspect Each Certificate:

For a quick summary (subject, issuer, and validity dates only).

Check RootCA Certificate:

openssl x509 -in rootCA/rootCA.crt -noout -subject -issuer -startdate -enddate

Expected Output (Root CA — self-signed (subject == issuer), valid 10 years):

subject=C=GG, ST=Coruscant, L=Imperial City, O=Galactic Empire, OU=Imperial Security Bureau,
        CN=Galactic Empire Root CA, emailAddress=emperor@galacticempire.com
issuer= C=GG, ST=Coruscant, L=Imperial City, O=Galactic Empire, OU=Imperial Security Bureau,
        CN=Galactic Empire Root CA, emailAddress=emperor@galacticempire.com
notBefore=May 13 14:27:23 2026 GMT
notAfter= May 10 14:27:23 2036 GMT

Intermediate CA Certificate (Intermediate CA — signed by Root CA (issuer == Root CA subject), valid 5 years):

openssl x509 -in intermediateCA/intermediateCA.crt -noout -subject -issuer -startdate -enddate

Expected Output:

subject=C=GG, ST=Coruscant, L=Death Star, O=Galactic Empire, OU=Imperial Intelligence,
        CN=Death Star Intermediate CA, emailAddress=darth.vader@galacticempire.com
issuer= C=GG, ST=Coruscant, L=Imperial City, O=Galactic Empire, OU=Imperial Security Bureau,
        CN=Galactic Empire Root CA, emailAddress=emperor@galacticempire.com
notBefore=May 13 14:27:26 2026 GMT
notAfter= May 12 14:27:26 2031 GMT

Server Leaf Certificate (Server (leaf) cert — signed by Intermediate CA (issuer == Intermediate CA subject))

openssl x509 -in server/server.crt -noout -subject -issuer -startdate -enddate

Expected Output:

subject=C=GG, ST=Tatooine, L=Mos Eisley, O=Rebel Alliance, OU=Rogue Squadron,
        CN=*.galacticempire.com, emailAddress=luke.skywalker@galacticempire.com
issuer= C=GG, ST=Coruscant, L=Death Star, O=Galactic Empire, OU=Imperial Intelligence,
        CN=Death Star Intermediate CA, emailAddress=darth.vader@galacticempire.com
notBefore=May 13 14:27:26 2026 GMT
notAfter= Aug 15 14:27:26 2028 GMT

You can also view the full details for each certificate using the following command:

openssl x509 -in rootCA/rootCA.crt          -text -noout
openssl x509 -in intermediateCA/intermediateCA.crt -text -noout
openssl x509 -in server/server.crt          -text -noout

IMPORTANT NOTES:

Confirm the `issuer` of the leaf cert matches the `subject` of the Intermediate CA cert.
Confirm the `issuer` of the Intermediate CA cert matches the `subject` of the Root CA cert.
Confirm the Root CA's `subject` and `issuer` are identical (self-signed).
Confirm all certificates are currently within their validity window.
Confirm the leaf cert's `CN` / SAN (`X509v3 Subject Alternative Name`) includes the hostname your Chef systems will use.

Step 2 — Verify the Intermediate CA Was Signed by the Root CA:

openssl verify -CAfile rootCA/rootCA.crt intermediateCA/intermediateCA.crt

Expected Output:

intermediateCA/intermediateCA.crt: OK

NOTE: Any output other than `OK` indicates a broken link between the Root CA and Intermediate CA. Do **not** proceed — obtain the correct certificates before continuing.

Step 3 — Verify the Server Leaf Certificate Was Signed by the Intermediate CA:

openssl verify -CAfile rootCA/rootCA.crt -untrusted intermediateCA/intermediateCA.crt server/server.crt

Expected Output:

server/server.crt: OK

NOTE: -untrusted`:** This flag passes the Intermediate CA as a "helper" (untrusted) certificate. OpenSSL uses it to build the chain path and traces trust back to the Root CA specified by `-CAfile`. The intermediate is verified as part of the process; it is not blindly trusted.

Step 4 — Validate the Full Chain File:

4a. Verify the server certificate against the full-chain bundle:

openssl verify -CAfile fullchain.crt server/server.crt

Expected Output:

server/server.crt: OK

4b. List all certificates in the full-chain file and confirm correct order and content:

openssl crl2pkcs7 -nocrl -certfile fullchain.crt | openssl pkcs7 -print_certs -noout

Expected Output (server cert first, then Intermediate CA, then Root CA):

subject=CN=*.galacticempire.com ...
issuer= CN=Death Star Intermediate CA ...

subject=CN=Death Star Intermediate CA ...
issuer= CN=Galactic Empire Root CA ...

subject=CN=Galactic Empire Root CA ...
issuer= CN=Galactic Empire Root CA ...

NOTE: If the order is different, or if any certificate is missing, reassemble the full-chain file in the correct order before proceeding.

Step 5 — Confirm the Private Key Matches the Server Certificate:

Both commands below must produce the **same MD5 hash**. A mismatch means the private key and certificate are not a pair and the TLS handshake will fail immediately.

openssl x509 -noout -modulus -in server/server.crt | openssl md5
openssl rsa  -noout -modulus -in server/server.key | openssl md5

Expected Output (hashes must be identical):

MD5(stdin)= 58ee276d1843adfeb82bc844066179ce
MD5(stdin)= 58ee276d1843adfeb82bc844066179ce

NOTE: If the hashes differ, you have the wrong private key. Source the correct key that was generated alongside the certificate before continuing.

Only once all five steps above pass without errors, should you proceed to actually update you certificate on Chef Automate, Chef Automate HA and/or Chef Infra Server.

If any of the above checks fail you should stop and ensure that you have the correct certificate chain and private keys before proceeding with any certificate changes on the respective chef system.

KEY NOTES:

Order of certificates in a PEM bundle matters: Always place the server (leaf) certificate first, the Intermediate CA second, and the Root CA last.
The Root CA must be trusted by all clients: If you are using an internal/private CA, distribute the Root CA certificate to all Chef Infra Client nodes (add it to the OS trust store or reference it in the `knife.rb` / `client.rb` using `ssl_ca_file` or `trusted_certs_dir`).
SAN is mandatory: Modern TLS libraries do not fall back to checking the `CN` field for hostname verification. The server certificate **must** have a `X509v3 Subject Alternative Name` that includes every hostname and IP address clients will use to reach the Chef system.
Wildcard certificates: A wildcard like `*.example.com` covers exactly one subdomain level (e.g., `automate.example.com`) but does **not** cover the apex domain (`example.com`) or multi-level subdomains (`a.b.example.com`) unless they are explicitly listed as additional SANs.
Validity window: Ensure the `Not After` date of the leaf certificate is sufficiently far in the future. It is common to deploy a certificate and only discover it expires in days rather than months because the wrong certificate was packaged.
Key size: Chef Automate and Chef Infra Server work with both 2048-bit and 4096-bit RSA keys. 2048-bit is the minimum recommended size; 4096-bit provides a higher security margin at a minor performance cost.

Workaround

Notes

The dummy certificates and their private keys that were used for the creation of this KB article are provided below for testing purposes:

Server Leaf certificate (Wildcard):

Server Leaf Certificate private key:

Intermediate Certificate:

Intermediate Certificate Private key:

Root CA certificate:

Root CA certificate Private key:

Full Certificate Chain:

Keyword Phrase

- certificate chain validation - certificate chain verification - chef automate certificate rotation - chef automate cert-rotate - chef automate HA certificate - chef infra server SSL certificate - unable to get local issuer certificate - SSL handshake failure chef - certificate signed by unknown authority - incomplete certificate chain - fullchain.crt order - intermediate CA missing - openssl verify certificate chain - private key certificate mismatch - openssl modulus md5 check - SAN subject alternative name chef - TLS certificate rotation chef - chef SSL certificate update - certificate trust chain - root CA intermediate CA leaf certificate

Last Modified Date

6/25/2026 3:43 PM

How to Validate a Certificate Chain Before Applying Certificates to Chef Automate, Chef Automate HA, and Chef Infra Server

Role

Also Called

Purpose

Field

What to Verify

Files(0)

Was this article helpful?

Was this article helpful?

How to Validate a Certificate Chain Before Applying Certificates to Chef Automate, Chef Automate HA, and Chef Infra Server

Information

Role

Also Called

Purpose

Field

What to Verify

Related Articles