Chef Automate HA certificate locations, verification and troubleshooting for on premise setup - Infra servers

For Chef Automate HA installation, the “Front End” load balancer is a critical pre-requisite. This article discusses the scenario of SSL termination on load balancer, which has 2 use cases of how certificates may be installed for consumption by the Chef Automate HA. This article focuses on the certificates of the Infra server frontend nodes only, same steps can be replicated for Automate frontends.

Jun 13, 2024
Knowledge

Title

URL Name

AutomateHA-certificate-locations-verificiation-and-troubleshooting-for-on-premise-setup

Article Number

000242299

Environment

Chef Automate 4.10.1 - On Premise set up

Question/Problem Description

How do I verify certificates for my Chef Automate cluster?

Steps to Reproduce

Clarifying Information

Error Message

Defect Number

Enhancement Number

Cause

Resolution

During Chef Automate HA installation, for the scenario of SSL termination on load balancer, there can be following 2 use cases.

1. Supply certificates (signed public internet based root CA) for the URL of the domain name for both Infra and Automate servers (assumes unique domain names for each, eg: (automate01.example.com, infra01.example.com ), remaining node certificates auto generated by AutomateHA.
2. Supply certificates (self signed - private root CA) for the URL of the domain name for both Infra and Automate servers (assumes unique domain names for each, eg (automate01.example.com, infra01.example.com ), remaining node certificates auto generated by AutomateHA.

The load balancer here is a customer load balancer - typically this will be an F5 device, or an HAProxy, which sits in front of the automate cluster on a customer network.

It is crucial that you validate the certificates during installation for your Automate HA instance to function as expected. Follow the below steps for the second use case: self signed - private root CA.

The architecture referenced here is shown below. This article focuses on the certificates of the Infra server frontend nodes only, same steps can be replicated for Automate frontends.

Arch

Login to the bastion host.

Check that you can reach the infra web server on port 443, (you check all x3 infra nodes, but normally just one is fine).

curl https://infra01.example.com

This will normally result in this output:

 curl: (60) SSL certificate problem: self-signed certificate in certificate chain

Add the -k option in curl to ignore the certificate chain issues (ie curl will not be able to check the certificate chain, but we do not need it to for basic level debug). More details here: https://curl.se/docs/sslcerts.html

curl -k https://infra01.example.com

Now start with verifying all the certificates by checking certificates on each end point (Load balancer, Infra nodes and Automate nodes.)

Customer supplied certificates (load balancer installed) will have the following 3 parts: Server Cert, Intermediate Cert and Root Cert
Chef AutomateHA auto generated certificates obtained after deployment of AutomateHA:

Find the cert locations by editing the following file and search for the certificates section
/hab/svc/automate-load-balancer/config/nginx.conf
It should be similar the following

server_name infra01.example.com; ssl_certificate /hab/svc/automate-load-balancer/data/infra.example.com.cert;

Each Infra server (service load balancer in effect) does not supply the certificate chain, it simply supplies the server certificate in response to an ssl connection. Each of infra01, infra02 will have a unique server certificate. Each web server has a unique certificate.
LoadBalancer certificate:

Run the following command:

openssl s_client -connect infra01.example.com:443 -showcerts -CApath ~/certs

Part of the sample output from above command:

---
Certificate chain
  0 s:C = FI, ST = Uusimaa, L = Vantaa, O = Progress, emailAddress =
copyright@Progress.com, CN = loadbalancer02.pri.vat.cld
    i:C = FI, ST = Uusimaa, L = Vantaa, O = Progress, emailAddress =
copyright@Progress.com, CN = internal-ca
    a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
    v:NotBefore: Aug 23 09:58:22 2023 GMT; NotAfter: Jan 13 09:58:22
2032 GMT
-----BEGIN CERTIFICATE-----
MIIFFDCCA/ygAwIBAgIBBDANBgkqhkiG9w0BAQsFADCBhDELMAkGA1UEBhMCRkkx
..
..
DzX7YbVUS28W+T5R6oUzZaPcTuyzZ+r4J0H8AlzyOGMldVyS2UZHeZ/RX9tPgcSu
AJJ3Ig6gry8=
-----END CERTIFICATE-----
  1 s:C = FI, ST = Uusimaa, L = Vantaa, O = Progress, emailAddress =
copyright@Progress.com, CN = internal-ca
    i:C = FI, ST = Uusimaa, L = Vantaa, O = Progress, emailAddress =
copyright@Progress.com, CN = internal-ca
    a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
    v:NotBefore: Aug 15 15:26:31 2023 GMT; NotAfter: Jan  5 15:26:31
2032 GMT
-----BEGIN CERTIFICATE-----
MIIEIzCCAwugAwIBAgIBATANBgkqhkiG9w0BAQsFADCBhDELMAkGA1UEBhMCRkkx
..
..
qm68mRkpQLTKBgywOEz4I2LsIUT6DVpoV7lR4lZRTDsRNzn6lavtfb+xVEpUgHvO
/bt544k8Uw==
-----END CERTIFICATE-----
  2 s:C = FI, ST = Uusimaa, L = Vantaa, O = Progress, emailAddress =
copyright@Progress.com, CN = internal-ca
    i:C = FI, ST = Uusimaa, L = Vantaa, O = Progress, emailAddress =
copyright@Progress.com, CN = internal-ca
    a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
    v:NotBefore: Aug 15 15:23:47 2023 GMT; NotAfter: Aug 12 15:23:47
2033 GMT
-----BEGIN CERTIFICATE-----
MIIEIzCCAwugAwIBAgIBADANBgkqhkiG9w0BAQsFADCBhDELMAkGA1UEBhMCRkkx
..
..
00ziAcvfQfSKQdaXDZU5n7vGekydcN/Wlw6Sp48aXbL4NQLYgQ+hnjLtN+UzXAYP
aIaQ8hMlBw==
-----END CERTIFICATE-----
---
Server certificate
subject=C = FI, ST = Uusimaa, L = Vantaa, O = Progress, emailAddress =
copyright@Progress.com, CN = loadbalancer02.pri.vat.cld
issuer=C = FI, ST = Uusimaa, L = Vantaa, O = Progress, emailAddress =
copyright@Progress.com, CN = internal-ca
---
No client certificate CA names sent
-----
Verify return code: 0 (ok)

The important points to notice in the output from the above command are as follows:-

There are 3 certs, in order, Load balancer cert, IntermediateCA, RootCA (earlier referred as Server Cert, Intermediate Cert and Root Cert) - Check the last 8 digits from the output of the above command to confirm the certificates match what you expect, you can generally confirm the last 8 digits of the certificate match the expected certificate, as a "quick check".
The Verify return code: 0 (ok) indicates the certificate in the chain is trusted.
You can add the -CApath ~/cert (this assumes you have your IntermediateCA and rootCA in this directory), if you choose not to add your certificate chain to your haproxy(or other load balancer)

Infra server certificate:

Run the following command:

openssl s_client -connect infra01.example.com:443 -showcerts

This will show the certs coming from the loadbalancer itself(not the backend servers, infra01, infra02)
Typically this will return 3 or more certificates in order as below

Server certificate(eg infra01.example.com)
IntermediateCA certificate(1 or more, depending on how many intermediate certs are in the certificate chain)
RootCA certificate

You should validate that all x3 certificates are what you would expect:

openssl s_client -showcerts -servername infra01.example.com -connect loadbalancer.example.com

This assumes the DNS alias infra01.example.com is the -servername (ie front end URL)and the actual ip address which hosts the certificate is the -connect, which in this example is the hostname of the loadbalancer node http://loadbalancer.example.com(LoadBalancer in this case)

4. Validate Status:
Validate that you get a successful response from the backend server, by going via load balancer or direct to the backend servers. - via load balancer(infra01.example.com), infra01,02 servers.

curl -k https://infra01.example.com/_status

Sample output below:

{"status":"pong","upstreams":{"chef_opensearch":"pong","chef_sql":"pong","oc_chef_authz":"pong","data_collector":"pong"},"keygen":{"keys":10,"max":10,"max_workers":10,"cur_max_workers":10,"inflight":0,"avail_workers":10,"start_size":2},"indexing":{"mode":"batch"}}

Default behaviour would be to get the above response if you connect directly to infra01.example.com. If so, then the infra01 node is functional. If you cannot get this response from infra01.example.com(ie via loadbalancer), then there is something wrong with either the load balancer or the load balancer connection to the backends.

Please use this article as a reference document only and for further help, get in touch with Chef support.

Workaround

Notes

Keyword Phrase

Last Modified Date

6/13/2024 6:33 AM

Chef Automate HA certificate locations, verification and troubleshooting for on premise setup - Infra servers

Files(0)

Was this article helpful?

Was this article helpful?

Chef Automate HA certificate locations, verification and troubleshooting for on premise setup - Infra servers

Information

Related Articles