How do I verify certificates for my Chef Automate cluster?

During Chef Automate HA installation, for the scenario of SSL termination on load balancer, there can be following 2 use cases. 

   1. Supply certificates (signed public internet based root CA) for the URL of the domain name for both Infra and Automate servers (assumes unique domain names for each, eg: (automate01.example.com, infra01.example.com ), remaining node certificates auto generated by AutomateHA.
    2. Supply certificates (self signed - private root CA) for the URL of the domain name for both Infra and Automate servers (assumes unique domain names for each, eg (automate01.example.com, infra01.example.com ), remaining node certificates auto generated by AutomateHA.

The load balancer here is a customer load balancer - typically this will be an F5 device, or an HAProxy, which sits in front of the automate cluster on a customer network.

It is crucial that you validate the certificates during installation for your Automate HA instance to function as expected. Follow the below steps for the second use case: self signed - private root CA.

The architecture referenced here is shown below. This article focuses on the certificates of the Automate server frontend nodes only, same steps can be replicated for Infra server frontends.


 

Login to the bastion host.

1. Check that you can reach the automate server on port 443, (you check all x3 Automate nodes, but normally just one is fine).

curl https://automate01.example.com

           This will normally result in this output:

 curl: (60) SSL certificate problem: self-signed certificate in certificate chain

2. Add the -k option in curl to ignore the certificate chain issues (ie curl will not be able to check the certificate chain, but we do not need it to for basic level debug). More details here: https://curl.se/docs/sslcerts.html

curl -k https://automate01.example.com

3. Now start with verifying all the certificates by checking certificates on each end point (Load balancer, Infra nodes and Automate nodes.)

• Customer supplied certificates (load balancer installed) will have the following 3 parts: Server Cert, Intermediate Cert and Root Cert
• Chef AutomateHA auto generated certificates obtained after deployment of AutomateHA:

Find the cert locations by editing the following file and search for the certificates section

/hab/svc/automate-load-balancer/config/nginx.conf

It should be similar the following

server_name automate01.example.com; ssl_certificate /hab/svc/automate-load-balancer/data/automate.example.com.cert;

• Each Automate server (service load balancer in effect) does not supply the certificate chain, it simply supplies the server certificate in response to an ssl connection. Each of automate01, automate02 will have a unique server certificate. Each web server has a unique certificate.
• LoadBalancer certificate:

Run the following command:

openssl s_client -connect automate01.example.com:443 -showcerts -CApath ~/certs

Part of the sample output from above command:

---
Certificate chain
  0 s:C = FI, ST = Uusimaa, L = Vantaa, O = Progress, emailAddress =
copyright@Progress.com, CN = loadbalancer02.pri.vat.cld
    i:C = FI, ST = Uusimaa, L = Vantaa, O = Progress, emailAddress =
copyright@Progress.com, CN = internal-ca
    a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
    v:NotBefore: Aug 23 09:58:22 2023 GMT; NotAfter: Jan 13 09:58:22
2032 GMT
-----BEGIN CERTIFICATE-----
MIIFFDCCA/ygAwIBAgIBBDANBgkqhkiG9w0BAQsFADCBhDELMAkGA1UEBhMCRkkx
..
..
DzX7YbVUS28W+T5R6oUzZaPcTuyzZ+r4J0H8AlzyOGMldVyS2UZHeZ/RX9tPgcSu
AJJ3Ig6gry8=
-----END CERTIFICATE-----
  1 s:C = FI, ST = Uusimaa, L = Vantaa, O = Progress, emailAddress =
copyright@Progress.com, CN = internal-ca
    i:C = FI, ST = Uusimaa, L = Vantaa, O = Progress, emailAddress =
copyright@Progress.com, CN = internal-ca
    a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
    v:NotBefore: Aug 15 15:26:31 2023 GMT; NotAfter: Jan  5 15:26:31
2032 GMT
-----BEGIN CERTIFICATE-----
MIIEIzCCAwugAwIBAgIBATANBgkqhkiG9w0BAQsFADCBhDELMAkGA1UEBhMCRkkx
..
..
qm68mRkpQLTKBgywOEz4I2LsIUT6DVpoV7lR4lZRTDsRNzn6lavtfb+xVEpUgHvO
/bt544k8Uw==
-----END CERTIFICATE-----
  2 s:C = FI, ST = Uusimaa, L = Vantaa, O = Progress, emailAddress =
copyright@Progress.com, CN = internal-ca
    i:C = FI, ST = Uusimaa, L = Vantaa, O = Progress, emailAddress =
copyright@Progress.com, CN = internal-ca
    a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
    v:NotBefore: Aug 15 15:23:47 2023 GMT; NotAfter: Aug 12 15:23:47
2033 GMT
-----BEGIN CERTIFICATE-----
MIIEIzCCAwugAwIBAgIBADANBgkqhkiG9w0BAQsFADCBhDELMAkGA1UEBhMCRkkx
..
..
00ziAcvfQfSKQdaXDZU5n7vGekydcN/Wlw6Sp48aXbL4NQLYgQ+hnjLtN+UzXAYP
aIaQ8hMlBw==
-----END CERTIFICATE-----
---
Server certificate
subject=C = FI, ST = Uusimaa, L = Vantaa, O = Progress, emailAddress =
copyright@Progress.com, CN = loadbalancer02.pri.vat.cld
issuer=C = FI, ST = Uusimaa, L = Vantaa, O = Progress, emailAddress =
copyright@Progress.com, CN = internal-ca
---
No client certificate CA names sent
-----
Verify return code: 0 (ok)

The important points to notice in the output from the above command are as follows:-

1. There are 3 certs, in order, Load balancer cert, IntermediateCA, RootCA (earlier referred as Server Cert, Intermediate Cert and Root Cert) - Check the last 8 digits from the output of the above command  to confirm the certificates match what you expect, you can generally confirm the last 8  digits of the certificate match the expected certificate, as a "quick check".
2. The Verify return code: 0 (ok) indicates the certificate in the chain is trusted.
3. You can add the -CApath ~/cert (this assumes you have your IntermediateCA and rootCA in this directory), if you choose not to add your certificate chain to your haproxy(or other load balancer)

• Automate server certificate:

Run the following command:

openssl s_client -connect automate01.example.com:443 -showcerts

This will show the certs coming from the loadbalancer itself(not the backend servers, automate01, automate02)
Typically this will return 3 or more certificates in order as below

1. Server certificate(eg automate01.example.com )
2. IntermediateCA certificate(1 or more, depending on how many intermediate certs are in the certificate chain)
3. RootCA certificate

You should validate that all x3 certificates are what you would expect:

openssl s_client -showcerts -servername automate01.example.com -connect loadbalancer.example.com

This assumes the DNS alias automate01.example.com is the -servername (ie front end URL)and the actual ip address which hosts the certificate is the -connect, which in this example is the hostname of the loadbalancer node http://loadbalancer.example.com(LoadBalancer in this case)

 4. Validate Status:
Validate that you get a successful response from the backend server, by going via load balancer or direct to the backend servers. - via load balancer(automate01.example.com), automate01,02 servers.                                                        

curl -k https://automate01.example.com/_status

Sample output below:

{"status":"pong","upstreams":{"chef_opensearch":"pong","chef_sql":"pong","oc_chef_authz":"pong","data_collector":"pong"},"keygen":{"keys":10,"max":10,"max_workers":10,"cur_max_workers":10,"inflight":0,"avail_workers":10,"start_size":2},"indexing":{"mode":"batch"}}

Default behaviour would be to get the above response if you connect directly to automate01.example.com. If so, then the automate01 node is functional. If you cannot get this response from automate01.example.com(ie via loadbalancer), then there is something wrong with either the load balancer or the load balancer connection to the backends.

Please use this article as a reference document only and for further help, get in touch with Chef support.