When initialising/creating a backup on a Chef Automate instance that has large amounts of data (100GB+) and no previous backup, you may see the snapshot creation process timeout with the following error from the CLI:

applications-service (sync 100.00%) (async 100.00%) 
authn-service (sync 100.00%) (async 100.00%) 
authz-service (sync 100.00%) (async 100.00%) 
automate-dex (sync 100.00%) (async 100.00%) 
automate-elasticsearch (sync 100.00%) (async 100.00%) 
automate-es-gateway (sync 100.00%) (async 100.00%) 
automate-gateway (sync 100.00%) (async 100.00%) 
automate-load-balancer (sync 100.00%) (async 100.00%) 
automate-pg-gateway (sync 100.00%) (async 100.00%) 
automate-postgresql (sync 100.00%) (async 100.00%) 
automate-ui (sync 100.00%) (async 100.00%) 
backup-gateway (sync 100.00%) (async 100.00%) 
compliance-service (sync 66.67%) (async 100.00%) 
config-mgmt-service (sync 100.00%) (async 100.00%) 
data-lifecycle-service (sync 100.00%) (async 100.00%) 
es-sidecar-service (sync 100.00%) (async 100.00%) 
event-service (sync 100.00%) (async 100.00%) 
local-user-service (sync 100.00%) (async 100.00%) 
pg-sidecar-service (sync 100.00%) (async 100.00%)

BackupError: An issue occurred when creating or restoring a backup: Streaming backup events failed: Unable to handle backup event: Backup failed. Check the Chef Automate logs for more information.

Specifically, errors within the run regarding the synchronisation of the data/reports for nodes and compliance may be stuck or unable to complete within the allotted time. These can be observed in the log as:

compliance-service (sync 66.67%) (async 100.00%) 
...
ingest-service (sync 50.00%) (async 100.00%) 

The error itself can be created when either the backup is manually or automatically halted. A manual halt would involve running the "chef-automate backup cancel" command in a parallel shell session. An automatic halt may occur as part of the default timeout (120 minutes / 7200 seconds) set for the backup window to complete.

BackupError: An issue occurred when creating or restoring a backup: Streaming backup events failed: Unable to handle backup event: Backup failed. Check the Chef Automate logs for more information.

You can also observe these messages in the journalctl logs on Chef Automate as:

# journalctl -uf chef-automate -f:
deployment-service.default(O): time="2020-12-12T12:00:12+01:00" level=error msg="Backup failed" backup_id=20200510190044 error="Failed to set status object to pending: Failed to commit .status: blob (key \"20200510190044/.status\") (code=Unknown): RequestCanceled: request context canceled\ncaused by: context deadline exceeded"

If you observe the additional below error in journalctl:

hab[3523]:
backup-gateway.default(O):
 API: SYSTEM() May 05 07:22:48 ip-10-10-10-20.us-west-2.compute.internal
 Time: 07:22:48 UTC 05/05/2020
 Error: remote error: tls: unknown certificate
 remoteAddr=127.0.0.1:60744, localAddr=127.0.0.1:10143 May 05 07:22:48
 1: cmd/http/listener.go:255:http.(*httpListener).start.func2()

And are running an older version of Chef Automate 2, it's likely that you have been impacted by revoked/expired certificates in the Elasticsearch backup client, which will need to be force-regenerated (see Resolution below).

Ensure you have read and understand the following topics before proceeding:

backup-space-requirements

backup-to-a-filesystem

backup-to-s3

Fully testing and understanding the requirements and implementation of your backup method will ensure that these do not play a part in a backup failure.

Please consider the following points around your back infrastructure:

• It's important to understand that sub-performant network/disk will increase the likelihood of passing the threshold of the backup window timeout mentioned above.
• Having ample disk space for your initial backup as well as sufficient IOPS to both write backups and continue to ingest data should be a consideration.
• In the event you are using S3 ensure that the outbound network bandwidth is ample and preferably prioritised so as to win when contesting bandwidth (the same could be said for using an NFS share or AWS storage gateway)
• In the event of a storage outage whilst a mounted volume is being used for backup we would also see similar hangs/failures. Please ensure that you have monitoring in place for any storage component preferred for backups.

To understand the total amount of data which would be stored we can complete the following:

du -hs /hab/svc/automate-elasticsearch/data/
83M     /hab/svc/automate-elasticsearch/data/

curl -XGET 'http://localhost:10144/_cat/allocation?v';
shards disk.indices disk.used disk.avail disk.total disk.percent host      ip        node
    20      143.6mb    11.2gb     70.7gb     81.9gb           13 127.0.0.1 127.0.0.1 O9fvXj7
    20                                                                               UNASSIGNED

The important thing is to remember that the contents of /hab/svc/automate-elasticsearch/data/ is not the only thing we need to consider in a backup. There will also be PostgreSQL data and additional snapshot metadata to consider. We would suggest to ensure you have ample room for the indices (which should give you a baseline indicator) + all incremental backups you need to store as part of your organisation's retention plans.

Improving the performance of the disk you're backing up to will decrease the time required for the backup process to complete. And may actually allow the backup to proceed.

If your storage used is over 100GB and you are taking the initial backup, it is likely that you will need to increase the timeout window beyond the default 7200 seconds. For every 50GB of data, we would add 1800 to the default. For example, if the system is performant and capable and needs to initialise a 150GB backup:

chef-automate backup create --debug --wait-timeout 9000

Should suffice. If the system is under high load and experiencing issues, we may be able to track this down with the following commands:

chef-automate status 

Ensure that the deployment-service is running and noting the PIDs for ingest-service and compliance-service

kill <INGEST_SERVICE_PID>  

Once complete, run

ss --options --numeric --tcp --all | grep WAIT | wc -l 

To confirm whether this has reduced the TIME_WAIT connections

kill <COMPLIANCE_SERVICE_PID> 

Once complete, run

ss --options --numeric --tcp --all | grep WAIT | wc -l 

To confirm whether this has reduced the TIME_WAIT connections

As long as the deployment-service is running, it should gracefully restart the killed services. If things are looking good, proceed to retry the backup with the correct timeout for the amount of data set. If necessary and viable, consider putting your Automate instance into maintenance mode whilst you complete the initial backup to ensure its success, see chef-automate maintenance mode for details.

If you did see the 'Error: remote error: tls: unknown certificate' mentioned in the above you will also need to remove the existing elasticsearch certificate:

rm -f /hab/svc/automate-elasticsearch/var/ca.jks

And run:

chef-automate restart-services

To restart Elasticsearch and automatically generate its certificates. Once this has completed, re-check the journalctl logs and retry a backup to ensure the issue is resolved.

Long Term Considerations:

• Scaling_Chef_Automate_Beyond_100000_Nodes
• data-lifecycle

The Elasticsearch backups are incremental snapshots of the database taken at that point in time. For Elasticsearch to create a new snapshot, it needs to access all of the previous snapshots to validate any data that is already stored in the backup as part of an earlier snapshot is up to date. If the earlier snapshots are not available the backup will fail, and it will be incomplete.

If you need to delete old backups, you will want to use the `chef-automate backup delete` command. This will tell Elasticsearch to delete all files that are associated with the deleted snapshot and are not used by any other snapshots.

If you are copying the backups to an offsite location, you also must ensure that the entire automate-elasticsearch-data directory is copied and restored as a single unit.