Chef Automate backup fails when the backup location changes from the default location

This article discusses the workaround for the Chef Automate (both standalone and HA) backup process when the backup location changes from the default location and it fails with error - 'failed to create snapshot'

Jan 12, 2024
Knowledge

Title

URL Name

Chef-Automate-backup-process-when-the-backup-location-changes-from-the-default-location

Article Number

000227756

Environment

Chef Automate 4.x.x

Question/Problem Description

When a back up of Chef Automate 4.x.x is taken, the first backup to the local file system is always a full backup. The subsequent backups to the same location are incremental backups. But when the backup location path is changed, the back up process fails and journalctl gives error details as below.

Steps to Reproduce

1. Take a back up of a new installed Automate server. 2. Now try to take a back up again with the back up path different than the first back up location.

Clarifying Information

Error Message

Nov 02 07:19:40 ec2-xx-x-xxx-xxx.ap-south-1.compute.amazonaws.com hab[26423]: deployment-service.default(O): time="2022-11-02T07:19:40Z" level=error msg="Backup failed" backup_id=20221102071939 error="es-sidecar -service failed to create snapshot: rpc error: code = Unknown desc = failed to create es snapshot \"20221102071939\" for service \"event-feed-service\": elastic: Error 500 (Internal Server Error): [chef-automate -es6-event-feed-service] Could not read repository data because the contents of the repository do not match its expected state. This is likely the result of either concurrently modifying the contents of the repo sitory by a process other than this cluster or an issue with the repository's underlying storage. The repository has been disabled to prevent corrupting its contents. To re-enable it and continue using it please remove the repository from the cluster and add it again to make the cluster recover the known state of the repository from its physical contents.

Defect Number

Enhancement Number

Cause

We can not change the backup location of Chef Automate because the opensearch in Automate tries to find certain snapshot in the default/original back up location which is not present at new backup location, i.e the reason "failed to create es snapshot “.

Resolution

Workaround

Follow the below step to take a full back up at a new location.

Update BKP-PATH with the new backup path on your file system and trigger the below script

indices=(
chef-automate-es6-automate-cs-oc-erchef
chef-automate-es6-compliance-service
chef-automate-es6-event-feed-service
chef-automate-es6-ingest-service
)
for index in ${indices[@]}; do
curl -XPUT -k -H 'Content-Type: application/json' http://localhost:10144/_snapshot/$index --data-binary @- << EOF
{
  "type" : "fs",
  "settings" : {
    "location" : "<BKP-PATH>/automate-elasticsearch-data/$index"
  }
}
EOF
done

After the script execution you will get output like below.

{"acknowledged":true}{"acknowledged":true}
{"acknowledged":true}{"acknowledged":true}

This script deletes all the existing snapshots at once. After that you can trigger the backup command again.

Further details on troubleshooting common issues faced on Chef Automate HA can be found at the official documentation .

Notes

This applies to both standalone and Automate HA set ups. For Automate HA, this needs to be done on any Automate node. The steps described here are applicable to back up process to local file system. For details on backup/restore with S3 buckets troubleshooting, refer the document here .

Keyword Phrase

Last Modified Date

1/12/2024 3:32 PM

Chef Automate backup fails when the backup location changes from the default location

Files(0)

Was this article helpful?

Was this article helpful?

Chef Automate backup fails when the backup location changes from the default location

Information

Related Articles