What are the steps and best practices to follow while upgrading chef-client?

Before attempting a chef-client upgrade, it is essential to evaluate whether your existing cookbooks will be compatible with the latest versions, and know how to use the tools provided in Chef Workstation to quickly iterate on and test those cookbooks against a target release.

The Chef Workstation comes with cookstyle embedded in it. The target client version can be updated in a yaml file (.rubocop.yml) inside your cookbook and then cookstyle can be run to check the compatibility of the cookbooks with the target client version. 

The command to run cookstyle is: 

cookstyle /path/to/cookbook

Many of the Cookstyle Cops include the ability to autocorrect violations. To autocorrect code, run the following from the cookbook directory:

cookstyle -a .

The details on leveraging various features in cookstyle can be found at: https://docs.chef.io/workstation/cookstyle/#run-cookstyle, https://github.com/chef/cookstyle and https://www.chef.io/webinars/prepare-for-new-chef-infra-client-releases-with-cookstyle

Another great way to test your cookbooks before going for the upgrade and testing cookbooks while developing code for a specific client version is using Test Kitchen.

Once you’re on Chef Client 15 or higher and have the latest Chef Workstation, you can use Test Kitchen attributes (https://docs.chef.io/workstation/config_yml_kitchen/) to be declarative about the versions you want to write code for. Adding the following to your kitchen.yml will ensure that the cookbook is compiled on the target version, as well as any deprecations will be flagged as errors, forcing the developer to fix those errors.

In your kitchen.yml:

# Set target version (latest by default) https://docs.chef.io/workstation/config_yml_kitchen/#new-provisioner-settings
product_version: latest
# Set deprecations as errors (https://docs.chef.io/workstation/config_yml_kitchen/#provisioner-settings)
provisioner:
  deprecations_as_errors: true

It is recommended to write Integration Tests within your cookbooks using InSpec and test it using Test Kitchen both in your development environment and also in your CI/CD pipelines to ensure the configurations in your recipes are being applied as expected.

In the CI/CD tool of your choice, incorporate cookstyle(lint/syntax testing) and Test Kitchen in your build and release pipelines. This ensures that there are no syntactical issues or regressions in the cookbooks being tested.
Sample pipelines for a few of the CI/CD tools can be found here; https://github.com/chef-cft/chef-examples/tree/main/examples/pipelines  

Finally, you can use your own tools of choice to upgrade or uninstall/reinstall with the desired new chef-client version. There are various ways to upgrade your chef-client like command line upgrade, using cookbooks and by the knife ssh command in Chef Workstation to execute the install script on multiple Linux, macOS, and Unix-based hosts at once. Please refer to https://docs.chef.io/upgrade_client/  for details.

You can also make use of the Chef client updater cookbook (https://github.com/chef-cookbooks/chef_client_updater) to upgrade your chef-client. Such an upgrade on Windows would need to include stopping any of the currently running version of the chef-client. There can never be two rubies/chef-clients running, otherwise the MSI on the Windows system will refuse to do the install properly. Most of the recent chef-client installs on Windows using this cookbook have also included a scheduled task install. That scheduled task would also need to be paused, so that it doesn't start up a new chef-client instance in the middle of an upgrade.

Upgrading requires testing. If you are going to upgrade, we recommend working through any changes needed for one cookbook at a time, not all cookbooks and nodes at once. When you find issues in the cookbooks, it is recommended to fix them and those fixes can be applied to the other cookbooks containing the same pattern.

We recommend to do extensive testing in a lower environment before doing any upgrade in the production environment. For example, if you are using policyfile and policy group way of deployment, first upgrade and test in lower policy groups and then propagate the changes to the wider policy groups. Similarly if you are using roles and environments way of deployment, first upgrade and test in lower environments like test/development and then propagate the changes to the higher environments like production.