Upgrading
Enter a short description here.
Abstract
Chapter 1. Upgrading using Helm charts
If you have installed Red Hat Advanced Cluster Security for Kubernetes by using Helm charts, to upgrade to the latest version of Red Hat Advanced Cluster Security for Kubernetes you must perform the following:
- Update the Helm chart.
- Update configuration files for the central-services Helm chart.
- Upgrade the central-services Helm chart.
- Update configuration files for the secured-cluster-services Helm chart.
- Upgrade the secured-cluster-services Helm chart.
1.1. Updating the Helm chart repository
You must always update Helm charts before upgrading to a new version of Red Hat Advanced Cluster Security for Kubernetes.
Prerequisites
- You must have already added the Red Hat Advanced Cluster Security for Kubernetes Helm chart repository.
Procedure
Update Red Hat Advanced Cluster Security for Kubernetes charts repository.
$ helm repo update
Verification
Run the following command to verify the added chart repository:
$ helm search repo -l rhacs/
1.2. Additional resources
1.3. Changing configuration options after deploying the central-services Helm chart
You can make changes to any configuration options after you have deployed the central-services Helm chart.
Procedure
-
Update the
values-public.yamlandvalues-private.yamlconfiguration files with new values. Run the
helm upgradecommand and specify the configuration files using the-foption:$ helm upgrade -n stackrox \ stackrox-central-services rhacs/central-services \ -f <path_to_values_public.yaml> \ -f <path_to_values_private.yaml>
NoteYou can also specify configuration values using the
--setor--set-fileparameters. However, these options are not saved, and it requires you to manually specify all the options again whenever you make changes.
1.4. Changing configuration options after deploying the secured-cluster-services Helm chart
You can make changes to any configuration options after you have deployed the secured-cluster-services Helm chart.
Procedure
-
Update the
values-public.yamlandvalues-private.yamlconfiguration files with new values. Run the
helm upgradecommand and specify the configuration files using the-foption:$ helm upgrade -n stackrox \ stackrox-secured-cluster-services rhacs/secured-cluster-services \ --reuse-values \ 1 -f <path_to_values_public.yaml> \ -f <path_to_values_private.yaml>- 1
- You must specify the
--reuse-valuesparameter, otherwise the Helm upgrade command resets all previously configured settings.
NoteYou can also specify configuration values using the
--setor--set-fileparameters. However, these options are not saved, and it requires you to manually specify all the options again whenever you make changes.
Chapter 2. Manually upgrading using the roxctl CLI
You can upgrade to the latest version of Red Hat Advanced Cluster Security for Kubernetes from a supported older version.
To upgrade Red Hat Advanced Cluster Security for Kubernetes to the latest version, you must perform the following:
- Backup the Central database
- Upgrade Central
-
Upgrade the
roxctlCLI - Upgrade Scanner
- Verify that all secured clusters are upgraded
2.1. Backing up the Central database
You can back up the Central database and use that backup for rolling back from a failed upgrade or data restoration in the case of an infrastructure disaster.
Prerequisites
-
You must have an API token with
readpermission for all resources of Red Hat Advanced Cluster Security for Kubernetes. The Analyst system role hasreadpermissions for all resources. -
You have installed the
roxctlCLI. -
You have configured the
ROX_API_TOKENand theROX_CENTRAL_ADDRESSenvironment variables.
Procedure
Run the backup command:
For Red Hat Advanced Cluster Security for Kubernetes 3.0.55 and newer:
$ roxctl -e "$ROX_CENTRAL_ADDRESS" central backup
For Red Hat Advanced Cluster Security for Kubernetes 3.0.54 and older:
$ roxctl -e "$ROX_CENTRAL_ADDRESS" central db backup
Additional resources
2.2. Upgrading the Central cluster
After you have backed up the Central database, the next step is to upgrade the Central cluster. This step includes upgrading Central, the roxctl CLI, and the Scanner.
2.2.1. Upgrading Central
You can update Central to the latest version by downloading and deploying the updated images.
Prerequisites
- If you deploy images from a private image registry, first push the new image into your private registry, and then replace your image registry for the commands in this section.
Procedure
Run the following commands to upgrade Central:
$ oc -n stackrox patch deploy/central -p '{"spec":{"template":{"spec":{"containers":[{"name":"central","env":[{"name":"ROX_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}}]}]}}}}' 1- 1
- If you use Kubernetes, enter
kubectlinstead ofoc.
$ oc -n stackrox patch deployment/scanner -p '{"spec":{"template":{"spec":{"containers":[{"name":"scanner","securityContext":{"runAsUser":65534}}]}}}}' 1- 1
- If you use Kubernetes, enter
kubectlinstead ofoc.
$ oc -n stackrox set image deploy/central central=registry.redhat.io/advanced-cluster-security/rhacs-main-rhel8:3.69.2 1- 1
- If you use Kubernetes, enter
kubectlinstead ofoc.
ImportantIf you are upgrading from Red Hat Advanced Cluster Security for Kubernetes 3.65.0, you must run the following additional command to create the
stackrox-central-diagnosticsrole:$ oc -n stackrox patch role stackrox-central-diagnostics -p '{"rules":[{"apiGroups":["*"],"resources":["deployments","daemonsets","replicasets","configmaps","services"],"verbs":["get","list"]}]}' 1- 1
- If you use Kubernetes, enter
kubectlinstead ofoc.
If you have not installed Red Hat Advanced Cluster Security for Kubernetes by using Helm or Operator, and you want to enable authentication using the OpenShift OAuth server, you must run the following additional command:
$ oc -n stackrox set env deploy/central ROX_ENABLE_OPENSHIFT_AUTH=true
$ oc -n stackrox patch serviceaccount/central -p ' { "metadata": { "annotations": { "serviceaccounts.openshift.io/oauth-redirecturi.main": "sso/providers/openshift/callback", "serviceaccounts.openshift.io/oauth-redirectreference.main": "{"kind":"OAuthRedirectReference","apiVersion":"v1","reference":{"kind":"Route","name":"central"}}" } } }'
2.2.2. Upgrading the roxctl CLI
To upgrade the roxctl CLI to the latest version you must uninstall the existing version of roxctl CLI and then install the latest version of the roxctl CLI.
2.2.2.1. Uninstalling the roxctl CLI
You can uninstall the roxctl CLI binary on Linux by using the following procedure.
Procedure
Find and delete the
roxctlbinary:$ ROXPATH=$(which roxctl) && rm -f $ROXPATH 1- 1
- Depending on your environment, you might need administrator rights to delete the
roxctlbinary.
2.2.2.2. Installing the roxctl CLI on Linux
You can install the roxctl CLI binary on Linux by using the following procedure.
Procedure
Download the latest version of the
roxctlCLI:$ curl -O https://mirror.openshift.com/pub/rhacs/assets/3.69.2/bin/Linux/roxctl
Make the
roxctlbinary executable:$ chmod +x roxctl
Place the
roxctlbinary in a directory that is on yourPATH:To check your
PATH, execute the following command:$ echo $PATH
Verification
Verify the
roxctlversion you have installed:$ roxctl version
2.2.2.3. Installing the roxctl CLI on macOS
You can install the roxctl CLI binary on macOS by using the following procedure.
Procedure
Download the latest version of the
roxctlCLI:$ curl -O https://mirror.openshift.com/pub/rhacs/assets/3.69.2/bin/Darwin/roxctl
Remove all extended attributes from the binary:
$ xattr -c roxctl
Make the
roxctlbinary executable:$ chmod +x roxctl
Place the
roxctlbinary in a directory that is on yourPATH:To check your
PATH, execute the following command:$ echo $PATH
Verification
Verify the
roxctlversion you have installed:$ roxctl version
2.2.2.4. Installing the roxctl CLI on Windows
You can install the roxctl CLI binary on Windows by using the following procedure.
Procedure
Download the latest version of the
roxctlCLI:$ curl -O https://mirror.openshift.com/pub/rhacs/assets/3.69.2/bin/Windows/roxctl.exe
Verification
Verify the
roxctlversion you have installed:$ roxctl version
After you upgrade the roxctl CLI you can upgrade Scanner.
2.2.3. Upgrading Scanner
You can update Scanner to the latest version by using the roxctl CLI.
Prerequisites
- If you deploy images from a private image registry, first push the new image into your private registry, and then replace your image registry for the commands in this section.
- If you used Red Hat UBI-based images when you installed Red Hat Advanced Cluster Security for Kubernetes, replace the image names for the commands in this section with the following UBI-based image names:
If you have created custom scanner configurations, you must apply those changes before updating the scanner configuration file:
$ roxctl -e "$ROX_CENTRAL_ADDRESS" scanner generate
$ oc apply -f scanner-bundle/scanner/02-scanner-03-tls-secret.yaml 1- 1
- If you use Kubernetes, enter
kubectlinstead ofoc.
$ oc apply -f scanner-bundle/scanner/02-scanner-04-scanner-config.yaml 1- 1
- If you use Kubernetes, enter
kubectlinstead ofoc.
Procedure
Update the Scanner image:
$ oc -n stackrox set image deploy/scanner scanner=registry.redhat.io/advanced-cluster-security/rhacs-scanner-rhel8:3.69.2 1- 1
- If you use Kubernetes, enter
kubectlinstead ofoc.
Update the Scanner database image:
$ oc -n stackrox set image deploy/scanner-db db=registry.redhat.io/advanced-cluster-security/rhacs-scanner-db-rhel8:3.69.2 1- 1
- If you use Kubernetes, enter
kubectlinstead ofoc.
$ oc -n stackrox set image deploy/scanner-db init-db=registry.redhat.io/advanced-cluster-security/rhacs-scanner-db-rhel8:3.69.2 1- 1
- If you use Kubernetes, enter
kubectlinstead ofoc.
Verification
Check that the new pods have deployed successfully:
$ oc get pod -n stackrox --watch 1- 1
- If you use Kubernetes, enter
kubectlinstead ofoc.
2.2.4. Verifying the Central cluster upgrade
After you have upgraded both Central and Scanner, verify that the Central cluster upgrade is complete.
Procedure
Check the Central logs:
$ oc logs -n stackrox deploy/central -c central 1- 1
- If you use Kubernetes, enter
kubectlinstead ofoc.
If the upgrade is successful, you will see output similar to the following:
No database restore directory found (this is not an error). Migrator: 2019/10/25 17:58:54: starting DB compaction Migrator: 2019/10/25 17:58:54: Free fraction of 0.0391 (40960/1048576) is < 0.7500. Will not compact badger 2019/10/25 17:58:54 INFO: All 1 tables opened in 2ms badger 2019/10/25 17:58:55 INFO: Replaying file id: 0 at offset: 846357 badger 2019/10/25 17:58:55 INFO: Replay took: 50.324µs badger 2019/10/25 17:58:55 DEBUG: Value log discard stats empty Migrator: 2019/10/25 17:58:55: DB is up to date. Nothing to do here. badger 2019/10/25 17:58:55 INFO: Got compaction priority: {level:0 score:1.73 dropPrefix:[]} version: 2019/10/25 17:58:55.189866 ensure.go:49: Info: Version found in the DB was current. We’re good to go!
2.3. Upgrading all secured clusters
After upgrading Central services, you must upgrade all secured clusters.
If you are using automatic upgrades:
- Update all your secured clusters by using automatic upgrades.
- Skip the instructions in this section and follow the instructions in the Verify upgrades and Revoking the API token sections.
- If you are not using automatic upgrades, you must run the instructions in this section on all secured clusters including the Central cluster.
To complete manual upgrades of each secured cluster running Sensor, Collector, and Admission Controller, follow the instructions in this section.
2.3.1. Update readiness probes
If you are upgrading from a version below Red Hat Advanced Cluster Security for Kubernetes 3.65.0, you must run the following additional command to update the readiness probe path. If you are running a higher version than 3.65, skip this step.
Procedure
Update the readiness probe path:
$ oc -n stackrox patch deploy/sensor -p '{"spec":{"template":{"spec":{"containers":[{"name":"sensor","readinessProbe":{"httpGet":{"path":"/ready"}}}]}}}}' 1- 1
- If you use Kubernetes, enter
kubectlinstead ofoc.
2.3.2. Updating OpenShift security context constraints
Depending on the version of Red Hat Advanced Cluster Security for Kubernetes you are upgrading to, you must update certain OpenShift Container Platform security context constraints (SCCs).
Run the commands in this section only if you are using Red Hat Advanced Cluster Security for Kubernetes with OpenShift Container Platform. Otherwise, skip the instructions in this section.
Procedure
Red Hat Advanced Cluster Security for Kubernetes 3.64.0 renames the SCCs. If you are upgrading from a version below Red Hat Advanced Cluster Security for Kubernetes 3.64.0, you must delete and reapply the SCCs, otherwise, skip this step:
Run the following commands to update Central:
$ oc apply -f - <<EOF kind: SecurityContextConstraints apiVersion: security.openshift.io/v1 metadata: name: stackrox-central labels: app.kubernetes.io/name: stackrox annotations: kubernetes.io/description: stackrox-central is the security constraint for the central server email: support@stackrox.com owner: stackrox allowHostDirVolumePlugin: false allowedCapabilities: [] allowHostIPC: false allowHostNetwork: false allowHostPID: false allowHostPorts: false allowPrivilegeEscalation: false allowPrivilegedContainer: false defaultAddCapabilities: [] fsGroup: type: MustRunAs ranges: - max: 4000 min: 4000 priority: 0 readOnlyRootFilesystem: true requiredDropCapabilities: [] runAsUser: type: MustRunAs uid: 4000 seLinuxContext: type: MustRunAs seccompProfiles: - '*' users: - system:serviceaccount:stackrox:central volumes: - '*' EOF$ oc delete scc central
Run the following commands to update Scanner:
$ oc apply -f - <<EOF kind: SecurityContextConstraints apiVersion: security.openshift.io/v1 metadata: name: stackrox-scanner labels: app.kubernetes.io/name: stackrox annotations: email: support@stackrox.com owner: stackrox kubernetes.io/description: stackrox-scanner is the security constraint for the Scanner container priority: 0 runAsUser: type: RunAsAny seLinuxContext: type: RunAsAny seccompProfiles: - '*' users: - system:serviceaccount:stackrox:scanner volumes: - '*' allowHostDirVolumePlugin: false allowedCapabilities: [] allowHostIPC: false allowHostNetwork: false allowHostPID: false allowHostPorts: false allowPrivilegeEscalation: false allowPrivilegedContainer: false defaultAddCapabilities: [] fsGroup: type: RunAsAny readOnlyRootFilesystem: false requiredDropCapabilities: [] EOF$ oc delete scc scanner
Run the following commands on each OpenShift Secured Cluster:
$ oc apply -f - <<EOF apiVersion: security.openshift.io/v1 kind: SecurityContextConstraints metadata: name: stackrox-admission-control labels: app.kubernetes.io/name: stackrox auto-upgrade.stackrox.io/component: "sensor" annotations: email: support@stackrox.com owner: stackrox kubernetes.io/description: stackrox-admission-control is the security constraint for the admission controller users: - system:serviceaccount:stackrox:admission-control priority: 0 runAsUser: type: RunAsAny seLinuxContext: type: RunAsAny seccompProfiles: - '*' supplementalGroups: type: RunAsAny fsGroup: type: RunAsAny groups: [] readOnlyRootFilesystem: true allowHostDirVolumePlugin: false allowHostIPC: false allowHostNetwork: false allowHostPID: false allowHostPorts: false allowPrivilegeEscalation: false allowPrivilegedContainer: false allowedCapabilities: [] defaultAddCapabilities: [] requiredDropCapabilities: [] volumes: - configMap - downwardAPI - emptyDir - secret --- apiVersion: security.openshift.io/v1 kind: SecurityContextConstraints metadata: name: stackrox-collector labels: app.kubernetes.io/name: stackrox auto-upgrade.stackrox.io/component: "sensor" annotations: email: support@stackrox.com owner: stackrox kubernetes.io/description: This SCC is based on privileged, hostaccess, and hostmount-anyuid users: - system:serviceaccount:stackrox:collector allowHostDirVolumePlugin: true allowPrivilegedContainer: true fsGroup: type: RunAsAny groups: [] priority: 0 readOnlyRootFilesystem: true runAsUser: type: RunAsAny seLinuxContext: type: RunAsAny seccompProfiles: - '*' supplementalGroups: type: RunAsAny allowHostIPC: false allowHostNetwork: false allowHostPID: false allowHostPorts: false allowPrivilegeEscalation: true allowedCapabilities: [] defaultAddCapabilities: [] requiredDropCapabilities: [] volumes: - configMap - downwardAPI - emptyDir - hostPath - secret --- apiVersion: security.openshift.io/v1 kind: SecurityContextConstraints metadata: name: stackrox-sensor labels: app.kubernetes.io/name: stackrox auto-upgrade.stackrox.io/component: "sensor" annotations: email: support@stackrox.com owner: stackrox kubernetes.io/description: stackrox-sensor is the security constraint for the sensor users: - system:serviceaccount:stackrox:sensor - system:serviceaccount:stackrox:sensor-upgrader priority: 0 runAsUser: type: RunAsAny seLinuxContext: type: RunAsAny seccompProfiles: - '*' supplementalGroups: type: RunAsAny fsGroup: type: RunAsAny groups: [] readOnlyRootFilesystem: true allowHostDirVolumePlugin: false allowHostIPC: false allowHostNetwork: false allowHostPID: false allowHostPorts: false allowPrivilegeEscalation: true allowPrivilegedContainer: false allowedCapabilities: [] defaultAddCapabilities: [] requiredDropCapabilities: [] volumes: - configMap - downwardAPI - emptyDir - secret EOF$ oc delete scc admission-control collector sensor
2.3.3. Updating other images
You must update the sensor, collector and compliance images on each secured cluster when not using automatic upgrades.
If you are using Kubernetes, use kubectl instead of oc for the commands listed in this procedure.
Procedure
Update the Sensor image:
$ oc -n stackrox set image deploy/sensor sensor=registry.redhat.io/advanced-cluster-security/rhacs-main-rhel8:3.69.2 1- 1
- If you use Kubernetes, enter
kubectlinstead ofoc.
Update the Compliance image:
$ oc -n stackrox set image ds/collector compliance=registry.redhat.io/advanced-cluster-security/rhacs-main-rhel8:3.69.2 1- 1
- If you use Kubernetes, enter
kubectlinstead ofoc.
Update the Collector image:
$ oc -n stackrox set image ds/collector collector=registry.redhat.io/advanced-cluster-security/rhacs-collector-rhel8:3.69.2 1- 1
- If you use Kubernetes, enter
kubectlinstead ofoc.
NoteIf you are using the collector slim image, run the following command instead:
$ oc -n stackrox set image ds/collector collector=registry.redhat.io/advanced-cluster-security/rhacs-collector-slim-rhel8:{rhacs-version}Update the admission control image:
$ oc -n stackrox set image deploy/admission-control admission-control=registry.redhat.io/advanced-cluster-security/rhacs-main-rhel8:3.69.2
2.3.4. Verifying secured cluster upgrade
After you have upgraded secured clusters, verify that the updated pods are working.
2.4. Rolling back Central
You can roll back to a previous version of Central if the upgrade to a new version is unsuccessful.
2.4.1. Rolling back Central normally
You can roll back to a previous version of Central if upgrading Red Hat Advanced Cluster Security for Kubernetes fails.
Prerequisites
- You must be using Red Hat Advanced Cluster Security for Kubernetes 3.0.57.0 or higher.
- Before you can perform a rollback, you must have free disk space available on your persistent storage. Red Hat Advanced Cluster Security for Kubernetes uses disk space to keep a copy of databases during the upgrade. If the disk space is not enough to store a copy and the upgrade fails, you will not be able to roll back to an earlier version.
Procedure
Run the following command to roll back to a previous version when an upgrade fails (before the Central service starts):
$ oc -n stackrox rollout undo deploy/central 1- 1
- If you use Kubernetes, enter
kubectlinstead ofoc.
2.5. Verifying upgrades
The updated Sensors and Collectors continue to report the latest data from each secured cluster.
The last time Sensor contacted Central is visible in the RHACS portal.
Procedure
- On the RHACS portal, navigate to Platform Configuration → System Health.
- Check to ensure that Sensor Upgrade shows clusters up to date with Central.
2.6. Revoking the API token
For security reasons, Red Hat recommends that you revoke the API token that you have used to complete Central database backup.
Prerequisites
- After the upgrade, you must reload the RHACS portal page and re-accept the certificate to continue using the RHACS portal.
Procedure
- On the RHACS portal, navigate to Platform Configuration → Integrations.
- Scroll down to the Authentication Tokens category, and click API Token.
- Select the checkbox in front of the token name that you want to revoke.
- Click Revoke.
- On the confirmation dialog box, click Confirm.