Chapter 5. Compliance Operator

Procedure

1. In the OpenShift Container Platform web console, navigate to Operators → OperatorHub.
2. Search for the Compliance Operator, then click Install.
3. Keep the default selection of Installation mode and namespace to ensure that the Operator will be installed to the openshift-compliance namespace.
4. Click Install.

1. Navigate to the Operators → Installed Operators page.
2. Check that the Compliance Operator is installed in the openshift-compliance namespace and its status is Succeeded.

1. Navigate to the Operators → Installed Operators page and inspect the Status column for any errors or failures.
2. Navigate to the Workloads → Pods page and check the logs in any pods in the openshift-compliance project that are reporting issues.

Procedure

1. Define a Namespace object:

   Example namespace-object.yaml

   apiVersion: v1
   kind: Namespace
   metadata:
     labels:
       openshift.io/cluster-monitoring: "true"
     name: openshift-compliance

2. Create the Namespace object:

   $ oc create -f namespace-object.yaml

3. Define an OperatorGroup object:

   Example operator-group-object.yaml

   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: compliance-operator
     namespace: openshift-compliance
   spec:
     targetNamespaces:
     - openshift-compliance

4. Create the OperatorGroup object:

   $ oc create -f operator-group-object.yaml

5. Define a Subscription object:

   Example subscription-object.yaml

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: compliance-operator-sub
     namespace: openshift-compliance
   spec:
     channel: "release-0.1"
     installPlanApproval: Automatic
     name: compliance-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace

6. Create the Subscription object:

   $ oc create -f subscription-object.yaml

Verification

1. Verify the installation succeeded by inspecting the CSV file:

   $ oc get csv -n openshift-compliance

2. Verify that the Compliance Operator is up and running:

   $ oc get deploy -n openshift-compliance

Procedure

1. Inspect the ScanSetting object by running:

   $ oc describe scansettings default -n openshift-compliance

   Example output

   apiVersion: compliance.openshift.io/v1alpha1
   kind: ScanSetting
   metadata:
     name: default
     namespace: openshift-compliance
   rawResultStorage:
     pvAccessModes:
     - ReadWriteOnce 1
     rotation: 3 2
     size: 1Gi 3
   roles:
   - worker 4
   - master 5
   scanTolerations: 6
     default:
     - operator: Exists
     schedule: 0 1 * * * 7

   1

   The Compliance Operator creates a persistent volume (PV) that contains the results of the scans. By default, the PV will use access mode ReadWriteOnce because the Compliance Operator cannot make any assumptions about the storage classes configured on the cluster. Additionally, ReadWriteOnce access mode is available on most clusters. If you need to fetch the scan results, you can do so by using a helper pod, which also binds the volume. Volumes that use the ReadWriteOnce access mode can be mounted by only one pod at time, so it is important to remember to delete the helper pods. Otherwise, the Compliance Operator will not be able to reuse the volume for subsequent scans.

   2

   The Compliance Operator keeps results of three subsequent scans in the volume; older scans are rotated.

   3

   The Compliance Operator will allocate one GB of storage for the scan results.

   4 5

   If the scan setting uses any profiles that scan cluster nodes, scan these node roles.

   6

   The default scan setting object also scans all the nodes.

   7

   The default scan setting object runs scans at 01:00 each day.

   As an alternative to the default scan setting, you can use default-auto-apply, which has the following settings:

   apiVersion: compliance.openshift.io/v1alpha1
   kind: ScanSetting
   metadata:
     name: default-auto-apply
     namespace: openshift-compliance
   autoUpdateRemediations: true 1
   autoApplyRemediations: true 2
   rawResultStorage:
     pvAccessModes:
       - ReadWriteOnce
     rotation: 3
     size: 1Gi
   schedule: 0 1 * * *
   roles:
     - worker
     - master
   scanTolerations:
     default:
     - operator: Exists

   1 2

   Setting autoUpdateRemediations and autoApplyRemediations flags to true allows you to easily create ScanSetting objects that auto-remediate without extra steps.

2. Create a ScanSettingBinding object that binds to the default ScanSetting object and scans the cluster using the cis and cis-node profiles. For example:

   apiVersion: compliance.openshift.io/v1alpha1
   kind: ScanSettingBinding
   metadata:
     name: cis-compliance
     namespace: openshift-compliance
   profiles:
     - name: ocp4-cis-node
       kind: Profile
       apiGroup: compliance.openshift.io/v1alpha1
     - name: ocp4-cis
       kind: Profile
       apiGroup: compliance.openshift.io/v1alpha1
   settingsRef:
     name: default
     kind: ScanSetting
     apiGroup: compliance.openshift.io/v1alpha1

3. Create the ScanSettingBinding object by running:

   $ oc create -f <file-name>.yaml -n openshift-compliance

   At this point in the process, the ScanSettingBinding object is reconciled and based on the Binding and the Bound settings. The Compliance Operator creates a ComplianceSuite object and the associated ComplianceScan objects.

4. Follow the compliance scan progress by running:

   $ oc get compliancescan -w -n openshift-compliance

   The scans progress through the scanning phases and eventually reach the DONE phase when complete. In most cases, the result of the scan is NON-COMPLIANT. You can review the scan results and start applying remediations to make the cluster compliant. See Managing Compliance Operator remediation for more information.

Procedure

1. Ensure that the lookup policy is set to local:

   $ oc patch is openscap-ocp4-ds \
       -p '{"spec":{"lookupPolicy":{"local":true}}}' \
       --type=merge
       imagestream.image.openshift.io/openscap-ocp4-ds patched
       -n openshift-compliance

2. Use the name of the ImageStreamTag for the ProfileBundle by retrieving the istag name:

   $ oc get istag -n openshift-compliance

   Example output

   NAME                  	IMAGE REFERENCE                                                                                                                                              	UPDATED
   openscap-ocp4-ds:latest   image-registry.openshift-image-registry.svc:5000/openshift-compliance/openscap-ocp4-ds@sha256:46d7ca9b7055fe56ade818ec3e62882cfcc2d27b9bf0d1cbae9f4b6df2710c96   3 minutes ago

3. Create the ProfileBundle:

   $ cat << EOF | oc create -f -
   apiVersion: compliance.openshift.io/v1alpha1
   kind: ProfileBundle
   metadata:
     name: mybundle
      spec:
        contentImage: openscap-ocp4-ds:latest
        contentFile: ssg-rhcos4-ds.xml
   EOF

Procedure

1. Browse the available rules for the Red Hat Enterprise Linux CoreOS (RHCOS) ProfileBundle:

   $ oc get rules.compliance -n openshift-compliance -l compliance.openshift.io/profile-bundle=rhcos4

2. Browse the available variables in the same ProfileBundle:

   $ oc get variables.compliance -n openshift-compliance -l compliance.openshift.io/profile-bundle=rhcos4

3. Create a tailored profile named nist-moderate-modified:

   1. Choose which rules you want to add to the nist-moderate-modified tailored profile. This example extends the rhcos4-moderate profile by disabling two rules and changing one value. Use the rationale value to describe why these changes were made:

      Example new-profile-node.yaml

      apiVersion: compliance.openshift.io/v1alpha1
      kind: TailoredProfile
      metadata:
        name: nist-moderate-modified
      spec:
        extends: rhcos4-moderate
        description: NIST moderate profile
        title: My modified NIST moderate profile
        disableRules:
        - name: rhcos4-file-permissions-var-log-messages
          rationale: The file contains logs of error messages in the system
        - name: rhcos4-account-disable-post-pw-expiration
          rationale: No need to check this as it comes from the IdP
        setValues:
        - name: rhcos4-var-selinux-state
          rationale: Organizational requirements
          value: permissive

      Table 5.2. Attributes for spec variables

      Attribute	Description
      extends	Name of the Profile object upon which this TailoredProfile is built.
      title	Human-readable title of the TailoredProfile.
      disableRules	A list of name and rationale pairs. Each name refers to a name of a rule object that is to be disabled. The rationale value is human-readable text describing why the rule is disabled.
      enableRules	A list of name and rationale pairs. Each name refers to a name of a rule object that is to be enabled. The rationale value is human-readable text describing why the rule is enabled.
      description	Human-readable text describing the TailoredProfile.
      setValues	A list of name, rationale, and value groupings. Each name refers to a name of the value set. The rationale is human-readable text describing the set. The value is the actual setting.

   2. Create the TailoredProfile object:

      $ oc create -n openshift-compliance -f new-profile-node.yaml 1

      1

      The TailoredProfile object is created in the default openshift-compliance namespace.

      Example output

      tailoredprofile.compliance.openshift.io/nist-moderate-modified created

4. Define the ScanSettingBinding object to bind the new nist-moderate-modified tailored profile to the default ScanSetting object.

   Example new-scansettingbinding.yaml

   apiVersion: compliance.openshift.io/v1alpha1
   kind: ScanSettingBinding
   metadata:
     name: nist-moderate-modified
   profiles:
     - apiGroup: compliance.openshift.io/v1alpha1
       kind: Profile
       name: ocp4-moderate
     - apiGroup: compliance.openshift.io/v1alpha1
       kind: TailoredProfile
       name: nist-moderate-modified
   settingsRef:
     apiGroup: compliance.openshift.io/v1alpha1
     kind: ScanSetting
     name: default

5. Create the ScanSettingBinding object:

   $ oc create -n openshift-compliance -f new-scansettingbinding.yaml

   Example output

   scansettingbinding.compliance.openshift.io/nist-moderate-modified created

1. Explore the ComplianceSuite object:

   $ oc get compliancesuites nist-moderate-modified -o json \
       | jq '.status.scanStatuses[].resultsStorage'
       {
         "name": "rhcos4-moderate-worker",
         "namespace": "openshift-compliance"
       }
       {
         "name": "rhcos4-moderate-master",
         "namespace": "openshift-compliance"
       }

   This shows the persistent volume claims where the raw results are accessible.

2. Verify the raw data location by using the name and namespace of one of the results:

   $ oc get pvc -n openshift-compliance rhcos4-moderate-worker

   Example output

   NAME                 	STATUS   VOLUME                                 	CAPACITY   ACCESS MODES   STORAGECLASS   AGE
   rhcos4-moderate-worker   Bound	pvc-548f6cfe-164b-42fe-ba13-a07cfbc77f3a   1Gi    	RWO        	gp2        	92m

3. Fetch the raw results by spawning a pod that mounts the volume and copying the results:

   Example pod

   apiVersion: "v1"
   kind: Pod
   metadata:
     name: pv-extract
   spec:
     containers:
       - name: pv-extract-pod
         image: registry.access.redhat.com/ubi8/ubi
         command: ["sleep", "3000"]
         volumeMounts:
         - mountPath: "/workers-scan-results"
           name: workers-scan-vol
     volumes:
       - name: workers-scan-vol
         persistentVolumeClaim:
           claimName: rhcos4-moderate-worker

4. After the pod is running, download the results:

   $ oc cp pv-extract:/workers-scan-results .

   Important

   Spawning a pod that mounts the persistent volume will keep the claim as Bound. If the volume’s storage class in use has permissions set to ReadWriteOnce, the volume is only mountable by one pod at a time. You must delete the pod upon completion, or it will not be possible for the Operator to schedule a pod and continue storing results in this location.

5. After the extraction is complete, the pod can be deleted:

   $ oc delete pod pv-extract

Procedure

1. List the nodes.

   $ oc get nodes

   Example output

   NAME                                       STATUS  ROLES  AGE    VERSION
   ip-10-0-128-92.us-east-2.compute.internal  Ready   master 5h21m  v1.23.3+d99c04f
   ip-10-0-158-32.us-east-2.compute.internal  Ready   worker 5h17m  v1.23.3+d99c04f
   ip-10-0-166-81.us-east-2.compute.internal  Ready   worker 5h17m  v1.23.3+d99c04f
   ip-10-0-171-170.us-east-2.compute.internal Ready   master 5h21m  v1.23.3+d99c04f
   ip-10-0-197-35.us-east-2.compute.internal  Ready   master 5h22m  v1.23.3+d99c04f

2. Add a label to nodes.

   $ oc label node ip-10-0-166-81.us-east-2.compute.internal node-role.kubernetes.io/<machine_config_pool_name>=

   Example output

   node/ip-10-0-166-81.us-east-2.compute.internal labeled

3. Create custom MachineConfigPool CR.

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfigPool
   metadata:
     name: <machine_config_pool_name>
     labels:
       pools.operator.machineconfiguration.openshift.io/<machine_config_pool_name>: '' 1
   spec:
     machineConfigSelector:
     matchExpressions:
     - {key: machineconfiguration.openshift.io/role, operator: In, values: [worker,<machine_config_pool_name>]}
     nodeSelector:
     matchLabels:
       node-role.kubernetes.io/<machine_config_pool_name>: ""

   1

   The labels field defines label name to add for Machine config pool(MCP).

4. Verify MCP created successfully.

   $ oc get mcp -w

Procedure

1. The example below uses the ocp4-ocp-allowed-registries-for-import rule, which would fail on a default OpenShift Container Platform installation. Inspect the rule oc get rule.compliance/ocp4-ocp-allowed-registries-for-import -oyaml, the rule is to limit the registries the users are allowed to import images from by setting the allowedRegistriesForImport attribute, The warning attribute of the rule also shows the API object checked, so it can be modified and remediate the issue:

   $ oc edit image.config.openshift.io/cluster

   Example output

   apiVersion: config.openshift.io/v1
   kind: Image
   metadata:
     annotations:
       release.openshift.io/create-only: "true"
     creationTimestamp: "2020-09-10T10:12:54Z"
     generation: 2
     name: cluster
     resourceVersion: "363096"
     selfLink: /apis/config.openshift.io/v1/images/cluster
     uid: 2dcb614e-2f8a-4a23-ba9a-8e33cd0ff77e
   spec:
     allowedRegistriesForImport:
     - domainName: registry.redhat.io
   status:
     externalRegistryHostnames:
     - default-route-openshift-image-registry.apps.user-cluster-09-10-12-07.devcluster.openshift.com
     internalRegistryHostname: image-registry.openshift-image-registry.svc:5000

2. Re-run the scan:

   $ oc annotate compliancescans/<scan_name> compliance.openshift.io/rescan=

Procedure

1. Search for any outdated remediations:

   $ oc get complianceremediations -lcomplianceoperator.openshift.io/outdated-remediation=

   Example output

   NAME                              STATE
   workers-scan-no-empty-passwords   Outdated

   The currently applied remediation is stored in the Outdated attribute and the new, unapplied remediation is stored in the Current attribute. If you are satisfied with the new version, remove the Outdated field. If you want to keep the updated content, remove the Current and Outdated attributes.

2. Apply the newer version of the remediation:

   $ oc patch complianceremediations workers-scan-no-empty-passwords --type json -p '[{"op":"remove", "path":/spec/outdated}]'

3. The remediation state will switch from Outdated to Applied:

   $ oc get complianceremediations workers-scan-no-empty-passwords

   Example output

   NAME                              STATE
   workers-scan-no-empty-passwords   Applied

4. The nodes will apply the newer remediation version and reboot.

Procedure

1. Set the apply flag to false:

   $ oc patch complianceremediations/<scan_name>-sysctl-net-ipv4-conf-all-accept-redirects -p '{"spec":{"apply":false}}' --type=merge

2. The remediation status will change to NotApplied and the composite MachineConfig object would be re-rendered to not include the remediation.

   Important

   All affected nodes with the remediation will be rebooted.

Procedure

1. Locate the scan-name and compliance check for the one-rule-tp-node-master-kubelet-eviction-thresholds-set-hard-imagefs-available remediation:

   $ oc get remediation one-rule-tp-node-master-kubelet-eviction-thresholds-set-hard-imagefs-available -o yaml

   Example output

   apiVersion: compliance.openshift.io/v1alpha1
   kind: ComplianceRemediation
   metadata:
     annotations:
       compliance.openshift.io/xccdf-value-used: var-kubelet-evictionhard-imagefs-available
     creationTimestamp: "2022-01-05T19:52:27Z"
     generation: 1
     labels:
       compliance.openshift.io/scan-name: one-rule-tp-node-master 1
       compliance.openshift.io/suite: one-rule-ssb-node
     name: one-rule-tp-node-master-kubelet-eviction-thresholds-set-hard-imagefs-available
     namespace: openshift-compliance
     ownerReferences:
     - apiVersion: compliance.openshift.io/v1alpha1
       blockOwnerDeletion: true
       controller: true
       kind: ComplianceCheckResult
       name: one-rule-tp-node-master-kubelet-eviction-thresholds-set-hard-imagefs-available
       uid: fe8e1577-9060-4c59-95b2-3e2c51709adc
     resourceVersion: "84820"
     uid: 5339d21a-24d7-40cb-84d2-7a2ebb015355
   spec:
     apply: true
     current:
       object:
         apiVersion: machineconfiguration.openshift.io/v1
         kind: KubeletConfig
         spec:
           kubeletConfig:
             evictionHard:
               imagefs.available: 10% 2
     outdated: {}
     type: Configuration
   status:
     applicationState: Applied

   1

   The scan name of the remediation.

   2

   The remediation that was added to the KubeletConfig objects.

   Note

   If the remediation invokes an evictionHard kubelet configuration, you must specify all of the evictionHard parameters: memory.available, nodefs.available, nodefs.inodesFree, imagefs.available, and imagefs.inodesFree. If you do not specify all parameters, only the specified parameters are applied and the remediation will not function properly.

2. Remove the remediation:

   1. Set apply to false for the remediation object:

      $ oc patch complianceremediations/one-rule-tp-node-master-kubelet-eviction-thresholds-set-hard-imagefs-available -p '{"spec":{"apply":false}}' --type=merge

   2. Using the scan-name, find the KubeletConfig object that the remediation was applied to:

      $ oc get kubeletconfig --selector compliance.openshift.io/scan-name=one-rule-tp-node-master

      Example output

      NAME                                 AGE
      compliance-operator-kubelet-master   2m34s

   3. Manually remove the remediation, imagefs.available: 10%, from the KubeletConfig object:

      $ oc edit KubeletConfig compliance-operator-kubelet-master

      Important

      All affected nodes with the remediation will be rebooted.

Procedure

1. Create the ConfigMap object from a file:

   $ oc create configmap <scan_name> --from-file=tailoring.xml=/path/to/the/tailoringFile.xml

2. Reference the tailoring file in a scan that belongs to a suite:

   apiVersion: compliance.openshift.io/v1alpha1
   kind: ComplianceSuite
   metadata:
     name: workers-compliancesuite
   spec:
     debug: true
     scans:
       - name: workers-scan
         profile: xccdf_org.ssgproject.content_profile_moderate
         content: ssg-rhcos4-ds.xml
         contentImage: quay.io/complianceascode/ocp4:latest
         debug: true
     tailoringConfigMap:
         name: <scan_name>
     nodeSelector:
       node-role.kubernetes.io/worker: ""

Procedure

1. Define the SCC in a YAML file named restricted-adjusted-compliance.yaml:

   SecurityContextConstraints object definition

     allowHostDirVolumePlugin: false
     allowHostIPC: false
     allowHostNetwork: false
     allowHostPID: false
     allowHostPorts: false
     allowPrivilegeEscalation: true
     allowPrivilegedContainer: false
     allowedCapabilities: null
     apiVersion: security.openshift.io/v1
     defaultAddCapabilities: null
     fsGroup:
       type: MustRunAs
     kind: SecurityContextConstraints
     metadata:
       name: restricted-adjusted-compliance
     priority: 30 1
     readOnlyRootFilesystem: false
     requiredDropCapabilities:
     - KILL
     - SETUID
     - SETGID
     - MKNOD
     runAsUser:
       type: MustRunAsRange
     seLinuxContext:
       type: MustRunAs
     supplementalGroups:
       type: RunAsAny
     users:
     - system:serviceaccount:openshift-compliance:api-resource-collector 2
     volumes:
     - configMap
     - downwardAPI
     - emptyDir
     - persistentVolumeClaim
     - projected
     - secret

   1

   The priority of this SCC must be higher than any other SCC that applies to the system:authenticated group.

   2

   Service Account used by Compliance Operator Scanner pod.

2. Create the SCC:

   $ oc create -f restricted-adjusted-compliance.yaml

   Example output

   securitycontextconstraints.security.openshift.io/restricted-adjusted-compliance created

Verification

1. Verify the SCC was created:

   $ oc get scc restricted-adjusted-compliance

   Example output

   NAME                             PRIV    CAPS         SELINUX     RUNASUSER        FSGROUP     SUPGROUP   PRIORITY   READONLYROOTFS   VOLUMES
   restricted-adjusted-compliance   false   <no value>   MustRunAs   MustRunAsRange   MustRunAs   RunAsAny   30         false            ["configMap","downwardAPI","emptyDir","persistentVolumeClaim","projected","secret"]

1. Remove CRDs that were installed by the Compliance Operator:

   1. Switch to the Administration → CustomResourceDefinitions page.
   2. Search for compliance.openshift.io in the Name field.

   3. Click the Options menu next to each of the following CRDs, and select Delete Custom Resource Definition:

      • ComplianceCheckResult
      • ComplianceRemediation
      • ComplianceScan
      • ComplianceSuite
      • ProfileBundle
      • Profile
      • Rule
      • ScanSettingBinding
      • ScanSetting
      • TailoredProfile
      • Variable

2. Remove the OpenShift Compliance project:

   1. Switch to the Home → Projects page.
   2. Click the Options menu next to the openshift-compliance project, and select Delete Project.
   3. Confirm the deletion by typing openshift-compliance in the dialog box, and click Delete.

1. Define your compliance scan requirements
2. Configure the compliance scan settings
3. Process compliance requirements with compliance scans settings
4. Monitor the compliance scans
5. Check the compliance scan results