Chapter 8. SmartState Analysis and OpenSCAP Compliance Checking
SmartState Analysis allows CloudForms to perform a deep introspection of OpenShift Container Platform container images to discover their contents and (for RHEL-based images) determine their OpenSCAP compliance.
SmartState Analysis is alternatively known as "fleecing"
SmartState Analysis of OpenShift Container Platform images uses the functionality of two CloudForms server roles, and an image inspector pod that runs in OpenShift. The first server role - SmartState Analysis - is performed by a Generic or Priority worker, depending on message priority. The second server role - SmartProxy - starts the embedded or coresident.[13] MiqSmartProxyWorker processes.
SmartState scanning of OpenShift Container Platform images is performed by an image-inspector pod that is pulled from a registry as required. The image-inspector dynamically downloads and uses the latest CVE definition/rules file from Red Hat before scanning RHEL-based images.
In CloudForms 4.6 the image-inspector repository, registry and version tag are optionally configurable in the provider’s Advanced Settings page. The proxy settings and CVE location that the image-inspector pod uses to download the rules file is also configurable here (see Figure 8.1, “Provider Advanced Settings”)
Figure 8.1. Provider Advanced Settings
8.1. SmartState Analysis Steps
Performing a SmartState Analysis of a container image involves the following steps:
- The SmartState Analysis worker calls the OpenShift Container Platform API to launch an image-inspector pod, and injects a command such as:
"/usr/bin/image-inspector", "--chroot", "--image=<registry>/<repository>/<image_name>@sha256:...", "--scan-type=openscap", "--serve=0.0.0.0:8080", "--dockercfg=/var/run/secrets/kubernetes.io/ ⏎ inspector-admin-secret-inspector-admin-dockercfg-42mjt/.dockercfg"
The --image switch tells the image-inspector which image to scan. The SmartState Analysis worker then waits for the resulting manageiq-img-scan-<id> pod to complete.
-
The image-inspector fetches the target (
--image) image and extracts it to a temporary directory such as/var/tmp/image-inspector-084630810. - If the scanned image is based on RHEL 5, 6,or 7, the image-inspector container retrieves the version-specific OpenSCAP CVE file such as com.redhat.rhsa-RHEL7.ds.xml.bz2 from the location specified in the advanced configuration (see Figure 8.1, “Provider Advanced Settings”). If not overridden the default location for the CVE file is www.redhat.com/security/data/metrics/ds/.
-
For a RHEL-based image the image-inspector container performs a CVE scan of the contents of the temporary directory and writes the results to another temporary directory such as
/var/tmp/image-inspector-scan-results-689112497. -
The image-inspector container serves the metadata and OpenSCAP results over WebDAV, on the socket specified by the
--serveargument, for examplewebdav://0.0.0.0:8080/api/v1/content/. - The SmartState Analysis worker queues a message for the SmartProxy worker to scan the metadata with a command line argument list similar to the following:
args=["<repository>/<image_name>", ⏎ "--- ⏎ :pod_namespace: management-infra ⏎ :pod_name: manageiq-img-scan-6db1d ⏎ :pod_port: 8080 ⏎ :guest_os: Linux\n"], ⏎ method_name="scan_metadata", ⏎ vm_guid="<registry>/<repository>/<image_name>", ⏎ category="system,software", ⏎ taskid="<task_id_assigned_to_scan>", ⏎ target_id=<container_image_id>, ⏎ target_type="ContainerImage"
- The SmartProxy worker connects to the WebDAV URI and retrieves the extracted pod package list, and if available the OpenSCAP scan results. It stores these in the VMDB with the data model for the container image.
- The SmartState Analysis worker calls the OpenShift Container Platform API to delete the image-inspector pod.
- If the OpenSCAP profile compliance policy profile has been added to the OpenShift Container Platform provider, the image’s compliance is evaluated. If the image is deemed non-compliant it is tagged with the following annotations:
images.openshift.io/deny-execution=true security.manageiq.org/failed-policy=openscap policy
8.2. Monitoring SmartState Analysis
The total time for each image scan can be determined from the time duration between the "request_containerimage_scan" and corresponding "containerimage_scan_complete" events being processed through automate, as follows:
... INFO -- : MIQ(MiqAeEngine.deliver) Delivering ⏎
{:event_type=>:request_containerimage_scan, ⏎
"MiqEvent::miq_event"=>1000001206765, :miq_event_id=>1000001206765, ⏎
"EventStream::event_stream"=>1000001206765, ⏎
:event_stream_id=>1000001206765} for object ⏎
[ContainerImage.1000000000671] with state [] to Automate
...
... INFO -- : MIQ(MiqAeEngine.deliver) Delivering ⏎
{:event_type=>"containerimage_scan_complete", ⏎
"MiqEvent::miq_event"=>1000001206774, :miq_event_id=>1000001206774, ⏎
"EventStream::event_stream"=>1000001206774, ⏎
:event_stream_id=>1000001206774} for object ⏎
[ContainerImage.1000000000671] with state [] to AutomateThis time includes the entire scan sequence of tasks, including image-inspector pod launch, and the time for it to pull the target container image from the registry.
8.3. Challenges of Scale
SmartState Analysis is a relatively time-consuming operation per container image. Many of the problems associated with scaling SmartState Analysis are related to performing many hundreds or thousands of analyses in a limited time window.
8.3.1. Identifying SmartState Analysis Problems
Problems with SmartState Analysis are logged to evm.log, and can usually be identified using the following bash command:
grep ':abort_job' evm.log
8.3.1.2. Failing to Download the image-inspector Container Image
The image-inspector container image is downloaded from the registry.access.redhat.com registry by default. If the OpenShift Container Platform cluster nodes do not have connectivity to this registry then the pod will fail to deploy. An error similar to the following will be seen in evm.log:
:event_type=>"CONTAINER_FAILED", :source=>"KUBERNETES", :timestamp=>"2017-12-14T16:18:45Z", ⏎ :message=>"Failed to pull image \"registry.access.redhat.com/openshift3/image-inspector:2.1\"
An alternative registry can be defined in the advanced settings for the provider (see Figure 8.1, “Provider Advanced Settings”)
8.3.1.3. Failing to Download the OpenSCAP CVE file
By default the OpenSCAP CVE file is downloaded from https://www.redhat.com/security/data/metrics/ds/. If the image-inspector pod does not have direct Internet connectivity, the download will timeout and an error similar to the following will be seen in evm.log:
job finished, Unable to run OpenSCAP: Unable to retreive the CVE file: ⏎ Could not download file https://www.redhat.com/security/data/metrics/ds/ ⏎ com.redhat.rhsa-RHEL7.ds.xml.bz2 ⏎ Get https://www.redhat.com/security/data/metrics/ds/com.redhat.rhsa-RHEL7.ds.xml.bz2: ⏎ dial tcp 23.214.47.223:443: i/o timeout
A proxy server and/or an alternative location for the CVE file can be defined in the advanced settings for the provider (see Figure 8.1, “Provider Advanced Settings”)
8.3.1.4. Non-RHEL Images
If the container image is based on a non-RHEL operating system, the image-inspector container writes the following message:
Unable to run OpenSCAP: Unable to get RHEL distribution number: could not find RHEL dist
8.4. Tuning SmartState Analysis
SmartState Analysis settings are stored in the :container_scanning section of the Configuration → Advanced settings, as follows:
:container_scanning: :scanning_job_timeout: 20.minutes :concurrent_per_ems: 3
The default value of :concurrent_per_ems is 3, which limits the number of concurrent container scans that can be carried out to any particular OpenShift Container Platform provider. This can be increased - with caution - to allow more scans to run concurrently.
8.4.1. Increasing the Number of SmartProxy Workers
The default number of "VM Analysis Collector" (MiqSmartProxyWorker) workers per CFME appliance/pod is 3. This can be increased to a maximum of 5, although consideration should be given to the additional CPU and memory requirements that an increased number of workers will place on an appliance. It may be more appropriate to add further appliances and scale horizontally.
CloudForms installations managing several thousand objects may benefit from dedicated CFME appliances or pods in the provider zones exclusively running the SmartState Analysis and SmartProxy roles.
Where did the comment section go?
Red Hat's documentation publication system recently went through an upgrade to enable speedier, more mobile-friendly content. We decided to re-evaluate our commenting platform to ensure that it meets your expectations and serves as an optimal feedback mechanism. During this redesign, we invite your input on providing feedback on Red Hat documentation via the discussion platform.