Integration with OpenShift Container Platform
Adding Red Hat OpenShift Container Platform (with Metrics Enabled) as a Container Provider
Abstract
Chapter 1. Overview
This guide walks you through adding a Red Hat OpenShift Container Platform cluster to a Red Hat CloudForms container provider catalog. This deployment focuses on enabling the OpenShift Container Platform cluster metrics plug-in, so that CloudForms can collect information from an OpenShift Container Platform cluster upon integration.
Each procedure in this guide is covered in greater detail in the Red Hat CloudForms and OpenShift Container Platform product documentation. However, links to the corresponding sections are provided for more detail.
The following sections will describe the required configuration for both products prior to integration.
Chapter 2. Prerequisites
This guide assumes that you have:
When enabling metrics on OpenShift Container Platform, you can store your metrics data on persistent or non-persistent storage. To use persistent storage, you need to provision a persistent volume specifically for this purpose before configuring the metrics components. See Persistent Volumes in the OpenShift Container Platform Architecture documentation for more information.
Chapter 3. Enabling OpenShift Container Platform Metrics
In order for CloudForms to collect OpenShift Container Platform node, pod, and container, you must first enable cluster metrics for your OpenShift cluster. This involves running the OpenShift Metrics services inside your cluster. If cluster metrics are already enabled on OpenShift, skip this section and proceed to Chapter 4, Retrieving the OpenShift Container Platform Management Token.
This section is an abridged version of a more detailed chapter, namely Enabling Cluster Metrics from the OpenShift Container Platform Installation and Configuration documentation. Refer to that chapter for more information.
3.1. Configuring the Required Service Accounts
Cluster metrics requires the following service accounts:
- metrics-deployer
- heapster
If you deployed OpenShift using openshift-ansible-3.0.20
, then the service account and roles required for enabling metrics will already be installed. You can skip this section and go to Section 3.2, “Configuring Metrics Components”.
To create these accounts:
- Log in as an administrator to any node within the OpenShift Container Platform cluster.
- Open a terminal.
Switch to the openshift-infra project:
$ oc project openshift-infra
Create a service account for the Metrics Deployer named metrics-deployer:
$ oc create -f - <<API apiVersion: v1 kind: ServiceAccount metadata: name: metrics-deployer secrets: - name: metrics-deployer API
As described in Section 3.2, “Configuring Metrics Components”, the Metrics Deployer uses the metrics-deployer service account. Configure the metrics-deployer account to have edit permissions in the openshift-infra project:
$ oadm policy add-role-to-user \ edit system:serviceaccount:openshift-infra:metrics-deployer
The heapster account will be automatically created in Section 3.2.1, “Deploying the Metrics Components”. However, pre-emptively grant it cluster-reader permission to the openshift-infra project:
$ oadm policy add-cluster-role-to-user \ cluster-reader system:serviceaccount:openshift-infra:heapster
3.2. Configuring Metrics Components
The Metrics Deployer installs and configures the components required for OpenShift Container Platform metrics. By default, the Metrics Deployer uses self-signed certificates to secure communication between components. This document assumes that you will use this default. For information on alternative secure communication configurations, see Using Secrets from the OpenShift Container Platform Installation and Configuration documentation.
This section is an abridged version of a more detailed chapter, namely Metrics Data Storage in the OpenShift Container Platform Installation and Configuration documentation. Refer to that chapter for more information on how to deploy the metrics plug-in using persistent and non-persistent storage.
3.2.1. Deploying the Metrics Components
OpenShift Container Platform uses Hawkular Metrics as its metrics engine. The Metrics Deployer will install the Hawkular Metrics service; however, you need to provide the external hostname so that CloudForms can reach the Hawkular Metrics service. The base configuration of the Metrics Deployer is defined in the /usr/share/openshift/examples/infrastructure-templates/enterprise/metrics-deployer.yaml
file.
Before deploying OpenShift metrics, choose a storage option, then log in as an administrator to any node within the OpenShift Container Platform cluster. From there, open a terminal and run the corresponding command:
- Deploying with persistent storage
With persistent storage, OpenShift metrics will be stored on a persistent volume. This offers metrics data protection by allowing it to survive a pod recreation or restart. OpenShift metrics requires a specifically configured persistent volume; see Persistent Volumes in the OpenShift Container Platform Architecture documentation.
$ oc new-app \ -f /usr/share/openshift/examples/infrastructure-templates/enterprise/metrics-deployer.yaml \ -p HAWKULAR_METRICS_HOSTNAME=HAWKULARHOST
- Deploying with non-persistent storage
With non-persistent storage, any stored metrics will be deleted when the pod is deleted. While it is much easier to run cluster metrics with non-persistent data, it does come with the risk of permanent data loss. So, while you no longer need to provision and configure a volume to store metric data, this does not offer the same protection as persistent storage.
$ oc new-app \ -f /usr/share/openshift/examples/infrastructure-templates/enterprise/metrics-deployer.yaml \ -p HAWKULAR_METRICS_HOSTNAME=HAWKULARHOST \ -p USE_PERSISTENT_STORAGE=false
For either command, replace HAWKULARHOST with the external hostname that CloudForms will use to reach the Hawkular Metrics service. HAWKULARHOST must be a fully-qualified domain name.
Either storage method deploys the required metrics components and creates the necessary service accounts. In particular, the metrics components will be configured to also use the specified HAWKULARHOST as its public endpoint.
3.2.2. Applying the Hawkular Metrics Settings to OpenShift Container Platform
After deploying the metrics components, configure OpenShift Container Platform to use them:
Open the OpenShift Master Configuration file at
/etc/origin/master/master-config.yaml
. Add the metricsPublicURL parameter to the assetConfig section, specifying the HAWKULARHOST you specified in Section 3.2.1, “Deploying the Metrics Components”:assetConfig: ... metricsPublicURL: "https://HAWKULARHOST/hawkular/metrics"
Restart your OpenShift Container Platform master host:
$ sudo systemctl restart atomic-openshift-master
Chapter 4. Retrieving the OpenShift Container Platform Management Token
After enabling cluster metrics on your OpenShift Container Platform deployment, retrieve the management token while you are still logged in to the OpenShift Container Platform host. This will be required later in Chapter 6, Adding OpenShift Container Platform as a Container Provider.
- For OpenShift Container Platform 3.3 or later
- Provide the token needed to add an OpenShift Container Platform 3.3 (or later) provider.
Run the following to obtain the token needed to add an OpenShift Container Platform 3.3 (or later) provider:
# oc sa get-token -n management-infra management-admin eyJhbGciOiJSUzI1NiI...
- For OpenShift Enterprise 3.2
- Provide the token needed to add an OpenShift Enterprise 3.2 provider.
Run the following to obtain the token needed to add an OpenShift Enterprise 3.2 provider:
# oc sa get-token -n management-infra management-admin eyJhbGciOiJSUzI1NiI...
- For OpenShift Enterprise 3.1
- Provide the token needed to add an OpenShift Enterprise 3.1 provider.
Run the following to obtain the token needed to add an OpenShift Enterprise 3.1 provider:
Obtain the
management
service account token name:# oc describe sa -n management-infra management-admin ... Tokens: management-admin-token-0f3fh management-admin-token-q7a87
Select and describe one of the tokens to retrieve the full token output, replacing
management-admin-token-0f3fh
with the name of your token:# oc describe secret -n management-infra management-admin-token-0f3fh ... Data ==== token: eyJhbGciOiJSUzI1NiI...
Chapter 5. Configuring Red Hat CloudForms
Configuring CloudForms involves two steps:
These steps are required to allow CloudForms to collect metrics from OpenShift Container Platform (Chapter 3, Enabling OpenShift Container Platform Metrics) and use them to perform a SmartState analysis. You can choose different servers to perform either function; the following sections assume that you will.
5.1. Configuring CloudForms Capacity and Utilization
For metrics collection to work properly, you also need to configure Red Hat CloudForms to allow for all three Capacity & Utilization server roles, which are available from the settings menu under → → . For more information on capacity and utilization collection, see Assigning the Capacity and Utilization Server Roles in the Deployment Planning Guide.
To enable these server roles:
- From the settings menu, select Configuration, then select the server to configure from → in the accordion menu on the left.
Navigate to the Server Roles list in the → section. From there, set the required capacity and utilization roles to ON, namely:
- Capacity & Utilization Coordinator
- Capacity & Utilization Data Collector
- Capacity & Utilization Data Processor
- Click Save.
Data collection is enabled immediately. However, the first collection begins 5 minutes after the server is started, and every 10 minutes after that. Therefore, the longest the collection takes after enabling the Capacity & Utilization Collector role is 10 minutes. The first collection from a particular provider may take a few minutes since Red Hat CloudForms is gathering data points going one month back in time.
For more information, see Capacity and Utilization Collection in the Deployment Planning Guide.
5.2. Enabling SmartState Analysis
After enabling the required server roles, enable SmartState analysis. See Smart State Analysis Support from the Support Matrix and Running a SmartState Analysis in the Managing Providers guide for more information.
Enabling SmartState analysis is similar to Section 5.1, “Configuring CloudForms Capacity and Utilization”, in that the procedure also involves enabling server roles on a specific server. To do so:
- From the settings menu, select Configuration.
- Select the server to configure from → in the left pane of the appliance.
Navigate to the Server Roles list in the → section. From there, set the appropriate SmartState roles to ON. Namely:
- SmartProxy
- SmartState Analysis
- Click Save.
Chapter 6. Adding OpenShift Container Platform as a Container Provider
At this point, you should now be ready to add your OpenShift cluster to Red Hat CloudForms as a container provider. To do so, prepare the token you retrieved in Chapter 4, Retrieving the OpenShift Container Platform Management Token and follow the procedure below:
- Navigate to → → .
-
Click
(Configuration), then click
(Add a New Containers Provider).
- Enter a Name for the provider.
- From the Type list, select OpenShift Container Platform.
-
Enter the appropriate Zone for the provider. If you do not specify a zone, it is set to
default
. - From the Alerts list, select Prometheus to enable external alerts. Selecting Prometheus adds an Alerts tab to the lower pane to configure the Prometheus service. Alerts are disabled by default.
- From the Metrics list, select Hawkular or Prometheus to collect capacity and utilization data, or leave as Disabled. Selecting Prometheus or Hawkular adds a Metrics tab to the lower pane for further configuration. Metrics are disabled by default.
In the Default tab, configure the following for the OpenShift provider:
Select a Security Protocol method to specify how to authenticate the provider:
- SSL: Authenticate the provider securely using a trusted Certificate Authority. Select this option if the provider has a valid SSL certificate and it is signed by a trusted Certificate Authority. No further configuration is required for this option.
SSL trusting custom CA: Authenticate the provider with a self-signed certificate. For this option, copy your provider’s CA certificate to the Trusted CA Certificates box in PEM format.
NoteYou can obtain your OpenShift Container Platform provider’s CA certificate for all endpoints (default, metrics, alerts) from
/etc/origin/master/ca.crt
. Paste the output (a block of text starting with-----BEGIN CERTIFICATE-----
) into the Trusted CA Certificates field.- SSL without validation: Authenticate the provider insecurely (not recommended).
Enter the Hostname (or IPv4 or IPv6 address) of the provider.
ImportantThe Hostname must use a unique fully qualified domain name.
-
Enter the API Port of the provider. The default port is
8443
. Enter a token for your provider in the Token box.
NoteTo obtain a token for your provider, run the
oc get secret
command on your provider; see Obtaining an OpenShift Container Platform Management Token.For example:
# oc get secret --namespace management-infra management-admin-token-8ixxs --template='{{index .data "ca.crt"}}' | base64 --decode
- Click Validate to confirm that Red Hat CloudForms can connect to the OpenShift Container Platform provider.
If you selected a metrics service, configure the service details in the Metrics tab:
Select a Security Protocol method to specify how to authenticate the service:
- SSL: Authenticate the provider securely using a trusted Certificate Authority. Select this option if the provider has a valid SSL certificate and it is signed by a trusted Certificate Authority. No further configuration is required for this option.
SSL trusting custom CA: Authenticate the provider with a self-signed certificate. For this option, copy your provider’s CA certificate to the Trusted CA Certificates box in PEM format.
NoteIn OpenShift, the default deployment of the router generates certificates during installation, which can be used with the SSL trusting custom CA option. Connecting a Hawkular endpoint with this option requires the CA certificate that the cluster uses for service certificates, which is stored in
/etc/origin/master/service-signer.crt
on the first master in a cluster.- SSL without validation: Authenticate the provider insecurely using SSL. (Not recommended)
- Enter the Hostname (or IPv4 or IPv6 address) of the provider, or use the Detect button to find the hostname.
-
Enter the API Port if your Hawkular provider uses a non-standard port for access. The default port is
443
. - Click Validate to confirm that Red Hat CloudForms can connect to the metrics endpoint.
For the Prometheus alerts service, add the Prometheus alerts endpoint in the Alerts tab:
Select a Security Protocol method to specify how to authenticate the service:
- SSL: Authenticate the provider securely using a trusted Certificate Authority. Select this option if the provider has a valid SSL certificate and it is signed by a trusted Certificate Authority. No further configuration is required for this option.
- SSL trusting custom CA: Authenticate the provider with a self-signed certificate. For this option, copy your provider’s CA certificate to the Trusted CA Certificates box in PEM format.
- SSL without validation: Authenticate the provider insecurely using SSL. (Not recommended)
- Enter the Hostname (or IPv4 or IPv6 address) or alert Route.
-
Enter the API Port if your Prometheus provider uses a non-standard port for access. The default port is
443
. - Click Validate to confirm that CloudForms can connect to the alerts service.
Click the Advanced tab to add image inspector settings for scanning container images on your provider using OpenSCAP.
Note- These settings control downloading the image inspector container image from the registry and obtaining the Common Vulnerabilities and Exposures (CVE) information (for effective scanning) via a proxy.
- CVE URL that CloudForms requires to be open for OpenSCAP scanning: https://www.redhat.com/security/data/metrics/ds/. This information is based on the source code of OpenSCAP.
- Enter the proxy information for the provider in either HTTP, HTTPS, or NO Proxy depending on your environment.
-
Enter the Image-Inspector Repository information. For example,
openshift3/image-inspector
. -
Enter the Image-Inspector Registry information. For example,
registry.access.redhat.com
. - Enter the Image-Inspector Tag value. A tag is a mark used to differentiate images in a repository, typically by the application version stored in the image.
-
Enter
https://www.redhat.com/security/data/metrics/ds/
in CVE location.
- Click Add.
You can also set global default image-inspector settings for all OpenShift providers in the advanced settings menu by editing the values under ems_kubernetes
, instead of setting this for each provider.
For example:
:image_inspector_registry: registry.access.redhat.com :image_inspector_repository: openshift3/image-inspector
Chapter 7. Container Image Scanning
7.1. Configuring Image Scanning
Red Hat CloudForms manages vulnerability scanning of container images. When an OpenShift provider is added, OpenShift images from the internal registry are discovered. To enable image scanning, perform the following configuration steps:
- Navigate to → → .
- Select the checkboxes of the OpenShift providers on which to enable scanning.
- From the Policy pull-down menu, click Manage Policies.
- Select the OpenSCAP profile checkbox.
- Click Save.
This action will trigger a SmartState analysis, or scan, of all images referenced by the OpenShift provider. The initial scan may take several hours to complete, depending on the number and size of images. The scan occurs in the OpenShift provider, which CloudForms receives and records in the database. OpenShift limits the number of scanning pods; only three images can be scanned simultaneously.
7.2. Scheduling A Recurring Scan
Software vulnerability databases are updated frequently. To apply these updates, a rescan is required. To schedule a recurring scan of container images:

- From the settings menu, select Configuration.
- From Schedules. → in the left pane of the appliance, select
- From the drop-down menu, click → .
- Type an arbitrary Name.
- Type an arbitrary Description.
- Ensure the Active checkbox is selected.
- In Action, select Container Image Analysis.
- In Filter, select All Container Images for Containers Provider, OpenShift.
- In Run, set the schedule as desired.
- Set the Time Zone, Starting Date, and Starting Time.
- Click Add.
7.3. Working with Images
7.3.1. Viewing Results
Image scanning results are displayed in each image summary page.
- Select → → .
- Click the desired image.
For an OpenSCAP HTML report, locate the Configuration section and select OpenSCAP HTML.

For compliance and scanning history information, locate the Compliance section and note the Status field or select Available from the History field.

7.3.2. Manual Scanning
SmartState analysis scanning may be initiated manually for images. From an image summary page, select
→ . Refreshing the image page will reflect the latest scan results and compliance history.7.3.3. Evaluating Compliance
If the image scan policy has been updated since the last scan, compliance conditions may be re-evaluated. From an image summary page, select
→ . Refreshing the image page will reflect the latest compliance history.7.3.4. Generating a Report on Images
You can output the results of an OpenSCAP scan of images to a report for an overview of the security risk level of images. The Images by Failed OpenSCAP Rule Results is included with CloudForms and shows whether the image has passed or failed OpenSCAP policy criteria, and the security risk.
You can also create a copy of this report and edit it to contain additional information, such as the project name where the image is used, to produce more useful results. See Editing a Report and See Reportable Fields in Red Hat CloudForms in Monitoring, Alerts, and Reporting for instructions on customizing reports.
To create a report showing image compliance:
- Navigate to → .
- Click the → accordion.
- Navigate to → → for a report showing which images have failed the OpenSCAP compliance.
-
Click
Queue.
The report generation is placed in the queue and its status shows in the reports page.
-
Click
(Refresh this page) to update the status.
- Navigate to the Saved Reports accordion, and click the report when it is completed.
Click on the report download buttons for the type of export you want. The report is automatically named with the type of report and date.
-
Click
(Download this report in text format) to download as text.
-
Click
(Download this report in CSV format) to download as a comma-separated file.
-
Click
(Download this report in PDF format) to download as PDF.
-
Click
7.4. OpenSCAP Policy Profile
Red Hat CloudForms is pre-configured with a default scanning policy profile. This includes conditions to scan and identify compliance, as well as annotate compliance failure. SmartState analysis is performed when new images are added to OpenShift.
7.4.1. Customizing the Scanning Policy Profile
The built-in OpenSCAP policy profile cannot be edited. You can, however, assign edited copies of its policies to a new policy profile. This will allow you to create a customized version of the built-in OpenSCAP policy profile.
To do so, you will first have to copy the policy you want to customize:
- Navigate to → .
- Click the Policies accordion, and select Container Image Compliance Policies, then click OpenSCAP.
-
Click
(Configuration), and an option to copy the policy should appear; for example,
(Copy this Container Image Policy).
- Click OK to confirm.
The new policy is created with a prefix of Copy of in its description, and it can be viewed in the Policies accordion.
You can now edit the copied policy. After editing copied policies, you can add them to a new policy profile. For instructions on how to edit policies, create a new policy profile, and add policies to it, see the Policies and Profiles guide. Once you have a customized policy profile, you can assign it to a containers provider.
7.5. Controlling OpenShift Pod Execution
Through the default policy profile, non-compliant images receive the control policy action Mark as Non-Compliant. This action annotates the image object (not to be confused with the imagestream object) with images.openshift.io/deny-execution=true. This annotation may be used to prevent nodes from running non-compliant images. Refer to the OpenShift Container Platform Image Policy documentation for configuration details.
7.6. Reference
More information about OpenSCAP, see visit the OpenSCAP web site.