Chapter 6. Clair Security Scanning
6.1. What is Clair?
Clair is a set of micro services that can be used with Red Hat Quay to perform vulnerability scanning of container images associated with a set of Linux operating systems. The micro services design of Clair makes it appropriate to run in a highly scalable configuration, where components can be scaled separately as appropriate for enterprise environments.
Clair uses the following vulnerability databases to scan for issues in your images:
- Alpine SecDB database
- AWS UpdateInfo
- Debian Oval database
- Oracle Oval database
- RHEL Oval database
- SUSE Oval database
- Ubuntu Oval database
- Pyup.io (python) database
For information on how Clair does security mapping with the different databases, see ClairCore Severity Mapping.
With the release of Red Hat Quay 3.4, the new Clair V4 (image registry.redhat.io/quay/clair-rhel8 fully replaces the prior Clair V2 (image quay.io/redhat/clair-jwt). See below for how to run V2 in read-only mode while V4 is updating.
6.1.1. Running Clair V4 and Clair V2 Simultaneously
While Clair V4 (registry.redhat.io/quay/clair-rhel8:v3.5.0) is the version of Clair that Red Hat Quay uses, both it and the prior Clair V2 (quay.io/redhat/clair-jwt) can run concurrently with Red Hat Quay. This is useful for existing Red Hat Quay deployments that have relied on Clair V2 but wish to have no interruption of scan results using Clair V4. All new image scans will happen in Clair V4 and existing images will be re-scanned automatically. When scan results are requested through Red Hat Quay, if the new Clair V4 results are not available, the existing Clair V2 results will be retrieved. Once the Clair V2 scan results are not needed, it may be decommissioned and removed from Red Hat Quay’s configuration.
The progress of rescanning images may be monitored via Red Hat Quay API. (Refer to Using The Quay API for details):
This will produce a simple JSON response with the percentage of completed manifests scanned by Clair V4:
Once the majority of the images in your registry have been scanned by Clair V4, the Clair V2 deployment should be removed entirely (both running containers and removal from config).
6.2. Setting Up Clair on a Red Hat Quay OpenShift deloyment
6.2.1. Deploying Via the Quay Operator
To set up Clair V4 on a new Red Hat Quay deployment on OpenShift, it is highly recommended to use the Quay Operator. By default, the Quay Operator will install or upgrade a Clair deployment along with your Red Hat Quay deployment and configure Clair security scanning automatically.
6.2.2. Manually Deploying Clair
To configure Clair V4 on an existing Red Hat Quay OpenShift deployment running Clair V2, first ensure Red Hat Quay has been upgraded to at least version 3.4.0. Then use the following steps to manually set up Clair V4 alongside Clair V2.
Set your current project to the name of the project in which Red Hat Quay is running. For example:
$ oc project quay-enterprise
Create a Postgres deployment file for Clair v4 (for example,
clairv4-postgres.yaml) as follows.
--- apiVersion: apps/v1 kind: Deployment metadata: name: clairv4-postgres namespace: quay-enterprise labels: quay-component: clairv4-postgres spec: replicas: 1 selector: matchLabels: quay-component: clairv4-postgres template: metadata: labels: quay-component: clairv4-postgres spec: volumes: - name: postgres-data persistentVolumeClaim: claimName: clairv4-postgres containers: - name: postgres image: postgres:11.5 imagePullPolicy: "IfNotPresent" ports: - containerPort: 5432 env: - name: POSTGRES_USER value: "postgres" - name: POSTGRES_DB value: "clair" - name: POSTGRES_PASSWORD value: "postgres" - name: PGDATA value: "/etc/postgres/data" volumeMounts: - name: postgres-data mountPath: "/etc/postgres" --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: clairv4-postgres labels: quay-component: clairv4-postgres spec: accessModes: - "ReadWriteOnce" resources: requests: storage: "5Gi" volumeName: "clairv4-postgres" --- apiVersion: v1 kind: Service metadata: name: clairv4-postgres labels: quay-component: clairv4-postgres spec: type: ClusterIP ports: - port: 5432 protocol: TCP name: postgres targetPort: 5432 selector: quay-component: clairv4-postgres
Deploy the postgres database as follows:
$ oc create -f ./clairv4-postgres.yaml
Create a Clair
config.yamlfile to use for Clair v4. For example:
introspection_addr: :8089 http_listen_addr: :8080 log_level: debug indexer: connstring: host=clairv4-postgres port=5432 dbname=clair user=postgres password=postgres sslmode=disable scanlock_retry: 10 layer_scan_concurrency: 5 migrations: true matcher: connstring: host=clairv4-postgres port=5432 dbname=clair user=postgres password=postgres sslmode=disable max_conn_pool: 100 run: "" migrations: true indexer_addr: clair-indexer # tracing and metrics trace: name: "jaeger" probability: 1 jaeger: agent_endpoint: "localhost:6831" service_name: "clair" metrics: name: "prometheus"
More information about Clair’s configuration format can be found in upstream Clair documentation.
Create a secret from the Clair
$ oc create secret generic clairv4-config-secret --from-file=./config.yaml
Create the Clair v4 deployment file (for example,
clair-combo.yaml) and modify it as necessary:
--- apiVersion: extensions/v1beta1 kind: Deployment metadata: labels: quay-component: clair-combo name: clair-combo spec: replicas: 1 selector: matchLabels: quay-component: clair-combo template: metadata: labels: quay-component: clair-combo spec: containers: - image: registry.redhat.io/quay/clair-rhel8:v3.5.0 1 imagePullPolicy: IfNotPresent name: clair-combo env: - name: CLAIR_CONF value: /clair/config.yaml - name: CLAIR_MODE value: combo ports: - containerPort: 8080 name: clair-http protocol: TCP - containerPort: 8089 name: clair-intro protocol: TCP volumeMounts: - mountPath: /clair/ name: config imagePullSecrets: - name: redhat-pull-secret restartPolicy: Always volumes: - name: config secret: secretName: clairv4-config-secret --- apiVersion: v1 kind: Service metadata: name: clairv4 2 labels: quay-component: clair-combo spec: ports: - name: clair-http port: 80 protocol: TCP targetPort: 8080 - name: clair-introspection port: 8089 protocol: TCP targetPort: 8089 selector: quay-component: clair-combo type: ClusterIP
Create the Clair v4 deployment as follows:
$ oc create -f ./clair-combo.yaml
config.yamlfile for your Red Hat Quay deployment to add the following entries at the end:
FEATURE_SECURITY_SCANNER: true SECURITY_SCANNER_V4_ENDPOINT: http://clairv4 1
- Identify the Clair v4 service endpoint
Redeploy the modified
config.yamlto the secret containing that file (for example,
$ oc delete secret quay-enterprise-config-secret $ oc create secret generic quay-enterprise-config-secret --from-file=./config.yaml
For the new
config.yamlto take effect, you need to restart the Red Hat Quay pods. Simply deleting the
quay-apppods causes pods with the updated configuration to be deployed.
At this point, images in any of the organizations identified in the namespace whitelist will be scanned by Clair v4.
6.3. Setting up Clair on a non-OpenShift Red Hat Quay deployment
For Red Hat Quay deployments not running on OpenShift, it is possible to configure Clair security scanning manually. Red Hat Quay deployments already running Clair V2 can use the instructions below to add Clair V4 to their deployment.
Deploy a (preferably fault-tolerant) Postgres database server. Note that Clair requires the
uuid-osspextension to be added to its Postgres database. If the user supplied in Clair’s
config.yamlhas the necessary privileges to create the extension then it will be added automatically by Clair itself. If not, then the extension must be added before starting Clair. If the extension is not present, the following error will be displayed when Clair attempts to start.
ERROR: Please load the "uuid-ossp" extension. (SQLSTATE 42501)
Create a Clair config file in a specific folder (e.g.
introspection_addr: :8089 http_listen_addr: :8080 log_level: debug indexer: connstring: host=clairv4-postgres port=5432 dbname=clair user=postgres password=postgres sslmode=disable scanlock_retry: 10 layer_scan_concurrency: 5 migrations: true matcher: connstring: host=clairv4-postgres port=5432 dbname=clair user=postgres password=postgres sslmode=disable max_conn_pool: 100 run: "" migrations: true indexer_addr: clair-indexer notifier: connstring: host=clairv4-postgres port=5432 dbname=clair user=postgres password=postgres sslmode=disable delivery_interval: 1m poll_interval: 5m migrations: true # tracing and metrics trace: name: "jaeger" probability: 1 jaeger: agent_endpoint: "localhost:6831" service_name: "clair" metrics: name: "prometheus"
More information about Clair’s configuration format can be found in upstream Clair documentation.
Run Clair via the container image, mounting in the configuration from the file you created.
$ podman run -p 8080:8080 -p 8089:8089 -e CLAIR_CONF=/clair/config.yaml -e CLAIR_MODE=combo -v /etc/clair4/config:/clair -d registry.redhat.io/quay/clair-rhel8:v3.5.0
- Follow the remaining instructions from the previous section for configuring Red Hat Quay to use the new Clair V4 endpoint.
Running multiple Clair containers in this fashion is also possible, but for deployment scenarios beyond a single container the use of a container orchestrator like Kubernetes or OpenShift is strongly recommended.
6.4. Using Clair
- Log in to your Red Hat Quay cluster and select an organization for which you have configured Clair scanning.
Select a repository from that organization that holds some images and select Tags from the left navigation. The following figure shows an example of a repository with two images that have been scanned:
If vulnerabilities are found, select to under the Security Scan column for the image to see either all vulnerabilities or those that are fixable. The following figure shows information on all vulnerabilities found:
6.5. Clair Notifications
When Clair received a new vulnerability affecting a previously indexed manifest, it will notify Red Hat Quay so that a new scan can be requested. Only the most severe vulnerabilities trigger a notification to avoid excessive scan requests. This notification mechanism is automatically set up when Clair is configured in Red Hat Quay’s configuration.
Clair notifications can also be set up for external consumption via AMQP and STOMP protocols. For details on how to set this up please consult the upstream Clair documentation.
6.6. Configuring Clair for Disconnected Environments
Clair utilizes a set of components called Updaters to handle the fetching and parsing of data from various vulnerability databases. These Updaters are set up by default to pull vulnerability data directly from the internet and work out of the box. For customers in disconnected environments without direct access to the internet this poses a problem. Clair supports these environments through the ability to work with different types of update workflows that take into account network isolation. Using the
clairctl command line utility, any process can easily fetch Updater data from the internet via an open host, securely transfer the data to an isolated host, and then import the Updater data on the isolated host into Clair itself.
The steps are as follows.
First ensure that your Clair configuration has disabled automated Updaters from running.
matcher: disable_updaters: true
Export out the latest Updater data to a local archive. This requires the
clairctltool which can be run directly as a binary, or via the Clair container image. Assume your Clair configuration is in
/etc/clairv4/config/config.yaml, to run via the container image:
$ podman run -it --rm -v /etc/clairv4/config:/cfg:Z -v $(pwd):/updaters:Z --entrypoint /bin/clairctl registry.redhat.io/quay/clair-rhel8:v3.5.0 export-updaters --config /cfg/config.yaml /updaters/updaters.gz
Note that you need to explicitly reference the Clair configuration. This will create the Updater archive in
/etc/clairv4/updaters/updaters.gz. If you want to ensure the archive was created without any errors from the source databases, you can supply the
clairctl. The archive file should be copied over to a volume that is accessible from the disconnected host running Clair. From the disconnected host, use the same procedure now to import the archive into Clair.
$ podman run -it --rm -v /etc/clairv4/config:/cfg:Z -v $(pwd):/updaters:Z --entrypoint /bin/clairctl registry.redhat.io/quay/clair-rhel8:v3.5.0 import-updaters --config /cfg/config.yaml /updaters/updaters.gz
6.7. Clair updater URLs
The following are the HTTP hosts and paths that Clair will attempt to talk to in a default configuration. This list is non-exhaustive, as some servers will issue redirects and some request URLs are constructed dynamically.