Troubleshooting Clair

Updated -

Table of Contents

Introduction

  • Red Hat Clair is an open-source vulnerability scanning tool designed to analyze container images for security vulnerabilities. It is commonly used in containerized environments to identify known vulnerabilities in the software components and libraries used within the containers. By detecting vulnerabilities early in the development or deployment process, Red Hat Clair helps ensure the security of containerized applications.

  • Indexing starts with submitting a Manifest to Clair. On receipt, Clair will fetch layers, scan their contents, and return an intermediate representation called an IndexReport. Manifests are Clair's representation of a container image. Clair leverages the fact that OCI Manifests and Layers are content-addressed to reduce duplicated work. Once a Manifest is indexed, the IndexReport is persisted for later retrieval.

  • Matching is taking an IndexReport and correlating vulnerabilities affecting the manifest the report represents. Clair is continually ingesting new security data and a request to the matcher will always provide you with the most up to date vulnerability analysis of an IndexReport.

  • Notifications - Clair implements a notification service. When new vulnerabilities are discovered, the notifier service will determine if these vulnerabilities affect any indexed Manifests. The notifier will then take action according to its configuration.

Troubleshooting Steps

  1. Verify image compatibility: Ensure that the images you are trying to scan are supported by the Clair scanner. Clair has certain requirements and does not support all image formats or configurations.

  2. For Clair behind proxy, confirm if clair updaters list are whitelisted in proxy/firewall setup.

  3. Update Clair scanner and dependencies: Make sure you are using the latest version of the Red Hat Clair scanner. Outdated versions may lack support for newer image formats or have known issues.

  • check clair version from logs:


    $ oc logs clair-pod

$ podman logs clair-container

{"level":"info",
"component":"main",
"version":"v4.5.1",

  1. Enable Debug mode for Clair from clair-config.yaml file and examine logs of each microservice for any error messages or warnings. By default debugging is on, if not changed.


    $ oc exec -it clair-pod-name -- cat /clair/config.yaml

$ podman exec -it clair-container-name cat /clair/config.yaml

http_listen_addr: :8081
introspection_addr: :8088
log_level: debug

  2. From the logs, check if there are errors from the updaters microsevice. Clair usually updates the vulnerability database every 30 minutes, that's the default value.

    $ oc logs clair-pod

$ podman logs clair-container

  3. Updating the CVE database can be a memory and CPU intensitve task, especially if there's a lot of CVEs that need to be parsed. If the resources are exhausted during this process, the system kernel can terminate the offending process. This should be visible either in Docker logs or in the system journal.


      May 14 21:48:14 vm-mtr3-live-k8s-00-ranchernode-4 kernel: [36611.338115] [ pid ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name

  May 14 21:48:14 vm-mtr3-live-k8s-00-ranchernode-4 kernel: [36611.338189] [25745]     0 25745      256        1    32768        0          -998 pause

  May 14 21:48:14 vm-mtr3-live-k8s-00-ranchernode-4 kernel: [36611.338191] [26445]     0 26445     2926      664    61440        0           937 bash

  May 14 21:48:14 vm-mtr3-live-k8s-00-ranchernode-4 kernel: [36611.338192] [26535]     0 26535    27823     3837   249856        0           937 supervisors

  May 14 21:48:14 vm-mtr3-live-k8s-00-ranchernode-4 kernel: [36611.338194] [26555]     0 26555   204146     4138   217088        0           937 jwtproxy

  May 14 21:48:14 vm-mtr3-live-k8s-00-ranchernode-4 kernel: [36611.338195] [26556]     0 26556   734467   386889  4165632        0           937 clair

  May 14 21:48:14 vm-mtr3-live-k8s-00-ranchernode-4 kernel: [36611.338227] Memory cgroup out of memory: Kill process 26556 (clair) score 1922 or sacrifice child

  May 14 21:48:14 vm-mtr3-live-k8s-00-ranchernode-4 kernel: [36611.339573] Killed process 26556 (clair) total-vm:2937868kB, anon-rss:1536364kB, file-rss:11192kB, shmem-rss:0kB

  May 14 21:48:14 vm-mtr3-live-k8s-00-ranchernode-4 kernel: [36611.396171] oom_reaper: reaped process 26556 (clair), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

  4. Check Clair Configuration: Ensure that the configuration parameters are correctly set and aligned with the desired behavior. Check for any misconfigurations or inconsistencies that could lead to issues.


    http_listen_addr: :8081
introspection_addr: :8088
log_level: debug
indexer:
  connstring: host=quay-server.example.com port=5433 dbname=clair user=clairuser password=clairpass sslmode=disable
  scanlock_retry: 10
  layer_scan_concurrency: 5
  migrations: true
matcher:
  connstring: host=quay-server.example.com port=5433 dbname=clair user=clairuser password=clairpass sslmode=disable
  max_conn_pool: 100
  run: ""
  migrations: true
  indexer_addr: clair-indexer
notifier:
  connstring: host=quay-server.example.com port=5433 dbname=clair user=clairuser password=clairpass sslmode=disable
  delivery_interval: 1m
  poll_interval: 5m
  migrations: true
auth:
  psk:
    key: "MTU5YzA4Y2ZkNzJoMQ=="
    iss: ["quay"]
# tracing and metrics
trace:
  name: "jaeger"
  probability: 1
  jaeger:
    agent_endpoint: "localhost:6831"
    service_name: "clair"
metrics:
  name: "prometheus"

  5. Check Clair configuration in Quay config.yaml file: Review the configuration settings for Clair to ensure they are properly set up. Specifically, verify that the correct image registry endpoint or repository is configured, and any necessary authentication credentials are provided.

     ...
 FEATURE_SECURITY_SCANNER
 FEATURE_SECURITY_NOTIFICATIONS
 SECURITY_SCANNER_V4_REINDEX_THRESHOLD
 SECURITY_SCANNER_V4_ENDPOINT
 SECURITY_SCANNER_V4_PSK
 SECURITY_SCANNER_INDEXING_INTERVAL
 SECURITY_SCANNER_ENDPOINT
 SECURITY_SCANNER_INDEXING_INTERVAL
 ...

  6. Inspect image metadata: The unsupported message might indicate that the scanner is unable to extract the necessary metadata from the image. Check if the image metadata is properly formatted and accessible.

Known Issues

Comments