Collector pods can't forward logs to Splunk due to weak CA certificate key in RHOCP 4

Solution Verified - Updated -

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4.10+
  • Red Hat OpenShift Logging (RHOL)
    • 5.6+

Issue

  • Collector pods can't send logs to external Splunk when CA certificate key is weak.

  • Collector pods stream below logs:

    2023-03-30T18:57:05.749071Z ERROR vector::internal_events::http_client: HTTP error. error=error trying to connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915:: CA certificate key too weak error_type="request_failed" stage="processing"
2023-03-30T18:57:05.749150Z ERROR vector::topology::builder: msg="Healthcheck: Failed Reason." error=Failed to make HTTP(S) request: error trying to connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915:: CA certificate key too weak component_kind="sink" component_type="splunk_hec" component_id=splunk_receiver component_name=splunk_receiver

Resolution

  • Generate a new CA certificate with key length of at least 2048 bits and then recreate the secret, further inject the CA certificate using ca-bundle.crt literal:

    $ oc project openshift-logging
$ oc create secret generic <secret-name> --from-file=ca-bundle.crt=<ca_bundle_file> --from-literal=hecToken=<HEC_Token>

  • Update clusterlogforwarder instance to have said secret for splunk type output:

    $ oc edit clusterlogforwarder instance -o yaml
----- Output Omitted -----
  outputs:
    - name: splunk-receiver 
      secret:
        name: <secret-name>     <<== Update secret name here 
      type: splunk 
      url: <http://splunk.hec.url:port>
----- Output Omitted -----

Root Cause

Cryptography advancements and the rise of quantum computing have rendered the 1024-bit RSA keys vulnerable to cyberattacks and are considered weak.

Diagnostic Steps

  • Check if collector pods are streaming logs as mentioned in issue section:

    $ oc logs -c collector-n openshift-logging
2023-03-30T18:57:05.749150Z ERROR vector::topology::builder: msg="Healthcheck: Failed Reason." error=Failed to make HTTP(S) request: error trying to connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915:: CA certificate key too weak component_kind="sink" component_type="splunk_hec" component_id=splunk_receiver component_name=splunk_receiver

  • Check the size of public key incorporated by CA certificate:

    $ openssl x509 -in ca-certificate.crt -text -noout | grep "Public Key"
RSA Public Key: (1024 bit)        // 1024 bit keys are considered weak

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments