AMQ Streams Operator KafkaMirrorMaker2 NullPointerException

Solution Verified - Updated -

Environment

  • Red Hat AMQ Streams
    • 2.3
  • Red Hat OpenShift Container Platform (RHOCP)
    • 4
  • Red Hat OpenShift on AWS (ROSA)
    • 4
  • Red Hat OpenShift Dedicated (OSD)
    • 4
  • Azure Red Hat OpenShift (ARO)
    • 4

Issue

  • AMQ Stream KafkaMirrorMaker2 instance status NotReady with reason NullPointerException.

    status:
  conditions:
    - lastTransitionTime: '2023-02-09T01:16:46.440400Z'
      reason: NullPointerException
      status: 'True'
      type: NotReady
  labelSelector: >-
    strimzi.io/cluster=npe-cluster-mirror-maker-2,strimzi.io/name=npe-cluster-mirror-maker-2-mirrormaker2,strimzi.io/kind=KafkaMirrorMaker2

  • Checking KafkaMirrorMaker2 pod logs and found error logs

    2023-02-09 00:56:46 ERROR AbstractOperator:247 - Reconciliation #41(watch) KafkaMirrorMaker2(kafka/npe-cluster-mirror-maker-2): createOrUpdate failed
java.lang.NullPointerException: null
  at io.strimzi.operator.common.Util.lambda$authTlsHash$6(Util.java:547) ~[io.strimzi.operator-common-0.28.0.redhat-00003.jar:0.28.0.redhat-00003]
...omitted...
2023-02-09 00:56:47 INFO  OperatorWatcher:38 - Reconciliation #42(watch) KafkaMirrorMaker2(kafka/npe-cluster-mirror-maker-2): KafkaMirrorMaker2 npe-cluster-mirror-maker-2 in namespace kafka was MODIFIED
2023-02-09 00:56:47 INFO  CrdOperator:113 - Reconciliation #41(watch) KafkaMirrorMaker2(kafka/npe-cluster-mirror-maker-2): Status of KafkaMirrorMaker2 npe-cluster-mirror-maker-2 in namespace kafka has been updated
2023-02-09 00:56:47 WARN  AbstractOperator:532 - Reconciliation #41(watch) KafkaMirrorMaker2(kafka/npe-cluster-mirror-maker-2): Failed to reconcile
java.lang.NullPointerException: null
  at io.strimzi.operator.common.Util.lambda$authTlsHash$6(Util.java:547) ~[io.strimzi.operator-common-0.28.0.redhat-00003.jar:0.28.0.redhat-00003]
...omitted...

Resolution

Check if TLS certificate secret names are correct or missing contents.

For example:

spec:
    clusters:
      - alias: target
        bootstrapServers: 'target-kafka-bootstrap:9093'
        tls:
          trustedCertificates:
            - certificate: ca.crt
              secretName: target-cluster-ca-cert
      - alias: mirror
        authentication:
          certificateAndKey:
            certificate: user.crt
            key: user.key
            secretName: mirror-mirror-maker-2-target-user
          type: tls
        bootstrapServers: 'mirror-kafka-brokers:9093'
        config:
          config.storage.replication.factor: -1
          offset.storage.replication.factor: -1
          status.storage.replication.factor: -1
        tls:
          trustedCertificates:
            - certificate: ca.crt
              secretName: mirror-cluster-ca

secretName: mirror-cluster-ca's correct name should be secretName: mirror-cluster-ca-cert

Root Cause

TLS certificate missing or wrong values.

Diagnostic Steps

Checking KafkaMirrorMaker2 pod logs and found error logs

java.lang.NullPointerException: null
    at io.strimzi.operator.common.Util.lambda$authTlsHash$6(Util.java:547)

This error message suggests that a NullPointerException occurred in the authTlsHash method of the Util class in the io.strimzi.operator.common package.

Looking at the source code the authTlsHash method generates an SHA-256 hash of the TLS authentication configuration for a Kafka client.

Source Code link: https://github.com/strimzi/strimzi-kafka-operator/blob/0.28.0/operator-common/src/main/java/io/strimzi/operator/common/Util.java#L539-L584

The $6 in the log message corresponds to the line number in the source code where the method is defined:

    public static String authTlsHash(String userName, String password, Map<String, Object> config) {
        try {
            // Build up the TLS configuration
            Map<String, Object> tlsConfig = new HashMap<>();
            tlsConfig.put("truststore", config.get("ssl.truststore.location"));
            tlsConfig.put("truststorePassword", config.get("ssl.truststore.password"));
            tlsConfig.put("keystore", config.get("ssl.keystore.location"));
            tlsConfig.put("keystorePassword", config.get("ssl.keystore.password"));
            tlsConfig.put("keyPassword", config.get("ssl.key.password"));

            // Convert the configuration to a JSON string and compute a SHA-256 hash
            String json = new ObjectMapper().writeValueAsString(tlsConfig);
            return DigestUtils.sha256Hex(json);
        } catch (JsonProcessingException e) {
            LOGGER.error("Unexpected error generating hash for user '{}' with TLS configuration: {}", userName, config, e);
            return null;
        }
    }

The authTlsHash method takes in three parameters: userName, password, and config. config is a map that contains the SSL/TLS configuration for the Kafka client. The method constructs a new map called tlsConfig containing the relevant fields for TLS authentication, and converts this map to a JSON string. The SHA-256 hash is then computed over the JSON string.

If the config parameter is null or does not contain all the necessary SSL/TLS configuration fields, the method will throw a NullPointerException on this line:

tlsConfig.put("truststore", config.get("ssl.truststore.location"));

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments