RHACS collector pod stuck into CLBO state due to libprotobuf contains invalid UTF-8 data

Solution Verified - Updated -

Environment

  • Red Hat Advanced Cluster Security
    • 4.5

Issue

  • Collector pods restarting very frequently in RHACS cluster and showing below error message:
[INFO 2024/08/01 10:29:15] Flushing container table
[INFO 2024/08/01 10:29:45] Flushing container table
[libprotobuf ERROR /build/third_party/protobuf/src/google/protobuf/wire_format_lite.cc:577] String field 'storage.ProcessSignal.args' contains invalid UTF-8 data when serializing a protocol buffer. Use the 'bytes' type if you intend to send raw bytes.
[INFO 2024/08/01 10:30:15] Flushing thread table
[INFO 2024/08/01 10:30:15] Flushing container table

Resolution

  • This is known to Red Hat and currently RHACS engineering team is troubleshooting this via a Bug.

Root Cause

  • Possibly server host running collector pods are using non UTF-8 compatible encoding for the arguments of some processes. This is still being investigated internally.

Diagnostic Steps

  • Check if the collector pods are stuck into CrashLoopBackOff state:
$ oc get pods -n stackrox
  • Now examine the RHACS collector container logs:
$ oc logs collector-xxxx -c collector

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments