SAP Edge Integration Cell (EIC) - Solace Broker Pods Fail to Start with POST Violation (nofile Limit)

Solution Verified - Updated -

Environment

  • SAP Integration Suite, Edge Integration Cell 8.48 or later (namespace edge-icell-services)
  • Red Hat OpenShift Container Platform 4.x

The requirement applies to any OpenShift 4.x version. ARO and ROSA need a separate review of their configuration policies.

Issue

Installing or upgrading to EIC 8.48 or later fails while deploying edge-icell-services because the Solace broker container starts with a hard open-file limit below 1048576. The Solace pods never become ready, and the broker shuts down after its startup resource check:

ERROR POST Violation [022]:Required system resource missing, Hard resource limit nofile 1048576 is required, 524288 detected
WARN POST Violation [023]:Recommended system resource missing, Soft resource limit nofiles 2448 is recommended, 1024 detected
Management API not yet accessible
Determining platform type: [ FAIL ]
System shutdown complete (Version 10.25.0.208)

Resolution

Proposed resolution; success on the reported cluster is pending confirmation.

Raise the CRI-O default ulimits on the worker nodes with a MachineConfig, following Adjust default ulimits on CRI-O using MachineConfig. The drop-in content is:

[crio.runtime]
default_ulimits = ["nofile=4096:1048576"]

If existing drop-ins already set default_ulimits, keep their other entries.

Save the following as 96-worker-crio-limit-files.yaml. Check that the Ignition version matches the target OCP release:

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  name: 96-worker-crio-limit-files
  labels:
    machineconfiguration.openshift.io/role: worker
spec:
  config:
    ignition:
      version: 3.2.0
    storage:
      files:
        - path: /etc/crio/crio.conf.d/99-eic-nofile.conf
          mode: 420
          overwrite: true
          contents:
            source: "data:,%5Bcrio.runtime%5D%0Adefault_ulimits%20%3D%20%5B%22nofile%3D4096%3A1048576%22%5D%0A"

The data URL encodes the TOML above. This changes the defaults for all containers in the worker pool and any custom pools that inherit from it, not just Solace.

Apply to an existing cluster

Applying the MachineConfig drains and reboots each worker in turn:

oc apply -f 96-worker-crio-limit-files.yaml
oc get machineconfigpools

Wait until the pool reports UPDATED=True, UPDATING=False, and DEGRADED=False, then retry the EIC deployment.

Node drain blocked by failed Solace pods

The Solace PodDisruptionBudget requires two available pods. If the Solace pods are failing, for example after a failed installation or upgrade, no Solace pod can be evicted. The drain then hangs on the node that hosts them: the pool stays UPDATING=True, the node stays SchedulingDisabled, and the machine-config-controller keeps logging:

oc logs -n openshift-machine-config-operator deployment/machine-config-controller \
  -c machine-config-controller | grep -i "disruption budget"

Cannot evict pod as it would violate the pod's disruption budget.

The pods are failing anyway, so delete them directly. A direct delete is not an eviction, so the PodDisruptionBudget does not block it:

oc get pods -n edge-icell-services -o wide \
  --field-selector spec.nodeName=<draining-node> | grep solace
oc delete pod <solace-pod> -n edge-icell-services

The StatefulSet may recreate a pod on a worker that has not been updated yet, which blocks the drain again when the rollout reaches that worker. Repeat the deletion as needed until the pool has finished updating, then retry the EIC deployment.

Include during cluster provisioning

There is no nofile option in install-config.yaml. Run openshift-install create manifests, copy the MachineConfig into the generated openshift/ directory, then continue with create cluster (IPI) or create ignition-configs (UPI). See the installation configuration guide for the target release.

Root Cause

Starting with EIC 8.48, the included Solace broker requires a hard RLIMIT_NOFILE of at least 1048576 and recommends a soft/hard pair of 4096:1048576 (Solace system scaling documentation). The container limits on the reported cluster were below these values:

Resource limit Observed Required
Soft nofile 1024 4096
Hard nofile 524288 1048576

Diagnostic Steps

  1. Find the Solace pods, their nodes, and the broker container name:

    oc get pods -n edge-icell-services -o wide | grep solace
oc get pod ssb-solace-0 -n edge-icell-services -o jsonpath='{.spec.containers[*].name}'

  2. While the broker container is running, check the Max open files limit of PID 1:

    oc exec -n edge-icell-services ssb-solace-0 -c <broker-container> -- cat /proc/1/limits

    If the container exits too quickly, check the startup log instead:

    oc logs -n edge-icell-services ssb-solace-0 -c <broker-container> --previous

  3. Before applying the fix, check whether existing MachineConfigs or CRI-O drop-ins under /etc/crio/crio.conf.d/ already set default_ulimits.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments