Unable to start VM and Portworx (PX) API pods are in CrashLoopBackOff state after upgrading to Openshift Container Platform 4.19.

Solution In Progress - Updated -

Environment

  • OpenShift Container Platform (OCP) 4.19
  • OpenShift Virtualization 4.19.3
  • Kubernetes version v1.32.7
  • Portworx version 3.2.2

Issue

  • Unable to start VM and Portworx (PX) API pods are in CrashLoopBackOff state after upgrading to Openshift Container Platform 4.19.

  • Event logs showed.

Warning   FileSystemModuleDependency           storagenode/openshift-worker-0   insmod: WARNING: could not insert module /var/lib/osd/pxfs/latest/12.px.ko: Invalid module format. Check dmesg for more information.
Warning   FileSystemDependency                 storagenode/openshift-worker-0   Failed to load PX filesystem dependencies for kernel 5.14.0-570.35.1.el9_6.x86_64.
Normal    Pulling                              pod/portworx-api-9cw7x           Pulling image "registry.k8s.io/pause:3.1"
Normal    Pulling                              pod/portworx-api-9cw7x           Pulling image "registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.12.0"
Warning   Unhealthy                            pod/portworx-api-9cw7x           Readiness probe failed: Get "http://127.0.0.1:17001/status": dial tcp 127.0.0.1:17001: connect: connection refused
Warning   Unhealthy                            pod/portworx-api-9cw7x           Liveness probe failed: Get "http://127.0.0.1:17001/status": dial tcp 127.0.0.1:17001: connect: connection refused

From dmesg:

[198118.658824] module px: .gnu.linkonce.this_module section size must match the kernel's built struct module size at run time

Resolution

  • Upgrade the Portworx Operator to 25.3.1 (v3.4.0) on Openshift Container Platform 4.19 It's also advisable to consult with Portworx support prior to upgrade.

Root Cause

Diagnostic Steps

  • Event log information from portworx namespace.
$ oc get events -n portworx
LAST SEEN   TYPE      REASON                               OBJECT                                                           MESSAGE
171m        Warning   FileSystemModuleDependency           storagenode/openshift-worker-0                                   insmod: WARNING: could not insert module /var/lib/osd/pxfs/latest/12.px.ko: Invalid module format. Check dmesg for more information.
156m        Warning   FileSystemModuleDependency           storagenode/openshift-worker-0                                   insmod: WARNING: could not insert module /var/lib/osd/pxfs/latest/12.px.ko: Invalid module format. Check dmesg for more information.
156m        Warning   FileSystemModuleDependency           storagenode/openshift-worker-0                                   insmod: WARNING: could not insert module /var/lib/osd/pxfs/latest/12.px.ko: Invalid module format. Check dmesg for more information.
156m        Warning   FileSystemDependency                 storagenode/openshift-worker-0                                   Failed to load PX filesystem dependencies for kernel 5.14.0-570.35.1.el9_6.x86_64.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments