[RHOCP 4] File integrity failing on file multus or istio-cni kubeconfig

Solution Verified - Updated -

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4

Issue

  • File integrity operator reports failed status for the file /etc/kubernetes/cni/net.d/multus.d/multus.kubeconfig
  • File integrity operator reports failed status for the file /etc/cni/multus/net.d/v2-3-istio-cni.kubeconfig

  • File integrity operator reporting file modification on each node, marking the node status as Failed:

    $ oc get fileintegritynodestatuses.fileintegrity.openshift.io -n openshift-file-integrity
NAME                                                   NODE            STATUS

infra-fileintegrity-infra0.xxx.xxx.xxx.xx     infra0.xxx.xxx.xxx.xx    Failed
infra-fileintegrity-infra1.xxx.xxx.xxx.xx     infra1.xxx.xxx.xxx.xx    Failed
master-fileintegrity-master0.xxx.xxx.xxx.xx   master0.xxx.xxx.xxx.xx   Failed
master-fileintegrity-master1.xxx.xxx.xxx.xx   master1.xxx.xxx.xxx.xx   Failed
master-fileintegrity-master2.xxx.xxx.xxx.xx   master2.xxx.xxx.xxx.xx   Failed
worker-fileintegrity-worker0.xxx.xxx.xxx.xx   worker0.xxx.xxx.xxx.xx   Failed
worker-fileintegrity-worker1.xxx.xxx.xxx.xx   worker1.xxx.xxx.xxx.xx   Failed
worker-fileintegrity-worker2.xxx.xxx.xxx.xx   worker2..xxx.xxx.xxx.xx  Failed

Resolution

  • Check if there exists customized AIDE(Advanced Intrusion Detection Environment) configuration file.

    $ oc get cm -n openshift-file-integrity

worker-aide-conf                                         1     100d
aide-worker-fileintegrity-worker2.xxx.xxx.xxx.xx-failed  1     100d

  • In the customized AIDE configuration file, make sure to add this location /hostroot/etc/kubernetes/cni/net.d/* under "OpenShift specific excludes" as shown below:

    $ oc get cm aide-worker-fileintegrity-worker0.xxxx-xxx-ocp.xxx.xxx.xx-failed -o yaml > <filename>.yaml

$ vi <filename>.yaml 
...
# OpenShift specific excludes
!/hostroot/opt/
!/hostroot/var
!/hostroot/etc/NetworkManager/system-connections/
!/hostroot/etc/mtab$
!/hostroot/etc/.*~
!/hostroot/etc/kubernetes/static-pod-resources
!/hostroot/etc/kubernetes/aide.*
!/hostroot/etc/kubernetes/manifests
!/hostroot/etc/docker/certs.d
!/hostroot/etc/selinux/targeted
!/hostroot/etc/openvswitch/conf.db
!/hostroot/etc/kubernetes/cni/net.d/*         <-----------------append this location
!/hostroot/etc/cni/multus/net.d/*             <-----------------append this location
!/hostroot/etc/machine-config-daemon/currentconfig$
!/hostroot/etc/pki/ca-trust/extracted/java/cacerts$
!/hostroot/etc/cvo/updatepayloads
...

$ oc replace -f <filename>.yaml

  • Confirm the changes are applied successfully

    $ oc get cm aide-worker-fileintegrity-worker0.xxxx-xxx-ocp.xxx.xxx.xx-failed -o yaml

Root Cause

Default FIO/AIDE config cannot exclude optional components of OpenShift, because by doing so it can open a door to exploit clusters that don’t have various optional components installed by modifying changes there. A user that is installing an optional component may create a custom AIDE config to exclude the directories that they know off as regularly modified by the optional operator they have installed.

Diagnostic Steps

  • Failed ConfigMap shows the below result:

    $ oc get cm aide-worker-fileintegrity-worker0.xxxx-xxx-ocp.xxx.xxx.xx-failed | grep -i "kubeconfig"

f   ...    .C... : /hostroot/etc/cni/multus/net.d/v2-3-istio-cni.kubeconfig
File: /hostroot/etc/cni/multus/net.d/v2-3-istio-cni.kubeconfig

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments