Chapter 4. Kernel Module Management Operator
Learn about the Kernel Module Management (KMM) Operator and how you can use it to deploy out-of-tree kernel modules and device plugins on OpenShift Container Platform clusters.
4.1. About the Kernel Module Management Operator
The Kernel Module Management (KMM) Operator manages, builds, signs, and deploys out-of-tree kernel modules and device plugins on OpenShift Container Platform clusters.
KMM adds a new Module CRD which describes an out-of-tree kernel module and its associated device plugin. You can use Module resources to configure how to load the module, define ModuleLoader images for kernel versions, and include instructions for building and signing modules for specific kernel versions.
KMM is designed to accommodate multiple kernel versions at once for any kernel module, allowing for seamless node upgrades and reduced application downtime.
4.2. Installing the Kernel Module Management Operator
As a cluster administrator, you can install the Kernel Module Management (KMM) Operator by using the OpenShift CLI or the web console.
The KMM Operator is supported on OpenShift Container Platform 4.12 and later. Installing KMM on version 4.11 does not require specific additional steps. For details on installing KMM on version 4.10 and earlier, see the section "Installing the Kernel Module Management Operator on earlier versions of OpenShift Container Platform".
4.2.1. Installing the Kernel Module Management Operator using the web console
As a cluster administrator, you can install the Kernel Module Management (KMM) Operator using the OpenShift Container Platform web console.
Procedure
- Log in to the OpenShift Container Platform web console.
Install the Kernel Module Management Operator:
- In the OpenShift Container Platform web console, click Operators → OperatorHub.
- Select Kernel Module Management Operator from the list of available Operators, and then click Install.
- On the Install Operator page, select the Installation mode as A specific namespace on the cluster.
-
From the Installed Namespace list, select the
openshift-kmmnamespace. - Click Install.
Verification
To verify that KMM Operator installed successfully:
- Navigate to the Operators → Installed Operators page.
Ensure that Kernel Module Management Operator is listed in the openshift-kmm project with a Status of InstallSucceeded.
NoteDuring installation, an Operator might display a Failed status. If the installation later succeeds with an InstallSucceeded message, you can ignore the Failed message.
Troubleshooting
To troubleshoot issues with Operator installation:
- Navigate to the Operators → Installed Operators page and inspect the Operator Subscriptions and Install Plans tabs for any failure or errors under Status.
-
Navigate to the Workloads → Pods page and check the logs for pods in the
openshift-kmmproject.
4.2.2. Installing the Kernel Module Management Operator by using the CLI
As a cluster administrator, you can install the Kernel Module Management (KMM) Operator by using the OpenShift CLI.
Prerequisites
- You have a running OpenShift Container Platform cluster.
-
You installed the OpenShift CLI (
oc). -
You are logged into the OpenShift CLI as a user with
cluster-adminprivileges.
Procedure
Install KMM in the
openshift-kmmnamespace:Create the following
NamespaceCR and save the YAML file, for example,kmm-namespace.yaml:apiVersion: v1 kind: Namespace metadata: name: openshift-kmm
Create the following
OperatorGroupCR and save the YAML file, for example,kmm-op-group.yaml:apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: kernel-module-management namespace: openshift-kmm
Create the following
SubscriptionCR and save the YAML file, for example,kmm-sub.yaml:apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: kernel-module-management namespace: openshift-kmm spec: channel: release-1.0 installPlanApproval: Automatic name: kernel-module-management source: redhat-operators sourceNamespace: openshift-marketplace startingCSV: kernel-module-management.v1.0.0
Create the subscription object by running the following command:
$ oc create -f kmm-sub.yaml
Verification
To verify that the Operator deployment is successful, run the following command:
$ oc get -n openshift-kmm deployments.apps kmm-operator-controller-manager
Example output
NAME READY UP-TO-DATE AVAILABLE AGE kmm-operator-controller-manager 1/1 1 1 97s
The Operator is available.
4.2.3. Installing the Kernel Module Management Operator on earlier versions of OpenShift Container Platform
The KMM Operator is supported on OpenShift Container Platform 4.12 and later. For version 4.10 and earlier, you must create a new SecurityContextConstraint object and bind it to the Operator’s ServiceAccount. As a cluster administrator, you can install the Kernel Module Management (KMM) Operator by using the OpenShift CLI.
Prerequisites
- You have a running OpenShift Container Platform cluster.
-
You installed the OpenShift CLI (
oc). -
You are logged into the OpenShift CLI as a user with
cluster-adminprivileges.
Procedure
Install KMM in the
openshift-kmmnamespace:Create the following
NamespaceCR and save the YAML file, for example,kmm-namespace.yamlfile:apiVersion: v1 kind: Namespace metadata: name: openshift-kmm
Create the following
SecurityContextConstraintobject and save the YAML file, for example,kmm-security-constraint.yaml:allowHostDirVolumePlugin: false allowHostIPC: false allowHostNetwork: false allowHostPID: false allowHostPorts: false allowPrivilegeEscalation: false allowPrivilegedContainer: false allowedCapabilities: - NET_BIND_SERVICE apiVersion: security.openshift.io/v1 defaultAddCapabilities: null fsGroup: type: MustRunAs groups: [] kind: SecurityContextConstraints metadata: name: restricted-v2 priority: null readOnlyRootFilesystem: false requiredDropCapabilities: - ALL runAsUser: type: MustRunAsRange seLinuxContext: type: MustRunAs seccompProfiles: - runtime/default supplementalGroups: type: RunAsAny users: [] volumes: - configMap - downwardAPI - emptyDir - persistentVolumeClaim - projected - secret
Bind the
SecurityContextConstraintobject to the Operator’sServiceAccountby running the following commands:$ oc apply -f kmm-security-constraint.yaml
$ oc adm policy add-scc-to-user kmm-security-constraint -z kmm-operator-controller-manager -n openshift-kmm
Create the following
OperatorGroupCR and save the YAML file, for example,kmm-op-group.yaml:apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: kernel-module-management namespace: openshift-kmm
Create the following
SubscriptionCR and save the YAML file, for example,kmm-sub.yaml:apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: kernel-module-management namespace: openshift-kmm spec: channel: release-1.0 installPlanApproval: Automatic name: kernel-module-management source: redhat-operators sourceNamespace: openshift-marketplace startingCSV: kernel-module-management.v1.0.0
Create the subscription object by running the following command:
$ oc create -f kmm-sub.yaml
Verification
To verify that the Operator deployment is successful, run the following command:
$ oc get -n openshift-kmm deployments.apps kmm-operator-controller-manager
Example output
NAME READY UP-TO-DATE AVAILABLE AGE kmm-operator-controller-manager 1/1 1 1 97s
The Operator is available.
4.3. Kernel module deployment
For each Module resource, Kernel Module Management (KMM) can create a number of DaemonSet resources:
-
One ModuleLoader
DaemonSetper compatible kernel version running in the cluster. -
One device plugin
DaemonSet, if configured.
The module loader daemon set resources run ModuleLoader images to load kernel modules. A module loader image is an OCI image that contains the .ko files and both the modprobe and sleep binaries.
When the module loader pod is created, the pod runs modprobe to insert the specified module into the kernel. It then enters a sleep state until it is terminated. When that happens, the ExecPreStop hook runs modprobe -r to unload the kernel module.
If the .spec.devicePlugin attribute is configured in a Module resource, then KMM creates a device plugin daemon set in the cluster. That daemon set targets:
-
Nodes that match the
.spec.selectorof theModuleresource. -
Nodes with the kernel module loaded (where the module loader pod is in the
Readycondition).
4.3.1. The Module custom resource definition
The Module custom resource definition (CRD) represents a kernel module that can be loaded on all or select nodes in the cluster, through a module loader image. A Module custom resource (CR) specifies one or more kernel versions with which it is compatible, and a node selector.
The compatible versions for a Module resource are listed under .spec.moduleLoader.container.kernelMappings. A kernel mapping can either match a literal version, or use regexp to match many of them at the same time.
The reconciliation loop for the Module resource runs the following steps:
-
List all nodes matching
.spec.selector. - Build a set of all kernel versions running on those nodes.
For each kernel version:
-
Go through
.spec.moduleLoader.container.kernelMappingsand find the appropriate container image name. If the kernel mapping hasbuildorsigndefined and the container image does not already exist, run the build, the signing job, or both, as needed. - Create a module loader daemon set with the container image determined in the previous step.
-
If
.spec.devicePluginis defined, create a device plugin daemon set using the configuration specified under.spec.devicePlugin.container.
-
Go through
Run
garbage-collecton:- Existing daemon set resources targeting kernel versions that are not run by any node in the cluster.
- Successful build jobs.
- Successful signing jobs.
4.3.2. Security and permissions
Loading kernel modules is a highly sensitive operation. After they are loaded, kernel modules have all possible permissions to do any kind of operation on the node.
4.3.2.1. ServiceAccounts and SecurityContextConstraints
Kernel Module Management (KMM) creates a privileged workload to load the kernel modules on nodes. That workload needs ServiceAccounts allowed to use the privileged SecurityContextConstraint (SCC) resource.
The authorization model for that workload depends on the namespace of the Module resource, as well as its spec.
-
If the
.spec.moduleLoader.serviceAccountNameor.spec.devicePlugin.serviceAccountNamefields are set, they are always used. If those fields are not set, then:
-
If the
Moduleresource is created in the operator’s namespace (openshift-kmmby default), then KMM uses its default, powerfulServiceAccountsto run the daemon sets. -
If the
Moduleresource is created in any other namespace, then KMM runs the daemon sets as the namespace’sdefaultServiceAccount. TheModuleresource cannot run a privileged workload unless you manually enable it to use theprivilegedSCC.
-
If the
openshift-kmm is a trusted namespace.
When setting up RBAC permissions, remember that any user or ServiceAccount creating a Module resource in the openshift-kmm namespace results in KMM automatically running privileged workloads on potentially all nodes in the cluster.
To allow any ServiceAccount to use the privileged SCC and therefore to run module loader or device plugin pods, use the following command:
$ oc adm policy add-scc-to-user privileged -z "${serviceAccountName}" [ -n "${namespace}" ]4.3.2.2. Pod security standards
OpenShift runs a synchronization mechanism that sets the namespace Pod Security level automatically based on the security contexts in use. No action is needed.
Additional resources
4.3.3. Example Module CR
The following is an annotated Module example:
apiVersion: kmm.sigs.x-k8s.io/v1beta1
kind: Module
metadata:
name: <my_kmod>
spec:
moduleLoader:
container:
modprobe:
moduleName: <my_kmod> 1
dirName: /opt 2
firmwarePath: /firmware 3
parameters: 4
- param=1
kernelMappings: 5
- literal: 6.0.15-300.fc37.x86_64
containerImage: some.registry/org/my-kmod:6.0.15-300.fc37.x86_64
- regexp: '^.+\fc37\.x86_64$' 6
containerImage: "some.other.registry/org/<my_kmod>:${KERNEL_FULL_VERSION}"
- regexp: '^.+$' 7
containerImage: "some.registry/org/<my_kmod>:${KERNEL_FULL_VERSION}"
build:
buildArgs: 8
- name: ARG_NAME
value: <some_value>
secrets:
- name: <some_kubernetes_secret> 9
baseImageRegistryTLS: 10
insecure: false
insecureSkipTLSVerify: false 11
dockerfileConfigMap: 12
name: <my_kmod_dockerfile>
sign:
certSecret:
name: <cert_secret> 13
keySecret:
name: <key_secret> 14
filesToSign:
- /opt/lib/modules/${KERNEL_FULL_VERSION}/<my_kmod>.ko
registryTLS: 15
insecure: false 16
insecureSkipTLSVerify: false
serviceAccountName: <sa_module_loader> 17
devicePlugin: 18
container:
image: some.registry/org/device-plugin:latest 19
env:
- name: MY_DEVICE_PLUGIN_ENV_VAR
value: SOME_VALUE
volumeMounts: 20
- mountPath: /some/mountPath
name: <device_plugin_volume>
volumes: 21
- name: <device_plugin_volume>
configMap:
name: <some_configmap>
serviceAccountName: <sa_device_plugin> 22
imageRepoSecret: 23
name: <secret_name>
selector:
node-role.kubernetes.io/worker: ""- 1 1 1
- Required.
- 2
- Optional.
- 3
- Optional: Copies
/firmware/*into/var/lib/firmware/on the node. - 4
- Optional.
- 5
- At least one kernel item is required.
- 6
- For each node running a kernel matching the regular expression, KMM creates a
DaemonSetresource running the image specified incontainerImagewith${KERNEL_FULL_VERSION}replaced with the kernel version. - 7
- For any other kernel, build the image using the Dockerfile in the
my-kmodConfigMap. - 8
- Optional.
- 9
- Optional: A value for
some-kubernetes-secretcan be obtained from the build environment at/run/secrets/some-kubernetes-secret. - 10
- Optional: Avoid using this parameter. If set to
true, the build is allowed to pull the image in the DockerfileFROMinstruction using plain HTTP. - 11
- Optional: Avoid using this parameter. If set to
true, the build will skip any TLS server certificate validation when pulling the image in the DockerfileFROMinstruction using plain HTTP. - 12
- Required.
- 13
- Required: A secret holding the public secureboot key with the key 'cert'.
- 14
- Required: A secret holding the private secureboot key with the key 'key'.
- 15
- Optional: Avoid using this parameter. If set to
true, KMM will be allowed to check if the container image already exists using plain HTTP. - 16
- Optional: Avoid using this parameter. If set to
true, KMM will skip any TLS server certificate validation when checking if the container image already exists. - 17
- Optional.
- 18
- Optional.
- 19
- Required: If the device plugin section is present.
- 20
- Optional.
- 21
- Optional.
- 22
- Optional.
- 23
- Optional: Used to pull module loader and device plugin images.
4.4. Using a ModuleLoader image
Kernel Module Management (KMM) works with purpose-built module loader images. These are standard OCI images that must satisfy the following requirements:
-
.kofiles must be located in/opt/lib/modules/${KERNEL_VERSION}. -
modprobeandsleepbinaries must be defined in the$PATHvariable.
4.4.1. Running depmod
If your module loader image contains several kernel modules and if one of the modules depends on another module, it is best practice to run depmod at the end of the build process to generate dependencies and map files.
You must have a Red Hat subscription to download the kernel-devel package.
Procedure
-
To generate
modules.depand.mapfiles for a specific kernel version, rundepmod -b /opt ${KERNEL_VERSION}.
4.4.1.1. Example Dockerfile
If you are building your image on OpenShift Container Platform, consider using the Driver Tool Kit (DTK).
For further information, see using an entitled build.
apiVersion: v1
kind: ConfigMap
metadata:
name: kmm-ci-dockerfile
data:
dockerfile: |
ARG DTK_AUTO
FROM ${DTK_AUTO} as builder
ARG KERNEL_VERSION
WORKDIR /usr/src
RUN ["git", "clone", "https://github.com/rh-ecosystem-edge/kernel-module-management.git"]
WORKDIR /usr/src/kernel-module-management/ci/kmm-kmod
RUN KERNEL_SRC_DIR=/lib/modules/${KERNEL_VERSION}/build make all
FROM registry.redhat.io/ubi8/ubi-minimal
ARG KERNEL_VERSION
RUN microdnf install kmod
COPY --from=builder /usr/src/kernel-module-management/ci/kmm-kmod/kmm_ci_a.ko /opt/lib/modules/${KERNEL_VERSION}/
COPY --from=builder /usr/src/kernel-module-management/ci/kmm-kmod/kmm_ci_b.ko /opt/lib/modules/${KERNEL_VERSION}/
RUN depmod -b /opt ${KERNEL_VERSION}Additional resources
4.4.2. Building in the cluster
KMM can build module loader images in the cluster. Follow these guidelines:
-
Provide build instructions using the
buildsection of a kernel mapping. -
Copy the
Dockerfilefor your container image into aConfigMapresource, under thedockerfilekey. -
Ensure that the
ConfigMapis located in the same namespace as theModule.
KMM checks if the image name specified in the containerImage field exists. If it does, the build is skipped.
Otherwise, KMM creates a Build resource to build your image. After the image is built, KMM proceeds with the Module reconciliation. See the following example.
# ...
- regexp: '^.+$'
containerImage: "some.registry/org/<my_kmod>:${KERNEL_FULL_VERSION}"
build:
buildArgs: 1
- name: ARG_NAME
value: <some_value>
secrets: 2
- name: <some_kubernetes_secret> 3
baseImageRegistryTLS:
insecure: false 4
insecureSkipTLSVerify: false 5
dockerfileConfigMap: 6
name: <my_kmod_dockerfile>
registryTLS:
insecure: false 7
insecureSkipTLSVerify: false 8- 1
- Optional.
- 2
- Optional.
- 3
- Will be mounted in the build pod as
/run/secrets/some-kubernetes-secret. - 4
- Optional: Avoid using this parameter. If set to
true, the build will be allowed to pull the image in the DockerfileFROMinstruction using plain HTTP. - 5
- Optional: Avoid using this parameter. If set to
true, the build will skip any TLS server certificate validation when pulling the image in the DockerfileFROMinstruction using plain HTTP. - 6
- Required.
- 7
- Optional: Avoid using this parameter. If set to
true, KMM will be allowed to check if the container image already exists using plain HTTP. - 8
- Optional: Avoid using this parameter. If set to
true, KMM will skip any TLS server certificate validation when checking if the container image already exists.
Additional resources
4.4.3. Using the Driver Toolkit
The Driver Toolkit (DTK) is a convenient base image for building build module loader images. It contains tools and libraries for the OpenShift version currently running in the cluster.
Procedure
Use DTK as the first stage of a multi-stage Dockerfile.
- Build the kernel modules.
-
Copy the
.kofiles into a smaller end-user image such asubi-minimal. To leverage DTK in your in-cluster build, use the
DTK_AUTObuild argument. The value is automatically set by KMM when creating theBuildresource. See the following example.ARG DTK_AUTO FROM ${DTK_AUTO} as builder ARG KERNEL_VERSION WORKDIR /usr/src RUN ["git", "clone", "https://github.com/rh-ecosystem-edge/kernel-module-management.git"] WORKDIR /usr/src/kernel-module-management/ci/kmm-kmod RUN KERNEL_SRC_DIR=/lib/modules/${KERNEL_VERSION}/build make all FROM registry.redhat.io/ubi8/ubi-minimal ARG KERNEL_VERSION RUN microdnf install kmod COPY --from=builder /usr/src/kernel-module-management/ci/kmm-kmod/kmm_ci_a.ko /opt/lib/modules/${KERNEL_VERSION}/ COPY --from=builder /usr/src/kernel-module-management/ci/kmm-kmod/kmm_ci_b.ko /opt/lib/modules/${KERNEL_VERSION}/ RUN depmod -b /opt ${KERNEL_VERSION}
Additional resources
4.5. Using signing with Kernel Module Management (KMM)
On a Secure Boot enabled system, all kernel modules (kmods) must be signed with a public/private key-pair enrolled into the Machine Owner’s Key (MOK) database. Drivers distributed as part of a distribution should already be signed by the distribution’s private key, but for kernel modules build out-of-tree, KMM supports signing kernel modules using the sign section of the kernel mapping.
For more details on using Secure Boot, see Generating a public and private key pair
Prerequisites
- A public private key pair in the correct (DER) format.
- At least one secure-boot enabled node with the public key enrolled in its MOK database.
-
Either a pre-built driver container image, or the source code and
Dockerfileneeded to build one in-cluster.
4.6. Adding the keys for secureboot
To use KMM Kernel Module Management (KMM) to sign kernel modules, a certificate and private key are required. For details on how to create these, see Generating a public and private key pair.
For details on how to extract the public and private key pair, see Signing kernel modules with the private key. Use steps 1 through 4 to extract the keys into files.
Procedure
Create the
sb_cert.cerfile that contains the certificate and thesb_cert.privfile that contains the private key:$ openssl req -x509 -new -nodes -utf8 -sha256 -days 36500 -batch -config configuration_file.config -outform DER -out my_signing_key_pub.der -keyout my_signing_key.priv
Add the files by using one of the following methods:
Add the files as secrets directly:
$ oc create secret generic my-signing-key --from-file=key=<my_signing_key.priv>
$ oc create secret generic my-signing-key-pub --from-file=key=<my_signing_key_pub.der>
Add the files by base64 encoding them:
$ cat sb_cert.priv | base64 -w 0 > my_signing_key2.base64
$ cat sb_cert.cer | base64 -w 0 > my_signing_key_pub.base64
Add the encoded text to a YAML file:
apiVersion: v1 kind: Secret metadata: name: my-signing-key-pub namespace: default 1 type: Opaque data: cert: <base64_encoded_secureboot_public_key> --- apiVersion: v1 kind: Secret metadata: name: my-signing-key namespace: default 2 type: Opaque data: key: <base64_encoded_secureboot_private_key>
Apply the YAML file:
$ oc apply -f <yaml_filename>
4.6.1. Checking the keys
After you have added the keys, you must check them to ensure they are set correctly.
Procedure
Check to ensure the public key secret is set correctly:
$ oc get secret -o yaml <certificate secret name> | awk '/cert/{print $2; exit}' | base64 -d | openssl x509 -inform der -textThis should display a certificate with a Serial Number, Issuer, Subject, and more.
Check to ensure the private key secret is set correctly:
$ oc get secret -o yaml <private key secret name> | awk '/key/{print $2; exit}' | base64 -dThis should display the key enclosed in the
-----BEGIN PRIVATE KEY-----and-----END PRIVATE KEY-----lines.
4.7. Signing a pre-built driver container
Use this procedure if you have a pre-built image, such as an image either distributed by a hardware vendor or built elsewhere.
The following YAML file adds the public/private key-pair as secrets with the required key names - key for the private key, cert for the public key. The cluster then pulls down the unsignedImage image, opens it, signs the kernel modules listed in filesToSign, adds them back, and pushes the resulting image as containerImage.
Kernel Module Management (KMM) should then deploy the DaemonSet that loads the signed kmods onto all the nodes that match the selector. The driver containers should run successfully on any nodes that have the public key in their MOK database, and any nodes that are not secure-boot enabled, which ignore the signature. They should fail to load on any that have secure-boot enabled but do not have that key in their MOK database.
Prerequisites
-
The
keySecretandcertSecretsecrets have been created.
Procedure
Apply the YAML file:
--- apiVersion: kmm.sigs.x-k8s.io/v1beta1 kind: Module metadata: name: example-module spec: moduleLoader: serviceAccountName: default container: modprobe: 1 moduleName: '<your module name>' kernelMappings: # the kmods will be deployed on all nodes in the cluster with a kernel that matches the regexp - regexp: '^.*\.x86_64$' # the container to produce containing the signed kmods containerImage: <image name e.g. quay.io/myuser/my-driver:<kernelversion>-signed> sign: # the image containing the unsigned kmods (we need this because we are not building the kmods within the cluster) unsignedImage: <image name e.g. quay.io/myuser/my-driver:<kernelversion> > keySecret: # a secret holding the private secureboot key with the key 'key' name: <private key secret name> certSecret: # a secret holding the public secureboot key with the key 'cert' name: <certificate secret name> filesToSign: # full path within the unsignedImage container to the kmod(s) to sign - /opt/lib/modules/4.18.0-348.2.1.el8_5.x86_64/kmm_ci_a.ko imageRepoSecret: # the name of a secret containing credentials to pull unsignedImage and push containerImage to the registry name: repo-pull-secret selector: kubernetes.io/arch: amd64
- 1
modprobe- The name of the kmod to load.
4.8. Building and signing a ModuleLoader container image
Use this procedure if you have source code and must build your image first.
The following YAML file builds a new container image using the source code from the repository. The image produced is saved back in the registry with a temporary name, and this temporary image is then signed using the parameters in the sign section.
The temporary image name is based on the final image name and is set to be <containerImage>:<tag>-<namespace>_<module name>_kmm_unsigned.
For example, using the following YAML file, Kernel Module Management (KMM) builds an image named example.org/repository/minimal-driver:final-default_example-module_kmm_unsigned containing the build with unsigned kmods and push it to the registry. Then it creates a second image named example.org/repository/minimal-driver:final that contains the signed kmods. It is this second image that is loaded by the DaemonSet object and deploys the kmods to the cluster nodes.
After it is signed, the temporary image can be safely deleted from the registry. It will be rebuilt, if needed.
Prerequisites
-
The
keySecretandcertSecretsecrets have been created.
Procedure
Apply the YAML file:
--- apiVersion: v1 kind: ConfigMap metadata: name: example-module-dockerfile namespace: default 1 data: Dockerfile: | ARG DTK_AUTO ARG KERNEL_VERSION FROM ${DTK_AUTO} as builder WORKDIR /build/ RUN git clone -b main --single-branch https://github.com/rh-ecosystem-edge/kernel-module-management.git WORKDIR kernel-module-management/ci/kmm-kmod/ RUN make FROM registry.access.redhat.com/ubi8/ubi:latest ARG KERNEL_VERSION RUN yum -y install kmod && yum clean all RUN mkdir -p /opt/lib/modules/${KERNEL_VERSION} COPY --from=builder /build/kernel-module-management/ci/kmm-kmod/*.ko /opt/lib/modules/${KERNEL_VERSION}/ RUN /usr/sbin/depmod -b /opt --- apiVersion: kmm.sigs.x-k8s.io/v1beta1 kind: Module metadata: name: example-module namespace: default 2 spec: moduleLoader: serviceAccountName: default 3 container: modprobe: moduleName: simple_kmod kernelMappings: - regexp: '^.*\.x86_64$' containerImage: < the name of the final driver container to produce> build: dockerfileConfigMap: name: example-module-dockerfile sign: keySecret: name: <private key secret name> certSecret: name: <certificate secret name> filesToSign: - /opt/lib/modules/4.18.0-348.2.1.el8_5.x86_64/kmm_ci_a.ko imageRepoSecret: 4 name: repo-pull-secret selector: # top-level selector kubernetes.io/arch: amd64
- 1 2
namespace- Replacedefaultwith a valid namespace.- 3
serviceAccountName- The defaultserviceAccountNamedoes not have the required permissions to run a module that is privileged. For information on creating a service account, see "Creating service accounts" in the "Additional resources" of this section.- 4
imageRepoSecret- Used asimagePullSecretsin theDaemonSetobject and to pull and push for the build and sign features.
Additional resources
For information on creating a service account, see Creating service accounts.
4.9. Debugging and troubleshooting
If the kmods in your driver container are not signed or are signed with the wrong key, then the container can enter a PostStartHookError or CrashLoopBackOff status. You can verify by running the oc describe command on your container, which displays the following message in this scenario:
modprobe: ERROR: could not insert '<your_kmod_name>': Required key not available
4.10. KMM firmware support
Kernel modules sometimes need to load firmware files from the file system. KMM supports copying firmware files from the ModuleLoader image to the node’s file system.
The contents of .spec.moduleLoader.container.modprobe.firmwarePath are copied into the /var/lib/firmware path on the node before running the modprobe command to insert the kernel module.
All files and empty directories are removed from that location before running the modprobe -r command to unload the kernel module, when the pod is terminated.
Additional resources
4.10.1. Configuring the lookup path on nodes
On OpenShift Container Platform nodes, the set of default lookup paths for firmwares does not include the /var/lib/firmware path.
Procedure
Use the Machine Config Operator to create a
MachineConfigcustom resource (CR) that contains the/var/lib/firmwarepath:apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig metadata: labels: machineconfiguration.openshift.io/role: worker 1 name: 99-worker-kernel-args-firmware-path spec: kernelArguments: - 'firmware_class.path=/var/lib/firmware'- 1
- You can configure the label based on your needs. In the case of single-node OpenShift, use either
control-paneormasterobjects.
-
By applying the
MachineConfigCR, the nodes are automatically rebooted.
Additional resources
4.10.2. Building a ModuleLoader image
Procedure
In addition to building the kernel module itself, include the binary firmware in the builder image:
FROM registry.redhat.io/ubi8/ubi-minimal as builder # Build the kmod RUN ["mkdir", "/firmware"] RUN ["curl", "-o", "/firmware/firmware.bin", "https://artifacts.example.com/firmware.bin"] FROM registry.redhat.io/ubi8/ubi-minimal # Copy the kmod, install modprobe, run depmod COPY --from=builder /firmware /firmware
4.10.3. Tuning the Module resource
Procedure
Set
.spec.moduleLoader.container.modprobe.firmwarePathin theModulecustom resource (CR):apiVersion: kmm.sigs.x-k8s.io/v1beta1 kind: Module metadata: name: my-kmod spec: moduleLoader: container: modprobe: moduleName: my-kmod # Required firmwarePath: /firmware 1- 1
- Optional: Copies
/firmware/*into/var/lib/firmware/on the node.
4.11. Troubleshooting KMM
When troubleshooting KMM installation issues, you can monitor logs to determine at which stage issues occur. Then, retrieve diagnostic data relevant to that stage.
4.11.1. Using the must-gather tool
The oc adm must-gather command is the preferred way to collect a support bundle and provide debugging information to Red Hat Support. Collect specific information by running the command with the appropriate arguments as described in the following sections.
Additional resources
4.11.1.1. Gathering data for KMM
Procedure
Gather the data for the KMM Operator controller manager:
Set the
MUST_GATHER_IMAGEvariable:$ export MUST_GATHER_IMAGE=$(oc get deployment -n openshift-kmm kmm-operator-controller-manager -ojsonpath='{.spec.template.spec.containers[?(@.name=="manager")].env[?(@.name=="RELATED_IMAGES_MUST_GATHER")].value}')NoteUse the
-n <namespace>switch to specify a namespace if you installed KMM in a custom namespace.Run the
must-gathertool:$ oc adm must-gather --image="${MUST_GATHER_IMAGE}" -- /usr/bin/gather
View the Operator logs:
$ oc logs -fn openshift-kmm deployments/kmm-operator-controller-manager
Example 4.1. Example output
I0228 09:36:37.352405 1 request.go:682] Waited for 1.001998746s due to client-side throttling, not priority and fairness, request: GET:https://172.30.0.1:443/apis/machine.openshift.io/v1beta1?timeout=32s I0228 09:36:40.767060 1 listener.go:44] kmm/controller-runtime/metrics "msg"="Metrics server is starting to listen" "addr"="127.0.0.1:8080" I0228 09:36:40.769483 1 main.go:234] kmm/setup "msg"="starting manager" I0228 09:36:40.769907 1 internal.go:366] kmm "msg"="Starting server" "addr"={"IP":"127.0.0.1","Port":8080,"Zone":""} "kind"="metrics" "path"="/metrics" I0228 09:36:40.770025 1 internal.go:366] kmm "msg"="Starting server" "addr"={"IP":"::","Port":8081,"Zone":""} "kind"="health probe" I0228 09:36:40.770128 1 leaderelection.go:248] attempting to acquire leader lease openshift-kmm/kmm.sigs.x-k8s.io... I0228 09:36:40.784396 1 leaderelection.go:258] successfully acquired lease openshift-kmm/kmm.sigs.x-k8s.io I0228 09:36:40.784876 1 controller.go:185] kmm "msg"="Starting EventSource" "controller"="Module" "controllerGroup"="kmm.sigs.x-k8s.io" "controllerKind"="Module" "source"="kind source: *v1beta1.Module" I0228 09:36:40.784925 1 controller.go:185] kmm "msg"="Starting EventSource" "controller"="Module" "controllerGroup"="kmm.sigs.x-k8s.io" "controllerKind"="Module" "source"="kind source: *v1.DaemonSet" I0228 09:36:40.784968 1 controller.go:185] kmm "msg"="Starting EventSource" "controller"="Module" "controllerGroup"="kmm.sigs.x-k8s.io" "controllerKind"="Module" "source"="kind source: *v1.Build" I0228 09:36:40.785001 1 controller.go:185] kmm "msg"="Starting EventSource" "controller"="Module" "controllerGroup"="kmm.sigs.x-k8s.io" "controllerKind"="Module" "source"="kind source: *v1.Job" I0228 09:36:40.785025 1 controller.go:185] kmm "msg"="Starting EventSource" "controller"="Module" "controllerGroup"="kmm.sigs.x-k8s.io" "controllerKind"="Module" "source"="kind source: *v1.Node" I0228 09:36:40.785039 1 controller.go:193] kmm "msg"="Starting Controller" "controller"="Module" "controllerGroup"="kmm.sigs.x-k8s.io" "controllerKind"="Module" I0228 09:36:40.785458 1 controller.go:185] kmm "msg"="Starting EventSource" "controller"="PodNodeModule" "controllerGroup"="" "controllerKind"="Pod" "source"="kind source: *v1.Pod" I0228 09:36:40.786947 1 controller.go:185] kmm "msg"="Starting EventSource" "controller"="PreflightValidation" "controllerGroup"="kmm.sigs.x-k8s.io" "controllerKind"="PreflightValidation" "source"="kind source: *v1beta1.PreflightValidation" I0228 09:36:40.787406 1 controller.go:185] kmm "msg"="Starting EventSource" "controller"="PreflightValidation" "controllerGroup"="kmm.sigs.x-k8s.io" "controllerKind"="PreflightValidation" "source"="kind source: *v1.Build" I0228 09:36:40.787474 1 controller.go:185] kmm "msg"="Starting EventSource" "controller"="PreflightValidation" "controllerGroup"="kmm.sigs.x-k8s.io" "controllerKind"="PreflightValidation" "source"="kind source: *v1.Job" I0228 09:36:40.787488 1 controller.go:185] kmm "msg"="Starting EventSource" "controller"="PreflightValidation" "controllerGroup"="kmm.sigs.x-k8s.io" "controllerKind"="PreflightValidation" "source"="kind source: *v1beta1.Module" I0228 09:36:40.787603 1 controller.go:185] kmm "msg"="Starting EventSource" "controller"="NodeKernel" "controllerGroup"="" "controllerKind"="Node" "source"="kind source: *v1.Node" I0228 09:36:40.787634 1 controller.go:193] kmm "msg"="Starting Controller" "controller"="NodeKernel" "controllerGroup"="" "controllerKind"="Node" I0228 09:36:40.787680 1 controller.go:193] kmm "msg"="Starting Controller" "controller"="PreflightValidation" "controllerGroup"="kmm.sigs.x-k8s.io" "controllerKind"="PreflightValidation" I0228 09:36:40.785607 1 controller.go:185] kmm "msg"="Starting EventSource" "controller"="imagestream" "controllerGroup"="image.openshift.io" "controllerKind"="ImageStream" "source"="kind source: *v1.ImageStream" I0228 09:36:40.787822 1 controller.go:185] kmm "msg"="Starting EventSource" "controller"="preflightvalidationocp" "controllerGroup"="kmm.sigs.x-k8s.io" "controllerKind"="PreflightValidationOCP" "source"="kind source: *v1beta1.PreflightValidationOCP" I0228 09:36:40.787853 1 controller.go:193] kmm "msg"="Starting Controller" "controller"="imagestream" "controllerGroup"="image.openshift.io" "controllerKind"="ImageStream" I0228 09:36:40.787879 1 controller.go:185] kmm "msg"="Starting EventSource" "controller"="preflightvalidationocp" "controllerGroup"="kmm.sigs.x-k8s.io" "controllerKind"="PreflightValidationOCP" "source"="kind source: *v1beta1.PreflightValidation" I0228 09:36:40.787905 1 controller.go:193] kmm "msg"="Starting Controller" "controller"="preflightvalidationocp" "controllerGroup"="kmm.sigs.x-k8s.io" "controllerKind"="PreflightValidationOCP" I0228 09:36:40.786489 1 controller.go:193] kmm "msg"="Starting Controller" "controller"="PodNodeModule" "controllerGroup"="" "controllerKind"="Pod"
4.11.1.2. Gathering data for KMM-Hub
Procedure
Gather the data for the KMM Operator hub controller manager:
Set the
MUST_GATHER_IMAGEvariable:$ export MUST_GATHER_IMAGE=$(oc get deployment -n openshift-kmm-hub kmm-operator-hub-controller-manager -ojsonpath='{.spec.template.spec.containers[?(@.name=="manager")].env[?(@.name=="RELATED_IMAGES_MUST_GATHER")].value}')NoteUse the
-n <namespace>switch to specify a namespace if you installed KMM in a custom namespace.Run the
must-gathertool:$ oc adm must-gather --image="${MUST_GATHER_IMAGE}" -- /usr/bin/gather -u
View the Operator logs:
$ oc logs -fn openshift-kmm-hub deployments/kmm-operator-hub-controller-manager
Example 4.2. Example output
I0417 11:34:08.807472 1 request.go:682] Waited for 1.023403273s due to client-side throttling, not priority and fairness, request: GET:https://172.30.0.1:443/apis/tuned.openshift.io/v1?timeout=32s I0417 11:34:12.373413 1 listener.go:44] kmm-hub/controller-runtime/metrics "msg"="Metrics server is starting to listen" "addr"="127.0.0.1:8080" I0417 11:34:12.376253 1 main.go:150] kmm-hub/setup "msg"="Adding controller" "name"="ManagedClusterModule" I0417 11:34:12.376621 1 main.go:186] kmm-hub/setup "msg"="starting manager" I0417 11:34:12.377690 1 leaderelection.go:248] attempting to acquire leader lease openshift-kmm-hub/kmm-hub.sigs.x-k8s.io... I0417 11:34:12.378078 1 internal.go:366] kmm-hub "msg"="Starting server" "addr"={"IP":"127.0.0.1","Port":8080,"Zone":""} "kind"="metrics" "path"="/metrics" I0417 11:34:12.378222 1 internal.go:366] kmm-hub "msg"="Starting server" "addr"={"IP":"::","Port":8081,"Zone":""} "kind"="health probe" I0417 11:34:12.395703 1 leaderelection.go:258] successfully acquired lease openshift-kmm-hub/kmm-hub.sigs.x-k8s.io I0417 11:34:12.396334 1 controller.go:185] kmm-hub "msg"="Starting EventSource" "controller"="ManagedClusterModule" "controllerGroup"="hub.kmm.sigs.x-k8s.io" "controllerKind"="ManagedClusterModule" "source"="kind source: *v1beta1.ManagedClusterModule" I0417 11:34:12.396403 1 controller.go:185] kmm-hub "msg"="Starting EventSource" "controller"="ManagedClusterModule" "controllerGroup"="hub.kmm.sigs.x-k8s.io" "controllerKind"="ManagedClusterModule" "source"="kind source: *v1.ManifestWork" I0417 11:34:12.396430 1 controller.go:185] kmm-hub "msg"="Starting EventSource" "controller"="ManagedClusterModule" "controllerGroup"="hub.kmm.sigs.x-k8s.io" "controllerKind"="ManagedClusterModule" "source"="kind source: *v1.Build" I0417 11:34:12.396469 1 controller.go:185] kmm-hub "msg"="Starting EventSource" "controller"="ManagedClusterModule" "controllerGroup"="hub.kmm.sigs.x-k8s.io" "controllerKind"="ManagedClusterModule" "source"="kind source: *v1.Job" I0417 11:34:12.396522 1 controller.go:185] kmm-hub "msg"="Starting EventSource" "controller"="ManagedClusterModule" "controllerGroup"="hub.kmm.sigs.x-k8s.io" "controllerKind"="ManagedClusterModule" "source"="kind source: *v1.ManagedCluster" I0417 11:34:12.396543 1 controller.go:193] kmm-hub "msg"="Starting Controller" "controller"="ManagedClusterModule" "controllerGroup"="hub.kmm.sigs.x-k8s.io" "controllerKind"="ManagedClusterModule" I0417 11:34:12.397175 1 controller.go:185] kmm-hub "msg"="Starting EventSource" "controller"="imagestream" "controllerGroup"="image.openshift.io" "controllerKind"="ImageStream" "source"="kind source: *v1.ImageStream" I0417 11:34:12.397221 1 controller.go:193] kmm-hub "msg"="Starting Controller" "controller"="imagestream" "controllerGroup"="image.openshift.io" "controllerKind"="ImageStream" I0417 11:34:12.498335 1 filter.go:196] kmm-hub "msg"="Listing all ManagedClusterModules" "managedcluster"="local-cluster" I0417 11:34:12.498570 1 filter.go:205] kmm-hub "msg"="Listed ManagedClusterModules" "count"=0 "managedcluster"="local-cluster" I0417 11:34:12.498629 1 filter.go:238] kmm-hub "msg"="Adding reconciliation requests" "count"=0 "managedcluster"="local-cluster" I0417 11:34:12.498687 1 filter.go:196] kmm-hub "msg"="Listing all ManagedClusterModules" "managedcluster"="sno1-0" I0417 11:34:12.498750 1 filter.go:205] kmm-hub "msg"="Listed ManagedClusterModules" "count"=0 "managedcluster"="sno1-0" I0417 11:34:12.498801 1 filter.go:238] kmm-hub "msg"="Adding reconciliation requests" "count"=0 "managedcluster"="sno1-0" I0417 11:34:12.501947 1 controller.go:227] kmm-hub "msg"="Starting workers" "controller"="imagestream" "controllerGroup"="image.openshift.io" "controllerKind"="ImageStream" "worker count"=1 I0417 11:34:12.501948 1 controller.go:227] kmm-hub "msg"="Starting workers" "controller"="ManagedClusterModule" "controllerGroup"="hub.kmm.sigs.x-k8s.io" "controllerKind"="ManagedClusterModule" "worker count"=1 I0417 11:34:12.502285 1 imagestream_reconciler.go:50] kmm-hub "msg"="registered imagestream info mapping" "ImageStream"={"name":"driver-toolkit","namespace":"openshift"} "controller"="imagestream" "controllerGroup"="image.openshift.io" "controllerKind"="ImageStream" "dtkImage"="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:df42b4785a7a662b30da53bdb0d206120cf4d24b45674227b16051ba4b7c3934" "name"="driver-toolkit" "namespace"="openshift" "osImageVersion"="412.86.202302211547-0" "reconcileID"="e709ff0a-5664-4007-8270-49b5dff8bae9"