The command "oc adm migrate storage" returns the error: "invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, ..."

Solution Verified - Updated -

Issue

As part of the prerequisites for the upgrade of an OCP cluster the documentation states:

The day before the upgrade, validate OpenShift Container Platform storage migration to ensure potential issues are resolved prior to the outage window:

$ oc adm migrate storage --include=* --loglevel=2 --confirm --config 
/etc/origin/master/admin.kubeconfig

When running the above command errors like the following appear:

E0924 04:04:32.348706 error:     -n dev-jef pods/pi2-5-nvrlv: Pod "pi2-5-nvrlv" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds` or `spec.tolerations` (only additions to existing tolerations)
{"Volumes":[{"Name":"default-token-zkcfz","HostPath":null,"EmptyDir":null,"GCEPersistentDisk":null,"AWSElasticBlockStore":null,"GitRepo":null,"Secret":{"SecretName":"default-token-zkcfz","Items":null,"DefaultMode":420,"Optional":null},"NFS":null,"ISCSI":null,"Glusterfs":null,"PersistentVolumeClaim":null,"RBD":null,"Quobyte":null,"FlexVolume":null,"Cinder":null,"CephFS":null,"Flocker":null,"DownwardAPI":null,"FC":null,"AzureFile":null,"ConfigMap":null,"VsphereVolume":null,"AzureDisk":null,"PhotonPersistentDisk":null,"Projected":null,"PortworxVolume":null,"ScaleIO":null,"StorageOS":null}],"InitContainers":null,"Containers":[{"Name":"pi2","Image":"registry.cluster.ocpcart1.ocp.tc.corp/openshift/rhel-tools:latest","Command":null,"Args":null,"WorkingDir":"","Ports":null,"EnvFrom":null,"Env":null,"Resources":{"Limits":

A: {"cpu":"2","memory":"2Gi"},"Requests":{"cpu":"100m","memory":"512Mi"}},"VolumeMounts":[{"Name":"default-token-zkcfz","ReadOnly":true,"MountPath":"/var/run/secrets/kubernetes.io/serviceaccount","SubPath":"","MountPropagation":null}],"VolumeDevices":null,"LivenessProbe":null,"ReadinessProbe":null,"Lifecycle":null,"TerminationMessagePath":"/dev/termination-log","TerminationMessagePolicy":"File","ImagePullPolicy":"Always","SecurityContext":{"Capabilities":{"Add":null,"Drop":["KILL","MKNOD","SETGID","SETUID"]},"Privileged":null,"SELinuxOptions":null,"RunAsUser":1000310000,"RunAsGroup":null,"RunAsNonRoot":null,"ReadOnlyRootFilesystem":null,"AllowPrivilegeEscalation":null},"Stdin":true,"StdinOnce":false,"TTY":true}],"RestartPolicy":"Always","TerminationGracePeriodSeconds":30,"ActiveDeadlineSeconds":null,"DNSPolicy":"ClusterFirst","NodeSelector":{"zone":"com"},"ServiceAccountName":"default","AutomountServiceAccountToken":null,"NodeName":"node-003.ocpcart1.ocp.tc.corp","SecurityContext":{"HostNetwork":false,"HostPID":false,"HostIPC":false,"ShareProcessNamespace":null,"SELinuxOptions":{"User":"","Role":"","Type":"","Level":"s0:c18,c2"},"RunAsUser":null,"RunAsGroup":null,"RunAsNonRoot":null,"SupplementalGroups":null,"FSGroup":1000310000,"Sysctls":null},"ImagePullSecrets":[{"Name":"default-dockercfg-tbk9h"}],"Hostname":"","Subdomain":"","Affinity":null,"SchedulerName":"default-scheduler","Tolerations":null,"HostAliases":null,"PriorityClassName":"","Priority":0,"DNSConfig":null,"ReadinessGates":null}

B: null,"Requests":null},"VolumeMounts":[{"Name":"default-token-zkcfz","ReadOnly":true,"MountPath":"/var/run/secrets/kubernetes.io/serviceaccount","SubPath":"","MountPropagation":null}],"VolumeDevices":null,"LivenessProbe":null,"ReadinessProbe":null,"Lifecycle":null,"TerminationMessagePath":"/dev/termination-log","TerminationMessagePolicy":"File","ImagePullPolicy":"Always","SecurityContext":{"Capabilities":{"Add":null,"Drop":["KILL","MKNOD","SETGID","SETUID"]},"Privileged":null,"SELinuxOptions":null,"RunAsUser":1000310000,"RunAsGroup":null,"RunAsNonRoot":null,"ReadOnlyRootFilesystem":null,"AllowPrivilegeEscalation":null},"Stdin":true,"StdinOnce":false,"TTY":true}],"RestartPolicy":"Always","TerminationGracePeriodSeconds":30,"ActiveDeadlineSeconds":null,"DNSPolicy":"ClusterFirst","NodeSelector":{"zone":"com"},"ServiceAccountName":"default","AutomountServiceAccountToken":null,"NodeName":"node-003.ocpcart1.ocp.tc.corp","SecurityContext":{"HostNetwork":false,"HostPID":false,"HostIPC":false,"ShareProcessNamespace":null,"SELinuxOptions":{"User":"","Role":"","Type":"","Level":"s0:c18,c2"},"RunAsUser":null,"RunAsGroup":null,"RunAsNonRoot":null,"SupplementalGroups":null,"FSGroup":1000310000,"Sysctls":null},"ImagePullSecrets":[{"Name":"default-dockercfg-tbk9h"}],"Hostname":"","Subdomain":"","Affinity":null,"SchedulerName":"default-scheduler","Tolerations":null,"HostAliases":null,"PriorityClassName":"","Priority":0,"DNSConfig":null,"ReadinessGates":null}

The information in the error includes a sort of diff between the current definition of the object in the etcd storage, the section prefixed by B:, and the new definition that is being stored, the section prefixed by A:. In the block just above A: you find the part of the object's definition that is common to both old and new versions.

In this example the "oc adm migrate storage" command is trying to save a new definition of the pod in which the Resources section inside spec changes from empty:

Resources:{"Limits":null,"Requests":null}

To a a set of specific values

Resources:{"Limits": {"cpu":"2","memory":"2Gi"},"Requests":{"cpu":"100m","memory":"512Mi"}}

This operation is not allowed as for the error message "pod updates may not change fields other than.."
The reason for this behaviour could be the presence of limitranges in the projects where the pods exist (see Root Cause section)

Environment

  • Red Hat OpenShift Container Platform
    • 3.11

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content