OpenShift Web Console pods constantly being redeployed
Environment
-
Red Hat OpenShift Container Platform. Versions:
- 4.21.
- 4.20.
-
Warning: a problem with similar symptoms was documented for earlier versions, but its root cause is different. For more information about that issue, see this link.
Issue
- Console pods become recreated every 10-45 minutes.
$ oc get pod -n openshift-console
NAME READY STATUS RESTARTS AGE
console-78f575c5b7-9spx9 1/1 Running 0 2m
console-78f575c5b7-rcw6g 1/1 Running 0 2m
downloads-5b8c85d48f-m42zg 1/1 Running 0 23h
downloads-5b8c85d48f-sfntc 1/1 Running 0 23h
- There should be
SyncLoopRefreshProgressingandDeploymentSyncDegradedmessages in a way similar to the one pasted below in the logs of the operator pod.
I0130 15:32:06.413640 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-console-operator", Name:"console-operator", UID:"e626c780-878f-4b89-9461-128aa81965ae", APIVersion:"apps/v1", ResourceVersion:
"", FieldPath:""}): type: 'Normal' reason: 'ConfigMapUpdated' Updated ConfigMap/console-config -n openshift-console:
cause by changes in data.console-config.yaml
I0130 15:32:06.420483 1 apps.go:155] Deployment "openshift-console/console" changes: {"metadata":{"annotations":{"console.openshift.io/console-config-version":"5317571","operator.openshift.io/spec-hash":"ab47360b77d51a243bc0ff894
acfc63fd5eefd10899262f56db3270f8cb94745"}},"spec":{"progressDeadlineSeconds":null,"revisionHistoryLimit":null,"template":{"metadata":{"annotations":{"console.openshift.io/console-config-version":"5317571"}},"spec":{"containers":[{"comm
and":["/opt/bridge/bin/bridge","--public-dir=/opt/bridge/static","--config=/var/console-config/console-config.yaml","--service-ca-file=/var/service-ca/service-ca.crt","--v=2"],"env":[{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVers
ion":"v1","fieldPath":"metadata.name"}}}],"image":"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:422e91c6e6f420673a7a7ed480f8e1123fabbf0f276c50d39e49de0f70dfe514","imagePullPolicy":"IfNotPresent","lifecycle":{"preStop":{"exec":
{"command":["sleep","25"]}}},"livenessProbe":{"failureThreshold":1,"httpGet":{"path":"/health","port":8443,"scheme":"HTTPS"},"periodSeconds":10,"successThreshold":1,"timeoutSeconds":10},"name":"console","ports":[{"containerPort":8443,"
name":"https","protocol":"TCP"}],"readinessProbe":{"failureThreshold":3,"httpGet":{"path":"/health","port":8443,"scheme":"HTTPS"},"periodSeconds":10,"successThreshold":1,"timeoutSeconds":1},"resources":{"requests":{"cpu":"10m","memory"
:"100Mi"}},"securityContext":{"allowPrivilegeEscalation":false,"capabilities":{"drop":["ALL"]},"readOnlyRootFilesystem":false},"startupProbe":{"failureThreshold":30,"httpGet":{"path":"/health","port":8443,"scheme":"HTTPS"},"periodSecon
ds":10,"successThreshold":1,"timeoutSeconds":10},"terminationMessagePolicy":"FallbackToLogsOnError","volumeMounts":[{"mountPath":"/var/serving-cert","name":"console-serving-cert","readOnly":true},{"mountPath":"/var/oauth-config","name"
:"console-oauth-config","readOnly":true},{"mountPath":"/var/console-config","name":"console-config","readOnly":true},{"mountPath":"/var/service-ca","name":"service-ca","readOnly":true},{"mountPath":"/etc/pki/ca-trust/extracted/pem","na
me":"trusted-ca-bundle","readOnly":true},{"mountPath":"/var/oauth-serving-cert","name":"oauth-serving-cert","readOnly":true}]}],"dnsPolicy":null,"serviceAccount":null,"volumes":[{"name":"console-serving-cert","secret":{"secretName":"co
nsole-serving-cert"}},{"name":"console-oauth-config","secret":{"secretName":"console-oauth-config"}},{"configMap":{"name":"console-config"},"name":"console-config"},{"configMap":{"name":"service-ca"},"name":"service-ca"},{"configMap":{
"items":[{"key":"ca-bundle.crt","path":"tls-ca-bundle.pem"}],"name":"trusted-ca-bundle"},"name":"trusted-ca-bundle"},{"configMap":{"name":"oauth-serving-cert"},"name":"oauth-serving-cert"}]}}}}
I0130 15:32:06.427216 1 warnings.go:110] "Warning: spec.template.spec.nodeSelector[node-role.kubernetes.io/master]: use \"node-role.kubernetes.io/control-plane\" instead"
E0130 15:32:06.427365 1 status.go:130] SyncLoopRefreshProgressing InProgress working toward version 4.22.0-ec.1, 2 replicas available
I0130 15:32:06.427635 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-console-operator", Name:"console-operator", UID:"e626c780-878f-4b89-9461-128aa81965ae", APIVersion:"apps/v1", ResourceVersion:
"", FieldPath:""}): type: 'Normal' reason: 'DeploymentUpdated' Updated Deployment.apps/console -n openshift-console because it changed
I0130 15:32:06.450639 1 status_controller.go:230] clusteroperator/console diff {"status":{"conditions":[{"lastTransitionTime":"2026-01-27T17:07:32Z","message":"All is well","reason":"AsExpected","status":"False","type":"Degraded"
},{"lastTransitionTime":"2026-01-30T15:32:06Z","message":"SyncLoopRefreshProgressing: working toward version 4.22.0-ec.1, 2 replicas available","reason":"SyncLoopRefresh_InProgress","status":"True","type":"Progressing"},{"lastTransitio
nTime":"2026-01-27T17:07:32Z","message":"All is well","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2026-01-30T08:14:45Z","message":"All is well","reason":"AsExpected","status":"True","type":"Upgradea
ble"},{"lastTransitionTime":"2026-01-27T16:53:37Z","reason":"NoData","status":"Unknown","type":"EvaluationConditionsDetected"}]}}
I0130 15:32:06.454436 1 apps.go:155] Deployment "openshift-console/console" changes: {"spec":{"progressDeadlineSeconds":null,"revisionHistoryLimit":null,"template":{"spec":{"containers":[{"command":["/opt/bridge/bin/bridge","--pu
blic-dir=/opt/bridge/static","--config=/var/console-config/console-config.yaml","--service-ca-file=/var/service-ca/service-ca.crt","--v=2"],"env":[{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name
"}}}],"image":"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:422e91c6e6f420673a7a7ed480f8e1123fabbf0f276c50d39e49de0f70dfe514","imagePullPolicy":"IfNotPresent","lifecycle":{"preStop":{"exec":{"command":["sleep","25"]}}},"livene
ssProbe":{"failureThreshold":1,"httpGet":{"path":"/health","port":8443,"scheme":"HTTPS"},"periodSeconds":10,"successThreshold":1,"timeoutSeconds":10},"name":"console","ports":[{"containerPort":8443,"name":"https","protocol":"TCP"}],"re
adinessProbe":{"failureThreshold":3,"httpGet":{"path":"/health","port":8443,"scheme":"HTTPS"},"periodSeconds":10,"successThreshold":1,"timeoutSeconds":1},"resources":{"requests":{"cpu":"10m","memory":"100Mi"}},"securityContext":{"allow
PrivilegeEscalation":false,"capabilities":{"drop":["ALL"]},"readOnlyRootFilesystem":false},"startupProbe":{"failureThreshold":30,"httpGet":{"path":"/health","port":8443,"scheme":"HTTPS"},"periodSeconds":10,"successThreshold":1,"timeout
Seconds":10},"terminationMessagePolicy":"FallbackToLogsOnError","volumeMounts":[{"mountPath":"/var/serving-cert","name":"console-serving-cert","readOnly":true},{"mountPath":"/var/oauth-config","name":"console-oauth-config","readOnly":t
rue},{"mountPath":"/var/console-config","name":"console-config","readOnly":true},{"mountPath":"/var/service-ca","name":"service-ca","readOnly":true},{"mountPath":"/etc/pki/ca-trust/extracted/pem","name":"trusted-ca-bundle","readOnly":t
rue},{"mountPath":"/var/oauth-serving-cert","name":"oauth-serving-cert","readOnly":true}]}],"dnsPolicy":null,"serviceAccount":null,"volumes":[{"name":"console-serving-cert","secret":{"secretName":"console-serving-cert"}},{"name":"conso
le-oauth-config","secret":{"secretName":"console-oauth-config"}},{"configMap":{"name":"console-config"},"name":"console-config"},{"configMap":{"name":"service-ca"},"name":"service-ca"},{"configMap":{"items":[{"key":"ca-bundle.crt","pat
h":"tls-ca-bundle.pem"}],"name":"trusted-ca-bundle"},"name":"trusted-ca-bundle"},{"configMap":{"name":"oauth-serving-cert"},"name":"oauth-serving-cert"}]}}}}
I0130 15:32:06.459548 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-console-operator", Name:"console-operator", UID:"e626c780-878f-4b89-9461-128aa81965ae", APIVersion:"apps/v1", ResourceVersion:
"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/console changed: Progressing changed from False to True ("SyncLoopRefreshProgressing: working toward version 4.22.0-ec.1, 2 replicas availabl
e")
E0130 15:32:06.461719 1 status.go:130] DeploymentSyncDegraded FailedApply Operation cannot be fulfilled on deployments.apps "console": the object has been modified; please apply your changes to the latest version and try again
I0130 15:32:06.461807 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-console-operator", Name:"console-operator", UID:"e626c780-878f-4b89-9461-128aa81965ae", APIVersion:"apps/v1", ResourceVersion:
"", FieldPath:""}): type: 'Warning' reason: 'DeploymentUpdateFailed' Failed to update Deployment.apps/console -n openshift-console: Operation cannot be fulfilled on deployments.apps "console": the object has been modified; please apply
your changes to the latest version and try again
E0130 15:32:06.477349 1 base_controller.go:279] "Unhandled Error" err="ConsoleOperator reconciliation failed: Operation cannot be fulfilled on deployments.apps \"console\": the object has been modified; please apply your changes
to the latest version and try again"
E0130 15:32:06.488808 1 status.go:130] SyncLoopRefreshProgressing InProgress working toward version 4.22.0-ec.1, 1 replicas available
I0130 15:32:06.553503 1 status_controller.go:230] clusteroperator/console diff {"status":{"conditions":[{"lastTransitionTime":"2026-01-27T17:07:32Z","message":"DeploymentSyncDegraded: Operation cannot be fulfilled on deployments.
apps \"console\": the object has been modified; please apply your changes to the latest version and try again","reason":"AsExpected","status":"False","type":"Degraded"},{"lastTransitionTime":"2026-01-30T15:32:06Z","message":"All is wel
l","reason":"AsExpected","status":"False","type":"Progressing"},{"lastTransitionTime":"2026-01-27T17:07:32Z","message":"All is well","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2026-01-30T08:14:45Z"
,"message":"All is well","reason":"AsExpected","status":"True","type":"Upgradeable"},{"lastTransitionTime":"2026-01-27T16:53:37Z","reason":"NoData","status":"Unknown","type":"EvaluationConditionsDetected"}]}}
I0130 15:32:06.822109 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-console-operator", Name:"console-operator", UID:"e626c780-878f-4b89-9461-128aa81965ae", APIVersion:"apps/v1", ResourceVersion:
"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/console changed: Degraded message changed from "All is well" to "DeploymentSyncDegraded: Operation cannot be fulfilled on deployments.apps \"
console\": the object has been modified; please apply your changes to the latest version and try again",Progressing changed from True to False ("All is well")
Resolution
-
This is a known bug: OCPBUGS-74872.
- At the time when this is being written, its fix is not still generally available. To get updated, check the bug ticket or contact Red Hat Support.
-
As a workaround, the OpenShift Data Foundation (ODF) Console plugin can be disabled.
- If the OpenShift Data Foundation (ODF) Console plugin is not installed in your cluster and you would like to investigate a workaround, please contact Red Hat Support.
Root Cause
- This problem happens as a consequence of the installation of some plugins. It is known that the OpenShift Data Foundation (ODF) Console plugin is one of them, but probably not the only one.
Diagnostic Steps
- Check the age of the Console pods:
$ oc get pod -n openshift-console
- Check the logs of the Console Operator pods:
- Replace the string
9b5dcd8dc-9c6rwwith the one showing up in the pod name running in the corresponding cluster.
- Replace the string
$ oc logs console-operator-9b5dcd8dc-9c6rw -n openshift-console-operator
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
Comments