OpenShift rpm-ostree stuck in UpdateDeployment
Issue
Node become stuck during MachineConfigDaemon (MCD) update due to an issue with rpm-ostree not completing successfully. Logs will show what looks like an infinte loop of starting and stopping of the rpm-ostree service.
Jul 14 13:09:16 worker05 root[2753771]: machine-config-daemon[2610060]: Starting update from rendered-worker-0c4a7b6581b0065d403c2c7e9589eab2 to rendered-worker-efe5644e2541c01d1fe192c364b40cf5: &{osUpdate:true kargs:false fips:false passwd:false files:true units:true kernelType:false extensions:false}
Jul 14 13:09:16 worker05 root[2753772]: machine-config-daemon[2610060]: Update prepared; beginning drain
Jul 14 13:09:16 worker05 root[2753791]: machine-config-daemon[2610060]: drain complete
Jul 14 13:09:30 worker05 rpm-ostree[2689365]: client(id:machine-config-operator dbus:1.111674 unit:crio-6ef8ee933b95235c515b6109cb6f7aba203e92e3e87c692703bfbb98c5225e2a.scope uid:0) added; new total=1
Jul 14 13:09:30 worker05 rpm-ostree[2689365]: client(id:machine-config-operator dbus:1.111674 unit:crio-6ef8ee933b95235c515b6109cb6f7aba203e92e3e87c692703bfbb98c5225e2a.scope uid:0) vanished; remaining=0
Jul 14 13:09:30 worker05 rpm-ostree[2689365]: client(id:machine-config-operator dbus:1.111675 unit:crio-6ef8ee933b95235c515b6109cb6f7aba203e92e3e87c692703bfbb98c5225e2a.scope uid:0) added; new total=1
Jul 14 13:09:30 worker05 rpm-ostree[2689365]: client(id:machine-config-operator dbus:1.111675 unit:crio-6ef8ee933b95235c515b6109cb6f7aba203e92e3e87c692703bfbb98c5225e2a.scope uid:0) vanished; remaining=0
Jul 14 13:10:30 worker05 root[2754611]: machine-config-daemon[2610060]: Starting update from rendered-worker-0c4a7b6581b0065d403c2c7e9589eab2 to rendered-worker-efe5644e2541c01d1fe192c364b40cf5: &{osUpdate:true kargs:false fips:false passwd:false files:true units:true kernelType:false extensions:false}
Jul 14 13:10:30 worker05 root[2754612]: machine-config-daemon[2610060]: Update prepared; beginning drain
Jul 14 13:10:31 worker05 root[2754619]: machine-config-daemon[2610060]: drain complete
Jul 14 13:11:10 worker05 rpm-ostree[2689365]: client(id:machine-config-operator dbus:1.111676 unit:crio-6ef8ee933b95235c515b6109cb6f7aba203e92e3e87c692703bfbb98c5225e2a.scope uid:0) added; new total=1
Jul 14 13:11:10 worker05 rpm-ostree[2689365]: client(id:machine-config-operator dbus:1.111676 unit:crio-6ef8ee933b95235c515b6109cb6f7aba203e92e3e87c692703bfbb98c5225e2a.scope uid:0) vanished; remaining=0
Jul 14 13:11:11 worker05 rpm-ostree[2689365]: client(id:machine-config-operator dbus:1.111677 unit:crio-6ef8ee933b95235c515b6109cb6f7aba203e92e3e87c692703bfbb98c5225e2a.scope uid:0) added; new total=1
Jul 14 13:11:11 worker05 rpm-ostree[2689365]: client(id:machine-config-operator dbus:1.111677 unit:crio-6ef8ee933b95235c515b6109cb6f7aba203e92e3e87c692703bfbb98c5225e2a.scope uid:0) vanished; remaining=0
Jul 14 13:12:11 worker05 root[2755785]: machine-config-daemon[2610060]: Starting update from rendered-worker-0c4a7b6581b0065d403c2c7e9589eab2 to rendered-worker-efe5644e2541c01d1fe192c364b40cf5: &{osUpdate:true kargs:false fips:false passwd:false files:true units:true kernelType:false extensions:false}
Jul 14 13:12:11 worker05 root[2755786]: machine-config-daemon[2610060]: Update prepared; beginning drain
Jul 14 13:12:12 worker05 root[2755795]: machine-config-daemon[2610060]: drain complete
Environment
OpenShift Container Platform
- 4.x
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.