Unable to bring one of my masters ready

Posted on

A couple of days ago one of my 3 masters went "not ready" and i've not been able to make it ready since. It is a 3.11 version of OCP. This is what i see: systemctl start atomic-openshift-node
Job for atomic-openshift-node.service failed because the control process exited with error code. See "systemctl status atomic-openshift-node.service" and "journalctl -xe" for details. "journalctl -xe" showes this: journalctl -xe
Oct 06 10:24:44 ocptstmstr601.tmw.com sshd[6147]: pam_unix(sshd:session): session closed for user ansible
Oct 06 10:24:44 ocptstmstr601.tmw.com systemd-logind[929]: Removed session 6.
-- Subject: Session 6 has been terminated
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

-- Documentation: http://www.freedesktop.org/wiki/Software/systemd/multiseat

-- A session with the ID 6 has been terminated.
Oct 06 10:24:44 ocptstmstr601.tmw.com systemd[1]: Removed slice User Slice of ansible.
-- Subject: Unit user-90008.slice has finished shutting down
-- Defined-By: systemd

-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

-- Unit user-90008.slice has finished shutting down.
Oct 06 10:24:48 ocptstmstr601.tmw.com container-storage-setup[6403]: INFO: Waiting for device /dev/mapper/docker--vg-docker--pool to be available. Wait time remaining is 40 seconds
Oct 06 10:24:50 ocptstmstr601.tmw.com telegraf[1451]: 2022-10-06T15:24:50Z E! [outputs.influxdb] when writing to [http://localhost:8086]: Post http://localhost:8086/write?db=telegraf: dial tcp [::1]:80
Oct 06 10:24:50 ocptstmstr601.tmw.com telegraf[1451]: 2022-10-06T15:24:50Z E! [agent] Error writing to output [influxdb]: could not write any address
Oct 06 10:24:52 ocptstmstr601.tmw.com goferd[1454]: [INFO][worker-0] gofer.messaging.adapter.proton.connection:131 - closed: proton+amqps://tbsat6cap601.tmw.com:5647
Oct 06 10:24:52 ocptstmstr601.tmw.com goferd[1454]: [INFO][worker-0] gofer.messaging.adapter.connect:28 - connecting: proton+amqps://tbsat6cap601.tmw.com:5647
Oct 06 10:24:52 ocptstmstr601.tmw.com goferd[1454]: [INFO][worker-0] gofer.messaging.adapter.proton.connection:87 - open: URL: amqps://tbsat6cap601.tmw.com:5647|SSL: ca: /etc/rhsm/ca/katello-default-ca
Oct 06 10:24:52 ocptstmstr601.tmw.com goferd[1454]: [INFO][worker-0] gofer.messaging.adapter.proton.connection:92 - opened: proton+amqps://tbsat6cap601.tmw.com:5647
Oct 06 10:24:52 ocptstmstr601.tmw.com goferd[1454]: [INFO][worker-0] gofer.messaging.adapter.connect:30 - connected: proton+amqps://tbsat6cap601.tmw.com:5647
Oct 06 10:24:52 ocptstmstr601.tmw.com goferd[1454]: [WARNING][worker-0] gofer.messaging.adapter.proton.reliability:48 - receiver aa6c7921-13ac-4138-bf19-6a8ee69c1704 from None closed due to: Condition(
Oct 06 10:24:53 ocptstmstr601.tmw.com container-storage-setup[6403]: INFO: Waiting for device /dev/mapper/docker--vg-docker--pool to be available. Wait time remaining is 35 seconds
Oct 06 10:24:58 ocptstmstr601.tmw.com container-storage-setup[6403]: INFO: Waiting for device /dev/mapper/docker--vg-docker--pool to be available. Wait time remaining is 30 seconds
Oct 06 10:25:00 ocptstmstr601.tmw.com telegraf[1451]: 2022-10-06T15:25:00Z E! [outputs.influxdb] when writing to [http://localhost:8086]: Post http://localhost:8086/write?db=telegraf: dial tcp [::1]:80
Oct 06 10:25:00 ocptstmstr601.tmw.com telegraf[1451]: 2022-10-06T15:25:00Z E! [agent] Error writing to output [influxdb]: could not write any address
Oct 06 10:25:02 ocptstmstr601.tmw.com snmpd[1458]: Connection from UDP: [172.26.7.221]:65300->[172.31.21.48]:161

Any thoughts as to what i might try?

Responses