RHV host rebooted during IBM SVC node upgrade
Issue
-
While upgrading the fw on one of the IBM SVC node, some of the paths to multipath devices got failed, this resulted in a fence/reboot of RHV host:
May 30 00:01:08 host-1 multipathd: 361234567891234567891234567891111: remaining active paths: 7 May 30 00:01:09 host-1 multipathd: checker failed path 8:64 in map 361234567891234567891234567891111 May 30 00:01:09 host-1 multipathd: 361234567891234567891234567891111: remaining active paths: 6 [...] May 30 00:01:12 host-1 sanlock[2857]: 2020-05-30 00:01:12 2536547 [3880]: s4 delta_renew read timeout 10 sec offset 0 /dev/111-584c-4d97-9259-1234567894f6/ids May 30 00:01:12 host-1 sanlock[2857]: 2020-05-30 00:01:12 2536547 [3880]: s4 renewal error -202 delta_length 10 last_success 2536517 [...] May 30 00:01:12 host-1 sanlock[2857]: 2020-05-30 00:01:12 2536547 [3880]: s4 renewal error -202 delta_length 10 last_success 2536517 May 30 00:01:33 host-1 sanlock[2857]: 2020-05-30 00:01:33 2536568 [3880]: s4 renewal error -202 delta_length 20 last_success 2536517 May 30 00:02:25 host-1 sanlock[2857]: 2020-05-30 00:02:25 2536620 [3880]: s4 renewal error -202 delta_length 10 last_success 2536589 May 30 00:02:45 host-1 sanlock[2857]: 2020-05-30 00:02:45 2536640 [3880]: s4 renewal error -202 delta_length 20 last_success 2536589 May 30 00:03:06 host-1 sanlock[2857]: 2020-05-30 00:03:06 2536661 [3880]: s4 renewal error -202 delta_length 20 last_success 2536589 May 30 00:03:14 host-1 sanlock[2857]: 2020-05-30 00:03:14 2536669 [2857]: s4 check_our_lease failed 80 [...] May 30 00:03:04 host-1 wdmd[2882]: test warning now 2536659 ping 2536649 close 2007488 renewal 2536589 expire 2536669 client 2857 sanlock_99a01e03-584c-4d97-9259-5bce901104f6:6 May 30 00:03:04 host-1 wdmd[2882]: /dev/watchdog0 closed unclean May 30 00:03:34 host-1 wdmd[2882]: test failed rem 30 now 2536689 ping 2536649 close 2536659 renewal 2536589 expire 2536669 client 2857 sanlock_99a01e03-584c-4d97-9259-5bce901104f6:6 [...]
Environment
- Red Hat Virtualization 4.x
- IBM 2145 storage.
- device-mapper-multipath
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.