RHV host rebooted during IBM SVC node upgrade

Solution Verified - Updated -

Issue

  • While upgrading the fw on one of the IBM SVC node, some of the paths to multipath devices got failed, this resulted in a fence/reboot of RHV host:

    May 30 00:01:08 host-1 multipathd: 361234567891234567891234567891111: remaining active paths: 7
    May 30 00:01:09 host-1 multipathd: checker failed path 8:64 in map 361234567891234567891234567891111
    May 30 00:01:09 host-1 multipathd: 361234567891234567891234567891111: remaining active paths: 6
    [...]
    May 30 00:01:12 host-1 sanlock[2857]: 2020-05-30 00:01:12 2536547 [3880]: s4 delta_renew read timeout 10 sec offset 0 /dev/111-584c-4d97-9259-1234567894f6/ids
    May 30 00:01:12 host-1 sanlock[2857]: 2020-05-30 00:01:12 2536547 [3880]: s4 renewal error -202 delta_length 10 last_success 2536517
    [...]
    May 30 00:01:12 host-1 sanlock[2857]: 2020-05-30 00:01:12 2536547 [3880]: s4 renewal error -202 delta_length 10 last_success 2536517
    May 30 00:01:33 host-1 sanlock[2857]: 2020-05-30 00:01:33 2536568 [3880]: s4 renewal error -202 delta_length 20 last_success 2536517
    May 30 00:02:25 host-1 sanlock[2857]: 2020-05-30 00:02:25 2536620 [3880]: s4 renewal error -202 delta_length 10 last_success 2536589
    May 30 00:02:45 host-1 sanlock[2857]: 2020-05-30 00:02:45 2536640 [3880]: s4 renewal error -202 delta_length 20 last_success 2536589
    May 30 00:03:06 host-1 sanlock[2857]: 2020-05-30 00:03:06 2536661 [3880]: s4 renewal error -202 delta_length 20 last_success 2536589
    May 30 00:03:14 host-1 sanlock[2857]: 2020-05-30 00:03:14 2536669 [2857]: s4 check_our_lease failed 80
    [...]
    May 30 00:03:04 host-1 wdmd[2882]: test warning now 2536659 ping 2536649 close 2007488 renewal 2536589 expire 2536669 client 2857 sanlock_99a01e03-584c-4d97-9259-5bce901104f6:6
    May 30 00:03:04 host-1 wdmd[2882]: /dev/watchdog0 closed unclean
    May 30 00:03:34 host-1 wdmd[2882]: test failed rem 30 now 2536689 ping 2536649 close 2536659 renewal 2536589 expire 2536669 client 2857 sanlock_99a01e03-584c-4d97-9259-5bce901104f6:6
    [...]
    

Environment

  • Red Hat Virtualization 4.x
  • IBM 2145 storage.
  • device-mapper-multipath

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content