OpenStack compute node/physical server reboots when SAN switch is rebooted OR fluctuations are encountered in connectivity to the storage.

Solution Verified - Updated -

Issue

  • OpenStack compute node/physical server reboots/crashes when SAN switch is rebooted OR fluctuations are encountered in connectivity to the storage.
    Vmcore logs :
[571553.711255] device-mapper: multipath: Failing path 128:176.
[571553.711265] device-mapper: multipath: Failing path 70:160.
[571553.711274] device-mapper: multipath: Failing path 71:64.
[571553.711283] device-mapper: multipath: Failing path 71:32.
[571553.711292] device-mapper: multipath: Failing path 71:208.
[571553.711300] device-mapper: multipath: Failing path 71:240.
[571553.711309] device-mapper: multipath: Failing path 128:144.
[571553.711317] device-mapper: multipath: Failing path 128:112.
[571553.711331] device-mapper: multipath: Failing path 128:96.
[571553.711342] device-mapper: multipath: Failing path 128:0.
[571553.719778] sd 15:0:9:0: alua: port group 01 state A preferred supports tolusnA
[571553.719939] sd 15:0:9:0: alua: port group 01 state A preferred supports tolusnA
[571553.721913] BUG: unable to handle kernel NULL pointer dereference at 0000000000000090
[571553.730230] IP: [<ffffffffc069ce13>] __lpfc_sli_release_iocbq_s4+0x63/0x260 [lpfc]
[571553.738094] PGD 800000af8eb13067 PUD bdd0685067 PMD 0
[571553.743514] Oops: 0000 [#1] SMP
...
[571553.893851]  wmi drm_panel_orientation_quirks dm_multipath nfit libnvdimm sunrpc dm_mirror dm_region_hash dm_log dm_mod
[571553.904439] CPU: 15 PID: 2674 Comm: lpfc_worker_2 Kdump: loaded Tainted: G           OE  ------------ T 3.10.0-1062.12.1.el7.x86_64 #1
[571553.917568] Hardware name: Dell Inc. PowerEdge R740/0JMK61, BIOS 2.6.4 04/09/2020
[571553.925642] task: ffff9fe7f067b150 ti: ffff9fe7e6fe4000 task.ti: ffff9fe7e6fe4000
[571553.933732] RIP: 0010:[<ffffffffc069ce13>]  [<ffffffffc069ce13>] __lpfc_sli_release_iocbq_s4+0x63/0x260 [lpfc]
[571553.944407] RSP: 0018:ffff9fe7e6fe7b20  EFLAGS: 00010046
[571553.950373] RAX: 0000000000100000 RBX: ffffa047eab88e00 RCX: 000000010040002e
[571553.958166] RDX: 0000000000000000 RSI: ffffa047eab88e00 RDI: ffffa047f4c1c000
[571553.965958] RBP: ffff9fe7e6fe7b50 R08: ffffa0478ffc7c40 R09: 000000010040002e
[571553.974153] R10: 000000008ffc7f01 R11: ffffa0478ffc7c40 R12: ffffa047f4c1c000
[571553.981969] R13: ffff9fe7f8874060 R14: ffffa047eab88e00 R15: ffffa047f4c1c000
[571553.989788] FS:  0000000000000000(0000) GS:ffffa0487d1c0000(0000) knlGS:0000000000000000
[571553.999033] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[571554.005500] CR2: 0000000000000090 CR3: 0000009a9fab4000 CR4: 00000000007627e0
[571554.013454] PKRU: 00000000
[571554.016910] Call Trace:
[571554.020139]  [<ffffffffc069f407>] lpfc_sli_release_iocbq+0x37/0x60 [lpfc]
[571554.027696]  [<ffffffffc06bd83e>] lpfc_els_free_iocb+0x14e/0x1d0 [lpfc]
[571554.035084]  [<ffffffffc06c1b03>] lpfc_cmpl_els_prli+0xe3/0x210 [lpfc]
[571554.042392]  [<ffffffffc06a60bd>] lpfc_sli_sp_handle_rspiocb+0x3fd/0x780 [lpfc]
[571554.050491]  [<ffffffffc06cea36>] ? lpfc_mbx_cmpl_reg_login+0xe6/0x160 [lpfc]
[571554.058423]  [<ffffffff9c2af1f5>] ? mod_timer+0x1b5/0x230
[571554.064628]  [<ffffffffc06b0172>] lpfc_sli_handle_slow_ring_event_s4+0x192/0x260 [lpfc]
[571554.073466]  [<ffffffffc06a03e2>] lpfc_sli_handle_slow_ring_event+0x12/0x20 [lpfc]
[571554.081862]  [<ffffffffc06d3afc>] lpfc_work_done+0x94c/0x14a0 [lpfc]
[571554.089046]  [<ffffffff9c9805c2>] ? __schedule+0x402/0x840
[571554.095378]  [<ffffffffc06d46c0>] lpfc_do_work+0x70/0x1e0 [lpfc]
[571554.102236]  [<ffffffff9c2c72e0>] ? wake_up_atomic_t+0x30/0x30
[571554.108927]  [<ffffffffc06d4650>] ? lpfc_work_done+0x14a0/0x14a0 [lpfc]
[571554.116398]  [<ffffffff9c2c61f1>] kthread+0xd1/0xe0
[571554.122246]  [<ffffffff9c2c6120>] ? insert_kthread_work+0x40/0x40
[571554.129181]  [<ffffffff9c98dd1d>] ret_from_fork_nospec_begin+0x7/0x21
[571554.136453]  [<ffffffff9c2c6120>] ? insert_kthread_work+0x40/0x40
[571554.143383] Code: 28 48 c7 00 00 00 00 00 4d 85 ed 0f 84 a6 00 00 00 8b 86 74 01 00 00 a9 00 00 80 00 0f 85 76 01 00 00 48 8b 97 98 02 00 00 a8 40 <4c> 8b b2 90 00 00 00 74 0b 41 83 7d 24 02 0f 85 19 01 00 00 4d
[571554.165427] RIP  [<ffffffffc069ce13>] __lpfc_sli_release_iocbq_s4+0x63/0x260 [lpfc]
[571554.173936]  RSP <ffff9fe7e6fe7b20>
[571554.178233] CR2: 0000000000000090

Environment

  • Red Hat Enterprise Linux 7
    • Emulex HBA

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content