OpenStack compute node reboots when SAN switch is rebooted OR fluctuations are encountered in connectivity to the storage.

Solution Unverified - Updated -

Issue

  • OpenStack compute node gets crash when SAN switch is rebooted OR fluctuations are encountered in connectivity to the storage.
    Vmcore logs :
[571553.711255] device-mapper: multipath: Failing path 128:176.
[571553.711265] device-mapper: multipath: Failing path 70:160.
[571553.711274] device-mapper: multipath: Failing path 71:64.
[571553.711283] device-mapper: multipath: Failing path 71:32.
[571553.711292] device-mapper: multipath: Failing path 71:208.
[571553.711300] device-mapper: multipath: Failing path 71:240.
[571553.711309] device-mapper: multipath: Failing path 128:144.
[571553.711317] device-mapper: multipath: Failing path 128:112.
[571553.711331] device-mapper: multipath: Failing path 128:96.
[571553.711342] device-mapper: multipath: Failing path 128:0.
[571553.719778] sd 15:0:9:0: alua: port group 01 state A preferred supports tolusnA
[571553.719939] sd 15:0:9:0: alua: port group 01 state A preferred supports tolusnA
[571553.721913] BUG: unable to handle kernel NULL pointer dereference at 0000000000000090
[571553.730230] IP: [<ffffffffc069ce13>] __lpfc_sli_release_iocbq_s4+0x63/0x260 [lpfc]
[571553.738094] PGD 800000af8eb13067 PUD bdd0685067 PMD 0
[571553.743514] Oops: 0000 [#1] SMP
...
[571553.893851]  wmi drm_panel_orientation_quirks dm_multipath nfit libnvdimm sunrpc dm_mirror dm_region_hash dm_log dm_mod
[571553.904439] CPU: 15 PID: 2674 Comm: lpfc_worker_2 Kdump: loaded Tainted: G           OE  ------------ T 3.10.0-1062.12.1.el7.x86_64 #1
[571553.917568] Hardware name: Dell Inc. PowerEdge R740/0JMK61, BIOS 2.6.4 04/09/2020
[571553.925642] task: ffff9fe7f067b150 ti: ffff9fe7e6fe4000 task.ti: ffff9fe7e6fe4000
[571553.933732] RIP: 0010:[<ffffffffc069ce13>]  [<ffffffffc069ce13>] __lpfc_sli_release_iocbq_s4+0x63/0x260 [lpfc]
[571553.944407] RSP: 0018:ffff9fe7e6fe7b20  EFLAGS: 00010046
[571553.950373] RAX: 0000000000100000 RBX: ffffa047eab88e00 RCX: 000000010040002e
[571553.958166] RDX: 0000000000000000 RSI: ffffa047eab88e00 RDI: ffffa047f4c1c000
[571553.965958] RBP: ffff9fe7e6fe7b50 R08: ffffa0478ffc7c40 R09: 000000010040002e
[571553.974153] R10: 000000008ffc7f01 R11: ffffa0478ffc7c40 R12: ffffa047f4c1c000
[571553.981969] R13: ffff9fe7f8874060 R14: ffffa047eab88e00 R15: ffffa047f4c1c000
[571553.989788] FS:  0000000000000000(0000) GS:ffffa0487d1c0000(0000) knlGS:0000000000000000
[571553.999033] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[571554.005500] CR2: 0000000000000090 CR3: 0000009a9fab4000 CR4: 00000000007627e0
[571554.013454] PKRU: 00000000
[571554.016910] Call Trace:
[571554.020139]  [<ffffffffc069f407>] lpfc_sli_release_iocbq+0x37/0x60 [lpfc]
[571554.027696]  [<ffffffffc06bd83e>] lpfc_els_free_iocb+0x14e/0x1d0 [lpfc]
[571554.035084]  [<ffffffffc06c1b03>] lpfc_cmpl_els_prli+0xe3/0x210 [lpfc]
[571554.042392]  [<ffffffffc06a60bd>] lpfc_sli_sp_handle_rspiocb+0x3fd/0x780 [lpfc]
[571554.050491]  [<ffffffffc06cea36>] ? lpfc_mbx_cmpl_reg_login+0xe6/0x160 [lpfc]
[571554.058423]  [<ffffffff9c2af1f5>] ? mod_timer+0x1b5/0x230
[571554.064628]  [<ffffffffc06b0172>] lpfc_sli_handle_slow_ring_event_s4+0x192/0x260 [lpfc]
[571554.073466]  [<ffffffffc06a03e2>] lpfc_sli_handle_slow_ring_event+0x12/0x20 [lpfc]
[571554.081862]  [<ffffffffc06d3afc>] lpfc_work_done+0x94c/0x14a0 [lpfc]
[571554.089046]  [<ffffffff9c9805c2>] ? __schedule+0x402/0x840
[571554.095378]  [<ffffffffc06d46c0>] lpfc_do_work+0x70/0x1e0 [lpfc]
[571554.102236]  [<ffffffff9c2c72e0>] ? wake_up_atomic_t+0x30/0x30
[571554.108927]  [<ffffffffc06d4650>] ? lpfc_work_done+0x14a0/0x14a0 [lpfc]
[571554.116398]  [<ffffffff9c2c61f1>] kthread+0xd1/0xe0
[571554.122246]  [<ffffffff9c2c6120>] ? insert_kthread_work+0x40/0x40
[571554.129181]  [<ffffffff9c98dd1d>] ret_from_fork_nospec_begin+0x7/0x21
[571554.136453]  [<ffffffff9c2c6120>] ? insert_kthread_work+0x40/0x40
[571554.143383] Code: 28 48 c7 00 00 00 00 00 4d 85 ed 0f 84 a6 00 00 00 8b 86 74 01 00 00 a9 00 00 80 00 0f 85 76 01 00 00 48 8b 97 98 02 00 00 a8 40 <4c> 8b b2 90 00 00 00 74 0b 41 83 7d 24 02 0f 85 19 01 00 00 4d
[571554.165427] RIP  [<ffffffffc069ce13>] __lpfc_sli_release_iocbq_s4+0x63/0x260 [lpfc]
[571554.173936]  RSP <ffff9fe7e6fe7b20>
[571554.178233] CR2: 0000000000000090

Environment

  • Red Hat Enterprise Linux 7
    • Emulex HBA

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In