OpenStack compute node/physical server reboots when SAN switch is rebooted OR fluctuations are encountered in connectivity to the storage.
Issue
- OpenStack compute node/physical server reboots/crashes when SAN switch is rebooted OR fluctuations are encountered in connectivity to the storage.
Vmcore logs :
[571553.711255] device-mapper: multipath: Failing path 128:176.
[571553.711265] device-mapper: multipath: Failing path 70:160.
[571553.711274] device-mapper: multipath: Failing path 71:64.
[571553.711283] device-mapper: multipath: Failing path 71:32.
[571553.711292] device-mapper: multipath: Failing path 71:208.
[571553.711300] device-mapper: multipath: Failing path 71:240.
[571553.711309] device-mapper: multipath: Failing path 128:144.
[571553.711317] device-mapper: multipath: Failing path 128:112.
[571553.711331] device-mapper: multipath: Failing path 128:96.
[571553.711342] device-mapper: multipath: Failing path 128:0.
[571553.719778] sd 15:0:9:0: alua: port group 01 state A preferred supports tolusnA
[571553.719939] sd 15:0:9:0: alua: port group 01 state A preferred supports tolusnA
[571553.721913] BUG: unable to handle kernel NULL pointer dereference at 0000000000000090
[571553.730230] IP: [<ffffffffc069ce13>] __lpfc_sli_release_iocbq_s4+0x63/0x260 [lpfc]
[571553.738094] PGD 800000af8eb13067 PUD bdd0685067 PMD 0
[571553.743514] Oops: 0000 [#1] SMP
...
[571553.893851] wmi drm_panel_orientation_quirks dm_multipath nfit libnvdimm sunrpc dm_mirror dm_region_hash dm_log dm_mod
[571553.904439] CPU: 15 PID: 2674 Comm: lpfc_worker_2 Kdump: loaded Tainted: G OE ------------ T 3.10.0-1062.12.1.el7.x86_64 #1
[571553.917568] Hardware name: Dell Inc. PowerEdge R740/0JMK61, BIOS 2.6.4 04/09/2020
[571553.925642] task: ffff9fe7f067b150 ti: ffff9fe7e6fe4000 task.ti: ffff9fe7e6fe4000
[571553.933732] RIP: 0010:[<ffffffffc069ce13>] [<ffffffffc069ce13>] __lpfc_sli_release_iocbq_s4+0x63/0x260 [lpfc]
[571553.944407] RSP: 0018:ffff9fe7e6fe7b20 EFLAGS: 00010046
[571553.950373] RAX: 0000000000100000 RBX: ffffa047eab88e00 RCX: 000000010040002e
[571553.958166] RDX: 0000000000000000 RSI: ffffa047eab88e00 RDI: ffffa047f4c1c000
[571553.965958] RBP: ffff9fe7e6fe7b50 R08: ffffa0478ffc7c40 R09: 000000010040002e
[571553.974153] R10: 000000008ffc7f01 R11: ffffa0478ffc7c40 R12: ffffa047f4c1c000
[571553.981969] R13: ffff9fe7f8874060 R14: ffffa047eab88e00 R15: ffffa047f4c1c000
[571553.989788] FS: 0000000000000000(0000) GS:ffffa0487d1c0000(0000) knlGS:0000000000000000
[571553.999033] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[571554.005500] CR2: 0000000000000090 CR3: 0000009a9fab4000 CR4: 00000000007627e0
[571554.013454] PKRU: 00000000
[571554.016910] Call Trace:
[571554.020139] [<ffffffffc069f407>] lpfc_sli_release_iocbq+0x37/0x60 [lpfc]
[571554.027696] [<ffffffffc06bd83e>] lpfc_els_free_iocb+0x14e/0x1d0 [lpfc]
[571554.035084] [<ffffffffc06c1b03>] lpfc_cmpl_els_prli+0xe3/0x210 [lpfc]
[571554.042392] [<ffffffffc06a60bd>] lpfc_sli_sp_handle_rspiocb+0x3fd/0x780 [lpfc]
[571554.050491] [<ffffffffc06cea36>] ? lpfc_mbx_cmpl_reg_login+0xe6/0x160 [lpfc]
[571554.058423] [<ffffffff9c2af1f5>] ? mod_timer+0x1b5/0x230
[571554.064628] [<ffffffffc06b0172>] lpfc_sli_handle_slow_ring_event_s4+0x192/0x260 [lpfc]
[571554.073466] [<ffffffffc06a03e2>] lpfc_sli_handle_slow_ring_event+0x12/0x20 [lpfc]
[571554.081862] [<ffffffffc06d3afc>] lpfc_work_done+0x94c/0x14a0 [lpfc]
[571554.089046] [<ffffffff9c9805c2>] ? __schedule+0x402/0x840
[571554.095378] [<ffffffffc06d46c0>] lpfc_do_work+0x70/0x1e0 [lpfc]
[571554.102236] [<ffffffff9c2c72e0>] ? wake_up_atomic_t+0x30/0x30
[571554.108927] [<ffffffffc06d4650>] ? lpfc_work_done+0x14a0/0x14a0 [lpfc]
[571554.116398] [<ffffffff9c2c61f1>] kthread+0xd1/0xe0
[571554.122246] [<ffffffff9c2c6120>] ? insert_kthread_work+0x40/0x40
[571554.129181] [<ffffffff9c98dd1d>] ret_from_fork_nospec_begin+0x7/0x21
[571554.136453] [<ffffffff9c2c6120>] ? insert_kthread_work+0x40/0x40
[571554.143383] Code: 28 48 c7 00 00 00 00 00 4d 85 ed 0f 84 a6 00 00 00 8b 86 74 01 00 00 a9 00 00 80 00 0f 85 76 01 00 00 48 8b 97 98 02 00 00 a8 40 <4c> 8b b2 90 00 00 00 74 0b 41 83 7d 24 02 0f 85 19 01 00 00 4d
[571554.165427] RIP [<ffffffffc069ce13>] __lpfc_sli_release_iocbq_s4+0x63/0x260 [lpfc]
[571554.173936] RSP <ffff9fe7e6fe7b20>
[571554.178233] CR2: 0000000000000090
Environment
- Red Hat Enterprise Linux 7
- Emulex HBA
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.