System crashes at 'cxiPanic+0x5c/0x70'

Issue

• System crashes at cxiPanic+0x5c/0x70

[1830585.551563] perf: interrupt took too long (7804 > 7768), lowering kernel.perf_event_max_sample_rate to 25000
[3171422.206753] GPFS logAssertFailed: TokenPermits(get_token_mode(), lm_want) file /project/spreltac504/build/rtac504s002a/src/avs/fs/mmfs/ts/kernext/ShHashV.C line 1367
[3171424.818593] <5>kp 24881: cxiPanic: ShHashV.C:1367:0:0:FFFFFFFFC0BCD200:LkObj::uXferMmapStartL: eff_token_mode ro and lm_want rf:TokenPermits(get_token_mode(), lm_want)
[3171424.818627] ------------[ cut here ]------------
[3171424.818644] kernel BUG at /usr/lpp/mmfs/src/gpl-linux/cxiSystem.c:2724!
[3171424.818659] invalid opcode: 0000 [#1] SMP
[3171424.818670] Modules linked in: tcp_lp fuse joydev mmfs26(OE) mmfslinux(OE) tracedev(OE) rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt bonding ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx5_ib dm_mirror ib_core dm_region_hash dm_log dm_mod vfat fat skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr cdc_eem usbnet mii ses enclosure hpwdt mei_me hpilo mei sg shpchp lpc_ich ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter nfsd binfmt_misc auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic sr_mod cdrom mgag200 i2c_algo_bit
[3171424.818869]  drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul mlx5_core crct10dif_common crc32c_intel drm ahci smartpqi libahci tg3 scsi_transport_sas libata devlink ptp i2c_core pps_core uas usb_storage
[3171424.818924] CPU: 0 PID: 24881 Comm: fluent_mpi.20.1 Tainted: G           OE  ------------   3.10.0-693.el7.x86_64 #1
[3171424.818946] Hardware name: HPE ProLiant DL360 Gen10/ProLiant DL360 Gen10, BIOS U32 03/09/2020
[3171424.818963] task: ffff8823f2260000 ti: ffff882304b2c000 task.ti: ffff882304b2c000
[3171424.818978] RIP: 0010:[<ffffffffc08be0dc>]  [<ffffffffc08be0dc>] cxiPanic+0x5c/0x70 [mmfslinux]
[3171424.819005] RSP: 0018:ffff882304b2f4a8  EFLAGS: 00010246
[3171424.819017] RAX: 000000000000009b RBX: ffffffffc0c00420 RCX: 0000000000000000
[3171424.819031] RDX: 0000000000000000 RSI: ffff8813efc0f8b8 RDI: ffff8813efc0f8b8
[3171424.819046] RBP: ffff882304b2f4b8 R08: 000000000000000a R09: 0000000000000000
[3171424.819060] R10: 0000000000000817 R11: ffff882304b2f1a6 R12: 0000000000000000
[3171424.819074] R13: ffffffffc0b93cd8 R14: 0000000000000000 R15: 0000000000000000
[3171424.819088] FS:  00007f103c227740(0000) GS:ffff8813efc00000(0000) knlGS:0000000000000000
[3171424.819104] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[3171424.819116] CR2: 00007f513bf80710 CR3: 00000013c3cd3000 CR4: 00000000003407f0
[3171424.819130] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[3171424.819144] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[3171424.819158] Stack:
[3171424.819164]  ffffffffc0c00c40 0000000000000557 ffff882304b2f518 ffffffffc0a1fb29
[3171424.819182]  ffffffffc0bcd200 ffff882304b2f508 ffffffffc08c4da7 ffffffffc0c00c40
[3171424.819198]  0000000200000000 ffffffffc0b8f1cb ffffc90039c20fd8 ffffc90039c20fd8
[3171424.819216] Call Trace:
[3171424.819265]  [<ffffffffc0a1fb29>] logAssertFailed+0x359/0x390 [mmfs26]
[3171424.819289]  [<ffffffffc08c4da7>] ? cxiBlockingMutexRelease+0x87/0xf0 [mmfslinux]
[3171424.819329]  [<ffffffffc0a134d8>] _ZN5LkObj16uXferMmapStartLkEP8CacheObjNS_12LockModeEnumE+0x158/0x200 [mmfs26]
[3171424.819371]  [<ffffffffc0a156ae>] _ZN5LkObj10lock_vfs_mEP8CacheObjNS_12LockModeEnumEiPj+0x33e/0x7c0 [mmfs26]
[3171424.819421]  [<ffffffffc0ac4dfd>] ? _ZN19NotGlobalMutexClass12kAcquireSlowEv+0x17d/0x270 [mmfs26]
[3171424.819446]  [<ffffffff810cdc7c>] ? dequeue_entity+0x11c/0x5d0
[3171424.819479]  [<ffffffffc0a03b21>] _Z13lockFileHeldMP15KernelOperationP8OpenFile7FileUIDN5LkObj12LockModeEnumEiPj+0xc1/0x520 [mmfs26]
[3171424.819525]  [<ffffffffc0a043da>] _Z9lockFile2P15KernelOperation17OperationLockMode7FileUIDN5LkObj12LockModeEnumES4_PP8OpenFileP10WhatLockediPj+0x2ca/0x570 [mmfs26]
[3171424.819583]  [<ffffffffc0a912c9>] _Z13gpfsQueueBufsP9_cxibuf_t+0x919/0x1440 [mmfs26]
[3171424.819608]  [<ffffffffc08c9ee9>] gpfs_filemap_nopage_internal+0x279/0x4f0 [mmfslinux]
[3171424.819634]  [<ffffffffc08ca176>] gpfs_filemap_fault+0x16/0x30 [mmfslinux]
[3171424.819654]  [<ffffffff811ad162>] __do_fault+0x52/0xe0
[3171424.820413]  [<ffffffff811adbbb>] do_cow_fault+0xab/0x290
[3171424.821002]  [<ffffffff811b1b3d>] handle_mm_fault+0x2bd/0xfa0
[3171424.821583]  [<ffffffff811b5258>] ? __vma_link_file+0x48/0x70
[3171424.822157]  [<ffffffff811b5e0d>] ? vma_link+0x7d/0xc0
[3171424.822727]  [<ffffffff811b704f>] ? vma_set_page_prot+0x3f/0x60
[3171424.823289]  [<ffffffff816affb4>] __do_page_fault+0x154/0x450
[3171424.823836]  [<ffffffff816b02e5>] do_page_fault+0x35/0x90
[3171424.824369]  [<ffffffff816ac508>] page_fault+0x28/0x30
[3171424.824889]  [<ffffffff81331c35>] ? __clear_user+0x25/0x50
[3171424.825395]  [<ffffffff81331c90>] clear_user+0x30/0x40
[3171424.825886]  [<ffffffff8125dc09>] padzero+0x29/0x40
[3171424.826364]  [<ffffffff8125fd1e>] load_elf_binary+0x8fe/0xe00
[3171424.826831]  [<ffffffff812d4079>] ? ima_bprm_check+0x49/0x50 
[3171424.827284]  [<ffffffff8125f420>] ? load_elf_library+0x220/0x220
[3171424.827729]  [<ffffffff81207ddd>] search_binary_handler+0xed/0x300
[3171424.828164]  [<ffffffff81209306>] do_execve_common.isra.25+0x5b6/0x6c0
[3171424.828595]  [<ffffffff812096a9>] SyS_execve+0x29/0x30
[3171424.829017]  [<ffffffff816b5569>] stub_execve+0x69/0xa0
[3171424.829431] Code: c0 83 3d 07 5f f4 ff 03 7e 1c 80 3d 69 c5 09 00 00 78 13 48 89 d9 31 d2 31 f6 bf 23 41 70 30 31 c0 e8 09 14 f4 ff 45 85 e4 75 02 <0f> 0b 48 89 de 48 c7 c7 38 77 8f c0 31 c0 e8 7d fa dd c0 90 0f
[3171424.830335] RIP  [<ffffffffc08be0dc>] cxiPanic+0x5c/0x70 [mmfslinux]
[3171424.830777]  RSP <ffff882304b2f4a8>