Soft lockups on mmfsd process - RIP on __do_page_fault
Issue
May 5 22:28:30 server-xyz1 kernel: BUG: soft lockup - CPU#2 stuck for 22s! [mmfsd:27251]
May 5 22:28:30 server-xyz1 kernel: Modules linked in: mmfs26(OE) mmfslinux(OE) tracedev(OE) nls_utf8 isofs loop ip6t_rpfilter ip6t_REJECT ipt_REJECT xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle xprtrdma iptable_security sunrpc iptable_raw ib_isert iptable_filter iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm intel_powerclamp coretemp intel_rapl kvm_intel kvm crc32_pclmul ghash_clmulni_intel ipmi_devintf aesni_intel ipmi_ssif iTCO_wdt ipmi_si
May 5 22:28:30 server-xyz1 kernel: iTCO_vendor_support lrw gf128mul ipmi_msghandler glue_helper ablk_helper cryptd sg lpc_ich mfd_core dcdbas sb_edac pcspkr mxm_wmi edac_core mei_me mei wmi shpchp acpi_power_meter tpm_crb ip_tables xfs libcrc32c sr_mod cdrom mlx4_ib mlx4_en ib_sa vxlan ib_mad ip6_udp_tunnel udp_tunnel ib_core ib_addr sd_mod crc_t10dif crct10dif_generic mgag200 mlx4_core crct10dif_pclmul syscopyarea crct10dif_common sysfillrect crc32c_intel sysimgblt drm_kms_helper ttm ixgbe igb ahci drm libahci mdio mpt3sas i2c_algo_bit ptp libata i2c_core raid_class pps_core megaraid_sas scsi_transport_sas dca dm_mirror dm_region_hash dm_log dm_mod
May 5 22:28:30 server-xyz1 kernel: CPU: 2 PID: 27251 Comm: mmfsd Tainted: G OE ------------ 3.10.0-327.el7.x86_64 #1
May 5 22:28:30 server-xyz1 kernel: Hardware name: Dell Inc. PowerEdge R730/072T6D, BIOS 2.0.2 03/15/2016
May 5 22:28:30 server-xyz1 kernel: task: ffff882024355c00 ti: ffff881f9f028000 task.ti: ffff881f9f028000
May 5 22:28:30 server-xyz1 kernel: RIP: 0010:[<ffffffff81640f4a>] [<ffffffff81640f4a>] __do_page_fault+0x27a/0x420
May 5 22:28:30 server-xyz1 kernel: RSP: 0018:ffff881f9f02bc18 EFLAGS: 00000202
May 5 22:28:30 server-xyz1 kernel: RAX: 0000000000000000 RBX: ffff881f9f02bc98 RCX: ffffffff8163d16c
May 5 22:28:30 server-xyz1 kernel: RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff881f9f02bc98
May 5 22:28:30 server-xyz1 kernel: RBP: ffff881f9f02bc60 R08: 0000000000000282 R09: 0000000000000001
May 5 22:28:30 server-xyz1 kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 00000000332e4153
May 5 22:28:30 server-xyz1 kernel: R13: ffff881f9f02bc98 R14: 0003000100000003 R15: ffffffff8162e07b
May 5 22:28:30 server-xyz1 kernel: FS: 00007fc180ee3700(0000) GS:ffff88103f220000(0000) knlGS:0000000000000000
May 5 22:28:30 server-xyz1 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 5 22:28:30 server-xyz1 kernel: CR2: 0000000001802580 CR3: 0000001fe4966000 CR4: 00000000003407e0
May 5 22:28:30 server-xyz1 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
May 5 22:28:30 server-xyz1 kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
May 5 22:28:30 server-xyz1 kernel: Stack:
May 5 22:28:30 server-xyz1 kernel: 0000000000000000 ffff882024355c00 0000000000000001 ffff8820243151b8
May 5 22:28:30 server-xyz1 kernel: ffff881f9f02bc98 0000000000000000 0000000000000c60 000000000000002c
May 5 22:28:30 server-xyz1 kernel: ffffffffa0ac1600 ffff881f9f02bc88 ffffffff81641113 0000000000000010
May 5 22:28:30 server-xyz1 kernel: Call Trace:
May 5 22:28:30 server-xyz1 kernel: [<ffffffff81641113>] do_page_fault+0x23/0x80
May 5 22:28:30 server-xyz1 kernel: [<ffffffff8163d408>] page_fault+0x28/0x30
May 5 22:28:30 server-xyz1 kernel: [<ffffffffa09cd682>] ? _ZN14BaseMutexClass15internalAcquireEm+0x82/0x380 [mmfs26]
May 5 22:28:30 server-xyz1 kernel: [<ffffffffa09cd670>] ? _ZN14BaseMutexClass15internalAcquireEm+0x70/0x380 [mmfs26]
May 5 22:28:30 server-xyz1 kernel: [<ffffffffa09cdb57>] ? kxAcquireMutex+0xa7/0xe0 [mmfs26]
May 5 22:28:30 server-xyz1 kernel: [<ffffffffa08b3a0e>] ? cxiCopyIn+0x6e/0x80 [mmfslinux]
May 5 22:28:30 server-xyz1 kernel: [<ffffffffa0a1e646>] ? _Z8ss_ioctljm+0x1506/0x16c0 [mmfs26]
May 5 22:28:30 server-xyz1 kernel: [<ffffffff811efd42>] ? user_path_at_empty+0x72/0xc0
May 5 22:28:30 server-xyz1 kernel: [<ffffffffa08c05e1>] ? ss_fs_unlocked_ioctl+0x171/0x420 [mmfslinux]
May 5 22:28:30 server-xyz1 kernel: [<ffffffff811f1ef5>] ? do_vfs_ioctl+0x2e5/0x4c0
May 5 22:28:30 server-xyz1 kernel: [<ffffffff8128bc6e>] ? file_has_perm+0xae/0xc0
May 5 22:28:30 server-xyz1 kernel: [<ffffffff8112483d>] ? call_rcu_sched+0x1d/0x20
May 5 22:28:30 server-xyz1 kernel: [<ffffffff811f2171>] ? SyS_ioctl+0xa1/0xc0
May 5 22:28:30 server-xyz1 kernel: [<ffffffff81645909>] ? system_call_fastpath+0x16/0x1b
May 5 22:28:30 server-xyz1 kernel: Code: 75 a8 41 f6 86 88 00 00 00 03 0f 85 08 fe ff ff 41 f6 86 91 00 00 00 02 41 bf 28 00 00 00 0f 84 05 fe ff ff fb 66 0f 1f 44 00 00 <e9> f9 fd ff ff 90 48 89 d9 4c 89 f2 be 01 00 00 00 bf 05 00 00
Environment
- Red Hat Enterprise Linux 7.1
- IBMs General Parallel File System (mmfsd, mmfs26, mmfslinux)
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.