Soft lockups on mmfsd process - RIP on __do_page_fault

Solution In Progress - Updated -

Issue

May  5 22:28:30 server-xyz1 kernel: BUG: soft lockup - CPU#2 stuck for 22s! [mmfsd:27251]
May  5 22:28:30 server-xyz1 kernel: Modules linked in: mmfs26(OE) mmfslinux(OE) tracedev(OE) nls_utf8 isofs loop ip6t_rpfilter ip6t_REJECT ipt_REJECT xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle xprtrdma iptable_security sunrpc iptable_raw ib_isert iptable_filter iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm intel_powerclamp coretemp intel_rapl kvm_intel kvm crc32_pclmul ghash_clmulni_intel ipmi_devintf aesni_intel ipmi_ssif iTCO_wdt ipmi_si
May  5 22:28:30 server-xyz1 kernel: iTCO_vendor_support lrw gf128mul ipmi_msghandler glue_helper ablk_helper cryptd sg lpc_ich mfd_core dcdbas sb_edac pcspkr mxm_wmi edac_core mei_me mei wmi shpchp acpi_power_meter tpm_crb ip_tables xfs libcrc32c sr_mod cdrom mlx4_ib mlx4_en ib_sa vxlan ib_mad ip6_udp_tunnel udp_tunnel ib_core ib_addr sd_mod crc_t10dif crct10dif_generic mgag200 mlx4_core crct10dif_pclmul syscopyarea crct10dif_common sysfillrect crc32c_intel sysimgblt drm_kms_helper ttm ixgbe igb ahci drm libahci mdio mpt3sas i2c_algo_bit ptp libata i2c_core raid_class pps_core megaraid_sas scsi_transport_sas dca dm_mirror dm_region_hash dm_log dm_mod
May  5 22:28:30 server-xyz1 kernel: CPU: 2 PID: 27251 Comm: mmfsd Tainted: G           OE  ------------   3.10.0-327.el7.x86_64 #1
May  5 22:28:30 server-xyz1 kernel: Hardware name: Dell Inc. PowerEdge R730/072T6D, BIOS 2.0.2 03/15/2016
May  5 22:28:30 server-xyz1 kernel: task: ffff882024355c00 ti: ffff881f9f028000 task.ti: ffff881f9f028000
May  5 22:28:30 server-xyz1 kernel: RIP: 0010:[<ffffffff81640f4a>]  [<ffffffff81640f4a>] __do_page_fault+0x27a/0x420
May  5 22:28:30 server-xyz1 kernel: RSP: 0018:ffff881f9f02bc18  EFLAGS: 00000202
May  5 22:28:30 server-xyz1 kernel: RAX: 0000000000000000 RBX: ffff881f9f02bc98 RCX: ffffffff8163d16c
May  5 22:28:30 server-xyz1 kernel: RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff881f9f02bc98
May  5 22:28:30 server-xyz1 kernel: RBP: ffff881f9f02bc60 R08: 0000000000000282 R09: 0000000000000001
May  5 22:28:30 server-xyz1 kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 00000000332e4153
May  5 22:28:30 server-xyz1 kernel: R13: ffff881f9f02bc98 R14: 0003000100000003 R15: ffffffff8162e07b
May  5 22:28:30 server-xyz1 kernel: FS:  00007fc180ee3700(0000) GS:ffff88103f220000(0000) knlGS:0000000000000000
May  5 22:28:30 server-xyz1 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May  5 22:28:30 server-xyz1 kernel: CR2: 0000000001802580 CR3: 0000001fe4966000 CR4: 00000000003407e0
May  5 22:28:30 server-xyz1 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
May  5 22:28:30 server-xyz1 kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
May  5 22:28:30 server-xyz1 kernel: Stack:
May  5 22:28:30 server-xyz1 kernel: 0000000000000000 ffff882024355c00 0000000000000001 ffff8820243151b8
May  5 22:28:30 server-xyz1 kernel: ffff881f9f02bc98 0000000000000000 0000000000000c60 000000000000002c
May  5 22:28:30 server-xyz1 kernel: ffffffffa0ac1600 ffff881f9f02bc88 ffffffff81641113 0000000000000010
May  5 22:28:30 server-xyz1 kernel: Call Trace:
May  5 22:28:30 server-xyz1 kernel: [<ffffffff81641113>] do_page_fault+0x23/0x80
May  5 22:28:30 server-xyz1 kernel: [<ffffffff8163d408>] page_fault+0x28/0x30
May  5 22:28:30 server-xyz1 kernel: [<ffffffffa09cd682>] ? _ZN14BaseMutexClass15internalAcquireEm+0x82/0x380 [mmfs26]
May  5 22:28:30 server-xyz1 kernel: [<ffffffffa09cd670>] ? _ZN14BaseMutexClass15internalAcquireEm+0x70/0x380 [mmfs26]
May  5 22:28:30 server-xyz1 kernel: [<ffffffffa09cdb57>] ? kxAcquireMutex+0xa7/0xe0 [mmfs26]
May  5 22:28:30 server-xyz1 kernel: [<ffffffffa08b3a0e>] ? cxiCopyIn+0x6e/0x80 [mmfslinux]
May  5 22:28:30 server-xyz1 kernel: [<ffffffffa0a1e646>] ? _Z8ss_ioctljm+0x1506/0x16c0 [mmfs26]
May  5 22:28:30 server-xyz1 kernel: [<ffffffff811efd42>] ? user_path_at_empty+0x72/0xc0
May  5 22:28:30 server-xyz1 kernel: [<ffffffffa08c05e1>] ? ss_fs_unlocked_ioctl+0x171/0x420 [mmfslinux]
May  5 22:28:30 server-xyz1 kernel: [<ffffffff811f1ef5>] ? do_vfs_ioctl+0x2e5/0x4c0
May  5 22:28:30 server-xyz1 kernel: [<ffffffff8128bc6e>] ? file_has_perm+0xae/0xc0
May  5 22:28:30 server-xyz1 kernel: [<ffffffff8112483d>] ? call_rcu_sched+0x1d/0x20
May  5 22:28:30 server-xyz1 kernel: [<ffffffff811f2171>] ? SyS_ioctl+0xa1/0xc0
May  5 22:28:30 server-xyz1 kernel: [<ffffffff81645909>] ? system_call_fastpath+0x16/0x1b
May  5 22:28:30 server-xyz1 kernel: Code: 75 a8 41 f6 86 88 00 00 00 03 0f 85 08 fe ff ff 41 f6 86 91 00 00 00 02 41 bf 28 00 00 00 0f 84 05 fe ff ff fb 66 0f 1f 44 00 00 <e9> f9 fd ff ff 90 48 89 d9 4c 89 f2 be 01 00 00 00 bf 05 00 00 

Environment

  • Red Hat Enterprise Linux 7.1
  • IBMs General Parallel File System (mmfsd, mmfs26, mmfslinux)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.