IBM gpfs (mmfs) module was unstable on RHEL-6 why ?.

Solution Verified - Updated -

Issue

  • Server did not respond to any network traffic. Remote login via ssh was not possible with below traces.
  • There are messages on unstable mmfs module in the logs, such as:
Nov 23 05:53:10  kernel: INFO: task mmfsd:24621 blocked for more than 120 seconds.
Nov 23 05:53:10 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 23 05:53:10  kernel: mmfsd         D 0000000000000002     0 24621   2489 0x00000080
Nov 23 05:53:10  kernel: ffff8805a4a3dbd8 0000000000000086 ffff88036e83ae40 ffffc9001604a608
Nov 23 05:53:10 kernel: ffff8805a4a3dc48 ffffffffa040c486 0000000000015f80 ffff88036ea06dc0
Nov 23 05:53:10  kernel: ffff880021dae6b8 ffff8805a4a3dfd8 000000000000f598 ffff880021dae6b8
Nov 23 05:53:10 kernel: Call Trace:
Nov 23 05:53:10  kernel: [<ffffffffa040c486>] ? cxiStartIO+0x496/0x580 [mmfslinux]
Nov 23 05:53:10  kernel: [<ffffffff8108e3ee>] ? prepare_to_wait+0x4e/0x80
Nov 23 05:53:10  kernel: [<ffffffffa0408ce4>] cxiWaitIO+0x134/0x1b0 [mmfslinux]
Nov 23 05:53:10  kernel: [<ffffffff8108e100>] ? autoremove_wake_function+0x0/0x40
Nov 23 05:53:10  kernel: [<ffffffffa04821df>] _ZN9DiskSched7localIOEPP15MBDoDiskIOParmsiiP15KernelOperation+0x20f/0x5a0 [mmfs26]
Nov 23 05:53:10  kernel: [<ffffffffa0482690>] ? _Z22LinuxIODoneIntCallbackPvj+0x0/0xd0 [mmfs26]
Nov 23 05:53:10  kernel: [<ffffffff81093e1f>] ? up+0x2f/0x50
Nov 23 05:53:10  kernel: [<ffffffffa0482604>] ? kxLocalIO+0x94/0x120 [mmfs26]
Nov 23 05:53:10  kernel: [<ffffffffa0411653>] ? cxiCopyIn+0x83/0xa0 [mmfslinux]
Nov 23 05:53:10  kernel: [<ffffffffa0538bdb>] ? _Z8ss_ioctljm+0x111b/0x1430 [mmfs26]
Nov 23 05:53:10  kernel: [<ffffffff8140d780>] ? sys_sendmsg+0x390/0x3a0
Nov 23 05:53:10  kernel: [<ffffffffa041df95>] ? ss_fs_unlocked_ioctl+0x75/0x380 [mmfslinux]
Nov 23 05:53:10 kernel: [<ffffffff81136088>] ? zap_page_range+0xd8/0xf0
Nov 23 05:53:10  kernel: [<ffffffff8100be6e>] ? reschedule_interrupt+0xe/0x20
Nov 23 05:53:10  kernel: [<ffffffff81184d92>] ? vfs_ioctl+0x22/0xa0
Nov 23 05:53:10  kernel: [<ffffffff81184f34>] ? do_vfs_ioctl+0x84/0x580
Nov 23 05:53:10  kernel: [<ffffffff811854b1>] ? sys_ioctl+0x81/0xa0
Nov 23 05:53:10  kernel: [<ffffffff8100b172>] ? system_call_fastpath+0x16/0x1b

Environment

  • RHEL-6 with IBM gpfs installed, mmfs module present in lsmod.
  • Must be running on a NUMA system (eg. AMD Opteron, newer Intel Xeon).

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content