IBM gpfs (mmfs) module was unstable on RHEL-6 why ?.
Issue
- Server did not respond to any network traffic. Remote login via ssh was not possible with below traces.
- There are messages on unstable mmfs module in the logs, such as:
Nov 23 05:53:10 kernel: INFO: task mmfsd:24621 blocked for more than 120 seconds.
Nov 23 05:53:10 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 23 05:53:10 kernel: mmfsd D 0000000000000002 0 24621 2489 0x00000080
Nov 23 05:53:10 kernel: ffff8805a4a3dbd8 0000000000000086 ffff88036e83ae40 ffffc9001604a608
Nov 23 05:53:10 kernel: ffff8805a4a3dc48 ffffffffa040c486 0000000000015f80 ffff88036ea06dc0
Nov 23 05:53:10 kernel: ffff880021dae6b8 ffff8805a4a3dfd8 000000000000f598 ffff880021dae6b8
Nov 23 05:53:10 kernel: Call Trace:
Nov 23 05:53:10 kernel: [<ffffffffa040c486>] ? cxiStartIO+0x496/0x580 [mmfslinux]
Nov 23 05:53:10 kernel: [<ffffffff8108e3ee>] ? prepare_to_wait+0x4e/0x80
Nov 23 05:53:10 kernel: [<ffffffffa0408ce4>] cxiWaitIO+0x134/0x1b0 [mmfslinux]
Nov 23 05:53:10 kernel: [<ffffffff8108e100>] ? autoremove_wake_function+0x0/0x40
Nov 23 05:53:10 kernel: [<ffffffffa04821df>] _ZN9DiskSched7localIOEPP15MBDoDiskIOParmsiiP15KernelOperation+0x20f/0x5a0 [mmfs26]
Nov 23 05:53:10 kernel: [<ffffffffa0482690>] ? _Z22LinuxIODoneIntCallbackPvj+0x0/0xd0 [mmfs26]
Nov 23 05:53:10 kernel: [<ffffffff81093e1f>] ? up+0x2f/0x50
Nov 23 05:53:10 kernel: [<ffffffffa0482604>] ? kxLocalIO+0x94/0x120 [mmfs26]
Nov 23 05:53:10 kernel: [<ffffffffa0411653>] ? cxiCopyIn+0x83/0xa0 [mmfslinux]
Nov 23 05:53:10 kernel: [<ffffffffa0538bdb>] ? _Z8ss_ioctljm+0x111b/0x1430 [mmfs26]
Nov 23 05:53:10 kernel: [<ffffffff8140d780>] ? sys_sendmsg+0x390/0x3a0
Nov 23 05:53:10 kernel: [<ffffffffa041df95>] ? ss_fs_unlocked_ioctl+0x75/0x380 [mmfslinux]
Nov 23 05:53:10 kernel: [<ffffffff81136088>] ? zap_page_range+0xd8/0xf0
Nov 23 05:53:10 kernel: [<ffffffff8100be6e>] ? reschedule_interrupt+0xe/0x20
Nov 23 05:53:10 kernel: [<ffffffff81184d92>] ? vfs_ioctl+0x22/0xa0
Nov 23 05:53:10 kernel: [<ffffffff81184f34>] ? do_vfs_ioctl+0x84/0x580
Nov 23 05:53:10 kernel: [<ffffffff811854b1>] ? sys_ioctl+0x81/0xa0
Nov 23 05:53:10 kernel: [<ffffffff8100b172>] ? system_call_fastpath+0x16/0x1b
Environment
- RHEL-6 with IBM gpfs installed, mmfs module present in lsmod.
- Must be running on a NUMA system (eg. AMD Opteron, newer Intel Xeon).
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.