nfs_idmap_request_key/call_usermodehelper_exec gets hung while trying to call userland helper

Solution Verified - Updated -

Issue

NFS calls gets hung while trying to decode server reply containing a key for ownership/group ownership attributes of file/s :

 #0 [ffff88200db07038] schedule at ffffffff815489d0
 #1 [ffff88200db07110] schedule_timeout at ffffffff81549945
 #2 [ffff88200db071c0] wait_for_common at ffffffff815495a3
 #3 [ffff88200db07250] wait_for_completion at ffffffff815496dd
 #4 [ffff88200db07260] call_usermodehelper_exec at ffffffff8109e9dc
 #5 [ffff88200db072b0] call_sbin_request_key at ffffffff812384f9
 #6 [ffff88200db07420] request_key_and_link at ffffffff81238024
 #7 [ffff88200db074c0] request_key at ffffffff812381f0
 #8 [ffff88200db07510] nfs_idmap_request_key at ffffffffa0469fe5 [nfs]
 #9 [ffff88200db07570] nfs_idmap_lookup_id at ffffffffa046a624 [nfs]
#10 [ffff88200db075c0] nfs_map_name_to_uid at ffffffffa046aa65 [nfs]
#11 [ffff88200db075f0] decode_getfattr_attrs at ffffffffa0464a07 [nfs]
#12 [ffff88200db07690] decode_getfattr_generic.clone.0 at ffffffffa0464acb [nfs]
#13 [ffff88200db076f0] decode_getfattr at ffffffffa0464c03 [nfs]
#14 [ffff88200db07700] nfs4_xdr_dec_access at ffffffffa04652b0 [nfs]
#15 [ffff88200db077a0] rpcauth_unwrap_resp at ffffffffa0398304 [sunrpc]
#16 [ffff88200db077f0] call_decode at ffffffffa038c9a9 [sunrpc]
#17 [ffff88200db07850] __rpc_execute at ffffffffa0396967 [sunrpc]
#18 [ffff88200db078c0] rpc_execute at ffffffffa0396ca1 [sunrpc]
#19 [ffff88200db078f0] rpc_run_task at ffffffffa038d455 [sunrpc]
#20 [ffff88200db07910] rpc_call_sync at ffffffffa038d572 [sunrpc]
#21 [ffff88200db07970] _nfs4_call_sync at ffffffffa0458d8e [nfs]
#22 [ffff88200db07980] _nfs4_proc_access at ffffffffa0457b2c [nfs]
#23 [ffff88200db07a60] nfs4_proc_access at ffffffffa0457bfb [nfs]
#24 [ffff88200db07ab0] nfs_do_access at ffffffffa04347bc [nfs]
#25 [ffff88200db07b40] nfs_permission at ffffffffa0434908 [nfs]
#26 [ffff88200db07b70] __link_path_walk at ffffffff811a9f93
#27 [ffff88200db07c50] __link_path_walk at ffffffff811aa3b3
#28 [ffff88200db07d30] path_walk at ffffffff811ab1fa
#29 [ffff88200db07d70] filename_lookup at ffffffff811ab40b
#30 [ffff88200db07db0] user_path_at at ffffffff811ac537
#31 [ffff88200db07e80] vfs_fstatat at ffffffff8119f8e0
#32 [ffff88200db07ee0] vfs_stat at ffffffff8119fa5b
#33 [ffff88200db07ef0] sys_newstat at ffffffff8119fa84
#34 [ffff88200db07f80] system_call_fastpath at ffffffff8100b0d2

Task hung messages will also appear :

INFO: task sapstartsrv:3869 blocked for more than 120 seconds.
  • another logs:
secfs2: Could not resolve user_id 7253963 to user_name
SEOS Syscall Monitor - ACTIVATED
INFO: task os_agent:9680 blocked for more than 120 seconds.
      Tainted: P           -- ------------    2.6.32-754.11.1.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
os_agent      D 0000000000000002     0  9680      1 0x00000000
 ffff880434bb73c8 0000000000000082 ffff880434bb7318 0000000039890040
 ffff880028318c00 ffff880028318c00 ffff880028318c00 0000000000000000
 0000000000000002 0000000000000002 ffff880434b025f8 ffff880434bb7fd8
Call Trace:
 [<ffffffff81559dd5>] schedule_timeout+0x215/0x2f0
 [<ffffffff810aaec6>] ? autoremove_wake_function+0x16/0x40
 [<ffffffff8106251c>] ? __wake_up_common+0x5c/0x90
 [<ffffffff81559a33>] wait_for_common+0x123/0x180
 [<ffffffff81070870>] ? default_wake_function+0x0/0x20
 [<ffffffff810a49c1>] ? __queue_work+0x41/0x50
 [<ffffffff81559b6d>] wait_for_completion+0x1d/0x20
 [<ffffffff810a2e7c>] call_usermodehelper_exec+0x10c/0x130
 [<ffffffff81241199>] call_sbin_request_key+0x259/0x310
 [<ffffffff8155a36e>] ? mutex_lock+0x1e/0x50
 [<ffffffff8123b2c3>] ? key_instantiate_and_link+0xa3/0xb0
 [<ffffffffa0610030>] ? pnfs_choose_layoutget_stateid+0x70/0x130 [nfs]
 [<ffffffff81240cc7>] request_key_and_link+0x347/0x400
 [<ffffffff81240e90>] request_key+0x50/0xa0
 [<ffffffffa060b4c5>] nfs_idmap_request_key+0xc5/0x180 [nfs]
 [<ffffffffa060bb04>] nfs_idmap_lookup_id+0x34/0x80 [nfs]
 [<ffffffffa060bea5>] nfs_map_group_to_gid+0x75/0xa0 [nfs]
 [<ffffffffa0605c19>] decode_getfattr_attrs+0xd69/0x1040 [nfs]
 [<ffffffffa0605f7b>] decode_getfattr_generic.clone.0+0x8b/0xf0 [nfs]
 [<ffffffffa06060b3>] decode_getfattr+0x13/0x20 [nfs]
 [<ffffffffa060669f>] nfs4_xdr_dec_getattr+0x8f/0xa0 [nfs]
 [<ffffffffa0606610>] ? nfs4_xdr_dec_getattr+0x0/0xa0 [nfs]
 [<ffffffffa05395e6>] rpcauth_unwrap_resp+0x86/0xc0 [sunrpc]
 [<ffffffffa0606610>] ? nfs4_xdr_dec_getattr+0x0/0xa0 [nfs]
 [<ffffffffa052d9f9>] call_decode+0x1b9/0x840 [sunrpc]
 [<ffffffff810aaf30>] ? wake_bit_function+0x0/0x50
 [<ffffffffa052d840>] ? call_decode+0x0/0x840 [sunrpc]
 [<ffffffffa0537ba9>] __rpc_execute+0x79/0x380 [sunrpc]
 [<ffffffff810aad37>] ? bit_waitqueue+0x17/0xd0
 [<ffffffffa0537f11>] rpc_execute+0x61/0xa0 [sunrpc]
 [<ffffffffa052e4b5>] rpc_run_task+0x75/0x90 [sunrpc]
 [<ffffffffa052e5d2>] rpc_call_sync+0x42/0x70 [sunrpc]
 [<ffffffffa05fa126>] _nfs4_call_sync+0x36/0x40 [nfs]
 [<ffffffffa05f29f2>] _nfs4_proc_getattr+0xb2/0xc0 [nfs]
 [<ffffffffa05f5916>] nfs4_proc_getattr+0x56/0x80 [nfs]
 [<ffffffffa05db3a9>] __nfs_revalidate_inode+0xe9/0x220 [nfs]
 [<ffffffffa05dc23e>] nfs_getattr+0xde/0x210 [nfs]
 [<ffffffff811a71c4>] vfs_getattr+0x54/0x90
 [<ffffffff811a7004>] ? cp_new_stat+0xe4/0x100
 [<ffffffff811a7264>] vfs_fstatat+0x64/0xa0
 [<ffffffff811a73cb>] vfs_stat+0x1b/0x20
 [<ffffffff811a73f4>] sys_newstat+0x24/0x50
 [<ffffffffa0350a3d>] ? am_policy_is_active+0x2d/0x40 [secfs2]
 [<ffffffffa031e8f8>] ? __am_prelookup+0x48/0xf0 [secfs2]
 [<ffffffff811a73d0>] ? sys_newstat+0x0/0x50
 [<ffffffffa031fbf3>] m_sys_stat+0x43/0x60 [secfs2]
 [<ffffffffa0669a2e>] my_stat+0x7e/0x100 [seos]
 [<ffffffff81564357>] system_call_fastpath+0x35/0x3a
 [<ffffffff8156427e>] ? system_call_after_swapgs+0xae/0x152

Environment

  • Red Hat Enterprise Linux 7
  • Red Hat Enterprise Linux 6

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In