nfs_idmap_request_key/call_usermodehelper_exec gets hung while trying to call userland helper

Solution Verified - Updated -

Issue

NFS calls gets hung while trying to decode server reply containing a key for ownership/group ownership attributes of file/s :

 #0 [ffff88200db07038] schedule at ffffffff815489d0
 #1 [ffff88200db07110] schedule_timeout at ffffffff81549945
 #2 [ffff88200db071c0] wait_for_common at ffffffff815495a3
 #3 [ffff88200db07250] wait_for_completion at ffffffff815496dd
 #4 [ffff88200db07260] call_usermodehelper_exec at ffffffff8109e9dc
 #5 [ffff88200db072b0] call_sbin_request_key at ffffffff812384f9
 #6 [ffff88200db07420] request_key_and_link at ffffffff81238024
 #7 [ffff88200db074c0] request_key at ffffffff812381f0
 #8 [ffff88200db07510] nfs_idmap_request_key at ffffffffa0469fe5 [nfs]
 #9 [ffff88200db07570] nfs_idmap_lookup_id at ffffffffa046a624 [nfs]
#10 [ffff88200db075c0] nfs_map_name_to_uid at ffffffffa046aa65 [nfs]
#11 [ffff88200db075f0] decode_getfattr_attrs at ffffffffa0464a07 [nfs]
#12 [ffff88200db07690] decode_getfattr_generic.clone.0 at ffffffffa0464acb [nfs]
#13 [ffff88200db076f0] decode_getfattr at ffffffffa0464c03 [nfs]
#14 [ffff88200db07700] nfs4_xdr_dec_access at ffffffffa04652b0 [nfs]
#15 [ffff88200db077a0] rpcauth_unwrap_resp at ffffffffa0398304 [sunrpc]
#16 [ffff88200db077f0] call_decode at ffffffffa038c9a9 [sunrpc]
#17 [ffff88200db07850] __rpc_execute at ffffffffa0396967 [sunrpc]
#18 [ffff88200db078c0] rpc_execute at ffffffffa0396ca1 [sunrpc]
#19 [ffff88200db078f0] rpc_run_task at ffffffffa038d455 [sunrpc]
#20 [ffff88200db07910] rpc_call_sync at ffffffffa038d572 [sunrpc]
#21 [ffff88200db07970] _nfs4_call_sync at ffffffffa0458d8e [nfs]
#22 [ffff88200db07980] _nfs4_proc_access at ffffffffa0457b2c [nfs]
#23 [ffff88200db07a60] nfs4_proc_access at ffffffffa0457bfb [nfs]
#24 [ffff88200db07ab0] nfs_do_access at ffffffffa04347bc [nfs]
#25 [ffff88200db07b40] nfs_permission at ffffffffa0434908 [nfs]
#26 [ffff88200db07b70] __link_path_walk at ffffffff811a9f93
#27 [ffff88200db07c50] __link_path_walk at ffffffff811aa3b3
#28 [ffff88200db07d30] path_walk at ffffffff811ab1fa
#29 [ffff88200db07d70] filename_lookup at ffffffff811ab40b
#30 [ffff88200db07db0] user_path_at at ffffffff811ac537
#31 [ffff88200db07e80] vfs_fstatat at ffffffff8119f8e0
#32 [ffff88200db07ee0] vfs_stat at ffffffff8119fa5b
#33 [ffff88200db07ef0] sys_newstat at ffffffff8119fa84
#34 [ffff88200db07f80] system_call_fastpath at ffffffff8100b0d2

Task hung messages will also appear :

INFO: task sapstartsrv:3869 blocked for more than 120 seconds.
  • another logs:
secfs2: Could not resolve user_id 7253963 to user_name
SEOS Syscall Monitor - ACTIVATED
INFO: task os_agent:9680 blocked for more than 120 seconds.
      Tainted: P           -- ------------    2.6.32-754.11.1.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
os_agent      D 0000000000000002     0  9680      1 0x00000000
 ffff880434bb73c8 0000000000000082 ffff880434bb7318 0000000039890040
 ffff880028318c00 ffff880028318c00 ffff880028318c00 0000000000000000
 0000000000000002 0000000000000002 ffff880434b025f8 ffff880434bb7fd8
Call Trace:
 [<ffffffff81559dd5>] schedule_timeout+0x215/0x2f0
 [<ffffffff810aaec6>] ? autoremove_wake_function+0x16/0x40
 [<ffffffff8106251c>] ? __wake_up_common+0x5c/0x90
 [<ffffffff81559a33>] wait_for_common+0x123/0x180
 [<ffffffff81070870>] ? default_wake_function+0x0/0x20
 [<ffffffff810a49c1>] ? __queue_work+0x41/0x50
 [<ffffffff81559b6d>] wait_for_completion+0x1d/0x20
 [<ffffffff810a2e7c>] call_usermodehelper_exec+0x10c/0x130
 [<ffffffff81241199>] call_sbin_request_key+0x259/0x310
 [<ffffffff8155a36e>] ? mutex_lock+0x1e/0x50
 [<ffffffff8123b2c3>] ? key_instantiate_and_link+0xa3/0xb0
 [<ffffffffa0610030>] ? pnfs_choose_layoutget_stateid+0x70/0x130 [nfs]
 [<ffffffff81240cc7>] request_key_and_link+0x347/0x400
 [<ffffffff81240e90>] request_key+0x50/0xa0
 [<ffffffffa060b4c5>] nfs_idmap_request_key+0xc5/0x180 [nfs]
 [<ffffffffa060bb04>] nfs_idmap_lookup_id+0x34/0x80 [nfs]
 [<ffffffffa060bea5>] nfs_map_group_to_gid+0x75/0xa0 [nfs]
 [<ffffffffa0605c19>] decode_getfattr_attrs+0xd69/0x1040 [nfs]
 [<ffffffffa0605f7b>] decode_getfattr_generic.clone.0+0x8b/0xf0 [nfs]
 [<ffffffffa06060b3>] decode_getfattr+0x13/0x20 [nfs]
 [<ffffffffa060669f>] nfs4_xdr_dec_getattr+0x8f/0xa0 [nfs]
 [<ffffffffa0606610>] ? nfs4_xdr_dec_getattr+0x0/0xa0 [nfs]
 [<ffffffffa05395e6>] rpcauth_unwrap_resp+0x86/0xc0 [sunrpc]
 [<ffffffffa0606610>] ? nfs4_xdr_dec_getattr+0x0/0xa0 [nfs]
 [<ffffffffa052d9f9>] call_decode+0x1b9/0x840 [sunrpc]
 [<ffffffff810aaf30>] ? wake_bit_function+0x0/0x50
 [<ffffffffa052d840>] ? call_decode+0x0/0x840 [sunrpc]
 [<ffffffffa0537ba9>] __rpc_execute+0x79/0x380 [sunrpc]
 [<ffffffff810aad37>] ? bit_waitqueue+0x17/0xd0
 [<ffffffffa0537f11>] rpc_execute+0x61/0xa0 [sunrpc]
 [<ffffffffa052e4b5>] rpc_run_task+0x75/0x90 [sunrpc]
 [<ffffffffa052e5d2>] rpc_call_sync+0x42/0x70 [sunrpc]
 [<ffffffffa05fa126>] _nfs4_call_sync+0x36/0x40 [nfs]
 [<ffffffffa05f29f2>] _nfs4_proc_getattr+0xb2/0xc0 [nfs]
 [<ffffffffa05f5916>] nfs4_proc_getattr+0x56/0x80 [nfs]
 [<ffffffffa05db3a9>] __nfs_revalidate_inode+0xe9/0x220 [nfs]
 [<ffffffffa05dc23e>] nfs_getattr+0xde/0x210 [nfs]
 [<ffffffff811a71c4>] vfs_getattr+0x54/0x90
 [<ffffffff811a7004>] ? cp_new_stat+0xe4/0x100
 [<ffffffff811a7264>] vfs_fstatat+0x64/0xa0
 [<ffffffff811a73cb>] vfs_stat+0x1b/0x20
 [<ffffffff811a73f4>] sys_newstat+0x24/0x50
 [<ffffffffa0350a3d>] ? am_policy_is_active+0x2d/0x40 [secfs2]
 [<ffffffffa031e8f8>] ? __am_prelookup+0x48/0xf0 [secfs2]
 [<ffffffff811a73d0>] ? sys_newstat+0x0/0x50
 [<ffffffffa031fbf3>] m_sys_stat+0x43/0x60 [secfs2]
 [<ffffffffa0669a2e>] my_stat+0x7e/0x100 [seos]
 [<ffffffff81564357>] system_call_fastpath+0x35/0x3a
 [<ffffffff8156427e>] ? system_call_after_swapgs+0xae/0x152

Environment

  • Red Hat Enterprise Linux 7
  • Red Hat Enterprise Linux 6

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content