RHEL6.7: machine hung, many tasks waiting on nfs4 idmapper, khelper stuck
Issue
- machine hung with many processes blocked in a backtrace waiting on nfs4 idmapping, with a backtrace similar to this
PID: 5905 TASK: ffff88083082f520 CPU: 7 COMMAND: "smbd"
#0 [ffff88010021b4e8] schedule at ffffffff81538130
#1 [ffff88010021b5c0] schedule_timeout at ffffffff81539085
#2 [ffff88010021b670] wait_for_common at ffffffff81538d03
#3 [ffff88010021b700] wait_for_completion at ffffffff81538e1d
#4 [ffff88010021b710] call_usermodehelper_exec at ffffffff8109961c
#5 [ffff88010021b760] call_sbin_request_key at ffffffff8122ef3e
#6 [ffff88010021b8c0] request_key_and_link at ffffffff8122ea75
#7 [ffff88010021b960] request_key at ffffffff8122ec40
#8 [ffff88010021b9b0] nfs_idmap_request_key at ffffffffa02e5f25 [nfs]
#9 [ffff88010021ba10] nfs_idmap_lookup_id at ffffffffa02e6554 [nfs]
#10 [ffff88010021ba60] nfs_map_name_to_uid at ffffffffa02e6995 [nfs]
- a khelper thread is stuck inside fork, which gets stuck due to migrate_pages trying to lock_page
PID: 68 TASK: ffff880834122040 CPU: 4 COMMAND: "khelper"
#0 [ffff880834137470] schedule at ffffffff81538130
#1 [ffff880834137548] io_schedule at ffffffff81538973
#2 [ffff880834137568] sync_page at ffffffff811275dd
#3 [ffff880834137578] __wait_on_bit_lock at ffffffff8153920a
#4 [ffff8808341375c8] __lock_page at ffffffff81127577
#5 [ffff880834137628] lock_page at ffffffff8117ce50
#6 [ffff880834137648] migrate_pages at ffffffff8117d9bb
#7 [ffff8808341376e8] compact_zone at ffffffff81172aa5
#8 [ffff8808341377c8] compact_zone_order at ffffffff811732fc
#9 [ffff880834137878] zone_reclaim at ffffffff811435c0
#10 [ffff880834137998] get_page_from_freelist at ffffffff811354cc
#11 [ffff880834137ac8] __alloc_pages_nodemask at ffffffff81137059
#12 [ffff880834137c18] copy_process at ffffffff81074fa8
#13 [ffff880834137ca8] do_fork at ffffffff810764c6
#14 [ffff880834137d68] kernel_thread at ffffffff8100c222
#15 [ffff880834137e38] worker_thread at ffffffff8109a7d0
#16 [ffff880834137ee8] kthread at ffffffff810a101e
Environment
- Red Hat Enterprise Linux 6
- seen on 2.6.32-573.1.1.el6
- nfs4 with idmapping
- seen with smbd
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.