RHEL6.7: machine hung, many tasks waiting on nfs4 idmapper, khelper stuck

Solution In Progress - Updated -

Issue

  • machine hung with many processes blocked in a backtrace waiting on nfs4 idmapping, with a backtrace similar to this
PID: 5905   TASK: ffff88083082f520  CPU: 7   COMMAND: "smbd"
 #0 [ffff88010021b4e8] schedule at ffffffff81538130
 #1 [ffff88010021b5c0] schedule_timeout at ffffffff81539085
 #2 [ffff88010021b670] wait_for_common at ffffffff81538d03
 #3 [ffff88010021b700] wait_for_completion at ffffffff81538e1d
 #4 [ffff88010021b710] call_usermodehelper_exec at ffffffff8109961c
 #5 [ffff88010021b760] call_sbin_request_key at ffffffff8122ef3e
 #6 [ffff88010021b8c0] request_key_and_link at ffffffff8122ea75
 #7 [ffff88010021b960] request_key at ffffffff8122ec40
 #8 [ffff88010021b9b0] nfs_idmap_request_key at ffffffffa02e5f25 [nfs]
 #9 [ffff88010021ba10] nfs_idmap_lookup_id at ffffffffa02e6554 [nfs]
#10 [ffff88010021ba60] nfs_map_name_to_uid at ffffffffa02e6995 [nfs]
  • a khelper thread is stuck inside fork, which gets stuck due to migrate_pages trying to lock_page
PID: 68     TASK: ffff880834122040  CPU: 4   COMMAND: "khelper"
 #0 [ffff880834137470] schedule at ffffffff81538130
 #1 [ffff880834137548] io_schedule at ffffffff81538973
 #2 [ffff880834137568] sync_page at ffffffff811275dd
 #3 [ffff880834137578] __wait_on_bit_lock at ffffffff8153920a
 #4 [ffff8808341375c8] __lock_page at ffffffff81127577
 #5 [ffff880834137628] lock_page at ffffffff8117ce50
 #6 [ffff880834137648] migrate_pages at ffffffff8117d9bb
 #7 [ffff8808341376e8] compact_zone at ffffffff81172aa5
 #8 [ffff8808341377c8] compact_zone_order at ffffffff811732fc
 #9 [ffff880834137878] zone_reclaim at ffffffff811435c0
#10 [ffff880834137998] get_page_from_freelist at ffffffff811354cc
#11 [ffff880834137ac8] __alloc_pages_nodemask at ffffffff81137059
#12 [ffff880834137c18] copy_process at ffffffff81074fa8
#13 [ffff880834137ca8] do_fork at ffffffff810764c6
#14 [ffff880834137d68] kernel_thread at ffffffff8100c222
#15 [ffff880834137e38] worker_thread at ffffffff8109a7d0
#16 [ffff880834137ee8] kthread at ffffffff810a101e

Environment

  • Red Hat Enterprise Linux 6
    • seen on 2.6.32-573.1.1.el6
  • nfs4 with idmapping
  • seen with smbd

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In