RHEL6.7: machine hung, many tasks waiting on nfs4 idmapper, khelper stuck

Solution In Progress - Updated -

Issue

  • machine hung with many processes blocked in a backtrace waiting on nfs4 idmapping, with a backtrace similar to this
PID: 5905   TASK: ffff88083082f520  CPU: 7   COMMAND: "smbd"
 #0 [ffff88010021b4e8] schedule at ffffffff81538130
 #1 [ffff88010021b5c0] schedule_timeout at ffffffff81539085
 #2 [ffff88010021b670] wait_for_common at ffffffff81538d03
 #3 [ffff88010021b700] wait_for_completion at ffffffff81538e1d
 #4 [ffff88010021b710] call_usermodehelper_exec at ffffffff8109961c
 #5 [ffff88010021b760] call_sbin_request_key at ffffffff8122ef3e
 #6 [ffff88010021b8c0] request_key_and_link at ffffffff8122ea75
 #7 [ffff88010021b960] request_key at ffffffff8122ec40
 #8 [ffff88010021b9b0] nfs_idmap_request_key at ffffffffa02e5f25 [nfs]
 #9 [ffff88010021ba10] nfs_idmap_lookup_id at ffffffffa02e6554 [nfs]
#10 [ffff88010021ba60] nfs_map_name_to_uid at ffffffffa02e6995 [nfs]
  • a khelper thread is stuck inside fork, which gets stuck due to migrate_pages trying to lock_page
PID: 68     TASK: ffff880834122040  CPU: 4   COMMAND: "khelper"
 #0 [ffff880834137470] schedule at ffffffff81538130
 #1 [ffff880834137548] io_schedule at ffffffff81538973
 #2 [ffff880834137568] sync_page at ffffffff811275dd
 #3 [ffff880834137578] __wait_on_bit_lock at ffffffff8153920a
 #4 [ffff8808341375c8] __lock_page at ffffffff81127577
 #5 [ffff880834137628] lock_page at ffffffff8117ce50
 #6 [ffff880834137648] migrate_pages at ffffffff8117d9bb
 #7 [ffff8808341376e8] compact_zone at ffffffff81172aa5
 #8 [ffff8808341377c8] compact_zone_order at ffffffff811732fc
 #9 [ffff880834137878] zone_reclaim at ffffffff811435c0
#10 [ffff880834137998] get_page_from_freelist at ffffffff811354cc
#11 [ffff880834137ac8] __alloc_pages_nodemask at ffffffff81137059
#12 [ffff880834137c18] copy_process at ffffffff81074fa8
#13 [ffff880834137ca8] do_fork at ffffffff810764c6
#14 [ffff880834137d68] kernel_thread at ffffffff8100c222
#15 [ffff880834137e38] worker_thread at ffffffff8109a7d0
#16 [ffff880834137ee8] kthread at ffffffff810a101e

Environment

  • Red Hat Enterprise Linux 6
    • seen on 2.6.32-573.1.1.el6
  • nfs4 with idmapping
  • seen with smbd

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content