NFS server is running at 100% CPU and eventually hangs

Issue

NFS server running at 100% CPU and locks up, following messages are printed in logs

Sep 22 09:53:25 hostname kernel: BUG: soft lockup - CPU#5 stuck for 10s! [rpc.mountd:6376]
...
...
Sep 22 09:53:25 hostname kernel: Pid: 6376, comm: rpc.mountd Not tainted 2.6.18-128.1.1.el5 #1
Sep 22 09:53:25 hostname kernel: RIP: 0010:[<ffffffff80064cb4>]  [<ffffffff80064cb4>] .text.lock.spinlock+0x2/0x30
...
...
Sep 22 09:53:25 hostname kernel: Call Trace:
Sep 22 09:53:25 hostname kernel:  [<ffffffff88591f9d>] :sunrpc:auth_domain_lookup+0xa0/0x12d
Sep 22 09:53:25 hostname kernel:  [<ffffffff800c8749>] zone_statistics+0x3e/0x6d
Sep 22 09:53:25 hostname kernel:  [<ffffffff88605a2c>] :nfsd:svc_export_parse+0xce/0x713
Sep 22 09:53:25 hostname kernel:  [<ffffffff8000f10b>] __alloc_pages+0x65/0x2ce
Sep 22 09:53:25 hostname kernel:  [<ffffffff80008dba>] __handle_mm_fault+0x74f/0xe5c
Sep 22 09:53:25 hostname kernel:  [<ffffffff80021afc>] __up_read+0x19/0x7f
Sep 22 09:53:25 hostname kernel:  [<ffffffff80066bcd>] do_page_fault+0x4fe/0x830
Sep 22 09:53:25 hostname kernel:  [<ffffffff8000dc29>] do_mmap_pgoff+0x431/0x7d7
Sep 22 09:53:25 hostname kernel:  [<ffffffff8000de64>] do_mmap_pgoff+0x66c/0x7d7
Sep 22 09:53:25 hostname kernel:  [<ffffffff8859558a>] :sunrpc:cache_write+0x8e/0xc0
Sep 22 09:53:25 hostname kernel:  [<ffffffff8001659e>] vfs_write+0xce/0x174
Sep 22 09:53:25 hostname kernel:  [<ffffffff80016e6b>] sys_write+0x45/0x6e
Sep 22 09:53:25 hostname kernel:  [<ffffffff8005d28d>] tracesys+0xd5/0xe0


Sep 22 09:54:16 hostname kernel: BUG: soft lockup - CPU#4 stuck for 10s! [nfsd:6366]
...
...
Sep 22 09:54:16 hostname kernel: Pid: 6366, comm: nfsd Not tainted 2.6.18-128.1.1.el5 #1
...
...
Sep 22 09:54:16 hostname kernel: Call Trace:
Sep 22 09:54:16 hostname kernel:  [<ffffffff8000c38a>] _atomic_dec_and_lock+0x39/0x57
Sep 22 09:54:16 hostname kernel:  [<ffffffff88592041>] :sunrpc:auth_domain_put+0x10/0x4d
Sep 22 09:54:16 hostname kernel:  [<ffffffff88592220>] :sunrpc:svcauth_unix_release+0x15/0x49
Sep 22 09:54:16 hostname kernel:  [<ffffffff8859209a>] :sunrpc:svc_authorise+0x1c/0x57
Sep 22 09:54:16 hostname kernel:  [<ffffffff8858f4e6>] :sunrpc:svc_process+0x4af/0x71b
Sep 22 09:54:16 hostname kernel:  [<ffffffff800646f5>] __down_read+0x12/0x92
Sep 22 09:54:16 hostname kernel:  [<ffffffff885fe5a1>] :nfsd:nfsd+0x0/0x2cb
Sep 22 09:54:16 hostname kernel:  [<ffffffff885fe746>] :nfsd:nfsd+0x1a5/0x2cb
Sep 22 09:54:16 hostname kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Sep 22 09:54:16 hostname kernel:  [<ffffffff885fe5a1>] :nfsd:nfsd+0x0/0x2cb
Sep 22 09:54:16 hostname kernel:  [<ffffffff885fe5a1>] :nfsd:nfsd+0x0/0x2cb
Sep 22 09:54:16 hostname kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11