RHEL6.8: rpciod kernel crash in worker_thread due to corrupt work_struct, __list_add corruption during mount GETPORT rpcbind call

Issue

• First there is a WARN_ON backtrace indicating that a list_head is corrupted

<4>------------[ cut here ]------------
<4>WARNING: at lib/list_debug.c:30 __list_add+0x8f/0xa0() (Tainted: P           -- ------------   )
<4>Hardware name: VMware Virtual Platform
<4>list_add corruption. prev->next should be next (ffffe8ffffc0e888), but was (null). (prev=ffff8806327bb438).
<4>Modules linked in: vsock(U) vmci(U) nfs lockd fscache auth_rpcgss nfs_acl sunrpc seos(P)(U) autofs4 ipv6 ppdev parport_pc parport microcode vmware_balloon sg i2c_piix4 shpchp ext4 jbd2 mbcache sd_mod crc_t10dif sr_mod cdrom vmxnet3 vmw_pvscsi pata_acpi ata_generic ata_piix vmwgfx ttm drm_kms_helper drm i2c_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: vmci]
<4>Pid: 4818, comm: mount.nfs Tainted: P           -- ------------    2.6.32-642.6.2.el6.x86_64 #1
<4>Call Trace:
<4> [<ffffffff8107c6f1>] ? warn_slowpath_common+0x91/0xe0
<4> [<ffffffff8107c7f6>] ? warn_slowpath_fmt+0x46/0x60
<4> [<ffffffff812ae86f>] ? __list_add+0x8f/0xa0
<4> [<ffffffff8109fe37>] ? insert_work+0x57/0xb0
<4> [<ffffffff81130d43>] ? mempool_alloc+0x63/0x140
<4> [<ffffffff810a04e6>] ? __queue_work+0x36/0x50
<4> [<ffffffff810a0602>] ? queue_work_on+0x42/0x60
<4> [<ffffffff810a07df>] ? queue_work+0x1f/0x30
<4> [<ffffffffa06a23ee>] ? rpc_make_runnable+0x7e/0x80 [sunrpc]
<4> [<ffffffffa06a2c90>] ? rpc_execute+0x50/0xa0 [sunrpc]
<4> [<ffffffffa0699455>] ? rpc_run_task+0x75/0x90 [sunrpc]
<4> [<ffffffffa06acc39>] ? rpcb_call_async+0x59/0x60 [sunrpc]
<4> [<ffffffffa06acf46>] ? rpcb_getport_async+0x306/0x520 [sunrpc]
<4> [<ffffffff81130ba5>] ? mempool_alloc_slab+0x15/0x20
<4> [<ffffffffa0697410>] ? call_bind+0x0/0x90 [sunrpc]
<4> [<ffffffffa069746c>] ? call_bind+0x5c/0x90 [sunrpc]
<4> [<ffffffffa06a2967>] ? __rpc_execute+0x77/0x350 [sunrpc]
<4> [<ffffffff810a6727>] ? bit_waitqueue+0x17/0xd0
<4> [<ffffffffa06a2ca1>] ? rpc_execute+0x61/0xa0 [sunrpc]
<4> [<ffffffffa0699455>] ? rpc_run_task+0x75/0x90 [sunrpc]
<4> [<ffffffffa0699572>] ? rpc_call_sync+0x42/0x70 [sunrpc]
<4> [<ffffffffa06995f2>] ? rpc_ping+0x52/0x70 [sunrpc]
<4> [<ffffffffa0699f68>] ? rpc_create+0x458/0x5b0 [sunrpc]
<4> [<ffffffff8117f76b>] ? cache_alloc_refill+0x15b/0x240
<4> [<ffffffffa073bccb>] ? nfs_create_rpc_client+0xcb/0x110 [nfs]
<4> [<ffffffffa073c0ac>] ? nfs_init_client+0x4c/0xb0 [nfs]
<4> [<ffffffffa073c686>] ? nfs_get_client+0x4c6/0x5a0 [nfs]
<4> [<ffffffffa06a3520>] ? __rpc_init_priority_wait_queue+0x80/0xb0 [sunrpc]
<4> [<ffffffffa073daaf>] ? nfs_create_server+0xcf/0x590 [nfs]
<4> [<ffffffffa074b13c>] ? nfs_get_sb+0x2dc/0x880 [nfs]
<4> [<ffffffff8119c6cb>] ? vfs_kern_mount+0x7b/0x1b0
<4> [<ffffffff8119c872>] ? do_kern_mount+0x52/0x130
<4> [<ffffffff811be7cb>] ? do_mount+0x2fb/0x930
<4> [<ffffffff811bee90>] ? sys_mount+0x90/0xe0
<4> [<ffffffffa036b121>] ? my_mount+0x91/0x1e0 [seos]
<4> [<ffffffff8100b0d2>] ? system_call_fastpath+0x16/0x1b
<4>---[ end trace 03ce1b060b406ac1 ]---

• Next the kernel panic with rpciod thread crashing in worker_thread with kernel NULL pointer dereference at (null)

<1>BUG: unable to handle kernel NULL pointer dereference at (null)
<1>IP: [<ffffffff8109fb6f>] worker_thread+0x13f/0x2a0
<4>PGD 0 
<4>Oops: 0002 [#1] SMP 
<4>last sysfs file: /sys/devices/system/cpu/online
<4>CPU 0 
<4>Modules linked in: vsock(U) vmci(U) nfs lockd fscache auth_rpcgss nfs_acl sunrpc seos(P)(U) autofs4 ipv6 ppdev parport_pc parport microcode vmware_balloon sg i2c_piix4 shpchp ext4 jbd2 mbcache sd_mod crc_t10dif sr_mod cdrom vmxnet3 vmw_pvscsi pata_acpi ata_generic ata_piix vmwgfx ttm drm_kms_helper drm i2c_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: vmci]
<4>
<4>Pid: 26918, comm: rpciod/0 Tainted: P        W  -- ------------    2.6.32-642.6.2.el6.x86_64 #1 VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform
<4>RIP: 0010:[<ffffffff8109fb6f>]  [<ffffffff8109fb6f>] worker_thread+0x13f/0x2a0
<4>RSP: 0018:ffff88063134be40  EFLAGS: 00010046
<4>RAX: ffff8806327bb438 RBX: ffffe8ffffc0e880 RCX: 0000000000000000
<4>RDX: ffff8806327bb430 RSI: ffff880630db4610 RDI: ffffe8ffffc0e880
<4>RBP: ffff88063134bee0 R08: 0000000000000000 R09: 0000000000000000
<4>R10: 0000000000000000 R11: 0000000000000003 R12: ffff880571222ab0
<4>R13: 0000000000000000 R14: ffff88063134bfd8 R15: ffffe8ffffc0e888
<4>FS:  0000000000000000(0000) GS:ffff880028200000(0000) knlGS:0000000000000000
<4>CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
<4>CR2: 0000000000000000 CR3: 0000000636967000 CR4: 00000000000007f0
<4>DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<4>DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
<4>Process rpciod/0 (pid: 26918, threadinfo ffff880631348000, task ffff880571222ab0)
<4>Stack:
<4> 0000000000000000 0000000000000000 ffff88063134be60 ffff880571223128
<4><d> ffff880571222ab0 ffff880571222ab0 ffff880571222ab0 ffffe8ffffc0e898
<4><d> 0000000000000000 ffff880571222ab0 ffffffff810a68a0 ffff88063134be98
<4>Call Trace:
<4> [<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
<4> [<ffffffff8109fa30>] ? worker_thread+0x0/0x2a0
<4> [<ffffffff810a640e>] kthread+0x9e/0xc0
<4> [<ffffffff8100c28a>] child_rip+0xa/0x20
<4> [<ffffffff810a6370>] ? kthread+0x0/0xc0
<4> [<ffffffff8100c280>] ? child_rip+0x0/0x20
<4>Code: 16 b4 00 48 83 ea 08 4c 8b 63 40 4c 8b 6a 18 45 85 c0 0f 85 de 00 00 00 48 8b 43 08 48 89 53 30 48 8b 30 48 8b 48 08 48 89 4e 08 <48> 89 31 48 89 00 48 89 40 08 c7 03 00 00 00 00 fb 66 0f 1f 44 
<1>RIP  [<ffffffff8109fb6f>] worker_thread+0x13f/0x2a0
<4> RSP <ffff88063134be40>
<4>CR2: 0000000000000000