RHEL6.8: rpciod kernel crash in worker_thread due to corrupt work_struct, __list_add corruption during mount GETPORT rpcbind call
Issue
- First there is a WARN_ON backtrace indicating that a list_head is corrupted
<4>------------[ cut here ]------------
<4>WARNING: at lib/list_debug.c:30 __list_add+0x8f/0xa0() (Tainted: P -- ------------ )
<4>Hardware name: VMware Virtual Platform
<4>list_add corruption. prev->next should be next (ffffe8ffffc0e888), but was (null). (prev=ffff8806327bb438).
<4>Modules linked in: vsock(U) vmci(U) nfs lockd fscache auth_rpcgss nfs_acl sunrpc seos(P)(U) autofs4 ipv6 ppdev parport_pc parport microcode vmware_balloon sg i2c_piix4 shpchp ext4 jbd2 mbcache sd_mod crc_t10dif sr_mod cdrom vmxnet3 vmw_pvscsi pata_acpi ata_generic ata_piix vmwgfx ttm drm_kms_helper drm i2c_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: vmci]
<4>Pid: 4818, comm: mount.nfs Tainted: P -- ------------ 2.6.32-642.6.2.el6.x86_64 #1
<4>Call Trace:
<4> [<ffffffff8107c6f1>] ? warn_slowpath_common+0x91/0xe0
<4> [<ffffffff8107c7f6>] ? warn_slowpath_fmt+0x46/0x60
<4> [<ffffffff812ae86f>] ? __list_add+0x8f/0xa0
<4> [<ffffffff8109fe37>] ? insert_work+0x57/0xb0
<4> [<ffffffff81130d43>] ? mempool_alloc+0x63/0x140
<4> [<ffffffff810a04e6>] ? __queue_work+0x36/0x50
<4> [<ffffffff810a0602>] ? queue_work_on+0x42/0x60
<4> [<ffffffff810a07df>] ? queue_work+0x1f/0x30
<4> [<ffffffffa06a23ee>] ? rpc_make_runnable+0x7e/0x80 [sunrpc]
<4> [<ffffffffa06a2c90>] ? rpc_execute+0x50/0xa0 [sunrpc]
<4> [<ffffffffa0699455>] ? rpc_run_task+0x75/0x90 [sunrpc]
<4> [<ffffffffa06acc39>] ? rpcb_call_async+0x59/0x60 [sunrpc]
<4> [<ffffffffa06acf46>] ? rpcb_getport_async+0x306/0x520 [sunrpc]
<4> [<ffffffff81130ba5>] ? mempool_alloc_slab+0x15/0x20
<4> [<ffffffffa0697410>] ? call_bind+0x0/0x90 [sunrpc]
<4> [<ffffffffa069746c>] ? call_bind+0x5c/0x90 [sunrpc]
<4> [<ffffffffa06a2967>] ? __rpc_execute+0x77/0x350 [sunrpc]
<4> [<ffffffff810a6727>] ? bit_waitqueue+0x17/0xd0
<4> [<ffffffffa06a2ca1>] ? rpc_execute+0x61/0xa0 [sunrpc]
<4> [<ffffffffa0699455>] ? rpc_run_task+0x75/0x90 [sunrpc]
<4> [<ffffffffa0699572>] ? rpc_call_sync+0x42/0x70 [sunrpc]
<4> [<ffffffffa06995f2>] ? rpc_ping+0x52/0x70 [sunrpc]
<4> [<ffffffffa0699f68>] ? rpc_create+0x458/0x5b0 [sunrpc]
<4> [<ffffffff8117f76b>] ? cache_alloc_refill+0x15b/0x240
<4> [<ffffffffa073bccb>] ? nfs_create_rpc_client+0xcb/0x110 [nfs]
<4> [<ffffffffa073c0ac>] ? nfs_init_client+0x4c/0xb0 [nfs]
<4> [<ffffffffa073c686>] ? nfs_get_client+0x4c6/0x5a0 [nfs]
<4> [<ffffffffa06a3520>] ? __rpc_init_priority_wait_queue+0x80/0xb0 [sunrpc]
<4> [<ffffffffa073daaf>] ? nfs_create_server+0xcf/0x590 [nfs]
<4> [<ffffffffa074b13c>] ? nfs_get_sb+0x2dc/0x880 [nfs]
<4> [<ffffffff8119c6cb>] ? vfs_kern_mount+0x7b/0x1b0
<4> [<ffffffff8119c872>] ? do_kern_mount+0x52/0x130
<4> [<ffffffff811be7cb>] ? do_mount+0x2fb/0x930
<4> [<ffffffff811bee90>] ? sys_mount+0x90/0xe0
<4> [<ffffffffa036b121>] ? my_mount+0x91/0x1e0 [seos]
<4> [<ffffffff8100b0d2>] ? system_call_fastpath+0x16/0x1b
<4>---[ end trace 03ce1b060b406ac1 ]---
- Next the kernel panic with rpciod thread crashing in worker_thread with
kernel NULL pointer dereference at (null)
<1>BUG: unable to handle kernel NULL pointer dereference at (null)
<1>IP: [<ffffffff8109fb6f>] worker_thread+0x13f/0x2a0
<4>PGD 0
<4>Oops: 0002 [#1] SMP
<4>last sysfs file: /sys/devices/system/cpu/online
<4>CPU 0
<4>Modules linked in: vsock(U) vmci(U) nfs lockd fscache auth_rpcgss nfs_acl sunrpc seos(P)(U) autofs4 ipv6 ppdev parport_pc parport microcode vmware_balloon sg i2c_piix4 shpchp ext4 jbd2 mbcache sd_mod crc_t10dif sr_mod cdrom vmxnet3 vmw_pvscsi pata_acpi ata_generic ata_piix vmwgfx ttm drm_kms_helper drm i2c_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: vmci]
<4>
<4>Pid: 26918, comm: rpciod/0 Tainted: P W -- ------------ 2.6.32-642.6.2.el6.x86_64 #1 VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform
<4>RIP: 0010:[<ffffffff8109fb6f>] [<ffffffff8109fb6f>] worker_thread+0x13f/0x2a0
<4>RSP: 0018:ffff88063134be40 EFLAGS: 00010046
<4>RAX: ffff8806327bb438 RBX: ffffe8ffffc0e880 RCX: 0000000000000000
<4>RDX: ffff8806327bb430 RSI: ffff880630db4610 RDI: ffffe8ffffc0e880
<4>RBP: ffff88063134bee0 R08: 0000000000000000 R09: 0000000000000000
<4>R10: 0000000000000000 R11: 0000000000000003 R12: ffff880571222ab0
<4>R13: 0000000000000000 R14: ffff88063134bfd8 R15: ffffe8ffffc0e888
<4>FS: 0000000000000000(0000) GS:ffff880028200000(0000) knlGS:0000000000000000
<4>CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
<4>CR2: 0000000000000000 CR3: 0000000636967000 CR4: 00000000000007f0
<4>DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<4>DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
<4>Process rpciod/0 (pid: 26918, threadinfo ffff880631348000, task ffff880571222ab0)
<4>Stack:
<4> 0000000000000000 0000000000000000 ffff88063134be60 ffff880571223128
<4><d> ffff880571222ab0 ffff880571222ab0 ffff880571222ab0 ffffe8ffffc0e898
<4><d> 0000000000000000 ffff880571222ab0 ffffffff810a68a0 ffff88063134be98
<4>Call Trace:
<4> [<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
<4> [<ffffffff8109fa30>] ? worker_thread+0x0/0x2a0
<4> [<ffffffff810a640e>] kthread+0x9e/0xc0
<4> [<ffffffff8100c28a>] child_rip+0xa/0x20
<4> [<ffffffff810a6370>] ? kthread+0x0/0xc0
<4> [<ffffffff8100c280>] ? child_rip+0x0/0x20
<4>Code: 16 b4 00 48 83 ea 08 4c 8b 63 40 4c 8b 6a 18 45 85 c0 0f 85 de 00 00 00 48 8b 43 08 48 89 53 30 48 8b 30 48 8b 48 08 48 89 4e 08 <48> 89 31 48 89 00 48 89 40 08 c7 03 00 00 00 00 fb 66 0f 1f 44
<1>RIP [<ffffffff8109fb6f>] worker_thread+0x13f/0x2a0
<4> RSP <ffff88063134be40>
<4>CR2: 0000000000000000
Environment
- Red Hat Enterprise Linux 6.8 (NFS client)
- seen on 2.6.32-642.6.2.el6
- 3 tainted modules, which are most likely third-party modules and unsupported by Red Hat.
For more information on 3rd party modules and how they affect support, see https://access.redhat.com/solutions/42843 and https://access.redhat.com/articles/1067
crash> mod -t
NAME TAINTS
seos P(U)
vmci (U)
vsock (U)
non-zero kernel taint value 201 PW
For explanation of taint values, see https://access.redhat.com/solutions/40594
crash> sys -t
TAINTED_MASK: 201 PW
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
