Kernel panics in stub_clone() function due to third party 'seos' module.
Environment
- Red Hat Enterprise Linux 6
Issue
- System panics with the following logs:
[ 82.411885] seos: module license 'Proprietary' taints kernel. <<----
[ 82.412370] Disabling lock debugging due to kernel taint
[ 82.414292] seos: applying 16k kernel stack fix up
[ 82.414742] seos: applying 16k kernel stack fix up
.....
[ 84.403061] SEOS Syscall Monitor - ACTIVATED
[ 84.407594] BUG: unable to handle kernel paging request at ffffffff81564710 <<--- kernel panic
[ 84.408081] IP: [<ffffffff81564726>] stub_clone+0x16/0x70
[ 84.408573] Kernel PGD 1a91067 PUD 1a95063 PMD 14001e1
[ 84.409066] User PGD 43ff62067 PUD 43ff65067 PMD 43ff66067 PTE 1564161
[ 84.409619] Oops: 0003 [#1] SMP
[ 84.410150] last sysfs file: /sys/devices/system/cpu/online
[ 84.410700] CPU 2
[ 84.410709] Modules linked in: seos(P)(U) vsock(U) vmci(U) ipv6 iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 iptable_filter
....
[ 84.414074] Pid: 9233, comm: seagent Tainted: P -- ------------ 2.6.32-754.3.5.el6.x86_64 #1 VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform
[ 84.415662] RIP: 0010:[<ffffffff81564726>] [<ffffffff81564726>] stub_clone+0x16/0x70
[ 84.416498] RSP: 0018:ffff8803a09bbf50 EFLAGS: 00010202
[ 84.417355] RAX: ffffffff81564710 RBX: 0000000000000000 RCX: 00007ff163ff99f0
[ 84.418229] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000001200011
[ 84.419113] RBP: 0000000000000000 R08: 00007ff163ff9720 R09: 0000000000002411
[ 84.420014] R10: 00007ff163ff99f0 R11: 00007ffef9471d30 R12: 0000000000000000
[ 84.420931] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[ 84.421822] FS: 00007ff163ff9720(0000) GS:ffff880028300000(0000) knlGS:0000000000000000
[ 84.422685] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 84.423524] CR2: ffffffff81564710 CR3: 0000000434c64000 CR4: 00000000000007e0
[ 84.424385] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 84.425212] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 84.426009] Process seagent (pid: 9233, threadinfo ffff8803a09b8000, task ffff88043277cab0)
...
[ 84.430103] Call Trace:
[ 84.430941] [<ffffffff81564357>] ? system_call_fastpath+0x35/0x3a
[ 84.431798] Code: 00 00 00 00 e9 34 ff ff ff 66 66 66 2e 0f 1f 84 00 00 00 00 00 65 4c 8b 1c 25 08 f7 00 00 4c 89 9c 24 a0 00 2d 12 e7 1e 84 24 a8 <00> 00 00 2b 00 00 00 48 c7 84 24 90 00 00 00 33 00 00 00 48 c7
[ 84.433770] RIP [<ffffffff81564726>] stub_clone+0x16/0x70
[ 84.434723] RSP <ffff8803a09bbf50>
[ 84.435678] CR2: ffffffff81564710
Resolution
- Engage
seos
module vendor for further investigate this issue.
Root Cause
- As per the backtrace output, 'seos' was calling stub_clone() which has passed invalid address to kernel.
Diagnostic Steps
- Backtrace of panic task :
crash> bt
PID: 9233 TASK: ffff88043277cab0 CPU: 2 COMMAND: "seagent"
#0 [ffff8803a09bbb40] machine_kexec at ffffffff81040f1b
#1 [ffff8803a09bbba0] crash_kexec at ffffffff810d6722
#2 [ffff8803a09bbc70] oops_end at ffffffff8155e2e0
#3 [ffff8803a09bbca0] no_context at ffffffff8105450b
#4 [ffff8803a09bbcf0] __bad_area_nosemaphore at ffffffff81054795
#5 [ffff8803a09bbd40] bad_area_nosemaphore at ffffffff81054863
#6 [ffff8803a09bbd50] __do_page_fault at ffffffff81055020
#7 [ffff8803a09bbe70] do_page_fault at ffffffff8156029e
#8 [ffff8803a09bbea0] page_fault at ffffffff8155d265
[exception RIP: stub_clone+22]
RIP: ffffffff81564726 RSP: ffff8803a09bbf50 RFLAGS: 00010202
RAX: ffffffff81564710 RBX: 0000000000000000 RCX: 00007ff163ff99f0
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000001200011
RBP: 0000000000000000 R8: 00007ff163ff9720 R9: 0000000000002411
R10: 00007ff163ff99f0 R11: 00007ffef9471d30 R12: 0000000000000000
R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#9 [ffff8803a09bbf50] system_call_fastpath at ffffffff81564357
RIP: 00007ff1628e1d2d RSP: 00007ffef9471d98 RFLAGS: 00010202
RAX: 0000000000000038 RBX: 00007ffef9471d30 RCX: 00007ff16286762e
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000001200011
RBP: 00007ffef9471d90 R8: 00007ff163ff9720 R9: 0000000000002411
R10: 00007ff163ff99f0 R11: 0000000000000246 R12: 0000000000000000
R13: 00007ffef9471d50 R14: 0000000000000000 R15: 0000000000000000
ORIG_RAX: 0000000000000038 CS: 0033 SS: 002b
- The panic task is 'seagent' PID (9233):
crash> set -p
PID: 9233
COMMAND: "seagent"
TASK: ffff88043277cab0 [THREAD_INFO: ffff8803a09b8000]
CPU: 2
STATE: TASK_RUNNING (PANIC)
crash> task -R mm
PID: 9233 TASK: ffff88043277cab0 CPU: 2 COMMAND: "seagent"
mm = 0xffff8803a0898e00,
crash> mm_struct.exe_file 0xffff8803a0898e00
exe_file = 0xffff8803a32a9dc0
crash> struct file.f_path.dentry 0xffff8803a32a9dc0
f_path.dentry = 0xffff8803a365ec00
crash> files -d 0xffff8803a365ec00
DENTRY INODE SUPERBLK TYPE PATH
ffff8803a365ec00 ffff8803a37e14e8 ffff88043310ec00 REG /usr/seos/bin/seagent <<----
- The kernel ring buffer shows that the proprietary (P) unsigned (U) kernel module
seos
was loaded and theseos
syscall monitor was activated just before the kernel panic.
crash> log
[ 82.411885] seos: module license 'Proprietary' taints kernel.
[ 82.412370] Disabling lock debugging due to kernel taint
.....
[ 82.419490] seos: applying 16k kernel stack fix up
[ 82.419928] seos: applying 16k kernel stack fix up
[ 84.403061] SEOS Syscall Monitor - ACTIVATED
[ 84.407594] BUG: unable to handle kernel paging request at ffffffff81564710 <<--- [kernel panicked]
- Details of unsigned (U) kernel module: [seos]
crash> mod -t
NAME TAINTS
vmci (U)
vsock (U)
seos P(U) <<--
crash> mod | grep -e NAME -e seos
MODULE NAME SIZE OBJECT FILE
ffffffffa0403100 seos 2873050 (not loaded) [CONFIG_KALLSYMS]
crash> module.name,version,srcversion ffffffffa0403100
name = "seos\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
version = 0x0
srcversion = 0xffff880436323480 "3322724B8462B94FB8B5E79"
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
Comments