Kernel panics in stub_clone() function due to third party 'seos' module.

Solution Unverified - Updated -

Environment

  • Red Hat Enterprise Linux 6

Issue

  • System panics with the following logs:
[   82.411885] seos: module license 'Proprietary' taints kernel. <<----
[   82.412370] Disabling lock debugging due to kernel taint
[   82.414292] seos: applying 16k kernel stack fix up
[   82.414742] seos: applying 16k kernel stack fix up
.....
[   84.403061] SEOS Syscall Monitor - ACTIVATED                                
[   84.407594] BUG: unable to handle kernel paging request at ffffffff81564710 <<--- kernel panic
[   84.408081] IP: [<ffffffff81564726>] stub_clone+0x16/0x70
[   84.408573] Kernel PGD 1a91067 PUD 1a95063 PMD 14001e1 
[   84.409066] User   PGD 43ff62067 PUD 43ff65067 PMD 43ff66067 PTE 1564161
[   84.409619] Oops: 0003 [#1] SMP 
[   84.410150] last sysfs file: /sys/devices/system/cpu/online
[   84.410700] CPU 2 
[   84.410709] Modules linked in: seos(P)(U) vsock(U) vmci(U) ipv6 iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 iptable_filter  
....
[   84.414074] Pid: 9233, comm: seagent Tainted: P           -- ------------    2.6.32-754.3.5.el6.x86_64 #1 VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform
[   84.415662] RIP: 0010:[<ffffffff81564726>]  [<ffffffff81564726>] stub_clone+0x16/0x70
[   84.416498] RSP: 0018:ffff8803a09bbf50  EFLAGS: 00010202
[   84.417355] RAX: ffffffff81564710 RBX: 0000000000000000 RCX: 00007ff163ff99f0
[   84.418229] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000001200011
[   84.419113] RBP: 0000000000000000 R08: 00007ff163ff9720 R09: 0000000000002411
[   84.420014] R10: 00007ff163ff99f0 R11: 00007ffef9471d30 R12: 0000000000000000
[   84.420931] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[   84.421822] FS:  00007ff163ff9720(0000) GS:ffff880028300000(0000) knlGS:0000000000000000
[   84.422685] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   84.423524] CR2: ffffffff81564710 CR3: 0000000434c64000 CR4: 00000000000007e0
[   84.424385] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   84.425212] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[   84.426009] Process seagent (pid: 9233, threadinfo ffff8803a09b8000, task ffff88043277cab0)
...
[   84.430103] Call Trace:
[   84.430941]  [<ffffffff81564357>] ? system_call_fastpath+0x35/0x3a
[   84.431798] Code: 00 00 00 00 e9 34 ff ff ff 66 66 66 2e 0f 1f 84 00 00 00 00 00 65 4c 8b 1c 25 08 f7 00 00 4c 89 9c 24 a0 00 2d 12 e7 1e 84 24 a8 <00> 00 00 2b 00 00 00 48 c7 84 24 90 00 00 00 33 00 00 00 48 c7 
[   84.433770] RIP  [<ffffffff81564726>] stub_clone+0x16/0x70
[   84.434723]  RSP <ffff8803a09bbf50>
[   84.435678] CR2: ffffffff81564710

Resolution

  • Engage seos module vendor for further investigate this issue.

Root Cause

  • As per the backtrace output, 'seos' was calling stub_clone() which has passed invalid address to kernel.

Diagnostic Steps

  • Backtrace of panic task :
crash> bt
PID: 9233   TASK: ffff88043277cab0  CPU: 2   COMMAND: "seagent"
 #0 [ffff8803a09bbb40] machine_kexec at ffffffff81040f1b
 #1 [ffff8803a09bbba0] crash_kexec at ffffffff810d6722
 #2 [ffff8803a09bbc70] oops_end at ffffffff8155e2e0
 #3 [ffff8803a09bbca0] no_context at ffffffff8105450b
 #4 [ffff8803a09bbcf0] __bad_area_nosemaphore at ffffffff81054795
 #5 [ffff8803a09bbd40] bad_area_nosemaphore at ffffffff81054863
 #6 [ffff8803a09bbd50] __do_page_fault at ffffffff81055020
 #7 [ffff8803a09bbe70] do_page_fault at ffffffff8156029e
 #8 [ffff8803a09bbea0] page_fault at ffffffff8155d265
    [exception RIP: stub_clone+22]
    RIP: ffffffff81564726  RSP: ffff8803a09bbf50  RFLAGS: 00010202
    RAX: ffffffff81564710  RBX: 0000000000000000  RCX: 00007ff163ff99f0
    RDX: 0000000000000000  RSI: 0000000000000000  RDI: 0000000001200011
    RBP: 0000000000000000   R8: 00007ff163ff9720   R9: 0000000000002411
    R10: 00007ff163ff99f0  R11: 00007ffef9471d30  R12: 0000000000000000
    R13: 0000000000000000  R14: 0000000000000000  R15: 0000000000000000
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 #9 [ffff8803a09bbf50] system_call_fastpath at ffffffff81564357
    RIP: 00007ff1628e1d2d  RSP: 00007ffef9471d98  RFLAGS: 00010202
    RAX: 0000000000000038  RBX: 00007ffef9471d30  RCX: 00007ff16286762e
    RDX: 0000000000000000  RSI: 0000000000000000  RDI: 0000000001200011
    RBP: 00007ffef9471d90   R8: 00007ff163ff9720   R9: 0000000000002411
    R10: 00007ff163ff99f0  R11: 0000000000000246  R12: 0000000000000000
    R13: 00007ffef9471d50  R14: 0000000000000000  R15: 0000000000000000
    ORIG_RAX: 0000000000000038  CS: 0033  SS: 002b
  • The panic task is 'seagent' PID (9233):
crash> set -p
    PID: 9233
COMMAND: "seagent"
   TASK: ffff88043277cab0  [THREAD_INFO: ffff8803a09b8000]
    CPU: 2
  STATE: TASK_RUNNING (PANIC)

crash> task -R mm
PID: 9233   TASK: ffff88043277cab0  CPU: 2   COMMAND: "seagent"
  mm = 0xffff8803a0898e00, 

crash> mm_struct.exe_file 0xffff8803a0898e00
  exe_file = 0xffff8803a32a9dc0

crash> struct file.f_path.dentry 0xffff8803a32a9dc0
  f_path.dentry = 0xffff8803a365ec00

crash> files -d 0xffff8803a365ec00
     DENTRY           INODE           SUPERBLK     TYPE PATH
ffff8803a365ec00 ffff8803a37e14e8 ffff88043310ec00 REG  /usr/seos/bin/seagent   <<----
  • The kernel ring buffer shows that the proprietary (P) unsigned (U) kernel module seos was loaded and the seos syscall monitor was activated just before the kernel panic.
crash> log
[   82.411885] seos: module license 'Proprietary' taints kernel.
[   82.412370] Disabling lock debugging due to kernel taint
 .....
[   82.419490] seos: applying 16k kernel stack fix up
[   82.419928] seos: applying 16k kernel stack fix up
[   84.403061] SEOS Syscall Monitor - ACTIVATED
[   84.407594] BUG: unable to handle kernel paging request at ffffffff81564710   <<---  [kernel panicked]
  • Details of unsigned (U) kernel module: [seos]
crash> mod -t
NAME   TAINTS
vmci   (U)
vsock  (U)
seos   P(U) <<--

crash> mod | grep -e NAME -e seos
     MODULE       NAME                   SIZE  OBJECT FILE
ffffffffa0403100  seos                2873050  (not loaded)  [CONFIG_KALLSYMS]

crash> module.name,version,srcversion ffffffffa0403100
  name = "seos\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
  version = 0x0
  srcversion = 0xffff880436323480 "3322724B8462B94FB8B5E79"

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments