Kernel panicked in "rfs_open" function of redirfs(U) module.

Solution Unverified - Updated -

Environment

  • Red Hat Enterprise Linux 6

  • redirfs Unsigned (U) module

Issue

  • Kernel panicked in "rfs_open" function of redirfs(U) module.
BUG: unable to handle kernel NULL pointer dereference at 0000000000000030
IP: [<ffffffffa03194f0>] rfs_open+0x50/0x400 [redirfs]
...
Pid: 28496, comm: vertica Not tainted 2.6.32-431.el6.x86_64 #1 Dell Inc. PowerEdge R720xd/0H5J4J
RIP: 0010:[<ffffffffa03194f0>]  [<ffffffffa03194f0>] rfs_open+0x50/0x400 [redirfs]

Resolution

  • Kernel panicked in "rfs_open" function of redirfs(U) module and it is shipped by third party vendor Kaspersky Lab.

  • Engage the third party module vendor for further details and fix for it.

Diagnostic Steps

  • Check system information:
        CPUS: 48
        DATE: Tue Nov 17 17:19:58 2015
      UPTIME: 87 days, 16:37:08
LOAD AVERAGE: 10.59, 6.04, 3.39
       TASKS: 1060
    NODENAME: hostname
     RELEASE: 2.6.32-431.el6.x86_64
     VERSION: #1 SMP Sun Nov 10 22:19:54 EST 2013
     MACHINE: x86_64  (2699 Mhz)
      MEMORY: 256 GB
       PANIC: "BUG: unable to handle kernel NULL pointer dereference at 0000000000000030"
         PID: 28496
     COMMAND: "vertica"
        TASK: ffff88404f3daae0  [THREAD_INFO: ffff88269dc2c000]
         CPU: 44
       STATE: TASK_RUNNING (PANIC)
  • Kernel Ring Buffer:
crash> log
...
INFO: task klnagent:6835 blocked for more than 120 seconds.
      Not tainted 2.6.32-431.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
klnagent      D 0000000000000019     0  6835  32174 0x00020080
 ffff883b90b59c78 0000000000000082 0000000000000000 0000000000000000
 ffff882984ae4ae0 00000000812250d0 ffffffffa00c68e0 0000000300000001
 ffff882984ae5098 ffff883b90b59fd8 000000000000fbc8 ffff882984ae5098
Call Trace:
 [<ffffffffa0448e4a>] Monitor_queue_add+0x14a/0x1b0 [kav4fs_oas]
 [<ffffffff81065df0>] ? default_wake_function+0x0/0x20
 [<ffffffffa04483ad>] ? Monitor_kernel_trusted_proc+0x1d/0xb0 [kav4fs_oas]
 [<ffffffffa04482e7>] Monitor_kernel_check_file+0x97/0x140 [kav4fs_oas]
 [<ffffffffa044773d>] kavoas_after_close+0x2dd/0x300 [kav4fs_oas]
 [<ffffffffa044a844>] klflt_release+0x24/0x40 [kav4fs_oas]
 [<ffffffffa031ed60>] rfs_postcall_flts+0x60/0x186 [redirfs]
 [<ffffffffa0319344>] rfs_release+0x114/0x270 [redirfs]
 [<ffffffff8109b2a0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff8118a725>] __fput+0xf5/0x210
 [<ffffffff8118a865>] fput+0x25/0x30
 [<ffffffff81185b8d>] filp_close+0x5d/0x90
 [<ffffffff81185c65>] sys_close+0xa5/0x100
 [<ffffffff81050c20>] sysenter_dispatch+0x7/0x2e
INFO: task klnagent:20017 blocked for more than 120 seconds.
      Not tainted 2.6.32-431.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
klnagent      D 0000000000000019     0 20017  32174 0x00020080
 ffff88213c475c78 0000000000000082 0000000000000000 0000000000000012
 ffff8822eed8d500 00000000812250d0 ffffffffa00c68e0 0000000300000001
 ffff8822eed8dab8 ffff88213c475fd8 000000000000fbc8 ffff8822eed8dab8
Call Trace:
 [<ffffffffa0448e4a>] Monitor_queue_add+0x14a/0x1b0 [kav4fs_oas]
 [<ffffffff81065df0>] ? default_wake_function+0x0/0x20
 [<ffffffffa04483ad>] ? Monitor_kernel_trusted_proc+0x1d/0xb0 [kav4fs_oas]
 [<ffffffffa04482e7>] Monitor_kernel_check_file+0x97/0x140 [kav4fs_oas]
 [<ffffffffa044773d>] kavoas_after_close+0x2dd/0x300 [kav4fs_oas]
 [<ffffffffa044a844>] klflt_release+0x24/0x40 [kav4fs_oas]
 [<ffffffffa031ed60>] rfs_postcall_flts+0x60/0x186 [redirfs]
 [<ffffffffa0319344>] rfs_release+0x114/0x270 [redirfs]
 [<ffffffff8109b2a0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff8118a725>] __fput+0xf5/0x210
 [<ffffffff8118a865>] fput+0x25/0x30
 [<ffffffff81185b8d>] filp_close+0x5d/0x90
 [<ffffffff81185c65>] sys_close+0xa5/0x100
 [<ffffffff81050c20>] sysenter_dispatch+0x7/0x2e
cdcwatch[4407]: segfault at 0 ip 00007f3cfb019296 sp 00007f3cfb00c430 error 4 in libgcc_s-3.2.3-20
BUG: unable to handle kernel NULL pointer dereference at 0000000000000030
IP: [<ffffffffa03194f0>] rfs_open+0x50/0x400 [redirfs]  <===== Here kernel panicked
PGD 401020d067 PUD 3b90b43067 PMD 0 
Oops: 0000 [#1] SMP 
last sysfs file: /sys/devices/system/cpu/online
CPU 44 
Modules linked in: kav4fs_oas(U) nfs lockd fscache auth_rpcgss nfs_acl sunrpc vfat fat usb_storage redirfs(U) mpt3sas mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf dell_rbu bonding 8021q garp stp llc ipv6 aesni_intel ablk_helper cryptd lrw glue_helper aes_x86_64 aes_generic xts gf128mul dm_crypt iTCO_wdt iTCO_vendor_support microcode dcdbas power_meter sg ixgbe mdio shpchp igb dca i2c_algo_bit i2c_core ptp pps_core lpc_ich mfd_core ext4 jbd2 mbcache sd_mod crc_t10dif wmi megaraid_sas dm_mirror dm_region_hash dm_log dm_mod [last unloaded: kav4fs_oas]

Pid: 28496, comm: vertica Not tainted 2.6.32-431.el6.x86_64 #1 Dell Inc. PowerEdge R720xd/0H5J4J
RIP: 0010:[<ffffffffa03194f0>]  [<ffffffffa03194f0>] rfs_open+0x50/0x400 [redirfs]
RSP: 0018:ffff88269dc2dc98  EFLAGS: 00010246
RAX: ffffffffa03204e0 RBX: ffff8840296f20c0 RCX: 0000000000000000
RDX: ffff88084fd91540 RSI: ffff8820517ef6c0 RDI: 0000000000000000
RBP: ffff88269dc2dd28 R08: ffff884016430d80 R09: ffff882da9b5ce40
R10: 0000000000000000 R11: ffffffffa00d2fc8 R12: ffff8820517ef6c0
R13: ffff882da9b5ce40 R14: 0000000000000000 R15: ffff8840296f20c0
FS:  00007f02a7fff700(0000) GS:ffff88011cac0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000030 CR3: 0000003d51d59000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process vertica (pid: 28496, threadinfo ffff88269dc2c000, task ffff88404f3daae0)
Stack:
 000001011f054000 ffff88054e076000 ffff88269dc2dcb8 ffff882e0842ec60
<d> ffff88269dc2dcf8 ffffffff810deb63 0000000000000000 0000000000000000
<d> ffff882600000000 ffffffff811aaa20 ffff882e0842ec60 ffff8820517ef6c0
Call Trace:
 [<ffffffff810deb63>] ? audit_copy_inode+0x83/0xc0
 [<ffffffff811aaa20>] ? mntput_no_expire+0x30/0x110
 [<ffffffffa03194a0>] ? rfs_open+0x0/0x400 [redirfs]
 [<ffffffff81185f7a>] __dentry_open+0x10a/0x360
 [<ffffffff8122752f>] ? security_inode_permission+0x1f/0x30
 [<ffffffff811862e4>] nameidata_to_filp+0x54/0x70
 [<ffffffff8119bb20>] do_filp_open+0x6d0/0xd20
 [<ffffffff810df8e5>] ? audit_filter_rules+0x3f5/0xdd0
 [<ffffffff810dea9a>] ? audit_alloc_name+0x6a/0xb0
 [<ffffffff811a8212>] ? alloc_fd+0x92/0x160
 [<ffffffff81185d29>] do_sys_open+0x69/0x140
 [<ffffffff81185e40>] sys_open+0x20/0x30
 [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
Code: 00 00 00 48 85 c0 74 0e 48 81 78 40 80 c7 31 a0 0f 84 d5 00 00 00 45 31 f6 49 8b 44 24 20 48 85 c0 74 08 48 8b 38 e8 d0 fa d9 e0 <49> 8b 46 30 48 85 c0 74 54 4c 8b 28 4d 85 ed 74 4c 65 8b 14 25 
RIP  [<ffffffffa03194f0>] rfs_open+0x50/0x400 [redirfs]
 RSP <ffff88269dc2dc98>
CR2: 0000000000000030
  • Backtrace of the panic task:
crash> bt
PID: 28496  TASK: ffff88404f3daae0  CPU: 44  COMMAND: "vertica"
 #0 [ffff88269dc2d860] machine_kexec at ffffffff81038f3b
 #1 [ffff88269dc2d8c0] crash_kexec at ffffffff810c5d92
 #2 [ffff88269dc2d990] oops_end at ffffffff8152b510
 #3 [ffff88269dc2d9c0] no_context at ffffffff8104a00b
 #4 [ffff88269dc2da10] __bad_area_nosemaphore at ffffffff8104a295
 #5 [ffff88269dc2da60] bad_area at ffffffff8104a3be
 #6 [ffff88269dc2da90] __do_page_fault at ffffffff8104ab6f
 #7 [ffff88269dc2dbb0] do_page_fault at ffffffff8152d45e
 #8 [ffff88269dc2dbe0] page_fault at ffffffff8152a815
    [exception RIP: rfs_open+80]
    RIP: ffffffffa03194f0  RSP: ffff88269dc2dc98  RFLAGS: 00010246
    RAX: ffffffffa03204e0  RBX: ffff8840296f20c0  RCX: 0000000000000000
    RDX: ffff88084fd91540  RSI: ffff8820517ef6c0  RDI: 0000000000000000
    RBP: ffff88269dc2dd28   R8: ffff884016430d80   R9: ffff882da9b5ce40
    R10: 0000000000000000  R11: ffffffffa00d2fc8  R12: ffff8820517ef6c0
    R13: ffff882da9b5ce40  R14: 0000000000000000  R15: ffff8840296f20c0
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 #9 [ffff88269dc2dd30] __dentry_open at ffffffff81185f7a
#10 [ffff88269dc2dd90] nameidata_to_filp at ffffffff811862e4
#11 [ffff88269dc2ddb0] do_filp_open at ffffffff8119bb20
#12 [ffff88269dc2df20] do_sys_open at ffffffff81185d29
#13 [ffff88269dc2df70] sys_open at ffffffff81185e40
#14 [ffff88269dc2df80] system_call_fastpath at ffffffff8100b072
    RIP: 0000003a1c6db2ed  RSP: 00007f027a37eff8  RFLAGS: 00010246
    RAX: 0000000000000002  RBX: ffffffff8100b072  RCX: 0000000000000000
    RDX: 0000000000000000  RSI: 0000000000040000  RDI: 0000000019bc9ac8
    RBP: 00007f027a37f880   R8: 0000000000000000   R9: 0000000000000000
    R10: 0000000000000010  R11: 0000000000000293  R12: ffffffff81185e40
    R13: ffff88269dc2df78  R14: 0000000019bc9ac8  R15: 0000000000000000
    ORIG_RAX: 0000000000000002  CS: 0033  SS: 002b
  • Tainted modules loaded:
crash> mod -t
NAME        TAINTS
redirfs     (U)  <==
kav4fs_oas  (U)

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.