Kernel panicked in "rfs_open" function of redirfs(U) module.
Environment
-
Red Hat Enterprise Linux 6
-
redirfs Unsigned (U) module
Issue
- Kernel panicked in "rfs_open" function of redirfs(U) module.
BUG: unable to handle kernel NULL pointer dereference at 0000000000000030
IP: [<ffffffffa03194f0>] rfs_open+0x50/0x400 [redirfs]
...
Pid: 28496, comm: vertica Not tainted 2.6.32-431.el6.x86_64 #1 Dell Inc. PowerEdge R720xd/0H5J4J
RIP: 0010:[<ffffffffa03194f0>] [<ffffffffa03194f0>] rfs_open+0x50/0x400 [redirfs]
Resolution
-
Kernel panicked in "rfs_open" function of redirfs(U) module and it is shipped by third party vendor Kaspersky Lab.
-
Engage the third party module vendor for further details and fix for it.
Diagnostic Steps
- Check system information:
CPUS: 48
DATE: Tue Nov 17 17:19:58 2015
UPTIME: 87 days, 16:37:08
LOAD AVERAGE: 10.59, 6.04, 3.39
TASKS: 1060
NODENAME: hostname
RELEASE: 2.6.32-431.el6.x86_64
VERSION: #1 SMP Sun Nov 10 22:19:54 EST 2013
MACHINE: x86_64 (2699 Mhz)
MEMORY: 256 GB
PANIC: "BUG: unable to handle kernel NULL pointer dereference at 0000000000000030"
PID: 28496
COMMAND: "vertica"
TASK: ffff88404f3daae0 [THREAD_INFO: ffff88269dc2c000]
CPU: 44
STATE: TASK_RUNNING (PANIC)
- Kernel Ring Buffer:
crash> log
...
INFO: task klnagent:6835 blocked for more than 120 seconds.
Not tainted 2.6.32-431.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
klnagent D 0000000000000019 0 6835 32174 0x00020080
ffff883b90b59c78 0000000000000082 0000000000000000 0000000000000000
ffff882984ae4ae0 00000000812250d0 ffffffffa00c68e0 0000000300000001
ffff882984ae5098 ffff883b90b59fd8 000000000000fbc8 ffff882984ae5098
Call Trace:
[<ffffffffa0448e4a>] Monitor_queue_add+0x14a/0x1b0 [kav4fs_oas]
[<ffffffff81065df0>] ? default_wake_function+0x0/0x20
[<ffffffffa04483ad>] ? Monitor_kernel_trusted_proc+0x1d/0xb0 [kav4fs_oas]
[<ffffffffa04482e7>] Monitor_kernel_check_file+0x97/0x140 [kav4fs_oas]
[<ffffffffa044773d>] kavoas_after_close+0x2dd/0x300 [kav4fs_oas]
[<ffffffffa044a844>] klflt_release+0x24/0x40 [kav4fs_oas]
[<ffffffffa031ed60>] rfs_postcall_flts+0x60/0x186 [redirfs]
[<ffffffffa0319344>] rfs_release+0x114/0x270 [redirfs]
[<ffffffff8109b2a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff8118a725>] __fput+0xf5/0x210
[<ffffffff8118a865>] fput+0x25/0x30
[<ffffffff81185b8d>] filp_close+0x5d/0x90
[<ffffffff81185c65>] sys_close+0xa5/0x100
[<ffffffff81050c20>] sysenter_dispatch+0x7/0x2e
INFO: task klnagent:20017 blocked for more than 120 seconds.
Not tainted 2.6.32-431.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
klnagent D 0000000000000019 0 20017 32174 0x00020080
ffff88213c475c78 0000000000000082 0000000000000000 0000000000000012
ffff8822eed8d500 00000000812250d0 ffffffffa00c68e0 0000000300000001
ffff8822eed8dab8 ffff88213c475fd8 000000000000fbc8 ffff8822eed8dab8
Call Trace:
[<ffffffffa0448e4a>] Monitor_queue_add+0x14a/0x1b0 [kav4fs_oas]
[<ffffffff81065df0>] ? default_wake_function+0x0/0x20
[<ffffffffa04483ad>] ? Monitor_kernel_trusted_proc+0x1d/0xb0 [kav4fs_oas]
[<ffffffffa04482e7>] Monitor_kernel_check_file+0x97/0x140 [kav4fs_oas]
[<ffffffffa044773d>] kavoas_after_close+0x2dd/0x300 [kav4fs_oas]
[<ffffffffa044a844>] klflt_release+0x24/0x40 [kav4fs_oas]
[<ffffffffa031ed60>] rfs_postcall_flts+0x60/0x186 [redirfs]
[<ffffffffa0319344>] rfs_release+0x114/0x270 [redirfs]
[<ffffffff8109b2a0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff8118a725>] __fput+0xf5/0x210
[<ffffffff8118a865>] fput+0x25/0x30
[<ffffffff81185b8d>] filp_close+0x5d/0x90
[<ffffffff81185c65>] sys_close+0xa5/0x100
[<ffffffff81050c20>] sysenter_dispatch+0x7/0x2e
cdcwatch[4407]: segfault at 0 ip 00007f3cfb019296 sp 00007f3cfb00c430 error 4 in libgcc_s-3.2.3-20
BUG: unable to handle kernel NULL pointer dereference at 0000000000000030
IP: [<ffffffffa03194f0>] rfs_open+0x50/0x400 [redirfs] <===== Here kernel panicked
PGD 401020d067 PUD 3b90b43067 PMD 0
Oops: 0000 [#1] SMP
last sysfs file: /sys/devices/system/cpu/online
CPU 44
Modules linked in: kav4fs_oas(U) nfs lockd fscache auth_rpcgss nfs_acl sunrpc vfat fat usb_storage redirfs(U) mpt3sas mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf dell_rbu bonding 8021q garp stp llc ipv6 aesni_intel ablk_helper cryptd lrw glue_helper aes_x86_64 aes_generic xts gf128mul dm_crypt iTCO_wdt iTCO_vendor_support microcode dcdbas power_meter sg ixgbe mdio shpchp igb dca i2c_algo_bit i2c_core ptp pps_core lpc_ich mfd_core ext4 jbd2 mbcache sd_mod crc_t10dif wmi megaraid_sas dm_mirror dm_region_hash dm_log dm_mod [last unloaded: kav4fs_oas]
Pid: 28496, comm: vertica Not tainted 2.6.32-431.el6.x86_64 #1 Dell Inc. PowerEdge R720xd/0H5J4J
RIP: 0010:[<ffffffffa03194f0>] [<ffffffffa03194f0>] rfs_open+0x50/0x400 [redirfs]
RSP: 0018:ffff88269dc2dc98 EFLAGS: 00010246
RAX: ffffffffa03204e0 RBX: ffff8840296f20c0 RCX: 0000000000000000
RDX: ffff88084fd91540 RSI: ffff8820517ef6c0 RDI: 0000000000000000
RBP: ffff88269dc2dd28 R08: ffff884016430d80 R09: ffff882da9b5ce40
R10: 0000000000000000 R11: ffffffffa00d2fc8 R12: ffff8820517ef6c0
R13: ffff882da9b5ce40 R14: 0000000000000000 R15: ffff8840296f20c0
FS: 00007f02a7fff700(0000) GS:ffff88011cac0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000030 CR3: 0000003d51d59000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process vertica (pid: 28496, threadinfo ffff88269dc2c000, task ffff88404f3daae0)
Stack:
000001011f054000 ffff88054e076000 ffff88269dc2dcb8 ffff882e0842ec60
<d> ffff88269dc2dcf8 ffffffff810deb63 0000000000000000 0000000000000000
<d> ffff882600000000 ffffffff811aaa20 ffff882e0842ec60 ffff8820517ef6c0
Call Trace:
[<ffffffff810deb63>] ? audit_copy_inode+0x83/0xc0
[<ffffffff811aaa20>] ? mntput_no_expire+0x30/0x110
[<ffffffffa03194a0>] ? rfs_open+0x0/0x400 [redirfs]
[<ffffffff81185f7a>] __dentry_open+0x10a/0x360
[<ffffffff8122752f>] ? security_inode_permission+0x1f/0x30
[<ffffffff811862e4>] nameidata_to_filp+0x54/0x70
[<ffffffff8119bb20>] do_filp_open+0x6d0/0xd20
[<ffffffff810df8e5>] ? audit_filter_rules+0x3f5/0xdd0
[<ffffffff810dea9a>] ? audit_alloc_name+0x6a/0xb0
[<ffffffff811a8212>] ? alloc_fd+0x92/0x160
[<ffffffff81185d29>] do_sys_open+0x69/0x140
[<ffffffff81185e40>] sys_open+0x20/0x30
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
Code: 00 00 00 48 85 c0 74 0e 48 81 78 40 80 c7 31 a0 0f 84 d5 00 00 00 45 31 f6 49 8b 44 24 20 48 85 c0 74 08 48 8b 38 e8 d0 fa d9 e0 <49> 8b 46 30 48 85 c0 74 54 4c 8b 28 4d 85 ed 74 4c 65 8b 14 25
RIP [<ffffffffa03194f0>] rfs_open+0x50/0x400 [redirfs]
RSP <ffff88269dc2dc98>
CR2: 0000000000000030
- Backtrace of the panic task:
crash> bt
PID: 28496 TASK: ffff88404f3daae0 CPU: 44 COMMAND: "vertica"
#0 [ffff88269dc2d860] machine_kexec at ffffffff81038f3b
#1 [ffff88269dc2d8c0] crash_kexec at ffffffff810c5d92
#2 [ffff88269dc2d990] oops_end at ffffffff8152b510
#3 [ffff88269dc2d9c0] no_context at ffffffff8104a00b
#4 [ffff88269dc2da10] __bad_area_nosemaphore at ffffffff8104a295
#5 [ffff88269dc2da60] bad_area at ffffffff8104a3be
#6 [ffff88269dc2da90] __do_page_fault at ffffffff8104ab6f
#7 [ffff88269dc2dbb0] do_page_fault at ffffffff8152d45e
#8 [ffff88269dc2dbe0] page_fault at ffffffff8152a815
[exception RIP: rfs_open+80]
RIP: ffffffffa03194f0 RSP: ffff88269dc2dc98 RFLAGS: 00010246
RAX: ffffffffa03204e0 RBX: ffff8840296f20c0 RCX: 0000000000000000
RDX: ffff88084fd91540 RSI: ffff8820517ef6c0 RDI: 0000000000000000
RBP: ffff88269dc2dd28 R8: ffff884016430d80 R9: ffff882da9b5ce40
R10: 0000000000000000 R11: ffffffffa00d2fc8 R12: ffff8820517ef6c0
R13: ffff882da9b5ce40 R14: 0000000000000000 R15: ffff8840296f20c0
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#9 [ffff88269dc2dd30] __dentry_open at ffffffff81185f7a
#10 [ffff88269dc2dd90] nameidata_to_filp at ffffffff811862e4
#11 [ffff88269dc2ddb0] do_filp_open at ffffffff8119bb20
#12 [ffff88269dc2df20] do_sys_open at ffffffff81185d29
#13 [ffff88269dc2df70] sys_open at ffffffff81185e40
#14 [ffff88269dc2df80] system_call_fastpath at ffffffff8100b072
RIP: 0000003a1c6db2ed RSP: 00007f027a37eff8 RFLAGS: 00010246
RAX: 0000000000000002 RBX: ffffffff8100b072 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000040000 RDI: 0000000019bc9ac8
RBP: 00007f027a37f880 R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000000010 R11: 0000000000000293 R12: ffffffff81185e40
R13: ffff88269dc2df78 R14: 0000000019bc9ac8 R15: 0000000000000000
ORIG_RAX: 0000000000000002 CS: 0033 SS: 002b
- Tainted modules loaded:
crash> mod -t
NAME TAINTS
redirfs (U) <==
kav4fs_oas (U)
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
