Kernel panics with the message "BUG: unable to handle kernel NULL pointer dereference at 0000000000000010" caused by a third party module 'appassure_vss'
Environment
- Red Hat Enterprise Linux 6
- Installed third-party modules : appassure_vss
Issue
- Server crash has occurred generating a vmcore where following logs were observed in the kernel ring buffer :
BUG: unable to handle kernel NULL pointer dereference at 0000000000000010
IP: [<ffffffffa0107149>] zcc9f3027ef+0x59/0xd0 [appassure_vss]
PGD 4c8ecf067 PUD 0
Oops: 0000 [#1] SMP
last sysfs file: /sys/devices/system/cpu/online
CPU 6
Modules linked in: lp nbd(U) autofs4 dm_switch(U) dm_queue_length dm_multipath be2iscsi iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi vsock(U) ppdev parport_pc parport microcode vmware_balloon vmxnet3 sg i2c_piix4 i2c_core vmci(U) shpchp appassure_vss(P)(U) ext4 jbd2 mbcache sd_mod crc_t10dif sr_mod cdrom vmw_pvscsi pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib]
Pid: 28657, comm: flush-253:8 Tainted: P --------------- 2.6.32-431.11.2.el6.x86_64 #1 VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform
RIP: 0010:[<ffffffffa0107149>] [<ffffffffa0107149>] zcc9f3027ef+0x59/0xd0 [appassure_vss]
RSP: 0018:ffff8801511f7430 EFLAGS: 00010246
RAX: 000000000671ed88 RBX: ffff88102562ca40 RCX: ffff88083c5728b8
RDX: 0000000000000008 RSI: 000000000671ed88 RDI: ffff88028fef7588
RBP: ffff8801511f7430 R08: 0000000000000000 R09: 000000000025ad88
R10: ffff88083c5728b0 R11: 0000000000000020 R12: ffff88083c5728b0
R13: ffff88102562ca50 R14: ffff880c440b2e40 R15: ffff8806b97fc0c0
FS: 0000000000000000(0000) GS:ffff880028380000(0000) knlGS:0000000000000000
CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 0000000000000010 CR3: 00000006bd485000 CR4: 00000000000407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process flush-253:8 (pid: 28657, threadinfo ffff8801511f6000, task ffff8801cfdd8040)
Stack:
ffff8801511f7460 ffffffffa010b177 00000000199631e8 ffff881027d9a3c0
<d> 00000000199631e8 0000000000000000 ffff8801511f74f0 ffffffffa010930b
<d> ffff880c440b2e40 ffff881027d9a3c4 ffff880c440b2e40 ffffffff81277bea
Call Trace:
[<ffffffffa010b177>] zdc26df746c+0x107/0x120 [appassure_vss]
[<ffffffffa010930b>] z446a9fb23a+0x11b/0x350 [appassure_vss]
[<ffffffff81277bea>] ? blk_throtl_bio+0x1ea/0x5f0
[<ffffffffa0107a8f>] z3a18f09fb9+0x6f/0xc0 [appassure_vss]
[<ffffffff81266e80>] generic_make_request+0x240/0x5a0
[<ffffffff811220d5>] ? mempool_alloc_slab+0x15/0x20
[<ffffffff81122273>] ? mempool_alloc+0x63/0x140
[<ffffffffa00b3e28>] ? __ext4_journal_stop+0x68/0xa0 [ext4]
[<ffffffff81267250>] submit_bio+0x70/0x120
[<ffffffff811bdddd>] submit_bh+0x11d/0x1f0
[<ffffffff811c0598>] __block_write_full_page+0x1c8/0x330
[<ffffffff811bf560>] ? end_buffer_async_write+0x0/0x190
[<ffffffffa009e300>] ? noalloc_get_block_write+0x0/0x60 [ext4]
[<ffffffffa009e300>] ? noalloc_get_block_write+0x0/0x60 [ext4]
[<ffffffff811c07e0>] block_write_full_page_endio+0xe0/0x120
[<ffffffffa0099bf0>] ? ext4_bh_delay_or_unwritten+0x0/0x30 [ext4]
[<ffffffff811c0835>] block_write_full_page+0x15/0x20
[<ffffffffa009fb32>] ext4_writepage+0x172/0x400 [ext4]
[<ffffffffa009ff07>] mpage_da_submit_io+0x147/0x1d0 [ext4]
[<ffffffffa00a235e>] mpage_da_map_and_submit+0x17e/0x470 [ext4]
[<ffffffff812884c5>] ? radix_tree_gang_lookup_tag_slot+0x95/0xe0
[<ffffffff8111f3d0>] ? find_get_pages_tag+0x40/0x130
[<ffffffffa00a26bd>] mpage_add_bh_to_extent+0x6d/0xf0 [ext4]
[<ffffffffa00a2a0f>] write_cache_pages_da+0x2cf/0x470 [ext4]
[<ffffffffa00a2e82>] ext4_da_writepages+0x2d2/0x620 [ext4]
[<ffffffff81134ca1>] do_writepages+0x21/0x40
[<ffffffff811b50cd>] writeback_single_inode+0xdd/0x290
[<ffffffff811b54cd>] writeback_sb_inodes+0xbd/0x170
[<ffffffff811b562b>] writeback_inodes_wb+0xab/0x1b0
[<ffffffff811b5a23>] wb_writeback+0x2f3/0x410
[<ffffffff81527f30>] ? thread_return+0x4e/0x76e
[<ffffffff81084d92>] ? del_timer_sync+0x22/0x30
[<ffffffff811b5ce5>] wb_do_writeback+0x1a5/0x240
[<ffffffff811b5de3>] bdi_writeback_task+0x63/0x1b0
[<ffffffff8109b117>] ? bit_waitqueue+0x17/0xd0
[<ffffffff811439a0>] ? bdi_start_fn+0x0/0x100
[<ffffffff81143a26>] bdi_start_fn+0x86/0x100
[<ffffffff811439a0>] ? bdi_start_fn+0x0/0x100
[<ffffffff8109aee6>] kthread+0x96/0xa0
[<ffffffff8100c20a>] child_rip+0xa/0x20
[<ffffffff8109ae50>] ? kthread+0x0/0xa0
[<ffffffff8100c200>] ? child_rip+0x0/0x20
Code: 48 39 c6 72 7d 4d 8b 40 10 4d 85 c0 74 1d 45 8b 48 08 45 85 c9 74 14 48 39 c6 73 de 4d 8b 40 10 45 89 c9 4c 01 c8 4d 85 c0 75 e3 <49> 8b 70 10 48 85 f6 74 33 44 8b 46 08 45 85 c0 74 2a 44 39 c2
RIP [<ffffffffa0107149>] zcc9f3027ef+0x59/0xd0 [appassure_vss]
RSP <ffff8801511f7430>
CR2: 0000000000000010
Resolution
- Red Hat does not have the source code of 'appassure_vss' and it in not shipped by us. As a result of this, Red Hat has no visibility into how it operates. So it is advised to contact the vendor of the third-party kernel modules for further investigation and troubleshooting of the issue.
Root Cause
- In this case, the panic occurred with the message "BUG: unable to handle kernel NULL pointer dereference at 0000000000000010". After inspecting the vmcore, it can be seen that the crash has occurred in a third party kernel module 'appassure_vss'.
Diagnostic Steps
- The analysis of the vmcore indicates the panic has occurred in a function called 'zcc9f3027ef' of a third party module 'appassure_vss'.
The Backtraces of the task running at the time panic
crash> bt
PID: 28657 TASK: ffff8801cfdd8040 CPU: 6 COMMAND: "flush-253:8"
#0 [ffff8801511f7020] machine_kexec at ffffffff81038f3b
#1 [ffff8801511f7080] crash_kexec at ffffffff810c5d82
#2 [ffff8801511f7150] oops_end at ffffffff8152bb30
#3 [ffff8801511f7180] no_context at ffffffff8104a00b
#4 [ffff8801511f71d0] __bad_area_nosemaphore at ffffffff8104a295
#5 [ffff8801511f7220] bad_area_nosemaphore at ffffffff8104a363
#6 [ffff8801511f7230] __do_page_fault at ffffffff8104aabf
#7 [ffff8801511f7350] do_page_fault at ffffffff8152da7e
#8 [ffff8801511f7380] page_fault at ffffffff8152ae35
[exception RIP: zcc9f3027ef+89]
RIP: ffffffffa0107149 RSP: ffff8801511f7430 RFLAGS: 00010246
RAX: 000000000671ed88 RBX: ffff88102562ca40 RCX: ffff88083c5728b8
RDX: 0000000000000008 RSI: 000000000671ed88 RDI: ffff88028fef7588
RBP: ffff8801511f7430 R8: 0000000000000000 R9: 000000000025ad88
R10: ffff88083c5728b0 R11: 0000000000000020 R12: ffff88083c5728b0
R13: ffff88102562ca50 R14: ffff880c440b2e40 R15: ffff8806b97fc0c0
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#9 [ffff8801511f7438] zdc26df746c at ffffffffa010b177 [appassure_vss]
#10 [ffff8801511f7468] z446a9fb23a at ffffffffa010930b [appassure_vss]
#11 [ffff8801511f74f8] z3a18f09fb9 at ffffffffa0107a8f [appassure_vss]
#12 [ffff8801511f7518] generic_make_request at ffffffff81266e80
#13 [ffff8801511f75f8] submit_bio at ffffffff81267250
#14 [ffff8801511f7648] submit_bh at ffffffff811bdddd
#15 [ffff8801511f7678] __block_write_full_page at ffffffff811c0598
#16 [ffff8801511f76f8] block_write_full_page_endio at ffffffff811c07e0
#17 [ffff8801511f7748] block_write_full_page at ffffffff811c0835
#18 [ffff8801511f7758] ext4_writepage at ffffffffa009fb32 [ext4]
#19 [ffff8801511f77a8] mpage_da_submit_io at ffffffffa009ff07 [ext4]
#20 [ffff8801511f7898] mpage_da_map_and_submit at ffffffffa00a235e [ext4]
#21 [ffff8801511f7978] mpage_add_bh_to_extent at ffffffffa00a26bd [ext4]
#22 [ffff8801511f79a8] write_cache_pages_da at ffffffffa00a2a0f [ext4]
#23 [ffff8801511f7ac8] ext4_da_writepages at ffffffffa00a2e82 [ext4]
#24 [ffff8801511f7bc8] do_writepages at ffffffff81134ca1
#25 [ffff8801511f7bd8] writeback_single_inode at ffffffff811b50cd
#26 [ffff8801511f7c18] writeback_sb_inodes at ffffffff811b54cd
#27 [ffff8801511f7c78] writeback_inodes_wb at ffffffff811b562b
#28 [ffff8801511f7cd8] wb_writeback at ffffffff811b5a23
#29 [ffff8801511f7dd8] wb_do_writeback at ffffffff811b5ce5
#30 [ffff8801511f7e68] bdi_writeback_task at ffffffff811b5de3
#31 [ffff8801511f7eb8] bdi_start_fn at ffffffff81143a26
#32 [ffff8801511f7ee8] kthread at ffffffff8109aee6
#33 [ffff8801511f7f48] kernel_thread at ffffffff8100c20a
The kernel ring buffer logs are as given below
crash> log
BUG: unable to handle kernel NULL pointer dereference at 0000000000000010
IP: [<ffffffffa0107149>] zcc9f3027ef+0x59/0xd0 [appassure_vss] >>>>>>>>>>
PGD 4c8ecf067 PUD 0
Oops: 0000 [#1] SMP
last sysfs file: /sys/devices/system/cpu/online
CPU 6
Modules linked in: lp nbd(U) autofs4 dm_switch(U) dm_queue_length dm_multipath be2iscsi iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi vsock(U) ppdev parport_pc parport microcode vmware_balloon vmxnet3 sg i2c_piix4 i2c_core vmci(U) shpchp appassure_vss(P)(U) ext4 jbd2 mbcache sd_mod crc_t10dif sr_mod cdrom vmw_pvscsi pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib]
Pid: 28657, comm: flush-253:8 Tainted: P --------------- 2.6.32-431.11.2.el6.x86_64 #1 VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform
RIP: 0010:[<ffffffffa0107149>] [<ffffffffa0107149>] zcc9f3027ef+0x59/0xd0 [appassure_vss]
RSP: 0018:ffff8801511f7430 EFLAGS: 00010246
RAX: 000000000671ed88 RBX: ffff88102562ca40 RCX: ffff88083c5728b8
RDX: 0000000000000008 RSI: 000000000671ed88 RDI: ffff88028fef7588
RBP: ffff8801511f7430 R08: 0000000000000000 R09: 000000000025ad88
R10: ffff88083c5728b0 R11: 0000000000000020 R12: ffff88083c5728b0
R13: ffff88102562ca50 R14: ffff880c440b2e40 R15: ffff8806b97fc0c0
FS: 0000000000000000(0000) GS:ffff880028380000(0000) knlGS:0000000000000000
CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 0000000000000010 CR3: 00000006bd485000 CR4: 00000000000407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process flush-253:8 (pid: 28657, threadinfo ffff8801511f6000, task ffff8801cfdd8040)
Stack:
ffff8801511f7460 ffffffffa010b177 00000000199631e8 ffff881027d9a3c0
<d> 00000000199631e8 0000000000000000 ffff8801511f74f0 ffffffffa010930b
<d> ffff880c440b2e40 ffff881027d9a3c4 ffff880c440b2e40 ffffffff81277bea
Call Trace:
[<ffffffffa010b177>] zdc26df746c+0x107/0x120 [appassure_vss]
[<ffffffffa010930b>] z446a9fb23a+0x11b/0x350 [appassure_vss]
[<ffffffff81277bea>] ? blk_throtl_bio+0x1ea/0x5f0
[<ffffffffa0107a8f>] z3a18f09fb9+0x6f/0xc0 [appassure_vss]
[<ffffffff81266e80>] generic_make_request+0x240/0x5a0
[<ffffffff811220d5>] ? mempool_alloc_slab+0x15/0x20
[<ffffffff81122273>] ? mempool_alloc+0x63/0x140
[<ffffffffa00b3e28>] ? __ext4_journal_stop+0x68/0xa0 [ext4]
[<ffffffff81267250>] submit_bio+0x70/0x120
[<ffffffff811bdddd>] submit_bh+0x11d/0x1f0
[<ffffffff811c0598>] __block_write_full_page+0x1c8/0x330
[<ffffffff811bf560>] ? end_buffer_async_write+0x0/0x190
[<ffffffffa009e300>] ? noalloc_get_block_write+0x0/0x60 [ext4]
[<ffffffffa009e300>] ? noalloc_get_block_write+0x0/0x60 [ext4]
[<ffffffff811c07e0>] block_write_full_page_endio+0xe0/0x120
[<ffffffffa0099bf0>] ? ext4_bh_delay_or_unwritten+0x0/0x30 [ext4]
[<ffffffff811c0835>] block_write_full_page+0x15/0x20
[<ffffffffa009fb32>] ext4_writepage+0x172/0x400 [ext4]
[<ffffffffa009ff07>] mpage_da_submit_io+0x147/0x1d0 [ext4]
[<ffffffffa00a235e>] mpage_da_map_and_submit+0x17e/0x470 [ext4]
[<ffffffff812884c5>] ? radix_tree_gang_lookup_tag_slot+0x95/0xe0
[<ffffffff8111f3d0>] ? find_get_pages_tag+0x40/0x130
[<ffffffffa00a26bd>] mpage_add_bh_to_extent+0x6d/0xf0 [ext4]
[<ffffffffa00a2a0f>] write_cache_pages_da+0x2cf/0x470 [ext4]
[<ffffffffa00a2e82>] ext4_da_writepages+0x2d2/0x620 [ext4]
[<ffffffff81134ca1>] do_writepages+0x21/0x40
[<ffffffff811b50cd>] writeback_single_inode+0xdd/0x290
[<ffffffff811b54cd>] writeback_sb_inodes+0xbd/0x170
[<ffffffff811b562b>] writeback_inodes_wb+0xab/0x1b0
[<ffffffff811b5a23>] wb_writeback+0x2f3/0x410
[<ffffffff81527f30>] ? thread_return+0x4e/0x76e
[<ffffffff81084d92>] ? del_timer_sync+0x22/0x30
[<ffffffff811b5ce5>] wb_do_writeback+0x1a5/0x240
[<ffffffff811b5de3>] bdi_writeback_task+0x63/0x1b0
[<ffffffff8109b117>] ? bit_waitqueue+0x17/0xd0
[<ffffffff811439a0>] ? bdi_start_fn+0x0/0x100
[<ffffffff81143a26>] bdi_start_fn+0x86/0x100
[<ffffffff811439a0>] ? bdi_start_fn+0x0/0x100
[<ffffffff8109aee6>] kthread+0x96/0xa0
[<ffffffff8100c20a>] child_rip+0xa/0x20
[<ffffffff8109ae50>] ? kthread+0x0/0xa0
[<ffffffff8100c200>] ? child_rip+0x0/0x20
Code: 48 39 c6 72 7d 4d 8b 40 10 4d 85 c0 74 1d 45 8b 48 08 45 85 c9 74 14 48 39 c6 73 de 4d 8b 40 10 45 89 c9 4c 01 c8 4d 85 c0 75 e3 <49> 8b 70 10 48 85 f6 74 33 44 8b 46 08 45 85 c0 74 2a 44 39 c2
RIP [<ffffffffa0107149>] zcc9f3027ef+0x59/0xd0 [appassure_vss]
RSP <ffff8801511f7430>
CR2: 0000000000000010
List of proprietary (P) unsigned (U) module is as follows :
crash> mod -t | grep U
nbd (U)
appassure_vss P(U) ------>>>>>>
vmci (U)
vsock (U)
dm_switch (U)
Additional details of the proprietary module 'appassure_vss'
crash> mod | grep -e NAME -e appassure_vss
MODULE NAME SIZE OBJECT FILE
ffffffffa010d560 appassure_vss 30601 (not loaded) [CONFIG_KALLSYMS]
crash> module.name,version,srcversion,gpgsig_ok ffffffffa010d560
name = "appassure_vss\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
version = 0x0
srcversion = 0xffff881023295d40 "35978333020E0AA10A152D9"
gpgsig_ok = 0
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
