RHEL 5.8: rsync to ext3 filesystem results in kernel panic with RIP value do_mpage_readpage+0x331/0x4a5

Issue

• rsync crashes in the kernel with the following backtrace in do_mpage_readpage called from ext3 code.

general protection fault: 0000 [1] SMP 
last sysfs file: /devices/pci0000:00/0000:00:00.0/irq
CPU 1 
Modules linked in: nfsd exportfs nfs_acl auth_rpcgss autofs4 hidp rfcomm l2cap bluetooth lockd sunrpc be2iscsi ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp bnx2i cnic uio cxgb3i libcxgbi cxgb3 8021q libiscsi_tcp libiscsi2 scsi_transport_iscsi2 scsi_transport_iscsi dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi acpi_memhotplug ac ipv6 xfrm_nalgo crypto_api lp sg floppy i2c_piix4 i2c_core parport_pc serio_raw pcspkr ide_cd e1000 parport tpm_tis cdrom shpchp tpm tpm_bios dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod ata_piix libata mptspi mptscsih mptbase scsi_transport_spi sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
Pid: 6011, comm: rsync Not tainted 2.6.18-308.el5 #1
RIP: 0010:[<ffffffff8002882c>]  [<ffffffff8002882c>] do_mpage_readpage+0x331/0x4a5
RSP: 0018:ffff8101b8bbdb28  EFLAGS: 00010246
RAX: 0000000000000000 RBX: 810000000083d000 RCX: 0000000000001000
RDX: 0000000000000000 RSI: ffff810011000000 RDI: 4924134c40e97000
RBP: ffff8101b8bbdc18 R08: ffff8101b8bbdc18 R09: ffff8101b8bbdc78
R10: 2d0a7c3332313039 R11: 2d2d2d2d2d2d2d2d R12: ffff81010ae33108
R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000001000
FS:  00002b24ba8fb550(0000) GS:ffff81010ef437c0(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000016e21898 CR3: 0000000177929000 CR4: 00000000000006e0
Process rsync (pid: 6011, threadinfo ffff8101b8bbc000, task ffff810329a5b820)
Stack:  0000000000000000 ffffffff8804ed74 ffff8101b8bbdc78 ffff8101b8bbdc80
 0000000100200200 0000000000000000 ffff8103380360c0 0000000080046c02
 0000000000001000 0000000000000001 0000000000000000 0000000000000001
Call Trace:
 [<ffffffff8804ed74>] :ext3:ext3_get_block+0x0/0xf8
 [<ffffffff800624c6>] __sched_text_start+0xf6/0xbd0
 [<ffffffff800f7a00>] mpage_readpage+0x47/0x5f
 [<ffffffff8804ed74>] :ext3:ext3_get_block+0x0/0xf8
 [<ffffffff8000e437>] current_fs_time+0x3b/0x40
 [<ffffffff8000d31d>] file_read_actor+0xbe/0x159
 [<ffffffff8000769e>] find_get_page+0x21/0x51
 [<ffffffff8000c461>] do_generic_mapping_read+0x20d/0x359
 [<ffffffff8000d25f>] file_read_actor+0x0/0x159
 [<ffffffff8000c6f9>] __generic_file_aio_read+0x14c/0x198
 [<ffffffff80016eeb>] generic_file_aio_read+0x36/0x3b
 [<ffffffff8000cf47>] do_sync_read+0xc7/0x104
 [<ffffffff80067235>] do_page_fault+0x4cc/0x842
 [<ffffffff800a3444>] autoremove_wake_function+0x0/0x2e
 [<ffffffff8000b72f>] vfs_read+0xcb/0x171
 [<ffffffff80011d49>] sys_read+0x45/0x6e
 [<ffffffff8005d28d>] tracesys+0xd5/0xe0


Code: f3 aa 75 1a f0 41 0f ba 2c 24 03 e9 45 01 00 00 83 7c 24 5c 
RIP  [<ffffffff8002882c>] do_mpage_readpage+0x331/0x4a5
 RSP <ffff8101b8bbdb28>

• Re-occorence produced the following vmcore information:

Kernel BUG at lib/list_debug.c:65
invalid opcode: 0000 [1] SMP 
last sysfs file: /devices/pci0000:00/0000:00:00.0/irq
CPU 0 
Modules linked in: nfsd exportfs nfs_acl auth_rpcgss autofs4 hidp rfcomm l2cap bluetooth lockd sunrpc be2iscsi ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp bnx2i cnic uio cxgb3i libcxgbi cxgb3 8021q libiscsi_tcp libiscsi2 scsi_transport_iscsi2 scsi_transport_iscsi vsock(U) vmmemctl(U) acpiphp dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi acpi_memhotplug ac ipv6 xfrm_nalgo crypto_api lp sg floppy i2c_piix4 i2c_core pcspkr serio_raw vmci(U) parport_pc parport e1000 ide_cd tpm_tis tpm cdrom tpm_bios shpchp pvscsi(U) vmxnet3(U) vmxnet(U) dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod ata_piix libata mptspi mptscsih mptbase scsi_transport_spi sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
Pid: 15322, comm: rsync Tainted: G     ---- 2.6.18-308.el5 #1
RIP: 0010:[<ffffffff8015a76d>]  [<ffffffff8015a76d>] list_del+0x21/0x6b
RSP: 0018:ffff8103e028dc58  EFLAGS: 00010082
RAX: 0000000000000058 RBX: ffff810106a085b0 RCX: ffffffff80323028
RDX: ffffffff80323028 RSI: 0000000000000000 RDI: ffffffff80323020
RBP: ffff810106a08588 R08: ffffffff80323028 R09: 0000000000000030
R10: 0000000000000000 R11: 0000000000000000 R12: ffff810000023600
R13: ffff810011006700 R14: 0000000000000006 R15: ffff8103e028dd88
FS:  00002ac9facd2550(0000) GS:ffffffff80431000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000015c23ed8 CR3: 00000003954f6000 CR4: 00000000000006e0
Process rsync (pid: 15322, threadinfo ffff8103e028c000, task ffff81014b077080)
Stack:  000000000000000e ffffffff800cd26d ffff8101058f6170 00000000000345dd
 00000000000345dd 00000000000345dc 0000000000001000 ffffffff8000c3c5
 ffff810000024c00 ffffffff8000d25f ffff8103e028de88 ffff81024dcdb6c0
Call Trace:
 [<ffffffff800cd26d>] __pagevec_mark_accessed+0xd8/0x15e
 [<ffffffff8000c3c5>] do_generic_mapping_read+0x171/0x359
 [<ffffffff8000d25f>] file_read_actor+0x0/0x159
 [<ffffffff8000c6f9>] __generic_file_aio_read+0x14c/0x198
 [<ffffffff80016eeb>] generic_file_aio_read+0x36/0x3b
 [<ffffffff8000cf47>] do_sync_read+0xc7/0x104
 [<ffffffff80067235>] do_page_fault+0x4cc/0x842
 [<ffffffff800a3444>] autoremove_wake_function+0x0/0x2e
 [<ffffffff800624c6>] __sched_text_start+0xf6/0xbd0
 [<ffffffff8000b72f>] vfs_read+0xcb/0x171
 [<ffffffff80011d49>] sys_read+0x45/0x6e
 [<ffffffff8005d28d>] tracesys+0xd5/0xe0


Code: 0f 0b 68 89 ab 2c 80 c2 41 00 eb fe 48 8b 07 48 8b 50 08 48 
RIP  [<ffffffff8015a76d>] list_del+0x21/0x6b
 RSP <ffff8103e028dc58>
Code: f3 aa 75 1a f0 41 0f ba 2c 24 03 e9 45 01 00 00 83 7c 24 5c 
RIP  [<ffffffff8002882c>] do_mpage_readpage+0x331/0x4a5
 RSP <ffff8101b8bbdb28>