System crash at cfq_group_served+0x103 due to divide error
Issue
- System crash at cfq_group_served+0x103 due to divide error
PID: 3457 TASK: ffff8800363e0000 CPU: 2 COMMAND: "mysqld"
#0 [ffff88007ab575d0] machine_kexec at ffffffff81059cdb
#1 [ffff88007ab57630] __crash_kexec at ffffffff81105182
#2 [ffff88007ab57700] crash_kexec at ffffffff81105270
#3 [ffff88007ab57718] oops_end at ffffffff8168ed88
#4 [ffff88007ab57740] die at ffffffff8102e93b
#5 [ffff88007ab57770] do_trap at ffffffff8168e440
#6 [ffff88007ab577c0] do_divide_error at ffffffff8102af4e
#7 [ffff88007ab57870] divide_error at ffffffff81697cce
[exception RIP: cfq_group_served+0x103]
RIP: ffffffff81310b03 RSP: ffff88007ab57920 RFLAGS: 00010006
RAX: 0000000028000000 RBX: ffff8800365c5000 RCX: ffff880036098408
RDX: 0000000000000000 RSI: ffff8800365c5000 RDI: ffff880036098408
RBP: ffff88007ab57a08 R8: 0000000000000000 R9: ffff8800361d20f8
R10: 0000000000000000 R11: 0000000000000050 R12: ffff8800361d2740
R13: 0000000000000028 R14: 0000000000000000 R15: ffff880036098400
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#8 [ffff88007ab57a10] __cfq_slice_expired at ffffffff81311330
#9 [ffff88007ab57ad0] cfq_preempt_queue at ffffffff81311772
#10 [ffff88007ab57b80] cfq_insert_request at ffffffff81314f9b
#11 [ffff88007ab57c40] __elv_add_request at ffffffff812e7bfc
#12 [ffff88007ab57c78] blk_flush_plug_list at ffffffff812eff4e
#13 [ffff88007ab57cd8] blk_finish_plug at ffffffff812f0374
#14 [ffff88007ab57cf0] ext4_writepages at ffffffffa018f8e9 [ext4]
#15 [ffff88007ab57e28] do_writepages at ffffffff8118d24e
#16 [ffff88007ab57e38] __filemap_fdatawrite_range at ffffffff81181e35
#17 [ffff88007ab57e88] filemap_write_and_wait_range at ffffffff81181f81
#18 [ffff88007ab57eb8] generic_file_fsync at ffffffff812260a6
#19 [ffff88007ab57ee8] ext4_sync_file at ffffffffa0186c83 [ext4]
#20 [ffff88007ab57f30] do_fsync at ffffffff8122fdb5
#21 [ffff88007ab57f70] sys_fdatasync at ffffffff812300a3
#22 [ffff88007ab57f80] system_call_fastpath at ffffffff816964c9
RIP: 00007fd44f602ddd RSP: 00007fd426aca000 RFLAGS: 00010246
RAX: 000000000000004b RBX: ffffffff816964c9 RCX: 0000000000000000
RDX: 0000000000000b45 RSI: 0000000000000000 RDI: 0000000000000009
RBP: 00007fd426acd0b0 R8: 00000000004ff800 R9: 000000001a8f290c
R10: 00007fd453da1218 R11: 0000000000000293 R12: ffffffff812300a3
R13: ffff88007ab57f78 R14: 0000000051eb851f R15: 0000000000000000
ORIG_RAX: 000000000000004b CS: 0033 SS: 002b
- Also, list corruption errors prior to the division error:
[166213.211890] ------------[ cut here ]------------
[166213.211933] WARNING: at lib/list_debug.c:29 __list_add+0x65/0xc0()
[166213.211949] list_add corruption. next->prev should be prev (ffff88007f097568), but was ffff880014793f50. (next=ffff880014793f50).
[166213.211958] Modules linked in: binfmt_misc nf_conntrack_ftp ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter dm_mirror dm_region_hash dm_log dm_mod ppdev iTCO_wdt iTCO_vendor_support intel_powerclamp coretemp i2c_i801 sg i2c_core pcspkr lpc_ich shpchp parport_pc parport tpm_infineon acpi_cpufreq ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic crct10dif_common ata_generic pata_acpi serio_raw ata_piix e1000e
[166213.212224] 8139too pata_jmicron 8139cp mii libata ptp pps_core video fjes
[166213.212269] CPU: 3 PID: 1947 Comm: mysqld Not tainted 3.10.0-514.el7.x86_64 #1
[166213.212282] Hardware name: To be filled by O.E.M. To be filled by O.E.M./Tiger Hill, BIOS 4.6.5 08/24/2015
[166213.212294] ffff88007a227d18 00000000b59777d8 ffff88007a227cd0 ffffffff81685eac
[166213.212315] ffff88007a227d08 ffffffff81085820 ffff880014791ff0 ffff880014793f50
[166213.212334] ffff88007f097568 ffff880014792764 0000000000000246 ffff88007a227d70
[166213.212353] Call Trace:
[166213.212382] [<ffffffff81685eac>] dump_stack+0x19/0x1b
[166213.212403] [<ffffffff81085820>] warn_slowpath_common+0x70/0xb0
[166213.212422] [<ffffffff810858bc>] warn_slowpath_fmt+0x5c/0x80
[166213.212441] [<ffffffff81332f15>] __list_add+0x65/0xc0
[166213.212461] [<ffffffff810ca937>] account_entity_enqueue+0xa7/0xb0
[166213.212484] [<ffffffff810d0549>] enqueue_entity+0x7a9/0xb60
[166213.212506] [<ffffffff813179a5>] ? cpumask_next_and+0x35/0x50
[166213.212525] [<ffffffff810d0959>] enqueue_task_fair+0x59/0x6c0
[166213.212547] [<ffffffff810c7d45>] ? sched_clock_cpu+0x85/0xc0
[166213.212566] [<ffffffff810bcf71>] enqueue_task+0x51/0x80
[166213.212584] [<ffffffff810c17b3>] activate_task+0x23/0x30
[166213.212603] [<ffffffff810c4835>] wake_up_new_task+0xc5/0x170
[166213.212621] [<ffffffff81084a38>] do_fork+0xe8/0x2c0
[166213.212639] [<ffffffff81084c96>] SyS_clone+0x16/0x20
[166213.212657] [<ffffffff81696819>] stub_clone+0x69/0x90
[166213.212678] [<ffffffff816964c9>] ? system_call_fastpath+0x16/0x1b
[166213.212692] ---[ end trace f1f13c8dba9bc521 ]---
Environment
- Red Hat Enterprise Linux 7
- kernel-3.10.0-514.el7
- CFQ scheduler
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.