System crash at cfq_group_served+0x103 due to divide error

Solution Unverified - Updated -

Issue

  • System crash at cfq_group_served+0x103 due to divide error
PID: 3457   TASK: ffff8800363e0000  CPU: 2   COMMAND: "mysqld"
 #0 [ffff88007ab575d0] machine_kexec at ffffffff81059cdb
 #1 [ffff88007ab57630] __crash_kexec at ffffffff81105182
 #2 [ffff88007ab57700] crash_kexec at ffffffff81105270
 #3 [ffff88007ab57718] oops_end at ffffffff8168ed88
 #4 [ffff88007ab57740] die at ffffffff8102e93b
 #5 [ffff88007ab57770] do_trap at ffffffff8168e440
 #6 [ffff88007ab577c0] do_divide_error at ffffffff8102af4e
 #7 [ffff88007ab57870] divide_error at ffffffff81697cce
    [exception RIP: cfq_group_served+0x103]
    RIP: ffffffff81310b03  RSP: ffff88007ab57920  RFLAGS: 00010006
    RAX: 0000000028000000  RBX: ffff8800365c5000  RCX: ffff880036098408
    RDX: 0000000000000000  RSI: ffff8800365c5000  RDI: ffff880036098408
    RBP: ffff88007ab57a08   R8: 0000000000000000   R9: ffff8800361d20f8
    R10: 0000000000000000  R11: 0000000000000050  R12: ffff8800361d2740
    R13: 0000000000000028  R14: 0000000000000000  R15: ffff880036098400
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 #8 [ffff88007ab57a10] __cfq_slice_expired at ffffffff81311330
 #9 [ffff88007ab57ad0] cfq_preempt_queue at ffffffff81311772
#10 [ffff88007ab57b80] cfq_insert_request at ffffffff81314f9b
#11 [ffff88007ab57c40] __elv_add_request at ffffffff812e7bfc
#12 [ffff88007ab57c78] blk_flush_plug_list at ffffffff812eff4e
#13 [ffff88007ab57cd8] blk_finish_plug at ffffffff812f0374
#14 [ffff88007ab57cf0] ext4_writepages at ffffffffa018f8e9 [ext4]
#15 [ffff88007ab57e28] do_writepages at ffffffff8118d24e
#16 [ffff88007ab57e38] __filemap_fdatawrite_range at ffffffff81181e35
#17 [ffff88007ab57e88] filemap_write_and_wait_range at ffffffff81181f81
#18 [ffff88007ab57eb8] generic_file_fsync at ffffffff812260a6
#19 [ffff88007ab57ee8] ext4_sync_file at ffffffffa0186c83 [ext4]
#20 [ffff88007ab57f30] do_fsync at ffffffff8122fdb5
#21 [ffff88007ab57f70] sys_fdatasync at ffffffff812300a3
#22 [ffff88007ab57f80] system_call_fastpath at ffffffff816964c9
    RIP: 00007fd44f602ddd  RSP: 00007fd426aca000  RFLAGS: 00010246
    RAX: 000000000000004b  RBX: ffffffff816964c9  RCX: 0000000000000000
    RDX: 0000000000000b45  RSI: 0000000000000000  RDI: 0000000000000009
    RBP: 00007fd426acd0b0   R8: 00000000004ff800   R9: 000000001a8f290c
    R10: 00007fd453da1218  R11: 0000000000000293  R12: ffffffff812300a3
    R13: ffff88007ab57f78  R14: 0000000051eb851f  R15: 0000000000000000
    ORIG_RAX: 000000000000004b  CS: 0033  SS: 002b
  • Also, list corruption errors prior to the division error:
[166213.211890] ------------[ cut here ]------------
[166213.211933] WARNING: at lib/list_debug.c:29 __list_add+0x65/0xc0()
[166213.211949] list_add corruption. next->prev should be prev (ffff88007f097568), but was ffff880014793f50. (next=ffff880014793f50).
[166213.211958] Modules linked in: binfmt_misc nf_conntrack_ftp ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter dm_mirror dm_region_hash dm_log dm_mod ppdev iTCO_wdt iTCO_vendor_support intel_powerclamp coretemp i2c_i801 sg i2c_core pcspkr lpc_ich shpchp parport_pc parport tpm_infineon acpi_cpufreq ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic crct10dif_common ata_generic pata_acpi serio_raw ata_piix e1000e
[166213.212224]  8139too pata_jmicron 8139cp mii libata ptp pps_core video fjes
[166213.212269] CPU: 3 PID: 1947 Comm: mysqld Not tainted 3.10.0-514.el7.x86_64 #1
[166213.212282] Hardware name: To be filled by O.E.M. To be filled by O.E.M./Tiger Hill, BIOS 4.6.5 08/24/2015
[166213.212294]  ffff88007a227d18 00000000b59777d8 ffff88007a227cd0 ffffffff81685eac
[166213.212315]  ffff88007a227d08 ffffffff81085820 ffff880014791ff0 ffff880014793f50
[166213.212334]  ffff88007f097568 ffff880014792764 0000000000000246 ffff88007a227d70
[166213.212353] Call Trace:
[166213.212382]  [<ffffffff81685eac>] dump_stack+0x19/0x1b
[166213.212403]  [<ffffffff81085820>] warn_slowpath_common+0x70/0xb0
[166213.212422]  [<ffffffff810858bc>] warn_slowpath_fmt+0x5c/0x80
[166213.212441]  [<ffffffff81332f15>] __list_add+0x65/0xc0
[166213.212461]  [<ffffffff810ca937>] account_entity_enqueue+0xa7/0xb0
[166213.212484]  [<ffffffff810d0549>] enqueue_entity+0x7a9/0xb60
[166213.212506]  [<ffffffff813179a5>] ? cpumask_next_and+0x35/0x50
[166213.212525]  [<ffffffff810d0959>] enqueue_task_fair+0x59/0x6c0
[166213.212547]  [<ffffffff810c7d45>] ? sched_clock_cpu+0x85/0xc0
[166213.212566]  [<ffffffff810bcf71>] enqueue_task+0x51/0x80
[166213.212584]  [<ffffffff810c17b3>] activate_task+0x23/0x30
[166213.212603]  [<ffffffff810c4835>] wake_up_new_task+0xc5/0x170
[166213.212621]  [<ffffffff81084a38>] do_fork+0xe8/0x2c0
[166213.212639]  [<ffffffff81084c96>] SyS_clone+0x16/0x20
[166213.212657]  [<ffffffff81696819>] stub_clone+0x69/0x90
[166213.212678]  [<ffffffff816964c9>] ? system_call_fastpath+0x16/0x1b
[166213.212692] ---[ end trace f1f13c8dba9bc521 ]---

Environment

  • Red Hat Enterprise Linux 7
  • kernel-3.10.0-514.el7
  • CFQ scheduler

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content