System crash at gen_pool_free+0xb8 while there are massive EDAC/MCE.
Issue
- System crash with EDAC logs and call trace as below.
[1877005.194279] EDAC sbridge MC1: HANDLING MCE MEMORY ERROR
[1877005.194280] EDAC sbridge MC1: CPU 8: Machine Check Event: 0 Bank 8: cc00008a000800c0
[1877005.194281] EDAC sbridge MC1: TSC 0
[1877005.194282] EDAC sbridge MC1: ADDR 3aa8194000
[1877005.194283] EDAC sbridge MC1: MISC 3182000040148c
[1877005.194284] EDAC sbridge MC1: PROCESSOR 0:206d7 TIME 1503336656 SOCKET 1 APIC 20
[1877005.194330] ------------[ cut here ]------------
[1877005.194363] kernel BUG at lib/genalloc.c:342!
[1877005.194394] invalid opcode: 0000 [#1] SMP
[1877005.194417] Modules linked in: iptable_filter ip_tables binfmt_misc vhost_net vhost macvtap macvlan ebt_arp ebtable_nat tun nfsv3 nfs fscache ebtable_filter ebtables ip6table_filter ip6_tables fuse scsi_transport_iscsi xt_physdev br_netfilter ipt_REJECT nf_reject_ipv4 xt_multiport softdog dm_mirror dm_region_hash dm_log intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass iTCO_wdt iTCO_vendor_support crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd mei_me mei i2c_i801 ses sb_edac enclosure pcspkr edac_core lpc_ich ntb dm_service_time ipmi_devintf sg ipmi_si ipmi_msghandler shpchp dm_multipath dm_mod nfsd auth_rpcgss nfs_acl lockd grace sunrpc xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ast drm_kms_helper syscopyarea crct10dif_pclmul crct10dif_common
[1877005.194964] sysfillrect sysimgblt crc32c_intel fb_sys_fops isci ttm qla2xxx ixgbe megaraid_sas libsas igb drm scsi_transport_sas mdio ptp libata scsi_transport_fc pps_core i2c_algo_bit dca i2c_core scsi_tgt fjes 8021q garp mrp bridge stp llc bonding [last unloaded: nf_defrag_ipv6]
[1877005.195133] CPU: 8 PID: 24699 Comm: kworker/8:0 Not tainted 3.10.0-514.10.2.el7.x86_64 #1
[1877005.195166] Hardware name: Quanta S210-X22RQ/S210-X22RQ, BIOS S2RQ3A22 05/14/2013
[1877005.195205] Workqueue: events mce_process_work
[1877005.195227] task: ffff885fbbf7ce70 ti: ffff884fd7e04000 task.ti: ffff884fd7e04000
[1877005.195257] RIP: 0010:[<ffffffff81333f98>] [<ffffffff81333f98>] gen_pool_free+0xb8/0xc0
[1877005.195298] RSP: 0018:ffff884fd7e07dc8 EFLAGS: 00010202
[1877005.195321] RAX: 0000000000000002 RBX: ffff8830ffc01400 RCX: 0000000000000006
[1877005.195350] RDX: 000000000003000c RSI: 0000000000000030 RDI: ffff8830ffc01430
[1877005.195388] RBP: ffff884fd7e07df0 R08: ffffffffffffffcf R09: 000000000000146b
[1877005.195416] R10: 0000000000000000 R11: ffff884fd7e07a76 R12: 0000000000000002
[1877005.195446] R13: ffff88607fc16480 R14: ffff88607fc1a700 R15: 0000000000000200
[1877005.195483] FS: 0000000000000000(0000) GS:ffff88607fc00000(0000) knlGS:0000000000000000
[1877005.195516] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1877005.195540] CR2: 00007f262233be90 CR3: 00000000019ba000 CR4: 00000000000427e0
[1877005.195568] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[1877005.195597] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[1877005.195628] Stack:
[1877005.195639] 000000067fc16480 ffff8830ffc01400 000000006b364d2c ffffffff81ccaa00
[1877005.195679] ffff885fb7eb4c80 ffff884fd7e07e08 ffffffff81043d5e ffffffff81cca820
[1877005.195718] ffff884fd7e07e18 ffffffff8104097e ffff884fd7e07e60 ffffffff810a810b
[1877005.195758] Call Trace:
[1877005.195776] [<ffffffff81043d5e>] mce_gen_pool_process+0x4e/0x60
[1877005.195803] [<ffffffff8104097e>] mce_process_work+0xe/0x10
[1877005.195839] [<ffffffff810a810b>] process_one_work+0x17b/0x470
[1877005.195864] [<ffffffff810a8f46>] worker_thread+0x126/0x410
[1877005.195887] [<ffffffff810a8e20>] ? rescuer_thread+0x460/0x460
[1877005.195917] [<ffffffff810b06ff>] kthread+0xcf/0xe0
[1877005.195939] [<ffffffff810b0630>] ? kthread_create_on_node+0x140/0x140
[1877005.195974] [<ffffffff81696958>] ret_from_fork+0x58/0x90
[1877005.195998] [<ffffffff810b0630>] ? kthread_create_on_node+0x140/0x140
[1877005.196033] Code: dc 75 28 44 89 e0 d3 e0 f0 01 43 10 48 8b 45 e8 65 48 33 04 25 28 00 00 00 75 0b 48 83 c4 18 5b 41 5c 5d c3 0f 0b e8 48 17 d5 ff <0f> 0b 0f 0b 0f 1f 40 00 55 48 89 e5 41 57 49 89 f7 be d0 80 00
[1877005.196320] RIP [<ffffffff81333f98>] gen_pool_free+0xb8/0xc0
[1877005.196347] RSP <ffff884fd7e07dc8>
Environment
- Red Hat Enterprise Linux 7
- 3.10.0-514.el7 ~ 3.10.0-693.el7 series of kernels
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.