System crash at gen_pool_free+0xb8 while there are massive EDAC/MCE.

Solution Verified - Updated -

Issue

  • System crash with EDAC logs and call trace as below.
[1877005.194279] EDAC sbridge MC1: HANDLING MCE MEMORY ERROR
[1877005.194280] EDAC sbridge MC1: CPU 8: Machine Check Event: 0 Bank 8: cc00008a000800c0
[1877005.194281] EDAC sbridge MC1: TSC 0 
[1877005.194282] EDAC sbridge MC1: ADDR 3aa8194000 
[1877005.194283] EDAC sbridge MC1: MISC 3182000040148c 
[1877005.194284] EDAC sbridge MC1: PROCESSOR 0:206d7 TIME 1503336656 SOCKET 1 APIC 20
[1877005.194330] ------------[ cut here ]------------
[1877005.194363] kernel BUG at lib/genalloc.c:342!
[1877005.194394] invalid opcode: 0000 [#1] SMP 
[1877005.194417] Modules linked in: iptable_filter ip_tables binfmt_misc vhost_net vhost macvtap macvlan ebt_arp ebtable_nat tun nfsv3 nfs fscache ebtable_filter ebtables ip6table_filter ip6_tables fuse scsi_transport_iscsi xt_physdev br_netfilter ipt_REJECT nf_reject_ipv4 xt_multiport softdog dm_mirror dm_region_hash dm_log intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass iTCO_wdt iTCO_vendor_support crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd mei_me mei i2c_i801 ses sb_edac enclosure pcspkr edac_core lpc_ich ntb dm_service_time ipmi_devintf sg ipmi_si ipmi_msghandler shpchp dm_multipath dm_mod nfsd auth_rpcgss nfs_acl lockd grace sunrpc xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ast drm_kms_helper syscopyarea crct10dif_pclmul crct10dif_common
[1877005.194964]  sysfillrect sysimgblt crc32c_intel fb_sys_fops isci ttm qla2xxx ixgbe megaraid_sas libsas igb drm scsi_transport_sas mdio ptp libata scsi_transport_fc pps_core i2c_algo_bit dca i2c_core scsi_tgt fjes 8021q garp mrp bridge stp llc bonding [last unloaded: nf_defrag_ipv6]
[1877005.195133] CPU: 8 PID: 24699 Comm: kworker/8:0 Not tainted 3.10.0-514.10.2.el7.x86_64 #1
[1877005.195166] Hardware name: Quanta S210-X22RQ/S210-X22RQ, BIOS S2RQ3A22 05/14/2013
[1877005.195205] Workqueue: events mce_process_work
[1877005.195227] task: ffff885fbbf7ce70 ti: ffff884fd7e04000 task.ti: ffff884fd7e04000
[1877005.195257] RIP: 0010:[<ffffffff81333f98>]  [<ffffffff81333f98>] gen_pool_free+0xb8/0xc0
[1877005.195298] RSP: 0018:ffff884fd7e07dc8  EFLAGS: 00010202
[1877005.195321] RAX: 0000000000000002 RBX: ffff8830ffc01400 RCX: 0000000000000006
[1877005.195350] RDX: 000000000003000c RSI: 0000000000000030 RDI: ffff8830ffc01430
[1877005.195388] RBP: ffff884fd7e07df0 R08: ffffffffffffffcf R09: 000000000000146b
[1877005.195416] R10: 0000000000000000 R11: ffff884fd7e07a76 R12: 0000000000000002
[1877005.195446] R13: ffff88607fc16480 R14: ffff88607fc1a700 R15: 0000000000000200
[1877005.195483] FS:  0000000000000000(0000) GS:ffff88607fc00000(0000) knlGS:0000000000000000
[1877005.195516] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1877005.195540] CR2: 00007f262233be90 CR3: 00000000019ba000 CR4: 00000000000427e0
[1877005.195568] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[1877005.195597] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[1877005.195628] Stack:
[1877005.195639]  000000067fc16480 ffff8830ffc01400 000000006b364d2c ffffffff81ccaa00
[1877005.195679]  ffff885fb7eb4c80 ffff884fd7e07e08 ffffffff81043d5e ffffffff81cca820
[1877005.195718]  ffff884fd7e07e18 ffffffff8104097e ffff884fd7e07e60 ffffffff810a810b
[1877005.195758] Call Trace:
[1877005.195776]  [<ffffffff81043d5e>] mce_gen_pool_process+0x4e/0x60
[1877005.195803]  [<ffffffff8104097e>] mce_process_work+0xe/0x10
[1877005.195839]  [<ffffffff810a810b>] process_one_work+0x17b/0x470
[1877005.195864]  [<ffffffff810a8f46>] worker_thread+0x126/0x410
[1877005.195887]  [<ffffffff810a8e20>] ? rescuer_thread+0x460/0x460
[1877005.195917]  [<ffffffff810b06ff>] kthread+0xcf/0xe0
[1877005.195939]  [<ffffffff810b0630>] ? kthread_create_on_node+0x140/0x140
[1877005.195974]  [<ffffffff81696958>] ret_from_fork+0x58/0x90
[1877005.195998]  [<ffffffff810b0630>] ? kthread_create_on_node+0x140/0x140
[1877005.196033] Code: dc 75 28 44 89 e0 d3 e0 f0 01 43 10 48 8b 45 e8 65 48 33 04 25 28 00 00 00 75 0b 48 83 c4 18 5b 41 5c 5d c3 0f 0b e8 48 17 d5 ff <0f> 0b 0f 0b 0f 1f 40 00 55 48 89 e5 41 57 49 89 f7 be d0 80 00 
[1877005.196320] RIP  [<ffffffff81333f98>] gen_pool_free+0xb8/0xc0
[1877005.196347]  RSP <ffff884fd7e07dc8>

Environment

  • Red Hat Enterprise Linux 7
  • 3.10.0-514.el7 ~ 3.10.0-693.el7 series of kernels

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content