Why system gets crash with soft lockup RIP: kmem_cache_alloc_node_trace ?
Issue
- System gets hang and
/var/log/messagesshowsINFO: task <process>:<pid> blocked for more than 120 seconds - When collected VMcore for this issue as explained in KCS 31453 , 382913 , below traces are observed:
BUG: soft lockup - CPU#44 stuck for 67s! [nfsiod:4692]
Modules linked in: ipmi_devintf nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand freq_table pcc_cpufreq ipv6 power_meter qlcnic netxen_nic microcode sg serio_raw iTCO_wdt iTCO_vendor_support hpilo hpwdt i7core_edac edac_core shpchp ext4 jbd2 mbcache sr_mod cdrom sd_mod crc_t10dif pata_acpi ata_generic ata_piix hpsa radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
irq event stamp: 1442694783
hardirqs last enabled at (1442694782): [<ffffffff811816d1>] kmem_cache_alloc_node_trace+0x1c1/0x280
hardirqs last disabled at (1442694783): [<ffffffff8100aeea>] save_args+0x6a/0x70
softirqs last enabled at (1442663466): [<ffffffff8107a861>] __do_softirq+0x151/0x210
softirqs last disabled at (1442664959): [<ffffffff8100c2cc>] call_softirq+0x1c/0x30
CPU 44
Modules linked in: ipmi_devintf nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand freq_table pcc_cpufreq ipv6 power_meter qlcnic netxen_nic microcode sg serio_raw iTCO_wdt iTCO_vendor_support hpilo hpwdt i7core_edac edac_core shpchp ext4 jbd2 mbcache sr_mod cdrom sd_mod crc_t10dif pata_acpi ata_generic ata_piix hpsa radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
Pid: 4692, comm: nfsiod Not tainted 2.6.32-358.23.2.el6.x86_64.debug #1 HP ProLiant DL580 G7
RIP: 0010:[<ffffffff811816d7>] [<ffffffff811816d7>] kmem_cache_alloc_node_trace+0x1c7/0x280
RSP: 0018:ffff880047203c30 EFLAGS: 00000246
RAX: 0000000055fdc67e RBX: ffff880047203c80 RCX: 0000000000000003
[...]
Call Trace:
<IRQ>
[<ffffffff811817e5>] ? __kmalloc_node_track_caller+0x55/0x70
[<ffffffff811817e5>] ? __kmalloc_node_track_caller+0x55/0x70
[<ffffffff8146a201>] ? __alloc_skb+0x71/0x190
[<ffffffff8146a384>] ? __netdev_alloc_skb+0x24/0x50
[<ffffffffa02e54e5>] ? qlcnic_alloc_rx_skb+0x35/0x1f0 [qlcnic]
[<ffffffffa02e7e18>] ? qlcnic_process_rcv_ring+0x358/0x900 [qlcnic]
[<ffffffffa02deca3>] ? qlcnic_rx_poll+0x33/0x90 [qlcnic]
[...]
Environment
- Red Hat Enterprise Linux 6.4
kernel-2.6.32-358.23.2.el6
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
