Why system gets crash with soft lockup RIP: kmem_cache_alloc_node_trace ?

Issue

• System gets hang and /var/log/messages shows INFO: task <process>:<pid> blocked for more than 120 seconds
• When collected VMcore for this issue as explained in KCS 31453 , 382913 , below traces are observed:

BUG: soft lockup - CPU#44 stuck for 67s! [nfsiod:4692]
Modules linked in: ipmi_devintf nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand freq_table pcc_cpufreq ipv6 power_meter qlcnic netxen_nic microcode sg serio_raw iTCO_wdt iTCO_vendor_support hpilo hpwdt i7core_edac edac_core shpchp ext4 jbd2 mbcache sr_mod cdrom sd_mod crc_t10dif pata_acpi ata_generic ata_piix hpsa radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
irq event stamp: 1442694783
hardirqs last  enabled at (1442694782): [<ffffffff811816d1>] kmem_cache_alloc_node_trace+0x1c1/0x280
hardirqs last disabled at (1442694783): [<ffffffff8100aeea>] save_args+0x6a/0x70
softirqs last  enabled at (1442663466): [<ffffffff8107a861>] __do_softirq+0x151/0x210
softirqs last disabled at (1442664959): [<ffffffff8100c2cc>] call_softirq+0x1c/0x30
CPU 44 
Modules linked in: ipmi_devintf nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand freq_table pcc_cpufreq ipv6 power_meter qlcnic netxen_nic microcode sg serio_raw iTCO_wdt iTCO_vendor_support hpilo hpwdt i7core_edac edac_core shpchp ext4 jbd2 mbcache sr_mod cdrom sd_mod crc_t10dif pata_acpi ata_generic ata_piix hpsa radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]

Pid: 4692, comm: nfsiod Not tainted 2.6.32-358.23.2.el6.x86_64.debug #1 HP ProLiant DL580 G7
RIP: 0010:[<ffffffff811816d7>]  [<ffffffff811816d7>] kmem_cache_alloc_node_trace+0x1c7/0x280
RSP: 0018:ffff880047203c30  EFLAGS: 00000246
RAX: 0000000055fdc67e RBX: ffff880047203c80 RCX: 0000000000000003
[...]
Call Trace:
 <IRQ> 
 [<ffffffff811817e5>] ? __kmalloc_node_track_caller+0x55/0x70
 [<ffffffff811817e5>] ? __kmalloc_node_track_caller+0x55/0x70
 [<ffffffff8146a201>] ? __alloc_skb+0x71/0x190
 [<ffffffff8146a384>] ? __netdev_alloc_skb+0x24/0x50
 [<ffffffffa02e54e5>] ? qlcnic_alloc_rx_skb+0x35/0x1f0 [qlcnic]
 [<ffffffffa02e7e18>] ? qlcnic_process_rcv_ring+0x358/0x900 [qlcnic]
 [<ffffffffa02deca3>] ? qlcnic_rx_poll+0x33/0x90 [qlcnic]
[...]