Soft lockup and system hang is observed on IBM System x3850 X5 (with Intel(R) Xeon(R) X7560 Processor) on Red Hat Enterprise Linux 5.3

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 5.3
  • IBM System x3850 X5
  • Intel(R) Xeon(R) X7560 Processor

Issue

  • System experienced soft lockup and hang situation

Resolution

  • Update to kernel-2.6.18-164.el5 (or later) which was made available via RHSA-2009:1243-3.

Root Cause

  • The backtrace shows that it is stuck in privcmd_ioctl(), meaning that userspace was doing a hypercall.

Diagnostic Steps

  • Collect sosreprot and vmcore file for analysis
  • Following messages observed in /var/log/messages file
kernel: BUG: soft lockup - CPU#23 stuck for 10s! [python:12899]
kernel: CPU 23:
kernel: Modules linked in: nls_utf8 loop nfs lockd fscache nfs_acl sunrpc xt_physdev netloop netbk blktap blkbk ipt_MASQUERADE iptable_nat ip_nat xt_state ip_conntrack nfnetlink ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables bridge autofs4 ipv6 xfrm_nalgo crypto_api bonding emcpdm(PU) emcpgpx(PU) emcpmpx(PU) emcp(PU) dm_multipath scsi_dh video hwmon backlight sbs i2c_ec button battery asus_acpi ac parport_pc lp parport sg pcspkr i2c_i801 i2c_core bnx2 serial_core e1000e cdc_ether usbnet dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod usb_storage qla2xxx scsi_transport_fc ata_piix libata shpchp mptsas mptscsih mptbase scsi_transport_sas sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
kernel: Pid: 12899, comm: python Tainted: P      2.6.18-128.el5xen #1
kernel: RIP: e030:[<ffffffff80206188>]  [<ffffffff80206188>] hypercall_page+0x188/0x1000
kernel: RSP: e02b:ffff88001d871e50  EFLAGS: 00000282
kernel: RAX: 000000000000000c RBX: ffff880060efeac0 RCX: ffffffff8020618a
kernel: RDX: 0000000000c00000 RSI: 000000004b517b20 RDI: 000000000055fcf6
kernel: RBP: 000000004b517ad0 R08: 000000004b517ba0 R09: 0000000000000000
kernel: R10: 0000000000c00018 R11: 0000000000000282 R12: 0000000000305000
kernel: R13: 0000000000000008 R14: 0000000000000008 R15: 0000000000000008
kernel: FS:  00002b388dede540(0063) GS:ffffffff805bab80(0000) knlGS:0000000000000000
kernel: CS:  e033 DS: 0000 ES: 0000
kernel:
kernel: Call Trace:
kernel:  [<ffffffff803aae18>] privcmd_ioctl+0x88/0x338
kernel:  [<ffffffff803164f4>] file_has_perm+0x94/0xa3
kernel:  [<ffffffff802437ba>] do_ioctl+0x21/0x6b
kernel:  [<ffffffff80231010>] vfs_ioctl+0x248/0x261
kernel:  [<ffffffff8024ddde>] sys_ioctl+0x59/0x78
kernel:  [<ffffffff8025f2f9>] tracesys+0xab/0xb6
kernel:
  • Processor (Intel Xeon X7560) used on the system is supported from RHEL5.4 onwards
  • Hardware (IBM System x3850 X5) is certified on RHEL5.4 onwards as per Cert id 577873 and 601378

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments