Why is my system stuck due to soft lockup, while nfs is used?
Issue
-
Soft lockups while running commands like 'mv' or 'find' using nfsv3 mounted filesystems:
kernel: CPU 12: kernel: Modules linked in: iptable_filter ip_tables x_tables nfs fscache nfs_acl lockd autofs4 mptctl mptbase ipmi_si(U) ipmi_devintf(U) ipmi_msghandler(U) hidp rfcomm l2cap bluetooth sunrpc dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec i2c_core dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport sr_mod cdrom sg hpilo e1000e bnx2 shpchp serio_raw pcspkr dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod ata_piix libata cciss sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd kernel: Pid: 11830, comm: java Tainted: G 2.6.18-194.el5 #1 kernel: RIP: 0010:[<ffffffff80048a97>] [<ffffffff80048a97>] lock_rename+0x3f/0xbd kernel: RSP: 0018:ffff810115ff1df8 EFLAGS: 00000206 kernel: RAX: ffff810297e483d8 RBX: ffff810279c6ba98 RCX: 0000000000000402 kernel: RDX: ffffffff8001283d RSI: ffff810279c6ba98 RDI: ffff810299ceb248 kernel: RBP: ffff81013b6b9320 R08: 0000000000000002 R09: ffff81042e2b8b80 kernel: R10: 0000000000000002 R11: ffffffff8002c728 R12: ffff810336a68810 kernel: R13: ffff81042e1813c0 R14: ffff81018840905a R15: 00000009f6922168 kernel: FS: 000000004388b940(0063) GS:ffff81010ebce6c0(0000) knlGS:0000000000000000 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 kernel: CR2: 00002aabaf2725d0 CR3: 00000002bb69d000 CR4: 00000000000006e0 kernel: kernel: Call Trace: kernel: [<ffffffff80048a85>] lock_rename+0x2d/0xbd kernel: [<ffffffff80036e60>] sys_renameat+0xd7/0x1eb kernel: [<ffffffff800b878c>] audit_syscall_entry+0x180/0x1b3 kernel: [<ffffffff8005e28d>] tracesys+0xd5/0xe0 -
top output, system log:
top - 10:56:21 up 27 days, 8:29, 8 users, load average: 47.24, 45.92, 39.87 Tasks: 673 total, 6 running, 667 sleeping, 0 stopped, 0 zombie Cpu(s): 2.1%us, 0.3%sy, 0.0%ni, 97.3%id, 0.3%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 16304952k total, 16168080k used, 136872k free, 54524k buffers Swap: 8191992k total, 292k used, 8191700k free, 5380468k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 3218 cbradm 16 0 69656 732 624 R 99.0 0.0 40:21.83 mv 14527 root 15 0 13160 1456 732 R 3.8 0.0 0:00.03 top 10031 useradm 16 0 5311m 3.4g 10m S 1.9 21.6 979:24.56 java 18992 useradm 24 0 5295m 4.1g 10m S 1.9 26.4 348:15.59 java 1 root 15 0 10372 700 592 S 0.0 0.0 0:03.19 init kernel: BUG: soft lockup - CPU#4 stuck for 60s! [mv:3218] kernel: BUG: soft lockup - CPU#4 stuck for 60s! [mv:3218] kernel: BUG: soft lockup - CPU#4 stuck for 60s! [mv:3218] kernel: BUG: soft lockup - CPU#4 stuck for 60s! [mv:3218] kernel: BUG: soft lockup - CPU#4 stuck for 60s! [mv:3218] kernel: BUG: soft lockup - CPU#4 stuck for 60s! [mv:3218] -
Why do entries like the following appear in my systemlog?
kernel: VFS: Lookup of 'exampledirectory' in nfs 0:18 would have caused loop
Environment
- Red Hat Enterprise Linux (RHEL) 5, minor release/update level before 8
- kernels prior to 2.6.18-274.14.1.el5
- NFS export mounted on multiple NFS clients
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.