Why is my system stuck due to soft lockup, while nfs is used?

Solution Verified - Updated -

Issue

  • Soft lockups while running commands like 'mv' or 'find' using nfsv3 mounted filesystems:

    kernel: CPU 12:
    kernel: Modules linked in: iptable_filter ip_tables x_tables nfs fscache nfs_acl lockd autofs4 mptctl mptbase ipmi_si(U) ipmi_devintf(U) ipmi_msghandler(U) hidp rfcomm l2cap bluetooth sunrpc dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec i2c_core dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport sr_mod cdrom sg hpilo e1000e bnx2 shpchp serio_raw pcspkr dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod ata_piix libata cciss sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
    kernel: Pid: 11830, comm: java Tainted: G      2.6.18-194.el5 #1
    kernel: RIP: 0010:[<ffffffff80048a97>]  [<ffffffff80048a97>] lock_rename+0x3f/0xbd
    kernel: RSP: 0018:ffff810115ff1df8  EFLAGS: 00000206
    kernel: RAX: ffff810297e483d8 RBX: ffff810279c6ba98 RCX: 0000000000000402
    kernel: RDX: ffffffff8001283d RSI: ffff810279c6ba98 RDI: ffff810299ceb248
    kernel: RBP: ffff81013b6b9320 R08: 0000000000000002 R09: ffff81042e2b8b80
    kernel: R10: 0000000000000002 R11: ffffffff8002c728 R12: ffff810336a68810
    kernel: R13: ffff81042e1813c0 R14: ffff81018840905a R15: 00000009f6922168
    kernel: FS:  000000004388b940(0063) GS:ffff81010ebce6c0(0000) knlGS:0000000000000000
    kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
    kernel: CR2: 00002aabaf2725d0 CR3: 00000002bb69d000 CR4: 00000000000006e0
    kernel:
    kernel: Call Trace:
    kernel:  [<ffffffff80048a85>] lock_rename+0x2d/0xbd
    kernel:  [<ffffffff80036e60>] sys_renameat+0xd7/0x1eb
    kernel:  [<ffffffff800b878c>] audit_syscall_entry+0x180/0x1b3
    kernel:  [<ffffffff8005e28d>] tracesys+0xd5/0xe0
    
  • top output, system log:

    top - 10:56:21 up 27 days,  8:29,  8 users,  load average: 47.24, 45.92, 39.87
    Tasks: 673 total,   6 running, 667 sleeping,   0 stopped,   0 zombie
    Cpu(s):  2.1%us,  0.3%sy,  0.0%ni, 97.3%id,  0.3%wa,  0.0%hi,  0.0%si,  0.0%st
    Mem:  16304952k total, 16168080k used,   136872k free,    54524k buffers
    Swap:  8191992k total,      292k used,  8191700k free,  5380468k cached
    
      PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                      
    3218 cbradm    16   0 69656  732  624 R 99.0  0.0  40:21.83 mv                                                                                            
    14527 root      15   0 13160 1456  732 R  3.8  0.0   0:00.03 top                                                                                           
    10031 useradm    16   0 5311m 3.4g  10m S  1.9 21.6 979:24.56 java                                                                                          
    18992 useradm    24   0 5295m 4.1g  10m S  1.9 26.4 348:15.59 java                                                                                          
        1 root      15   0 10372  700  592 S  0.0  0.0   0:03.19 init
    
    kernel: BUG: soft lockup - CPU#4 stuck for 60s! [mv:3218]
    kernel: BUG: soft lockup - CPU#4 stuck for 60s! [mv:3218]
    kernel: BUG: soft lockup - CPU#4 stuck for 60s! [mv:3218]
    kernel: BUG: soft lockup - CPU#4 stuck for 60s! [mv:3218]
    kernel: BUG: soft lockup - CPU#4 stuck for 60s! [mv:3218]
    kernel: BUG: soft lockup - CPU#4 stuck for 60s! [mv:3218]
    
  • Why do entries like the following appear in my systemlog?

    kernel: VFS: Lookup of 'exampledirectory' in nfs 0:18 would have caused loop
    

Environment

  • Red Hat Enterprise Linux (RHEL) 5, minor release/update level before 8
  • kernels prior to 2.6.18-274.14.1.el5
  • NFS export mounted on multiple NFS clients

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.