'ls' does not respond on nfs-mounted filesystem.

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 5.7
  • kernel-2.6.18-274.el5

Issue

  • Our customer faced a problem that ls command does not respond on nfs-mounted filesystem. The customer wanted us to investigate from the view of nfs server.
  • We asked the customer to collect nfs debug information and tcudump on nfs server. From tcpdump nfs server was sending GETATTR reply finally, so I thought it was a nfs client issue at first. However, from flow of nfs debug information it looks that nfs processing stopped in midstream.
  Jun 20 12:21:36 nfsserver kernel: svc: got len=140
  Jun 20 12:21:36 nfsserver kernel: svc: svc_authenticate (1)
  Jun 20 12:21:36 nfsserver kernel: svc: server ffff810251bab800, socket ffff81026cb6e880, inuse=2
  Jun 20 12:21:36 nfsserver kernel: svc: socket ffff810187f096c0 busy, not enqueued
  Jun 20 12:21:36 nfsserver kernel: svc: got len=-11
  Jun 20 12:21:36 nfsserver kernel: svc: calling dispatcher
  Jun 20 12:21:36 nfsserver kernel: svc: server ffff810251bab800 waiting for data (to = 3600000)
  Jun 20 12:21:36 nfsserver kernel: nfsd_dispatch: vers 3 proc 1
  Jun 20 12:21:36 nfsserver kernel: nfsd: GETATTR(3)  20: 01000001 02000800 033e0347 03460002 cdd5bf78 00000000
  Jun 20 12:21:36 nfsserver kernel: nfsd: fh_verify(20: 01000001 02000800 033e0347 03460002 cdd5bf78 00000000)
  Jun 20 12:21:36 nfsserver kernel: svc: service ffff81027d970000, releasing skb ffff810121121480
  Jun 20 12:21:36 nfsserver kernel: svc: socket ffff81026cb6e880 sendto([ffff810278b62000 112... ], 112) = 112 (addr d505010a)
  Jun 20 12:21:36 nfsserver kernel: svc: server ffff81027d970000 waiting for data (to = 3600000)
  Jun 20 12:21:36 nfsserver kernel: svc: socket ffff81026cb6e880(inet ffff810187f096c0), write_space busy=0
  Jun 20 12:21:38 nfsserver kernel: svc: socket ffff81026cb6e880(inet ffff810187f096c0), count=140, busy=0
  Jun 20 12:21:38 nfsserver kernel: svc: socket ffff810187f096c0 served by daemon ffff81027d970000
  Jun 20 12:21:38 nfsserver kernel: svc: server ffff81027d970000, socket ffff81026cb6e880, inuse=1
  Jun 20 12:21:38 nfsserver kernel: svc: socket ffff810187f096c0 served by daemon ffff810251bab800
  Jun 20 12:21:38 nfsserver kernel: svc: server ffff810251bab800, socket ffff81026cb6e880, inuse=2
  Jun 20 12:21:38 nfsserver kernel: svc: got len=-11
  Jun 20 12:21:38 nfsserver kernel: svc: got len=132
  Jun 20 12:21:38 nfsserver kernel: svc: server ffff810251bab800 waiting for data (to = 3600000)
  Jun 20 12:21:38 nfsserver kernel: svc: svc_authenticate (1)
  Jun 20 12:21:38 nfsserver kernel: svc: calling dispatcher
  Jun 20 12:21:38 nfsserver kernel: nfsd_dispatch: vers 3 proc 1
* Jun 20 12:21:38 nfsserver kernel: nfsd: GETATTR(3)  12: 00000001 02000800 033e0347 00000000 00000000 00000000
* Jun 20 12:21:38 nfsserver kernel: nfsd: fh_verify(12: 00000001 02000800 033e0347 00000000 00000000 00000000)
* Jun 20 12:21:38 nfsserver kernel: svc: service ffff81027d970000, releasing skb ffff810121121c80
* Jun 20 12:21:38 nfsserver kernel: svc: socket ffff81026cb6e880 sendto([ffff8102596f5000 112... ], 112) = 112 (addr d505010a)
* Jun 20 12:21:38 nfsserver kernel: svc: server ffff81027d970000 waiting for data (to = 3600000)
* Jun 20 12:21:38 nfsserver kernel: svc: socket ffff81026cb6e880(inet ffff810187f096c0), write_space busy=0
  Jun 20 12:21:47 nfsserver kernel: NFSD: laundromat service - starting
  Jun 20 12:21:47 nfsserver kernel: NFSD: laundromat_main - sleeping for 90 seconds
  Jun 20 12:23:17 nfsserver kernel: NFSD: laundromat service - starting
  Jun 20 12:23:17 nfsserver kernel: NFSD: laundromat_main - sleeping for 90 seconds
  Jun 20 12:24:47 nfsserver kernel: NFSD: laundromat service - starting
  Jun 20 12:24:47 nfsserver kernel: NFSD: laundromat_main - sleeping for 90 seconds
  Jun 20 12:26:17 nfsserver kernel: NFSD: laundromat service - starting
  Jun 20 12:26:17 nfsserver kernel: NFSD: laundromat_main - sleeping for 90 seconds
  Jun 20 12:27:47 nfsserver kernel: NFSD: laundromat service - starting
  Jun 20 12:27:47 nfsserver kernel: NFSD: laundromat_main - sleeping for 90 seconds
  • According to the customer, he is migrating the RHEL server from old server. It worked correctly on the old server. We do not get information of nfs client, but maybe I think it is Linux since ls command is used.
  • Also, ls without '-l' option run correctly if mouting with -o nolock.
# mount -t nfs -o nolock 192.168.160.73:/home/user/csv /home/user/csv_TEST/

Resolution

  • According to the customer, this issue occurs only when a nfs client is Asianux 3 (miracle linux) and does not occur when a client is rhel3 or CentOS5.9. Therefore, it seems clients relate to the cause of this problem.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.