NFS locking fails with "lockd: unexpected unlock status: 9 or 150994944"

Solution Unverified - Updated -

Environment

  • Red Hat Enterprise Linux 6 - NFS Client
  • Kernel 2.6.32-573
  • NetApp Clustered C-Mode Data ONTap (CDOT) - NFS Server
  • NFS Network File System v3

Issue

  • The following messages are printed to /var/log/messages periodically:
kernel: lockd: unexpected unlock status: 9
kernel: lockd: unexpected unlock status: 150994944

kernel: lockd: unexpected unlock status: 7
kernel: lockd: unexpected unlock status: 117440512

Resolution

There are two issues:

1) When a valid error is returned and why this error is returned, the resolution is unknown at this point.
It's suspected to be an issue within the NetApp filer operating system.

2) To fix the invalid error number (like 117440512 or 150994944), update kernel to 2.6.32-642.el6 or later to fix it.

Root Cause

  • When an invalid error number is logged, this was because of the network byte order;
        lockd: properly convert be32 values in debug messages

        lockd: server returns status 50331648
        it's quite hard to understand that number in this message is 3 in big endian

Diagnostic Steps

  • Decode the "9" error message:
kernel: lockd: unexpected unlock status: 9

include/linux/lockd/nlm.h
/* Return states for NLM */
enum {
        NLM_LCK_GRANTED                 = 0,
        NLM_LCK_DENIED                  = 1,
        NLM_LCK_DENIED_NOLOCKS          = 2,
        NLM_LCK_BLOCKED                 = 3,
        NLM_LCK_DENIED_GRACE_PERIOD     = 4,
#ifdef CONFIG_LOCKD_V4
        NLM_DEADLCK                     = 5,
        NLM_ROFS                        = 6,
        NLM_STALE_FH                    = 7,
        NLM_FBIG                        = 8,
        NLM_FAILED                      = 9,
#endif
};

This implies the Network Lock Manager failed.

  • Decode the "150994944" error message
kernel: lockd: unexpected unlock status: 150994944

The hexadecimal representation of 150994944 is 0x 9000 0000.

It's likely this means the same thing (ie: 9), the network byte order is just around the wrong way.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments