Timestamp is incorrect when creating files in VMware vSAN NFS server

Solution Unverified - Updated -

Environment

  • VMware vSAN (NFS server): not sure which specific versions are affected
  • RedHat Enterprise Linux (RHEL) 8 and 9 (NFS client)
  • NFS version >= 4.1

Issue

  • When creating a new file in a NFS share coming from a vSAN NFS server the access and modify timestamp of the file are incorrect.

  • This only happens when the timestamp of the file is not explicitly set by the command that was run to create the file:

    cp foo bar     # bar has incorrect timestamp
cp -p foo bar  # bar has a correct timestamp since `-p` makes copy set the timestamp of bar to the same value of foo
touch baz      # baz has correct timestamp since touch explicitly sets the timestamp after creating the file

Resolution

  • Solution: The problem is caused by a bug in the NFS server and there is not a client side solution to the problem at this point
  • Workaround: force the application to explicitly set the timestamp after file creation
  • Workaround: problem shouldn't be present if the NFS protocol is downgraded to use version 4.0 since this version does not support EXCLUSIVE4_1

Root Cause

  • The problem is caused by the NFS server incorrectly setting the Time_Access and Time_Modify in the suppattr_exclcreat attribute, but then using them to store the verifier in EXCLUSIVE4_1 open calls.

  • Per RFC 8111 Section 18.16.3 the attributes set in suppattr_exclcreat shouldn't be used to store the verifier, since this will make the client unable to know for sure where the verifier was stored:

    ...
    In NFSv4.1, EXCLUSIVE4 has been deprecated in favor of EXCLUSIVE4_1. Unlike EXCLUSIVE4, attributes may be provided in the EXCLUSIVE4_1 case, but because the server may use attributes of the target object to store the verifier, the set of allowable attributes may be fewer than the set of attributes SETATTR allows. The allowable attributes for EXCLUSIVE4_1 are indicated in the suppattr_exclcreat (Section 5.8.1.14) attribute. If the client attempts to set in cva_attrs an attribute that is not in suppattr_exclcreat, the server MUST return NFS4ERR_INVAL. The response field, attrset, indicates both which attributes the server set from cva_attrs and which attributes the server used to store the verifier. As described in Section 18.16.4, the client can compare cva_attrs.attrmask with attrset to determine which attributes were used to store the verifier.
    ...

  • The NFS protocol requires the NFS client to set the values of the attributes where the verifier was stored (RFC 8111 Section 18.16.4):

    After the client has performed a successful exclusive create, the attrset response indicates which attributes were used to store the verifier. If EXCLUSIVE4 was used, the attributes set in attrset were used for the verifier. If EXCLUSIVE4_1 was used, the client determines the attributes used for the verifier by comparing attrset with cva_attrs.attrmask; any bits set in the former but not the latter identify the attributes used to store the verifier. The client MUST immediately send a SETATTR to set attributes used to store the verifier. Until it does so, the attributes used to store the verifier cannot be relied upon. The subsequent SETATTR MUST NOT occur in the same COMPOUND request as the OPEN.

  • But Since Time_Access and Time_Modify appear in suppattr_exclcreat the RedHat NFS client assumes this fields can't contain the verifier, and does not issue the subsequent SETATTR.

  • Since the SETATTR was never received Time_Access and Time_Modify continue to store the verifier value which produce the incorrect timestamp when interpreted as date.

Diagnostic Steps

Limited diagnostic:

  • Create a new file on the server and check if the access and modify timestamps are incorrect:

    $ cd <NFS share>
$ cp /usr/share/fedora-logos/fedora_logo.svg .
$ stat fedora_logo.svg
  ...
Access: 1992-02-21 18:42:39.000000000 +0100   # <<<
Modify: 2084-07-33 17:00:01.000000000 +0100   # <<<
Change: 2024-11-22 09:59:31.379002396 +0100
 Birth: 2024-11-22 09:59:31.341002394 +0100

Full diagnostic:

  • Unmount the problematic network share

  • Start network capture for traffic going to the problematic NFS server

    sudo tcpdump -i <interfact> host <NFS server> and port 2049

  • Mount the NFS share again and create a file:

    $ cd <NFS share>
$ cp /usr/share/fedora-logos/fedora_logo.svg .

  • Stop the network capture

  • Check the suppattr_exclcreat value send by the server to see if it contains Time_Access and Time_Modify

    $ tshark -r <capture file> -Y 'nfs and rpc.msgtyp == 1 and nfs.attr == 75' -O nfs

...
Remote Procedure Call, Type:Reply XID:0xabcdef77
Network File System, Ops(3): SEQUENCE PUTFH GETATTR
    [Program Version: 4]
    [V4 Procedure: COMPOUND (1)]
    Status: NFS4_OK (0)
    Tag: <EMPTY>
    Operations (count: 3)
        Opcode: SEQUENCE (53)
        Opcode: PUTFH (22)
        Opcode: GETATTR (9)
            Status: NFS4_OK (0)
            Attr mask[0]: 0x00002065 (Supported_Attrs, FH_Expire_Type, Link_Support, Symlink_Support, ACLSupport)
            Attr mask[2]: 0x00000800 (Suppattr_ExclCreat)
                reqd_attr: Suppattr_ExclCreat (75)
                    Attr mask[0]: 0xfdffbfff (Supported_Attrs, Type, FH_Expire_Type, Change, Size, Link_Support, Symlink_Support, Named_Attr, FSID, Unique_Handles, Lease_Time, RDAttr_Error, ACL, ACLSupport, CanSetTime, Case_Insensitive, Case_Preserving, Chown
                    Attr mask[1]: 0x40b8be3e (Mode, No_Trunc, NumLinks, Owner, Owner_Group, RawDev, Space_Avail, Space_Free, Space_Total, Space_Used, Time_Access, Time_Delta, Time_Metadata, Time_Modify, Mounted_on_FileId, FS_Layout_Type)
                        ...
                        reco_attr: Time_Access (47)             <<<<
                        ...
                        reco_attr: Time_Modify (53)             <<<< 
                        ...
                    Attr mask[2]: 0x00050802 (Layout_blksize, Suppattr_ExclCreat, Security_Label, Xattr_Support)
    [Main Opcode: GETATTR (9)]
...

  • Check if the server stored the verifier in the Time_Access and Time_Modify attributes during the open:

    $ tshark -r 0050-03949234-1016.tar.gz/03949234-1016/tcpdump.pcap -Y 'nfs and frame.number in {341..342}' -O nfs    # Only relevant subtree shown

...
Remote Procedure Call, Type:Call XID:0xaaaaaaaa
Network File System, Ops(5): SEQUENCE, PUTFH, OPEN, GETFH, GETATTR
    [Program Version: 4]
    [V4 Procedure: COMPOUND (1)]
    Tag: <EMPTY>
    minorversion: 1
    Operations (count: 5): SEQUENCE, PUTFH, OPEN, GETFH, GETATTR
        Opcode: SEQUENCE (53)
        Opcode: PUTFH (22)
        Opcode: OPEN (18)
            ...
            Open Type: OPEN4_CREATE (1)
                Create Mode: EXCLUSIVE4_1 (3)
                verifier: 0xf2ca1bb6c7e90700       <<< SEE THE VERIFIER
                Attr mask: 0x00000002 (Mode)
                    reco_attr: Mode (33)
                        mode: 0644, Name: Unknown, Read permission for owner, Write permission for owner, Read permission for group, Read permission for others
            ...
        Opcode: GETFH (10)
        Opcode: GETATTR (9)
    [Main Opcode: OPEN (18)]

...
Remote Procedure Call, Type:Reply XID:0xaaaaaaaa
Network File System, Ops(5): SEQUENCE PUTFH OPEN GETFH GETATTR
    [Program Version: 4]
    [V4 Procedure: COMPOUND (1)]
    Status: NFS4_OK (0)
    Tag: <EMPTY>
    Operations (count: 5)
        Opcode: SEQUENCE (53)
        Opcode: PUTFH (22)
        Opcode: OPEN (18)
            ...
            Attr mask: 0x00208000 (Time_Access, Time_Modify)     <<<<
                reco_attr: Time_Access (47)
                reco_attr: Time_Modify (53)
            ...
        Opcode: GETFH (10)
        Opcode: GETATTR (9)
            Status: NFS4_OK (0)
            Attr mask[0]: 0x0010011a (Type, Change, Size, FSID, FileId)
            Attr mask[1]: 0x00b0a23a (Mode, NumLinks, Owner, Owner_Group, RawDev, Space_Used, Time_Access, Time_Metadata, Time_Modify, Mounted_on_FileId)
                ...
                reco_attr: Time_Access (47)
                    seconds: 3055274738     #  <<<< 0xb61bcaf2 first part of the verifier
                    nseconds: 0
                ...
                reco_attr: Time_Modify (53)
                    seconds: 518599         #  <<<< 0x7e9c7 second part of the verifier
                    nseconds: 0
                ...
    [Main Opcode: OPEN (18)]

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments