Attaching of manually provisioned local storage in vSphere fails with "is not registered as a CNS volume"

Issue

• After manually creating a vSphere local volume using govc and when trying to attacah it to a Pod, this fails with the following error message (The input volume <ID> is not registered as a CNS volume):

  AttachVolume.Attach failed for volume "example" : rpc error: code = Internal desc = failed to attach disk: "15459512-92ee-48a5-8ebd-e1cda1d7ad4d" with node: "8b84586d-068c-4bdc-b27c-c73e3362a77f" err failed to attach cns volume: "15459512-92ee-48a5-8ebd-e1cda1d7ad4d" to node vm: "VirtualMachine:vm-12345 [VirtualCenterHost: hypervisor01.example.com, UUID: 8b84586d-068c-4bdc-b27c-c73e3362a77f, Datacenter: Datacenter [Datacenter: Datacenter:datacenter-1, VirtualCenterHost: host.example.com]]". fault: "(*types.LocalizedMethodFault)(0xc00137ae00)({\n DynamicData: (types.DynamicData) {\n },\n Fault: (types.BaseMethodFault) <nil>,\n LocalizedMessage: (string) (len=188) \"Volume with ID 15459512-92ee-48a5-8ebd-e1cda1d7ad4d is not registered as CNS Volume. Error message: The input volume 15459512-92ee-48a5-8ebd-e1cda1d7ad4d is not registered as a CNS volume.\"\n})\n". opId: "4e4ae6a5")
  

• The vsphere-syncer container in the vmware-vsphere-csi-driver-controller Pod in the openshift-cluster-csi-drivers namespace shows the following errors:

  2024-05-27T11:29:19.652935001Z {"level":"info","time":"2024-05-27T11:29:19.652862237Z","caller":"syncer/metadatasyncer.go:1846","msg":"PVUpdated: Verified volume: \"37df7156-4504-47ec-a69d-bf597d436287\" is not marked as container volume in CNS. Calling CreateVolume with BackingID to mark volume as Container Volume.","TraceId":"3367ba6e-01af-4f67-ade8-119945476224"}
  2024-05-27T11:29:19.663138043Z {"level":"error","time":"2024-05-27T11:29:19.663049683Z","caller":"volume/util.go:242","msg":"CNS CreateVolume failed from vCenter \"example.obfuscated.com\" with err: ServerFaultCode: The object or item referred to could not be found." [..]
  [..]
  2024-05-27T11:29:19.663188215Z {"level":"error","time":"2024-05-27T11:29:19.663149845Z","caller":"syncer/metadatasyncer.go:1875","msg":"PVUpdated: Failed to create disk xxxxxxxxxxxxxxxxxxxxx-e820000004002a-csitest with error ServerFaultCode: The object or item referred to could not be found."