Creation of cinder volume from snapshot with Netapp NFS volume backend fails with 'chmod permission error'

Issue

• Cinder volume creation is failing with permission denied error:

  Stdout: u"Unexpected error while running command.\nCommand: chmod 660 /path/to/volume\nExit code: 1\n
  Stdout: u''\nStderr: 'chmod: changing permissions of /path/to/volume: Permission denied\\n'"
  

Reproducer Steps:

First method:

• Create a cinder volume and verify in NFS filesystem that cinder volume is created. It's showing the ownership cinder:cinder.

# cinder create --display_name vol9_test 10

# ls -ld /var/lib/cinder/mnt/afc4b156a118a2e05xxxxxxxxxxxx/volume-5557f96b-f62c-429d-xxxx-xxxxxxxxx
-rw-rw----. 1 cinder cinder 21474836480 Aug 26 08:41 /var/lib/cinder/mnt/afc4b156a118a2e05xxxxxxxxxxxx/volume-5557f96b-f62c-429d-xxxx-xxxxxxxxx

• Create a new instance and attach a cinder volume to the instance. Ownership of the volume is changed to root:root from cinder:cinder.

# nova --insecure boot --flavor m1.small --key_name test --image 94359bb4-38d2-48c3-a703-892f4aaa4f05 --nic net-id=364782c5-69e7-44ab-9470-xxxxxxxxxxx instance_test

# ls -ld /var/lib/cinder/mnt/afc4b156a118a2e05xxxxxxxxxxxx/volume-5557f96b-f62c-429d-xxxx-xxxxxxxxx
-rw-rw----. 1 107 107 21474836480 Aug 26 08:41 /var/lib/cinder/mnt/afc4b156a118a2e05xxxxxxxxxxxx/volume-5557f96b-f62c-429d-xxxx-xxxxxxxxx

• Stop the instance and detach a volume from instance. Ownership of volume is still root:root.

# nova stop instance_test
Request to stop server instance_test has been accepted.

# nova volume-detach instance_test 5557f96b-f62c-429d-xxxx-xxxxxxxxx

# ls -ld /var/lib/cinder/mnt/afc4b156a118a2e05xxxxxxxxxxxx/volume-5557f96b-f62c-429d-xxxx-xxxxxxxxx
-rw-rw----. 1 root root 21474836480 Aug 26 08:41 /var/lib/cinder/mnt/afc4b156a118a2e05xxxxxxxxxxxx/volume-5557f96b-f62c-429d-xxxx-xxxxxxxxx

• Create a snapshot of the volume. Snapshot is generated with root:root ownership similar to cinder volume ownership from last step.

# cinder snapshot-create --display-name snap_vol10_test vol10_test

# cinder snapshot-list
|                  ID                  |              Volume ID               |   Status  |            Display Name            | Size |
| 9e8cf73c-c414-4aa4-b45d-19975cf4d7a7 | 5f973d6b-159f-4d28-9161-b55adda44017 | available |           snap_vol10_test           |  20  |

# ls -ld /var/lib/cinder/mnt/afc4b156a118a2e05xxxxxxxxxxxx/snapshot-14951e67-2ab7-498b-aad9-5a19a011b476
-rw-rw----. 1 root root 21474836480 Aug 26 09:40 /var/lib/cinder/mnt/afc4b156a118a2e05xxxxxxxxxxxx/snapshot-14951e67-2ab7-498b-aad9-5a19a011b476

• Trying to generate a new cinder volume from snapshot is getting failed.

# cinder --insecure create --snapshot-id 14951e67-2ab7-498b-aad9-5a19a011b476 --display-name vol_test_11 20

Following message reported in /var/log/cinder/volume.log of controller node because it's not able to change the ownership from root:root to cinder:cinder.

2016-08-26 09:54:06.488 17049 TRACE oslo_messaging.rpc.dispatcher Stderr: 'chmod: changing permissions of \xe2\x80\x98/var/lib/cinder/mnt/afc4b156a118a2e05xxxxxxxxxxxx/volume-c9ddafc5-9215-457e-a846-0bcf8ed215ac\xe2\x80\x99: Permission denied\n'

Second method:

• Create a new instance with "Boot from image (creates a new volume)".
• Snapshot from created volume in step before with --force option because this volume is in use for the instance.
• Create a new instance from this snapshot with "Boot from volume snapshot (creates a new volume)"

Everytime while creating cinder volume from snapshot is getting failed because of permission issue.