Every other image upload fails with "410 Gone Error in store configuration. adding images to store is disabled."

Issue

• We are having consistent upload errors when trying to upload images. We are getting the error:

410 Gone
Error in store configuration. Adding images to store is disabled.
    (HTTP N/A)

• Typically, we just try the upload again and it succeeds, then we delete the failed image and life goes on. However, we need a permanent solution.

• Also, on failures, the /var/log/glance/api.log shows:

2020-03-03 16:45:36.471 3453 ERROR glance_store._drivers.filesystem [-] Permission to write in /var/lib/glance/images/ denied
2020-03-03 16:45:36.471 3453 ERROR glance_store._drivers.filesystem None
2020-03-03 16:45:36.471 3453 ERROR glance_store._drivers.filesystem 
2020-03-03 16:45:36.472 3453 WARNING glance_store.driver [-] Failed to configure store correctly: Store filesystem could not be configured correctly. Reason: Permission to write in /var/lib/glance/images/ denied Disabling add method.

• Upload is failing with :

[stack@undercloud ~]$ date; time glance image-create --name Testing-23-Test-Image1 --file ~/tenant-images/itte.CentOS-7-x86_64-GenericCloud-2003.qcow2 --disk-format qcow2 - -container-format bare --progress
Thu Jul  9 14:51:04 UTC 2020
[=============================>] 100%
+------------------+--------------------------------------+
| Property         | Value                                |
+------------------+--------------------------------------+
| checksum         | None                                 |
| container_format | bare                                 |
| created_at       | 2020-07-09T14:50:58Z                 |
| disk_format      | qcow2                                |
| id               | 525a4c2b-b02d-482a-9138-71a83b527d2e |
| min_disk         | 0                                    |
| min_ram          | 0                                    |
| name             | Testing-23-Test-Image1               |
| owner            | 87e55c90ba20400e866480f5a989d52a     |
| protected        | False                                |
| size             | None                                 |
| status           | queued                               |
| tags             | []                                   |
| updated_at       | 2020-07-09T14:50:58Z                 |
| virtual_size     | None                                 |
| visibility       | private                              |
+------------------+--------------------------------------+
410 Gone
Error in store configuration. Adding images to store is disabled.
    (HTTP N/A)

real    0m6.312s
user    0m2.537s
sys     0m2.354s

• Sometimes it works:

[stack@undercloud ~]$ date; time glance image-create --name Testing-23-Test-Image1 --file ~/tenant-images/itte.CentOS-7-x86_64-GenericCloud-2003.qcow2 --disk-format qcow2 - -container-format bare --progress
Thu Jul  9 14:51:13 UTC 2020
[=============================>] 100%
+------------------+--------------------------------------------------------------------+
| Property         | Value                                                              |
+------------------+--------------------------------------------------------------------+
| checksum         | 3dd858ad7c7448b6dd4ad43cd0ba241e                                   |
| container_format | bare                                                               |
| created_at       | 2020-07-09T14:51:07Z                                               |
| direct_url       | file:///var/lib/glance/images/abed753d-a195-484c-af58-02aadfffd6ee |
| disk_format      | qcow2                                                              |
| id               | abed753d-a195-484c-af58-02aadfffd6ee                               |
| min_disk         | 0                                                                  |
| min_ram          | 0                                                                  |
| name             | Testing-23-Test-Image1                                             |
| owner            | 87e55c90ba20400e866480f5a989d52a                                   |
| protected        | False                                                              |
| size             | 858783744                                                          |
| status           | active                                                             |
| tags             | []                                                                 |
| updated_at       | 2020-07-09T14:51:13Z                                               |
| virtual_size     | None                                                               |
| visibility       | private                                                            |
+------------------+--------------------------------------------------------------------+

real    0m8.831s
user    0m2.977s
sys     0m2.389s

• How can we eliminate the errors to make all file uploads consistent?

• All images are served from a NFS server .