swift authentication error when (re)deploying an overcloud with 404 Client Error: Not Found for url or 401 Client Error: Unauthorized for url in Red Hat OpenStack Platform

Issue

The deployment is stuck early on. When going to the stuck node and verifying the logs, the following shows up:

[root@overcloud-cephstorage-1 ~]# journalctl -u os-collect-config -f
-- Logs begin at Tue 2017-02-14 16:55:44 EST. --
Feb 14 17:32:17 overcloud-cephstorage-1.localdomain os-collect-config[2630]: /var/lib/os-collect-config/local-data not found. Skipping
Feb 14 17:32:17 overcloud-cephstorage-1.localdomain os-collect-config[2630]: No local metadata found (['/var/lib/os-collect-config/local-data'])
Feb 14 17:32:47 overcloud-cephstorage-1.localdomain os-collect-config[2630]: 404 Client Error: Not Found for url: http://192.0.2.1:8080/v1/AUTH_903fc547d61649f1b4ea75285bebb3b0/ov-efiddjx5xy-1-xerow2zifeni-CephStorage-aezu7vmtr6zs/7741efdc-cd1f-46b7-b50d-e2462a2bd308?temp_url_sig=4acc1a3f2d5490516afdfb7cc95778f00d0df024&temp_url_expires=2147483586
Feb 14 17:32:47 overcloud-cephstorage-1.localdomain os-collect-config[2630]: Source [request] Unavailable.
Feb 14 17:32:47 overcloud-cephstorage-1.localdomain os-collect-config[2630]: /var/lib/os-collect-config/local-data not found. Skipping
Feb 14 17:32:47 overcloud-cephstorage-1.localdomain os-collect-config[2630]: No local metadata found (['/var/lib/os-collect-config/local-data'])
Feb 14 17:33:18 overcloud-cephstorage-1.localdomain os-collect-config[2630]: 404 Client Error: Not Found for url: http://192.0.2.1:8080/v1/AUTH_903fc547d61649f1b4ea75285bebb3b0/ov-efiddjx5xy-1-xerow2zifeni-CephStorage-aezu7vmtr6zs/7741efdc-cd1f-46b7-b50d-e2462a2bd308?temp_url_sig=4acc1a3f2d5490516afdfb7cc95778f00d0df024&temp_url_expires=2147483586
Feb 14 17:33:18 overcloud-cephstorage-1.localdomain os-collect-config[2630]: Source [request] Unavailable.
Feb 14 17:33:18 overcloud-cephstorage-1.localdomain os-collect-config[2630]: /var/lib/os-collect-config/local-data not found. Skipping
Feb 14 17:33:18 overcloud-cephstorage-1.localdomain os-collect-config[2630]: No local metadata found (['/var/lib/os-collect-config/local-data'])

Alternatively, one may also get the following, if the authentication parameters are wrong, but the URL is correct. This, for example, may happen if during an upgrade from one OSP version to another, something goes wrong and the undercloud is reverted to a snapshot which was taken prior to the start of the upgrade process.

    Feb 01 21:51:19 overcloud-compute-3.localdomain os-collect-config[6025]: Source [request] Unavailable.
    Feb 01 21:51:19 overcloud-compute-3.localdomain os-collect-config[6025]: /var/lib/os-collect-config/local-data not found. Skipping
    Feb 01 21:51:19 overcloud-compute-3.localdomain os-collect-config[6025]: No local metadata found (['/var/lib/os-collect-config/local-data'])
    Feb 01 21:51:49 overcloud-compute-3.localdomain os-collect-config[6025]: 401 Client Error: Unauthorized for url: http://192.168.0.1:8080/v1/AUTH_3b0748ea36024be9870c8f37f562aa75/ov-st4paidvl5-3-kaata7gmiaro-NovaCompute-nvxqpu6i256z/259f7e00-11de-4107-9fe9-ac70b04fa80f?temp_url_sig=a03417ebccb74ea5a8958f2fca0f1b0a1a31113a&temp_url_expires=2147483587
    Feb 01 21:51:49 overcloud-compute-3.localdomain os-collect-config[6025]: Source [request] Unavailable.
    Feb 01 21:51:49 overcloud-compute-3.localdomain os-collect-config[6025]: /var/lib/os-collect-config/local-data not found. Skipping
    Feb 01 21:51:49 overcloud-compute-3.localdomain os-collect-config[6025]: No local metadata found (['/var/lib/os-collect-config/local-data'])
    Feb 01 21:52:20 overcloud-compute-3.localdomain os-collect-config[6025]: 401 Client Error: Unauthorized for url: http://192.168.0.1:8080/v1/AUTH_3b0748ea36024be9870c8f37f562aa75/ov-st4paidvl5-3-kaata7gmiaro-NovaCompute-nvxqpu6i256z/259f7e00-11de-4107-9fe9-ac70b04fa80f?temp_url_sig=a03417ebccb74ea5a8958f2fca0f1b0a1a31113a&temp_url_expires=2147483587
    Feb 01 21:52:20 overcloud-compute-3.localdomain os-collect-config[6025]: Source [request] Unavailable.
    Feb 01 21:52:20 overcloud-compute-3.localdomain os-collect-config[6025]: /var/lib/os-collect-config/local-data not found. Skipping
    Feb 01 21:52:20 overcloud-compute-3.localdomain os-collect-config[6025]: No local metadata found (['/var/lib/os-collect-config/local-data'])