"rhui-manager" throws error while syncing all repositories on RHUA server, even though entitlement certificate is valid

Issue

• How to generate and download latest "Content / Entitlement" certificate for RHUA server from customer portal ?

• rhui-manager fails to sync repositories

  -= Repository Synchronization Status =-
  Last Refreshed: 12:24:55
  
  Next Sync                    Last Sync                    Last Result         
  ------------------------------------------------------------------------------
  Red Hat Enterprise Linux 5 Server - Supplementary from RHUI (Debug RPMs) (5Server-i386)
  09-23-2013 13:44             08-15-2013 13:44             Error      
  
  Red Hat Enterprise Linux 5 Server - Supplementary from RHUI (Debug RPMs) (5Server-x86_64)
  09-23-2013 13:44             08-15-2013 13:44             Error      
  
  Red Hat Enterprise Linux 5 Server - Supplementary from RHUI (RPMs) (5Server-i386)
  09-23-2013 13:44             08-15-2013 13:46             Error      
  
  Red Hat Enterprise Linux 5 Server - Supplementary from RHUI (RPMs) (5Server-x86_64)
  09-23-2013 13:44             08-15-2013 13:47             Error      
  
  Red Hat Enterprise Linux 5 Server - Supplementary from RHUI (Source RPMs) (5Server-i386)
  09-23-2013 13:44             08-15-2013 13:45             Error      
  
  Red Hat Enterprise Linux 5 Server - Supplementary from RHUI (Source RPMs) (5Server-x86_64)
  09-23-2013 13:44             08-15-2013 13:45             Error      
  
  Red Hat Enterprise Linux 5 Server from RHUI (Debug RPMs) (5Server-i386)
  09-23-2013 17:44             08-15-2013 11:52             Error      
  
  Red Hat Enterprise Linux 5 Server from RHUI (Debug RPMs) (5Server-x86_64)
  09-23-2013 17:44             08-15-2013 11:55             Error      
  
  Red Hat Enterprise Linux 5 Server from RHUI (RPMs) (5Server-i386)
  09-23-2013 17:37             08-15-2013 18:00             Error      
  
  Red Hat Enterprise Linux 5 Server from RHUI (RPMs) (5Server-x86_64)
  09-23-2013 17:37             08-15-2013 18:06             Error      
  

• After upgrading to RHUI 4.x, syncing repositories from CDN caused error:

  --snip--
  Aug  5 03:28:10 prodrhua pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): cdn.redhat.com
  Aug  5 03:28:11 prodrhua pulp: pulp.server.async.tasks:INFO: [2a2873d8] Task failed : [1a4321d8-4d8f-48d0-87878-8319a71a72e9] : Error retrieving metadata: Forbidden
  --snip--
  

• Below error message seen in /root/.rhui/rhui.log

  Connecting to RHUA [rhua.example.com]...
  Successfully connected to [rhua.example.com]
  This certificate is invalid or expired.
  None
  This certificate is invalid or expired.
  None
  Loading the dict with registered subscriptions
  

• A sudden repository sync failure has occurred, and the issue is ongoing.

  /var/log/messages in rhua.
  
  Nov  1 15:55:05 rhua4 pulpcore-worker[XXX]: pulp [YYY]: pulpcore.tasking.tasks:INFO: Starting task ZZZ
  Nov  1 15:55:05 rhua4 pulpcore-worker[XXX]: pulp [YYY]: pulp_rpm.app.tasks.synchronizing:INFO: Synchronizing: repository=rhel-7-server-rhui-rpms-7Server-x86_64 remote=rhel-7-server-rhui-rpms-7Server-x86_64
  Nov  1 15:55:05 rhua4 pulpcore-worker[XXX]: Giving up download_wrapper(...) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 403, message='Forbidden', url=URL('https://cdn.redhat.com/content/dist/rhel/rhui/server/7/7Server/x86_64/os/repodata/repomd.xml'))
  Nov  1 15:55:05 rhua4 pulpcore-worker[XXX]: pulp [YYY]: backoff:ERROR: Giving up download_wrapper(...) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 403, message='Forbidden', url=URL('https://cdn.redhat.com/content/dist/rhel/rhui/server/7/7Server/x86_64/os/repodata/repomd.xml'))
  Nov  1 15:55:05 rhua4 pulpcore-worker[XXX]: pulp [YYY]: pulp_rpm.app.tasks.synchronizing:INFO: Attempting to resolve a true url from potential mirrolist url 'https://cdn.redhat.com/content/dist/rhel/rhui/server/7/7Server/x86_64/os'
  Nov  1 15:55:05 rhua4 pulpcore-worker[XXX]: Giving up download_wrapper(...) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 403, message='Forbidden', url=URL('https://cdn.redhat.com/content/dist/rhel/rhui/server/7/7Server/x86_64/os'))
  Nov  1 15:55:05 rhua4 pulpcore-worker[XXX]: pulp [YYY]: backoff:ERROR: Giving up download_wrapper(...) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 403, message='Forbidden', url=URL('https://cdn.redhat.com/content/dist/rhel/rhui/server/7/7Server/x86_64/os'))
  Nov  1 15:55:05 rhua4 pulpcore-worker[XXX]: pulp [YYY]: pulpcore.tasking.tasks:INFO: Task ZZZ failed (403, message='Forbidden', url=URL('https://cdn.redhat.com/content/dist/rhel/rhui/server/7/7Server/x86_64/os'))
  Nov  1 15:55:05 rhua4 pulpcore-worker[XXX]: pulp [YYY]: pulpcore.tasking.tasks:INFO:   File "/usr/lib/python3.9/site-packages/pulpcore/tasking/tasks.py", line 66, in _execute_task
  Nov  1 15:55:05 rhua4 pulpcore-worker[XXX]:    result = func(*args, **kwargs)
  Nov  1 15:55:05 rhua4 pulpcore-worker[XXX]:  File "/usr/lib/python3.9/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 487, in synchronize
  Nov  1 15:55:05 rhua4 pulpcore-worker[XXX]:    remote_url = fetch_remote_url(remote, url)
  Nov  1 15:55:05 rhua4 pulpcore-worker[XXX]:  File "/usr/lib/python3.9/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 306, in fetch_remote_url
  Nov  1 15:55:05 rhua4 pulpcore-worker[XXX]:    remote_url = fetch_mirror(remote)
  Nov  1 15:55:05 rhua4 pulpcore-worker[XXX]:  File "/usr/lib/python3.9/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 255, in fetch_mirror
  Nov  1 15:55:05 rhua4 pulpcore-worker[XXX]:    result = downloader.fetch()
  Nov  1 15:55:05 rhua4 pulpcore-worker[XXX]:  File "/usr/lib/python3.9/site-packages/pulpcore/download/base.py", line 185, in fetch
  Nov  1 15:55:05 rhua4 pulpcore-worker[XXX]:    result = asyncio.get_event_loop().run_until_complete(self.run())
  Nov  1 15:55:05 rhua4 pulpcore-worker[XXX]:  File "/usr/lib64/python3.9/asyncio/base_events.py", line 647, in run_until_complete
  Nov  1 15:55:05 rhua4 pulpcore-worker[XXX]:    return future.result()
  Nov  1 15:55:05 rhua4 pulpcore-worker[XXX]:  File "/usr/lib/python3.9/site-packages/pulpcore/download/http.py", line 273, in run
  Nov  1 15:55:05 rhua4 pulpcore-worker[XXX]:    return await download_wrapper()
  Nov  1 15:55:05 rhua4 pulpcore-worker[XXX]:  File "/usr/lib/python3.9/site-packages/backoff/_async.py", line 151, in retry
  Nov  1 15:55:05 rhua4 pulpcore-worker[XXX]:    ret = await target(*args, **kwargs)
  Nov  1 15:55:05 rhua4 pulpcore-worker[XXX]:  File "/usr/lib/python3.9/site-packages/pulpcore/download/http.py", line 258, in download_wrapper
  Nov  1 15:55:05 rhua4 pulpcore-worker[XXX]:    return await self._run(extra_data=extra_data)
  Nov  1 15:55:05 rhua4 pulpcore-worker[XXX]:  File "/usr/lib/python3.9/site-packages/pulp_rpm/app/downloaders.py", line 117, in _run
  Nov  1 15:55:05 rhua4 pulpcore-worker[XXX]:    self.raise_for_status(response)
  Nov  1 15:55:05 rhua4 pulpcore-worker[XXX]:  File "/usr/lib/python3.9/site-packages/pulp_rpm/app/downloaders.py", line 102, in raise_for_status
  Nov  1 15:55:05 rhua4 pulpcore-worker[XXX]:    response.raise_for_status()
  Nov  1 15:55:05 rhua4 pulpcore-worker[XXX]:  File "/usr/lib64/python3.9/site-packages/aiohttp/client_reqrep.py", line 1004, in raise_for_status
  Nov  1 15:55:05 rhua4 pulpcore-worker[XXX]:    raise ClientResponseError(