Minor update failed on first controller

Issue

• When performing a minor update from 16.1 to 16.1.3, the openstack overcloud update run --stack overcloud --limit Controller --playbook allstep failed. A custom Horizon container image is in use, it was discovered that this image had an incorrect prefix. Even after correcting this and re-running the update run command, the deploy failed with this error:

Ansible failed, check log at /var/log/containers/mistral/package_update.log.
2021-02-18 14:41:12.573 580556 ERROR tripleoclient.v1.overcloud_update.MinorUpdateRun [-] Exception occured while running the command: RuntimeError: Update failed with: Ansible failed, check log at /var/log/containers/mistral/package_update.log.
2021-02-18 14:41:12.573 580556 ERROR tripleoclient.v1.overcloud_update.MinorUpdateRun Traceback (most recent call last):
2021-02-18 14:41:12.573 580556 ERROR tripleoclient.v1.overcloud_update.MinorUpdateRun   File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 32, in run
2021-02-18 14:41:12.573 580556 ERROR tripleoclient.v1.overcloud_update.MinorUpdateRun     super(Command, self).run(parsed_args)
2021-02-18 14:41:12.573 580556 ERROR tripleoclient.v1.overcloud_update.MinorUpdateRun   File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run
2021-02-18 14:41:12.573 580556 ERROR tripleoclient.v1.overcloud_update.MinorUpdateRun     return super(Command, self).run(parsed_args)
2021-02-18 14:41:12.573 580556 ERROR tripleoclient.v1.overcloud_update.MinorUpdateRun   File "/usr/lib/python3.6/site-packages/cliff/command.py", line 185, in run
2021-02-18 14:41:12.573 580556 ERROR tripleoclient.v1.overcloud_update.MinorUpdateRun     return_code = self.take_action(parsed_args) or 0
2021-02-18 14:41:12.573 580556 ERROR tripleoclient.v1.overcloud_update.MinorUpdateRun   File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_update.py", line 219, in take_action
2021-02-18 14:41:12.573 580556 ERROR tripleoclient.v1.overcloud_update.MinorUpdateRun     priv_key=key)
2021-02-18 14:41:12.573 580556 ERROR tripleoclient.v1.overcloud_update.MinorUpdateRun   File "/usr/lib/python3.6/site-packages/tripleoclient/utils.py", line 1369, in run_update_ansible_action
2021-02-18 14:41:12.573 580556 ERROR tripleoclient.v1.overcloud_update.MinorUpdateRun     verbosity=verbosity, extra_vars=extra_vars)
2021-02-18 14:41:12.573 580556 ERROR tripleoclient.v1.overcloud_update.MinorUpdateRun   File "/usr/lib/python3.6/site-packages/tripleoclient/workflows/package_update.py", line 127, in update_ansible
2021-02-18 14:41:12.573 580556 ERROR tripleoclient.v1.overcloud_update.MinorUpdateRun     raise RuntimeError('Update failed with: {}'.format(payload['message']))
2021-02-18 14:41:12.573 580556 ERROR tripleoclient.v1.overcloud_update.MinorUpdateRun RuntimeError: Update failed with: Ansible failed, check log at /var/log/containers/mistral/package_update.log.
2021-02-18 14:41:12.573 580556 ERROR tripleoclient.v1.overcloud_update.MinorUpdateRun 
2021-02-18 14:41:12.575 580556 ERROR openstack [-] Update failed with: Ansible failed, check log at /var/log/containers/mistral/package_update.log.: RuntimeError: Update failed with: Ansible failed, check log at /var/log/containers/mistral/package_update.log.
2021-02-18 14:41:12.576 580556 INFO osc_lib.shell [-] END return value: 1

• At this point the first controller has been updated but the other two are still running the older packages and container images.