Overcloud deployment failed while adding new compute host

Issue

• We were adding new compute 'compute-5' in the existing Openstack cluster and it failed.

• Ansible showed error on the compute hosts which were deployed earlier and running successfully without any issue. From the ansible error it shows something like - 'operation not supported':

ok: [overcloud-novacompute-1]
ok: [overcloud-novacompute-0]
FAILED - RETRYING: Wait for containers to start for step 3 using paunch (1182 retries left).
FAILED - RETRYING: Wait for containers to start for step 3 using paunch (1182 retries left).
FAILED - RETRYING: Wait for containers to start for step 3 using paunch (1181 retries left).
FAILED - RETRYING: Wait for containers to start for step 3 using paunch (1181 retries left).
fatal: [overcloud-novacompute-3]: FAILED! => {"ansible_job_id": "550578584937.40657", "attempts": 21, "changed": false, "cmd": "/home/tripleo-admin/.ansible/tmp/ansible-tmp-1595620234.1668646-62397654733318/AnsiballZ_paunch.py", "data": "", "finished": 1, "msg": "Traceback (most recent call last):\n  File \"/tmp/ansible_async_wrapper_payload_782cct13/__main__.py\", line 159, in _run_module\n  File \"/tmp/ansible_async_wrapper_payload_782cct13/__main__.py\", line 93, in _filter_non_json_lines\nValueError: No start of json char found\n", "stderr": "Removed /etc/systemd/system/multi-user.target.wants/tripleo_iscsid.service.\nRemoved /etc/systemd/system/timers.target.wants/tripleo_iscsid_healthcheck.timer.\nRemoved /etc/systemd/system/tripleo_iscsid.service.requires/tripleo_iscsid_healthcheck.timer.\nRemoved /etc/systemd/system/multi-user.target.wants/tripleo_nova_libvirt.service.\nRemoved /etc/systemd/system/timers.target.wants/tripleo_nova_libvirt_healthcheck.timer.\nRemoved /etc/systemd/system/tripleo_nova_libvirt.service.requires/tripleo_nova_libvirt_healthcheck.timer.\nRemoved /etc/systemd/system/multi-user.target.wants/tripleo_nova_virtlogd.service.\nRemoved /etc/systemd/system/timers.target.wants/tripleo_nova_virtlogd_healthcheck.timer.\nRemoved /etc/systemd/system/tripleo_nova_virtlogd.service.requires/tripleo_nova_virtlogd_healthcheck.timer.\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=nova_statedir_owner', '--filter', 'label=config_id=tripleo_step3', '--format', '{{.Names}}']\" - retrying without config_id\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=nova_statedir_owner', '--format', '{{.Names}}']\"\nError running ['podman', 'run', '--name', 'nova_statedir_owner', '--label', 'config_id=tripleo_step3', '--label', 'container_name=nova_statedir_owner', '--label', 'managed_by=tripleo-Compute', '--label', 'config_data={\"command\": \"/container-config-scripts/pyshim.sh /container-config-scripts/nova_statedir_ownership.py\", \"detach\": false, \"environment\": {\"TRIPLEO_DEPLOY_IDENTIFIER\": \"1595618735\", \"__OS_DEBUG\": \"false\"}, \"image\": \"casadirector0.ctlplane.casa-systems.com:8787/rhosp-rhel8/openstack-nova-compute:16.0-107\", \"net\": \"none\", \"privileged\": false, \"user\": \"root\", \"volumes\": [\"/var/lib/nova:/var/lib/nova:shared,z\", \"/var/lib/container-config-scripts/:/container-config-scripts/:z\"]}', '--conmon-pidfile=/var/run/nova_statedir_owner.pid', '--log-driver', 'k8s-file', '--log-opt', 'path=/var/log/containers/stdouts/nova_statedir_owner.log', '--env=TRIPLEO_DEPLOY_IDENTIFIER=1595618735', '--env=__OS_DEBUG=false', '--net=none', '--privileged=false', '--user=root', '--volume=/var/lib/nova:/var/lib/nova:shared,z', '--volume=/var/lib/container-config-scripts/:/container-config-scripts/:z', '--cpuset-cpus=0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71', 'casadirector0.ctlplane.casa-systems.com:8787/rhosp-rhel8/openstack-nova-compute:16.0-107', '/container-config-scripts/pyshim.sh', '/container-config-scripts/nova_statedir_ownership.py']. [126]\n\nstdout: \nstderr: Error: relabel failed \"/var/lib/nova\": operation not supported\n\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=nova_virtlogd', '--filter', 'label=config_id=tripleo_step3', '--format', '{{.Names}}']\" - retrying without config_id\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=nova_virtlogd', '--format', '{{.Names}}']\"\nCreated symlink /etc/systemd/system/multi-user.target.wants/tripleo_nova_virtlogd.service → /etc/systemd/system/tripleo_nova_virtlogd.service.\nJob for tripleo_nova_virtlogd.service failed because the control process exited with error code.\nSee \"systemctl status tripleo_nova_virtlogd.service\" and \"journalctl -xe\" for details.\nJob for tripleo_nova_virtlogd.service failed because the control process exited with error code.\nSee \"systemctl status tripleo_nova_virtlogd.service\" and \"journalctl -xe\" for details.\nJob for tripleo_nova_virtlogd.service failed because the control process exited with error code.\nSee \"systemctl status tripleo_nova_virtlogd.service\" and \"journalctl -xe\" for details.\nJob for tripleo_nova_virtlogd.service failed because the control process exited with error code.\nSee \"systemctl status tripleo_nova_virtlogd.service\" and \"journalctl -xe\" for details.\nJob for tripleo_nova_virtlogd.service failed because the control process exited with error code.\nSee \"systemctl status tripleo_nova_virtlogd.service\" and \"journalctl -xe\" for details.\nsystemctl failed\nTraceback (most recent call last):\n  File \"/usr/lib/python3.6/site-packages/paunch/utils/systemctl.py\", line 32, in systemctl\n    subprocess.check_call(cmd)\n  File \"/usr/lib64/python3.6/subprocess.py\", line 311, in check_call\n    raise CalledProcessError(retcode, cmd)\nsubprocess.CalledProcessError: Command '['systemctl', 'enable', '--now', 'tripleo_nova_virtlogd']' returned non-zero exit status 1.\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/usr/lib/python3.6/site-packages/paunch/utils/systemctl.py\", line 75, in enable\n    systemctl(cmd, log)\n  File \"/usr/lib/python3.6/site-packages/paunch/utils/systemctl.py\", line 34, in systemctl\n    raise SystemctlException(str(err))\npaunch.utils.systemctl.SystemctlException: Command '['systemctl', 'enable', '--now', 'tripleo_nova_virtlogd']' returned non-zero exit status 1.\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/usr/lib/python3.6/site-packages/paunch/utils/systemd.py\", line 112, in service_create\n    systemctl.enable(service, now=True)\n  File \"/usr/lib/python3.6/site-packages/tenacity/__init__.py\", line 292, in wrapped_f\n    return self.call(f, *args, **kw)\n  File \"/usr/lib/python3.6/site-packages/tenacity/__init__.py\", line 358, in call\n    do = self.iter(retry_state=retry_state)\n  File \"/usr/lib/python3.6/site-packages/tenacity/__init__.py\", line 331, in iter\n    raise retry_exc.reraise()\n  File \"/usr/lib/python3.6/site-packages/tenacity/__init__.py\", line 167, in reraise\n    raise self.last_attempt.result()\n  File \"/usr/lib64/python3.6/concurrent/futures/_base.py\", line 425, in result\n    return self.__get_result()\n  File \"/usr/lib64/python3.6/concurrent/futures/_base.py\", line 384, in __get_result\n    raise self._exception\n  File \"/usr/lib/python3.6/site-packages/tenacity/__init__.py\", line 361, in call\n    result = fn(*args, **kwargs)\n  File \"/usr/lib/python3.6/site-packages/paunch/utils/systemctl.py\", line 79, in enable\n    raise SystemctlException(str(err))\npaunch.utils.systemctl.SystemctlException: Command '['systemctl', 'enable', '--now', 'tripleo_nova_virtlogd']' returned non-zero exit status 1.\nTraceback (most recent call last):\n  File \"/usr/lib/python3.6/site-packages/paunch/utils/systemctl.py\", line 32, in systemctl\n    subprocess.check_call(cmd)\n  File \"/usr/lib64/python3.6/subprocess.py\", line 311, in check_call\n    raise CalledProcessError(retcode, cmd)\nsubprocess.CalledProcessError: Command '['systemctl', 'enable', '--now', 'tripleo_nova_virtlogd']' returned non-zero exit status 1.\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/usr/lib/python3.6/site-packages/paunch/utils/systemctl.py\", line 75, in enable\n    systemctl(cmd, log)\n  File \"/usr/lib/python3.6/site-packages/paunch/utils/systemctl.py\", line 34, in systemctl\n    raise SystemctlException(str(err))\npaunch.utils.systemctl.SystemctlException: Command '['systemctl', 'enable', '--now', 'tripleo_nova_virtlogd']' returned non-zero exit status 1.\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/home/tripleo-admin/.ansible/tmp/ansible-tmp-1595620234.1668646-62397654733318/AnsiballZ_paunch.py\", line 114, in <module>\n    _ansiballz_main()\n  File \"/home/tripleo-admin/.ansible/tmp/ansible-tmp-1595620234.1668646-62397654733318/AnsiballZ_paunch.py\", line 106, in _ansiballz_main\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File \"/home/tripleo-admin/.ansible/tmp/ansible-tmp-1595620234.1668646-62397654733318/AnsiballZ_paunch.py\", line 49, in invoke_module\n    imp.load_module('__main__', mod, module, MOD_DESC)\n  File \"/usr/lib64/python3.6/imp.py\", line 235, in load_module\n    return load_source(name, filename, file)\n  File \"/usr/lib64/python3.6/imp.py\", line 170, in load_source\n    module = _exec(spec, sys.modules[name])\n  File \"<frozen importlib._bootstrap>\", line 618, in _exec\n  File \"<frozen importlib._bootstrap_external>\", line 678, in exec_module\n  File \"<frozen importlib._bootstrap>\", line 219, in _call_with_frames_removed\n  File \"/tmp/ansible_paunch_payload_yovazksi/__main__.py\", line 228, in <module>\n  File \"/tmp/ansible_paunch_payload_yovazksi/__main__.py\", line 224, in main\n  File \"/tmp/ansible_paunch_payload_yovazksi/__main__.py\", line 156, in __init__\n  File \"/tmp/ansible_paunch_payload_yovazksi/__main__.py\", line 179, in paunch_apply\n  File \"/usr/lib/python3.6/site-packages/paunch/__init__.py\", line 80, in apply\n    return builder.apply()\n  File \"/usr/lib/python3.6/site-packages/paunch/builder/base.py\", line 150, in apply\n    log=self.log)\n  File \"/usr/lib/python3.6/site-packages/paunch/utils/systemd.py\", line 112, in service_create\n    systemctl.enable(service, now=True)\n  File \"/usr/lib/python3.6/site-packages/tenacity/__init__.py\", line 292, in wrapped_f\n    return self.call(f, *args, **kw)\n  File \"/usr/lib/python3.6/site-packages/tenacity/__init__.py\", line 358, in call\n    do = self.iter(retry_state=retry_state)\n  File \"/usr/lib/python3.6/site-packages/tenacity/__init__.py\", line 331, in iter\n    raise retry_exc.reraise()\n  File \"/usr/lib/python3.6/site-packages/tenacity/__init__.py\", line 167, in reraise\n    raise self.last_attempt.result()\n  File \"/usr/lib64/python3.6/concurrent/futures/_base.py\", line 425, in result\n    return self.__get_result()\n  File \"/usr/lib64/python3.6/concurrent/futures/_base.py\", line 384, in __get_result\n    raise self._exception\n  File \"/usr/lib/python3.6/site-packages/tenacity/__init__.py\", line 361, in call\n    result = fn(*args, **kwargs)\n  File \"/usr/lib/python3.6/site-packages/paunch/utils/systemctl.py\", line 79, in enable\n    raise SystemctlException(str(err))\npaunch.utils.systemctl.SystemctlException: Command '['systemctl', 'enable', '--now', 'tripleo_nova_virtlogd']' returned non-zero exit status 1.\n", "stderr_lines": ["Removed /etc/systemd/system/multi-user.target.wants/tripleo_iscsid.service.", "Removed /etc/systemd/system/timers.target.wants/tripleo_iscsid_healthcheck.timer.", "Removed /etc/systemd/system/tripleo_iscsid.service.requires/tripleo_iscsid_healthcheck.timer.", "Removed /etc/systemd/system/multi-user.target.wants/tripleo_nova_libvirt.service.", "Removed /etc/systemd/system/timers.target.wants/tripleo_nova_libvirt_healthcheck.timer.", "Removed /etc/systemd/system/tripleo_nova_libvirt.service.requires/tripleo_nova_libvirt_healthcheck.timer.", "Removed /etc/systemd/system/multi-user.target.wants/tripleo_nova_virtlogd.service.", "Removed /etc/systemd/system/timers.target.wants/tripleo_nova_virtlogd_healthcheck.timer.", "Removed /etc/systemd/system/tripleo_nova_virtlogd.service.requires/tripleo_nova_virtlogd_healthcheck.timer.", "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=nova_statedir_owner', '--filter', 'label=config_id=tripleo_step3', '--format', '{{.Names}}']\" - retrying without config_id", "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=nova_statedir_owner', '--format', '{{.Names}}']\"", "Error running ['podman', 'run', '--name', 'nova_statedir_owner', '--label', 'config_id=tripleo_step3', '--label', 'container_name=nova_statedir_owner', '--label', 'managed_by=tripleo-Compute', '--label', 'config_data={\"command\": \"/container-config-scripts/pyshim.sh /container-config-scripts/nova_statedir_ownership.py\", \"detach\": false, \"environment\": {\"TRIPLEO_DEPLOY_IDENTIFIER\": \"1595618735\", \"__OS_DEBUG\": \"false\"}, \"image\": \"casadirector0.ctlplane.casa-systems.com:8787/rhosp-rhel8/openstack-nova-compute:16.0-107\", \"net\": \"none\", \"privileged\": false, \"user\": \"root\", \"volumes\": [\"/var/lib/nova:/var/lib/nova:shared,z\", \"/var/lib/container-config-scripts/:/container-config-scripts/:z\"]}', '--conmon-pidfile=/var/run/nova_statedir_owner.pid', '--log-driver', 'k8s-file', '--log-opt', 'path=/var/log/containers/stdouts/nova_statedir_owner.log', '--env=TRIPLEO_DEPLOY_IDENTIFIER=1595618735', '--env=__OS_DEBUG=false', '--net=none', '--privileged=false', '--user=root', '--volume=/var/lib/nova:/var/lib/nova:shared,z', '--volume=/var/lib/container-config-scripts/:/container-config-scripts/:z', '--cpuset-cpus=0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71', 'casadirector0.ctlplane.casa-systems.com:8787/rhosp-rhel8/openstack-nova-compute:16.0-107', '/container-config-scripts/pyshim.sh', '/container-config-scripts/nova_statedir_ownership.py']. [126]", "", "stdout: ", "stderr: Error: relabel failed \"/var/lib/nova\": operation not supported", "", "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=nova_virtlogd', '--filter', 'label=config_id=tripleo_step3', '--format', '{{.Names}}']\" - retrying without config_id", "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=nova_virtlogd', '--format', '{{.Names}}']\"", "Created symlink /etc/systemd/system/multi-user.target.wants/tripleo_nova_virtlogd.service → /etc/systemd/system/tripleo_nova_virtlogd.service.", "Job for tripleo_nova_virtlogd.service failed because the control process exited with error code.", "See \"systemctl status tripleo_nova_virtlogd.service\" and \"journalctl -xe\" for details.", "Job for tripleo_nova_virtlogd.service failed because the control process exited with error code.", "See \"systemctl status tripleo_nova_virtlogd.service\" and \"journalctl -xe\" for details.", "Job for tripleo_nova_virtlogd.service failed because the control process exited with error code.", "See \"systemctl status tripleo_nova_virtlogd.service\" and \"journalctl -xe\" for details.", "Job for tripleo_nova_virtlogd.service failed because the control process exited with error code.", "See \"systemctl status tripleo_nova_virtlogd.service\" and \"journalctl -xe\" for details.", "Job for tripleo_nova_virtlogd.service failed because the control process exited with error code.", "See \"systemctl status tripleo_nova_virtlogd.service\" and \"journalctl -xe\" for details.", "systemctl failed", "Traceback (most recent call last):", "  File \"/usr/lib/python3.6/site-packages/paunch/utils/systemctl.py\", line 32, in systemctl", "    subprocess.check_call(cmd)", "  File \"/usr/lib64/python3.6/subprocess.py\", line 311, in check_call", "    raise CalledProcessError(retcode, cmd)", "subprocess.CalledProcessError: Command '['systemctl', 'enable', '--now', 'tripleo_nova_virtlogd']' returned non-zero exit status 1.", "", "During handling of the above exception, another exception occurred:", "", "Traceback (most recent call last):", "  File \"/usr/lib/python3.6/site-packages/paunch/utils/systemctl.py\", line 75, in enable", "    systemctl(cmd, log)", "  File \"/usr/lib/python3.6/site-packages/paunch/utils/systemctl.py\", line 34, in systemctl", "    raise SystemctlException(str(err))", "paunch.utils.systemctl.SystemctlException: Command '['systemctl', 'enable', '--now', 'tripleo_nova_virtlogd']' returned non-zero exit status 1.", "", "During handling of the above exception, another exception occurred:", "", "Traceback (most recent call last):", "  File \"/usr/lib/python3.6/site-packages/paunch/utils/systemd.py\", line 112, in service_create", "    systemctl.enable(service, now=True)", "  File \"/usr/lib/python3.6/site-packages/tenacity/__init__.py\", line 292, in wrapped_f", "    return self.call(f, *args, **kw)", "  File \"/usr/lib/python3.6/site-packages/tenacity/__init__.py\", line 358, in call", "    do = self.iter(retry_state=retry_state)", "  File \"/usr/lib/python3.6/site-packages/tenacity/__init__.py\", line 331, in iter", "    raise retry_exc.reraise()", "  File \"/usr/lib/python3.6/site-packages/tenacity/__init__.py\", line 167, in reraise", "    raise self.last_attempt.result()", "  File \"/usr/lib64/python3.6/concurrent/futures/_base.py\", line 425, in result", "    return self.__get_result()", "  File \"/usr/lib64/python3.6/concurrent/futures/_base.py\", line 384, in __get_result", "    raise self._exception", "  File \"/usr/lib/python3.6/site-packages/tenacity/__init__.py\", line 361, in call", "    result = fn(*args, **kwargs)", "  File \"/usr/lib/python3.6/site-packages/paunch/utils/systemctl.py\", line 79, in enable", "    raise SystemctlException(str(err))", "paunch.utils.systemctl.SystemctlException: Command '['systemctl', 'enable', '--now', 'tripleo_nova_virtlogd']' returned non-zero exit status 1.", "Traceback (most recent call last):", "  File \"/usr/lib/python3.6/site-packages/paunch/utils/systemctl.py\", line 32, in systemctl", "    subprocess.check_call(cmd)", "  File \"/usr/lib64/python3.6/subprocess.py\", line 311, in check_call", "    raise CalledProcessError(retcode, cmd)", "subprocess.CalledProcessError: Command '['systemctl', 'enable', '--now', 'tripleo_nova_virtlogd']' returned non-zero exit status 1.", "", "During handling of the above exception, another exception occurred:", "", "Traceback (most recent call last):", "  File \"/usr/lib/python3.6/site-packages/paunch/utils/systemctl.py\", line 75, in enable", "    systemctl(cmd, log)", "  File \"/usr/lib/python3.6/site-packages/paunch/utils/systemctl.py\", line 34, in systemctl", "    raise SystemctlException(str(err))", "paunch.utils.systemctl.SystemctlException: Command '['systemctl', 'enable', '--now', 'tripleo_nova_virtlogd']' returned non-zero exit status 1.", "", "During handling of the above exception, another exception occurred:", "", "Traceback (most recent call last):", "  File \"/home/tripleo-admin/.ansible/tmp/ansible-tmp-1595620234.1668646-62397654733318/AnsiballZ_paunch.py\", line 114, in <module>", "    _ansiballz_main()", "  File \"/home/tripleo-admin/.ansible/tmp/ansible-tmp-1595620234.1668646-62397654733318/AnsiballZ_paunch.py\", line 106, in _ansiballz_main", "    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)", "  File \"/home/tripleo-admin/.ansible/tmp/ansible-tmp-1595620234.1668646-62397654733318/AnsiballZ_paunch.py\", line 49, in invoke_module", "    imp.load_module('__main__', mod, module, MOD_DESC)", "  File \"/usr/lib64/python3.6/imp.py\", line 235, in load_module", "    return load_source(name, filename, file)", "  File \"/usr/lib64/python3.6/imp.py\", line 170, in load_source", "    module = _exec(spec, sys.modules[name])", "  File \"<frozen importlib._bootstrap>\", line 618, in _exec", "  File \"<frozen importlib._bootstrap_external>\", line 678, in exec_module", "  File \"<frozen importlib._bootstrap>\", line 219, in _call_with_frames_removed", "  File \"/tmp/ansible_paunch_payload_yovazksi/__main__.py\", line 228, in <module>", "  File \"/tmp/ansible_paunch_payload_yovazksi/__main__.py\", line 224, in main", "  File \"/tmp/ansible_paunch_payload_yovazksi/__main__.py\", line 156, in __init__", "  File \"/tmp/ansible_paunch_payload_yovazksi/__main__.py\", line 179, in paunch_apply", "  File \"/usr/lib/python3.6/site-packages/paunch/__init__.py\", line 80, in apply", "    return builder.apply()", "  File \"/usr/lib/python3.6/site-packages/paunch/builder/base.py\", line 150, in apply", "    log=self.log)", "  File \"/usr/lib/python3.6/site-packages/paunch/utils/systemd.py\", line 112, in service_create", "    systemctl.enable(service, now=True)", "  File \"/usr/lib/python3.6/site-packages/tenacity/__init__.py\", line 292, in wrapped_f", "    return self.call(f, *args, **kw)", "  File \"/usr/lib/python3.6/site-packages/tenacity/__init__.py\", line 358, in call", "    do = self.iter(retry_state=retry_state)", "  File \"/usr/lib/python3.6/site-packages/tenacity/__init__.py\", line 331, in iter", "    raise retry_exc.reraise()", "  File \"/usr/lib/python3.6/site-packages/tenacity/__init__.py\", line 167, in reraise", "    raise self.last_attempt.result()", "  File \"/usr/lib64/python3.6/concurrent/futures/_base.py\", line 425, in result", "    return self.__get_result()", "  File \"/usr/lib64/python3.6/concurrent/futures/_base.py\", line 384, in __get_result", "    raise self._exception", "  File \"/usr/lib/python3.6/site-packages/tenacity/__init__.py\", line 361, in call", "    result = fn(*args, **kwargs)", "  File \"/usr/lib/python3.6/site-packages/paunch/utils/systemctl.py\", line 79, in enable", "    raise SystemctlException(str(err))", "paunch.utils.systemctl.SystemctlException: Command '['systemctl', 'enable', '--now', 'tripleo_nova_virtlogd']' returned non-zero exit status 1."]}
FAILED - RETRYING: Wait for containers to start for step 3 using paunch (1180 retries left).
FAILED - RETRYING: Wait for containers to start for step 3 using paunch (1179 retries left).

FAILED - RETRYING: Wait for containers to start for step 3 using paunch (1178 retries left).
FAILED - RETRYING: Wait for containers to start for step 3 using paunch (1177 retries left).
FAILED - RETRYING: Wait for containers to start for step 3 using paunch (1176 retries left).
FAILED - RETRYING: Wait for containers to start for step 3 using paunch (1175 retries left).
FAILED - RETRYING: Wait for containers to start for step 3 using paunch (1174 retries left).
FAILED - RETRYING: Wait for containers to start for step 3 using paunch (1173 retries left).
FAILED - RETRYING: Wait for containers to start for step 3 using paunch (1172 retries left).
FAILED - RETRYING: Wait for containers to start for step 3 using paunch (1171 retries left).
FAILED - RETRYING: Wait for containers to start for step 3 using paunch (1170 retries left).
FAILED - RETRYING: Wait for containers to start for step 3 using paunch (1169 retries left).

FAILED - RETRYING: Wait for containers to start for step 3 using paunch (1168 retries left).
FAILED - RETRYING: Wait for containers to start for step 3 using paunch (1167 retries left).
FAILED - RETRYING: Wait for containers to start for step 3 using paunch (1166 retries left).
FAILED - RETRYING: Wait for containers to start for step 3 using paunch (1165 retries left).
ok: [overcloud-controller-0]

NO MORE HOSTS LEFT *************************************************************

PLAY RECAP *********************************************************************
overcloud-controller-0     : ok=279  changed=113  unreachable=0    failed=0    skipped=118  rescued=0    ignored=1
overcloud-novacompute-0    : ok=226  changed=83   unreachable=0    failed=0    skipped=99   rescued=0    ignored=1
overcloud-novacompute-1    : ok=222  changed=83   unreachable=0    failed=0    skipped=99   rescued=0    ignored=1
overcloud-novacompute-2    : ok=221  changed=83   unreachable=0    failed=1    skipped=99   rescued=0    ignored=1
overcloud-novacompute-3    : ok=221  changed=83   unreachable=0    failed=1    skipped=99   rescued=0    ignored=1
overcloud-novacompute-4    : ok=221  changed=83   unreachable=0    failed=1    skipped=99   rescued=0    ignored=1
undercloud                 : ok=13   changed=7    unreachable=0    failed=0    skipped=10   rescued=0    ignored=0

Friday 24 July 2020  15:52:30 -0400 (0:01:55.995)       0:21:38.828 ***********
===============================================================================

• Error similar to these are seen in /var/log/messages:

Jul 27 06:32:39 overcloud-novacompute-3 systemd[1]: tripleo_nova_virtlogd.service: Control process exited, code=exited status=125
Jul 27 06:32:39 overcloud-novacompute-3 systemd[1]: tripleo_nova_virtlogd.service: Failed with result 'exit-code'.
Jul 27 06:32:39 overcloud-novacompute-3 systemd[1]: Failed to start nova_virtlogd container.
Jul 27 06:32:39 overcloud-novacompute-3 paunch-start-podman-container[12671]: Error: unable to start container "nova_compute": error mounting storage for container aa0c7f2c92b377b5c1839bf4e0ebfb26b6f98592c91588ca4a090e738db78242: error creating overlay mount to /var/lib/containers/storage/overlay/0079f7fe18e6671845f9bc2e66e3a1ec444daf1ef4b42d5e59a8d4349e6effdf/merged: invalid argument
Jul 27 06:32:39 overcloud-novacompute-3 paunch-start-podman-container[12671]: Error starting podman container nova_compute: 125
Jul 27 06:32:39 overcloud-novacompute-3 systemd[1]: tripleo_nova_compute.service: Control process exited, code=exited status=125
Jul 27 06:32:39 overcloud-novacompute-3 systemd[1]: tripleo_nova_compute.service: Failed with result 'exit-code'.
Jul 27 06:32:39 overcloud-novacompute-3 systemd[1]: Failed to start nova_compute container.
Jul 27 06:32:39 overcloud-novacompute-3 cloud-init[12917]: Cloud-init v. 18.5 running 'modules:final' at Mon, 27 Jul 2020 13:32:39 +0000. Up 25.09 seconds.
Jul 27 06:32:39 overcloud-novacompute-3 cloud-init[12917]: Cloud-init v. 18.5 finished at Mon, 27 Jul 2020 13:32:39 +0000. Datasource DataSourceConfigDrive [net,ver=2][source=/dev/sda1].  Up 25.13 seconds
Jul 27 06:32:39 overcloud-novacompute-3 kernel: tmpfs: Bad mount option context
Jul 27 06:32:39 overcloud-novacompute-3 kernel: tmpfs: Bad mount option context
Jul 27 06:32:39 overcloud-novacompute-3 systemd[1]: Started Execute cloud user/final scripts.
Jul 27 06:32:39 overcloud-novacompute-3 systemd[1]: tripleo_nova_virtlogd.service: Service RestartSec=100ms expired, scheduling restart.
Jul 27 06:32:39 overcloud-novacompute-3 systemd[1]: tripleo_nova_virtlogd.service: Scheduled restart job, restart counter is at 1.
Jul 27 06:32:39 overcloud-novacompute-3 systemd[1]: tripleo_nova_compute.service: Service RestartSec=100ms expired, scheduling restart.
Jul 27 06:32:39 overcloud-novacompute-3 systemd[1]: tripleo_nova_compute.service: Scheduled restart job, restart counter is at 1.
Jul 27 06:32:39 overcloud-novacompute-3 systemd[1]: Stopped nova_compute container.
Jul 27 06:32:39 overcloud-novacompute-3 podman[12666]: Error: unable to start container "logrotate_crond": failed to mount shm tmpfs "/var/lib/containers/storage/overlay-containers/294e224e788417920707e40a60207ced3caa4f1b9eb03203da9eda65ee4ca155/userdata/shm": invalid argument
Jul 27 06:32:39 overcloud-novacompute-3 systemd[1]: Starting nova_compute container...
Jul 27 06:32:39 overcloud-novacompute-3 systemd[1]: Stopped nova_compute container healthcheck.
Jul 27 06:32:39 overcloud-novacompute-3 systemd[1]: Stopping nova_compute container healthcheck.
Jul 27 06:32:39 overcloud-novacompute-3 systemd[1]: Started nova_compute container healthcheck.
Jul 27 06:32:39 overcloud-novacompute-3 systemd[1]: Stopped nova_virtlogd container.
Jul 27 06:32:39 overcloud-novacompute-3 systemd[1]: Starting nova_virtlogd container...
Jul 27 06:32:39 overcloud-novacompute-3 systemd[1]: tripleo_logrotate_crond.service: Control process exited, code=exited status=125
Jul 27 06:32:39 overcloud-novacompute-3 systemd[1]: tripleo_logrotate_crond.service: Failed with result 'exit-code'.
Jul 27 06:32:39 overcloud-novacompute-3 systemd[1]: Failed to start logrotate_crond container.
Jul 27 06:32:39 overcloud-novacompute-3 podman[12667]: Error: unable to start container "ovn_metadata_agent": failed to mount shm tmpfs "/var/lib/containers/storage/overlay-containers/690f5239e12c0102351d49be18d08ec47a3b89eb2788fdd15d039ed92b98b110/userdata/shm": invalid argument