Overcloud update fails during puppet host config step 1

Solution Unverified - Updated -

Issue

During an openstack overcloud update, the update fails with errors similar to the following:

2020-12-09 02:40:00Z [overcloud]: UPDATE_FAILED  Resource UPDATE failed: Error: resources.AllNodesDeploySteps.resources.ControllerDeployment_Step1.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 2

 Stack overcloud UPDATE_FAILED

overcloud.AllNodesDeploySteps.ControllerDeployment_Step1.0:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: 9da545d9-28dc-4fc4-8b76-2d358a60abcd
  status: UPDATE_FAILED
  status_reason: |
    Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2
  deploy_stdout: |
    ...
            "Warning: /Stage[main]/Tripleo::Firewall/Exec[nonpersistent_v6_rules_cleanup]: Skipping because of failed dependencies",
            "Warning: /Stage[main]/Tripleo::Firewall/Exec[nonpersistent_ironic_inspector_pxe_filter_v4_rules_cleanup]: Skipping because of failed dependencies",
            "Warning: /Stage[main]/Tripleo::Firewall/Exec[nonpersistent_ironic_inspector_pxe_filter_v6_rules_cleanup]: Skipping because of failed dependencies"
        ]
    }
        to retry, use: --limit @/var/lib/heat-config/heat-config-ansible/3ad89b15-6425-415b-bd90-45d9f627abcd_playbook.retry

    PLAY RECAP *********************************************************************
    localhost                  : ok=27   changed=10   unreachable=0    failed=1

    (truncated, view all with --long)
  deploy_stderr: |

Heat Stack update failed.
Heat Stack update failed.

And also

    TASK [Debug output for task: Run puppet host configuration for step 1] *********
    fatal: [localhost]: FAILED! => {
        "failed_when_result": true,
        "outputs.stdout_lines|default([])|union(outputs.stderr_lines|default([]))": [
            "Notice: hiera(): Cannot load backend module_data: cannot load such file -- hiera/backend/module_data_backend",
            "Notice: Scope(Class[Tripleo::Firewall::Post]): At this stage, all network traffic is blocked.",
            "Notice: Compiled catalog for ospcontrol-ctc-a107d.ctc.pa.tesprod.comcast.net in environment production in 3.18 seconds",
            "Notice: /Stage[main]/Tripleo::Profile::Base::Kernel/Kmod::Load[nf_conntrack_proto_sctp]/Exec[modprobe nf_conntrack_proto_sctp]/returns: executed successfully",
            "Notice: /Stage[main]/Tripleo::Profile::Base::Docker/Service[docker]: Dependency Augeas[docker-daemon.json-debug] has failures: true",
            "Notice: /Stage[main]/Pacemaker::Stonith/Pacemaker::Property[Disable STONITH]/Pcmk_property[property--stonith-enabled]/ensure: created",
            "Notice: /Stage[main]/Ssh::Server::Config/Concat[/etc/ssh/sshd_config]/File[/etc/ssh/sshd_config]/content: content changed '{md5}f8044ed53107372d76ab3978d355de0f' to '{md5}3534841fdb8db5b58d66600a60bf3759'",
            "Notice: /Stage[main]/Ssh::Server::Service/Service[sshd]: Triggered 'refresh' from 2 events",
            "Notice: /Stage[main]/Tripleo::Firewall::Post/Tripleo::Firewall::Rule[998 log all]/Firewall[998 log all ipv4]: Dependency Augeas[docker-daemon.json-debug] has failures: true",
            "Notice: /Stage[main]/Tripleo::Firewall::Post/Tripleo::Firewall::Rule[998 log all]/Firewall[998 log all ipv6]: Dependency Augeas[docker-daemon.json-debug] has failures: true",
            "Notice: /Stage[main]/Tripleo::Firewall::Post/Tripleo::Firewall::Rule[999 drop all]/Firewall[999 drop all ipv4]: Dependency Augeas[docker-daemon.json-debug] has failures: true",
            "Notice: /Stage[main]/Tripleo::Firewall::Post/Tripleo::Firewall::Rule[999 drop all]/Firewall[999 drop all ipv6]: Dependency Augeas[docker-daemon.json-debug] has failures: true",
            "Notice: /Stage[main]/Firewall::Linux::Redhat/File[/etc/sysconfig/iptables]: Dependency Augeas[docker-daemon.json-debug] has failures: true",
            "Notice: /Stage[main]/Firewall::Linux::Redhat/File[/etc/sysconfig/ip6tables]: Dependency Augeas[docker-daemon.json-debug] has failures: true",
            "Notice: /Stage[main]/Tripleo::Firewall/Exec[nonpersistent_v4_rules_cleanup]: Dependency Augeas[docker-daemon.json-debug] has failures: true",
            "Notice: /Stage[main]/Tripleo::Firewall/Exec[nonpersistent_v6_rules_cleanup]: Dependency Augeas[docker-daemon.json-debug] has failures: true",
            "Notice: /Stage[main]/Tripleo::Firewall/Exec[nonpersistent_ironic_inspector_pxe_filter_v4_rules_cleanup]: Dependency Augeas[docker-daemon.json-debug] has failures: true",
            "Notice: /Stage[main]/Tripleo::Firewall/Exec[nonpersistent_ironic_inspector_pxe_filter_v6_rules_cleanup]: Dependency Augeas[docker-daemon.json-debug] has failures: true",

Environment

  • Red Hat OpenStack Platform 13

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content