Why does deleting a host fail if the capsule that it is registered to is down?

Solution Verified - Updated -

Environment

  • Red Hat Satellite 6.8.
  • Red Hat Satellite 6.9.4 and earlier

Issue

  • Attempting to delete a host fails while the capsule that the host is registered to is down or unavailable:
  • The capsule has already been decommissioned and is no longer available.
  • How can I delete the host?

Resolution

  • This issue was resolved with the Red Hat Satellite 6.9.5 release and errata RHBA-2021:3387. Please upgrade to version 6.9.5 or later.

  • Sometimes this occurs when wrong organization and location is selected for the host. Ensure the same registered capsule server organization and location is assigned for the host.

Root Cause

Starting in Satellite 6.8 and early Satellite 6.9, Satellite attempts to delete the SSH host key of the host being deleted from the capsule, to prevent future failures or remote execution if the host was to be re-provisioned and registered back to Satellite:

The attempt to delete the SSH host key from the capsule fails because the capsule is no longer available:

2020-12-14T16:13:13 [W|app|] Remove SSH known hosts for host.example.com task failed with the following error: ERF12-6886 [ProxyAPI::ProxyException]: Unable to remove host from known hosts ([SocketError]: Failed to open TCP connection to capsule.example.com:9090 (getaddrinfo: Name or service not known)) for Capsule https://capsule.example.com:9090/ssh
2020-12-14T16:13:13 [W|app|] Rolling back due to a problem: [#<Orchestration::Task:0x0000000011f995f0 @name="Remove SSH known hosts for host.example.com", @id="ssh_remove_known_hosts_interface_10.33.8.73_2", @status="failed", @priority=200, @action=[#<Nic::Bridge id: 15, mac: "52:54:00:14:4d:08", ip: "10.33.8.73", type: "Nic::Bridge", name: "host.example.com", host_id: 8, subnet_id: 1, domain_id: 1, attrs: {"bridge"=>true}, created_at: "2020-12-14 14:51:07", updated_at: "2020-12-14 15:03:18", provider: nil, username: nil, password: nil, virtual: true, link: true, identifier: "br0", tag: "", attached_to: "", managed: true, mode: "balance-rr", attached_devices: "", bond_options: "", primary: true, provision: true, compute_attributes: {}, execution: true, ip6: "", subnet6_id: nil>, :drop_from_known_hosts, 2], @created=1607962393.0890071, @timestamp=2020-12-14 16:13:13 UTC>]
2020-12-14T16:13:13 [I|bac|] Task {label: Actions::Katello::Host::Destroy, id: e81809da-1c33-4a0c-84ae-ab20536a62e9, execution_plan_id: a8f006df-a5ac-420d-90d3-b49336a39e42} state changed: stopped  result: success
2020-12-14T16:13:13 [I|bac|] Task {label: Actions::Katello::Host::Destroy, id: e81809da-1c33-4a0c-84ae-ab20536a62e9, execution_plan_id: a8f006df-a5ac-420d-90d3-b49336a39e42} state changed: stopped  result: success
2020-12-14T16:13:13 [I|bac|] Task {label: Actions::BulkAction, id: 22c644c1-e0d7-49c2-88f0-514ca9140cf1, execution_plan_id: 9d74e080-ea14-4eb0-8c8d-ca6e30a39790} state changed: stopped  result: success
2020-12-14T16:13:13 [I|bac|] Task {label: Actions::BulkAction, id: 22c644c1-e0d7-49c2-88f0-514ca9140cf1, execution_plan_id: 9d74e080-ea14-4eb0-8c8d-ca6e30a39790} state changed: stopped  result: success

Diagnostic Steps

It should be noted as well that the Satellite remote_execution_global_proxy global setting can also cause a failed host deletion if an unrelated capsule is down.

"Search for remote execution Capsule outside of the Capsules assigned to the host. The search will be limited to the host's organization and location."

With this behaviour enabled, the cleanup of the known_hosts will attempt to reach beyond just the capsule to which the host is registered. So an unrelated downed capsule may also contribute to a failure to remove a host. Setting this to "no"/"false" should focus this known_hosts removal only to the pertinent capsule.

Attachments

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments