HCP installation on kubevirt is failing in RHOCP 4

Issue

• Installing HCP cluster fails for ingress operator not able to connect to the canary route for Bad Gateway error:

  ERROR   operator.ingress_controller controller/controller.go:119    got retryable error; requeueing {"after": "1m0s", "error": "IngressController is degraded: CanaryChecksSucceeding=False (CanaryChecksRepetitiveFailures: Canary route checks for the default ingress controller are failing. Last 2 error messages:\nerror sending canary HTTP Request: Timeout: Get \"https://canary-openshift-ingress-canary.apps.hcp0.apps.example.com\": context deadline exceeded (Client.Timeout exceeded while awaiting headers) (x4 over 5h0m8s)\nerror sending canary HTTP request to \"canary-openshift-ingress-canary.apps.hcp0.apps.example.com\": Get \"https://canary-openshift-ingress-canary.apps.hcp0.apps.example.com\": Bad Gateway (x273 over 5h4m25s))"}
  

• konnectivity-proxy is receiving server misbehaving while trying to resolve the canary host:

  WARN: Error dialing to canary-openshift-ingress-canary.apps.hcp0.apps.example.com:443: failed to resolve name canary-openshift-ingress-canary.apps.hcp0.apps.example.com: failed to look up name canary-openshift-ingress-canary.apps.hcp0.apps.example.com from guest cluster cluster-dns: failed to resolve "canary-openshift-ingress-canary.apps.hcp0.apps.example.com": lookup canary-openshift-ingress-canary.apps.hcp0.apps.example.com on 172.30.0.10:53: server misbehaving
  

• From the DNS HCP pod the Management Cluster DNS Service IP is unreachable and the DNS logs are reporting timeout:

  [ERROR] plugin/errors: 2 canary-openshift-ingress-canary.apps.hcp0.apps.example.com. AAAA: dial tcp 172.30.0.10:53: i/o timeout