Velero pods constantly gets killed and recreated when using OADP in ROSA

Solution Verified - Updated -

Environment

  • Red Hat OpenShift Services in AWS (ROSA)
    • 4.X

Issue

  • Velero pods recreating
velero-7b5bbc7dd7-p7qz4                             0/1     PodInitializing   0          4s
velero-7b5bbc7dd7-pxxtv                             0/1     Pending           0          0s
velero-7b5bbc7dd7-rdmgr                             0/1     PodInitializing   0          5s
  • Message when describing the DataProtectionApplication (DPA)
 message: deployments.apps "velero" not found

Resolution

Disable the target namespace from Instana configuration.

Root Cause

When running the 3rd party software Instana, the software automatically injects a InitContainer in the Velero pod, which causes the constantly kill and recreation of the Velero pods.

Diagnostic Steps

  • Injected InitContainer:
$ oc get pods -n <oadp-namespace>  <velero-pod> -o json | jq '.status.initContainerStatuses[] | {Container: .name, Ready: .ready, Reason: .state.waiting.reason}'
{
  "Container": "instana-instrumentation-init",
  "Ready": false,
  "Reason": "PodInitializing"
}
  • Labels can be verified:
    labels: 
      control-plane: controller-manager
      instana-autotrace-applied: "true"   <<<<<<
      instana-autotrace-transformation: DefaultDeploymentTransformation <<<<<<
      instana-autotrace-version: 1.242.1 <<<<<<
      olm.deployment-spec-hash: 74b56bbb96
      olm.owner: oadp-operator.v1.1.4
      olm.owner.kind: ClusterServiceVersion
      olm.owner.namespace: oadp-velero

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments