Pods in the namespace "openshift-azure-logging" are in error after cluster upgrade

Issue

• After cluster upgrade, the pods in the namespace openshift-azure-logging appear as CrashLoopBackOff or in Error:

  $ oc get pods -n openshift-azure-logging
  NAME         READY   STATUS             RESTARTS   AGE
  mdsd-dsn57   1/2     CrashLoopBackOff   1235       50d
  mdsd-hpfvs   1/2     CrashLoopBackOff   1236       50d
  mdsd-jx8fl   1/2     Error              1236       50d
  mdsd-m7nn4   1/2     CrashLoopBackOff   1226       50d
  mdsd-pc8dv   1/2     CrashLoopBackOff   1235       50d
  mdsd-vclsp   1/2     CrashLoopBackOff   1238       50d
  

  The pods logs present the errors below:

  $ oc logs mdsd-dsn57 -n openshift-azure-logging -c mdsd
  2024-10-29T06:49:24.032615419Z 2024-10-29T06:49:24.0325860Z: GcsMgr failed to download start up config. Retrying...
  2024-10-29T06:49:24.094687763Z 2024-10-29T06:49:24.0946390Z: Error: request failed with exception: Failed to read HTTP status line; error code: generic:1
  2024-10-29T06:49:24.094687763Z 2024-10-29T06:49:24.0946710Z: MdsRestInterface::QueryGcsConfigInfo() failed
  
  $ oc logs mdsd-dsn57 -n openshift-azure-logging -c fluentbit
  2024-10-29T06:53:44.053015432Z [2024/10/29 06:53:44] [error] [output:forward:forward.0] no upstream connections available
  2024-10-29T06:53:44.053864510Z [2024/10/29 06:53:44] [error] [output:forward:forward.0] no upstream connections available
  2024-10-29T06:53:44.067461621Z [2024/10/29 06:53:44] [ warn] [engine] chunk '1-1730184817.62316623.flb' cannot be retried: task_id=3, input=tail.1 > output=forward.0
  2024-10-29T06:53:44.067461621Z [2024/10/29 06:53:44] [ warn] [engine] failed to flush chunk '1-1730184823.64213287.flb', retry in 7 seconds: task_id=10, input=tail.1 > output=forward.0 (out_id=0)