opentelemetry-operator-controller-manager crashlooping after upgrading in RHOCP 4

Solution Verified - Updated -

Issue

  • After receiving the upgrade to opentelemetry-operator.v0.74.0-5 the opentelemetry-operator-controller-manager pod has started to crash with the error:

    {"level":"info","ts":"2023-05-11T12:43:06.332498677Z","logger":"controllers.OpenTelemetryCollector","msg":"couldn't determine metrics port from configuration, using 8888 default value","opentelemetrycollector":"collector-system/otel-cluster","error":"missing port in address"}
    {"level":"info","ts":"2023-05-11T12:43:06.356630367Z","msg":"Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference","controller":"opentelemetrycollector","controllerGroup":"opentelemetry.io","controllerKind":"OpenTelemetryCollector","OpenTelemetryCollector":{"name":"otel-cluster","namespace":"collector-system"},"namespace":"collector-system","name":"otel-cluster","reconcileID":"1dcd5ff8-4811-4b00-8df3-3d758f2fed3d"}
    panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference
    [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x214ddbf]
    goroutine 730 [running]:
    sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
    /remote-source/otelop/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:119 +0x1fa
    panic({0x2604f00, 0x43493a0})
    /usr/lib/golang/src/runtime/panic.go:884 +0x212
    github.com/open-telemetry/opentelemetry-operator/pkg/collector.HorizontalPodAutoscaler({{0x2f45620, 0xc000630c90}, {{0x2f5d438, 0xc00029a750}, 0x0}, {0xc000312410, 0x45}, {0xc0003124b0, 0x49}, {0xc0003125a0, ...}, ...}, ...)
    /remote-source/otelop/app/pkg/collector/horizontalpodautoscaler.go:135 +0xc5f
    github.com/open-telemetry/opentelemetry-operator/pkg/collector/reconcile.HorizontalPodAutoscalers({_, _}, {{0x2f67d88, 0xc000b75260}, {0x2f58520, 0xc000c485c0}, 0xc000a1b180, {{0x2f5d438, 0xc004877ec0}, 0x0}, ...})
    /remote-source/otelop/app/pkg/collector/reconcile/horizontalpodautoscaler.go:41 +0xd8
    github.com/open-telemetry/opentelemetry-operator/controllers.(*OpenTelemetryCollectorReconciler).RunTasks(_, {_, _}, {{0x2f67d88, 0xc000b75260}, {0x2f58520, 0xc000c485c0}, 0xc000a1b180, {{0x2f5d438, 0xc004877ec0}, ...}, ...})
    /remote-source/otelop/app/controllers/opentelemetrycollector_controller.go:223 +0x1d2
    github.com/open-telemetry/opentelemetry-operator/controllers.(*OpenTelemetryCollectorReconciler).Reconcile(0xc000c5e300, {0x2f59798, 0xc004877e90}, {{{0xc000fa21f0, 0x10}, {0xc000fa21e0, 0xc}}})
    /remote-source/otelop/app/controllers/opentelemetrycollector_controller.go:211 +0x3a6
    sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x2f59798?, {0x2f59798?, 0xc004877e90?}, {{{0xc000fa21f0?, 0x23cffc0?}, {0xc000fa21e0?, 0x0?}}})
    /remote-source/otelop/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:122 +0xc8
    sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc00086c320, {0x2f596f0, 0xc000c48500}, {0x2704620?, 0xc000d82080?})
    /remote-source/otelop/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:323 +0x38f
    sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc00086c320, {0x2f596f0, 0xc000c48500})
    /remote-source/otelop/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:274 +0x1d9
    sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
    /remote-source/otelop/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:235 +0x85
    created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
    /remote-source/otelop/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:231 +0x333
    

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4
  • OpenTelemetry Operator
    • 0.74.0-5
  • Red Hat OpenShift Distributed Tracing (RHODST)
    • 2.8

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content