opentelemetry-operator-controller-manager crashlooping after upgrading in RHOCP 4
Issue
-
After receiving the upgrade to
opentelemetry-operator.v0.74.0-5theopentelemetry-operator-controller-managerpod has started to crash with the error:{"level":"info","ts":"2023-05-11T12:43:06.332498677Z","logger":"controllers.OpenTelemetryCollector","msg":"couldn't determine metrics port from configuration, using 8888 default value","opentelemetrycollector":"collector-system/otel-cluster","error":"missing port in address"} {"level":"info","ts":"2023-05-11T12:43:06.356630367Z","msg":"Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference","controller":"opentelemetrycollector","controllerGroup":"opentelemetry.io","controllerKind":"OpenTelemetryCollector","OpenTelemetryCollector":{"name":"otel-cluster","namespace":"collector-system"},"namespace":"collector-system","name":"otel-cluster","reconcileID":"1dcd5ff8-4811-4b00-8df3-3d758f2fed3d"} panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x214ddbf] goroutine 730 [running]: sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1() /remote-source/otelop/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:119 +0x1fa panic({0x2604f00, 0x43493a0}) /usr/lib/golang/src/runtime/panic.go:884 +0x212 github.com/open-telemetry/opentelemetry-operator/pkg/collector.HorizontalPodAutoscaler({{0x2f45620, 0xc000630c90}, {{0x2f5d438, 0xc00029a750}, 0x0}, {0xc000312410, 0x45}, {0xc0003124b0, 0x49}, {0xc0003125a0, ...}, ...}, ...) /remote-source/otelop/app/pkg/collector/horizontalpodautoscaler.go:135 +0xc5f github.com/open-telemetry/opentelemetry-operator/pkg/collector/reconcile.HorizontalPodAutoscalers({_, _}, {{0x2f67d88, 0xc000b75260}, {0x2f58520, 0xc000c485c0}, 0xc000a1b180, {{0x2f5d438, 0xc004877ec0}, 0x0}, ...}) /remote-source/otelop/app/pkg/collector/reconcile/horizontalpodautoscaler.go:41 +0xd8 github.com/open-telemetry/opentelemetry-operator/controllers.(*OpenTelemetryCollectorReconciler).RunTasks(_, {_, _}, {{0x2f67d88, 0xc000b75260}, {0x2f58520, 0xc000c485c0}, 0xc000a1b180, {{0x2f5d438, 0xc004877ec0}, ...}, ...}) /remote-source/otelop/app/controllers/opentelemetrycollector_controller.go:223 +0x1d2 github.com/open-telemetry/opentelemetry-operator/controllers.(*OpenTelemetryCollectorReconciler).Reconcile(0xc000c5e300, {0x2f59798, 0xc004877e90}, {{{0xc000fa21f0, 0x10}, {0xc000fa21e0, 0xc}}}) /remote-source/otelop/app/controllers/opentelemetrycollector_controller.go:211 +0x3a6 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x2f59798?, {0x2f59798?, 0xc004877e90?}, {{{0xc000fa21f0?, 0x23cffc0?}, {0xc000fa21e0?, 0x0?}}}) /remote-source/otelop/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:122 +0xc8 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc00086c320, {0x2f596f0, 0xc000c48500}, {0x2704620?, 0xc000d82080?}) /remote-source/otelop/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:323 +0x38f sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc00086c320, {0x2f596f0, 0xc000c48500}) /remote-source/otelop/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:274 +0x1d9 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2() /remote-source/otelop/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:235 +0x85 created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 /remote-source/otelop/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:231 +0x333
Environment
- Red Hat OpenShift Container Platform (RHOCP)
- 4
- OpenTelemetry Operator
- 0.74.0-5
- Red Hat OpenShift Distributed Tracing (RHODST)
- 2.8
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.