Jaeger-ingester pod in CrashLoopBackoff due to segmentation fault

Solution In Progress - Updated -

Issue

  • jaeger-ingester pod in CrashLoopBackOff when searched with key=value pair in jaeger dashboard with the following segmentation fault.

    panic: runtime error: invalid memory address or nil pointer dereference
    [signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0xe59e46]
    2023-08-08T13:27:06.837744566Z
    goroutine 1119 [running]:
    github.com/jaegertracing/jaeger/plugin/storage/es/spanstore/dbmodel.FromDomain.convertProcess(...)
    2023-08-08T13:27:06.837857279Z /remote-source/jaeger/app/plugin/storage/es/spanstore/dbmodel/from_domain.go:123
    github.com/jaegertracing/jaeger/plugin/storage/es/spanstore/dbmodel.FromDomain.convertSpanEmbedProcess({0x80?, 0xc00037e4e0?, {0x15c3998?, 0xfea212?}}, 0xc0003a4ff0)
    /remote-source/jaeger/app/plugin/storage/es/spanstore/dbmodel/from_domain.go:64 +0x126
    github.com/jaegertracing/jaeger/plugin/storage/es/spanstore/dbmodel.FromDomain.FromDomainEmbedProcess(...)
    /remote-source/jaeger/app/plugin/storage/es/spanstore/dbmodel/from_domain.go:43
    github.com/jaegertracing/jaeger/plugin/storage/es/spanstore.(*SpanWriter).WriteSpan(0xc000221bc0, {0x0?, 0x02023-08-08T13:27:06.837933603Z ?}, 0xc0003a4ff0)
    /remote-source/jaeger/app/plugin/storage/es/spanstore/writer.go:152 +0x7a
    github.com/jaegertracing/jaeger/cmd/ingester/app/processor.KafkaSpanProcessor.Process({{0x15c7ce0, 0x1ebc378}, {0x15c7b60, 0xc000221bc02023-08-08T13:27:06.837961974Z }, {0x0, 0x0}}, {0x15ca6e0?2023-08-08T13:27:06.837986234Z , 0xc000436f00?})
    /remote-source/jaeger/app/cmd/ingester/app/processor/span_processor.go:67 +0xd32023-08-08T13:27:06.838053731Z
    github.com/jaegertracing/jaeger/cmd/ingester/app/processor/decorator.(*retryDecorator).Process(0xc00018dd50, {0x15ca6e0, 0xc000436f00})
    /remote-source/jaeger/app/cmd/ingester/app/processor/decorator/retry.go:109 +0x37
    github.com/jaegertracing/jaeger/cmd/ingester/app/consumer.(*comittingProcessor).Process(0xc00023cea0, {0x15ca6e0, 0xc000436f00}2023-08-08T13:27:06.838104194Z )
    /remote-source/jaeger/app/cmd/ingester/app/consumer/committing_processor.go:44 +0x5e
    github.com/jaegertracing/jaeger/cmd/ingester/app/processor.(*metricsDecorator).Process(2023-08-08T13:27:06.838151127Z 0xc00020efc0, {0x15ca6e0, 0xc000436f00})
    /remote-source/jaeger/app/cmd/ingester/app/processor/metrics_decorator.go2023-08-08T13:27:06.838175100Z :44 +0x5b
    github.com/jaegertracing/jaeger/cmd/ingester/app/processor.(*ParallelProcessor).Start.func1()
    /remote-source/jaeger/app/cmd/ingester/app/processor/parallel_processor.go2023-08-08T13:27:06.838199534Z :58 +0x42
    created by github.com/jaegertracing/jaeger/cmd/ingester/app/processor.(*ParallelProcessor).Start
    /remote-source/jaeger/app/cmd/ingester/app/processor/parallel_processor.go2023-08-08T13:27:06.838223140Z :54 +0xf5
    

Environment

  • Red Hat OpenShift Container Platform
    • 4.10+
  • Red Hat OpenShift distributed tracing platform Operator
    • v1.36.0

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content