Image Registry pruning job is failing in OCP 4

Solution Verified - Updated -

Issue

  • Image registry Operator went degraded due to a failed image pruner pod with the following error.

    I0514 00:00:10.386992       1 merged_client_builder.go:121] Using in-cluster configuration
    I0514 00:01:47.303417       1 request.go:995] Stream error http2.StreamError{StreamID:0xf, Code:0x2, Cause:error(nil)} when reading response body, may be caused by closed connection.
    F0514 00:01:47.303543       1 helpers.go:115] error: stream error when reading response body, may be caused by closed connection. Please retry. Original error: stream error: stream ID 15; INTERNAL_ERROR
    goroutine 1 [running]:
    k8s.io/klog/v2.stacks(0xc00019a001, 0xc0278a6f00, 0xcb, 0x1c4)
    

    [...]
    /go/src/github.com/openshift/oc/vendor/github.com/spf13/cobra/command.go:950 +0x375
    github.com/spf13/cobra.(*Command).Execute(...)
    /go/src/github.com/openshift/oc/vendor/github.com/spf13/cobra/command.go:887
    main.main()
    /go/src/github.com/openshift/oc/cmd/oc/oc.go:110 +0x885

    goroutine 19 [chan receive]:
    k8s.io/klog/v2.(*loggingT).flushDaemon(0x4d37f20)
        /go/src/github.com/openshift/oc/vendor/k8s.io/klog/v2/klog.go:1131 +0x8b
    created by k8s.io/klog/v2.init.0
        /go/src/github.com/openshift/oc/vendor/k8s.io/klog/v2/klog.go:416 +0xd8
    
    goroutine 11 [chan receive]:
    k8s.io/klog.(*loggingT).flushDaemon(0x4d37e40)
        /go/src/github.com/openshift/oc/vendor/k8s.io/klog/klog.go:1010 +0x8b
    created by k8s.io/klog.init.0
        /go/src/github.com/openshift/oc/vendor/k8s.io/klog/klog.go:411 +0xd8
    
    goroutine 16 [select]:
    k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x31cf578, 0x3488a60, 0xc001428090, 0x1, 0xc0001a8300)
    

    [...]
    /go/src/github.com/openshift/oc/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:81 +0x4f
    created by k8s.io/component-base/logs.InitLogs
    /go/src/github.com/openshift/oc/vendor/k8s.io/component-base/logs/logs.go:58 +0x8a

    goroutine 82 [select, 1 minutes]:
    io.(*pipe).Read(0xc00184a840, 0xc00142e000, 0x1000, 0x1000, 0x2aabea0, 0x1, 0xc00142e000)
        /opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/io/pipe.go:57 +0xe7
    io.(*PipeReader).Read(0xc000375aa0, 0xc00142e000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/io/pipe.go:134 +0x4c
    bufio.(*Scanner).Scan(0xc00183d400, 0x0)
        /opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/bufio/scan.go:214 +0xa9
    github.com/openshift/oc/pkg/cli/admin/mustgather.newPrefixWriter.func1(0xc00183d400, 0x348a480, 0xc00019a008, 0x2fe8b51, 0x17)
        /go/src/github.com/openshift/oc/pkg/cli/admin/mustgather/mustgather.go:424 +0x13e
    created by github.com/openshift/oc/pkg/cli/admin/mustgather.newPrefixWriter
        /go/src/github.com/openshift/oc/pkg/cli/admin/mustgather/mustgather.go:423 +0x1d0
    
    goroutine 85 [IO wait]:
    internal/poll.runtime_pollWait(0x7fd724141630, 0x72, 0x348d7a0)
    

    [...]
    /go/src/github.com/openshift/oc/vendor/golang.org/x/net/http2/transport.go:1794 +0xd8
    golang.org/x/net/http2.(ClientConn).readLoop(0xc001b1e480)
    /go/src/github.com/openshift/oc/vendor/golang.org/x/net/http2/transport.go:1716 +0x6f
    created by golang.org/x/net/http2.(
    Transport).newClientConn
    /go/src/github.com/openshift/oc/vendor/golang.org/x/net/http2/transport.go:695 +0x66e

  • Container Image Registry Operator is degraded with error message reporting:

    Error from server: rpc error: code = ResourceExhausted desc = grpc: trying to send message larger than max (2199809298 vs. 2147483647)
    

Environment

  • Red Hat OpenShift Container Platform (RHOCP) 4

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content