RHOCP, openshift-image-registry-operator: panic: runtime error: slice bounds out of range [3:2] [recovered]

Solution Verified - Updated -

Environment

  • Red Hat Openshift Container Platform (RHOCP) 4.x
  • Image registry using S3 backend storage

Issue

  • Cluster upgrade stalling from 4.8.x towards 4.9.x
  • openshift-image-registry-operator: panic: runtime error: slice bounds out of range [3:2] [recovered]

Resolution

Disclaimer: Links contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.

  • verify that the regionEndpoint configuration is not specifying the protocol

    $ oc get configs.imageregistry.operator.openshift.io/cluster -ojsonpath="{.spec.storage.s3.regionEndpoint}"
registry.example.com:443

  • specify the regionEndpoint including the protocol according to the AWS S3 specification

    $ oc patch configs.imageregistry.operator.openshift.io/cluster \
    --type='json' \
    --patch='[
            {"op":"replace", "path":"/spec/storage/s3/regionEndpoint", "value":"https://registry.example.com:443"}
        ]'

  • Amazon S3 Regional Endpoints definition https://docs.aws.amazon.com/general/latest/gr/rande.html#regional-endpoints

Root Cause

  • In configs.imageregistry.operator.openshift.io/cluster, the regionEndpoint needs to be specified including the protocol

  • unhandled input violation in GetURIPath in file aws/signer/internal/v4/util.go function strings.Split(u.Opaque, "/")[:3]

    func GetURIPath(u *url.URL) string {
    var uri string
    if len(u.Opaque) > 0 {
        uri = "/" + strings.Join(strings.Split(u.Opaque, "/")[3:], "/")
    } else {
        uri = u.EscapedPath()
    }
 [... output omitted ...]

  • regionEndpoint without protocol violates the expected input containing three forward slashes "/"

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments