Resolving Blob Upload Failures in OpenShift to JFROG Artifactory Integration

Solution Verified - Updated -

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4
  • Disconnected environment
  • JFROG Artifactory Docker-repository

Issue

Uploading the OpenShift image repository to the JFROG Artifactory Docker-repository fails with "unknown: Not Found" errors for multiple blobs.

  info: Planning completed in 630ms
  error: unable to upload blob sha256:77d2bc8dbd642f8b2e4a185e37bfb7a28a63aa4d0c058ae9ebd903e2d0b049d0 to 
  alias.of.our.artifactory/openshift: unknown: Not Found
  error: unable to push file://openshift/release: failed to upload blob

Resolution

  • Verify Artifactory Configuration

    • Access the Artifactory Web UI

    • Log in to the Artifactory web interface using admin credentials.

    • Determine the Image Format

    • Select the Docker repository in question and ensure:
    • Determine the Image Format
    • Inspect the image format using tools like skopeo, crane, or even Docker/Podman.

    • Using skopeo:

      $ skopeo inspect docker://<image_name>

    • Using crane:

      $ crane manifest <image_name>

    • Check Repository Configuration in Artifactory:

    • Navigate to Admin -> Repositories -> Local in the Artifactory UI`

    • Select the Docker repository and ensure the following based on the image format identified:

      • If the image is in Docker v2-2 format, ensure that the repository is set up to accept Docker v2 images.
      • If the image is in OCI format, ensure that the repository is configured to accept OCI images.

    • For OCI Images

    • Some versions of Artifactory may require specific settings to fully support OCI images. Make sure the repository is configured to handle OCI manifests, which might be a different setting from Docker v2-2.

    • The repository type is set to Docker.
    • The repository supports the Docker v2-2 image format.

    • The storage quota is sufficient for the image sizes.

    • Command-Line Verification (Optional)

    • Use curl to verify the repository settings

      $ curl -u <username>:<password> 
  "https://<artifactory_url>/artifactory/api/repositories/<repository_name>"

    • Curl Command for HTTP/HTTPS Connectivity

    • Check if the Artifactory URL is reachable via HTTPS

      $ curl -I "https://<artifactory_url>"

  • Correct Usage of oc image mirror Command

    • Find the command

      $ oc image mirror -a /path/to/pull-secret.json --from-dir=/var/lib/openshift/mirror 
  "file://openshift/release:4.15.16*" alias.of.our.artifactory/openshift

  • Inspect Artifactory Logs

    • Access Logs via the Web UI
    • Navigate to Admin -> Advanced -> System Logs in the Artifactory UI.

    • Look for entries related to Docker registry activities, especially around the time of the error.

    • Check Logs on the Artifactory Server (if you have access)

    • Access the server where Artifactory is installed and check logs typically located at /var/opt/jfrog/artifactory/logs.
    • Review artifactory.log and request.log for detailed error messages

  • Retry the Upload

    • Run the Mirroring Command Again
    • Retry the upload using the correct oc image mirror command

  • Delete the repository and try image push again.**

    • Identify the repository in the registry for which the image push is failing.
    • Delete that repository.
    • Try to push the image again.

  • Check the filesystem/storage for any kind of corruption.

    • Recommended storage for the registry would be object storage.

Note: JFROG and other third-party registries are not tested with OpenShift Container Platform. Refer to the official documentation: Red Hat does not test third party registries with OpenShift Container Platform

Root Cause

  • The unknown: Not Found error usually occurs due to a mismatch between the image format (e.g., OCI vs Docker v2-2) and the Artifactory repository configuration, network/proxy issues blocking access, or specific blobs being corrupted or exceeding storage quotas in Artifactory.

Diagnostic Steps

  • Verify Artifactory Repository Configuration

    • The error unknown: Not Found when uploading blobs suggests a mismatch between the image format and what the Artifactory repository is configured to accept.

    • Ensure the repository supports the image format (Docker v2-2 or OCI). In the Artifactory UI, go to Admin -> Repositories -> Local and ensure that the repository is configured correctly based on the image format.

    • Use skopeo to inspect the image format:

         $ skopeo inspect docker://<image_name>

    • If the image is in OCI format, ensure the repository is configured to accept OCI images.

         error: unable to upload blob sha256:<blob_id> to alias.of.our.artifactory/openshift: unknown: Not Found

  • Check Repository API and Logs

  • The error unknown: Not Found might be related to Artifactory misconfiguration or an issue preventing blob uploads.

  • Check Repository Settings: Use curl to query the repository and verify it's configured to accept uploads:

    $ curl -u <username>:<password> "https://<artifactory_url>/artifactory/api/repositories/<repository_name>"

  • Review Logs: In Artifactory UI, navigate to Admin -> Advanced -> System Logs or check /var/opt/jfrog/artifactory/logs/artifactory.log to find specific errors related to the failed blob uploads.

    error: unable to upload blob sha256:ca98956a16c493150c65f04743a873ea0d94866c350f15356fadceeef58c6fed: unknown: Not Found

  • Retry the Upload

  • The error unable to push file://openshift/release: failed to upload blob may indicate an issue with the initial push attempt.

  • Retry the oc image mirror command after verifying the repository configuration:

    $ oc image mirror -a /path/to/pull-secret.json --from-dir=/var/lib/openshift/mirror "file://openshift/release:4.15.16*" alias.of.our.artifactory/openshift

  • If specific blobs still fail, investigate the integrity of those blobs using skopeo inspect:

    skopeo inspect docker://<blob_id>

These steps directly address the errors you are encountering and provide targeted actions to resolve them.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments