OpenShift 4 upgrade fails if cluster has custom SCC with `readOnlyRootFilesystem` set to `true`

Solution Verified - Updated -

Issue

  • A custom SCC with the readOnlyRootFilesystem flag set to true is associated with the pod version-XXXX-XXXX-XXXX, causing it to be unable to write hostPath /etc/cvo/updatepayloads and resulting in a failure of the upgrade process.
  • The following message is shown in the clusterversion resource:

        message: 'Retrieving payload failed version="4.10.XX" image="quay.io/openshift-release-dev/ocp-release@sha256:nnnnnnnnnnnnnnnnnnnnnnnnnnnnnn"
        failure=Unable to download and prepare the update: deadline exceeded, reason:
        "DeadlineExceeded", message: "Job was active longer than specified deadline"'
      reason: RetrievePayload
      status: "False"
      type: ReleaseAccepted
    
  • A ReleaseAccepted=False error is shown when running oc adm upgrade:

    $ oc adm upgrade
    Cluster version is 4.10.YY
    
    ReleaseAccepted=False
    
    Reason: RetrievePayload
    Message: Retrieving payload failed version="4.10.XX" image="quay.io/openshift-release-dev/ocp-release@sha256:nnnnnnnnnnnnnnnnnnnnnnnnnnnnnn" failure=Unable to download and prepare the update: deadline exceeded, reason: "DeadlineExceeded", message: "Job was active longer than specified deadline"
    
  • The following error messages are shown in the logs of pod version-xxxxx-xxxxx-xxxxx, located in the openshift-cluster-version namespace:

    mv: inter-device move failed: '/manifests' to '/etc/cvo/updatepayloads/XXXXYYYYZZZZ/manifests/manifests'; unable to remove target: Directory not empty
    
    mv: cannot remove '/manifests/0000_00_cluster-version-operator_XX_yyyyyyyyyyy.yaml': Read-only file system
    

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4.10

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content