Can't build or push to the registry after upgrading to OSE 3.1.1

Solution Verified - Updated -

Issue

My docker-registry is backed by an NFS persistent volume and runs with a privileged service account.

After upgrading the registry to version 3.1.1.6 I'm having issues with pushing images to the docker registry after a build:

809 F0220 14:11:55.066620       1 builder.go:185] Error: build error: Failed to push image. Response from registry is: Received unexpected HTTP status: 500 Internal Server Error

Registry registry's logs point to a permission problem with the NFS storage, however no changes were done to NFS.

# oc logs docker-registry-12345
...
time="2016-02-20T15:16:43.269414364-05:00" level=debug msg="(*linkedBlobStore).Create" go.version=go1.4.2 http.request.host="172.30.77.44:5000" http.request.id=3caf378c-8100-4df8-9a86-74b47ae6db3f http.request.method=POST http.request.remoteaddr="10.1.1.1:42747" http.request.uri="/v2/test/nodejs-example/blobs/uploads/" http.request.useragent="docker/1.8.2-el7 go/go1.4.2 kernel/3.10.0-327.4.5.el7.x86_64 os/linux arch/amd64" instance.id=74a32f03-788a-4a16-92fd-39b3c98ba443 vars.name="test/nodejs-example"
10.1.1.1 - - [20/Feb/2016:15:16:43 -0500] "POST /v2/test/nodejs-example/blobs/uploads/ HTTP/1.1" 500 187 "" "docker/1.8.2-el7 go/go1.4.2 kernel/3.10.0-327.4.5.el7.x86_64 os/linux arch/amd64"
time="2016-02-20T15:16:43.290856856-05:00" level=debug msg="filesystem.PutContent(\"/docker/registry/v2/repositories/test/nodejs-example/_uploads/d4961557-0a66-420a-90aa-2687e98aa919/startedat\")" go.version=go1.4.2 http.request.host="172.30.77.44:5000" http.request.id=3caf378c-8100-4df8-9a86-74b47ae6db3f http.request.method=POST http.request.remoteaddr="10.1.1.1:42747" http.request.uri="/v2/test/nodejs-example/blobs/uploads/" http.request.useragent="docker/1.8.2-el7 go/go1.4.2 kernel/3.10.0-327.4.5.el7.x86_64 os/linux arch/amd64" instance.id=74a32f03-788a-4a16-92fd-39b3c98ba443 trace.duration=21.358757ms trace.file="/builddir/build/BUILD/atomic-openshift-git-0.b57e8bd/_thirdpartyhacks/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).PutContent" trace.id=66c33f3b-fa51-4643-bc6c-f60a253264c2 trace.line=95 vars.name="test/nodejs-example"
time="2016-02-20T15:16:43.290983715-05:00" level=error msg="response completed with error" err.code=UNKNOWN err.detail="filesystem: mkdir /registry/docker/registry/v2/repositories/test: permission denied" err.message="unknown error" go.version=go1.4.2 http.request.host="172.30.77.44:5000" http.request.id=3caf378c-8100-4df8-9a86-74b47ae6db3f http.request.method=POST http.request.remoteaddr="10.1.1.1:42747" http.request.uri="/v2/test/nodejs-example/blobs/uploads/" http.request.useragent="docker/1.8.2-el7 go/go1.4.2 kernel/3.10.0-327.4.5.el7.x86_64 os/linux arch/amd64" http.response.contenttype="application/json; charset=utf-8" http.response.duration=54.295539ms http.response.status=500 http.response.written=187 instance.id=74a32f03-788a-4a16-92fd-39b3c98ba443 vars.name="test/nodejs-example"

Note the err "filesystem: ... permission denied" message above.

Environment

  • OpenShift Enterprise 3.1.1 or later, upgraded from an earlier version
  • docker-registry deployment using a privileged security context constraint (SCC)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content