Can't build or push to the registry after upgrading to OSE 3.1.1
Issue
My docker-registry is backed by an NFS persistent volume and runs with a privileged service account.
After upgrading the registry to version 3.1.1.6 I'm having issues with pushing images to the docker registry after a build:
809 F0220 14:11:55.066620 1 builder.go:185] Error: build error: Failed to push image. Response from registry is: Received unexpected HTTP status: 500 Internal Server Error
Registry registry's logs point to a permission problem with the NFS storage, however no changes were done to NFS.
# oc logs docker-registry-12345
...
time="2016-02-20T15:16:43.269414364-05:00" level=debug msg="(*linkedBlobStore).Create" go.version=go1.4.2 http.request.host="172.30.77.44:5000" http.request.id=3caf378c-8100-4df8-9a86-74b47ae6db3f http.request.method=POST http.request.remoteaddr="10.1.1.1:42747" http.request.uri="/v2/test/nodejs-example/blobs/uploads/" http.request.useragent="docker/1.8.2-el7 go/go1.4.2 kernel/3.10.0-327.4.5.el7.x86_64 os/linux arch/amd64" instance.id=74a32f03-788a-4a16-92fd-39b3c98ba443 vars.name="test/nodejs-example"
10.1.1.1 - - [20/Feb/2016:15:16:43 -0500] "POST /v2/test/nodejs-example/blobs/uploads/ HTTP/1.1" 500 187 "" "docker/1.8.2-el7 go/go1.4.2 kernel/3.10.0-327.4.5.el7.x86_64 os/linux arch/amd64"
time="2016-02-20T15:16:43.290856856-05:00" level=debug msg="filesystem.PutContent(\"/docker/registry/v2/repositories/test/nodejs-example/_uploads/d4961557-0a66-420a-90aa-2687e98aa919/startedat\")" go.version=go1.4.2 http.request.host="172.30.77.44:5000" http.request.id=3caf378c-8100-4df8-9a86-74b47ae6db3f http.request.method=POST http.request.remoteaddr="10.1.1.1:42747" http.request.uri="/v2/test/nodejs-example/blobs/uploads/" http.request.useragent="docker/1.8.2-el7 go/go1.4.2 kernel/3.10.0-327.4.5.el7.x86_64 os/linux arch/amd64" instance.id=74a32f03-788a-4a16-92fd-39b3c98ba443 trace.duration=21.358757ms trace.file="/builddir/build/BUILD/atomic-openshift-git-0.b57e8bd/_thirdpartyhacks/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).PutContent" trace.id=66c33f3b-fa51-4643-bc6c-f60a253264c2 trace.line=95 vars.name="test/nodejs-example"
time="2016-02-20T15:16:43.290983715-05:00" level=error msg="response completed with error" err.code=UNKNOWN err.detail="filesystem: mkdir /registry/docker/registry/v2/repositories/test: permission denied" err.message="unknown error" go.version=go1.4.2 http.request.host="172.30.77.44:5000" http.request.id=3caf378c-8100-4df8-9a86-74b47ae6db3f http.request.method=POST http.request.remoteaddr="10.1.1.1:42747" http.request.uri="/v2/test/nodejs-example/blobs/uploads/" http.request.useragent="docker/1.8.2-el7 go/go1.4.2 kernel/3.10.0-327.4.5.el7.x86_64 os/linux arch/amd64" http.response.contenttype="application/json; charset=utf-8" http.response.duration=54.295539ms http.response.status=500 http.response.written=187 instance.id=74a32f03-788a-4a16-92fd-39b3c98ba443 vars.name="test/nodejs-example"
Note the err "filesystem: ... permission denied" message above.
Environment
- OpenShift Enterprise 3.1.1 or later, upgraded from an earlier version
docker-registry
deployment using a privileged security context constraint (SCC)
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.