Failure to push external image to internal OpenShift docker registry backed by AWS S3

Issue

Conditions of failure attempting to push image:

• Pushing from external workstation using docker or podman to internal OpenShift registry
• OpenShift internal registry is backed by AWS S3 bucket
• Smaller image pushes appear to work fine. Images with layers larger than 100MB fail.
• Failure is either EOF or 504 Gateway Timeout
  

An example output:

$ sudo docker -D --log-level debug  push docker-registry-default.apps.ocp.example.com/testproj/test:latestThe push refers to a repository [docker-registry-default.apps.ocp.example.com/testproj/test]
193a94256642: Pushing [==================================================>] 542.2 MB
6d6356604420: Layer already exists 
c2dc8b8960d7: Layer already exists 
EOF

Log messages from the docker client side daemon during the push:

2019-05-29 May 21:20:07 localhost dockerd-current: time="2019-05-29T21:20:07.599745354-06:00" level=error msg="Upload failed, retrying: EOF"
2019-05-29 May 21:22:36 localhost dockerd-current: time="2019-05-29T21:22:36.471705703-06:00" level=error msg="Upload failed, retrying: EOF"
2019-05-29 May 21:24:52 localhost dockerd-current: time="2019-05-29T21:24:52.469247915-06:00" level=error msg="Upload failed, retrying: EOF"
2019-05-29 May 21:27:09 localhost dockerd-current: time="2019-05-29T21:27:09.603556168-06:00" level=error msg="Upload failed, retrying: EOF"
2019-05-29 May 21:29:35 localhost dockerd-current: time="2019-05-29T21:29:35.466561966-06:00" level=error msg="Upload failed: EOF"
2019-05-29 May 21:29:35 localhost dockerd-current: time="2019-05-29T21:29:35.468127954-06:00" level=error msg="Attempting next endpoint for push after error: EOF"

Alternatively, you may see a 504 Gateway Timeout error:

$ sudo docker -D --log-level debug  push docker-registry-default.apps.ocp.example.com/testproj/test:latest
The push refers to a repository [docker-registry-default.apps.ocp.example.com/testproj/test]
d6d55944ae7d: Pushing [==================================================>] 199.2 MB
4275739fdcbd: Pushing [=================================>                 ] 230.3 MB/340.4 MB
0d7b0f030dd9: Pushed
c613b100be16: Pushed
24d85c895b6b: Pushing [==================================================>] 215.8 MB
received unexpected HTTP status: 504 Gateway Time-out