getting 504 gateway timeout error frequently when trying to pull/push images to the internal registry.

Issue

• While pulling/pushing images to internal image registry observed 504 gateway timeout error very frequently
• Below messages are being observed in image registry pod logs:

time="2024-04-03T05:01:13.063117265Z" level=error msg="response completed with error" err.code=unknown err.detail="s3aws: SerializationError: failed to unmarshal error message\n\tstatus code: 504, request id: , host id: \ncaused by: UnmarshalError: failed to unmarshal error message

• podman command is throwing error:

$ podman pull default-route-openshift-image-registry.apps.domain.com/my-repo/my-image:v1.2.3
<<snip>>
Trying to pull default-route-openshift-image-registry.apps.domain.com/my-repo/my-image:v1.2.3...
Error: initializing source docker://default-route-openshift-image-registry.apps.domain.com/my-repo/my-image:v1.2.3: reading manifest v1.2.3 in default-route-openshift-image-registry.apps.domain.com/my-repo/my-image: received unexpected HTTP status: 504 Gateway Time-out