crictl cannot pull images, but podman can

Latest response

we have a RHCOS+openshift4.4 container platform using vSphere/VmWare .

Issue: crictl cannot pull images from internal dockerhub.internal.com repository. It can pull images from hub.docker.com

However, podman can pull from dockerhub.internal.com and hub.docker.com

but, both crictl and podman can pull images from cloud repo hub.docker.com. ie. crictl pull mango:latest and podman pull mongo:latest work fine.

following command to pull image from internal repo/registry fails

crictl pull dockerhub.internal.com/mongo:2.2
. . . 
pulling image failed: rpc error:  code =  unknown desc = error pinging docker registry 

has anyone faced this situation.

Responses

Hi tk,

can you run the command with the --debug/-d flag?

$ crictl pull dockerhub.internal.com/mongo:2.2 -D

The podman pull works fine: (sorry, example is with camunda, not mongo)

podman pull dockerhub.internal.com/cx-bac-bpa-internal-docker/camunda:2.2

however did the command as you suggested:

crictl pull dockerhub.internal.com/cx-bac-bpa-internal-docker/camunda:2.2 -D

FATA[0150] pulling image failed: rpc error: code = Unknown desc = error pinging docker registry dockerhub.internal.com: Get https://dockerhub.internal.com/v2/: Bad Gateway

I wonder if there is anything set for CRIO in etc/sysconfig/crio , perhaps proxy.

I agree with checking this as a starting point; but rather I'd wonder if something is set on the system that's not being honored by crictl

Stephen, Hevellyn,
Thanks a LOT for your replies and support.

went through /etc/crio/crio.conf. As you suggested on proxy may be an issue, I was looking for a proxy config or a network settings that points to a proxy server. There is no proxy settings at this file under section [crio.network].
file: /etc/crio/crio.conf section network

[crio.network]
# Path to the directory where CNI configuration files are located.
network_dir = "/etc/kubernetes/cni/net.d/"   

no proxy settings in files under /etc/kubernetes/cni/net.d/.

The clusterwide proxy are setup at ignition files while installing the workers.
perhaps that is why podman does not complain as it uses cluster wide proxy settings. (HTTP_PROXY, HTTPS_PROXY and NO_PROXY settings).
However, if that is the case, crio should also pick up the cluster wide proxy settings. Am I right?

Thanks.

file: /etc/sysconfig/crio-metrics
CRIO_METRICS_OPTIONS="--enable-metrics=true --metrics-port=9537"

file: /etc/sysconfig/crio-network
CRIO_NETWORK_OPTIONS=

file: /etc/sysconfig/crio-storage
CRIO_STORAGE_OPTIONS=

dig command output for dockerhub.internal.com: The dig on dockerhub shows these possible paths: dockerhub.ci.engit.internal.com must be an alias to dockerhub.internal.com as ip address is same.

$ dig +short dockerhub.internal.com
dockerhub.ci.engit.internal.com.
dockerhub.geo.ci.engit.internal.com.
dockerhub.rtp.geo.ci.engit.internal.com.
64.102.x.x
10.83.x.x

the following throws me for a loop:

https://dockerhub.internal.com/v2/: Bad Gateway

what happens when you do a raw curl against the registry?

HI Stephen, posting a curl response here.:

curl -Ss -k https://dockerhub.internal.com/cx-bac-bpa-internal-docker/camunda:2.2.1-b
output

<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 500 </title>
</head>
<body>
<h2>HTTP ERROR: 500</h2>
<p>Problem accessing /cx-bac-bpa-internal-docker/camunda:2.2.1-b. Reason:
<pre>    com.internal.shim.InvalidPathException</pre></p>
<hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.z-SNAPSHOT</a><hr/>
</body>
</html>

just for consistency's sake here, what do the logs say?

oc logs deployments/image-registry

the namespace openshift-image-registry has an operator pod running as

$ oc describe deployments -n openshift-image-registry
Name:                   cluster-image-registry-operator
Namespace:              openshift-image-registry
CreationTimestamp:      Sun, 04 Oct 2020 14:42:45 -0500
Labels:                 <none>
Annotations:            config.openshift.io/inject-proxy: cluster-image-registry-operator
                        deployment.kubernetes.io/revision: 3
. . .

under this namespace, i have to get the pod and then get into container to get the logs.

$ oc get pods
NAME                                               READY   STATUS    RESTARTS   AGE
cluster-image-registry-operator-75fcf7c6db-ldhxh   2/2     Running   0          10d

Then get the logs of the container inside this pod as:

oc logs cluster-image-registry-operator-75fcf7c6db-ldhxh cluster-image-registry-operator

I1016 18:34:09.650406      15 clusteroperator.go:98] event from workqueue successfully processed
I1016 18:34:09.651357      15 clusteroperator.go:98] event from workqueue successfully processed
I1016 18:34:10.548299      15 controller.go:215] object changed: *v1.Config, Name=cluster (status=true): changed:status.conditions.0.lastTransitionTime={"2020-10-16T18:34:09Z" -> "2020-10-16T18:34:10Z"}
I1016 18:34:10.574769      15 controller.go:255] event from workqueue successfully processed
I1016 18:34:10.575008      15 controllerimagepruner.go:270] event from image pruner workqueue successfully processed
I1016 18:34:10.626109      15 generator.go:59] object *v1.ClusterOperator, Name=image-registry updated: changed:metadata.resourceVersion={"5752715" -> "5752721"}, changed:metadata.selfLink={"/apis/config.openshift.io/v1/clusteroperators/image-registry" -> "/apis/config.openshift.io/v1/clusteroperators/image-registry/status"}, changed:status.conditions.0.lastTransitionTime={"2020-10-16T18:34:09Z" -> "2020-10-16T18:34:10Z"}
I1016 18:34:10.626131      15 clusteroperator.go:98] event from workqueue successfully processed
I1016 18:34:10.648369      15 clusteroperator.go:98] event from workqueue successfully processed
I1016 18:34:11.599609      15 controller.go:215] object changed: *v1.Config, Name=cluster (status=true): changed:status.conditions.0.lastTransitionTime={"2020-10-16T18:34:10Z" -> "2020-10-16T18:34:11Z"}
I1016 18:34:11.614895      15 controller.go:255] event from workqueue successfully processed
I1016 18:34:11.617026      15 controllerimagepruner.go:270] event from image pruner workqueue successfully processed
I1016 18:34:11.704645      15 generator.go:59] object *v1.ClusterOperator, Name=image-registry updated: changed:metadata.resourceVersion={"5752721" -> "5752732"}, changed:metadata.selfLink={"/apis/config.openshift.io/v1/clusteroperators/image-registry" -> "/apis/config.openshift.io/v1/clusteroperators/image-registry/status"}, changed:status.conditions.0.lastTransitionTime={"2020-10-16T18:34:10Z" -> "2020-10-16T18:34:11Z"}
I1

Does this make sense?

Yeah, it's fine - I was hoping to get lucky and we'd see an internal error - but we don't.

long and short it looks like Jetty is having an issue with the image path, but there's no obvious reason why - especially if podman works. I'm going to say that unfortunately, sounds like you need to open a support ticket on this one.

Hi Stephen,

I was doubting on where the crio sets up its proxy. Found it. At the workers, they are at /etc/systemd/system/crio.service.d/10-default-env.conf: (picked up from ignition config files during install)

[Service]
Environment=HTTP_PROXY=http://12.12.12.50:8888
Environment=HTTPS_PROXY=http://12.12.12.50:8888
Environment=NO_PROXY=.cluster.local,.robot.com,.svc,10.0.0.0/16,10.254.0.0/16,127.0.0.1,172.30.0.0/16,api-int.ocp4.robot.com,etcd-0.ocp4.robot.com,etcd-1.ocp4.robot.com,etcd-2.ocp4.robot.com,localhost

Yes, will raise a ticket with Jetty admin group. Thanks a LOT for your patience. Have a good weekend.