"noProxy" variable is not referenced during the OCP IPI Baremetal Installation

Solution Verified - Updated -

Environment

  • Red Hat OpenShift Container Platform (OCP) 4.8

Issue

  • Variable noProxy has been set to download the OS image from an internal repository, but curl command used by the ironic-coreos-downloader process is not able get the image as proxy configuration is being referenced even with the variable noProxy is exported correctly during the downloading process.

Resolution

  • Verify the internal registry IP where OS image is located in file install-config.yaml:
   $ grep -i OSImage install-config.yaml 
    bootstrapOSImage: http://10.190.5.3/rhcos-48-qemu.x86_64.qcow2.gz?sha256=5d21de07eb45ff169d5780b9401c9745738d39dea357b928a0b83355cb6d088a
    clusterOSImage: http://10.190.5.3/rhcos-openstack.qcow2.gz?sha256=28af4abb51171eb374691e2a11cba5134fe47db896062e38f9c697e53ba8edfb
  • Add the IP to the list of the variable noProxy:
  $ grep -i Proxy install-config.yaml
    proxy:
      httpProxy: http://pxuser:pxpsswd@proxy.testbmipi.es:18080
      httpsProxy: http://pxuser:pxpsswd@proxy.testbmipi.es:18080
      noProxy: .testbmipi.es,.testbmipi,10.5.1.0/8,10.190.5.0/23,10.190.5.3

Root Cause

  • As per the documentation referenced to proxy-config-openshift network CIDRs can be specified as a value in the list of variable noProxy, but curl used by the ironic-coreos-downloader process is not recognizing it therefore the proxy is used as default.
  • Seems to be a limitation of curl or wget tools.
  • Flaw reported in bug 1990556.

Diagnostic Steps

  • Internal repository is reachable from the provisoner node:
$ curl -I http://10.190.5.3/rhcos-48-qemu.x86_64.qcow2.gz
HTTP/1.1 200 OK
...
ETag: "8188c73d-3c955b97-5c7ba3d02ee40"
Accept-Ranges: bytes
Content-Length: 1016421271
Strict-Transport-Security: max-age=15768000, includeSubDomains
Connection: close
Content-Type: application/x-gzip
  • The installer is not able to mount boot image on masters nodes.
  • Log ironic.log reports that image can't be found:
   $ grep "404 Not Found" bootstrap/journals/ironic.log 
   Aug 10 06:47:13 localhost.localdomain startironic.sh[2175]: curl: (22) The requested URL returned error: 404 Not Found
   ...
   Aug 10 06:51:07 localhost.localdomain startironic.sh[2175]: curl: (22) The requested URL returned error: 404 Not Found
   Aug 10 06:51:08 localhost.localdomain startironic.sh[2175]: curl: (22) The requested URL returned error: 404 Not Found
  • Log coreos-downloader-xxx.log reports that images can't be uncompressed:
   $ grep -ri  "not in gzip format" bootstrap/pods/coreos-downloader-53e87bcfecad.log
   gzip: rhcos-openstack.qcow2.gz: not in gzip format

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.