Usage of port 22623 in OpenShift 4

Solution Verified - Updated -

Environment

  • Red Hat OpenShift Container Platform
    • 4.1 to 4.5

Issue

  • What is the purpose of port 22623 in OpenShift Container Platform 4?
  • What is the URL associated with port 22623?

  • Error adding worker nodes to the cluster:

    internal server error(ignition: GET error: Get "https://api-int.[cluster-name].[base-domain]:22623/config/worker":dial tcp: lookup api-int.[cluster-name].[base-domain] on 10.0.0.1:53: connect: network is unreachable

Resolution

That port is used by the machine config operator to provide the configuration (in the form of ignition files) to master and worker nodes. At the time of installation, the machine-config-server is first set up on bootstrap, and after that, it runs on all masters to provide config to worker nodes.

The URLs used with the port 22623 are:

https://api-int.[cluster-name].[base-domain]:22623/config/master
https://api-int.[cluster-name].[base-domain]:22623/config/worker
https://api.[cluster-name].[base-domain]:22623/config/master
https://api.[cluster-name].[base-domain]:22623/config/worker

Root Cause

The port 22623 is required for fetching the Ignition config for new machines in the cluster from master nodes.
However note that since OpenShift 4.5 that port is blocked by a firewall rule once the node is deployed

Diagnostic Steps

  • Check the master and worker ignition files using curl:
$ curl -k https://api-int.[cluster-name].[base-domain]:22623/config/master
$ curl -k https://api-int.[cluster-name].[base-domain]:22623/config/worker
$ curl -k https://api.[cluster-name].[base-domain]:22623/config/master
$ curl -k https://api.[cluster-name].[base-domain]:22623/config/worker
  • The Ignition files are not available on port 6443:
$ curl -k https://api-int.[cluster-name].[base-domain]:6443/config/worker
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/config/worker\"",
  "reason": "Forbidden",
  "details": {

  },
  "code": 403

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments