Chapter 6. Configuring a cluster-wide proxy

If you are using an existing Virtual Private Cloud (VPC), you can configure a cluster-wide proxy during a Red Hat OpenShift Service on AWS (ROSA) cluster installation or after the cluster is installed. When you enable a proxy, the core cluster components are denied direct access to the internet, but the proxy does not affect user workloads.

Note

Only cluster system egress traffic is proxied, including calls to the cloud provider API.

If you use a cluster-wide proxy, you are responsible for maintaining the availability of the proxy to the cluster. If the proxy becomes unavailable, then it might impact the health and supportability of the cluster.

6.1. Prerequisites for configuring a cluster-wide proxy

To configure a cluster-wide proxy, you must meet the following requirements. These requirements are valid when you configure a proxy during installation or postinstallation.

General requirements

  • You are the cluster owner.
  • Your account has sufficient privileges.
  • You have an existing Virtual Private Cloud (VPC) for your cluster.
  • The proxy can access the VPC for the cluster and the private subnets of the VPC. The proxy is also accessible from the VPC for the cluster and from the private subnets of the VPC.

  • You have added the following endpoints to your VPC endpoint:

    • ec2.<aws_region>.amazonaws.com
    • elasticloadbalancing.<aws_region>.amazonaws.com

    • s3.<aws_region>.amazonaws.com

      These endpoints are required to complete requests from the nodes to the AWS EC2 API. Because the proxy works at the container level and not at the node level, you must route these requests to the AWS EC2 API through the AWS private network. Adding the public IP address of the EC2 API to your allowlist in your proxy server is not enough.

      Note

      When using a cluster-wide proxy, you must configure the s3.<aws_region>.amazonaws.com endpoint as type Gateway. Also, you can configure the ec2.<aws_region>.amazonaws.com and elasticloadbalancing.<aws_region>.amazonaws.com endpoints only as type Interface.

Network requirements

  • If your proxy re-encrypts egress traffic, you must create exclusions to the domain and port combinations. The following table offers guidance into these exceptions.

    • Your proxy must exclude re-encrypting the following OpenShift URLs:

      AddressProtocol/PortFunction

      observatorium-mst.api.openshift.com

      https/443

      Required. Used for Managed OpenShift-specific telemetry.

      sso.redhat.com

      https/443

      The https://cloud.redhat.com/openshift site uses authentication from sso.redhat.com to download the cluster pull secret and use Red Hat SaaS solutions to facilitate monitoring of your subscriptions, cluster inventory, and chargeback reporting.

    • Your proxy must exclude re-encrypting the following site reliability engineering (SRE) and management URLs:

      AddressProtocol/PortFunction

      *.osdsecuritylogs.splunkcloud.com

      OR

      inputs1.osdsecuritylogs.splunkcloud.cominputs2.osdsecuritylogs.splunkcloud.cominputs4.osdsecuritylogs.splunkcloud.cominputs5.osdsecuritylogs.splunkcloud.cominputs6.osdsecuritylogs.splunkcloud.cominputs7.osdsecuritylogs.splunkcloud.cominputs8.osdsecuritylogs.splunkcloud.cominputs9.osdsecuritylogs.splunkcloud.cominputs10.osdsecuritylogs.splunkcloud.cominputs11.osdsecuritylogs.splunkcloud.cominputs12.osdsecuritylogs.splunkcloud.cominputs13.osdsecuritylogs.splunkcloud.cominputs14.osdsecuritylogs.splunkcloud.cominputs15.osdsecuritylogs.splunkcloud.com

      tcp/9997

      Used by the splunk-forwarder-operator as a log forwarding endpoint to be used by Red Hat SRE for log-based alerting.

      http-inputs-osdsecuritylogs.splunkcloud.com

      https/443

      Used by the splunk-forwarder-operator as a log forwarding endpoint to be used by Red Hat SRE for log-based alerting.

    Important

    The use of a proxy server to perform TLS re-encryption is currently not supported if the server is acting as a transparent forward proxy where it is not configured on-cluster via the --http-proxy or --https-proxy arguments.

    A transparent forward proxy intercepts the cluster traffic, but it is not actually configured on the cluster itself.

Additional Resources

6.2. Responsibilities for additional trust bundles

If you supply an additional trust bundle, you are responsible for the following requirements:

  • Ensuring that the contents of the additional trust bundle are valid
  • Ensuring that the certificates, including intermediary certificates, contained in the additional trust bundle have not expired
  • Tracking the expiry and performing any necessary renewals for certificates contained in the additional trust bundle
  • Updating the cluster configuration with the updated additional trust bundle

6.3. Configuring a proxy during installation

You can configure an HTTP or HTTPS proxy when you install a Red Hat OpenShift Service on AWS (ROSA) cluster into an existing Virtual Private Cloud (VPC). You can configure the proxy during installation by using Red Hat OpenShift Cluster Manager or the ROSA CLI (rosa).

6.3.1. Configuring a proxy during installation using OpenShift Cluster Manager

If you are installing a Red Hat OpenShift Service on AWS (ROSA) cluster into an existing Virtual Private Cloud (VPC), you can use Red Hat OpenShift Cluster Manager to enable a cluster-wide HTTP or HTTPS proxy during installation.

Prior to the installation, you must verify that the proxy is accessible from the VPC that the cluster is being installed into. The proxy must also be accessible from the private subnets of the VPC.

For detailed steps to configure a cluster-wide proxy during installation by using OpenShift Cluster Manager, see Creating a cluster with customizations by using OpenShift Cluster Manager.

6.3.2. Configuring a proxy during installation using the CLI

If you are installing a Red Hat OpenShift Service on AWS (ROSA) cluster into an existing Virtual Private Cloud (VPC), you can use the ROSA CLI (rosa) to enable a cluster-wide HTTP or HTTPS proxy during installation.

The following procedure provides details about the ROSA CLI (rosa) arguments that are used to configure a cluster-wide proxy during installation. For general installation steps using the ROSA CLI, see Creating a cluster with customizations using the CLI.

Prerequisites

  • You have verified that the proxy is accessible from the VPC that the cluster is being installed into. The proxy must also be accessible from the private subnets of the VPC.

Procedure

  • Specify a proxy configuration when you create your cluster: 

    $ rosa create cluster \
 <other_arguments_here> \
 --additional-trust-bundle-file <path_to_ca_bundle_file> \ 1 2 3
 --http-proxy http://<username>:<password>@<ip>:<port> \ 4 5
 --https-proxy https://<username>:<password>@<ip>:<port> \ 6 7
 --no-proxy example.com 8
    1 4 6
    The additional-trust-bundle-file, http-proxy, and https-proxy arguments are all optional.
    2
    If you use the additional-trust-bundle-file argument without an http-proxy or https-proxy argument, the trust bundle is added to the trust store and used to verify cluster system egress traffic. In that scenario, the bundle is not configured to be used with a proxy.
    3
    The additional-trust-bundle-file argument is a file path pointing to a bundle of PEM-encoded X.509 certificates, which are all concatenated together. The additionalTrustBundle parameter is required unless the identity certificate of the proxy is signed by an authority from the RHCOS trust bundle. If you use an MITM transparent proxy network that does not require additional proxy configuration but requires additional CAs, you must provide the MITM CA certificate.
    5 7
    The http-proxy and https-proxy arguments must point to a valid URL.
    8
    A comma-separated list of destination domain names, IP addresses, or network CIDRs to exclude proxying.

    Preface a domain with . to match subdomains only. For example, .y.com matches x.y.com, but not y.com. Use * to bypass proxy for all destinations. If you scale up workers that are not included in the network defined by the networking.machineNetwork[].cidr field from the installation configuration, you must add them to this list to prevent connection issues.

    This field is ignored if neither the httpProxy or httpsProxy fields are set.

Additional Resources

6.4. Configuring a proxy after installation

You can configure an HTTP or HTTPS proxy after you install a Red Hat OpenShift Service on AWS (ROSA) cluster into an existing Virtual Private Cloud (VPC). You can configure the proxy after installation by using Red Hat OpenShift Cluster Manager or the ROSA CLI (rosa).

6.4.1. Configuring a proxy after installation using OpenShift Cluster Manager

You can use Red Hat OpenShift Cluster Manager to add a cluster-wide proxy configuration to an existing Red Hat OpenShift Service on AWS cluster in a Virtual Private Cloud (VPC).

You can also use OpenShift Cluster Manager to update an existing cluster-wide proxy configuration. For example, you might need to update the network address for the proxy or replace the additional trust bundle if any of the certificate authorities for the proxy expire.

Important

The cluster applies the proxy configuration to the control plane and compute nodes. While applying the configuration, each cluster node is temporarily placed in an unschedulable state and drained of its workloads. Each node is restarted as part of the process.

Prerequisites

  • You have an Red Hat OpenShift Service on AWS cluster .
  • Your cluster is deployed in a VPC.

Procedure

  1. Navigate to OpenShift Cluster Manager and select your cluster.
  2. Under the Virtual Private Cloud (VPC) section on the Networking page, click Edit cluster-wide proxy.

  3. On the Edit cluster-wide proxy page, provide your proxy configuration details:

    1. Enter a value in at least one of the following fields:

      • Specify a valid HTTP proxy URL.
      • Specify a valid HTTPS proxy URL.

      • In the Additional trust bundle field, provide a PEM encoded X.509 certificate bundle. If you are replacing an existing trust bundle file, select Replace file to view the field. The bundle is added to the trusted certificate store for the cluster nodes. An additional trust bundle file is required unless the identity certificate for the proxy is signed by an authority from the Red Hat Enterprise Linux CoreOS (RHCOS) trust bundle.

        If you use an MITM transparent proxy network that does not require additional proxy configuration but requires additional certificate authorities (CAs), you must provide the MITM CA certificate.

        Note

        If you upload an additional trust bundle file without specifying an HTTP or HTTPS proxy URL, the bundle is set on the cluster but is not configured to be used with the proxy.

    2. Click Confirm.

Verification

  • Under the Virtual Private Cloud (VPC) section on the Networking page, verify that the proxy configuration for your cluster is as expected.

6.4.2. Configuring a proxy after installation using the CLI

You can use the Red Hat OpenShift Service on AWS (ROSA) CLI (rosa) to add a cluster-wide proxy configuration to an existing ROSA cluster in a Virtual Private Cloud (VPC).

You can also use rosa to update an existing cluster-wide proxy configuration. For example, you might need to update the network address for the proxy or replace the additional trust bundle if any of the certificate authorities for the proxy expire.

Important

The cluster applies the proxy configuration to the control plane and compute nodes. While applying the configuration, each cluster node is temporarily placed in an unschedulable state and drained of its workloads. Each node is restarted as part of the process.

Prerequisites

  • You have installed and configured the latest ROSA (rosa) and OpenShift (oc) CLIs on your installation host.
  • You have a ROSA cluster that is deployed in a VPC.

Procedure

  • Edit the cluster configuration to add or update the cluster-wide proxy details: 

    $ rosa edit cluster \
 --cluster $CLUSTER_NAME \
 --additional-trust-bundle-file <path_to_ca_bundle_file> \ 1 2 3
 --http-proxy http://<username>:<password>@<ip>:<port> \ 4 5
 --https-proxy https://<username>:<password>@<ip>:<port> \ 6 7
  --no-proxy example.com 8
    1 4 6
    The additional-trust-bundle-file, http-proxy, and https-proxy arguments are all optional.
    2
    If you use the additional-trust-bundle-file argument without an http-proxy or https-proxy argument, the trust bundle is added to the trust store and used to verify cluster system egress traffic. In that scenario, the bundle is not configured to be used with a proxy.
    3
    The additional-trust-bundle-file argument is a file path pointing to a bundle of PEM-encoded X.509 certificates, which are all concatenated together. The additionalTrustBundle parameter is required unless the identity certificate of the proxy is signed by an authority from the RHCOS trust bundle. If you use an MITM transparent proxy network that does not require additional proxy configuration but requires additional CAs, you must provide the MITM CA certificate.
    Note

    You should not attempt to change the proxy or additional trust bundle configuration on the cluster directly. These changes must be applied by using the ROSA CLI (rosa) or Red Hat OpenShift Cluster Manager. Any changes that are made directly to the cluster will be reverted automatically.

    5 7
    The http-proxy and https-proxy arguments must point to a valid URL.
    8
    A comma-separated list of destination domain names, IP addresses, or network CIDRs to exclude proxying.

    Preface a domain with . to match subdomains only. For example, .y.com matches x.y.com, but not y.com. Use * to bypass proxy for all destinations. If you scale up workers that are not included in the network defined by the networking.machineNetwork[].cidr field from the installation configuration, you must add them to this list to prevent connection issues.

    This field is ignored if neither the httpProxy or httpsProxy fields are set.

Verification

  1. List the status of the machine config pools and verify that they are updated: 

    $ oc get machineconfigpools

    Example output

    NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
master   rendered-master-d9a03f612a432095dcde6dcf44597d90   True      False      False      3              3                   3                     0                      31h
worker   rendered-worker-f6827a4efe21e155c25c21b43c46f65e   True      False      False      6              6                   6                     0                      31h

  2. Display the proxy configuration for your cluster and verify that the details are as expected: 

    $ oc get proxy cluster -o yaml

    Example output

    apiVersion: config.openshift.io/v1
kind: Proxy
spec:
  httpProxy: http://proxy.host.domain:<port>
  httpsProxy: https://proxy.host.domain:<port>
  <...more...>
status:
  httpProxy: http://proxy.host.domain:<port>
  httpsProxy: https://proxy.host.domain:<port>
  <...more...>

6.5. Removing a cluster-wide proxy

You can remove your cluster-wide proxy by using the ROSA CLI. After removing the cluster, you should also remove any trust bundles that are added to the cluster.

6.5.1. Removing the cluster-wide proxy using CLI

You must use the Red Hat OpenShift Service on AWS (ROSA) CLI, rosa, to remove the proxy’s address from your cluster.

Prerequisites

  • You must have cluster administrator privileges.
  • You have installed the ROSA CLI (rosa).

Procedure

  • Use the rosa edit command to modify the proxy. You must pass empty strings to the --http-proxy and --https-proxy arguments to clear the proxy from the cluster: 

    $ rosa edit cluster -c <cluster_name> --http-proxy "" --https-proxy ""
    Note

    While your proxy might only use one of the proxy arguments, the empty fields are ignored, so passing empty strings to both the --http-proxy and --https-proxy arguments do not cause any issues.

    Example Output

    I: Updated cluster <cluster_name>

Verification

  • You can verify that the proxy has been removed from the cluster by using the rosa describe command: 

    $ rosa describe cluster -c <cluster_name>

    Before removal, the proxy IP displays in a proxy section: 

    Name:                       <cluster_name>
ID:                         <cluster_internal_id>
External ID:                <cluster_external_id>
OpenShift Version:          4.0
Channel Group:              stable
DNS:                        <dns>
AWS Account:                <aws_account_id>
API URL:                    <api_url>
Console URL:                <console_url>
Region:                     us-east-1
Multi-AZ:                   false
Nodes:
 - Control plane:           3
 - Infra:                   2
 - Compute:                 2
Network:
 - Type:                    OVNKubernetes
 - Service CIDR:            <service_cidr>
 - Machine CIDR:            <machine_cidr>
 - Pod CIDR:                <pod_cidr>
 - Host Prefix:             <host_prefix>
Proxy:
 - HTTPProxy:               <proxy_url>
Additional trust bundle:    REDACTED

    After removing the proxy, the proxy section is removed: 

    Name:                       <cluster_name>
ID:                         <cluster_internal_id>
External ID:                <cluster_external_id>
OpenShift Version:          4.0
Channel Group:              stable
DNS:                        <dns>
AWS Account:                <aws_account_id>
API URL:                    <api_url>
Console URL:                <console_url>
Region:                     us-east-1
Multi-AZ:                   false
Nodes:
 - Control plane:           3
 - Infra:                   2
 - Compute:                 2
Network:
 - Type:                    OVNKubernetes
 - Service CIDR:            <service_cidr>
 - Machine CIDR:            <machine_cidr>
 - Pod CIDR:                <pod_cidr>
 - Host Prefix:             <host_prefix>
Additional trust bundle:    REDACTED

6.5.2. Removing certificate authorities on a Red Hat OpenShift Service on AWS cluster

You can remove certificate authorities (CA) from your cluster with the Red Hat OpenShift Service on AWS (ROSA) CLI, rosa.

Prerequisites

  • You must have cluster administrator privileges.
  • You have installed the ROSA CLI (rosa).
  • Your cluster has certificate authorities added.

Procedure

  • Use the rosa edit command to modify the CA trust bundle. You must pass empty strings to the --additional-trust-bundle-file argument to clear the trust bundle from the cluster: 

    $ rosa edit cluster -c <cluster_name> --additional-trust-bundle-file ""

    Example Output

    I: Updated cluster <cluster_name>

Verification

  • You can verify that the trust bundle has been removed from the cluster by using the rosa describe command: 

    $ rosa describe cluster -c <cluster_name>

    Before removal, the Additional trust bundle section appears, redacting its value for security purposes: 

    Name:                       <cluster_name>
ID:                         <cluster_internal_id>
External ID:                <cluster_external_id>
OpenShift Version:          4.0
Channel Group:              stable
DNS:                        <dns>
AWS Account:                <aws_account_id>
API URL:                    <api_url>
Console URL:                <console_url>
Region:                     us-east-1
Multi-AZ:                   false
Nodes:
 - Control plane:           3
 - Infra:                   2
 - Compute:                 2
Network:
 - Type:                    OVNKubernetes
 - Service CIDR:            <service_cidr>
 - Machine CIDR:            <machine_cidr>
 - Pod CIDR:                <pod_cidr>
 - Host Prefix:             <host_prefix>
Proxy:
 - HTTPProxy:               <proxy_url>
Additional trust bundle:    REDACTED

    After removing the proxy, the Additional trust bundle section is removed: 

    Name:                       <cluster_name>
ID:                         <cluster_internal_id>
External ID:                <cluster_external_id>
OpenShift Version:          4.0
Channel Group:              stable
DNS:                        <dns>
AWS Account:                <aws_account_id>
API URL:                    <api_url>
Console URL:                <console_url>
Region:                     us-east-1
Multi-AZ:                   false
Nodes:
 - Control plane:           3
 - Infra:                   2
 - Compute:                 2
Network:
 - Type:                    OVNKubernetes
 - Service CIDR:            <service_cidr>
 - Machine CIDR:            <machine_cidr>
 - Pod CIDR:                <pod_cidr>
 - Host Prefix:             <host_prefix>
Proxy:
 - HTTPProxy:               <proxy_url>