• Multicast packets sent by a pod will be delivered to all other pods in the project, regardless of NetworkPolicy objects. Pods might be able to communicate over multicast even when they cannot communicate over unicast.
• Multicast packets sent by a pod in one project will never be delivered to pods in any other project, even if there are NetworkPolicy objects that allow communication between the projects.

• Multicast packets sent by a pod will be delivered to all other pods in the project.
• Multicast packets sent by a pod in one project will be delivered to pods in other projects only if each project is joined together and multicast is enabled in each joined project.

Prerequisites

• Install the OpenShift CLI (oc).
• You must log in to the cluster with a user that has the cluster-admin or the dedicated-admin role.

Procedure

• Run the following command to enable multicast for a project. Replace <namespace> with the namespace for the project you want to enable multicast for.

  $ oc annotate netnamespace <namespace> \
      netnamespace.network.openshift.io/multicast-enabled=true

Prerequisites

• You have verified that the proxy is accessible from the VPC that the cluster is being installed into. The proxy must also be accessible from the private subnets of the VPC.

Procedure

• Specify a proxy configuration when you create your cluster:

  $ rosa create cluster \
   <other_arguments_here> \
   --additional-trust-bundle-file <path_to_ca_bundle_file> \ 1 2 3
   --http-proxy http://<username>:<password>@<ip>:<port> \ 4 5
   --https-proxy https://<username>:<password>@<ip>:<port> \ 6 7
   --no-proxy example.com 8

  1 4 6

  The additional-trust-bundle-file, http-proxy, and https-proxy arguments are all optional.

  2

  If you use the additional-trust-bundle-file argument without an http-proxy or https-proxy argument, the trust bundle is added to the trust store and used to verify cluster system egress traffic. In that scenario, the bundle is not configured to be used with a proxy.

  3

  The additional-trust-bundle-file argument is a file path pointing to a bundle of PEM-encoded X.509 certificates, which are all concatenated together. The additionalTrustBundle parameter is required unless the identity certificate of the proxy is signed by an authority from the RHCOS trust bundle. If you use an MITM transparent proxy network that does not require additional proxy configuration but requires additional CAs, you must provide the MITM CA certificate.

  5 7

  The http-proxy and https-proxy arguments must point to a valid URL.

  8

  A comma-separated list of destination domain names, IP addresses, or network CIDRs to exclude proxying.

  Preface a domain with . to match subdomains only. For example, .y.com matches x.y.com, but not y.com. Use * to bypass proxy for all destinations. If you scale up workers that are not included in the network defined by the networking.machineNetwork[].cidr field from the installation configuration, you must add them to this list to prevent connection issues.

  This field is ignored if neither the httpProxy or httpsProxy fields are set.

Prerequisites

• You have an Red Hat OpenShift Service on AWS cluster .
• Your cluster is deployed in a VPC.

Verification

• Under the Virtual Private Cloud (VPC) section on the Networking page, verify that the proxy configuration for your cluster is as expected.

Prerequisites

• You have installed and configured the latest ROSA (rosa) and OpenShift (oc) CLIs on your installation host.
• You have a ROSA cluster that is deployed in a VPC.

Procedure

• Edit the cluster configuration to add or update the cluster-wide proxy details:

  $ rosa edit cluster \
   --cluster $CLUSTER_NAME \
   --additional-trust-bundle-file <path_to_ca_bundle_file> \ 1 2 3
   --http-proxy http://<username>:<password>@<ip>:<port> \ 4 5
   --https-proxy https://<username>:<password>@<ip>:<port> \ 6 7
    --no-proxy example.com 8

  1 4 6

  The additional-trust-bundle-file, http-proxy, and https-proxy arguments are all optional.

  2

  If you use the additional-trust-bundle-file argument without an http-proxy or https-proxy argument, the trust bundle is added to the trust store and used to verify cluster system egress traffic. In that scenario, the bundle is not configured to be used with a proxy.

  3

  The additional-trust-bundle-file argument is a file path pointing to a bundle of PEM-encoded X.509 certificates, which are all concatenated together. The additionalTrustBundle parameter is required unless the identity certificate of the proxy is signed by an authority from the RHCOS trust bundle. If you use an MITM transparent proxy network that does not require additional proxy configuration but requires additional CAs, you must provide the MITM CA certificate.

  Note

  You should not attempt to change the proxy or additional trust bundle configuration on the cluster directly. These changes must be applied by using the ROSA CLI (rosa) or Red Hat OpenShift Cluster Manager. Any changes that are made directly to the cluster will be reverted automatically.

  5 7

  The http-proxy and https-proxy arguments must point to a valid URL.

  8

  A comma-separated list of destination domain names, IP addresses, or network CIDRs to exclude proxying.

  Preface a domain with . to match subdomains only. For example, .y.com matches x.y.com, but not y.com. Use * to bypass proxy for all destinations. If you scale up workers that are not included in the network defined by the networking.machineNetwork[].cidr field from the installation configuration, you must add them to this list to prevent connection issues.

  This field is ignored if neither the httpProxy or httpsProxy fields are set.

Prerequisites

• You must have cluster administrator privileges.
• You have installed the Red Hat OpenShift Service on AWS rosa CLI tool.

Procedure

• Use the rosa edit command to modify the proxy. You must pass empty strings to the --http-proxy and --https-proxy arguments to clear the proxy from the cluster:

  $ rosa edit cluster -c <cluster_name> --http-proxy "" --https-proxy ""

  Note

  While your proxy might only use one of the proxy arguments, the empty fields are ignored, so passing empty strings to both the --http-proxy and --https-proxy arguments do not cause any issues.

  Example Output

  I: Updated cluster <cluster_name>

Verification

• You can verify that the proxy has been removed from the cluster by using the rosa describe command:

  $ rosa describe cluster -c <cluster_name>

  Before removal, the proxy IP displays in a proxy section:

  Name:                       <cluster_name>
  ID:                         <cluster_internal_id>
  External ID:                <cluster_external_id>
  OpenShift Version:          4.13.0
  Channel Group:              stable
  DNS:                        <dns>
  AWS Account:                <aws_account_id>
  API URL:                    <api_url>
  Console URL:                <console_url>
  Region:                     us-east-1
  Multi-AZ:                   false
  Nodes:
   - Control plane:           3
   - Infra:                   2
   - Compute:                 2
  Network:
   - Type:                    OVNKubernetes
   - Service CIDR:            <service_cidr>
   - Machine CIDR:            <machine_cidr>
   - Pod CIDR:                <pod_cidr>
   - Host Prefix:             <host_prefix>
  Proxy:
   - HTTPProxy:               <proxy_url>
  Additional trust bundle:    REDACTED

  After removing the proxy, the proxy section is removed:

  Name:                       <cluster_name>
  ID:                         <cluster_internal_id>
  External ID:                <cluster_external_id>
  OpenShift Version:          4.13.0
  Channel Group:              stable
  DNS:                        <dns>
  AWS Account:                <aws_account_id>
  API URL:                    <api_url>
  Console URL:                <console_url>
  Region:                     us-east-1
  Multi-AZ:                   false
  Nodes:
   - Control plane:           3
   - Infra:                   2
   - Compute:                 2
  Network:
   - Type:                    OVNKubernetes
   - Service CIDR:            <service_cidr>
   - Machine CIDR:            <machine_cidr>
   - Pod CIDR:                <pod_cidr>
   - Host Prefix:             <host_prefix>
  Additional trust bundle:    REDACTED

Prerequisites

• You must have cluster administrator privileges.
• You have installed the rosa CLI tool.
• Your cluster has certificate authorities added.

Procedure

• Use the rosa edit command to modify the CA trust bundle. You must pass empty strings to the --additional-trust-bundle-file argument to clear the trust bundle from the cluster:

  $ rosa edit cluster -c <cluster_name> --additional-trust-bundle-file ""

  Example Output

  I: Updated cluster <cluster_name>

Verification

• You can verify that the trust bundle has been removed from the cluster by using the rosa describe command:

  $ rosa describe cluster -c <cluster_name>

  Before removal, the Additional trust bundle section appears, redacting its value for security purposes:

  Name:                       <cluster_name>
  ID:                         <cluster_internal_id>
  External ID:                <cluster_external_id>
  OpenShift Version:          4.13.0
  Channel Group:              stable
  DNS:                        <dns>
  AWS Account:                <aws_account_id>
  API URL:                    <api_url>
  Console URL:                <console_url>
  Region:                     us-east-1
  Multi-AZ:                   false
  Nodes:
   - Control plane:           3
   - Infra:                   2
   - Compute:                 2
  Network:
   - Type:                    OVNKubernetes
   - Service CIDR:            <service_cidr>
   - Machine CIDR:            <machine_cidr>
   - Pod CIDR:                <pod_cidr>
   - Host Prefix:             <host_prefix>
  Proxy:
   - HTTPProxy:               <proxy_url>
  Additional trust bundle:    REDACTED

  After removing the proxy, the Additional trust bundle section is removed:

  Name:                       <cluster_name>
  ID:                         <cluster_internal_id>
  External ID:                <cluster_external_id>
  OpenShift Version:          4.13.0
  Channel Group:              stable
  DNS:                        <dns>
  AWS Account:                <aws_account_id>
  API URL:                    <api_url>
  Console URL:                <console_url>
  Region:                     us-east-1
  Multi-AZ:                   false
  Nodes:
   - Control plane:           3
   - Infra:                   2
   - Compute:                 2
  Network:
   - Type:                    OVNKubernetes
   - Service CIDR:            <service_cidr>
   - Machine CIDR:            <machine_cidr>
   - Pod CIDR:                <pod_cidr>
   - Host Prefix:             <host_prefix>
  Proxy:
   - HTTPProxy:               <proxy_url>

• Deny all traffic:

  To make a project deny by default, add a NetworkPolicy object that matches all pods but accepts no traffic:

  kind: NetworkPolicy
  apiVersion: networking.k8s.io/v1
  metadata:
    name: deny-by-default
  spec:
    podSelector: {}
    ingress: []

• Only allow connections from the Red Hat OpenShift Service on AWS Ingress Controller:

  To make a project allow only connections from the Red Hat OpenShift Service on AWS Ingress Controller, add the following NetworkPolicy object.

  apiVersion: networking.k8s.io/v1
  kind: NetworkPolicy
  metadata:
    name: allow-from-openshift-ingress
  spec:
    ingress:
    - from:
      - namespaceSelector:
          matchLabels:
            network.openshift.io/policy-group: ingress
    podSelector: {}
    policyTypes:
    - Ingress

• Only accept connections from pods within a project:

  To make pods accept connections from other pods in the same project, but reject all other connections from pods in other projects, add the following NetworkPolicy object:

  kind: NetworkPolicy
  apiVersion: networking.k8s.io/v1
  metadata:
    name: allow-same-namespace
  spec:
    podSelector: {}
    ingress:
    - from:
      - podSelector: {}

• Only allow HTTP and HTTPS traffic based on pod labels:

  To enable only HTTP and HTTPS access to the pods with a specific label (role=frontend in following example), add a NetworkPolicy object similar to the following:

  kind: NetworkPolicy
  apiVersion: networking.k8s.io/v1
  metadata:
    name: allow-http-and-https
  spec:
    podSelector:
      matchLabels:
        role: frontend
    ingress:
    - ports:
      - protocol: TCP
        port: 80
      - protocol: TCP
        port: 443

• Accept connections by using both namespace and pod selectors:

  To match network traffic by combining namespace and pod selectors, you can use a NetworkPolicy object similar to the following:

  kind: NetworkPolicy
  apiVersion: networking.k8s.io/v1
  metadata:
    name: allow-pod-and-namespace-both
  spec:
    podSelector:
      matchLabels:
        name: test-pods
    ingress:
      - from:
        - namespaceSelector:
            matchLabels:
              project: project_name
          podSelector:
            matchLabels:
              name: test-pods

• Reduce the number of OVS flow rules by using namespaces to contain groups of pods that need to be isolated.

  NetworkPolicy objects that select a whole namespace, by using the namespaceSelector or an empty podSelector, generate only a single OVS flow rule that matches the VXLAN virtual network ID (VNID) of the namespace.

• Keep the pods that do not need to be isolated in their original namespace, and move the pods that require isolation into one or more different namespaces.
• Create additional targeted cross-namespace network policies to allow the specific traffic that you do want to allow from the isolated pods.

• For network policies with the same spec.podSelector spec, it is more efficient to use one network policy with multiple ingress or egress rules, than multiple network policies with subsets of ingress or egress rules.

• Every ingress or egress rule based on the podSelector or namespaceSelector spec generates the number of OVS flows proportional to number of pods selected by network policy + number of pods selected by ingress or egress rule. Therefore, it is preferable to use the podSelector or namespaceSelector spec that can select as many pods as you need in one rule, instead of creating individual rules for every pod.

  For example, the following policy contains two rules:

  apiVersion: networking.k8s.io/v1
  kind: NetworkPolicy
  metadata:
   name: test-network-policy
  spec:
    podSelector: {}
    ingress:
      - from:
         - podSelector:
             matchLabels:
               role: frontend
      - from:
         - podSelector:
             matchLabels:
               role: backend

  The following policy expresses those same two rules as one:

  apiVersion: networking.k8s.io/v1
  kind: NetworkPolicy
  metadata:
   name: test-network-policy
  spec:
    podSelector: {}
    ingress:
      - from:
         - podSelector:
             matchExpressions:
               - {key: role, operator: In, values: [frontend, backend]}

  The same guideline applies to the spec.podSelector spec. If you have the same ingress or egress rules for different network policies, it might be more efficient to create one network policy with a common spec.podSelector spec. For example, the following two policies have different rules:

  metadata:
   name: policy1
  spec:
   podSelector:
     matchLabels:
       role: db
    ingress:
      - from:
         - podSelector:
             matchLabels:
               role: frontend
  
  metadata:
   name: policy2
  spec:
   podSelector:
     matchLabels:
       role: client
    ingress:
      - from:
         - podSelector:
             matchLabels:
               role: frontend

  The following network policy expresses those same two rules as one:

  metadata:
   name: policy3
  spec:
   podSelector:
     matchExpressions:
       - {key: role, operator: In, values: [db, client]}
    ingress:
      - from:
         - podSelector:
             matchLabels:
               role: frontend

  You can apply this optimization when only multiple selectors are expressed as one. In cases where selectors are based on different labels, it may not be possible to apply this optimization. In those cases, consider applying some new labels for network policy optimization specifically.

• Creating a network policy

Prerequisites

• You logged in to OpenShift Cluster Manager Hybrid Cloud Console.
• You created an Red Hat OpenShift Service on AWS cluster.
• You configured an identity provider for your cluster.
• You added your user account to the configured identity provider.

Prerequisites

• Your cluster uses a network plugin that supports NetworkPolicy objects, such as the OpenShift SDN network provider with mode: NetworkPolicy set. This mode is the default for OpenShift SDN.
• You installed the OpenShift CLI (oc).
• You are logged in to the cluster with a user with admin privileges.

Prerequisites

• You installed the OpenShift CLI (oc).
• You are logged in as an administrator.
• You have a web application that exposes a port and a TCP endpoint listening for traffic on the port.

Verification

• To verify that the route resource that you created, run the following command:

  $ oc get routes -o yaml <name of resource> 1

  1

  In this example, the route is named hello-openshift.

Prerequisites

• You need a deployed Ingress Controller on a running cluster.

• Enable HSTS per-route
• Disable HSTS per-route
• Enforce HSTS per-domain, for a set of domains, or use namespace labels in combination with domains

Prerequisites

• You have a service that you want to expose.
• You have access to the OpenShift CLI (oc).

Verification

• Verify that Red Hat OpenShift Service on AWS has created the expected route for the Ingress object by running the following command:

  $ oc get routes -o yaml

  Example output

  apiVersion: v1
  items:
  - apiVersion: route.openshift.io/v1
    kind: Route
    metadata:
      name: frontend-j9sdd 1
      ...
    spec:
    ...
      tls: 2
        insecureEdgeTerminationPolicy: Redirect
        termination: edge 3
    ...

  1

  The name of the route includes the name of the Ingress object followed by a random suffix.

  2

  In order to use the default certificate, the route should not specify spec.certificate.

  3

  The route should specify the edge termination policy.

Prerequisites

• You may have a certificate/key pair in PEM-encoded files, where the certificate is valid for the route host.
• You may have a separate CA certificate in a PEM-encoded file that completes the certificate chain.
• You must have a separate destination CA certificate in a PEM-encoded file.
• You must have a service that you want to expose.

Prerequisites

• You must have a certificate/key pair in PEM-encoded files, where the certificate is valid for the route host.
• You may have a separate CA certificate in a PEM-encoded file that completes the certificate chain.
• You must have a separate destination CA certificate in a PEM-encoded file.
• You must have a service that you want to expose.

• Create a secure Route resource using reencrypt TLS termination and a custom certificate:

  $ oc create route reencrypt --service=frontend --cert=tls.crt --key=tls.key --dest-ca-cert=destca.crt --ca-cert=ca.crt --hostname=www.example.com

  If you examine the resulting Route resource, it should look similar to the following:

  YAML Definition of the Secure Route

  apiVersion: route.openshift.io/v1
  kind: Route
  metadata:
    name: frontend
  spec:
    host: www.example.com
    to:
      kind: Service
      name: frontend
    tls:
      termination: reencrypt
      key: |-
        -----BEGIN PRIVATE KEY-----
        [...]
        -----END PRIVATE KEY-----
      certificate: |-
        -----BEGIN CERTIFICATE-----
        [...]
        -----END CERTIFICATE-----
      caCertificate: |-
        -----BEGIN CERTIFICATE-----
        [...]
        -----END CERTIFICATE-----
      destinationCACertificate: |-
        -----BEGIN CERTIFICATE-----
        [...]
        -----END CERTIFICATE-----

  See oc create route reencrypt --help for more options.

Prerequisites

• You must have a certificate/key pair in PEM-encoded files, where the certificate is valid for the route host.
• You may have a separate CA certificate in a PEM-encoded file that completes the certificate chain.
• You must have a service that you want to expose.

• Create a secure Route resource using edge TLS termination and a custom certificate.

  $ oc create route edge --service=frontend --cert=tls.crt --key=tls.key --ca-cert=ca.crt --hostname=www.example.com

  If you examine the resulting Route resource, it should look similar to the following:

  YAML Definition of the Secure Route

  apiVersion: route.openshift.io/v1
  kind: Route
  metadata:
    name: frontend
  spec:
    host: www.example.com
    to:
      kind: Service
      name: frontend
    tls:
      termination: edge
      key: |-
        -----BEGIN PRIVATE KEY-----
        [...]
        -----END PRIVATE KEY-----
      certificate: |-
        -----BEGIN CERTIFICATE-----
        [...]
        -----END CERTIFICATE-----
      caCertificate: |-
        -----BEGIN CERTIFICATE-----
        [...]
        -----END CERTIFICATE-----

  See oc create route edge --help for more options.

Prerequisites

• You must have a service that you want to expose.

Procedure

• Create a Route resource:

  $ oc create route passthrough route-passthrough-secured --service=frontend --port=8080

  If you examine the resulting Route resource, it should look similar to the following:

  A Secured Route Using Passthrough Termination

  apiVersion: route.openshift.io/v1
  kind: Route
  metadata:
    name: route-passthrough-secured 1
  spec:
    host: www.example.com
    port:
      targetPort: 8080
    tls:
      termination: passthrough 2
      insecureEdgeTerminationPolicy: None 3
    to:
      kind: Service
      name: frontend

  1

  The name of the object, which is limited to 63 characters.

  2

  The termination field is set to passthrough. This is the only required tls field.

  3

  Optional insecureEdgeTerminationPolicy. The only valid values are None, Redirect, or empty for disabled.

  The destination pod is responsible for serving certificates for the traffic at the endpoint. This is currently the only method that can support requiring client certificates, also known as two-way authentication.