Chapter 13. Submariner

The submariner-addon component is a technology preview feature.

Submariner is an open source tool that can be used with Red Hat Advanced Cluster Management for Kubernetes to provide direct networking between two or more Kubernetes clusters in your environment, either on-premises or in the cloud. For more information about Submariner, see Submariner.

You can enable Submariner on the OpenShift Container Platform clusters that are hosted in the following environments:

  • Amazon Web Services
  • Google Cloud Platform
  • Microsoft Azure
  • IBM Cloud
  • VMware vSphere
  • Bare metal
  • Red Hat OpenShift Dedicated

Red Hat Advanced Cluster Management for Kubernetes provides a Submariner component that you can deploy in your environment by using your hub cluster.

13.1. Prerequisites

Ensure that you have the following prerequisites before using Submariner:

  • A Red Hat Advanced Cluster Management hub cluster that is running on Red Hat OpenShift Container Platform version 4.5, or later, with Kubernetes version 1.17, or later.
  • A credential for accessing the hub cluster with cluster administrator permissions.
  • Two or more OpenShift Container Platform managed clusters that are running on OpenShift Container Platform version 4.4, or later, with Kubernetes version 1.17, or later, and are managed by the Red Hat Advanced Cluster Management hub cluster.
  • Pod and Service Classless Inter-Domain Routing (CIDR) between the clusters that do not overlap.
  • IP connectivity must be configured between the Gateway nodes. When connecting two clusters, at least one of the clusters must have a publicly routable IP address designated to the Gateway node.
  • Firewall configuration across all nodes in each of the managed clusters must allow 4800/UDP in both directions.

    Note: This is configured automatically when your clusters are deployed in an Amazon Web Services environment, but must be configured manually for clusters on other environments and for the firewalls that protect private clouds like bare metal and VMware vSphere.

  • Firewall configuration on the Gateway nodes allows ingress 8080/TCP so the other nodes in the cluster can access it.
  • Firewall configuration open for UDP/4500, UDP/500, and any other ports that are used for IPSec traffic on the gateway nodes.

See Submariner prerequisites for more detailed information about the prerequisites.

13.2. Preparing the hosts to deploy Submariner

Before deploying Submariner with Red Hat Advanced Cluster Management for Kubernetes, you must prepare the clusters on the hosting environment for the connection. The requirements vary by the hosting environment, so follow the instructions for your hosting environment.

13.2.1. Preparing Amazon Web Services to deploy Submariner

There are two ways that you can configure the OpenShift Container Platform cluster that is hosted on Amazon Web Services to integrate with a Submariner deployment. Select one of these options to prepare for the connection:

  • Method 1

    You can use the SubmarinerConfig API to build the cluster environment. With this method, the submariner-addon configures the environment, so you use your configurations and cloud provider credentials in your SubmarinerConfig definition.

    Note: This method is only supported when the cluster is on Amazon Web Services, and not on other environments.

    Create a YAML file that contains the following content:

    apiVersion: v1
    kind: Secret
    metadata:
        name: <cloud-provider-credential-secret-name>
        namespace: <managed-cluster-namespace>
    type: Opaque
    data:
        aws_access_key_id: <aws-access-key-id>
        aws_secret_access_key: <aws-secret-access-key>

    The format of name is the same as the credential secret name that you used to provision the cluster with Red Hat Advanced Cluster Management.

    If you have already configured your Submariner cluster environment manually, include the configurations in the your SubmarinerConfig addition. In this example, the IPSecIKEPort is set to 501, and the IPSecNATTPort is set to 4501:

    apiVersion: submarineraddon.open-cluster-management.io/v1alpha1
    kind: SubmarinerConfig
    metadata:
        name: <config-name>
        namespace: <managed-cluster-namespace>
    spec:
        IPSecIKEPort: 501
        IPSecNATTPort: 4501
        ...
  • Method 2

    You can use the script file, prep_for_subm.sh, that is provided on the Submariner website to update your OpenShift Container Platform installer-provisioned Amazon Web Services infrastructure for Submariner deployments. See Prepare AWS Clusters for Submariner for the script and instructions for running it.

13.2.2. Preparing Google Cloud Platform to deploy Submariner

To prepare the clusters on your Google Cloud Platform for deploying the Submariner component, complete the following steps:

  1. Create the inbound and outbound firewall rules on your Google Cloud Platform to open the IPsec IKE (by default 500/UDP) and NAT traversal ports (by default 4500/UDP) to enable Submariner communication:

    $ gcloud compute firewall-rules create <name> --network=<network-name> --allow=udp:<ipsec-port> --direction=IN
    $ gcloud compute firewall-rules create <rule-name> --network=<network-name> --allow=udp:<ipsec-port>  --direction=OUT

    Replace rule-name with your rule name.

    Replace network-name with your Google Cloud Platform cluster network name.

    Replace ipsec-port with your IPsec port.

  2. Create the inbound and outbound firewall rules on your Google Cloud Platform to open the 4800/UDP port to encapsulate Pod traffic from the worker and master nodes to the Submariner Gateway nodes:

    $ gcloud compute firewall-rules create <name> --network=<network-name> --allow=udp:4800 --direction=IN
    $ gcloud compute firewall-rules create <name> --network=<network-name> --allow=udp:4800 --direction=OUT

    Replace name with your rule name.

    Replace network-name with your Google Cloud Platform cluster network name.

  3. Create the inbound and outbound firewall rules on your Google Cloud Platform to open the 8080/TCP port to export metrics service from the Submariner Gateway nodes:

    $ gcloud compute firewall-rules create <name> --network=<network-name> --allow=tcp:8080 --direction=IN
    $ gcloud compute firewall-rules create <name> --network=<network-name> --allow=tcp:8080 --direction=OUT

    Replace name with your rule name.

    Replace network-name with your Google Cloud Platform cluster network name.

13.2.3. Preparing Microsoft Azure to deploy Submariner

To prepare the clusters on your Microsoft Azure for deploying the Submariner component, complete the following steps:

  1. Create the inbound and outbound firewall rules on your Microsoft Azure environment to open the IP security IKE (by default 500/UDP) and NAT traversal ports (by default 4500/UDP) to enable Submariner communication:

    # create inbound nat rule
    $ az network lb inbound-nat-rule create --lb-name <lb-name> \
    --resource-group <res-group> \
    --name <name> \
    --protocol Udp --frontend-port <ipsec-port> \
    --backend-port <ipsec-port> \
    --frontend-ip-name <frontend-ip-name>
    
    # add your vm network interface to the created inbound nat rule
    $ az network nic ip-config inbound-nat-rule add \
    --lb-name <lb-name> --resource-group <res-group> \
    --inbound-nat-rule <nat-name> \
    --nic-name <nic-name> --ip-config-name <pipConfig>

    Replace lb-name with your load balancer name.

    Replace res-group with your resource group name.

    Replace nat-name with your load balancing inbound NAT rule name.

    Replace ipsec-port with your IPsec port.

    Replace pipConfig with your cluster frontend IP configuration name.

    Replace nic-name with your network interface card (NIC) name.

  2. Create one load balancing inbound NAT rules to forward Submariner gateway metrics service request:

    # create inbound nat rule
    $ az network lb inbound-nat-rule create --lb-name <lb-name> \
    --resource-group <res-group> \
    --name <name> \
    --protocol Tcp --frontend-port 8080 --backend-port 8080 \
    --frontend-ip-name <frontend-ip-name>
    
    # add your vm network interface to the created inbound nat rule
    $ az network nic ip-config inbound-nat-rule add \
    --lb-name <lb-name> --resource-group <res-group> \
    --inbound-nat-rule <nat-name> \
    --nic-name <nic-name> --ip-config-name <pipConfig>

    Replace lb-name with your load balancer name.

    Replace res-group with your resource group name.

    Replace nat-name with your load balancing inbound NAT rule name.

    Replace pipConfig with your cluster frontend IP configuration name.

    Replace nic-name with your network interface card (NIC) name.

  3. Create network security groups {NSG) security rules on your Azure to open IPsec IKE (by default 500/UDP) and NAT traversal ports (by default 4500/UDP) for Submariner:

    $ az network nsg rule create --resource-group <res-group> \
    --nsg-name <nsg-name> --priority <priority> \
    --name <name> --direction Inbound --access Allow \
    --protocol Udp --destination-port-ranges <ipsec-port>
    
    $ az network nsg rule create --resource-group <res-group> \
    --nsg-name <nsg-name> --priority <priority> \
    --name <name> --direction Outbound --access Allow \
    --protocol Udp --destination-port-ranges <ipsec-port>

    Replace res-group with your resource group name.

    Replace nsg-name with your NSG name.

    Replace priority with your rule priority.

    Replace name with your rule name.

    Replace ipsec-port with your IPsec port.

  4. Create the NSG rules to open 4800/UDP port to encapsulate pod traffic from the worker and master nodes to the Submariner Gateway nodes:

    $ az network nsg rule create --resource-group <res-group> \
    --nsg-name <nsg-name> --priority <priority> \
    --name <name> --direction Inbound --access Allow \
    --protocol Udp --destination-port-ranges 4800 \
    
    $ az network nsg rule create --resource-group <res-group> \
    --nsg-name <nsg-name> --priority <priority> \
    --name <name> --direction Outbound --access Allow \
    --protocol Udp --destination-port-ranges 4800

    Replace res-group with your resource group name.

    Replace nsg-name with your NSG name.

    Replace priority with your rule priority.

    Replace name with your rule name.

  5. Create the NSG rules to open 8080/TCP port to export metrics service from the Submariner Gateway nodes:

    $ az network nsg rule create --resource-group <res-group> \
    --nsg-name <nsg-name> --priority <priority> \
    --name <name> --direction Inbound --access Allow \
    --protocol Tcp --destination-port-ranges 8080 \
    
    $ az network nsg rule create --resource-group <res-group> \
    --nsg-name <nsg-name> --priority <priority> \
    --name <name> --direction Outbound --access Allow \
    --protocol Udp --destination-port-ranges 8080

    Replace res-group with your resource group name.

    Replace nsg-name with your NSG name.

    Replace priority with your rule priority.

    Replace name with your rule name.

13.2.4. Preparing IBM Cloud to deploy Submariner

There are two kinds of Red Hat OpenShift Kubernetes Service (ROKS) on IBM Cloud: the classic cluster and the second generation of compute infrastructure in a virtual private cloud (VPC). Submariner cannot run on the classic ROKS cluster since cannot configure the IPSec ports for the classic cluster.

To configure the ROKS clusters on a VPC to use Submariner, complete the steps in the following links:

  1. Before creating a cluster, specify subnets for pods and services, which avoids overlapping CIDRs with other clusters. Make sure there are no overlapping pods and services CIDRs between clusters if you are using an existing cluster.See VPC Subnets for the procedure.
  2. Attach a public gateway to subnets used in the cluster. See Public Gateway for the procedure.
  3. Create inbound rules for the default security group of the cluster by completing the steps in Security Group. Ensure that the firewall allows inbound and outbound traffic on UDP/4500 and UDP/500 ports for Gateway nodes, and allows inbound and outbound UDP/4800 for all the other nodes.
  4. Label a node that has the public gateway as submariner.io/gateway=true in the cluster.
  5. Refer to Calico to configure Calico CNI by creating IPPools in the cluster.

13.2.5. Preparing Red Hat OpenShift Dedicated to deploy Submariner

Red Hat OpenShift Dedicated supports clusters that were provisioned by AWS and Google Cloud Platform.

13.2.5.1. Preparing Red Hat OpenShift Dedicated to deploy Submariner on AWS

To configure the AWS clusters on Red Hat OpenShift Dedicated, complete the following steps:

  1. Submit a support ticket to the Red Hat OpenShift Hosted SRE Support team to grant cluster-admin group access to the Red Hat OpenShift Dedicated cluster. The default access of dedicated-admin does not have the permission that is required the create a MachineSet.
  2. After the group is created, add the user name to the cluster-admin group that you created by completing the steps in Granting the cluster-admin role to users in the Red Hat OpenShift Dedicated documentation.
  3. Complete the prerequisites that are listed in the Preparing Amazon Web Services to deploy Submariner.
  4. Configure the credentials of the user osdCcsAdmin, so you can use that as a service account.

13.2.5.2. Preparing Red Hat OpenShift Dedicated to deploy Submariner on Google Cloud Platform

To configure the Google Cloud Platform clusters on Red Hat OpenShift Dedicated, complete the following steps:

  1. Complete the prerequisites in Preparing Google Cloud Platform to deploy Submariner.
  2. Configure a service account named osd-ccs-admin that you can use to manage the deployment.

13.2.6. Preparing to deploy Submariner on VMware vSphere or bare metal

To prepare the VMware vSphere and bare metal clusters for deploying Submariner, complete the following steps:

  1. Configure a publicly routable IP address that is designated to the gateway node on at least one of the clusters.
  2. Ensure that ports for IP security are open. The default ports are 4500/UDP and 500/UDP. If the default ports are blocked by a firewall, configure a pair of custom ports that are available, like 4501/UDP and 501/UDP.

13.3. Deploying Submariner

The submariner-addon component is a technology preview feature.

Complete the following steps to deploy Submariner:

  1. Log on to your hub cluster with cluster administrator permissions.
  2. Ensure that you have completed the applicable preparations. See Submariner for the requirements.
  3. Create a ManagedClusterSet on the hub cluster by using the instructions provided in ManagedClusterSets. The submariner-addon creates a namespace called submariner-clusterset-<clusterset-name>-broker and deploys the Submariner Broker to it. The name of the ManagedClusterSet replaces <clusterset-name> in the namespace name. Your entry for the ManagedClusterSet should resemble the following content:

    apiVersion: cluster.open-cluster-management.io/v1alpha1
    kind: ManagedClusterSet
    metadata:
      name: <ManagedClusterSet-name>

    Replace ManagedClusterSet-name with a name for the ManagedClusterSet that you are creating.

  4. Enable Submariner to provide communication between managed clusters by entering the following command:

    oc label managedclusters <managedcluster-name> "cluster.open-cluster-management.io/submariner-agent=true" --overwrite

    Replace managedcluster-name with the name of the managed cluster that you want to use with Submariner.

  5. Add the managed clusters to the ManagedClusterSet by entering the following command:

    oc label managedclusters <managedcluster-name> "cluster.open-cluster-management.io/clusterset=<ManagedClusterSet-name>" --overwrite

    Replace managedcluster-name with the name of the managed cluster that you want to add to the ManagedClusterSet. Replace ManagedClusterSet-name with the name of the ManagedClusterSet to which you want to add the managed cluster.

  6. Repeat those steps for all of the managed clusters that you want to add to the ManagedClusterSet.

13.3.1. Deploying Submariner to an AWS OpenShift Container Platform cluster

Red Hat Advanced Cluster Management for Kubernetes version 2.2 supports automatic deployment of Submariner to managed OpenShift Container Platform cluster that are deployed on Amazon Web Services.

Complete the following steps to deploy Submariner:

  1. Log on to your hub cluster with cluster administrator permissions.

    • For a cluster that was created with Red Hat Advanced Cluster Management, continue with step 2.
    • For a cluster that was imported to Red Hat Advanced Cluster Management, skip to step 3.
  2. For AWS clusters that were created by Red Hat Advanced Cluster Management for Kubernetes, create a file called submarinerconfig.yaml with the following SubmarinerConfig content:

    apiVersion: submarineraddon.open-cluster-management.io/v1alpha1
    kind: SubmarinerConfig
    metadata:
      name: submariner
      namespace: <your-cluster-namespace>
    spec:
      credentialsSecret:
        name: <your-cluster-name>-aws-creds
      subscriptionConfig:
        channel: alpha
        startingCSV: submariner.v0.8.1
    • Replace your-cluster-namespace with your cluster’s namespace.
    • Replace your-cluster-name with the name of your cluster. The file for your-cluster-namespace aws-cloud-credentials is stored in the Hive cluster namespace.

      Skip to step 4.

  3. For AWS clusters that were created by OpenShift Container Platform but imported into Red Hat Advanced Cluster Management, create the aws-cloud-credentials.

    1. Apply the following secret to the managed cluster namespace.

      apiVersion: v1
      kind: Secret
      metadata:
        name: aws-cloud-credentials
        namespace: <your-cluster-namespace>
      type: Opaque
      data:
        aws_access_key_id: <your-aws_access_key_id>
        aws_secret_access_key: <your-aws_secret_access_key>
    2. Create a file called submarinerconfig.yaml with the following SubmarinerConfig content:

      apiVersion: submarineraddon.open-cluster-management.io/v1alpha1
      kind: SubmarinerConfig
      metadata:
        name: subconfig
        namespace: <your-cluster-namespace>
      spec:
        credentialsSecret:
          name: <aws-cloud-credentials>
        subscriptionConfig:
          channel: alpha
          startingCSV: submariner.v0.8.1

      Replace aws-cloud-credential with the AWS credential that you created in the previous step.

  4. Apply the SubmarinerConfig resource to the managed cluster namespace on the hub cluster by entering the following command:

    oc apply -f submarinerconfig.yaml
  5. Create a ManagedClusterSet on the hub cluster.

    apiVersion: cluster.open-cluster-management.io/v1alpha1
    kind: ManagedClusterSet
    metadata:
      name: my-clusterset
  6. Add the managed clusters to the ManagedClusterSet by entering the following command:

    oc label managedclusters <managedcluster-name> "cluster.open-cluster-management.io/clusterset=<ManagedClusterSet-name>" --overwrite
  7. Run the following command to allow Red Hat Advanced Cluster Management to automatically deploy Submariner:app-name:

    oc label managedclusters <managedcluster-name> "cluster.open-cluster-management.io/submariner-agent=true" --overwrite

13.4. Enabling service discovery for Submariner

The submariner-addon component is a technology preview feature.

After Submariner is deployed into the same environment as your managed clusters, the routes are configured for secure IP routing between the pod and services across the clusters in the ManagedClusterSet. To make a service from a cluster visible and discoverable to other clusters in the ManagedClusterSet, you must create a ServiceExport object. After a service is exported with a ServiceExport object, you can access the the service by the following format: <service>.<namespace>.svc.clusterset.local. If multiple clusters export a service with the same name, and from the same namespace, they are recognized by other clusters as a single logical service.

This example uses the nginx service in the default namespace, but you can discover any Kubernetes ClusterIP service or headless service:

  1. Apply an instance of the nginx service on a managed cluster that is in the ManagedClusterSet by entering the following commands:

    oc -n default create deployment nginx --image=nginxinc/nginx-unprivileged:stable-alpine
    oc -n default expose deployment nginx --port=8080
  2. Export the service by creating a ServiceExport entry that resembles the following content in the YAML file:

    apiVersion: multicluster.x-k8s.io/v1alpha1
    kind: ServiceExport
    metadata:
      name: <service-name>
      namespace: <service-namespace>

    Replace service-name with the name of the service that you are exporting. In this example, it is nginx. Replace service-namespace with the name of the namespace where the service is located. In this example, it is default.

  3. Run the following command from a different managed cluster to confirm that it can access the nginx service:

    oc -n default run --generator=run-pod/v1 tmp-shell --rm -i --tty --image quay.io/submariner/nettest -- /bin/bash curl nginx.default.svc.clusterset.local:8080

The nginx service discovery is now configured for Submariner.