Chapter 13. Submariner
The submariner-addon
component is a technology preview feature.
Submariner is an open source tool that can be used with Red Hat Advanced Cluster Management for Kubernetes to provide direct networking between two or more Kubernetes clusters in your environment, either on-premises or in the cloud. For more information about Submariner, see Submariner.
You can enable Submariner on the OpenShift Container Platform clusters that are hosted in the following environments:
- Amazon Web Services
- Google Cloud Platform
- Microsoft Azure
- IBM Cloud
- VMware vSphere
- Bare metal
- Red Hat OpenShift Dedicated
Red Hat Advanced Cluster Management for Kubernetes provides a Submariner component that you can deploy in your environment by using your hub cluster.
13.1. Prerequisites
Ensure that you have the following prerequisites before using Submariner:
- A Red Hat Advanced Cluster Management hub cluster that is running on Red Hat OpenShift Container Platform version 4.5, or later, with Kubernetes version 1.17, or later.
-
A credential for accessing the hub cluster with
cluster administrator
permissions. - Two or more OpenShift Container Platform managed clusters that are running on OpenShift Container Platform version 4.4, or later, with Kubernetes version 1.17, or later, and are managed by the Red Hat Advanced Cluster Management hub cluster.
- Pod and Service Classless Inter-Domain Routing (CIDR) between the clusters that do not overlap.
- IP connectivity must be configured between the Gateway nodes. When connecting two clusters, at least one of the clusters must have a publicly routable IP address designated to the Gateway node.
Firewall configuration across all nodes in each of the managed clusters must allow 4800/UDP in both directions.
Note: This is configured automatically when your clusters are deployed in an Amazon Web Services environment, but must be configured manually for clusters on other environments and for the firewalls that protect private clouds like bare metal and VMware vSphere.
- Firewall configuration on the Gateway nodes allows ingress 8080/TCP so the other nodes in the cluster can access it.
- Firewall configuration open for UDP/4500, UDP/500, and any other ports that are used for IPSec traffic on the gateway nodes.
See Submariner prerequisites for more detailed information about the prerequisites.
13.2. Preparing the hosts to deploy Submariner
Before deploying Submariner with Red Hat Advanced Cluster Management for Kubernetes, you must prepare the clusters on the hosting environment for the connection. The requirements vary by the hosting environment, so follow the instructions for your hosting environment.
13.2.1. Preparing Amazon Web Services to deploy Submariner
There are two ways that you can configure the OpenShift Container Platform cluster that is hosted on Amazon Web Services to integrate with a Submariner deployment. Select one of these options to prepare for the connection:
Method 1
You can use the
SubmarinerConfig
API to build the cluster environment. With this method, thesubmariner-addon
configures the environment, so you use your configurations and cloud provider credentials in yourSubmarinerConfig
definition.Note: This method is only supported when the cluster is on Amazon Web Services, and not on other environments.
Create a YAML file that contains the following content:
apiVersion: v1 kind: Secret metadata: name: <cloud-provider-credential-secret-name> namespace: <managed-cluster-namespace> type: Opaque data: aws_access_key_id: <aws-access-key-id> aws_secret_access_key: <aws-secret-access-key>
The format of
name
is the same as the credential secret name that you used to provision the cluster with Red Hat Advanced Cluster Management.If you have already configured your Submariner cluster environment manually, include the configurations in the your
SubmarinerConfig
addition. In this example, theIPSecIKEPort
is set to501
, and theIPSecNATTPort
is set to4501
:apiVersion: submarineraddon.open-cluster-management.io/v1alpha1 kind: SubmarinerConfig metadata: name: <config-name> namespace: <managed-cluster-namespace> spec: IPSecIKEPort: 501 IPSecNATTPort: 4501 ...
Method 2
You can use the script file,
prep_for_subm.sh
, that is provided on the Submariner website to update your OpenShift Container Platform installer-provisioned Amazon Web Services infrastructure for Submariner deployments. See Prepare AWS Clusters for Submariner for the script and instructions for running it.
13.2.2. Preparing Google Cloud Platform to deploy Submariner
To prepare the clusters on your Google Cloud Platform for deploying the Submariner component, complete the following steps:
Create the inbound and outbound firewall rules on your Google Cloud Platform to open the IPsec IKE (by default 500/UDP) and NAT traversal ports (by default 4500/UDP) to enable Submariner communication:
$ gcloud compute firewall-rules create <name> --network=<network-name> --allow=udp:<ipsec-port> --direction=IN $ gcloud compute firewall-rules create <rule-name> --network=<network-name> --allow=udp:<ipsec-port> --direction=OUT
Replace
rule-name
with your rule name.Replace
network-name
with your Google Cloud Platform cluster network name.Replace
ipsec-port
with your IPsec port.Create the inbound and outbound firewall rules on your Google Cloud Platform to open the 4800/UDP port to encapsulate Pod traffic from the worker and master nodes to the Submariner Gateway nodes:
$ gcloud compute firewall-rules create <name> --network=<network-name> --allow=udp:4800 --direction=IN $ gcloud compute firewall-rules create <name> --network=<network-name> --allow=udp:4800 --direction=OUT
Replace
name
with your rule name.Replace
network-name
with your Google Cloud Platform cluster network name.Create the inbound and outbound firewall rules on your Google Cloud Platform to open the 8080/TCP port to export metrics service from the Submariner Gateway nodes:
$ gcloud compute firewall-rules create <name> --network=<network-name> --allow=tcp:8080 --direction=IN $ gcloud compute firewall-rules create <name> --network=<network-name> --allow=tcp:8080 --direction=OUT
Replace
name
with your rule name.Replace
network-name
with your Google Cloud Platform cluster network name.
13.2.3. Preparing Microsoft Azure to deploy Submariner
To prepare the clusters on your Microsoft Azure for deploying the Submariner component, complete the following steps:
Create the inbound and outbound firewall rules on your Microsoft Azure environment to open the IP security IKE (by default 500/UDP) and NAT traversal ports (by default 4500/UDP) to enable Submariner communication:
# create inbound nat rule $ az network lb inbound-nat-rule create --lb-name <lb-name> \ --resource-group <res-group> \ --name <name> \ --protocol Udp --frontend-port <ipsec-port> \ --backend-port <ipsec-port> \ --frontend-ip-name <frontend-ip-name> # add your vm network interface to the created inbound nat rule $ az network nic ip-config inbound-nat-rule add \ --lb-name <lb-name> --resource-group <res-group> \ --inbound-nat-rule <nat-name> \ --nic-name <nic-name> --ip-config-name <pipConfig>
Replace
lb-name
with your load balancer name.Replace
res-group
with your resource group name.Replace
nat-name
with your load balancing inbound NAT rule name.Replace
ipsec-port
with your IPsec port.Replace
pipConfig
with your cluster frontend IP configuration name.Replace
nic-name
with your network interface card (NIC) name.Create one load balancing inbound NAT rules to forward Submariner gateway metrics service request:
# create inbound nat rule $ az network lb inbound-nat-rule create --lb-name <lb-name> \ --resource-group <res-group> \ --name <name> \ --protocol Tcp --frontend-port 8080 --backend-port 8080 \ --frontend-ip-name <frontend-ip-name> # add your vm network interface to the created inbound nat rule $ az network nic ip-config inbound-nat-rule add \ --lb-name <lb-name> --resource-group <res-group> \ --inbound-nat-rule <nat-name> \ --nic-name <nic-name> --ip-config-name <pipConfig>
Replace
lb-name
with your load balancer name.Replace
res-group
with your resource group name.Replace
nat-name
with your load balancing inbound NAT rule name.Replace
pipConfig
with your cluster frontend IP configuration name.Replace
nic-name
with your network interface card (NIC) name.Create network security groups {NSG) security rules on your Azure to open IPsec IKE (by default 500/UDP) and NAT traversal ports (by default 4500/UDP) for Submariner:
$ az network nsg rule create --resource-group <res-group> \ --nsg-name <nsg-name> --priority <priority> \ --name <name> --direction Inbound --access Allow \ --protocol Udp --destination-port-ranges <ipsec-port> $ az network nsg rule create --resource-group <res-group> \ --nsg-name <nsg-name> --priority <priority> \ --name <name> --direction Outbound --access Allow \ --protocol Udp --destination-port-ranges <ipsec-port>
Replace
res-group
with your resource group name.Replace
nsg-name
with your NSG name.Replace
priority
with your rule priority.Replace
name
with your rule name.Replace
ipsec-port
with your IPsec port.Create the NSG rules to open 4800/UDP port to encapsulate pod traffic from the worker and master nodes to the Submariner Gateway nodes:
$ az network nsg rule create --resource-group <res-group> \ --nsg-name <nsg-name> --priority <priority> \ --name <name> --direction Inbound --access Allow \ --protocol Udp --destination-port-ranges 4800 \ $ az network nsg rule create --resource-group <res-group> \ --nsg-name <nsg-name> --priority <priority> \ --name <name> --direction Outbound --access Allow \ --protocol Udp --destination-port-ranges 4800
Replace
res-group
with your resource group name.Replace
nsg-name
with your NSG name.Replace
priority
with your rule priority.Replace
name
with your rule name.Create the NSG rules to open 8080/TCP port to export metrics service from the Submariner Gateway nodes:
$ az network nsg rule create --resource-group <res-group> \ --nsg-name <nsg-name> --priority <priority> \ --name <name> --direction Inbound --access Allow \ --protocol Tcp --destination-port-ranges 8080 \ $ az network nsg rule create --resource-group <res-group> \ --nsg-name <nsg-name> --priority <priority> \ --name <name> --direction Outbound --access Allow \ --protocol Udp --destination-port-ranges 8080
Replace
res-group
with your resource group name.Replace
nsg-name
with your NSG name.Replace
priority
with your rule priority.Replace
name
with your rule name.
13.2.4. Preparing IBM Cloud to deploy Submariner
There are two kinds of Red Hat OpenShift Kubernetes Service (ROKS) on IBM Cloud: the classic cluster and the second generation of compute infrastructure in a virtual private cloud (VPC). Submariner cannot run on the classic ROKS cluster since cannot configure the IPSec ports for the classic cluster.
To configure the ROKS clusters on a VPC to use Submariner, complete the steps in the following links:
- Before creating a cluster, specify subnets for pods and services, which avoids overlapping CIDRs with other clusters. Make sure there are no overlapping pods and services CIDRs between clusters if you are using an existing cluster.See VPC Subnets for the procedure.
- Attach a public gateway to subnets used in the cluster. See Public Gateway for the procedure.
- Create inbound rules for the default security group of the cluster by completing the steps in Security Group. Ensure that the firewall allows inbound and outbound traffic on UDP/4500 and UDP/500 ports for Gateway nodes, and allows inbound and outbound UDP/4800 for all the other nodes.
-
Label a node that has the public gateway as
submariner.io/gateway=true
in the cluster. - Refer to Calico to configure Calico CNI by creating IPPools in the cluster.
13.2.5. Preparing Red Hat OpenShift Dedicated to deploy Submariner
Red Hat OpenShift Dedicated supports clusters that were provisioned by AWS and Google Cloud Platform.
13.2.5.1. Preparing Red Hat OpenShift Dedicated to deploy Submariner on AWS
To configure the AWS clusters on Red Hat OpenShift Dedicated, complete the following steps:
-
Submit a support ticket to the Red Hat OpenShift Hosted SRE Support team to grant
cluster-admin
group access to the Red Hat OpenShift Dedicated cluster. The default access ofdedicated-admin
does not have the permission that is required the create aMachineSet
. -
After the group is created, add the user name to the
cluster-admin
group that you created by completing the steps in Granting the cluster-admin role to users in the Red Hat OpenShift Dedicated documentation. - Complete the prerequisites that are listed in the Preparing Amazon Web Services to deploy Submariner.
-
Configure the credentials of the user
osdCcsAdmin
, so you can use that as a service account.
13.2.5.2. Preparing Red Hat OpenShift Dedicated to deploy Submariner on Google Cloud Platform
To configure the Google Cloud Platform clusters on Red Hat OpenShift Dedicated, complete the following steps:
- Complete the prerequisites in Preparing Google Cloud Platform to deploy Submariner.
-
Configure a service account named
osd-ccs-admin
that you can use to manage the deployment.
13.2.6. Preparing to deploy Submariner on VMware vSphere or bare metal
To prepare the VMware vSphere and bare metal clusters for deploying Submariner, complete the following steps:
- Configure a publicly routable IP address that is designated to the gateway node on at least one of the clusters.
- Ensure that ports for IP security are open. The default ports are 4500/UDP and 500/UDP. If the default ports are blocked by a firewall, configure a pair of custom ports that are available, like 4501/UDP and 501/UDP.
13.3. Deploying Submariner
The submariner-addon
component is a technology preview feature.
Complete the following steps to deploy Submariner:
- Log on to your hub cluster with cluster administrator permissions.
- Ensure that you have completed the applicable preparations. See Submariner for the requirements.
Create a
ManagedClusterSet
on the hub cluster by using the instructions provided in ManagedClusterSets. Thesubmariner-addon
creates a namespace calledsubmariner-clusterset-<clusterset-name>-broker
and deploys the Submariner Broker to it. The name of the ManagedClusterSet replaces <clusterset-name> in the namespace name. Your entry for theManagedClusterSet
should resemble the following content:apiVersion: cluster.open-cluster-management.io/v1alpha1 kind: ManagedClusterSet metadata: name: <ManagedClusterSet-name>
Replace ManagedClusterSet-name with a name for the
ManagedClusterSet
that you are creating.Enable Submariner to provide communication between managed clusters by entering the following command:
oc label managedclusters <managedcluster-name> "cluster.open-cluster-management.io/submariner-agent=true" --overwrite
Replace
managedcluster-name
with the name of the managed cluster that you want to use with Submariner.Add the managed clusters to the
ManagedClusterSet
by entering the following command:oc label managedclusters <managedcluster-name> "cluster.open-cluster-management.io/clusterset=<ManagedClusterSet-name>" --overwrite
Replace
managedcluster-name
with the name of the managed cluster that you want to add to theManagedClusterSet
. ReplaceManagedClusterSet-name
with the name of theManagedClusterSet
to which you want to add the managed cluster.-
Repeat those steps for all of the managed clusters that you want to add to the
ManagedClusterSet
.
13.3.1. Deploying Submariner to an AWS OpenShift Container Platform cluster
Red Hat Advanced Cluster Management for Kubernetes version 2.2 supports automatic deployment of Submariner to managed OpenShift Container Platform cluster that are deployed on Amazon Web Services.
Complete the following steps to deploy Submariner:
Log on to your hub cluster with cluster administrator permissions.
- For a cluster that was created with Red Hat Advanced Cluster Management, continue with step 2.
- For a cluster that was imported to Red Hat Advanced Cluster Management, skip to step 3.
For AWS clusters that were created by Red Hat Advanced Cluster Management for Kubernetes, create a file called
submarinerconfig.yaml
with the followingSubmarinerConfig
content:apiVersion: submarineraddon.open-cluster-management.io/v1alpha1 kind: SubmarinerConfig metadata: name: submariner namespace: <your-cluster-namespace> spec: credentialsSecret: name: <your-cluster-name>-aws-creds subscriptionConfig: channel: alpha startingCSV: submariner.v0.8.1
-
Replace
your-cluster-namespace
with your cluster’s namespace. Replace
your-cluster-name
with the name of your cluster. The file foryour-cluster-namespace
aws-cloud-credentials
is stored in the Hive cluster namespace.Skip to step 4.
-
Replace
For AWS clusters that were created by OpenShift Container Platform but imported into Red Hat Advanced Cluster Management, create the
aws-cloud-credentials
.Apply the following secret to the managed cluster namespace.
apiVersion: v1 kind: Secret metadata: name: aws-cloud-credentials namespace: <your-cluster-namespace> type: Opaque data: aws_access_key_id: <your-aws_access_key_id> aws_secret_access_key: <your-aws_secret_access_key>
Create a file called
submarinerconfig.yaml
with the followingSubmarinerConfig
content:apiVersion: submarineraddon.open-cluster-management.io/v1alpha1 kind: SubmarinerConfig metadata: name: subconfig namespace: <your-cluster-namespace> spec: credentialsSecret: name: <aws-cloud-credentials> subscriptionConfig: channel: alpha startingCSV: submariner.v0.8.1
Replace
aws-cloud-credential
with the AWS credential that you created in the previous step.
Apply the
SubmarinerConfig
resource to the managed cluster namespace on the hub cluster by entering the following command:oc apply -f submarinerconfig.yaml
Create a ManagedClusterSet on the hub cluster.
apiVersion: cluster.open-cluster-management.io/v1alpha1 kind: ManagedClusterSet metadata: name: my-clusterset
Add the managed clusters to the
ManagedClusterSet
by entering the following command:oc label managedclusters <managedcluster-name> "cluster.open-cluster-management.io/clusterset=<ManagedClusterSet-name>" --overwrite
Run the following command to allow Red Hat Advanced Cluster Management to automatically deploy Submariner:app-name:
oc label managedclusters <managedcluster-name> "cluster.open-cluster-management.io/submariner-agent=true" --overwrite
13.4. Enabling service discovery for Submariner
The submariner-addon
component is a technology preview feature.
After Submariner is deployed into the same environment as your managed clusters, the routes are configured for secure IP routing between the pod and services across the clusters in the ManagedClusterSet
. To make a service from a cluster visible and discoverable to other clusters in the ManagedClusterSet
, you must create a ServiceExport
object. After a service is exported with a ServiceExport
object, you can access the the service by the following format: <service>.<namespace>.svc.clusterset.local
. If multiple clusters export a service with the same name, and from the same namespace, they are recognized by other clusters as a single logical service.
This example uses the nginx
service in the default
namespace, but you can discover any Kubernetes ClusterIP
service or headless service:
Apply an instance of the
nginx
service on a managed cluster that is in theManagedClusterSet
by entering the following commands:oc -n default create deployment nginx --image=nginxinc/nginx-unprivileged:stable-alpine oc -n default expose deployment nginx --port=8080
Export the service by creating a
ServiceExport
entry that resembles the following content in the YAML file:apiVersion: multicluster.x-k8s.io/v1alpha1 kind: ServiceExport metadata: name: <service-name> namespace: <service-namespace>
Replace
service-name
with the name of the service that you are exporting. In this example, it isnginx
. Replaceservice-namespace
with the name of the namespace where the service is located. In this example, it isdefault
.Run the following command from a different managed cluster to confirm that it can access the
nginx
service:oc -n default run --generator=run-pod/v1 tmp-shell --rm -i --tty --image quay.io/submariner/nettest -- /bin/bash curl nginx.default.svc.clusterset.local:8080
The nginx
service discovery is now configured for Submariner.