Chapter 7. Multiple networks

Procedure

1. Add an annotation to the Pod object. Only one of the following annotation formats can be used:

   1. To attach an additional network without any customization, add an annotation with the following format. Replace <network> with the name of the additional network to associate with the Pod:

      metadata:
        annotations:
          k8s.v1.cni.cncf.io/networks: <network>[,<network>,...] 1

      1

      To specify more than one additional network, separate each network with a comma. Do not include whitespace between the comma. If you specify the same additional network multiple times, that Pod will have multiple network interfaces attached to that network.

   2. To attach an additional network with customizations, add an annotation with the following format:

      metadata:
        annotations:
          k8s.v1.cni.cncf.io/networks: |-
            [
              {
                "name": "<network>", 1
                "namespace": "<namespace>", 2
                "default-route": ["<default-route>"] 3
              }
            ]

      1

      Specify the name of the additional network defined by a NetworkAttachmentDefinition CR.

      2

      Specify the namespace where the NetworkAttachmentDefinition CR is defined.

      3

      Optional: Specify an override for the default route, such as 192.168.17.1.

2. To create the Pod, enter the following command. Replace <name> with the name of the Pod.

   $ oc create -f <name>.yaml

3. Optional: To Confirm that the annotation exists in the Pod CR, enter the following command, replacing <name> with the name of the Pod.

   $ oc get pod <name> -o yaml

   In the following example, the example-pod Pod is attached to the net1 additional network:

   $ oc get pod example-pod -o yaml
   apiVersion: v1
   kind: Pod
   metadata:
     annotations:
       k8s.v1.cni.cncf.io/networks: macvlan-bridge
       k8s.v1.cni.cncf.io/networks-status: |- 1
         [{
             "name": "openshift-sdn",
             "interface": "eth0",
             "ips": [
                 "10.128.2.14"
             ],
             "default": true,
             "dns": {}
         },{
             "name": "macvlan-bridge",
             "interface": "net1",
             "ips": [
                 "20.2.2.100"
             ],
             "mac": "22:2f:60:a5:f8:00",
             "dns": {}
         }]
     name: example-pod
     namespace: default
   spec:
     ...
   status:
     ...

   1

   The k8s.v1.cni.cncf.io/networks-status parameter is a JSON array of objects. Each object describes the status of an additional network attached to the Pod. The annotation value is stored as a plain text value.

1. Edit the Pod resource definition by running the following command. Replace <name> with the name of the Pod to edit.

   $ oc edit pod <name>

2. Update the annotations mapping to remove the additional network from the Pod by performing one of the following actions:

   • To remove all additional networks from a Pod, remove the k8s.v1.cni.cncf.io/networks parameter from the Pod resource definition as in the following example:

     apiVersion: v1
     kind: Pod
     metadata:
       name: example-pod
       annotations: {}
     spec:
       containers:
       - name: example-pod
         command: ["/bin/bash", "-c", "sleep 2000000000000"]
         image: centos/tools

   • To remove a specific additional network from a Pod, update the k8s.v1.cni.cncf.io/networks parameter by removing the name of the NetworkAttachmentDefinition for the additional network.

3. Optional: Confirm that the Pod is no longer attached to the additional network by running the following command. Replace <name> with the name of the Pod.

   $ oc describe pod <name>

   In the following example, the example-pod Pod is attached only to the default cluster network.

   $ oc describe pod example-pod

   Example output

   Name:               example-pod
   ...
   Annotations:        k8s.v1.cni.cncf.io/networks-status:
                         [{
                             "name": "openshift-sdn",
                             "interface": "eth0",
                             "ips": [
                                 "10.131.0.13"
                             ],
                             "default": true, 1
                             "dns": {}
                         }]
   Status:             Running
   ...

   1

   Only the default cluster network is attached to the Pod.

1. Edit the CNO CR by running the following command:

   $ oc edit networks.operator.openshift.io cluster

2. Modify the CR that you are creating by adding the configuration for the additional network you are creating, as in the following example CR.

   The following YAML configures the bridge CNI plug-in:

   apiVersion: operator.openshift.io/v1
   kind: Network
   metadata:
     name: cluster
   spec:
     additionalNetworks: 1
     - name: test-network-1
       namespace: test-1
       type: Raw
       rawCNIConfig: '{
         "cniVersion": "0.3.1",
         "name": "test-network-1",
         "type": "bridge",
         "ipam": {
           "type": "static",
           "addresses": [
             {
               "address": "191.168.1.7"
             }
           ]
         }
       }'

   1

   Specify the configuration for the additional network attachment definition.

3. Save your changes and quit the text editor to commit your changes.

4. Optional: Confirm that the CNO created the NetworkAttachmentDefinition CR by running the following command. There might be a delay before the CNO creates the CR.

   $ oc get network-attachment-definitions -n <namespace>

   Example output

   NAME                 AGE
   test-network-1       14m

1. Edit the CNO CR by running the following command:

   $ oc edit networks.operator.openshift.io cluster

2. Modify the CR that you are creating by adding the configuration for the additional network you are creating, as in the following example CR.

   The following YAML configures the macvlan CNI plug-in:

   apiVersion: operator.openshift.io/v1
   kind: Network
   metadata:
     name: cluster
   spec:
     additionalNetworks: 1
     - name: test-network-1
       namespace: test-1
       type: SimpleMacvlan
       simpleMacvlanConfig:
         ipamConfig:
           type: static
           staticIPAMConfig:
             addresses:
             - address: 10.1.1.7

   1

   Specify the configuration for the additional network attachment definition.

3. Save your changes and quit the text editor to commit your changes.

4. Optional: Confirm that the CNO created the NetworkAttachmentDefinition CR by running the following command. There might be a delay before the CNO creates the CR.

   $ oc get network-attachment-definitions -n <namespace>

   Example output

   NAME                 AGE
   test-network-1       14m

1. Edit the CNO CR by running the following command:

   $ oc edit networks.operator.openshift.io cluster

2. Modify the CR that you are creating by adding the configuration for the additional network you are creating, as in the following example CR.

   The following YAML configures the ipvlan CNI plug-in:

   apiVersion: operator.openshift.io/v1
   kind: Network
   metadata:
     name: cluster
   spec:
     additionalNetworks: 1
     - name: test-network-1
       namespace: test-1
       type: Raw
       rawCNIConfig: '{
         "cniVersion": "0.3.1",
         "name": "test-network-1",
         "type": "ipvlan",
         "master": "eth1",
         "mode": "l2",
         "ipam": {
           "type": "static",
           "addresses": [
             {
               "address": "191.168.1.7"
             }
           ]
         }
       }'

   1

   Specify the configuration for the additional network attachment definition.

3. Save your changes and quit the text editor to commit your changes.

4. Optional: Confirm that the CNO created the NetworkAttachmentDefinition CR by running the following command. There might be a delay before the CNO creates the CR.

   $ oc get network-attachment-definitions -n <namespace>

   Example output

   NAME                 AGE
   test-network-1       14m

1. Edit the CNO CR by running the following command:

   $ oc edit networks.operator.openshift.io cluster

2. Modify the CR that you are creating by adding the configuration for the additional network you are creating, as in the following example CR.

   The following YAML configures the host-device CNI plug-in:

   apiVersion: operator.openshift.io/v1
   kind: Network
   metadata:
     name: cluster
   spec:
     additionalNetworks: 1
     - name: test-network-1
       namespace: test-1
       type: Raw
       rawCNIConfig: '{
         "cniVersion": "0.3.1",
         "name": "test-network-1",
         "type": "host-device",
         "device": "eth1"
       }'

   1

   Specify the configuration for the additional network attachment definition.

3. Save your changes and quit the text editor to commit your changes.

4. Optional: Confirm that the CNO created the NetworkAttachmentDefinition CR by running the following command. There might be a delay before the CNO creates the CR.

   $ oc get network-attachment-definitions -n <namespace>

   Example output

   NAME                 AGE
   test-network-1       14m

1. Run the following command to edit the Cluster Network Operator (CNO) CR in your default text editor:

   $ oc edit networks.operator.openshift.io cluster

2. In the additionalNetworks collection, update the additional network with your changes.
3. Save your changes and quit the text editor to commit your changes.

4. Optional: Confirm that the CNO updated the NetworkAttachmentDefinition CR by running the following command. Replace <network-name> with the name of the additional network to display. There might be a delay before the CNO updates the NetworkAttachmentDefinition CR to reflect your changes.

   $ oc get network-attachment-definitions <network-name> -o yaml

   For example, the following console output displays a NetworkAttachmentDefinition that is named net1:

   $ oc get network-attachment-definitions net1 -o go-template='{{printf "%s\n" .spec.config}}'
   { "cniVersion": "0.3.1", "type": "macvlan",
   "master": "ens5",
   "mode": "bridge",
   "ipam":       {"type":"static","routes":[{"dst":"0.0.0.0/0","gw":"10.128.2.1"}],"addresses":[{"address":"10.128.2.100/23","gateway":"10.128.2.1"}],"dns":{"nameservers":["172.30.0.10"],"domain":"us-west-2.compute.internal","search":["us-west-2.compute.internal"]}} }

1. Edit the Cluster Network Operator (CNO) in your default text editor by running the following command:

   $ oc edit networks.operator.openshift.io cluster

2. Modify the CR by removing the configuration from the additionalNetworks collection for the network attachment definition you are removing.

   apiVersion: operator.openshift.io/v1
   kind: Network
   metadata:
     name: cluster
   spec:
     additionalNetworks: [] 1

   1

   If you are removing the configuration mapping for the only additional network attachment definition in the additionalNetworks collection, you must specify an empty collection.

3. Save your changes and quit the text editor to commit your changes.

4. Optional: Confirm that the additional network CR was deleted by running the following command:

   $ oc get network-attachment-definition --all-namespaces

Procedure

1. Create the following PtpConfig CR, and then save the YAML in the <name>-ptp-config.yaml file. Replace <name> with the name for this configuration.

   apiVersion: ptp.openshift.io/v1
   kind: PtpConfig
   metadata:
     name: <name> 1
     namespace: openshift-ptp 2
   spec:
     profile: 3
     - name: "profile1" 4
       interface: "ens787f1" 5
       ptp4lOpts: "-s -2" 6
       phc2sysOpts: "-a -r" 7
     recommend: 8
     - profile: "profile1" 9
       priority: 10 10
       match: 11
       - nodeLabel: "node-role.kubernetes.io/worker" 12
         nodeName: "dev-worker-0" 13

   1

   Specify a name for the PtpConfig CR.

   2

   Specify the namespace where the PTP Operator is installed.

   3

   Specify an array of one or more profile objects.

   4

   Specify the name of a profile object which is used to uniquely identify a profile object.

   5

   Specify the network interface name to use by the ptp4l service, for example ens787f1.

   6

   Specify system config options for the ptp4l service, for example -s -2. This should not include the interface name -i <interface> and service config file -f /etc/ptp4l.conf because these will be automatically appended.

   7

   Specify system config options for the phc2sys service, for example -a -r.

   8

   Specify an array of one or more recommend objects which define rules on how the profile should be applied to nodes.

   9

   Specify the profile object name defined in the profile section.

   10

   Specify the priority with an integer value between 0 and 99. A larger number gets lower priority, so a priority of 99 is lower than a priority of 10. If a node can be matched with multiple profiles according to rules defined in the match field, the profile with the higher priority will be applied to that node.

   11

   Specify match rules with nodeLabel or nodeName.

   12

   Specify nodeLabel with the key of node.Labels from the node object.

   13

   Specify nodeName with node.Name from the node object.

2. Create the CR by running the following command:

   $ oc create -f <filename> 1

   1

   Replace <filename> with the name of the file you created in the previous step.

3. Optional: Check that the PtpConfig profile is applied to nodes that match with nodeLabel or nodeName.

   $ oc get pods -n openshift-ptp -o wide

   Example output

   NAME                            READY   STATUS    RESTARTS   AGE   IP               NODE           NOMINATED NODE   READINESS GATES
   linuxptp-daemon-4xkbb           1/1     Running   0          43m   192.168.111.15   dev-worker-0   <none>           <none>
   linuxptp-daemon-tdspf           1/1     Running   0          43m   192.168.111.11   dev-master-0   <none>           <none>
   ptp-operator-657bbb64c8-2f8sj   1/1     Running   0          43m   10.128.0.116     dev-master-0   <none>           <none>
   
   $ oc logs linuxptp-daemon-4xkbb -n openshift-ptp
   I1115 09:41:17.117596 4143292 daemon.go:107] in applyNodePTPProfile
   I1115 09:41:17.117604 4143292 daemon.go:109] updating NodePTPProfile to:
   I1115 09:41:17.117607 4143292 daemon.go:110] ------------------------------------
   I1115 09:41:17.117612 4143292 daemon.go:102] Profile Name: profile1 1
   I1115 09:41:17.117616 4143292 daemon.go:102] Interface: ens787f1    2
   I1115 09:41:17.117620 4143292 daemon.go:102] Ptp4lOpts: -s -2       3
   I1115 09:41:17.117623 4143292 daemon.go:102] Phc2sysOpts: -a -r     4
   I1115 09:41:17.117626 4143292 daemon.go:116] ------------------------------------
   I1115 09:41:18.117934 4143292 daemon.go:186] Starting phc2sys...
   I1115 09:41:18.117985 4143292 daemon.go:187] phc2sys cmd: &{Path:/usr/sbin/phc2sys Args:[/usr/sbin/phc2sys -a -r] Env:[] Dir: Stdin:<nil> Stdout:<nil> Stderr:<nil> ExtraFiles:[] SysProcAttr:<nil> Process:<nil> ProcessState:<nil> ctx:<nil> lookPathErr:<nil> finished:false childFiles:[] closeAfterStart:[] closeAfterWait:[] goroutine:[] errch:<nil> waitDone:<nil>}
   I1115 09:41:19.118175 4143292 daemon.go:186] Starting ptp4l...
   I1115 09:41:19.118209 4143292 daemon.go:187] ptp4l cmd: &{Path:/usr/sbin/ptp4l Args:[/usr/sbin/ptp4l -m -f /etc/ptp4l.conf -i ens787f1 -s -2] Env:[] Dir: Stdin:<nil> Stdout:<nil> Stderr:<nil> ExtraFiles:[] SysProcAttr:<nil> Process:<nil> ProcessState:<nil> ctx:<nil> lookPathErr:<nil> finished:false childFiles:[] closeAfterStart:[] closeAfterWait:[] goroutine:[] errch:<nil> waitDone:<nil>}
   ptp4l[102189.864]: selected /dev/ptp5 as PTP clock
   ptp4l[102189.886]: port 1: INITIALIZING to LISTENING on INIT_COMPLETE
   ptp4l[102189.886]: port 0: INITIALIZING to LISTENING on INIT_COMPLETE

   1

   Profile Name is the name that is applied to node dev-worker-0.

   2

   Interface is the PTP device specified in the profile1 interface field. The ptp4l service runs on this interface.

   3

   Ptp4lOpts are the ptp4l sysconfig options specified in profile1 Ptp4lOpts field.

   4

   Phc2sysOpts are the phc2sys sysconfig options specified in profile1 Phc2sysOpts field.