Chapter 10. Custom network interface templates

This chapter follows on from the concepts and procedures outlined in Chapter 8, Basic network isolation. The purpose of this chapter is to demonstrate how to create a set of custom network interface template to suit nodes in your environment. This includes:

  • The environment file to enable network isolation (/usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml).
  • The environment file to configure network defaults (/usr/share/openstack-tripleo-heat-templates/environments/network-environment.yaml).
  • Templates to define your NIC layout for each node. The overcloud core template collection contains a set of defaults for different use cases. In this situation, you will render a default a basis for your custom templates.
  • A custom environment file to enable NICs. This example uses a custom environment file (/home/stack/templates/custom-network-configuration.yaml) that references your custom interface templates.
  • Any additional environment files to customize your networking parameters.
  • If using customizing your networks, a custom network_data file.
  • If creating additional or custom composable networks, a custom network_data file and a custom roles_data file.

10.1. Custom network architecture

The default NIC templates might not suit a specific network configuration. For example, you might want to create your own custom NIC template that suits a specific network layout. You might aim to separate the control services and data services on to separate NICs. In this situation, the service to NIC assignments result in the following mapping:

  • NIC1 (Provisioning):

    • Provisioning / Control Plane
  • NIC2 (Control Group)

    • Internal API
    • Storage Management
    • External (Public API)
  • NIC3 (Data Group)

    • Tenant Network (VXLAN tunneling)
    • Tenant VLANs / Provider VLANs
    • Storage
    • External VLANs (Floating IP/SNAT)
  • NIC4 (Management)

    • Management

10.2. Rendering default network interface templates for customization

For the purposes of simplifying the configuration of custom interface templates, this procedure shows you how to render the Jinja2 syntax of a default NIC template. This way you can use the rendered templates as a basis for your custom configuration.

Procedure

  1. Render a copy of the openstack-tripleo-heat-templates collection using the process-templates.py script:

    $ cd /usr/share/openstack-tripleo-heat-templates
    $ ./tools/process-templates.py -o ~/openstack-tripleo-heat-templates-rendered

    This converts all Jinja2 templates to their rendered YAML versions and saves the results to ~/openstack-tripleo-heat-templates-rendered.

    If using a custom network file or custom roles file, you can include these files using the -n and -r options respectively. For example:

    $ ./tools/process-templates.py -o ~/openstack-tripleo-heat-templates-rendered -n /home/stack/network_data.yaml -r /home/stack/roles_data.yaml
  2. Copy the multiple NIC example:

    $ cp -r ~/openstack-tripleo-heat-templates-rendered/network/config/multiple-nics/ ~/templates/custom-nics/
  3. You can edit the template set in custom-nics to suit your own network configuration.

10.3. Network interface architecture

This section explores the architecture of the custom NIC templates in custom-nics and provides recommendations on editing them.

Parameters

The parameters section contains all network configuration parameters for network interfaces. This includes information such as subnet ranges and VLAN IDs. This section should remain unchanged as the Heat template inherits values from its parent template. However, you can modify the values for some parameters using a network environment file.

Resources

The resources section is where the main network interface configuration occurs. In most cases, the resources section is the only one that requires editing. Each resources section begins with the following header:

resources:
  OsNetConfigImpl:
    type: OS::Heat::SoftwareConfig
    properties:
      group: script
      config:
        str_replace:
          template:
            get_file: /usr/share/openstack-tripleo-heat-templates/network/scripts/run-os-net-config.sh
          params:
            $network_config:
              network_config:

This runs a script (run-os-net-config.sh) that creates a configuration file for os-net-config to use for configuring network properties on a node. The network_config section contains the custom network interface data sent to the run-os-net-config.sh script. You arrange this custom interface data in a sequence based on the type of device.

Important

If creating custom NIC templates, you must set the run-os-net-config.sh script location to an absolute location for each NIC template. The script is located at /usr/share/openstack-tripleo-heat-templates/network/scripts/run-os-net-config.sh on the undercloud.

10.4. Network interface reference

The following sections define the network interface types and the parameters used in each.

interface

Defines a single network interface. The configuration defines each interface using either the actual interface name ("eth0", "eth1", "enp0s25") or a set of numbered interfaces ("nic1", "nic2", "nic3").

For example:

  - type: interface
    name: nic2

Table 10.1. interface options

OptionDefaultDescription

name

 

Name of the Interface

use_dhcp

False

Use DHCP to get an IP address

use_dhcpv6

False

Use DHCP to get a v6 IP address

addresses

 

A list of IP addresses assigned to the interface

routes

 

A list of routes assigned to the interface. See routes.

mtu

1500

The maximum transmission unit (MTU) of the connection

primary

False

Defines the interface as the primary interface

defroute

True

Use a default route provided by the DHCP service. Only applies when use_dhcp or use_dhcpv6 is enabled.

persist_mapping

False

Write the device alias configuration instead of the system names

dhclient_args

None

Arguments to pass to the DHCP client

dns_servers

None

List of DNS servers to use for the interface

vlan

Defines a VLAN. Use the VLAN ID and subnet passed from the parameters section.

For example:

  - type: vlan
    vlan_id:{get_param: ExternalNetworkVlanID}
    addresses:
      - ip_netmask: {get_param: ExternalIpSubnet}

Table 10.2. vlan options

OptionDefaultDescription

vlan_id

 

The VLAN ID

device

 

The parent device to attach the VLAN. Use this parameter when the VLAN is not a member of an OVS bridge. For example, use this parameter to attach the VLAN to a bonded interface device.

use_dhcp

False

Use DHCP to get an IP address

use_dhcpv6

False

Use DHCP to get a v6 IP address

addresses

 

A list of IP addresses assigned to the VLAN

routes

 

A list of routes assigned to the VLAN. See routes.

mtu

1500

The maximum transmission unit (MTU) of the connection

primary

False

Defines the VLAN as the primary interface

defroute

True

Use a default route provided by the DHCP service. Only applies when use_dhcp or use_dhcpv6 is enabled.

persist_mapping

False

Write the device alias configuration instead of the system names

dhclient_args

None

Arguments to pass to the DHCP client

dns_servers

None

List of DNS servers to use for the VLAN

ovs_bond

Defines a bond in Open vSwitch to join two or more interfaces together. This helps with redundancy and increases bandwidth.

For example:

          - type: ovs_bond
            name: bond1
            members:
            - type: interface
              name: nic2
            - type: interface
              name: nic3

Table 10.3. ovs_bond options

OptionDefaultDescription

name

 

Name of the bond

use_dhcp

False

Use DHCP to get an IP address

use_dhcpv6

False

Use DHCP to get a v6 IP address

addresses

 

A list of IP addresses assigned to the bond

routes

 

A list of routes assigned to the bond. See routes.

mtu

1500

The maximum transmission unit (MTU) of the connection

primary

False

Defines the interface as the primary interface

members

 

A sequence of interface objects to use in the bond

ovs_options

 

A set of options to pass to OVS when creating the bond

ovs_extra

 

A set of options to to set as the OVS_EXTRA parameter in the bond’s network configuration file

defroute

True

Use a default route provided by the DHCP service. Only applies when use_dhcp or use_dhcpv6 is enabled.

persist_mapping

False

Write the device alias configuration instead of the system names

dhclient_args

None

Arguments to pass to the DHCP client

dns_servers

None

List of DNS servers to use for the bond

ovs_bridge

Defines a bridge in Open vSwitch, which connects multiple interface, ovs_bond, and vlan objects together. The external bridge also uses two special values for parameters:

  • bridge_name, which is replaced with the external bridge name.
  • interface_name, which is replaced with the external interface.

For example:

      - type: ovs_bridge
        name: bridge_name
        addresses:
        - ip_netmask:
            list_join:
            - /
            - - {get_param: ControlPlaneIp}
              - {get_param: ControlPlaneSubnetCidr}
        members:
          - type: interface
            name: interface_name
      - type: vlan
        device: bridge_name
        vlan_id:
          {get_param: ExternalNetworkVlanID}
        addresses:
          - ip_netmask:
              {get_param: ExternalIpSubnet}
Note

The OVS bridge connects to the Neutron server in order to get configuration data. If the OpenStack control traffic (typically the Control Plane and Internal API networks) is placed on an OVS bridge, then connectivity to the Neutron server gets lost whenever OVS is upgraded or the OVS bridge is restarted by the admin user or process. This will cause some downtime. If downtime is not acceptable under these circumstances, then the Control group networks should be placed on a separate interface or bond rather than on an OVS bridge:

  • A minimal setting can be achieved, when you put the Internal API network on a VLAN on the provisioning interface and the OVS bridge on a second interface.
  • If you want bonding, you need at least two bonds (four network interfaces). The control group should be placed on a Linux bond (Linux bridge). If the switch does not support LACP fallback to a single interface for PXE boot, then this solution requires at least five NICs.

Table 10.4. ovs_bridge options

OptionDefaultDescription

name

 

Name of the bridge

use_dhcp

False

Use DHCP to get an IP address

use_dhcpv6

False

Use DHCP to get a v6 IP address

addresses

 

A list of IP addresses assigned to the bridge

routes

 

A list of routes assigned to the bridge. See routes.

mtu

1500

The maximum transmission unit (MTU) of the connection

members

 

A sequence of interface, VLAN, and bond objects to use in the bridge

ovs_options

 

A set of options to pass to OVS when creating the bridge

ovs_extra

 

A set of options to to set as the OVS_EXTRA parameter in the bridge’s  network configuration file

defroute

True

Use a default route provided by the DHCP service. Only applies when use_dhcp or use_dhcpv6 is enabled.

persist_mapping

False

Write the device alias configuration instead of the system names

dhclient_args

None

Arguments to pass to the DHCP client

dns_servers

None

List of DNS servers to use for the bridge

linux_bond

Defines a Linux bond that joins two or more interfaces together. This helps with redundancy and increases bandwidth. Make sure to include the kernel-based bonding options in the bonding_options parameter.

For example:

      - type: linux_bond
        name: bond1
        members:
        - type: interface
          name: nic2
          primary: true
        - type: interface
          name: nic3
        bonding_options: "mode=802.3ad"

Note that nic2 uses primary: true. This ensures the bond uses the MAC address for nic2.

Table 10.5. linux_bond options

OptionDefaultDescription

name

 

Name of the bond

use_dhcp

False

Use DHCP to get an IP address

use_dhcpv6

False

Use DHCP to get a v6 IP address

addresses

 

A list of IP addresses assigned to the bond

routes

 

A list of routes assigned to the bond. See routes.

mtu

1500

The maximum transmission unit (MTU) of the connection

primary

False

Defines the interface as the primary interface.

members

 

A sequence of interface objects to use in the bond

bonding_options

 

A set of options when creating the bond.

defroute

True

Use a default route provided by the DHCP service. Only applies when use_dhcp or use_dhcpv6 is enabled.

persist_mapping

False

Write the device alias configuration instead of the system names

dhclient_args

None

Arguments to pass to the DHCP client

dns_servers

None

List of DNS servers to use for the bond

linux_bridge

Defines a Linux bridge, which connects multiple interface, linux_bond, and vlan objects together. The external bridge also uses two special values for parameters:

  • bridge_name, which is replaced with the external bridge name.
  • interface_name, which is replaced with the external interface.

For example:

      - type: linux_bridge
        name: bridge_name
        addresses:
          - ip_netmask:
              list_join:
                - /
                - - {get_param: ControlPlaneIp}
                  - {get_param: ControlPlaneSubnetCidr}
        members:
          - type: interface
            name: interface_name
      - type: vlan
        device: bridge_name
        vlan_id:
          {get_param: ExternalNetworkVlanID}
        addresses:
          - ip_netmask:
              {get_param: ExternalIpSubnet}

Table 10.6. linux_bridge options

OptionDefaultDescription

name

 

Name of the bridge

use_dhcp

False

Use DHCP to get an IP address

use_dhcpv6

False

Use DHCP to get a v6 IP address

addresses

 

A list of IP addresses assigned to the bridge

routes

 

A list of routes assigned to the bridge. See routes.

mtu

1500

The maximum transmission unit (MTU) of the connection

members

 

A sequence of interface, VLAN, and bond objects to use in the bridge

defroute

True

Use a default route provided by the DHCP service. Only applies when use_dhcp or use_dhcpv6 is enabled.

persist_mapping

False

Write the device alias configuration instead of the system names

dhclient_args

None

Arguments to pass to the DHCP client

dns_servers

None

List of DNS servers to use for the bridge

routes

Defines a list of routes to apply to a network interface, VLAN, bridge, or bond.

For example:

  - type: interface
    name: nic2
    ...
    routes:
      - ip_netmask: 10.1.2.0/24
        default: true
        next_hop:
          get_param: EC2MetadataIp
OptionDefaultDescription

ip_netmask

None

IP and netmask of the destination network.

default

False

Sets this this route to a default route. Equivalent to setting ip_netmask: 0.0.0.0/0.

next_hop

None

The IP address of the router used to reach the destination network.

10.5. Example network interface layout

The following snippet for a possible Controller node NIC template demonstrates how to configure the custom network scenario to keep the control group apart from the OVS bridge:

resources:
  OsNetConfigImpl:
    type: OS::Heat::SoftwareConfig
    properties:
      group: script
      config:
        str_replace:
          template:
            get_file: /usr/share/openstack-tripleo-heat-templates/network/scripts/run-os-net-config.sh
          params:
            $network_config:
              network_config:

              # NIC 1 - Provisioning
              - type: interface
                name: nic1
                use_dhcp: false
                addresses:
                - ip_netmask:
                    list_join:
                    - /
                    - - get_param: ControlPlaneIp
                      - get_param: ControlPlaneSubnetCidr
                routes:
                - ip_netmask: 169.254.169.254/32
                  next_hop:
                    get_param: EC2MetadataIp

              # NIC 2 - Control Group
              - type: interface
                name: nic2
                use_dhcp: false
              - type: vlan
                device: nic2
                vlan_id:
                  get_param: InternalApiNetworkVlanID
                addresses:
                - ip_netmask:
                    get_param: InternalApiIpSubnet
              - type: vlan
                device: nic2
                vlan_id:
                  get_param: StorageMgmtNetworkVlanID
                addresses:
                - ip_netmask:
                    get_param: StorageMgmtIpSubnet
              - type: vlan
                device: nic2
                vlan_id:
                  get_param: ExternalNetworkVlanID
                addresses:
                - ip_netmask:
                    get_param: ExternalIpSubnet
                routes:
                - default: true
                  next_hop:
                    get_param: ExternalInterfaceDefaultRoute

              # NIC 3 - Data Group
              - type: ovs_bridge
                name: bridge_name
                dns_servers:
                  get_param: DnsServers
                members:
                - type: interface
                  name: nic3
                  primary: true
                - type: vlan
                  vlan_id:
                    get_param: StorageNetworkVlanID
                  addresses:
                  - ip_netmask:
                      get_param: StorageIpSubnet
                - type: vlan
                  vlan_id:
                    get_param: TenantNetworkVlanID
                  addresses:
                  - ip_netmask:
                      get_param: TenantIpSubnet

                # NIC 4 - Management
                - type: interface
                  name: nic4
                  use_dhcp: false
                  addresses:
                  - ip_netmask: {get_param: ManagementIpSubnet}
                  routes:
                  - default: true
                    next_hop: {get_param: ManagementInterfaceDefaultRoute}

This template uses four network interfaces and assigns a number of tagged VLAN devices to the numbered interfaces, nic1 to nic4. On nic3 it creates the OVS bridge that hosts the Storage and Tenant networks. As a result, it creates the following layout:

  • NIC1 (Provisioning):

    • Provisioning / Control Plane
  • NIC2 (Control Group)

    • Internal API
    • Storage Management
    • External (Public API)
  • NIC3 (Data Group)

    • Tenant Network (VXLAN tunneling)
    • Tenant VLANs / Provider VLANs
    • Storage
    • External VLANs (Floating IP/SNAT)
  • NIC4 (Management)

    • Management

10.6. Network interface template considerations for custom networks

When using composable networks, the process-templates.py script renders the static templates to include networks and roles defined in your network_data and roles_data files. Check the rendered NIC templates and ensure it contains:

  • Static file for each roles, including custom composable networks.
  • Each static file for each role contains the correct network definitions.

Each static file requires all the parameter definitions for any custom networks even if the network is not used on the role. Check to make sure the rendered templates contain these parameters. For example, if a StorageBackup network is added to only the Ceph nodes, the parameters section in NIC configuration templates for all roles must also include this definition:

parameters:
  ...
  StorageBackupIpSubnet:
    default: ''
    description: IP address/subnet on the external network
    type: string
  ...

You can also include the parameters definitions for VLAN IDs and/or gateway IP, if needed:

parameters:
  ...
  StorageBackupNetworkVlanID:
    default: 60
    description: Vlan ID for the management network traffic.
    type: number
  StorageBackupDefaultRoute:
	  description: The default route of the storage backup network.
	  type: string
  ...

The IpSubnet parameter for the custom network appears in the parameter definitions for each role. However, since the Ceph role might be the only role that uses the StorageBackup network, only the NIC configuration template for the Ceph role would make use of the StorageBackup parameters in the network_config section of the template.

      $network_config:
        network_config:
        - type: interface
          name: nic1
          use_dhcp: false
          addresses:
          - ip_netmask:
              get_param: StorageBackupIpSubnet

10.7. Custom network environment file

The custom network environment file (in this case, /home/stack/templates/custom-network-configuration.yaml) is a Heat environment file that describes the Overcloud’s network environment and points to the custom network interface configuration templates. You can define the subnets and VLANs for your network along with IP address ranges. You can then customize these values for the local environment.

The resource_registry section contains references to the custom network interface templates for each node role. Each resource registered uses the following format:

  • OS::TripleO::[ROLE]::Net::SoftwareConfig: [FILE]

[ROLE] is the role name and [FILE] is the respective network interface template for that particular role. For example:

resource_registry:
  OS::TripleO::Controller::Net::SoftwareConfig: /home/stack/templates/custom-nics/controller.yaml

The parameter_defaults section contains a list of parameters that define the network options for each network type.

10.8. Network environment parameters

The following table is a list of parameters you can use in a network environment file’s parameter_defaults section to override the default parameter values in your NIC templates.

ParameterDescriptionType

ControlPlaneDefaultRoute

The IP address of the router on the Control Plane, which is used as a default route for roles other than the Controller nodes by default. Set to the undercloud IP if using IP masquerade instead of a router.

string

ControlPlaneSubnetCidr

The CIDR netmask of the IP network used on the Control Plane. If the Control Plane network uses 192.168.24.0/24, the CIDR is 24.

string (though is always a number)

*NetCidr

The full network and CIDR netmask for a particular network. The default is automatically set to the network’s ip_subnet setting in the network_data file. For example: InternalApiNetCidr: 172.16.0.0/24

string

*AllocationPools

"The IP allocation range for a particular network. The default is automatically set to the network’s allocation_pools setting in the network_data file. For example: InternalApiAllocationPools: [{'start': '172.16.0.10', 'end': '172.16.0.200'}]

hash

*NetworkVlanID

The node’s VLAN ID for on a particular network. The default is set automatically to the network’s vlan setting in the network_data file. For example: InternalApiNetworkVlanID: 201

number

*InterfaceDefaultRoute

The router address for a particular network, which you can use as a default route for roles or used for routes to other networks. The default is automatically set to the network’s gateway_ip setting in the network_data file. For example: InternalApiInterfaceDefaultRoute: 172.16.0.1

string

DnsServers

A list of DNS servers added to resolv.conf. Usually allows a maximum of 2 servers.

comma delimited list

EC2MetadataIp

The IP address of the metadata server used to provision overcloud nodes. Set to the IP address of the undercloud on the Control Plane.

string

BondInterfaceOvsOptions

The options for bonding interfaces. For example: BondInterfaceOvsOptions: "bond_mode=balance-slb"

string

NeutronExternalNetworkBridge

Legacy value for the name of the external bridge to use for OpeNStack Networking (neutron). This value is empty by default, which allows for multiple physical bridges to be defined in the NeutronBridgeMappings. This should not normally be overridden.

string

NeutronFlatNetworks

Defines the flat networks to configure in neutron plugins. Defaults to "datacentre" to permit external network creation. For example: NeutronFlatNetworks: "datacentre"

string

NeutronBridgeMappings

The logical to physical bridge mappings to use. Defaults to mapping the external bridge on hosts (br-ex) to a physical name (datacentre). You would refer to the logical name when creating OpenStack Networking (neutron) provider networks or floating IP networks. For example NeutronBridgeMappings: "datacentre:br-ex,tenant:br-tenant"

string

NeutronPublicInterface

Defines the interface to bridge onto br-ex for network nodes when not using network isolation. Usually not used except in small deployments with only two networks. For example: NeutronPublicInterface: "eth0"

string

NeutronNetworkType

TThe tenant network type for OpenStack Networking (neutron). To specify multiple values, use a comma separated list. The first type specified is used until all available networks are exhausted, then the next type is used. For example: NeutronNetworkType: "vxlan"

string

NeutronTunnelTypes

The tunnel types for the neutron tenant network. To specify multiple values, use a comma separated string. For example: NeutronTunnelTypes: 'gre,vxlan'

string / comma separated list

NeutronTunnelIdRanges

Ranges of GRE tunnel IDs to make available for tenant network allocation. For example: NeutronTunnelIdRanges "1:1000"

string

NeutronVniRanges

Ranges of VXLAN VNI IDs to make available for tenant network allocation. For example: NeutronVniRanges: "1:1000"

string

NeutronEnableTunnelling

Defines whether to enable or completely disable all tunnelled networks. Leave this enabled unless you are sure you will never want to create tunelled networks. Defaults to enabled.

Boolean

NeutronNetworkVLANRanges

The ML2 and Open vSwitch VLAN mapping range to support. Defaults to permitting any VLAN on the datacentre physical network. To specify multiple values, use a comma separated list. For example: NeutronNetworkVLANRanges: "datacentre:1:1000,tenant:100:299,tenant:310:399"

string

NeutronMechanismDrivers

The mechanism drivers for the neutron tenant network. Defaults to "ovn". To specify multiple values, use a comma-separated string. For example: NeutronMechanismDrivers: 'openvswitch,l2population'

string / comma separated list

10.9. Example custom network environment file

The following is an example of an environment file to enable your NIC templates and set custom parameters.

resource_registry:
  OS::TripleO::BlockStorage::Net::SoftwareConfig:
    /home/stack/templates/nic-configs/cinder-storage.yaml
  OS::TripleO::Compute::Net::SoftwareConfig:
    /home/stack/templates/nic-configs/compute.yaml
  OS::TripleO::Controller::Net::SoftwareConfig:
    /home/stack/templates/nic-configs/controller.yaml
  OS::TripleO::ObjectStorage::Net::SoftwareConfig:
    /home/stack/templates/nic-configs/swift-storage.yaml
  OS::TripleO::CephStorage::Net::SoftwareConfig:
    /home/stack/templates/nic-configs/ceph-storage.yaml

parameter_defaults:
  # Gateway router for the provisioning network (or Undercloud IP)
  ControlPlaneDefaultRoute: 192.0.2.254
  # The IP address of the EC2 metadata server. Generally the IP of the Undercloud
  EC2MetadataIp: 192.0.2.1
  # Define the DNS servers (maximum 2) for the overcloud nodes
  DnsServers: ["8.8.8.8","8.8.4.4"]
  NeutronExternalNetworkBridge: "''"

10.10. Enabling network isolation with custom NICs

This procedure show how to enable network isolation using custom NIC templates.

Procedure

  1. When running the openstack overcloud deploy command, make sure to include:

    • The custom network_data file.
    • The rendered file name of the default network isolation.
    • The rendered file name of the default network environment file.
    • The custom environment network configuration that includes resource references to your custom NIC templates.
    • Any additional environment files relevant to your configuration.

For example:

$ openstack overcloud deploy --templates \
    ...
    -n /home/stack/network_data.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/network-environment.yaml \
    -e /home/stack/templates/custom-network-configuration.yaml \
    ...
  • Include the network-isolation.yaml file first, then the network-environment.yaml file. The subsequent custom-network-configuration.yaml overrides the OS::TripleO::[ROLE]::Net::SoftwareConfig resources from the previous two files..
  • If using composable networks, include the network_data and roles_data files with this command.