9.4. Hosts and Networking

9.4.1. Network Manager Stateful Configuration (nmstate)

Version 4.4 of Red Hat Virtualization (RHV) uses Network Manager Stateful Configuration (nmstate) to configure networking for RHV hosts that are based on RHEL 8. RHV version 4.3 and earlier use interface configuration (ifcfg) network scripts to manage host networking.

To use nmstate, upgrade the Red Hat Virtualization Manager and hosts as described in the RHV Upgrade Guide.

As an administrator, you do not need to install or configure nmstate. It is enabled by default and runs in the background.

Important

Always use RHV Manager to modify the network configuration of hosts in your clusters. Otherwise, you might create an unsupported configuration.

The change to nmstate is nearly transparent. It only changes how you configure host networking in the following ways:

  • After you add a host to a cluster, always use the RHV Manager to modify the host network.
  • Modifying the host network without using the Manager can create an unsupported configuration.
  • To fix an unsupported configuration, you replace it with a supported one by using the Manager to synchronize the host network. For details, see Synchronizing Host Networks.
  • The only situation where you modify host networks outside the Manager is to configure a static route on a host. For more details, see Adding a static route on a host.

The change to nmstate improves how RHV Manager applies configuration changes you make in Cockpit and Anaconda before adding the host to the Manager. This fixes some issues, such as BZ#1680970 Static IPv6 Address is lost on host deploy if NM manages the interface.

Important

If you use dnf or yum to manually update the nmstate package, restart vdsmd and supervdsmd on the host. For example:

+

# dnf update nmstate
# systemctl restart vdsmd supervdsmd
Important

If you use dnf or yum to manually update the Network Manager package, restart NetworkManager on the host. For example:

+

# dnf update NetworkManager
# systemctl restart NetworkManager

9.4.2. Refreshing Host Capabilities

When a network interface card is added to a host, the capabilities of the host must be refreshed to display that network interface card in the Manager.

Procedure

  1. Click ComputeHosts and select a host.
  2. Click ManagementRefresh Capabilities.

The list of network interface cards in the Network Interfaces tab for the selected host is updated. Any new network interface cards can now be used in the Manager.

9.4.3. Editing Host Network Interfaces and Assigning Logical Networks to Hosts

You can change the settings of physical host network interfaces, move the management network from one physical host network interface to another, and assign logical networks to physical host network interfaces. Bridge and ethtool custom properties are also supported.

Warning

The only way to change the IP address of a host in Red Hat Virtualization is to remove the host and then to add it again.

To change the VLAN settings of a host, see Section 9.4.5, “Editing a Host’s VLAN Settings”.

Important

You cannot assign logical networks offered by external providers to physical host network interfaces; such networks are dynamically assigned to hosts as they are required by virtual machines.

Note

If the switch has been configured to provide Link Layer Discovery Protocol (LLDP) information, you can hover your cursor over a physical network interface to view the switch port’s current configuration. This can help to prevent incorrect configuration. Check the following information prior to assigning logical networks:

  • Port Description (TLV type 4) and System Name (TLV type 5) help to detect to which ports and on which switch the host’s interfaces are patched.
  • Port VLAN ID shows the native VLAN ID configured on the switch port for untagged ethernet frames. All VLANs configured on the switch port are shown as VLAN Name and VLAN ID combinations.

Procedure

  1. Click ComputeHosts.
  2. Click the host’s name. This opens the details view.
  3. Click the Network Interfaces tab.
  4. Click Setup Host Networks.
  5. Optionally, hover your cursor over host network interface to view configuration information provided by the switch.
  6. Attach a logical network to a physical host network interface by selecting and dragging the logical network into the Assigned Logical Networks area next to the physical host network interface.

    Note

    If a NIC is connected to more than one logical network, only one of the networks can be non-VLAN. All the other logical networks must be unique VLANs.

  7. Configure the logical network:

    1. Hover your cursor over an assigned logical network and click the pencil icon. This opens the Edit Management Network window.
    2. From the IPv4 tab, select a Boot Protocol from None, DHCP, or Static. If you selected Static, enter the IP, Netmask / Routing Prefix, and the Gateway.

      Note

      For IPv6, only static IPv6 addressing is supported. To configure the logical network, select the IPv6 tab and make the following entries:

      • Set Boot Protocol to Static.
      • For Routing Prefix, enter the length of the prefix using a forward slash and decimals. For example: /48
      • IP: The complete IPv6 address of the host network interface. For example: 2001:db8::1:0:0:6
      • Gateway: The source router’s IPv6 address. For example: 2001:db8::1:0:0:1
      Note

      If you change the host’s management network IP address, you must reinstall the host for the new IP address to be configured.

      Each logical network can have a separate gateway defined from the management network gateway. This ensures traffic that arrives on the logical network will be forwarded using the logical network’s gateway instead of the default gateway used by the management network.

      Important

      Set all hosts in a cluster to use the same IP stack for their management network; either IPv4 or IPv6 only. Dual stack is not supported.

    3. Use the QoS tab to override the default host network quality of service. Select Override QoS and enter the desired values in the following fields:

      • Weighted Share: Signifies how much of the logical link’s capacity a specific network should be allocated, relative to the other networks attached to the same logical link. The exact share depends on the sum of shares of all networks on that link. By default this is a number in the range 1-100.
      • Rate Limit [Mbps]: The maximum bandwidth to be used by a network.
      • Committed Rate [Mbps]: The minimum bandwidth required by a network. The Committed Rate requested is not guaranteed and will vary depending on the network infrastructure and the Committed Rate requested by other networks on the same logical link.
    4. To configure a network bridge, click the Custom Properties tab and select bridge_opts from the drop-down list. Enter a valid key and value with the following syntax: key=value. Separate multiple entries with a whitespace character. The following keys are valid, with the values provided as examples. For more information on these parameters, see Section B.1, “Explanation of bridge_opts Parameters”.

      forward_delay=1500
      group_addr=1:80:c2:0:0:0
      group_fwd_mask=0x0
      hash_max=512
      hello_time=200
      max_age=2000
      multicast_last_member_count=2
      multicast_last_member_interval=100
      multicast_membership_interval=26000
      multicast_querier=0
      multicast_querier_interval=25500
      multicast_query_interval=13000
      multicast_query_response_interval=1000
      multicast_query_use_ifaddr=0
      multicast_router=1
      multicast_snooping=1
      multicast_startup_query_count=2
      multicast_startup_query_interval=3125
    5. To configure ethernet properties, click the Custom Properties tab and select ethtool_opts from the drop-down list. Enter a valid value using the format of the command-line arguments of ethtool. For example:

      --coalesce em1 rx-usecs 14 sample-interval 3 --offload em2 rx on lro on tso off --change em1 speed 1000 duplex half

      This field can accept wildcards. For example, to apply the same option to all of this network’s interfaces, use:

      --coalesce * rx-usecs 14 sample-interval 3

      The ethtool_opts option is not available by default; you need to add it using the engine configuration tool. See Section B.2, “How to Set Up Red Hat Virtualization Manager to Use Ethtool” for more information. For more information on ethtool properties, see the manual page by typing man ethtool in the command line.

    6. To configure Fibre Channel over Ethernet (FCoE), click the Custom Properties tab and select fcoe from the drop-down list. Enter a valid key and value with the following syntax: key=value. At least enable=yes is required. You can also add dcb= and auto_vlan=[yes|no]. Separate multiple entries with a whitespace character. The fcoe option is not available by default; you need to add it using the engine configuration tool. See Section B.3, “How to Set Up Red Hat Virtualization Manager to Use FCoE” for more information.

      Note

      A separate, dedicated logical network is recommended for use with FCoE.

    7. To change the default network used by the host from the management network (ovirtmgmt) to a non-management network, configure the non-management network’s default route. See Section 9.1.5, “Configuring a Non-Management Logical Network as the Default Route” for more information.
    8. If your logical network definition is not synchronized with the network configuration on the host, select the Sync network check box. For more information about unsynchronized hosts and how to synchronize them, see Section 9.4.4, “Synchronizing Host Networks”.
  8. Select the Verify connectivity between Host and Engine check box to check network connectivity. This action only works if the host is in maintenance mode.
  9. Click OK.
Note

If not all network interface cards for the host are displayed, click ManagementRefresh Capabilities to update the list of network interface cards available for that host.

Troubleshooting

In some cases, making multiple concurrent changes to a host network configuration using the Setup Host Networks window or setupNetwork command fails with an Operation failed: [Cannot setup Networks]. Another Setup Networks or Host Refresh process in progress on the host. Please try later.] error in the event log. This error indicates that some of the changes were not configured on the host. This happens because, to preserve the integrity of the configuration state, only a single setup network command can be processed at a time. Other concurrent configuration commands are queued for up to a default timeout of 20 seconds. To help prevent the above failure from happening, use the engine-config command to increase the timeout period of SetupNetworksWaitTimeoutSeconds beyond 20 seconds. For example:

# engine-config --set SetupNetworksWaitTimeoutSeconds=40

9.4.4. Synchronizing Host Networks

The Manager defines a network interface as out-of-sync when the definition of the interface on the host differs from the definitions stored by the Manager.

Out-of-sync networks appear with an Out-of-sync icon out of sync in the host’s Network Interfaces tab and with this icon out of sync setup in the Setup Host Networks window.

When a host’s network is out of sync, the only activities that you can perform on the unsynchronized network in the Setup Host Networks window are detaching the logical network from the network interface or synchronizing the network.

Understanding How a Host Becomes out-of-sync

A host will become out of sync if:

  • You make configuration changes on the host rather than using the the Edit Logical Networks window, for example:

    • Changing the VLAN identifier on the physical host.
    • Changing the Custom MTU on the physical host.
  • You move a host to a different data center with the same network name, but with different values/parameters.
  • You change a network’s VM Network property by manually removing the bridge from the host.
Important

If you change the network’s MTU settings, you must propagate this change to the running virtual machines on the network: Hot unplug and replug every virtual machine’s vNIC that should apply the MTU setting, or restart the virtual machines. Otherwise, these interfaces fail when the virtual machine migrates to another host. For more information, see https://access.redhat.com/solutions/4540631 and BZ#1766414.

Preventing Hosts from Becoming Unsynchronized

Following these best practices will prevent your host from becoming unsynchronized:

  1. Use the Administration Portal to make changes rather than making changes locally on the host.
  2. Edit VLAN settings according to the instructions in Section 9.4.5, “Editing a Host’s VLAN Settings”.

Synchronizing Hosts

Synchronizing a host’s network interface definitions involves using the definitions from the Manager and applying them to the host. If these are not the definitions that you require, after synchronizing your hosts update their definitions from the Administration Portal. You can synchronize a host’s networks on three levels:

  • Per logical network
  • Per host
  • Per cluster

Synchronizing Host Networks on the Logical Network Level

  1. Click ComputeHosts.
  2. Click the host’s name. This opens the details view.
  3. Click the Network Interfaces tab.
  4. Click Setup Host Networks.
  5. Hover your cursor over the unsynchronized network and click the pencil icon. This opens the Edit Network window.
  6. Select the Sync network check box.
  7. Click OK to save the network change.
  8. Click OK to close the Setup Host Networks window.

Synchronizing a Host’s Networks on the Host level

  • Click the Sync All Networks button in the host’s Network Interfaces tab to synchronize all of the host’s unsynchronized network interfaces.

Synchronizing a Host’s Networks on the Cluster level

  • Click the Sync All Networks button in the cluster’s Logical Networks tab to synchronize all unsynchronized logical network definitions for the entire cluster.
Note

You can also synchronize a host’s networks via the REST API. See syncallnetworks in the REST API Guide.

9.4.5. Editing a Host’s VLAN Settings

To change the VLAN settings of a host, the host must be removed from the Manager, reconfigured, and re-added to the Manager.

To keep networking synchronized, do the following:

  1. Put the host in maintenance mode.
  2. Manually remove the management network from the host. This will make the host reachable over the new VLAN.
  3. Add the host to the cluster. Virtual machines that are not connected directly to the management network can be migrated between hosts safely.

The following warning message appears when the VLAN ID of the management network is changed:

Changing certain properties (e.g. VLAN, MTU) of the management network could lead to loss of connectivity to hosts in the data center, if its underlying network infrastructure isn't configured to accommodate the changes. Are you sure you want to proceed?

Proceeding causes all of the hosts in the data center to lose connectivity to the Manager and causes the migration of hosts to the new management network to fail. The management network will be reported as "out-of-sync".

Important

If you change the management network’s VLAN ID, you must reinstall the host to apply the new VLAN ID.

9.4.6. Adding Multiple VLANs to a Single Network Interface Using Logical Networks

Multiple VLANs can be added to a single network interface to separate traffic on the one host.

Important

You must have created more than one logical network, all with the Enable VLAN tagging check box selected in the New Logical Network or Edit Logical Network windows.

Adding Multiple VLANs to a Network Interface using Logical Networks

  1. Click ComputeHosts.
  2. Click the host’s name. This opens the details view.
  3. Click the Network Interfaces tab.
  4. Click Setup Host Networks.
  5. Drag your VLAN-tagged logical networks into the Assigned Logical Networks area next to the physical network interface. The physical network interface can have multiple logical networks assigned due to the VLAN tagging.
  6. Edit the logical networks:

    1. Hover your cursor over an assigned logical network and click the pencil icon.
    2. If your logical network definition is not synchronized with the network configuration on the host, select the Sync network check box.
    3. Select a Boot Protocol:

      • None
      • DHCP
      • Static
    4. Provide the IP and Subnet Mask.
    5. Click OK.
  7. Select the Verify connectivity between Host and Engine check box to run a network check; this will only work if the host is in maintenance mode.
  8. Click OK.

Add the logical network to each host in the cluster by editing a NIC on each host in the cluster. After this is done, the network will become operational.

This process can be repeated multiple times, selecting and editing the same network interface each time on each host to add logical networks with different VLAN tags to a single network interface.