What causes routing table 30400 to be created automatically in RHEL 8 on AWS which causes connectivity issues?

Root Cause

• When running a virtual machine in a public cloud environment, it is desirable to automatically configure the network of that VM. In simple setups, the VM only has one network interface and the public cloud supports automatic configuration via DHCP, DHCP6, or IPv6 Autoconf. However, the virtual machine might have multiple network interfaces, or multiple IP addresses and IP subnets on one interface which cannot be configured via DHCP. Also, the administrator may reconfigure the network while the machine is running. NetworkManager's nm-cloud-setup is a tool that automatically picks up such configuration in cloud environments and updates the network configuration of the host.

• nm-cloud-setup configures the network by fetching the configuration from the well-known metadata server of the cloud provider. That means, it already needs the network configured to the point where it can reach the metadata server. Commonly that means, that a simple connection profile is activated that possibly uses DHCP to get the primary IP address. NetworkManager will create such a profile for ethernet devices automatically if it is not configured otherwise via the "no-auto-default" setting in NetworkManager.conf. One possible alternative may be to create such an initial profile with nmcli device connect "$DEVICE" or nmcli connection add type ethernet.

• By setting the user-data org.freedesktop.nm-cloud-setup.skip=yes on the profile, nm-cloud-setup will skip the device.

• nm-cloud-setup modifies the run time configuration akin to nmcli device modify. With this approach, the configuration is not persisted and only preserved until the device disconnects. More information on nm-cloud-setup

Amazon EC2 (AWS):

For AWS, the tools try to fetch configuration from http://169.254.169.254/. Currently, it only configures IPv4 and does nothing about IPv6. It will do the following.

- First, fetch http://169.254.169.254/latest/meta-data/ to determine whether the expected API is present. This determines whether the EC2 environment is detected and whether to proceed to configure the host using EC2 metadata.

- Fetch http://169.254.169.254/2018-09-24/meta-data/network/interfaces/macs/ to get the list of available interfaces. Interfaces are identified by their MAC address.

- Then for each interface fetch http://169.254.169.254/2018-09-24/meta-data/network/interfaces/macs/$MAC/subnet-ipv4-cidr-block and http://169.254.169.254/2018-09-24/meta-data/network/interfaces/macs/$MAC/local-ipv4s. Thereby we get a list of local IPv4 addresses and one CIDR subnet block.

- Then nm-cloud-setup iterates over all interfaces for which it could fetch IP configuration. If no ethernet device for the respective MAC address is found, it is skipped. Also, if the device is currently not activated in NetworkManager or if the currently activated profile has a user-data org.freedesktop.nm-cloud-setup.skip=yes, it is skipped.

- Then, the tool will change the runtime configuration of the device.

- Add static IPv4 addresses for all the configured addresses from local-ipv4s with prefix length according to subnet-ipv4-cidr-block. For example, we might have here 2 IP addresses like "192.0.2.1/24,198.51.100.1/24".

- Choose a routeing table 30400 + the index of the interface and add a default route 0.0.0.0/0. The gateway is the first IP address in the CIDR subnet block. For example, we might get a route "0.0.0.0/0 192.0.2.1 10 table=30401".

- Finally, add a policy routing rule for each address. For example "priority 30401 from 192.0.2.1/32 table 30401, priority 30401 from 192.0.2.1/32 table 30401".

With above example, this roughly corresponds for interface eth0 to nmcli device modify "eth0" ipv4.addresses "192.0.2.1/24,192.0.2.1/24" ipv4.routes "0.0.0.0/0 192.0.2.1 10 table=30401" ipv4.routing-rules "priority 30401 from 192.0.2.1/32 table 30401, priority 30401 from 192.0.2.1/32 table 30401". Note that this replaces the previous addresses, routes, and rules with the new information. But also note that this only changes the run time configuration of the device. The connection profile on the disk is not affected.