ssh protocol error between subnets in same tenant when CIPSO is enabled in the guests

Issue

• We are running PitBull Linux in a single tenant. PitBull is based on RHEL with additional security lock downs including network labeling using CIPSO.

• Our configuration looks good. Our physical switches and nics are all configured with mtu sizes of 9000. We are using OVS bridges with bonding. On the tenant, we have 2 networks that are connected to the same router. That router also connects to the external network. The mtu sizes on the private tenant networks are defaulting to 1450 as well as the tenant instances when we spin them up.

• The issue we are having is that when we ssh from instance A on network A to instance B on network B, with CIPSO turned on inside the instance VM's, we get a protocol error. We are not seeing any additional information with verbose turned on in ssh. We also do not see traffic even hitting the other instance if we run a packet capture on the destination host. Almost like it dies on the src side. Either on outgoing or incoming gateway on the private nets.

• However, if we turn CIPSO off on both sides, ssh works as expected. Also running straight RHEL between subnets works as well with no issues.

• Our initial thought is that there may be a mtu sizing issue with the additional data being added to packets with CIPSO turned on. We tried to modify mtu on both private networks and instances, both higher, 1600, as well as lower, 1200-1442. Neither worked. Our PitBull partner recommended setting both the private networks and instances mtu to 1442. It also did not work.

• We were hoping Red Hat might have some additional insight that might help. Here are some things we are not sure of:

1. Could the physical nic hardware be an issue in that it could not support CIPSO packets?
2. We tried to change the mtu size on the physical switches by running the following.  We were not sure that it took.  Interesting if we tried to bump it to 9000, the network took it, but not the instance.
    openstack network set --mtu 1500 <net ID>
3. Can you point us to any network troubleshooting docs or notes that will allow us to packet capture from the controler/compute layer in that we can connect to the network namespace or the private gateways or even the instances.