RHEV: VMs lost network connectivity after RHEL Hypervisor upgraded to 6.7

Issue

VMs lost network connectivity when migrated to RHEL hypervisors running kernel-2.6.32-573.el6 or later.
Migrating VM back to a not upgraded hypervisor simply made it accessible again.
DHCP replies/offers are not forwarded to KVM Guest.
PXE boot in the KVM guest times out.
Hypervisor bridge not forwarding ARP requests to virtual machine vnetX.
Virtual Machines can only ping other Virtual Machines running in the same Host.
Hosted-Engine can only ping the host it is running on, cannot connect to other Hosts.
Server cannot communicated unless NIC is in promiscuous mode

Cisco UCS
Red Hat Virtualization 3.4, 3.5, 3.6 and 4.x
Red Hat Enterprise Linux 6.7, 6.8 based Hypervisor
- kernel-2.6.32-573.el6 (and higher)
Red Hat Enterprise Linux 7.x based Hypervisor
- kernel-3.10.0-123.el7 (and higher)
  - driver: enic version 2.1.1.67

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.