vEPC components such as SGW and PGW may be placed either centrally at the core datacenter (National Datacenter) or regionally to serve the local cell sites or exit points. The eNodeBs are collocated with the tower (Cell Site). In a 4G/LTE network, deciding which SGW (Ingress into the network) to use is typically based on subscriber location (Tracking Area Code). The subscriber connection terminates on the PGW. The decision of which PGW to assign to the subscriber depends on the context (APN) and other factors. The actual assignment may be done based on DNS lookup. This is shown in Figure 7.

Because of this reason, the SGW and the PGWs could be placed in regional datacenters closer to the service they need to provide. This approach is taken by many operators. However, in many deployments, the gateways, MME and other functions may be collocated in a centralized national data-center. GiLAN and IMS services are typically deployed only in the core of the mobile network.

Regardless of which way it is deployed, it is important to have redundancy built in at a datacenter level. Additionally geo-redundancy should also be factored in so that similar functions can be leveraged from other datacenters in case of a catastrophic failure at one of the datacenters. It should be noted that apart from using traditional routing to reroute traffic to the gateways/elements that became active due to the failure of the primary, SDN and Software Defined Wide Area Network (SD-WAN) technology could be employed to reprogram the underlying layer 2 and layer 3 networks to provide the appropriate level of bandwidth and functionality needed.

Figure 7: Datacenter redundancy and deployment of vEPC

For lightweight All In One (AIO) deployments (shown in Figure 8, common datacenter design practice should be followed. Don’t place active and standby VNFs on the same chassis/host. Follow datacenter and cloud deployment best practices:

Figure 8: Lightweight, AIO VPC deployment

Multi-server deployments regardless of whether they run on top of KVM or a full fledged OpenStack deployed in HA mode should follow the above mentioned Chassis/Blade redundancy best practices. In addition, HA best practices should be followed at the application layer. Typically, VPC VNFs have dedicated VMs acting as Control Modules which handles control traffic and the data traffic is typically handled by switching or Service Modules. Depending on whether a single chassis is being deployed or multiple chassis are being deployed, it will be important to place the control modules and service modules to allow for high availability and redundancy.

Figure 9: Single Chassis deployment of control and switching modules.

Figure 9 shows deployment of VPC VNF on a single blade server chassis. Multiple blade server will distribute the control modules (CMs) across the chassis so that each chassis to mitigate failure. The number of control and switching modules and their deployment best practices will come from the VNF vendors. This can be seen in Figure 10.

Figure 10: Multi-chassis VNF HA Deployment

GiLAN elements that form logical service chains (SFC), can be deployed the following manner:

IMS elements will be similar to GiLAN in the sense that they are made up of some discrete elements required to perform the various functions. Care has to be taken to ensure that hardware or network failure does not isolate functions. Basic principles of HA and best practices apply to for deployment of all virtual mobile network VNFs.

Datacenter topologies have changed and evolved over time. Prior approach was three layered and consisted of Access, Aggregation(Distribution) and Core layers (Figure 11.) Access layers typically consisted of layer 2 switches where the servers connected. These switches connected into layer 2/layer 3 switches which were then connected up to pure layer 3 routers that formed the core.

Figure 11: Traditional three-tier datacenter topology

This approach served as the classic datacenter topology for many years. However, it had limitations and complexity typically from running Spanning Tree Protocol (STP) which offers protection against layer 2 loops but could leave the network with half the capacity due to blocked ports, not to mention complexity of troubleshooting STP! Today’s datacenter has been replaced by what is referred to as leaf-spine architecture. In leaf-spine architecture, servers are dual-homed to two leaf switches. The leaf as well as spine switches are not connected to each other but form a full mesh between each other as shown in Figure 12.

Figure 12: Leaf and spine datacenter topology

Leaf-spine can be either layer 2 or layer 3. In a layer 2 approach, all connections are active/active. In leaf-spine architecture, Transparent Interconnection of Lots of Links (TrILL) or Shortest Path Bridging (SPB) are used instead of STP. However, leaf-spine architecture with layer 3 has the advantage of simplicity. In such a layer-3-only network, hosts may run a routing agent that will announce /32 to the leafs. Equal Cost MultiPath (ECMP) is used for evenly distributing traffic across links. It is possible to use Open Shortest Path First (OSPF) or Border Gateway Protocol (BGP) to distribute routes across the datacenter fabric. BGP is becoming a favorite with new enhancements such as BGP unnumbered which makes configuration a breeze. BGP is also proven to be highly scalable and reliable as it has been carrying Internet Routes for decades now. It should be noted however, that OpenStack networking is typically layer 2 based (VLAN, VxLAN). Most OpenStack deployments including for NFV/virtual mobile networks are layer 2 based. Red Hat OpenStack Platform uses the Neutron networking service to create and deploy tenant networks. It should be noted that routed (layer 3) leaf-spine deployments are not currently supported by OpenStack Platform director. Director installations have the following pre-defined networks (https://access.redhat.com/documentation/en/red-hat-openstack-platform/10/paged/director-installation-and-usage/chapter-3-planning-your-overcloud):

Each of these networks will be mapped to IP subnets and typically VLANs. Details of networking for vEPC is discussed under the networking section.

For the layer 2 based design, it is recommended when possible to use link bundling using Link Aggregation Control Protocol (LACP) on the servers and using Multi-chassis Link Aggregation (MLAG) on the switches. In this scheme, two (or more) NICs are bonded (using Linux LACP bonding). One of the NICs is connected to one leaf switch and the other NIC to a different leaf switch. The two switches will run MLAG between each other. This will ensure that failure of a single switch will not cause the server to be isolated. Figure 13 shows how this is connected

Figure 13: High availability server connection

NIC 0 on all servers is or Intelligent Platform Management Interface (IPMI) that is used to configure the servers and to console into them. NIC1 is for Out of band management (OOB). That is used to connect to the servers directly over IP.

NICs 3 & 4 of the server carry Internal traffic including storage, internal API, etc. They are bonded. NIC 3 connects up to the switch on the top, whereas NIC 4 connects to the bottom switch. The two switches run MLAG. The bonded NIC will trunk all the VLANs for internal traffic. Similarly NICs 5 & 6 may be bonded to carry external traffic (accessible to the outside world). It should be noted however, that current deployments of vEPC use Single Root Input Output Virtualization (SR-IOV) for datapath (data plane) and in order to use SR-IOV, bonding should not be done at the host level. NIC bonding at guest VM/VNF level is however supported for SR-IOV and can be be used by VNFs to mitigate risk of uplink switch failure.

Network Equipment Providers (NEPs) are looking to move away from SR-IOV based vEPC solution in the future and are looking to use Open Virtual Switch (OVS) + Data Plane Development Kit (DPDK) as their fast datapath. NIC bonding at the host level will be supported for OVS+DPDK thus allowing NICs 5 & 6 to be bonded as well.

It should be noted that if another pair of NICs is available, it is a good idea to split off the storage network to have its own bond if we are using a CEPH cluster as our storage backend. If we are using external/cloud storage this is not required.

Red Hat OpenStack Platform offers several features to support HA. These features are supported and can be deployed using Red Hat OpenStack director. Since OpenStack control plane and services mostly reside on the controller nodes, HA features are targeted towards the controller nodes. OpenStack HA features that are employed for HA are:

OpenStack HA may be deployed in one of two modes:

More details on Red Hat OpenStack Platform HA can be obtained in the document “UNDERSTANDING RED HAT OPENSTACK PLATFORM HIGH AVAILABILITY” which is available on the Red Hat customer portal at http://access.redhat.com. Additionally, it is important to note that with Red Hat OpenStack Platform 10 release (Newton), another HA component is introduced known as “systemd”. Some services will be managed by Pacemaker whereas others will be managed by systemd. Also there will be common controllers vs. dedicated controllers. These will be discussed in release-specific addendums to this architecture document where release-specific features and their configurations will be covered.

vEPC and virtual mobile NEPs in general expect the VIM (OpenStack) to be running in full HA mode as vEPC could be running mission critical services such as 911 and typically hosts millions of subscribers per instance.

NEPs have had resilience and high availability in mind through design and deployment of virtualized mobile networks (vEPC, GiLAN and IMS). The purpose built hardware had HA features built into the products from the get-go all the way from the backplane design, deployment of control modules and service modules (data plane processors) to port redundancy to in service software upgrades. With the advent of virtualized solutions (NFV), the control and service modules became VMs that use the “Internal Networks” created in OpenStack to communicate with each other. However, the application logic that existed to manage availability and detect failure of service modules continue to exist in the virtualized solution as well. Control modules typically keep tab of the deployed and available service modules, use some load-balancing criteria to distribute traffic and assign service modules to subscriber flows. N+1 or N+N redundancy depending on design makes certain service modules active while one or more service modules may serve as standby, thus allowing these standby modules to take over upon failure of an active service module. Since typically these applications use some sort of heart-beat mechanism to keep track of the service module liveliness, whether they run in a virtual environment or not becomes immaterial. Additionally, session recovery between physical chassis will continue to be supported in the virtual environment. For this typically, some control plane protocol is used to determine active and standby nodes, failure detection and state change from standby to active based on some failure condition.