LISP Deployment Considerations in Data Center Networks

Data Center Networks require support for segmentation, mobility and scale as described in . These requirements can be addressed with the use of overlay networks. The LISP control plane can be used for the creation of network overlays that support mobility and segmentation at large scale in Layer 2, Layer 3 and combined Layer2/Layer3 overlays as described in and . The needs for overlay provided services are typically different within and across Data Centers versus the needs for Data Center Wide Area Network connectivity. LISP is relevant in both of these areas. The use of LISP as a control protocol for the creation of overlays can be optimized for the specific topologies that are relevant in the context of Data Center Networks within and across Data Centers. This document discusses different deployment options for LISP in the context of different data center topologies, the implications of the use of the different deployment options are part of the discussion. Data Center Networks include an intra-Data-Center fabric topology as well as inter-Data-Center and Wide Area Network components. The topologies found inside the Data Center Network fabric are designed in a deterministic manner with a very high degree of symmetry. This high degree of symmetry assures that a multitude of paths exist between any two end-points in the data center and that these paths are of equal cost and deterministic latency. As a result of such designs, traffic patterns within or across the data center are guaranteed to traverse specific tiers of the network. A reference Data Center Network inclusive of an intra-Data-Center fabric topology as well as inter-Data-Center and Wide Area Network components is described in . The predictability of the different possible data paths in the Data Center Network opens opportunities for optimizations in the deployment of demand based control plane models such as LISP. In the specific case of LISP, some of the interesting deployment options are centered around the topological location of the mapping-system and xTRs, equally interesting is the option of separating the LISP control-plane from the encapsulation/decapsulation functions in an xTR. Different deployment scenarios are discussed in and .

Proxy Reply Mode: Map Servers may reply to a map-request directly, rather than forwarding the map-request to the registering ETR Fabric Manager: Orchestration tool in charge of provisioning and monitoring Fabric Network nodes. WAN Manager: Orchestration tool in charge of provisioning and monitoring WAN/DCI Network nodes. For definition of NVO3 related terms, notably Virtual Network (VN), Virtual Network Identifier (VNI), Network Virtualization Edge (NVE), Data Center (DC), please consult . For definitions of LISP related terms, notably Map-Request, Map-Reply, Ingress Tunnel Router (ITR), Egress Tunnel Router (ETR), Map-Server (MS) and Map-Resolver (MR) please consult the LISP specification .

The reference topology that will be used for our discussion is a folded clos topology. The folded clos is considered to be the prevalent topology in large-scale data centers requiring the use of overlays. Although other topologies may be utilized within the data center, most of such topologies may be modeled as a folded clos or collection of clos topologies. The reference clos topology is illustrated in . The connectivity depicted in the diagram is not exhaustive; every Leaf node has at least one connection to every Spine node, effectively providing at least N equal cost paths between any two Leaf nodes, where N is the number of spine nodes.

The clos topology is a multi-stage topology. In the folded clos instantiation, there is an ingress stage, a middle stage and an egress stage. The elements of the topology are described as: Spine Nodes: Nodes that make the middle stage of the clos. In the absence of failures, each Spine Node is connected to every Leaf Node. Leaf Nodes: Nodes that make the ingress and egress stage of the clos. Each Leaf Node is connected to every Spine Node. In the case of a link or Spine Node failure, a Leaf Node will remain connected to the surviving Spine Nodes over the remaining healthy links. The fabric will continue to operate under such condition of partial connectivity. vLeaf Nodes: Virtual Leaf Nodes are those network nodes that connect to the Leaf Nodes. vLeaf Nodes are usually in the form of virtual switches/routers running on the physical compute end-points hosts. Many vLeaf nodes may connect to a single Leaf node. Likewise, a vLeaf node may connect to multiple Leaf nodes for redundancy purposes. WAN Routers: Routers attached to leaf nodes in the clos topology to provide connectivity outside the clos over a WAN or Data Center Interconnect (DCI). WAN Routers are usually deployed in pairs for resiliency and they are also usually attached to at least two Leaf nodes in a redundant topology. WAN routers participate in a Wide Area Network (WAN) or Data Center Interconnect (DCI). bLeaf: Border Leaf nodes are Leaf nodes that attach to the WAN routers. WAN: The Wide Area Network is the network over which client sites as well as Disaster Recovery sites connect to the data center. DCI: Data Center Interconnect is a Metropolitan Area Network over which multiple Data Centers connect to each other. There is an assumption of bound latency over the DCI. end-points: Hosts that connect to the Data Center Fabric. End-points can be physical (P) or virtual (V) V: Virtual end-points such as Virtual Machines. P: Physical end-points such as non-virtualized servers.

The main LISP functions to be deployed in the Data Center fabric topology are: xTRs Map-Servers Map-Resolvers The Map-Server and Map-Resolver functions will generally be co-located on the same network nodes, we will refer to the co-located function as Map-Server/Resolver.

LISP xTRs perform both Data Plane as well as Control Plane tasks. From the Data Plane perspective the xTRs are tasked with encapsulating (ITRs) or decapsulating (ETRs) traffic. From the Control Plane perspective the tasks of registering mappings and requesting/caching mappings are handled by the ETRs and ITRs respectively. It is possible to split the roles of ETR and ITR to different network nodes. The most prevalent deployment model combines the roles of ETR and ITR onto a single device. We will focus our discussion on the combined ETR/ITR model. A device that serves as both ETR and ITR is generally referred to as an xTR. It is also possible to split the Data Plane and Control Plane responsibilities and distribute them across different network nodes. Some of the deployment options enabled by this separation are discussed in the following sections:

The full xTR function including both Control and Data Plane can be deployed at the vLeaf nodes. The entire clos including Leaf nodes and Spine nodes simply serves as an underlay transport to the LISP overlay. The advantage of distributing the xTR role to the vLeaf is mainly that the map-cache state at each xTR is minimized by virtue of it being distributed to a larger number of nodes. Amongst the challenges are: Operational implications of managing a larger number of overlay end-points. Large number of xTRs sending requests and registrations to the mapping system has implications on the scale requirements and limits of the mapping system (servers and resolvers) vLeaves do not service non-virtualized Physical end-points

The control and data plane tasks can be separated and distributed amongst Leaf and vLeaf. In this approach the vLeaf can retain the Data Plane encap/decap function while the LISP signaling is done by the Leaf. Thus, the Leaf will proxy the signaling for the vLeaf nodes connected to it as follows Map-register messages will be issued by the Leaf with the corresponding RLOC addresses of the vLeaf nodes. Map-request messages will be sent by the ITR vLeaf to its upstream Leaf and then relayed by the Leaf to the Map-resolver on behalf of the vLeaf. The mapping system, when not in proxy reply mode, will forward the map-request directly to the RLOC of the vLeaf node with which the EID being requested was registered. Map-reply messages will be sent to the requesting Leaf and then relayed to the corresponding vLeaf where the mapping is cached. This scheme allows the caching of state to remain fully distributed and also enables more granular distribution of the mappings amongst a larger number of ETRs. The scheme also allows the consolidation of the signaling at the Leaf nodes where map-registers and map-requests can be batched. The scheme does however present the challenge of maintaining mappings in the mapping system for an increased number of xTRs since it will track the larger number of xTRs at the vLeaf level rather than the smaller number of xTRs at the Leaf level, this can be alleviated by fully proxying the vLeaf LISP functions at the Leaf as discussed in .

Both the data plane and control plane xTR function may reside on the Leaf nodes. This deployment model leverages the LISP control plane without any changes and enables interworking of different hypervisor overlays across a normalized LISP fabric. vLeaf nodes may encapsulate traffic into the Fabric in different ways (802.1Q, VXLAN, NVGRE, etc) depending on the hypervisor they may use. It is the role of the Leaf node, in this model, to terminate any encapsulation being used by the vLeaf nodes, extract any metadata information, such as the VNID [I-D.ietf-nvo3-framework], from the headers imposed at the vLeaf nodes and normalize the different encapsulations received from different vLeaf nodes to a common LISP encapsulation amongst Leaf nodes. The LISP encapsulation at the Leaf nodes will encode the VNIDs extracted from the encapsulated vLeaf traffic as instance-ids in the LISP overlay. The mapping of the VNIDs imposed by the vLeaf nodes to LISP instance-ids is provisioned at each Leaf/xTR node either manually or by an automated provisioning system linked to an orchestration system with visibility into the different hypervisor overlays and the LISP overlay. The ability to normalize different hypervisor overlays is predicated on the use and exchange of Universally Unique Identifiers (UUIDs) as the constant parameter identifying a common segment across different overlay administrative domains that may use a variety of VNIDs to encapsulate traffic for the same segment. The assignment and administration of tenant UUIDs is an important orchestration task and outside the scope of this document.

Within the data center, we will assume that the functions of map-server and map-resolver are co-located on the same nodes. We will refer to such nodes as Map-Server/Resolver. Resiliency is important in the deployment of Map-Server/Resolvers for the Data Center. Multiple Map-Server/Resolvers must be deployed for redundancy. Depending on the topological location of the Map-Server/Resolver nodes, there may be different deployment requirements to achieve appropriate reachability to the multiple Map-Server/Resolvers involved.

The Map-Server/Resolver function can be deployed out of band on one or more network nodes that are either reachable in the WAN or attached to a leaf as end-points but are not part of the clos topology. Out of band Map-Server/Resolvers will only be exposed to control plane traffic. Clusters of Map-Server/Resolver nodes can be deployed by having ETRs register to multiple Map-Server addresses and ITRs issue map-requests to an anycast address that is shared by the map-resolver function of the Map-Server/Resolver nodes in the cluster. Thus, the map-resolver function can use a common anycast IP address across all Map-Server/Resolver nodes, separate unicast IP addresses will be assigned the map-server component in each node. ETRs registering to the map-servers should be configured to send map-registers to all map-server IP addresses. ITRs should issue map-requests to the shared anycast IP address of the map-resolver component of the cluster. Alternatively, clusters of Map-Server/Resolver nodes can be deployed by having ETRs register to one Map-Server and use an alternate mechanism to allow the synchronization of this registration across all Map-Servers in the cluster. In this case ETRs will be configured to register to a single IP address, this would likely be a shared anycast IP address amongst the Map-Servers in order to ensure resiliency. ITRs will continue to issue requests to a Map-Resolver shared anycast IP. The creation of Map-Server/Resolver resiliency clusters based on IP addressing is a topologically agnostic deployment model which fits the out of band model well and gives a high degree of flexibility for the deployment of the necessary mapping system nodes.

The Map-Server/Resolver function can be deployed in-line on one or more network nodes that participate in the clos topology either as Spine nodes or Leaf nodes. In-line Map-Server/Resolvers will receive both Control Plane and Data Plane traffic.

The Map-Server/Resolver function can be deployed on Spine nodes. In order to guarantee resiliency, Map-Server/Resolvers are deployed on at least two Spine nodes, but preferably on all Spine nodes. Any map-register messages must be sent to all Map-Server/Resolver enabled Spine nodes in order to ensure synchronization of the mapping system. Therefore in a system where all Spine nodes host the Map-Server/Resolver functionality, all Spine nodes will have complete mapping state for all EIDs in the fabric. Alternatively, map-register messages can be sent to a single Map-Server and the state may be relayed by other methods to other Map-Servers in the Spine. When all Spine nodes host the Map-Server/Resolver functionality, all data plane traffic that cuts across different leaf nodes will traverse a Spine node that has the full LISP mapping state for the fabric. In this deterministic topology it is possible to implement avoid transient drops that may occur when looking up destinations that have not been previously cached at the ITRs. In order to avoid transient drops during Mapping System lookups, ITRs (Leaf or vLeaf nodes) without a pre-existing cache entry for a particular destination must encapsulate and forward the traffic with the unknown destination to the Spine. Since the Spine has full location information, it knows which ETR to send the traffic to, so it encapsulates and forwards the traffic accordingly. Simultaneously, the ITR may send a map-request to the anycast address for the map-resolvers that are present across the spine. The ITR will continue to send the traffic for the unknown destination towards the Spine until a mapping for the destination is received in a map-reply and cached at the ITR, at which point the ITR will start encapsulating traffic directly to the appropriate ETR. In order to forward traffic for unknown destinations to the Spine, the ITR must LISP encapsulate the unknown destination traffic towards the anycast IP address configured for the map-resolver function on the Spine nodes or it may hash the traffic to select a discrete RLOC for a specific Spine node. It is important that traffic be LISP encapsulated in order to preserve the semantics of instance-id and other parameters that may be encoded in the LISP header. An alternative implementation could leverage the data plane traffic in order to reduce the amount of control plane messages that are exchanged. For instance, when traffic with unknown destinations is sent to the spine, rather than waiting to receive a map-request for the unknown destination, the spine could react to the reception of the unknown destination data plane traffic by sending a map-reply to the source ITR with the mappings for the corresponding unknown destination. This effectively replaces the map-request messaging with data plane events. Either way this is implemented, the main benefit of the deployment model is the ability to avoid packet drops while mapping system lookups are completed.

The Map-Server/Resolver function could be deployed at Leaf nodes. In this scenario it may be beneficial to make the different Leaf hosted Map-Server/Resolvers authoritative for specific prefixes or instance-ids (VNIs) and allow the distribution of the mapping state across the different Leaf nodes. One mechanism to achieve the distribution of the mapping state across the different leaf nodes is to have different Leaf nodes be the authority for a particular prefix or instance-id in a Delegated Database Tree (DDT)[I-D.ietf-lisp-ddt]. The Leaf nodes can be arranged in pairs for redundancy and the association of a particular prefix or instance-id to specific Leaf nodes is achieved by configuration of the Leaf nodes and the DDT. This type of in-line deployment of the Map-Server/Resolver could, in theory, also provide a no-drop LISP lookup service at the expense of maintaining all LISP mapping state at all Leaf nodes. In the case of a no-drop LISP lookup service, the Map-Server/Resolver function would be deployed in the same way as explained for deployment in the Spine and the system would therefore not benefit from the distribution of state at the Leaf nodes.

The placement of IP workloads within and across Data Centers has a high degree of entropy, which renders existing topology congruent routing summarization methods ineffective in containing the explosion of routing state in the Data Center WAN and Data Center Interconnect. The problem of entropy in the placement of workloads across Data Centers is analogous to the challenge of prefix disaggregation found on the Internet. The seminal motivation behind the development of LISP was the mitigation of the scale issues caused by the disaggregation of user prefixes in the Internet, the multi-Data Center problem is of similar nature, hence LISP is a very good fit to address such scale problem. In the WAN, LISP will be in charge of handling reachability information between Branch sites and IP workloads that are distributed across different Data Centers. The use of LISP allows the scalable handling of very granular location information as IP workloads are deployed in diverse Data Centers. The management domain is usually a single one for the LISP WAN environment and the diverse Data Centers involved may be managed independently. The diversity of management domains is a key deployment consideration which is found in the WAN and DCI connectivity scenarios and requires a certain level of federation between the different domains as will be discussed in the next few sections. In the Data Center Interconnect (DCI), LISP provides reachability and policies (such as inbound load-balancing) between Data Centers for those cases in which IP workloads must communicate with each other across Data Centers. The communication may happen between Data Centers in independent administrative domains, with a single administrative domain being a more specific instantiation of the general case. In the WAN and DCI, as inside the Data Center, the main LISP functions to deploy are: xTRs Map-Servers Map-Resolvers PxTRs Map-Servers/Map-Resolvers will be distributed across Data Centers and over the WAN following the DDT model. DDT prefix delegation provides a way to parition different administrative domains, providing a very basic form of federation. xTRs will normally be deployed at the edge of the Data Center on the WAN/DCI routers. PxTR location may vary, but in general will be placed as close as possible to Internet gateways, or other non LISP-speaking sources. See [RFC6832] for considerations on Interworking with non-LISP sites. Particularly interesting in the deployment of LISP is the type of handoff between the Data Center Fabric and the devices providing the WAN/DCI services. Different handoff options are discussed in . Of further relevance is the careful evaluation of the different interoperability scenarios between the Data Center Fabric and the WAN/DCI that are discussed in .

There are a couple of approaches to realizing the handoff between the Fabric and the WAN: Two-tier: Border-Leaf to WAN-router peering Single-tier: Consolidated border-Leaf/WAN-router function

In the two-tier approach the border-Leaf and WAN-router peer over a normalized handoff. There is clear administrative delineation at the handoff between Fabric and WAN where the Fabric Manager is authoritative solely over the border-Leaf and the WAN Manager is authoritative solely over the WAN-router. The border-Leaf and WAN router may enter a peering relationship and exchange routes and other attributes of their respective overlay services. Segments (VNs) in the Fabric overlay will map at the border-Leaf to a normalized segment over the handoff that can be realized by use of VRFs or VLANs interconnected over an 802.1Q trunked set of links or over an overlay encapsulation such as VXLAN, GRE, NVGRE, MPLS, LISP or other. These segments will in turn be mapped at the WAN-router to their corresponding segments in the WAN service (for example, a VRF in an IP MPLS VPN or a VRF/instance-id segment in LISP). Reachability and other information can be exchanged between the border-Leaf nodes and the WAN routers over the normalized handoff using an extensible routing protocol such as MP-BGP or IS-IS.

In the single tier approach, the border-Leaf and the WAN-router are the same device. Mapping of segments (VNs) in one domain to segments in the other domain is achieved by sharing certain components between the two domains. One good example is a VRF that routes traffic between the Fabric overlay domain and the WAN overlay domain at the consolidated border-Leaf/WAN-router device. The device that consolidates the border-Leaf and WAN-router roles is managed by two entities: the Fabric manager and the WAN manager. Delineation of management authority must be established carefully and precisely to separate WAN relevant elements within the device from Fabric relevant elements and their corresponding administration. There will be a series of common components where the mapping between the domains happens (e.g. VRFs). Policy to resolve any conflicts related to the administration of common components in the handoff must be in place.

To integrate a LISP based Fabric with the LISP WAN it is possible to merge the Fabric LISP domain with the WAN LISP domain. To achieve this, the Map-Servers/Resolvers for the Fabric can join the DDT structure in the WAN. The merging of the domains assumes that the Fabric ITR/Leaf nodes are reachable in the underlay (RLOC space) from the WAN. In this model the xTRs are the Fabric Leaf nodes as well as the Branch Routers at the different WAN sites and an end-to-end overlay is realized between xTRs cutting across WAN and Fabric. Although the domains are merged into one, the hierarchical structure of the DDT provides some isolation of the control plane and failure isolation. The granular state for the Fabric is maintained solely on the Fabric Map-Server/Resolvers and LISP signaling for the Fabric xTRs will be scoped and serviced by the Fabric Map-Server/Resolvers. The domain isolation achieved with this model may not be sufficient for specific deployments, it is possible to follow the model described for a disjoint RLOC space in even if the RLOC space is contiguous.

There are scenarios in which the RLOC space in the Fabric is not part of the WAN routing space. In these scenarios the LISP overlay realized inside the Fabric must terminate at the border of the Fabric and should be mapped to a WAN overlay that terminates at the border of the WAN. The handoff between the two domains may be a single-tier handoff. In this case, the WAN/DCI device and bLeaf are the same device. The WAN/DCI device will be an xTR in two domains: Fabric and WAN/DCI. In one deployment model the xTR may receive Map-Notifications from the WAN Mapping System and these may trigger registrations into the Fabric Mapping System and vice versa. This model basically uses the xTR as an exchange point between the two LISP control plane instances. From the data plane perspective the xTR acts as an RTR (Re-encapsulating Tunnel Router). A different deployment model may rely heavily on Mapping System intelligence and may assume that the mapping systems are federated and that the location of the requesting ITR gives the mapping system enough context to provide map-replies that will steer traffic to the appropriate RTRs as packets progress through the different Data Center edges. In the disjoint RLOC scenario, the handoff between the two domains may alternatively be a two-tier handoff with separate border-Leaf and WAN-router devices. The exchange of routing information between the bLeaf and WAN-router may be achieved by dynamic routing protocols such as MP-BGP or IS-IS. The routes received by either device must be registered into the respective mapping system. Similarly, any mappings in one domain must be advertised as routes on the handoff to the other domain. When the LISP implementation does not support redistribution of routes into LISP and the converse redistribution of Mappings into routing protocols, the use of static routes and static mappings can help the interoperability of the two domains.

In order to support mobility, Data Center Fabrics are either built as pure Layer 2 Fabrics or, when built as Layer 3 networks, they rely on host routing to achieve mobility in the routed Fabric. The implication of the use of Layer 3 in the Fabric is that every Leaf will advertise host routes for the devices that are directly connected to the Leaf and all Leaf nodes will hold complete state for all end-points that attach to the fabric. When multiple Fabrics are interconnected with a DCI/WAN service, end-points that are reachable outside the fabric can be represented inside the fabric by a default route advertised by the bLeaves into the Fabric. The routing is thus made of specific host routes for end-points attached to the local Fabric and any end-points attached to other Fabrics will not be known specifically, but would be assumed to be reachable via the default route that points to the bLeaf. Another way of describing this model is to simply state that all local end-points are explicitly known and that traffic destined to any unknown destinations will be default forwarded to the bLeaf. The bLeaf can be a LISP xTR (in the single-tier model) and it would receive default routed traffic for which it can issue map-requests and encapsulate to the appropriate destination as the traffic is received. Altenatively, the bLeaf may handoff the traffic to a WAN/DCI router that provides the required xTR functionality. The advantages of this model are several: Remote state is kept out of the local fabric by using default routing xTRs in the LISP WAN/DCI only resolve and cache mappings for active flows as expected of an on-demand control plane Host routes terminate at the bLeaf and are kept outside of the routing tables in the WAN/DCI as they are handled by LISP instead

In this scenario, the prefixes handled by LISP inside the Fabric must be advertised to the Non-LISP WAN/DCI. In order to do so, the bLeaf must be able to pull all EIDs that are registered with the Fabric's mapping system and advertise these EIDs in a routing protocol. If the EIDs are easily summarizable, it may be sufficient to simply add a static route at the bLeaf. More generally, the EIDs may not be easily summarizable and may even change dynamically. When the EIDs are not summarizable or static, a mechanism for the bLeaf to subscribe to the mapping system and download all EIDs dynamically is required. Since LISP does not provide such subscription service, one option is to run a push protocol such as BGP between the mapping system and the bLeaf, since all registered EIDs are known at the Map-Server/Resolver, the Map-Server/Resolver may redistribute these EIDs into BGP and the required information may be advertised over BGP to the xTRs on the bLeaves. Another option is for the Map-Server-Resolver to be located at the bLeaves and have the EIDs be redistributed from LISP into a routing protocol on the same device.

defines a set of security mechanisms that provide origin authentication, integrity and anti-replay protection to LISP's EID-to-RLOC mapping data conveyed via mapping lookup process. LISP-SEC also enables verification of authorization on EID-prefix claims in Map-Reply messages. Additional security mechanisms to protect the LISP Map-Register messages are defined in . The security of the Mapping System Infrastructure depends on the particular mapping database used. The specification, as an example, defines a public-key based mechanism that provides origin authentication and integrity protection to the LISP DDT protocol.

This document has no IANA implications

The authors want to thank Yves Hertoghs for the early review, insightful comments and suggestions.