Network Working Group R. Callon Request for Comments: 4110 Juniper Networks Category: Informational M. Suzuki NTT Corporation July 2005 A Framework for Layer 3 Provider-Provisioned Virtual Private Networks (PPVPNs) Status of This Memo This memo provides information for the Internet community. It does not specify an Internet standard of any kind. Distribution of this memo is unlimited. Copyright Notice Copyright (C) The Internet Society (2005). Abstract This document provides a framework for Layer 3 Provider-Provisioned Virtual Private Networks (PPVPNs). This framework is intended to aid in the standardization of protocols and mechanisms for support of layer 3 PPVPNs. It is the intent of this document to produce a coherent description of the significant technical issues that are important in the design of layer 3 PPVPN solutions. Selection of specific approaches, making choices regarding engineering tradeoffs, and detailed protocol specification, are outside of the scope of this framework document. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.1. Objectives of the Document . . . . . . . . . . . . . . . 3 1.2. Overview of Virtual Private Networks . . . . . . . . . . 4 1.3. Types of VPNs. . . . . . . . . . . . . . . . . . . . . . 7 1.3.1. CE- vs PE-based VPNs . . . . . . . . . . . . . . 8 1.3.2. Types of PE-based VPNs . . . . . . . . . . . . . 9 1.3.3. Layer 3 PE-based VPNs. . . . . . . . . . . . . . 10 1.4. Scope of the Document. . . . . . . . . . . . . . . . . . 10 1.5. Terminology. . . . . . . . . . . . . . . . . . . . . . . 11 1.6. Acronyms . . . . . . . . . . . . . . . . . . . . . . . . 13 2. Reference Models . . . . . . . . . . . . . . . . . . . . . . . 14 2.1. Reference Model for Layer 3 PE-based VPN . . . . . . . . 14 2.1.1. Entities in the Reference Model. . . . . . . . . 16 2.1.2. Relationship Between CE and PE . . . . . . . . . 18 Callon & Suzuki Informational [Page 1] RFC 4110 A Framework for L3 PPVPNs July 2005 2.1.3. Interworking Model . . . . . . . . . . . . . . . 19 2.2. Reference Model for Layer 3 Provider-Provisioned CE-based VPN . . . . . . . . . . . . . . . . . . . . . . 21 2.2.1. Entities in the Reference Model. . . . . . . . . 22 3. Customer Interface . . . . . . . . . . . . . . . . . . . . . . 23 3.1. VPN Establishment at the Customer Interface. . . . . . . 23 3.1.1. Layer 3 PE-based VPN . . . . . . . . . . . . . . 23 3.1.1.1. Static Binding . . . . . . . . . . . . 24 3.1.1.2. Dynamic Binding. . . . . . . . . . . . 24 3.1.2. Layer 3 Provider-Provisioned CE-based VPN. . . . 25 3.2. Data Exchange at the Customer Interface. . . . . . . . . 25 3.2.1. Layer 3 PE-based VPN . . . . . . . . . . . . . . 25 3.2.2. Layer 3 Provider-Provisioned CE-based VPN. . . . 26 3.3. Customer Visible Routing . . . . . . . . . . . . . . . . 26 3.3.1. Customer View of Routing for Layer 3 PE-based VPNs . . . . . . . . . . . . . . . . . . . . . . 26 3.3.1.1. Routing for Intranets . . . . . . . . 27 3.3.1.2. Routing for Extranets . . . . . . . . 28 3.3.1.3. CE and PE Devices for Layer 3 PE-based VPNs. . . . . . . . . . . . . 29 3.3.2. Customer View of Routing for Layer 3 Provider- Provisioned CE-based VPNs. . . . . . . . . . . . 29 3.3.3. Options for Customer Visible Routing . . . . . . 30 4. Network Interface and SP Support of VPNs . . . . . . . . . . . 32 4.1. Functional Components of a VPN . . . . . . . . . . . . . 32 4.2. VPN Establishment and Maintenance. . . . . . . . . . . . 34 4.2.1. VPN Discovery . . . . . . . . . . . . . . . . . 35 4.2.1.1. Network Management for Membership Information. . . . . . . . . . . . . . 35 4.2.1.2. Directory Servers. . . . . . . . . . . 36 4.2.1.3. Augmented Routing for Membership Information. . . . . . . . . . . . . . 36 4.2.1.4. VPN Discovery for Inter-SP VPNs. . . . 37 4.2.2. Constraining Distribution of VPN Routing Information . . . . . . . . . . . . . . . . . . 38 4.2.3. Controlling VPN Topology . . . . . . . . . . . . 38 4.3. VPN Tunneling . . . . . . . . . . . . . . . . . . . . . 40 4.3.1. Tunnel Encapsulations. . . . . . . . . . . . . . 40 4.3.2. Tunnel Multiplexing. . . . . . . . . . . . . . . 41 4.3.3. Tunnel Establishment . . . . . . . . . . . . . . 42 4.3.4. Scaling and Hierarchical Tunnels . . . . . . . . 43 4.3.5. Tunnel Maintenance . . . . . . . . . . . . . . . 45 4.3.6. Survey of Tunneling Techniques . . . . . . . . . 46 4.3.6.1. GRE . . . . . . . . . . . . . . . . . 46 4.3.6.2. IP-in-IP Encapsulation . . . . . . . . 47 4.3.6.3. IPsec. . . . . . . . . . . . . . . . . 48 4.3.6.4. MPLS . . . . . . . . . . . . . . . . . 49 4.4. PE-PE Distribution of VPN Routing Information. . . . . . 51 Callon & Suzuki Informational [Page 2] RFC 4110 A Framework for L3 PPVPNs July 2005 4.4.1. Options for VPN Routing in the SP. . . . . . . . 52 4.4.2. VPN Forwarding Instances . . . . . . . . . . . . 52 4.4.3. Per-VPN Routing . . . . . . . . . . . . . . . . 53 4.4.4. Aggregated Routing Model . . . . . . . . . . . . 54 4.4.4.1. Aggregated Routing with OSPF or IS-IS. 55 4.4.4.2. Aggregated Routing with BGP. . . . . . 56 4.4.5. Scalability and Stability of Routing with Layer 3 PE-based VPNs. . . . . . . . . . . . . . . . . 59 4.5. Quality of Service, SLAs, and IP Differentiated Services 61 4.5.1. IntServ/RSVP . . . . . . . . . . . . . . . . . . 61 4.5.2. DiffServ . . . . . . . . . . . . . . . . . . . . 62 4.6. Concurrent Access to VPNs and the Internet . . . . . . . 62 4.7. Network and Customer Management of VPNs. . . . . . . . . 63 4.7.1. Network and Customer Management. . . . . . . . . 63 4.7.2. Segregated Access of VPN Information . . . . . . 64 5. Interworking Interface . . . . . . . . . . . . . . . . . . . . 66 5.1. Interworking Function. . . . . . . . . . . . . . . . . . 66 5.2. Interworking Interface . . . . . . . . . . . . . . . . . 66 5.2.1. Tunnels at the Interworking Interface. . . . . . 67 5.3. Support of Additional Services . . . . . . . . . . . . . 68 5.4. Scalability Discussion . . . . . . . . . . . . . . . . . 69 6. Security Considerations. . . . . . . . . . . . . . . . . . . . 69 6.1. System Security. . . . . . . . . . . . . . . . . . . . . 70 6.2. Access Control . . . . . . . . . . . . . . . . . . . . . 70 6.3. Endpoint Authentication . . . . . . . . . . . . . . . . 70 6.4. Data Integrity . . . . . . . . . . . . . . . . . . . . . 71 6.5. Confidentiality. . . . . . . . . . . . . . . . . . . . . 71 6.6. User Data and Control Data . . . . . . . . . . . . . . . 72 6.7. Security Considerations for Inter-SP VPNs . . . . . . . 72 Appendix A: Optimizations for Tunnel Forwarding. . . . . . . . . . 73 A.1. Header Lookups in the VFIs . . . . . . . . . . . . . . . 73 A.2. Penultimate Hop Popping for MPLS . . . . . . . . . . . . 73 A.3. Demultiplexing to Eliminate the Tunnel Egress VFI Lookup 74 Acknowledgments. . . . . . . . . . . . . . . . . . . . . . . . . . 75 Normative References . . . . . . . . . . . . . . . . . . . . . . . 76 Informative References . . . . . . . . . . . . . . . . . . . . . . 76 Contributors' Addresses. . . . . . . . . . . . . . . . . . . . . . 80 1. Introduction 1.1. Objectives of the Document This document provides a framework for Layer 3 Provider-Provisioned Virtual Private Networks (PPVPNs). This framework is intended to aid in standardizing protocols and mechanisms to support interoperable layer 3 PPVPNs. Callon & Suzuki Informational [Page 3] RFC 4110 A Framework for L3 PPVPNs July 2005 The term "provider-provisioned VPNs" refers to Virtual Private Networks (VPNs) for which the Service Provider (SP) participates in management and provisioning of the VPN, as defined in section 1.3. There are multiple ways in which a provider can participate in managing and provisioning a VPN; therefore, there are multiple different types of PPVPNs. The framework document discusses layer 3 VPNs (as defined in section 1.3). First, this document provides a reference model for layer 3 PPVPNs. Then technical aspects of layer 3 PPVPN operation are discussed, first from the customer's point of view, then from the providers point of view. Specifically, this includes discussion of the technical issues which are important in the design of standards and mechanisms for the operation and support of layer 3 PPVPNs. Furthermore, technical aspects of layer 3 PPVPN interworking are clarified. Finally, security issues as they apply to layer 3 PPVPNs are addressed. This document takes a "horizontal description" approach. For each technical issue, it describes multiple approaches. To specify a particular PPVPN strategy, one must choose a particular way of solving each problem, but this document does not make choices, and does not select any particular approach to support VPNs. The "vertical description" approach is taken in other documents, viz., in the documents that describe particular PPVPN solutions. Note that any specific solution will need to make choices based on SP requirements, customer needs, implementation cost, and engineering tradeoffs. Solutions will need to chose between flexibility (supporting multiple options) and conciseness (selection of specific options in order to simplify implementation and deployment). While a framework document can discuss issues and criteria which are used as input to these choices, the specific selection of a solution is outside of the scope of a framework document. 1.2. Overview of Virtual Private Networks The term "Virtual Private Network" (VPN) refers to a set of communicating sites, where (a) communication between sites outside the set and sites inside the set is restricted, but (b) communication between sites in the VPN takes place over a network infrastructure that is also used by sites which are not in the VPN. The fact that the network infrastructure is shared by multiple VPNs (and possibly also by non-VPN traffic) is what distinguishes a VPN from a private network. We will refer to this shared network infrastructure as the "VPN Backbone". Callon & Suzuki Informational [Page 4] RFC 4110 A Framework for L3 PPVPNs July 2005 The logical structure of the VPN, such as addressing, topology, connectivity, reachability, and access control, is equivalent to part of or all of a conventional private network using private facilities [RFC2764] [VPN-2547BIS]. In this document, we are concerned only with the case where the shared network infrastructure (VPN backbone) is an IP and/or MPLS network. Further, we are concerned only with the case where the Service Provider's edge devices, whether at the provider edge (PE) or at the Customer Edge (CE), determine how to route VPN traffic by looking at the IP and/or MPLS headers of the packets they receive from the customer's edge devices; this is the distinguishing feature of Layer 3 VPNs. In some cases, one SP may offer VPN services to another SP. The former SP is known as a carrier of carriers, and the service it offers is known as "carrier of carriers" service. In this document, in cases where the customer could be either an enterprise or SP network, we will make use of the term "customer" to refer to the user of the VPN services. Similarly we will use the term "customer network" to refer to the user's network. VPNs may be intranets, in which the multiple sites are under the control of a single customer administration, such as multiple sites of a single company. Alternatively, VPNs may be extranets, in which the multiple sites are controlled by administrations of different customers, such as sites corresponding to a company, its suppliers, and its customers. Figure 1.1. illustrates an example network, which will be used in the discussions below. PE1 and PE2 are Provider Edge devices within an SP network. CE1, CE2, and CE3 are Customer Edge devices within a customer network. Routers r3, r4, r5, and r6 are IP routers internal to the customer sites. Callon & Suzuki Informational [Page 5] RFC 4110 A Framework for L3 PPVPNs July 2005 ............ ................. ............ . . . . . . . +---+ +-------+ +-------+ +---+ . . r3---| | | | | |----|CE2|---r5 . . | | | | | | +---+ . . |CE1|----| PE1 | | PE2 | : . . | | | | | | +---+ . . r4---| | | | | |----|CE3|---r6 . . +---+ +-------+ +-------+ +---+ . . Customer . . Service . . Customer . . site 1 . . provider(s) . . site 2 . ............ ................. ............ Figure 1.1.: VPN interconnecting two sites. In many cases, Provider Edge (PE) and Customer Edge (CE) devices may be either routers or LSRs. In this document, the Service Providers' network is an IP or MPLS network. It is desired to interconnect the customer network sites via the Service Providers' network. Some VPN solutions require that the VPN service be provided either over a single SP network, or over a small set of closely cooperating SP networks. Other VPN solutions are intended to allow VPN service to be provided over an arbitrary set of minimally cooperating SP networks (i.e., over the public Internet). In many cases, customer networks will make use of private IP addresses [RFC1918] or other non-unique IP address (i.e., unregistered addresses); there is no guarantee that the IP addresses used in the customer network are globally unique. The addresses used in one customer's network may overlap the addresses used in others. However, a single PE device can be used to provide VPN service to multiple customer networks, even if those customer networks have overlapping addresses. In PE-based layer 3 VPNs, the PE devices may route the VPN traffic based on the customer addresses found in the IP headers; this implies that the PE devices need to maintain a level of isolation between the packets from different customer networks. In CE-based layer 3 VPNs, the PEs do not make routing decisions based on the customer's private addresses, so this issue does not arise. For either PE or CE-based VPNs, the fact that the VPNs do not necessarily use globally unique address spaces also implies that IP packets from a customer network cannot be transmitted over the SP network in their native form. Instead, some form of encapsulation/tunneling must be used. Callon & Suzuki Informational [Page 6] RFC 4110 A Framework for L3 PPVPNs July 2005 Tunneling is also important for other reasons, such as providing isolation between different customer networks, allowing a wide range of protocols to be carried over an SP network, etc. Different QoS and security characteristics may be associated with different tunnels. 1.3. Types of VPNs This section describes multiple types of VPNs, and some of the engineering tradeoffs between different types. It is not up to this document to decide between different types of VPNs. Different types of VPNs may be appropriate in different situations. There is a wide spectrum of types of possible VPNs, and it is difficult to split the types of VPNs into clearly distinguished categories. As an example, consider a company making use of a private network, with several sites interconnected via leased lines. All routing is done via routers which are internal to the private network. At some point, the administrator of the private network might decide to replace the leased lines by ATM links (using an ATM service from an SP). Here again all IP-level routing is done between customer premises routers, and managed by the private network administrator. In order to reduce the network management burden on the private network, the company may decide to make use of a provider-provisioned CE devices [VPN-CE]. Here the operation of the network might be unchanged, except that the CE devices would be provided by and managed by an SP. The SP might decide that it is too difficult to manually configure each CE-CE link. This might lead the SP to replace the ATM links with a layer 2 VPN service between CE devices [VPN-L2]. Auto- discovery might be used to simplify configuration of links between CE devices, and an MPLS service might be used between CE devices instead of an ATM service (for example, to take advantage of the provider's high speed IP or MPLS backbone). After a while the SP might decide that it is too much trouble to be managing a large number of devices at the customers' premises, and might instead physically move these routers to be on the provider premises. Each edge router at the provider premises might nonetheless be dedicated to a single VPN. The operation might remain unchanged (except that links from the edge routers to other routers in the private network become MAN links instead of LAN links, and the link from the edge routers to provider core routers become LAN links Callon & Suzuki Informational [Page 7] RFC 4110 A Framework for L3 PPVPNs July 2005 instead of MAN links). The routers in question can now be considered to be provider edge routers, and the service provided by the SP has now become essentially a layer 3 VPN service. In order to minimize the cost of equipment, the provider might decide to replace several dedicated PE devices with a single physical router with the capability of running virtual routers (VR) [VPN-VR]. Protocol operation may remain unchanged. In this case the provider is offering a layer 3 VPN service making use of a VR capability. Note that autodiscovery might be used in a manner which is very similar to how it had been done in the layer 2 VPN case described above (for example, BGP might be used between VRs for discovery of other VRs supporting the same VPN). Finally, in order to simplify operation of routing protocols for the private network over the SP network, the provider might decide to aggregate multiple instances of routing into a single instance of BGP [VPN-2547BIS]. In practice it is highly unlikely that any one network would actually evolve through all of these approaches at different points in time. However, this example illustrates that there is a continuum of possible approaches, and each approach is relatively similar to at least some of the other possible approaches for supporting VPN services. Some techniques (such as auto-discovery of VPN sites) may be common between multiple approaches. 1.3.1. CE- vs PE-based VPNs The term "CE-based VPN" (or Customer Edge-based Virtual Private Network) refers to an approach in which the PE devices do not know anything about the routing or the addressing of the customer networks. The PE devices offer a simple IP service, and expect to receive IP packets whose headers contain only globally unique IP addresses. What makes a CE-based VPN into a Provider-Provisioned VPN is that the SP takes on the task of managing and provisioning the CE devices [VPN-CE]. In CE-based VPNs, the backbone of the customer network is a set of tunnels whose endpoints are the CE devices. Various kinds of tunnels may be used (e.g., GRE, IP-in-IP, IPsec, L2TP, MPLS), the only overall requirement being that sending a packet through the tunnel requires encapsulating it with a new IP header whose addresses are globally unique. For customer provisioned CE-based VPNs, provisioning and management of the tunnels is the responsibility of the customer network administration. Typically, this makes use of manual configuration of Callon & Suzuki Informational [Page 8] RFC 4110 A Framework for L3 PPVPNs July 2005 the tunnels. In this case the customer is also responsible for operation of the routing protocol between CE devices. (Note that discussion of customer provisioned CE-based VPNs is out of scope of the document). For provider-provisioned CE-based VPNs, provisioning and management of the tunnels is the responsibility of the SP. In this case the provider may also configure routing protocols on the CE devices. This implies that routing in the private network is partially under the control of the customer, and partially under the control of the SP. For CE-based VPNs (whether customer or provider-provisioned) routing in the customer network treats the tunnels as layer 2 links. In a PE-based VPN (or Provider Edge-based Virtual Private Network), customer packets are carried through the SP networks in tunnels, just as they are in CE-based VPNs. However, in a PE-based VPN, the tunnel endpoints are the PE devices, and the PE devices must know how to route the customer packets, based on the IP addresses that they carry. In this case, the CE devices themselves do not have to have any special VPN capabilities, and do not even have to know that they are part of a VPN. In this document we will use the generic term "VPN Edge Device" to refer to the device, attached to both the customer network and the VPN backbone, that performs the VPN-specific functions. In the case of CE-based VPNs, the VPN Edge Device is a CE device. In the case of PE-based VPNs, the VPN Edge Device is a PE device. 1.3.2. Types of PE-based VPNs Different types of PE-based VPNs may be distinguished by the service offered. o Layer 3 service When a PE receives a packet from a CE, it determines how to forward the packet by considering both the packet's incoming link, and the layer 3 information in the packet's header. o Layer 2 service When a PE receives a frame from a CE, it determines how to forward the packet by considering both the packet's incoming link, and the layer 2 information in the frame header (such as FR, ATM, or MAC header). (Note that discussion of layer 2 service is out of scope of the document). Callon & Suzuki Informational [Page 9] RFC 4110 A Framework for L3 PPVPNs July 2005 1.3.3. Layer 3 PE-based VPNs A layer 3 PE-based VPN is one in which the SP takes part in IP level forwarding based on the customer network's IP address space. In general, the customer network is likely to make use of private and/or non-unique IP addresses. This implies that at least some devices in the provider network needs to understand the IP address space as used in the customer network. Typically this knowledge is limited to the PE devices which are directly attached to the customer. In a layer 3 PE-based VPN, the provider will need to participate in some aspects of management and provisioning of the VPNs, such as ensuring that the PE devices are configured to support the correct VPNs. This implies that layer 3 PE-based VPNs are by definition provider-provisioned VPNs. Layer 3 PE-based VPNs have the advantage that they offload some aspects of VPN management from the customer network. From the perspective of the customer network, it looks as if there is just a normal network; specific VPN functionality is hidden from the customer network. Scaling of the customer network's routing might also be improved, since some layer 3 PE-based VPN approaches avoid the need for the customer's routing algorithm to see "N squared" (actually N*(N-1)/2) point to point duplex links between N customer sites. However, these advantages come along with other consequences. Specifically, the PE devices must have some knowledge of the routing, addressing, and layer 3 protocols of the customer networks to which they attach. One consequence is that the set of layer 3 protocols which can be supported by the VPN is limited to those supported by the PE (which in practice means, limited to IP). Another consequence is that the PE devices have more to do, and the SP has more per-customer management to do. An SP may offer a range of layer 3 PE-based VPN services. At one end of the range is a service limited to simply providing connectivity (optionally including QoS support) between specific customer network sites. This is referred to as "Network Connectivity Service". There is a spectrum of other possible services, such as firewalls, user or site of origin authentication, and address assignment (e.g., using Radius or DHCP). 1.4. Scope of the Document This framework document will discuss methods for providing layer 3 PE-based VPNs and layer 3 provider-provisioned CE-based VPNs. This may include mechanisms which will can be used to constrain Callon & Suzuki Informational [Page 10] RFC 4110 A Framework for L3 PPVPNs July 2005 connectivity between sites, including the use and placement of firewalls, based on administrative requirements [PPVPN-REQ] [L3VPN-REQ]. Similarly the use and placement of NAT functionality is discussed. However, this framework document will not discuss methods for additional services such as firewall administration and address assignment. A discussion of specific firewall mechanisms and policies, and detailed discussion of NAT functionality, are outside of the scope of this document. This document does not discuss those forms of VPNs that are outside of the scope of the IETF Provider-Provisioned VPN working group. Specifically, this document excludes discussion of PPVPNs using VPN native (non-IP, non-MPLS) protocols as the base technology used to provide the VPN service (e.g., native ATM service provided using ATM switches with ATM signaling). However, this does not mean to exclude multiprotocol access to the PPVPN by customers. 1.5. Terminology Backdoor Links: Links between CE devices that are provided by the end customer rather than the SP; may be used to interconnect CE devices in multiple-homing arrangements. CE-based VPN: An approach in which all the VPN-specific procedures are performed in the CE devices, and the PE devices are not aware in any way that some of the traffic they are processing is VPN traffic. Customer: A single organization, corporation, or enterprise that administratively controls a set of sites belonging to a VPN. Customer Edge (CE) Device: The equipment on the customer side of the SP-customer boundary (the customer interface). IP Router: A device which forwards IP packets, and runs associated IP routing protocols (such as OSPF, IS-IS, RIP, BGP, or similar protocols). An IP router might optionally also be an LSR. The term "IP router" is often abbreviated as "router". Label Switching Router: A device which forwards MPLS packets and runs associated IP routing and signaling protocols (such as LDP, RSVP-TE, CR-LDP, OSPF, IS-IS, or similar protocols). A label switching router is also an IP router. PE-Based VPNs: The PE devices know that certain traffic is VPN traffic. They forward the traffic (through tunnels) based on the destination IP address of the packet, and optionally on based on other information in the IP header of the packet. The PE devices are Callon & Suzuki Informational [Page 11] RFC 4110 A Framework for L3 PPVPNs July 2005 themselves the tunnel endpoints. The tunnels may make use of various encapsulations to send traffic over the SP network (such as, but not restricted to, GRE, IP-in-IP, IPsec, or MPLS tunnels). Private Network: A network which allows communication between a restricted set of sites, over an IP backbone that is used only to carry traffic to and from those sites. Provider Edge (PE) Device: The equipment on the SP side of the SP-customer boundary (the customer interface). Provider-Provisioned VPNs (PPVPNs): VPNs, whether CE-based or PE-based, that are actively managed by the SP rather than by the end customer. Route Reflectors: An SP-owned network element that is used to distribute BGP routes to the SP's BGP-enabled routers. Virtual Private Network (VPN): Restricted communication between a set of sites, making use of an IP backbone which is shared by traffic that is not going to or coming from those sites. Virtual Router (VR): An instance of one of a number of logical routers located within a single physical router. Each logical router emulates a physical router using existing mechanisms and tools for configuration, operation, accounting, and maintenance. VPN Forwarding Instance (VFI): A logical entity that resides in a PE that includes the router information base and forwarding information base for a VPN. VPN Backbone: IP and/or MPLS network which is used to carry VPN traffic between the customer sites of a particular VPN. VPN Edge Device: Device, attached to both the VPN backbone and the customer network, which performs VPN-specific functions. For PE-based VPNs, this is the PE device; for CE-based VPNs, this is the CE device. VPN Routing: Routing that is specific to a particular VPN. VPN Tunnel: A logical link between two PE or two CE entities, used to carry VPN traffic, and implemented by encapsulating packets that are transmitted between those two entities. Callon & Suzuki Informational [Page 12] RFC 4110 A Framework for L3 PPVPNs July 2005 1.6. Acronyms ATM Asynchronous Transfer Mode BGP Border Gateway Protocol CE Customer Edge CLI Command Line Interface CR-LDP Constraint-based Routing Label Distribution Protocol EBGP External Border Gateway Protocol FR Frame Relay GRE Generic Routing Encapsulation IBGP Internal Border Gateway Protocol IKE Internet Key Exchange IGP Interior Gateway Protocol (e.g., RIP, IS-IS and OSPF are all IGPs) IP Internet Protocol (same as IPv4) IPsec Internet Protocol Security protocol IPv4 Internet Protocol version 4 (same as IP) IPv6 Internet Protocol version 6 IS-IS Intermediate System to Intermediate System routing protocol L2TP Layer 2 Tunneling Protocol LAN Local Area Network LDAP Lightweight Directory Access Protocol LDP Label Distribution Protocol LSP Label Switched Path LSR Label Switching Router MIB Management Information Base MPLS Multi Protocol Label Switching NBMA Non-Broadcast Multi-Access NMS Network Management System OSPF Open Shortest Path First routing protocol P Provider equipment PE Provider Edge PPVPN Provider-Provisioned VPN QoS Quality of Service RFC Request For Comments RIP Routing Information Protocol RSVP Resource Reservation Protocol RSVP-TE Resource Reservation Protocol with Traffic Engineering Extensions SNMP Simple Network Management Protocol SP Service Provider VFI VPN Forwarding Instance VPN Virtual Private Network VR Virtual Router Callon & Suzuki Informational [Page 13] RFC 4110 A Framework for L3 PPVPNs July 2005 2. Reference Models This section describes PPVPN reference models. The purpose of discussing reference models is to clarify the common components and pieces that are needed to build and deploy a PPVPN. Two types of VPNs, layer 3 PE-based VPN and layer 3 provider-provisioned CE-based VPN are covered in separated sections below. 2.1. Reference Model for Layer 3 PE-based VPN This subsection describes functional components and their relationship for implementing layer 3 PE-based VPN. Figure 2.1 shows the reference model for layer 3 PE-based VPNs and Figures 2.2 and 2.3 show relationship between entities in the reference model. As shown in Figure 2.1, the customer interface is defined as the interface which exists between CE and PE devices, and the network interface is defined as the interface which exists between a pair of PE devices. Figure 2.2 illustrates a single logical tunnel between each pair of VFIs supporting the same VPN. Other options are possible. For example, a single tunnel might occur between two PEs, with multiple per-VFI tunnels multiplexed over the PE to PE tunnel. Similarly, there may be multiple tunnels between two VFIs, for example to optimize forwarding within the VFI. Other possibilities will be discussed later in this framework document. Callon & Suzuki Informational [Page 14] RFC 4110 A Framework for L3 PPVPNs July 2005 +---------+ +------------------------------------+ +---------+ | | | | | | | | | +------+ +------+ : +------+ +------+ : | | | | | | : | CE | | CE | : | | | P | | PE | : |device| |device| : +------+ VPN tunnel : |router| |device| : | of | | of |-:--| |================:===============| |--:-|VPN A| |VPN A| : | | : +------+ +------+ : +------+ +------+ : | PE | : | | : | +------+ : |device| Network interface | | : | | CE | : | | : +------+ : +------+ |device|-:--| |================:===============| |--:-| CE | | of | : +------+ : VPN tunnel | PE | : |device| |VPN B| : | | |device| : | of | +------+ : | | +------------+ +------------+ | | : |VPN B| | : | | | Customer | | Network | +------+ : +------+ |Customer | | | management | | management | | | : | |interface| | | function | | function | | |Customer | | | | +------------+ +------------+ | |interface| | | | | | | +---------+ +------------------------------------+ +---------+ | Access | |<---------- SP network(s) --------->| | Access | | network | | single or multiple SP domains | | network | Figure 2.1: Reference model for layer 3 PE-based VPN. +----------+ +----------+ +-----+ |PE device | |PE device | +-----+ | CE | | | | | | CE | | dev | Access | +------+ | | +------+ | Access | dev | | of | conn. | |VFI of| | VPN tunnel | |VFI of| | conn. | of | |VPN A|----------|VPN A |======================|VPN A |----------|VPN A| +-----+ | +------+ | | +------+ | +-----+ | | | | +-----+ Access | +------+ | | +------+ | Access +-----+ | CE | conn. | |VFI of| | VPN tunnel | |VFI of| | conn. | CE | | dev |----------|VPN B |======================|VPN B |----------| dev | | of | | +------+ | | +------+ | | of | |VPN B| | | | | |VPN B| +-----+ +----------+ +----------+ +-----+ Figure 2.2: Relationship between entities in reference model (1). Callon & Suzuki Informational [Page 15] RFC 4110 A Framework for L3 PPVPNs July 2005 +----------+ +----------+ +-----+ |PE device | |PE device | +-----+ | CE | | | | | | CE | | dev | Access | +------+ | | +------+ | Access | dev | | of | conn. | |VFI of| | | |VFI of| | conn. | of | |VPN A|----------|VPN A | | | |VPN A |----------|VPN A| +-----+ | +------+\| Tunnel |/+------+ | +-----+ | >==================< | +-----+ Access | +------+/| |\+------+ | Access +-----+ | CE | conn. | |VFI of| | | |VFI of| | conn. | CE | | dev |----------|VPN B | | | |VPN B |----------| dev | | of | | +------+ | | +------+ | | of | |VPN B| | | | | |VPN B| +-----+ +----------+ +----------+ +-----+ Figure 2.3: Relationship between entities in reference model (2). 2.1.1. Entities in the Reference Model The entities in the reference model are described below. o Customer edge (CE) device In the context of layer 3 provider-provisioned PE-based VPNs, a CE device may be a router, LSR, or host that has no VPN-specific functionality. It is attached via an access connection to a PE device. o P router A router within a provider network which is used to interconnect PE devices, but which does not have any VPN state and does not have any direct attachment to CE devices. o Provider edge (PE) device In the context of layer 3 provider-provisioned PE-based VPNs, a PE device implements one or more VFIs and maintains per-VPN state for the support of one or more VPNs. It may be a router, LSR, or other device that includes VFIs and provider edge VPN functionality such as provisioning, management, and traffic classification and separation. (Note that access connections are terminated by VFIs from the functional point of view). A PE device is attached via an access connection to one or more CE devices. Callon & Suzuki Informational [Page 16] RFC 4110 A Framework for L3 PPVPNs July 2005 o Customer site A customer site is a set of users that have mutual IP reachability without use of a VPN backbone that goes beyond the site. o SP networks An SP network is an IP or MPLS network administered by a single service provider. o Access connection An access connection represents an isolated layer 2 connectivity between a CE device and a PE device. Access connections can be, e.g., dedicated physical circuits, logical circuits (such as FR, ATM, and MAC), or IP tunnels (e.g., using IPsec, L2TP, or MPLS). o Access network An access network provides access connections between CE and PE devices. It may be a TDM network, layer 2 network (e.g., FR, ATM, and Ethernet), or IP network over which access is tunneled (e.g., using L2TP [RFC2661] or MPLS). o VPN tunnel A VPN tunnel is a logical link between two VPN edge devices. A VPN packet is carried on a tunnel by encapsulating it before transmitting it over the VPN backbone. Multiple VPN tunnels at one level may be hierarchically multiplexed into a single tunnel at another level. For example, multiple per- VPN tunnels may be multiplexed into a single PE to PE tunnel (e.g., GRE, IP-in-IP, IPsec, or MPLS tunnel). This is illustrated in Figure 2.3. See section 4.3 for details. o VPN forwarding instance (VFI) A single PE device is likely to be connected to a number of CE devices. The CE devices are unlikely to all be in the same VPN. The PE device must therefore maintain a separate forwarding instances for each VPN to which it is connected. A VFI is a logical entity, residing in a PE, that contains the router information base and forwarding information base for a VPN. The interaction between routing and VFIs is discussed in section 4.4.2. Callon & Suzuki Informational [Page 17] RFC 4110 A Framework for L3 PPVPNs July 2005 o Customer management function The customer management function supports the provisioning of customer specific attributes, such as customer ID, personal information (e.g., name, address, phone number, credit card number, and etc.), subscription services and parameters, access control policy information, billing and statistical information, and etc. The customer management function may use a combination of SNMP manager, directory service (e.g., LDAP [RFC3377]), or proprietary network management system. o Network management function The network management function supports the provisioning and monitoring of PE or CE device attributes and their relationships. The network management function may use a combination of SNMP manager, directory service (e.g., LDAP [RFC3377]), or proprietary network management system. 2.1.2. Relationship Between CE and PE For robustness, a CE device may be connected to more than one PE device, resulting in a multi-homing arrangement. Four distinct types of multi-homing arrangements, shown in Figure 2.4, may be supported. Callon & Suzuki Informational [Page 18] RFC 4110 A Framework for L3 PPVPNs July 2005 +---------------- +--------------- | | +------+ +------+ +---------| PE | +---------| PE | | |device| | |device| SP network | +------+ | +------+ +------+ | +------+ | | CE | | | CE | +--------------- |device| | SP network |device| +--------------- +------+ | +------+ | | +------+ | +------+ | | PE | | | PE | +---------|device| +---------|device| SP network +------+ +------+ | | +---------------- +--------------- This type includes a CE device connected to a PE device via two access connections. (a) (b) +---------------- +--------------- | | +------+ +------+ +------+ +------+ | CE |-----| PE | | CE |-----| PE | |device| |device| |device| |device| SP network +------+ +------+ +------+ +------+ | | | | | Backdoor | | Backdoor +--------------- | link | SP network | link +--------------- | | | | +------+ +------+ +------+ +------+ | CE | | PE | | CE | | PE | |device|-----|device| |device|-----|device| SP network +------+ +------+ +------+ +------+ | | +---------------- +--------------- (c) (d) Figure 2.4: Four types of double-homing arrangements. 2.1.3. Interworking Model It is quite natural to assume that multiple different layer 3 VPN approaches may be implemented, particularly if the VPN backbone includes more than one SP network. For example, (1) each SP chooses one or more layer 3 PE-based VPN approaches out of multiple vendor's implementations, implying that different SPs may choose different Callon & Suzuki Informational [Page 19] RFC 4110 A Framework for L3 PPVPNs July 2005 approaches; and (2) an SP may deploy multiple networks of layer 3 PE-based VPNs (e.g., an old network and a new network). Thus it is important to allow interworking of layer 3 PE-based VPNs making use of multiple different layer 3 VPN approaches. There are three scenarios that enable layer 3 PE-based VPN interworking among different approaches. o Interworking function This scenario enables interworking using a PE that is located at one or more points which are logically located between VPNs based on different layer 3 VPN approaches. For example, this PE may be located on the boundary between SP networks which make use of different layer 3 VPN approaches [VPN-DISC]. A PE at one of these points is called an interworking function (IWF), and an example configuration is shown in Figure 2.5. +------------------+ +------------------+ | | | | +------+ VPN tunnel +------+ VPN tunnel +------+ | |==============| |==============| | | | | | | | | PE | | PE | | PE | | | |device| | | |device| |(IWF) | |device| | | VPN tunnel | | VPN tunnel | | | |==============| |==============| | +------+ +------+ +------+ | | | | +------------------+ +------------------+ |<-VPN approach 1->| |<-VPN approach 2->| Figure 2.5: Interworking function. o Interworking interface This scenario enables interworking using tunnels between PEs supporting by different layer 3 VPN approaches. As shown in Figure 2.6, interworking interface is defined as the interface which exists between a pair of PEs and connects two SP networks implemented with different approaches. This interface is similar to the customer interface located between PE and CE, but the interface is supported by tunnels to identify VPNs, while the customer interface is supported by access connections. Callon & Suzuki Informational [Page 20] RFC 4110 A Framework for L3 PPVPNs July 2005 +------------------+ +------------------+ | | : | | +------+ VPN tunnel +------+Tunnel: +------+ VPN tunnel +------+ | |============| |======:======| |============| | | | | | : | | | | | PE | | PE | : | PE | | PE | | | | | : | | | | |device| |device| : |device| |device| | | VPN tunnel | |Tunnel: | | VPN tunnel | | | |============| |======:======| |============| | +------+ +------+ : +------+ +------+ | | : | | +------------------+ Interworking +------------------+ |<-VPN approach 1->| interface |<-VPN approach 2->| Figure 2.6: Interworking interface. o Customer-based interworking If some customer site has a CE attached to one kind of VPN, and a CE attached to another kind, communication between the two kinds of VPN occurs automatically. 2.2. Reference Model for Layer 3 Provider-Provisioned CE-based VPN This subsection describes functional components and their relationship for implementing layer 3 provider-provisioned CE-based VPN. Figure 2.7 shows the reference model for layer 3 provider-provisioned CE-based VPN. As shown in Figure 2.7, the customer interface is defined as the interface which exists between CE and PE devices. In this model, a CE device maintains one or more VPN tunnel endpoints, and a PE device has no VPN-specific functionality. As a result, the interworking issues of section 2.1.3 do not arise. Callon & Suzuki Informational [Page 21] RFC 4110 A Framework for L3 PPVPNs July 2005 +---------+ +------------------------------------+ +---------+ | | | | | | | | | +------+ +------+ : +------+ +------+ : | | | | | | : | CE | | CE | : | | | P | | PE | : |device| |device| : +------+ VPN tunnel |router| |device| : | of | | of |=:====================================================:=|VPN A| |VPN A| : | | +------+ +------+ : +------+ +------+ : | PE | | | : | +------+ : |device| | | : | | CE | : | | VPN tunnel +------+ : +------+ |device|=:====================================================:=| CE | | of | : +------+ | PE | : |device| |VPN B| : | | |device| : | of | +------+ : | | +------------+ +------------+ | | : |VPN B| | : | | | Customer | | Network | +------+ : +------+ |Customer | | | management | | management | | | : | |interface| | | function | | function | | |Customer | | | | +------------+ +------------+ | |interface| | | | | | | +---------+ +------------------------------------+ +---------+ | Access | |<---------- SP network(s) --------->| | Access | | network | | | | network | Figure 2.7: Reference model for layer 3 provider-provisioned CE-based VPN. 2.2.1. Entities in the Reference Model The entities in the reference model are described below. o Customer edge (CE) device In the context of layer 3 provider-provisioned CE-based VPNs, a CE device provides layer 3 connectivity to the customer site. It may be a router, LSR, or host that maintains one or more VPN tunnel endpoints. A CE device is attached via an access connection to a PE device and usually located at the edge of a customer site or co-located on an SP premises. o P router (see section 2.1.1) o Provider edge (PE) device In the context of layer 3 provider-provisioned CE-based VPNs, a PE device may be a router, LSR, or other device that has no VPN-specific functionality. It is attached via an access connection to one or more CE devices. Callon & Suzuki Informational [Page 22] RFC 4110 A Framework for L3 PPVPNs July 2005 o Customer Site (see section 2.1.1) o SP networks An SP network is a network administrated by a single service provider. It is an IP or MPLS network. In the context of layer 3 provider-provisioned CE-based VPNs, the SP network consists of the SP's network and the SP's management functions that manage both its own network and the customer's VPN functions on the CE device. o Access connection (see section 2.1.1) o Access network (see section 2.1.1) o VPN tunnel A VPN tunnel is a logical link between two entities which is created by encapsulating packets within an encapsulating header for purpose of transmission between those two entities for support of VPNs. In the context of layer 3 provider-provisioned CE-based VPNs, a VPN tunnel is an IP tunnel (e.g., using GRE, IP-in-IP, IPsec, or L2TP) or an MPLS tunnel between two CE devices over the SP's network. o Customer management function (see section 2.1.1) o Network management function The network management function supports the provisioning and monitoring of PE or CE device attributes and their relationships, covering PE and CE devices that define the VPN connectivity of the customer VPNs. The network management function may use a combination of SNMP manager, directory service (e.g., LDAP [RFC3377]), or proprietary network management system. 3. Customer Interface 3.1. VPN Establishment at the Customer Interface 3.1.1. Layer 3 PE-based VPN It is necessary for each PE device to know which CEs it is attached to, and what VPNs each CE is associated with. VPN membership refers to the association of VPNs, CEs, and PEs. A given CE belongs to one or more VPNs. Each PE is therefore Callon & Suzuki Informational [Page 23] RFC 4110 A Framework for L3 PPVPNs July 2005 associated with a set of VPNs, and a given VPN has a set of associated PEs which are supporting that VPN. If a PE has at least one attached CE belonging to a given VPN, then state information for that VPN (e.g., the VPN routes) must exist on that PE. The set of VPNs that exist on a PE may change over time as customer sites are added to or removed from the VPNs. In some layer 3 PE-based PPVPN schemes, VPN membership information (i.e., information about which PEs are attached to which VPNs) is explicitly distributed. In others, the membership information is inferred from other information that is distributed. Different schemes use the membership information in different ways, e.g., some to determine what set of tunnels to set up, some to constrain the distribution of VPN routing information. A VPN site may be added or deleted as a result of a provisioning operation carried out by the network administrator, or may be dynamically added or deleted as a result of a subscriber initiated operation; thus VPN membership information may be either static or dynamic, as discussed below. 3.1.1.1. Static Binding Static binding occurs when a provisioning action binds a particular PE-CE access link to a particular VPN. For example, a network administrator may set up a dedicated link layer connection, such as an ATM VCC or a FR DLCI, between a PE device and a CE device. In this case the binding between a PE-CE access connection and a particular VPN to fixed at provisioning time, and remains the same until another provisioning action changes the binding. 3.1.1.2. Dynamic Binding Dynamic binding occurs when some real-time protocol interaction causes a particular PE-CE access link to be temporarily bound to a particular VPN. For example, a mobile user may dial up the provider network and carry out user authentication and VPN selection procedures. Then the PE to which the user is attached is not one permanently associated with the user, but rather one that is typically geographically close to where the mobile user happens to be. Another example of dynamic binding is that of a permanent access connection between a PE and a CE at a public facility such as a hotel or conference center, where the link may be accessed by multiple users in turn, each of which may wish to connect to a different VPN. To support dynamically connected users, PPP and RADIUS are commonly used, as these protocols provide for user identification, authentication and VPN selection. Other mechanisms are also Callon & Suzuki Informational [Page 24] RFC 4110 A Framework for L3 PPVPNs July 2005 possible. For example a user's HTTP traffic may be initially intercepted by a PE and diverted to a provider hosted web server. After a dialogue that includes user authentication and VPN selection, the user can then be connected to the required VPN. This is sometimes referred to as a "captive portal". Independent of the particular mechanisms used for user authentication and VPN selection, an implication of dynamic binding is that a user for a given VPN may appear at any PE at any time. Thus VPN membership may change at any time as a result of user initiated actions, rather than as a result of network provisioning actions. This suggests that there needs to be a way to distribute membership information rapidly and reliably when these user-initiated actions take place. 3.1.2. Layer 3 Provider-Provisioned CE-based VPN In layer 3 provider-provisioned CE-based VPNs, the PE devices have no knowledge of the VPNs. A PE device attached to a particular VPN has no knowledge of the addressing or routing information of that specific VPN. CE devices have IP or MPLS connectivity via a connection to a PE device, which just provides ordinary connectivity to the global IP address space or to an address space which is unique in a particular SPs network. The IP connectivity may be via a static binding, or via some kind of dynamic binding. The establishment of the VPNs is done at each CE device, making use of the IP or MPLS connectivity to the others. Therefore, it is necessary for a given CE device to know which other CE devices belong to the same VPN. In this context, VPN membership refers to the association of VPNs and CE devices. 3.2. Data Exchange at the Customer Interface 3.2.1. Layer 3 PE-based VPN For layer 3 PE-based VPNs, the exchange is normal IP packets, transmitted in the same form which is available for interconnecting routers in general. For example, IP packets may be exchanged over Ethernet, SONET, T1, T3, dial-up lines, and any other link layer available to the router. It is important to note that those link layers are strictly local to the interface for the purpose of carrying IP packets, and are terminated at each end of the customer interface. The IP packets may contain addresses which, while unique within the VPN, are not unique on the VPN backbone. Optionally, the data exchange may use MPLS to carry the IP packets. Callon & Suzuki Informational [Page 25] RFC 4110 A Framework for L3 PPVPNs July 2005 3.2.2. Layer 3 Provider-Provisioned CE-based VPN The data exchanged at the customer interface are always normal IP packets that are routable on the VPN backbone, and whose addresses are unique on the VPN backbone. Optionally, MPLS frames can be used, if the appropriate label-switched paths exist across the VPN backbone. The PE device does not know whether these packets are VPN packets or not. At the current time, MPLS is not commonly offered as a customer-visible service, so that CE-based VPNs most commonly make use of IP services. 3.3. Customer Visible Routing Once VPN tunnels are set up between pairs of VPN edge devices, it is necessary to set up mechanisms which ensure that packets from the customer network get sent through the proper tunnels. This routing function must be performed by the VPN edge device. 3.3.1. Customer View of Routing for Layer 3 PE-based VPNs There is a PE-CE routing interaction which enables a PE to obtain those addresses, from the customer network, that are reachable via the CE. The PE-CE routing interaction also enables a CE device to obtain those addresses, from the customer network, which are reachable via the PE; these will generally be addresses that are at other sites in the customer network. The PE-CE routing interaction can make use of static routing, an IGP (such as RIP, OSPF, IS-IS, etc.), or BGP. If the PE-CE interaction is done via an IGP, the PE will generally maintain at least several independent IGP instances; one for the backbone routing, and one for each VPN. Thus the PE participates in the IGP of the customer VPNs, but the CE does not participate in the backbone's IGP. If the PE-CE interaction is done via BGP, the PE MAY support one instance of BGP for each VPN, as well as an additional instance of BGP for the public Internet routes. Alternatively, the PE might support a single instance of BGP, using, e.g., different BGP Address Families to distinguish the public Internet routes from the VPN routes. Routing information which a PE learns from a CE in a particular VPN must be forwarded to the other PEs that are attached to the same VPN. Those other PEs must then forward the information in turn to the other CEs of that VPN. Callon & Suzuki Informational [Page 26] RFC 4110 A Framework for L3 PPVPNs July 2005 The PE-PE routing distribution can be done as part of the same routing instance to which the PE-CE interface belongs. Alternatively, it can be done via a different routing instance, possibly using a different routing algorithm. In this case, the PE must redistribute VPN routes from one routing instance to another. Note that VPN routing information is never distributed to the P routers. VPN routing information is known at the edge of the VPN backbone, but not in the core. If the VPN's IGP is different than the routing algorithm running on the CE-PE link, then the CE must support two routing instances, and must redistribute the VPN's routes from one instance to the other (e.g., [VPN-BGP-OSPF]). In the case of layer 3 PE-based VPNs a single PE device is likely to provide service for several different VPNs. Since different VPNs may have address spaces which are not mutually unique, a PE device must have several forwarding tables, in general one for each VPN to which it is attached. These will be referred to as VPN Forwarding Instances (VFIs). Each VFI is a logical entity internal to the PE device. VFIs are defined in section 2.1.1, and discussed in more detail in section 4.4.2. The scaling and management of the customer network (as well as the operation of the VPN) will depend upon the implementation approach and the manner in which routing is done. 3.3.1.1. Routing for Intranets In the intranet case all of the sites to be interconnected belong to the same administration (for example, the same company). The options for routing within a single customer network include: o A single IGP area (using OSPF, IS-IS, or RIP) o Multiple areas within a single IGP o A separate IGP within each site, with routes redistributed from each site to backbone routing (i.e., to a backbone as seen by the customer network). Note that these options look at routing from the perspective of the overall routing in the customer network. This list does not specify whether PE device is considered to be in a site or not. This issue is discussed below. Callon & Suzuki Informational [Page 27] RFC 4110 A Framework for L3 PPVPNs July 2005 A single IGP area (such as a single OSPF area, a single IS-IS area, or a single instance of RIP between routers) may be used. One could have, all routers within the customer network (including the PEs, or more precisely, including a VFI within each PE) appear within a single area. Tunnels between the PEs could also appear as normal links. In some cases the multi-level hierarchy of OSPF or IS-IS may be used. One way to apply this to VPNs would be to have each site be a single OSPF or IS-IS area. The VFIs will participate in routing within each site as part of that area. The VFIs may then be interconnected as the backbone (OSPF area 0 or IS-IS level 2). If OSPF is used, the VFIs therefore appear to the customer network as area border routers. If IS-IS is used, the VFIs therefore participate in level 1 routing within the local area, and appear to the customer network as if they are level 2 routers in the backbone. Where an IGP is used across the entire network, it is straightforward for VPN tunnels, access connections, and backdoor links to be mixed in a network. Given that OSPF or IS-IS metrics will be assigned to all links, paths via alternate links can be compared and the shortest cost path will be used regardless of whether it is via VPN tunnels, access connections, or backdoor links. If multiple sites of a VPN do not use a common IGP, or if the backbone does not use the same common IGP as the sites, then special procedures may be needed to ensure that routes to/from other sites are treated as intra-area routes, rather than as external routes (depending upon the VPN approach taken). Another option is to operate each site as a separate routing domain. For example each site could operate as a single OSPF area, a single IS-IS area, or a RIP domain. In this case the per-site routing domains will need to redistribute routes into a backbone routing domain (Note: in this context the "backbone routing domain" refers to a backbone as viewed by the customer network). In this case it is optional whether or not the VFIs participate in the routing within each site. 3.3.1.2. Routing for Extranets In the extranet case the sites to be interconnected belong to multiple different administrations. In this case IGPs (such as OSPF, IS-IS, or RIP) are normally not used across the interface between organizations. Either static routes or BGP may be used between sites. If the customer network administration wishes to maintain control of routing between its site and other networks, then either Callon & Suzuki Informational [Page 28] RFC 4110 A Framework for L3 PPVPNs July 2005 static routing or BGP may be used across the customer interface. If the customer wants to outsource all such control to the provider, then an IGP or static routes may be used at this interface. The use of BGP between sites allows for policy based routing between sites. This is particularly useful in the extranet case. Note that private IP addresses or non-unique IP address (e.g., unregistered addresses) should not be used for extranet communication. 3.3.1.3. CE and PE Devices for Layer 3 PE-based VPNs When using a single IGP area across an intranet, the entire customer network participates in a single area of an IGP. In this case, for layer 3 PE-based VPNs both CE and PE devices participate as normal routers within the area. The other options make a distinction between routing within a site, and routing between sites. In this case, a CE device would normally be considered as part of the site where it is located. However, there is an option regarding how the PE devices should be considered. In some cases, from the perspective of routing within the customer network, a PE device (or more precisely a VFI within a PE device) may be considered to be internal to the same area or routing domain as the site to which it is attached. This simplifies the management responsibilities of the customer network administration, since inter-area routing would be handled by the provider. For example, from the perspective of routing within the customer network, the CE devices may be the area border or AS boundary routers of the IGP area. In this case, static routing, BGP, or whatever routing is used in the backbone, may be used across the customer interface. 3.3.2. Customer View of Routing for Layer 3 Provider-Provisioned CE-based VPNs For layer 3 provider-provisioned CE-based VPNs, the PE devices are not aware of the set of addresses which are reachable at particular customer sites. The CE and PE devices do not exchange the customer's routing information. Customer sites that belong to the same VPN may exchange routing information through the CE-CE VPN tunnels that appear, to the customers IGP, as router adjacencies. Alternatively, instead of Callon & Suzuki Informational [Page 29] RFC 4110 A Framework for L3 PPVPNs July 2005 exchanging routing information through the VPN tunnels, the SP's management system may take care of the configuration of the static route information of one site towards the other sites in the VPN. Routing within the customer site may be done in any possible way, using any kind of routing protocols (see section 3.3.3). As the CE device receives an IP or MPLS service from the SP, the CE and PE devices may exchange routing information that is meaningful within the SP routing realm. Moreover, as the forwarding of tunneled customer packets in the SP network will be based on global IP forwarding, the routes to the various CE devices must be known in the entire SP's network. This means that a CE device may need to participate in two different routing processes: o routing in its own private network (VPN routing), within its own site and with the other VPN sites through the VPN tunnels, possibly using private addresses. o routing in the SP network (global routing), as such peering with its PE. However, in many scenarios, the use of static/default routes at the CE-PE interface might be all the global routing that is required. 3.3.3. Options for Customer Visible Routing The following technologies are available for the exchange of routing information. o Static routing Routing tables may be configured through a management system. o RIP (Routing Information Protocol) [RFC2453] RIP is an interior gateway protocol and is used within an autonomous system. It sends out routing updates at regular intervals and whenever the network topology changes. Routing information is then propagated by the adjacent routers to their neighbors and thus to the entire network. A route from a source to a destination is the path with the least number of routers. This number is called the "hop count" and its maximum value is 15. This implies that RIP is suitable for a small- or medium-sized networks. Callon & Suzuki Informational [Page 30] RFC 4110 A Framework for L3 PPVPNs July 2005 o OSPF (Open Shortest Path First) [RFC2328] OSPF is an interior gateway protocol and is applied to a single autonomous system. Each router distributes the state of its interfaces and neighboring routers as a link state advertisement, and maintains a database describing the autonomous system's topology. A link state is advertised every 30 minutes or when the topology is reconfigured. Each router maintains an identical topological database, from which it constructs a tree of shortest paths with itself as the root. The algorithm is known as the Shortest Path First or SPF. The router generates a routing table from the tree of shortest paths. OSPF supports a variable length subnet mask, which enables effective use of the IP address space. OSPF allows sets of networks to be grouped together into an area. Each area has its own topological database. The topology of the area is invisible from outside its area. The areas are interconnected via a "backbone" network. The backbone network distributes routing information between the areas. The area routing scheme can reduce the routing traffic and compute the shortest path trees and is indispensable for larger scale networks. Each multi-access network with multiple routers attached has a designated router. The designated router generates a link state advertisement for the multi-access network and synchronizes the topological database with other adjacent routers in the area. The concept of designated router can thus reduce the routing traffic and compute shortest path trees. To achieve high availability, a backup designated router is used. o IS-IS (intermediate system to intermediate system) [RFC1195] IS-IS is a routing protocol designed for the OSI (Open Systems Interconnection) protocol suites. Integrated IS-IS is derived from IS-IS in order to support the IP protocol. In the Internet community, IS-IS means integrated IS-IS. In this, a link state is advertised over a connectionless network service. IS-IS has the same basic features as OSPF. They include: link state advertisement and maintenance of a topological database within an area, calculation of a tree of shortest paths, generation of a routing table from a tree of shortest paths, the area routing scheme, a designated router, and a variable length subnet mask. Callon & Suzuki Informational [Page 31] RFC 4110 A Framework for L3 PPVPNs July 2005 o BGP-4 (Border Gateway Protocol version 4) [RFC1771] BGP-4 is an exterior gateway protocol and is applied to the routing of inter-autonomous systems. A BGP speaker establishes a session with other BGP speakers and advertises routing information to them. A session may be an External BGP (EBGP) that connects two BGP speakers within different autonomous systems, or an internal BGP (IBGP) that connects two BGP speakers within a single autonomous system. Routing information is qualified with path attributes, which differentiate routes for the purpose of selecting an appropriate one from possible routes. Also, routes are grouped by the community attribute [RFC1997] [BGP-COM]. The IBGP mesh size tends to increase dramatically with the number of BGP speakers in an autonomous system. BGP can reduce the number of IBGP sessions by dividing the autonomous system into smaller autonomous systems and grouping them into a single confederation [RFC3065]. Route reflection is another way to reduce the number of IBGP sessions [RFC1966]. BGP divides the autonomous system into clusters. Each cluster establishes the IBGP full mesh within itself, and designates one or more BGP speakers as "route reflectors," which communicate with other clusters via their route reflectors. Route reflectors in each cluster maintain path and attribute information across the autonomous system. The autonomous system still functions like a fully meshed autonomous system. On the other hand, confederations provide finer control of routing within the autonomous system by allowing for policy changes across confederation boundaries, while route reflection requires the use of identical policies. BGP-4 has been extended to support IPv6, IPX, and others as well as IPv4 [RFC2858]. Multiprotocol BGP-4 carries routes from multiple "address families". 4. Network Interface and SP Support of VPNs 4.1. Functional Components of a VPN The basic functional components of an implementation of a VPN are: o A mechanism to acquire VPN membership/capability information o A mechanism to tunnel traffic between VPN sites o For layer 3 PE-based VPNs, a means to learn customer routes, distribute them between the PEs, and to advertise reachable destinations to customer sites. Callon & Suzuki Informational [Page 32] RFC 4110 A Framework for L3 PPVPNs July 2005 Based on the actual implementation, these functions could be implemented on a per-VPN basis or could be accomplished via a common mechanism shared by all VPNs. For instance, a single process could handle the routing information for all the VPNs or a separate process may be created for each VPN. Logically, the establishment of a VPN can be thought of as composed of the following three stages. In the first stage, the VPN edge devices learn of each other. In the second stage, they establish tunnels to each other. In the third stage, they exchange routing information with each other. However, not all VPN solutions need be decomposed into these three stages. For example, in some VPN solutions, tunnels are not established after learning membership information; rather, pre-existing tunnels are selected and used. Also, in some VPN solutions, the membership information and the routing information are combined. In the membership/capability discovery stage, membership and capability information needs to be acquired to determine whether two particular VPN edge devices support any VPNs in common. This can be accomplished, for instance, by exchanging VPN identifiers of the configured VPNs at each VPN edge device. The capabilities of the VPN edge devices need to be determined, in order to be able to agree on a common mechanism for tunneling and/or routing. For instance, if site A supports both IPsec and MPLS as tunneling mechanisms and site B supports only MPLS, they can both agree to use MPLS for tunneling. In some cases the capability information may be determined implicitly, for example some SPs may implement a single VPN solution. Likewise, the routing information for VPNs can be distributed using the methods discussed in section 4.4. In the tunnel establishment stage, mechanisms may need to be invoked to actually set up the tunnels. With IPsec, for instance, this could involve the use of IKE to exchange keys and policies for securing the data traffic. However, if IP tunneling, e.g., is used, there may not be any need to explicitly set up tunnels; if MPLS tunnels are used, they may be pre-established as part of normal MPLS functioning. In the VPN routing stage, routing information for the VPN sites must be exchanged before data transfer between the sites can take place. Based on the VPN model, this could involve the use of static routes, IGPs such as OSPF/ISIS/RIP, or an EGP such as BGP. VPN membership and capability information can be distributed from a central management system, using protocols such as, e.g., LDAP. Alternatively, it can be distributed manually. However, as manual configuration does not scale and is error prone, its use is discouraged. As a third alternative, VPN information can be Callon & Suzuki Informational [Page 33] RFC 4110 A Framework for L3 PPVPNs July 2005 distributed via protocols that ensure automatic and consistent distribution of information in a timely manner, much as routing protocols do for routing information. This may suggest that the information be carried in routing protocols themselves, though only if this can be done without negatively impacting the essential routing functions. It can be seen that quite a lot of information needs to be exchanged in order to establish and maintain a VPN. The scaling and stability consequences need to be analyzed for any VPN approach. While every VPN solution must address the functionality of all three components, the combinations of mechanisms used to provide the needed functionality, and the order in which different pieces of functionality are carried out, may differ. For layer 3 provider-provisioned CE-based VPNs, the VPN service is offering tunnels between CE devices. IP routing for the VPN is done by the customer network. With these solutions, the SP is involved in the operation of the membership/capability discovery stage and the tunnel establishment stage. The IP routing functional component may be entirely up to the customer network, or alternatively, the SP's management system may be responsible for the distribution of the reachability information of the VPN sites to the other sites of the same VPN. 4.2. VPN Establishment and Maintenance For a layer 3 provider-provisioned VPN the SP is responsible for the establishment and maintenance of the VPNs. Many different approaches and schemes are possible in order to provide layer 3 PPVPNs, however there are some generic problems that any VPN solution must address, including: o For PE-based VPNs, when a new site is added to a PE, how do the other PEs find out about it? When a PE first gets attached to a given VPN, how does it determine which other PEs are attached to the same VPN. For CE-based VPNs, when a new site is added, how does its CE find out about all the other CEs at other sites of the same VPN? o In order for layer 3 PE-based VPNs to scale, all routes for all VPNs cannot reside on all PEs. How is the distribution of VPN routing information constrained so that it is distributed to only those devices that need it? Callon & Suzuki Informational [Page 34] RFC 4110 A Framework for L3 PPVPNs July 2005 o An administrator may wish to provision different topologies for different VPNs (e.g., a full mesh or a hub & spoke topology). How is this achieved? This section looks at some of these generic problems and at some of the mechanisms that can be used to solve them. 4.2.1. VPN Discovery Mechanisms are needed to acquire information that allows the establishment and maintenance of VPNs. This may include, for example, information on VPN membership, topology, and VPN device capabilities. This information may be statically configured, or distributed by an automated protocol. As a result of the operation of these mechanisms and protocols, a device is able to determine where to set up tunnels, and where to advertise the VPN routes for each VPN. With a physical network, the equivalent problem can by solved by the control of the physical interconnection of links, and by having a router run a discovery/hello protocol over its locally connected links. With VPNs both the routers and the links (tunnels) may be logical entities, and thus some other mechanisms are needed. A number of different approaches are possible for VPN discovery. One scheme uses the network management system to configure and provision the VPN edge devices. This approach can also be used to distribute VPN discovery information, either using proprietary protocols or using standard management protocols and MIBs. Another approach is where the VPN edge devices act as clients of a centralized directory or database server that contains VPN discovery information. Another possibility is where VPN discovery information is piggybacked onto a routing protocol running between the VPN edge devices [VPN-DISC]. 4.2.1.1. Network Management for Membership Information SPs use network management extensively to configure and monitor the various devices that are spread throughout their networks. This approach could be also used for distributing VPN related information. A network management system (either centralized or distributed) could be used by the SP to configure and provision VPNs on the VPN edge devices at various locations. VPN configuration information could be entered into a network management application and distributed to the remote sites via the same means used to distribute other network management information. This approach is most natural when all the devices that must be provisioned are within a single SP's network, Callon & Suzuki Informational [Page 35] RFC 4110 A Framework for L3 PPVPNs July 2005 since the SP has access to all VPN edge devices in its domain. Security and access control are important, and could be achieved for example using SNMPv3, SSH, or IPsec tunnels. 4.2.1.2. Directory Servers An SP typically needs to maintain a database of VPN configuration/membership information, regardless of the mechanisms used to distribute it. LDAPv3 [RFC3377] is a standard directory protocol which makes it possible to use a common mechanism for both storing such information and distributing it. To facilitate interoperability between different implementations, as well as between the management systems of different SPs, a standard schema for representing VPN membership and configuration information would have to be developed. LDAPv3 supports authentication of messages and associated access control, which can be used to limit access to VPN information to authorized entities. 4.2.1.3. Augmented Routing for Membership Information Extensions to the use of existing BGP mechanisms, for distribution of VPN membership information, are proposed in [VPN-2547BIS]. In that scheme, BGP is used to distribute VPN routes, and each route carries a set of attributes which indicate the VPN (or VPNs) to which the route belongs. This allows the VPN discovery information and routing information to be combined in a single protocol. Information needed to establish per-VPN tunnels can also be carried as attributes of the routes. This makes use of the BGP protocol's ability to effectively carry large amounts of routing information. It is also possible to use BGP to distribute just the membership/capability information, while using a different technique to distribute the routing. BGP's update message would be used to indicate that a PE is attached to a particular VPN; BGP's withdraw message would be used to indicate that a PE has ceased to be attached to a particular VPN. This makes use of the BGP protocol's ability to dynamically distribute real-time changes in a reliable and fairly rapid manner. In addition, if a BGP route reflector is used, PEs never have to be provisioned with each other's IP addresses at all. Both cases make use of BGP's mechanisms, such as route filters, for constraining the distribution of information. Augmented routing may be done in combination with aggregated routing, as discussed in section 4.4.4. Of course, when using BGP for distributing any kind of VPN-specific information, one must ensure Callon & Suzuki Informational [Page 36] RFC 4110 A Framework for L3 PPVPNs July 2005 that one is not disrupting the classical use of BGP for distributing public Internet routing information. For further discussion of this, see the discussion of aggregated routing, section 4.4.4. 4.2.1.4. VPN Discovery for Inter-SP VPNs When two sites of a VPN are connected to different SP networks, the SPs must support a common mechanism for exchanging membership/capability information. This might make use of manual configuration or automated exchange of information between the SPs. Automated exchange may be facilitated if one or more mechanisms for VPN discovery are standardized and supported across the multiple SPs. Inter-SP trust relationships will need to be established, for example to determine which information and how much information about the VPNs may be exchanged between SPs. In some cases different service providers may deploy different approaches for VPN discovery. Where this occurs, this implies that for multi-SP VPNs, some manual coordination and configuration may be necessary. The amount of information which needs to be shared between SPs may vary greatly depending upon the number of size of the multi-SP VPNs. The SPs will therefore need to determine and agree upon the expected amount of membership information to be exchanged, and the dynamic nature of this information. Mechanisms may also be needed to authenticate the VPN membership information. VPN information should be distributed only to places where it needs to go, whether that is intra-provider or inter-provider. In this way, the distribution of VPN information is unlike the distribution of inter-provider routing information, as the latter needs to be distributed throughout the Internet. In addition, the joint support of a VPN by two SPs should not require any third SP to maintain state for that VPN. Again, notice the difference with respect to inter-provider routing; in inter-provider routing: sending traffic from one SP to another may indeed require routing state in a third SP. As one possible example: Suppose that there are two SPs A and C, which want to support a common VPN. Suppose that A and C are interconnected via SP B. In this case B will need to know how to route traffic between A and C, and therefore will need to know something about A and C (such as enough routing information to forward IP traffic and/or connect MPLS LSPs between PEs or route reflectors in A and C). However, for scaling purposes it is desirable that B not need to know VPN-specific information about the VPNs which are supported by A and C. Callon & Suzuki Informational [Page 37] RFC 4110 A Framework for L3 PPVPNs July 2005 4.2.2. Constraining Distribution of VPN Routing Information In layer 3 provider-provisioned CE-based VPNs, the VPN tunnels connect CE devices. In this case, distribution of IP routing information occurs between CE devices on the customer sites. No additional constraints on the distribution of VPN routing information are necessary. In layer 3 PE-based VPNs, however, the PE devices must be aware of VPN routing information (for the VPNs to which they are attached). For scalability reasons, one does not want a scheme in which all PEs contain all routes for all VPNs. Rather, only the PEs that are attached to sites in a given VPN should contain the routing information for that VPN. This means that the distribution of VPN routing information between PE devices must be constrained. As VPN membership may change dynamically, it is necessary to have a mechanism that allows VPN route information to be distributed to any PE where there is an attached user for that VPN, and allows for the removal of this information when it is no longer needed. In the Virtual Router scheme, per-VPN tunnels must be established before any routes for a VPN are distributed, and the routes are then distributed through those tunnels. Thus by establishing the proper set of tunnels, one implicitly constrains and controls the distribution of per-VPN routing information. In this scheme, the distribution of membership information consists of the set of VPNs that exists on each PE, as well as information about the desired topology. This enables a PE to determine the set of remote PEs to which it must establish tunnels for a particular VPN. In the aggregated routing scheme (see section 4.4.4), the distribution of VPN routing information is constrained by means of route filtering. As VPN membership changes on a PE, the route filters in use between the PE and its peers can be adjusted. Each peer may then adjust the filters in use with each of its peers in turn, and thus the changes propagate across the network. When BGP is used, this filtering may take place at route reflectors as discussed in section 4.4.4. 4.2.3. Controlling VPN Topology The topology for a VPN consists of a set of nodes interconnected via tunnels. The topology may be a full mesh, a hub and spoke topology, or an arbitrary topology. For a VPN the set of nodes will include all VPN edge devices that have attached sites for that VPN. Naturally, whatever the topology, all VPN sites are reachable from each other; the topology simply constrains the way traffic is routed Callon & Suzuki Informational [Page 38] RFC 4110 A Framework for L3 PPVPNs July 2005 among the sites. For example, in one topology traffic between site A and site B goes from one to the other directly over the VPN backbone; in another topology, traffic from site A to site B must traverse site C before reaching site B. The simplest topology is a full mesh, where a tunnel exists between every pair of VPN edge devices. If we assume the use of point-to- point tunnels (rather than multipoint-to-point), then with a full mesh topology there are N*(N-1)/2 duplex tunnels or N*(N-1) simplex tunnels for N VPN edge devices. Each tunnel consumes some resources at a VPN edge device, and depending on the type of tunnel, may or may not consume resources in intermediate routers or LSRs. One reason for using a partial mesh topology is to reduce the number of tunnels a VPN edge device, and/or the network, needs to support. Another reason is to support the scenario where an administrator requires all traffic from certain sites to traverse some particular site for policy or control reasons, such as to force traffic through a firewall, or for monitoring or accounting purposes. Note that the topologies used for each VPN are separate, and thus the same VPN edge device may be part of a full mesh topology for one VPN, and of a partial mesh topology for another VPN. An example of where a partial mesh topology could be suitable is for a VPN that supports a large number of telecommuters and a small number of corporate sites. Most traffic will be between telecommuters and the corporate sites, not between pairs of telecommuters. A hub and spoke topology for the VPN would thus map onto the underlying traffic flow, with the telecommuters attached to spoke VPN edge devices and the corporate sites attached to hub VPN edge devices. Traffic between telecommuters is still supported, but this traffic traverses a hub VPN edge device. The selection of a topology for a VPN is an administrative choice, but it is useful to examine protocol mechanisms that can be used to automate the construction of the desired topology, and thus reduce the amount of configuration needed. To this end it is useful for a VPN edge device to be able to advertise per-VPN topology information to other VPN edge devices. It may be simplest to advertise this at the same time as the membership information is advertised, using the same mechanisms. A simple scheme is where a VPN edge device advertises itself either as a hub or as a spoke, for each VPN that it has. When received by other VPN edge devices this information can be used when determining whether to establish a tunnel. A more comprehensive scheme allows a VPN edge device to advertise a set of topology groups, with tunnels established between a pair of VPN edge devices if they have a group in common. Callon & Suzuki Informational [Page 39] RFC 4110 A Framework for L3 PPVPNs July 2005 4.3. VPN Tunneling VPN solutions use tunneling in order to transport VPN packets across the VPN backbone, from one VPN edge device to another. There are different types of tunneling protocols, different ways of establishing and maintaining tunnels, and different ways to associate tunnels with VPNs (e.g., shared versus dedicated per-VPN tunnels). Sections 4.3.1 through 4.3.5 discusses some common characteristics shared by all forms of tunneling, and some common problems to which tunnels provide a solution. Section 4.3.6 provides a survey of available tunneling techniques. Note that tunneling protocol issues are generally independent of the mechanisms used for VPN membership and VPN routing. One motivation for the use of tunneling is that the packet addressing used in a VPN may have no relation to the packet addressing used between the VPN edge devices. For example the customer VPN traffic could use non-unique or private IP addressing [RFC1918]. Also an IPv6 VPN could be implemented across an IPv4 provider backbone. As such the packet forwarding between the VPN edge devices must use information other than that contained in the VPN packets themselves. A tunneling protocol adds additional information, such an extra header or label, to a VPN packet, and this additional information is then used for forwarding the packet between the VPN edge devices. Another capability optionally provided by tunneling is that of isolation between different VPN traffic flows. The QoS and security requirements for these traffic flows may differ, and can be met by using different tunnels with the appropriate characteristics. This allows a provider to offer different service characteristics for traffic in different VPNs, or to subsets of traffic flows within a single VPN. The specific tunneling protocols considered in this section are GRE, IP-in-IP, IPsec, and MPLS, as these are the most suitable for carrying VPN traffic across the VPN backbone. Other tunneling protocols, such as L2TP [RFC2661], may be used as access tunnels, carrying traffic between a PE and a CE. As backbone tunneling is independent of and orthogonal to access tunneling, protocols for the latter are not discussed here. 4.3.1. Tunnel Encapsulations All tunneling protocols use an encapsulation that adds additional information to the encapsulated packet; this information is used for forwarding across the VPN backbone. Examples are provided in section 4.3.6. Callon & Suzuki Informational [Page 40] RFC 4110 A Framework for L3 PPVPNs July 2005 One characteristic of a tunneling protocol is whether per-tunnel state is needed in the SP network in order to forward the encapsulated packets. For IP tunneling schemes (GRE, IP-in-IP, and IPsec) per-tunnel state is completely confined to the VPN edge devices. Other routers are unaware of the tunnels, and forward according to the IP header. For MPLS, per-tunnel state is needed, since the top label in the label stack must be examined and swapped by intermediate LSRs. The amount of state required can be minimized by hierarchical multiplexing, and by use of multi-point to point tunnels, as discussed below. Another characteristic is the tunneling overhead introduced. With IPsec the overhead may be considerable as it may include, for example, an ESP header, ESP trailer and an additional IP header. The other mechanisms listed use less overhead, with MPLS being the most lightweight. The overhead inherent in any tunneling mechanism may result in additional IP packet fragmentation, if the resulting packet is too large to be carried by the underlying link layer. As such it is important to report any reduced MTU sizes via mechanisms such as path MTU discovery in order to avoid fragmentation wherever possible. Yet another characteristic is something we might call "transparency to the Internet". IP-based encapsulation can carry be used to carry a packet anywhere in the Internet. MPLS encapsulation can only be used to carry a packet on IP networks that support MPLS. If an MPLS-encapsulated packet must cross the networks of multiple SPs, the adjacent SPs must bilateral agreements to accept MPLS packets from each other. If only a portion of the path across the backbone lacks MPLS support, then an MPLS-in-IP encapsulation can be used to move the MPLS packets across that part of the backbone. However, this does add complexity. On the other hand, MPLS has efficiency advantages, particularly in environments where encapsulations may need to be nested. Transparency to the Internet is sometimes a requirement, but sometimes not. This depends on the sort of service which a SP is offering to its customer. 4.3.2. Tunnel Multiplexing When a tunneled packet arrives at the tunnel egress, it must be possible to infer the packet's VPN from its encapsulation header. In MPLS encapsulations, this must be inferred from the packet's label stack. In IP-based encapsulations, this can be inferred from some combination of the IP source address, the IP destination address, and a "multiplexing field" in the encapsulation header. The multiplexing Callon & Suzuki Informational [Page 41] RFC 4110 A Framework for L3 PPVPNs July 2005 field might be one which was explicitly designed for multiplexing, or one that wasn't originally designed for this but can be pushed into service as a multiplexing field. For example: o GRE: Packets associated to VPN by source IP address, destination IP address, and Key field, although the key field was originally intended for authentication. o IP-in-IP: Packets associated to VPN by IP destination address in outer header. o IPsec: Packets associated to VPN by IP source address, IP destination address, and SPI field. o MPLS: Packets associated to VPN by label stack. Note that IP-in-IP tunneling does not have a real multiplexing field, so a different IP destination address must be used for every VPN supported by a given PE. In the other IP-based encapsulations, a given PE need have only a single IP address, and the multiplexing field is used to distinguish the different VPNs supported by a PE. Thus the IP-in-IP solution has the significant disadvantage that it requires the allocation and assignment of a potentially large number of IP addresses, all of which have to be reachable via backbone routing. In the following, we will use the term "multiplexing field" to refer to whichever field in the encapsulation header must is used to distinguish different VPNs at a given PE. In the IP-in-IP encapsulation, this is the destination IP address field, in the other encapsulations it is a true multiplexing field. 4.3.3. Tunnel Establishment When tunnels are established, the tunnel endpoints must agree on the multiplexing field values which are to be used to indicate that particular packets are in particular VPNs. The use of "well known" or explicitly provisioned values would not scale well as the number of VPNs increases. So it is necessary to have some sort of protocol interaction in which the tunnel endpoints agree on the multiplexing field values. For some tunneling protocols, setting up a tunnel requires an explicit exchange of signaling messages. Generally the multiplexing field values would be agreed upon as part of this exchange. For example, if an IPsec encapsulation is used, the SPI field plays the role of the multiplexing field, and IKE signaling is used to distribute the SPI values; if an MPLS encapsulation is used, LDP, Callon & Suzuki Informational [Page 42] RFC 4110 A Framework for L3 PPVPNs July 2005 CR-LDP or RSVP-TE can be used to distribute the MPLS label value used as the multiplexing field. Information about the identity of the VPN with which the tunnel is to be associated needs to be exchanged as part of the signaling protocol (e.g., a VPN-ID can be carried in the signaling protocol). An advantage of this approach is that per-tunnel security, QoS and other characteristics may also be negotiable via the signaling protocol. A disadvantage is that the signaling imposes overhead, which may then lead to scalability considerations, discussed further below. For some tunneling protocols, there is no explicit protocol interaction that sets up the tunnel, and the multiplexing field values must be exchanged in some other way. For example, for MPLS tunnels, MPLS labels can be piggybacked on the protocols used to distribute VPN routes or VPN membership information. GRE and IP-in-IP have no associated signaling protocol, and thus by necessity the multiplexing values are distributed via some other mechanism, such as via configuration, control protocol, or piggybacked in some manner on a VPN membership protocol. The resources used by the different tunneling establishment mechanisms may vary. With a full mesh VPN topology, and explicit signaling, each VPN edge device has to establish a tunnel to all the other VPN edge devices for in each VPN. The resources needed for this on a VPN edge device may be significant, and issues such as the time needed to recover following a device failure may need to be taken into account, as the time to recovery includes the time needed to reestablish a large number of tunnels. 4.3.4. Scaling and Hierarchical Tunnels If tunnels require state to be maintained in the core of the network, it may not be feasible to set up per-VPN tunnels between all adjacent devices that are adjacent in some VPN topology. This would violate the principle that there is no per-VPN state in the core of the network, and would make the core scale poorly as the number of VPNs increases. For example, MPLS tunnels require that core network devices maintain state for the topmost label in the label stack. If every core router had to maintain one or more labels for every VPN, scaling would be very poor. There are also scaling considerations related to the use of explicit signaling for tunnel establishment. Even if the tunneling protocol does not maintain per tunnel state in the core, the number of tunnels that a single VPN edge device needs to handle may be large, as this grows according to the number of VPNs and the number of neighbors per VPN. One way to reduce the number of tunnels in a network is to use Callon & Suzuki Informational [Page 43] RFC 4110 A Framework for L3 PPVPNs July 2005 a VPN topology other than a full mesh. However this may not always be desirable, and even with hub and spoke topologies the hubs VPN edge devices may still need to handle large numbers of tunnels. If the core routers need to maintain any per-tunnel state at all, scaling can be greatly improved by using hierarchical tunnels. One tunnel can be established between each pair of VPN edge devices, and multiple VPN-specific tunnels can then be carried through the single "outer" tunnel. Now the amount of state is dependent only on the number of VPN edge devices, not on the number of VPNs. Scaling can be further improved by having the outer tunnels be multipoint-to-point "merging" tunnels. Now the amount of state to be maintained in the core is on the order of the number of VPN edge devices, not on the order of the square of that number. That is, the amount of tunnel state is roughly equivalent to the amount of state needed to maintain IP routes to the VPN edge devices. This is almost (if not quite) as good as using tunnels which do not require any state to be maintained in the core. Using hierarchical tunnels may also reduce the amount of state to be maintained in the VPN edge devices, particularly if maintaining the outer tunnels requires more state than maintaining the per-VPN tunnels that run inside the outer tunnels. There are other factors relevant to determining the number of VPN edge to VPN edge "outer" tunnels to use. While using a single such tunnel has the best scaling properties, using more than one may allow different QoS capabilities or different security characteristics to be used for different traffic flows (from the same or from different VPNs). When tunnels are used hierarchically, the tunnels in the hierarchy may all be of the same type (e.g., an MPLS label stack) or they may be of different types (e.g., a GRE tunnel carried inside an IPsec tunnel). One example using hierarchical tunnels is the establishment of a number of different IPsec security associations, providing different levels of security between a given pair of VPN edge devices. Per-VPN GRE tunnels can then be grouped together and then carried over the appropriate IPsec tunnel, rather than having a separate IPsec tunnel per-VPN. Another example is the use of an MPLS label stack. A single PE-PE LSP is used to carry all the per-VPN LSPs. The mechanisms used for label establishment are typically different. The PE-PE LSP could be established using LDP, as part or normal backbone operation, with the per-VPN LSP labels established by piggybacking on VPN routing (e.g., using BGP) discussed in sections 3.3.1.3 and 4.1. Callon & Suzuki Informational [Page 44] RFC 4110 A Framework for L3 PPVPNs July 2005 4.3.5. Tunnel Maintenance Once a tunnel is established it is necessary to know that the tunnel is operational. Mechanisms are needed to detect tunnel failures, and to respond appropriately to restore service. There is a potential issue regarding propagation of failures when multiple tunnels are multiplexed hierarchically. Suppose that multiple VPN-specific tunnels are multiplexed inside a single PE to PE tunnel. In this case, suppose that routing for the VPN is done over the VPN-specific tunnels (as may be the case for CE-based and VR approaches). Suppose that the PE to PE tunnel fails. In this case multiple VPN-specific tunnels may fail, and layer 3 routing may simultaneously respond for each VPN using the failed tunnel. If the PE to PE tunnel is subsequently restored, there may then be multiple VPN-specific tunnels and multiple routing protocol instances which also need to recover. Each of these could potentially require some exchange of control traffic. When a tunnel fails, if the tunnel can be restored quickly, it might therefore be preferable to restore the tunnel without any response by high levels (such as other tunnels which were multiplexed inside the failed tunnels). By having high levels delay response to a lower level failed tunnel, this may limit the amount of control traffic needed to completely restore correct service. However, if the failed tunnel cannot be quickly restored, then it is necessary for the tunnels or routing instances multiplexed over the failed tunnel to respond, and preferable for them to respond quickly and without explicit action by network operators. With most layer 3 provider-provisioned CE-based VPNs and the VR scheme, a per-VPN instance of routing is running over the tunnel, thus any loss of connectivity between the tunnel endpoints will be detected by the VPN routing instance. This allows rapid detection of tunnel failure. Careful adjustment of timers might be needed to avoid failure propagation as discussed the above. With the aggregated routing scheme, there isn't a per-VPN instance of routing running over the tunnel, and therefore some other scheme to detect loss of connectivity is needed in the event that the tunnel cannot be rapidly restored. Failure of connectivity in a tunnel can be very difficult to detect reliably. Among the mechanisms that can be used to detect failure are loss of the underlying connectivity to the remote endpoint (as indicated, e.g., by "no IP route to host" or no MPLS label), timeout of higher layer "hello" mechanisms (e.g., IGP hellos, when the tunnel is an adjacency in some IGP), and timeout of keep alive mechanisms in Callon & Suzuki Informational [Page 45] RFC 4110 A Framework for L3 PPVPNs July 2005 the tunnel establishment protocols (if any). However, none of these techniques provides completely reliable detection of all failure modes. Additional monitoring techniques may also be necessary. With hierarchical tunnels it may suffice to only monitor the outermost tunnel for loss of connectivity. However there may be failure modes in a device where the outermost tunnel is up but one of the inner tunnels is down. 4.3.6. Survey of Tunneling Techniques Tunneling mechanisms provide isolated communication between two CE-PE devices. Available tunneling mechanisms include (but are not limited to): GRE [RFC2784] [RFC2890], IP-in-IP encapsulation [RFC2003] [RFC2473], IPsec [RFC2401] [RFC2402], and MPLS [RFC3031] [RFC3035]. Note that the following subsections address tunnel overhead to clarify the risk of fragmentation. Some SP networks contain layer 2 switches that enforce the standard/default MTU of 1500 bytes. In this case, any encapsulation whatsoever creates a significant risk of fragmentation. However, layer 2 switch vendors are in general aware of IP tunneling as well as stacked VLAN overhead, thus many switches practically allow an MTU of approximately 1512 bytes now. In this case, up to 12 bytes of encapsulation can be used before there is any risk of fragmentation. Furthermore, to improve TCP and NFS performance, switches that support 9K bytes "jumbo frames" are also on the market. In this case, there is no risk of fragmentation. 4.3.6.1. GRE [RFC2784] [RFC2890] Generic Routing Encapsulation (GRE) specifies a protocol for encapsulating an arbitrary payload protocol over an arbitrary delivery protocol [RFC2784]. In particular, it can be used where both the payload and the delivery protocol are IP as is the case in layer 3 VPNs. A GRE tunnel is a tunnel whose packets are encapsulated by GRE. o Multiplexing The GRE specification [RFC2784] does not explicitly support multiplexing. But the key field extension to GRE is specified in [RFC2890] and it may be used as a multiplexing field. Callon & Suzuki Informational [Page 46] RFC 4110 A Framework for L3 PPVPNs July 2005 o QoS/SLA GRE itself does not have intrinsic QoS/SLA capabilities, but it inherits whatever capabilities exist in the delivery protocol (IP). Additional mechanisms, such as Diffserv or RSVP extensions [RFC2746], can be applied. o Tunnel setup and maintenance There is no standard signaling protocol for setting up and maintaining GRE tunnels. o Large MTUs and minimization of tunnel overhead When GRE encapsulation is used, the resulting packet consists of a delivery protocol header, followed by a GRE header, followed by the payload packet. When the delivery protocol is IPv4, and if the key field is not present, GRE encapsulation adds at least 28 bytes of overhead (36 bytes if key field extension is used.) o Security GRE encapsulation does not provide any significant security. The optional key field can be used as a clear text password to aid in the detection of misconfigurations, but it does not provide integrity or authentication. An SP network which supports VPNs must do extensive IP address filtering at its borders to prevent spoofed packets from penetrating the VPNs. If multi-provider VPNs are being supported, it may be difficult to set up these filters. 4.3.6.2. IP-in-IP Encapsulation [RFC2003] [RFC2473] IP-in-IP specifies the format and procedures for IP-in-IP encapsulation. This allows an IP datagram to be encapsulated within another IP datagram. That is, the resulting packet consists of an outer IP header, followed immediately by the payload packet. There is no intermediate header as in GRE. [RFC2003] and [RFC2473] specify IPv4 and IPv6 encapsulations respectively. Once the encapsulated datagram arrives at the intermediate destination (as specified in the outer IP header), it is decapsulated, yielding the original IP datagram, which is then delivered to the destination indicated by the original destination address field. Callon & Suzuki Informational [Page 47] RFC 4110 A Framework for L3 PPVPNs July 2005 o Multiplexing The IP-in-IP specifications don't explicitly support multiplexing. But if a different IP address is used for every VPN then the IP address field can be used for this purpose. (See section 4.3.2 for detail). o QoS/SLA IP-in-IP itself does not have intrinsic QoS/SLA capabilities, but of course it inherits whatever capabilities exist for IP. Additional mechanisms, such as RSVP extensions [RFC2764] or DiffServ extensions [RFC2983], may be used with it. o Tunnel setup and maintenance There is no standard setup and maintenance protocol for IP-in-IP. o Large MTUs and minimization of tunnel overhead When the delivery protocol is IPv4, IP-in-IP adds at least 20 bytes of overhead. o Security IP-in-IP encapsulation does not provide any significant security. An SP network which supports VPNs must do extensive IP address filtering at its borders to prevent spoofed packets from penetrating the VPNs. If multi-provider VPNs are being supported, it may be difficult to set up these filters. 4.3.6.3. IPsec [RFC2401] [RFC2402] [RFC2406] [RFC2409] IP Security (IPsec) provides security services at the IP layer [RFC2401]. It comprises authentication header (AH) protocol [RFC2402], encapsulating security payload (ESP) protocol [RFC2406], and Internet key exchange (IKE) protocol [RFC2409]. AH protocol provides data integrity, data origin authentication, and an anti-replay service. ESP protocol provides data confidentiality and limited traffic flow confidentiality. It may also provide data integrity, data origin authentication, and an anti-replay service. AH and ESP may be used in combination. IPsec may be employed in either transport or tunnel mode. In transport mode, either an AH or ESP header is inserted immediately after the payload packet's IP header. In tunnel mode, an IP packet is encapsulated with an outer IP packet header. Either an AH or ESP header is inserted between them. AH and ESP establish a Callon & Suzuki Informational [Page 48] RFC 4110 A Framework for L3 PPVPNs July 2005 unidirectional secure communication path between two endpoints, which is called a security association. In tunnel mode, PE-PE tunnel (or a CE-CE tunnel) consists of a pair of unidirectional security associations. The IPsec and IKE protocols are used for setting up IPsec tunnels. o Multiplexing The SPI field of AH and ESP is used to multiplex security associations (or tunnels) between two peer devices. o QoS/SLA IPsec itself does not have intrinsic QoS/SLA capabilities, but it inherits whatever mechanisms exist for IP. Other mechanisms such as "RSVP Extensions for IPsec Data Flows" [RFC2207] or DiffServ extensions [RFC2983] may be used with it. o Tunnel setup and maintenance The IPsec and IKE protocols are used for the setup and maintenance of tunnels. o Large MTUs and minimization of tunnel overhead IPsec transport mode adds at least 8 bytes of overhead. IPsec tunnel mode adds at least 28 bytes of overhead. IPsec transport mode adds minimal overhead. In PE-based PPVPNs, the processing overhead of IPsec (due to its cryptography) may limit the PE's performance, especially if privacy is being provided; this is not generally an issue in CE-based PPVPNs. o Security When IPsec tunneling is used in conjunction with IPsec's cryptographic capabilities, excellent authentication and integrity functions can be provided. Privacy can also be optionally provided. 4.3.6.4. MPLS [RFC3031] [RFC3032] [RFC3035] Multiprotocol Label Switching (MPLS) is a method for forwarding packets through a network. Routers at the edge of a network apply simple labels to packets. A label may be inserted between the data link and network headers, or may be carried in the data link header (e.g., the VPI/VCI field in an ATM header). Routers in the network Callon & Suzuki Informational [Page 49] RFC 4110 A Framework for L3 PPVPNs July 2005 switch packets according to the labels, with minimal lookup overhead. A path, or a tunnel in the PPVPN, is called a "label switched path (LSP)". o Multiplexing LSPs may be multiplexed within other LSPs. o QoS/SLA MPLS does not have intrinsic QoS or SLA management mechanisms, but bandwidth may be allocated to LSPs, and their routing may be explicitly controlled. Additional techniques such as DiffServ and DiffServ aware traffic engineering may be used with it [RFC3270] [MPLS-DIFF-TE]. QoS capabilities from IP may be inherited. o Tunnel setup and maintenance LSPs are set up and maintained by LDP (Label Distribution Protocol), RSVP (Resource Reservation Protocol) [RFC3209], or BGP. o Large MTUs and minimization of tunnel overhead. MPLS encapsulation adds four bytes per label. VPN-2547BIS's [VPN-2547BIS] approach uses at least two labels for encapsulation and adds minimal overhead. o Encapsulation MPLS packets may optionally be encapsulated in IP or GRE, for cases where it is desirable to carry MPLS packets over an IP-only infrastructure. o Security MPLS encapsulation does not provide any significant security. An SP which is providing VPN service can refuse to accept MPLS packets from outside its borders. This provides the same level of assurance as would be obtained via IP address filtering when IP-based encapsulations are used. If a VPN is jointly provided by multiple SPs, care should be taken to ensure that a labeled packet is accepted from a neighboring router in another SP only if its top label is one which was actually distributed to that router. Callon & Suzuki Informational [Page 50] RFC 4110 A Framework for L3 PPVPNs July 2005 o Applicability MPLS is the only one of the encapsulation techniques that cannot be guaranteed to run over any IP network. Hence it would not be applicable when transparency to the Internet is a requirement. If the VPN backbone consists of several cooperating SP networks which support MPLS, then the adjacent networks may support MPLS at their interconnects. If two cooperating SP networks which support MPLS are separated by a third which does not support MPLS, then MPLS-in-IP or MPLS-in-IPsec tunneling may be done between them. 4.4. PE-PE Distribution of VPN Routing Information In layer 3 PE-based VPNs, PE devices examine the IP headers of packets they receive from the customer networks. Forwarding is based on routing information received from the customer network. This implies that the PE devices need to participate in some manner in routing for the customer network. Section 3.3 discussed how routing would be done in the customer network, including the customer interface. In this section, we discuss ways in which the routing information from a particular VPN may be passed, over the shared VPN backbone, among the set of PEs attaching to that VPN. The PEs needs to distribute two types of routing information to each other: (i) Public Routing: routing information which specifies how to reach addresses on the VPN backbone (i.e., "public addresses"); call this "public routing information" (ii) VPN Routing: routing information obtained from the CEs, which specifies how to reach addresses ("private addresses") that are in the VPNs. The way in which routing information in the first category is distributed is outside the scope of this document; we discuss only the distribution of routing information in the second category. Of course, one of the requirements for distributing VPN routing information is that it be kept separate and distinct from the public information. Another requirement is that the distribution of VPN routing information not destabilize or otherwise interfere with the distribution of public routing information. Similarly, distribution of VPN routing information associated with one VPN should not destabilize or otherwise interfere with the operation of other VPNs. These requirements are, for example, relevant in the case that a private network might be suffering from instability or other problems with its internal routing, which might be propagated to the VPN used to support that private network. Callon & Suzuki Informational [Page 51] RFC 4110 A Framework for L3 PPVPNs July 2005 Note that this issue does not arise in CE-based VPNs, as in CE-based VPNs, the PE devices do not see packets from the VPN until after the packets haven been encapsulated in an outer header that has only public addresses. 4.4.1. Options for VPN Routing in the SP The following technologies can be used for exchanging VPN routing information discussed in sections 3.3.1.3 and 4.1. o Static routing o RIP [RFC2453] o OSPF [RFC2328] o BGP-4 [RFC1771] 4.4.2. VPN Forwarding Instances (VFIs) In layer 3 PE-based VPNs, the PE devices receive unencapsulated IP packets from the CE devices, and the PE devices use the IP destination addresses in these packets to help make their forwarding decisions. In order to do this properly, the PE devices must obtain routing information from the customer networks. This implies that the PE device participates in some manner in the customer network's routing. In layer 3 PE-based VPNs, a single PE device connected to several CE devices that are in the same VPN, and it