< draft-fang-l3vpn-virtual-pe-framework-00.txt   draft-fang-l3vpn-virtual-pe-framework-01.txt >
INTERNET-DRAFT Luyuan Fang INTERNET-DRAFT Luyuan Fang
Intended Status: Informational Rex Fernando Intended Status: Informational David Ward
Expires: April 15, 2013 Cisco Expires: April 22, 2013 Rex Fernando
Cisco
Maria Napierala Maria Napierala
AT&T AT&T
Nabil Bitar
Verizon
Dhananjaya Rao
Cisco
October 15, 2012 October 22, 2012
BGP L3VPN Virtual PE Framework BGP L3VPN Virtual PE Framework
draft-fang-l3vpn-virtual-pe-framework-00 draft-fang-l3vpn-virtual-pe-framework-01
Abstract Abstract
This document describes a framework for BGP/MPLS L3VPN with virtual This document describes a framework for BGP/MPLS L3VPN with virtual
PE solutions. It provides the information on control plane, data PE solutions. It provides functional description of the control plane
plane of the virtual PE solutions, and its interaction with other and data plane of the virtual PE solutions. It also describes
network elements. The solution supports further control and interactions among the vPE solutions and other network elements. The
forwarding separation from traditional L3VPN solutions. It allows the virtual PE solutions support further control plane and forwarding
L3VPN functions extended to application end systems for large scale plane separation when compared with traditional L3VPN PE solutions.
and efficient IP application support. It allows the L3VPN functions to be extended to application end
devices for large scale and efficient IP application support.
Status of this Memo Status of this Memo
This Internet-Draft is submitted to IETF in full conformance with the This Internet-Draft is submitted to IETF in full conformance with the
provisions of BCP 78 and BCP 79. provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as other groups may also distribute working documents as
Internet-Drafts. Internet-Drafts.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at The list of current Internet-Drafts can be accessed at
http://www.ietf.org/1id-abstracts.html http://www.ietf.org/1id-abstracts.html
INTERNET DRAFT <Document Title> <Issue Date>
The list of Internet-Draft Shadow Directories can be accessed at The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html http://www.ietf.org/shadow.html
Copyright and License Notice Copyright and License Notice
INTERNET DRAFT <Document Title> <Issue Date>
Copyright (c) 2012 IETF Trust and the persons identified as the Copyright (c) 2012 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License. described in the Simplified BSD License.
Table of Contents Table of Contents
1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.1 Terminology . . . . . . . . . . . . . . . . . . . . . . . . 4
1.2 Scope of the document . . . . . . . . . . . . . . . . . . . 3 1.2 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.3 Terminology . . . . . . . . . . . . . . . . . . . . . . . . 4 1.3 Scope of the document . . . . . . . . . . . . . . . . . . . 6
2. Virtual PE Architecture and Reference Model . . . . . . . . . . 4 2. Virtual PE Architecture and Reference Model . . . . . . . . . . 6
2.1 Virtual PE . . . . . . . . . . . . . . . . . . . . . . . . . 4 2.1 Virtual PE . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.2 Architecture reference model . . . . . . . . . . . . . . . . 5 2.2 Architecture reference model . . . . . . . . . . . . . . . . 7
3. Control Plane . . . . . . . . . . . . . . . . . . . . . . . . . 8 3. Control Plane . . . . . . . . . . . . . . . . . . . . . . . . . 10
3.1 Router server of vPE . . . . . . . . . . . . . . . . . . . . 9 3.1 vPE Control Plane . . . . . . . . . . . . . . . . . . . . . 10
3.2 Control plane options of vPE . . . . . . . . . . . . . . . . 9 3.1 Route server of vPE . . . . . . . . . . . . . . . . . . . . 11
3.3 Use of router reflector . . . . . . . . . . . . . . . . . . 10 3.3 Use of router reflector . . . . . . . . . . . . . . . . . . 11
3.4 Use of RT constraint . . . . . . . . . . . . . . . . . . . . 10 3.4 Use of RT constraint . . . . . . . . . . . . . . . . . . . . 12
4. Forwarding Plane . . . . . . . . . . . . . . . . . . . . . . . 10 4. Forwarding Plane . . . . . . . . . . . . . . . . . . . . . . . 12
4.1 Virtual Interface . . . . . . . . . . . . . . . . . . . . . 10 4.1 Virtual Interface . . . . . . . . . . . . . . . . . . . . . 12
4.2 VPN forwarder . . . . . . . . . . . . . . . . . . . . . . . 10 4.2 VPN forwarder . . . . . . . . . . . . . . . . . . . . . . . 12
4.3 Encapsulation . . . . . . . . . . . . . . . . . . . . . . . 11 4.3 Encapsulation . . . . . . . . . . . . . . . . . . . . . . . 12
4.4 Optimal forwarding . . . . . . . . . . . . . . . . . . . . . 11 4.4 Optimal forwarding . . . . . . . . . . . . . . . . . . . . . 13
5. Addressing . . . . . . . . . . . . . . . . . . . . . . . . . . 12 5. Addressing . . . . . . . . . . . . . . . . . . . . . . . . . . 14
5.1 IPv4 and IPv6 support . . . . . . . . . . . . . . . . . . . 12 5.1 IPv4 and IPv6 support . . . . . . . . . . . . . . . . . . . 14
5.2 Address space separation . . . . . . . . . . . . . . . . . . 12 5.2 Address space separation . . . . . . . . . . . . . . . . . . 14
7. Security Considerations . . . . . . . . . . . . . . . . . . . 12 6.0 Inter-connection considerations . . . . . . . . . . . . . . 14
8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 12 7. Security Considerations . . . . . . . . . . . . . . . . . . . 15
9. References . . . . . . . . . . . . . . . . . . . . . . . . . . 12 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 15
9.1 Normative References . . . . . . . . . . . . . . . . . . . 12 9. References . . . . . . . . . . . . . . . . . . . . . . . . . . 15
9.2 Informative References . . . . . . . . . . . . . . . . . . 13 9.1 Normative References . . . . . . . . . . . . . . . . . . . 15
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 13
INTERNET DRAFT <Document Title> <Issue Date> INTERNET DRAFT <Document Title> <Issue Date>
1 Introduction 9.2 Informative References . . . . . . . . . . . . . . . . . . 16
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 16
Network virtualization is to provide multiple individual network
services by sharing common available network resources. Network
virtualization is not a new concept, for example, BGP layer 3 Virtual
Private Networks (L3 VPNs) have been widely deployed to provide
network based, service provider provisioned VPNs. It provides routing
isolation and allow address overlapping among different VPNs while
forward traffic over common network infrastructure.
The recent development of server virtulization, provided the new
opportunities for the virtual Provider Edge (vPE) solution on
application end-system. The virtual PE solution can be powerful and
attractive to service providers and enterprises in the fast growing
cloud computing service and intelligent mobility environment.
1.1 Overview
L3 VPN Virtual Provider Edge may provide full or partial L3VPN PE
functions or partial PE functions. The virtual PE has two components
- control and forwarding. The control component can be a software
instance resides in a physical device, most commonly seen in the end-
system devices where multiple applications are supported, e.g.,
mobile application server in a wireless call center, or an end-system
in a SP virtual Private Cloud (vPC) data center, or a host in an
Financial back-office.
The architecture and protocols defined in IETF for BGP/MPLS L3VPN
[RFC4364] is the foundation for virtual PE solution. Certain protocol
extensions or integration may be needed to support the virtual PE
solutions.
This document defines a framework for using standard protocols to INTERNET DRAFT <Document Title> <Issue Date>
build BGP L3 VPN with virtual PE solutions. The goal is to support
large scale deployment and reduce operational complexity. The
targeted environment can be virtualized wireless providers call
centers, large scale service providers data centers, large enterprise
central and branch facilities, and service provider managed
services.
1.2 Scope of the document 1 Introduction
This focus of this document is BGP L3VPN virtual PE solutions. Network virtualization is to provide multiple individual network
services through shared common network resources. Network
virtualization is not a new concept. For example, BGP/MPLS layer 3
Virtual Private Networks (L3VPNs) [RFC4364] have been widely deployed
to provide network based virtual private network services. It
provides routing isolation and forwarding separation for individual
VPNs, allow IP address overlapping among different VPNs while
forwarding traffic over common network infrastructure.
It is assumed that the readers are familiar with BGP/MPLS L3VPN Network virtualization enables the support of multiple isolated
technologies, the base technology and operation will not be covered individual networks over a common network infrastructure. Network
in this document. virtualization is not a new concept. For example, BGP/MPLS IP Virtual
Private Network (IP VPNs) [RFC4364] have been widely deployed to
provide network based, service provider provisioned IP VPNs for
multiple customers with overlapping IP address spaces over a common
service provider IP/MPLS network. BGP/MPLS IPVPNs provide routing
isolation among customers and allow address overlapping among
different VPNs by having per-customer Virtual Routing and Forwarding
Instance (VRF) at a service provided Edge (PE), while forwarding
customer traffic over a common IP/MPLS network infrastructure.
INTERNET DRAFT <Document Title> <Issue Date> With the advent of compute capabilities and the proliferation of
virtualization in end devices for systems and applications, PE
functionality virtualization on such end device is becoming feasible,
and in some cases attractive for scale and efficiency. Scale and
efficiency are crutial factors in the cloud computing environment
supporting various applications and services, and in traditional
service provider space.
The following network elements are in scope: BGP L3VPN vPE; the The virtual Provider Edge (vPE) solution described in this document
interaction of vPE with other network elements, including BGP L3VPN is to extend the functionality of BGP/MPLS L3VPN to the application
physical PE, physical BGP Route Reflector (RR) and virtual BGP Route end device.
Reflector (vRR), and Autonomic System Border Router (ASBR), external
controller, provisioning/orchestration systems. Definitions of
protocol extensions is out of scope of this document.
1.3 Terminology 1.1 Terminology
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC 2119 [RFC2119]. document are to be interpreted as described in RFC 2119 [RFC2119].
Term Definition
----------- --------------------------------------------------
3GPP 3rd Generation Partnership Project (3GPP)
AS Autonomous Systems AS Autonomous Systems
ASBR Autonomous Systems Border Router ASBR Autonomous Systems Border Router
INTERNET DRAFT <Document Title> <Issue Date>
BGP Border Gateway Protocol BGP Border Gateway Protocol
End-Device A device where Guest OS and Host OS/Hypervisor ED End device: where Guest OS, Host OS/Hypervisor,
may reside, aka End-system applications, VMs, and virtual router may reside
Forwarder L3VPN forwarding function
GRE Generic Routing Encapsulation GRE Generic Routing Encapsulation
IaaS Infrastructure as a Service IaaS Infrastructure as a Service
IRS Interface to Routing System IRS Interface to Routing System
LTE Long Term Evolution LTE Long Term Evolution
MP-BGP Multi-Protocol Border Gateway Protocol
PCEF Policy Charging and Enforcement Function PCEF Policy Charging and Enforcement Function
P Provider backbone router
RR Route Reflector RR Route Reflector
RT Route Target RT Route Target
RTC RT Constraint RTC RT Constraint
ToR Top-of-Rack switch ToR Top-of-Rack switch
VM Virtual Machine VM Virtual Machine
Hypervisor Virtual Machine Manager Hypervisor Virtual Machine Manager
VM Virtual Machine VM Virtual Machine
SDN Software Defined Network SDN Software Defined Network
VI Virtual Interface VI Virtual Interface
vCE virtual Customer Router
vPC virtual Private Cloud vPC virtual Private Cloud
vPE virtual Provider Edge vPE virtual Provider Edge
VPN Virtual Private Network VPN Virtual Private Network
vRR virtual Route Reflector vRR virtual Route Reflector1.2 Scope of the document
WAN Wide Area Network
2. Virtual PE Architecture and Reference Model Virtual PE is a PE resides in an end device (e.g., a server) along
with client/application VMs.
2.1 Virtual PE Through out this document, the term virtual PE (vPE) is used to
denote BGP/MPLS L3VPN virtual Provider Edge router.
A virtual PE is a PE with control instance and a forwarding 1.2 Motivation
components reside in a shared physical device where multiple
applications are supported. The control and forwarding components are The recent rapid adoption of Cloud Services by enterprises and the
decoupled, they may reside in the same or different physical devices. phenomenal growth of mobile IP applications accelerate the needs to
extend the L3VPN capability to the end devices. For example,
Enterprise customers requested Service Providers to extend and
integrate their L3VPN services available in the WAN into the new
Cloud services; large enterprise have existing L3VPN deployment are
extending them into their data centers; mobile providers are adopting
L3VPN into their 3GPP Mobile infrastructure are looking to extend the
L3VPNs to their end device of their call processing center.
The virtual PE solution described in this document is aimed to meet
the following key requirement [I-D.fang-l3vpn-end-system-req].
INTERNET DRAFT <Document Title> <Issue Date> INTERNET DRAFT <Document Title> <Issue Date>
A key motivation of using virtual PE solution is to place the L3VPN 1) Support end device multi-tenancy, per tenant routing isolation and
termination point as close as possible to where the traffic separation.
services/applications reside; and to take the advantage of control
and forwarding decoupling for better scalability and allow flexible 2) Support large scale L3VPNs in service network, upto tens of
routing policy control and fast provisioning. In many cases, the thousands of end devices and Millions of VMs in the single service
virtual PE is placed in the service end systems where the Virtual network, e.g., a data center.
Machines running various applications.
3) Support end-to-end L3VPN connectivity, e.g. L3VPN can start from a
service network end device, connect to a corresponding L3VPN in the
WAN, and terminate in another service network end device.
4) Decoupling control plane and forwarding for network virtualization
and abstraction.
L3VPN is the proven technologies which is capable of providing
routing and forwarding separation, and it is proven with large scale
deployment (e.g. supporting 7-8 million L3VPN routes in single
Service Provider networks today).
By extending L3VPN solution to the end device with vPE solution,
application end-to-end (VM to VM, applications to the end user) L3VPN
connectivity cab be achieved, and well as the true network
virtualization and abstraction.
The architecture and protocols defined in BGP/MPLS IP VPN [RFC4364]
is the foundation for virtual PE extension. Certain protocol
extensions or integration may be needed to support the virtual PE
solutions.
1.3 Scope of the document
It is assumed that the readers are familiar with BGP/MPLS IP VPN
[RFC4364] terms and technologies, the base technology and its
operation are not reviewed in details in this document.
The following network elements are discussed in this document: the
concept of BGP L3VPN vPE; the interaction of vPE with other network
elements, including BGP L3VPN physical PE, physical or virtual BGP
Route Reflectors (RR, vRR), and Autonomous System Border Router
(ASBR), Service Network gateway routers, external controllers,
provisioning/orchestration systems, and the vPE inter-connections
with other non L3VPN networks.
The definitions of protocols extensions are out of the scope of this
document.
2. Virtual PE Architecture and Reference Model
INTERNET DRAFT <Document Title> <Issue Date>
2.1 Virtual PE
As defined in [RFC4364], a L3VPN is created by applying policies to As defined in [RFC4364], a L3VPN is created by applying policies to
form a subset of sites among all sites connected the backbone form a subset of sites among all sites connected the backbone
network. It is collection of "sites". A site can be considered as a network. It is collection of "sites". A site can be considered as a
set of IP systems maintain IP inter-connectivity without connecting set of IP systems maintain IP inter-connectivity without connecting
through the backbone. The typical use of L3VPM has been to inter- through the backbone. The typical use of L3VPM has been to inter-
connect different sites of an Enterprise networks through Service connect different sites of an Enterprise networks through Service
Provider's L3VPN in the WAN. Provider's L3VPNs in the WAN.
The recent rapid adoption of Cloud Services, and the phenomenal A virtual PE (vPE) is a PE instance which resides in one or more
growth of mobile IP applications, accelerate the needs to extend the physical devices, it is commonly placed in a network service (e.g. a
VPN capability to the end-systems. Enterprise customers requests Data Center) end device (e.g., a Server) where the client/application
Service Providers to extend the L3VPN services in the WAN into the VMs are hosted. The control and forwarding components of the vPE are
new Cloud services supported by various Data Center technologies. decoupled, they may reside in the same physical device or in
Mobile providers who have already adopted L3VPN into the Mobile different physical devices.
infrastructure are looking to support service virtualization with
L3VPN on the end-system of mobile applications. In the case that a vPE is in a Data Center server along with
client/application VMs, one can view the vPE to VM relationship as a
typical PE-CE relationship. Unlike a regular physical PE, vPE allows
L3VPN control plane and forwarding function residing on different
physical devices. The full MP-BGP control plane may reside on the end
device, or may be external to the end device, e.g., in a BGP L3VPN
boarder router (ASBR)/DC gateway router, a Route Reflector (RR), or
an external controller.
Virtual PE solution allows the placement of L3VPN termination point
right inside the end device (e.g., a server). In this case, the vPE
to CE (VM) connection is internal to the device. If both control and
forwarding elements are placed on the end device, L3VPN routing and
forwarding starts from the end device, the eliminate the needs for
additional process in the next hop (e.g., layer2 and layer 3
integration). This approach helps to simplify the operation and
improve the routing and forwarding efficiency in large scale
deployment.
Another important benefit is that vPE solution allows full control
and forwarding decoupling for scale and achieving true network
virtualization to allow network abstraction, flexible and dynamic
policy control, quick service turn up time and VM mobility support.
2.2 Architecture reference model 2.2 Architecture reference model
Figure 1 illustrate the topology that vPE is reside inside the end- Figure 1 illustrate the topology that vPE is reside in the end device
system where the applications are hosted. where the applications are hosted.
INTERNET DRAFT <Document Title> <Issue Date> INTERNET DRAFT <Document Title> <Issue Date>
+--------------------------------------------+ +--------------------------------------------+
| | | |
| +-----------+ | | +-----------+ |
| |vPE| End | | | |vPE| End | |
| |---+ System| | | |---+ Device| |
| +---------+ +-----------+ | | +---------+ +-----------+ |
.------. | |Transport| +-----------+ | .------. | |Transport| +-----------+ |
( ) | +-------+ | Device | |vPE| End | | ( ) | +-------+ | Device | |vPE| End | |
( ) | |Gateway| +---------+ |---+ System| | ( ) | |Gateway| +---------+ |---+ Device| |
: :--+-| PE | +-----------+ | : :--+-| PE | +-----------+ |
: : | +-------+ +---------+ +-----------+ | : : | +-------+ +---------+ +-----------+ |
: IP/MPLS : | |Transport| |vPE| End | | : IP/MPLS : | |Transport| |vPE| End | |
: WAN : | +-------+ | Device | |---+ System| | : WAN : | +-------+ | Device | |---+ Device| |
: :--+-|Gateway| +---------+ +-----------+ | : :--+-|Gateway| +---------+ +-----------+ |
( ) | | PE | +-----------+ | ( ) | | PE | +-----------+ |
( ) | +-------+ +---------+ |vPE| End | | ( ) | +-------+ +---------+ |vPE| End | |
'------' | |Transport| |---+ System| | '------' | |Transport| |---+ Device| |
| | Device | +-----------+ | | | Device | +-----------+ |
| +---------+ +-----------+ | | +---------+ +-----------+ |
| |vPE| End | | | |vPE| End | |
| |---+ System| | | |---+ Device| |
| +-----------+ | | +-----------+ |
| Virtualized Service Network | | Virtualized Service Network |
+--------------------------------------------+ +--------------------------------------------+
Figure 1. Virtualized Service network with vPE Figure 1. Virtualized Service network with vPE
The Virtualized Service Network in Figure 1 consists of WAN gateway The Virtualized Service Network in Figure 1 consists of WAN gateway
PE devices, transport devices, and end-systems. PE devices, transport devices, and end devices. In some networks, it
is feasible the VPN Gateways may be implemented as vPEs as well.
Examples of service network may be a network that supports cloud Examples of service network may be a network that supports cloud
computing services, mobile call centers, and SP or enterprise private computing services, mobile call centers, and SP or enterprise data
or public data centers. centers.
The transport devices in the service network do not participate Note that the transport devices in the service network in the diagram
L3VPNs, they do not maintain the L3VPN states, they are not aware of do not participate L3VPNs, they function similar as P routers in MPLS
the L3VPN in the network. back bone, they do not maintain the L3VPN states, and are not L3VPN
aware.
INTERNET DRAFT <Document Title> <Issue Date> INTERNET DRAFT <Document Title> <Issue Date>
+----------------------------------------------------+ +----------------------------------------------------+
| +---------+ +---------+ +---------+ +---------+ | | +---------+ +---------+ +---------+ +---------+ |
| | VM1 | | VM2 | | VM47 | | VM48 | | | | VM1 | | VM2 | | VM47 | | VM48 | |
| |(VPN Red)| |(VPN Grn)|... |(VPN Grn)| |(VPN Blu)| | | |(VPN Red)| |(VPN Grn)|... |(VPN Grn)| |(VPN Blu)| |
| +----+----+ +---+-----+ +----+----+ +----+----+ | | +----+----+ +---+-----+ +----+----+ +----+----+ |
| | | | | | | | | | | |
| +---+ | +-------------+ +---+ | | +---+ | +-------------+ +---+ |
| | | | | | | | | | | |
to | +---+------+-+---------------------+---+ | to | +---+------+-+---------------------+---+ |
Gateway| | | | | | | Gateway| | | | | | | |
PE | | +-+-+ ++-++ +---+ +-+-+ | | PE | | +-+-+ ++-++ +---+ +-+-+ | |
| | |VRF| |VRF| ....... |VRF| |VRF| | | | | |VRF| |VRF| ....... |VRF| |VRF| | |
<------+------+ |Red| |Grn| |Yel| |Blu| | | <------+------+ |Red| |Grn| |Yel| |Blu| | |
| | +---+ +---+ +---+ +---+ | | | | +---+ +---+ +---+ +---+ | |
| | L3 VPN virtual PE | | | | L3 VPN virtual PE | |
| +--------------------------------------+ | | +--------------------------------------+ |
| | | |
| End-system | | End Device |
+----------------------------------------------------+ +----------------------------------------------------+
Figure 2. Virtual L3 VPN PE logical connections in the End-system Figure 2. VM in end device to VRF in vPE mapping
An end-system shown in Figure 2 is a virtualized server which hosting
multiple VMs, the a virtual PE resides in the end-system. The vPE
supports multiple VRFs, VRF Red, VRF Grn, VRF Yel, VRF Blu, etc. Each
VM is associated to a particular VRF as a member of the particular
VPN. For example, VM1 is associated to VRF Red, VM2 and VM47 are
associated to RFC Grn, etc. Routing isolations apply between VPNs.
The vPE connectivity relationship between vPE to VM is similar like An end device shown in Figure 2 is a virtualized server or system
the PE to CE in a regular BGP L3VPNs. which hosts multiple VMs, the virtual PE resides in the end device.
The vPE supports multiple VRFs, VRF Red, VRF Grn, VRF Yel, VRF Blu,
etc. Each client or application VM is associated to a particular VRF
as a member of the particular VPN. For example, VM1 is associated to
VRF Red, VM2 and VM47 are associated to RFC Grn, etc. Routing
isolation applies between VPNs for multi-tenancy support. For
example, VM1 and VM2 can not communicate with each other in a simple
intranet L3VPN topology as shown in the configuration.
VM1 and VM2 can not connect to each other in a simple intranet VPN The vPE connectivity relationship between vPE and the application VM
topology as shown in the configuration. The L3VPN provides routing is similar to the PE to CE relationship in a regular BGP L3VPNs.
isolation among different VPNs.
INTERNET DRAFT <Document Title> <Issue Date> INTERNET DRAFT <Document Title> <Issue Date>
+----------------------+ +----------------------------+ +----------------------+ +----------------------------+
| | | +-----------+ | | | | +-----------+ |
+------+--+ | +-----+ +-----+ | | +-----+ | +---+| | +------+--+ | +-----+ +-----+ | | +-----+ | +---+| |
|VPN +--+ | | | | | | | |+---+| +---+ |VM1|| | |VPN +--+ | | | | | | | |+---+| +---+ |VM1|| |
|Red |CE|-+-|+---+| |+---+| | | ||VRF|| |vPE| +---+| | |Red |CE|-+-|+---+| |+---+| | | ||VRF|| |vPE| +---+| |
|Site A+--+ | ||VRF|| ||VRF|| | | ||Red|| +---+ |VM2|| | |Site A+--+ | ||VRF|| ||VRF|| | | ||Red|| +---+ |VM2|| |
+------+--+ | ||Red|| ||Red|| | | |+---+| | +---+| | +------+--+ | ||Red|| ||Red|| | | |+---+| |Server+---+| |
| |+---+| |+---+|-+---+-|+---+| +-----------+ | | |+---+| |+---+|-+---+-|+---+| +-----------+ |
| |+---+| |+---+| | | ||VRF|| +-----------+ | | |+---+| |+---+| | | ||VRF|| +-----------+ |
+------+--+ | ||VRF|| ||VRF|| | | ||Grn|| | +---+| | +------+--+ | ||VRF|| ||VRF|| | | ||Grn|| | +---+| |
|VPN +--+-+-||Grn|| ||Grn|| | | |+---+| +---+ |VM1|| | |VPN +--+-+-||Grn|| ||Grn|| | | |+---+| +---+ |VM1|| |
|Grn |CE| | |+---+| |+---+| | | |GWay | |vPE| +---+| | |Grn |CE| | |+---+| |+---+| | | |GWay | |vPE| +---+| |
|Site A+--+ | | PE | | PE | | | | PE | +---+ |VM2|| | |Site A+--+ | | PE | | PE | | | | PE | +---+ |VM2|| |
+------+--+ | +-----+ +-----+ | | +-----+ | +---+| | +------+--+ | +-----+ +-----+ | | +-----+ |Server+---+| |
| | | +-----------+ | | | | +-----------+ |
| IP/MPLS WAN | |Virtualized Service Network | | IP/MPLS WAN | |Virtualized Data Center |
+----------------------+ +----------------------------+ +----------------------+ +----------------------------+
|eBGP | | | | Figure 3. Connecting Enterprise CE to DC VM over WAN
|Static| MP-BGP | MP-BGP |(Options) |
|<---->|<--------------->|<------>|<--------->|
Inter-AS
Option B
Figure 3. L3VPN Logical Connection protocols
Options for protocol between Gateway PE to vPE on the end-system:
1. MP-BGP. The example of connection from an Enterprise site to application VMs
through vPE on the end device of a SP provisioned virtualized data
center.
2. XMPP or other extensible messaging protocols which are familiar by There are multiple options for VPN control plane signaling between
computing work force. the Gateway PE to vPE on the server within the data center. It can
use MP-BGP as in regular L3VPN, or use other extensible IP messaging
protocols defined in IETF, or use controller direct signaling as a
SDN approach.
3. Network Controller The inter-connection from DC Gateway PE to MPLS WAN may use one of
the Inter-AS options if they are in different ASes. Option B may be
more practical for the reasons it is more scalable than Option A, and
more restricted than Option C. Consider route aggregation with Option
B if both sides have large number of routes.
Options for protocols between IP/MPLS WAN network and Virtualized The connection between backbone VPN to VPN CE on the left hand side
service network depending on the design. Inter-AS with Option B is is regular L3VPN connection, e-BGP, or static, or other protocols can
an example. be used.
3. Control Plane 3. Control Plane
L3VPN control protocol is MP-BGP as defined in [RFC4364]. 3.1 vPE Control Plane
When extending L3VPN into service network, supporting MP-BGP on the
end-system may or may not be practical, alternatives are also
INTERNET DRAFT <Document Title> <Issue Date>
considered here.
3.1 Router server of vPE
A virtual PE consist the control plane element and the forwarding
plane element. Since the proposed solution decoupled the two element,
they may or may not reside on the same physical device.
The Route Server of L3VPN vPE is a software application that
implements the BGP/MPLS L3VPN PE control plane functionality.
In the case other control/signaling/messaging protocol are used (see
options listed in the next sub-section), the route server is also the
server of the particular protocol(s), it interacts with VPN forwarder
(see the section on forwarding).
3.2 Control plane options of vPE The vPE control plane can be distributed or centralized.
a. MP-BGP 1) Distributed control plane
MP-BGP can be used in service network where both the end system INTERNET DRAFT <Document Title> <Issue Date>
implementation and operation work force has the knowledge and skills
to support it. In this case, the design and deployment is very
similar to what applies to regular L3VPN deployment.
b. Extensible messaging protocols vPE participates in underlay routing through IGP protocols: ISIS or
OSPF.
It is redeemed as a good alternative for bridging the gap between vPE participates in overlay L3VPN control protocol: MP-BGP
Gateway PE and the vPE where operators could not support BGP. [RFC4364].
Although the modern end-systems may be able to support MP-BGP, the While MP-BGP is the de facto preferred choice between vPE and
operational support of the computing and storage community often may gateway-PE, using extensible signaling messaging protocols can be
not have the adequate BGP skills or willingness to step up to BGP alternative, such technologies have been proposed for this segment of
operation. Since this is a common situation today, an light weight, signaling [I-D.ietf-l3vpn-end-system].
extensible IP protocol is very welcome by the cloud service
communities. One example of such protocol is to use XMPP to signal
L3VPN connectivity between the Gateway PE the virtual PE on the end-
systems. The technology solution is described in details in [I-
D.ietf-l3vpn-end-system].
c. Controller 2. Centralized routing controller
This is a SDN approach. In the virtual PE implementation, not only This is a SDN approach. In the virtual PE implementation, not only
the service network infrastructure and the VPN overlay networks are the service network infrastructure and the VPN overlay networks are
decoupled, but also the vPE control plane and data plane are decoupled, but also the vPE control plane and data plane are
physically decoupled. The control plane directing the data flow may physically decoupled. The control plane directing the data flow may
reside elsewhere, such a centralized controller. This requires reside elsewhere, such a centralized controller. This requires
standard interface to routing system (IRS). The IRS work is in standard interface to routing system (IRS). The Interface to Routing
System (IRS) is work in progress in IETF [I-D.ward-irs-framework],
[I-D.rfernando-irs-fw-req].
INTERNET DRAFT <Document Title> <Issue Date> 3.1 Route server of vPE
progress in IETF, [ID.ward-irs-framework], [ID.rfernando-irs- A virtual PE consist the control plane element and the forwarding
framework-requirement]. plane element. Since the proposed solution decoupled the two element,
they may or may not reside on the same physical device.
The Route Server of L3VPN vPE is a software application that
implements the BGP/MPLS L3VPN PE control plane functionality.
In the case other control/signaling/messaging protocol are used, the
route server is also the server of the particular protocol(s), it
interacts with VPN forwarder.
3.3 Use of router reflector 3.3 Use of router reflector
Modern service networks can be very large in scale, the very large Modern service networks can be very large in scale. For example, the
data centers can easily pass the scale of SP backbone VPN networks. number of VPNs routes in a very large data centers can pass the scale
The end-systems and therefore the vPEs may be several thousand in a of those in SP backbone VPN networks. There are may be tens of
single service network. thousands of end devices in a single service network.
Use of Router Reflector (RR) is necessary in large scale L3VPN Use of Router Reflector (RR) is necessary in large scale L3VPN
networks to avoid full iBGP mesh among all vPEs and PEs. The L3 VPN networks to avoid full iBGP mesh among all vPEs and PEs. The L3 VPN
routes can be partitioned to a set of RRs, the partition techniques routes can be partitioned to a set of RRs, the partition techniques
are detailed in [RFC4364]. are detailed in [RFC4364].
When RR is residing a device which is partitioned to support multi- When RR is residing in a physical device, e.g., a server, which is
functions and application, the RR become virtualized RR (vRR). Since
RR's is control plane only, a physical or virtualized server with INTERNET DRAFT <Document Title> <Issue Date>
large scale of computing power and memory can be a perfect candidate
as host of vRRs. The vRR can be in Gateway PE, or an end-system. partitioned to support multi-functions and client/applications VMs,
the RR becomes virtualized RR (vRR). Since RR's performs control
plane only, a physical or virtualized server with large scale of
computing power and memory can be a good candidate as host of vRRs.
The vRR can also reside be in Gateway PE, or in an end device.
Redundant RR design is even more important in when using vRR.
3.4 Use of RT constraint 3.4 Use of RT constraint
The Route Target Constraint [RFC4684] is a powerful tool for VPN The Route Target Constraint (RT Constraint, RTC) [RFC4684] is a
route filtering. With RT constraint, only the BGP receiver (a PE, powerful tool for VPN selective L3VPN route distribution. With RT
vPE, RR, vRR, ASBRs, etc.) with the particular L3VPN routes will Constraint, only the BGP receiver (e.g, PE/vPE/RR/vRR/ASBRs, etc.)
receive the route update. It is critical to use RT constraint with the particular L3VPNs will receive the route update for the
particularly in large scale development. corresponding VPNs. It is critical to use RT constraint to support
large scale L3VPN development.
4. Forwarding Plane 4. Forwarding Plane
4.1 Virtual Interface 4.1 Virtual Interface
Virtual Interface (VI) is an interface in an end-system which is used Virtual Interface (VI) is an interface in an end device which is used
for connecting the vPE to the VMs or other applications in the end- for connecting the vPE to the application VMs in the end device. The
system. The later can be viewed as CEs in traditional L3VPN scenario. latter cab be treated as CEs in the regular L3VPN's view.
4.2 VPN forwarder 4.2 VPN forwarder
VPN Forwarder is the forwarding component of a vPE solution. VPN Forwarder is the forwarding component of a vPE.
The VPN forwarder location options: The VPN forwarder location options:
1) within the end-system where the virtual interface is 1) within the end device where the virtual interface and application
VMs are.
2) in an external device of end-system which the end-system connect
to, for example, a ToR.
INTERNET DRAFT <Document Title> <Issue Date> 2) in an external device which the end device connect to, for
example, a Top of the Rack (ToR) in a data center.
Multiple factors should be considered for the location of the VPN Multiple factors should be considered for the location of the VPN
forwarder, including device capability, overall economy, forwarder, including device capability, overall solution economics,
QoS/firewall/NAT placement, optimal forwarding, latency and QoS/firewall/NAT placement, optimal forwarding, latency and
performance, etc. There are design trade offs, it is worth the effort performance, operation impact, etc. There are design trade offs, it
to study the traffic pattern and forwarding looking trend in your own is worth the effort to study the traffic pattern and forwarding
unique service network as part of the exercise. looking trend in your own unique service network as part of the
exercise.
4.3 Encapsulation 4.3 Encapsulation
There are two existing standardized forwarding options for BGP/MPLS There are two existing standardized encapsulation/forwarding options
L3VPN. for BGP/MPLS L3VPN.
INTERNET DRAFT <Document Title> <Issue Date>
1. MPLS Encapsulation, [RFC3032]. 1. MPLS Encapsulation, [RFC3032].
2. Encapsulating MPLS in IP or Generic Routing Encapsulation 2. Encapsulating MPLS in IP or Generic Routing Encapsulation
(GRE), [RFC4023]. (GRE), [RFC4023].
The most common BGP/MPLS L3VPNs deployment in SP networks are using The most common BGP/MPLS L3VPNs deployment in SP networks are using
MPLS forwarding. This requires MPLS to be deployed in the network. It MPLS forwarding. This requires MPLS, e.g., Label Switched Protocol
is proven to scale and provide various security mechanism to against (LDP) [RFC5036] to be deployed in the network. It is proven to scale,
attacks. and it comes with various security mechanisms to protect network
against attacks.
The service network environment, such as a data center, is different However, the service network environment, such as a data center, is
than Service Provider VPN networks or large enterprise backbones. different than Service Provider VPN networks or large enterprise
MPLS deployment may or may not be feasible. Two major challenges for backbones. MPLS deployment may or may not be feasible. Two major
MPLS deployment in the new environment: 1) the capabilities for the challenges for MPLS deployment in this new environment: 1) the
end-system devices or transport/forwarding devices; 2) the workforce capabilities of the end devices and the transport/forwarding devices;
skill set. 2) the workforce skill set.
Encapsulating MPLS in IP or GRE tunnel may be more practical in many Encapsulating MPLS in IP or GRE tunnel [RFC4023] may often be more
cases. But bare in mind, IP or GRE tunnel does not provide the same practical in most data center, and computing environment. Note that
level of security mechanism as MPLS forwarding. when IP encapsulations are used, the associated security
considerations must be analyzed carefully.
There are new encapsulation proposals for service network/Data center In addition, there are new encapsulation proposals for service
as work in progress in IETF, including several UDP based network/Data center currently as work in progress in IETF, including
encapsulations proposals and some TCP based proposals. These several UDP based encapsulations proposals and some TCP based
mechanism may be considered as alternative to MPLS and IP/GRE encap. proposal. These overlay encapsulations can be suitable alternatives
for a vPE, considering the availability and leverage of support in
virtual and physical devices.
4.4 Optimal forwarding 4.4 Optimal forwarding
As reported by many large cloud service operators, the traffic As reported by many large cloud service operators, the traffic
pattern in their data centers are dominated by East-West traffic pattern in their data centers were dominated by East-West across
(between the end-systems hosting different applications) than North- subnet traffic (between the end device hosting different applications
South traffic (going in and out the DC to the WAN). This is a key in different subnets) than North-South traffic (going in and out the
DC to the WAN) or switched traffic within subnets. This is a primary
reason that many large scale new design has moved away from reason that many large scale new design has moved away from
traditional L2 design to L3. traditional L2 design to L3.
When forwarding the traffic within the same VPN, the end-system When forwarding the traffic within the same VPN, the vPE should be
able to provide direct communication among the VMs/application
INTERNET DRAFT <Document Title> <Issue Date> senders/receivers without the need of going through gateway devices.
If it is on the same end device, the traffic should not need to leave
should be able to access directly between the VMs/application the same device. If it is on different end device, optimal routing
sender/receivers without the need of going through gateway devices.
If it is on the same end-system, the traffic should not need to leave
the same device. If it is on different end-system, optimal routing
should be applied. should be applied.
When multiple VPNs need to be accessed to accomplish the task the When multiple VPNs need to be accessed to accomplish the task the
user requested (this is common too), the end-system virtual
interfaces should be able to directly access multiple VPNs in INTERNET DRAFT <Document Title> <Issue Date>
extranet VPN style without the need of Gateway facilitation. BGP
L3VPN policy control is the tool to support this function. user requested (this is common too), the end device virtual
interfaces should be able to directly access multiple VPNs via use of
extranet VPN techniques without the need of Gateway facilitation. Use
BGP L3VPN policy control mechanisms to support this function.
5. Addressing 5. Addressing
5.1 IPv4 and IPv6 support 5.1 IPv4 and IPv6 support
Both IPv4 and IPv6 should be supported in the virtual PE solution. Both IPv4 and IPv6 should be supported in the virtual PE solution.
This may present challenging to older devices, but may not be issues This may present challenging to older devices, but may not be issues
to newer forwarding devices and servers. A server is replaced much to newer forwarding devices and servers. A server is replaced much
more frequently than a network router/switch in the infrastructure more frequently than a network router/switch in the infrastructure
network. Newer equipment most likely support IPv6. network, newer equipment should be capable of IPv6 support.
5.2 Address space separation 5.2 Address space separation
The addresses used for L3VPNs in the service network should be in The addresses used for L3VPNs in the service network should be in
separate address blocks than the ones used the underlay separate address blocks than the ones used the underlay
infrastructure of the service network. This practice is to protect infrastructure of the service network. This practice is to protect
the service network infrastructure being attacked if the attacker the service network infrastructure being attacked if the attacker
gain access of the tenant VPNs. gain access of the tenant VPNs.
Similarity, the addresses used for the service network, e.g., a cloud Similarity, the addresses used for the service network, e.g., a cloud
service center of a SP, should be separated from the WAN backbone service center of a SP, should be separated from the WAN backbone
addresses space, for security reasons. addresses space, for security reasons.
6.0 Inter-connection considerations
There are also deployment scenarios that L3VPN may not be supported
in every segment of the networks to provide end-to-end L3VPN
connectivity, a L3VPN vPE may be reachable only via an intermediate
inter-connecting network, interconnection may be needed in these
cases.
When multiple technologies are employed in the overall solution, a
clear demarcation should be preserved at the inter-connecting points.
The problems encountered in one domain should not impact the other
domains.
From L3VPN point of view: A L3VPN vPE that implements [RFC4364] is a
component of L3VPN network only. A L3VPN VRF on physical PE or vPE
contains IP routes only, including routes learnt over the locally
attached network.
As described earlier in this document, the L3VPN vPE should ideally
be located as close to the "customer" edge devices. For cases, where
INTERNET DRAFT <Document Title> <Issue Date>
this is not possible, simple existing "L3VPN CE connectivity"
mechanisms should be used, such as static, or direct VM attachments
such as described in the vCE option below.
Consider the following scenarios when BGP MPLS VPN technology is
considered as whole or partial deployment:
Scenario 1: All VPN sites (CEs/VMs) support IP connectivity. The best
suited BGP solution is to use L3 VPNs [RFC4364] for all sites with PE
and/or vPE solutions. This is a straightforward case.
Scenario 2: Legacy layer 2 connectivity must be supported in certain
sites/CEs/VMs, and the rest sites/CEs/VMs need only 3 connectivity.
One can consider to use combined vPE and vCE solution to solved the
problem. Use L3VPN for all sites with IP connectivity, and use a
physical or virtual CE (vCE, may reside on the end device) to
aggregate the L2 sites which, for example, are in a single container
in a data center. The CE/vCE can be considered as inter-connecting
point, where the L2 network are terminated and the corresponding
routes for connectivity of the L2 network are inserted into L3VPN
VRF. The L2 aspect is transparent to the L3VPN in this case.
Reducing operation complicity and maintaining the robustness of the
solution are the primary reasons for the recommendations.
7. Security Considerations 7. Security Considerations
To be added. vPE solution presented a virtualized L3VPN PE model. There are
potential implications to L3VPN control plane, forwarding plane, and
management plane. Security considerations are currently under study,
will be included in the future revisions.
8. IANA Considerations 8. IANA Considerations
None. None.
9. References 9. References
9.1 Normative References 9.1 Normative References
INTERNET DRAFT <Document Title> <Issue Date>
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997. Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC3032] Rosen, E., Tappan, D., Fedorkow, G., Rekhter, Y., [RFC3032] Rosen, E., Tappan, D., Fedorkow, G., Rekhter, Y.,
Farinacci, D., Li, T., and A. Conta, "MPLS Label Stack Farinacci, D., Li, T., and A. Conta, "MPLS Label Stack
Encoding", RFC 3032, January 2001. Encoding", RFC 3032, January 2001.
INTERNET DRAFT <Document Title> <Issue Date>
[RFC4023] Worster, T., Rekhter, Y., and E. Rosen, Ed., [RFC4023] Worster, T., Rekhter, Y., and E. Rosen, Ed.,
"Encapsulating MPLS in IP or Generic Routing Encapsulation "Encapsulating MPLS in IP or Generic Routing Encapsulation
(GRE)", RFC 4023, March 2005. (GRE)", RFC 4023, March 2005.
[RFC4271] Rekhter, Y., Ed., Li, T., Ed., and S. Hares, Ed., "A [RFC4271] Rekhter, Y., Ed., Li, T., Ed., and S. Hares, Ed., "A
Border Gateway Protocol 4 (BGP-4)", RFC 4271, January Border Gateway Protocol 4 (BGP-4)", RFC 4271, January
2006. 2006.
[RFC4364] Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual Private [RFC4364] Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual Private
Networks (VPNs)", RFC 4364, February 2006. Networks (VPNs)", RFC 4364, February 2006.
[RFC4684] Marques, P., Bonica, R., Fang, L., Martini, L., Raszuk, [RFC4684] Marques, P., Bonica, R., Fang, L., Martini, L., Raszuk,
R., Patel, K., and J. Guichard, "Constrained Route R., Patel, K., and J. Guichard, "Constrained Route
Distribution for Border Gateway Protocol/MultiProtocol Distribution for Border Gateway Protocol/MultiProtocol
Label Switching (BGP/MPLS) Internet Protocol (IP) Virtual Label Switching (BGP/MPLS) Internet Protocol (IP) Virtual
Private Networks (VPNs)", RFC 4684, November 2006. Private Networks (VPNs)", RFC 4684, November 2006.
[RFC5036] Andersson, L., Ed., Minei, I., Ed., and B. Thomas, Ed.,
"LDP Specification", RFC 5036, October 2007.
[I-D.ietf-l3vpn-end-system] Marques, P., Fang, L., Pan, P., Shukla, [I-D.ietf-l3vpn-end-system] Marques, P., Fang, L., Pan, P., Shukla,
A., Napierala, M., "BGP-signaled end-system IP/VPNs", A., Napierala, M., "BGP-signaled end-system IP/VPNs",
draft-ietf-l3vpn-end-system-00, October 2012. draft-ietf-l3vpn-end-system-00, October 2012.
9.2 Informative References 9.2 Informative References
[ID.ward-irs-framework] Atlas, A., Nadeau, T., Ward. D., "Interface [I-D.fang-l3vpn-end-system-req] Napierala, M., adn Fang, L.,
"Requirements for Extending BGP/MPLS VPNs to End-Systems",
draft-fang-l3vpn-end-system-requirements-00, Oct. 2012.
[I-D.ward-irs-framework] Atlas, A., Nadeau, T., Ward. D., "Interface
to the Routing System Framework", draft-ward-irs- to the Routing System Framework", draft-ward-irs-
framework-00, July 2012. framework-00, July 2012.
[ID.rfernando-irs-framework-requirement] Fernando, R., Medved, J., [I-D.rfernando-irs-fw-req] Fernando, R., Medved, J., Ward, D., Atlas,
Ward, D., Atlas, A., Rijsman, B., "IRS Framework A., Rijsman, B., "IRS Framework Requirements", draft-
Requirements", draft-rfernando-irs-framework-requirement- rfernando-irs-framework-requirement-00, Oct. 2012.
00, Oct., 2012.
Authors' Addresses Authors' Addresses
Luyuan Fang Luyuan Fang
Cisco Cisco
111 Wood Ave. South 111 Wood Ave. South
Iselin, NJ 08830
INTERNET DRAFT <Document Title> <Issue Date> INTERNET DRAFT <Document Title> <Issue Date>
Iselin, NJ 08830
Email: lufang@cisco.com Email: lufang@cisco.com
David Ward
Cisco
170 W Tasman Dr
San Jose, CA 95134
Email: wardd@cisco.com
Rex Fernando Rex Fernando
Cisco Cisco
170 W Tasman Dr 170 W Tasman Dr
San Jose, CA San Jose, CA
Email: rex@cisco.com Email: rex@cisco.com
Maria Napierala Maria Napierala
AT&T AT&T
200 Laurel Avenue 200 Laurel Avenue
Middletown, NJ 07748 Middletown, NJ 07748
Email: mnapierala@att.com Email: mnapierala@att.com
Nabil Bitar
Verizon
40 Sylvan Road
Waltham, MA 02145
Email: nabil.bitar@verizon.com
Dhananjaya Rao
Cisco
170 W Tasman Dr
San Jose, CA
Email: dhrao@cisco.com
 End of changes. 91 change blocks. 
274 lines changed or deleted 413 lines changed or added

This html diff was produced by rfcdiff 1.48. The latest version is available from http://tools.ietf.org/tools/rfcdiff/