Network Virtualization Overlays Working R. Schott Group Deutsche Telekom Internet-Draft Q. Wu Intended status: Informational Huawei Expires: August 22, 2013 February 18, 2013 Network Virtualization Overlay Architecture draft-fw-nvo3-server2vcenter-01.txt Abstract Multiple virtual machines (VMs) created in a single physical platform Or vServer greatly improve the efficiency of data centers by enabling more work from less hardware. Multiple vServer and associated virtual machines work together as one cluster make good use of resources of each vServer that are scattered into different data centers or vServers. VMs have their lifecycles from VM creation, VM Power on to VM Power off and VM deletion. The VMs may also move across the participating virtualization hosts (e.g., the virtualization server, hypervisor). This document discusses how VMs, vServers and overlay network are managed by leveraging control plane function and management plane function and desired signaling functionalities for Network Virtualization Overlay. Status of this Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at http://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on August 22, 2013. Copyright Notice Copyright (c) 2013 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Schott & Wu Expires August 22, 2013 [Page 1] Internet-Draft NVO3 Architecture February 2013 Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Schott & Wu Expires August 22, 2013 [Page 2] Internet-Draft NVO3 Architecture February 2013 Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 5 2.1. Standards Language . . . . . . . . . . . . . . . . . . . . 5 3. Discussions . . . . . . . . . . . . . . . . . . . . . . . . . 6 3.1. VM awareness and VM movement awareness . . . . . . . . . . 6 3.2. Why VM migration . . . . . . . . . . . . . . . . . . . . . 6 3.3. Who manages VM . . . . . . . . . . . . . . . . . . . . . . 7 3.4. VM Grouping . . . . . . . . . . . . . . . . . . . . . . . 7 3.5. What VM information should be managed . . . . . . . . . . 8 3.6. Who Triggers or Controls VM Movements . . . . . . . . . . 9 3.7. VM Monitoring . . . . . . . . . . . . . . . . . . . . . . 9 4. Use Cases . . . . . . . . . . . . . . . . . . . . . . . . . . 10 4.1. On Demand Network Provision Automation within the data center . . . . . . . . . . . . . . . . . . . . . . . . . . 10 4.2. Large inter-data centers Layer 2 interconnection and data forwarding . . . . . . . . . . . . . . . . . . . . . 11 4.3. Enable multiple data centers present as one . . . . . . . 12 4.4. VM migration and mobility across data centers . . . . . . 13 5. General Network Virtualization Architecture . . . . . . . . . 15 5.1. NVE (Network Virtualization Edge Function) . . . . . . . . 16 5.2. vServer (virtualization Server) . . . . . . . . . . . . . 17 5.3. vCenter (management plane function) . . . . . . . . . . . 17 5.4. nDirector (Control plane function) . . . . . . . . . . . . 17 6. vServer to vCenter management interface . . . . . . . . . . . 19 6.1. VM Creation . . . . . . . . . . . . . . . . . . . . . . . 19 6.2. VM Termination . . . . . . . . . . . . . . . . . . . . . . 19 6.3. VM Registration . . . . . . . . . . . . . . . . . . . . . 19 6.4. VM Unregistration . . . . . . . . . . . . . . . . . . . . 19 6.5. VM Bulk Registration . . . . . . . . . . . . . . . . . . . 19 6.6. VM Bulk Unregistration . . . . . . . . . . . . . . . . . . 19 6.7. VM Configuration Modification . . . . . . . . . . . . . . 20 6.8. VM Profile Lookup/Discovery . . . . . . . . . . . . . . . 20 6.9. VM Relocation . . . . . . . . . . . . . . . . . . . . . . 20 6.10. VM Replication . . . . . . . . . . . . . . . . . . . . . . 20 6.11. VM Report . . . . . . . . . . . . . . . . . . . . . . . . 20 7. nDirector to NVE Edge control interface . . . . . . . . . . . 22 8. vServer to NVE Edge control interface . . . . . . . . . . . . 23 9. Security Considerations . . . . . . . . . . . . . . . . . . . 24 10. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 25 11. Contributors . . . . . . . . . . . . . . . . . . . . . . . . . 26 12. References . . . . . . . . . . . . . . . . . . . . . . . . . . 27 12.1. Normative References . . . . . . . . . . . . . . . . . . . 27 12.2. Informative References . . . . . . . . . . . . . . . . . . 27 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 28 Schott & Wu Expires August 22, 2013 [Page 3] Internet-Draft NVO3 Architecture February 2013 1. Introduction Multiple virtual machines (VMs) created in a single physical platform greatly improve the efficiency of data centers by enabling more work from less hardware. Multiple vServer and associated virtual machines work together as one cluster make good use of resources of each vServer that are scattered into different data centers or vServers. VMs have their lifecycles from VM creation, VM startup to VM poweroff and VM deletion. The VMs may also move across the participating virtualization hosts (e.g., the virtualization server or hypervisor). One example is, as the workload on one physical server increases or physical server needs upgrade, VMs can be moved to other available lightweight-workload servers to ensure that service level agreement and response time requirements are met. We call this VM movement or relocation as VM migration. When the workload decreases, the VMs can be moved back, allowing the unused server powered off to save cost and energy. Another example is as one tenant moves, VMs associated with this tenant may also move to the place that is more close to the tenant and provides better user experience (e.g., larger bandwidth with lower latency). We call such movements as VM mobility. VM migration refers to the transfer of a VM image including memory, storage and network connectivity while VM mobility refers to sending data to the moving tenant associated with the VM and emphasizes service non-disruption during a tenant's movement. This document advocates the distinction between VM mobility and VM migration, both important notions in VM management. The implication is that different signaling or protocols for VM mobility and VM migration might be chosen to automate Network Management for VM Movement, thus possibly reusing the existing protocols or schemes to manage VM migration and VM mobility separately. Unfortunately we sometimes mixed them up or don't distinct VM migration management from VM mobility management and intend to utilize one common protocol to support both VM migration and VM mobility, which seems to simplify the overall protocol design but it is difficult or impossible to run one such protocol across both VM mobility management system that manages VM mobility and VM management platform that manages VM attributes. This document discusses how VMs,vServer and overlay network to which VMs are connecting are managed, signaling for VM, overlay network management and argues VMs need management or control functionality support but can be managed without VM mobility functionality support. Schott & Wu Expires August 22, 2013 [Page 4] Internet-Draft NVO3 Architecture February 2013 2. Terminology 2.1. Standards Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119]. Schott & Wu Expires August 22, 2013 [Page 5] Internet-Draft NVO3 Architecture February 2013 3. Discussions 3.1. VM awareness and VM movement awareness Virtual machines usually operate under the control of a server virtualization software residing on the physical compute server. The server virtualization software is commonly referred to as 'hypervisor'. The hypervisor is the container of the VM and provides shared compute/memory/storage and network connectivity to each VM that it hosts. Therefore the hypervisor or the virtualized server MUST be aware of VMs that it hosts. However it should not be aware of VMs that it doesn't host. When VMs hosted in different virtualization servers need to communicate each other, packets from one VM will be forwarded by a virtual switch within the virtualization server or the hypervisor to other VMs on another virtualization server. Since the virtual switch resides within the hypervisor or virtualization server, the rule on VM awareness applied to the hypervisor should apply to virtual switch too. Unlike VM awareness, VM movement awareness is the capability of knowing the location update of the VM. For example, when a VM moves out of the hypervisor and goes to another host, the original hypervisor that hosts the VM is aware of VM movement or location changing but may not be able to keep track of the new location after the VM moves. Therefore one external party that maintains the mapping between the VM's identity and the VM's current location is needed which keeps track VM movements. 3.2. Why VM migration VM migration refers to VM movement within a virtual environment in response to events, conditions or based on requirements. The events or conditions could be, for example, very high workloads experienced by the VMs or upgrades of the physical server or virtualization server, load balancing between virtualization servers. The requirements could be, for example, low power and low cost requirements or service continuity requirement. When a VM is moved without service disruption, we usually call this VM movement as VM mobility. However it is very difficult to provide transparent VM mobility support since it not only needs to keep connection uninterrupted but also needs to move the whole VM image from one place to another place, which may take a long down time (e.g., more than 400 ms) and can be noticed by the end user. Fortunately, VMs may be migrated without VM mobility support. For example, a server manager or administrator can move a running virtual machine or application between different physical machines without disconnecting the client or application if the client or application Schott & Wu Expires August 22, 2013 [Page 6] Internet-Draft NVO3 Architecture February 2013 supports VM suspending and resuming operation or stopped at the source before the movement and restart at the destination after movement. In some case when VM mobility is really needed, it is recommended that one copy of VM SHOULD be replicated from the source to the destination and during VM replication, thus the VM running on the source should not be affected. When VM replication to the destination completes and the VM on the destination restarts, the VM on the source can be stopped. However how the VM on the destination coordinates with the VM on the source to know whom the latter is communicating with is a challenging issue. 3.3. Who manages VM To ensure the quality of applications (e.g., real-time applications) or provide the same service level agreement, the VM's state(i.e., the network attributes and policies associated with the VM ) should be moved with the VM as well when the VM moves across participating virtualization hosts (e.g., virtualization server or hypervisor). These network attributes associated with VM should be enforced on the physical switch and the virtual switch corresponding to VM to avoid security and access threats. The hypervisor or the virtualization server may maintain the network attributes for each VM. However when VMs are moved from the previous server to the new server, the old server and the new server may have no means to find each other. Therefore one external party called VM network management system (e.g., Cloud Broker) is needed and should get involved to coordinate between the old server and the new server to establish the association between network attributes/policies and the VM's identity. If the VM management system does not span across data center and the VM is moved between data centers, the VM network management system in one data center may also need to coordinate with VM network management system in another data center. 3.4. VM Grouping VM grouping significantly simplifies the administration tasks when managing large numbers of virtual machines, as new VMs are simply added to existing groups. With grouping, similar VMs can be grouped together and assigned with the same networking policies to all members of the group to ensure consistent allocation of resources and security measures to meet service level goals. Members of the group retain the group attributes wherever they are located or move within the virtual environment, providing a basis for dynamic policy assignments. VM groups can be maintained or distributed on the virtualization server or can be maintained on a centralized place, e.g., the VM management platform that manages all the virtualization Schott & Wu Expires August 22, 2013 [Page 7] Internet-Draft NVO3 Architecture February 2013 servers in the data center. VM groups maintained on each virtualization server may change at any time due to various VM operations (e.g., VM adding, VM removing, VM moving). Therefore VM groups need to be synchronized with the central VM management platform. Profiles containing network configurations such as VLAN, traffic shaping and ACLs for VM groups can be automatically synchronized to the central VM management platform as well. This way, consistent network policies can be enforced regardless of the VM's location. 3.5. What VM information should be managed The ability to identify VMs within the physical hosts is very important. With the ability to identify each VM uniquely, the administrator can apply the same philosophy to VMs as used with physical servers. VLAN and QoS settings can be provisioned and ACL attributes can be set at a VM level with permit and deny actions based on layer 2 to layer 4 information. In the VM environment, a VM is usually identified by MAC or IP address and belongs to one tenant. Typically, one tenant may possess of one VM or a group of VMs in one virtual network or several groups of VMs distributed in multiple virtual networks. On the request of the tenant, a VM can be added, removed and moved by the virtualization server or the hypervisor. When the VM moves, the network attributes or configuration attributes associated with the VM should also be moved with the VM as well to ensure that the service level agreement and response times are met. These network attributes include access and tunnel policies and (L2 and/or L3) forwarding functions and should be moved with the VM information. We use Virtual Network Instance ID to represent those network attributes. One tenant has at least one Virtual Network ID. Therefore each tenant should at least include the following information: o vCenter Name or Identifier o vServer Name or Identifier o Virtual Network ID (VNID) o VLAN tag value o VM Group ID o VM MAC/IP address Note that Tenant may have tenant ID which could be combination of these information. Schott & Wu Expires August 22, 2013 [Page 8] Internet-Draft NVO3 Architecture February 2013 3.6. Who Triggers or Controls VM Movements VM can be moved within the virtual environment in response to events or conditions. An issue here is who triggers and controls VM movement? In a small scale or large scale data center, the server administrator is usually not aware of VM movement and may respond quickly to system fault or server overload and move a virtual machine or a group of VMs to different physical machines. However it is hard for the server administrator to response to dynamic VM movement and creation since he doesn't keep track of VM movements. In large scale data centers, the server administrator may be more hesitated to utilize VM movements because of the time demands of managing the related networking requirements. Therefore automated solutions that safely create and move virtual machines and free VM network or Server administrators from their responsibilities is highly required. The external party (i.e., the control or management plane function) is needed to play the role of server administrator and should support keeping track of VM movement and response quickly to dynamic VM creation and movement. When one tenant moves from one place to another place, VM movement associated tenant should be informed to the control or management plane function. When one tenant requests to improve the quality of application and shorten the response time, the control or management function can trigger VM being moved to the server that is closer to the user. 3.7. VM Monitoring In order to sort out bad VMs, VM monitoring is very important. The VM monitor mechanism keeps track of the availability of VMs and their resource entitlements and utilization, e.g., CPU utilization, Disk and memory utilization, network utilization, network storage utilization,. It ensures that there is no overloading of resources whereby many service requests cannot be simultaneously fulfilled due to limited resource available. VM monitor is also useful for server administrations and report the status information of VMs or VM groups in each server to the VM management and provision system. Schott & Wu Expires August 22, 2013 [Page 9] Internet-Draft NVO3 Architecture February 2013 4. Use Cases 4.1. On Demand Network Provision Automation within the data center The Tenant Alice is logging into user portal via her laptop and request playing Cloud gaming using VM she has already rented, the request is redirected to the provision system vCenter, the vCenter retrieves service configuration information and locate which vServer the VM belongs to and then Provision resources for VM running on that vServer. The vServer signals the VM operation parameter update to the NVE to which the VM is connecting. In turn, the NVE device interacts with the DC nDirector to configure policy and populate the forwarding table to each network element (e.g., ToR,DC GW), in the path from the Tenant End System to the NVE Device. In addition, the DC nDirector may also populate the mapping table to map the destination address (either L2 or L3) of a packet received from a VM into the corresponding destination address of the NVE device. +--------+ +--------+ | | | | | Tenant | | User | |(Client)|--------> Portal | | | | | --------- +---|----+ +---+----+ /--- - -----\ | | /// \\\ | | // \\ | +---+---+/ +---------+ \\ | | | Signaling Path | | \ | |vCenter+--------------------- vServer | \ | | |1.Resource Provision| | | | +-------+ +---------+ | | |VM VM VM | | | +----+----+ | | | | | +---------+ | | -------------| DC | +----+----+ | |nDirector| Data Path | | | | /Oracle |<----------------->|NVEDevice| | | | | | / +---------+ +---------+ / 2.Network Policy control // \\ // \\\ /// \----- -----/ --------- Schott & Wu Expires August 22, 2013 [Page 10] Internet-Draft NVO3 Architecture February 2013 4.2. Large inter-data centers Layer 2 interconnection and data forwarding When the tenant Alice using VM1 in data center1 communicates with tenant Bob using VM2 in data center2, the VM1 should already know layer2 identity of VM2, however the VM1 may not know which NVE Edge the VM2 is placed behind, in order to learn the location of the remote NVE Device associated VM2, the mapping table is needed on the local NVE Device associated with VM1 which is used to map the final destination(i.e.,the identity of VM2) to the destination address of the remote NVE device associated with VM2. In order to realize this, the nDirector should tell the local NVE device associated with VM1 about layer 3 location identifier of remote NVE device associated with VM2 and establish mapping between layer 2 VM2 identity and layer 3 identity of the remote NVE Edge associated with VM2. In addition, the nDirector may tell all the remote NVE devices associated with the VM which the VM1 is communicating with to establish the mapping between layer 2 VM1 identity and layer 3 identity of the local NVE Device associated with VM1. When this is done, the data packet from VM1 can be sent to the NVE device associated with VM1, the NVE Device associated with VM1 can identify layer 2 frame targeted for remote destination based on established mapping table, encapsulates it into IP packet and transmit it across layer 3 network. After the packet arrives at the remote NVE Edge, the remove NVE Edge device decapsulates layer 3 packet, take out layer 2 frame and forward it to ultimate destination VM2. Schott & Wu Expires August 22, 2013 [Page 11] Internet-Draft NVO3 Architecture February 2013 +---------+ | DC | |nDirector| +----------+ /Oracle |-----------+ | | | | | +---------+ | | ---+---- ---+--- ---- | ---- /---- | ----\ /// | \\\ // | \\ // | \\ // +-------+---+ \\ / +------+----+ \ / | | \ / | | \ / | NVE Edge1 | \ / | NVE Edge2 | \ / | | \ | | | | | +-----------+ || +-----------+ | | || | | +---------+ +---------+ || +---------+ +---------+ | | | | | | || | | | | | | | vServer | | vServer | || | vServer | | vServer | | | | | | | || | | | | | | +---------+ +---------+ || +---------+ +---------+ | | |VM VM VM | |VM VM VM | || |VM VM VM | |VM VM VM | | | +---------+ +---------+ | \ +---------+ +---------+ / \ / \ / \ / \ / \ / \\ // \\ // \\\ /// \\ // ---- ---- \---- ----/ -------- ------- 4.3. Enable multiple data centers present as one In order to support more data centers interconnection and enable more efficient use of resources in each data center, multiple data centers may closely coordinate with each other to better load balancing capability and work as one large DC with the involvement of the nDirector that manages DCs, e.g., DC nDirector in each data center may coordinate with each other and form one common control plane. Schott & Wu Expires August 22, 2013 [Page 12] Internet-Draft NVO3 Architecture February 2013 ----------------------- //////// \\\\\\\\ |/// \\\| | | | Internet Backbone | | | |\\\ ///| \\\\\\\\ //////// ----------------------- +-------------------------------------------------------------+ |One Unified Director | | +-------+ +-------+ +---------+ | | |egress | |egress | | egress | | | | GW1/ | | GW2/ | | GW3/ | | | |nDirector |nDirector |nDirector| | | +----+--+ +--+----+ +----+----+ | +-----------+-----------------+-----------------+-------------+ | | | /---+--\ /--+---\ /--+---\ //+--------+\\ //+--------+\\ //+--------+\\ | | DC1 | | | | DC2 | | | | DC3 | | | | | | | | | | | | | | \\+--------+// \\+--------+// \\+--------+// \------/ \------/ \------/ 4.4. VM migration and mobility across data centers The Tenant Alice is using VM1 in data center 1 to communicate with the tenant Bob who is using VM9 in data center 2. For business reason, the tenant Alice travels to the Bob's city where the data center 2 situates but still use VM1 in the data center 1 to communicate with the tenant Bob. In order to provide better user experience, the VM1 may be move from vServer 1 to the new vServer3 in the data center 2 which is more close to where the tenant Alice is located. The vCenter can get involved to interact with data center 1 and data center2 and help replicate and relocate VM1 to the new location. In the meanwhile ,when VM movement is done, the NVE device connecting to VM1 and associated with vServer 3 should interact with the nDirector to update mapping table maintained in the nDirector by the new NVE device location associated with VM1. In turn, the nDirector should update the mapping tables in all the NVE device associated with the VM which VM1 is communicating with. Schott & Wu Expires August 22, 2013 [Page 13] Internet-Draft NVO3 Architecture February 2013 +---------+ | | |nDirector| +----------+ /Oracle |-----------+ | | | | | +---------+ | | ---+---- +--------+ ---+--- ---- | -|vCenter2| /---- | ----\ /// DC2 | +--------+ // DC1 | +--------+ // +-----+--------+ \\ // +-------+---+ |vCenter1| / | | \ / | | +--------+ / +---------+ +---------+\ / | NVE Edge1 | / |NVE Edge3| |NVE Edge4| / | | \ | +----+----+ +----+----+ | | +-------+---+ -|| | | | | +------+------| || | | | | +-----+---+ +-----+---+ || +---------+ +----+----+ | | | | | | || | | | | | | | vServer1| | vServer2| || | vServer3| | vServer4| | | | | | | || | | | | | | +---------+ +---------+ || +---------+ +---------+ | | |VM1VM2VM3| |VM4VM5VM6| || |VM1VM7VM8| | VM9 | | | +---------+ +---------+ | \ +---------+ +---------+ / \ / \ / ----------------------------/ ---------------------------- +--------+ +--------+ | | | | | Tenant | +------------------------> | Tenant | | Alice | | Alice | +--------+ +--------+ Schott & Wu Expires August 22, 2013 [Page 14] Internet-Draft NVO3 Architecture February 2013 5. General Network Virtualization Architecture When Multiple virtual machines (VMs) created in one vServer,VM can be managed under this vServer. However vServer can not be isolated node since VM can be moved from one to another vServer under the same or different data center which is beyond the control of the vServer who create that VM. We envision the Network virtualization architecture to consist of vServers (virtualization servers),nDirector and vCenters (the aforementioned VM and vServer management platform) and NVE Edges. The vCenter is placed on the management plane within each data center and can be used to manage a large number of vServers in each data center. The vServer is connecting to NVE Edge in its own data center either directly or via a switched network (typically Ethernet). The nDirector is placed on the control plane and manage one or multiple data centers. When the nDirector manages multiple data centers, the nDirector should interact with all the NVE Edges in each data center to facilitate large inter-data center Layer 2 interconnection, VM migration and mobility across data centers and enabling multiple data centers work and present as one. Schott & Wu Expires August 22, 2013 [Page 15] Internet-Draft NVO3 Architecture February 2013 ... .... .................... ................... . DC1 +--------+ +---------+. . | | | |. . |vServer1+------+NVE Edge1+---| . +---| | | |. | . | +--------+ +---------+. | . +--------+ | |VM VM VM| . | . | | | ---------+ . | . |vCenter1+---| . | . | | | +--------+ +---------+. | . +--------+ | | | | |. | . | |vServer2+------|NVE Edge2+---| . +---| | | |. | . +--------+ +---------+. | . |VM VM VM| . | +---------+ . ---------+ . | |nDirector| ... .... .................... .................. +-+ /Oracle | ... .... .................... .................. | | | .DC2 +--------+ . | +---------+ . | | . | . ---|vServer3+---| . | . | | | | . | . | +--------+ | +---------+. | . +--------+ | |VM VM VM| | | |. | . | | | ---------+ +--|NVE Edgex+---| . |vCenter2|--+ | | |. . | | | +--------+ | +---------+. . +--------+ | | | | . . | |vServerx+---| . . +---| | . . +--------+ . . |VM VM VM| . . ---------+ . . .... .................... ..................... Network Virtualization Architecture 5.1. NVE (Network Virtualization Edge Function) As defined in section 1.2 of [I.D-ietf-nvo3-framework],it is a network entity that sits on the edge of the NVO3 network and could be implemented as part of a virtual switch within a hypervisor, a physical switch or router, a Network Service Appliance(e.g.,NAT/ FW).When VM1 connecting to one NVE Edge want to communicate with the other VMs which are connecting to the other NVE Edges, the NVE Edge associated with VM1 should distribute the mapping between layer 2 identity of VM1 and NVE Edge associated with VM1 by the nDirector to all the NVE Edges associated with VMs which VM1 is communicating Schott & Wu Expires August 22, 2013 [Page 16] Internet-Draft NVO3 Architecture February 2013 with. In addition, the NVE Edge associated with VM1 either interact with the nDirector or learn from the other NVE Edges who is distributing mapping table through the nDirector to build mapping table between layer 2 identity of VMs which VM1 is communicating with the NVE Edge associated with VMs which VM1 is communicating with and based on such mapping table to forward the data packets. 5.2. vServer (virtualization Server) The vServer is served as a platform for running virtual machines and is installed on the physical hardware in a virtualized environment and provide physical hardware resource dynamically to the virtual machines as needed. It is also referred to as "the virtualization server" or hypervisor. It may get instructions from provision systems (i.e.,vCenters)to create, modify, terminate VM for each tenant. It may also interact with the NVE Edge to inform the NVE about the map or association between vserver, virtual machine and network connection. This interaction can also be used to release association between vServer and the NVE Edge. 5.3. vCenter (management plane function) The vCenter is served as a platform for managing in one data center not only assignment of virtual machines to the vServer but also assignment of resources to the virtual machines and provide a single control point to the data center. It unifies the resources from individual vServer to be shared among virtual machines in the entire data center. It may interact with vServer to allocate virtual machines to the vServer and monitor performance of each vServer and each VM in the data center. The vCenter may maintain the map from vServer to Network connection which contain not not only vServer configurations such as vServer name, vServer IP address port number but also VM configurations for each tenant end system associated with that vServer. When vCenter hierarchy is used, the root vCenter who has global view may interact with the child vCenter to decide which child vCenter is responsible for assigning the virtual machine to which vServer based on topological information and resource utilization information in each data center and local policy information. 5.4. nDirector (Control plane function) The nDirector is implemented as part of DC Gateway and sits on top of the vCenter in each data center and is served as orchestrator layer to allow layer 2 interconnection and forwarding between data centers and enable multiple data centers to present as one. The nDirector may interact with the NVE Edge to populate forwarding table in the path from the NVE Edge Device to the Tenant End System and react to Schott & Wu Expires August 22, 2013 [Page 17] Internet-Draft NVO3 Architecture February 2013 the NVE request to assign network attributes such as VLAN,ACL, QoS parameters on all the network elements in the path from NVE device to the Tenant End System and manipulates the QoS control information in the path between the NVE Edges associated with VMs in communication. In addition, the nDirector may distribute mapping table between layer 2 identity of VM and the NVE Edge associate with that VM to all the other NVE Edges and maintain such mapping table in the nDirector. Schott & Wu Expires August 22, 2013 [Page 18] Internet-Draft NVO3 Architecture February 2013 6. vServer to vCenter management interface 6.1. VM Creation vCenter requests vServer to create a new virtual machine and allocate the resource for its execution. 6.2. VM Termination vCenter requests vServer to delete a virtual machine and clean up the underlying resources for that virtual machine. 6.3. VM Registration When a VM is created for one tenant in the vServer, the vServer may create VM profile for this tenant containing VM identity,VNID, port,VID and registers the VM configuration associated with this tenant to the vCenter. Upon receiving such a registration request, vCenter should check if it has already established VM profile for the corresponding tenant: if yes, vCenter should update the existing VM profile for that tenant, otherwise vCenter should create a new VM profile for that tenant. 6.4. VM Unregistration When a VM is removed for one tenant from the vServer, the vServer may remove VM profile for this tenant containing VM identity, VNID, port,VID and deregisters the VM configuration associated with that tenant to the vCenter. Upon receiving such a deregistration request, vCenter should check if it has already established VM profile for that tenant: if yes, vCenter should remove the existing VM profile for that tenant,otherwise other vCenter should report alert to the vServer. 6.5. VM Bulk Registration When a large number of VMs are created in one vServer and share the same template, the vSever may create a profile for a group of these VMs and send a bulk registration request containing the group identifier and associated VM profile to the vCenter. Upon receiving such a bulk registration request, vCenter should create or update the profile for a group of these VMs. 6.6. VM Bulk Unregistration When a large number of VMs are removed in one vServer and share the same template, the vSever may remove a profile for a group of these VMs and send a bulk unregistration request containing the group Schott & Wu Expires August 22, 2013 [Page 19] Internet-Draft NVO3 Architecture February 2013 identifier and associated VM profile to the vCenter. Upon receiving such a bulk registration request, vCenter should remove the profile for a group of these VMs. 6.7. VM Configuration Modification vCenter requests vServer to update a virtual machine and reallocate the resource for its execution. 6.8. VM Profile Lookup/Discovery When a VM1 in one vServer want to communicate with one VM2 in another vServer, the client associated with VM1 should check with vCenter based on VM2 identity to see if the profile for that VM2 already exists and which server maintains that VM configuration. If yes, vCenter should reply to the the client with the address or name of the vServer which the VM2 is situated in. 6.9. VM Relocation When vCenter is triggered to move one VM or a group of VMs from one source vServer to another destination vServer, the vCenter should send a VM relocation request to both vServers and updates its profile to indicate the new vServer that maintains the VM configuration for that VM or a group of those VMs. The relocation request will trigger the VM image to be moved from the source vServer to the destination vServer. 6.10. VM Replication One tenant moves between vServers or between data centers and may, as the internet user, want to access applications via the VM without service disruption. In order to achieve this, he can choose to access applications via the same VM without moving the VM when he moves. However, the VM he is using may be far away from where he stays. In order to provide better user experience, the tenant may request vCenter through the nDirector to move VM to the vServer that is more close to where he stays and keeps the service uninterrupted. In such case, the vCenter may interact with the vServer that hosts the original VM to chooses one vServer that is closer to the tenant and moves one copy of the VM image to the destination vServer. 6.11. VM Report When one VM is created, moved, added, removed from the vServer, the VM monitor should be enabled to report the status information and resource availability of that VM to the vCenter. In this case, vCenter can know which server is overloaded, which server is unused Schott & Wu Expires August 22, 2013 [Page 20] Internet-Draft NVO3 Architecture February 2013 or least used. Schott & Wu Expires August 22, 2013 [Page 21] Internet-Draft NVO3 Architecture February 2013 7. nDirector to NVE Edge control interface Signaling between the nDirector and NVE Device can be used to do three things: Enforce the network policy for each VM in the path from the NVE Edge associated with VM to the Tenant End System. Populate forwarding table in the path from the NVE Edge associated with VM to the Tenant End System in the data center. Populate mapping table in each NVE Edge that is in the virtual network across data centers under the control of the nDirector. One could reuse existing protocols, among them NetConf,SNMP,RSVP,Radius,Diameter, to signal the messages between nDirector and NVE Edges. The nDirector need to know which NVE Edges belong to the same virtual network and then the distribute the routes between these NVE Edges to each NVE Edges belonging to the same virtual network. In additional the nDirector may interact with the NVE Edge and the associated overlay network in the data center in response to the provision request from the NVE Edge and populate forwarding table to the associated overlay Network elements in the data path from the Tenant End System to the NVE Edge and install network policy to the network elements in the data path between the Tenant End System and the NVE Edge. For details of Signaling control/forward plane information between network virtualization edges (NVEs) , please see [I.D-wu-nvo3-nve2nve]. Schott & Wu Expires August 22, 2013 [Page 22] Internet-Draft NVO3 Architecture February 2013 8. vServer to NVE Edge control interface Signaling between vServer and NVE Edge is used to establish mapping between the vServer who host VM and network connection which VM relies on. For more details signaling and operation, please see relevant NVO3 draft. Schott & Wu Expires August 22, 2013 [Page 23] Internet-Draft NVO3 Architecture February 2013 9. Security Considerations Threats may arise when VMs move into a hostile VM environment, e.g., when the VM identity is exploited by adversaries to launch denial of service or Phishing attacks[Phishing]. Further details are to be explored in the future version of this document. Schott & Wu Expires August 22, 2013 [Page 24] Internet-Draft NVO3 Architecture February 2013 10. IANA Considerations This document has no actions for IANA. Schott & Wu Expires August 22, 2013 [Page 25] Internet-Draft NVO3 Architecture February 2013 11. Contributors Thank Xiaoming Fu for helping provide input to the initial draft of this document. Schott & Wu Expires August 22, 2013 [Page 26] Internet-Draft NVO3 Architecture February 2013 12. References 12.1. Normative References [I.D-ietf-nvo3-framework] Lasserre, M., "Framework for DC Network Virtualization", ID draft-ietf-nvo3-framework-00, September 2012. [I.D-wu-nvo3-nve2nve] Wu, Q., "Signaling control/forward plane information between network virtualization edges (NVEs)", ID draft-wu-nvo3-nve2nve-00, 2013. [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", March 1997. 12.2. Informative References [I.D-kompella-nvo3-server2nve] Kompella, K., "Using Signaling to Simplify Network Virtualization Provisioning", ID draft-kompella-nvo3-server2nve, July 2012. [Phishing] "http://kea.hubpages.com/hub/What-is-Phishing". Schott & Wu Expires August 22, 2013 [Page 27] Internet-Draft NVO3 Architecture February 2013 Authors' Addresses Roland Schott Deutsche Telekom Laboratories Deutsche-Telekom-Allee 7 Darmstadt 64295 Germany Email: Roland.Schott@telekom.de Qin Wu Huawei 101 Software Avenue, Yuhua District Nanjing, Jiangsu 210012 China Email: sunseawq@huawei.com Schott & Wu Expires August 22, 2013 [Page 28]