Network Working Group Liang. Ou Internet-Draft Huamin. Jin Intended status: Informational China Telecom Expires: April 16, 2013 M. Chen Huawei Technologies Co., Ltd October 13, 2012 Content Delivery Network Based Virtual Machine Migration Scheme draft-ou-cdn-vm-migration-scheme-00 Abstract Virtual machine migration is a critical requirement in today's data centers to support services dynamically deployed in distributed infrastructures. To keep the uniform environment required by application software, large number of networking based solutions have been presented to provide data center interconnection for virtual resource management. Aiming at simplifying networking complexities in off-site service deployment, this document presents a framework based on content delivery network for supporting virtual machine migration over the legacy wide area network. Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119]. Status of this Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at http://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on April 16, 2013. Copyright Notice Ou, et al. Expires April 16, 2013 [Page 1] Internet-Draft CDN based VM Migration scheme October 2012 Copyright (c) 2012 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4 3. CDN based Migration Architecture . . . . . . . . . . . . . . . 4 3.1. Functional Architecture . . . . . . . . . . . . . . . . . 4 3.2. Protocols . . . . . . . . . . . . . . . . . . . . . . . . 6 3.3. Mapping Tables in RMG . . . . . . . . . . . . . . . . . . 7 4. Use Cases . . . . . . . . . . . . . . . . . . . . . . . . . . 8 4.1. Batch Migration . . . . . . . . . . . . . . . . . . . . . 8 4.2. Multi-nodes Backup . . . . . . . . . . . . . . . . . . . . 9 4.3. Virtual Desktop Migration . . . . . . . . . . . . . . . . 11 5. Implementation Consideration . . . . . . . . . . . . . . . . . 11 5.1. Discussion . . . . . . . . . . . . . . . . . . . . . . . . 11 5.2. Experimental Work . . . . . . . . . . . . . . . . . . . . 12 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 12 7. Security Considerations . . . . . . . . . . . . . . . . . . . 12 8. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 12 9. Normative References . . . . . . . . . . . . . . . . . . . . . 12 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 12 Ou, et al. Expires April 16, 2013 [Page 2] Internet-Draft CDN based VM Migration scheme October 2012 1. Introduction In an infrastructure-as-a-service (IaaS) system, IT infrastructure is deployed in a provider's data center as virtual machine (VM). Managing a VM's full life-cycle mainly includes setting up networks dynamically for groups of VMs and their storage requirements, such as VM disk image deployment or on-the-fly software environment creation. With IaaS cloud growing popularity, VM migration technology has been widely used in off-site database backup, new services deployment and server system maintenance or upgrade. Basically, VM migration is categorized into two types: live-migration (real-time) and off-line migration (non real-time). The former migration procedure is typically controlled by VM manager, which involves real-time status synchronization, including networks, memory and storage, between two hosts where VMs locate. Due to strict requirement to network performance, most former methods (application- level or process-level) are only limited within a data center instead of data center interconnection (DCI) scope to prevent poor migration performance or system exception. Not migrating VM in run-time, the later VM migration can be implemented by sending VM image files (static disk image and dynamic snapshot) from source host to destination. That is to say, file-level data transfer between hosts is efficient enough for non real-time VM migration when network quality is favorable. It also means dependence on expensive and sophisticated DCI solutions is largely reduced. As a fast file delivery and cache system, content delivery network (CDN) provides not only high performance data transfer channel but also easy-to-go control plane. Thus, DCI solution should take CDN based scheme into consideration for VM migration. This document only focus on overall framework, scenarios and implementation consideration for CDN based VM migration. Completed comparisons between VM migration approaches and related networking issues are outside the scope of this document. Motivation behind this document can be concluded as follows:. 1.Provide a simple transition scheme before evolving to future's cost efficient DCI solution in large scale, and give a simple option to those who don't want to take risk of complex DCI in current phrase. 2.Protect investment on legacy infrastructure. Typically, for some Asia telecom carries, who not only own broadband network and data center infrastructure but also a national wide CDN. Comparing CDN system upgrade, it is a big burden to construct a new DCI specified data plane over underlying broadband network in the near future. Ou, et al. Expires April 16, 2013 [Page 3] Internet-Draft CDN based VM Migration scheme October 2012 2. Terminology Virtual Resource Node (VRN): Data centers in which VM migration happens are abstracted as VRN. VRN consists of host, storage and distributed file system in a data center. Before migrating from one to another, virtual resource corresponding to VM is stored in the format of VM image files in VRN, and then is delivered from source VRN to destination VRN across network. Cloud Virtual Machine Manager (CVMM): Different from typical virtual machine management (VMM) located in data center internally, CVMM is an infrastructure manager that deploys virtualized services on both local pool of resources and external IaaS cloud to support virtual resource life-cycle management. CDN Management Node (CMN): CMN provides content management for cache node in CDN network, including content adding, content routing, content delivery, resource discovery function. Only one CMN in a CDN system. CDN Cache Node (CCN): CCN carries VM images for migration received from source VRN, and deliver those images to destination VRN via its high speed data channel provided by CDN interconnected network which is typically constructed over legacy L3 network. Any cache node in legacy CDN system could be a CCN which could receive/deliver image files from/to VRN.Thus, the CCN is also partitioned into source CCN and destination CCN. Resource Migration Gateway (RMG): RMG is a control system that supports virtual resources in a cloud to migrate across wide area network via CDN system. Its interface with CVMM parses migration instruction and reports related migration status. While the interface with CMN is used to apply for data channel and cache space from CDN for arriving VM images. RMG also finishes some conversion or mapping operation, including content /image format, content/image ID, content location and content routing, between CDN system and VM file system. 3. CDN based Migration Architecture 3.1. Functional Architecture Ou, et al. Expires April 16, 2013 [Page 4] Internet-Draft CDN based VM Migration scheme October 2012 +-----------+ +------+ +------------+ Cloud | VRN (SRC) |<------>| CVMM |<------>| VRN (DEST) | Layer +-----------+ +------+ +------------+ | | | | +------+ | .............|..............| RMG |..............|.......... | +------+ | | | | CDN +-----------+ +------+ +-----------+ Layer | CCN (SRC) |<------>| CMN |<------>| CCN (DEST)| +-----------+ +------+ +-----------+ .............|....................................|......... | | Network +-----------------------------------------------+ Layer | Physical Network | +-----------------------------------------------+ RMG - Resource Migration Gateway CCN - CDN Cache Node VRN - Virtual Resource Node CMN - CDN Management Node CVMM - Cloud Virtual Machine Manager Figure 1 CDN Based VM Migration Architecture The functional architecture (as depicted in the Figure 1) describes how RMG, cloud manager and CDN management element collaborate on fulfilling VM migration via CDN system which set up over legacy physical L2/L3 network. Here, one key role of the RMG is to decouple the signaling dependency between cloud system and CDN system. In terms of mechanism outlined below, the VM images are able to be transferred from upper cloud layer (SRC VRN) to CDN and network layer, and then back to the cloud layer (DEST VRN). The typical workflow is summarized as following steps: 1. CVMM receives VM migration instruction in two ways: manually requested by cloud administrator or automatically triggered by user applications. In either case, CVMM records current virtual resource status and clones necessary disk image from the physical machine that the VM is hosted. 2. CVMM identifies this migration operation belong to local or remote. If it is remote migration, CVMM sends a migration request which also contains source and destination information about VRNs. 3. RMG checks its internal VRN-CCN mapping table to get information about the CCN directly connected with SRC or DEST VRN. Then, RMG sends request to CMN with dedicated protocol which includes SRC and DEST nodes information about CCN and size or identification of VM Ou, et al. Expires April 16, 2013 [Page 5] Internet-Draft CDN based VM Migration scheme October 2012 image. 4. Receiving message from RMG, CMN first requires whether the DEST CCN or network is busy or not. If the DEST is congested, CMN will suggest RMG to modify the destination for VRN. Otherwise, it will check its internal database to judge if CDN system has cached such image file before. If there is an existing image copy, CMN will push the image file to DEST CCN, and then informs RMG to receive the image from the VRN connected. 5. If there is no VM image in CDN system at current phrase, CMN thereafter responds the RMG's requesting for image transport. And RMG send permitting message for VM migration to CVMM as well. Finally, CVMM informs VRNs (SRC and DEST) to start up its file transfer process and keep status for both. CVMM, RMG and CMN maintain necessary dialog during images transfer phrase to refresh status messages. 6. Once the DEST VRN successfully receives image file from DEST CCN, new VM is created in the DEST VRN's internal physical host. 3.2. Protocols +--------+ C-R +-------+ R-C +-------+ | CVMM |<--------->| RMG |<--------->| CMN | +--------+ Protocol +-------+ Protocol +-------+ Negotiation Protocol between CVMM, RMG and CMN +--------+ V-C +-------+ | VRN |<--------->| CCN | +--------+ Protocol +-------+ Transport Protocol between VRN and CCN Figure 2 Reference Protocols There are two type protocols in suggest CDN based VM migration scheme (see Figure 2): control protocol between CVMM,RMG and CMN, and data transfer protocol between VRN and CCN. 1. C-R Protocol TBD 2. R-C Protocol TBD Ou, et al. Expires April 16, 2013 [Page 6] Internet-Draft CDN based VM Migration scheme October 2012 3. V-C Protocol TBD 3.3. Mapping Tables in RMG +-------+--------+--------+--------+--------+-------+--------+-------+ |CCN ID | CCN | VRN ID | VRN |HOST ID | VM ID |IMAGE ID| Status| | |IP addr.| |IP addr.| | | | | +-------+--------+--------+--------+--------+-------+--------+-------+ | | | | | | | | | +-------+--------+--------+--------+--------+-------+--------+-------+ | SH 1# |10.3.1.2| DC 2# |1.1.3.2 | west1 | Xen-10|Linux-2 | SRC | +-------+--------+--------+--------+--------+-------+--------+-------+ | BJ 2# |3.3.1.2 | DC 1# |1.1.3.2 | west1 | kvm-07|Win2k-1 | DEST | +-------+--------+--------+--------+--------+-------+--------+-------+ Tabel 1: CDN-VRN Content Mapping Table +-------+--------+-------+--------+--------+--------+-------+--------+ | SRC |SRC CCN | SRC |SRC VRN| DEST |DEST CCN| DEST |DEST VRN| |CCN ID |IP addr.| VRN ID|IP addr.| CCN ID |IP addr.|VRN ID |IP addr.| +-------+--------+-------+--------+--------+--------+-------+--------+ | | | | | | | | | +-------+--------+-------+--------+--------+--------+-------+--------+ | | | | | | | | | +-------+--------+-------+--------+--------+--------+-------+--------+ Tabel 2: CDN-VRN Location Mapping Table Figure 3 Mapping Tables in RMG Figure 3 illustrates mapping tables stored in RMG to reflect how to assign VRN's VM images to CCN cache system 1. CDN-VRN Content Mapping Table CCN ID: the identification of CCN supporting VMs migration, directly connecting to a VRN with VRN ID CCN IP addr.: the IP address of CCN with CCN ID VRN ID: the identification of VRN initiating VMs migration, connecting to a CCN with CCN ID VRN IP addr.: the IP address of a concrete CCN with VRN ID Host ID: the identification of host initiating VM migration in a VRN Ou, et al. Expires April 16, 2013 [Page 7] Internet-Draft CDN based VM Migration scheme October 2012 with VRN ID VM ID: the identification of VM to be migrated in a host with HOST ID Image ID: the identification of image file describing a VM with VM ID Status: source or destination node Others: TBD 2. CDN-VRN Location Mapping Table This table shows how to record the source and destination binding relation between VRN-CCN pairs in VM migration duration. 4. Use Cases 4.1. Batch Migration City A City B +-----------+ +------+ +------------+ Cloud | Src VRN |<------>| CVMM |<------>| Dest VRN | | (*v1,*v2) | | | |(*v1',*v2') | Layer +-----------+ +------+ +------------+ | | ^ |(*v1',*v2') +------+ |(*v1',*v2') .............|..............| RMG |..............|.......... | +------+ | V | | CDN +-----------+ +------+ +-----------+ Layer | Src CCN |<------>| CMN |<------>| Dest CCN | |(*v1',*v2')| | | |(*v1',*v2')| +-----------+ +------+ +-----------+ | ^ .............|....................................|......... V | Network +-----+-----------------------------------------+ | | | | | +------------------>-----------------+ | Layer | Physical Network | +-----------------------------------------------+ (*v1,*v2): Two Original VM image "v1" and "v2" in a host (*v1',*v2''): VM image copied from images "v1"and "v2" Figure 4 Batch migrating VM Images from City A to City B Ou, et al. Expires April 16, 2013 [Page 8] Internet-Draft CDN based VM Migration scheme October 2012 Typically, a service deployed in a data center is hosted in different servers or VMs which might belong to different administrative domain. And these virtual or physical resources are orchestrated by cloud manager. Hence all requirements shown in this scenario for service migration are able to be abstracted at the granularity of VMs. Specifically speaking, it is a batch process for multiple VM images transfer via high speed network. This scenario demonstrates how to implement batch migration for VMs images when dedicated service moves from one data center to another geographically separated in two cities. As depicted in Figure 4, image data "v1"and "v2"are cloned in the Src VRN as "v1'"and "v2'". Image copies and then are fast send to Dest VRN over CDN system in which the high speed links for cache nodes are pre-constructed and carefully optimized over physical network. Obviously, VM image transfer over CDN system is better than VRN to VRN connected network in quality and bandwidth. Most steps for negotiation protocol and data transfer are the same as workflows mentioned in Section 3.1. 6. Once the DEST VRN successfully receives image file from DEST CCN, new VM is created in the DEST VRN's internal physical host. 4.2. Multi-nodes Backup Ou, et al. Expires April 16, 2013 [Page 9] Internet-Draft CDN based VM Migration scheme October 2012 +-------------+ City B City A City C | +-------+ | +------------+ +-------------+ +------------+ | | CVMM | | | Dest VRN 1 | | Src VRN | | Dest VRN 2 | | +-------+ |--->| (*v') | | (*v) | | (*v') | | | | +------------+ +-------------+ +------------+ | +-------+ | ^ | ^ | | RMG | | | (*v') | (*v') |(*v') | +-------+ | | V | | | | +------------+ +-------------+ +------------+ | +-------+ | | Dest CCN 1 | | Src CCN | | Dest CCN 2 | | | CMN | |--->| (*v') | | (*v') | | (*v') | | +-------+ | +------------+ +-------------+ +------------+ | Management | ^ | | ^ | Platform | |(*v') (*v')| |(*v') |(*v') +-------------+ | | | | +-----+----------------+-----+---------------+------+ | | | | | | | +-------<--------+ +--------->-----+ | | Physical IP Network | +---------------------------------------------------+ (*v): Original VM image to be migrated in a host (*v'):VM image copied from (*v) Figure 5 VM Image Delivered to Multiple VRNs (Off-site Backup Service) This case illustrates an off-site disaster backup service - that is, how to simultaneously deliver VM image to multiple data centers in different cities. All major steps for control protocol are mentioned in Section 3.1. According to steps in Section 3.1, the management platform will negotiate mutually to discover VRN-CCN pairs matched in terms of mapping table in the RMG before a real migration operation occurs. Once the VRN-CCN pairs are determined, as shown in Figure 5, Src VRN which locates in city A will receive backup instruction from CVMM. It clones the coresponding VM image as "v'" and transfer it to Src CCN connected directly. Even if the image "v'" should be sent to two destination nodes, Src VRN just send "v'" to Src CCN one time only. Because Src CCN "understands" this migration operation with a fork destination. Therefore, the "v' " image will be duplicated inside Src CCN and delivered to corresponding Dest CCNs along with the routes pre- configured by CDN system. Finally, Dest VRN fetches the same image "v'" respectively from Dest CCNs connected. The more backup nodes in the backup services, the higher transfer efficiency will be gained from CDN based migration scheme mentioned in this document. Ou, et al. Expires April 16, 2013 [Page 10] Internet-Draft CDN based VM Migration scheme October 2012 4.3. Virtual Desktop Migration This section illustrates a experience enhancement application for virtual desktop user, which migrates a remote virtual desktop (VM instance) to a local host existing in local CCN. TBD 5. Implementation Consideration 5.1. Discussion 1. Status Synchronization The effort to implement this scheme mainly exists in control protocols between CVMM and RMG, RMG-CMN, and necessary modification on CVMM and CMN system. Such modification may include revising status synchronization mechanism in VMM and re-defining content discovery flow in CMN. Since the scheme we presented is an off-line oriented solution, modification work for migration operations in a CVMM are kept in application level rather than revising hypervisor level. 2. Live-migration or offline-migration One may argue that the scheme mentioned in this document is only suitable for offline-migration cases in which active VMs must be shut down by cloud manager before migration initiates. In view of poor performance in WAN environment and complexity for DCI technologies, offline-migration across WAN is acceptable choice. Considering existence of diverse migration levels, such as progress-level or memory-level, a revised approach for this CDN based scheme is to use live-migration technologies for real status synchronization as supplement. While those static big image files are still transferred in term of our scheme. 3. Routing, Addressing and Content Discovery This scheme just maintains a simple table to configure static mapping between content and routing manually. It cannot deal with dynamic adding/removing either CDN nodes or VRNs. Moreover, maintaining two resource systems information and related location/routing information, RMG is easy to become the bottleneck in the whole system. 4. Engineering issues Ou, et al. Expires April 16, 2013 [Page 11] Internet-Draft CDN based VM Migration scheme October 2012 This scheme is only suitable for those hosting service providers or carriers who retain their own CDN to improve utilization of CDN system and to avoid large scale investment on DCI devices at current phrase. Another issue is that not all data centers are close enough to CDN cache nodes. Thus, connections between data centers and CDN node will become a burden for operators 5. Others TBD 5.2. Experimental Work TBD (Part of initial experiment work have been done to validate mechanism of the scheme, which is based on open source software for CDN and cloud management ) 6. IANA Considerations This document makes no request of IANA. Note to RFC Editor: this section may be removed on publication as an RFC. 7. Security Considerations TBD 8. Acknowledgements 9. Normative References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. Ou, et al. Expires April 16, 2013 [Page 12] Internet-Draft CDN based VM Migration scheme October 2012 Authors' Addresses Liang Ou China Telecom 109, Zhongshan Ave. West Guangzhou, China Email: oul@gsta.com Huamin Jin China Telecom 109, Zhongshan Ave. West Guangzhou, China Email: jhm@gsta.com Mach(Guoyi) Chen Huawei Technologies Co., Ltd No. 3 Xinxi Road, Shang-di, Hai-dian District Beijing 100085 China Email: mach@huawei.com Ou, et al. Expires April 16, 2013 [Page 13]