Network Working Group Ning So (UTD) Internet Draft Young Lee (Huawei) Intended status: Informational Dave McDysan (Verizon) Greg Bernstein (Grotto) Tae Yeon Kim (ETRI) Kohei Shiomoto (NTT) Oscar Gonzalez de Dios (Telefonica) October 13, 2010 Problem Statement for Network Aware Application Resource Assignment and Mobility (NA-ARAM) in Data Center Environments draft-so-network-aware-application-problem-00.txt Status of this Memo This Internet-Draft is submitted to IETF in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. This Internet-Draft will expire on April 13, 2011. Copyright Notice Copyright (c) 2010 IETF Trust and the persons identified as the document authors. All rights reserved. So & Lee, et al. Expires April 13, 2011 [Page 1] Internet-Draft Network Aware Application Resource Assignment & Mobility October 2010 This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Abstract Data Centers offer various application services to end-users such as video gaming, cloud computing and others. As the data centers used to provide application services are distributed geographically around a network many decisions made in the control and management of application services such as where to instantiate another service instance or to which server a new client is assigned, should be made jointly with knowledge of the underlying network resources that are available and with knowledge of the application resources (e.g., server load conditions. Currently such decisions are made with very little or no information concerning the underlying network used to deliver those services. Hence such decisions are inherently sub-optimal. In addition the lack of network awareness may result in not meeting the end-user application service objectives. Table of Contents 1. Introduction..................................................3 1.1. Terminology...............................................3 2. Network Contexts...............................................4 3. Problem Statement..............................................7 4. High-level requirements........................................8 4.1. End-User to Application/DC Provider Communication.........9 4.2. Inter DC communication....................................9 4.3. Data Center-Network Stratum Communication (NS Query)......9 4.3.1. Application Profile.................................10 4.3.2. Network Load Data to be queried.....................10 4.3.3. Responses to NS Query from network to application...11 5. Security Considerations.......................................11 6. References....................................................11 Author's Addresses...............................................14 Intellectual Property Statement..................................14 Disclaimer of Validity...........................................15 So & Lee Expires April 13, 2011 [Page 2] Internet-Draft Network Aware Application Resource Assignment & Mobility October 2010 1. Introduction Data Centers offer application services to end-users such as video gaming [WoWAct], [WoWHrs], [GameServ] and [GroupGame], cloud computing [CostCloud], grid application [GFD-122] and others. Many application services offered by Data Center to end-users make significant use of the underlying networks resources in the form of bandwidth consumption used to carry the actual traffic between a data center and the end-users. It is a common practice for the same application stratum service to be offered by multiple geographically dispersed. One of the major drivers for operating multiple Data Centers is allowing the application to be closer to the end-users, so that the overall service performance and the user experience can be enhanced. As the application servers are distributed geographically across many Data Centers for various reasons (e.g., load balancing), the decision as which server to select for an application request from end-users can negatively affect the quality of experience (QoE) of the users if not done correctly. In addition, the internal infrastructure supporting the applications (e.g., virtual machines) may be actively distributed within and/or between data centers. This practice can improve the utilization of the servers, and prevent service performance degradation during the peak usage time period, and during equipment failures. When instantiating new virtual machines or migrating existing virtual machines across data centers, the underlying network loading conditions within a data center (LAN) or between data centers (MAN/WAN) need to be considered in the decision making process. This document describes problems encountered by distributed applications and provides the motivation for the coordinated resource allocation and management across application stratum and network stratum. 1.1. Terminology This section describes key terminology used in this document. Application Stratum -- The functional block which manages and controls application resources and provides application resources to a variety of clients/end-users. So & Lee Expires April 13, 2011 [Page 3] Internet-Draft Network Aware Application Resource Assignment & Mobility October 2010 Application Profile -- The characteristics of the application from a network traffic perspective and the QoS requirements that the application service will require from the network. Application Resources -- Non-network resources critical to achieving the application service functionality. Examples include: caches, mirrors, application specific servers, content, large data sets, and computing power. Application Service -- A networked application offered to a variety of clients. Network Stratum -- The functional block which manages and controls network resources and provides transport of data between clients/end- users and application sources. Network Resources -- Resources of any layer 3 or below network such as bandwidth, links, paths, path processing (creation, deletion, and management), network databases, path computation, admission control, resource reservation, etc. 2. Network Contexts This section discusses the network contexts and the key components that will help understand problems that will be discussed in the subsequent section. Figure 1 below depicts overarching network architecture in which to show the key components. So & Lee Expires April 13, 2011 [Page 4] Internet-Draft Network Aware Application Resource Assignment & Mobility October 2010 --------------- ---------- | DC 1 | | End-user |. . . . .>| o o o | | | | \|/ | ---------- | O | | ----- --|------ | | | | | -----------------|----------- | / | \ | / ..........O PE1 \ -------------- | | . | | o o o DC 2 | | | PE4 . PE2 | | \|/ | ----|---O.........................O---|---|---O | | . | | | | . PE3 | -------------- \ ..........O Carrier's / \ | Network / ---------------|------------- | --------|------ | O | | /|\ | | o o o | | DC 3 | --------------- Figure 1. Data Center Architecture Figure 1 shows three distinct entities: . End-users . Application/Data Center Networks . Carrier Network (access network is not shown for brevity's sake) There are a number of factors that need be considered in choosing the right server in the right data center for an application request or in instantiating new VMs/applications or migrating existing VMs/applications: . Server level load condition in a data center . Intra data center network condition So & Lee Expires April 13, 2011 [Page 5] Internet-Draft Network Aware Application Resource Assignment & Mobility October 2010 . Carrier MAN/WAN network condition . User Condition Details of potential constraining factors for each of the list above are as follows: . Server level load condition o Virtual Machine (VM) supportability limitations o CPU utilization; o Memory segmentation and consumption; o Application limitations such as max number of simultaneous instances of the application supportability; o Storage access speed (disk, RAM, etc.); o Environment considerations such as server temperature, power load, and electrical cost at the time; o Operational and managerial considerations such as location of peer servers and storages. o VM to NIC switching in a virtual switch . Intra-DC network condition o Server NIC to Top of Rack (ToR) Switch; o TOR switch to Layer 2 Switch - link and node level; o Between L2 Switches and L2 switch to L2 core/gateway switch/router - link and node level; o L2 gateway router to provider edge (PE) router - link and node level. . Carrier MAN/WAN network condition (the networks that connect the data centers and connect end users to the data centers) o Type of networks and the technical capabilities of the networks; o Bandwidth capabilities and availability; o latency; o jitter; o packet loss; o And other Network Performance Objective (NPO) as defined in section 5 of [ITU-T Y.1541]. So & Lee Expires April 13, 2011 [Page 6] Internet-Draft Network Aware Application Resource Assignment & Mobility October 2010 . User condition o User access capabilities and limitations (e.g., user terminal information such as codec for video application); o User location; o Optional user preferences (for some application, user may be able to specify its preferences. For example, the preferred server location for gaming). It is worth to note that the service performance objectives of many new/emerging applications are real-time and interactive applications that require bandwidth guarantee and/or dynamic bandwidth allocation/modification with a strict latency requirement. Such application examples are real-time video distribution services, conferencing and gaming, grid computing, and etc. The need for the coordinated resource allocation and management in all the factors including server level load condition, intra DC network condition, the underlying carrier network condition, and the user condition is paramount in order to meet increasingly sophisticated end-user service needs. 3. Problem Statement In the current Intra-Data Center network, the server selection for an application/VM is done by load-balancer. The load balancer is aware of a certain level of server usage data (e.g., the number simultaneous instances of the application usage) and distributes the application requests based on that data. However, the current load balancing technology is insufficient in providing an optimal decision across multiple VLANs and multiple Data Centers. This capability is often referred to as global load balancing. First of all, there is no standard solution for the communication exchange among load balancers located in different Data Centers. This implies that load balancers from different vendors cannot communicate to each other. So & Lee Expires April 13, 2011 [Page 7] Internet-Draft Network Aware Application Resource Assignment & Mobility October 2010 Secondly, load balancers know little about the underlying network conditions listed in the previous section. Nor is it user condition aware. In some cases, application controllers can estimate network load based on ping latency, and network topology based on trace routes in the Internet, based on the assumption that the underlying transport network is an IP network, and the routing is based on simple IP forwarding. In reality, the carrier's routing schemes are likely to include IP tunneling or MPLS tunneling on top of or in place of IP forwarding. In some cases, the actual network may be VPN, MPLS-TE or GMPLS-TE networks where trace route does not work. This implies that network status estimation technique made from application stratum cannot be accurate. Thus, application resource allocation to end-users can suffer sub-optimality and fail to meet performance objective for the application. When migrating existing VMs/applications from one data center to another, the underlying network load condition in LAN/MAN/WAN can be constraining factors. Migration of VMs/applications, for instance, typically requires a high-speed data transfer across LAN/MAN/WAN to minimize service impact. Application controllers responsible for this operation is not aware of LAN/MAN/WAN network conditions. Another issue is that there is no standard way for an application controller in Data Center to ask for network stratum in a way suitable for a third party, i.e., an entity "outside the network". Even if the network load information is available, few networks would want to reveal the details of the information, such as topology, link bandwidth availability, latency, packet loss, etc, to the outside entity. This warrants some works on abstraction from network side to preserve the privacy of network stratum details from the application stratum. This capability is referred to as Network Stratum Query (NS Query) in this document. 4. High-level requirements This section discusses high-level requirements to support network- aware application mobility in the data center environments. So & Lee Expires April 13, 2011 [Page 8] Internet-Draft Network Aware Application Resource Assignment & Mobility October 2010 4.1. End-User to Application/DC Provider Communication End-users are the entities that make requests to the Front-end (FE) servers, such as load balancers, proxy servers, and etc. Once resources are allocated or made available, they consume application resources offered by application/Data Center provider. End-users may communicate several things to the FE servers: . Required application: this may be a simple URL request; . Optional user preference about server: examples are . Specific server identity or location. This is applicable to gaming where gamers can access game server information and can specify server location. . Additional security requirements; . Modified application specs. For example, a mobile user may request a video download with reduced resolution to conserve battery. . Optional end-user terminal information, e.g., codec for video application. 4.2. Inter DC communication A couple of communication protocols are required in order for the optimal network-aware global load balancing to be achieved. . Server/Application Status Exchange between Data Centers . Ns Query that allows application to factor in network conditions between end-user and the server of choice. (NS Query will be elaborated in detail in Section 4.3) 4.3. Data Center-Network Stratum Communication (NS Query) The FE server plays the key role in choosing the optimal server for an application request. The FE server can function as an application gateway to network and runs NS Query. So & Lee Expires April 13, 2011 [Page 9] Internet-Draft Network Aware Application Resource Assignment & Mobility October 2010 4.3.1. Application Profile The application Stratum needs to provide the application profile to network. Example service profile information that can be useful to network to understand is as follows: . End user IP address; . User access router IP address; . Authentication Profile: Authentication Key; . Bandwidth Profile: Minimum bandwidth required for the application; . Connectivity Profile: P-P, P-MP, Anycast (Multi-destination); . Directionality of the connectivity: unidirectional, bi- directional; . Path Estimation Objective Function: Min latency, etc. Additional profile information can be added depending on the network capability. 4.3.2. Network Load Data to be queried The query should be able to ask all the network condition information listed in Section 1. Note that this can be asked in a different way. For example, the query can simply ask: . Can you route x amount of b/w (in a particular point in network) within y ms of latency? . Can you route x amount of b/w (in a particular point in network) with no packet loss? So & Lee Expires April 13, 2011 [Page 10] Internet-Draft Network Aware Application Resource Assignment & Mobility October 2010 4.3.3. Responses to NS Query from network to application Upon receiving the network query from application, the network should be able to perform the following functions: . Using the given location mapping information (e.g., the server location and the end-user location or a set of server locations in different data centers), the network should be able to derive its network condition data. Note: How to estimate and generate the network condition data by the network is beyond the scope of this draft, and should be addressed in other pertinent WGs. . The network should be able to formulate the abstraction of the requested network condition data in response to the NS Query request from application. 5. Security Considerations TBD 6. IANA Considerations This informational document does not make any requests for IANA action. 7. References 7.1. Informative References [RFC2261] D. Harrington, et al., "An Architecture for Describing SNMP Management Frameworks," January, 1998. [RFC2265] B. Wijnen, et al., "View-based Access Control Model (VACM) for the Simple Network Management Protocol (SNMP)," January, 1998. [Y.2011] General principles and general reference model for Next Generation Networks, October, 2004. [Y.2012] Functional Requirements and architecture of the NGN, April, 2010. So & Lee Expires April 13, 2011 [Page 11] Internet-Draft Network Aware Application Resource Assignment & Mobility October 2010 [GameMirror] S.D. Webb, S. Soh, and W. Lau, "Enhanced mirrored servers for network games," Proceedings of the 6th ACM SIGCOMM workshop on Network and system support for games, Melbourne, Australia: ACM, 2007, pp. 117-122. [GameServ]P. Quax, J. Dierckx, B. Cornelissen, G. Vansichem, and W. Lamotte, "Dynamic server allocation in a real-life deployable communications architecture for networked games," Proceedings of the 7th ACM SIGCOMM Workshop on Network and System Support for Games, Worcester, Massachusetts: ACM, 2008, pp. 66-71. [GameTrf] J. Farber, "Network game traffic modeling," Proceedings of the 1st workshop on Network and system support for games, Braunschweig, Germany: ACM, 2002, pp. 53-57. [GroupGame] K. Vik, C. Griwodz, and P. Halvorsen, "Applicability of group communication for increased scalability in MMOGs," Proceedings of 5th ACM SIGCOMM workshop on Network and system support for games, Singapore: ACM, 2006, p. 2. [IPTV] A.A. Mahimkar, Z. Ge, A. Shaikh, J. Wang, J. Yates, Y. Zhang, and Q. Zhao, "Towards automated performance diagnosis in a large IPTV network," Proceedings of the ACM SIGCOMM 2009 conference on Data communication, Barcelona, Spain: ACM, 2009, pp. 231-242. [MPSel] S. Gargolinski, C.S. Pierre, and M. Claypool, "Game server selection for multiple players," Proceedings of 4th ACM SIGCOMM workshop on Network and system support for games, Hawthorne, NY: ACM, 2005, pp. 1-6. [PartState] P.B. Beskow, K. Vik, P. Halvorsen, and C. Griwodz, "Latency reduction by dynamic core selection and partial migration of game state," Proceedings of the 7th ACM SIGCOMM Workshop on Network and System Support for Games, Worcester, Massachusetts: ACM, 2008, pp. 79-84. [ServVoD] N. Carlsson and D.L. Eager, "Server selection in large- scale video-on-demand systems," ACM Trans. Multimedia Comput. Commun. Appl., vol. 6, 2010, pp. 1-26. [ServMulti] Zongming Fei, M. Ammar, and E. Zegura, "Multicast server selection: problems, complexity, and solutions," Selected Areas in Communications, IEEE Journal on, vol. 20, 2002, pp. 1399-1413. So & Lee Expires April 13, 2011 [Page 12] Internet-Draft Network Aware Application Resource Assignment & Mobility October 2010 [WoWHrs] P. Tarng, K. Chen, and P. Huang, "An analysis of WoW players' game hours," Proceedings of the 7th ACM SIGCOMM Workshop on Network and System Support for Games, Worcester, Massachusetts: ACM, 2008, pp. 47-52. [WoWAct] M. Suznjevic, M. Matijasevic, and O. Dobrijevic, "Action specific Massive Multiplayer Online Role Playing Games traffic analysis: case study of World of Warcraft," Proceedings of the 7th ACM SIGCOMM Workshop on Network and System Support for Games, Worcester, Massachusetts: ACM, 2008, pp. 106-107. [GFD-122] Tiziana Ferrari (editor), "Grid Network services Use Cases from the e-Science Community", GFD-I-122, Open Grid Forum, December 12, 2007. [CostCloud] A. Greenberg, J. Hamilton, D. Maltz, and P. Patel, "The cost of a cloud: research problems in data center networks," ACM SIGCOMM, Vol. 39, Number1, January 2009. So & Lee Expires April 13, 2011 [Page 13] Internet-Draft Network Aware Application Resource Assignment & Mobility October 2010 Author's Addresses Ning So (Editor) Univerity of Texas at Dallas Email: ningso@yahoo.com Young Lee (Editor) Huawei Technologies 1700 Alma Drive, Suite 500 Plano, TX 75075 USA Phone: (972) 509-5599 Email: ylee@huawei.com Dave McDysan Verizon Business Email: dave.mcdysan@verizon.com Greg M. Bernstein Grotto Networking Fremont California, USA Phone: (510) 573-2237 Email: gregb@grotto-networking.com Tae Yeon Kim ETRI tykim@etri.or.kr Kohei Shiomoto NTT Email : shiomoto.kohei@lab.ntt.co.jp Oscar Gonzalez de Dios Telefonica Email : ogondio@tid.es Intellectual Property Statement The IETF Trust takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in any IETF Document or the extent to which any license under such rights might or might not be available; nor does it So & Lee Expires April 13, 2011 [Page 14] Internet-Draft Network Aware Application Resource Assignment & Mobility October 2010 represent that it has made any independent effort to identify any such rights. Copies of Intellectual Property disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement any standard or specification contained in an IETF Document. Please address the information to the IETF at ietf-ipr@ietf.org. Disclaimer of Validity All IETF Documents and the information contained therein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION THEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Acknowledgment Funding for the RFC Editor function is currently provided by the Internet Society. So & Lee Expires April 13, 2011 [Page 15]