Network Working Group Ning So (UTD) Internet Draft Young Lee (Huawei) Intended status: Informational Dave McDysan (Verizon) Greg Bernstein (Grotto) Tae Yeon Kim (ETRI) Kohei Shiomoto (NTT) Oscar Gonzalez de Dios (Telefonica) April 20, 2011 Problem Statement for Network Aware Application Resource Assignment and Mobility (NA-ARAM) in Data Center Environments draft-so-network-aware-application-problem-02.txt Status of this Memo This Internet-Draft is submitted to IETF in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. This Internet-Draft will expire on October 20, 2011. Copyright Notice Copyright (c) 2011 IETF Trust and the persons identified as the document authors. All rights reserved. So & Lee, et al. Expires October 20, 2011 [Page 1] Internet-Draft Network Aware Application Resource Assignment & Mobility April 2011 This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Abstract Data Centers offer various application services to end-users such as video gaming, cloud computing and others. Since the data centers used to provide application services are distributed geographically around a network, many decisions made in the control and management of application services, such as where to instantiate another service instance or to which server a new client is assigned, should be made with the knowledge of the underlying network resources availability and the application resources conditions. Currently such decisions are made with very little or no information concerning the underlying network used to deliver those services. Hence such decisions are inherently sub-optimal. In addition the lack of network awareness may result in not meeting the end-user application service objectives. Table of Contents 1. Introduction........................................ 3 1.1. Terminology..................................... 3 2. Network and Application Contexts........................ 4 2.1. Server level load condition........................ 6 2.2. Intra-DC network condition......................... 6 2.3. Carrier MAN/WAN network condition (the networks that connect the data centers and connect end users to the data centers).. 6 2.4. User condition................................... 7 3. Problem Statement .................................... 7 4. High-level requirements................................ 8 4.1. End-User to Application/DC Provider Communication...... 9 4.2. Inter DC communication............................ 9 4.3. Data Center-Network Stratum Communication (NS Query).... 9 4.3.1. Application Profile.......................... 10 4.3.2. Network Load Data to be queried................ 10 4.3.3. Responses to NS Query from network to application. 11 5. Security Considerations............................... 11 6. IANA Considerations.................................. 11 7. References......................................... 11 7.1. Informative References........................... 11 Author's Addresses..................................... 13 So & Lee Expires October 20, 2011[Page 2] Internet-Draft Network Aware Application Resource Assignment & Mobility April 2011 Intellectual Property Statement .......................... 13 Disclaimer of Validity.................................. 14 1. Introduction Data Centers offer application services to end-users such as video gaming [WoWAct], [WoWHrs], [GameServ] and [GroupGame], cloud computing [CostCloud], grid application [GFD-122] and others. Many application services offered by Data Center to end-users make significant use of the underlying networks resources in the form of bandwidth consumption used to carry the actual traffic between a data center and the end-users. It is a common practice for the same application stratum service to be offered by multiple geographically dispersed data centers. One of the major drivers for operating multiple Data Centers is allowing the application to be closer to the end-users, so that the overall service performance and the user experience can be enhanced. As the application servers are distributed geographically across many Data Centers for various reasons (e.g., load balancing), the decision as which server to select for an application request from end-users can negatively affect the quality of experience (QoE) of the users if not done correctly. In addition, the internal infrastructure supporting the applications (e.g., virtual machines) may be actively distributed within and/or between data centers. This practice can improve the utilization of the servers, and prevent service performance degradation during the peak usage time period, and during equipment failures. When instantiating new virtual machines or migrating existing virtual machines across data centers, the underlying network loading conditions within a data center (LAN) or between data centers (MAN/WAN) need to be considered in the decision making process. This document describes problems encountered by distributed applications and provides the motivation for the coordinated resource allocation and management across application stratum and network stratum. 1.1. Terminology This section describes key terminology used in this document. Application Stratum -- The functional block which manages and controls application resources and provides application resources to a variety of clients/end-users. So & Lee Expires October 20, 2011[Page 3] Internet-Draft Network Aware Application Resource Assignment & Mobility April 2011 Application Profile -- The characteristics of the application from a network traffic perspective and the QoS requirements that the application service will require from the network. Application Resources -- Non-network resources critical to achieving the application service functionality. Examples include: caches, mirrors, application specific servers, content, large data sets, and computing power. Application Service -- A networked application offered to a variety of clients. Application Controller - The central controller that makes assignment the decision as which server to select for an application request from end-users. This is also known as global load balancer. Network Stratum -- The functional block which manages and controls network resources and provides transport of data between clients/end- users and application sources. Network Resources -- Resources of any layer 3 or below network such as bandwidth, links, paths, path processing (creation, deletion, and management), network databases, path computation, admission control, resource reservation, etc. 2. Network and Application Contexts This section discusses the network and application contexts and the key components that will help understand problems that will be discussed in the subsequent section. Figure 1 below depicts overarching network and application architecture in which to show the key components. So & Lee Expires October 20, 2011[Page 4] Internet-Draft Network Aware Application Resource Assignment & Mobility April 2011 ,-----. --------------- ---------- / App \ | DC 1 | | End-user |. . .>( Control ) | o o o | | | \ / | \|/ | ---------- `-----' | O | | ----- --|------ | | | | | --------------------------|-- | / PE1 | \ | / ...................O \ -------------- | | . | | o o o DC 2 | | | PE4 . PE2 | | \|/ | ----|---O.........................O---|---|---O | | . | | | | . PE3 | -------------- \ ..........O Carrier / \ | Network / ---------------|------------- | --------|------ | O | | /|\ | | o o o | | DC 3 | --------------- Figure 1. Data Center Architecture Figure 1 shows four distinct entities: . End-users . Application Controller (Global Load Balancer) . Data Center Networks . Carrier Network (access network is not shown for brevity's sake) There are a number of factors that need be considered in choosing the right server in the right data center for an application request or in instantiating new VMs/applications or migrating existing VMs/applications: . Server level load condition in a data center So & Lee Expires October 20, 2011[Page 5] Internet-Draft Network Aware Application Resource Assignment & Mobility April 2011 . Intra data center network condition . Carrier MAN/WAN network condition . User Condition Details of potential constraining factors for each of the list above are as follows: 2.1. Server level load condition o Virtual Machine (VM) supportability limitations o CPU utilization; o Memory segmentation and consumption; o Application limitations such as max number of simultaneous instances of the application supportability; o Storage access speed (disk, RAM, etc.); o Environment considerations such as server temperature, power load, and electrical cost at the time; o Operational and managerial considerations such as location of peer servers and storages. o VM to NIC switching in a virtual switch 2.2. Intra-DC network condition o Server NIC to Top of Rack (ToR) Switch; o TOR switch to Layer 2 Switch - link and node level; o Between L2 Switches and L2 switch to L2 core/gateway switch/router - link and node level; o L2 gateway router to provider edge (PE) router - link and node level. 2.3. Carrier MAN/WAN network condition (the networks that connect the data centers and connect end users to the data centers) o Type of networks and the technical capabilities of the networks; o Bandwidth capabilities and availability; o latency; o jitter; o packet loss; So & Lee Expires October 20, 2011[Page 6] Internet-Draft Network Aware Application Resource Assignment & Mobility April 2011 o And other Network Performance Objective (NPO) as defined in section 5 of [ITU-T Y.1541]. 2.4. User condition o User access capabilities and limitations (e.g., user terminal information such as codec for video application); o User location; o Optional user preferences (for some application, user may be able to specify its preferences. For example, the preferred server location for gaming). It is worth to note that the service performance objectives of many new/emerging applications are real-time and interactive applications that require bandwidth guarantee and/or dynamic bandwidth allocation/modification with a strict latency requirement. Such application examples are real-time video distribution services, conferencing and gaming, grid computing, and etc. The need for the coordinated resource allocation and management in all the factors including server level load condition, intra DC network condition, the underlying carrier network condition, and the user condition is paramount in order to meet increasingly sophisticated end-user service needs. 3. Problem Statement In the current Intra-Data Center network, the server selection for an application/VM is done by load-balancer. The load balancer is aware of a certain level of server usage data (e.g., the number simultaneous instances of the application usage) and distributes the application requests based on that data. However, the current load balancing technology is insufficient in providing an optimal decision across multiple VLANs and multiple Data Centers. This capability is often referred to as global load balancing. First of all, there is no standard solution for the communication exchange among load balancers located in different Data Centers. This implies that load balancers from different vendors cannot communicate to each other. So & Lee Expires October 20, 2011[Page 7] Internet-Draft Network Aware Application Resource Assignment & Mobility April 2011 Secondly, load balancers know little about the underlying network conditions listed in the previous section. Nor is it user condition aware. In some cases, application controllers can estimate network load based on ping latency, and network topology based on trace routes in the Internet, based on the assumption that the underlying transport network is an IP network, and the routing is based on simple IP forwarding. In reality, the carrier's routing schemes are likely to include IP tunneling or MPLS tunneling on top of or in place of IP forwarding. In some cases, the actual network may be VPN, MPLS-TE or GMPLS-TE networks where trace route does not work. This implies that network status estimation technique made from application stratum may have limited accuracy. Thus, application resource allocation to end-users can suffer sub-optimality and fail to meet performance objective for the application. When migrating existing VMs/applications from one data center to another, the underlying network load condition in LAN/MAN/WAN can be constraining factors. Migration of VMs/applications, for instance, typically requires a high-speed data transfer across LAN/MAN/WAN to minimize service impact. Application controllers responsible for this operation is not aware of LAN/MAN/WAN network conditions. Another issue is that there is no standard way for an application control gateway in Data Center to ask for network stratum in a way suitable for a third party, i.e., an entity "outside the network". Even if the network load information is available, few networks would want to reveal the details of the information, such as topology, link bandwidth availability, latency, packet loss, etc, to the outside entity. This warrants some works on abstraction from network side to preserve the privacy of network stratum details from the application stratum. This capability is referred to as Network Stratum Query (NS Query) in this document. 4. High-level requirements This section discusses high-level requirements to support network- aware application resource assignment and mobility in the data center environments. So & Lee Expires October 20, 2011[Page 8] Internet-Draft Network Aware Application Resource Assignment & Mobility April 2011 4.1. End-User to Application/DC Provider Communication End-users are the entities that make requests to the Application Controller (aka, global load balancer, or front-end server). As Figure 1 shows, the Application Controller interfaces users and makes server assignment decision for the user application. Once resources are allocated or made available, they consume application resources offered by application/Data Center provider. End-users may communicate several things to the Application Controller: . Required application: this may be a simple URL request; . Specific server identity or location. This is applicable to gaming where gamers can access game server information and can specify the user's preference for server location. . Additional security requirements; . Modified application specs. For example, a mobile user may request a video download with reduced resolution to conserve battery. . Optional end-user terminal information, e.g., codec for video application. 4.2. Inter DC communication A couple of new communication protocols are required in order for the optimal network-aware global load balancing to be achieved. . Server/Application Status needs to be made available to the Application Controller (Global Load Balancer) from each of the Data Centers where the application servers are located (See section 2.1 for details); and . Intra-DC network conditions defined in Section 2.2 need to be made available to the Application Controller (Global Load Balancer) from each of the Data Centers where the application servers are located. 4.3. Data Center-Network Stratum Communication (NS Query) The Application Controller plays the key role in choosing the optimal server for an application request. The Application Controller can So & Lee Expires October 20, 2011[Page 9] Internet-Draft Network Aware Application Resource Assignment & Mobility April 2011 interface with an application gateway that interfaces with network and runs NS Query. The details of NS Query are addressed in a separate draft [NS Query]. 4.3.1. Application Profile The application Stratum needs to provide the application profile to network. Example service profile information that can be useful to network to understand is as follows: . End user IP address; . User access router IP address; . Authentication Profile: Authentication Key; . Bandwidth Profile: Minimum bandwidth required for the application; . Connectivity Profile: P-P, P-MP, Anycast (Multi-destination); . Directionality of the connectivity: unidirectional, bi- directional; . Path Estimation Objective Function: Min latency, etc. Additional profile information can be added depending on the network capability. 4.3.2. Network Load Data to be queried The query should be able to ask all the network condition information listed in Section 1. Note that this can be asked in a different way. For example, the query can simply ask: . Can you route x amount of b/w (in a particular point in network) within y ms of latency? . Can you route x amount of b/w (in a particular point in network) with no packet loss? So & Lee Expires October 20, 2011[Page 10] Internet-Draft Network Aware Application Resource Assignment & Mobility April 2011 4.3.3. Responses to NS Query from network to application Upon receiving the network query from application, the network should be able to perform the following functions: . Using the given location mapping information (e.g., the server location and the end-user location or a set of server locations in different data centers), the network should be able to derive its network condition data. Note: How to estimate and generate the network condition data by the network is beyond the scope of this draft, and should be addressed in other pertinent WGs. . The network should be able to formulate the abstraction of the requested network condition data in response to the NS Query request from application. 5. Security Considerations TBD 6. IANA Considerations This informational document does not make any requests for IANA action. 7. References 7.1. Informative References [RFC2261] D. Harrington, et al., "An Architecture for Describing SNMP Management Frameworks," January, 1998. [RFC2265] B. Wijnen, et al., "View-based Access Control Model (VACM) for the Simple Network Management Protocol (SNMP)," January, 1998. [Y.2011] General principles and general reference model for Next Generation Networks, October, 2004. [Y.2012] Functional Requirements and architecture of the NGN, April, 2010. So & Lee Expires October 20, 2011[Page 11] Internet-Draft Network Aware Application Resource Assignment & Mobility April 2011 [GameServ]P. Quax, J. Dierckx, B. Cornelissen, G. Vansichem, and W. Lamotte, "Dynamic server allocation in a real-life deployable communications architecture for networked games," Proceedings of the 7th ACM SIGCOMM Workshop on Network and System Support for Games, Worcester, Massachusetts: ACM, 2008, pp. 66-71. [GroupGame] K. Vik, C. Griwodz, and P. Halvorsen, "Applicability of group communication for increased scalability in MMOGs," Proceedings of 5th ACM SIGCOMM workshop on Network and system support for games, Singapore: ACM, 2006, p. 2. [WoWHrs] P. Tarng, K. Chen, and P. Huang, "An analysis of WoW players' game hours," Proceedings of the 7th ACM SIGCOMM Workshop on Network and System Support for Games, Worcester, Massachusetts: ACM, 2008, pp. 47-52. [WoWAct] M. Suznjevic, M. Matijasevic, and O. Dobrijevic, "Action specific Massive Multiplayer Online Role Playing Games traffic analysis: case study of World of Warcraft," Proceedings of the 7th ACM SIGCOMM Workshop on Network and System Support for Games, Worcester, Massachusetts: ACM, 2008, pp. 106-107. [GFD-122] Tiziana Ferrari (editor), "Grid Network services Use Cases from the e-Science Community", GFD-I-122, Open Grid Forum, December 12, 2007. [CostCloud] A. Greenberg, J. Hamilton, D. Maltz, and P. Patel, "The cost of a cloud: research problems in data center networks," ACM SIGCOMM, Vol. 39, Number1, January 2009. [NS Query] Y. Lee, et. al., "Problem Statement for Network Stratum Query," draft-lee-network-stratum-query-problem, work in progress. [Y.1541] Network performance objectives for IP-based services, February, 2002. So & Lee Expires October 20, 2011[Page 12] Internet-Draft Network Aware Application Resource Assignment & Mobility April 2011 Author's Addresses Ning So (Editor) Univerity of Texas at Dallas Email: ningso@yahoo.com Young Lee (Editor) Huawei Technologies 1700 Alma Drive, Suite 500 Plano, TX 75075 USA Phone: (972) 509-5599 Email: ylee@huawei.com Dave McDysan Verizon Business Email: dave.mcdysan@verizon.com Greg M. Bernstein Grotto Networking Fremont California, USA Phone: (510) 573-2237 Email: gregb@grotto-networking.com Tae Yeon Kim ETRI tykim@etri.or.kr Kohei Shiomoto NTT Email : shiomoto.kohei@lab.ntt.co.jp Oscar Gonzalez de Dios Telefonica Email : ogondio@tid.es Intellectual Property Statement The IETF Trust takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in any IETF Document or the extent to which any license under such rights might or might not be available; nor does it So & Lee Expires October 20, 2011[Page 13] Internet-Draft Network Aware Application Resource Assignment & Mobility April 2011 represent that it has made any independent effort to identify any such rights. Copies of Intellectual Property disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement any standard or specification contained in an IETF Document. Please address the information to the IETF at ietf-ipr@ietf.org. Disclaimer of Validity All IETF Documents and the information contained therein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION THEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Acknowledgment Funding for the RFC Editor function is currently provided by the Internet Society. So & Lee Expires October 20, 2011[Page 14]