idnits 2.17.1 draft-morton-bmwg-virtual-net-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (February 14, 2014) is 3718 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Unused Reference: 'RFC2330' is defined on line 232, but no explicit reference was found in the text == Unused Reference: 'RFC2679' is defined on line 239, but no explicit reference was found in the text == Unused Reference: 'RFC2680' is defined on line 242, but no explicit reference was found in the text == Unused Reference: 'RFC2681' is defined on line 245, but no explicit reference was found in the text == Unused Reference: 'RFC3393' is defined on line 248, but no explicit reference was found in the text == Unused Reference: 'RFC3432' is defined on line 252, but no explicit reference was found in the text == Unused Reference: 'RFC4737' is defined on line 256, but no explicit reference was found in the text == Unused Reference: 'RFC5357' is defined on line 260, but no explicit reference was found in the text == Unused Reference: 'RFC5905' is defined on line 264, but no explicit reference was found in the text == Unused Reference: 'RFC5481' is defined on line 273, but no explicit reference was found in the text == Unused Reference: 'RFC6248' is defined on line 276, but no explicit reference was found in the text == Unused Reference: 'RFC6390' is defined on line 280, but no explicit reference was found in the text ** Obsolete normative reference: RFC 2679 (Obsoleted by RFC 7679) ** Obsolete normative reference: RFC 2680 (Obsoleted by RFC 7680) Summary: 2 errors (**), 0 flaws (~~), 13 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group A. Morton 3 Internet-Draft AT&T Labs 4 Intended status: Informational February 14, 2014 5 Expires: August 18, 2014 7 Considerations for Benchmarking Virtual Network Functions and Their 8 Infrastructure 9 draft-morton-bmwg-virtual-net-00 11 Abstract 13 BMWG has traditionally conducted laboratory characterization of 14 dedicated physical implementations of internetworking functions. 15 This memo investigates additional considerations when network 16 functions are virtualized and performed in commodity off-the-shelf 17 hardware. 19 Requirements Language 21 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 22 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 23 document are to be interpreted as described in RFC 2119 [RFC2119]. 25 Status of This Memo 27 This Internet-Draft is submitted in full conformance with the 28 provisions of BCP 78 and BCP 79. 30 Internet-Drafts are working documents of the Internet Engineering 31 Task Force (IETF). Note that other groups may also distribute 32 working documents as Internet-Drafts. The list of current Internet- 33 Drafts is at http://datatracker.ietf.org/drafts/current/. 35 Internet-Drafts are draft documents valid for a maximum of six months 36 and may be updated, replaced, or obsoleted by other documents at any 37 time. It is inappropriate to use Internet-Drafts as reference 38 material or to cite them other than as "work in progress." 40 This Internet-Draft will expire on August 18, 2014. 42 Copyright Notice 44 Copyright (c) 2014 IETF Trust and the persons identified as the 45 document authors. All rights reserved. 47 This document is subject to BCP 78 and the IETF Trust's Legal 48 Provisions Relating to IETF Documents 49 (http://trustee.ietf.org/license-info) in effect on the date of 50 publication of this document. Please review these documents 51 carefully, as they describe your rights and restrictions with respect 52 to this document. Code Components extracted from this document must 53 include Simplified BSD License text as described in Section 4.e of 54 the Trust Legal Provisions and are provided without warranty as 55 described in the Simplified BSD License. 57 Table of Contents 59 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 60 2. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 61 3. New Considerations . . . . . . . . . . . . . . . . . . . . . 3 62 3.1. Hardware Components . . . . . . . . . . . . . . . . . . . 3 63 3.2. Configuration Parameters . . . . . . . . . . . . . . . . 3 64 3.3. Testing Strategies . . . . . . . . . . . . . . . . . . . 4 65 4. Security Considerations . . . . . . . . . . . . . . . . . . . 5 66 5. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 5 67 6. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 5 68 7. References . . . . . . . . . . . . . . . . . . . . . . . . . 5 69 7.1. Normative References . . . . . . . . . . . . . . . . . . 5 70 7.2. Informative References . . . . . . . . . . . . . . . . . 6 71 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 7 73 1. Introduction 75 BMWG has traditionally conducted laboratory characterization of 76 dedicated physical implementations of internetworking functions. The 77 Black-box Benchmarks of Throughput, Latency, Forwarding Rates and 78 others have served our industry for many years. [RFC1242] and 79 [RFC2544] are the cornerstones of the work. 81 A set of development goals is to reduce costs while increasing 82 flexibility of network devices, and drastically accelerate their 83 deployment. Network Function Virtualization has the promise to 84 achieve these goals, and therefore has garnered much attention. It 85 now seems certain that some network functions will be virtualized 86 following the success of cloud computing and virtual desktops 87 supported by sufficient network path capacity, performance,and 88 widespread deployment; many of the same techniques will be brought to 89 bear. 91 See http://www.etsi.org/technologies-clusters/technologies/nfv for 92 more background, for example, the white papers there may be a useful 93 starting place. 95 2. Scope 97 This memo investigates additional methodological considerations 98 necessary when benchmarking Virtual Network Functions (VNF) 99 instantiated and hosted in commodity off-the-shelf hardware (COTS). 101 A clearly related goal: the benchmarks for the capacity of COTS to 102 host a plurality of VNF instances should be investigated. 104 A non-goal is any overlap with traditional computer benchmark 105 development and their specific metrics (SPECmark suites such as 106 SPECCPU). 108 3. New Considerations 110 This section lists the new considerations which must be addressed to 111 benchmark VNF(s) and their supporting infrastructure. 113 3.1. Hardware Components 115 New Hardware devices will become part of the test set-up. 117 1. High volume server platforms (COTS, possibly with virtual 118 technology enhancements). 120 2. Large capacity, and high speed, high reliability storage systems. 122 3. Network Interface ports specially designed for efficient service 123 of many virtual NICs. 125 4. High capacity Ethernet Switches. 127 Labs conducting comparisons of different VNFs may be able to use the 128 same hardware platform over many studies, until the steady march of 129 innovations overtakes their capabilities (as happens with the lab's 130 traffic generation and testing devices today). 132 3.2. Configuration Parameters 134 It will be necessary to configure and document the settings for the 135 entire COTS platform, including: 137 o number of server blades (shelf occupation) 139 o CPUs 141 o caches 142 o storage system 144 o I/O 146 as well as configurations that support the devices which host the VNF 147 itself: 149 o Hypervisor 151 o Virtual Machine 153 o Infrastructure Virtual Network 155 and finally, the VNF itself, with items such as: 157 o specific function being implemented in VNF 159 o number of VNF components in the service function chain 161 o number of physical interfaces and links transited in the service 162 function chain 164 3.3. Testing Strategies 166 The concept of characterizing performance at capacity limits may 167 change. For example: 169 1. It may be more representative of system capacity to characterize 170 the case where Virtual Machines (VM, hosting the VNF) are 171 operating at 50% Utilization, and therefore sharing the "real" 172 processing power across many VMs. 174 2. Another important case stems from the need for partitioning 175 functions. A noisy neighbor (VM hosting a VNF in an infinite 176 loop) would ideally be isolated and the performance of other VMs 177 would continue according to their specifications. 179 3. System errors will likely occur as transients, implying a 180 distribution of performance characteristics with a long tail 181 (like latency), leading to the need for longer-term tests of each 182 set of configuration and test parameters. 184 4. The desire for Elasticity and flexibility among network functions 185 will include tests where there is constant flux in the VM 186 instances. Requests for new VMs and Releases for VMs hosting 187 VNFs no longer needed would be an normal operational condition. 189 5. All physical things can fail, and benchmarking efforts can also 190 examine recovery aided by the virtual architecture with different 191 approaches to resiliency. 193 4. Security Considerations 195 Benchmarking activities as described in this memo are limited to 196 technology characterization using controlled stimuli in a laboratory 197 environment, with dedicated address space and the constraints 198 specified in the sections above. 200 The benchmarking network topology will be an independent test setup 201 and MUST NOT be connected to devices that may forward the test 202 traffic into a production network, or misroute traffic to the test 203 management network. 205 Further, benchmarking is performed on a "black-box" basis, relying 206 solely on measurements observable external to the DUT/SUT. 208 Special capabilities SHOULD NOT exist in the DUT/SUT specifically for 209 benchmarking purposes. Any implications for network security arising 210 from the DUT/SUT SHOULD be identical in the lab and in production 211 networks. 213 5. IANA Considerations 215 No IANA Action is requested at this time. 217 6. Acknowledgements 219 The author acknowledges an encouraging conversation on this topic 220 with Mukhtiar Shaikh and Ramki Krishnan in November 2013. 222 7. References 224 7.1. Normative References 226 [RFC1242] Bradner, S., "Benchmarking terminology for network 227 interconnection devices", RFC 1242, July 1991. 229 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 230 Requirement Levels", BCP 14, RFC 2119, March 1997. 232 [RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis, 233 "Framework for IP Performance Metrics", RFC 2330, May 234 1998. 236 [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for 237 Network Interconnect Devices", RFC 2544, March 1999. 239 [RFC2679] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way 240 Delay Metric for IPPM", RFC 2679, September 1999. 242 [RFC2680] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way 243 Packet Loss Metric for IPPM", RFC 2680, September 1999. 245 [RFC2681] Almes, G., Kalidindi, S., and M. Zekauskas, "A Round-trip 246 Delay Metric for IPPM", RFC 2681, September 1999. 248 [RFC3393] Demichelis, C. and P. Chimento, "IP Packet Delay Variation 249 Metric for IP Performance Metrics (IPPM)", RFC 3393, 250 November 2002. 252 [RFC3432] Raisanen, V., Grotefeld, G., and A. Morton, "Network 253 performance measurement with periodic streams", RFC 3432, 254 November 2002. 256 [RFC4737] Morton, A., Ciavattone, L., Ramachandran, G., Shalunov, 257 S., and J. Perser, "Packet Reordering Metrics", RFC 4737, 258 November 2006. 260 [RFC5357] Hedayat, K., Krzanowski, R., Morton, A., Yum, K., and J. 261 Babiarz, "A Two-Way Active Measurement Protocol (TWAMP)", 262 RFC 5357, October 2008. 264 [RFC5905] Mills, D., Martin, J., Burbank, J., and W. Kasch, "Network 265 Time Protocol Version 4: Protocol and Algorithms 266 Specification", RFC 5905, June 2010. 268 7.2. Informative References 270 [RFC1242] Bradner, S., "Benchmarking terminology for network 271 interconnection devices", RFC 1242, July 1991. 273 [RFC5481] Morton, A. and B. Claise, "Packet Delay Variation 274 Applicability Statement", RFC 5481, March 2009. 276 [RFC6248] Morton, A., "RFC 4148 and the IP Performance Metrics 277 (IPPM) Registry of Metrics Are Obsolete", RFC 6248, April 278 2011. 280 [RFC6390] Clark, A. and B. Claise, "Guidelines for Considering New 281 Performance Metric Development", BCP 170, RFC 6390, 282 October 2011. 284 Author's Address 286 Al Morton 287 AT&T Labs 288 200 Laurel Avenue South 289 Middletown,, NJ 07748 290 USA 292 Phone: +1 732 420 1571 293 Fax: +1 732 368 1192 294 Email: acmorton@att.com 295 URI: http://home.comcast.net/~acmacm/