idnits 2.17.1 draft-dcn-bmwg-containerized-infra-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack an IANA Considerations section. (See Section 2.2 of https://www.ietf.org/id-info/checklist for how to handle the case when there are no actions for IANA.) Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document doesn't use any RFC 2119 keywords, yet has text resembling RFC 2119 boilerplate text. -- The document date (March 7, 2019) is 1849 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- No issues found here. Summary: 1 error (**), 0 flaws (~~), 2 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Benchmarking Methodology Working Group K. Sun 3 Internet-Draft H. Yang 4 Intended status: Informational Y. Park 5 Expires: September 8, 2019 Y. Kim 6 Soongsil University 7 W. Lee 8 ETRI 9 March 7, 2019 11 Considerations for Benchmarking Network Performance in Containerized 12 Infrastructures 13 draft-dcn-bmwg-containerized-infra-00 15 Abstract 17 This draft describes benchmarking considerations for a containerized 18 infrastructure. In a containerized infrastructure, Virtualized 19 Network Functions(VNFs) are deployed on operating-system-level 20 virtualization platform by abstracting the user namespace as opposed 21 to virtualization using a hypervisor. Leveraging this, the system 22 configurations and networking scenarios for VNF benchmarking will be 23 partially changed by way of resource allocation and network port 24 binding between a physical host and VNFs. In this draft we compare 25 the state of the art in container networking architecture with 26 networking on VM-based virtualized systems, and provide several test 27 scenarios for network performance in containerized infrastructure. 29 Status of This Memo 31 This Internet-Draft is submitted in full conformance with the 32 provisions of BCP 78 and BCP 79. 34 Internet-Drafts are working documents of the Internet Engineering 35 Task Force (IETF). Note that other groups may also distribute 36 working documents as Internet-Drafts. The list of current Internet- 37 Drafts is at https://datatracker.ietf.org/drafts/current/. 39 Internet-Drafts are draft documents valid for a maximum of six months 40 and may be updated, replaced, or obsoleted by other documents at any 41 time. It is inappropriate to use Internet-Drafts as reference 42 material or to cite them other than as "work in progress." 44 This Internet-Draft will expire on September 8, 2019. 46 Copyright Notice 48 Copyright (c) 2019 IETF Trust and the persons identified as the 49 document authors. All rights reserved. 51 This document is subject to BCP 78 and the IETF Trust's Legal 52 Provisions Relating to IETF Documents 53 (https://trustee.ietf.org/license-info) in effect on the date of 54 publication of this document. Please review these documents 55 carefully, as they describe your rights and restrictions with respect 56 to this document. Code Components extracted from this document must 57 include Simplified BSD License text as described in Section 4.e of 58 the Trust Legal Provisions and are provided without warranty as 59 described in the Simplified BSD License. 61 Table of Contents 63 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 64 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3 65 3. Benchmarking Consideration . . . . . . . . . . . . . . . . . 3 66 3.1. Comparison with VM based Infrastructure . . . . . . . . . 3 67 3.2. Additional Considerations for Container Networking . . . 5 68 4. Test Scenarios . . . . . . . . . . . . . . . . . . . . . . . 7 69 5. Security Considerations . . . . . . . . . . . . . . . . . . . 7 70 6. Informative References . . . . . . . . . . . . . . . . . . . 7 71 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 7 73 1. Introduction 75 The Benchmarking Methodology Working Group(BMWG) has recently 76 expanded its benchmarking scope from Physical Network Function(PNF) 77 running on dedicated hardware system to Network Function 78 Virtualization(NFV) infrastructure and Virtualized Network 79 Function(VNF). [RFC8172] described considerations for configuring 80 NFV infrastructure and benchmarking metrics, and [RFC8204] gives 81 guidelines for benchmarking virtual switch which connects VNFs in 82 Open Platform for NFV(OPNFV). 84 Recently NFV infrastructure has evolved to include a lightweight 85 virtualized platform called the containerized infrastructure, where 86 VNFs share the same host Operating System(OS) and they are logically 87 isolated by using a different namespace. While previous NFV 88 infrastructure uses a hypervisor to allocate resources for Virtual 89 Machine(VMs) and instantiate VNFs on it, the containerized 90 infrastructure virtualizes resources without a hypervisor, therefore 91 making containers very lightweight and more efficient in 92 infrastructure resource utilization compared to a VM based NFV 93 infrastructure. When we consider benchmarking for VNFs in the 94 containerized infrastructure, it may have a different Device Under 95 Test(DUT) configuration compared with both black-box benchmarking and 96 VM-based NFV infrastructure as described in [RFC8172]. Accordingly, 97 additional configuration parameters and testing strategies may be 98 required. 100 In the containerized infrastructure, a VNF network is implemented by 101 running both switch and router functions in the host system. For 102 example, the internal communication between VNFs in the same host 103 uses the L2 bridge function, while communication with external 104 node(s) uses the L3 router function. For container networking, the 105 host system may use a virtual switch(vSwitch), but other options 106 exist. In the [ETSI-TST-009], they describe differences in 107 networking structure between VM-based and container-based 108 infrastructure. Occasioned by these differences, deployment 109 scenarios for testing network performance described in [RFC8204] may 110 be partially applied to the containerized infrastructure, but other 111 scenarios may be required. 113 In this draft, we describe differences and additional considerations 114 for benchmarking containerized infrastructure based on [RFC8172] and 115 [RFC8204]. In particular, we focus on differences in system 116 configuration parameters and networking configurations of the 117 containerized infrastructure compared with VM-based NFV 118 infrastructure. Note that, although the detailed configurations of 119 both infrastructures differ, the new benchmarks and metrics defined 120 in [RFC8172] can be equally applied in containerized infrastructure 121 from a generic-NFV point of view, and therefore defining additional 122 metrics or methodologies is out of scope. 124 2. Terminology 126 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 127 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 128 document is to be interpreted as described in [RFC2119]. This 129 document uses the terminology described in [RFC8172], [RFC8204], 130 [ETSI-TST-009]. 132 3. Benchmarking Consideration 134 3.1. Comparison with VM based Infrastructure 136 For benchmarking of containerized infrastructure, as mentioned in 137 [RFC8172], the basic approach is to reuse existing benchmarks 138 developed within the BMWG. Various network function specifications 139 already defined in BMWG should still be applied to containerized VNFs 140 for performance comparison with physical network functions and VM 141 based VNFs. 143 +---------------------------------+ +--------------------------------+ 144 |+--------------+ +--------------+| |+------------+ +------------+| 145 || Guest VM | | Guest VM || || Container | | Container || 146 ||+------------+| |+------------+|| ||+----------+| |+----------+|| 147 ||| APP || || APP ||| ||| APP || || APP ||| 148 ||+------------+| |+------------+|| ||+----------+| |+----------+|| 149 ||+------------+| |+------------+|| ||+----------+| |+----------+|| 150 |||Guest Kernel|| ||Guest Kernel||| ||| Bin/Libs || || Bin/Libs ||| 151 ||+------------+| |+------------+|| ||+----------+| |+----------+|| 152 |+------^-------+ +-------^------+| |+-----^------+ +------^-----+| 153 |+------|-----------------|------+| |+-----|------------------|-----+| 154 || | Hypervisor | || || |+----------------+| || 155 |+------|-----------------|------+| || ||Container Engine|| || 156 |+------|-----------------|------+| || |+----------------+| || 157 || | Host OS Kernel | || || | Host OS Kernel | || 158 |+------|-----------------|-----+|| |+-----|------------------|-----+| 159 | +--v-----------------v--+ | | +---v------------------v---+ | 160 +----| physical network |----+ +--| physical network |--+ 161 +-----------------------+ +--------------------------+ 162 (a) VM-Based Infrastructure (b) Containerized Infrastructure 164 Figure 1: Comparison of NFV Infrastructures 166 In Figure 1, we describe two different NFV architectures: VM-based 167 and Containerized. A major distinction between containerized 168 infrastructure and VM based infrastructure is that with the former, 169 all VNFs share the same host resources including but not limited to 170 computing, storage and networking resources, as well as the host 171 Operating System(OS), kernel and libraries. The absence of the guest 172 OS and the hypervisor, necessitates the following considerations that 173 occur in the test environment: 175 o Concerning hardware for containerized infrastructure, all 176 components described in [RFC8172] can be part of the test setup. 177 While the capabilities of servers and storage should meet the minimum 178 requirements for testing, it is possible to deploy a test environment 179 with less capabilities than in a VM based infrastructure. 181 o About configuration parameters, containerized infrastructure needs 182 specified management system instead of hypervisor(e.g. Linux 183 Container, Docker Engine). 185 o In the VM based infrastructure, each VM has packet processing in 186 the kernel of the guest OS through its own CPU threads, virtualized 187 and assigned by hypervisor. On the other hand, containerized VNFs 188 use the host CPU without virtualization. Different CPU resource 189 assignment methods may have different CPU utilization perspectives 190 for VNF performance benchmarking. 192 o From a Memory Management Unit(MMU) point of view, there is a 193 difference in how the paging process is conducted between two 194 environments. The main difference lies in the isolated nature of the 195 OS for VM-based VNFs. In the containerized infrastructure, memory 196 paging which processes conversion between physical address and 197 virtual address is affected by the host resource directly. Thus, 198 memory usage of each VNFs is more dependent on the host resource 199 capabilities than in VM-based VNFs. 201 o Some network drivers may have varying dependencies for each 202 environment. For example, a vhost-net driver used in a guest OS 203 cannot be used for a container; on the other hand, a veth driver can 204 be only applicable within a containerized infrastructure. 206 3.2. Additional Considerations for Container Networking 208 In the containerized infrastructure, there are various network 209 architectures depending on the deployment environment and models. 210 Since container networking typically involves using virtual switch 211 functions, base network configuration parameters for container 212 networking benchmarks are mostly similar with VM based VNF networking 213 described in [RFC8204]. Additional considerations for container 214 networking are described as follows: 216 o Networking depends on deployment models: Containerized VNFs have 217 several deployment models. Containerized VNFs can be deployed as a 218 cluster called POD by Kubernetes, otherwise each VNF can be deployed 219 separately using Docker. In former case, there is only one external 220 network interface for a POD which contains more than one VNF. An 221 alternative deployment model considers a scenario in which 222 containerized VNFs or PODs are running on VM-based infrastructure. 223 Figure 2 shows briefly differences of network architectures based on 224 deployment models. [ETSI-TST-009] describes in more detail the 225 differences between them. Other deployment models are classified 226 bases on whether containerized VNFs are deployed on baremetal or 227 inside of the VM. 229 +---------------------------------------------------------------------+ 230 | Baremetal Node | 231 | | 232 | +--------------+ +--------------+ +-------------- + +-------------+ | 233 | | | | POD | | VM | | VM | | 234 | | | |+------------+| |+-------------+| | +------+ | | 235 | | Container | || Container || ||Container VNF|| | | PODs | | | 236 | | VNF | || VNFs || |+-----^-------+| | +---^--+ | | 237 | | | |+------------+| | | | | | | | 238 | | +------+ | | +------+ | | +--v---+ | | +---v--+ | | 239 | +---| veth |---+ +---| veth |---+ +---|virtio|----+ +--|virtio|---+ | 240 | +--^---+ +---^--+ +--^---+ +---^--+ | 241 | | | | | | 242 | | | +--v---+ +---v--+ | 243 | +------|-----------------|------------|vhost |---------|vhost |---+ | 244 | | | | +--^---+ +---^--+ | | 245 | | | | | | | | 246 | | +--v---+ +---v--+ +--v---+ +---v--+ | | 247 | | +-| veth |---------| veth |---------| Tap |---------| Tap |-+ | | 248 | | | +--^---+ +---^--+ +--^---+ +---^--+ | | | 249 | | | | | vSwitch | | | | | 250 | | | +--|-----------------|---------------|-----------------|--+ | | | 251 | | +-| | | Bridge | | |-+ | | 252 | | +--|-----------------|---------------|-----------------|--+ | | 253 | | | +---------+ | +--|-----------------|---+ | | 254 | | | |Container| | | | Hypervisor | | | | 255 | | | | Engine | | | | | | | | 256 | | | +---------+ | +--|-----------------|---+ | | 257 | | | | Host Kernel | | | | 258 | +------|-----------------|---------------|-----------------|------+ | 259 | +--v-----------------v---------------v-----------------v--+ | 260 +-----| physical network |-----+ 261 +---------------------------------------------------------+ 263 Figure 2: Examples of Networking Architecture based on Deployment 264 Models 266 o Network Plug-ins: In the containerized infrastructure, specific 267 networking functions can be supported by attaching various plug-ins. 268 Container Network Model(CNM) and Container Network Interface(CNI) are 269 currently the most popular network plug-ins. According each network 270 plug-in, they have different runtime structure or accessibilities to 271 namespace. Actual testing results may vary depending on plug-in 272 types and its supporting drivers. 274 o Network Types: To enhance forwarding capabilities, similar to the 275 VM based infrastructure, the containerized infrastructure can also 276 employ use of specific networking technologies such as SR-IOV. 278 4. Test Scenarios 280 TBD 282 5. Security Considerations 284 TBD 286 6. Informative References 288 [ETSI-TST-009] 289 "Network Functions Virtualisation (NFV) Release 3; 290 Testing; Specification of Networking Benchmarks and 291 Measurement Methods for NFVI", October 2018. 293 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 294 Requirement Levels", RFC 2119, March 1997. 296 [RFC8172] Morton, A., "Considerations for Benchmarking Virtual 297 Network Functions and Their Infrastructure", RFC 8172, 298 July 2017. 300 [RFC8204] Tahhan, M., O'Mahony, B., and A. Morton, "Benchmarking 301 Virtual Switches in the Open Platform for NFV (OPNFV)", 302 RFC 8204, September 2017. 304 Authors' Addresses 306 Kyoungjae Sun 307 School of Electronic Engineering 308 Soongsil University 309 369, Sangdo-ro, Dongjak-gu 310 Seoul, Seoul 06978 311 Republic of Korea 313 Phone: +82 10 3643 5627 314 EMail: gomjae@dcn.ssu.ac.kr 316 Hyunsik Yang 317 School of Electronic Engineering 318 Soongsil University 319 369, Sangdo-ro, Dongjak-gu 320 Seoul, Seoul 06978 321 Republic of Korea 323 Phone: +82 10 9005 7439 324 EMail: yangun@dcn.ssu.ac.kr 325 Youngki Park 326 School of Electronic Engineering 327 Soongsil University 328 369, Sangdo-ro, Dongjak-gu 329 Seoul, Seoul 06978 330 Republic of Korea 332 Phone: +82 10 4281 0720 333 EMail: ykpark@dcn.ssu.ac.kr 335 Younghan Kim 336 School of Electronic Engineering 337 Soongsil University 338 369, Sangdo-ro, Dongjak-gu 339 Seoul, Seoul 06978 340 Republic of Korea 342 Phone: +82 10 2691 0904 343 EMail: younghak@ssu.ac.kr 345 Wangbong Lee 346 ETRI 347 ETRI 348 161, Gajeong-ro, Yoosung-gu 349 Dajeon, Dajeon 34129 350 Republic of Korea 352 Phone: +82 10 5336 2323 353 EMail: leewb@etri.re.kr