idnits 2.17.1 draft-dcn-bmwg-containerized-infra-01.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack an IANA Considerations section. (See Section 2.2 of https://www.ietf.org/id-info/checklist for how to handle the case when there are no actions for IANA.) Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document doesn't use any RFC 2119 keywords, yet has text resembling RFC 2119 boilerplate text. -- The document date (July 8, 2019) is 1744 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- No issues found here. Summary: 1 error (**), 0 flaws (~~), 2 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Benchmarking Methodology Working Group K. Sun 3 Internet-Draft H. Yang 4 Intended status: Informational Y. Park 5 Expires: January 9, 2020 Y. Kim 6 Soongsil University 7 W. Lee 8 ETRI 9 July 8, 2019 11 Considerations for Benchmarking Network Performance in Containerized 12 Infrastructures 13 draft-dcn-bmwg-containerized-infra-01 15 Abstract 17 This draft describes benchmarking considerations for the 18 containerized infrastructure. In the containerized infrastructure, 19 Virtualized Network Functions(VNFs) are deployed on operating-system- 20 level virtualization platform by abstracting the user namespace as 21 opposed to virtualization using a hypervisor. Leveraging this, the 22 system configurations and networking scenarios for benchmarking will 23 be partially changed by the way in which the resource allocation and 24 network technologies speicifed for containerized VNFs. In this draft 25 we compare the state of the art in a container networking 26 architecture with networking on VM-based virtualized systems, and 27 provide several test scenarios in the containerized infrastructure. 29 Status of This Memo 31 This Internet-Draft is submitted in full conformance with the 32 provisions of BCP 78 and BCP 79. 34 Internet-Drafts are working documents of the Internet Engineering 35 Task Force (IETF). Note that other groups may also distribute 36 working documents as Internet-Drafts. The list of current Internet- 37 Drafts is at https://datatracker.ietf.org/drafts/current/. 39 Internet-Drafts are draft documents valid for a maximum of six months 40 and may be updated, replaced, or obsoleted by other documents at any 41 time. It is inappropriate to use Internet-Drafts as reference 42 material or to cite them other than as "work in progress." 44 This Internet-Draft will expire on January 9, 2020. 46 Copyright Notice 48 Copyright (c) 2019 IETF Trust and the persons identified as the 49 document authors. All rights reserved. 51 This document is subject to BCP 78 and the IETF Trust's Legal 52 Provisions Relating to IETF Documents 53 (https://trustee.ietf.org/license-info) in effect on the date of 54 publication of this document. Please review these documents 55 carefully, as they describe your rights and restrictions with respect 56 to this document. Code Components extracted from this document must 57 include Simplified BSD License text as described in Section 4.e of 58 the Trust Legal Provisions and are provided without warranty as 59 described in the Simplified BSD License. 61 Table of Contents 63 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 64 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3 65 3. Benchmarking Considerations . . . . . . . . . . . . . . . . . 3 66 3.1. Comparison with the VM-based Infrastructure . . . . . . . 3 67 3.2. Container Networking Classification . . . . . . . . . . . 5 68 3.3. Resource Considerations . . . . . . . . . . . . . . . . . 8 69 4. Benchmarking Scenarios for the Containerized Infrastructure . 9 70 5. Additional Considerations . . . . . . . . . . . . . . . . . . 12 71 6. Security Considerations . . . . . . . . . . . . . . . . . . . 13 72 7. Acknkowledgement . . . . . . . . . . . . . . . . . . . . . . 13 73 8. Informative References . . . . . . . . . . . . . . . . . . . 13 74 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 14 76 1. Introduction 78 The Benchmarking Methodology Working Group(BMWG) has recently 79 expanded its benchmarking scope from Physical Network Function(PNF) 80 running on dedicated hardware system to Network Function 81 Virtualization(NFV) infrastructure and Virtualized Network 82 Function(VNF). [RFC8172] described considerations for configuring 83 NFV infrastructure and benchmarking metrics, and [RFC8204] gives 84 guidelines for benchmarking virtual switch which connects VNFs in 85 Open Platform for NFV(OPNFV). 87 Recently NFV infrastructure has evolved to include a lightweight 88 virtualized platform called the containerized infrastructure, where 89 VNFs share the same host Operating System(OS) and they are logically 90 isolated by using a different namespace. While previous NFV 91 infrastructure uses a hypervisor to allocate resources for Virtual 92 Machine(VMs) and instantiate VNFs on it, the containerized 93 infrastructure virtualizes resources without a hypervisor, therefore 94 making containers very lightweight and more efficient in 95 infrastructure resource utilization compared to the VM-based NFV 96 infrastructure. When we consider benchmarking for VNFs in the 97 containerized infrastructure, it may have a different System Under 98 Test(SUT) and Device Under Test(DUT) configuration compared with both 99 black-box benchmarking and VM-based NFV infrastructure as described 100 in [RFC8172]. Accordingly, additional configuration parameters and 101 testing strategies may be required. 103 In the containerized infrastructure, a VNF network is implemented by 104 running both switch and router functions in the host system. For 105 example, the internal communication between VNFs in the same host 106 uses the L2 bridge function, while communication with external 107 node(s) uses the L3 router function. For container networking, the 108 host system may use a virtual switch(vSwitch), but other options 109 exist. In the [ETSI-TST-009], they describe differences in 110 networking structure between the VM-based and the containerized 111 infrastructure. Occasioned by these differences, deployment 112 scenarios for testing network performance described in [RFC8204] may 113 be partially applied to the containerized infrastructure, but other 114 scenarios may be required. 116 In this draft, we describe differences and additional considerations 117 for benchmarking containerized infrastructure based on [RFC8172] and 118 [RFC8204]. In particular, we focus on differences in system 119 configuration parameters and networking configurations of the 120 containerized infrastructure compared with VM-based NFV 121 infrastructure. Note that, although the detailed configurations of 122 both infrastructures differ, the new benchmarks and metrics defined 123 in [RFC8172] can be equally applied in containerized infrastructure 124 from a generic-NFV point of view, and therefore defining additional 125 metrics or methodologies is out of scope. 127 2. Terminology 129 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 130 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 131 document is to be interpreted as described in [RFC2119]. This 132 document uses the terminology described in [RFC8172], [RFC8204], 133 [ETSI-TST-009]. 135 3. Benchmarking Considerations 137 3.1. Comparison with the VM-based Infrastructure 139 For the benchmarking of the containerized infrastructure, as 140 mentioned in [RFC8172], the basic approach is to reuse existing 141 benchmarking methods developed within the BMWG. Various network 142 function specifications defined in BMWG should still be applied to 143 containerized VNF(C-VNF)s for the performance comparison with 144 physical network functions and VM-based VNFs. 146 +---------------------------------+ +--------------------------------+ 147 |+--------------+ +--------------+| |+------------+ +------------+| 148 || Guest VM | | Guest VM || || Container | | Container || 149 ||+------------+| |+------------+|| ||+----------+| |+----------+|| 150 ||| APP || || APP ||| ||| APP || || APP ||| 151 ||+------------+| |+------------+|| ||+----------+| |+----------+|| 152 ||+------------+| |+------------+|| ||+----------+| |+----------+|| 153 |||Guest Kernel|| ||Guest Kernel||| ||| Bin/Libs || || Bin/Libs ||| 154 ||+------------+| |+------------+|| ||+----------+| |+----------+|| 155 |+------^-------+ +-------^------+| |+-----^------+ +------^-----+| 156 |+------|-----------------|------+| |+-----|------------------|-----+| 157 || | Hypervisor | || || |+----------------+| || 158 |+------|-----------------|------+| || ||Container Engine|| || 159 |+------|-----------------|------+| || |+----------------+| || 160 || | Host OS Kernel | || || | Host OS Kernel | || 161 |+------|-----------------|-----+|| |+-----|------------------|-----+| 162 | +--v-----------------v--+ | | +---v------------------v---+ | 163 +----| physical network |----+ +--| physical network |--+ 164 +-----------------------+ +--------------------------+ 165 (a) VM-Based Infrastructure (b) Containerized Infrastructure 167 Figure 1: Comparison of NFV Infrastructures 169 In Figure 1, we describe two different NFV architectures: VM-based 170 and Containerized. A major distinction between the containerized and 171 the VM-based infrastructure is that with the former, all VNFs share 172 same host resources including but not limited to computing, storage 173 and networking resources, as well as the host Operating System(OS), 174 kernel and libraries. The absence of the guest OS and the 175 hypervisor, necessitates the following considerations that occur in 176 the test environment: 178 o When we consider hardware configurations for the containerized 179 infrastructure, all components described in [RFC8172] can be part of 180 the test setup. While capabilities of servers and storages should 181 meet the minimum requirements for testing, it is possible to deploy a 182 test environment with less capabilities than in the VM-based 183 infrastructure. 185 o About configuration parameters, the containerized infrastructure 186 needs specified management system instead of a hypervisor(e.g. Linux 187 Container, Docker Engine). 189 o In the VM-based infrastructure, each VM manipulates packets in the 190 kernel of the guest OS through its own CPU threads, virtualized and 191 assigned by the hypervisor. On the other hand, C-VNFs use the host 192 CPU without virtualization. Different CPU resource assignment 193 methods may have different CPU utilization perspectives for the 194 performance benchmarking. 196 o From a Memory Management Unit(MMU) point of view, there are 197 differences in how the paging process is conducted between two 198 environments. The main difference lies in the isolated nature of the 199 OS for VM-based VNFs. In the containerized infrastructure, memory 200 paging which processes conversion between physical address and 201 virtual address is affected by the host resource directly. Thus, 202 memory usage of each C-VNFs is more dependent on the host resource 203 capabilities than in VM-based VNFs. 205 3.2. Container Networking Classification 207 Container networking services are provided as network plugins. 208 Basically, using them, network services are deployed by using 209 isolation environment from container runtime through the host 210 namespace, creating virtual interface, allocating interface and IP 211 address to C-VNF. Since the containerized infrastructure has 212 defferent network architecture depending on its using plugins, it is 213 necessary to specify the plugin used in the infrastructure. There 214 are two proposed models for configuring network interfaces for 215 containers as belows; 217 o CNM(Container Networking Model) proposed by Docker, using 218 libnetwork which provides an interface between the Docker daemon and 219 network drivers. 221 o CNI(Conainer Network Interface) proposed by CoreOS, describing 222 network configuration files in JSON format and plugins are 223 instantiated as new namespaces. Kubernetes uses CNI for providing 224 network service. 226 Regardless of both CNM and CNI, container network model can be 227 classified into kernel space network model and user space network 228 model according to the location of network service creation. In case 229 of kernel-based network model, network interfaces are created in 230 kernel space so that data packets should be processed in network 231 stack of host kernel before transferring packets to the C-VNF running 232 in user space. On the other hand, using user-based network model, 233 data packets from physical network port are bypassed kernel 234 processing and delivered directly to user space. Specific 235 technologies for each network model and example of network 236 architecture are written as follows: 238 o Kernel space network model: Docker Network[Docker-network], Flannel 239 Network[Flannel], Calico[Calico], OVS(OpenvSwitch)[OVS], OVN(Open 240 Virtual Network)[OVN], eBPF[eBPF] 242 +------------------------------------------------------------------+ 243 | User Space | 244 | +-----------+ +-----------+ | 245 | | Container | | Container | | 246 | | +-------+ | | +-------+ | | 247 | +-| eth |-+ +-| eth |-+ | 248 | +--^----+ +----^--+ | 249 | | +------------------------------------------+ | | 250 | | | vSwitch | | | 251 | | | +--------------------------------------+ | | | 252 | | | | +--v---v---v--+ | | | | 253 | | | |bridge | tag[n] | | | | | 254 | | | | +--^-------^--+ | | | | 255 | | | +--^-------------|-------|-----------^-+ | | | 256 | | | | +---+ +---+ | | | | 257 | | | | +------ v-----+ +-------v----+ | | | | 258 | | | | |tunnel bridge| | flat bridge | | | | | 259 | | | | +------^------+ +-------^-----+ | | | | 260 | | +--- |--------|----------------|-------|---+ | | 261 ---------|------ |--------|----------------|-------|------|--------- 262 | +----|-------|--------|----------------|-------|------|----+ | 263 | | +--v-------v--+ | | +--v------v--+ | | 264 | | | veth | | | | veth | | | 265 | | +---^---------+ | | +---^--------+ | | 266 | | Kernel Datapath | | | | 267 | +---------------------|----------------|-------------------+ | 268 | | | | 269 | Kernel Space +--v----------------v--+ | 270 +----------------------| NIC |--------------------+ 271 +----------------------+ 273 Figure 2: Examples of Kernel Space Network Model 275 o User space network model - Device pass-through model: SR- 276 IOV[SR-IOV] 277 +------------------------------------------------------------------+ 278 | User Space | 279 | +-----------------+ +-----------------+ | 280 | | Container | | Container | | 281 | | +-------------+ | | +-------------+ | | 282 | +-| vf driver |-+ +-| vf driver |-+ | 283 | +-----^-------+ +------^------+ | 284 | | | | 285 -------------|---------------------------------------|-------------- 286 | +---------+ +---------+ | 287 | +------|-------------------|------+ | 288 | | +----v-----+ +-----v----+ | | 289 | | | virtual | | virtual | | | 290 | | | function | | function | | | 291 | Kernel Space | +----^-----+ NIC +-----^----+ | | 292 +---------------| | | |----------------+ 293 | +----v-------------------v----+ | 294 | | Classify and Queue | | 295 | +-----------------------------+ | 296 +---------------------------------+ 298 Figure 3: Examples of User Space Netowrk Model - Device Pass-through 300 - vSwitch model: ovs-dpdk[ovs-dpdk], vpp[vpp], netmap[netmap] 301 +------------------------------------------------------------------+ 302 | User Space | 303 | +-----------------+ +-----------------+ | 304 | | Container | | Container | | 305 | | +-------------+ | | +-------------+ | | 306 | +-| virtio-user |-+ +-| virtio-user |-+ | 307 | +-----^-------+ +-------^-----+ | 308 | | | | 309 | +---------+ +---------+ | 310 | +-----------------|--------------------|-----------------+ | 311 | | vSwitch | | | | 312 | | +-------v-----+ +-----v-------+ | | 313 | | | virtio-user | | virtio-user | | | 314 | | +-------^-----+ +-----^-------+ | | 315 | | +------------|--------------------|-------------+ | | 316 | | | +--v--------------------v---+ | | | 317 | | |bridge | tag[n] | | | | 318 | | | +------------^--------------+ | | | 319 | | +----------------------|------------------------+ | | 320 | | +-------v--------+ | | 321 | | | dpdk0 bridge | | | 322 | | +-------^--------+ | | 323 | +---------------------------|----------------------------+ | 324 | +-------v--------+ | 325 | | DPDK PMD | | 326 | +-------^--------+ | 327 ---------------------------------|---------------------------------- 328 | Kernel Space +-----v------+ | 329 +--------------------------| NIC |--------------------------+ 330 +------------+ 332 Figure 4: Examples of User Space Netowrk Model - vSwitch Model using 333 DPDK 335 3.3. Resource Considerations 337 In the containerized infrastructure, resource utilization and 338 isolation may have different charateristics compared with the VM- 339 based infrastructure. Some details are listed as follows: 341 o Hugepage: When using Cent OS or RedHat OS in the VM-based 342 infrastructure, Hugepage should be set to at least 1G byte. In the 343 containerized infrastructure, container is isolated in the 344 application level so that administrators can set Hugepage more 345 granular level(e.g 2M, 4M, ...). In addition, since the increase of 346 the Hugepage can affect the Translation Lookaside Buffer (TLB) miss, 347 the value of the Hugepage should be taken into account in the 348 performance measurement. Moreover, benchmarking results may vary 349 according to Hugepage set value of kernel space model and user space 350 model in the containerized infrastructure so that Hugepage values 351 should be considered when we configure test environment. 353 o NUMA: NUMA technology can be used both in the VM-based and 354 containerized infrastructure. However, the containerized 355 infrastructure provides more variable options than the VM-based 356 infrastructure such as kernel memory, user memory, and CPU setting. 357 Instantiation of C-VNFs is somewhat non-deterministic and apparently 358 NUMA-Node agnostic, which is one way of saying that performance will 359 likely vary whenever this instantiation is performed. So, when we 360 use NUMA in the containerized infratructure, repeated instantiation 361 and testing to quantify the performance variation is required. 363 o RX/TX Multiple-Queue: RX/TX Multiple-Queue technology[Multique], 364 which enables packet sending/receiving processing to scale with 365 number of available vcpus of guest VM, may be used to enhance network 366 performance in the VM-based infrastructure. However, RX/TX Multiple- 367 Queue technology is not supported in the containerized infrastructure 368 yet. 370 4. Benchmarking Scenarios for the Containerized Infrastructure 372 Figure 5 shows briefly differences of network architectures based on 373 deployment models. Basically, on baremetal, C-VNFs can be deployed 374 as a cluster called POD by Kubernetes, otherwise each C-VNF can be 375 deployed separately using Docker. In former case, there is only one 376 external network interface even a POD contains more than one C-VNF. 377 An additional deployment model considers a scenario in which C-VNFs 378 or PODs are running on VM. In our draft, we define new 379 terminologies; BMP which is Pod on baremetal and VMP which is Pod on 380 VM. 382 +---------------------------------------------------------------------+ 383 | Baremetal Node | 384 | | 385 | +--------------+ +--------------+ +-------------- + +-------------+ | 386 | | | | POD | | VM | | VM | | 387 | | | |+------------+| |+-------------+| | +-------+ | | 388 | | C-VNF(A) | || C-VNFs(B) || || C-VNFs(C) || | |PODs(D)| | | 389 | | | |+------------+| |+-----^-------+| | +---^---+ | | 390 | | | | | | | | | | | | 391 | | +------+ | | +------+ | | +--v---+ | | +---v--+ | | 392 | +---| veth |---+ +---| veth |---+ +---|virtio|----+ +--|virtio|---+ | 393 | +--^---+ +---^--+ +--^---+ +---^--+ | 394 | | | | | | 395 | | | +--v---+ +---v--+ | 396 | +------|-----------------|------------|vhost |---------|vhost |---+ | 397 | | | | +--^---+ +---^--+ | | 398 | | | | | | | | 399 | | +--v---+ +---v--+ +--v---+ +---v--+ | | 400 | | +-| veth |---------| veth |---------| Tap |---------| Tap |-+ | | 401 | | | +--^---+ +---^--+ +--^---+ +---^--+ | | | 402 | | | | | vSwitch | | | | | 403 | | | +--|-----------------|---------------|-----------------|--+ | | | 404 | | +-| | | Bridge | | |-+ | | 405 | | +--|-----------------|---------------|-----------------|--+ | | 406 | | | +---------+ | +--|-----------------|---+ | | 407 | | | |Container| | | | Hypervisor | | | | 408 | | | | Engine | | | | | | | | 409 | | | +---------+ | +--|-----------------|---+ | | 410 | | | | Host Kernel | | | | 411 | +------|-----------------|---------------|-----------------|------+ | 412 | +--v-----------------v---------------v-----------------v--+ | 413 +-----| physical network |-----+ 414 +---------------------------------------------------------+ 416 Figure 5: Examples of Networking Architecture based on Deployment 417 Models - (A)C-VNF on Baremetal (B)Pod on Baremetal(BMP) (C)C-VNF on 418 VM (D)Pod on VM(VMP) 420 In [ETSI-TST-009], they described data plane test scenarios in a 421 single host. In that document, there are two scenario for 422 containerized infrastucture; Container2Container which is internal 423 communication between two containers in the same Pod, and Pod2Pod 424 model which is communication between two containers runnung in 425 different Pods. According to our new terminologies, we can call 426 Pod2Pod model as BMP2BMP scenario. When we consider container 427 running on VM as an additional deployment option, there can be more 428 single host test scenarios as follows; 429 o BMP2VMP scenario 431 +---------------------------------------------------------------------+ 432 | HOST +-----------------------------+ | 433 | |VM +-------------------+ | | 434 | | | C-VNF | | | 435 | +--------------------+ | | +--------------+ | | | 436 | | C-VNF | | | | Logical Port | | | | 437 | | +--------------+ | | +-+--^-------^---+--+ | | 438 | | | Logical Port | | | +----|-------|---+ | | 439 | +-+--^-------^---+---+ | | Logical Port | | | 440 | | | +---+----^-------^---+--------+ | 441 | | | | | | 442 | +----v-------|----------------------------|-------v-------------+ | 443 | | l----------------------------l | | 444 | | Data Plane Networking | | 445 | | (Kernel or User space) | | 446 | +----^--------------------------------------------^-------------+ | 447 | | | | 448 | +----v------+ +----v------+ | 449 | | Phy Port | | Phy Port | | 450 | +-----------+ +-----------+ 451 +-------^--------------------------------------------^----------------+ 452 | | 453 +-------v--------------------------------------------v----------------+ 454 | | 455 | Traffic Generator | 456 | | 457 +---------------------------------------------------------------------+ 459 Figure 6: Single Host Test Scenario - BMP2VMP 461 o VMP2VMP scenario 463 +---------------------------------------------------------------------+ 464 | HOST | 465 | +-----------------------------+ +-----------------------------+ | 466 | |VM +-------------------+ | |VM +-------------------+ | | 467 | | | C-VNF | | | | C-VNF | | | 468 | | | +--------------+ | | | | +--------------+ | | | 469 | | | | Logical Port | | | | | | Logical Port | | | | 470 | | +-+--^-------^---+--+ | | +-+--^-------^---+--+ | | 471 | | +----|-------|---+ | | +----|-------|---+ | | 472 | | | Logical Port | | | | Logical Port | | | 473 | +---+----^-------^---+--------+ +---+----^-------^---+--------+ | 474 | | | | | | 475 | +--------v-------v------------------------|-------v-------------+ | 476 | | l------------------------l | | 477 | | Data Plane Networking | | 478 | | (Kernel or User space) | | 479 | +----^--------------------------------------------^-------------+ | 480 | | | | 481 | +----v------+ +----v------+ | 482 | | Phy Port | | Phy Port | | 483 | +-----------+ +-----------+ | 484 +-------^--------------------------------------------^----------------+ 485 | | 486 +-------v--------------------------------------------v----------------+ 487 | | 488 | Traffic Generator | 489 | | 490 +---------------------------------------------------------------------+ 492 Figure 7: Single Host Test Scenario - VMP2VMP 494 5. Additional Considerations 496 When we consider benchmarking for not only containerized but also VM- 497 based infrastucture and network functions, benchmarking scenarios may 498 contain various operational use cases. Traditional black-box 499 benchmarking is focused to measure in-out performance of packet from 500 physical network ports, since hardware is tightly coupled with its 501 function and only single function is running on its dedicated 502 hardware. However, in the NFV environment, the physical network port 503 commonly will be connected to multiple VNFs(i.e. Multiple PVP test 504 setup architecture was described in [ETSI-TST-009]) rather than 505 dedicated to a single VNF. Therefore, benchmarking scenarios should 506 reflect operational considerations such as number of VNFs or network 507 services defined by a set of VNFs in a single host. 508 [service-density], which proposed a way for measuring performance of 509 multiple NFV service instances at a varied service density on a 510 single host, is one example of these operational benchmarking 511 aspects. 513 6. Security Considerations 515 TBD 517 7. Acknkowledgement 519 We would like to thanks people Al, Maciek and Luis who reviewed and 520 gave comments of previous draft. 522 8. Informative References 524 [Calico] "Project Calico", July 2019, 525 . 527 [Docker-network] 528 "Docker, Libnetwork design", July 2019, 529 . 531 [eBPF] "eBPF, extended Berkeley Packet Filter", July 2019, 532 . 534 [ETSI-TST-009] 535 "Network Functions Virtualisation (NFV) Release 3; 536 Testing; Specification of Networking Benchmarks and 537 Measurement Methods for NFVI", October 2018. 539 [Flannel] "flannel 0.10.0 Documentation", July 2019, 540 . 542 [Multique] 543 "Multiqueue virtio-net", July 2019, 544 . 546 [netmap] "Netmap: a framework for fast packet I/O", July 2019, 547 . 549 [OVN] "How to use Open Virtual Networking with Kubernetes", July 550 2019, . 552 [OVS] "Open Virtual Switch", July 2019, 553 . 555 [ovs-dpdk] 556 "Open vSwitch with DPDK", July 2019, 557 . 560 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 561 Requirement Levels", RFC 2119, March 1997. 563 [RFC8172] Morton, A., "Considerations for Benchmarking Virtual 564 Network Functions and Their Infrastructure", RFC 8172, 565 July 2017. 567 [RFC8204] Tahhan, M., O'Mahony, B., and A. Morton, "Benchmarking 568 Virtual Switches in the Open Platform for NFV (OPNFV)", 569 RFC 8204, September 2017. 571 [service-density] 572 Konstantynowicz, M. and P. Mikus, "NFV Service Density 573 Benchmarking", March 2019, . 576 [SR-IOV] "SRIOV for Container-networking", July 2019, 577 . 579 [vpp] "VPP with Containers", July 2019, . 582 Authors' Addresses 584 Kyoungjae Sun 585 School of Electronic Engineering 586 Soongsil University 587 369, Sangdo-ro, Dongjak-gu 588 Seoul, Seoul 06978 589 Republic of Korea 591 Phone: +82 10 3643 5627 592 EMail: gomjae@dcn.ssu.ac.kr 593 Hyunsik Yang 594 School of Electronic Engineering 595 Soongsil University 596 369, Sangdo-ro, Dongjak-gu 597 Seoul, Seoul 06978 598 Republic of Korea 600 Phone: +82 10 9005 7439 601 EMail: yangun@dcn.ssu.ac.kr 603 Youngki Park 604 School of Electronic Engineering 605 Soongsil University 606 369, Sangdo-ro, Dongjak-gu 607 Seoul, Seoul 06978 608 Republic of Korea 610 Phone: +82 10 4281 0720 611 EMail: ykpark@dcn.ssu.ac.kr 613 Younghan Kim 614 School of Electronic Engineering 615 Soongsil University 616 369, Sangdo-ro, Dongjak-gu 617 Seoul, Seoul 06978 618 Republic of Korea 620 Phone: +82 10 2691 0904 621 EMail: younghak@ssu.ac.kr 623 Wangbong Lee 624 ETRI 625 ETRI 626 161, Gajeong-ro, Yoosung-gu 627 Dajeon, Dajeon 34129 628 Republic of Korea 630 Phone: +82 10 5336 2323 631 EMail: leewb@etri.re.kr