idnits 2.17.1
draft-dcn-bmwg-containerized-infra-02.txt:
Checking boilerplate required by RFC 5378 and the IETF Trust (see
https://trustee.ietf.org/license-info):
----------------------------------------------------------------------------
No issues found here.
Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt:
----------------------------------------------------------------------------
No issues found here.
Checking nits according to https://www.ietf.org/id-info/checklist :
----------------------------------------------------------------------------
** The document seems to lack an IANA Considerations section. (See Section
2.2 of https://www.ietf.org/id-info/checklist for how to handle the case
when there are no actions for IANA.)
Miscellaneous warnings:
----------------------------------------------------------------------------
== The copyright year in the IETF Trust and authors Copyright Line does not
match the current year
== The document doesn't use any RFC 2119 keywords, yet has text resembling
RFC 2119 boilerplate text.
-- The document date (July 23, 2019) is 1732 days in the past. Is this
intentional?
Checking references for intended status: Informational
----------------------------------------------------------------------------
No issues found here.
Summary: 1 error (**), 0 flaws (~~), 2 warnings (==), 1 comment (--).
Run idnits with the --verbose option for more detailed information about
the items above.
--------------------------------------------------------------------------------
2 Benchmarking Methodology Working Group K. Sun
3 Internet-Draft H. Yang
4 Intended status: Informational Y. Park
5 Expires: January 24, 2020 Y. Kim
6 Soongsil University
7 W. Lee
8 ETRI
9 July 23, 2019
11 Considerations for Benchmarking Network Performance in Containerized
12 Infrastructures
13 draft-dcn-bmwg-containerized-infra-02
15 Abstract
17 This draft describes benchmarking considerations for the
18 containerized infrastructure. In the containerized infrastructure,
19 Virtualized Network Functions(VNFs) are deployed on operating-system-
20 level virtualization platform by abstracting the user namespace as
21 opposed to virtualization using a hypervisor. Leveraging this, the
22 system configurations and networking scenarios for benchmarking will
23 be partially changed by the way in which the resource allocation and
24 network technologies specified for containerized VNFs. In this draft
25 we compare the state of the art in a container networking
26 architecture with networking on VM-based virtualized systems, and
27 provide several test scenarios in the containerized infrastructure.
29 Status of This Memo
31 This Internet-Draft is submitted in full conformance with the
32 provisions of BCP 78 and BCP 79.
34 Internet-Drafts are working documents of the Internet Engineering
35 Task Force (IETF). Note that other groups may also distribute
36 working documents as Internet-Drafts. The list of current Internet-
37 Drafts is at https://datatracker.ietf.org/drafts/current/.
39 Internet-Drafts are draft documents valid for a maximum of six months
40 and may be updated, replaced, or obsoleted by other documents at any
41 time. It is inappropriate to use Internet-Drafts as reference
42 material or to cite them other than as "work in progress."
44 This Internet-Draft will expire on January 24, 2020.
46 Copyright Notice
48 Copyright (c) 2019 IETF Trust and the persons identified as the
49 document authors. All rights reserved.
51 This document is subject to BCP 78 and the IETF Trust's Legal
52 Provisions Relating to IETF Documents
53 (https://trustee.ietf.org/license-info) in effect on the date of
54 publication of this document. Please review these documents
55 carefully, as they describe your rights and restrictions with respect
56 to this document. Code Components extracted from this document must
57 include Simplified BSD License text as described in Section 4.e of
58 the Trust Legal Provisions and are provided without warranty as
59 described in the Simplified BSD License.
61 Table of Contents
63 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2
64 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3
65 3. Benchmarking Considerations . . . . . . . . . . . . . . . . . 3
66 3.1. Comparison with the VM-based Infrastructure . . . . . . . 3
67 3.2. Container Networking Classification . . . . . . . . . . . 5
68 3.3. Resource Considerations . . . . . . . . . . . . . . . . . 8
69 4. Benchmarking Scenarios for the Containerized Infrastructure . 9
70 5. Additional Considerations . . . . . . . . . . . . . . . . . . 12
71 6. Security Considerations . . . . . . . . . . . . . . . . . . . 13
72 7. Acknkowledgement . . . . . . . . . . . . . . . . . . . . . . 13
73 8. Informative References . . . . . . . . . . . . . . . . . . . 13
74 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 14
76 1. Introduction
78 The Benchmarking Methodology Working Group(BMWG) has recently
79 expanded its benchmarking scope from Physical Network Function(PNF)
80 running on dedicated hardware system to Network Function
81 Virtualization(NFV) infrastructure and Virtualized Network
82 Function(VNF). [RFC8172] described considerations for configuring
83 NFV infrastructure and benchmarking metrics, and [RFC8204] gives
84 guidelines for benchmarking virtual switch which connects VNFs in
85 Open Platform for NFV(OPNFV).
87 Recently NFV infrastructure has evolved to include a lightweight
88 virtualized platform called the containerized infrastructure, where
89 VNFs share the same host Operating System(OS) and they are logically
90 isolated by using a different namespace. While previous NFV
91 infrastructure uses a hypervisor to allocate resources for Virtual
92 Machine(VMs) and instantiate VNFs on it, the containerized
93 infrastructure virtualizes resources without a hypervisor, therefore
94 making containers very lightweight and more efficient in
95 infrastructure resource utilization compared to the VM-based NFV
96 infrastructure. When we consider benchmarking for VNFs in the
97 containerized infrastructure, it may have a different System Under
98 Test(SUT) and Device Under Test(DUT) configuration compared with both
99 black-box benchmarking and VM-based NFV infrastructure as described
100 in [RFC8172]. Accordingly, additional configuration parameters and
101 testing strategies may be required.
103 In the containerized infrastructure, a VNF network is implemented by
104 running both switch and router functions in the host system. For
105 example, the internal communication between VNFs in the same host
106 uses the L2 bridge function, while communication with external
107 node(s) uses the L3 router function. For container networking, the
108 host system may use a virtual switch(vSwitch), but other options
109 exist. In the [ETSI-TST-009], they describe differences in
110 networking structure between the VM-based and the containerized
111 infrastructure. Occasioned by these differences, deployment
112 scenarios for testing network performance described in [RFC8204] may
113 be partially applied to the containerized infrastructure, but other
114 scenarios may be required.
116 In this draft, we describe differences and additional considerations
117 for benchmarking containerized infrastructure based on [RFC8172] and
118 [RFC8204]. In particular, we focus on differences in system
119 configuration parameters and networking configurations of the
120 containerized infrastructure compared with VM-based NFV
121 infrastructure. Note that, although the detailed configurations of
122 both infrastructures differ, the new benchmarks and metrics defined
123 in [RFC8172] can be equally applied in containerized infrastructure
124 from a generic-NFV point of view, and therefore defining additional
125 metrics or methodologies is out of scope.
127 2. Terminology
129 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
130 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
131 document is to be interpreted as described in [RFC2119]. This
132 document uses the terminology described in [RFC8172], [RFC8204],
133 [ETSI-TST-009].
135 3. Benchmarking Considerations
137 3.1. Comparison with the VM-based Infrastructure
139 For the benchmarking of the containerized infrastructure, as
140 mentioned in [RFC8172], the basic approach is to reuse existing
141 benchmarking methods developed within the BMWG. Various network
142 function specifications defined in BMWG should still be applied to
143 containerized VNF(C-VNF)s for the performance comparison with
144 physical network functions and VM-based VNFs.
146 +---------------------------------+ +--------------------------------+
147 |+--------------+ +--------------+| |+------------+ +------------+|
148 || Guest VM | | Guest VM || || Container | | Container ||
149 ||+------------+| |+------------+|| ||+----------+| |+----------+||
150 ||| APP || || APP ||| ||| APP || || APP |||
151 ||+------------+| |+------------+|| ||+----------+| |+----------+||
152 ||+------------+| |+------------+|| ||+----------+| |+----------+||
153 |||Guest Kernel|| ||Guest Kernel||| ||| Bin/Libs || || Bin/Libs |||
154 ||+------------+| |+------------+|| ||+----------+| |+----------+||
155 |+------^-------+ +-------^------+| |+-----^------+ +------^-----+|
156 |+------|-----------------|------+| |+-----|------------------|-----+|
157 || | Hypervisor | || || |+----------------+| ||
158 |+------|-----------------|------+| || ||Container Engine|| ||
159 |+------|-----------------|------+| || |+----------------+| ||
160 || | Host OS Kernel | || || | Host OS Kernel | ||
161 |+------|-----------------|-----+|| |+-----|------------------|-----+|
162 | +--v-----------------v--+ | | +---v------------------v---+ |
163 +----| physical network |----+ +--| physical network |--+
164 +-----------------------+ +--------------------------+
165 (a) VM-Based Infrastructure (b) Containerized Infrastructure
167 Figure 1: Comparison of NFV Infrastructures
169 In Figure 1, we describe two different NFV architectures: VM-based
170 and Containerized. A major distinction between the containerized and
171 the VM-based infrastructure is that with the former, all VNFs share
172 same host resources including but not limited to computing, storage
173 and networking resources, as well as the host Operating System(OS),
174 kernel and libraries. The absence of the guest OS and the
175 hypervisor, necessitates the following considerations that occur in
176 the test environment:
178 o When we consider hardware configurations for the containerized
179 infrastructure, all components described in [RFC8172] can be part of
180 the test setup. While capabilities of servers and storages should
181 meet the minimum requirements for testing, it is possible to deploy a
182 test environment with less capabilities than in the VM-based
183 infrastructure.
185 o About configuration parameters, the containerized infrastructure
186 needs specified management system instead of a hypervisor(e.g. Linux
187 Container, Docker Engine).
189 o In the VM-based infrastructure, each VM manipulates packets in the
190 kernel of the guest OS through its own CPU threads, virtualized and
191 assigned by the hypervisor. On the other hand, C-VNFs use the host
192 CPU without virtualization. Different CPU resource assignment
193 methods may have different CPU utilization perspectives for the
194 performance benchmarking.
196 o From a Memory Management Unit(MMU) point of view, there are
197 differences in how the paging process is conducted between two
198 environments. The main difference lies in the isolated nature of the
199 OS for VM-based VNFs. In the containerized infrastructure, memory
200 paging which processes conversion between physical address and
201 virtual address is affected by the host resource directly. Thus,
202 memory usage of each C-VNFs is more dependent on the host resource
203 capabilities than in VM-based VNFs.
205 3.2. Container Networking Classification
207 Container networking services are provided as network plugins.
208 Basically, using them, network services are deployed by using
209 isolation environment from container runtime through the host
210 namespace, creating virtual interface, allocating interface and IP
211 address to C-VNF. Since the containerized infrastructure has
212 different network architecture depending on its using plugins, it is
213 necessary to specify the plugin used in the infrastructure. There
214 are two proposed models for configuring network interfaces for
215 containers as below;
217 o CNM(Container Networking Model) proposed by Docker, using
218 libnetwork which provides an interface between the Docker daemon and
219 network drivers.
221 o CNI(Container Network Interface) proposed by CoreOS, describing
222 network configuration files in JSON format and plugins are
223 instantiated as new namespaces. Kubernetes uses CNI for providing
224 network service.
226 Regardless of both CNM and CNI, container network model can be
227 classified into kernel space network model and user space network
228 model according to the location of network service creation. In case
229 of kernel-based network model, network interfaces are created in
230 kernel space so that data packets should be processed in network
231 stack of host kernel before transferring packets to the C-VNF running
232 in user space. On the other hand, using user-based network model,
233 data packets from physical network port are bypassed kernel
234 processing and delivered directly to user space. Specific
235 technologies for each network model and example of network
236 architecture are written as follows:
238 o Kernel space network model: Docker Network[Docker-network], Flannel
239 Network[Flannel], Calico[Calico], OVS(OpenvSwitch)[OVS], OVN(Open
240 Virtual Network)[OVN], eBPF[eBPF]
242 +------------------------------------------------------------------+
243 | User Space |
244 | +-----------+ +-----------+ |
245 | | Container | | Container | |
246 | | +-------+ | | +-------+ | |
247 | +-| eth |-+ +-| eth |-+ |
248 | +--^----+ +----^--+ |
249 | | +------------------------------------------+ | |
250 | | | vSwitch | | |
251 | | | +--------------------------------------+ | | |
252 | | | | +--v---v---v--+ | | | |
253 | | | |bridge | tag[n] | | | | |
254 | | | | +--^-------^--+ | | | |
255 | | | +--^-------------|-------|-----------^-+ | | |
256 | | | | +---+ +---+ | | | |
257 | | | | +------ v-----+ +-------v----+ | | | |
258 | | | | |tunnel bridge| | flat bridge | | | | |
259 | | | | +------^------+ +-------^-----+ | | | |
260 | | +--- |--------|----------------|-------|---+ | |
261 ---------|------ |--------|----------------|-------|------|---------
262 | +----|-------|--------|----------------|-------|------|----+ |
263 | | +--v-------v--+ | | +--v------v--+ | |
264 | | | veth | | | | veth | | |
265 | | +---^---------+ | | +---^--------+ | |
266 | | Kernel Datapath | | | |
267 | +---------------------|----------------|-------------------+ |
268 | | | |
269 | Kernel Space +--v----------------v--+ |
270 +----------------------| NIC |--------------------+
271 +----------------------+
273 Figure 2: Examples of Kernel Space Network Model
275 o User space network model / Device pass-through model: SR-
276 IOV[SR-IOV]
277 +------------------------------------------------------------------+
278 | User Space |
279 | +-----------------+ +-----------------+ |
280 | | Container | | Container | |
281 | | +-------------+ | | +-------------+ | |
282 | +-| vf driver |-+ +-| vf driver |-+ |
283 | +-----^-------+ +------^------+ |
284 | | | |
285 -------------|---------------------------------------|--------------
286 | +---------+ +---------+ |
287 | +------|-------------------|------+ |
288 | | +----v-----+ +-----v----+ | |
289 | | | virtual | | virtual | | |
290 | | | function | | function | | |
291 | Kernel Space | +----^-----+ NIC +-----^----+ | |
292 +---------------| | | |----------------+
293 | +----v-------------------v----+ |
294 | | Classify and Queue | |
295 | +-----------------------------+ |
296 +---------------------------------+
298 Figure 3: Examples of User Space Network Model - Device Pass-through
300 o User space network model / vSwitch model: ovs-dpdk[ovs-dpdk],
301 vpp[vpp], netmap[netmap]
302 +------------------------------------------------------------------+
303 | User Space |
304 | +-----------------+ +-----------------+ |
305 | | Container | | Container | |
306 | | +-------------+ | | +-------------+ | |
307 | +-| virtio-user |-+ +-| virtio-user |-+ |
308 | +-----^-------+ +-------^-----+ |
309 | | | |
310 | +---------+ +---------+ |
311 | +-----------------|--------------------|-----------------+ |
312 | | vSwitch | | | |
313 | | +-------v-----+ +-----v-------+ | |
314 | | | virtio-user | | virtio-user | | |
315 | | +-------^-----+ +-----^-------+ | |
316 | | +------------|--------------------|-------------+ | |
317 | | | +--v--------------------v---+ | | |
318 | | |bridge | tag[n] | | | |
319 | | | +------------^--------------+ | | |
320 | | +----------------------|------------------------+ | |
321 | | +-------v--------+ | |
322 | | | dpdk0 bridge | | |
323 | | +-------^--------+ | |
324 | +---------------------------|----------------------------+ |
325 | +-------v--------+ |
326 | | DPDK PMD | |
327 | +-------^--------+ |
328 ---------------------------------|----------------------------------
329 | Kernel Space +-----v------+ |
330 +--------------------------| NIC |--------------------------+
331 +------------+
333 Figure 4: Examples of User Space Network Model - vSwitch Model using
334 DPDK
336 3.3. Resource Considerations
338 In the containerized infrastructure, resource utilization and
339 isolation may have different characteristics compared with the VM-
340 based infrastructure. Some details are listed as follows:
342 o Hugepage: When using Cent OS or RedHat OS in the VM-based
343 infrastructure, Hugepage should be set to at least 1G byte. In the
344 containerized infrastructure, container is isolated in the
345 application level so that administrators can set Hugepage more
346 granular level(e.g 2M, 4M, ...). In addition, since the increase of
347 the Hugepage can affect the Translation Lookaside Buffer (TLB) miss,
348 the value of the Hugepage should be taken into account in the
349 performance measurement. Moreover, benchmarking results may vary
350 according to Hugepage set value of kernel space model and user space
351 model in the containerized infrastructure so that Hugepage values
352 should be considered when we configure test environment.
354 o NUMA: NUMA technology can be used both in the VM-based and
355 containerized infrastructure. However, the containerized
356 infrastructure provides more variable options than the VM-based
357 infrastructure such as kernel memory, user memory, and CPU setting.
358 Instantiation of C-VNFs is somewhat non-deterministic and apparently
359 NUMA-Node agnostic, which is one way of saying that performance will
360 likely vary whenever this instantiation is performed. So, when we
361 use NUMA in the containerized infrastructure, repeated instantiation
362 and testing to quantify the performance variation is required.
364 o RX/TX Multiple-Queue: RX/TX Multiple-Queue technology[Multique],
365 which enables packet sending/receiving processing to scale with
366 number of available vcpus of guest VM, may be used to enhance network
367 performance in the VM-based infrastructure. However, RX/TX Multiple-
368 Queue technology is not supported in the containerized infrastructure
369 yet.
371 4. Benchmarking Scenarios for the Containerized Infrastructure
373 Figure 5 shows briefly differences of network architectures based on
374 deployment models. Basically, on bare metal, C-VNFs can be deployed
375 as a cluster called POD by Kubernetes, otherwise each C-VNF can be
376 deployed separately using Docker. In former case, there is only one
377 external network interface even a POD contains more than one C-VNF.
378 An additional deployment model considers a scenario in which C-VNFs
379 or PODs are running on VM. In our draft, we define new
380 terminologies; BMP which is Pod on bare metal and VMP which is Pod on
381 VM.
383 +---------------------------------------------------------------------+
384 | Baremetal Node |
385 | |
386 | +--------------+ +--------------+ +-------------- + +-------------+ |
387 | | | | POD | | VM | | VM | |
388 | | | |+------------+| |+-------------+| | +-------+ | |
389 | | C-VNF(A) | || C-VNFs(B) || || C-VNFs(C) || | |PODs(D)| | |
390 | | | |+------------+| |+-----^-------+| | +---^---+ | |
391 | | | | | | | | | | | |
392 | | +------+ | | +------+ | | +--v---+ | | +---v--+ | |
393 | +---| veth |---+ +---| veth |---+ +---|virtio|----+ +--|virtio|---+ |
394 | +--^---+ +---^--+ +--^---+ +---^--+ |
395 | | | | | |
396 | | | +--v---+ +---v--+ |
397 | +------|-----------------|------------|vhost |---------|vhost |---+ |
398 | | | | +--^---+ +---^--+ | |
399 | | | | | | | |
400 | | +--v---+ +---v--+ +--v---+ +---v--+ | |
401 | | +-| veth |---------| veth |---------| Tap |---------| Tap |-+ | |
402 | | | +--^---+ +---^--+ +--^---+ +---^--+ | | |
403 | | | | | vSwitch | | | | |
404 | | | +--|-----------------|---------------|-----------------|--+ | | |
405 | | +-| | | Bridge | | |-+ | |
406 | | +--|-----------------|---------------|-----------------|--+ | |
407 | | | +---------+ | +--|-----------------|---+ | |
408 | | | |Container| | | | Hypervisor | | | |
409 | | | | Engine | | | | | | | |
410 | | | +---------+ | +--|-----------------|---+ | |
411 | | | | Host Kernel | | | |
412 | +------|-----------------|---------------|-----------------|------+ |
413 | +--v-----------------v---------------v-----------------v--+ |
414 +-----| physical network |-----+
415 +---------------------------------------------------------+
417 Figure 5: Examples of Networking Architecture based on Deployment
418 Models - (A)C-VNF on Baremetal (B)Pod on Baremetal(BMP) (C)C-VNF on
419 VM (D)Pod on VM(VMP)
421 In [ETSI-TST-009], they described data plane test scenarios in a
422 single host. In that document, there are two scenarios for
423 containerized infrastructure; Container2Container which is internal
424 communication between two containers in the same Pod, and Pod2Pod
425 model which is communication between two containers running in
426 different Pods. According to our new terminologies, we can call
427 Pod2Pod model as BMP2BMP scenario. When we consider container
428 running on VM as an additional deployment option, there can be more
429 single host test scenarios as follows;
430 o BMP2VMP scenario
432 +---------------------------------------------------------------------+
433 | HOST +-----------------------------+ |
434 | |VM +-------------------+ | |
435 | | | C-VNF | | |
436 | +--------------------+ | | +--------------+ | | |
437 | | C-VNF | | | | Logical Port | | | |
438 | | +--------------+ | | +-+--^-------^---+--+ | |
439 | | | Logical Port | | | +----|-------|---+ | |
440 | +-+--^-------^---+---+ | | Logical Port | | |
441 | | | +---+----^-------^---+--------+ |
442 | | | | | |
443 | +----v-------|----------------------------|-------v-------------+ |
444 | | l----------------------------l | |
445 | | Data Plane Networking | |
446 | | (Kernel or User space) | |
447 | +----^--------------------------------------------^-------------+ |
448 | | | |
449 | +----v------+ +----v------+ |
450 | | Phy Port | | Phy Port | |
451 | +-----------+ +-----------+
452 +-------^--------------------------------------------^----------------+
453 | |
454 +-------v--------------------------------------------v----------------+
455 | |
456 | Traffic Generator |
457 | |
458 +---------------------------------------------------------------------+
460 Figure 6: Single Host Test Scenario - BMP2VMP
462 o VMP2VMP scenario
464 +---------------------------------------------------------------------+
465 | HOST |
466 | +-----------------------------+ +-----------------------------+ |
467 | |VM +-------------------+ | |VM +-------------------+ | |
468 | | | C-VNF | | | | C-VNF | | |
469 | | | +--------------+ | | | | +--------------+ | | |
470 | | | | Logical Port | | | | | | Logical Port | | | |
471 | | +-+--^-------^---+--+ | | +-+--^-------^---+--+ | |
472 | | +----|-------|---+ | | +----|-------|---+ | |
473 | | | Logical Port | | | | Logical Port | | |
474 | +---+----^-------^---+--------+ +---+----^-------^---+--------+ |
475 | | | | | |
476 | +--------v-------v------------------------|-------v-------------+ |
477 | | l------------------------l | |
478 | | Data Plane Networking | |
479 | | (Kernel or User space) | |
480 | +----^--------------------------------------------^-------------+ |
481 | | | |
482 | +----v------+ +----v------+ |
483 | | Phy Port | | Phy Port | |
484 | +-----------+ +-----------+ |
485 +-------^--------------------------------------------^----------------+
486 | |
487 +-------v--------------------------------------------v----------------+
488 | |
489 | Traffic Generator |
490 | |
491 +---------------------------------------------------------------------+
493 Figure 7: Single Host Test Scenario - VMP2VMP
495 5. Additional Considerations
497 When we consider benchmarking for not only containerized but also VM-
498 based infrastructure and network functions, benchmarking scenarios
499 may contain various operational use cases. Traditional black-box
500 benchmarking is focused to measure in-out performance of packet from
501 physical network ports, since hardware is tightly coupled with its
502 function and only single function is running on its dedicated
503 hardware. However, in the NFV environment, the physical network port
504 commonly will be connected to multiple VNFs(i.e. Multiple PVP test
505 setup architecture was described in [ETSI-TST-009]) rather than
506 dedicated to a single VNF. Therefore, benchmarking scenarios should
507 reflect operational considerations such as number of VNFs or network
508 services defined by a set of VNFs in a single host.
509 [service-density], which proposed a way for measuring performance of
510 multiple NFV service instances at a varied service density on a
511 single host, is one example of these operational benchmarking
512 aspects.
514 6. Security Considerations
516 TBD
518 7. Acknkowledgement
520 We would like to thank Al, Maciek and Luis who reviewed and gave
521 comments of previous draft.
523 8. Informative References
525 [Calico] "Project Calico", July 2019,
526 .
528 [Docker-network]
529 "Docker, Libnetwork design", July 2019,
530 .
532 [eBPF] "eBPF, extended Berkeley Packet Filter", July 2019,
533 .
535 [ETSI-TST-009]
536 "Network Functions Virtualisation (NFV) Release 3;
537 Testing; Specification of Networking Benchmarks and
538 Measurement Methods for NFVI", October 2018.
540 [Flannel] "flannel 0.10.0 Documentation", July 2019,
541 .
543 [Multique]
544 "Multiqueue virtio-net", July 2019,
545 .
547 [netmap] "Netmap: a framework for fast packet I/O", July 2019,
548 .
550 [OVN] "How to use Open Virtual Networking with Kubernetes", July
551 2019, .
553 [OVS] "Open Virtual Switch", July 2019,
554 .
556 [ovs-dpdk]
557 "Open vSwitch with DPDK", July 2019,
558 .
561 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
562 Requirement Levels", RFC 2119, March 1997.
564 [RFC8172] Morton, A., "Considerations for Benchmarking Virtual
565 Network Functions and Their Infrastructure", RFC 8172,
566 July 2017.
568 [RFC8204] Tahhan, M., O'Mahony, B., and A. Morton, "Benchmarking
569 Virtual Switches in the Open Platform for NFV (OPNFV)",
570 RFC 8204, September 2017.
572 [service-density]
573 Konstantynowicz, M. and P. Mikus, "NFV Service Density
574 Benchmarking", March 2019, .
577 [SR-IOV] "SRIOV for Container-networking", July 2019,
578 .
580 [vpp] "VPP with Containers", July 2019, .
583 Authors' Addresses
585 Kyoungjae Sun
586 School of Electronic Engineering
587 Soongsil University
588 369, Sangdo-ro, Dongjak-gu
589 Seoul, Seoul 06978
590 Republic of Korea
592 Phone: +82 10 3643 5627
593 EMail: gomjae@dcn.ssu.ac.kr
594 Hyunsik Yang
595 School of Electronic Engineering
596 Soongsil University
597 369, Sangdo-ro, Dongjak-gu
598 Seoul, Seoul 06978
599 Republic of Korea
601 Phone: +82 10 9005 7439
602 EMail: yangun@dcn.ssu.ac.kr
604 Youngki Park
605 School of Electronic Engineering
606 Soongsil University
607 369, Sangdo-ro, Dongjak-gu
608 Seoul, Seoul 06978
609 Republic of Korea
611 Phone: +82 10 4281 0720
612 EMail: ykpark@dcn.ssu.ac.kr
614 Younghan Kim
615 School of Electronic Engineering
616 Soongsil University
617 369, Sangdo-ro, Dongjak-gu
618 Seoul, Seoul 06978
619 Republic of Korea
621 Phone: +82 10 2691 0904
622 EMail: younghak@ssu.ac.kr
624 Wangbong Lee
625 ETRI
626 ETRI
627 161, Gajeong-ro, Yoosung-gu
628 Dajeon, Dajeon 34129
629 Republic of Korea
631 Phone: +82 10 5336 2323
632 EMail: leewb@etri.re.kr