idnits 2.17.1 draft-mkonstan-nf-service-density-01.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack an Introduction section. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document seems to lack the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords. (The document does seem to have the reference to RFC 2119 which the ID-Checklist requires). -- The document date (July 08, 2019) is 1725 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Unused Reference: 'RFC8174' is defined on line 1117, but no explicit reference was found in the text == Unused Reference: 'NFVbench' is defined on line 1173, but no explicit reference was found in the text Summary: 1 error (**), 0 flaws (~~), 4 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Benchmarking Working Group M. Konstantynowicz, Ed. 3 Internet-Draft P. Mikus, Ed. 4 Intended status: Informational Cisco Systems 5 Expires: January 9, 2020 July 08, 2019 7 NFV Service Density Benchmarking 8 draft-mkonstan-nf-service-density-01 10 Abstract 12 Network Function Virtualization (NFV) system designers and operators 13 continuously grapple with the problem of qualifying performance of 14 network services realised with software Network Functions (NF) 15 running on Commercial-Off-The-Shelf (COTS) servers. One of the main 16 challenges is getting repeatable and portable benchmarking results 17 and using them to derive deterministic operating range that is 18 production deployment worthy. 20 This document specifies benchmarking methodology for NFV services 21 that aims to address this problem space. It defines a way for 22 measuring performance of multiple NFV service instances, each 23 composed of multiple software NFs, and running them at a varied 24 service "packing" density on a single server. 26 The aim is to discover deterministic usage range of NFV system. In 27 addition specified methodology can be used to compare and contrast 28 different NFV virtualization technologies. 30 Status of This Memo 32 This Internet-Draft is submitted in full conformance with the 33 provisions of BCP 78 and BCP 79. 35 Internet-Drafts are working documents of the Internet Engineering 36 Task Force (IETF). Note that other groups may also distribute 37 working documents as Internet-Drafts. The list of current Internet- 38 Drafts is at https://datatracker.ietf.org/drafts/current/. 40 Internet-Drafts are draft documents valid for a maximum of six months 41 and may be updated, replaced, or obsoleted by other documents at any 42 time. It is inappropriate to use Internet-Drafts as reference 43 material or to cite them other than as "work in progress." 45 This Internet-Draft will expire on January 9, 2020. 47 Copyright Notice 49 Copyright (c) 2019 IETF Trust and the persons identified as the 50 document authors. All rights reserved. 52 This document is subject to BCP 78 and the IETF Trust's Legal 53 Provisions Relating to IETF Documents 54 (https://trustee.ietf.org/license-info) in effect on the date of 55 publication of this document. Please review these documents 56 carefully, as they describe your rights and restrictions with respect 57 to this document. Code Components extracted from this document must 58 include Simplified BSD License text as described in Section 4.e of 59 the Trust Legal Provisions and are provided without warranty as 60 described in the Simplified BSD License. 62 Table of Contents 64 1. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3 65 2. Motivation . . . . . . . . . . . . . . . . . . . . . . . . . 4 66 2.1. Problem Description . . . . . . . . . . . . . . . . . . . 4 67 2.2. Proposed Solution . . . . . . . . . . . . . . . . . . . . 4 68 3. NFV Service . . . . . . . . . . . . . . . . . . . . . . . . . 5 69 3.1. Topology . . . . . . . . . . . . . . . . . . . . . . . . 6 70 3.2. Configuration . . . . . . . . . . . . . . . . . . . . . . 8 71 3.3. Packet Path(s) . . . . . . . . . . . . . . . . . . . . . 9 72 4. Virtualization Technology . . . . . . . . . . . . . . . . . . 12 73 5. Host Networking . . . . . . . . . . . . . . . . . . . . . . . 13 74 6. NFV Service Density Matrix . . . . . . . . . . . . . . . . . 14 75 7. Compute Resource Allocation . . . . . . . . . . . . . . . . . 15 76 8. NFV Service Data-Plane Benchmarking . . . . . . . . . . . . . 19 77 9. Sample NFV Service Density Benchmarks . . . . . . . . . . . . 19 78 9.1. Intrepreting the Sample Results . . . . . . . . . . . . . 20 79 9.2. Benchmarking MRR Throughput . . . . . . . . . . . . . . . 20 80 9.3. VNF Service Chain . . . . . . . . . . . . . . . . . . . . 20 81 9.4. CNF Service Chain . . . . . . . . . . . . . . . . . . . . 21 82 9.5. CNF Service Pipeline . . . . . . . . . . . . . . . . . . 22 83 9.6. Sample Results: FD.io CSIT . . . . . . . . . . . . . . . 23 84 9.7. Sample Results: CNCF/CNFs . . . . . . . . . . . . . . . . 24 85 9.8. Sample Results: OPNFV NFVbench . . . . . . . . . . . . . 26 86 10. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 26 87 11. Security Considerations . . . . . . . . . . . . . . . . . . . 26 88 12. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 26 89 13. References . . . . . . . . . . . . . . . . . . . . . . . . . 27 90 13.1. Normative References . . . . . . . . . . . . . . . . . . 27 91 13.2. Informative References . . . . . . . . . . . . . . . . . 27 92 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 28 94 1. Terminology 96 o NFV: Network Function Virtualization, a general industry term 97 describing network functionality implemented in software. 99 o NFV service: a software based network service realized by a 100 topology of interconnected constituent software network function 101 applications. 103 o NFV service instance: a single instantiation of NFV service. 105 o Data-plane optimized software: any software with dedicated threads 106 handling data-plane packet processing e.g. FD.io VPP (Vector 107 Packet Processor), OVS-DPDK. 109 o Packet Loss Ratio (PLR): ratio of packets received relative to 110 packets transmitted over the test trial duration, calculated using 111 formula: PLR = ( pkts_transmitted - pkts_received ) / 112 pkts_transmitted. For bi-directional throughput tests aggregate 113 PLR is calculated based on the aggregate number of packets 114 transmitted and received. 116 o Packet Throughput Rate: maximum packet offered load DUT/SUT 117 forwards within the specified Packet Loss Ratio (PLR). In many 118 cases the rate depends on the frame size processed by DUT/SUT. 119 Hence packet throughput rate MUST be quoted with specific frame 120 size as received by DUT/SUT during the measurement. For bi- 121 directional tests, packet throughput rate should be reported as 122 aggregate for both directions. Measured in packets-per-second 123 (pps) or frames-per-second (fps), equivalent metrics. 125 o Non Drop Rate (NDR): maximum packet/bandwith throughput rate 126 sustained by DUT/SUT at PLR equal zero (zero packet loss) specific 127 to tested frame size(s). MUST be quoted with specific packet size 128 as received by DUT/SUT during the measurement. Packet NDR 129 measured in packets-per-second (or fps), bandwidth NDR expressed 130 in bits-per-second (bps). 132 o Partial Drop Rate (PDR): maximum packet/bandwith throughput rate 133 sustained by DUT/SUT at PLR greater than zero (non-zero packet 134 loss) specific to tested frame size(s). MUST be quoted with 135 specific packet size as received by DUT/SUT during the 136 measurement. Packet PDR measured in packets-per-second (or fps), 137 bandwidth PDR expressed in bits-per-second (bps). 139 o Maximum Receive Rate (MRR): packet/bandwidth rate regardless of 140 PLR sustained by DUT/SUT under specified Maximum Transmit Rate 141 (MTR) packet load offered by traffic generator. MUST be quoted 142 with both specific packet size and MTR as received by DUT/SUT 143 during the measurement. Packet MRR measured in packets-per-second 144 (or fps), bandwidth MRR expressed in bits-per-second (bps). 146 2. Motivation 148 2.1. Problem Description 150 Network Function Virtualization (NFV) system designers and operators 151 continuously grapple with the problem of qualifying performance of 152 network services realised with software Network Functions (NF) 153 running on Commercial-Off-The-Shelf (COTS) servers. One of the main 154 challenges is getting repeatable and portable benchmarking results 155 and using them to derive deterministic operating range that is 156 production deployment worthy. 158 Lack of well defined and standardised NFV centric performance 159 methodology and metrics makes it hard to address fundamental 160 questions that underpin NFV production deployments: 162 1. What NFV service and how many instances can run on a single 163 compute node? 165 2. How to choose the best compute resource allocation scheme to 166 maximise service yield per node? 168 3. How do different NF applications compare from the service density 169 perspective? 171 4. How do the virtualisation technologies compare e.g. Virtual 172 Machines, Containers? 174 Getting answers to these points should allow designers to make data 175 based decisions about the NFV technology and service design best 176 suited to meet requirements of their use cases. Thereby obtained 177 benchmarking data would aid in selection of the most appropriate NFV 178 infrastructure design and platform and enable more accurate capacity 179 planning, an important element for commercial viability of the NFV 180 service. 182 2.2. Proposed Solution 184 The primary goal of the proposed benchmarking methodology is to focus 185 on NFV technologies used to construct NFV services. More 186 specifically to i) measure packet data-plane performance of multiple 187 NFV service instances while running them at varied service "packing" 188 densities on a single server and ii) quantify the impact of using 189 multiple NFs to construct each NFV service instance and introducing 190 multiple packet processing hops and links on each packet path. 192 The overarching aim is to discover a set of deterministic usage 193 ranges that are of interest to NFV system designers and operators. 194 In addition, specified methodology can be used to compare and 195 contrast different NFV virtualisation technologies. 197 In order to ensure wide applicability of the benchmarking 198 methodology, the approach is to separate NFV service packet 199 processing from the shared virtualisation infrastructure by 200 decomposing the software technology stack into three building blocks: 202 +-------------------------------+ 203 | NFV Service | 204 +-------------------------------+ 205 | Virtualization Technology | 206 +-------------------------------+ 207 | Host Networking | 208 +-------------------------------+ 210 Figure 1. NFV software technology stack. 212 Proposed methodology is complementary to existing NFV benchmarking 213 industry efforts focusing on vSwitch benchmarking [RFC8204], [TST009] 214 and extends the benchmarking scope to NFV services. 216 This document does not describe a complete benchmarking methodology, 217 instead it is focusing on the system under test configuration. Each 218 of the compute node configurations identified in this document is to 219 be evaluated for NFV service data-plane performance using existing 220 and/or emerging network benchmarking standards. This may include 221 methodologies specified in [RFC2544], [TST009], 222 [draft-vpolak-mkonstan-bmwg-mlrsearch] and/or 223 [draft-vpolak-bmwg-plrsearch]. 225 3. NFV Service 227 It is assumed that each NFV service instance is built of one or more 228 constituent NFs and is described by: topology, configuration and 229 resulting packet path(s). 231 Each set of NFs forms an independent NFV service instance, with 232 multiple sets present in the host. 234 3.1. Topology 236 NFV topology describes the number of network functions per service 237 instance, and their inter-connections over packet interfaces. It 238 includes all point-to-point virtual packet links within the compute 239 node, Layer-2 Ethernet or Layer-3 IP, including the ones to host 240 networking data-plane. 242 Theoretically, a large set of possible NFV topologies can be realised 243 using software virtualisation topologies, e.g. ring, partial -/full- 244 mesh, star, line, tree, ladder. In practice however, only a few 245 topologies are in the actual use as NFV services mostly perform 246 either bumps-in-a-wire packet operations (e.g. security filtering/ 247 inspection, monitoring/telemetry) and/or inter-site forwarding 248 decisions (e.g. routing, switching). 250 Two main NFV topologies have been identified so far for NFV service 251 density benchmarking: 253 1. Chain topology: a set of NFs connect to host data-plane with 254 minimum of two virtual interfaces each, enabling host data-plane 255 to facilitate NF to NF service chain forwarding and provide 256 connectivity with external network. 258 2. Pipeline topology: a set of NFs connect to each other in a line 259 fashion with edge NFs homed to host data-plane. Host data-plane 260 provides connectivity with external network. 262 In both cases multiple NFV service topologies are running in 263 parallel. Both topologies are shown in figures 2. and 3. below. 265 NF chain topology: 267 +-----------------------------------------------------------+ 268 | Host Compute Node | 269 | | 270 | SmNF1 SmNF2 SmNFn Service-m | 271 | ... ... ... ... | 272 | S2NF1 S2NF2 S2NFn Service-2 | 273 | +--------+ +--------+ +--------+ | 274 | | S1NF1 | | S1NF2 | | S1NFn | | 275 | | | | | .... | | Service-1 | 276 | | | | | | | | 277 | +-+----+-+ +-+----+-+ + + +-+----+-+ | 278 | | | | | | | | | Virtual | 279 | | |<-CS->| |<-CS->| |<-CS->| | Interfaces | 280 | +-+----+------+----+------+----+------+----+-+ | 281 | | | CS: Chain | 282 | | | Segment | 283 | | Host Data-Plane | | 284 | +-+--+----------------------------------+--+-+ | 285 | | | | | | 286 +-----------------------------------------------------------+ 287 | | | | Physical 288 | | | | Interfaces 289 +---+--+----------------------------------+--+--------------+ 290 | | 291 | Traffic Generator | 292 | | 293 +-----------------------------------------------------------+ 295 Figure 2. NF chain topology forming a service instance. 297 NF pipeline topology: 299 +-----------------------------------------------------------+ 300 | Host Compute Node | 301 | | 302 | SmNF1 SmNF2 SmNFn Service-m | 303 | ... ... ... ... | 304 | S2NF1 S2NF2 S2NFn Service-2 | 305 | +--------+ +--------+ +--------+ | 306 | | S1NF1 | | S1NF2 | | S1NFn | | 307 | | +--+ +--+ .... +--+ | Service1 | 308 | | | | | | | | 309 | +-+------+ +--------+ +------+-+ | 310 | | | Virtual | 311 | |<-Pipeline Edge Pipeline Edge->| Interfaces | 312 | +-+----------------------------------------+-+ | 313 | | | | 314 | | | | 315 | | Host Data-Plane | | 316 | +-+--+----------------------------------+--+-+ | 317 | | | | | | 318 +-----------------------------------------------------------+ 319 | | | | Physical 320 | | | | Interfaces 321 +---+--+----------------------------------+--+--------------+ 322 | | 323 | Traffic Generator | 324 | | 325 +-----------------------------------------------------------+ 327 Figure 3. NF pipeline topology forming a service instance. 329 3.2. Configuration 331 NFV configuration includes all packet processing functions in NFs 332 including Layer-2, Layer-3 and/or Layer-4-to-7 processing as 333 appropriate to specific NF and NFV service design. L2 sub- interface 334 encapsulations (e.g. 802.1q, 802.1ad) and IP overlay encapsulation 335 (e.g. VXLAN, IPSec, GRE) may be represented here too as appropriate, 336 although in most cases they are used as external encapsulation and 337 handled by host networking data-plane. 339 NFV configuration determines logical network connectivity that is 340 Layer-2 and/or IPv4/IPv6 switching/routing modes, as well as NFV 341 service specific aspects. In the context of NFV density benchmarking 342 methodology the initial focus is on logical network connectivity 343 between the NFs, and no NFV service specific configurations. NF 344 specific functionality is emulated using IPv4/IPv6 routing. 346 Building on the two identified NFV topologies, two common NFV 347 configurations are considered: 349 1. Chain configuration: 351 * Relies on chain topology to form NFV service chains. 353 * NF packet forwarding designs: 355 + IPv4/IPv6 routing. 357 * Requirements for host data-plane: 359 + L2 switching with L2 forwarding context per each NF chain 360 segment, or 362 + IPv4/IPv6 routing with IP forwarding context per each NF 363 chain segment or per NF chain. 365 2. Pipeline configuration: 367 * Relies on pipeline topology to form NFV service pipelines. 369 * Packet forwarding designs: 371 + IPv4/IPv6 routing. 373 * Requirements for host data-plane: 375 + L2 switching with L2 forwarding context per each NF 376 pipeline edge link, or 378 + IPv4/IPv6 routing with IP forwarding context per each NF 379 pipeline edge link or per NF pipeline. 381 3.3. Packet Path(s) 383 NFV packet path(s) describe the actual packet forwarding path(s) used 384 for benchmarking, resulting from NFV topology and configuration. 385 They are aimed to resemble true packet forwarding actions during the 386 NFV service lifecycle. 388 Based on the specified NFV topologies and configurations two NFV 389 packet paths are taken for benchmarking: 391 1. Snake packet path 393 * Requires chain topology and configuration. 395 * Packets enter the NFV chain through one edge NF and progress 396 to the other edge NF of the chain. 398 * Within the chain, packets follow a zigzagging "snake" path 399 entering and leaving host data-plane as they progress through 400 the NF chain. 402 * Host data-plane is involved in packet forwarding operations 403 between NIC interfaces and edge NFs, as well as between NFs in 404 the chain. 406 2. Pipeline packet path 408 * Requires pipeline topology and configuration. 410 * Packets enter the NFV chain through one edge NF and progress 411 to the other edge NF of the pipeline. 413 * Within the chain, packets follow a straight path entering and 414 leaving subsequent NFs as they progress through the NF 415 pipeline. 417 * Host data-plane is involved in packet forwarding operations 418 between NIC interfaces and edge NFs only. 420 Both packet paths are shown in figures below. 422 Snake packet path: 424 +-----------------------------------------------------------+ 425 | Host Compute Node | 426 | | 427 | SmNF1 SmNF2 SmNFn Service-m | 428 | ... ... ... ... | 429 | S2NF1 S2NF2 S2NFn Service-2 | 430 | +--------+ +--------+ +--------+ | 431 | | S1NF1 | | S1NF2 | | S1NFn | | 432 | | | | | .... | | Service1 | 433 | | XXXX | | XXXX | | XXXX | | 434 | +-+X--X+-+ +-+X--X+-+ +X X+ +-+X--X+-+ | 435 | |X X| |X X| |X X| |X X| Virtual | 436 | |X X| |X X| |X X| |X X| Interfaces | 437 | +-+X--X+------+X--X+------+X--X+------+X--X+-+ | 438 | | X XXXXXXXXXX XXXXXXXXXX XXXXXXXXXX X | | 439 | | X X | | 440 | | X Host Data-Plane X | | 441 | +-+X-+----------------------------------+-X+-+ | 442 | |X | | X| | 443 +----X--------------------------------------X---------------+ 444 |X | | X| Physical 445 |X | | X| Interfaces 446 +---+X-+----------------------------------+-X+--------------+ 447 | | 448 | Traffic Generator | 449 | | 450 +-----------------------------------------------------------+ 452 Figure 4. Snake packet path thru NF chain topology. 454 Pipeline packet path: 456 +-----------------------------------------------------------+ 457 | Host Compute Node | 458 | | 459 | SmNF1 SmNF2 SmNFn Service-m | 460 | ... ... ... ... | 461 | S2NF1 S2NF2 S2NFn Service-2 | 462 | +--------+ +--------+ +--------+ | 463 | | S1NF1 | | S1NF2 | | S1NFn | | 464 | | +--+ +--+ .... +--+ | Service1 | 465 | | XXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXX | | 466 | +--X-----+ +--------+ +-----X--+ | 467 | |X X| Virtual | 468 | |X X| Interfaces | 469 | +-+X--------------------------------------X+-+ | 470 | | X X | | 471 | | X X | | 472 | | X Host Data-Plane X | | 473 | +-+X-+----------------------------------+-X+-+ | 474 | |X | | X| | 475 +----X--------------------------------------X---------------+ 476 |X | | X| Physical 477 |X | | X| Interfaces 478 +---+X-+----------------------------------+-X+--------------+ 479 | | 480 | Traffic Generator | 481 | | 482 +-----------------------------------------------------------+ 484 Figure 5. Pipeline packet path thru NF pipeline topology. 486 In all cases packets enter NFV system via shared physical NIC 487 interfaces controlled by shared host data-plane, are then associated 488 with specific NFV service (based on service discriminator) and 489 subsequently are cross- connected/switched/routed by host data-plane 490 to and through NF topologies per one of the above listed schemes. 492 4. Virtualization Technology 494 NFV services are built of composite isolated NFs, with virtualisation 495 technology providing the workload isolation. Following 496 virtualisation technology types are considered for NFV service 497 density benchmarking: 499 1. Virtual Machines (VMs) 501 * Relying on host hypervisor technology e.g. KVM, ESXi, Xen. 503 * NFs running in VMs are referred to as VNFs. 505 2. Containers 507 * Relying on Linux container technology e.g. LXC, Docker. 509 * NFs running in Containers are referred to as CNFs. 511 Different virtual interface types are available to VNFs and CNFs: 513 1. VNF 515 * virtio-vhostuser: fully user-mode based virtual interface. 517 * virtio-vhostnet: involves kernel-mode based backend. 519 2. CNF 521 * memif: fully user-mode based virtual interface. 523 * af_packet: involves kernel-mode based backend. 525 * (add more common ones) 527 5. Host Networking 529 Host networking data-plane is the central shared resource that 530 underpins creation of NFV services. It handles all of the 531 connectivity to external physical network devices through physical 532 network connections using NICs, through which the benchmarking is 533 done. 535 Assuming that NIC interface resources are shared, here is the list of 536 widely available host data-plane options for providing packet 537 connectivity to/from NICs and constructing NFV chain and pipeline 538 topologies and configurations: 540 o Linux Kernel-Mode Networking. 542 o Linux User-Mode vSwitch. 544 o Virtual Machine vSwitch. 546 o Linux Container vSwitch. 548 o SRIOV NIC Virtual Function - note: restricted support for chain 549 and pipeline topologies, as it requires hair-pinning through the 550 NIC and oftentimes also through external physical switch. 552 Analysing properties of each of these options and their Pros/Cons for 553 specified NFV topologies and configurations is outside the scope of 554 this document. 556 From all listed options, performance optimised Linux user-mode 557 vswitch deserves special attention. Linux user-mode switch decouples 558 NFV service from the underlying NIC hardware, offers rich multi- 559 tenant functionality and most flexibility for supporting NFV 560 services. But in the same time it is consuming compute resources and 561 is harder to benchmark in NFV service density scenarios. 563 Following sections focus on using Linux user-mode vSwitch, focusing 564 on its performance benchmarking at increasing levels of NFV service 565 density. 567 6. NFV Service Density Matrix 569 In order to evaluate performance of multiple NFV services running on 570 a compute node, NFV service instances are benchmarked at increasing 571 density, allowing to construct an NFV Service Density Matrix. 572 Table below shows an example of such a matrix, capturing number of 573 NFV service instances (row indices), number of NFs per service 574 instance (column indices) and resulting total number of NFs (values). 576 NFV Service Density - NF Count View 578 SVC 001 002 004 006 008 00N 579 001 1 2 4 6 8 1*N 580 002 2 4 8 12 16 2*N 581 004 4 8 16 24 32 4*N 582 006 6 12 24 36 48 6*N 583 008 8 16 32 48 64 8*N 584 00M M*1 M*2 M*4 M*6 M*8 M*N 586 RowIndex: Number of NFV Service Instances, 1..M. 587 ColumnIndex: Number of NFs per NFV Service Instance, 1..N. 588 Value: Total number of NFs running in the system. 590 In order to deliver good and repeatable network data-plane 591 performance, NFs and host data-plane software require direct access 592 to critical compute resources. Due to a shared nature of all 593 resources on a compute node, a clearly defined resource allocation 594 scheme is defined in the next section to address this. 596 In each tested configuration host data-plane is a gateway between the 597 external network and the internal NFV network topologies. Offered 598 packet load is generated and received by an external traffic 599 generator per usual benchmarking practice. 601 It is proposed that benchmarks are done with the offered packet load 602 distributed equally across all configured NFV service instances. 603 This approach should provide representative benchmarking data for 604 each tested topology and configuraiton, and a good guesstimate of 605 maximum performance required for capacity planning. 607 Following sections specify compute resource allocation, followed by 608 examples of applying NFV service density methodology to VNF and CNF 609 benchmarking use cases. 611 7. Compute Resource Allocation 613 Performance optimized NF and host data-plane software threads require 614 timely execution of packet processing instructions and are very 615 sensitive to any interruptions (or stalls) to this execution e.g. cpu 616 core context switching, or cpu jitter. To that end, NFV service 617 density methodology treats controlled mapping ratios of data plane 618 software threads to physical processor cores with directly allocated 619 cache hierarchies as the first order requirement. 621 Other compute resources including memory bandwidth and PCIe bandwidth 622 have lesser impact and as such are subject for further study. For 623 more detail and deep-dive analysis of software data plane performance 624 and impact on different shared compute resources is available in 625 [BSDP]. 627 It is assumed that NFs as well as host data-plane (e.g. vswitch) are 628 performance optimized, with their tasks executed in two types of 629 software threads: 631 o data-plane - handling data-plane packet processing and forwarding, 632 time critical, requires dedicated cores. To scale data-plane 633 performance, most NF apps use multiple data-plane threads and rely 634 on NIC RSS (Receive Side Scaling), virtual interface multi-queue 635 and/or integrated software hashing to distribute packets across 636 the data threads. 638 o main-control - handling application management, statistics and 639 control-planes, less time critical, allows for core sharing. For 640 most NF apps this is a single main thread, but often statistics 641 (counters) and various control protocol software are run in 642 separate threads. 644 Core mapping scheme described below allocates cores for all threads 645 of specified type belonging to each NF app instance, and separately 646 lists number of threads to a number of logical/physical core mappings 647 for processor configurations with enabled/disabled Symmetric Multi- 648 Threading (SMT) (e.g. AMD SMT, Intel Hyper-Threading). 650 If NFV service density benchmarking is run on server nodes with 651 Symmetric Multi-Threading (SMT) (e.g. AMD SMT, Intel Hyper- 652 Threading) for higher performance and efficiency, logical cores 653 allocated to data- plane threads should be allocated as pairs of 654 sibling logical cores corresponding to the hyper-threads running on 655 the same physical core. 657 Separate core ratios are defined for mapping threads of vSwitch and 658 NFs. In order to get consistent benchmarking results, the mapping 659 ratios are enforced using Linux core pinning. 661 +-------------+--------+----------+----------------+----------------+ 662 | application | thread | app:core | threads/pcores | threads/lcores | 663 | | type | ratio | (SMT disabled) | map (SMT | 664 | | | | | enabled) | 665 +-------------+--------+----------+----------------+----------------+ 666 | vSwitch-1c | data | 1:1 | 1DT/1PC | 2DT/2LC | 667 | | | | | | 668 | | main | 1:S2 | 1MT/S2PC | 1MT/1LC | 669 | | | | | | 670 | | | | | | 671 | | | | | | 672 | vSwitch-2c | data | 1:2 | 2DT/2PC | 4DT/4LC | 673 | | | | | | 674 | | main | 1:S2 | 1MT/S2PC | 1MT/1LC | 675 | | | | | | 676 | | | | | | 677 | | | | | | 678 | vSwitch-4c | data | 1:4 | 4DT/4PC | 8DT/8LC | 679 | | | | | | 680 | | main | 1:S2 | 1MT/S2PC | 1MT/1LC | 681 | | | | | | 682 | | | | | | 683 | | | | | | 684 | NF-0.5c | data | 1:S2 | 1DT/S2PC | 1DT/1LC | 685 | | | | | | 686 | | main | 1:S2 | 1MT/S2PC | 1MT/1LC | 687 | | | | | | 688 | | | | | | 689 | | | | | | 690 | NF-1c | data | 1:1 | 1DT/1PC | 2DT/2LC | 691 | | | | | | 692 | | main | 1:S2 | 1MT/S2PC | 1MT/1LC | 693 | | | | | | 694 | | | | | | 695 | | | | | | 696 | NF-2c | data | 1:2 | 2DT/2PC | 4DT/4LC | 697 | | | | | | 698 | | main | 1:S2 | 1MT/S2PC | 1MT/1LC | 699 +-------------+--------+----------+----------------+----------------+ 701 o Legend to table 703 * Header row 705 + application - network application with optimized data-plane, 706 a vSwitch or Network Function (NF) application. 708 + thread type - either "data", short for data-plane; or 709 "main", short for all main-control threads. 711 + app:core ratio - ratio of per application instance threads 712 of specific thread type to physical cores. 714 + threads/pcores (SMT disabled) - number of threads of 715 specific type (DT for data-plane thread, MT for main thread) 716 running on a number of physical cores, with SMT disabled. 718 + threads/lcores map (SMT enabled) - number of threads of 719 specific type (DT, MT) running on a number of logical cores, 720 with SMT enabled. Two logical cores per one physical core. 722 * Content rows 724 + vSwitch-(1c|2c|4c) - vSwitch with 1 physical core (or 2, or 725 4) allocated to its data-plane software worker threads. 727 + NF-(0.5c|1c|2c) - NF application with half of a physical 728 core (or 1, or 2) allocated to its data-plane software 729 worker threads. 731 + Sn - shared core, sharing ratio of (n). 733 + DT - data-plane thread. 735 + MT - main-control thread. 737 + PC - physical core, with SMT/HT enabled has many (mostly 2 738 today) logical cores associated with it. 740 + LC - logical core, if more than one lc get allocated in sets 741 of two sibling logical cores running on the same physical 742 core. 744 + SnPC - shared physical core, sharing ratio of (n). 746 + SnLC - shared logical core, sharing ratio of (n). 748 Maximum benchmarked NFV service densities are limited by a number of 749 physical cores on a compute node. 751 A sample physical core usage view is shown in the matrix below. 753 NFV Service Density - Core Usage View 754 vSwitch-1c, NF-1c 756 SVC 001 002 004 006 008 010 757 001 2 3 6 9 12 15 758 002 3 6 12 18 24 30 759 004 6 12 24 36 48 60 760 006 9 18 36 54 72 90 761 008 12 24 48 72 96 120 762 010 15 30 60 90 120 150 764 RowIndex: Number of NFV Service Instances, 1..10. 765 ColumnIndex: Number of NFs per NFV Service Instance, 1..10. 766 Value: Total number of physical processor cores used for NFs. 768 8. NFV Service Data-Plane Benchmarking 770 NF service density scenarios should have their data-plane performance 771 benchmarked using existing and/or emerging network benchmarking 772 standards as noted earlier. 774 Following metrics should be measured (or calculated) and reported: 776 o Packet throughput rate (packets-per-second) 778 * Specific to tested packet size or packet sequence (e.g. some 779 type of packet size mix sent in recurrent sequence). 781 * Applicable types of throughput rate: NDR, PDR, MRR. 783 o (Calculated) Bandwidth throughput rate (bits-per-second) 784 corresponding to the measured packet throughput rate. 786 o Packet one-way latency (seconds) 788 * Measured at different packet throughput rates load e.g. light, 789 medium, heavy. 791 Listed metrics should be itemized per service instance and per 792 direction (e.g. forward/reverse) for latency. 794 9. Sample NFV Service Density Benchmarks 796 To illustrate defined NFV service density applicability, following 797 sections describe three sets of NFV service topologies and 798 configurations that have been benchmarked in open-source: i) in 799 [LFN-FDio-CSIT], a continuous testing and data-plane benchmarking 800 project, ii) as part of CNCF CNF Testbed initiative 801 [CNCF-CNF-Testbed] and iii) in OPNFV NFVbench project. 803 In the first two cases each NFV service instance definition is based 804 on the same set of NF applications, and varies only by network 805 addressing configuration to emulate multi-tenant operating 806 environment. 808 OPNFV NFVbench project is focusing on benchmarking the actual 809 production deployments that are aligned with OPNFV specifications. 811 9.1. Intrepreting the Sample Results 813 TODO How to interpret and avoid misreading included results? And how 814 to avoid falling into the trap of using these results to draw 815 generilized conclusions about performance of different virtualization 816 technologies, e.g. VM and Containers, irrespective of deployment 817 scenarios and what VNFs and CNFs are in the actual use. 819 9.2. Benchmarking MRR Throughput 821 Initial NFV density throughput benchmarks have been performed using 822 Maximum Receive Rate (MRR) test methodology defined and used in FD.io 823 CSIT. 825 MRR tests measure the packet forwarding rate under specified Maximum 826 Transmit Rate (MTR) packet load offered by traffic generator over a 827 set trial duration, regardless of packet loss ratio (PLR). MTR for 828 specified Ethernet frame size was set to the bi-directional link 829 rate, 2x 10GbE in referred results. 831 Tests were conducted with two traffic profiles: i) continuous stream 832 of 64B frames, ii) continuous stream of IMIX sequence of (7x 64B, 4x 833 570B, 1x 1518B), all sizes are L2 untagged Ethernet. 835 NFV service topologies tested include: VNF service chains, CNF 836 service chains and CNF service pipelines. 838 9.3. VNF Service Chain 840 VNF Service Chain (VSC) topology is tested with KVM hypervisor 841 (Ubuntu 18.04-LTS), with NFV service instances consisting of NFs 842 running in VMs (VNFs). Host data-plane is provided by FD.io VPP 843 vswitch. Virtual interfaces are virtio-vhostuser. Snake forwarding 844 packet path is tested using [TRex] traffic generator, see figure. 846 +-----------------------------------------------------------+ 847 | Host Compute Node | 848 | | 849 | +--------+ +--------+ +--------+ | 850 | | S1VNF1 | | S1VNF2 | | S1VNFn | | 851 | | | | | .... | | Service1 | 852 | | XXXX | | XXXX | | XXXX | | 853 | +-+X--X+-+ +-+X--X+-+ +-+X--X+-+ | 854 | |X X| |X X| |X X| Virtual | 855 | |X X| |X X| |X X| |X X| Interfaces | 856 | +-+X--X+------+X--X+------+X--X+------+X--X+-+ | 857 | | X XXXXXXXXXX XXXXXXXXXX XXXXXXXXXX X | | 858 | | X X | | 859 | | X FD.io VPP vSwitch X | | 860 | +-+X-+----------------------------------+-X+-+ | 861 | |X | | X| | 862 +----X--------------------------------------X---------------+ 863 |X | | X| Physical 864 |X | | X| Interfaces 865 +---+X-+----------------------------------+-X+--------------+ 866 | | 867 | Traffic Generator (TRex) | 868 | | 869 +-----------------------------------------------------------+ 871 Figure 6. VNF service chain test setup. 873 9.4. CNF Service Chain 875 CNF Service Chain (CSC) topology is tested with Docker containers 876 (Ubuntu 18.04-LTS), with NFV service instances consisting of NFs 877 running in Containers (CNFs). Host data-plane is provided by FD.io 878 VPP vswitch. Virtual interfaces are memif. Snake forwarding packet 879 path is tested using [TRex] traffic generator, see figure. 881 +-----------------------------------------------------------+ 882 | Host Compute Node | 883 | | 884 | +--------+ +--------+ +--------+ | 885 | | S1CNF1 | | S1CNF2 | | S1CNFn | | 886 | | | | | .... | | Service1 | 887 | | XXXX | | XXXX | | XXXX | | 888 | +-+X--X+-+ +-+X--X+-+ +-+X--X+-+ | 889 | |X X| |X X| |X X| Virtual | 890 | |X X| |X X| |X X| |X X| Interfaces | 891 | +-+X--X+------+X--X+------+X--X+------+X--X+-+ | 892 | | X XXXXXXXXXX XXXXXXXXXX XXXXXXXXXX X | | 893 | | X X | | 894 | | X FD.io VPP vSwitch X | | 895 | +-+X-+----------------------------------+-X+-+ | 896 | |X | | X| | 897 +----X--------------------------------------X---------------+ 898 |X | | X| Physical 899 |X | | X| Interfaces 900 +---+X-+----------------------------------+-X+--------------+ 901 | | 902 | Traffic Generator (TRex) | 903 | | 904 +-----------------------------------------------------------+ 906 Figure 7. CNF service chain test setup. 908 9.5. CNF Service Pipeline 910 CNF Service Pipeline (CSP) topology is tested with Docker containers 911 (Ubuntu 18.04-LTS), with NFV service instances consisting of NFs 912 running in Containers (CNFs). Host data-plane is provided by FD.io 913 VPP vswitch. Virtual interfaces are memif. Pipeline forwarding 914 packet path is tested using [TRex] traffic generator, see figure. 916 +-----------------------------------------------------------+ 917 | Host Compute Node | 918 | | 919 | +--------+ +--------+ +--------+ | 920 | | S1NF1 | | S1NF2 | | S1NFn | | 921 | | +--+ +--+ .... +--+ | Service1 | 922 | | XXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXX | | 923 | +--X-----+ +--------+ +-----X--+ | 924 | |X X| Virtual | 925 | |X X| Interfaces | 926 | +-+X--------------------------------------X+-+ | 927 | | X X | | 928 | | X X | | 929 | | X FD.io VPP vSwitch X | | 930 | +-+X-+----------------------------------+-X+-+ | 931 | |X | | X| | 932 +----X--------------------------------------X---------------+ 933 |X | | X| Physical 934 |X | | X| Interfaces 935 +---+X-+----------------------------------+-X+--------------+ 936 | | 937 | Traffic Generator (TRex) | 938 | | 939 +-----------------------------------------------------------+ 941 Figure 8. CNF service chain test setup. 943 9.6. Sample Results: FD.io CSIT 945 FD.io CSIT project introduced NFV density benchmarking in release 946 CSIT-1904 and published results for the following NFV service 947 topologies and configurations: 949 1. VNF Service Chains 951 * VNF: DPDK-L3FWD v19.02 953 + IPv4 forwarding 955 + NF-1c 957 * vSwitch: VPP v19.04-release 959 + L2 MAC switching 961 + vSwitch-1c, vSwitch-2c 963 * frame sizes: 64B, IMIX 965 2. CNF Service Chains 967 * CNF: VPP v19.04-release 969 + IPv4 routing 971 + NF-1c 973 * vSwitch: VPP v19.04-release 975 + L2 MAC switching 977 + vSwitch-1c, vSwitch-2c 979 * frame sizes: 64B, IMIX 981 3. CNF Service Pipelines 983 * CNF: VPP v19.04-release 985 + IPv4 routing 987 + NF-1c 989 * vSwitch: VPP v19.04-release 991 + L2 MAC switching 993 + vSwitch-1c, vSwitch-2c 995 * frame sizes: 64B, IMIX 997 More information is available in FD.io CSIT-1904 report, with 998 specific references listed below: 1000 o Testbed: [CSIT-1904-testbed-2n-skx] 1002 o Test environment: [CSIT-1904-test-enviroment] 1004 o Methodology: [CSIT-1904-nfv-density-methodology] 1006 o Results: [CSIT-1904-nfv-density-results] 1008 9.7. Sample Results: CNCF/CNFs 1010 CNCF CI team introduced a CNF testbed initiative focusing on 1011 benchmaring NFV density with open-source network applications running 1012 as VNFs and CNFs. Following NFV service topologies and 1013 configurations have been tested to date: 1015 1. VNF Service Chains 1017 * VNF: VPP v18.10-release 1019 + IPv4 routing 1021 + NF-1c 1023 * vSwitch: VPP v18.10-release 1025 + L2 MAC switching 1027 + vSwitch-1c, vSwitch-2c 1029 * frame sizes: 64B, IMIX 1031 2. CNF Service Chains 1033 * CNF: VPP v18.10-release 1035 + IPv4 routing 1037 + NF-1c 1039 * vSwitch: VPP v18.10-release 1041 + L2 MAC switching 1043 + vSwitch-1c, vSwitch-2c 1045 * frame sizes: 64B, IMIX 1047 3. CNF Service Pipelines 1049 * CNF: VPP v18.10-release 1051 + IPv4 routing 1053 + NF-1c 1055 * vSwitch: VPP v18.10-release 1057 + L2 MAC switching 1059 + vSwitch-1c, vSwitch-2c 1061 * frame sizes: 64B, IMIX 1063 More information is available in CNCF CNF Testbed github, with 1064 summary test results presented in summary markdown file, references 1065 listed below: 1067 o Results: [CNCF-CNF-Testbed-Results] 1069 9.8. Sample Results: OPNFV NFVbench 1071 TODO Add short NFVbench based test description, and NFVbench sweep 1072 chart with single VM per service instance: Y-axis packet throughput 1073 rate or bandwidth throughput rate, X-axis number of concurrent 1074 service instances. 1076 10. IANA Considerations 1078 No requests of IANA. 1080 11. Security Considerations 1082 Benchmarking activities as described in this memo are limited to 1083 technology characterization of a DUT/SUT using controlled stimuli in 1084 a laboratory environment, with dedicated address space and the 1085 constraints specified in the sections above. 1087 The benchmarking network topology will be an independent test setup 1088 and MUST NOT be connected to devices that may forward the test 1089 traffic into a production network or misroute traffic to the test 1090 management network. 1092 Further, benchmarking is performed on a "black-box" basis, relying 1093 solely on measurements observable external to the DUT/SUT. 1095 Special capabilities SHOULD NOT exist in the DUT/SUT specifically for 1096 benchmarking purposes. Any implications for network security arising 1097 from the DUT/SUT SHOULD be identical in the lab and in production 1098 networks. 1100 12. Acknowledgements 1102 Thanks to Vratko Polak of FD.io CSIT project and Michael Pedersen of 1103 the CNCF Testbed initiative for their contributions and useful 1104 suggestions. Extended thanks to Alec Hothan of OPNFV NFVbench 1105 project for numerous comments, suggestions and references to his/team 1106 work in the OPNFV/NVFbench project. 1108 13. References 1110 13.1. Normative References 1112 [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for 1113 Network Interconnect Devices", RFC 2544, 1114 DOI 10.17487/RFC2544, March 1999, 1115 . 1117 [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 1118 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, 1119 May 2017, . 1121 13.2. Informative References 1123 [BSDP] "Benchmarking Software Data Planes Intel(R) Xeon(R) 1124 Skylake vs. Broadwell", March 2019, . 1128 [CNCF-CNF-Testbed] 1129 "Cloud native Network Function (CNF) Testbed", July 2019, 1130 . 1132 [CNCF-CNF-Testbed-Results] 1133 "CNCF CNF Testbed: NFV Service Density Benchmarking", 1134 December 2018, . 1138 [CSIT-1904-nfv-density-methodology] 1139 "FD.io CSIT Test Methodology: NFV Service Density", June 1140 2019, 1141 . 1144 [CSIT-1904-nfv-density-results] 1145 "FD.io CSIT Test Results: NFV Service Density", June 2019, 1146 . 1149 [CSIT-1904-test-enviroment] 1150 "FD.io CSIT Test Environment", June 2019, 1151 . 1154 [CSIT-1904-testbed-2n-skx] 1155 "FD.io CSIT Test Bed", June 2019, 1156 . 1159 [draft-vpolak-bmwg-plrsearch] 1160 "Probabilistic Loss Ratio Search for Packet Throughput 1161 (PLRsearch)", July 2019, 1162 . 1164 [draft-vpolak-mkonstan-bmwg-mlrsearch] 1165 "Multiple Loss Ratio Search for Packet Throughput 1166 (MLRsearch)", July 2019, . 1169 [LFN-FDio-CSIT] 1170 "Fast Data io, Continuous System Integration and Testing 1171 Project", July 2019, . 1173 [NFVbench] 1174 "NFVbench Data Plane Performance Measurement Features", 1175 July 2019, . 1179 [RFC8204] Tahhan, M., O'Mahony, B., and A. Morton, "Benchmarking 1180 Virtual Switches in the Open Platform for NFV (OPNFV)", 1181 RFC 8204, DOI 10.17487/RFC8204, September 2017, 1182 . 1184 [TRex] "TRex Low-Cost, High-Speed Stateful Traffic Generator", 1185 July 2019, . 1188 [TST009] "ETSI GS NFV-TST 009 V3.1.1 (2018-10), Network Functions 1189 Virtualisation (NFV) Release 3; Testing; Specification of 1190 Networking Benchmarks and Measurement Methods for NFVI", 1191 October 2018, . 1194 Authors' Addresses 1196 Maciek Konstantynowicz (editor) 1197 Cisco Systems 1199 Email: mkonstan@cisco.com 1200 Peter Mikus (editor) 1201 Cisco Systems 1203 Email: pmikus@cisco.com