idnits 2.17.1 draft-ietf-bmwg-sdn-controller-benchmark-meth-02.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 544 has weird spacing: '...warding plane...' == Line 638 has weird spacing: '...ddition to...' -- The document date (July 8, 2016) is 2848 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Unused Reference: 'RFC2544' is defined on line 1291, but no explicit reference was found in the text == Unused Reference: 'RFC2330' is defined on line 1294, but no explicit reference was found in the text == Unused Reference: 'RFC6241' is defined on line 1298, but no explicit reference was found in the text == Unused Reference: 'RFC6020' is defined on line 1302, but no explicit reference was found in the text == Unused Reference: 'RFC5440' is defined on line 1306, but no explicit reference was found in the text == Unused Reference: 'I-D.sdn-controller-benchmark-term' is defined on line 1312, but no explicit reference was found in the text == Unused Reference: 'I-D.i2rs-architecture' is defined on line 1320, but no explicit reference was found in the text == Unused Reference: 'OpenContrail' is defined on line 1325, but no explicit reference was found in the text == Unused Reference: 'OpenDaylight' is defined on line 1329, but no explicit reference was found in the text == Outdated reference: A later version (-10) exists of draft-ietf-bmwg-sdn-controller-benchmark-term-02 == Outdated reference: A later version (-15) exists of draft-ietf-i2rs-architecture-09 Summary: 0 errors (**), 0 flaws (~~), 14 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 Internet-Draft Bhuvaneswaran Vengainathan 2 Network Working Group Anton Basil 3 Intended Status: Informational Veryx Technologies 4 Expires: January 8, 2017 Mark Tassinari 5 Hewlett-Packard 6 Vishwas Manral 7 Nano Sec 8 Sarah Banks 9 VSS Monitoring 10 July 8, 2016 12 Benchmarking Methodology for SDN Controller Performance 13 draft-ietf-bmwg-sdn-controller-benchmark-meth-02 15 Abstract 17 This document defines the methodologies for benchmarking control 18 plane performance of SDN controllers. Terminology related to 19 benchmarking SDN controllers is described in the companion 20 terminology document. SDN controllers have been implemented with 21 many varying designs in order to achieve their intended network 22 functionality. Hence, the authors have taken the approach of 23 considering an SDN controller as a black box, defining the 24 methodology in a manner that is agnostic to protocols and network 25 services supported by controllers. The intent of this document is to 26 provide a standard mechanism to measure the performance of all 27 controller implementations. 29 Status of this Memo 31 This Internet-Draft is submitted in full conformance with the 32 provisions of BCP 78 and BCP 79. 34 Internet-Drafts are working documents of the Internet Engineering 35 Task Force (IETF). Note that other groups may also distribute 36 working documents as Internet-Drafts. The list of current Internet- 37 Drafts is at http://datatracker.ietf.org/drafts/current. 39 Internet-Drafts are draft documents valid for a maximum of six 40 months and may be updated, replaced, or obsoleted by other documents 41 at any time. It is inappropriate to use Internet-Drafts as reference 42 material or to cite them other than as "work in progress. 44 This Internet-Draft will expire on January 8, 2017. 46 Copyright Notice 48 Copyright (c) 2016 IETF Trust and the persons identified as the 49 document authors. All rights reserved. 51 This document is subject to BCP 78 and the IETF Trust's Legal 52 Provisions Relating to IETF Documents 53 (http://trustee.ietf.org/license-info) in effect on the date of 54 publication of this document. Please review these documents 55 carefully, as they describe your rights and restrictions with 56 respect to this document. Code Components extracted from this 57 document must include Simplified BSD License text as described in 58 Section 4.e of the Trust Legal Provisions and are provided without 59 warranty as described in the Simplified BSD License. 61 Table of Contents 63 1. Introduction ................................................ 3 64 2. Scope ....................................................... 4 65 3. Test Setup .................................................. 4 66 3.1. Test setup - Controller working in Standalone Mode....... 5 67 3.2. Test setup - Controller working in Cluster Mode.......... 6 68 4. Test Considerations ......................................... 7 69 4.1. Network Topology ........................................ 7 70 4.2. Test Traffic ........................................... 7 71 4.3. Connection Setup ........................................ 7 72 4.4. Measurement Point Specification and Recommendation....... 8 73 4.5. Connectivity Recommendation ............................. 8 74 4.6. Test Repeatability ...................................... 8 75 5. Benchmarking Tests .......................................... 9 76 5.1. Performance ............................................ 9 77 5.1.1. Network Topology Discovery Time .................... 9 78 5.1.2. Asynchronous Message Processing Time .............. 11 79 5.1.3. Asynchronous Message Processing Rate .............. 12 80 5.1.4. Reactive Path Provisioning Time ................... 14 81 5.1.5. Proactive Path Provisioning Time .................. 15 82 5.1.6. Reactive Path Provisioning Rate ................... 16 83 5.1.7. Proactive Path Provisioning Rate .................. 18 84 5.1.8. Network Topology Change Detection Time ............ 19 85 5.2. 6.2 Scalability ........................................ 20 86 5.2.1. Control Session Capacity .......................... 20 87 5.2.2. Network Discovery Size ............................ 21 88 5.2.3. 6.2.3 Forwarding Table Capacity ................... 22 89 5.3. 6.3 Security .......................................... 24 90 5.3.1. 6.3.1 Exception Handling .......................... 24 91 5.3.2. Denial of Service Handling ........................ 25 93 5.4. Reliability ........................................... 26 94 5.4.1. Controller Failover Time .......................... 26 95 5.4.2. Network Re-Provisioning Time ...................... 28 96 6. References ................................................. 29 97 6.1. Normative References ................................... 29 98 6.2. Informative References ................................. 30 99 7. IANA Considerations ........................................ 30 100 8. Security Considerations ..................................... 30 101 9. Acknowledgments ............................................ 31 102 Appendix A. Example Test Topologies ............................ 32 103 A.1. Leaf-Spine Topology - Three Tier Network Architecture... 32 104 A.2. Leaf-Spine Topology - Two Tier Network Architecture..... 32 105 Appendix B. Benchmarking Methodology using OpenFlow Controllers. 33 106 B.1. Protocol Overview ...................................... 33 107 B.2. Messages Overview ...................................... 33 108 B.3. Connection Overview .................................... 33 109 B.4. Performance Benchmarking Tests ......................... 34 110 B.4.1. Network Topology Discovery Time ................... 34 111 B.4.2. Asynchronous Message Processing Time .............. 35 112 B.4.3. Asynchronous Message Processing Rate .............. 36 113 B.4.4. Reactive Path Provisioning Time ................... 37 114 B.4.5. Proactive Path Provisioning Time .................. 38 115 B.4.6. Reactive Path Provisioning Rate ................... 39 116 B.4.7. Proactive Path Provisioning Rate .................. 40 117 B.4.8. Network Topology Change Detection Time ............ 41 118 B.5. Scalability ........................................... 42 119 B.5.1. Control Sessions Capacity ......................... 42 120 B.5.2. Network Discovery Size ............................ 42 121 B.5.3. Forwarding Table Capacity ......................... 43 122 B.6. Security .............................................. 45 123 B.6.1. Exception Handling ................................ 45 124 B.6.2. Denial of Service Handling ........................ 46 125 B.7. Reliability ........................................... 48 126 B.7.1. Controller Failover Time .......................... 48 127 B.7.2. Network Re-Provisioning Time ...................... 49 128 Authors' Addresses ............................................ 52 130 1. Introduction 132 This document provides generic methodologies for benchmarking SDN 133 controller performance. An SDN controller may support many 134 northbound and southbound protocols, implement a wide range of 135 applications, and work solely, or as a group to achieve the desired 136 functionality. This document considers an SDN controller as a black 137 box, regardless of design and implementation. The tests defined in 138 the document can be used to benchmark SDN controller for 139 performance, scalability, reliability and security independent of 140 northbound and southbound protocols. These tests can be performed on 141 an SDN controller running as a virtual machine (VM) instance or on a 142 bare metal server. This document is intended for those who want to 143 measure the SDN controller performance as well as compare various 144 SDN controllers performance. 146 Conventions used in this document 148 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 149 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 150 document are to be interpreted as described in RFC 2119. 152 2. Scope 154 This document defines methodology to measure the networking metrics 155 of SDN controllers. For the purpose of this memo, the SDN controller 156 is a function that manages and controls Network Devices. Any SDN 157 controller without a control capability is out of scope for this 158 memo. The tests defined in this document enable benchmarking of SDN 159 Controllers in two ways; as a standalone controller and as a cluster 160 of homogeneous controllers. These tests are recommended for 161 execution in lab environments rather than in live network 162 deployments. Performance benchmarking of a federation of controllers 163 is beyond the scope of this document. 165 3. Test Setup 167 The tests defined in this document enable measurement of an SDN 168 controllers performance in standalone mode and cluster mode. This 169 section defines common reference topologies that are later referred 170 to in individual tests. 172 3.1. Test setup - Controller working in Standalone Mode 174 +-----------------------------------------------------------+ 175 | Application Plane Test Emulator | 176 | | 177 | +-----------------+ +-------------+ | 178 | | Application | | Service | | 179 | +-----------------+ +-------------+ | 180 | | 181 +-----------------------------+(I2)-------------------------+ 182 | 183 | 184 | (Northbound interface) 185 +-------------------------------+ 186 | +----------------+ | 187 | | SDN Controller | | 188 | +----------------+ | 189 | | 190 | Device Under Test (DUT) | 191 +-------------------------------+ 192 | (Southbound interface) 193 | 194 | 195 +-----------------------------+(I1)-------------------------+ 196 | | 197 | +-----------+ +-----------+ | 198 | | Network |l1 ln-1| Network | | 199 | | Device 1 |---- .... ----| Device n | | 200 | +-----------+ +-----------+ | 201 | |l0 |ln | 202 | | | | 203 | | | | 204 | +---------------+ +---------------+ | 205 | | Test Traffic | | Test Traffic | | 206 | | Generator | | Generator | | 207 | | (TP1) | | (TP2) | | 208 | +---------------+ +---------------+ | 209 | | 210 | Forwarding Plane Test Emulator | 211 +-----------------------------------------------------------+ 213 Figure 1 215 3.2. Test setup - Controller working in Cluster Mode 217 +-----------------------------------------------------------+ 218 | Application Plane Test Emulator | 219 | | 220 | +-----------------+ +-------------+ | 221 | | Application | | Service | | 222 | +-----------------+ +-------------+ | 223 | | 224 +-----------------------------+(I2)-------------------------+ 225 | 226 | 227 | (Northbound interface) 228 +---------------------------------------------------------+ 229 | | 230 | ------------------ ------------------ | 231 | | SDN Controller 1 | <--E/W--> | SDN Controller n | | 232 | ------------------ ------------------ | 233 | | 234 | Device Under Test (DUT) | 235 +---------------------------------------------------------+ 236 | (Southbound interface) 237 | 238 | 239 +-----------------------------+(I1)-------------------------+ 240 | | 241 | +-----------+ +-----------+ | 242 | | Network |l1 ln-1| Network | | 243 | | Device 1 |---- .... ----| Device n | | 244 | +-----------+ +-----------+ | 245 | |l0 |ln | 246 | | | | 247 | | | | 248 | +---------------+ +---------------+ | 249 | | Test Traffic | | Test Traffic | | 250 | | Generator | | Generator | | 251 | | (TP1) | | (TP2) | | 252 | +---------------+ +---------------+ | 253 | | 254 | Forwarding Plane Test Emulator | 255 +-----------------------------------------------------------+ 257 Figure 2 259 4. Test Considerations 261 4.1. Network Topology 263 The test cases SHOULD use Leaf-Spine topology with at least 1 264 Network Device in the topology for benchmarking. The test traffic 265 generators TP1 and TP2 SHOULD be connected to the first and the last 266 leaf Network Device. If a test case uses test topology with 1 267 Network Device, the test traffic generators TP1 and TP2 SHOULD be 268 connected to the same node. However to achieve a complete 269 performance characterization of the SDN controller, it is 270 recommended that the controller be benchmarked for many network 271 topologies and a varying number of Network Devices. This document 272 includes a few sample test topologies, defined in Section 10 - 273 Appendix A for reference. Further, care should be taken to make sure 274 that a loop prevention mechanism is enabled either in the SDN 275 controller, or in the network when the topology contains redundant 276 network paths. 278 4.2. Test Traffic 280 Test traffic is used to notify the controller about the arrival of 281 new flows. The test cases SHOULD use multiple frame sizes as 282 recommended in RFC2544 for benchmarking. 284 4.3. Test Emulator Requirements 286 The Test Emulator SHOULD time stamp the transmitted and received 287 control messages to/from the controller on the established network 288 connections. The test cases use these values to compute the 289 controller processing time. 291 4.4. Connection Setup 293 There may be controller implementations that support unencrypted and 294 encrypted network connections with Network Devices. Further, the 295 controller may have backward compatibility with Network Devices 296 running older versions of southbound protocols. It is recommended 297 that the controller performance be measured with one or more 298 applicable connection setup methods defined below. 300 1.Unencrypted connection with Network Devices, running same 301 protocol version. 302 2.Unencrypted connection with Network Devices, running different 303 protocol versions. 304 Example: 306 a.Controller running current protocol version and switch 307 running older protocol version 308 b.Controller running older protocol version and switch 309 running current protocol version 310 3.Encrypted connection with Network Devices, running same 311 protocol version 312 4.Encrypted connection with Network Devices, running different 313 protocol versions. 314 Example: 315 a.Controller running current protocol version and switch 316 running older protocol version 317 b.Controller running older protocol version and switch 318 running current protocol version 320 4.5. Measurement Point Specification and Recommendation 322 The measurement accuracy depends on several factors including the 323 point of observation where the indications are captured. For 324 example, the notification can be observed at the controller or test 325 emulator. The test operator SHOULD make the observations/ 326 measurements at the interfaces of test emulator unless it is 327 explicitly mentioned otherwise in the individual test. 329 4.6. Connectivity Recommendation 331 The SDN controller in the test setup SHOULD be connected directly 332 with the forwarding and the management plane test emulators to avoid 333 any delays or failure introduced by the intermediate devices during 334 benchmarking tests. 336 4.7. Test Repeatability 338 To increase the confidence in measured result, it is recommended 339 that each test SHOULD be repeated a minimum of 10 times. 341 Test Reporting 343 Each test has a reporting format that contains some global and 344 identical reporting components, and some individual components that 345 are specific to individual tests. The following test configuration 346 parameters and controller settings parameters MUST be reflected in 347 the test report. 349 Test Configuration Parameters: 351 1.Controller name and version 352 2.Northbound protocols and versions 353 3.Southbound protocols and versions 354 4.Controller redundancy mode (Standalone or Cluster Mode) 355 5.Connection setup (Unencrypted or Encrypted) 356 6.Network Topology (Mesh or Tree or Linear) 357 7.Network Device Type (Physical or Virtual or Emulated) 358 8.Number of Nodes 359 9.Number of Links 360 10.Test Traffic Type 361 11.Controller System Configuration (e.g., CPU, Memory, Operating 362 System, Interface Speed etc.,) 363 12.Reference Test Setup (e.g., Section 3.1 etc.,) 365 Controller Settings Parameters: 366 1.Topology re-discovery timeout 367 2.Controller redundancy mode (e.g., active-standby etc.,) 369 To ensure the repeatability of test, the following capabilities of 370 test emulator SHOULD be reported 372 1.Maximum number of Network Devices that the forwarding plane 373 emulates 374 2.Control message processing time (e.g., Topology Discovery 375 Messages) 377 One way to determine the above two values are to simulate the 378 required control sessions and messages from the control plane. 380 5. Benchmarking Tests 382 5.1. Performance 384 5.1.1. Network Topology Discovery Time 386 Objective: 388 The time taken by controller(s) to determine the complete network 389 topology, defined as the interval starting with the first discovery 390 message from the controller(s) at its Southbound interface, ending 391 with all features of the static topology determined. 393 Reference Test Setup: 395 The test SHOULD use one of the test setups described in section 3.1 396 or section 3.2 of this document. 398 Prerequisite: 400 1. The controller MUST support network discovery. 401 2. Tester should be able to retrieve the discovered topology 402 information either through the controller's management interface, 403 or northbound interface to determine if the discovery was 404 successful and complete. 405 3. Ensure that the controller's topology re-discovery timeout has 406 been set to the maximum value to avoid initiation of re-discovery 407 process in the middle of the test. 409 Procedure: 411 1. Ensure that the controller is operational, its network 412 applications, northbound and southbound interfaces are up and 413 running. 414 2. Establish the network connections between controller and Network 415 Devices. 416 3. Record the time for the first discovery message (Tm1) received 417 from the controller at forwarding plane test emulator interface 418 I1. 419 4. Query the controller every 3 seconds to obtain the discovered 420 network topology information through the northbound interface or 421 the management interface and compare it with the deployed network 422 topology information. 423 5. Stop the test when the discovered topology information matches the 424 deployed network topology, or when the discovered topology 425 information for 3 consecutive queries return the same details. 426 6. Record the time last discovery message (Tmn) sent to controller 427 from the forwarding plane test emulator interface (I1) when the 428 test completed successfully. (e.g., the topology matches). 430 Measurement: 432 Topology Discovery Time Tr1 = Tmn-Tm1. 434 Tr1 + Tr2 + Tr3 .. Trn 435 Average Topology Discovery Time = ----------------------- 436 Total Test Iterations 438 Reporting Format: 440 The Topology Discovery Time results MUST be reported in the format 441 of a table, with a row for each successful iteration. The last row 442 of the table indicates the average Topology Discovery Time. 444 If this test is repeated with varying number of nodes over the same 445 topology, the results SHOULD be reported in the form of a graph. The 446 X coordinate SHOULD be the Number of nodes (N), the Y coordinate 447 SHOULD be the average Topology Discovery Time. 449 If this test is repeated with same number of nodes over different 450 topologies, the results SHOULD be reported in the form of a graph. 451 The X coordinate SHOULD be the Topology Type, the Y coordinate 452 SHOULD be the average Topology Discovery Time. 454 5.1.2. Asynchronous Message Processing Time 456 Objective: 458 The time taken by controller(s) to process an asynchronous message, 459 defined as the interval starting with an asynchronous message from a 460 network device after the discovery of all the devices by the 461 controller(s), ending with a response message from the controller(s) 462 at its Southbound interface. 464 Reference Test Setup: 466 This test SHOULD use one of the test setup described in section 3.1 467 or section 3.2 of this document. 469 Prerequisite: 471 1.The controller MUST have completed the network topology discovery 472 for the connected Network Devices. 474 Procedure: 476 1.Generate asynchronous messages from every connected Network 477 Device, to the SDN controller, one at a time in series from the 478 forwarding plane test emulator for the test duration. 479 2.Record every request transmit (T1) timestamp and the 480 corresponding response (R1) received timestamp at the 481 forwarding plane test emulator interface (I1) for every 482 successful message exchange. 484 Measurement: 486 (R1-T1) + (R2-T2)..(Rn-Tn) 487 Asynchronous Message Processing Time Tr1 = ----------------------- 488 Nrx 490 Where Nrx is the total number of successful messages exchanged 491 Tr1 + Tr2 + Tr3..Trn 492 Average Asynchronous Message Processing Time= -------------------- 493 Total Test Iterations 495 Reporting Format: 497 The Asynchronous Message Processing Time results MUST be reported in 498 the format of a table with a row for each iteration. The last row of 499 the table indicates the average Asynchronous Message Processing 500 Time. 502 The report should capture the following information in addition to 503 the configuration parameters captured in section 5. - Successful 504 messages exchanged (Nrx) 506 If this test is repeated with varying number of nodes with same 507 topology, the results SHOULD be reported in the form of a graph. The 508 X coordinate SHOULD be the Number of nodes (N), the Y coordinate 509 SHOULD be the average Asynchronous Message Processing Time. 511 If this test is repeated with same number of nodes using different 512 topologies, the results SHOULD be reported in the form of a graph. 513 The X coordinate SHOULD be the Topology Type, the Y coordinate 514 SHOULD be the average Asynchronous Message Processing Time. 516 5.1.3. Asynchronous Message Processing Rate 518 Objective: 520 The maximum number of asynchronous messages (session aliveness check 521 message, new flow arrival notification message etc.) that the 522 controller(s) can process, defined as the number of asynchronous 523 messages the controller(s) can process at its Southbound interface 524 between the start of the test and the expiry of given test duration 526 Reference Test Setup: 528 The test SHOULD use one of the test setups described in section 3.1 529 or section 3.2 of this document. 531 Prerequisite: 533 1. The controller MUST have completed the network topology discovery 534 for the connected Network Devices. 536 Procedure: 538 1. Generate asynchronous messages continuously at the maximum 539 possible rate on the established connections from all the 540 connected Network Devices in the forwarding plane test emulator 541 for the Test Duration (Td). 542 2. Record the total number of responses received from the controller 543 (Nrx) as well as the number of messages sent(Ntx) to the 544 controller within the test duration(Td) at the forwarding plane 545 test emulator interface (I1). 547 Measurement: 549 Nrx 550 Asynchronous Message Processing Rate Tr1 = ----- 551 Td 553 Tr1 + Tr2 + Tr3..Trn 554 Average Asynchronous Message Processing Rate= -------------------- 555 Total Test Iterations 557 Loss Ratio = (Ntx-Nrx)/100. 559 Reporting Format: 561 The Asynchronous Message Processing Rate results MUST be reported in 562 the format of a table with a row for each iteration. The last row of 563 the table indicates the average Asynchronous Message Processing 564 Rate. 566 The report should capture the following information in addition to 567 the configuration parameters captured in section 5. 569 - Offered rate (Ntx) 571 - Loss Ratio 573 If this test is repeated with varying number of nodes over same 574 topology, the results SHOULD be reported in the form of a graph. The 575 X coordinate SHOULD be the Number of nodes (N), the Y coordinate 576 SHOULD be the average Asynchronous Message Processing Rate. 578 If this test is repeated with same number of nodes over different 579 topologies, the results SHOULD be reported in the form of a graph. 580 The X coordinate SHOULD be the Topology Type, the Y coordinate 581 SHOULD be the average Asynchronous Message Processing Rate. 583 5.1.4. Reactive Path Provisioning Time 585 Objective: 587 The time taken by the controller to setup a path reactively between 588 source and destination node, defined as the interval starting with 589 the first flow provisioning request message received by the 590 controller(s), ending with the last flow provisioning response 591 message sent from the controller(s) at it Southbound interface. 593 Reference Test Setup: 595 The test SHOULD use one of the test setups described in section 3.1 596 or section 3.2 of this document. 598 Prerequisite: 600 1. The controller MUST contain the network topology information for 601 the deployed network topology. 602 2. The controller should have the knowledge about the location of 603 destination endpoint for which the path has to be provisioned. 604 This can be achieved through dynamic learning or static 605 provisioning. 606 3. Ensure that the default action for 'flow miss' in Network Device 607 is configured to 'send to controller'. 608 4. Ensure that each Network Device in a path requires the controller 609 to make the forwarding decision while paving the entire path. 611 Procedure: 613 1. Send a single traffic stream from the test traffic generator TP1 614 to test traffic generator TP2. 615 2. Record the time of the first flow provisioning request message 616 sent to the controller (Tsf1) from the Network Device at the 617 forwarding plane test emulator interface (I1). 618 3. Wait for the arrival of first traffic frame at the Traffic 619 Endpoint TP2 or the expiry of test duration (Td). 620 4. Record the time of the last flow provisioning response message 621 received from the controller (Tdf1) to the Network Device at the 622 forwarding plane test emulator interface (I1). 624 Measurement: 626 Reactive Path Provisioning Time Tr1 = Tdf1-Tsf1. 628 Tr1 + Tr2 + Tr3 .. Trn 629 Average Reactive Path Provisioning Time = ----------------------- 630 Total Test Iterations 632 Reporting Format: 634 The Reactive Path Provisioning Time results MUST be reported in the 635 format of a table with a row for each iteration. The last row of the 636 table indicates the Average Reactive Path Provisioning Time 638 The report should capture the following information in addition to 639 the configuration parameters captured in section 5. 641 - Number of Network Devices in the path 643 5.1.5. Proactive Path Provisioning Time 645 Objective: 647 The time taken by the controller to setup a path proactively between 648 source and destination node, defined as the interval starting with 649 the first proactive flow provisioned in the controller(s) at its 650 Northbound interface, ending with the last flow provisioning 651 response message sent from the controller(s) at it Southbound 652 interface. 654 Reference Test Setup: 656 The test SHOULD use one of the test setups described in section 3.1 657 or section 3.2 of this document. 659 Prerequisite: 661 1. The controller MUST contain the network topology information for 662 the deployed network topology. 663 2. The controller should have the knowledge about the location of 664 destination endpoint for which the path has to be provisioned. 665 This can be achieved through dynamic learning or static 666 provisioning. 667 3. Ensure that the default action for flow miss in Network Device is 668 'drop'. 670 Procedure: 672 1. Send a single traffic stream from test traffic generator TP1 to 673 TP2. 674 2. Install the flow entries to reach from test traffic generator TP1 675 to the test traffic generator TP2 through controller's northbound 676 or management interface. 677 3. Wait for the arrival of first traffic frame at the test traffic 678 generator TP2 or the expiry of test duration (Td). 679 4. Record the time when the proactive flow is provisioned in the 680 Controller (Tsf1) at the management plane test emulator interface 681 I2. 682 5. Record the time of the last flow provisioning message received 683 from the controller (Tdf1) at the forwarding plane test emulator 684 interface I1. 686 Measurement: 688 Proactive Flow Provisioning Time Tr1 = Tdf1-Tsf1. 690 Tr1 + Tr2 + Tr3 .. Trn 691 Average Proactive Path Provisioning Time = ----------------------- 692 Total Test Iterations 694 Reporting Format: 696 The Proactive Path Provisioning Time results MUST be reported in the 697 format of a table with a row for each iteration. The last row of the 698 table indicates the Average Proactive Path Provisioning Time. 700 The report should capture the following information in addition to 701 the configuration parameters captured in section 5. 703 - Number of Network Devices in the path 705 5.1.6. Reactive Path Provisioning Rate 707 Objective: 709 The maximum number of independent paths a controller can 710 concurrently establish between source and destination nodes 711 reactively, defined as the number of paths provisioned by the 712 controller(s) at its Southbound interface for the flow provisioning 713 requests received for path provisioning at its Southbound interface 714 between the start of the test and the expiry of given test duration. 716 Reference Test Setup: 718 The test SHOULD use one of the test setups described in section 3.1 719 or section 3.2 of this document. 721 Prerequisite: 723 1. The controller MUST contain the network topology information for 724 the deployed network topology. 725 2. The controller should have the knowledge about the location of 726 destination addresses for which the paths have to be provisioned. 727 This can be achieved through dynamic learning or static 728 provisioning. 729 3. Ensure that the default action for 'flow miss' in Network Device 730 is configured to 'send to controller'. 731 4. Ensure that each Network Device in a path requires the controller 732 to make the forwarding decision while provisioning the entire 733 path. 735 Procedure: 737 1. Send traffic with unique source and destination addresses from 738 test traffic generator TP1. 739 2. Record total number of unique traffic frames (Ndf) received at the 740 test traffic generator TP2 within the test duration (Td). 742 Measurement: 744 Ndf 745 Reactive Path Provisioning Rate Tr1 = ------ 746 Td 748 Tr1 + Tr2 + Tr3 .. Trn 749 Average Reactive Path Provisioning Rate = ------------------------ 750 Total Test Iterations 752 Reporting Format: 754 The Reactive Path Provisioning Rate results MUST be reported in the 755 format of a table with a row for each iteration. The last row of the 756 table indicates the Average Reactive Path Provisioning Rate. 758 The report should capture the following information in addition to 759 the configuration parameters captured in section 5. 761 - Number of Network Devices in the path 762 - Offered rate 764 5.1.7. Proactive Path Provisioning Rate 766 Objective: 768 Measure the maximum number of independent paths a controller can 769 concurrently establish between source and destination nodes 770 proactively, defined as the number of paths provisioned by the 771 controller(s) at its Southbound interface for the paths provisioned 772 in its Northbound interface between the start of the test and the 773 expiry of given test duration . 775 Reference Test Setup: 777 The test SHOULD use one of the test setups described in section 3.1 778 or section 3.2 of this document. 780 Prerequisite: 782 1. The controller MUST contain the network topology information for 783 the deployed network topology. 785 2. The controller should have the knowledge about the location of 786 destination addresses for which the paths have to be provisioned. 787 This can be achieved through dynamic learning or static 788 provisioning. 790 3. Ensure that the default action for flow miss in Network Device is 791 'drop'. 793 Procedure: 795 1. Send traffic continuously with unique source and destination 796 addresses from test traffic generator TP1. 798 2. Install corresponding flow entries to reach from simulated 799 sources at the test traffic generator TP1 to the simulated 800 destinations at test traffic generator TP2 through controller's 801 northbound or management interface. 803 3. Record total number of unique traffic frames received Ndf) at the 804 test traffic generator TP2 within the test duration (Td). 806 Measurement: 808 Ndf 809 Proactive Path Provisioning Rate Tr1 = ------ 810 Td 812 Tr1 + Tr2 + Tr3 .. Trn 813 Average Proactive Path Provisioning Rate = ----------------------- 814 Total Test Iterations 816 Reporting Format: 818 The Proactive Path Provisioning Rate results MUST be reported in the 819 format of a table with a row for each iteration. The last row of the 820 table indicates the Average Proactive Path Provisioning Rate. 822 The report should capture the following information in addition to 823 the configuration parameters captured in section 5. 825 - Number of Network Devices in the path 827 - Offered rate 829 5.1.8. Network Topology Change Detection Time 831 Objective: 833 The amount of time required for the controller to detect any changes 834 in the network topology, defined as the interval starting with the 835 notification message received by the controller(s) at its Southbound 836 interface, ending with the first topology rediscovery messages sent 837 from the controller(s) at its Southbound interface. 839 Reference Test Setup: 841 The test SHOULD use one of the test setups described in section 3.1 842 or section 3.2 of this document. 844 Prerequisite: 846 1. The controller MUST have discovered the network topology 847 information for the deployed network topology. 849 2. The periodic network discovery operation should be configured to 850 twice the Test duration (Td) value. 852 Procedure: 854 1. Trigger a topology change event by bringing down an active 855 Network Device in the topology. 857 2. Record the time when the first topology change notification is 858 sent to the controller (Tcn) at the forwarding plane test emulator 859 interface (I1). 861 3. Stop the test when the controller sends the first topology re- 862 discovery message to the Network Device or the expiry of test 863 interval (Td). 865 4. Record the time when the first topology re-discovery message is 866 received from the controller (Tcd) at the forwarding plane test 867 emulator interface (I1) 869 Measurement: 871 Network Topology Change Detection Time Tr1 = Tcd-Tcn. 873 Tr1 + Tr2 + Tr3 .. Trn 874 Average Network Topology Change Detection Time = ------------------ 875 Total Test Iterations 877 Reporting Format: 879 The Network Topology Change Detection Time results MUST be reported 880 in the format of a table with a row for each iteration. The last 881 row of the table indicates the average Network Topology Change Time. 883 5.2. 6.2 Scalability 885 5.2.1. Control Session Capacity 887 Objective: 889 Measure the maximum number of control sessions the controller can 890 maintain, defined as the number of sessions that the controller can 891 accept from network devices, starting with the first control 892 session, ending with the last control session that the controller(s) 893 accepts at its Southbound interface. 895 Reference Test Setup: 897 The test SHOULD use one of the test setups described in section 3.1 898 or section 3.2 of this document. 900 Procedure: 902 1. Establish control connection with controller from every Network 903 Device emulated in the forwarding plane test emulator. 904 2. Stop the test when the controller starts dropping the control 905 connection. 906 3. Record the number of successful connections established with the 907 controller (CCn) at the forwarding plane test emulator. 909 Measurement: 911 Control Sessions Capacity = CCn. 913 Reporting Format: 915 The Control Session Capacity results MUST be reported in addition to 916 the configuration parameters captured in section 5. 918 5.2.2. Network Discovery Size 920 Objective: 922 Measure the network size (number of nodes, links and hosts) that a 923 controller can discover, defined as the size of a network that the 924 controller(s) can discover, starting from a network topology given 925 by the user for discovery, ending with the topology that the 926 controller(s) could successfully discover. 928 Reference Test Setup: 930 The test SHOULD use one of the test setups described in section 3.1 931 or section 3.2 of this document. 933 Prerequisite: 935 1. The controller MUST support automatic network discovery. 936 2. Tester should be able to retrieve the discovered topology 937 information either through controller's management interface or 938 northbound interface. 940 Procedure: 942 1. Establish the network connections between controller and network 943 nodes. 944 2. Query the controller for the discovered network topology 945 information and compare it with the deployed network topology 946 information. 947 3. 3a. Increase the number of nodes by 1 when the comparison is 948 successful and repeat the test. 949 4. 3b. Decrease the number of nodes by 1 when the comparison fails 950 and repeat the test. 951 5. Continue the test until the comparison of step 3b is successful. 952 6. Record the number of nodes for the last iteration (Ns) where the 953 topology comparison was successful. 955 Measurement: 957 Network Discovery Size = Ns. 959 Reporting Format: 961 The Network Discovery Size results MUST be reported in addition to 962 the configuration parameters captured in section 5. 964 5.2.3. 6.2.3 Forwarding Table Capacity 966 Objective: 968 Measure the maximum number of flow entries a controller can manage 969 in its Forwarding table. 971 Reference Test Setup: 973 The test SHOULD use one of the test setups described in section 3.1 974 or section 3.2 of this document. 976 Prerequisite: 978 1. The controller Forwarding table should be empty. 979 2. Flow Idle time MUST be set to higher or infinite value. 980 3. The controller MUST have completed network topology discovery. 981 4. Tester should be able to retrieve the forwarding table information 982 either through controller's management interface or northbound 983 interface. 985 Procedure: 987 Reactive Flow Provisioning Mode: 989 1. Send bi-directional traffic continuously with unique source and/or 990 destination addresses from test traffic generators TP1 and TP2 at 991 the asynchronous message processing rate of controller. 992 2. Query the controller at a regular interval (e.g., 5 seconds) for 993 the number of learnt flow entries from its northbound interface. 994 3. Stop the test when the retrieved value is constant for three 995 consecutive iterations and record the value received from the last 996 query (Nrp). 998 Proactive Flow Provisioning Mode: 1000 1. Install unique flows continuously through controller's northbound 1001 or management interface until a failure response is received from 1002 the controller. 1003 2. Record the total number of successful responses (Nrp). 1005 Note: 1007 Some controller designs for proactive flow provisioning mode may 1008 require the switch to send flow setup requests in order to generate 1009 flow setup responses. In such cases, it is recommended to generate 1010 bi-directional traffic for the provisioned flows. 1012 Measurement: 1014 Proactive Flow Provisioning Mode: 1016 Max Flow Entries = Total number of flows provisioned (Nrp) 1018 Reactive Flow Provisioning Mode: 1020 Max Flow Entries = Total number of learnt flow entries (Nrp) 1022 Forwarding Table Capacity = Max Flow Entries. 1024 Reporting Format: 1026 The Forwarding Table Capacity results MUST be tabulated with the 1027 following information in addition to the configuration parameters 1028 captured in section 5. 1030 - Provisioning Type (Proactive/Reactive) 1032 5.3. 6.3 Security 1034 5.3.1. 6.3.1 Exception Handling 1036 Objective: 1038 Determine the effect of handling error packets and notifications on 1039 performance tests. The impact MUST be measured for the following 1040 performance tests 1042 a. Path Provisioning Rate 1044 b. Path Provisioning Time 1046 c. Network Topology Change Detection Time 1048 Reference Test Setup: 1050 The test SHOULD use one of the test setups described in section 3.1 1051 or section 3.2 of this document. 1053 Prerequisite: 1055 1. This test MUST be performed after obtaining the baseline 1056 measurement results for the above performance tests. 1057 2. Ensure that the invalid messages are not dropped by the 1058 intermediate devices connecting the controller and Network 1059 Devices. 1061 Procedure: 1063 1. Perform the above listed performance tests and send 1% of messages 1064 from the Asynchronous Message Processing Rate as invalid messages 1065 from the connected Network Devices emulated at the forwarding 1066 plane test emulator. 1067 2. Perform the above listed performance tests and send 2% of messages 1068 from the Asynchronous Message Processing Rate as invalid messages 1069 from the connected Network Devices emulated at the forwarding 1070 plane test emulator. 1072 Note: 1074 Invalid messages can be frames with incorrect protocol fields or any 1075 form of failure notifications sent towards controller. 1077 Measurement: 1079 Measurement MUST be done as per the equation defined in the 1080 corresponding performance test measurement section. 1082 Reporting Format: 1084 The Exception Handling results MUST be reported in the format of 1085 table with a column for each of the below parameters and row for 1086 each of the listed performance tests. 1088 - Without Exceptions 1090 - With 1% Exceptions 1092 - With 2% Exceptions 1094 5.3.2. Denial of Service Handling 1096 Objective: 1098 Determine the effect of handling DoS attacks on performance and 1099 scalability tests the impact MUST be measured for the following 1100 tests: 1102 a. Path Provisioning Rate 1104 b. Path Provisioning Time 1106 c. Network Topology Change Detection Time 1108 d. Network Discovery Size 1110 Reference Test Setup: 1112 The test SHOULD use one of the test setups described in section 3.1 1113 or section 3.2 of this document. 1115 Prerequisite: 1117 This test MUST be performed after obtaining the baseline measurement 1118 results for the above tests. 1120 Procedure: 1122 1. Perform the listed tests and launch a DoS attack towards 1123 controller while the test is running. 1125 Note: 1127 DoS attacks can be launched on one of the following interfaces. 1129 a. Northbound (e.g., Sending a huge number of requests on 1130 northbound interface) 1131 b. Management (e.g., Ping requests to controller's management 1132 interface) 1133 c. Southbound (e.g., TCP SYNC messages on southbound interface) 1135 Measurement: 1137 Measurement MUST be done as per the equation defined in the 1138 corresponding test's measurement section. 1140 Reporting Format: 1142 The DoS Attacks Handling results MUST be reported in the format of 1143 table with a column for each of the below parameters and row for 1144 each of the listed tests. 1146 - Without any attacks 1148 - With attacks 1150 The report should also specify the nature of attack and the 1151 interface. 1153 5.4. Reliability 1155 5.4.1. Controller Failover Time 1157 Objective: 1159 The time taken to switch from an active controller to the backup 1160 controller, when the controllers work in redundancy mode and the 1161 active controller fails, defined as the interval starting with the 1162 active controller bringing down, ending with the first re-discovery 1163 message received from the new controller at its Southbound 1164 interface. 1166 Reference Test Setup: 1168 The test SHOULD use the test setup described in section 3.2 of this 1169 document. 1171 Prerequisite: 1173 1. Master controller election MUST be completed. 1174 2. Nodes are connected to the controller cluster as per the 1175 Redundancy Mode (RM). 1176 3. The controller cluster should have completed the network topology 1177 discovery. 1178 4. The Network Device MUST send all new flows to the controller when 1179 it receives from the test traffic generator. 1180 5. Controller should have learnt the location of destination (D1) at 1181 test traffic generator TP2. 1183 Procedure: 1185 1. Send uni-directional traffic continuously with incremental 1186 sequence number and source addresses from test traffic generator 1187 TP1 at the rate that the controller processes without any drops. 1188 2. Ensure that there are no packet drops observed at the test traffic 1189 generator TP2. 1190 3. Bring down the active controller. 1191 4. Stop the test when a first frame received on TP2 after failover 1192 operation. 1193 5. Record the time at which the last valid frame received (T1) at 1194 test traffic generator TP2 before sequence error and the first 1195 valid frame received (T2) after the sequence error at TP2 1197 Measurement: 1199 Controller Failover Time = (T2 - T1) 1201 Packet Loss = Number of missing packet sequences. 1203 Reporting Format: 1205 The Controller Failover Time results MUST be tabulated with the 1206 following information. 1208 - Number of cluster nodes 1209 - Redundancy mode 1211 - Controller Failover 1213 - Time Packet Loss 1215 - Cluster keep-alive interval 1217 5.4.2. Network Re-Provisioning Time 1219 Objective: 1221 The time taken to re-route the traffic by the Controller, when there 1222 is a failure in existing traffic paths, defined as the interval 1223 starting from the first failure notification message received by the 1224 controller, ending with the last flow re-provisioning message sent 1225 by the controller at its Southbound interface. 1227 Reference Test Setup: 1229 This test SHOULD use one of the test setup described in section 3.1 1230 or section 3.2 of this document. 1232 Prerequisite: 1233 1. Network with the given number of nodes and redundant paths MUST be 1234 deployed. 1235 2. Ensure that the controller MUST have knowledge about the location 1236 of test traffic generators TP1 and TP2. 1237 3. Ensure that the controller does not pre-provision the alternate 1238 path in the emulated Network Devices at the forwarding plane test 1239 emulator. 1241 Procedure: 1243 1. Send bi-directional traffic continuously with unique sequence 1244 number from TP1 and TP2. 1245 2. Bring down a link or switch in the traffic path. 1246 3. Stop the test after receiving first frame after network re- 1247 convergence. 1248 4. Record the time of last received frame prior to the frame loss at 1249 TP2 (TP2-Tlfr) and the time of first frame received after the 1250 frame loss at TP2 (TP2-Tffr). 1252 5. Record the time of last received frame prior to the frame loss at 1253 TP1 (TP1-Tlfr) and the time of first frame received after the 1254 frame loss at TP1 (TP1-Tffr). 1256 Measurement: 1258 Forward Direction Path Re-Provisioning Time (FDRT) 1259 = (TP2-Tffr - TP2-Tlfr) 1261 Reverse Direction Path Re-Provisioning Time (RDRT) 1262 = (TP1-Tffr - TP1-Tlfr) 1264 Network Re-Provisioning Time = (FDRT+RDRT)/2 1266 Forward Direction Packet Loss = Number of missing sequence frames 1267 at TP1 1269 Reverse Direction Packet Loss = Number of missing sequence frames 1270 at TP2 1272 Reporting Format: 1274 The Network Re-Provisioning Time results MUST be tabulated with the 1275 following information. 1277 - Number of nodes in the primary path 1279 - Number of nodes in the alternate path 1281 - Network Re-Provisioning Time 1283 - Forward Direction Packet Loss 1285 - Reverse Direction Packet Loss 1287 6. References 1289 6.1. Normative References 1291 [RFC2544] S. Bradner, J. McQuaid, "Benchmarking Methodology for 1292 Network Interconnect Devices",RFC 2544, March 1999. 1294 [RFC2330] V. Paxson, G. Almes, J. Mahdavi, M. Mathis, 1295 "Framework for IP Performance Metrics",RFC 2330, 1296 May 1998. 1298 [RFC6241] R. Enns, M. Bjorklund, J. Schoenwaelder, A. Bierman, 1299 "Network Configuration Protocol (NETCONF)",RFC 6241, 1300 July 2011. 1302 [RFC6020] M. Bjorklund, "YANG - A Data Modeling Language for 1303 the Network Configuration Protocol (NETCONF)", RFC 6020, 1304 October 2010 1306 [RFC5440] JP. Vasseur, JL. Le Roux, "Path Computation Element (PCE) 1307 Communication Protocol (PCEP)", RFC 5440, March 2009. 1309 [OpenFlow Switch Specification] ONF,"OpenFlow Switch Specification" 1310 Version 1.4.0 (Wire Protocol 0x05), October 14, 2013. 1312 [I-D.sdn-controller-benchmark-term] Bhuvaneswaran.V, Anton Basil, 1313 Mark.T, Vishwas Manral, Sarah Banks, "Terminology for 1314 Benchmarking SDN Controller Performance", 1315 draft-ietf-bmwg-sdn-controller-benchmark-term-02 1316 (Work in progress), July 8, 2016 1318 6.2. Informative References 1320 [I-D.i2rs-architecture] A. Atlas, J. Halpern, S. Hares, D. Ward, 1321 T. Nadeau, "An Architecture for the Interface to the 1322 Routing System", draft-ietf-i2rs-architecture-09 1323 (Work in progress), March 6, 2015 1325 [OpenContrail] Ankur Singla, Bruno Rijsman, "OpenContrail 1326 Architecture Documentation", 1327 http://opencontrail.org/opencontrail-architecture-documentation 1329 [OpenDaylight] OpenDaylight Controller:Architectural Framework, 1330 https://wiki.opendaylight.org/view/OpenDaylight_Controller 1332 7. IANA Considerations 1334 This document does not have any IANA requests. 1336 8. Security Considerations 1338 Benchmarking tests described in this document are limited to the 1339 performance characterization of controller in lab environment with 1340 isolated network. 1342 9. Acknowledgments 1344 The authors would like to thank the following individuals for 1345 providing their valuable comments to the earlier versions of this 1346 document: Al Morton (AT&T), Sandeep Gangadharan (HP), M. Georgescu 1347 (NAIST), Andrew McGregor (Google), Scott Bradner (Harvard 1348 University), Jay Karthik (Cisco), Ramakrishnan (Dell), Khasanov 1349 Boris (Huawei), Brian Castelli (Spirent) 1351 This document was prepared using 2-Word-v2.0.template.dot. 1353 Appendix A. Example Test Topologies 1355 A.1. Leaf-Spine Topology - Three Tier Network Architecture 1357 +----------+ 1358 | SDN | 1359 | Node | (Core) 1360 +----------+ 1361 / \ 1362 / \ 1363 +------+ +------+ 1364 | SDN | | SDN | (Spine) 1365 | Node |.. | Node | 1366 +------+ +------+ 1367 / \ / \ 1368 / \ / \ 1369 l1 / / \ ln-1 1370 / / \ \ 1371 +--------+ +-------+ 1372 | SDN | | SDN | 1373 | Node |.. | Node | (Leaf) 1374 +--------+ +-------+ 1376 A.2. Leaf-Spine Topology - Two Tier Network Architecture 1378 +------+ +------+ 1379 | SDN | | SDN | (Spine) 1380 | Node |.. | Node | 1381 +------+ +------+ 1382 / \ / \ 1383 / \ / \ 1384 l1 / / \ ln-1 1385 / / \ \ 1386 +--------+ +-------+ 1387 | SDN | | SDN | 1388 | Node |.. | Node | (Leaf) 1389 +--------+ +-------+ 1391 Appendix B. Benchmarking Methodology using OpenFlow Controllers 1393 This section gives an overview of OpenFlow protocol and provides 1394 test methodology to benchmark SDN controllers supporting OpenFlow 1395 southbound protocol. 1397 B.1. Protocol Overview 1399 OpenFlow is an open standard protocol defined by Open Networking 1400 Foundation (ONF), used for programming the forwarding plane of 1401 network switches or routers via a centralized controller. 1403 B.2. Messages Overview 1405 OpenFlow protocol supports three messages types namely controller- 1406 to-switch, asynchronous and symmetric. 1408 Controller-to-switch messages are initiated by the controller and 1409 used to directly manage or inspect the state of the switch. These 1410 messages allow controllers to query/configure the switch (Features, 1411 Configuration messages), collect information from switch (Read-State 1412 message), send packets on specified port of switch (Packet-out 1413 message), and modify switch forwarding plane and state (Modify- 1414 State, Role-Request messages etc.). 1416 Asynchronous messages are generated by the switch without a 1417 controller soliciting them. These messages allow switches to update 1418 controllers to denote an arrival of new flow (Packet-in), switch 1419 state change (Flow-Removed, Port-status) and error (Error). 1421 Symmetric messages are generated in either direction without 1422 solicitation. These messages allow switches and controllers to set 1423 up connection (Hello), verify for liveness (Echo) and offer 1424 additional functionalities (Experimenter). 1426 B.3. Connection Overview 1428 OpenFlow channel is used to exchange OpenFlow message between an 1429 OpenFlow switch and an OpenFlow controller. The OpenFlow channel 1430 connection can be setup using plain TCP or TLS. By default, a switch 1431 establishes single connection with SDN controller. A switch may 1432 establish multiple parallel connections to single controller 1433 (auxiliary connection) or multiple controllers to handle controller 1434 failures and load balancing. 1436 B.4. Performance Benchmarking Tests 1438 B.4.1. Network Topology Discovery Time 1440 Procedure: 1442 Network Devices OpenFlow SDN 1443 Controller Application 1444 | | | 1445 | | | 1447 | | | 1448 | | | 1450 | | | 1451 | OFPT_HELLO Exchange | | 1452 |<-------------------------->| | 1453 | | | 1454 | PACKET_OUT with LLDP | | 1455 | to all switches | | 1456 (Tm1)|<---------------------------| | 1457 | | | 1458 | PACKET_IN with LLDP| | 1459 | rcvd from switch-1| | 1460 |--------------------------->| | 1461 | | | 1462 | PACKET_IN with LLDP| | 1463 | rcvd from switch-2| | 1464 |--------------------------->| | 1465 | . | | 1466 | . | | 1467 | | | 1468 | PACKET_IN with LLDP| | 1469 | rcvd from switch-n| | 1470 (Tmn)|--------------------------->| | 1471 | | | 1472 | | | 1474 | | | 1475 | | Query the controller for| 1476 | | discovered n/w topo.(Di)| 1477 | |<--------------------------| 1478 | | | 1479 | | | 1481 | | | 1483 Legend: 1485 NB: Northbound 1486 SB: Southbound 1487 OF: OpenFlow 1488 Tm1: Time of reception of first LLDP message from controller 1489 Tmn: Time of last LLDP message sent to controller 1491 Discussion: 1493 The Network Topology Discovery Time can be obtained by calculating 1494 the time difference between the first PACKET_OUT with LLDP message 1495 received from the controller (Tm1) and the last PACKET_IN with LLDP 1496 message sent to the controller (Tmn) when the comparison is 1497 successful. 1499 B.4.2. Asynchronous Message Processing Time 1501 Procedure: 1503 Network Devices OpenFlow SDN 1504 Controller Application 1505 | | | 1506 |PACKET_IN with single | | 1507 |OFP match header | | 1508 (T0)|--------------------------->| | 1509 | | | 1510 | PACKET_OUT with single OFP | | 1511 | action header | | 1512 (R0)|<---------------------------| | 1513 | . | | 1514 | . | | 1515 | . | | 1516 | | | 1517 |PACKET_IN with single OFP | | 1518 |match header | | 1519 (Tn)|--------------------------->| | 1520 | | | 1521 | PACKET_OUT with single OFP | | 1522 | action header| | 1523 (Rn)|<---------------------------| | 1524 | | | 1525 | | | 1527 | | | 1528 | | | 1531 | | | 1533 Legend: 1535 T0,T1, ..Tn are PACKET_IN messages transmit timestamps. 1536 R0,R1, ..Rn are PACKET_OUT messages receive timestamps. 1537 Nrx : Number of successful PACKET_IN/PACKET_OUT message 1538 exchanges 1540 Discussion: 1542 The Asynchronous Message Processing Time will be obtained by sum of 1543 ((R0-T0),(R1-T1)..(Rn - Tn))/ Nrx. 1545 B.4.3. Asynchronous Message Processing Rate 1547 Procedure: 1549 Network Devices OpenFlow SDN 1550 Controller Application 1551 | | | 1552 |PACKET_IN with multiple OFP | | 1553 |match headers | | 1554 |--------------------------->| | 1555 | | | 1556 | PACKET_OUT with multiple | | 1557 | OFP action headers| | 1558 |<---------------------------| | 1559 | | | 1560 |PACKET_IN with multiple OFP | | 1561 |match headers | | 1562 |--------------------------->| | 1563 | | | 1564 | PACKET_OUT with multiple | | 1565 | OFP action headers| | 1566 |<---------------------------| | 1567 | . | | 1568 | . | | 1569 | . | | 1570 | | | 1571 |PACKET_IN with multiple OFP | | 1572 |match headers | | 1573 |--------------------------->| | 1574 | | | 1575 | PACKET_OUT with multiple | | 1576 | OFP action headers| | 1577 |<---------------------------| | 1578 | | | 1579 | | | 1581 | | | 1582 | | | 1584 | | | 1586 Discussion: 1588 The Asynchronous Message Processing Rate will be obtained by 1589 calculating the number of OFP action headers received in all 1590 PACKET_OUT messages during the test duration. 1592 B.4.4. Reactive Path Provisioning Time 1594 Procedure: 1596 Test Traffic Test Traffic Network Devices OpenFlow 1597 Generator TP1 Generator TP2 Controller 1598 | | | | 1599 | |G-ARP (D1) | | 1600 | |--------------------->| | 1601 | | | | 1602 | | |PACKET_IN(D1) | 1603 | | |------------------>| 1604 | | | | 1605 |Traffic (S1,D1) | | 1606 (Tsf1)|----------------------------------->| | 1607 | | | | 1608 | | | | 1609 | | | | 1610 | | |PACKET_IN(S1,D1) | 1611 | | |------------------>| 1612 | | | | 1613 | | | FLOW_MOD(D1) | 1614 | | |<------------------| 1615 | | | | 1616 | |Traffic (S1,D1) | | 1617 | (Tdf1)|<---------------------| | 1618 | | | | 1620 Legend: 1622 G-ARP: Gratuitous ARP message. 1623 Tsf1: Time of first frame sent from TP1 1624 Tdf1: Time of first frame received from TP2 1626 Discussion: 1628 The Reactive Path Provisioning Time can be obtained by finding the 1629 time difference between the transmit and receive time of the traffic 1630 (Tsf1-Tdf1). 1632 B.4.5. Proactive Path Provisioning Time 1634 Procedure: 1636 Test Traffic Test Traffic Network Devices OpenFlow SDN 1637 Generator TP1 Generator TP2 Controller Application 1638 | | | | | 1639 | |G-ARP (D1) | | | 1640 | |-------------->| | | 1641 | | | | | 1642 | | |PACKET_IN(D1) | | 1643 | | |--------------->| | 1644 | | | | | 1645 |Traffic (S1,D1) | | | 1646 Tsf1)|---------------------------->| | | 1647 | | | | | 1648 | | | | | 1650 | | | | | 1651 | | | FLOW_MOD(D1) | | 1652 | | |<---------------| | 1653 | | | | | 1654 | |Traffic (S1,D1)| | | 1655 | (Tdf1)|<--------------| | | 1656 | | | | | 1658 Legend: 1660 G-ARP: Gratuitous ARP message. 1661 Tsf1: Time of first frame sent from TP1 1662 Tdf1: Time of first frame received from TP2 1664 Discussion: 1666 The Proactive Path Provisioning Time can be obtained by finding the 1667 time difference between the transmit and receive time of the traffic 1668 (Tsf1-Tdf1). 1670 B.4.6. Reactive Path Provisioning Rate 1672 Procedure: 1674 Test Traffic Test Traffic Network Devices OpenFlow 1675 Generator TP1 Generator TP2 Controller 1676 | | | | 1677 | | | | 1678 | | | | 1679 | |G-ARP (D1..Dn) | | 1680 | |--------------------| | 1681 | | | | 1682 | | |PACKET_IN(D1..Dn) | 1683 | | |--------------------->| 1684 | | | | 1685 |Traffic (S1..Sn,D1..Dn) | | 1686 |--------------------------------->| | 1687 | | | | 1688 | | |PACKET_IN(S1.Sn,D1.Dn)| 1689 | | |--------------------->| 1690 | | | | 1691 | | | FLOW_MOD(S1) | 1692 | | |<---------------------| 1693 | | | | 1694 | | | FLOW_MOD(D1) | 1695 | | |<---------------------| 1696 | | | | 1697 | | | FLOW_MOD(S2) | 1698 | | |<---------------------| 1699 | | | | 1700 | | | FLOW_MOD(D2) | 1701 | | |<---------------------| 1702 | | | . | 1703 | | | . | 1704 | | | | 1705 | | | FLOW_MOD(Sn) | 1706 | | |<---------------------| 1707 | | | | 1708 | | | FLOW_MOD(Dn) | 1709 | | |<---------------------| 1710 | | | | 1711 | | Traffic (S1..Sn, | | 1712 | | D1..Dn)| | 1713 | |<-------------------| | 1714 | | | | 1715 | | | | 1717 Legend: 1719 G-ARP: Gratuitous ARP 1720 D1..Dn: Destination Endpoint 1, Destination Endpoint 2 .... 1721 Destination Endpoint n 1722 S1..Sn: Source Endpoint 1, Source Endpoint 2 .., Source 1723 Endpoint n 1725 Discussion: 1727 The Reactive Path Provisioning Rate can be obtained by finding the 1728 total number of frames received at TP2 after the test duration. 1730 B.4.7. Proactive Path Provisioning Rate 1732 Procedure: 1734 Test Traffic Test Traffic Network Devices OpenFlow SDN 1735 Generator TP1 Generator TP2 Controller Application 1736 | | | | | 1737 | |G-ARP (D1..Dn) | | | 1738 | |-------------->| | | 1739 | | | | | 1740 | | |PACKET_IN(D1.Dn)| | 1741 | | |--------------->| | 1742 | | | | | 1743 |Traffic (S1..Sn,D1..Dn) | | | 1744 Tsf1)|---------------------------->| | | 1745 | | | | | 1746 | | | | | 1748 | | | | | 1749 | | | | . | 1750 | | | | | 1752 | | | | | 1753 | | | FLOW_MOD(S1) | | 1754 | | |<---------------| | 1755 | | | | | 1756 | | | FLOW_MOD(D1) | | 1757 | | |<---------------| | 1758 | | | | | 1759 | | | . | | 1760 | | | FLOW_MOD(Sn) | | 1761 | | |<---------------| | 1762 | | | | | 1763 | | | FLOW_MOD(Dn) | | 1764 | | |<---------------| | 1765 | | | | | 1766 | |Traffic (S1.Sn,| | | 1767 | | D1.Dn)| | | 1768 | (Tdf1)|<--------------| | | 1769 | | | | | 1771 Legend: 1773 G-ARP: Gratuitous ARP 1774 D1..Dn: Destination Endpoint 1, Destination Endpoint 2 .... 1775 Destination Endpoint n 1776 S1..Sn: Source Endpoint 1, Source Endpoint 2 .., Source 1777 Endpoint n 1779 Discussion: 1781 The Proactive Path Provisioning Rate can be obtained by finding the 1782 total number of frames received at TP2 after the test duration 1784 B.4.8. Network Topology Change Detection Time 1786 Procedure: 1788 Network Devices OpenFlow SDN 1789 Controller Application 1790 | | | 1791 | | | 1793 | | | 1794 T0 |PORT_STATUS with link down | | 1795 | from S1 | | 1796 |--------------------------->| | 1797 | | | 1798 |First PACKET_OUT with LLDP | | 1799 |to OF Switch | | 1800 T1 |<---------------------------| | 1801 | | | 1802 | | | 1805 Discussion: 1807 The Network Topology Change Detection Time can be obtained by 1808 finding the difference between the time the OpenFlow switch S1 sends 1809 the PORT_STATUS message (T0) and the time that the OpenFlow 1810 controller sends the first topology re-discovery message (T1) to 1811 OpenFlow switches. 1813 B.5. Scalability 1815 B.5.1. Control Sessions Capacity 1817 Procedure: 1819 Network Devices OpenFlow 1820 Controller 1821 | | 1822 | OFPT_HELLO Exchange for Switch 1 | 1823 |<------------------------------------->| 1824 | | 1825 | OFPT_HELLO Exchange for Switch 2 | 1826 |<------------------------------------->| 1827 | . | 1828 | . | 1829 | . | 1830 | OFPT_HELLO Exchange for Switch n | 1831 |X<----------------------------------->X| 1832 | | 1834 Discussion: 1836 The value of Switch n-1 will provide Control Sessions Capacity. 1838 B.5.2. Network Discovery Size 1840 Procedure: 1842 Network Devices OpenFlow SDN 1843 Controller Application 1844 | | | 1845 | | | 1847 | | | 1848 | OFPT_HELLO Exchange | | 1849 |<-------------------------->| | 1850 | | | 1851 | PACKET_OUT with LLDP | | 1852 | to all switches | | 1853 |<---------------------------| | 1854 | | | 1855 | PACKET_IN with LLDP| | 1856 | rcvd from switch-1| | 1857 |--------------------------->| | 1858 | | | 1859 | PACKET_IN with LLDP| | 1860 | rcvd from switch-2| | 1861 |--------------------------->| | 1862 | . | | 1863 | . | | 1864 | | | 1865 | PACKET_IN with LLDP| | 1866 | rcvd from switch-n| | 1867 |--------------------------->| | 1868 | | | 1869 | | | 1871 | | | 1872 | | Query the controller for| 1873 | | discovered n/w topo.(N1)| 1874 | |<--------------------------| 1875 | | | 1876 | | | 1878 | | | 1879 | | | 1882 | | | 1884 Legend: 1886 n/w topo: Network Topology 1887 OF: OpenFlow 1889 Discussion: 1891 The value of N1 provides the Network Discovery Size value. The test 1892 duration can be set to the stipulated time within which the user 1893 expects the controller to complete the discovery process. 1895 B.5.3. Forwarding Table Capacity 1896 Procedure: 1898 Test Traffic Network Devices OpenFlow SDN 1899 Generator TP1 Controller Application 1900 | | | | 1901 | | | | 1902 |G-ARP (H1..Hn) | | | 1903 |----------------->| | | 1904 | | | | 1905 | |PACKET_IN(D1..Dn) | | 1906 | |------------------>| | 1907 | | | | 1908 | | || 1909 | | | | 1910 | | | |(F1) 1912 | | | | 1913 | | || 1914 | | | | 1915 | | | |(F2) 1917 | | | | 1918 | | || 1919 | | | | 1920 | | | |(F3) 1922 | | | | 1923 | | | | 1925 | | | | 1927 Legend: 1929 G-ARP: Gratuitous ARP 1930 H1..Hn: Host 1 .. Host n 1931 FWD: Forwarding Table 1933 Discussion: 1935 Query the controller forwarding table entries for multiple times 1936 until the three consecutive queries return the same value. The last 1937 value retrieved from the controller will provide the Forwarding 1938 Table Capacity value. The query interval is user configurable. The 5 1939 seconds shown in this example is for representational purpose. 1941 B.6. Security 1943 B.6.1. Exception Handling 1945 Procedure: 1947 Test Traffic Test Traffic Network Devices OpenFlow SDN 1948 Generator TP1 Generator TP2 Controller Application 1949 | | | | | 1950 | |G-ARP (D1..Dn) | | | 1951 | |------------------>| | | 1952 | | | | | 1953 | | |PACKET_IN(D1..Dn)| | 1954 | | |---------------->| | 1955 | | | | | 1956 |Traffic (S1..Sn,D1..Dn) | | | 1957 |----------------------------->| | | 1958 | | | | | 1959 | | |PACKET_IN(S1..Sa,| | 1960 | | | D1..Da)| | 1961 | | |---------------->| | 1962 | | | | | 1963 | | |PACKET_IN(Sa+1.. | | 1964 | | |.Sn,Da+1..Dn) | | 1965 | | |(1% incorrect OFP| | 1966 | | | Match header)| | 1967 | | |---------------->| | 1968 | | | | | 1969 | | | FLOW_MOD(D1..Dn)| | 1970 | | |<----------------| | 1971 | | | | | 1972 | | | FLOW_MOD(S1..Sa)| | 1973 | | | OFP headers| | 1974 | | |<----------------| | 1975 | | | | | 1976 | |Traffic (S1..Sa, | | | 1977 | | D1..Da)| | | 1978 | |<------------------| | | 1979 | | | | | 1980 | | | | | 1983 | | | | | 1984 | | | | | 1987 | | | | | 1988 | | | | | 1992 | | | | | 1993 | | | | | 1996 | | | | | 1998 Legend: 2000 G-ARP: Gratuitous ARP 2001 PACKET_IN(Sa+1..Sn,Da+1..Dn): OpenFlow PACKET_IN with wrong 2002 version number 2003 Rn1: Total number of frames received at Test Port 2 with 2004 1% incorrect frames 2005 Rn2: Total number of frames received at Test Port 2 with 2006 2% incorrect frames 2008 Discussion: 2010 The traffic rate sent towards OpenFlow switch from Test Port 1 2011 should be 1% higher than the Path Programming Rate. Rn1 will provide 2012 the Path Provisioning Rate of controller at 1% of incorrect frames 2013 handling and Rn2 will provide the Path Provisioning Rate of 2014 controller at 2% of incorrect frames handling. 2016 The procedure defined above provides test steps to determine the 2017 effect of handling error packets on Path Programming Rate. Same 2018 procedure can be adopted to determine the effects on other 2019 performance tests listed in this benchmarking tests. 2021 B.6.2. Denial of Service Handling 2023 Procedure: 2025 Test Traffic Test Traffic Network Devic OpenFlow SDN 2026 Generator TP1 Generator TP2 Controller Application 2027 | | | | | 2028 | |G-ARP (D1..Dn) | | | 2029 | |------------------>| | | 2030 | | | | | 2031 | | |PACKET_IN(D1..Dn)| | 2032 | | |---------------->| | 2033 | | | | | 2034 |Traffic (S1..Sn,D1..Dn) | | | 2035 |----------------------------->| | | 2036 | | | | | 2037 | | |PACKET_IN(S1..Sn,| | 2038 | | | D1..Dn)| | 2039 | | |---------------->| | 2040 | | | | | 2041 | | |TCP SYN Attack | | 2042 | | |from a switch | | 2043 | | |---------------->| | 2044 | | | | | 2045 | | |FLOW_MOD(D1..Dn) | | 2046 | | |<----------------| | 2047 | | | | | 2048 | | | FLOW_MOD(S1..Sn)| | 2049 | | | OFP headers| | 2050 | | |<----------------| | 2051 | | | | | 2052 | |Traffic (S1..Sn, | | | 2053 | | D1..Dn)| | | 2054 | |<------------------| | | 2055 | | | | | 2056 | | | | | 2059 | | | | | 2060 | | | | | 2063 | | | | | 2065 Legend: 2067 G-ARP: Gratuitous ARP 2069 Discussion: 2071 TCP SYN attack should be launched from one of the emulated/simulated 2072 OpenFlow Switch. Rn1 provides the Path Programming Rate of 2073 controller uponhandling denial of service attack. 2075 The procedure defined above provides test steps to determine the 2076 effect of handling denial of service on Path Programming Rate. Same 2077 procedure can be adopted to determine the effects on other 2078 performance tests listed in this benchmarking tests. 2080 B.7. Reliability 2082 B.7.1. Controller Failover Time 2084 Procedure: 2086 Test Traffic Test Traffic Network Device OpenFlow SDN 2087 Generator TP1 Generator TP2 Controller Application 2088 | | | | | 2089 | |G-ARP (D1) | | | 2090 | |------------>| | | 2091 | | | | | 2092 | | |PACKET_IN(D1) | | 2093 | | |---------------->| | 2094 | | | | | 2095 |Traffic (S1..Sn,D1) | | | 2096 |-------------------------->| | | 2097 | | | | | 2098 | | | | | 2099 | | |PACKET_IN(S1,D1) | | 2100 | | |---------------->| | 2101 | | | | | 2102 | | |FLOW_MOD(D1) | | 2103 | | |<----------------| | 2104 | | |FLOW_MOD(S1) | | 2105 | | |<----------------| | 2106 | | | | | 2107 | |Traffic (S1,D1)| | | 2108 | |<------------| | | 2109 | | | | | 2110 | | |PACKET_IN(S2,D1) | | 2111 | | |---------------->| | 2112 | | | | | 2113 | | |FLOW_MOD(S2) | | 2114 | | |<----------------| | 2115 | | | | | 2116 | | |PACKET_IN(Sn-1,D1)| | 2117 | | |---------------->| | 2118 | | | | | 2119 | | |PACKET_IN(Sn,D1) | | 2120 | | |---------------->| | 2121 | | | . | | 2122 | | | . | | 2125 | | | FLOW_MOD(Sn-1) | | 2126 | | | <-X----------| | 2127 | | | | | 2128 | | |FLOW_MOD(Sn) | | 2129 | | |<----------------| | 2130 | | | | | 2131 | |Traffic (Sn,D1)| | | 2132 | |<------------| | | 2133 | | | | | 2134 | | | | | 2139 Legend: 2141 G-ARP: Gratuitous ARP. 2143 Discussion: 2145 The time difference between the last valid frame received before the 2146 traffic loss and the first frame received after the traffic loss 2147 will provide the controller failover time. 2149 If there is no frame loss during controller failover time, the 2150 controller failover time can be deemed negligible. 2152 B.7.2. Network Re-Provisioning Time 2154 Procedure: 2156 Test Traffic Test Traffic Network Devices OpenFlow SDN 2157 Generator TP1 Generator TP2 Controller Application 2158 | | | | | 2159 | |G-ARP (D1) | | | 2160 | |-------------->| | | 2161 | | | | | 2162 | | |PACKET_IN(D1) | | 2163 | | |---------------->| | 2164 | G-ARP (S1) | | | 2165 |---------------------------->| | | 2166 | | | | | 2167 | | |PACKET_IN(S1) | | 2168 | | |---------------->| | 2169 | | | | | 2170 |Traffic (S1,D1,Seq.no (1..n))| | | 2171 |---------------------------->| | | 2172 | | | | | 2173 | | |PACKET_IN(S1,D1) | | 2174 | | |---------------->| | 2175 | | | | | 2176 | |Traffic (D1,S1,| | | 2177 | | Seq.no (1..n))| | | 2178 | |-------------->| | | 2179 | | | | | 2180 | | |PACKET_IN(D1,S1) | | 2181 | | |---------------->| | 2182 | | | | | 2183 | | |FLOW_MOD(D1) | | 2184 | | |<----------------| | 2185 | | | | | 2186 | | |FLOW_MOD(S1) | | 2187 | | |<----------------| | 2188 | | | | | 2189 | |Traffic (S1,D1,| | | 2190 | | Seq.no(1))| | | 2191 | |<--------------| | | 2192 | | | | | 2193 | |Traffic (S1,D1,| | | 2194 | | Seq.no(2))| | | 2195 | |<--------------| | | 2196 | | | | | 2197 | | | | | 2198 | Traffic (D1,S1,Seq.no(1))| | | 2199 |<----------------------------| | | 2200 | | | | | 2201 | Traffic (D1,S1,Seq.no(2))| | | 2202 |<----------------------------| | | 2203 | | | | | 2204 | Traffic (D1,S1,Seq.no(x))| | | 2205 |<----------------------------| | | 2206 | | | | | 2207 | |Traffic (S1,D1,| | | 2208 | | Seq.no(x))| | | 2209 | |<--------------| | | 2210 | | | | | 2211 | | | | | 2212 | | | | | 2216 | | | | | 2217 | | |PORT_STATUS(Sa) | | 2218 | | |---------------->| | 2219 | | | | | 2220 | |Traffic (S1,D1,| | | 2221 | | Seq.no(n-1))| | | 2222 | | X<-----------| | | 2223 | | | | | 2224 | Traffic (D1,S1,Seq.no(n-1))| | | 2225 | X------------------------| | | 2226 | | | | | 2227 | | | | | 2228 | | |FLOW_MOD(D1) | | 2229 | | |<----------------| | 2230 | | | | | 2231 | | |FLOW_MOD(S1) | | 2232 | | |<----------------| | 2233 | | | | | 2234 | Traffic (D1,S1,Seq.no(n))| | | 2235 |<----------------------------| | | 2236 | | | | | 2237 | |Traffic (S1,D1,| | | 2238 | | Seq.no(n))| | | 2239 | |<--------------| | | 2240 | | | | | 2241 | | | | | 2246 Legend: 2248 G-ARP: Gratuitous ARP message. 2249 Seq.no: Sequence number. 2250 Sa: Neighbour switch of the switch that was brought down. 2252 Discussion: 2254 The time difference between the last valid frame received before the 2255 traffic loss (Packet number with sequence number x) and the first 2256 frame received after the traffic loss (packet with sequence number 2257 n) will provide the network path re-provisioning time. 2259 Note that the test is valid only when the controller provisions the 2260 alternate path upon network failure. 2262 Authors' Addresses 2264 Bhuvaneswaran Vengainathan 2265 Veryx Technologies Inc. 2266 1 International Plaza, Suite 550 2267 Philadelphia 2268 PA 19113 2270 Email: bhuvaneswaran.vengainathan@veryxtech.com 2272 Anton Basil 2273 Veryx Technologies Inc. 2274 1 International Plaza, Suite 550 2275 Philadelphia 2276 PA 19113 2278 Email: anton.basil@veryxtech.com 2280 Mark Tassinari 2281 Hewlett-Packard, 2282 8000 Foothills Blvd, 2283 Roseville, CA 95747 2285 Email: mark.tassinari@hpe.com 2287 Vishwas Manral 2288 Nano Sec, 2289 CA 2291 Email: vishwas.manral@gmail.com 2293 Sarah Banks 2294 VSS Monitoring 2295 930 De Guigne Drive, 2296 Sunnyvale, CA 2298 Email: sbanks@encrypted.net