idnits 2.17.1 draft-ietf-bmwg-sdn-controller-benchmark-term-02.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (July 8, 2016) is 2849 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Unused Reference: 'RFC7426' is defined on line 886, but no explicit reference was found in the text == Unused Reference: 'RFC4689' is defined on line 891, but no explicit reference was found in the text == Unused Reference: 'RFC2330' is defined on line 895, but no explicit reference was found in the text == Unused Reference: 'I-D.sdn-controller-benchmark-meth' is defined on line 902, but no explicit reference was found in the text == Unused Reference: 'OpenContrail' is defined on line 910, but no explicit reference was found in the text == Unused Reference: 'OpenDaylight' is defined on line 914, but no explicit reference was found in the text == Outdated reference: A later version (-09) exists of draft-ietf-bmwg-sdn-controller-benchmark-meth-02 Summary: 0 errors (**), 0 flaws (~~), 8 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 Internet-Draft Bhuvaneswaran Vengainathan 2 Network Working Group Anton Basil 3 Intended Status: Informational Veryx Technologies 4 Expires: January 8, 2017 Mark Tassinari 5 Hewlett-Packard 6 Vishwas Manral 7 Nano Sec 8 Sarah Banks 9 VSS Monitoring 10 July 8, 2016 12 Terminology for Benchmarking SDN Controller Performance 13 draft-ietf-bmwg-sdn-controller-benchmark-term-02 15 Abstract 17 This document defines terminology for benchmarking an SDN 18 controller's control plane performance. It extends the terminology 19 already defined in RFC 7426 for the purpose of benchmarking SDN 20 controllers. The terms provided in this document help to benchmark 21 SDN controller's performance independent of the controller's 22 supported protocols and/or network services. A mechanism for 23 benchmarking the performance of SDN controllers is defined in the 24 companion methodology document. These two documents provide a 25 standard mechanism to measure and evaluate the performance of 26 various controller implementations. 28 Status of this Memo 30 This Internet-Draft is submitted in full conformance with the 31 provisions of BCP 78 and BCP 79. 33 Internet-Drafts are working documents of the Internet Engineering 34 Task Force (IETF). Note that other groups may also distribute 35 working documents as Internet-Drafts. The list of current Internet- 36 Drafts is at http://datatracker.ietf.org/drafts/current. 38 Internet-Drafts are draft documents valid for a maximum of six 39 months and may be updated, replaced, or obsoleted by other documents 40 at any time. It is inappropriate to use Internet-Drafts as reference 41 material or to cite them other than as "work in progress. 43 This Internet-Draft will expire on January 8, 2017. 45 Copyright Notice 47 Copyright (c) 2016 IETF Trust and the persons identified as the 48 document authors. All rights reserved. 50 This document is subject to BCP 78 and the IETF Trust's Legal 51 Provisions Relating to IETF Documents 52 (http://trustee.ietf.org/license-info) in effect on the date of 53 publication of this document. Please review these documents 54 carefully, as they describe your rights and restrictions with 55 respect to this document. Code Components extracted from this 56 document must include Simplified BSD License text as described in 57 Section 4.e of the Trust Legal Provisions and are provided without 58 warranty as described in the Simplified BSD License. 60 Table of Contents 62 1. Introduction ................................................ 3 63 2. Term Definitions ............................................ 4 64 2.1. SDN Terms .............................................. 4 65 2.1.1. Flow .............................................. 4 66 2.1.2. Northbound Interface............................... 4 67 2.1.3. Controller Forwarding Table........................ 4 68 2.1.4. Proactive Flow Provisioning Mode................... 5 69 2.1.5. Reactive Flow Provisioning Mode.................... 5 70 2.1.6. Path .............................................. 6 71 2.1.7. Standalone Mode.................................... 6 72 2.1.8. Cluster/Redundancy Mode............................ 6 73 2.1.9. Asynchronous Message............................... 7 74 2.1.10. Test Traffic Generator............................ 7 75 2.2. Test Configuration/Setup Terms.......................... 8 76 2.2.1. Number of Network Devices.......................... 8 77 2.2.2. Test Iterations.................................... 8 78 2.2.3. Test Duration...................................... 8 79 2.2.4. Number of Cluster nodes............................ 9 80 2.3. Benchmarking Terms...................................... 9 81 2.3.1. Performance........................................ 9 82 2.3.1.1. Network Topology Discovery Time............... 9 83 2.3.1.2. Asynchronous Message Processing Time.......... 10 84 2.3.1.3. Asynchronous Message Processing Rate.......... 10 85 2.3.1.4. Reactive Path Provisioning Time .............. 11 86 2.3.1.5. Proactive Path Provisioning Time ............. 11 87 2.3.1.6. Reactive Path Provisioning Rate .............. 12 88 2.3.1.7. Proactive Path Provisioning Rate ............. 12 89 2.3.1.8. Network Topology Change Detection Time........ 13 90 2.3.2. Scalability ....................................... 14 91 2.3.2.1. Control Sessions Capacity .................... 14 92 2.3.2.2. Network Discovery Size ....................... 14 93 2.3.2.3. Forwarding Table Capacity .................... 15 94 2.3.3. Security ......................................... 15 95 2.3.3.1. Exception Handling ........................... 15 96 2.3.3.2. Denial of Service Handling ................... 16 97 2.3.4. Reliability ....................................... 16 98 2.3.4.1. Controller Failover Time ..................... 16 99 2.3.4.2. Network Re-Provisioning Time ................. 16 100 3. Test Setup ................................................. 17 101 3.1. Test setup - Controller working in Standalone Mode...... 18 102 3.2. Test setup - Controller working in Cluster Mode......... 19 103 4. Test Coverage .............................................. 20 104 5. References ................................................. 21 105 5.1. Normative References ................................... 21 106 5.2. Informative References ................................. 21 107 6. IANA Considerations ........................................ 21 108 7. Security Considerations ..................................... 21 109 8. Acknowledgements ........................................... 22 110 9. Authors' Addresses ......................................... 22 112 1. Introduction 114 Software Defined Networking (SDN) is a networking architecture in 115 which network control is decoupled from the underlying forwarding 116 function and is placed in a centralized location called the SDN 117 controller. The SDN controller abstracts the underlying network and 118 offers a global view of the overall network to applications and 119 business logic. Thus, an SDN controller provides the flexibility to 120 program, control, and manage network behaviour dynamically through 121 standard interfaces. Since the network controls are logically 122 centralized, the need to benchmark the SDN controller performance 123 becomes significant. This document defines terms to benchmark 124 various controller designs for performance, scalability, reliability 125 and security, independent of northbound and southbound protocols. 127 Conventions used in this document 129 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 130 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 131 document are to be interpreted as described in RFC 2119. 133 2. Term Definitions 135 2.1. SDN Terms 137 The terms defined in this section are extensions to the terms 138 defined in RFC 7426 ''Software-Defined Networking (SDN): Layers and 139 Architecture Terminology''. This RFC should be referred before 140 attempting to make use of this document. 142 2.1.1. Flow 144 Definition: 145 The definition of Flow is same as microflows defined in RFC 4689 146 Section 3.1.5. 148 Discussion: 149 A flow can be set of packets having same source address, destination 150 address, source port and destination port, or any of these 151 combinations. 153 Measurement Units: 154 N/A 156 See Also: 157 None 159 2.1.2. Northbound Interface 161 Definition: 162 The definition of northbound interface is same Service Interface 163 defined in RFC 7426. 165 Discussion: 166 The northbound interface allows SDN applications and orchestration 167 systems to program and retrieve the network information through the 168 SDN controller. 170 Measurement Units: 171 N/A 173 See Also: 174 None 176 2.1.3. Controller Forwarding Table 178 Definition: 180 A controller forwarding table contains flow entries learned in one 181 of two ways: first, entries could be learned from traffic received 182 through the data plane, or, second, these entries could be 183 statically provisioned on the controller, and distributed to devices 184 via the southbound interface. 186 Discussion: 187 The controller forwarding table has an aging mechanism which will be 188 applied only for dynamically learnt entries. 190 Measurement Units: 191 N/A 193 See Also: 194 None 196 2.1.4. Proactive Flow Provisioning Mode 198 Definition: 199 Controller programming flows in Network Devices based on the flow 200 entries provisioned through controller's northbound interface. 202 Discussion: 203 Orchestration systems and SDN applications can define the network 204 forwarding behaviour by programming the controller using proactive 205 flow provisioning. The controller can then program the Network 206 Devices with the pre-provisioned entries. 208 Measurement Units: 209 N/A 211 See Also: 212 None 214 2.1.5. Reactive Flow Provisioning Mode 216 Definition: 217 Controller programming flows in Network Devices based on the traffic 218 received from Network Devices through controller's southbound 219 interface 221 Discussion: 222 The SDN controller dynamically decides the forwarding behaviour 223 based on the incoming traffic from the Network Devices. The 224 controller then programs the Network Devices using Reactive Flow 225 Provisioning. 227 Measurement Units: 228 N/A 230 See Also: 231 None 233 2.1.6. Path 235 Definition: 236 Refer to Section 5 in RFC 2330. 238 Discussion: 239 None 241 Measurement Units: 242 N/A 244 See Also: 245 None 247 2.1.7. Standalone Mode 249 Definition: 250 Single controller handling all control plane functionalities without 251 redundancy, or the ability to provide high availability and/or 252 automatic failover. 254 Discussion: 255 In standalone mode, one controller manages one or more network 256 domains. 258 Measurement Units: 259 N/A 261 See Also: 262 None 264 2.1.8. Cluster/Redundancy Mode 266 Definition: 267 A group of 2 or more controllers handling all control plane 268 functionalities. 270 Discussion: 272 In cluster mode, multiple controllers are teamed together for the 273 purpose of load sharing and/or high availability. The controllers in 274 the group may work in active/standby (master/slave) or active/active 275 (equal) mode depending on the intended purpose. 277 Measurement Units: 278 N/A 280 See Also: 281 None 283 2.1.9. Asynchronous Message 285 Definition: 286 Any message from the Network Device that is generated for network 287 events. 289 Discussion: 290 Control messages like flow setup request and response message is 291 classified as asynchronous message. The controller has to return a 292 response message. Note that the Network Device will not be in 293 blocking mode and continues to send/receive other control messages 295 Measurement Units: 296 N/A 298 See Also: 299 None 301 2.1.10. Test Traffic Generator 303 Definition: 304 Test Traffic Generator is an entity that generates/receives network 305 traffic. 307 Discussion: 308 Test Traffic Generator can be an entity that interfaces with Network 309 Devices to send/receive real-time network traffic. 311 Measurement Units: 312 N/A 314 See Also: 315 None 317 2.2. Test Configuration/Setup Terms 319 2.2.1. Number of Network Devices 321 Definition: 322 The number of Network Devices present in the defined test topology. 324 Discussion: 325 The Network Devices defined in the test topology can be deployed 326 using real hardware or emulated in hardware platforms. 328 Measurement Units: 329 N/A 331 See Also: 332 None 334 2.2.2. Test Iterations 336 Definition: 337 The number of times the test needs to be repeated. 339 Discussion: 340 The test needs to be repeated for multiple iterations to obtain a 341 reliable metric. It is recommended that this test SHOULD be 342 performed for at least 10 iterations to increase the confidence in 343 measured result. 345 Measurement Units: 346 N/A 348 See Also: 349 None 351 2.2.3. Test Duration 353 Definition: 354 Defines the duration of test trails for each iteration. 356 Discussion: 357 Test duration forms the basis for stop criteria for benchmarking 358 tests. Test not completed within this time interval is considered as 359 incomplete. 361 Measurement Units: 362 seconds 364 See Also: 365 None 367 2.2.4. Number of Cluster nodes 369 Definition: 370 Defines the number of controllers present in the controller cluster. 372 Discussion: 373 This parameter is relevant when testing the controller performance 374 in clustering/teaming mode. The number of nodes in the cluster MUST 375 be greater than 1. 377 Measurement Units: 378 N/A 380 See Also: 381 None 383 2.3. Benchmarking Terms 385 This section defines metrics for benchmarking the SDN controller. 386 The procedure to perform the defined metrics is defined in the 387 accompanying methodology document. 389 2.3.1. Performance 391 2.3.1.1. Network Topology Discovery Time 393 Definition: 394 The time taken by controller(s) to determine the complete network 395 topology, defined as the interval starting with the first discovery 396 message from the controller(s) at its Southbound interface, ending 397 with all features of the static topology determined. 399 Discussion: 400 Network topology discovery is key for the SDN controller to 401 provision and manage the network. So it is important to measure how 402 quickly the controller discovers the topology to learn the current 403 network state. This benchmark is obtained by presenting a network 404 topology (Tree, Mesh or Linear) with the given number of nodes to 405 the controller and wait for the discovery process to complete .It is 406 expected that the controller supports network discovery mechanism 407 and uses protocol messages for its discovery process. 409 Measurement Units: 410 milliseconds 412 See Also: 413 None 415 2.3.1.2. Asynchronous Message Processing Time 417 Definition: 418 The time taken by controller(s) to process an asynchronous message, 419 defined as the interval starting with an asynchronous message from a 420 network device after the discovery of all the devices by the 421 controller(s), ending with a response message from the controller(s) 422 at its Southbound interface. 424 Discussion: 425 For SDN to support dynamic network provisioning, it is important to 426 measure how quickly the controller responds to an event triggered 427 from the network. The event could be any notification messages 428 generated by an Network Device upon arrival of a new flow, link down 429 etc. This benchmark is obtained by sending asynchronous messages 430 from every connected Network Devices one at a time for the defined 431 test duration. This test assumes that the controller will respond to 432 the received asynchronous message. 434 Measurement Units: 435 milliseconds 437 See Also: 438 None 440 2.3.1.3. Asynchronous Message Processing Rate 442 Definition: 443 The maximum number of asynchronous messages that the controller(s) 444 can process, defined as the number of asynchronous messages the 445 controller(s) can process at its Southbound interface between the 446 start of the test and the expiry of given test duration.. 448 Discussion: 450 As SDN assures flexible network and agile provisioning, it is 451 important to measure how many network events that the controller can 452 handle at a time. This benchmark is obtained by sending asynchronous 453 messages from every connected Network Devices at full connection 454 capacity for the given test duration. This test assumes that the 455 controller will respond to all the received asynchronous messages. 457 Measurement Units: 458 Messages processed per second. 460 See Also: 461 None 463 2.3.1.4. Reactive Path Provisioning Time 465 Definition: 466 The time taken by the controller to setup a path reactively between 467 source and destination node, defined as the interval starting with 468 the first flow provisioning request message received by the 469 controller(s), ending with the last flow provisioning response 470 message sent from the controller(s) at it Southbound interface. 472 Discussion: 473 As SDN supports agile provisioning, it is important to measure how 474 fast that the controller provisions an end-to-end flow in the 475 dataplane. The benchmark is obtained by sending traffic from a 476 source endpoint to the destination endpoint, finding the time 477 difference between the first and the last flow provisioning message 478 exchanged between the controller and the Network Devices for the 479 traffic path. 481 Measurement Units: 482 milliseconds. 484 See Also: 485 None 487 2.3.1.5. Proactive Path Provisioning Time 489 Definition: 490 The time taken by the controller to setup a path proactively between 491 source and destination node, defined as the interval starting with 492 the first proactive flow provisioned in the controller(s) at its 493 Northbound interface, ending with the last flow provisioning 494 response message sent from the controller(s) at it Southbound 495 interface. 497 Discussion: 498 For SDN to support pre-provisioning of traffic path from 499 application, it is important to measure how fast that the controller 500 provisions an end-to-end flow in the dataplane. The benchmark is 501 obtained by provisioning a flow on controller's northbound interface 502 for the traffic to reach from a source to a destination endpoint, 503 finding the time difference between the first and the last flow 504 provisioning message exchanged between the controller and the 505 Network Devices for the traffic path. 507 Measurement Units: 508 milliseconds. 510 See Also: 511 None 513 2.3.1.6. Reactive Path Provisioning Rate 515 Definition: 516 The maximum number of independent paths a controller can 517 concurrently establish between source and destination nodes 518 reactively, defined as the number of paths provisioned by the 519 controller(s) at its Southbound interface for the flow provisioning 520 requests received for path provisioning at its Southbound interface 521 between the start of the test and the expiry of given test duration 523 Discussion: 524 For SDN to support agile traffic forwarding, it is important to 525 measure how many end-to-end flows that the controller could setup in 526 the dataplane. This benchmark is obtained by sending traffic each 527 with unique source and destination pairs from the source Network 528 Device and determine the number of frames received at the 529 destination Network Device. 531 Measurement Units: 532 Paths provisioned per second. 534 See Also: 535 None 537 2.3.1.7. Proactive Path Provisioning Rate 539 Definition: 540 Measure the maximum number of independent paths a controller can 541 concurrently establish between source and destination nodes 542 proactively, defined as the number of paths provisioned by the 543 controller(s) at its Southbound interface for the paths provisioned 544 in its Northbound interface between the start of the test and the 545 expiry of given test duration 547 Discussion: 548 For SDN to support pre-provisioning of traffic path for a larger 549 network from the application, it is important to measure how many 550 end-to-end flows that the controller could setup in the dataplane. 551 This benchmark is obtained by sending traffic each with unique 552 source and destination pairs from the source Network Device. Program 553 the flows on controller's northbound interface for traffic to reach 554 from each of the unique source and destination pairs and determine 555 the number of frames received at the destination Network Device. 557 Measurement Units: 558 Paths provisioned per second. 560 See Also: 561 None 563 2.3.1.8. Network Topology Change Detection Time 565 Definition: 566 The amount of time required for the controller to detect any changes 567 in the network topology, defined as the interval starting with the 568 notification message received by the controller(s) at its Southbound 569 interface, ending with the first topology rediscovery messages sent 570 from the controller(s) at its Southbound interface. 572 Discussion: 573 In order to for the controller to support fast network failure 574 recovery, it is critical to measure how fast the controller is able 575 to detect any network-state change events. This benchmark is 576 obtained by triggering a topology change event and measuring the 577 time controller takes to detect and initiate a topology re-discovery 578 process. 580 Measurement Units: 581 milliseconds 583 See Also: 584 None 586 2.3.2. Scalability 588 2.3.2.1. Control Sessions Capacity 590 Definition: 591 Measure the maximum number of control sessions the controller can 592 maintain, defined as the number of sessions that the controller can 593 accept from network devices, starting with the first control 594 session, ending with the last control session that the controller(s) 595 accepts at its Southbound interface. 597 Discussion: 598 Measuring the controller's control sessions capacity is important to 599 determine the controller's system and bandwidth resource 600 requirements. This benchmark is obtained by establishing control 601 session with the controller from each of the Network Device until it 602 fails. The number of sessions that were successfully established 603 will provide the Control Sessions Capacity. 605 Measurement Units: 606 N/A 608 See Also: 609 None 611 2.3.2.2. Network Discovery Size 613 Definition: 614 Measure the network size (number of nodes, links and hosts) that a 615 controller can discover, defined as the size of a network that the 616 controller(s) can discover, starting from a network topology given 617 by the user for discovery, ending with the topology that the 618 controller(s) could successfully discover. 620 Discussion: 621 For optimal network planning, it is key to measure the maximum 622 network size that the controller can discover. This benchmark is 623 obtained by presenting an initial set of Network Devices for 624 discovery to the controller. Based on the initial discovery, the 625 number of Network Devices is increased or decreased to determine the 626 maximum nodes that the controller can discover. 628 Measurement Units: 629 N/A 631 See Also: 632 None 634 2.3.2.3. Forwarding Table Capacity 636 Definition: 637 The maximum number of flow entries that a controller can manage in 638 its Forwarding table. 640 Discussion: 641 It is significant to measure the capacity of controller's Forwarding 642 Table to determine the number of flows that controller could forward 643 without flooding/dropping. This benchmark is obtained by 644 continuously presenting the controller with new flow entries through 645 reactive or proactive flow provisioning mode until the forwarding 646 table becomes full. The maximum number of nodes that the controller 647 can hold in its Forwarding Table will provide Forwarding Table 648 Capacity. 650 Measurement Units: 651 Maximum number of flow entries managed. 653 See Also: 654 None 656 2.3.3. Security 658 2.3.3.1. Exception Handling 660 Definition: 661 To determine the effect of handling error packets and notifications 662 on performance tests. 664 Discussion: 665 This benchmark test is to be performed after obtaining the baseline 666 performance of the performance tests defined in Section 2.3.1. This 667 benchmark determines the deviation from the baseline performance due 668 to the handling of error or failure messages from the connected 669 Network Devices. 671 Measurement Units: 672 N/A 674 See Also: 675 None 677 2.3.3.2. Denial of Service Handling 679 Definition: 680 To determine the effect of handling denial of service (DoS) attacks 681 on performance and scalability tests. 683 Discussion: 684 This benchmark test is to be performed after obtaining the baseline 685 performance of the performance and scalability tests defined in 686 section 2.3.1 and section 2.3.1.. This benchmark determines the 687 deviation from the baseline performance due to the handling of 688 denial of service attacks on controller. 690 Measurement Units: 691 Deviation of baseline metrics while handling Denial of Service 692 Attacks. 694 See Also: 695 None 697 2.3.4. Reliability 699 2.3.4.1. Controller Failover Time 701 Definition: 702 The time taken to switch from an active controller to the backup 703 controller, when the controllers work in redundancy mode and the 704 active controller fails, defined as the interval starting with the 705 active controller bringing down, ending with the first re-discovery 706 message received from the new controller at its Southbound 707 interface. 709 Discussion: 710 This benchmark determine the impact of provisioning new flows when 711 controllers are teamed and the active controller fails. 713 Measurement Units: 714 milliseconds. 716 See Also: 717 None 719 2.3.4.2. Network Re-Provisioning Time 721 Definition: 723 The time taken to re-route the traffic by the Controller, when there 724 is a failure in existing traffic paths, defined as the interval 725 starting from the first failure notification message received by the 726 controller, ending with the last flow re-provisioning message sent 727 by the controller at its Southbound interface . 729 Discussion: 730 This benchmark determines the controller's re-provisioning ability 731 upon network failures. This benchmark test assumes the following: 732 i. Network topology supports redundant path between 733 source and destination endpoints. 734 ii. Controller does not pre-provision the redundant path. 736 Measurement Units: 737 milliseconds. 739 See Also: 740 None 742 3. Test Setup 744 This section provides common reference topologies that are later 745 referred to in individual tests defined in the companion methodology 746 document. 748 3.1. Test setup - Controller working in Standalone Mode 750 +-----------------------------------------------------------+ 751 | Application Plane Test Emulator | 752 | | 753 | +-----------------+ +-------------+ | 754 | | Application | | Service | | 755 | +-----------------+ +-------------+ | 756 | | 757 +-----------------------------+(I2)-------------------------+ 758 | 759 | 760 | (Northbound interface) 761 +-------------------------------+ 762 | +----------------+ | 763 | | SDN Controller | | 764 | +----------------+ | 765 | | 766 | Device Under Test (DUT) | 767 +-------------------------------+ 768 | (Southbound interface) 769 | 770 | 771 +-----------------------------+(I1)-------------------------+ 772 | | 773 | +-----------+ +-----------+ | 774 | | Network |l1 ln-1| Network | | 775 | | Device 1 |---- .... ----| Device n | | 776 | +-----------+ +-----------+ | 777 | |l0 |ln | 778 | | | | 779 | | | | 780 | +---------------+ +---------------+ | 781 | | Test Traffic | | Test Traffic | | 782 | | Generator | | Generator | | 783 | | (TP1) | | (TP2) | | 784 | +---------------+ +---------------+ | 785 | | 786 | Forwarding Plane Test Emulator | 787 +-----------------------------------------------------------+ 789 Figure 1 791 3.2. Test setup - Controller working in Cluster Mode 793 +-----------------------------------------------------------+ 794 | Application Plane Test Emulator | 795 | | 796 | +-----------------+ +-------------+ | 797 | | Application | | Service | | 798 | +-----------------+ +-------------+ | 799 | | 800 +-----------------------------+(I2)-------------------------+ 801 | 802 | 803 | (Northbound interface) 804 +---------------------------------------------------------+ 805 | | 806 | ------------------ ------------------ | 807 | | SDN Controller 1 | <--E/W--> | SDN Controller n | | 808 | ------------------ ------------------ | 809 | | 810 | Device Under Test (DUT) | 811 +---------------------------------------------------------+ 812 | (Southbound interface) 813 | 814 | 815 +-----------------------------+(I1)-------------------------+ 816 | | 817 | +-----------+ +-----------+ | 818 | | Network |l1 ln-1| Network | | 819 | | Device 1 |---- .... ----| Device n | | 820 | +-----------+ +-----------+ | 821 | |l0 |ln | 822 | | | | 823 | | | | 824 | +---------------+ +---------------+ | 825 | | Test Traffic | | Test Traffic | | 826 | | Generator | | Generator | | 827 | | (TP1) | | (TP2) | | 828 | +---------------+ +---------------+ | 829 | | 830 | Forwarding Plane Test Emulator | 831 +-----------------------------------------------------------+ 833 Figure 2 835 4. Test Coverage 837 + -----------------------------------------------------------------+ 838 | | Speed | Scalability | Reliability | 839 + -----------+-------------------+---------------+-----------------+ 840 | | 1. Network Topolo-|1. Network | | 841 | | -gy Discovery | Discovery | | 842 | | | Size | | 843 | | 2. Reactive Path | | | 844 | | Provisioning | | | 845 | | Time | | | 846 | | | | | 847 | | 3. Proactive Path | | | 848 | | Provisioning | | | 849 | Setup | Time | | | 850 | | | | | 851 | | 4. Reactive Path | | | 852 | | Provisioning | | | 853 | | Rate | | | 854 | | | | | 855 | | 5. Proactive Path | | | 856 | | Provisioning | | | 857 | | Rate | | | 858 | | | | | 859 +------------+-------------------+---------------+-----------------+ 860 | | 1. Asynchronous |1. Control |1. Network | 861 | | Message Proces-| Sessions | Topology | 862 | | -sing Rate | Capacity | Change | 863 | | | | Detection Time| 864 | | 2. Asynchronous |2. Forwarding | | 865 | | Message Proces-| Table |2. Exception | 866 | | -sing Time | Capacity | Handling | 867 | Operational| | | | 868 | | | |3. Denial of | 869 | | | | Service | 870 | | | | Handling | 871 | | | | | 872 | | | |4. Network Re- | 873 | | | | Provisioning | 874 | | | | Time | 875 | | | | | 876 +------------+-------------------+---------------+-----------------+ 877 | | | | | 878 | Tear Down | | |1. Controller | 879 | | | | Failover Time | 880 +------------+-------------------+---------------+-----------------+ 882 5. References 884 5.1. Normative References 886 [RFC7426] E. Haleplidis, K. Pentikousis, S. Denazis, J. Hadi Salim, 887 D. Meyer, O. Koufopavlou "Software-Defined Networking 888 (SDN): Layers and Architecture Terminology", RFC 7426, 889 January 2015. 891 [RFC4689] S. Poretsky, J. Perser, S. Erramilli, S. Khurana 892 "Terminology for Benchmarking Network-layer Traffic 893 Control Mechanisms", RFC 4689, October 2006. 895 [RFC2330] V. Paxson, G. Almes, J. Mahdavi, M. Mathis, 896 "Framework for IP Performance Metrics", RFC 2330, 897 May 1998. 899 [OpenFlow Switch Specification] ONF,"OpenFlow Switch Specification" 900 Version 1.4.0 (Wire Protocol 0x05), October 14, 2013. 902 [I-D.sdn-controller-benchmark-meth] Bhuvaneswaran.V, Anton Basil, 903 Mark.T, Vishwas Manral, Sarah Banks "Benchmarking 904 Methodology for SDN Controller Performance", 905 draft-ietf-bmwg-sdn-controller-benchmark-meth-02 906 (Work in progress), July 8, 2016 908 5.2. Informative References 910 [OpenContrail] Ankur Singla, Bruno Rijsman, "OpenContrail 911 Architecture Documentation", 912 http://opencontrail.org/opencontrail-architecture-documentation 914 [OpenDaylight] OpenDaylight Controller:Architectural Framework, 915 https://wiki.opendaylight.org/view/OpenDaylight_Controller 917 6. IANA Considerations 919 This document does not have any IANA requests. 921 7. Security Considerations 923 Security issues are not discussed in this memo. 925 8. Acknowledgements 927 The authors would like to acknowledge Al Morton (AT&T) for the 928 significant contributions to the earlier versions of this document. 929 The authors would like to thank the following individuals for 930 providing their valuable comments to the earlier versions of this 931 document: Sandeep Gangadharan (HP), M. Georgescu (NAIST), Andrew 932 McGregor (Google), Scott Bradner (Harvard University), Jay Karthik 933 (Cisco), Ramakrishnan (Dell), Khasanov Boris (Huawei). 935 9. Authors' Addresses 937 Bhuvaneswaran Vengainathan 938 Veryx Technologies Inc. 939 1 International Plaza, Suite 550 940 Philadelphia 941 PA 19113 943 Email: bhuvaneswaran.vengainathan@veryxtech.com 945 Anton Basil 946 Veryx Technologies Inc. 947 1 International Plaza, Suite 550 948 Philadelphia 949 PA 19113 951 Email: anton.basil@veryxtech.com 953 Mark Tassinari 954 Hewlett-Packard, 955 8000 Foothills Blvd, 956 Roseville, CA 95747 958 Email: mark.tassinari@hpe.com 960 Vishwas Manral 961 Nano Sec, 962 CA 964 Email: vishwas.manral@gmail.com 966 Sarah Banks 967 VSS Monitoring 969 Email: sbanks@encrypted.net