idnits 2.17.1 draft-ietf-bmwg-evpntest-05.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- == It seems as if not all pages are separated by form feeds - found 0 form feeds but 28 pages Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** There are 43 instances of too long lines in the document, the longest one being 61 characters in excess of 72. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 211 has weird spacing: '...nerator sende...' == Line 212 has weird spacing: '...eceiver of la...' == The document seems to lack the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords -- however, there's a paragraph with a matching beginning. Boilerplate error? (The document does seem to have the reference to RFC 2119 which the ID-Checklist requires). -- The document date (March 11, 2020) is 1507 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Missing Reference: 'RFC8174' is mentioned on line 106, but not defined == Unused Reference: 'RFC2119' is defined on line 1231, but no explicit reference was found in the text == Unused Reference: 'RFC2544' is defined on line 1236, but no explicit reference was found in the text == Unused Reference: 'RFC2899' is defined on line 1241, but no explicit reference was found in the text == Unused Reference: 'RFC7432' is defined on line 1247, but no explicit reference was found in the text == Unused Reference: 'RFC7623' is defined on line 1252, but no explicit reference was found in the text Summary: 1 error (**), 0 flaws (~~), 11 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Internet Engineering Task Force S. Jacob, Ed. 3 Internet-Draft K. Tiruveedhula 4 Intended status: Informational Juniper Networks 5 Expires: September 12, 2020 March 11, 2020 7 Benchmarking Methodology for EVPN and PBB-EVPN 8 draft-ietf-bmwg-evpntest-05 10 Abstract 12 This document defines methodologies for benchmarking EVPN and PBB- 13 EVPN performance.EVPN is defined in RFC 7432, and is being deployed 14 in Service Provider networks.Specifically, this document defines 15 the methodologies for benchmarking EVPN/PBB-EVPN convergence, data 16 plane performance, and control plane performance. 18 Status of This Memo 20 This Internet-Draft is submitted in full conformance with the 21 provisions of BCP 78 and BCP 79. 23 Internet-Drafts are working documents of the Internet Engineering 24 Task Force (IETF). Note that other groups may also distribute 25 working documents as Internet-Drafts. The list of current Internet- 26 Drafts is at https://datatracker.ietf.org/drafts/current/. 28 Internet-Drafts are draft documents valid for a maximum of six months 29 and may be updated, replaced, or obsoleted by other documents at any 30 time. It is inappropriate to use Internet-Drafts as reference 31 material or to cite them other than as "work in progress." 33 This Internet-Draft will expire on September 12, 2020. 35 Copyright Notice 37 Copyright (c) 2020 IETF Trust and the persons identified as the 38 document authors. All rights reserved. 40 This document is subject to BCP 78 and the IETF Trust's Legal 41 Provisions Relating to IETF Documents 42 (https://trustee.ietf.org/license-info) in effect on the date of 43 publication of this document. Please review these documents 44 carefully, as they describe your rights and restrictions with respect 45 to this document. Code Components extracted from this document must 46 include Simplified BSD License text as described in Section 4.e of 47 the Trust Legal Provisions and are provided without warranty as 48 described in the Simplified BSD License. 50 Table of Contents 52 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 53 1.1. Requirements Language . . . . . . . . . . . . . . . . . . 3 54 1.2. Terminologies . . . . . . . . . . . . . . . . . . . . . . 3 55 2. Test Topology . . . . . . . . . . . . . . . . . . . . . . . . 4 56 3. Test Cases for EVPN Benchmarking . . . . . . . . . . . . . . 7 57 3.1. Data Plane MAC Learning . . . . . . . . . . . . . . . . . 7 58 3.2. Control Plane MAC Learning . . . . . . . . . . . . . . . 8 59 3.3. MAC Flush-Local Link Failure and Relearning . . . . . . . 9 60 3.4. MAC Flush-Remote Link Failure and Relearning. . . . . . . 10 61 3.5. MAC Aging . . . . . . . . . . . . . . . . . . . . . . . . 11 62 3.6. Remote MAC Aging . . . . . . . . . . . . . . . . . . . . 12 63 3.7. Control and Data plane MAC Learning . . . . . . . . . . . 12 64 3.8. High Availability. . . . . . . . . . . . . . . . . . . . 13 65 3.9. ARP/ND Scale . . . . . . . . . . . . . . . . . . . . . . 14 66 3.10. Scaling of Services . . . . . . . . . . . . . . . . . . . 15 67 3.11. Scale Convergence . . . . . . . . . . . . . . . . . . . . 16 68 3.12. SOAK Test. . . . . . . . . . . . . . . . . . . . . . . . 17 69 4. Test Cases for PBB-EVPN Benchmarking . . . . . . . . . . . . 18 70 4.1. Data Plane Local MAC Learning . . . . . . . . . . . . . . 18 71 4.2. Data Plane Remote MAC Learning . . . . . . . . . . . . . 18 72 4.3. MAC Flush-Local Link Failure . . . . . . . . . . . . . . 19 73 4.4. MAC Flush-Remote Link Failure . . . . . . . . . . . . . . 20 74 4.5. MAC Aging . . . . . . . . . . . . . . . . . . . . . . . . 21 75 4.6. Remote MAC Aging. . . . . . . . . . . . . . . . . . . . . 22 76 4.7. Local and Remote MAC Learning . . . . . . . . . . . . . . 23 77 4.8. High Availability . . . . . . . . . . . . . . . . . . . . 23 78 4.9. Scale . . . . . . . . . . . . . . . . . . . . . . . . . . 24 79 4.10. Scale Convergence . . . . . . . . . . . . . . . . . . . . 25 80 4.11. Soak Test . . . . . . . . . . . . . . . . . . . . . . . . 26 81 5. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . 27 82 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 27 83 7. Security Considerations . . . . . . . . . . . . . . . . . . . 27 84 8. References . . . . . . . . . . . . . . . . . . . . . . . . . 27 85 8.1. Normative References . . . . . . . . . . . . . . . . . . 27 86 8.2. Informative References . . . . . . . . . . . . . . . . . 28 87 Appendix A. Appendix . . . . . . . . . . . . . . . . . . . . . . 28 88 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 28 90 1. Introduction 92 EVPN is defined in RFC 7432, and describes BGP MPLS based Ethernet 93 VPNs (EVPN). PBB-EVPN is defined in RFC 7623, discusses how Ethernet 94 Provider backbone Bridging can be combined with EVPNs to provide a 95 new/combined solution. This draft defines methodologies that can be 96 used to benchmark both RFC 7432 and RFC 7623 solutions.Further, 97 this draft provides methodologies for benchmarking the performance of 98 EVPN data and control planes, MAC learning, MAC flushing, MAC aging, 99 convergence, high availability, and scale. 101 1.1. Requirements Language 103 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 104 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 105 document are to be interpreted as described in RFC 8174 [RFC8174]. 107 1.2. Terminologies 109 All-Active Redundancy Mode: When all PEs attached to an Ethernet 110 segment are allowed to forward known unicast traffic to/from that 111 Ethernet segment for a given VLAN, then the Ethernet segment is 112 defined to be operating in All-Active redundancy mode. 114 AA: All Active mode 116 CE: Customer Router/Devices/Switch. 118 DF: Designated Forwarder 120 DUT: Device under test. 122 Ethernet Segment (ES): When a customer site (device or network) is 123 connected to one or more PEs via a set of Ethernet links, then that 124 set of links is referred to as an 'Ethernet segment'. 126 EVI: An EVPN instance spanning the Provider Edge (PE) devices 127 participating in that EVPN. 129 Ethernet Segment Identifier (ESI): A unique non-zero identifier that 130 identifies an Ethernet segment is called an 'Ethernet Segment 131 Identifier'. 133 Ethernet Tag: An Ethernet tag identifies a particular broadcast 134 domain, e.g., a VLAN. An EVPN instance consists of one or more 135 broadcast domains. 137 Interface: Physical interface of a router/switch. 139 IRB: Integrated routing and bridging interface 141 MAC: Media Access Control addresses on a PE. 143 MHPE2: Multi homed Provider Edge router 2. 145 MHPE1: Multi homed Provider Edge router 1. 147 SHPE3: Single homed Provider Edge Router 3. 149 PE: Provider Edge device. 151 P: Provider Router. 153 RR: Route Reflector. 155 RT: Traffic Generator. 157 Sub Interface: Each physical Interfaces is subdivided into Logical 158 units. 160 SA: Single Active 162 Single-Active Redundancy Mode: When only a single PE, among all the 163 PEs attached to an Ethernet segment, is allowed to forward traffic 164 to/from that Ethernet segment for a given VLAN, then the Ethernet 165 segment is defined to be operating in Single-Active redundancy mode. 167 2. Test Topology 169 There are five routers in the Test setup.SHPE3, RR/P, MHPE1 and 170 MHPE2 emulating a service provider network.CE is a customer device 171 connected to MHPE1 and MHPE2, it is configured with bridge domains in 172 multiple vlans. The traffic generator is connected to CE and 173 SHPE3.The MHPE1 acts as DUT.The traffic generator will be used as 174 sender and receiver of traffic.The DUT will be the reference point 175 for all the test cases. MHPE1 and MHPE2 are mulihome routers 176 connected to CE running single active mode.The traffic generator 177 will be generating traffic at 10% of the line rate. 179 +----------------+ +-------------------------+ 180 | | | | 181 | | |Traffic Generator sender/| 182 | SHPE3 | |receiver of layer 2 traffic| 183 | +-----------------+ with multiple Vlans | 184 | | +-------------------------+ 185 +---------+------+ 186 | Core Link 187 | 188 +--------+-----+ 189 | | 190 | RR/P | 191 | | Core link 192 | +----------------+ 193 +--+-----------+ | 194 | | 195 | core link | 196 | | 197 +-------------+---+ ++------------------+ 198 | | | | 199 | | | MHPE2 | 200 |MHPE1(DUT) | | | 201 | | | | 202 | | | | 203 +-----------------+------+ +-----+-------------------+ 204 | | 205 PE-CE link | | PE-CE link 206 | | 207 | | 208 | | 209 | | 210 +-----+----------+----+ +----------------------------+ 211 | CE/Layer 2 bridge +-----------| Traffic Generator sender/ | 212 | | |receiver of layer 2 traffic| 213 | | | with multiple Vlans | 214 | | +----------------------------+ 215 | | 216 +---------------------+ 218 Topology 1 220 Test Setup 222 Figure 1 224 +-----------------+---------------------+---------------------+---------------------+----------------------+-----------------------+ 225 | | | | | | | 226 | | | | | | | 227 | | | | | | | 228 | | | | | | | 229 | Mode | | | |Receiver | | 230 | | Test |Traffic Direction |Sender | | | 231 | | | | | | | 232 | | | | | | | 233 | | | | | | | 234 +----------------------------------------------------------------------------------------------------------------------------------+ 235 | | | | | | | 236 | | | | | SHPE3 | | 237 |Single Active | Local MAC | |CE | |Layer 2 traffic | 238 | | Learning | Uni | | | | 239 | | | | | | multiple MAC | 240 | | | | | | | 241 +-----------------------------------------------------------------------------------------------------------------------------------+ 242 | | | | | | | 243 |Single Active | Remote MAC | | | CE |Layer 2 traffic | 244 | | Learning | uni | SHPE3 | | | 245 | | | | | |multiple MAC | 246 | | | | | | ++ 247 +----------------------------------------------------------------------------------------------------------------------------------+ 248 | | | | | | | 249 |Single Active | Scale Convergence | Bi | | CE/SHPE3 | | 250 | | | | CE/SHPE3 | |Layer 2 traffic | 251 | | Local& Remote | | | |multiple MAC& vlans | 252 | | Learning | | | | | 253 +-----------------+---------------------+---------------------+--------------------------------------------+-----------------------+ 254 | 255 ++ 257 Table showing the traffic directions of various EVPN/PBB-EVPN 258 benchmarking test cases. Depends on the test scenario the traffic 259 can be uni/bi directional generated by the traffic generator. 261 Figure 2 263 Test Setup Configurations: 265 SHPE3 is configured with Interior Gateway protocols like OPSF or IS- 266 IS for underlay, LDP for MPLS support,Interior Border Gateway with 267 EVPN address family for overlay support.This router must be 268 configured with N EVPN/PBB-EVPN instances for testing.Traffic 269 generator is connected to this router for sending and receiving 270 traffic. 272 RR is configured with Interior Gateway protocols like OPSF or IS-IS 273 for underlay, LDP for MPLS support,Interior Border Gateway with EVPN 274 address family for overlay support.This router acts as a provider 275 router and as a route reflector. 277 MHPE1 is configured with Interior Gateway protocols like OPSF or IS- 278 IS for underlay, LDP for MPLS support,Interior Border Gateway with 279 EVPN address family for overlay support.This router must be 280 configured with N EVPN/PBB-EVPN instances for testing.This router 281 is configured with ESI per vlan or ESI per interface.It is 282 functioning as multi homing PE working on Single Active EVPN 283 mode.This router serves as the DUT and it is connected to CE.MHPE1 284 is acting as DUT for all the test cases. 286 MHPE2 is configured with Interior Gateway protocols like OPSF or IS- 287 IS for underlay, LDP for MPLS support,Interior Border Gateway with 288 EVPN address family for overlay support.This router must be 289 configured with N EVPN/PBB-EVPN instances for testing.This router 290 is configured with ESI per vlan or ESI per interface.It is 291 functioning as multi homing PE working on Single Active EVPN mode. 292 It is connected to CE. 294 CE is acting as bridge configured with multiple vlans,the same vlans 295 are configured on MHPE1,MHPE2,SHPE3.Traffic generator is connected 296 to CE. the traffic generator acts as sender or receiver of traffic. 298 Depending up on the test scenarios the traffic generators will be 299 used to generate uni directional or bi directional flows. 301 The above configuration will be serving as the base configuration for 302 all test cases. 304 3. Test Cases for EVPN Benchmarking 306 3.1. Data Plane MAC Learning 308 Objective: 310 Measure the time taken to learn the Data Plane MAC in DUT. 312 Topology : Topology 1 314 Procedure: 316 The data plane MAC learning can be measured using the parameters 317 defined in RFC 2889 section 5.8. 319 Confirm the DUT is up and running with EVPN. 321 Traffic generator connected to CE must send frames with "X" different 322 source and destination MAC address for one vlan, the same vlan must 323 be present in all the devices except RR. 325 Send "X" unicast frames from CE to MHPE1(DUT) for one EVPN instance 326 working in SA mode. 328 The DUT will learn these "X" MAC in data plane. 330 Measurement : 332 Measure the time taken to learn "X" MAC locally in DUT evpn MAC 333 table.The data plane measurement is taken by considering DUT as 334 black box. The range of MAC are known from traffic generator,the 335 same must be learned in DUT, the time taken to learn "X" MAC is 336 measured.The measurement is carried out using external server which 337 polls the DUT using automated scripts. 339 The test is repeated for "N" times and the values are collected.The 340 MAC learning rate is calculated by averaging the values obtained from 341 "N" samples. "N" is an arbitrary number to get a sufficient sample. 342 The time measured for each sample is denoted by T1,T2...Tn. 344 MAC learning rate = (T1+T2+..Tn)/N 346 3.2. Control Plane MAC Learning 348 Objective: 350 Measure the time taken to learn the control plane MAC. 352 Topology : Topology 1 354 Procedure: 356 Confirm the DUT is up and running with EVPN. 358 Traffic generator connected to SHPE3 must send frames with "X" 359 different source and destination MAC address for one vlan, the same 360 vlan must be present in all the devices except RR. 362 Ensure the frames must be destined to one EVPN instance. 364 The DUT will learn these "X" MAC in control plane. 366 Measurement : 368 Measure the time taken by the DUT to learn the "X" MAC in the data 369 plane.The test is repeated for "N" times and the values are 370 collected. The remote MAC learning rate is calculated by averaging 371 the values obtained from "N" samples. "N" is an arbitrary number to 372 get a sufficient sample. The time measured for each sample is 373 denoted by T1,T2...Tn.The measurement is carried out using external 374 server which polls the DUT using automated scripts. 376 MAC learning rate = (T1+T2+..Tn)/N 378 3.3. MAC Flush-Local Link Failure and Relearning 380 Objective: 382 Measure the time taken to flush the Data Plane MAC and the time taken 383 to relearn the same amount of MAC. 385 Topology : Topology 1 387 Procedure: 389 Confirm the DUT is up and running with EVPN. 391 Send X frames with X different source and destination MAC addresses 392 to DUT from CE using traffic generator for one vlan. 394 Ensure the DUT learns all X MAC addresses in data plane. 396 Fail the DUT-CE link and measure the time taken to flush these X MAC 397 from the EVPN MAC table. 399 Bring up the link which was made Down(the link between DUT and 400 CE).Measure time taken by the DUT to relearn these "X" MAC. 402 The DUT and MHPE2 are running SA mode. 404 Measurement : 406 Measure the time taken for flushing these X MAC addresses. Measure 407 the time taken to relearn these X MAC in DUT.The test is repeated for 408 "N" times and the values are collected. The flush and the relearning 409 time is calculated by averaging the values obtained by "N" 410 samples."N" is an arbitrary number to get a sufficient sample.The 411 time measured for each sample is denoted by T1,T2...Tn.The 412 measurement is carried out using external server which polls the DUT 413 using automated scripts. 415 Flush rate = (T1+T2+..Tn)/N 417 Relearning rate = (T1+T2+..Tn)/N 419 3.4. MAC Flush-Remote Link Failure and Relearning. 421 Objective: 423 Measure the time taken to flush the Control plane MAC learned in DUT 424 during remote link failure and the time taken to relearn. 426 Topology : Topology 1 428 Procedure: 430 Confirm the DUT is up and running with EVPN. 432 Send X frames with X different source and destination MAC addresses 433 to DUT from SHPE3 using traffic generator for one vlan. 435 Bring down the link between SHPE3 and traffic generator. 437 SHPE3 will withdraw the routes from DUT due to link failure. 439 Measure the time taken to flush the DUT EVPN MAC table. The DUT and 440 MHPE2 are running SA mode. 442 Bring up the link which was made Down(the link between SHPE3 and 443 traffic generator). 445 Measure time taken by the DUT to relearn these "X" MAC from control 446 plane. 448 Measurement : 450 Measure the time taken to flush X remote MAC from EVPN MAC table of 451 the DUT.Measure the time taken to relearn these X MAC in DUT.The 452 test is repeated for "N" times and the values are collected.The 453 flush rate is calculated by averaging the values obtained by "N" 454 samples. "N" is an arbitrary number to get a sufficient sample.The 455 time measured for each sample is denoted by T1,T2...Tn.The 456 measurement is carried out using external server which polls the DUT 457 using automated scripts. 459 Flush rate = (T1+T2+..Tn)/N 461 Relearning rate = (T1+T2+..Tn)/N 463 3.5. MAC Aging 465 Objective: 467 To measure the MAC aging time. 469 Topology : Topology 1 471 Procedure: 473 Confirm the DUT is up and running with EVPN. 475 Send X frames with X different source and destination MAC addresses 476 to DUT from CE using traffic generator for one vlan. 478 Ensure these X MAC addresses are learned in DUT. 480 Then stop the traffic. 482 Ensure the DUT and other devices in the test are using the default 483 timers for aging. 485 Measure the time taken to flush X MAC from DUT EVPN MAC table due to 486 aging. 488 The DUT and MHPE2 are running SA mode. 490 Measurement : 492 Measure the time taken to flush X MAC addresses due to aging. The 493 test is repeated for "N" times and the values are collected. The 494 aging is calculated averaging the values obtained by "N" samples. 495 "N" is an arbitrary number to get a sufficient sample.The time 496 measured for each sample is denoted by T1,T2...Tn. The measurement 497 is carried out using external server which polls the DUT using 498 automated scripts. 500 Aging time for X MAC in sec = (T1+T2+..Tn)/N 502 3.6. Remote MAC Aging 504 Objective: 506 Measure the control plane learned MAC aging time. 508 Topology : Topology 1 510 Procedure: 512 Confirm the DUT is up and running with EVPN. 514 Send X frames with X different source and destination MAC addresses 515 to DUT from SHPE3 using traffic generator for one vlan. 517 Ensure these X MAC addresses are learned in DUT via control plane. 519 Then stop the traffic. 521 Ensure the DUT and other devices in the test are using the default 522 timers for aging. 524 Measure the time taken to flush X MAC from DUT EVPN MAC table due to 525 aging. 527 The DUT and MHPE2 are running SA mode. 529 Measurement : 531 Measure the time taken to flush X remote MAC learned in DUT EVPN MAC 532 table due to aging.The test is repeated for "N" times and the values 533 are collected.The aging is calculated by averaging the values 534 obtained by "N" samples."N" is an arbitrary number to get a 535 sufficient sample. The time measured for each sample is denoted by 536 T1,T2...Tn.The measurement is carried out using external server 537 which polls the DUT using automated scripts. 539 Aging time for X MAC in sec = (T1+T2+..Tn)/N 541 3.7. Control and Data plane MAC Learning 543 Objective: 545 To record the time taken to learn both local and remote MAC. 547 Topology : Topology 1 548 Procedure: 550 Confirm the DUT is up and running with EVPN. 552 Send X frames with X different source and destination MAC addresses 553 to DUT from SHPE3 using traffic generator for one vlan. 555 Send X frames with different source and destination MAC addresses 556 from traffic generator connected to CE for one vlan. 558 The source and destination addresses of flows must be complimentary 559 to have unicast flows. 561 Measure the time taken by the DUT to learn 2X in EVPN MAC table. 563 DUT and MHPE2 are running in SA mode. 565 Measurement : 567 Measure the time taken to learn 2X MAC addresses in DUT EVPN MAC 568 table.The test is repeated for "N" times and the values are 569 collected. The MAC learning time is calculated by averaging the 570 values obtained by "N" samples."N" is an arbitrary number to get a 571 sufficient sample.The time measured for each sample is denoted by 572 T1,T2...Tn. The measurement is carried out using external server 573 which polls the DUT using automated scripts 575 MAC learning rate = (T1+T2+..Tn)/N 577 3.8. High Availability. 579 Objective: 581 Measure traffic loss during routing engine fail over. 583 Topology : Topology 1 585 Procedure: 587 Confirm the DUT is up and running with EVPN. 589 Send X frames from CE to DUT from traffic generator with X different 590 source and destination MAC addresses. 592 Send X frames from traffic generator to SHPE3 with X different source 593 and destination MAC addresses, so that 2X MAC address will be learned 594 in the DUT. 596 There is a bi directional traffic flow with X pps in each direction. 598 Ensure the DUT learn 2X MAC. 600 Then do a routing engine fail-over. 602 Measurement : 604 The expectation of the test is 0 traffic loss with no change in the 605 DF role. DUT should not withdraw any routes.But in cases where the 606 DUT is not property synchronized between master and standby,due to 607 that packet loss are observed. In that scenario the packet loss is 608 measured.The test is repeated for "N" times and the values are 609 collected.The packet loss is calculated by averaging the values 610 obtained by "N" samples."N" is an arbitrary number to get a 611 sufficient sample. The time measured for each sample is denoted by 612 T1,T2...Tn. The measurement is carried out using external server 613 which polls the DUT using automated scripts to ensure the DUT learned 614 2X MAC. The packet drop is measured using traffic generator. 616 Packet loss in sec with 2X MAC addresses = (T1+T2+..Tn)/N 618 3.9. ARP/ND Scale 620 Measure the DUT scaling limit of ARP/ND. 622 Objective: 624 Measure the ARP/ND scale of the DUT. 626 Topology : Topology 1 628 Procedure: 630 Confirm the DUT is up and running with EVPN. 632 Send X arp/neighbor discovery(ND) from the traffic generator to DUT 633 with different sender ip/ipv6,MAC addresses to the target IRB address 634 configured in EVPN instance. 636 The EVPN instance learns the MAC+ip and MAC+ipv6 addresses from these 637 request and advertise as type 2 MAC+ip/MAC+ipv6 route to remote 638 provide edge routers which have same EVPN configurations. 640 The value of X must be increased at a incremental value of 5% of X, 641 till the limit is reached.The limit is where the DUT cant learn any 642 more type 2 MAC+ip/MAC+ipv6.The test must be separately conducted for 643 arp and ND. 645 Measurement : 647 Measure the scale limit of type 2 MAC+ip/MAC+ipv6 route which DUT can 648 learn.The test is repeated for "N" times and the values are 649 collected.The scale limit is calculated by averaging the values 650 obtained by "N" samples for both MAC+ip and MAC+ipv6."N" is an 651 arbitrary number to get a sufficient sample.The scale value 652 obtained by each sample be v1,v2..vn.The measurement is carried out 653 using external server which polls the DUT using automated scripts to 654 find the scale limit of MAC+ipv4/MAC+ipv6. 656 Scale limit for MAC+ip = (v1+v2+..vn)/N 658 Scale limit for MAC+ipv6 = (v1+v2+..vn)/N 660 3.10. Scaling of Services 662 Objective: 664 Measure the scale of EVPN instances that a DUT can hold. 666 Topology : Topology 1 668 Procedure: 670 Confirm the DUT is up and running with EVPN. 672 The DUT,MHPE2 and SHPE3 are scaled to "N" EVI. 674 Ensure routes received from MHPE2 and SHPE3 for "N" EVI in the DUT. 676 Then increment the scale of N by 5% of N till the limit is reached. 678 The limit is where the DUT cant learn any EVPN routes from its peers. 680 Measurement : 682 There should not be any loss of route types 1,2,3 and 4 in DUT. DUT 683 must relearn all type 1,2,3 and 4 from remote routers.The DUT must 684 be subjected to various values of N to find the optimal scale limit. 685 The scope of the test is find out the maximum evpn instance that a 686 DUT can hold.The measurement is carried out using external server 687 which polls the DUT using automated scripts to find the scale limit 688 of EVPN instances. 690 3.11. Scale Convergence 692 Objective: 694 Measure the convergence time of DUT when the DUT is scaled with EVPN 695 instance along with traffic. 697 Topology : Topology 1 699 Procedure: 701 Confirm the DUT is up and running with EVPN. 703 Scale N EVIs in DUT,SHPE3 and MHPE2. 705 Send F frames to DUT from CE using traffic generator with X different 706 source and destination MAC addresses for N EVI's. 708 Send F frames from traffic generator to SHPE3 with X different source 709 and destination MAC addresses. 711 There will be 2X number of MAC addresses will be learned in DUT EVPN 712 MAC table. 714 There is a bi directional traffic flow with F pps in each direction. 716 Then clear the BGP neighbors in the DUT. 718 Once the BGP session is in established state in DUT. 720 Measure the time taken to learn 2X MAC address in DUT MAC table. 722 Measurement : 724 The DUT must learn 2X MAC addresses. Measure the time taken to learn 725 2X MAC in DUT.The test is repeated for "N" times and the values are 726 collected. The convergence time is calculated by averaging the 727 values obtained by "N" samples."N" is an arbitrary number to get a 728 sufficient sample.The time measured for each sample is denoted by 729 T1,T2...Tn.The measurement is carried out using external server which 730 polls the DUT using automated scripts. 732 Time taken to learn 2X MAC in DUT = (T1+T2+..Tn)/N 734 3.12. SOAK Test. 736 Objective: 738 This test is carried out to measure the stability of the DUT in a 739 scaled environment with traffic over a period of time "T'". In each 740 interval "t1" the DUT CPU usage, memory usage are measured.The DUT 741 is checked for any crashes during this time period. 743 Topology : Topology 1 745 Procedure: 747 Confirm the DUT is up and running with EVPN. 749 Scale N EVI's in DUT,SHPE3 and MHPE2.Send F frames to DUT from CE 750 using traffic generator with different X source and destination MAC 751 addresses for N EVI's. 753 Send F frames from traffic generator to SHPE3 with X different source 754 and destination MAC addresses. 756 There will be 2X number of MAC addresses will be learned in DUT EVPN 757 MAC table. 759 There is a bi directional traffic flow with F pps in each direction. 761 The DUT must run with traffic for 24 hours. 763 Every hour check for memory leak in EVPN process, CPU usage and 764 crashes in DUT. 766 Measurement : 768 Take the hourly reading of CPU, process memory. There should not be 769 any leak, crashes, CPU spikes. Th CPU spike is determined as the CPU 770 usage which shoots at 40 to 50 percent of the average usage. The 771 average value vary from device to device. Memory leak is determined 772 by increase usage of the memory for EVPN process. The expectation is 773 under steady state the memory usage for EVPN process should not 774 increase. The measurement is carried out using external server which 775 polls the DUT using automated scripts which captures the CPU usage 776 and process memory. 778 4. Test Cases for PBB-EVPN Benchmarking 780 4.1. Data Plane Local MAC Learning 782 Objective: 784 Measure the time taken to learn the Data Plane MAC in DUT. 786 Topology : Topology 1 788 Procedure: 790 Confirm the DUT is up and running with PBB-EVPN. 792 Traffic generator connected to CE must send frames with "X" different 793 source and destination MAC address for one vlan, the same vlan must 794 be present in all the devices except RR. 796 Send "X" unicast frames from CE to MHPE1(DUT) for one PBB-EVPN 797 instance working in SA mode. 799 The DUT will learn these "X" MAC in data plane. 801 Measurement : 803 Measure the time taken to learn "X" MAC locally in DUT PBB-EVPN MAC 804 table. The data plane measurement is taken by considering DUT as 805 black box. The range of MAC are known from traffic generator,the 806 same must be learned in DUT, the time taken to learn "X" MAC is 807 measured.The measurement is carried out using external server which 808 polls the DUT using automated scripts. 810 The test is repeated for "N" times and the values are collected.The 811 MAC learning rate is calculated by averaging the values obtained from 812 "N" samples. "N" is an arbitrary number to get a sufficient sample. 813 The time measured for each sample is denoted by T1,T2...Tn. 815 MAC learning rate = (T1+T2+..Tn)/N 817 4.2. Data Plane Remote MAC Learning 819 Objective: 821 To Record the time taken to learn the remote MAC. 823 Topology : Topology 1 824 Procedure: 826 Confirm the DUT is up and running with PBB-EVPN. 828 Traffic generator connected to SHPE3 must send frames with "X" 829 different source and destination MAC address for one vlan, the same 830 vlan must be present in all the devices except RR. 832 Ensure the frames must be destined to one PBB-EVPN instance. 834 The DUT will learn these "X" MAC in data plane. 836 Measurement : 838 Measure the time taken by the DUT to learn the "X" MAC in the data 839 plane.The test is repeated for "N" times and the values are 840 collected. The remote MAC learning rate is calculated by averaging 841 the values obtained from "N" samples. "N" is an arbitrary number to 842 get a sufficient sample. The time measured for each sample is 843 denoted by T1,T2...Tn.The measurement is carried out using external 844 server which polls the DUT using automated scripts. 846 MAC learning rate = (T1+T2+..Tn)/N 848 4.3. MAC Flush-Local Link Failure 850 Objective: 852 Measure the time taken to flush the locally learned MAC and the time 853 taken to relearn the same amount of MAC. 855 Topology : Topology 1 857 Procedure: 859 Confirm the DUT is up and running with PBB-EVPN. 861 Send X frames with X different source and destination MAC addresses 862 to DUT from CE using traffic generator for one vlan. 864 Ensure the DUT learns all X MAC addresses in data plane. 866 Fail the DUT-CE link and measure the time taken to flush these X MAC 867 from the PBB-EVPN MAC table. 869 Bring up the link which was made Down(the link between DUT and 870 CE).Measure time taken by the DUT to relearn these "X" MAC. 872 The DUT and MHPE2 are running SA mode. 874 Measurement : 876 Measure the time taken for flushing these X MAC addresses. Measure 877 the time taken to relearn these X MAC in DUT.The test is repeated for 878 "N" times and the values are collected. The flush and the relearning 879 time is calculated by averaging the values obtained by "N" 880 samples."N" is an arbitrary number to get a sufficient sample. The 881 time measured for each sample is denoted by T1,T2...Tn. The 882 measurement is carried out using external server which polls the DUT 883 using automated scripts. 885 Flush rate = (T1+T2+..Tn)/N 887 Relearning rate = (T1+T2+..Tn)/N 889 4.4. MAC Flush-Remote Link Failure 891 Objective: 893 Measure the time taken to flush the remote MAC learned in DUT due to 894 remote link failure and relearning it. 896 Topology : Topology 1 898 Procedure: 900 confirm the DUT is up and running with PBB-EVPN. 902 Send X frames with X different source and destination MAC addresses 903 to DUT from SHPE3 using traffic generator for one vlan. 905 Bring down the link between SHPE3 and traffic generator. 907 Measure the time taken to flush the DUT PBB-EVPN MAC table. The DUT 908 and MHPE2 are running SA mode. 910 Bring up the link which was made Down(the link between SHPE3 and 911 traffic generator). 913 Measure time taken by the DUT to relearn these "X" MAC 915 Measurement : 917 Measure the time taken to flush X remote MAC from PBB-EVPN MAC table 918 of the DUT.Measure the time taken to relearn these X MAC in DUT. The 919 test is repeated for "N" times and the values are collected. The 920 flush rate is calculated by averaging the values obtained by "N" 921 samples. "N" is an arbitrary number to get a sufficient sample. The 922 time measured for each sample is denoted by T1,T2...Tn.The 923 measurement is carried out using external server which polls the DUT 924 using automated scripts. 926 Flush rate = (T1+T2+..Tn)/N 928 Relearning rate = (T1+T2+..Tn)/N 930 4.5. MAC Aging 932 Objective: 934 Measure the MAC aging time. 936 Topology : Topology 1 938 Procedure: 940 Confirm the DUT is up and running with PBB-EVPN. 942 Send X frames with X different source and destination MAC addresses 943 to DUT from CE using traffic generator for one vlan. 945 Ensure these X MAC addresses are learned in DUT. 947 Then stop the traffic. 949 Ensure the DUT and other devices in the test are using the default 950 timers for aging. 952 Measure the time taken to flush X MAC from DUT PBB-EVPN MAC table due 953 to aging. 955 The DUT and MHPE2 are running SA mode. 957 Measurement : 959 Measure the time taken to flush X MAC addresses due to aging. The 960 test is repeated for "N" times and the values are collected. The 961 aging is calculated averaging the values obtained by "N" samples. 962 "N" is an arbitrary number to get a sufficient sample. The time 963 measured for each sample is denoted by T1,T2...Tn. The measurement 964 is carried out using external server which polls the DUT using 965 automated scripts. 967 Aging time for X MAC in sec = (T1+T2+..Tn)/N 969 4.6. Remote MAC Aging. 971 Objective: 973 Measure the remote MAC aging time. 975 Topology : Topology 1 977 Procedure: 979 Confirm the DUT is up and running with PBB-EVPN. 981 Send X frames with X different source and destination MAC addresses 982 to DUT from SHPE3 using traffic generator for one vlan. 984 Ensure these X MAC addresses are learned in DUT. 986 Then stop the traffic. 988 Ensure the DUT and other devices in the test are using the default 989 timers for aging. 991 Measure the time taken to flush X MAC from DUT PBB-EVPN MAC table due 992 to aging. 994 The DUT and MHPE2 are running SA mode. 996 Measurement : 998 Measure the time taken to flush X remote MAC learned in DUT EVPN MAC 999 table due to aging.The test is repeated for "N" times and the values 1000 are collected. The aging is calculated by averaging the values 1001 obtained by "N" samples."N" is an arbitrary number to get a 1002 sufficient sample. The time measured for each sample is denoted by 1003 T1,T2...Tn. The measurement is carried out using external server 1004 which polls the DUT using automated scripts. 1006 Aging time for X MAC in sec = (T1+T2+..Tn)/N 1008 4.7. Local and Remote MAC Learning 1010 Objective: 1012 Measure the time taken to learn both local and remote MAC. 1014 Topology : Topology 1 1016 Procedure: 1018 Confirm the DUT is up and running with PBB-EVPN. 1020 Send X frames with X different source and destination MAC addresses 1021 to DUT from SHPE3 using traffic generator for one vlan. 1023 Send X frames with different source and destination MAC addresses 1024 from traffic generator connected to CE for one vlan. 1026 The source and destination addresses of flows must be complimentary 1027 to have unicast flows. 1029 Measure the time taken by the DUT to learn 2X in PBB-EVPN MAC table. 1031 DUT and MHPE2 are running in SA mode. 1033 Measurement : 1035 Measure the time taken to learn 2X MAC addresses in DUT PBB-EVPN MAC 1036 table. The test is repeated for "N" times and the values are 1037 collected. The MAC learning time is calculated by averaging the 1038 values obtained by "N" samples."N" is an arbitrary number to get a 1039 sufficient sample. The time measured for each sample is denoted by 1040 T1,T2...Tn. The measurement is carried out using external server 1041 which polls the DUT using automated scripts 1043 MAC learning rate = (T1+T2+..Tn)/N 1045 4.8. High Availability 1047 Objective: 1049 Measure traffic loss during routing engine failover. 1051 Topology : Topology 1 1053 Procedure: 1055 Confirm the DUT is up and running with PBB-EVPN. 1057 Send X frames from CE to DUT from traffic generator with X different 1058 source and destination MAC addresses. 1060 Send X frames from traffic generator to SHPE3 with X different source 1061 and destination MAC addresses, so that 2X MAC address will be learned 1062 in the DUT. 1064 There is a bi directional traffic flow with X pps in each direction. 1066 Ensure the DUT learn 2X MAC. 1068 Then do a routing engine fail-over. 1070 Measurement : 1072 The expectation of the test is 0 traffic loss with no change in the 1073 DF role. DUT should not withdraw any routes.But in cases where the 1074 DUT is not property synchronized between master and standby,due to 1075 that packet loss are observed. In that scenario the packet loss is 1076 measured.The test is repeated for "N" times and the values are 1077 collected.The packet loss is calculated by averaging the values 1078 obtained by "N" samples."N" is an arbitrary number to get a 1079 sufficient sample. The time measured for each sample is denoted by 1080 T1,T2...Tn. The measurement is carried out using external server 1081 which polls the DUT using automated scripts to ensure the DUT learned 1082 2X MAC. The packet drop is measured using traffic generator. 1084 Packet loss in sec with 2X MAC addresses = (T1+T2+..Tn)/N 1086 4.9. Scale 1088 Objective: 1090 Measure the scale limit of DUT for PBB-EVPN. 1092 Topology : Topology 1 1094 Procedure: 1096 Confirm the DUT is up and running with PBB-EVPN. 1098 The DUT,MHPE2 and SHPE3 are scaled to "N" PBB-EVI. 1100 Ensure routes received from MHPE2 and SHPE3 for "N" PBB-EVI in the 1101 DUT. 1103 Then increment the scale of N by 5% of N till the limit is reached. 1105 The limit is where the DUT cant learn any EVPN routes from its peers. 1107 Measurement : 1109 There should not be any loss of route types 2,3 and 4 in DUT. DUT 1110 must relearn all type 2,3 and 4 from remote routers. The DUT must be 1111 subjected to various values of N to find the optimal scale limit. 1112 The scope of the test is find out the maximum evpn instance that a 1113 DUT can hold.The measurement is carried out using external server 1114 which polls the DUT using automated scripts to find the scale limit 1115 of PBB-EVPN instances. 1117 4.10. Scale Convergence 1119 Objective: 1121 To measure the convergence time of DUT when the DUT is scaled with 1122 EVPN instance along with traffic. 1124 Topology : Topology 1 1126 Procedure: 1128 Confirm the DUT is up and running with PBB-EVPN. 1130 Scale N PBB-EVIs in DUT,SHPE3 and MHPE2. 1132 Send F frames to DUT from CE using traffic generator with X different 1133 source and destination MAC addresses for N PBB-EVI's. 1135 Send F frames from traffic generator to SHPE3 with X different source 1136 and destination MAC addresses. 1138 There will be 2X number of MAC addresses will be learned in DUT PBB- 1139 EVPN MAC table. 1141 There is a bi directional traffic flow with F pps in each direction. 1143 Then clear the BGP neighbors in the DUT. 1145 Once the BGP session is in established state in DUT. 1147 Measure the time taken to learn 2X MAC address in DUT MAC table. 1149 Measurement : 1151 The DUT must learn 2X MAC addresses. Measure the time taken to learn 1152 2X MAC in DUT.The test is repeated for "N" times and the values are 1153 collected. The convergence time is calculated by averaging the 1154 values obtained by "N" samples. "N" is an arbitrary number to get a 1155 sufficient sample.The time measured for each sample is denoted by 1156 T1,T2...Tn.The measurement is carried out using external server which 1157 polls the DUT using automated scripts. 1159 Time taken to learn 2X MAC in DUT = (T1+T2+..Tn)/N 1161 4.11. Soak Test 1163 Objective: 1165 To measure the stability of the DUT in a scaled environment with 1166 traffic. 1168 Topology : Topology 1 1170 Procedure: 1172 Confirm the DUT is up and running with PBB-EVPN. 1174 Scale N PBB-EVI's in DUT,SHPE3 and MHPE2.Send F frames to DUT from CE 1175 using traffic generator with different X source and destination MAC 1176 addresses for N EVI's. 1178 Send F frames from traffic generator to SHPE3 with X different source 1179 and destination MAC addresses. 1181 There will be 2X number of MAC addresses will be learned in DUT PBB- 1182 EVPN MAC table. 1184 There is a bi directional traffic flow with F pps in each direction. 1186 The DUT must run with traffic for 24 hours. 1188 Every hour check for memory leak in PBB-EVPN process,CPU usage and 1189 crashes in DUT. 1191 Measurement : 1193 Take the hourly reading of CPU, process memory. There should not be 1194 any leak, crashes, CPU spikes. Th CPU spike is determined as the CPU 1195 usage which shoots at 40 to 50 percent of the average usage. The 1196 average value vary from device to device. Memory leak is determined 1197 by increase usage of the memory for PBB-EVPN process. The 1198 expectation is under steady state the memory usage for PBB-EVPN 1199 process should not increase. The measurement is carried out using 1200 external server which polls the DUT using automated scripts which 1201 captures the CPU usage and process memory. 1203 5. Acknowledgements 1205 We would like to thank Fioccola Giuseppe of Telecom Italia reviewing 1206 our draft and commenting it. We would like to thank Sarah Banks for 1207 guiding and mentoring us. 1209 6. IANA Considerations 1211 This memo includes no request to IANA. 1213 7. Security Considerations 1215 The benchmarking tests described in this document are limited to the 1216 performance characterization of controllers in a lab environment with 1217 isolated networks. The benchmarking network topology will be an 1218 independent test setup and MUST NOT be connected to devices that may 1219 forward the test traffic into a production network or misroute 1220 traffic to the test management network. Further, benchmarking is 1221 performed on a "black-box" basis, relying solely on measurements 1222 observable external to the controller. Special capabilities SHOULD 1223 NOT exist in the controller specifically for benchmarking purposes. 1224 Any implications for network security arising from the controller 1225 SHOULD be identical in the lab and in production networks. 1227 8. References 1229 8.1. Normative References 1231 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1232 Requirement Levels", BCP 14, RFC 2119, 1233 DOI 10.17487/RFC2119, March 1997, 1234 . 1236 [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for 1237 Network Interconnect Devices", RFC 2544, 1238 DOI 10.17487/RFC2544, March 1999, 1239 . 1241 [RFC2899] Ginoza, S., "Request for Comments Summary RFC Numbers 1242 2800-2899", RFC 2899, DOI 10.17487/RFC2899, May 2001, 1243 . 1245 8.2. Informative References 1247 [RFC7432] Sajassi, A., Ed., Aggarwal, R., Bitar, N., Isaac, A., 1248 Uttaro, J., Drake, J., and W. Henderickx, "BGP MPLS-Based 1249 Ethernet VPN", RFC 7432, DOI 10.17487/RFC7432, February 1250 2015, . 1252 [RFC7623] Sajassi, A., Ed., Salam, S., Bitar, N., Isaac, A., and W. 1253 Henderickx, "Provider Backbone Bridging Combined with 1254 Ethernet VPN (PBB-EVPN)", RFC 7623, DOI 10.17487/RFC7623, 1255 September 2015, . 1257 Appendix A. Appendix 1259 Authors' Addresses 1261 Sudhin Jacob (editor) 1262 Juniper Networks 1263 Bangalore 1264 India 1266 Phone: +91 8061212543 1267 Email: sjacob@juniper.net 1269 Kishore Tiruveedhula 1270 Juniper Networks 1271 10 Technology Park Dr 1272 Westford, MA 01886 1273 USA 1275 Phone: +1 9785898861 1276 Email: kishoret@juniper.net