idnits 2.17.1 draft-ietf-bmwg-evpntest-07.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- == It seems as if not all pages are separated by form feeds - found 0 form feeds but 28 pages Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** There are 32 instances of too long lines in the document, the longest one being 16 characters in excess of 72. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 215 has weird spacing: '...nerator sende...' == Line 216 has weird spacing: '...eceiver of la...' == The document seems to lack the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords -- however, there's a paragraph with a matching beginning. Boilerplate error? (The document does seem to have the reference to RFC 2119 which the ID-Checklist requires). -- The document date (February 2, 2021) is 1179 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Missing Reference: 'RFC7632' is mentioned on line 110, but not defined == Unused Reference: 'RFC2544' is defined on line 1228, but no explicit reference was found in the text == Unused Reference: 'RFC2899' is defined on line 1233, but no explicit reference was found in the text == Unused Reference: 'RFC7623' is defined on line 1248, but no explicit reference was found in the text Summary: 1 error (**), 0 flaws (~~), 9 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Internet Engineering Task Force S. Jacob, Ed. 3 Internet-Draft K. Tiruveedhula 4 Intended status: Informational Juniper Networks 5 Expires: August 6, 2021 February 2, 2021 7 Benchmarking Methodology for EVPN and PBB-EVPN 8 draft-ietf-bmwg-evpntest-07 10 Abstract 12 This document defines methodologies for benchmarking EVPN and PBB- 13 EVPN performance. EVPN is defined in RFC 7432, and is being deployed 14 in Service Provider networks. Specifically, this document defines 15 the methodologies for benchmarking EVPN/PBB-EVPN convergence, data 16 plane performance, and control plane performance. 18 Status of This Memo 20 This Internet-Draft is submitted in full conformance with the 21 provisions of BCP 78 and BCP 79. 23 Internet-Drafts are working documents of the Internet Engineering 24 Task Force (IETF). Note that other groups may also distribute 25 working documents as Internet-Drafts. The list of current Internet- 26 Drafts is at https://datatracker.ietf.org/drafts/current/. 28 Internet-Drafts are draft documents valid for a maximum of six months 29 and may be updated, replaced, or obsoleted by other documents at any 30 time. It is inappropriate to use Internet-Drafts as reference 31 material or to cite them other than as "work in progress." 33 This Internet-Draft will expire on August 6, 2021. 35 Copyright Notice 37 Copyright (c) 2021 IETF Trust and the persons identified as the 38 document authors. All rights reserved. 40 This document is subject to BCP 78 and the IETF Trust's Legal 41 Provisions Relating to IETF Documents 42 (https://trustee.ietf.org/license-info) in effect on the date of 43 publication of this document. Please review these documents 44 carefully, as they describe your rights and restrictions with respect 45 to this document. Code Components extracted from this document must 46 include Simplified BSD License text as described in Section 4.e of 47 the Trust Legal Provisions and are provided without warranty as 48 described in the Simplified BSD License. 50 Table of Contents 52 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 53 1.1. Requirements Language . . . . . . . . . . . . . . . . . . 3 54 1.2. Terminologies . . . . . . . . . . . . . . . . . . . . . . 3 55 2. Test Topology . . . . . . . . . . . . . . . . . . . . . . . . 4 56 3. Test Cases for EVPN Benchmarking . . . . . . . . . . . . . . 7 57 3.1. Data Plane MAC Learning . . . . . . . . . . . . . . . . . 7 58 3.2. Control Plane MAC Learning . . . . . . . . . . . . . . . 8 59 3.3. MAC Flush-Local Link Failure and Relearning . . . . . . . 9 60 3.4. MAC Flush-Remote Link Failure and Relearning. . . . . . . 10 61 3.5. MAC Aging . . . . . . . . . . . . . . . . . . . . . . . . 11 62 3.6. Remote MAC Aging . . . . . . . . . . . . . . . . . . . . 11 63 3.7. Control and Data plane MAC Learning . . . . . . . . . . . 12 64 3.8. High Availability. . . . . . . . . . . . . . . . . . . . 13 65 3.9. ARP/ND Scale . . . . . . . . . . . . . . . . . . . . . . 14 66 3.10. Scaling of Services . . . . . . . . . . . . . . . . . . . 15 67 3.11. Scale Convergence . . . . . . . . . . . . . . . . . . . . 15 68 3.12. SOAK Test. . . . . . . . . . . . . . . . . . . . . . . . 16 69 4. Test Cases for PBB-EVPN Benchmarking . . . . . . . . . . . . 17 70 4.1. Data Plane Local MAC Learning . . . . . . . . . . . . . . 17 71 4.2. Data Plane Remote MAC Learning . . . . . . . . . . . . . 18 72 4.3. MAC Flush-Local Link Failure . . . . . . . . . . . . . . 19 73 4.4. MAC Flush-Remote Link Failure . . . . . . . . . . . . . . 20 74 4.5. MAC Aging . . . . . . . . . . . . . . . . . . . . . . . . 21 75 4.6. Remote MAC Aging. . . . . . . . . . . . . . . . . . . . . 21 76 4.7. Local and Remote MAC Learning . . . . . . . . . . . . . . 22 77 4.8. High Availability . . . . . . . . . . . . . . . . . . . . 23 78 4.9. Scale . . . . . . . . . . . . . . . . . . . . . . . . . . 24 79 4.10. Scale Convergence . . . . . . . . . . . . . . . . . . . . 25 80 4.11. Soak Test . . . . . . . . . . . . . . . . . . . . . . . . 26 81 5. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 26 82 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 27 83 7. Security Considerations . . . . . . . . . . . . . . . . . . . 27 84 8. References . . . . . . . . . . . . . . . . . . . . . . . . . 27 85 8.1. Normative References . . . . . . . . . . . . . . . . . . 27 86 8.2. Informative References . . . . . . . . . . . . . . . . . 27 87 Appendix A. Appendix . . . . . . . . . . . . . . . . . . . . . . 28 88 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 28 90 1. Introduction 92 EVPN is defined in RFC 7432, and describes BGP MPLS based Ethernet 93 VPNs (EVPN). PBB-EVPN is defined in RFC 7623, discusses how Ethernet 94 Provider backbone Bridging can be combined with EVPNs to provide a 95 new/combined solution. This draft defines methodologies that can be 96 used to benchmark both RFC 7432 and RFC 7623 solutions. Further, 97 this draft provides methodologies for benchmarking the performance of 98 EVPN data and control planes, MAC learning, MAC flushing, MAC aging, 99 convergence, high availability, and scale. 101 1.1. Requirements Language 103 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 104 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 105 document are to be interpreted as described in RFC 8174 [RFC8174]. 107 1.2. Terminologies 109 Most of the terminology used in this documents comes from [RFC7432] 110 and [RFC7632]. 112 All-Active Redundancy Mode: When all PEs attached to an Ethernet 113 segment are allowed to forward known unicast traffic to/from that 114 Ethernet segment for a given VLAN, then the Ethernet segment is 115 defined to be operating in All-Active redundancy mode. 117 AA: All Active mode 119 CE: Customer Router/Devices/Switch. 121 DF: Designated Forwarder 123 DUT: Device under test. 125 Ethernet Segment (ES): When a customer site (device or network) is 126 connected to one or more PEs via a set of Ethernet links, then that 127 set of links is referred to as an 'Ethernet segment'. 129 EVI: An EVPN instance spanning the Provider Edge (PE) devices 130 participating in that EVPN. 132 Ethernet Segment Identifier (ESI): A unique non-zero identifier that 133 identifies an Ethernet segment is called an 'Ethernet Segment 134 Identifier'. 136 Ethernet Tag: An Ethernet tag identifies a particular broadcast 137 domain, e.g., a VLAN. An EVPN instance consists of one or more 138 broadcast domains. 140 Interface: Physical interface of a router/switch. 142 IRB: Integrated routing and bridging interface 144 MAC: Media Access Control addresses on a PE. 146 MHPE2: Multi homed Provider Edge router 2. 148 MHPE1: Multi homed Provider Edge router 1. 150 SHPE3: Single homed Provider Edge Router 3. 152 PE: Provider Edge device. 154 P: Provider Router. 156 RR: Route Reflector. 158 RT: Traffic Generator. 160 Sub Interface: Each physical Interfaces is subdivided in to a set of 161 Logical units. 163 SA: Single Active 165 Single-Active Redundancy Mode: When a single PE (among all the PEs 166 attached to an Ethernet segment) is the only PE allowed to forward 167 traffic to/from a given Ethernet segment for a given VLAN, then that 168 Ethernet segment is defined to be operating in Single-Active 169 redundancy mode. 171 2. Test Topology 173 There are five routers in the Test setup. SHPE3, RR/P, MHPE1 and 174 MHPE2 emulating a service provider network. CE is a customer device 175 connected to MHPE1 and MHPE2. it is configured with bridge domains in 176 multiple VLANS. The traffic generator is connected to the CE and 177 SHPE3. The MHPE1 acts as DUT. The traffic generator will be used as 178 sender and receiver of traffic. The test measurements are taken from 179 the DUT. MHPE1 and MHPE2 are multi-homed routers connected to CE 180 running single active mode. The traffic generator will be generating 181 traffic at 10% of the line rate. 183 +----------------+ +---------------------------+ 184 | | | | 185 | | |Traffic Generator sender/ | 186 | SHPE3 | |receiver of layer 2 traffic| 187 | +----------------| with multiple Vlans | 188 | | +---------------------------+ 189 +---------+------+ 190 | Core Link 191 | 192 +--------+-----+ 193 | | 194 | RR/P +----------------------+ 195 | | Core link | 196 | | | 197 +--+-----------+ | 198 | | 199 | core link | 200 | | 201 +-------------+---+ +------+------------+ 202 | | | | 203 | | | | 204 | MHPE1(DUT) | | MHPE2 | 205 | | | | 206 | | | | 207 +-----------------+------+ +-----+-------------------+ 208 | | 209 PE-CE link | | PE-CE link 210 | | 211 | | 212 | | 213 | | 214 +-----+----------+----+ +----------------------------+ 215 | | | Traffic Generator sender/ | 216 | | |receiver of layer 2 traffic| 217 | CE/Layer 2 bridge +-----------+ with multiple Vlans | 218 | | +----------------------------+ 219 | | 220 +---------------------+ 222 Topology 1 224 Test Setup 226 Figure 1 228 +--------------+------------------+-------------+---------+-----------+----------------+ 229 | | | | | | | 230 | Mode | Test | Traffic | Sender | Receiver | | 231 | | | Direction | | | | 232 +--------------------------------------------------------------------------------------+ 233 | | | | | | | 234 |Single Active | Local MAC | | CE | SHPFE3 |Layer 2 traffic | 235 | | Learning | Uni | | | | 236 | | | | | | multiple MAC | 237 +--------------------------------------------------------------------------------------+ 238 | | | | | | | 239 |Single Active | Remote MAC | | | CE |Layer 2 traffic | 240 | | Learning | Uni | SHPE3 | | | 241 | | | | | |multiple MAC | 242 +--------------------------------------------------------------------------------------+ 243 | | | | | | | 244 |Single Active | Scale Convergence| Bi | | CE/SHPE3 | | 245 | | | |CE/SHPE3 | |Layer 2 traffic | 246 | | Local& Remote | | | |multiple MAC & | 247 | | Learning | | | | vlans | 248 +--------------+------------------+-------------+---------+-----------+----------------+ 250 Table showing the traffic directions of various EVPN/PBB-EVPN 251 benchmarking test cases. Depending on the test scenario, the traffic 252 can be uni-directional or bi-directional (configured in the traffic 253 generator). 255 Figure 2 257 Test Setup Configurations: 259 SHPE3 is configured with Interior Gateway protocols like OPSF or IS- 260 IS for underlay, LDP for MPLS support, Interior Border Gateway with 261 EVPN address family for overlay support. This router must be 262 configured with N EVPN/PBB-EVPN instances for testing. Traffic 263 generator is connected to this router for sending and receiving 264 traffic. 266 RR is configured with Interior Gateway protocols like OPSF or IS-IS 267 for underlay, LDP for MPLS support, Interior Border Gateway with EVPN 268 address family for overlay support. This router function as both 269 provider router and a route reflector. 271 MHPE1 is configured with Interior Gateway protocols like OPSF or IS- 272 IS for underlay, LDP for MPLS support, Interior Border Gateway with 273 EVPN address family for overlay support. This router must be 274 configured with N EVPN/PBB-EVPN instances for testing. This router 275 is configured with ESI per vlan or ESI per interface. It is 276 functioning as multi homing PE working on Single Active EVPN mode. 277 This router serves as the DUT and it is connected to CE. MHPE1 is 278 acting as DUT for all the test cases. 280 MHPE2 is configured with Interior Gateway protocols like OPSF or IS- 281 IS for underlay, LDP for MPLS support,Interior Border Gateway with 282 EVPN address family for overlay support. This router must be 283 configured with N EVPN/PBB-EVPN instances for testing. This router 284 is configured with ESI per vlan or ESI per interface. It is 285 functioning as multi homing PE working on Single Active EVPN mode. 286 It is connected to CE. 288 CE is acting as bridge configured with multiple vlans. The same 289 vlans are configured on MHPE1,MHPE2,SHPE3. traffic generator is 290 connected to CE. the traffic generator acts as sender or receiver of 291 traffic. 293 Depending up on the test scenarios the traffic generators will be 294 used to generate uni directional or bi directional flows. 296 The above configuration will be serving as the base configuration for 297 all test cases. 299 The X is used as variable to denote scale factor of the testing 300 parameters.It must be in the multiples of 100. 302 3. Test Cases for EVPN Benchmarking 304 3.1. Data Plane MAC Learning 306 Objective: 308 Measure the time taken to learn the Data Plane MAC in DUT. 310 Topology : Topology 1 312 Procedure: 314 The data plane MAC learning can be measured using the parameters 315 defined in RFC 2889 section 5.8. 317 Confirm the DUT is up and running with EVPN. 319 Traffic generator connected to CE must send frames with X different 320 source and destination MAC address for one vlan, the same vlan must 321 be present in all the devices except RR. 323 Send X unicast frames from CE to MHPE1(DUT) for one EVPN instance 324 working in SA mode. 326 The DUT will learn these X MAC in data plane. 328 Measurement : 330 Measure the time taken to learn X MAC locally in DUT evpn MAC table. 331 The data plane measurement is taken by considering DUT as black box. 332 The range of MAC are known from traffic generator, the same must be 333 learned in DUT, the time taken to learn X MAC is measured. The 334 measurement is carried out using external server which polls the DUT 335 using automated scripts. 337 The test is repeated for N times and the values are collected. The 338 MAC learning rate is calculated by averaging the values obtained from 339 N samples. N is an arbitrary number to get a sufficient sample. The 340 time measured for each sample is denoted by T1,T2...Tn. 342 MAC learning rate = (T1+T2+..Tn)/N 344 3.2. Control Plane MAC Learning 346 Objective: 348 Measure the time taken to learn the control plane MAC. 350 Topology : Topology 1 352 Procedure: 354 Confirm the DUT is up and running with EVPN. 356 Traffic generator connected to SHPE3 must send frames with X 357 different source and destination MAC address for one vlan, the same 358 vlan must be present in all the devices except RR. 360 Ensure the frames must be destined to one EVPN instance. 362 The DUT will learn these X MAC in control plane. 364 Measurement : 366 Measure the time taken by the DUT to learn the X MAC in the data 367 plane. The test is repeated for N times and the values are 368 collected. The remote MAC learning rate is calculated by averaging 369 the values obtained from N samples. N is an arbitrary number to get 370 a sufficient sample. The time measured for each sample is denoted by 371 T1,T2...Tn. The measurement is carried out using external server 372 which polls the DUT using automated scripts. 374 MAC learning rate = (T1+T2+..Tn)/N 376 3.3. MAC Flush-Local Link Failure and Relearning 378 Objective: 380 Measure the time taken to flush the Data Plane MAC and the time taken 381 to relearn the same amount of MAC. 383 Topology : Topology 1 385 Procedure: 387 Confirm the DUT is up and running with EVPN. 389 Send X frames with X different source and destination MAC addresses 390 to DUT from CE using traffic generator for one vlan. 392 Ensure the DUT learns all X MAC addresses in data plane. 394 Fail the DUT-CE link and measure the time taken to flush these X MAC 395 from the EVPN MAC table. 397 Bring up the link which was made Down(the link between DUT and CE). 398 Measure time taken by the DUT to relearn these X MAC. 400 The DUT and MHPE2 are running SA mode. 402 Measurement : 404 Measure the time taken for flushing these X MAC addresses. Measure 405 the time taken to relearn these X MAC in DUT. The test is repeated 406 for N times and the values are collected. The flush and the 407 relearning time is calculated by averaging the values obtained by N 408 samples. N is an arbitrary number to get a sufficient sample. The 409 time measured for each sample is denoted by T1,T2...Tn. The 410 measurement is carried out using external server which polls the DUT 411 using automated scripts. 413 Flush rate = (T1+T2+..Tn)/N 415 Relearning rate = (T1+T2+..Tn)/N 417 3.4. MAC Flush-Remote Link Failure and Relearning. 419 Objective: 421 Measure the time taken to flush the Control plane MAC learned in DUT 422 during remote link failure and the time taken to relearn. 424 Topology : Topology 1 426 Procedure: 428 confirm the DUT is up and running with EVPN. 430 Send X frames with X different source and destination MAC addresses 431 to DUT from SHPE3 using traffic generator for one vlan. 433 Bring down the link between SHPE3 and traffic generator. 435 SHPE3 will withdraw the routes from DUT due to link failure. 437 Measure the time taken to flush the DUT EVPN MAC table. The DUT and 438 MHPE2 are running SA mode. 440 Bring up the link which was made Down(the link between SHPE3 and 441 traffic generator). 443 Measure time taken by the DUT to relearn these X MAC from control 444 plane. 446 Measurement : 448 Measure the time taken to flush X remote MAC from EVPN MAC table of 449 the DUT. Measure the time taken to relearn these X MAC in DUT. The 450 test is repeated for N times and the values are collected. The flush 451 rate is calculated by averaging the values obtained by N samples. N 452 is an arbitrary number to get a sufficient sample. The time measured 453 for each sample is denoted by T1,T2...Tn. The measurement is carried 454 out using external server which polls the DUT using automated 455 scripts. 457 Flush rate = (T1+T2+..Tn)/N 459 Relearning rate = (T1+T2+..Tn)/N 461 3.5. MAC Aging 463 Objective: 465 To measure the MAC aging time. 467 Topology : Topology 1 469 Procedure: 471 Confirm the DUT is up and running with EVPN. 473 Send X frames with X different source and destination MAC addresses 474 to DUT from CE using traffic generator for one vlan. 476 Ensure these X MAC addresses are learned in DUT. 478 Then stop the traffic. 480 Ensure the DUT and other devices in the test are using the default 481 timers for aging. 483 Measure the time taken to flush X MAC from DUT EVPN MAC table due to 484 aging. 486 The DUT and MHPE2 are running SA mode. 488 Measurement : 490 Measure the time taken to flush X MAC addresses due to aging. The 491 test is repeated for N times and the values are collected. The aging 492 is calculated by averaging the values obtained by N samples. N is an 493 arbitrary number to get a sufficient sample. The time measured for 494 each sample is denoted by T1,T2...Tn. The measurement is carried out 495 using external server which polls the DUT using automated scripts. 497 Aging time for X MAC in sec = (T1+T2+..Tn)/N 499 3.6. Remote MAC Aging 501 Objective: 503 Measure the control plane learned MAC aging time. 505 Topology : Topology 1 507 Procedure: 509 Confirm the DUT is up and running with EVPN. 511 Send X frames with X different source and destination MAC addresses 512 to DUT from SHPE3 using traffic generator for one vlan. 514 Ensure these X MAC addresses are learned in DUT via control plane. 516 Then stop the traffic. 518 Ensure the DUT and other devices in the test are using the default 519 timers for aging. 521 Measure the time taken to flush X MAC from DUT EVPN MAC table due to 522 aging. 524 The DUT and MHPE2 are running SA mode. 526 Measurement : 528 Measure the time taken to flush X remote MAC learned in DUT EVPN MAC 529 table due to aging. The test is repeated for N times and the values 530 are collected. The aging is calculated by averaging the values 531 obtained by N samples. N is an arbitrary number to get a sufficient 532 sample. The time measured for each sample is denoted by T1,T2...Tn. 533 The measurement is carried out using external server which polls the 534 DUT using automated scripts. 536 Aging time for X MAC in sec = (T1+T2+..Tn)/N 538 3.7. Control and Data plane MAC Learning 540 Objective: 542 To record the time taken to learn both local and remote MAC. 544 Topology : Topology 1 546 Procedure: 548 Confirm the DUT is up and running with EVPN. 550 Send X frames with X different source and destination MAC addresses 551 to DUT from SHPE3 using traffic generator for one vlan. 553 Send X frames with different source and destination MAC addresses 554 from traffic generator connected to CE for one vlan. 556 The source and destination addresses of flows must be complimentary 557 to have unicast flows. 559 Measure the time taken by the DUT to learn 2X in EVPN MAC table. 561 DUT and MHPE2 are running in SA mode. 563 Measurement : 565 Measure the time taken to learn 2X MAC addresses in DUT EVPN MAC 566 table. The test is repeated for N times and the values are 567 collected. The MAC learning time is calculated by averaging the 568 values obtained by N samples. N is an arbitrary number to get a 569 sufficient sample. The time measured for each sample is denoted by 570 T1,T2...Tn. The measurement is carried out using external server 571 which polls the DUT using automated scripts 573 MAC learning rate = (T1+T2+..Tn)/N 575 3.8. High Availability. 577 Objective: 579 Measure traffic loss during routing engine fail over. 581 Topology : Topology 1 583 Procedure: 585 Confirm the DUT is up and running with EVPN. 587 Send X frames from CE to DUT from traffic generator with X different 588 source and destination MAC addresses. 590 Send X frames from traffic generator to SHPE3 with X different source 591 and destination MAC addresses, so that 2X MAC address will be learned 592 in the DUT. 594 There is a bi directional traffic flow with X pps in each direction. 596 Ensure the DUT learn 2X MAC. 598 Then do a routing engine fail-over. 600 Measurement : 602 The expectation of the test is 0 traffic loss with no change in the 603 DF role. DUT should not withdraw any routes. But in cases where the 604 DUT is not property synchronized between master and standby, due to 605 that packet loss are observed. In that scenario the packet loss is 606 measured.The test is repeated for N times and the values are 607 collected. The packet loss is calculated by averaging the values 608 obtained by N samples. N is an arbitrary number to get a sufficient 609 sample. The time measured for each sample is denoted by T1,T2...Tn. 610 The measurement is carried out using external server which polls the 611 DUT using automated scripts to ensure the DUT learned 2X MAC. The 612 packet drop is measured using traffic generator. 614 Packet loss in sec with 2X MAC addresses = (T1+T2+..Tn)/N 616 3.9. ARP/ND Scale 618 Measure the DUT scaling limit of ARP/ND. 620 Objective: 622 Measure the ARP/ND scale of the DUT. 624 Topology : Topology 1 626 Procedure: 628 Confirm the DUT is up and running with EVPN. 630 Send X arp/neighbor discovery(ND) from the traffic generator to DUT 631 with different sender ip/ipv6,MAC addresses to the target IRB address 632 configured in EVPN instance. 634 The EVPN instance learns the MAC+ip and MAC+ipv6 addresses from these 635 request and advertise as type 2 MAC+ip/MAC+ipv6 route to remote 636 provide edge routers which have same EVPN configurations. 638 The value of X must be increased at a incremental value of 5% of X, 639 till the limit is reached. The limit is where the DUT cant learn any 640 more type 2 MAC+ip/MAC+ipv6.The test must be separately conducted for 641 arp and ND. 643 Measurement : 645 Measure the scale limit of type 2 MAC+ip/MAC+ipv6 route which DUT can 646 learn. The test is repeated for N times and the values are 647 collected. The scale limit is calculated by averaging the values 648 obtained by N samples for both MAC+ip and MAC+ipv6. N is an 649 arbitrary number to get a sufficient sample. The scale value 650 obtained by each sample be v1,v2..vn. The measurement is carried out 651 using external server which polls the DUT using automated scripts to 652 find the scale limit of MAC+ipv4/MAC+ipv6. 654 Scale limit for MAC+ip = (v1+v2+..vn)/N 656 Scale limit for MAC+ipv6 = (v1+v2+..vn)/N 658 3.10. Scaling of Services 660 Objective: 662 Measure the scale of EVPN instances that a DUT can hold. 664 Topology : Topology 1 666 Procedure: 668 Confirm the DUT is up and running with EVPN. 670 The DUT, MHPE2 and SHPE3 are scaled to N EVI. 672 Ensure routes received from MHPE2 and SHPE3 for N EVI in the DUT. 674 Then increment the scale of N by 5% of N till the limit is reached. 676 The limit is where the DUT cant learn any EVPN routes from its peers. 678 Measurement : 680 There should not be any loss of route types 1,2,3 and 4 in DUT. DUT 681 must relearn all type 1, 2, 3 and 4 from remote routers. The DUT 682 must be subjected to various values of N to find the optimal scale 683 limit. The scope of the test is find out the maximum evpn instance 684 that a DUT can hold. The measurement is carried out using external 685 server which polls the DUT using automated scripts to find the scale 686 limit of EVPN instances. 688 3.11. Scale Convergence 690 Objective: 692 Measure the convergence time of DUT when the DUT is scaled with EVPN 693 instance along with traffic. 695 Topology : Topology 1 696 Procedure: 698 Confirm the DUT is up and running with EVPN. 700 Scale N EVIs in DUT,SHPE3 and MHPE2. 702 Send F frames to DUT from CE using traffic generator with X different 703 source and destination MAC addresses for N EVI's. 705 Send F frames from traffic generator to SHPE3 with X different source 706 and destination MAC addresses. 708 There will be 2X number of MAC addresses will be learned in DUT EVPN 709 MAC table. 711 There is a bi directional traffic flow with F pps in each direction. 713 Then clear the BGP neighbors in the DUT. 715 Once the BGP session is in established state in DUT. 717 Measure the time taken to learn 2X MAC address in DUT MAC table. 719 Measurement : 721 The DUT must learn 2X MAC addresses. Measure the time taken to learn 722 2X MAC in DUT. The test is repeated for N times and the values are 723 collected. The convergence time is calculated by averaging the 724 values obtained by N samples. N is an arbitrary number to get a 725 sufficient sample.The time measured for each sample is denoted by 726 T1,T2...Tn. The measurement is carried out using external server 727 which polls the DUT using automated scripts. 729 Time taken to learn 2X MAC in DUT = (T1+T2+..Tn)/N 731 3.12. SOAK Test. 733 Objective: 735 This test is carried out to measure the stability of the DUT in a 736 scaled environment with traffic over a period of time "T'". In each 737 interval "t1" the DUT CPU usage, memory usage are measured. The DUT 738 is checked for any crashes during this time period. 740 Topology : Topology 1 742 Procedure: 744 Confirm the DUT is up and running with EVPN. 746 Scale N EVI's in DUT, SHPE3 and MHPE2.Send F frames to DUT from CE 747 using traffic generator with different X source and destination MAC 748 addresses for N EVI's. 750 Send F frames from traffic generator to SHPE3 with X different source 751 and destination MAC addresses. 753 There will be 2X number of MAC addresses will be learned in DUT EVPN 754 MAC table. 756 There is a bi directional traffic flow with F pps in each direction. 758 The DUT must run with traffic for 24 hours. 760 Every hour check for memory leak in EVPN process, CPU usage and 761 crashes in DUT. 763 Measurement : 765 Take the hourly reading of CPU, process memory. There should not be 766 any leak, crashes, CPU spikes. The CPU spike is determined as the 767 CPU usage which shoots at 40 to 50 percent of the average usage. The 768 average value vary from device to device. Memory leak is determined 769 by increase usage of the memory for EVPN process. The expectation is 770 under steady state the memory usage for EVPN process should not 771 increase. The measurement is carried out using external server which 772 polls the DUT using automated scripts which captures the CPU usage 773 and process memory. 775 4. Test Cases for PBB-EVPN Benchmarking 777 4.1. Data Plane Local MAC Learning 779 Objective: 781 Measure the time taken to learn the Data Plane MAC in DUT. 783 Topology : Topology 1 785 Procedure: 787 Confirm the DUT is up and running with PBB-EVPN. 789 Traffic generator connected to CE must send frames with X different 790 source and destination MAC address for one vlan, the same vlan must 791 be present in all the devices except RR. 793 Send X unicast frames from CE to MHPE1(DUT) for one PBB-EVPN instance 794 working in SA mode. 796 The DUT will learn these X MAC in data plane. 798 Measurement : 800 Measure the time taken to learn X MAC locally in DUT PBB-EVPN MAC 801 table. The data plane measurement is taken by considering DUT as 802 black box. The range of MAC are known from traffic generator,the 803 same must be learned in DUT, the time taken to learn X MAC is 804 measured.The measurement is carried out using external server which 805 polls the DUT using automated scripts. 807 The test is repeated for N times and the values are collected. The 808 MAC learning rate is calculated by averaging the values obtained from 809 N samples. N is an arbitrary number to get a sufficient sample. The 810 time measured for each sample is denoted by T1,T2...Tn. 812 MAC learning rate = (T1+T2+..Tn)/N 814 4.2. Data Plane Remote MAC Learning 816 Objective: 818 To Record the time taken to learn the remote MAC. 820 Topology : Topology 1 822 Procedure: 824 Confirm the DUT is up and running with PBB-EVPN. 826 Traffic generator connected to SHPE3 must send frames with X 827 different source and destination MAC address for one vlan, the same 828 vlan must be present in all the devices except RR. 830 Ensure the frames must be destined to one PBB-EVPN instance. 832 The DUT will learn these X MAC in data plane. 834 Measurement : 836 Measure the time taken by the DUT to learn the X MAC in the data 837 plane. The test is repeated for N times and the values are 838 collected. The remote MAC learning rate is calculated by averaging 839 the values obtained from N samples. N is an arbitrary number to get 840 a sufficient sample. The time measured for each sample is denoted by 841 T1,T2...Tn. The measurement is carried out using external server 842 which polls the DUT using automated scripts. 844 MAC learning rate = (T1+T2+..Tn)/N 846 4.3. MAC Flush-Local Link Failure 848 Objective: 850 Measure the time taken to flush the locally learned MAC and the time 851 taken to relearn the same amount of MAC. 853 Topology : Topology 1 855 Procedure: 857 Confirm the DUT is up and running with PBB-EVPN. 859 Send X frames with X different source and destination MAC addresses 860 to DUT from CE using traffic generator for one vlan. 862 Ensure the DUT learns all X MAC addresses in data plane. 864 Fail the DUT-CE link and measure the time taken to flush these X MAC 865 from the PBB-EVPN MAC table. 867 Bring up the link which was made Down(the link between DUT and 868 CE).Measure time taken by the DUT to relearn these X MAC. 870 The DUT and MHPE2 are running SA mode. 872 Measurement : 874 Measure the time taken for flushing these X MAC addresses. Measure 875 the time taken to relearn these X MAC in DUT. The test is repeated 876 for N times and the values are collected. The flush and the 877 relearning time is calculated by averaging the values obtained by N 878 samples. N is an arbitrary number to get a sufficient sample. The 879 time measured for each sample is denoted by T1,T2...Tn. The 880 measurement is carried out using external server which polls the DUT 881 using automated scripts. 883 Flush rate = (T1+T2+..Tn)/N 885 Relearning rate = (T1+T2+..Tn)/N 887 4.4. MAC Flush-Remote Link Failure 889 Objective: 891 Measure the time taken to flush the remote MAC learned in DUT due to 892 remote link failure and relearning it. 894 Topology : Topology 1 896 Procedure: 898 confirm the DUT is up and running with PBB-EVPN. 900 Send X frames with X different source and destination MAC addresses 901 to DUT from SHPE3 using traffic generator for one vlan. 903 Bring down the link between SHPE3 and traffic generator. 905 Measure the time taken to flush the DUT PBB-EVPN MAC table. The DUT 906 and MHPE2 are running SA mode. 908 Bring up the link which was made Down(the link between SHPE3 and 909 traffic generator). 911 Measure time taken by the DUT to relearn these X MAC 913 Measurement : 915 Measure the time taken to flush X remote MAC from PBB-EVPN MAC table 916 of the DUT. Measure the time taken to relearn these X MAC in DUT. 917 The test is repeated for N times and the values are collected. The 918 flush rate is calculated by averaging the values obtained by N 919 samples. N is an arbitrary number to get a sufficient sample. The 920 time measured for each sample is denoted by T1,T2...Tn. The 921 measurement is carried out using external server which polls the DUT 922 using automated scripts. 924 Flush rate = (T1+T2+..Tn)/N 926 Relearning rate = (T1+T2+..Tn)/N 928 4.5. MAC Aging 930 Objective: 932 Measure the MAC aging time. 934 Topology : Topology 1 936 Procedure: 938 Confirm the DUT is up and running with PBB-EVPN. 940 Send X frames with X different source and destination MAC addresses 941 to DUT from CE using traffic generator for one vlan. 943 Ensure these X MAC addresses are learned in DUT. 945 Then stop the traffic. 947 Ensure the DUT and other devices in the test are using the default 948 timers for aging. 950 Measure the time taken to flush X MAC from DUT PBB-EVPN MAC table due 951 to aging. 953 The DUT and MHPE2 are running SA mode. 955 Measurement : 957 Measure the time taken to flush X MAC addresses due to aging. The 958 test is repeated for N times and the values are collected. The aging 959 is calculated averaging the values obtained by N samples. N is an 960 arbitrary number to get a sufficient sample. The time measured for 961 each sample is denoted by T1,T2...Tn. The measurement is carried out 962 using external server which polls the DUT using automated scripts. 964 Aging time for X MAC in sec = (T1+T2+..Tn)/N 966 4.6. Remote MAC Aging. 968 Objective: 970 Measure the remote MAC aging time. 972 Topology : Topology 1 974 Procedure: 976 Confirm the DUT is up and running with PBB-EVPN. 978 Send X frames with X different source and destination MAC addresses 979 to DUT from SHPE3 using traffic generator for one vlan. 981 Ensure these X MAC addresses are learned in DUT. 983 Then stop the traffic. 985 Ensure the DUT and other devices in the test are using the default 986 timers for aging. 988 Measure the time taken to flush X MAC from DUT PBB-EVPN MAC table due 989 to aging. 991 The DUT and MHPE2 are running SA mode. 993 Measurement : 995 Measure the time taken to flush X remote MAC learned in DUT EVPN MAC 996 table due to aging. The test is repeated for N times and the values 997 are collected. The aging is calculated by averaging the values 998 obtained by N samples. N is an arbitrary number to get a sufficient 999 sample. The time measured for each sample is denoted by T1,T2...Tn. 1000 The measurement is carried out using external server which polls the 1001 DUT using automated scripts. 1003 Aging time for X MAC in sec = (T1+T2+..Tn)/N 1005 4.7. Local and Remote MAC Learning 1007 Objective: 1009 Measure the time taken to learn both local and remote MAC. 1011 Topology : Topology 1 1013 Procedure: 1015 Confirm the DUT is up and running with PBB-EVPN. 1017 Send X frames with X different source and destination MAC addresses 1018 to DUT from SHPE3 using traffic generator for one vlan. 1020 Send X frames with different source and destination MAC addresses 1021 from traffic generator connected to CE for one vlan. 1023 The source and destination addresses of flows must be complimentary 1024 to have unicast flows. 1026 Measure the time taken by the DUT to learn 2X in PBB-EVPN MAC table. 1028 DUT and MHPE2 are running in SA mode. 1030 Measurement : 1032 Measure the time taken to learn 2X MAC addresses in DUT PBB-EVPN MAC 1033 table. The test is repeated for N times and the values are 1034 collected. The MAC learning time is calculated by averaging the 1035 values obtained by N samples. N is an arbitrary number to get a 1036 sufficient sample. The time measured for each sample is denoted by 1037 T1,T2...Tn. The measurement is carried out using external server 1038 which polls the DUT using automated scripts 1040 MAC learning rate = (T1+T2+..Tn)/N 1042 4.8. High Availability 1044 Objective: 1046 Measure traffic loss during routing engine failover. 1048 Topology : Topology 1 1050 Procedure: 1052 Confirm the DUT is up and running with PBB-EVPN. 1054 Send X frames from CE to DUT from traffic generator with X different 1055 source and destination MAC addresses. 1057 Send X frames from traffic generator to SHPE3 with X different source 1058 and destination MAC addresses, so that 2X MAC address will be learned 1059 in the DUT. 1061 There is a bi directional traffic flow with X pps in each direction. 1063 Ensure the DUT learn 2X MAC. 1065 Then do a routing engine fail-over. 1067 Measurement : 1069 The expectation of the test is 0 traffic loss with no change in the 1070 DF role. DUT should not withdraw any routes.But in cases where the 1071 DUT is not property synchronized between master and standby, due to 1072 that packet loss are observed. In that scenario the packet loss is 1073 measured. The test is repeated for N times and the values are 1074 collected. The packet loss is calculated by averaging the values 1075 obtained by N samples. N is an arbitrary number to get a sufficient 1076 sample. The time measured for each sample is denoted by T1,T2...Tn. 1077 The measurement is carried out using external server which polls the 1078 DUT using automated scripts to ensure the DUT learned 2X MAC. The 1079 packet drop is measured using traffic generator. 1081 Packet loss in sec with 2X MAC addresses = (T1+T2+..Tn)/N 1083 4.9. Scale 1085 Objective: 1087 Measure the scale limit of DUT for PBB-EVPN. 1089 Topology : Topology 1 1091 Procedure: 1093 Confirm the DUT is up and running with PBB-EVPN. 1095 The DUT,MHPE2 and SHPE3 are scaled to N PBB-EVI. 1097 Ensure routes received from MHPE2 and SHPE3 for N PBB-EVI in the DUT. 1099 Then increment the scale of N by 5% of N till the limit is reached. 1101 The limit is where the DUT cant learn any EVPN routes from its peers. 1103 Measurement : 1105 There should not be any loss of route types 2,3 and 4 in DUT. DUT 1106 must relearn all type 2, 3 and 4 from remote routers. The DUT must 1107 be subjected to various values of N to find the optimal scale limit. 1108 The scope of the test is find out the maximum evpn instance that a 1109 DUT can hold. The measurement is carried out using external server 1110 which polls the DUT using automated scripts to find the scale limit 1111 of PBB-EVPN instances. 1113 4.10. Scale Convergence 1115 Objective: 1117 To measure the convergence time of DUT when the DUT is scaled with 1118 EVPN instance along with traffic. 1120 Topology : Topology 1 1122 Procedure: 1124 Confirm the DUT is up and running with PBB-EVPN. 1126 Scale N PBB-EVIs in DUT,SHPE3 and MHPE2. 1128 Send F frames to DUT from CE using traffic generator with X different 1129 source and destination MAC addresses for N PBB-EVI's. 1131 Send F frames from traffic generator to SHPE3 with X different source 1132 and destination MAC addresses. 1134 There will be 2X number of MAC addresses will be learned in DUT PBB- 1135 EVPN MAC table. 1137 There is a bi directional traffic flow with F pps in each direction. 1139 Then clear the BGP neighbors in the DUT. 1141 Once the BGP session is in established state in DUT. 1143 Measure the time taken to learn 2X MAC address in DUT MAC table. 1145 Measurement : 1147 The DUT must learn 2X MAC addresses. Measure the time taken to learn 1148 2X MAC in DUT. The test is repeated for N times and the values are 1149 collected. The convergence time is calculated by averaging the 1150 values obtained by N samples. N is an arbitrary number to get a 1151 sufficient sample.The time measured for each sample is denoted by 1152 T1,T2...Tn. The measurement is carried out using external server 1153 which polls the DUT using automated scripts. 1155 Time taken to learn 2X MAC in DUT = (T1+T2+..Tn)/N 1157 4.11. Soak Test 1159 Objective: 1161 To measure the stability of the DUT in a scaled environment with 1162 traffic. 1164 Topology : Topology 1 1166 Procedure: 1168 Confirm the DUT is up and running with PBB-EVPN. 1170 Scale N PBB-EVI's in DUT,SHPE3 and MHPE2.Send F frames to DUT from CE 1171 using traffic generator with different X source and destination MAC 1172 addresses for N EVI's. 1174 Send F frames from traffic generator to SHPE3 with X different source 1175 and destination MAC addresses. 1177 There will be 2X number of MAC addresses will be learned in DUT PBB- 1178 EVPN MAC table. 1180 There is a bi directional traffic flow with F pps in each direction. 1182 The DUT must run with traffic for 24 hours. 1184 Every hour check for memory leak in PBB-EVPN process,CPU usage and 1185 crashes in DUT. 1187 Measurement : 1189 Take the hourly reading of CPU, process memory. There should not be 1190 any leak, crashes, CPU spikes. The CPU spike is determined as the 1191 CPU usage which shoots at 40 to 50 percent of the average usage. The 1192 average value vary from device to device. Memory leak is determined 1193 by increase usage of the memory for PBB-EVPN process. The 1194 expectation is under steady state the memory usage for PBB-EVPN 1195 process should not increase. The measurement is carried out using 1196 external server which polls the DUT using automated scripts which 1197 captures the CPU usage and process memory. 1199 5. Acknowledgments 1201 We would like to thank Fioccola Giuseppe of Telecom Italia reviewing 1202 our draft and commenting it. We would like to thank Sarah Banks for 1203 guiding and mentoring us. We take the opportunity to thank Al for 1204 reviewing our draft and gave us valuable comments. 1206 6. IANA Considerations 1208 This memo includes no request to IANA. 1210 7. Security Considerations 1212 The benchmarking tests described in this document are limited to the 1213 performance characterization of controllers in a lab environment with 1214 isolated networks. The benchmarking network topology will be an 1215 independent test setup and MUST NOT be connected to devices that may 1216 forward the test traffic into a production network or misroute 1217 traffic to the test management network. Further, benchmarking is 1218 performed on a "black-box" basis, relying solely on measurements 1219 observable external to the controller. Special capabilities SHOULD 1220 NOT exist in the controller specifically for benchmarking purposes. 1221 Any implications for network security arising from the controller 1222 SHOULD be identical in the lab and in production networks. 1224 8. References 1226 8.1. Normative References 1228 [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for 1229 Network Interconnect Devices", RFC 2544, 1230 DOI 10.17487/RFC2544, March 1999, 1231 . 1233 [RFC2899] Ginoza, S., "Request for Comments Summary RFC Numbers 1234 2800-2899", RFC 2899, DOI 10.17487/RFC2899, May 2001, 1235 . 1237 [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 1238 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, 1239 May 2017, . 1241 8.2. Informative References 1243 [RFC7432] Sajassi, A., Ed., Aggarwal, R., Bitar, N., Isaac, A., 1244 Uttaro, J., Drake, J., and W. Henderickx, "BGP MPLS-Based 1245 Ethernet VPN", RFC 7432, DOI 10.17487/RFC7432, February 1246 2015, . 1248 [RFC7623] Sajassi, A., Ed., Salam, S., Bitar, N., Isaac, A., and W. 1249 Henderickx, "Provider Backbone Bridging Combined with 1250 Ethernet VPN (PBB-EVPN)", RFC 7623, DOI 10.17487/RFC7623, 1251 September 2015, . 1253 Appendix A. Appendix 1255 Authors' Addresses 1257 Sudhin Jacob (editor) 1258 Juniper Networks 1259 Bangalore 1260 India 1262 Phone: +91 8061212543 1263 Email: sjacob@juniper.net 1265 Kishore Tiruveedhula 1266 Juniper Networks 1267 10 Technology Park Dr 1268 Westford, MA 01886 1269 USA 1271 Phone: +1 9785898861 1272 Email: kishoret@juniper.net