idnits 2.17.1 draft-ietf-bmwg-evpntest-08.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** There are 32 instances of too long lines in the document, the longest one being 16 characters in excess of 72. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 216 has weird spacing: '...nerator sende...' == Line 217 has weird spacing: '...eceiver of la...' == The document seems to lack the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords -- however, there's a paragraph with a matching beginning. Boilerplate error? (The document does seem to have the reference to RFC 2119 which the ID-Checklist requires). -- The document date (May 29, 2021) is 1034 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Missing Reference: 'RFC7632' is mentioned on line 111, but not defined == Unused Reference: 'RFC2544' is defined on line 1232, but no explicit reference was found in the text == Unused Reference: 'RFC2899' is defined on line 1237, but no explicit reference was found in the text == Unused Reference: 'RFC7623' is defined on line 1252, but no explicit reference was found in the text Summary: 1 error (**), 0 flaws (~~), 8 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Internet Engineering Task Force S. Jacob, Ed. 3 Internet-Draft Independent 4 Intended status: Informational K. Tiruveedhula 5 Expires: November 30, 2021 Juniper Networks 6 May 29, 2021 8 Benchmarking Methodology for EVPN and PBB-EVPN 9 draft-ietf-bmwg-evpntest-08 11 Abstract 13 This document defines methodologies for benchmarking EVPN and PBB- 14 EVPN performance. EVPN is defined in RFC 7432, and is being deployed 15 in Service Provider networks. Specifically, this document defines 16 the methodologies for benchmarking EVPN/PBB-EVPN convergence, data 17 plane performance, and control plane performance. 19 Status of This Memo 21 This Internet-Draft is submitted in full conformance with the 22 provisions of BCP 78 and BCP 79. 24 Internet-Drafts are working documents of the Internet Engineering 25 Task Force (IETF). Note that other groups may also distribute 26 working documents as Internet-Drafts. The list of current Internet- 27 Drafts is at https://datatracker.ietf.org/drafts/current/. 29 Internet-Drafts are draft documents valid for a maximum of six months 30 and may be updated, replaced, or obsoleted by other documents at any 31 time. It is inappropriate to use Internet-Drafts as reference 32 material or to cite them other than as "work in progress." 34 This Internet-Draft will expire on November 30, 2021. 36 Copyright Notice 38 Copyright (c) 2021 IETF Trust and the persons identified as the 39 document authors. All rights reserved. 41 This document is subject to BCP 78 and the IETF Trust's Legal 42 Provisions Relating to IETF Documents 43 (https://trustee.ietf.org/license-info) in effect on the date of 44 publication of this document. Please review these documents 45 carefully, as they describe your rights and restrictions with respect 46 to this document. Code Components extracted from this document must 47 include Simplified BSD License text as described in Section 4.e of 48 the Trust Legal Provisions and are provided without warranty as 49 described in the Simplified BSD License. 51 Table of Contents 53 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 54 1.1. Requirements Language . . . . . . . . . . . . . . . . . . 3 55 1.2. Terminologies . . . . . . . . . . . . . . . . . . . . . . 3 56 2. Test Topology . . . . . . . . . . . . . . . . . . . . . . . . 4 57 3. Test Cases for EVPN Benchmarking . . . . . . . . . . . . . . 7 58 3.1. Data Plane MAC Learning . . . . . . . . . . . . . . . . . 7 59 3.2. Control Plane MAC Learning . . . . . . . . . . . . . . . 8 60 3.3. MAC Flush-Local Link Failure and Relearning . . . . . . . 9 61 3.4. MAC Flush-Remote Link Failure and Relearning. . . . . . . 10 62 3.5. MAC Aging . . . . . . . . . . . . . . . . . . . . . . . . 11 63 3.6. Remote MAC Aging . . . . . . . . . . . . . . . . . . . . 11 64 3.7. Control and Data plane MAC Learning . . . . . . . . . . . 12 65 3.8. High Availability. . . . . . . . . . . . . . . . . . . . 13 66 3.9. ARP/ND Scale . . . . . . . . . . . . . . . . . . . . . . 14 67 3.10. Scaling of Services . . . . . . . . . . . . . . . . . . . 15 68 3.11. Scale Convergence . . . . . . . . . . . . . . . . . . . . 15 69 3.12. SOAK Test. . . . . . . . . . . . . . . . . . . . . . . . 16 70 4. Test Cases for PBB-EVPN Benchmarking . . . . . . . . . . . . 17 71 4.1. Data Plane Local MAC Learning . . . . . . . . . . . . . . 17 72 4.2. Data Plane Remote MAC Learning . . . . . . . . . . . . . 18 73 4.3. MAC Flush-Local Link Failure . . . . . . . . . . . . . . 19 74 4.4. MAC Flush-Remote Link Failure . . . . . . . . . . . . . . 20 75 4.5. MAC Aging . . . . . . . . . . . . . . . . . . . . . . . . 21 76 4.6. Remote MAC Aging. . . . . . . . . . . . . . . . . . . . . 21 77 4.7. Local and Remote MAC Learning . . . . . . . . . . . . . . 22 78 4.8. High Availability . . . . . . . . . . . . . . . . . . . . 23 79 4.9. Scale . . . . . . . . . . . . . . . . . . . . . . . . . . 24 80 4.10. Scale Convergence . . . . . . . . . . . . . . . . . . . . 25 81 4.11. Soak Test . . . . . . . . . . . . . . . . . . . . . . . . 26 82 5. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 27 83 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 27 84 7. Security Considerations . . . . . . . . . . . . . . . . . . . 27 85 8. References . . . . . . . . . . . . . . . . . . . . . . . . . 27 86 8.1. Normative References . . . . . . . . . . . . . . . . . . 27 87 8.2. Informative References . . . . . . . . . . . . . . . . . 27 88 Appendix A. Appendix . . . . . . . . . . . . . . . . . . . . . . 28 89 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 28 91 1. Introduction 93 EVPN is defined in RFC 7432, and describes BGP MPLS based Ethernet 94 VPNs (EVPN). PBB-EVPN is defined in RFC 7623, discusses how Ethernet 95 Provider backbone Bridging can be combined with EVPNs to provide a 96 new/combined solution. This draft defines methodologies that can be 97 used to benchmark both RFC 7432 and RFC 7623 solutions. Further, 98 this draft provides methodologies for benchmarking the performance of 99 EVPN data and control planes, MAC learning, MAC flushing, MAC aging, 100 convergence, high availability, and scale. 102 1.1. Requirements Language 104 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 105 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 106 document are to be interpreted as described in RFC 8174 [RFC8174]. 108 1.2. Terminologies 110 Most of the terminology used in this documents comes from [RFC7432] 111 and [RFC7632]. 113 All-Active Redundancy Mode: When all PEs attached to an Ethernet 114 segment are allowed to forward known unicast traffic to/from that 115 Ethernet segment for a given VLAN, then the Ethernet segment is 116 defined to be operating in All-Active redundancy mode. 118 AA: All Active mode 120 CE: Customer Router/Devices/Switch. 122 DF: Designated Forwarder 124 DUT: Device under test. 126 Ethernet Segment (ES): When a customer site (device or network) is 127 connected to one or more PEs via a set of Ethernet links, then that 128 set of links is referred to as an 'Ethernet segment'. 130 EVI: An EVPN instance spanning the Provider Edge (PE) devices 131 participating in that EVPN. 133 Ethernet Segment Identifier (ESI): A unique non-zero identifier that 134 identifies an Ethernet segment is called an 'Ethernet Segment 135 Identifier'. 137 Ethernet Tag: An Ethernet tag identifies a particular broadcast 138 domain, e.g., a VLAN. An EVPN instance consists of one or more 139 broadcast domains. 141 Interface: Physical interface of a router/switch. 143 IRB: Integrated routing and bridging interface 145 MAC: Media Access Control addresses on a PE. 147 MHPE2: Multi homed Provider Edge router 2. 149 MHPE1: Multi homed Provider Edge router 1. 151 SHPE3: Single homed Provider Edge Router 3. 153 PE: Provider Edge device. 155 P: Provider Router. 157 RR: Route Reflector. 159 RT: Traffic Generator. 161 Sub Interface: Each physical Interfaces is subdivided in to a set of 162 Logical units. 164 SA: Single Active 166 Single-Active Redundancy Mode: When a single PE (among all the PEs 167 attached to an Ethernet segment) is the only PE allowed to forward 168 traffic to/from a given Ethernet segment for a given VLAN, then that 169 Ethernet segment is defined to be operating in Single-Active 170 redundancy mode. 172 2. Test Topology 174 There are five routers in the Test setup. SHPE3, RR/P, MHPE1 and 175 MHPE2 emulating a service provider network. CE is a customer device 176 connected to MHPE1 and MHPE2. it is configured with bridge domains in 177 multiple VLANS. The traffic generator is connected to the CE and 178 SHPE3. The MHPE1 acts as DUT. The traffic generator will be used as 179 sender and receiver of traffic. The test measurements are taken from 180 the DUT. MHPE1 and MHPE2 are multi-homed routers connected to CE 181 running single active mode. The traffic generator will be generating 182 traffic at 10% of the line rate. 184 +----------------+ +---------------------------+ 185 | | | | 186 | | |Traffic Generator sender/ | 187 | SHPE3 | |receiver of layer 2 traffic| 188 | +----------------| with multiple Vlans | 189 | | +---------------------------+ 190 +---------+------+ 191 | Core Link 192 | 193 +--------+-----+ 194 | | 195 | RR/P +----------------------+ 196 | | Core link | 197 | | | 198 +--+-----------+ | 199 | | 200 | core link | 201 | | 202 +-------------+---+ +------+------------+ 203 | | | | 204 | | | | 205 | MHPE1(DUT) | | MHPE2 | 206 | | | | 207 | | | | 208 +-----------------+------+ +-----+-------------------+ 209 | | 210 PE-CE link | | PE-CE link 211 | | 212 | | 213 | | 214 | | 215 +-----+----------+----+ +----------------------------+ 216 | | | Traffic Generator sender/ | 217 | | |receiver of layer 2 traffic| 218 | CE/Layer 2 bridge +-----------+ with multiple Vlans | 219 | | +----------------------------+ 220 | | 221 +---------------------+ 223 Topology 1 225 Test Setup 227 Figure 1 229 +--------------+------------------+-------------+---------+-----------+----------------+ 230 | | | | | | | 231 | Mode | Test | Traffic | Sender | Receiver | | 232 | | | Direction | | | | 233 +--------------------------------------------------------------------------------------+ 234 | | | | | | | 235 |Single Active | Local MAC | | CE | SHPFE3 |Layer 2 traffic | 236 | | Learning | Uni | | | | 237 | | | | | | multiple MAC | 238 +--------------------------------------------------------------------------------------+ 239 | | | | | | | 240 |Single Active | Remote MAC | | | CE |Layer 2 traffic | 241 | | Learning | Uni | SHPE3 | | | 242 | | | | | |multiple MAC | 243 +--------------------------------------------------------------------------------------+ 244 | | | | | | | 245 |Single Active | Scale Convergence| Bi | | CE/SHPE3 | | 246 | | | |CE/SHPE3 | |Layer 2 traffic | 247 | | Local& Remote | | | |multiple MAC & | 248 | | Learning | | | | vlans | 249 +--------------+------------------+-------------+---------+-----------+----------------+ 251 Table showing the traffic directions of various EVPN/PBB-EVPN 252 benchmarking test cases. Depending on the test scenario, the traffic 253 can be uni-directional or bi-directional (configured in the traffic 254 generator). 256 Figure 2 258 Test Setup Configurations: 260 SHPE3 is configured with Interior Gateway protocols like OPSF or IS- 261 IS for underlay, LDP for MPLS support, Interior Border Gateway with 262 EVPN address family for overlay support. This router must be 263 configured with N EVPN/PBB-EVPN instances for testing. Traffic 264 generator is connected to this router for sending and receiving 265 traffic. 267 RR is configured with Interior Gateway protocols like OPSF or IS-IS 268 for underlay, LDP for MPLS support, Interior Border Gateway with EVPN 269 address family for overlay support. This router function as both 270 provider router and a route reflector. 272 MHPE1 is configured with Interior Gateway protocols like OPSF or IS- 273 IS for underlay, LDP for MPLS support, Interior Border Gateway with 274 EVPN address family for overlay support. This router must be 275 configured with N EVPN/PBB-EVPN instances for testing. This router 276 is configured with ESI per vlan or ESI per interface. It is 277 functioning as multi homing PE working on Single Active EVPN mode. 278 This router serves as the DUT and it is connected to CE. MHPE1 is 279 acting as DUT for all the test cases. 281 MHPE2 is configured with Interior Gateway protocols like OPSF or IS- 282 IS for underlay, LDP for MPLS support,Interior Border Gateway with 283 EVPN address family for overlay support. This router must be 284 configured with N EVPN/PBB-EVPN instances for testing. This router 285 is configured with ESI per vlan or ESI per interface. It is 286 functioning as multi homing PE working on Single Active EVPN mode. 287 It is connected to CE. 289 CE is acting as bridge configured with multiple vlans. The same 290 vlans are configured on MHPE1,MHPE2,SHPE3. traffic generator is 291 connected to CE. the traffic generator acts as sender or receiver of 292 traffic. 294 Depending up on the test scenarios the traffic generators will be 295 used to generate uni directional or bi directional flows. 297 The above configuration will be serving as the base configuration for 298 all test cases. 300 The X is used as variable to denote scale factor of the testing 301 parameters.It must be in the multiples of 100. 303 3. Test Cases for EVPN Benchmarking 305 3.1. Data Plane MAC Learning 307 Objective: 309 Measure the time taken to learn the Data Plane MAC in DUT. 311 Topology : Topology 1 313 Procedure: 315 The data plane MAC learning can be measured using the parameters 316 defined in RFC 2889 section 5.8. 318 Confirm the DUT is up and running with EVPN. 320 Traffic generator connected to CE must send frames with X different 321 source and destination MAC address for one vlan, the same vlan must 322 be present in all the devices except RR. 324 Send X unicast frames from CE to MHPE1(DUT) for one EVPN instance 325 working in SA mode. 327 The DUT will learn these X MAC in data plane. 329 Measurement : 331 Measure the time taken to learn X MAC locally in DUT EVPN MAC table. 332 The data plane measurement is taken by considering DUT as black box. 333 The range of MAC are known from traffic generator, the same must be 334 learned in DUT, the time taken to learn X MAC is measured. The 335 measurement is carried out using external server which polls the DUT 336 using automated scripts. 338 The test is repeated for N times and the values are collected. The 339 MAC learning rate is calculated by averaging the values obtained from 340 N samples. N is an arbitrary number to get a sufficient sample. The 341 time measured for each sample is denoted by T1,T2...Tn. 343 MAC learning rate = (T1+T2+..Tn)/N 345 3.2. Control Plane MAC Learning 347 Objective: 349 Measure the time taken to learn the control plane MAC. 351 Topology : Topology 1 353 Procedure: 355 Confirm the DUT is up and running with EVPN. 357 Traffic generator connected to SHPE3 must send frames with X 358 different source and destination MAC address for one vlan, the same 359 vlan must be present in all the devices except RR. 361 Ensure the frames must be destined to one EVPN instance. 363 The DUT will learn these X MAC in control plane. 365 Measurement : 367 Measure the time taken by the DUT to learn the X MAC in the data 368 plane. The test is repeated for N times and the values are 369 collected. The remote MAC learning rate is calculated by averaging 370 the values obtained from N samples. N is an arbitrary number to get 371 a sufficient sample. The time measured for each sample is denoted by 372 T1,T2...Tn. The measurement is carried out using external server 373 which polls the DUT using automated scripts. 375 MAC learning rate = (T1+T2+..Tn)/N 377 3.3. MAC Flush-Local Link Failure and Relearning 379 Objective: 381 Measure the time taken to flush the Data Plane MAC and the time taken 382 to relearn the same amount of MAC. 384 Topology : Topology 1 386 Procedure: 388 Confirm the DUT is up and running with EVPN. 390 Send X frames with X different source and destination MAC addresses 391 to DUT from CE using traffic generator for one vlan. 393 Ensure the DUT learns all X MAC addresses in data plane. 395 Fail the DUT-CE link and measure the time taken to flush these X MAC 396 from the EVPN MAC table. 398 Bring up the link which was made Down(the link between DUT and CE). 399 Measure time taken by the DUT to relearn these X MAC. 401 The DUT and MHPE2 are running SA mode. 403 Measurement : 405 Measure the time taken for flushing these X MAC addresses. Measure 406 the time taken to relearn these X MAC in DUT. The test is repeated 407 for N times and the values are collected. The flush and the 408 relearning time is calculated by averaging the values obtained by N 409 samples. N is an arbitrary number to get a sufficient sample. The 410 time measured for each sample is denoted by T1,T2...Tn. The 411 measurement is carried out using external server which polls the DUT 412 using automated scripts. 414 Flush rate = (T1+T2+..Tn)/N 416 Relearning rate = (T1+T2+..Tn)/N 418 3.4. MAC Flush-Remote Link Failure and Relearning. 420 Objective: 422 Measure the time taken to flush the Control plane MAC learned in DUT 423 during remote link failure and the time taken to relearn. 425 Topology : Topology 1 427 Procedure: 429 Confirm the DUT is up and running with EVPN. 431 Send X frames with X different source and destination MAC addresses 432 to DUT from SHPE3 using traffic generator for one vlan. 434 Bring down the link between SHPE3 and traffic generator. 436 SHPE3 will withdraw the routes from DUT due to link failure. 438 Measure the time taken to flush the DUT EVPN MAC table. The DUT and 439 MHPE2 are running SA mode. 441 Bring up the link which was made Down(the link between SHPE3 and 442 traffic generator). 444 Measure time taken by the DUT to relearn these X MAC from control 445 plane. 447 Measurement : 449 Measure the time taken to flush X remote MAC from EVPN MAC table of 450 the DUT. Measure the time taken to relearn these X MAC in DUT. The 451 test is repeated for N times and the values are collected. The flush 452 rate is calculated by averaging the values obtained by N samples. N 453 is an arbitrary number to get a sufficient sample. The time measured 454 for each sample is denoted by T1,T2...Tn. The measurement is carried 455 out using external server which polls the DUT using automated 456 scripts. 458 Flush rate = (T1+T2+..Tn)/N 460 Relearning rate = (T1+T2+..Tn)/N 462 3.5. MAC Aging 464 Objective: 466 To measure the MAC Aging time. 468 Topology : Topology 1 470 Procedure: 472 Confirm the DUT is up and running with EVPN. 474 Send X frames with X different source and destination MAC addresses 475 to DUT from CE using traffic generator for one vlan. 477 Ensure these X MAC addresses are learned in DUT. 479 Then stop the traffic. 481 Ensure the DUT and other devices in the test are using the default 482 timers for aging. 484 Measure the time taken to flush X MAC from DUT EVPN MAC table due to 485 aging. 487 The DUT and MHPE2 are running SA mode. 489 Measurement : 491 Measure the time taken to flush X MAC addresses due to aging. The 492 test is repeated for N times and the values are collected. The aging 493 is calculated by averaging the values obtained by N samples. N is an 494 arbitrary number to get a sufficient sample. The time measured for 495 each sample is denoted by T1,T2...Tn. The measurement is carried out 496 using external server which polls the DUT using automated scripts. 498 Aging time for X MAC in sec = (T1+T2+..Tn)/N 500 3.6. Remote MAC Aging 502 Objective: 504 Measure the control plane learned MAC Aging time. 506 Topology : Topology 1 508 Procedure: 510 Confirm the DUT is up and running with EVPN. 512 Send X frames with X different source and destination MAC addresses 513 to DUT from SHPE3 using traffic generator for one vlan. 515 Ensure these X MAC addresses are learned in DUT via control plane. 517 Then stop the traffic. 519 Ensure the DUT and other devices in the test are using the default 520 timers for aging. 522 Measure the time taken to flush X MAC from DUT EVPN MAC table due to 523 aging. 525 The DUT and MHPE2 are running SA mode. 527 Measurement : 529 Measure the time taken to flush X remote MAC learned in DUT EVPN MAC 530 table due to aging. The test is repeated for N times and the values 531 are collected. The aging is calculated by averaging the values 532 obtained by N samples. N is an arbitrary number to get a sufficient 533 sample. The time measured for each sample is denoted by T1,T2...Tn. 534 The measurement is carried out using external server which polls the 535 DUT using automated scripts. 537 Aging time for X MAC in sec = (T1+T2+..Tn)/N 539 3.7. Control and Data plane MAC Learning 541 Objective: 543 To record the time taken to learn both local and remote MAC. 545 Topology : Topology 1 547 Procedure: 549 Confirm the DUT is up and running with EVPN. 551 Send X frames with X different source and destination MAC addresses 552 to DUT from SHPE3 using traffic generator for one vlan. 554 Send X frames with different source and destination MAC addresses 555 from traffic generator connected to CE for one vlan. 557 The source and destination addresses of flows must be complimentary 558 to have unicast flows. 560 Measure the time taken by the DUT to learn 2X in EVPN MAC table. 562 DUT and MHPE2 are running in SA mode. 564 Measurement : 566 Measure the time taken to learn 2X MAC addresses in DUT EVPN MAC 567 table. The test is repeated for N times and the values are 568 collected. The MAC learning time is calculated by averaging the 569 values obtained by N samples. N is an arbitrary number to get a 570 sufficient sample. The time measured for each sample is denoted by 571 T1,T2...Tn. The measurement is carried out using external server 572 which polls the DUT using automated scripts 574 MAC learning rate = (T1+T2+..Tn)/N 576 3.8. High Availability. 578 Objective: 580 Measure traffic loss during routing engine fail over. 582 Topology : Topology 1 584 Procedure: 586 Confirm the DUT is up and running with EVPN. 588 Send X frames from CE to DUT from traffic generator with X different 589 source and destination MAC addresses. 591 Send X frames from traffic generator to SHPE3 with X different source 592 and destination MAC addresses, so that 2X MAC address will be learned 593 in the DUT. 595 There is a bi directional traffic flow with X pps in each direction. 597 Ensure the DUT learn 2X MAC. 599 Then do a routing engine fail-over. 601 Measurement : 603 The expectation of the test is 0 traffic loss with no change in the 604 DF role. DUT should not withdraw any routes. But in cases where the 605 DUT is not property synchronised between master and standby, due to 606 that packet loss are observed. In that scenario the packet loss is 607 measured.The test is repeated for N times and the values are 608 collected. The packet loss is calculated by averaging the values 609 obtained by N samples. N is an arbitrary number to get a sufficient 610 sample. The time measured for each sample is denoted by T1,T2...Tn. 611 The measurement is carried out using external server which polls the 612 DUT using automated scripts to ensure the DUT learned 2X MAC. The 613 packet drop is measured using traffic generator. 615 Packet loss in sec with 2X MAC addresses = (T1+T2+..Tn)/N 617 3.9. ARP/ND Scale 619 Measure the DUT scaling limit of ARP/ND. 621 Objective: 623 Measure the ARP/ND scale of the DUT. 625 Topology : Topology 1 627 Procedure: 629 Confirm the DUT is up and running with EVPN. 631 Send X arp/neighbour discovery(ND) from the traffic generator to DUT 632 with different sender ip/ipv6, MAC addresses to the target IRB 633 address configured in EVPN instance. 635 The EVPN instance learns the MAC+ip and MAC+ipv6 addresses from these 636 request and advertise as type 2 MAC+ip/MAC+ipv6 route to remote 637 provide edge routers which have same EVPN configurations. 639 The value of X must be increased at a incremental value of 5% of X, 640 till the limit is reached. The limit is where the DUT cant learn any 641 more type 2 MAC+ip/MAC+ipv6.The test must be separately conducted for 642 arp and ND. 644 Measurement : 646 Measure the scale limit of type 2 MAC+ip/MAC+ipv6 route which DUT can 647 learn. The test is repeated for N times and the values are 648 collected. The scale limit is calculated by averaging the values 649 obtained by N samples for both MAC+ip and MAC+ipv6. N is an 650 arbitrary number to get a sufficient sample. The scale value 651 obtained by each sample be v1,v2..vn. The measurement is carried out 652 using external server which polls the DUT using automated scripts to 653 find the scale limit of MAC+ipv4/MAC+ipv6. 655 Scale limit for MAC+ip = (v1+v2+..vn)/N 657 Scale limit for MAC+ipv6 = (v1+v2+..vn)/N 659 3.10. Scaling of Services 661 Objective: 663 Measure the scale of EVPN instances that a DUT can hold. 665 Topology : Topology 1 667 Procedure: 669 Confirm the DUT is up and running with EVPN. 671 The DUT, MHPE2 and SHPE3 are scaled to N EVI. 673 Ensure routes received from MHPE2 and SHPE3 for N EVI in the DUT. 675 Then increment the scale of N by 5% of N till the limit is reached. 677 The limit is where the DUT cant learn any EVPN routes from its peers. 679 Measurement : 681 There should not be any loss of route types 1,2,3 and 4 in DUT. DUT 682 must relearn all type 1, 2, 3 and 4 from remote routers. The DUT 683 must be subjected to various values of N to find the optimal scale 684 limit. The scope of the test is find out the maximum EVPN instance 685 that a DUT can hold. The measurement is carried out using external 686 server which polls the DUT using automated scripts to find the scale 687 limit of EVPN instances. 689 3.11. Scale Convergence 691 Objective: 693 Measure the convergence time of DUT when the DUT is scaled with EVPN 694 instance along with traffic. 696 Topology : Topology 1 697 Procedure: 699 Confirm the DUT is up and running with EVPN. 701 Scale N EVIs in DUT,SHPE3 and MHPE2. 703 Send F frames to DUT from CE using traffic generator with X different 704 source and destination MAC addresses for N EVI's. 706 Send F frames from traffic generator to SHPE3 with X different source 707 and destination MAC addresses. 709 There will be 2X number of MAC addresses will be learned in DUT EVPN 710 MAC table. 712 There is a bi directional traffic flow with F pps in each direction. 714 Then clear the BGP neighbours in the DUT. 716 Once the BGP session is in established state in DUT. 718 Measure the time taken to learn 2X MAC address in DUT MAC table. 720 Measurement : 722 The DUT must learn 2X MAC addresses. Measure the time taken to learn 723 2X MAC in DUT. The test is repeated for N times and the values are 724 collected. The convergence time is calculated by averaging the 725 values obtained by N samples. N is an arbitrary number to get a 726 sufficient sample.The time measured for each sample is denoted by 727 T1,T2...Tn. The measurement is carried out using external server 728 which polls the DUT using automated scripts. 730 Time taken to learn 2X MAC in DUT = (T1+T2+..Tn)/N 732 3.12. SOAK Test. 734 Objective: 736 This test is carried out to measure the stability of the DUT in a 737 scaled environment with traffic over a period of time "T'". In each 738 interval "t1" the DUT CPU usage, memory usage are measured. The DUT 739 is checked for any crashes during this time period. 741 Topology : Topology 1 743 Procedure: 745 Confirm the DUT is up and running with EVPN. 747 Scale N EVI's in DUT, SHPE3 and MHPE2.Send F frames to DUT from CE 748 using traffic generator with different X source and destination MAC 749 addresses for N EVI's. 751 Send F frames from traffic generator to SHPE3 with X different source 752 and destination MAC addresses. 754 There will be 2X number of MAC addresses will be learned in DUT EVPN 755 MAC table. 757 There is a bi directional traffic flow with F pps in each direction. 759 The DUT must run with traffic for 24 hours. 761 Every hour check for memory leak in EVPN process, CPU usage and 762 crashes in DUT. 764 Measurement : 766 Take the hourly reading of CPU, process memory. There should not be 767 any memory leak, crashes, CPU spikes. The CPU spike is determined as 768 the sudden increase of CPU usage to 100 percent compared to the 769 average usage. The average value vary from device to device. Memory 770 leak is determined as the increase of memory usage with respect to 771 time. The expectation is under steady state the memory usage for 772 EVPN process should not increase with respect to time. The 773 measurement is carried out using external server which polls the DUT 774 using automated scripts which captures the CPU usage and process 775 memory. 777 4. Test Cases for PBB-EVPN Benchmarking 779 4.1. Data Plane Local MAC Learning 781 Objective: 783 Measure the time taken to learn the Data Plane MAC in DUT. 785 Topology : Topology 1 787 Procedure: 789 Confirm the DUT is up and running with PBB-EVPN. 791 Traffic generator connected to CE must send frames with X different 792 source and destination MAC address for one vlan, the same vlan must 793 be present in all the devices except RR. 795 Send X unicast frames from CE to MHPE1(DUT) for one PBB-EVPN instance 796 working in SA mode. 798 The DUT will learn these X MAC in data plane. 800 Measurement : 802 Measure the time taken to learn X MAC locally in DUT PBB-EVPN MAC 803 table. The data plane measurement is taken by considering DUT as 804 black box. The range of MAC are known from traffic generator, the 805 same must be learned in DUT, the time taken to learn X MAC is 806 measured.The measurement is carried out using external server which 807 polls the DUT using automated scripts. 809 The test is repeated for N times and the values are collected. The 810 MAC learning rate is calculated by averaging the values obtained from 811 N samples. N is an arbitrary number to get a sufficient sample. The 812 time measured for each sample is denoted by T1,T2...Tn. 814 MAC learning rate = (T1+T2+..Tn)/N 816 4.2. Data Plane Remote MAC Learning 818 Objective: 820 To Record the time taken to learn the remote MAC. 822 Topology : Topology 1 824 Procedure: 826 Confirm the DUT is up and running with PBB-EVPN. 828 Traffic generator connected to SHPE3 must send frames with X 829 different source and destination MAC address for one vlan, the same 830 vlan must be present in all the devices except RR. 832 Ensure the frames must be destined to one PBB-EVPN instance. 834 The DUT will learn these X MAC in data plane. 836 Measurement : 838 Measure the time taken by the DUT to learn the X MAC in the data 839 plane. The test is repeated for N times and the values are 840 collected. The remote MAC learning rate is calculated by averaging 841 the values obtained from N samples. N is an arbitrary number to get 842 a sufficient sample. The time measured for each sample is denoted by 843 T1,T2...Tn. The measurement is carried out using external server 844 which polls the DUT using automated scripts. 846 MAC learning rate = (T1+T2+..Tn)/N 848 4.3. MAC Flush-Local Link Failure 850 Objective: 852 Measure the time taken to flush the locally learned MAC and the time 853 taken to relearn the same amount of MAC. 855 Topology : Topology 1 857 Procedure: 859 Confirm the DUT is up and running with PBB-EVPN. 861 Send X frames with X different source and destination MAC addresses 862 to DUT from CE using traffic generator for one vlan. 864 Ensure the DUT learns all X MAC addresses in data plane. 866 Fail the DUT-CE link and measure the time taken to flush these X MAC 867 from the PBB-EVPN MAC table. 869 Bring up the link which was made Down(the link between DUT and 870 CE).Measure time taken by the DUT to relearn these X MAC. 872 The DUT and MHPE2 are running SA mode. 874 Measurement : 876 Measure the time taken for flushing these X MAC addresses. Measure 877 the time taken to relearn these X MAC in DUT. The test is repeated 878 for N times and the values are collected. The flush and the 879 relearning time is calculated by averaging the values obtained by N 880 samples. N is an arbitrary number to get a sufficient sample. The 881 time measured for each sample is denoted by T1,T2...Tn. The 882 measurement is carried out using external server which polls the DUT 883 using automated scripts. 885 Flush rate = (T1+T2+..Tn)/N 887 Relearning rate = (T1+T2+..Tn)/N 889 4.4. MAC Flush-Remote Link Failure 891 Objective: 893 Measure the time taken to flush the remote MAC learned in DUT due to 894 remote link failure and relearning it. 896 Topology : Topology 1 898 Procedure: 900 confirm the DUT is up and running with PBB-EVPN. 902 Send X frames with X different source and destination MAC addresses 903 to DUT from SHPE3 using traffic generator for one vlan. 905 Bring down the link between SHPE3 and traffic generator. 907 Measure the time taken to flush the DUT PBB-EVPN MAC table. The DUT 908 and MHPE2 are running SA mode. 910 Bring up the link which was made Down(the link between SHPE3 and 911 traffic generator). 913 Measure time taken by the DUT to relearn these X MAC 915 Measurement : 917 Measure the time taken to flush X remote MAC from PBB-EVPN MAC table 918 of the DUT. Measure the time taken to relearn these X MAC in DUT. 919 The test is repeated for N times and the values are collected. The 920 flush rate is calculated by averaging the values obtained by N 921 samples. N is an arbitrary number to get a sufficient sample. The 922 time measured for each sample is denoted by T1,T2...Tn. The 923 measurement is carried out using external server which polls the DUT 924 using automated scripts. 926 Flush rate = (T1+T2+..Tn)/N 928 Relearning rate = (T1+T2+..Tn)/N 930 4.5. MAC Aging 932 Objective: 934 Measure the MAC Aging time. 936 Topology : Topology 1 938 Procedure: 940 Confirm the DUT is up and running with PBB-EVPN. 942 Send X frames with X different source and destination MAC addresses 943 to DUT from CE using traffic generator for one vlan. 945 Ensure these X MAC addresses are learned in DUT. 947 Then stop the traffic. 949 Ensure the DUT and other devices in the test are using the default 950 timers for aging. 952 Measure the time taken to flush X MAC from DUT PBB-EVPN MAC table due 953 to aging. 955 The DUT and MHPE2 are running SA mode. 957 Measurement : 959 Measure the time taken to flush X MAC addresses due to aging. The 960 test is repeated for N times and the values are collected. The aging 961 is calculated averaging the values obtained by N samples. N is an 962 arbitrary number to get a sufficient sample. The time measured for 963 each sample is denoted by T1,T2...Tn. The measurement is carried out 964 using external server which polls the DUT using automated scripts. 966 Aging time for X MAC in sec = (T1+T2+..Tn)/N 968 4.6. Remote MAC Aging. 970 Objective: 972 Measure the remote MAC Aging time. 974 Topology : Topology 1 976 Procedure: 978 Confirm the DUT is up and running with PBB-EVPN. 980 Send X frames with X different source and destination MAC addresses 981 to DUT from SHPE3 using traffic generator for one vlan. 983 Ensure these X MAC addresses are learned in DUT. 985 Then stop the traffic. 987 Ensure the DUT and other devices in the test are using the default 988 timers for aging. 990 Measure the time taken to flush X MAC from DUT PBB-EVPN MAC table due 991 to aging. 993 The DUT and MHPE2 are running SA mode. 995 Measurement : 997 Measure the time taken to flush X remote MAC learned in DUT EVPN MAC 998 table due to aging. The test is repeated for N times and the values 999 are collected. The aging is calculated by averaging the values 1000 obtained by N samples. N is an arbitrary number to get a sufficient 1001 sample. The time measured for each sample is denoted by T1,T2...Tn. 1002 The measurement is carried out using external server which polls the 1003 DUT using automated scripts. 1005 Aging time for X MAC in sec = (T1+T2+..Tn)/N 1007 4.7. Local and Remote MAC Learning 1009 Objective: 1011 Measure the time taken to learn both local and remote MAC. 1013 Topology : Topology 1 1015 Procedure: 1017 Confirm the DUT is up and running with PBB-EVPN. 1019 Send X frames with X different source and destination MAC addresses 1020 to DUT from SHPE3 using traffic generator for one vlan. 1022 Send X frames with different source and destination MAC addresses 1023 from traffic generator connected to CE for one vlan. 1025 The source and destination addresses of flows must be complimentary 1026 to have unicast flows. 1028 Measure the time taken by the DUT to learn 2X in PBB-EVPN MAC table. 1030 DUT and MHPE2 are running in SA mode. 1032 Measurement : 1034 Measure the time taken to learn 2X MAC addresses in DUT PBB-EVPN MAC 1035 table. The test is repeated for N times and the values are 1036 collected. The MAC learning time is calculated by averaging the 1037 values obtained by N samples. N is an arbitrary number to get a 1038 sufficient sample. The time measured for each sample is denoted by 1039 T1,T2...Tn. The measurement is carried out using external server 1040 which polls the DUT using automated scripts 1042 MAC learning rate = (T1+T2+..Tn)/N 1044 4.8. High Availability 1046 Objective: 1048 Measure traffic loss during routing engine failover. 1050 Topology : Topology 1 1052 Procedure: 1054 Confirm the DUT is up and running with PBB-EVPN. 1056 Send X frames from CE to DUT from traffic generator with X different 1057 source and destination MAC addresses. 1059 Send X frames from traffic generator to SHPE3 with X different source 1060 and destination MAC addresses, so that 2X MAC address will be learned 1061 in the DUT. 1063 There is a bi directional traffic flow with X pps in each direction. 1065 Ensure the DUT learn 2X MAC. 1067 Then do a routing engine fail-over. 1069 Measurement : 1071 The expectation of the test is 0 traffic loss with no change in the 1072 DF role. DUT should not withdraw any routes.But in cases where the 1073 DUT is not property synchronised between master and standby, due to 1074 that packet loss are observed. In that scenario the packet loss is 1075 measured. The test is repeated for N times and the values are 1076 collected. The packet loss is calculated by averaging the values 1077 obtained by N samples. N is an arbitrary number to get a sufficient 1078 sample. The time measured for each sample is denoted by T1,T2...Tn. 1079 The measurement is carried out using external server which polls the 1080 DUT using automated scripts to ensure the DUT learned 2X MAC. The 1081 packet drop is measured using traffic generator. 1083 Packet loss in sec with 2X MAC addresses = (T1+T2+..Tn)/N 1085 4.9. Scale 1087 Objective: 1089 Measure the scale limit of DUT for PBB-EVPN. 1091 Topology : Topology 1 1093 Procedure: 1095 Confirm the DUT is up and running with PBB-EVPN. 1097 The DUT,MHPE2 and SHPE3 are scaled to N PBB-EVI. 1099 Ensure routes received from MHPE2 and SHPE3 for N PBB-EVI in the DUT. 1101 Then increment the scale of N by 5% of N till the limit is reached. 1103 The limit is where the DUT cant learn any EVPN routes from its peers. 1105 Measurement : 1107 There should not be any loss of route types 2,3 and 4 in DUT. DUT 1108 must relearn all type 2, 3 and 4 from remote routers. The DUT must 1109 be subjected to various values of N to find the optimal scale limit. 1110 The scope of the test is find out the maximum EVPN instance that a 1111 DUT can hold. The measurement is carried out using external server 1112 which polls the DUT using automated scripts to find the scale limit 1113 of PBB-EVPN instances. 1115 4.10. Scale Convergence 1117 Objective: 1119 To measure the convergence time of DUT when the DUT is scaled with 1120 EVPN instance along with traffic. 1122 Topology : Topology 1 1124 Procedure: 1126 Confirm the DUT is up and running with PBB-EVPN. 1128 Scale N PBB-EVIs in DUT,SHPE3 and MHPE2. 1130 Send F frames to DUT from CE using traffic generator with X different 1131 source and destination MAC addresses for N PBB-EVI's. 1133 Send F frames from traffic generator to SHPE3 with X different source 1134 and destination MAC addresses. 1136 There will be 2X number of MAC addresses will be learned in DUT PBB- 1137 EVPN MAC table. 1139 There is a bi directional traffic flow with F pps in each direction. 1141 Then clear the BGP neighbours in the DUT. 1143 Once the BGP session is in established state in DUT. 1145 Measure the time taken to learn 2X MAC address in DUT MAC table. 1147 Measurement : 1149 The DUT must learn 2X MAC addresses. Measure the time taken to learn 1150 2X MAC in DUT. The test is repeated for N times and the values are 1151 collected. The convergence time is calculated by averaging the 1152 values obtained by N samples. N is an arbitrary number to get a 1153 sufficient sample.The time measured for each sample is denoted by 1154 T1,T2...Tn. The measurement is carried out using external server 1155 which polls the DUT using automated scripts. 1157 Time taken to learn 2X MAC in DUT = (T1+T2+..Tn)/N 1159 4.11. Soak Test 1161 Objective: 1163 To measure the stability of the DUT in a scaled environment with 1164 traffic. 1166 Topology : Topology 1 1168 Procedure: 1170 Confirm the DUT is up and running with PBB-EVPN. 1172 Scale N PBB-EVI's in DUT,SHPE3 and MHPE2.Send F frames to DUT from CE 1173 using traffic generator with different X source and destination MAC 1174 addresses for N EVI's. 1176 Send F frames from traffic generator to SHPE3 with X different source 1177 and destination MAC addresses. 1179 There will be 2X number of MAC addresses will be learned in DUT PBB- 1180 EVPN MAC table. 1182 There is a bi directional traffic flow with F pps in each direction. 1184 The DUT must run with traffic for 24 hours. 1186 Every hour check for memory leak in PBB-EVPN process,CPU usage and 1187 crashes in DUT. 1189 Measurement : 1191 Take the hourly reading of CPU, process memory. There should not be 1192 any memory leak, crashes, CPU spikes. The CPU spike is determined as 1193 the sudden increase of CPU usage to 100 percent compared to the 1194 average usage. The average value vary from device to device. Memory 1195 leak is determined as the increase of memory usage with respect to 1196 time. The expectation is under steady state the memory usage for 1197 PBB-EVPN process should not increase with respect to time. The 1198 measurement is carried out using external server which polls the DUT 1199 using automated scripts which captures the CPU usage and process 1200 memory. 1202 5. Acknowledgments 1204 We would like to thank Fioccola Giuseppe of Telecom Italia reviewing 1205 our draft and commenting it. We would like to thank Sarah Banks for 1206 guiding and mentoring us. We take the opportunity to thank Al for 1207 reviewing our draft and gave us valuable comments. 1209 6. IANA Considerations 1211 This memo includes no request to IANA. 1213 7. Security Considerations 1215 The benchmarking tests described in this document are limited to the 1216 performance characterisation of controllers in a lab environment with 1217 isolated networks. The benchmarking network topology will be an 1218 independent test setup and MUST NOT be connected to devices that may 1219 forward the test traffic into a production network or misroute 1220 traffic to the test management network. Further, benchmarking is 1221 performed on a "black-box" basis, relying solely on measurements 1222 observable external to the controller. Security features mentioned 1223 in the RFC 7432 will affect the test results. Special capabilities 1224 SHOULD NOT exist in the controller specifically for benchmarking 1225 purposes. Any implications for network security arising from the 1226 controller SHOULD be identical in the lab and in production networks. 1228 8. References 1230 8.1. Normative References 1232 [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for 1233 Network Interconnect Devices", RFC 2544, 1234 DOI 10.17487/RFC2544, March 1999, 1235 . 1237 [RFC2899] Ginoza, S., "Request for Comments Summary RFC Numbers 1238 2800-2899", RFC 2899, DOI 10.17487/RFC2899, May 2001, 1239 . 1241 [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 1242 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, 1243 May 2017, . 1245 8.2. Informative References 1247 [RFC7432] Sajassi, A., Ed., Aggarwal, R., Bitar, N., Isaac, A., 1248 Uttaro, J., Drake, J., and W. Henderickx, "BGP MPLS-Based 1249 Ethernet VPN", RFC 7432, DOI 10.17487/RFC7432, February 1250 2015, . 1252 [RFC7623] Sajassi, A., Ed., Salam, S., Bitar, N., Isaac, A., and W. 1253 Henderickx, "Provider Backbone Bridging Combined with 1254 Ethernet VPN (PBB-EVPN)", RFC 7623, DOI 10.17487/RFC7623, 1255 September 2015, . 1257 Appendix A. Appendix 1259 Authors' Addresses 1261 Sudhin Jacob (editor) 1262 Independent 1263 Bangalore 1264 India 1266 Phone: +91 8061212543 1267 Email: sudhinjacob@rediffmail.com 1269 Kishore Tiruveedhula 1270 Juniper Networks 1271 10 Technology Park Dr 1272 Westford, MA 01886 1273 USA 1275 Phone: +1 9785898861 1276 Email: kishoret@juniper.net