Benchmarking Methodology WG R.Papneja Internet Draft Isocore Intended status: Informational Expires: April 2011 B.Parise Cisco Systems Susan Hares Huawei October 18, 2010 Basic BGP Convergence Benchmarking Methodology for Data Plane Convergence draft-papneja-bgp-basic-dp-convergence-00.txt Abstract BGP is widely deployed and used by several service providers as the default Inter AS routing protocol. It is of utmost importance to ensure that when a BGP peer or a downstream link of a BGP peer fails, the alternate paths are rapidly used and routes via these alternate paths are installed. This document provides the basic BGP Benchmarking Methodology using existing BGP Convergence Terminology, RFC-4098. Status of this Memo This Internet-Draft is submitted to IETF in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html This Internet-Draft will expire on April 18, 2009. Papneja, et al. Expires April 18, 2011 [Page 1] Internet-Draft BGP Convergence Methodology October 2010 Copyright Notice Copyright (c) 2010 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. This document may contain material from IETF Documents or IETF Contributions published or made publicly available before November 10, 2008. The person(s) controlling the copyright in some of this material may not have granted the IETF Trust the right to allow modifications of such material outside the IETF Standards Process. Without obtaining an adequate license from the person(s) controlling the copyright in such materials, this document may not be modified outside the IETF Standards Process, and derivative works of it may not be created outside the IETF Standards Process, except to format it for publication as an RFC or to translate it into languages other than English. Table of Contents 1. Introduction 3 1.1. Document Scope 5 2. Existing definitions and requirements 5 3. Test Topologies 6 3.1. General Reference Topology 6 4. Test Considerations 8 4.1. Number of Peers 8 4.2. Number of Routes per Peer 8 4.3. Policy Processing/Reconfiguration 9 4.4. Configured Parameters (Timers, etc..) 9 4.5. Interface Types 10 4.6. Measurement Accuracy 10 4.7. Measurement Statistics 11 4.8. Authentication 11 4.9. Convergence Events 11 4.10. High Availability 12 Papneja, et al. Expires April 18, 2011 [Page 2] Internet-Draft BGP Convergence Methodology October 2010 5. Test Cases 12 5.1. Basic Convergence Tests 12 5.1.1. RIB-IN Convergence 12 5.1.2. RIB-OUT Convergence 14 5.1.3. eBGP Convergence 15 5.1.4. iBGP Convergence 15 5.1.5. eBGP Multihop Convergence 16 5.2. BGP Failure/Convergence Events 17 5.2.1. Physical Link Failure on DUT End 17 5.2.2. Physical Link Failure on Remote/Emulator End 18 5.2.3. ECMP Link Failure on DUT End 19 5.3. BGP Adjacency Failure (Non-Physical Link Failure) on Emulator 19 5.4. BGP Hard Reset Test cases 20 5.4.1. BGP Non-Recovering Hard Reset Event on DUT 20 5.5. BGP Soft Reset 22 5.6. BGP Route Withdrawal Convergence Time 23 5.7. BGP Path Attribute Change Convergence Time 25 5.8. BGP Graceful Restart Convergence Time 26 6. Reporting Format 28 7. Security Considerations 30 8. IANA Considerations 31 9. References 31 9.1. Normative References 31 9.2. Informative References 31 Authors Addresses 33 1. Introduction This document defines the methodology for benchmarking data plane FIB convergence performance of BGP in router and switches for simple topologies of 3 or 4 nodes. The methodology proposed in this document applies to both IPv4 and IPv6 and if a particular test is unique to one version, it is marked accordingly. For IPv6 benchmarking the device under test will require the support of Multi-Protocol BGP (MP-BGP) [RFC2858, RFC2545]. The scope of this companion document is limited to basic BGP protocol FIB convergence measurements. BGP extensions outside of Papneja, et al. Expires April 18, 2011 [Page 3] Internet-Draft BGP Convergence Methodology October 2010 carrying IPv6 in (MP-BGP) [RFC2858, RFC2545] are outside the scope of this document. Interaction with IGPs (IGP interworking) is outside the scope of this document. 1.1 Precise Benchmarking definition Since benchmarking is science of precision, let us restate the purpose of this document in benchmarking terms. This document defines methodology to test - data plane convergence on a single BGP device that supports the BGP [RFC4271] functionality; - in test topology of 3 or 4 nodes, - using Basic BGP. Data plane convergence is defined as the completion of all FIB changes so that all forwarded traffic now takes the new proposed route. RFC 4098 defines the terms BGP device, FIB and the forwarded traffic. Data plane convergence is different than control plane convergence within a node. Basic BGP is defined as RFC 4271 functional with Multi-Protocol BGP (MP-BGP) [RFC2858, RFC2545] for IPv6. The use of other extensions of BGP to support layer-2, layer-3 virtual private networks (VPN) are out of scope of this document. The terminology used in this document is defined in [RFC4098]. One additional term is defined in this draft: data plane BGP convergence. 1.2 Purpose of BGP FIB (data plane) convergence In the current Internet architecture the Inter-Autonomous System (inter-AS) transit is primarily available through BGP. To maintain a reliable connectivity within intra-domains or across inter-domains, fast recovery from failures remains most critical. To ensure minimal traffic losses, many service providers are requiring BGP implementations to converge the entire Internet routing table within sub-seconds at FIB level. Furthermore, to compare these numbers amongst various devices, service providers are also looking at ways to standardize the convergence measurement methods. This document offers test methods for simple topologies. These simple tests will provide a quick high- Papneja, et al. Expires April 18, 2011 [Page 4] Internet-Draft BGP Convergence Methodology October 2010 level check, of the BGP data plane convergence across multiple implementations. 1.2 Control Plane Convergence The convergence of BGP occurs at two levels: RIB and FIB convergence. RFC 4098 defines terms for BGP control plane convergence. Methodologies which test control plane convergence are out of scope for this draft. 1.3 Benchmarking Testing In order to ensure that the results obtained in tests are repeatable, careful setup of initial conditions and exact steps are required. This document proposes these initial conditions, test steps, and result checking. To ensure uniformity of the results all optional parameters SHOULD be disabled and all settings SHOULD be changed to default, these may include BGP timers as well. 2. Existing definitions and requirements RFC 1242, "Benchmarking Terminology for Network Interconnect Devices" [RFC1242] and RFC 2285, "Benchmarking Terminology for LAN Switching Devices" [RFC2285] SHOULD be reviewed in conjunction with this document. WLAN-specific terms and definitions are also provided in Clauses 3 and 4 of the IEEE 802.11 standard [802.11]. Commonly used terms may also be found in RFC 1983 [RFC1983]. For the sake of clarity and continuity, this document adopts the general template for benchmarking terminology set out in Section 2 of RFC 1242. Definitions are organized in alphabetical order, and grouped into sections for ease of reference. The following terms are assumed to be taken as defined in RFC 1242 [RFC1242]: Throughput, Latency, Constant Load, Frame Loss Rate, and Overhead Behavior. In addition, the following terms are taken as defined in RFC 2285 [RFC2285]: Forwarding Rates, Maximum Forwarding Rate, Loads, Device Under Test (DUT), and System Under Test (SUT). Papneja, et al. Expires April 18, 2011 [Page 5] Internet-Draft BGP Convergence Methodology October 2010 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119]. 3. Test Topologies This section describes simple test setups for use in BGP benchmarking tests measuring convergence of the FIB (data plane) after the BGP updates has been received. These simple test nodes have 3 or 4 nodes with the following configuration: 1. Basic Test Setup 2. Three node setup for iBGP or eBGP convergence 3. Setup for eBGP multihop test scenario 4. Four node setup for iBGP or eBGP convergence Individual tests refer to these topologies. Figures 1-4 use the following conventions AS-X: Autonomous System X Loopback Int: Loopback interface on the BGP enabled device R2: Helper router 3.1. General Reference Topology Emulator acts as 1 or more BGP peers for different testcases. ----------- ------------- | | traffic interfaces | | | |-----------------------1---- | tx | | |-----------------------2---- | tr1 | | |-----------------------3-----| tr2 | | DUT | routing interfaces | Emulator | | | | | | Drr1|--------------------------- |Err1 | | | BGP Peering | | | Drr2|---------------------------- |Err2 | | | BGP Peering | | ----------- ------------- Figure 1 Basic Test Setup Papneja, et al. Expires April 18, 2011 [Page 6] Internet-Draft BGP Convergence Methodology October 2010 ------------- ------------- ------------- | | | | | | | | | | | | | HLP | | DUT | | Emulator | | (AS-X) |--------| (AS-Y) |-----------| (AS-Z) | | | | | | | | | | | | | | | | | | | ------------- ------------- ------------- Figure 2 Three Node Setup for eBGP and iBGP Convergence ------------- ------------- ------------- | | | | | | | | | | | | | R1 | | DUT | | Emulator | | (AS-X) |--------| (AS-Y) |-----------| (AS-Z) | | | | | | | | | | | | | | | | | | | ------------- ------------- ------------- |Loopback-Int |Loopback-Int | | + + Figure 3 BGP Convergence for eBGP Multihop Scenario ---------- ---------- ---------- ---------- | | | | | | | | | | | | | | | | | R1 | | DUT2 | | DUT1 | |Emulator | | (AS-X) |-----| (AS-X) |-----| (AS-Y) |-----| (AS-Z) | | | | | | | | | | | | | | | | | | | | | | | | | ---------- ---------- ---------- ---------- Figure 4 Four Node Setup for EBGP and IBGP Convergence Papneja, et al. Expires April 18, 2011 [Page 7] Internet-Draft BGP Convergence Methodology October 2010 4. Test Considerations The test cases for measuring convergence for iBGP and eBGP are different. Both iBGP and eBGP use different mechanisms to advertise, install and learn the routes. Typically, an iBGP route on the DUT is installed and exported only when the next-hop is reachable. For eBGP the route is installed on the DUT with the remote interface address as the next-hop with the exception of the multihop case. 4.1. Number of Peers Number of Peers is defined as the number of BGP neighbors or sessions the DUT has at the beginning of the test. The peers are established before the tests begin. The relationship could be either, iBGP or eBGP peering depending upon the test case requirement. The DUT establishes one or more BGP sessions with one more emulated routers or helper nodes. Additional peers can be added based on the testing requirements. The number of peers enabled during the testing should be well documented in the report matrix. 4.2. Number of Routes per Peer It Number of Routes per Peer is defined as the number of routes advertized or learnt by the DUT per session or through neighbor relationship with an emulator or helper node. The tester, emulating as neighbor MUST advertise at least one route per peer. Each test must run must identify the route stream in terms of route packing, route mixture, and number of routes. This route stream must be well documented in the reporting stream. RFC 4098 defines these terms. It is RECOMMENDED that the user may consider advertizing the entire current Internet routing table per peering session using an Internet route mixture with unique or non-unique routes. If multiple peers are used, it is important to precisely document the timing sequence between the peer sending routes (as defined in RFC 4098). Papneja, et al. Expires April 18, 2011 [Page 8] Internet-Draft BGP Convergence Methodology October 2010 4.3. Policy Processing/Reconfiguration The DUT MUST run one baseline test where policy is Minimum policy as defined in RFC 4098. Additional runs may be done with policy set-up before the tests begin. Exact policy settings should be documented as part of the test. 4.4. Configured Parameters (Timers, etc..) There are configured parameters and timers that may impact the measured BGP convergence times. The benchmark metrics MAY be measured at any fixed values for these configured parameters. It is RECOMMENDED these configure parameters have two settings: a) basic-test, and b)values as expected in the operational network. All optional BGP settings MUST be kept consistent across iterations of any specific tests Examples of the configured parameters that may impact measured BGP convergence time include, but are not limited to: 1. Interface failure detection timer 2. BGP Keepalive timer 3. BGP Holdtime 4. BGP update delay timer 5. ConnectRetry timer 6. TCP Segment Size 7. Minimum Route Advertisement Interval (MRAI) 8. MinASOriginationInterval (MAOI) 9. Route Flap Dampening parameters 10. TCP MD5 The basic-test settings for the parameters should be: Papneja, et al. Expires April 18, 2011 [Page 9] Internet-Draft BGP Convergence Methodology October 2010 1. Interface failure detection timer (0 ms) 2. BGP Keepalive timer (1 min) 3. BGP Holdtime (3 min) 4. BGP update delay timer (0 s) 5. ConnectRetry timer (1 s) 6. TCP Segment Size (4096) 7. Minimum Route Advertisement Interval (MRAI)(0 s) 8. MinASOriginationInterval (MAOI) (0 s) 9. Route Flap Dampening parameters (off) 10. TCP MD5 (off) 4.5. Interface Types The type of media dictate which test cases may be executed, each interface type has unique mechanism for detecting link failures and the speed at which that mechanism operates will influence the measurement results. All interfaces MUST be of the same media and throughput for each test case. 4.6. Measurement Accuracy Since observed packet loss is used to measure the route convergence time, the time between two successive packets offered to each individual route is the highest possible accuracy of any packet-loss based measurement. When packet jitter is much less than the convergence time, it is a negligible source of error and hence it will be treated as within tolerance. An exterior measurement on the input media (such Ethernet)is defined by this specification. Papneja, et al. Expires April 18, 2011 [Page 10] Internet-Draft BGP Convergence Methodology October 2010 4.7. Measurement Statistics The benchmark measurements may vary for each trial, due to the statistical nature of timer expirations, CPU scheduling, etc. It is recommended to repeat the test multiple times. Evaluation of the test data must be done with an understanding of generally accepted testing practices regarding repeatability, variance and statistical significance of a small number of trials. For any repeated tests that are averaged to remove variance, all parameters MUST remain the same. 4.8. Authentication Authentication in BGP is done using the TCP MD5 Signature Option [RFC2385]. The processing of the MD5 hash, particularly in devices with a large number of BGP peers and a large amount of update traffic, can have an impact on the control plane of the device. If authentication is enabled, it SHOULD be documented correctly in the reporting format 4.9. Convergence Events Convergence events or triggers are defined as abnormal occurrences in the network, which initiate route flapping in the network, and hence forces the re-convergence of a steady state network. In a real network, a series of convergence events may cause convergence latency operators desire to test. These convergence events must be defined in terms of the sequences defined in RFC 4098. This basic document begins all tests with a router initial set-up. Additional documents will define BGP data plane convergence based on peer initialization. The convergence events may or may not be tied to the actual failure A Soft Reset (RFC 4098) does not clear the RIB or FIB tables. A Hard reset clears the BGP peer sessions, the RIB tables, and FIB tables. Papneja, et al. Expires April 18, 2011 [Page 11] Internet-Draft BGP Convergence Methodology October 2010 4.10. High Availability Due to the different Non-Stop-Routing (sometimes referred to High- Availability) solutions available from different vendors, it is RECOMMENDED that any redundancy available in the routing processors should be disabled during the convergence measurements. 5. Test Cases All tests defined under this section assume the following: BGP peers should be brought to BGP Peer established state. a. Furthermore the traffic generation and routing should be verified in the topology 5.1. Basic Convergence Tests These test cases measure characteristics of a BGP implementation in non-failure scenarios like: a. RIB-IN Convergence b. RIB-OUT Convergence c. eBGP Convergence d. iBGP Convergence 5.1.1. RIB-IN Convergence Objective: This test measures the convergence time taken to receive and install a route in RIB using BGP Reference Test Setup: This test uses the setup as shown in figure 1 Procedure: Papneja, et al. Expires April 18, 2011 [Page 12] Internet-Draft BGP Convergence Methodology October 2010 a. All variables affecting Convergence should be set to a basic test state (as defined in section 4-4). b. Establish BGP adjacency between DUT and peer x of Emulator; c. To ensure adjacency establishment, wait for 3 KeepAlives from the DUT or a configurable delay before proceeding with the rest of the test; d. Start the traffic from the Emulator peer-x towards the DUT targeted at a routes specified in route mixture (ex. route A) Initially no traffic SHOULD be observed on the egress interface as the route A is not installed in the forwarding database of the DUT. e. Advertise route A from the Peer-x to the DUT and record the time; This is Tup(EMx,Rt-A). (nick-name XMT-Rt-time) f. Record the time when the route-A from Peer-x is received at the DUT. This Tup(DUT,Rt-A). It is nick named is RCV-Rt-time g. Record the time when the traffic targeted towards route A is received by Emulator on appropriate traffic egress interface. rd If 3 party route (traffic-egress 2), or BGP peer route interfaces. This is TR(TDx,Rt-A). This is "nick-named" DUT-XMT-Data-Time. h. The difference between the Tup(TDx,RT-A) and traffic received time (TR (TDr, Rt-A) is the FIB Convergence Time for route-A in the route mixture. A full convergence for the route update is the measurement between the 1st route (Route-A) and the last route (Rt-last) Route update convergence is TR(TDr, RT-last)- Tup(DUT, Rt-A) or (DUT-XMT-Data-Time - RCV-Rt-Time)(rt-A) Papneja, et al. Expires April 18, 2011 [Page 13] Internet-Draft BGP Convergence Methodology October 2010 Note: It is recommended that a single test with the same route mixture be repeated several times. A report should provide the Stand deviation of all tests and the average. Running tests with a varying number of routes and route mixtures is important to get a full characterization of a single peer. 5.1.2. RIB-OUT Convergence Objective: This test measures the convergence time taken by an implementation to receive, install and advertise a route using BGP Reference Test Setup: This test uses the setup as shown in figure 2 Procedure: a. The Helper node (HLP) run same version of BGP as DUT; b. All devices MUST be synchronized using NTP or some local reference clock; c. All configuration variables for HLP, DUT, and Emulator SHOULD be set to the same values. These values MAY be basic-test or a unique set completely described in the test set-up. d. Establish BGP adjacency between DUT and Emulator e. Establish BGP adjacency between DUT and Helper Node f. To ensure adjacency establishment, wait for 3 KeepAlives from the DUT or a configurable delay before proceeding with the rest of the test g. Start the traffic from the Emulator towards the Helper Node targeted at a specific route say route A. Initially no traffic SHOULD be observed on the egress interface as the route-A is not installed in the forwarding database of the DUT h. Advertise routeA from the Emulator to the DUT and note the time. This is Tup(EMx, Route-A). (nick-name EM-XMT-Rt-Time) Papneja, et al. Expires April 18, 2011 [Page 14] Internet-Draft BGP Convergence Methodology October 2010 i. Record when Route-A is received by DUT. This is Tup(DUTr, Route-A). (nick-name DUT-RCV-Rt-Time) j. Record the time when the ROUTE forward by DUT toward the Helper node. This is Tup(DUTx, Rt-A). (nick-name DUT-XMT-Rt-Time). k. Record the time when the traffic targeted towards route-A is received on the Route Egress Interface toward peer-X. This is TR(EMr, Route-A). (nick-name DUT-XMT-Data Time). FIB convergence = (DUT-RCV-Rt-Time - DUT-XMT-Data-Time). RIB convergence = (DUT-RCV-Rt-Time - DUT-XMT-Rt-Time). Convergence for a route stream is characterized by a) Individual route convergence for FIB, RIB b) All route convergence of FIB-convergence =DUT-RCV-Rt-Time(A)-DUT-XMT-Data-Time(last) RIB-convergence =DUT-RCV-Rt-Time(A)-DUT-XMT-Rt-Time(last) 5.1.3. eBGP Convergence Objective: This test measures the convergence time taken by an implementation to receive, install and advertise a route in an eBGP Scenario Reference Test Setup: This test uses the setup as shown in figure 2, and the scenarios described in RIB-IN and RIB-OUT are applicable to this test case. 5.1.4. iBGP Convergence Objective: Papneja, et al. Expires April 18, 2011 [Page 15] Internet-Draft BGP Convergence Methodology October 2010 This test measures the convergence time taken by an implementation to receive, install and advertise a route in an iBGP Scenario Reference Test Setup: This test uses the setup as shown in figure 2, and the test scenarios listed in RIB-IN and RIB-OUT are applicable to this test case. 5.1.5. eBGP Multihop Convergence Objective This test measures the convergence time taken by an implementation to receive, install and advertise a route in an eBGP Multihop Scenario Reference Test Setup: This test uses the setup as shown in figure 3. Two DUTs are used along with a helper node. Procedure: a. The DUT2 is the same model as DUT and runs the same BGP implementation as DUT. b. All devices to be synchronized using NTP c. All variables affecting Convergence like authentication, policies, timers should be set to basic-settings. d. All 3 devices, DUT, Emulator and Helper Node are configured as different Autonomous Systems e. Loopback Interfaces configured on DUT and Helper Node and connectivity is established between them using any config options available on the DUT f. Establish BGP adjacency between DUT1 and Emulator g. Establish BGP adjacency between DUT2 and Helper Node h. Establish BGP adjacency between DUT 1 and DUT 2 i. To ensure adjacency establishment, wait for 3 KeepAlives from the DUT1 and DUT2 or a configurable delay before proceeding with the rest of the test j. Start the traffic from the Emulator towards the Helper Node targeted at a specific route say routeA. Papneja, et al. Expires April 18, 2011 [Page 16] Internet-Draft BGP Convergence Methodology October 2010 k. Initially no traffic SHOULD be observed on the egress interface as the routeA is not installed in the forwarding database of the DUT l. Advertise routeA from the Emulator to the DUT and note the time. (Tup(EMx,RouteA) - this is nicknamed (Route-Rec-time). m. Record the time when the traffic targeted towards routeA is received from Egress Interface of DUT on emulator. This is TR(EMr,DUT), nicknamed (Data Receive time) n. The following equation represents the FIB Convergence multi- node eBGP Multihop Convergence Time = (Rt-RecTime - Data-RcvTime). Note: It is recommended that the test be repeated with varying number of routes and route mixtures. With each set route mixture, the test should be repeated multiple times. The results should record average, mean, Standard Deviation. 5.2. BGP Failure/Convergence Events 5.2.1. Physical Link Failure on DUT End Objective: This test measures the route convergence time due to local link failure event at DUT's Local Interface Reference Test Setup: This test uses the setup as shown in figure 1. Shutdown event is defined as an administrative shutdown event on the DUT. Procedure: a. All variables affecting Convergence like authentication, policies, timers should be set to basic-test policy. Papneja, et al. Expires April 18, 2011 [Page 17] Internet-Draft BGP Convergence Methodology October 2010 b. Establish 2 BGP adjacencies from DUT to Emulator, one over the peer interface and the other using a second peer interface. c. Advertise the same route, route A over both the adjacencies and (Tx1)Interface to be the preferred next hop. d. To ensure adjacency establishment, wait for 3 KeepAlives from the DUT or a configurable delay before proceeding with the rest of the test. e. Start the traffic from the Emulator towards the DUT targeted at a specific route say route A. Initially traffic would be observed on the best egress route (Err1) instead of Trr2 f. Trigger the shutdown event of Best Egress Interface on DUT (Drr1). g. Measure the Convergence Time for the event to be detected and traffic to be forwarded to Next-Best Egress Interface (rr2). Time = Data-detect(rr2) - Shutdown time. h. Stop the offered load and wait for the queues to drain and Restart i. Bring up the link on DUT Best Egress Interface j. Measure the convergence time taken for the traffic to be rerouted from (rr2) to Best Interface (rr1) Time = Data-Detect(rr1) - Shutdown-time. k. It is recommended that the test be repeated with varying number of routes and route mixtures or with number of routes & route mixtures closer to what is deployed in operational networks 5.2.2. Physical Link Failure on Remote/Emulator End Objective: This test measures the route convergence time due to local link failure event at Tester's Local Interface Papneja, et al. Expires April 18, 2011 [Page 18] Internet-Draft BGP Convergence Methodology October 2010 Reference Test Setup: This test uses the setup as shown in figure 1. Shutdown event is defined as shutdown of the local interface of Tester via logical shutdown event. The procedure used in 5.2.1 is used for the termination. 5.2.3. ECMP Link Failure on DUT End Objective: This test measures the route convergence time due to local link failure event at ECMP Member. The FIB configuration and BGP is set to allow two ECMP routes to be installed. However, policy directs the routes to be sent only over one of the paths. Reference Test Setup: This test uses the setup as shown in figure 1, and the procedure uses 5.2.1. 5.3. BGP Adjacency Failure (Non-Physical Link Failure) on Emulator Objective: This test measures the route convergence time due to BGP Adjacency Failure on Emulator Reference Test Setup: This test uses the setup as shown in figure 1 Procedure: a. All variables affecting Convergence like authentication, policies, timers should be basic-policy set. Papneja, et al. Expires April 18, 2011 [Page 19] Internet-Draft BGP Convergence Methodology October 2010 b. Establish 2 BGP adjacencies from DUT to Emulator, one over the Best Egress Interface and the other using the Next-Best Egress Interface c. Advertise the same route, routeA over both the adjacencies and make Best Egress Interface to be the preferred next hop d. To ensure adjacency establishment, wait for 3 KeepAlives from the DUT or a configurable delay before proceeding with the rest of the test e. Start the traffic from the Emulator towards the DUT targeted at a specific route say routeA. Initially traffic would be observed on the Best Egress interface f. Remove BGP adjacency via a software adjacency down on the Emulator on the Best Egress Interface Time = BGPadj-down-time - nicknamed BGPpeer-down. g. Measure the Convergence Time for the event to be detected and traffic to be forwarded to Next-Best Egress Interface This time is Tr-rr2 nicknamed - TR2-traffic-on Convergence = TR2-traffic-on - BGPpeer-down h. Stop the offered load and wait for the queues to drain and Restart i. Bring up BGP adjacency on the Emulator over the Best Egress Interface Time = BGP-adj-up - nicknamed BGPpeer-up j. Measure the convergence time taken for the traffic to be rerouted to Best Interface Time = Tr-rr1 is nicknamed TR1-traffic-on. 5.4. BGP Hard Reset Test cases 5.4.1. BGP Non-Recovering Hard Reset Event on DUT Objective: Papneja, et al. Expires April 18, 2011 [Page 20] Internet-Draft BGP Convergence Methodology October 2010 This test measures the route convergence time due to Hard Reset on the DUT Reference Test Setup: This test uses the setup as shown in figure 1 Procedure: a. The requirement for this test case is that the Hard Reset Event should be non-recovering and should affect only the adjacency between DUT and Emulator on the Best Egress Interface b. All variables affecting SHOULD be set to basic-test values c. Establish 2 BGP adjacencies from DUT to Emulator, one over the Best Egress Interface and the other using the Next-Best Egress Interface d. Advertise the same route, routeA over both the adjacencies and make Best Egress Interface to be the preferred next hop e. To ensure adjacency establishment, wait for 3 KeepAlives from the DUT or a configurable delay before proceeding with the rest of the test f. Start the traffic from the Emulator towards the DUT targeted at a specific route say routeA. Initially traffic would be observed on the Best Egress interface g. Trigger the Hard Reset event of Best Egress Interface on DUT h. Measure the Convergence Time for the event to be detected and traffic to be forwarded to Next-Best Egress Interface Time of convergence = time-traffic flow - time-reset. i. Stop the offered load and wait for the queues to drain and Restart j. It is recommended that the test be repeated with varying number of routes and route mixtures or with number of routes & route mixtures closer to what is deployed in operational networks k. When varying number of routes are used, convergence Time is measured using the Loss Derived method [IGP-Data] Papneja, et al. Expires April 18, 2011 [Page 21] Internet-Draft BGP Convergence Methodology October 2010 l. Convergence Time in this scenario is influenced by Failure detection time on Tester, BGP Keep Alive Time and routing, forwarding table update time 5.5. BGP Soft Reset Objective: This test measures the route convergence time taken by an implementation to service a BGP Route Refresh message and advertise a route Reference Test Setup: This test uses the setup as shown in figure 2 Procedure: a. The BGP implementation on DUT & Helper Node needs to support BGP Route Refresh Capability [RFC 2918] b. All devices to be synchronized using NTP c. All variables affecting Convergence like authentication, policies, timers should be set to basic-test defaults. d. DUT and Helper Node are configured in the same Autonomous System whereas Emulator is configured under a different Autonomous System e. Establish BGP adjacency between DUT and Emulator f. Establish BGP adjacency between DUT and Helper Node g. To ensure adjacency establishment, wait for 3 KeepAlives from the DUT or a configurable delay before proceeding with the rest of the test h. Configure a policy under BGP on Helper Node to deny routes received from DUT i. Advertise routeA from the Emulator to the DUT j. The DUT will try to advertise the route to Helper Node will be denied k. Wait for 3 KeepAlives l. Start the traffic from the Emulator towards the Helper Node targeted at a specific route say routeA. Initially no traffic Papneja, et al. Expires April 18, 2011 [Page 22] Internet-Draft BGP Convergence Methodology October 2010 would be observed on the Egress interface, as routeA is not present m. Remove the policy on Helper Node and issue a Route Refresh request towards DUT. Note the timestamp of this event. This is the RefreshTime n. Record the time when the traffic targeted towards routeA is received on the Egress Interface. This is RecTime o. The following equation represents the Route Refresh Convergence Time per route i. Route Refresh Convergence Time = (RecTime - RefreshTime) 5.6. BGP Route Withdrawal Convergence Time Objective: This test measures the route convergence time taken by an implementation to service a BGP Withdraw message and advertise the withdraw Reference Test Setup: This test uses the setup as shown in figure 2 Procedure: a. This test consists of 2 steps to determine the Total Withdraw Processing Time b. Step 1: i. All devices to be synchronized using NTP ii. All variables should be set to basic-test parameters. iii. DUT and Helper Node are configured in the same Autonomous System whereas Emulator is configured under a different Autonomous System iv. Establish BGP adjacency between DUT and Emulator v. To ensure adjacency establishment, wait for 3 KeepAlives from the DUT or a configurable delay before proceeding with the rest of the test vi. Start the traffic from the Emulator towards the DUT targeted at a specific route say routeA. Initially no Papneja, et al. Expires April 18, 2011 [Page 23] Internet-Draft BGP Convergence Methodology October 2010 traffic would be observed on the Egress interface as the routeA is not present on DUT. vii. Advertise routeA from the Emulator to the DUT viii. The traffic targeted towards routeA is received on the Egress Interface ix. Now the Tester sends request to withdraw routeA to DUT. TRx(Awith) nicknamed WdrawTime1 x. Record the time when no traffic is observed on the Egress Interface. This is the RouteRemoveTime1(A) WdrawConvTime1 = RouteRemoveTime1(A) xi. The difference between the RouteRemoveTime1 and WdrawTime1 is the WdrawConvTime1 c. Step 2: i. Continuing from Step 1, re-advertise routeA back to DUT from Tester ii. The DUT will try to advertise the routeA to Helper Node (assumption there exists a session between DUT and helper node) iii. Start the traffic from the Emulator towards the Helper Node targeted at a specific route say routeA. Traffic would be observed on the Egress interface after routeA is received by the Helper Node WATime=time traffic first flows iv. Now the Tester sends a request to withdraw routeA to DUT. This is the WdrawTime2 WAWtime-TRx(RouteA) = is nicknamed WdrawTime2 v. DUT processes the withdraw and sends it to Helper Node vi. Record the time when no traffic is observed on the Egress Interface of Helper Node. This is the Papneja, et al. Expires April 18, 2011 [Page 24] Internet-Draft BGP Convergence Methodology October 2010 TR-WAW(DUT,RouteA) = RouteRemoveTime2 vii. Total withdraw processing time is TotalWdrawTime = ((RouteRemoveTime2 - WdrawTime2) - WdrawConvTime1) 5.7. BGP Path Attribute Change Convergence Time Objective: This test measures the route convergence time taken by an implementation to service a BGP Path Attribute Change Reference Test Setup: This test uses the setup as shown in figure 1 Procedure: a. This test only applies to Well-Known Mandatory Attributes like Origin, AS Path, Next Hop b. In each iteration of test only one of these mandatory attributes need to be varied whereas the others remain the same c. All devices to be synchronized using NTP d. All variables should be set to basic-test parameters e. Advertise the route, routeA over the Best Egress Interface only, making it the preferred next hop f. To ensure adjacency establishment, wait for 3 KeepAlives from the DUT or a configurable delay before proceeding with the rest of the test g. Start the traffic from the Emulator towards the DUT targeted at the specific route say routeA. Initially traffic would be observed on the Best Egress interface h. Now advertise the same route routeA on the Next-Best Egress Interface but by varying one of the well-known mandatory attributes to have a preferred value over that interface. The Papneja, et al. Expires April 18, 2011 [Page 25] Internet-Draft BGP Convergence Methodology October 2010 other values need to be same as what was advertised on the Best-Egress adjacency. TRx(Path-Change) = Path Change Event Time i. Measure the Convergence Time for the event to be detected and traffic to be forwarded to Next-Best Egress Interface DUT(Path-Change, RouteA) = Path-switch time Convergence = Path-switch time - Path Change Event Time. j. Stop the offered load and wait for the queues to drain and Restart 5.8. BGP Graceful Restart Convergence Time Objective: This test measures the route convergence time taken by an implementation during a Graceful Restart Event Reference Test Setup: This test uses the setup as shown in figure 4 Procedure: a. It measures the time taken by an implementation to service a BGP Graceful Restart Event and advertise a route b. The Helper Nodes are the same model as DUT and run the same BGP implementation as DUT c. The BGP implementation on DUT & Helper Node needs to support BGP Graceful Restart Mechanism [RFC4724] d. All devices to be synchronized using NTP e. All variables are set to basic-test values. f. DUT and Helper Node-1 are configured in the same Autonomous System whereas Emulator and Helper Node-2 are configured under different Autonomous Systems g. Establish BGP adjacency between DUT and Helper Nodes Papneja, et al. Expires April 18, 2011 [Page 26] Internet-Draft BGP Convergence Methodology October 2010 h. Establish BGP adjacency between Helper Node-2 and Emulator i. To ensure adjacency establishment, wait for 3 KeepAlives from the DUT or a configurable delay before proceeding with the rest of the test j. Configure a policy under BGP on Helper Node-1 to deny routes received from DUT k. Advertise routeA from the Emulator to Helper Node-2 l. Helper Node-2 advertises the route to DUT and DUT will try to advertise the route to Helper Node-1 which will be denied m. Wait for 3 KeepAlives n. Start the traffic from the Emulator towards the Helper Node-1 targeted at the specific route say routeA. Initially no traffic would be observed on the Egress interface as the routeA is not present o. Perform a Graceful Restart Trigger Event on DUT and note the time. This is the GREventTime p. Remove the policy on Helper Node-1 q. Record the time when the traffic targeted towards routeA is received on the Egress Interface. TRr(DUT, routeA). This is nicknamed RecTime. r. The following equation represents the Graceful Restart Convergence Time i. Graceful Restart Convergence Time = ((GREventTime - RecTime) - RIB-IN) s. It is assumed in this test case that after a Switchover is triggered on the DUT, it will not have any cycles to process BGP Refresh messages. The reason for this assumption is that there is a narrow window of time where after switchover when we remove the policy from Helper Node -1, implementations might generate Route-Refresh automatically and this request might be serviced before the DUT actually switches over and reestablishes BGP adjacencies with the peers Papneja, et al. Expires April 18, 2011 [Page 27] Internet-Draft BGP Convergence Methodology October 2010 6. Reporting Format For each test case, it is recommended that the reporting tables below are completed and all time values SHOULD be reported with resolution as specified in [RFC 4098]. Parameter Units Test case Test case number Test topology 1,2,3 or 4 Parallel links Number of parallel links Interface type GigE, POS, ATM, other Convergence Event Hard reset, Soft reset, link failure, or other defined eBGP sessions Number of eBGP sessions iBGP sessions Number of iBGP sessions eBGP neighbor Number of eBGP neighbors iBGP neighbor Number of iBGP neighbors Routes per peer Number of routes Total unique routes Number of routes Total non-unique routes Number of routes IGP configured ISIS, OSPF, static, or other Route Mixture Description of Route mixture Route Packing Number of routes in an update Policy configured Yes, No Packet size offered to the DUT Bytes Offered load Packets per second Packet sampling interval on Seconds tester Forwarding delay threshold Seconds Timer value configured on DUT Interface failure indication Seconds delay Hold timer Seconds MinRouteAdvertisementInterval Seconds (MRAI) MinASOriginationInterval Seconds (MAOI) Keepalive Seconds ConnectRetry Seconds TCP Parameters for DUT and Papneja, et al. Expires April 18, 2011 [Page 28] Internet-Draft BGP Convergence Methodology October 2010 tester MSS Bytes Slow start threshold Bytes Maximum window size Bytes Test Details: a. If the Offered Load matches a subset of routes, describe how this subset is selected. b. Describe how the Convergence Event is applied; does it cause instantaneous traffic loss or not. c. If there is any policy configured, describe the configured policy. Complete the table below for the initial Convergence Event and the reversion Convergence Event. Parameter Unit Conversion Event Initial or reversion Traffic Forwarding Metrics Total number of packets Number of packets offered to DUT Total number of packets Number of packets forwarded by DUT Connectivity Packet Loss Number of packets Convergence Packet Loss Number of packets Out-of-order packets Number of packets Duplicate packets Number of packets Convergence Benchmarks Rate-derived Method[IGP- Data]: First route convergence Seconds time Full convergence time Seconds Loss-derived Method [IGP- Data]: Loss-derived convergence Seconds time Route-Specific Loss-Derived Method: Papneja, et al. Expires April 18, 2011 [Page 29] Internet-Draft BGP Convergence Methodology October 2010 Minimum R-S convergence Seconds time Maximum R-S convergence Seconds time Median R-S convergence Seconds time Average R-S convergence Seconds time Loss of Connectivity Benchmarks Loss-derived Method: Loss-derived loss of Seconds connectivity period Route-Specific loss-derived Method: Minimum LoC period [n] Array of seconds Minimum Route LoC period Seconds Maximum Route LoC period Seconds Median Route LoC period Seconds Average Route LoC period Seconds 7. Security Considerations Benchmarking activities as described in this memo are limited to technology characterization using controlled stimuli in a laboratory environment, with dedicated address space and the constraints specified in the sections above. The benchmarking network topology will be an independent test setup and MUST NOT be connected to devices that may forward the test traffic into a production network, or misroute traffic to the test management network. Further, benchmarking is performed on a "black-box" basis, relying solely on measurements observable external to the DUT/SUT. Special capabilities SHOULD NOT exist in the DUT/SUT specifically for benchmarking purposes. Any implications for network security arising from the DUT/SUT SHOULD be identical in the lab and in production networks. Papneja, et al. Expires April 18, 2011 [Page 30] Internet-Draft BGP Convergence Methodology October 2010 8. IANA Considerations This document requires no IANA considerations. 9. References 9.1. Normative References [RFC1771] Rekhter, Y. and Li, T., "A Border Gateway Protocol 4(BGP- 4)", RFC 4271, March 1995. [RFC4098] Berkowitz, H. et al., "Terminology for benchmarking BGP device convergence in control plane", RFC4098, June 2005 [RFC1242] Bradner, S., "Benchmarking terminology for network interconnection devices", RFC 1242, July 1991. [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for Network Interconnect Devices", RFC 2544, March 1999 [IGP-Data] Poretsky, S. et al., "Terminology for benchmarking Link- state IGP data plane convergence," draft-ietf-bmwg-igp- dataplane-conv-term-21, May 2010 9.2. Informative References [RFC2858] Bates, T. et al., "Multiprotocol extensions for BGP-4," RFC 2858, June 2000 [RFC2545] Marques, P. and F. Dupont, "Use of BGP-4 multiprotocol extensions for IPv6 Inter-Domain Routing," RFC2545, March 1999 [RFC3107] Rekhter, Y. and E. Rosen, "Carrying label information in BGP-4," RFC 3107, May 2001 [RFC2385] Heffernan, A., "Protection of BGP Sessions via the TCP MD5 signature option," RFC2385, August 1998 [RFC2918] Chen, E., "Route Refresh capability for BGP-4," RFC 2918, September 2000 Papneja, et al. Expires April 18, 2011 [Page 31] Internet-Draft BGP Convergence Methodology October 2010 [RFC4724] Sangli, S. et al., "Graceful restart mechanism for BGP," RFC 4724, Jan 2007 [RFC1983] Malkin, G.,"Internet Users' Glossary", RFC 1983, August 1996 Papneja, et al. Expires April 18, 2011 [Page 32] Internet-Draft BGP Convergence Methodology October 2010 Authors Addresses Rajiv Papneja Isocore 12359 Sunrise Valley Dr. STE100 Reston, VA 20191 Phone: +1.703.860.9273 Email: rpapneja@isocore.com Bhavani Parise Cisco Systems 170 West Tasman Drive San Jose, CA 95134 Phone: +408-853-6346 Email: bhavani@cisco.com Susan Hares Huawei Technologies (USA) 2330 Central Expressway Santa Clara, CA 95050 Phone: +408-330-4581 Cell: +1-734-604-0332 Email shares@huawei.com Eric Brendel Independent Consultant 154 3rd St, Fair Haven, NJ 07704 Phone:+1.732.895.1504 Email: brendel@pektel.com Mohan Nanduri Microsoft 12012 Sunset Hills Rd. Reston, VA 20190 Phone: +1.703.627.6455 Email: mnanduri@microsoft.com Jay Karthik Cisco Systems 170 West Tasman Drive San Jose, CA 95134 Phone: +1.978.319.0527 Email: jkarthik@cisco.com Papneja, et al. Expires April 18, 2011 [Page 33]