TOC 
Network Working GroupS. Poretsky
Internet-DraftAllot Communications
Intended status: InformationalB. Imhoff
Expires: September 9, 2010Juniper Networks
 K. Michielsen
 Cisco Systems
 March 08, 2010


Benchmarking Methodology for Link-State IGP Data Plane Route Convergence
draft-ietf-bmwg-igp-dataplane-conv-meth-20

Abstract

This document describes the methodology for benchmarking Link-State Interior Gateway Protocol (IGP) Route Convergence. The methodology is to be used for benchmarking IGP convergence time through externally observable (black box) data plane measurements. The methodology can be applied to any link-state IGP, such as ISIS and OSPF.

Status of this Memo

This Internet-Draft is submitted to IETF in full conformance with the provisions of BCP 78 and BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as “work in progress.”

The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt.

The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html.

This Internet-Draft will expire on September 9, 2010.

Copyright Notice

Copyright (c) 2010 IETF Trust and the persons identified as the document authors. All rights reserved.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the BSD License.

This document may contain material from IETF Documents or IETF Contributions published or made publicly available before November 10, 2008. The person(s) controlling the copyright in some of this material may not have granted the IETF Trust the right to allow modifications of such material outside the IETF Standards Process. Without obtaining an adequate license from the person(s) controlling the copyright in such materials, this document may not be modified outside the IETF Standards Process, and derivative works of it may not be created outside the IETF Standards Process, except to format it for publication as an RFC or to translate it into languages other than English.



Table of Contents

1.  Introduction and Scope
2.  Existing Definitions
3.  Test Topologies
    3.1.  Test topology for local changes
    3.2.  Test topology for remote changes
    3.3.  Test topology for local ECMP changes
    3.4.  Test topology for remote ECMP changes
    3.5.  Test topology for Parallel Link changes
4.  Convergence Time and Loss of Connectivity Period
    4.1.  Convergence Events without instant traffic loss
    4.2.  Loss of Connectivity
5.  Test Considerations
    5.1.  IGP Selection
    5.2.  Routing Protocol Configuration
    5.3.  IGP Topology
    5.4.  Timers
    5.5.  Interface Types
    5.6.  Offered Load
    5.7.  Measurement Accuracy
    5.8.  Measurement Statistics
    5.9.  Tester Capabilities
6.  Selection of Convergence Time Benchmark Metrics and Methods
    6.1.  Loss-Derived Method
        6.1.1.  Tester capabilities
        6.1.2.  Benchmark Metrics
        6.1.3.  Measurement Accuracy
    6.2.  Rate-Derived Method
        6.2.1.  Tester Capabilities
        6.2.2.  Benchmark Metrics
        6.2.3.  Measurement Accuracy
    6.3.  Route-Specific Loss-Derived Method
        6.3.1.  Tester Capabilities
        6.3.2.  Benchmark Metrics
        6.3.3.  Measurement Accuracy
7.  Reporting Format
8.  Test Cases
    8.1.  Interface failures
        8.1.1.  Convergence Due to Local Interface Failure
        8.1.2.  Convergence Due to Remote Interface Failure
        8.1.3.  Convergence Due to ECMP Member Local Interface Failure
        8.1.4.  Convergence Due to ECMP Member Remote Interface Failure
        8.1.5.  Convergence Due to Parallel Link Interface Failure
    8.2.  Other failures
        8.2.1.  Convergence Due to Layer 2 Session Loss
        8.2.2.  Convergence Due to Loss of IGP Adjacency
        8.2.3.  Convergence Due to Route Withdrawal
    8.3.  Administrative changes
        8.3.1.  Convergence Due to Local Adminstrative Shutdown
        8.3.2.  Convergence Due to Cost Change
9.  Security Considerations
10.  IANA Considerations
11.  Acknowledgements
12.  Normative References
§  Authors' Addresses




 TOC 

1.  Introduction and Scope

This document describes the methodology for benchmarking Link-State Interior Gateway Protocol (IGP) convergence. The motivation and applicability for this benchmarking is described in [Po09a] (Poretsky, S., “Considerations for Benchmarking Link-State IGP Data Plane Route Convergence,” March 2009.). The terminology to be used for this benchmarking is described in [Po09t] (Poretsky, S. and B. Imhoff, “Terminology for Benchmarking Link-State IGP Data Plane Route Convergence,” July 2009.).

IGP convergence time is measured on the data plane at the Tester by observing packet loss through the DUT. All factors contributing to convergence time are accounted for by measuring on the data plane, as discussed in [Po09a] (Poretsky, S., “Considerations for Benchmarking Link-State IGP Data Plane Route Convergence,” March 2009.). The test cases in this document are black-box tests that emulate the network events that cause convergence, as described in [Po09a] (Poretsky, S., “Considerations for Benchmarking Link-State IGP Data Plane Route Convergence,” March 2009.).

The methodology described in this document can be applied to IPv4 and IPv6 traffic and link-state IGPs such as ISIS [Ca90] (Callon, R., “Use of OSI IS-IS for routing in TCP/IP and dual environments,” December 1990.)[Ho08] (Hopps, C., “Routing IPv6 with IS-IS,” October 2008.), OSPF [Mo98] (Moy, J., “OSPF Version 2,” April 1998.)[Co08] (Coltun, R., Ferguson, D., Moy, J., and A. Lindem, “OSPF for IPv6,” July 2008.), and others.



 TOC 

2.  Existing Definitions

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14, RFC 2119 [Br97] (Bradner, S., “Key words for use in RFCs to Indicate Requirement Levels,” March 1997.). RFC 2119 defines the use of these key words to help make the intent of standards track documents as clear as possible. While this document uses these keywords, this document is not a standards track document.

This document uses much of the terminology defined in [Po09t] (Poretsky, S. and B. Imhoff, “Terminology for Benchmarking Link-State IGP Data Plane Route Convergence,” July 2009.) and uses existing terminology defined in other BMWG work. Examples include, but are not limited to:



Throughput [Ref.[Br91] (Bradner, S., “Benchmarking terminology for network interconnection devices,” July 1991.), section 3.17]
Device Under Test (DUT) [Ref.[Ma98] (Mandeville, R., “Benchmarking Terminology for LAN Switching Devices,” February 1998.), section 3.1.1]
System Under Test (SUT) [Ref.[Ma98] (Mandeville, R., “Benchmarking Terminology for LAN Switching Devices,” February 1998.), section 3.1.2]
Out-of-Order Packet [Ref.[Po06] (Poretsky, S., Perser, J., Erramilli, S., and S. Khurana, “Terminology for Benchmarking Network-layer Traffic Control Mechanisms,” October 2006.), section 3.3.4]
Duplicate Packet [Ref.[Po06] (Poretsky, S., Perser, J., Erramilli, S., and S. Khurana, “Terminology for Benchmarking Network-layer Traffic Control Mechanisms,” October 2006.), section 3.3.5]
Stream [Ref.[Po06] (Poretsky, S., Perser, J., Erramilli, S., and S. Khurana, “Terminology for Benchmarking Network-layer Traffic Control Mechanisms,” October 2006.), section 3.3.2]
Loss Period [Ref.[Ko02] (Koodli, R. and R. Ravikanth, “One-way Loss Pattern Sample Metrics,” August 2002.), section 4]
Forwarding Delay [Ref.[Po06] (Poretsky, S., Perser, J., Erramilli, S., and S. Khurana, “Terminology for Benchmarking Network-layer Traffic Control Mechanisms,” October 2006.), section 3.2.4]
Jitter [Ref.[Po06] (Poretsky, S., Perser, J., Erramilli, S., and S. Khurana, “Terminology for Benchmarking Network-layer Traffic Control Mechanisms,” October 2006.), section 3.2.5]



 TOC 

3.  Test Topologies



 TOC 

3.1.  Test topology for local changes

Figure 1 (IGP convergence test topology for local changes) shows the test topology to measure IGP convergence time due to local Convergence Events such as Local Interface failure (Section 8.1.1 (Convergence Due to Local Interface Failure)), layer 2 session failure (Section 8.2.1 (Convergence Due to Layer 2 Session Loss)), and IGP adjacency failure (Section 8.2.2 (Convergence Due to Loss of IGP Adjacency)). This topology is also used to measure IGP convergence time due to the route withdrawal (Section 8.2.3 (Convergence Due to Route Withdrawal)), and route cost change (Section 8.3.2 (Convergence Due to Cost Change)) Convergence Events. IGP adjacencies MUST be established between Tester and DUT, one on the Preferred Egress Interface and one on the Next-Best Egress Interface. For this purpose the Tester emulates two routers, each establishing one adjacency with the DUT. An IGP adjacency SHOULD be established on the Ingress Interface between Tester and DUT.



---------       Ingress Interface         ----------
|       |<--------------------------------|        |
|       |                                 |        |
|       |    Preferred Egress Interface   |        |
|  DUT  |-------------------------------->| Tester |
|       |                                 |        |
|       |-------------------------------->|        |
|       |    Next-Best Egress Interface   |        |
---------                                 ----------
 Figure 1: IGP convergence test topology for local changes 

Figure 2 (IGP convergence test topology for local changes with non-ECMP to ECMP convergence) shows the test topology to measure IGP convergence time due to local Convergence Events with a non-ECMP Preferred Egress Interface and ECMP Next-Best Egress Interfaces (Section 8.1.1 (Convergence Due to Local Interface Failure)). In this topology, the DUT is configured with each Next-Best Egress interface as a member of a single ECMP set. The Preferred Egress Interface is not a member of an ECMP set. The Tester emulates N+1 next-hop routers, one router for the Preferred Egress Interface and N routers for the members of the ECMP set. IGP adjacencies MUST be established between Tester and DUT, one on the Preferred Egress Interface, an one on each member of the ECMP set. For this purpose each of the N+1 routers emulated by the Tester establishes one adjacency with the DUT. An IGP adjacency SHOULD be established on the Ingress Interface between Tester and DUT. When the test specifies to observe the Next-Best Egress Interface statistics, the combined statistics for all ECMP members should be observed.



---------       Ingress Interface         ----------
|       |<--------------------------------|        |
|       |    Preferred Egress Interface   |        |
|       |-------------------------------->|        |
|       |     ECMP set interface 1        |        |
|  DUT  |-------------------------------->| Tester |
|       |               .                 |        |
|       |               .                 |        |
|       |-------------------------------->|        |
|       |     ECMP set interface N        |        |
---------                                 ----------
 Figure 2: IGP convergence test topology for local changes with non-ECMP to ECMP convergence 



 TOC 

3.2.  Test topology for remote changes

Figure 3 (IGP convergence test topology for remote changes) shows the test topology to measure IGP convergence time due to Remote Interface failure (Section 8.1.2 (Convergence Due to Remote Interface Failure)). In this topology the two routers R1 and R2 are considered System Under Test (SUT) and SHOULD be identically configured devices of the same model. IGP adjacencies MUST be established between Tester and SUT, one on the Preferred Egress Interface and one on the Next-Best Egress Interface. For this purpose the Tester emulates one or two routers. An IGP adjacency SHOULD be established on the Ingress Interface between Tester and SUT. In this topology there is a possibility of a transient microloop between R1 and R2 during convergence.



         ------                      ----------
         |    |  Preferred           |        |
------   | R2 |--------------------->|        |
|    |-->|    |  Egress Interface    |        |
|    |   ------                      |        |
| R1 |                               | Tester |
|    |           Next-Best           |        |
|    |------------------------------>|        |
------           Egress Interface    |        |
   ^                                 ----------
   |                                     |
   ---------------------------------------
               Ingress Interface
 Figure 3: IGP convergence test topology for remote changes 

Figure 4 (IGP convergence test topology for remote changes with non-ECMP to ECMP convergence) shows the test topology to measure IGP convergence time due to remote Convergence Events with a non-ECMP Preferred Egress Interface and ECMP Next-Best Egress Interfaces (Section 8.1.2 (Convergence Due to Remote Interface Failure)). In this topology the two routers R1 and R2 are considered System Under Test (SUT) and MUST be identically configured devices of the same model. Router R1 is configured with each Next-Best Egress interface as a member of the same ECMP set. The Preferred Egress Interface of R1 is not a member of an ECMP set. The Tester emulates N+1 next-hop routers, one for R2 and one for each member of the ECMP set. IGP adjacencies MUST be established between Tester and SUT, one on each egress interface of SUT. For this purpose each of the N+1 routers emulated by the Tester establishes one adjacency with the SUT. An IGP adjacency SHOULD be established on the Ingress Interface between Tester and SUT. In this topology there is a possibility of a transient microloop between R1 and R2 during convergence. When the test specifies to observe the Next-Best Egress Interface statistics, the combined statistics for all ECMP members should be observed.



                          ------     ----------
                          |    |     |        |
------      Preferred     | R2 |---->|        |
|    |------------------->|    |     |        |
|    |  Egress Interface  ------     |        |
|    |                               |        |
|    |      ECMP set interface 1     |        |
| R1 |------------------------------>| Tester |
|    |               .               |        |
|    |               .               |        |
|    |               .               |        |
|    |------------------------------>|        |
------      ECMP set interface N     |        |
   ^                                 ----------
   |                                     |
   ---------------------------------------
               Ingress Interface
 Figure 4: IGP convergence test topology for remote changes with non-ECMP to ECMP convergence 



 TOC 

3.3.  Test topology for local ECMP changes

Figure 5 (IGP convergence test topology for local ECMP changes) shows the test topology to measure IGP convergence time due to local Convergence Events of a member of an Equal Cost Multipath (ECMP) set (Section 8.1.3 (Convergence Due to ECMP Member Local Interface Failure)). In this topology, the DUT is configured with each egress interface as a member of a single ECMP set and the Tester emulates N next-hop routers, one router for each member. IGP adjacencies MUST be established between Tester and DUT, one on each member of the ECMP set. For this purpose each of the N routers emulated by the Tester establishes one adjacency with the DUT. An IGP adjacency SHOULD be established on the Ingress Interface between Tester and DUT. When the test specifies to observe the Next-Best Egress Interface statistics, the combined statistics for all ECMP members except the one affected by the Convergence Event, should be observed.



---------       Ingress Interface         ----------
|       |<--------------------------------|        |
|       |                                 |        |
|       |     ECMP set interface 1        |        |
|       |-------------------------------->|        |
|  DUT  |               .                 | Tester |
|       |               .                 |        |
|       |               .                 |        |
|       |-------------------------------->|        |
|       |     ECMP set interface N        |        |
---------                                 ----------
 Figure 5: IGP convergence test topology for local ECMP changes 



 TOC 

3.4.  Test topology for remote ECMP changes

Figure 6 (IGP convergence test topology for remote ECMP changes) shows the test topology to measure IGP convergence time due to remote Convergence Events of a member of an Equal Cost Multipath (ECMP) set (Section 8.1.4 (Convergence Due to ECMP Member Remote Interface Failure)). In this topology the two routers R1 and R2 are considered System Under Test (SUT) and MUST be identically configured devices of the same model. Router R1 is configured with each egress interface as a member of a single ECMP set and the Tester emulates N next-hop routers, one router for each member. IGP adjacencies MUST be established between Tester and SUT, one on each egress interface of SUT. For this purpose each of the N routers emulated by the Tester establishes one adjacency with the SUT. An IGP adjacency SHOULD be established on the Ingress Interface between Tester and SUT. In this topology there is a possibility of a transient microloop between R1 and R2 during convergence. When the test specifies to observe the Next-Best Egress Interface statistics, the combined statistics for all ECMP members except the one affected by the Convergence Event, should be observed.



                          ------     ----------
                          |    |     |        |
------      ECMP set      | R2 |---->|        |
|    |------------------->|    |     |        |
|    |      Interface 1   ------     |        |
|    |                               |        |
|    |      ECMP set interface 2     |        |
| R1 |------------------------------>| Tester |
|    |               .               |        |
|    |               .               |        |
|    |               .               |        |
|    |------------------------------>|        |
------      ECMP set interface N     |        |
   ^                                 ----------
   |                                     |
   ---------------------------------------
               Ingress Interface
 Figure 6: IGP convergence test topology for remote ECMP changes 



 TOC 

3.5.  Test topology for Parallel Link changes

Figure 7 (IGP convergence test topology for Parallel Link changes) shows the test topology to measure IGP convergence time due to local Convergence Events with members of a Parallel Link (Section 8.1.5 (Convergence Due to Parallel Link Interface Failure )). In this topology, the DUT is configured with each egress interface as a member of a Parallel Link and the Tester emulates the single next-hop router. IGP adjacencies MUST be established on all N members of the Parallel Link between Tester and DUT. For this purpose the router emulated by the Tester establishes N adjacencies with the DUT. An IGP adjacency SHOULD be established on the Ingress Interface between Tester and DUT. When the test specifies to observe the Next-Best Egress Interface statistics, the combined statistics for all Parallel Link members except the one affected by the Convergence Event, should be observed.



---------       Ingress Interface         ----------
|       |<--------------------------------|        |
|       |                                 |        |
|       |     Parallel Link Interface 1   |        |
|       |-------------------------------->|        |
|  DUT  |               .                 | Tester |
|       |               .                 |        |
|       |               .                 |        |
|       |-------------------------------->|        |
|       |     Parallel Link Interface N   |        |
---------                                 ----------
 Figure 7: IGP convergence test topology for Parallel Link changes 



 TOC 

4.  Convergence Time and Loss of Connectivity Period

Two concepts will be highlighted in this section: convergence time and loss of connectivity period.

The Route Convergence [Po09t] (Poretsky, S. and B. Imhoff, “Terminology for Benchmarking Link-State IGP Data Plane Route Convergence,” July 2009.) time indicates the period in time between the Convergence Event Instant [Po09t] (Poretsky, S. and B. Imhoff, “Terminology for Benchmarking Link-State IGP Data Plane Route Convergence,” July 2009.) and the instant in time the DUT is ready to forward traffic for a specific route on its Next-Best Egress Interface and maintains this state for the duration of the Sustained Convergence Validation Time [Po09t] (Poretsky, S. and B. Imhoff, “Terminology for Benchmarking Link-State IGP Data Plane Route Convergence,” July 2009.). To measure Route Convergence time, the Convergence Event Instant and the traffic received from the Next-Best Egress Interface need to be observed.

The Route Loss of Connectivity Period [Po09t] (Poretsky, S. and B. Imhoff, “Terminology for Benchmarking Link-State IGP Data Plane Route Convergence,” July 2009.) indicates the time during which traffic to a specific route is lost following a Convergence Event until Full Convergence [Po09t] (Poretsky, S. and B. Imhoff, “Terminology for Benchmarking Link-State IGP Data Plane Route Convergence,” July 2009.) completes. This Route Loss of Connectivity Period can consist of one or more Loss Periods [Ko02] (Koodli, R. and R. Ravikanth, “One-way Loss Pattern Sample Metrics,” August 2002.). For the testcases described in this document it is expected to have a single Loss Period. To measure Route Loss of Connectivity Period, the traffic received from the Preferred Egress Interface and the traffic received from the Next-Best Egress Interface need to be observed.

The Route Loss of Connectivity Period is most important since that has a direct impact on the network user's application performance.

In general the Route Convergence time is larger than or equal to the Route Loss of Connectivity Period. Depending on which Convergence Event occurs and how this Convergence Event is applied, traffic for a route may still be forwarded over the Preferred Egress Interface after the Convergence Event Instant, before converging to the Next-Best Egress Interface. In that case the Route Loss of Connectivity Period is shorter than the Route Convergence time.

At least one condition needs to be fulfilled for Route Convergence time to be equal to Route Loss of Connectivity Period. The condition is that the Convergence Event causes an instantaneous traffic loss for the measured route. A fiber cut on the Preferred Egress Interface is an example of such a Convergence Event.

A second condition applies to Route Convergence time measurements based on Connectivity Packet Loss [Po09t] (Poretsky, S. and B. Imhoff, “Terminology for Benchmarking Link-State IGP Data Plane Route Convergence,” July 2009.). This second condition is that there is only a single Loss Period during Route Convergence. For the testcases described in this document this is expected to be the case.



 TOC 

4.1.  Convergence Events without instant traffic loss

To measure convergence time benchmarks for Convergence Events caused by a Tester, such as an IGP cost change, the Tester MAY start to discard all traffic received from the Preferred Egress Interface at the Convergence Event Instant, or MAY separately observe packets received from the Preferred Egress Interface prior to the Convergence Event Instant. This way these Convergence Events can be treated the same as Convergence Events that cause instantaneous traffic loss.

To measure convergence time benchmarks without instantaneous traffic loss (either real or induced by the Tester) at the Convergence Event Instant, such as a reversion of a link failure Convergence Event, the Tester SHALL only observe packet statistics on the Next-Best Egress Interface. If using the Rate-Derived method to benchmark convergence times for such Convergence Events, the Tester MUST collect a timestamp at the Convergence Event Instant. If using a loss-derived method to benchmark convergence times for such Convergence Events, the Tester MUST measure the period in time between the Start Traffic Instant and the Convergence Event Instant. To measure this period in time the Tester can collect timestamps at the Start Traffic Instant and the Convergence Event Instant.

The Convergence Event Instant together with the receive rate observations on the Next-Best Egress Interface allow to derive the convergence time benchmarks using the Rate-Derived Method [Po09t] (Poretsky, S. and B. Imhoff, “Terminology for Benchmarking Link-State IGP Data Plane Route Convergence,” July 2009.).

By observing lost packets on the Next-Best Egress Interface only, the observed packet loss is the number of lost packets between Traffic Start Instant and Convergence Recovery Instant. To measure convergence times using a loss-derived method, packet loss between the Convergence Event Instant and the Convergence Recovery Instant is needed. The time between Traffic Start Instant and Convergence Event Instant must be accounted for. An example may clarify this.

Figure 8 illustrates a Convergence Event without instantaneous traffic loss for all routes. The top graph shows the Forwarding Rate over all routes, the bottom graph shows the Forwarding Rate for a single route Rta. Some time after the Convergence Event Instant, Forwarding Rate observed on the Preferred Egress Interface starts to decrease. In the example, route Rta is the first route to experience packet loss at time Ta. Some time later, the Forwarding Rate observed on the Next-Best Egress Interface starts to increase. In the example, route Rta is the first route to complete convergence at time Ta'.



     ^
Fwd  |
Rate |-------------                    ............
     |             \                  .
     |              \                .
     |               \              .
     |                \            .
     |.................-.-.-.-.-.-.----------------
     +----+-------+---------------+----------------->
     ^    ^       ^               ^             time
    T0   CEI      Ta              Ta'

     ^
Fwd  |
Rate |-------------               .................
Rta  |            |               .
     |            |               .
     |.............-.-.-.-.-.-.-.-.----------------
     +----+-------+---------------+----------------->
     ^    ^       ^               ^             time
    T0   CEI      Ta              Ta'

     Preferred Egress Interface: ---
     Next-Best Egress Interface: ...

With T0 the Start Traffic Instant; CEI the Convergence Event Instant; Ta the time instant traffic loss for route Rta starts; Ta' the time instant traffic loss for route Rta ends.

 Figure 8 

If only packets received on the Next-Best Egress Interface are observed, the duration of the packet loss period for route Rta can be calculated from the received packets as in Equation 1. Since the Convergence Event Instant is the start time for convergence time measurement, the period in time between T0 and CEI needs to be subtracted from the calculated result to become the convergence time, as in Equation 2.

Next-Best Egress Interface packet loss period
    = (packets transmitted
        - packets received from Next-Best Egress Interface) / tx rate
    = Ta' - T0

Equation 1

convergence time
    = Next-Best Egress Interface packet loss period - (CEI - T0)
    = Ta' - CEI

Equation 2



 TOC 

4.2.  Loss of Connectivity

Route Loss of Connectivity Period SHOULD be measured using the Route-Specific Loss-Derived Method. Since the start instant and end instant of the Route Loss of Connectivity Period can be different for each route, these can not be accurately derived by only observing global statistics over all routes. An example may clarify this.

Following a Convergence Event, route Rta is the first route for which packet loss starts, the Route Loss of Connectivity Period for route Rta starts at time Ta. Route Rtb is the last route for which packet loss starts, the Route Loss of Connectivity Period for route Rtb starts at time Tb with Tb>Ta.



     ^
Fwd  |
Rate |--------                       -----------
     |        \                     /
     |         \                   /
     |          \                 /
     |           \               /
     |            ---------------
     +------------------------------------------>
              ^   ^             ^    ^      time
             Ta   Tb           Ta'   Tb'
                               Tb''  Ta''

 Figure 9: Example Route Loss Of Connectivity Period 

If the DUT implementation would be such that Route Rta would be the first route for which traffic loss ends at time Ta' with Ta'>Tb. Route Rtb would be the last route for which traffic loss ends at time Tb' with Tb'>Ta'. By using only observing global traffic statistics over all routes, the minimum Route Loss of Connectivity Period would be measured as Ta'-Ta. The maximum calculated Route Loss of Connectivity Period would be Tb'-Ta. The real minimum and maximum Route Loss of Connectivity Periods are Ta'-Ta and Tb'-Tb. Illustrating this with the numbers Ta=0, Tb=1, Ta'=3, and Tb'=5, would give a LoC Period between 3 and 5 derived from the global traffic statistics, versus the real LoC Period between 3 and 4.

If the DUT implementation would be such that route Rtb would be the first for which packet loss ends at time Tb'' and route Rta would be the last for which packet loss ends at time Ta'', then the minimum and maximum Route Loss of Connectivity Periods derived by observing only global traffic statistics would be Tb''-Ta, and Ta''-Ta. The real minimum and maximum Route Loss of Connectivity Periods are Tb''-Tb and Ta''-Ta. Illustrating this with the numbers Ta=0, Tb=1, Ta''=5, Tb''=3, would give a LoC Period between 3 and 5 derived from the global traffic statistics, versus the real LoC Period between 2 and 5.

The two implementation variations in the above example would result in the same derived minimum and maximum Route Loss of Connectivity Periods when only observing the global packet statistics, while the real Route Loss of Connectivity Periods are different.



 TOC 

5.  Test Considerations



 TOC 

5.1.  IGP Selection

The test cases described in Section 8 (Test Cases) MAY be used for link-state IGPs, such as ISIS or OSPF. The IGP convergence time test methodology is identical.



 TOC 

5.2.  Routing Protocol Configuration

The obtained results for IGP convergence time may vary if other routing protocols are enabled and routes learned via those protocols are installed. IGP convergence times SHOULD be benchmarked without routes installed from other protocols.



 TOC 

5.3.  IGP Topology

The Tester emulates a single IGP topology. The DUT establishes IGP adjacencies with one or more of the emulated routers in this single IGP topology emulated by the Tester. See test topology details in Section 3 (Test Topologies). The emulated topology SHOULD only be advertised on the DUT egress interfaces.

The number of IGP routes and number of nodes in the topology, and the type of topology will impact the measured IGP convergence time. To obtain results similar to those that would be observed in an operational network, it is RECOMMENDED that the number of installed routes and nodes closely approximate that of the network (e.g. thousands of routes with tens or hundreds of nodes).

The number of areas (for OSPF) and levels (for ISIS) can impact the benchmark results.



 TOC 

5.4.  Timers

There are timers that may impact the measured IGP convergence times. The benchmark metrics MAY be measured at any fixed values for these timers. To obtain results similar to those that would be observed in an operational network, it is RECOMMENDED to configure the timers with the values as configured in the operational network.

Examples of timers that may impact measured IGP convergence time include, but are not limited to:

Interface failure indication

IGP hello timer

IGP dead-interval or hold-timer

LSA or LSP generation delay

LSA or LSP flood packet pacing

SPF delay



 TOC 

5.5.  Interface Types

All test cases in this methodology document MAY be executed with any interface type. The type of media may dictate which test cases may be executed. Each interface type has a unique mechanism for detecting link failures and the speed at which that mechanism operates will influence the measurement results. All interfaces MUST be the same media and Throughput [Br91] (Bradner, S., “Benchmarking terminology for network interconnection devices,” July 1991.)[Br99] (Bradner, S. and J. McQuaid, “Benchmarking Methodology for Network Interconnect Devices,” March 1999.) for each test case. All interfaces SHOULD be configured as point-to-point.



 TOC 

5.6.  Offered Load

The Throughput of the device, as defined in [Br91] (Bradner, S., “Benchmarking terminology for network interconnection devices,” July 1991.) and benchmarked in [Br99] (Bradner, S. and J. McQuaid, “Benchmarking Methodology for Network Interconnect Devices,” March 1999.) at a fixed packet size, needs to be determined over the preferred path and over the next-best path. The Offered Load SHOULD be the minimum of the measured Throughput of the device over the primary path and over the backup path. The packet size is selectable and MUST be recorded. Packet size is measured in bytes and includes the IP header and payload.

The destination addresses for the Offered Load MUST be distributed such that all routes or a statistically representative subset of all routes are matched and each of these routes is offered an equal share of the Offered Load. It is RECOMMENDED to send traffic matching all routes, but a statistically representative subset of all routes can be used if required.

In the Remote Interface failure testcases using topologies 3 (IGP convergence test topology for remote changes), 4 (IGP convergence test topology for remote changes with non-ECMP to ECMP convergence), and 6 (IGP convergence test topology for remote ECMP changes) there is a possibility of a transient microloop between R1 and R2 during convergence. The TTL or Hop Limit value of the packets sent by the Tester may influence the benchmark measurements since it determines which device in the topology may send an ICMP Time Exceeded Message for looped packets.

The duration of the Offered Load MUST be greater than the convergence time plus the Sustained Convergence Validation Time.

Offered load should send a packet to each destination before sending another packet to the same destination. It is RECOMMENDED that the packets are transmitted in a round-robin fashion with a uniform interpacket delay.



 TOC 

5.7.  Measurement Accuracy

Since packet loss is observed to measure the Route Convergence Time, the time between two successive packets offered to each individual route is the highest possible accuracy of any packet loss based measurement. The higher the traffic rate offered to each route the higher the possible measurement accuracy.

Also see Section 6 (Selection of Convergence Time Benchmark Metrics and Methods) for method-specific measurement accuracy.



 TOC 

5.8.  Measurement Statistics

The benchmark measurements may vary for each trial, due to the statistical nature of timer expirations, cpu scheduling, etc. Evaluation of the test data must be done with an understanding of generally accepted testing practices regarding repeatability, variance and statistical significance of a small number of trials.



 TOC 

5.9.  Tester Capabilities

It is RECOMMENDED that the Tester used to execute each test case has the following capabilities:

  1. Ability to establish IGP adjacencies and advertise a single IGP topology to one or more peers.
  2. Ability to measure Forwarding Delay, Duplicate Packets and Out-of-Order Packets.
  3. An internal time clock to control timestamping, time measurements, and time calculations.
  4. Ability to distinguish traffic load received on the Preferred and Next-Best Interfaces [Po09t] (Poretsky, S. and B. Imhoff, “Terminology for Benchmarking Link-State IGP Data Plane Route Convergence,” July 2009.).
  5. Ability to disable or tune specific Layer-2 and Layer-3 protocol functions on any interface(s).

The Tester MAY be capable to make non-data plane convergence observations and use those observations for measurements. The Tester MAY be capable to send and receive multiple traffic Streams [Po06] (Poretsky, S., Perser, J., Erramilli, S., and S. Khurana, “Terminology for Benchmarking Network-layer Traffic Control Mechanisms,” October 2006.).

Also see Section 6 (Selection of Convergence Time Benchmark Metrics and Methods) for method-specific capabilities.



 TOC 

6.  Selection of Convergence Time Benchmark Metrics and Methods

Different convergence time benchmark methods MAY be used to measure convergence time benchmark metrics. The Tester capabilities are important criteria to select a specific convergence time benchmark method. The criteria to select a specific benchmark method include, but are not limited to:



Tester capabilities: Sampling Interval, number of Stream statistics to collect
Measurement accuracy: Sampling Interval, Offered Load, number of routes
Test specification: number of routes
DUT capabilities: Throughput, Jitter



 TOC 

6.1.  Loss-Derived Method



 TOC 

6.1.1.  Tester capabilities

The Offered Load SHOULD consist of a single Stream [Po06] (Poretsky, S., Perser, J., Erramilli, S., and S. Khurana, “Terminology for Benchmarking Network-layer Traffic Control Mechanisms,” October 2006.). If sending multiple Streams, the measured packet loss statistics for all Streams MUST be added together.

In order to verify Full Convergence completion and the Sustained Convergence Validation Time, the Tester MUST measure Forwarding Rate each Packet Sampling Interval.

The total number of packets lost between the start of the traffic and the end of the Sustained Convergence Validation Time is used to calculate the Loss-Derived Convergence Time.



 TOC 

6.1.2.  Benchmark Metrics

The Loss-Derived Method can be used to measure the Loss-Derived Convergence Time, which is the average convergence time over all routes, and to measure the Loss-Derived Loss of Connectivity Period, which is the average Route Loss of Connectivity Period over all routes.



 TOC 

6.1.3.  Measurement Accuracy

The actual value falls within the accuracy interval [-(number of destinations/Offered Load), +(number of destinations/Offered Load)] around the value as measured using the Loss-Derived Method.



 TOC 

6.2.  Rate-Derived Method



 TOC 

6.2.1.  Tester Capabilities

The Offered Load SHOULD consist of a single Stream. If sending multiple Streams, the measured traffic rate statistics for all Streams MUST be added together.

The Tester measures Forwarding Rate each Sampling Interval. The Packet Sampling Interval influences the observation of the different convergence time instants. If the Packet Sampling Interval is large compared to the time between the convergence time instants, then the different time instants may not be easily identifiable from the Forwarding Rate observation. The presence of Jitter [Po06] (Poretsky, S., Perser, J., Erramilli, S., and S. Khurana, “Terminology for Benchmarking Network-layer Traffic Control Mechanisms,” October 2006.) may cause fluctuations of the Forwarding Rate observation and can prevent correct observation of the different convergence time instants.

The Packet Sampling Interval MUST be larger than or equal to the time between two consecutive packets to the same destination. For maximum accuracy the value for the Packet Sampling Interval SHOULD be as small as possible, but the presence of Jitter may enforce using a larger Packet Sampling Interval. The Packet Sampling Interval MUST be reported.

Jitter causes fluctuations in the number of received packets during each Packet Sampling Interval. To account for the presence of Jitter in determining if a convergence instant has been reached, Jitter SHOULD be observed during each Packet Sampling Interval. The minimum and maximum number of packets expected in a Packet Sampling Interval in presence of Jitter can be calculated with Equation 3.

number of packets expected in a Packet Sampling Interval
  in presence of Jitter
    = expected number of packets without Jitter
      +/-(max Jitter during Packet Sampling Interval * Offered Load)

Equation 3

To determine if a convergence instant has been reached the number of packets received in a Packet Sampling Interval is compared with the range of expected number of packets calculated in Equation 3.



 TOC 

6.2.2.  Benchmark Metrics

The Rate-Derived Method SHOULD be used to measure First Route Convergence Time and Full Convergence Time. It SHOULD NOT be used to measure Loss of Connectivity Period (see Section 4 (Convergence Time and Loss of Connectivity Period)).



 TOC 

6.2.3.  Measurement Accuracy

The measurement accuracy interval of the Rate-Derived Method depends on the metric being measured or calculated and the characteristics of the related transition. Jitter [Po06] (Poretsky, S., Perser, J., Erramilli, S., and S. Khurana, “Terminology for Benchmarking Network-layer Traffic Control Mechanisms,” October 2006.) adds uncertainty to the amount of packets received in a Packet Sampling Interval and this uncertainty adds to the measurement error. The effect of Jitter is not accounted for in the calculation of the accuracy intervals below. Jitter is of importance for the convergence instants were a variation in Forwarding Rate needs to be observed (Convergence Recovery Instant and for topologies with ECMP also Convergence Event Instant and First Route Convergence Instant).

If the Convergence Event Instant is observed on the dataplane using the Rate Derived Method, it needs to be instantaneous for all routes (see Section 4.1 (Convergence Events without instant traffic loss)). The actual value of the Convergence Event Instant falls within the accuracy interval [–(Packet Sampling Interval + 1/Offered Load), +0] around the value as measured using the Rate-Derived Method.

If the Convergence Recovery Transition is non-instantaneous for all routes then the actual value of the First Route Convergence Instant falls within the accuracy interval [–(Packet Sampling Interval + time between two consecutive packets to the same destination), +0] around the value as measured using the Rate-Derived Method, and the actual value of the Convergence Recovery Instant falls within the accuracy interval [–(2 * Packet Sampling Interval), –(Packet Sampling Interval - time between two consecutive packets to the same destination)] around the value as measured using the Rate-Derived Method.

The term “time between two consecutive packets to the same destination” is added in the above accuracy intervals since packets are sent in a particular order to all destinations in a stream and when part of the routes experience packet loss, it is unknown where in the transmit cycle packets to these routes are sent. This uncertainty adds to the error.

The accuracy intervals of the derived metrics First Route Convergence Time and Rate-Derived Convergence Time are calculated from the above convergence instants accuracy intervals. The actual value of First Route Convergence Time falls within the accuracy interval [–(Packet Sampling Interval + time between two consecutive packets to the same destination), +(Packet Sampling Interval + 1/Offered Load)] around the calculated value. The actual value of Rate-Derived Convergence Time falls within the accuracy interval [–(2 * Packet Sampling Interval), +(time between two consecutive packets to the same destination + 1/Offered Load)] around the calculated value.



 TOC 

6.3.  Route-Specific Loss-Derived Method



 TOC 

6.3.1.  Tester Capabilities

The Offered Load consists of multiple Streams. The Tester MUST measure packet loss for each Stream separately.

In order to verify Full Convergence completion and the Sustained Convergence Validation Time, the Tester MUST measure packet loss each Packet Sampling Interval. This measurement at each Packet Sampling Interval MAY be per Stream.

Only the total packet loss measured per Stream at the end of the Sustained Convergence Validation Time is used to calculate the benchmark metrics with this method.



 TOC 

6.3.2.  Benchmark Metrics

The Route-Specific Loss-Derived Method SHOULD be used to measure Route-Specific Convergence Times. It is the RECOMMENDED method to measure Route Loss of Connectivity Period.

Under the conditions explained in Section 4 (Convergence Time and Loss of Connectivity Period), First Route Convergence Time and Full Convergence Time as benchmarked using Rate-Derived Method, may be equal to the minimum resp. maximum of the Route-Specific Convergence Times.



 TOC 

6.3.3.  Measurement Accuracy

The actual value falls within the accuracy interval [-(number of destinations/Offered Load), +(number of destinations/Offered Load)] around the value as measured using the Route-Specific Loss-Derived Method.



 TOC 

7.  Reporting Format

For each test case, it is recommended that the reporting tables below are completed and all time values SHOULD be reported with resolution as specified in [Po09t] (Poretsky, S. and B. Imhoff, “Terminology for Benchmarking Link-State IGP Data Plane Route Convergence,” July 2009.).



ParameterUnits
Test Case test case number
Test Topology Test Topology Figure number
IGP (ISIS, OSPF, other)
Interface Type (GigE, POS, ATM, other)
Packet Size offered to DUT bytes
Offered Load packets per second
IGP Routes advertised to DUT number of IGP routes
Nodes in emulated network number of nodes
Number of Parallel or ECMP links number of links
Number of Routes measured number of routes
Packet Sampling Interval on Tester seconds
Forwarding Delay Threshold seconds
   
Timer Values configured on DUT:  
 Interface failure indication delay seconds
 IGP Hello Timer seconds
 IGP Dead-Interval or hold-time seconds
 LSA Generation Delay seconds
 LSA Flood Packet Pacing seconds
 LSA Retransmission Packet Pacing seconds
 SPF Delay seconds

Test Details:

If the Offered Load matches a subset of routes, describe how this subset is selected.

Describe how the Convergence Event is applied; does it cause instantaneous traffic loss or not.

Complete the table below for the initial Convergence Event and the reversion Convergence Event.



ParameterUnits
Conversion Event (initial or reversion)
   
Traffic Forwarding Metrics:  
 Total number of packets offered to DUT number of Packets
 Total number of packets forwarded by DUT number of Packets
 Connectivity Packet Loss number of Packets
 Convergence Packet Loss number of Packets
 Out-of-Order Packets number of Packets
 Duplicate Packets number of Packets
   
Convergence Benchmarks:  
 Rate-Derived Method:  
  First Route Convergence Time seconds
  Full Convergence Time seconds
 Loss-Derived Method:  
  Loss-Derived Convergence Time seconds
 Route-Specific Loss-Derived Method:  
  Route-Specific Convergence Time[n] array of seconds
  Minimum R-S Convergence Time seconds
  Maximum R-S Convergence Time seconds
  Median R-S Convergence Time seconds
  Average R-S Convergence Time seconds
   
Loss of Connectivity Benchmarks:  
 Loss-Derived Method:  
  Loss-Derived Loss of Connectivity Period seconds
 Route-Specific Loss-Derived Method:  
  Route LoC Period[n] array of seconds
  Minimum Route LoC Period seconds
  Maximum Route LoC Period seconds
  Median Route LoC Period seconds
  Average Route LoC Period seconds



 TOC 

8.  Test Cases

It is RECOMMENDED that all applicable test cases be performed for best characterization of the DUT. The test cases follow a generic procedure tailored to the specific DUT configuration and Convergence Event [Po09t] (Poretsky, S. and B. Imhoff, “Terminology for Benchmarking Link-State IGP Data Plane Route Convergence,” July 2009.). This generic procedure is as follows:

  1. Establish DUT and Tester configurations and advertise an IGP topology from Tester to DUT.
  2. Send Offered Load from Tester to DUT on ingress interface.
  3. Verify traffic is routed correctly. Verify if traffic is forwarded without drops, without Out-of-Order Packets, and without exceeding the Forwarding Delay Threshold [Po06] (Poretsky, S., Perser, J., Erramilli, S., and S. Khurana, “Terminology for Benchmarking Network-layer Traffic Control Mechanisms,” October 2006.).
  4. Introduce Convergence Event [Po09t] (Poretsky, S. and B. Imhoff, “Terminology for Benchmarking Link-State IGP Data Plane Route Convergence,” July 2009.).
  5. Measure First Route Convergence Time [Po09t] (Poretsky, S. and B. Imhoff, “Terminology for Benchmarking Link-State IGP Data Plane Route Convergence,” July 2009.).
  6. Measure Full Convergence Time [Po09t] (Poretsky, S. and B. Imhoff, “Terminology for Benchmarking Link-State IGP Data Plane Route Convergence,” July 2009.).
  7. Stop Offered Load.
  8. Measure Route-Specific Convergence Times, Loss-Derived Convergence Time, Route LoC Periods, and Loss-Derived LoC Period [Po09t] (Poretsky, S. and B. Imhoff, “Terminology for Benchmarking Link-State IGP Data Plane Route Convergence,” July 2009.).
  9. Wait sufficient time for queues to drain. The duration of this time period is equal to the Forwarding Delay Threshold. In absence of a Forwarding Delay Threshold specification the duration of this time period is 2 seconds [Br99] (Bradner, S. and J. McQuaid, “Benchmarking Methodology for Network Interconnect Devices,” March 1999.).
  10. Restart Offered Load.
  11. Reverse Convergence Event.
  12. Measure First Route Convergence Time.
  13. Measure Full Convergence Time.
  14. Stop Offered Load.
  15. Measure Route-Specific Convergence Times, Loss-Derived Convergence Time, Route LoC Periods, and Loss-Derived LoC Period.


 TOC 

8.1.  Interface failures



 TOC 

8.1.1.  Convergence Due to Local Interface Failure

Objective

To obtain the IGP convergence times due to a Local Interface failure event. The Next-Best Egress Interface can be a single interface (Figure 1 (IGP convergence test topology for local changes)) or an ECMP set (Figure 2 (IGP convergence test topology for local changes with non-ECMP to ECMP convergence)). The test with ECMP topology (Figure 2 (IGP convergence test topology for local changes with non-ECMP to ECMP convergence)) is OPTIONAL.

Procedure

  1. Advertise an IGP topology from Tester to DUT using the topology shown in Figure 1 (IGP convergence test topology for local changes) or 2 (IGP convergence test topology for local changes with non-ECMP to ECMP convergence).
  2. Send Offered Load from Tester to DUT on ingress interface.
  3. Verify traffic is forwarded over Preferred Egress Interface.
  4. Remove link on DUT's Preferred Egress Interface. This is the Convergence Event.
  5. Measure First Route Convergence Time.
  6. Measure Full Convergence Time.
  7. Stop Offered Load.
  8. Measure Route-Specific Convergence Times and Loss-Derived Convergence Time.
  9. Wait sufficient time for queues to drain.
  10. Restart Offered Load.
  11. Restore link on DUT's Preferred Egress Interface.
  12. Measure First Route Convergence Time.
  13. Measure Full Convergence Time.
  14. Stop Offered Load.
  15. Measure Route-Specific Convergence Times, Loss-Derived Convergence Time, Route LoC Periods, and Loss-Derived LoC Period.

Results

The measured IGP convergence time may be influenced by the link failure indication time, LSA/LSP delay, LSA/LSP generation time, LSA/LSP flood packet pacing, SPF delay, SPF execution time, and routing and forwarding tables update time [Po09a] (Poretsky, S., “Considerations for Benchmarking Link-State IGP Data Plane Route Convergence,” March 2009.).



 TOC 

8.1.2.  Convergence Due to Remote Interface Failure

Objective

To obtain the IGP convergence time due to a Remote Interface failure event. The Next-Best Egress Interface can be a single interface (Figure 3 (IGP convergence test topology for remote changes)) or an ECMP set (Figure 4 (IGP convergence test topology for remote changes with non-ECMP to ECMP convergence)). The test with ECMP topology (Figure 4 (IGP convergence test topology for remote changes with non-ECMP to ECMP convergence)) is OPTIONAL.

Procedure

  1. Advertise an IGP topology from Tester to SUT using the topology shown in Figure 3 (IGP convergence test topology for remote changes) or 4 (IGP convergence test topology for remote changes with non-ECMP to ECMP convergence).
  2. Send Offered Load from Tester to SUT on ingress interface.
  3. Verify traffic is forwarded over Preferred Egress Interface.
  4. Remove link on Tester's interface [Po09t] (Poretsky, S. and B. Imhoff, “Terminology for Benchmarking Link-State IGP Data Plane Route Convergence,” July 2009.) connected to SUT's Preferred Egress Interface. This is the Convergence Event.
  5. Measure First Route Convergence Time.
  6. Measure Full Convergence Time.
  7. Stop Offered Load.
  8. Measure Route-Specific Convergence Times and Loss-Derived Convergence Time.
  9. Wait sufficient time for queues to drain.
  10. Restart Offered Load.
  11. Restore link on Tester's interface connected to DUT's Preferred Egress Interface.
  12. Measure First Route Convergence Time.
  13. Measure Full Convergence Time.
  14. Stop Offered Load.
  15. Measure Route-Specific Convergence Times, Loss-Derived Convergence Time, Route LoC Periods, and Loss-Derived LoC Period.

Results

The measured IGP convergence time may be influenced by the link failure indication time, LSA/LSP delay, LSA/LSP generation time, LSA/LSP flood packet pacing, SPF delay, SPF execution time, and routing and forwarding tables update time. This test case may produce Stale Forwarding [Po09t] (Poretsky, S. and B. Imhoff, “Terminology for Benchmarking Link-State IGP Data Plane Route Convergence,” July 2009.) due to a transient microloop between R1 and R2 during convergence, which may increase the measured convergence times and loss of connectivity periods.



 TOC 

8.1.3.  Convergence Due to ECMP Member Local Interface Failure

Objective

To obtain the IGP convergence time due to a Local Interface link failure event of an ECMP Member.

Procedure

  1. Advertise an IGP topology from Tester to DUT using the test setup shown in Figure 5 (IGP convergence test topology for local ECMP changes).
  2. Send Offered Load from Tester to DUT on ingress interface.
  3. Verify traffic is forwarded over the DUT's ECMP member interface that will be failed in the next step.
  4. Remove link on one of the DUT's ECMP member interfaces. This is the Convergence Event.
  5. Measure First Route Convergence Time.
  6. Measure Full Convergence Time.
  7. Stop Offered Load.
  8. Measure Route-Specific Convergence Times and Loss-Derived Convergence Time. At the same time measure Out-of-Order Packets [Po06] (Poretsky, S., Perser, J., Erramilli, S., and S. Khurana, “Terminology for Benchmarking Network-layer Traffic Control Mechanisms,” October 2006.) and Duplicate Packets [Po06] (Poretsky, S., Perser, J., Erramilli, S., and S. Khurana, “Terminology for Benchmarking Network-layer Traffic Control Mechanisms,” October 2006.).
  9. Wait sufficient time for queues to drain.
  10. Restart Offered Load.
  11. Restore link on DUT's ECMP member interface.
  12. Measure First Route Convergence Time.
  13. Measure Full Convergence Time.
  14. Stop Offered Load.
  15. Measure Route-Specific Convergence Times, Loss-Derived Convergence Time, Route LoC Periods, and Loss-Derived LoC Period. At the same time measure Out-of-Order Packets [Po06] (Poretsky, S., Perser, J., Erramilli, S., and S. Khurana, “Terminology for Benchmarking Network-layer Traffic Control Mechanisms,” October 2006.) and Duplicate Packets [Po06] (Poretsky, S., Perser, J., Erramilli, S., and S. Khurana, “Terminology for Benchmarking Network-layer Traffic Control Mechanisms,” October 2006.).

Results

The measured IGP Convergence time may be influenced by link failure indication time, LSA/LSP delay, LSA/LSP generation time, LSA/LSP flood packet pacing, SPF delay, SPF execution time, and routing and forwarding tables update time [Po09a] (Poretsky, S., “Considerations for Benchmarking Link-State IGP Data Plane Route Convergence,” March 2009.).



 TOC 

8.1.4.  Convergence Due to ECMP Member Remote Interface Failure

Objective

To obtain the IGP convergence time due to a Remote Interface link failure event for an ECMP Member.

Procedure

  1. Advertise an IGP topology from Tester to DUT using the test setup shown in Figure 6 (IGP convergence test topology for remote ECMP changes).
  2. Send Offered Load from Tester to DUT on ingress interface.
  3. Verify traffic is forwarded over the DUT's ECMP member interface that will be failed in the next step.
  4. Remove link on Tester's interface to R2. This is the Convergence Event Trigger.
  5. Measure First Route Convergence Time.
  6. Measure Full Convergence Time.
  7. Stop Offered Load.
  8. Measure Route-Specific Convergence Times and Loss-Derived Convergence Time. At the same time measure Out-of-Order Packets [Po06] (Poretsky, S., Perser, J., Erramilli, S., and S. Khurana, “Terminology for Benchmarking Network-layer Traffic Control Mechanisms,” October 2006.) and Duplicate Packets [Po06] (Poretsky, S., Perser, J., Erramilli, S., and S. Khurana, “Terminology for Benchmarking Network-layer Traffic Control Mechanisms,” October 2006.).
  9. Wait sufficient time for queues to drain.
  10. Restart Offered Load.
  11. Restore link on Tester's interface to R2.
  12. Measure First Route Convergence Time.
  13. Measure Full Convergence Time.
  14. Stop Offered Load.
  15. Measure Route-Specific Convergence Times, Loss-Derived Convergence Time, Route LoC Periods, and Loss-Derived LoC Period. At the same time measure Out-of-Order Packets [Po06] (Poretsky, S., Perser, J., Erramilli, S., and S. Khurana, “Terminology for Benchmarking Network-layer Traffic Control Mechanisms,” October 2006.) and Duplicate Packets [Po06] (Poretsky, S., Perser, J., Erramilli, S., and S. Khurana, “Terminology for Benchmarking Network-layer Traffic Control Mechanisms,” October 2006.).

Results

The measured IGP convergence time may influenced by the link failure indication time, LSA/LSP delay, LSA/LSP generation time, LSA/LSP flood packet pacing, SPF delay, SPF execution time, and routing and forwarding tables update time. This test case may produce Stale Forwarding [Po09t] (Poretsky, S. and B. Imhoff, “Terminology for Benchmarking Link-State IGP Data Plane Route Convergence,” July 2009.) due to a transient microloop between R1 and R2 during convergence, which may increase the measured convergence times and loss of connectivity periods.



 TOC 

8.1.5.  Convergence Due to Parallel Link Interface Failure

Objective

To obtain the IGP convergence due to a local link failure event for a member of a parallel link. The links can be used for data load balancing

Procedure

  1. Advertise an IGP topology from Tester to DUT using the test setup shown in Figure 7 (IGP convergence test topology for Parallel Link changes).
  2. Send Offered Load from Tester to DUT on ingress interface.
  3. Verify traffic is forwarded over the parallel link member that will be failed in the next step.
  4. Remove link on one of the DUT's parallel link member interfaces. This is the Convergence Event.
  5. Measure First Route Convergence Time.
  6. Measure Full Convergence Time.
  7. Stop Offered Load.
  8. Measure Route-Specific Convergence Times and Loss-Derived Convergence Time. At the same time measure Out-of-Order Packets [Po06] (Poretsky, S., Perser, J., Erramilli, S., and S. Khurana, “Terminology for Benchmarking Network-layer Traffic Control Mechanisms,” October 2006.) and Duplicate Packets [Po06] (Poretsky, S., Perser, J., Erramilli, S., and S. Khurana, “Terminology for Benchmarking Network-layer Traffic Control Mechanisms,” October 2006.).
  9. Wait sufficient time for queues to drain.
  10. Restart Offered Load.
  11. Restore link on DUT's Parallel Link member interface.
  12. Measure First Route Convergence Time.
  13. Measure Full Convergence Time.
  14. Stop Offered Load.
  15. Measure Route-Specific Convergence Times, Loss-Derived Convergence Time, Route LoC Periods, and Loss-Derived LoC Period. At the same time measure Out-of-Order Packets [Po06] (Poretsky, S., Perser, J., Erramilli, S., and S. Khurana, “Terminology for Benchmarking Network-layer Traffic Control Mechanisms,” October 2006.) and Duplicate Packets [Po06] (Poretsky, S., Perser, J., Erramilli, S., and S. Khurana, “Terminology for Benchmarking Network-layer Traffic Control Mechanisms,” October 2006.).

Results

The measured IGP convergence time may be influenced by the link failure indication time, LSA/LSP delay, LSA/LSP generation time, LSA/LSP flood packet pacing, SPF delay, SPF execution time, and routing and forwarding tables update time [Po09a] (Poretsky, S., “Considerations for Benchmarking Link-State IGP Data Plane Route Convergence,” March 2009.).



 TOC 

8.2.  Other failures



 TOC 

8.2.1.  Convergence Due to Layer 2 Session Loss

Objective

To obtain the IGP convergence time due to a local layer 2 loss.

Procedure

  1. Advertise an IGP topology from Tester to DUT using the topology shown in Figure 1 (IGP convergence test topology for local changes).
  2. Send Offered Load from Tester to DUT on ingress interface.
  3. Verify traffic is routed over Preferred Egress Interface.
  4. Remove Layer 2 session from DUT's Preferred Egress Interface. This is the Convergence Event.
  5. Measure First Route Convergence Time.
  6. Measure Full Convergence Time.
  7. Stop Offered Load.
  8. Measure Route-Specific Convergence Times, Loss-Derived Convergence Time, Route LoC Periods, and Loss-Derived LoC Period.
  9. Wait sufficient time for queues to drain.
  10. Restart Offered Load.
  11. Restore Layer 2 session on DUT's Preferred Egress Interface.
  12. Measure First Route Convergence Time.
  13. Measure Full Convergence Time.
  14. Stop Offered Load.
  15. Measure Route-Specific Convergence Times, Loss-Derived Convergence Time, Route LoC Periods, and Loss-Derived LoC Period.

Results

The measured IGP Convergence time may be influenced by the Layer 2 failure indication time, LSA/LSP delay, LSA/LSP generation time, LSA/LSP flood packet pacing, SPF delay, SPF execution time, and routing and forwarding tables update time [Po09a] (Poretsky, S., “Considerations for Benchmarking Link-State IGP Data Plane Route Convergence,” March 2009.).

Discussion

Configure IGP timers such that the IGP adjacency does not time out before layer 2 failure is detected.

To measure convergence time, traffic SHOULD start dropping on the Preferred Egress Interface on the instant the layer 2 session is removed. Alternatively the Tester SHOULD record the time the instant layer 2 session is removed and traffic loss SHOULD only be measured on the Next-Best Egress Interface. For loss-derived benchmarks the time of the Start Traffic Instant SHOULD be recorded as well. See Section 4.1 (Convergence Events without instant traffic loss).



 TOC 

8.2.2.  Convergence Due to Loss of IGP Adjacency

Objective

To obtain the IGP convergence time due to loss of an IGP Adjacency.

Procedure

  1. Advertise an IGP topology from Tester to DUT using the topology shown in Figure 1 (IGP convergence test topology for local changes).
  2. Send Offered Load from Tester to DUT on ingress interface.
  3. Verify traffic is routed over Preferred Egress Interface.
  4. Remove IGP adjacency from the Preferred Egress Interface while the layer 2 session MUST be maintained. This is the Convergence Event.
  5. Measure First Route Convergence Time.
  6. Measure Full Convergence Time.
  7. Stop Offered Load.
  8. Measure Route-Specific Convergence Times, Loss-Derived Convergence Time, Route LoC Periods, and Loss-Derived LoC Period.
  9. Wait sufficient time for queues to drain.
  10. Restart Offered Load.
  11. Restore IGP session on DUT's Preferred Egress Interface.
  12. Measure First Route Convergence Time.
  13. Measure Full Convergence Time.
  14. Stop Offered Load.
  15. Measure Route-Specific Convergence Times, Loss-Derived Convergence Time, Route LoC Periods, and Loss-Derived LoC Period.

Results

The measured IGP Convergence time may be influenced by the IGP Hello Interval, IGP Dead Interval, LSA/LSP delay, LSA/LSP generation time, LSA/LSP flood packet pacing, SPF delay, SPF execution time, and routing and forwarding tables update time [Po09a] (Poretsky, S., “Considerations for Benchmarking Link-State IGP Data Plane Route Convergence,” March 2009.).

Discussion

Configure layer 2 such that layer 2 does not time out before IGP adjacency failure is detected.

To measure convergence time, traffic SHOULD start dropping on the Preferred Egress Interface on the instant the IGP adjacency is removed. Alternatively the Tester SHOULD record the time the instant the IGP adjacency is removed and traffic loss SHOULD only be measured on the Next-Best Egress Interface. For loss-derived benchmarks the time of the Start Traffic Instant SHOULD be recorded as well. See Section 4.1 (Convergence Events without instant traffic loss).



 TOC 

8.2.3.  Convergence Due to Route Withdrawal

Objective

To obtain the IGP convergence time due to route withdrawal.

Procedure

  1. Advertise an IGP topology from Tester to DUT using the topology shown in Figure 1 (IGP convergence test topology for local changes). The routes that will be withdrawn MUST be a set of leaf routes advertised by at least two nodes in the emulated topology. The topology SHOULD be such that before the withdrawal the DUT prefers the leaf routes advertised by a node "nodeA" via the Preferred Egress Interface, and after the withdrawal the DUT prefers the leaf routes advertised by a node "nodeB" via the Next-Best Egress Interface.
  2. Send Offered Load from Tester to DUT on Ingress Interface.
  3. Verify traffic is routed over Preferred Egress Interface.
  4. The Tester withdraws the set of IGP leaf routes from nodeA. This is the Convergence Event. The withdrawal update message SHOULD be a single unfragmented packet. If the routes cannot be withdrawn by a single packet, the messages SHOULD be sent using the same pacing characteristics as the DUT. The Tester MAY record the time it sends the withdrawal message(s).
  5. Measure First Route Convergence Time.
  6. Measure Full Convergence Time.
  7. Stop Offered Load.
  8. Measure Route-Specific Convergence Times, Loss-Derived Convergence Time, Route LoC Periods, and Loss-Derived LoC Period.
  9. Wait sufficient time for queues to drain.
  10. Restart Offered Load.
  11. Re-advertise the set of withdrawn IGP leaf routes from nodeA emulated by the Tester. The update message SHOULD be a single unfragmented packet. If the routes cannot be advertised by a single packet, the messages SHOULD be sent using the same pacing characteristics as the DUT. The Tester MAY record the time it sends the update message(s).
  12. Measure First Route Convergence Time.
  13. Measure Full Convergence Time.
  14. Stop Offered Load.
  15. Measure Route-Specific Convergence Times, Loss-Derived Convergence Time, Route LoC Periods, and Loss-Derived LoC Period.

Results

The measured IGP convergence time is influenced by SPF or route calculation delay, SPF or route calculation execution time, and routing and forwarding tables update time [Po09a] (Poretsky, S., “Considerations for Benchmarking Link-State IGP Data Plane Route Convergence,” March 2009.).

Discussion

To measure convergence time, traffic SHOULD start dropping on the Preferred Egress Interface on the instant the routes are withdrawn by the Tester. Alternatively the Tester SHOULD record the time the instant the routes are withdrawn and traffic loss SHOULD only be measured on the Next-Best Egress Interface. For loss-derived benchmarks the time of the Start Traffic Instant SHOULD be recorded as well. See Section 4.1 (Convergence Events without instant traffic loss).



 TOC 

8.3.  Administrative changes



 TOC 

8.3.1.  Convergence Due to Local Adminstrative Shutdown

Objective

To obtain the IGP convergence time due to taking the DUT's Local Interface administratively out of service.

Procedure

  1. Advertise an IGP topology from Tester to DUT using the topology shown in Figure 1 (IGP convergence test topology for local changes).
  2. Send Offered Load from Tester to DUT on ingress interface.
  3. Verify traffic is routed over Preferred Egress Interface.
  4. Take the DUT's Preferred Egress Interface administratively out of service. This is the Convergence Event.
  5. Measure First Route Convergence Time.
  6. Measure Full Convergence Time.
  7. Stop Offered Load.
  8. Measure Route-Specific Convergence Times, Loss-Derived Convergence Time, Route LoC Periods, and Loss-Derived LoC Period.
  9. Wait sufficient time for queues to drain.
  10. Restart Offered Load.
  11. Restore Preferred Egress Interface by administratively enabling the interface.
  12. Measure First Route Convergence Time.
  13. Measure Full Convergence Time.
  14. Stop Offered Load.
  15. Measure Route-Specific Convergence Times, Loss-Derived Convergence Time, Route LoC Periods, and Loss-Derived LoC Period.
  16. It is possible that no measured packet loss will be observed for this test case.

Results

The measured IGP Convergence time may be influenced by LSA/LSP delay, LSA/LSP generation time, LSA/LSP flood packet pacing, SPF delay, SPF execution time, and routing and forwarding tables update time [Po09a] (Poretsky, S., “Considerations for Benchmarking Link-State IGP Data Plane Route Convergence,” March 2009.).



 TOC 

8.3.2.  Convergence Due to Cost Change

Objective

To obtain the IGP convergence time due to route cost change.

Procedure

  1. Advertise an IGP topology from Tester to DUT using the topology shown in Figure 1 (IGP convergence test topology for local changes).
  2. Send Offered Load from Tester to DUT on ingress interface.
  3. Verify traffic is routed over Preferred Egress Interface.
  4. The Tester, emulating the neighbor node, increases the cost for all IGP routes at DUT's Preferred Egress Interface so that the Next-Best Egress Interface becomes preferred path. The update message advertising the higher cost MUST be a single unfragmented packet. This is the Convergence Event. The Tester MAY record the time it sends the update message advertising the higher cost on the Preferred Egress Interface.
  5. Measure First Route Convergence Time.
  6. Measure Full Convergence Time.
  7. Stop Offered Load.
  8. Measure Route-Specific Convergence Times, Loss-Derived Convergence Time, Route LoC Periods, and Loss-Derived LoC Period.
  9. Wait sufficient time for queues to drain.
  10. Restart Offered Load.
  11. The Tester, emulating the neighbor node, decreases the cost for all IGP routes at DUT's Preferred Egress Interface so that the Preferred Egress Interface becomes preferred path. The update message advertising the lower cost MUST be a single unfragmented packet.
  12. Measure First Route Convergence Time.
  13. Measure Full Convergence Time.
  14. Stop Offered Load.
  15. Measure Route-Specific Convergence Times, Loss-Derived Convergence Time, Route LoC Periods, and Loss-Derived LoC Period.

Results

The measured IGP Convergence time may be influenced by SPF delay, SPF execution time, and routing and forwarding tables update time [Po09a] (Poretsky, S., “Considerations for Benchmarking Link-State IGP Data Plane Route Convergence,” March 2009.).

Discussion

To measure convergence time, traffic SHOULD start dropping on the Preferred Egress Interface on the instant the cost is changed by the Tester. Alternatively the Tester SHOULD record the time the instant the cost is changed and traffic loss SHOULD only be measured on the Next-Best Egress Interface. For loss-derived benchmarks the time of the Start Traffic Instant SHOULD be recorded as well. See Section 4.1 (Convergence Events without instant traffic loss).



 TOC 

9.  Security Considerations

Benchmarking activities as described in this memo are limited to technology characterization using controlled stimuli in a laboratory environment, with dedicated address space and the constraints specified in the sections above.

The benchmarking network topology will be an independent test setup and MUST NOT be connected to devices that may forward the test traffic into a production network, or misroute traffic to the test management network.

Further, benchmarking is performed on a "black-box" basis, relying solely on measurements observable external to the DUT/SUT.

Special capabilities SHOULD NOT exist in the DUT/SUT specifically for benchmarking purposes. Any implications for network security arising from the DUT/SUT SHOULD be identical in the lab and in production networks.



 TOC 

10.  IANA Considerations

This document requires no IANA considerations.



 TOC 

11.  Acknowledgements

Thanks to Sue Hares, Al Morton, Kevin Dubray, Ron Bonica, David Ward, Peter De Vriendt, Anuj Dewagan and the BMWG for their contributions to this work.



 TOC 

12. Normative References

[Br91] Bradner, S., “Benchmarking terminology for network interconnection devices,” RFC 1242, July 1991 (TXT).
[Br97] Bradner, S., “Key words for use in RFCs to Indicate Requirement Levels,” BCP 14, RFC 2119, March 1997 (TXT, HTML, XML).
[Br99] Bradner, S. and J. McQuaid, “Benchmarking Methodology for Network Interconnect Devices,” RFC 2544, March 1999 (TXT).
[Ca90] Callon, R., “Use of OSI IS-IS for routing in TCP/IP and dual environments,” RFC 1195, December 1990 (TXT, PS).
[Co08] Coltun, R., Ferguson, D., Moy, J., and A. Lindem, “OSPF for IPv6,” RFC 5340, July 2008 (TXT).
[Ho08] Hopps, C., “Routing IPv6 with IS-IS,” RFC 5308, October 2008 (TXT).
[Ko02] Koodli, R. and R. Ravikanth, “One-way Loss Pattern Sample Metrics,” RFC 3357, August 2002 (TXT).
[Ma98] Mandeville, R., “Benchmarking Terminology for LAN Switching Devices,” RFC 2285, February 1998 (TXT, HTML, XML).
[Mo98] Moy, J., “OSPF Version 2,” STD 54, RFC 2328, April 1998 (TXT, HTML, XML).
[Po06] Poretsky, S., Perser, J., Erramilli, S., and S. Khurana, “Terminology for Benchmarking Network-layer Traffic Control Mechanisms,” RFC 4689, October 2006 (TXT).
[Po09a] Poretsky, S., “Considerations for Benchmarking Link-State IGP Data Plane Route Convergence,” draft-ietf-bmwg-igp-dataplane-conv-app-17 (work in progress), March 2009 (TXT).
[Po09t] Poretsky, S. and B. Imhoff, “Terminology for Benchmarking Link-State IGP Data Plane Route Convergence,” draft-ietf-bmwg-igp-dataplane-conv-term-18 (work in progress), July 2009 (TXT).


 TOC 

Authors' Addresses

  Scott Poretsky
  Allot Communications
  67 South Bedford Street, Suite 400
  Burlington, MA 01803
  USA
Phone:  + 1 508 309 2179
Email:  sporetsky@allot.com
  
  Brent Imhoff
  Juniper Networks
  1194 North Mathilda Ave
  Sunnyvale, CA 94089
  USA
Phone:  + 1 314 378 2571
Email:  bimhoff@planetspork.com
  
  Kris Michielsen
  Cisco Systems
  6A De Kleetlaan
  Diegem, BRABANT 1831
  Belgium
Email:  kmichiel@cisco.com