idnits 2.17.1 draft-villamizar-mpls-forwarding-02.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document seems to lack the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords. (The document does seem to have the reference to RFC 2119 which the ID-Checklist requires). -- The document date (March 30, 2013) is 4044 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Outdated reference: A later version (-08) exists of draft-ietf-pwe3-mpls-eth-oam-iwk-07 == Outdated reference: A later version (-07) exists of draft-ietf-tictoc-1588overmpls-04 -- Obsolete informational reference (is this intentional?): RFC 4379 (Obsoleted by RFC 8029) -- Obsolete informational reference (is this intentional?): RFC 6424 (Obsoleted by RFC 8029) -- Obsolete informational reference (is this intentional?): RFC 6829 (Obsoleted by RFC 8029) Summary: 0 errors (**), 0 flaws (~~), 4 warnings (==), 4 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 MPLS C. Villamizar, Ed. 3 Internet-Draft OCCNC 4 Intended status: Informational K. Kompella 5 Expires: October 1, 2013 Contrail Systems 6 S. Amante 7 Level 3 Communications, Inc. 8 A. Malis 9 Verizon 10 C. Pignataro 11 Cisco 12 March 30, 2013 14 MPLS Forwarding Compliance and Performance Requirements 15 draft-villamizar-mpls-forwarding-02 17 Abstract 19 This document provides guidelines for implementors regarding MPLS 20 forwarding and a basis for evaluations of forwarding implementations. 21 Guidelines cover many aspects of MPLS forwarding. Topics are 22 highlighted where implementors might potentially overlook practical 23 requirements which are unstated or underemphasized or are optional 24 for conformance to RFCs. 26 Status of this Memo 28 This Internet-Draft is submitted in full conformance with the 29 provisions of BCP 78 and BCP 79. 31 Internet-Drafts are working documents of the Internet Engineering 32 Task Force (IETF). Note that other groups may also distribute 33 working documents as Internet-Drafts. The list of current Internet- 34 Drafts is at http://datatracker.ietf.org/drafts/current/. 36 Internet-Drafts are draft documents valid for a maximum of six months 37 and may be updated, replaced, or obsoleted by other documents at any 38 time. It is inappropriate to use Internet-Drafts as reference 39 material or to cite them other than as "work in progress." 41 This Internet-Draft will expire on October 1, 2013. 43 Copyright Notice 45 Copyright (c) 2013 IETF Trust and the persons identified as the 46 document authors. All rights reserved. 48 This document is subject to BCP 78 and the IETF Trust's Legal 49 Provisions Relating to IETF Documents 50 (http://trustee.ietf.org/license-info) in effect on the date of 51 publication of this document. Please review these documents 52 carefully, as they describe your rights and restrictions with respect 53 to this document. Code Components extracted from this document must 54 include Simplified BSD License text as described in Section 4.e of 55 the Trust Legal Provisions and are provided without warranty as 56 described in the Simplified BSD License. 58 Table of Contents 60 1. Introduction and Document Scope . . . . . . . . . . . . . . . 4 61 1.1. Acronyms . . . . . . . . . . . . . . . . . . . . . . . . . 4 62 1.2. Use of Requirements Language . . . . . . . . . . . . . . . 8 63 1.3. Apparent Misconceptions . . . . . . . . . . . . . . . . . 9 64 1.4. Target Audience . . . . . . . . . . . . . . . . . . . . . 10 65 2. Forwarding Issues . . . . . . . . . . . . . . . . . . . . . . 10 66 2.1. Forwarding Basics . . . . . . . . . . . . . . . . . . . . 10 67 2.1.1. MPLS Reserved Labels . . . . . . . . . . . . . . . . . 11 68 2.1.2. MPLS Differentiated Services . . . . . . . . . . . . . 12 69 2.1.3. Time Synchronization . . . . . . . . . . . . . . . . . 13 70 2.1.4. Uses of Multiple Label Stack Entries . . . . . . . . . 13 71 2.1.5. MPLS Link Bundling . . . . . . . . . . . . . . . . . . 14 72 2.1.6. MPLS Hierarchy . . . . . . . . . . . . . . . . . . . . 14 73 2.1.7. MPLS Fast Reroute (FRR) . . . . . . . . . . . . . . . 14 74 2.1.8. Pseudowire Encapsulation . . . . . . . . . . . . . . . 15 75 2.1.8.1. Pseudowire Sequence Number . . . . . . . . . . . . 15 76 2.1.9. Layer-2 and Layer-3 VPN . . . . . . . . . . . . . . . 16 77 2.2. MPLS Multicast . . . . . . . . . . . . . . . . . . . . . . 16 78 2.3. Packet Rates . . . . . . . . . . . . . . . . . . . . . . . 17 79 2.4. MPLS Multipath Techniques . . . . . . . . . . . . . . . . 19 80 2.4.1. Pseudowire Control Word . . . . . . . . . . . . . . . 20 81 2.4.2. Large Microflows . . . . . . . . . . . . . . . . . . . 20 82 2.4.3. Pseudowire Flow Label . . . . . . . . . . . . . . . . 21 83 2.4.4. MPLS Entropy Label . . . . . . . . . . . . . . . . . . 21 84 2.4.5. Fields Used for Multipath . . . . . . . . . . . . . . 22 85 2.4.5.1. MPLS Fields in Multipath . . . . . . . . . . . . . 22 86 2.4.5.2. IP Fields in Multipath . . . . . . . . . . . . . . 23 87 2.4.5.3. Fields Used in Flow Label . . . . . . . . . . . . 25 88 2.4.5.4. Fields Used in Entropy Label . . . . . . . . . . . 25 89 2.5. MPLS-TP and UHP . . . . . . . . . . . . . . . . . . . . . 26 90 2.6. Local Delivery of Packets . . . . . . . . . . . . . . . . 26 91 2.6.1. DoS Protection . . . . . . . . . . . . . . . . . . . . 26 92 2.6.2. MPLS OAM . . . . . . . . . . . . . . . . . . . . . . . 28 93 2.6.3. Pseudowire OAM . . . . . . . . . . . . . . . . . . . . 29 94 2.6.4. MPLS-TP OAM . . . . . . . . . . . . . . . . . . . . . 30 95 2.6.5. MPLS OAM and Layer-2 OAM Interworking . . . . . . . . 31 96 2.6.6. Extent of OAM Support by Hardware . . . . . . . . . . 32 97 2.7. Number and Size of Flows . . . . . . . . . . . . . . . . . 32 98 3. Questions for Suppliers . . . . . . . . . . . . . . . . . . . 33 99 3.1. Basic Compliance . . . . . . . . . . . . . . . . . . . . . 33 100 3.2. Basic Performance . . . . . . . . . . . . . . . . . . . . 35 101 3.3. Multipath Capabilities and Performance . . . . . . . . . . 35 102 3.4. Pseudowire Capabilities and Performance . . . . . . . . . 36 103 3.5. Entropy Label Support and Performance . . . . . . . . . . 36 104 3.6. DoS Protection . . . . . . . . . . . . . . . . . . . . . . 37 105 3.7. OAM Capabilities and Performance . . . . . . . . . . . . . 37 106 4. Forwarding Compliance and Performance Testing . . . . . . . . 37 107 4.1. Basic Compliance . . . . . . . . . . . . . . . . . . . . . 38 108 4.2. Basic Performance . . . . . . . . . . . . . . . . . . . . 38 109 4.3. Multipath Capabilities and Performance . . . . . . . . . . 39 110 4.4. Pseudowire Capabilities and Performance . . . . . . . . . 40 111 4.5. Entropy Label Support and Performance . . . . . . . . . . 40 112 4.6. DoS Protection . . . . . . . . . . . . . . . . . . . . . . 41 113 4.7. OAM Capabilities and Performance . . . . . . . . . . . . . 42 114 5. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 42 115 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 43 116 7. Security Considerations . . . . . . . . . . . . . . . . . . . 43 117 8. References . . . . . . . . . . . . . . . . . . . . . . . . . . 43 118 8.1. Normative References . . . . . . . . . . . . . . . . . . . 43 119 8.2. Informative References . . . . . . . . . . . . . . . . . . 44 120 Appendix A. Organization of References Section . . . . . . . . . 49 121 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 49 123 1. Introduction and Document Scope 125 The initial purpose of this document was to address concerns raised 126 on the MPLS WG mailing list about shortcomings in implementations of 127 MPLS forwarding. Documenting existing misconceptions and potential 128 pitfalls might potentially avoid repeating past mistakes. The 129 document has grown to address a broad set of forwarding requirements. 131 The focus of this document is MPLS forwarding, base pseudowire 132 forwarding, and MPLS OAM. The use of pseudowire control word, and 133 sequence number are discussed. Specific pseudowire AC and NSP are 134 out of scope. Specific pseudowire applications, such as various 135 forms of VPN, are out of scope. 137 MPLS support for multipath techniques is considered essential by many 138 service providers and is useful for other high capacity networks. In 139 order to obtain sufficient entropy from MPLS traffic service 140 providers and others find it essential for the MPLS implementation to 141 interpret the MPLS payload as IPv4 or IPv6 based on the contents of 142 the first nibble of payload. The use of IP addresses, the IP 143 protocol field, and UDP and TCP port number fields in multipath load 144 balancing are considered within scope. The use of any other IP 145 protocol fields, such as tunneling protocols carried within IP, are 146 out of scope. 148 Implementation details are a local matter and are out of scope. Most 149 interfaces today operate at 1 Gb/s or greater. It is assumed that 150 all forwarding operations are implemented in specialized forwarding 151 hardware rather than on a special purpose processor. This is often 152 referred to as "fast path" and "slow path" processing. Some 153 recommendations are made regarding implemeting control or management 154 plane functionality in specialized hardware or with limited 155 assistance from specialized hardware. This advise is based on 156 expected control or management protocol loads and on the need for 157 denial of service (DoS) protection. 159 1.1. Acronyms 161 The following acronyms are used. 163 AC Attachment Circuit ([RFC3985]) 165 ACH Associated Channel Header (pseudowires) 167 ACK Acknowledgement (TCP flag and type of packet) 168 AIS Alarm Indication Signal (MPLS-TP OAM) 170 ATM Asynchronous Transfer Mode (legacy switched circuits) 172 BFD Bidirectional Forwarding Detection 174 BGP Border Gateway Protocol 176 CC-CV Connectivity Check and Connectivity Verification 178 CE Customer Edge (LDP) 180 CPU Central Processing Unit (computer or microprocessor) 182 CT Class Type ([RFC4124]) 184 CW Control Word ([RFC4385]) 186 DCCP Datagram Congestion Control Protocol 188 DDoS Distributed Denial of Service 190 DM Delay Measurement (MPLS-TP OAM) 192 DSCP Differentiated Services Code Point ([RFC2474]) 194 DWDM Dense Wave Division Multiplexing 196 DoS Denial of Service 198 E-LSP EXP-Inferred-PSC LSP ([RFC3270]) 200 EBGP External BGP 202 ECMP Equal Cost Multi-Path 204 ECN Explicit Congestion Notification ([RFC3168] and [RFC5129]) 206 EL Entropy Label ([RFC6790]) 208 ELI Entropy Label Indicator ([RFC6790]) 210 EXP Experimental (field in MPLS renamed to TC in [RFC5462]) 212 FEC Forwarding Equivalence Classes (LDP), also Forward Error 213 Correction in other context 215 FR Frame Relay (legacy switched circuits) 217 FRR Fast Reroute ([RFC4090]) 219 G-ACh Generic Associated Channel ([RFC5586]) 221 GAL Generic Associated Channel Label ([RFC5586]) 223 GFP Generic Framing Protocol (used in OTN) 225 GMPLS Generalized MPLS ([RFC3471]) 227 GTSM Generalized TTL Security Mechanism ([RFC5082]) 229 Gb/s Gigabits per second (billion bits per second) 231 IANA Internet Assigned Numbers Authority 233 ILM Incoming Label Map ([RFC3031]) 235 IP Internet Protocol 237 IPVPN Internet Protocol VPN 239 IPv4 Internet Protocol version 4 241 IPv6 Internet Protocol version 6 243 L-LSP Label-Only-Inferred-PSC LSP ([RFC3270]) 245 L2VPN Layer 2 VPN 247 LDP Label Distribution Protocol ([RFC5036]) 249 LER Label Edge Router ([RFC3031]) 251 LM Loss Measurement (MPLS-TP OAM) 253 LSP Label Switched Path ([RFC3031]) 255 LSR Label Switching Router ([RFC3031]) 257 MP2MP Multipoint to Point 259 MPLS MultiProtocol Label Switching ([RFC3031]) 260 MPLS-TP MPLS Transport Profile ([RFC5317]) 262 Mb/s Megabits per second (million bits per second) 264 NSP Native Service Processing ([RFC3985]) 266 NTP Network Time Protocol 268 OAM Operations, Administration, and Maintenance ([RFC6291]) 270 OOB Out-of-band (not carried within a data channel) 272 OTN Optical Transport Network 274 P Provider router (LDP) 276 P2MP Point to Multi-Point 278 PE Provider Edge router (LDP) 280 PHB Per-Hop-Behavior ([RFC2475]) 282 PHP Penultimate Hop Popping ([RFC3443]) 284 POS Packet over SONET 286 PSC This acronym has multiple interpretations. 288 1. Packet Switch Capable ([RFC3471] 290 2. PHB Scheduling Class ([RFC3270]) 292 3. Protection State Coordination (MPLS-TP linear protection) 294 PTP Precision Time Protocol 296 PW Pseudowire 298 QoS Quality of Service 300 RA Router Alert ([RFC3032]) 302 RDI Remote Defect Indication (MPLS-TP OAM) 304 RSVP-TE RSVP Traffic Engineering 305 RTP Real-Time Transport Protocol 307 SCTP Stream Control Transmission Protocol 309 SDH Synchronous Data Hierarchy (European SONET, a form of TDM) 311 SONET Synchronous Optical Network (US SDH, a form of TDM) 313 T-LDP Targeted LDP (LDP sessions over more than one hop) 315 TC Traffic Class ([RFC5462]) 317 TCP Transmission Control Protocol 319 TDM Time-Division Multiplexing (legacy encapsulations) 321 TOS Type of Service (see [RFC2474]) 323 TTL Time-to-live (a field in IP and MPLS headers) 325 UDP User Datagram Protocol 327 UHP Ultimate Hop Popping (opposite of PHP) 329 VCCV Virtual Circuit Connectivity Verification ([RFC5085]) 331 VLAN Virtual Local Area Network (Ethernet) 333 VOQ Virtual Output Queuing (switch fabric design) 335 VPN Virtual Private Network 337 WG Working Group 339 1.2. Use of Requirements Language 341 This document is informational. The key words "MUST", "MUST NOT", 342 "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", 343 "RECOMMENDED", "MAY", and "OPTIONAL" are used only where the 344 requirement is specified in an existing RFC. These keywords SHOULD 345 be interpreted as described in RFC 2119 [RFC2119]. 347 Advice given in this document does not use the upper case RFC 2119 348 keywords, except where explicitly noted that the keywords indicate 349 that operator experiences indicate a requirement, but there are no 350 existing RFC requirements. Such advice may be ignored by 351 implementations. Similarly, implementations not claiming conformance 352 to specific RFCs may ignore the requirements of those RFCs. In both 353 cases, implementators may be doing so at their own peril. 355 1.3. Apparent Misconceptions 357 In early generations of forwarding silicon (which might now be behind 358 us), there apparently were some misconceptions about MPLS. The 359 following statements provide clarifications. 361 1. There are practical reasons to have more than one or two labels 362 in an MPLS label stack. Under some circumstances the label stack 363 can become quite deep. See Section 2.1. 365 2. The label stack MUST be considered to be arbitrarily deep. 366 Section 3.27.4. "Hierarchy: LSP Tunnels within LSPs" of RFC 3031 367 [RFC3031] states "The label stack mechanism allows LSP tunneling 368 to nest to any depth." If a bottom of the label stack cannot be 369 found, but sufficient number of labels exist to forward, an LSR 370 MUST forward the packet. An LSR MUST NOT assume the packet is 371 malformed unless the end of packet is found before bottom of 372 stack. See Section 2.1. 374 3. In networks where deep label stacks are encountered, they are not 375 rare. Full packet rate performance is required regardless of 376 label stack depth, except where multiple pop operations are 377 required. See Section 2.1. 379 4. Research has shown that long bursts of short packets with 40 byte 380 or 44 byte IP payload sizes in these bursts are quite common. 381 This is due to TCP ACK compression [ACK-compression]. 383 A. A forwarding engine SHOULD, if practical, be able to sustain 384 an arbitrarily long sequence of small packets arriving at 385 full interface rate. 387 B. If indefinite full packet rate for small packets is not 388 practical, a forwarding engine MUST be able to buffer a long 389 sequence of small packets inbound to the on-chip decision 390 engine and sustain full interface rate for some reasonable 391 average packet rate. Absent this small on-chip buffering, 392 QoS agnostic packet drops can occur. 394 See Section 2.3. 396 5. The implementor and system designer MUST support pseudowire 397 control word if MPLS-TP is supported or if ACH is being used on a 398 pseudowire [RFC5586]. The implementor and system designer SHOULD 399 support pseudowire control word if MPLS-TP and [RFC5586] are not 400 used [RFC5085]. Deployments SHOULD enable pseudowire control 401 word. See Section 2.4.1. 403 6. The implementor and system designer SHOULD support adding a 404 pseudowire Flow Label [RFC6391]. Deployments MAY enable this 405 feature for appropriate pseudowire types. See Section 2.4.3. 407 7. The implementor and system designer SHOULD support adding an MPLS 408 entropy label [RFC6790]. Deployments MAY enable this feature. 409 See Section 2.4.4. 411 1.4. Target Audience 413 This document is intended for multiple audiences: implementor 414 (implementing MPLS forwarding in silicon or in software); systems 415 designer (putting together a MPLS forwarding systems); deployer 416 (running an MPLS network). These guidelines are intended to serve 417 the following purposes: 419 1. Explain what to do and what not to do when a deep label stack is 420 encountered. (audience: implementor) 422 2. Highlight pitfalls to look for when implementing an MPLS 423 forwarding chip. (audience: implementor) 425 3. Provide a checklist of features and performance specifications to 426 request. (audience: systems designer, deployer) 428 4. Provide a set of tests to perform. (audience: systems designer, 429 deployer). 431 The implementor, systems designer, and deployer have a transitive 432 supplier customer relationship. It is in the best interest of the 433 supplier to review their product against their customer's checklist 434 and customer's customer's checklist if applicable. 436 2. Forwarding Issues 438 A brief review of forwarding issues is provided in the subsections 439 that follow. This section provides some background on why some of 440 these requirements exist. The questions to ask of suppliers and 441 testing is covered in the following sections, Section 3 and 442 Section 4. 444 2.1. Forwarding Basics 446 Basic MPLS architecture and MPLS encapsulation, and therefore packet 447 forwarding are defined in [RFC3031] and [RFC3032]. RFC3031 and 448 RFC3032 are somewhat LDP centric. RSVP-TE supports traffic 449 engineering (TE) and fast reroute, features that LDP lacks. The base 450 document for RSVP-TE based MPLS is [RFC3209]. 452 A few RFCs update RFC3032. Those with impact on forwarding include 453 the following. 455 1. TTL processing is clarified in [RFC3443]. 457 2. The use of MPLS Explicit NULL is modified in [RFC4182]. 459 3. Differentiated Services is supported by [RFC3270] and [RFC4124]. 460 The "EXP" field is renamed to "Traffic Class" in [RFC5462], 461 removing any misconception that it was available for 462 experimentation or could be ignored. 464 4. ECN is supported by [RFC5129]. 466 5. The MPLS G-ACh and GAL are defined in [RFC5586]. 468 Other RFCs have implications to MPLS Forwarding and do not update 469 RFC3032 or RFC3209, including: 471 1. The pseudowire (PW) Associated Channel Header (ACH), defined by 472 [RFC5085], later generalized by the MPLS G-ACh [RFC5586]. 474 2. The entropy label indicator (ELI) and entropy label (EL) are 475 defined by [RFC6790]. 477 A few RFCs update RFC3209. Those that are listed as updating RFC3209 478 generally impact only RSVP-TE signaling. Forwarding is modified by 479 major extension built upon RFC3209. 481 RFCs which impact forwarding are discussed in the following 482 subsections. 484 2.1.1. MPLS Reserved Labels 486 [RFC3032] specifies that label values 0-15 are reserved labels with 487 special meanings. Three values of NULL label are defined (two of 488 which are later updated by [RFC4182]) and a router-alert label is 489 defined. The original intent was that reserved labels, except the 490 NULL labels, could be sent to the routing engine CPU rather than be 491 processed in forwarding hardware. Hardware support is required by 492 new RFCs such as those defining entropy label and OAM processed as a 493 result of receiving a GAL. For new reserved labels, some 494 accommodation is needed for LSR that will send the labels to a 495 general purpose CPU or other highly programmable hardware. For 496 example, ELI will only be sent to LSR which have signaled support for 497 [RFC6790] and high OAM packet rate must be negotiated among 498 endpoints. 500 [RFC3429] reserves a label for ITU-T Y.1711, however Y.1711 does not 501 work with multipath and its use is strongly discouraged. 503 The current list of reserved labels can be found on the 504 "Multiprotocol Label Switching Architecture (MPLS) Label Values" 505 registry reachable at IANA's pages at . 507 When an unknown reserved label is encountered or a reserved label not 508 directly handled in forwarding hardware is encountered, the packet 509 should be sent to a general purpose CPU by default. If this 510 capability is supported, there must be an option to either drop or 511 rate limit such packets on a per reserved label value basis. 513 2.1.2. MPLS Differentiated Services 515 [RFC2474] deprecates the IP Type of Service (TOS) and IP Precedence 516 (Prec) fields and replaces them with the Differentiated Services 517 Field more commonly known as the Differentiated Services Code Point 518 (DSCP) field. [RFC2475] defines the Differentiated Services 519 architecture, which in other forum is often called a Quality of 520 Service (QoS) architecture. 522 MPLS uses the Traffic Class (TC) field to support Differentiated 523 Services [RFC5462]. There are two primary documents describing how 524 DSCP is mapped into TC. 526 1. [RFC3270] defines E-LSP and L-LSP. E-LSP use a static mapping of 527 DSCP into TC. L-LSP uses a per LSP mapping of DSCP into TC, with 528 one PHB Scheduling Class (PSC) per L-LSP. Each PSC can use 529 multiple Per-Hop Behavior (PHB) values. For example, the Assured 530 Forwarding service defines three PSC, each with three PHB 531 [RFC2597]. 533 2. [RFC4124] defines assignment of a class-type (CT) to an LSP, 534 where a per CT static mapping of TC to PHB is used. [RFC4124] 535 provides a means to support up to eight E-LSP-like mappings of 536 DSCP to TC. 538 To meet Differentiated Services requirements specified in [RFC3270], 539 the following forwarding requirements must be met. An ingress LER 540 MUST be able to select an LSP and then apply a per LSP map of DSCP 541 into TC. A midpoint LSR MUST be able to apply a per LSP map of TC to 542 PHB. The number of mappings supported will be far less than the 543 number of LSP supported. 545 2.1.3. Time Synchronization 547 PTP or NTP may be carried over MPLS [I-D.ietf-tictoc-1588overmpls]. 548 Generally NTP will be carried within IP with IP carried in MPLS 549 [RFC5905]. Both PTP and NTP benefit from accurate time stamping of 550 incoming packets and the ability to insert accurate time stamps in 551 outgoing packets. PTP correction which occurs when forwarding 552 requires updating a timestamp compensation field based on the 553 difference between packet arrival at an LSR and packet transmit time 554 at that same LSR. 556 Since the label stack depth may vary, hardware should allow a 557 timestamp to be placed in an outgoing packet at any specified byte 558 position. It may be necessary to modify layer-2 checksums or frame 559 check sequences after insertion. PTP and NTP timestamp formats 560 differ slightly. If NTP or PTP is carried over UDP/IP or UDP/IP/ 561 MPLS, the UDP checksum will also have to be updated. 563 Accurate time synchronization in addition to being generally useful 564 is required for MPLS-TP delay measurement (DM) OAM. See 565 Section 2.6.4. 567 2.1.4. Uses of Multiple Label Stack Entries 569 MPLS deployments in the early part of the prior decade (circa 2000) 570 tended to support either LDP or RSVP-TE. LDP was favored by some for 571 its ability to scale to a very large number of PE devices at the edge 572 of the network, without adding deployment complexity. RSVP-TE was 573 favored, generally in the network core, where traffic engineering 574 and/or fast reroute were considered important. 576 Both LDP and RSVP-TE are used simultaneously within major Service 577 Provider networks using a technique known as "LDP over RSVP-TE 578 Tunneling". This technique allows service providers to carry LDP 579 tunnels, originating and terminating at PE's, inside of RSVP-TE 580 tunnels, generally between Inter-City P routers, to take advantage of 581 Traffic Engineering and Fast Re-Route on more expensive Inter-City 582 and Inter-Continental Transport paths. LDP over RSVP-TE tunneling 583 requires a minimum of two MPLS labels: one each for LDP and RSVP-TE. 585 The use of MPLS FRR [RFC4090] might add one more label to MPLS 586 traffic, but only when FRR protection was in use. If LDP over 587 RSVP-TE is in use, and FRR protection is in use, then at least three 588 MPLS labels are present on the label stack on the links through which 589 the Bypass LSP traverses. FRR is covered in Section 2.1.7. 591 LDP L2VPN, LDP IPVPN, BGP L2VPN, and BGP IPVPN added support for VPN 592 services that are deployed in the vast majority of service providers. 594 These VPN services added yet another label, bringing the label stack 595 depth (when FRR is active) to four. 597 Pseudowires and VPN are discussed in further detail in Section 2.1.8 598 and Section 2.1.9. 600 2.1.5. MPLS Link Bundling 602 MPLS Link Bundling was the first RFC to address the need for multiple 603 parallel links between nodes [RFC4201]. MPLS Link Bundling is 604 notable in that it tried not to change MPLS forwarding, except in 605 specifying the "All-Ones" component link. MPLS Link Bundling is 606 seldom if ever deployed. Instead multipath techniques described in 607 Section 2.4 are used. 609 2.1.6. MPLS Hierarchy 611 MPLS hierarchy is defined in [RFC4206]. Although RFC4206 is 612 considered part of GMPLS, the Packet Switching Capable (PSC) portion 613 of the MPLS hierarchy are applicable to MPLS and may be supported in 614 an otherwise GMPLS free implementation. The MPLS PSC hierarchy 615 remains the most likely means of providing further scaling in an 616 RSVP-TE MPLS network, particularly where the network is designed to 617 provide RSVP-TE connectivity to the edges. This is the case for 618 envisioned MPLS-TP networks. The use of the MPLS PSC hierarchy can 619 add as many as four labels to a label stack, though it is likely that 620 only one layer of PSC will be used in the near future. 622 2.1.7. MPLS Fast Reroute (FRR) 624 Fast reroute is defined by [RFC4090]. Two significantly different 625 methods are the "One-to-One Backup" method which uses the "Detour 626 LSP" and the " Facility Backup" which uses a "bypass tunnel". These 627 are commonly referred to as the detour and bypass methods 628 respectively. 630 The detour method makes use of a presignaled LSP. Hardware 631 assistance is needed for detour FRR only if necessary to accomplish 632 local repair of a large number of LSP within the 10s of milliseconds 633 target. For each affected LSP a swap operation must be reprogrammed 634 or otherwise switched over. The use of detour FRR doubles the number 635 of LSP terminating at any given hop and will increase the number of 636 LSP within a network by a factor dependent on the average detour path 637 length. 639 The bypass method makes use of a tunnel that is unused when no fault 640 exists but may carry many LSP when a local repair is required. There 641 is no presignaling indicating which working LSP will be diverted into 642 any specific bypass LSP. The egress LSR of the bypass LSP MUST use 643 platform label space (as defined in [RFC3031]) so that an LSP working 644 path on any give interface can be backed up using a bypass LSP 645 terminating on any other interface. Hardware assistance is needed if 646 necessary to accomplish local repair of a large number of LSP within 647 the 10s of milliseconds target. For each affected LSP a swap 648 operation must be reprogrammed or otherwise switched over with an 649 additional push of the bypass LSP label. In addition the use of 650 platform label space impacts the size of the LSR ILM for LSR with a 651 very large number of interfaces. 653 2.1.8. Pseudowire Encapsulation 655 The pseudowire (PW) architecture is defined in [RFC3985]. A 656 pseudowire, when carried over MPLS, adds one or more additional label 657 entries to the MPLS label stack. A PW Control Word is defined in 658 [RFC4385] with motivation for defining the control word in [RFC4928]. 659 The PW Associated Channel defined in [RFC4385] is used for OAM in 660 [RFC5085]. The PW Flow Label is defined in [RFC6391] and is 661 discussed further in this document in Section 2.4.3. 663 There are numerous pseudowire encapsulations, supporting emulation of 664 services such as Frame Relay, ATM, Ethernet, TDM, and SONET/SDH over 665 packet switched networks (PSNs) using IP or MPLS. 667 The pseudowire encapsulation is out of scope for this document. 668 Pseudowire impact on MPLS forwarding at midpoint LSR is within scope. 669 The impact on ingress MPLS push and egress MPLS UHP pop are within 670 scope. While pseudowire encapsulation is out of scope, some advice 671 is given on sequence number support. 673 2.1.8.1. Pseudowire Sequence Number 675 Pseudowire (PW) sequence number support is most important for PW 676 payload types with a high expectation of in-order delivery. 677 Resequencing support, rather than dropping at egress on out of order 678 arrival, is most important for PW payload types with a high 679 expectation of lossless delivery. For example, TDM payloads require 680 sequence number support and require resequencing support. The same 681 is true of ATM CBR service. ATM VBR or ABR may have somewhat relaxed 682 requirements, but generally require ATM Early Packet Discard (EPD) or 683 ATM Partial Packet Discard (PPD) [ATM-EPD-and-PPD]. Though sequence 684 number support and resequencing support are beneficial to PW packet 685 oriented payloads such as FR and Ethernet, they are highly desirable 686 but not as strongly required. 688 Packet reorder should be rare except in a small number of 689 circumstances, most of which are due to network design or equipment 690 design errors: 692 1. The most common case is where reordering occurs is rare, 693 occurring only when a network or equipment fault forces traffic 694 on a new path with different delay. The packet loss that 695 accompanies a network or equipment fault is generally more 696 disruptive than any reordering which may occur. 698 2. A path change can be caused by reasons other than a network or 699 equipment fault, such as administrative routing change. This may 700 result in packet reordering but generally without any packet 701 loss. 703 3. If the edge is not using pseudowire control word (CW) and the 704 core is using multipath, reordering will be far more common. If 705 this is occurring, the best solution is to use CW on the edge, 706 rather than try to fix the reordering using resequencing. 708 4. Another avoidable case is where some core equipment has multipath 709 and for some reason insists on periodically installing a new 710 random number as the multipath hash seed. If supporting MPLS-TP, 711 equipment MUST provide a means to disable periodic hash reseeding 712 and deployments MUST disable periodic hash reseeding. Even if 713 not supporting MPLS-TP, equipment should provide a means to 714 disable periodic hash reseeding and deployments should disable 715 periodic hash reseeding. 717 2.1.9. Layer-2 and Layer-3 VPN 719 Layer-2 VPN [RFC4664] and Layer-3 VPN [RFC4110] add one or more label 720 entry to the MPLS label stack. VPN encapsulations are out of scope 721 for this document. Its impact on forwarding at midpoint LSR are 722 within scope. 724 Any of these services may be used on an MPLS entropy label enabled 725 ingress and egress (see Section 2.4.4 for discussion of entropy 726 label) which would add an additional label to the MPLS label stack. 727 The need to provide a useful entropy label value impacts the 728 requirements of the VPN ingress LER but is out of scope for this 729 document. 731 2.2. MPLS Multicast 733 MPLS Multicast encapsulation is clarified in [RFC5332]. MPLS 734 Multicast may be signaled using RSVP-TE [RFC4875] or LDP [RFC6388]. 736 [RFC4875] defines a root initiated RSVP-TE LSP setup rather than leaf 737 initiated join used in IP multicast. [RFC6388] defines a leaf 738 initiated LDP setup. Both [RFC4875] and [RFC6388] define point to 739 multipoint (P2MP) LSP setup. [RFC6388] also defined multipoint to 740 multipoint (MP2MP) LSP setup. 742 The P2MP LSP have a single source. An LSR may be a leaf node, an 743 intermediate node, or a "bud" node. A bud serves as both a leaf and 744 intermediate. At a leaf an MPLS pop is performed. The payload may 745 be a IP Multicast packet that requires further replication. At an 746 intermediate node a MPLS swap operation is performed. The bud 747 requires that both a pop operation and a swap operation be performed 748 for the same incoming packet. 750 One strategy to support P2MP functionality is to pop at the LSR 751 ingress and then optionally push labels at each LSR egress. A given 752 LSR egress chip may support multiple egress interfaces, each of which 753 requires a copy, but each with a different set of added labels and 754 layer-2 encapsulation. Some physical interfaces may have multiple 755 sub-interfaces (such as Ethernet VLAN or channelized interfaces) each 756 requiring a copy. 758 If packet replication is performed at LSR ingress, then the ingress 759 interface performance may suffer. If the packet replication is 760 performed within a LSR switching fabric and at LSR egress, congestion 761 of egress interfaces cannot make use of backpressure to ingress 762 interfaces using techniques such as virtual output queuing (VOQ). If 763 buffering is primarily supported at egress, then the need for 764 backpressure is minimized. There may be no good solution for high 765 volumes of multicast traffic if VOQ is used. 767 MP2MP LSP differ in that any branch may provide an input, including a 768 leaf. Packets must be replicated onto all other branches. This 769 forwarding is often implemented as multiple P2MP forwarding trees, 770 one for each potential input. 772 2.3. Packet Rates 774 While average packet size of Internet traffic may be large, long 775 sequences of small packets have both been predicted in theory and 776 observed in practice. Traffic compression and TCP ACK compression 777 can conspire to create long sequences of packets of 40-44 bytes in 778 payload length. If carried over Ethernet, the 64 byte minimum 779 payload applies, yielding a packet rate of approximately 150 Mpps 780 (million packets per second) for the duration of the burst on a 781 nominal 100 Gb/s link. The peak rate is higher for other 782 encapsulations can be as high as 250 Mpps (for example IP or MPLS 783 encapsulated using GFP over OTN ODU4). 785 It is also possible that the packet rates for a minimum payload size, 786 such as 64 byte (64B) payload for Ethernet, is acceptable, but the 787 rate declines for other packet sizes, such as 65B payload. There are 788 other packet rates of interest besides TCP ACK. For example, a TCP 789 ACK carried over an Ethernet PW over MPLS over Ethernet may occupy 790 82B or 82B plus an increment of 4B if additional MPLS labels are 791 present. 793 A graph of packet rate vs. packet size often displays a sawtooth. 794 The sawtooth is commonly due to a memory bottleneck and memory 795 widths, sometimes internal cache, but often a very wide external 796 buffer memory interface. In some cases it may be due to a fabric 797 transfer width. A fine packing, rounding up to the nearest 8B or 16B 798 will result in a fine sawtooth with small degradation for 65B, and 799 even less for 82B packets. A course packing, rounding up to 64B can 800 yield a sharper drop in performance for 65B packets, or perhaps more 801 important, a larger drop for 82B packets. 803 The loss of some TCP ACK packets are not the primary concern when 804 such a burst occurs. When a burst occurs, any other packets, 805 regardless of packet length and packet QoS are dropped once on-chip 806 input buffers prior to the decision engine are exceeded. Buffers in 807 front of the packet decision engine are often very small or non- 808 existent (less than one packet of buffer) causing significant QoS 809 agnostic packet drop. 811 Internet service providers and content providers generally specify 812 full rate forwarding with 40 byte payload packets as a requirement. 813 This requirement often can be waived if the provider can be convinced 814 that when long sequence of short packets occur no packets will be 815 dropped. 817 Many equipment suppliers have pointed out that the extra cost in 818 designing hardware capable of processing the minimum size packets at 819 full line rate is significant for very high speed interfaces. If 820 hardware is not capable of processing the minimum size packets at 821 full line rate, then that hardware MUST be capable of handling large 822 burst of small packets, a condition which is often observed. This 823 level of performance is necessary to meet Differentiated Services 824 [RFC2475] requirements for without it, packets are lost prior to 825 inspection of the IP DSCP field [RFC2474] or MPLS TC field [RFC5462]. 827 With adequate on-chip buffers before the packet decision engine, an 828 LSR can absorb a long sequence of short packets. Even if the output 829 is slowed to the point where light congestion occurs, the packets, 830 having cleared the decision process, can make use of larger VOQ or 831 output side buffers and be dealt with according to configured QoS 832 treatment, rather than dropped completely at random. 834 These on-chip buffers need not contribute significant delay since 835 they are only used when the packet decision engine is unable to keep 836 up, not in response to congestion, plus these buffers are quite 837 small. For example, an on-chip buffer capable of handling 4K packets 838 of 64 bytes in length, or 256KB, corresponds to 2 msec on a 10 Mb/s 839 link and 0.2 usec on a 100 Gb/s link. If the packet decision engine 840 is capable of handling packets at 90% of the full rate for small 841 packets, then the maximum added delay is 0.2 msec and 20 nsec 842 respectively, and this delay only applies if a 4K burst of short 843 packets occurs. When no burst of short packets was being processed, 844 no delay is added. 846 Packet rate requirements apply regardless of which network tier 847 equipment is deployed in. Whether deployed in the network core or 848 near the network edges, one of the two conditions MUST be met: 850 1. Packets must be processed at full line rate with minimum sized 851 packets. -OR- 853 2. Packets must be processed at a rate well under generally accepted 854 average packet sizes, with sufficient buffering prior to the 855 packet decision engine to accommodate long bursts of small 856 packets. 858 2.4. MPLS Multipath Techniques 860 In any large provider, service providers and content providers, hash 861 based multipath techniques are used in the core and in the edge. In 862 many of these providers hash based multipath is also used in the 863 larger metro networks. 865 The most common multipath techniques are ECMP applied at the IP 866 forwarding level, Ethernet LAG with inspection of the IP payload, and 867 multipath on links carrying both IP and MPLS, where the IP header is 868 inspected below the MPLS label stack. In most core networks, the 869 vast majority of traffic is MPLS encapsulated. 871 In order to support an adequately balanced load distribution across 872 multiple links, IP header information must be used. Common practice 873 today is to reinspect the IP headers at each LSR and use the label 874 stack and IP header information in a hash performed at each LSR. 875 Further details are provided in Section 2.4.5. 877 The use of this technique is so ubiquitous in provider networks that 878 lack of support for multipath makes any product unsuitable for use in 879 large core networks. This will continue to be the case in the near 880 future, even as deployment of MPLS entropy label begins to relax the 881 core LSR multipath performance requirements given the existing 882 deployed base of edge equipment without the ability to add an entropy 883 label. 885 A generation of edge equipment supporting the ability to add an MPLS 886 entropy label is needed before the performance requirements for core 887 LSR can be relaxed. However, it is likely that two generations of 888 deployment in the future will allow core LSR to support full packet 889 rate only when a relatively small number of MPLS labels need to be 890 inspected before hashing. For now, don't count on it. 892 Common practice today is to reinspect the packet at each LSR and 893 information from the packet combined with a hash seed that is 894 selected by each LSR. Where flow labels or entropy labels are used, 895 a hash seed must be used when creating these labels. 897 2.4.1. Pseudowire Control Word 899 Within the core of a network some form of multipath is almost certain 900 to be used. Multipath techniques deployed today are likely to be 901 looking beneath the label stack for an opportunity to hash on IP 902 addresses. 904 A pseudowire encapsulated at a network edge must have a means to 905 prevent reordering within the core if the pseudowire will be crossing 906 a network core, or any part of a network topology where multipath is 907 used (see [RFC4385] and [RFC4928]). 909 Not supporting the ability to encapsulate a pseudowire with a control 910 word may lock a product out from consideration. A pseudowire 911 capability without control word support might be sufficient for 912 applications that are strictly both intra-metro and low bandwidth. 913 However a provider with other applications will very likely not 914 tolerate having equipment which can only support a subset of their 915 pseudowire needs. 917 2.4.2. Large Microflows 919 Where multipath makes use of a simple hash and simple load balance 920 such as modulo or other fixed allocation (see Section 2.4) the 921 presence of large microflows that each consumes 10% of the capacity 922 of a component link of a potentially congested composite link, one 923 such microflow can upset the traffic balance and more than one can in 924 effect reduce the effective capacity of the entire composite link by 925 more than 10%. 927 When even a very small number of large microflows are present, there 928 is a significant probability that more than one of these large 929 microflows could fall on the same component link. If the traffic 930 contribution from large microflows is small, the probability for 931 three or more large microflows on the same component link drops 932 significantly. Therefore in a network where a significant number of 933 parallel 10 Gb/s links exists, even a 1 Gb/s pseudowire or other 934 large microflow that could not otherwise be subdivided into smaller 935 flows should carry a flow label or entropy label if possible. 937 Active management of the hash space to better accommodate large 938 microflows has been implemented and deployed in the past, however 939 such techniques are out of scope for this document. 941 2.4.3. Pseudowire Flow Label 943 Unlike a pseudowire control word, a pseudowire flow label [RFC6391], 944 is required only for relatively large capacity pseudowires. There 945 are many cases where a pseudowire flow label makes sense. Any 946 service such as a VPN which carries IP traffic within a pseudowire 947 can make use of a pseudowire flow label. 949 Any pseudowire carried over MPLS which makes use of the pseudowire 950 control word and does not carry a flow label is in effect a single 951 microflow (in [RFC2475] terms). 953 2.4.4. MPLS Entropy Label 955 The MPLS entropy label simplifies flow group identification [RFC6790] 956 in the network core. Prior to the MPLS entropy label core LSR needed 957 to inspect the entire label stack and often the IP headers to provide 958 an adequate distribution of traffic when using multipath techniques 959 (see Section 2.4.5). With the use of MPLS entropy label, a hash can 960 be performed closer to network edges, placed in the label stack, and 961 used within the network core. 963 The MPLS entropy label is capable of avoiding full label stack and 964 payload inspection within the core where performance levels are most 965 difficult to achieve (see Section 2.3). The label stack inspection 966 can be terminated as soon as the first entropy label is encountered, 967 which is generally after a small number of labels are inspected. 969 In order to provide these benefits in the core, LSR closer to the 970 edge must be capable of adding an entropy label. This support may 971 not be required in the access tier, the tier closest to the customer, 972 but is likely to be required in the edge or the border to the network 973 core. LSR peering with external networks will also need to be able 974 to add an entropy label. 976 2.4.5. Fields Used for Multipath 978 The most common multipath techniques are based on a hash over a set 979 of fields. Regardless of whether a hash is used or some other method 980 is used, the there is a limited set of fields which can safely be 981 used for multipath. 983 2.4.5.1. MPLS Fields in Multipath 985 If the "outer" or "first" layer of encapsulation is MPLS, then label 986 stack entries are used in the hash. Within a finite amount of time 987 (and for small packets arriving at high speed that time can quite 988 limited) only a finite number of label entries can be inspected. 989 Pipelined or parallel architectures improve this, but the limit is 990 still finite. 992 The following guidelines are provided for use of MPLS fields in 993 multipath load balancing. 995 1. Only the 20 bit label field SHOULD be used. The TTL field SHOULD 996 NOT be used. The S bit MUST NOT be used. The TC field (formerly 997 EXP) MUST NOT be used. See below this list for reasons. 999 2. If an ELI label is found, then if the LSR supports entropy label, 1000 the EL label field in the next label entry (the EL) SHOULD be 1001 used and label entries below that label SHOULD NOT be used and 1002 the MPLS payload SHOULD NOT be used. See below this list for 1003 reasons. 1005 3. Reserved labels (label values 0-15) MUST NOT be used. In 1006 particular, GAL and RA MUST NOT be used so that OAM traffic 1007 follows the same path as payload packets with the same label 1008 stack. 1010 4. The most entropy is generally found in the label stack entries 1011 near the bottom of the label stack (innermost label, closest to 1012 S=1 bit). If the entire label stack cannot be used (or entire 1013 stack up to an EL), then it is better to use as many labels as 1014 possible closest to the bottom of stack. 1016 5. If no ELI is encountered, and the first nibble of payload 1017 contains a 4 (IPv4) or 6 (IPv6), an implementation SHOULD support 1018 the ability to interpret the payload as IPv4 or IPv6 and extract 1019 and use appropriate fields from the IP headers. This feature is 1020 considered a hard requirement by many service providers. If 1021 supported, there MUST be a way to disable it (if, for example, PW 1022 without CW are used). This ability to disable this feature is 1023 considered a hard requirement by many service providers. 1025 Therefore an implementation has a very strong incentive to 1026 support both options. 1028 6. A label which is popped at egress (UHP pop) SHOULD NOT be used. 1029 A label which is popped at the penultimate hop (PHP pop) SHOULD 1030 be used. 1032 Apparently some chips have made use of the TC (formerly EXP) bits as 1033 a source of entropy. This is very harmful since it will reorder 1034 Assured Forwarding (AF) traffic [RFC2597] when a subset does not 1035 conform to the configured rates and is remarked but not dropped at a 1036 prior LSR. Traffic which uses MPLS ECN [RFC5129] can also be 1037 reordered if TC is used for entropy. Therefore, as stated in the 1038 guidelines above, the TC field (formerly EXP) MUST NOT be used in 1039 multipath load balancing as it violates Differentiated Services 1040 Ordered Aggregate (OA) requirements in these two instances. 1042 Use of the MPLS label entry S bit would result in putting OAM traffic 1043 on a different path if the addition of a GAL at the bottom of stack 1044 removed the S bit from the prior label. 1046 If an ELI label is found, then if the LSR supports entropy label, the 1047 EL label field in the next label entry (the EL) SHOULD be used and 1048 the search for additional entropy within the packet SHOULD be 1049 terminated. Failure to terminate the search will impact client 1050 MPLS-TP LSP carried within server MPLS LSP. A network operator has 1051 the option to use administrative attributes as a means to identify 1052 LSR which do not terminate the entropy search at the first EL. 1053 Administrative attributes are defined in [RFC3209]. Some 1054 configuration is required to support this. 1056 If the label removed by a PHP pop is not used, then for any PW for 1057 which CW is used, there is no basis for multipath load split. In 1058 some networks it is infeasible to put all PW traffic on one component 1059 link. Any PW which does not use CW will be improperly split 1060 regardless of whether the label removed by a PHP pop is used. 1061 Therefore the PHP pop label SHOULD be used as recommended above. 1063 2.4.5.2. IP Fields in Multipath 1065 Inspecting the IP payload provides the most entropy in provider 1066 networks. The practice of looking past the bottom of stack label for 1067 an IP payload is well accepted and documented in [RFC4928] and in 1068 other RFCs. 1070 Where IP is mentioned in the document, both IPv4 and IPv6 apply. All 1071 LSRs MUST fully support IPv6. 1073 When information in the IP header is used, the following guidelines 1074 apply: 1076 1. Both the IP source address and IP destination address SHOULD be 1077 used. There MAY be an option to reverse the order of these 1078 addresses, improving the ability to provide symmetric paths in 1079 some cases. Many service providers require that both addresses 1080 be used. 1082 2. Implementations SHOULD allow inspection of the IP protocol field 1083 and use of the UDP or TCP port numbers. For many service 1084 providers this feature is considered mandatory, particularly for 1085 enterprise, data center, or edge equipment. If this feature is 1086 provided, it SHOULD be possible to disable use of TCP and UDP 1087 ports. Many service providers consider it a hard requirement 1088 that use of UDP and TCP ports can be disabled. Therefore there 1089 is a stong incentive for implementations to provide both options. 1091 3. Equipment suppliers MUST NOT make assumptions that because the IP 1092 version field is equal to 4 (an IPv4 packet) that the IP protocol 1093 will either be TCP (IP protocol 6) or UDP (IP protocol 17) and 1094 blindly fetch the data at the offset where the TCP or UDP ports 1095 would be found. With IPv6, TCP and UDP port numbers are not at 1096 fixed offsets. With IPv4 packets carrying IP options, TCP and 1097 UDP port numbers are not at fixed offsets. 1099 4. The IPv6 header flow field SHOULD be used. This is the explicit 1100 purpose of the IPv6 flow field, however observed flow fields 1101 rarely contains a non-zero value. Some uses of the flow field 1102 have been defined such as [RFC6438]. In the absense of MPLS 1103 encapsulation, the IPv6 flow field can serve a role equivalent to 1104 entropy label. 1106 5. Support other protocols that share a common Layer-4 header such 1107 as RTP, UDP-lite, SCTP and DCCP SHOULD be provided, particularly 1108 for edge or access equipment where additional entropy may be 1109 needed. Equipment SHOULD also use RTP, UDP-lite, SCTP and DCCP 1110 headers when creating an entropy label. 1112 6. The following IP header fields should not or must not be used: 1114 A. Similar to avoiding TC in MPLS, the IP DSCP, and ECN bits 1115 MUST NOT be used. 1117 B. The IPv4 TTL or IPv6 Hop Count SHOULD NOT be used. 1119 C. Note that the IP TOS field was deprecated ([RFC0791] was 1120 updated by [RFC2474]). No part of the IP DSCP field can be 1121 used (formerly IP PREC and IP TOS bits). 1123 7. Some IP encapsulations support tunneling, such as IP-in-IP, GRE, 1124 L2TPv3, and IPSEC. These provide a greater source of entropy 1125 which some provider networks carrying large amounts of tunneled 1126 traffic may need. The use of tunneling header information is out 1127 of scope for this document. 1129 This document makes the following recommendations. These 1130 recommendations are not required to claim compliance to any existing 1131 RFC therefore implementors are free to ignore them, but due to 1132 service provider requirements may be doing so at their own peril. 1133 The use of IP addresses MUST be supported and TCP and UDP ports 1134 (conditional on the protocol field and properly located) MUST be 1135 supported. The ability to disable use of UDP and TCP ports MUST be 1136 available. Though potentially very useful in some networks, it is 1137 uncommon to support using payloads of tunneling protocols carried 1138 over IP. Though the use of tunneling protocol header information is 1139 out of scope for this document, it is not discouraged. 1141 2.4.5.3. Fields Used in Flow Label 1143 The ingress to a pseudowire (PW) can extract information from the 1144 payload being encapsulated to create a flow label. [RFC6391] 1145 references IP carried in Ethernet as an example. The Native Service 1146 Processing (NSP) function defined in [RFC3985] differs with 1147 pseudowire type. It is in the NSP function where information for a 1148 specific type of PW can be extracted for use in a flow label. Which 1149 fields to use for any given PW NSP is out of scope for this document. 1151 2.4.5.4. Fields Used in Entropy Label 1153 An entropy label is added at the ingress to an LSP. The payload 1154 being encapsulated is most often MPLS, a PW, or IP. The payload type 1155 is identified by the layer-2 encapsulation (Ethernet, GFP, POS, etc). 1157 If the payload is MPLS, then the information used to create an 1158 entropy label is the same information used for local load balancing 1159 (see Section 2.4.5.1). This information MUST be extracted for use in 1160 generating an entropy label even if the LSR local egress interface is 1161 not a multipath. 1163 Of the non-MPLS payload types, only payloads that are forwarded are 1164 of interest. For example, ARP is not forwarded and CNLP (used only 1165 for ISIS) is not forwarded. 1167 The non-MPLS payload type of greatest interest are IPv4 and IPv6. 1168 The guidelines in Section 2.4.5.2 apply to fields used to create and 1169 entropy label. 1171 The IP tunneling protocols mentioned in Section 2.4.5.2 may be more 1172 applicable to generation of an entropy label at edge or access where 1173 deep packet inspection is practical due to lower interface speeds 1174 than in the core where deep packet inspection may be impractical. 1176 2.5. MPLS-TP and UHP 1178 MPLS-TP introduces forwarding demands that will be extremely 1179 difficult to meet in a core network. Most troublesome is the 1180 requirement for Ultimate Hop Popping (UHP, the opposite of 1181 Penultimate Hop Popping or PHP). Using UHP opens the possibility of 1182 one or more MPLS pop operation plus an MPLS swap operation for each 1183 packet. The potential for multiple lookups and multiple counter 1184 instances per packet exists. 1186 As networks grow and tunneling of LDP LSPs into RSVP-TE LSPs is used, 1187 and/or RSVP-TE hierarchy is used, the requirement to perform one or 1188 two or more MPLS pop operations plus a MPLS swap operation (and 1189 possibly a push or two) increases. If MPLS-TP LM (link monitoring) 1190 OAM is enabled at each layer, then a packet and byte count MUST be 1191 maintained for each pop and swap operation so as to offer OAM for 1192 each layer. 1194 2.6. Local Delivery of Packets 1196 There are a number of situations in which packets are destined to a 1197 local address or where a return packet must be generated. There is a 1198 need to mitigate the potential for outage as a result of either 1199 attacks on network infrastructure, or in some cases unintentional 1200 misconfiguration resulting in processor overload. Some hardware 1201 assistance is needed for all traffic destined to the general purpose 1202 CPU that is used in MPLS control protocol processing or network 1203 management protocol processing and in most cases to other general 1204 purpose CPUs residing on an LSR. This is due to the ease of 1205 overwhelming such a processor with traffic arriving on LSR high speed 1206 interfaces, whether the traffic is malicious or not. 1208 Denial of service (DoS) protection is an area requiring hardware 1209 support that is often overlooked or inadequately considered. 1210 Hardware assist is also needed for OAM, particularly the more 1211 demanding MPLS-TP OAM. 1213 2.6.1. DoS Protection 1215 Modern equipment supports a number of control plane and management 1216 plane protocols. Generally no single means of protecting network 1217 equipment from denial of service (DoS) attacks is sufficient, 1218 particularly for high speed interfaces. This problem is not specific 1219 to MPLS, but is a topic that cannot be ignored when implementing or 1220 evaluating MPLS implementations. 1222 Two types of protections are often cited as primary means of 1223 protecting against attacks of all kinds. 1225 Isolated Control/Management Traffic 1226 Control and Management traffic can be carried out-of-band (OOB), 1227 meaning not intermixed with payload. For MPLS use of G-ACh and 1228 GAL to carry control and management traffic provides a means of 1229 isolation from potentially malicious payload. Used alone, the 1230 compromise of a single node, including a small computer at a 1231 network operations center, could compromise an entire network. 1232 Implementations which send all G-ACh/GAL traffic directly to a 1233 routing engine CPU are subject to DoS attack as a result of such 1234 a compromise. 1236 Cryptographic Authentication 1237 Cryptographic authentication can very effectively prevent 1238 malicious injection of control or management traffic. 1239 Cryptographic authentication can is some circumstances be subject 1240 to DoS attack by overwhelming the capacity of the decryption with 1241 a high volume of malicious traffic. For very low speed 1242 interfaces, cryptographic authentication can be performed by the 1243 general purpose CPU used as a routing engine. For all other 1244 cases, cryptographic hardware may be needed. For very high speed 1245 interfaces, even cryptographic hardware can be overwhelmed. 1247 Some control and management protocols are often carried with payload 1248 traffic. This is commonly the case with BGP, T-LDP, and SNMP. It is 1249 often the case with RSVP-TE. Even when carried over G-ACh/GAL 1250 additional measures can reduce the potential for a minor breach to be 1251 leveraged to a full network attack. 1253 Some of the additional protections are supported by hardware packet 1254 filtering. 1256 GTSM 1257 [RFC5082] defines a mechanism that uses the IPv4 TTL or IPv6 Hop 1258 Limit fields to insure control traffic that can only originate 1259 from an immediate neighbor is not forged and originating from a 1260 distant source. GTSM can be applies to many control protocols 1261 which are routable, for example LDP [RFC6720]. 1263 IP Filtering 1264 At the very minimum, packet filtering plus classification and use 1265 of multiple queues supporting rate limiting is needed for traffic 1266 that could potentially be sent to a general purpose CPU used as a 1267 routing engine. The first level of filtering only allows 1268 connections to be initiated from specific IP prefixes to specific 1269 destination ports and then preferably passes traffic directly to 1270 a cryptographic engine and/or rate limits. The second level of 1271 filtering passes connected traffic, such as TCP connections 1272 having received at least one authenticated SYN or having been 1273 locally initiated. The second level of filtering only passes 1274 traffic to specific address and port pairs to be checked for 1275 cryptographic authentication. 1277 The cryptographic authentication is generally the last resort in DoS 1278 attack mitigation. If a packet must be first sent to a general 1279 purpose CPU, then sent to a cryptographic engine, a DoS attack is 1280 possible on high speed interfaces. Only where hardware can identify 1281 a signature and the portion of packet covered by the signature is 1282 cryptographic authentication highly beneficial in protecting against 1283 DoS attacks. 1285 For chips supporting multiple 100 Gb/s interfaces, only a very large 1286 number of parallel cryptographic engines can provide the processing 1287 capacity to handle a large scale DoS or distributed DoS (DDoS) 1288 attack. For many forwarding chips this much processing power 1289 requires significant chip real estate and power, and therefore 1290 reduces system space and power density. For this reason, 1291 cryptographic authentication is not considered a viable first line of 1292 defense. 1294 For some networks the first line of defense is some means of 1295 supporting OOB control and management traffic. In the past this OOB 1296 channel migh make use of overhead bits in SONET or OTN or a dedicated 1297 DWDM wavelength. G-ACh and GAL provide an alternative OOB mechanism 1298 which is independent of underlying layers. In other networks, 1299 including most IP/MPLS networks, perimeter filtering serves a similar 1300 purpose, though less effective without extreme vigilance. 1302 A second line of defense is filtering, including GTSM. For protocols 1303 such as EBGP, GTSM and other filtering is often the first line of 1304 defense. Cryptographic authentication is usually the last line of 1305 defense and insufficient by itself to mitigate DoS or DDoS attacks. 1307 2.6.2. MPLS OAM 1309 [RFC4377] defines requirements for MPLS OAM that predate MPLS-TP. 1310 [RFC4379] defines what is commonly referred to as LSP Ping and LSP 1311 Traceroute. [RFC4379] is updated by [RFC6424] supporting MPLS 1312 tunnels and stitched LSP and P2MP LSP. [RFC4379] is updated by 1313 [RFC6425] supporting P2MP LSP. [RFC4379] is updated by [RFC6426] to 1314 support MPLS-TP connectivity verification (CV) and route tracing. 1316 [RFC4950] extends the ICMP format to support TTL expiration that may 1317 occur when using IP traceroute within an MPLS tunnel. The ICMP 1318 message generation can be implemented in forwarding hardware, but if 1319 sent to a general purpose CPU must be rate limited to avoid a 1320 potential denial or service (DoS) attack. 1322 [RFC5880] defines Bidirectional Forwarding Detection (BFD), a 1323 protocol intended to detect faults in the bidirectional path between 1324 two forwarding engines. [RFC5884] and [RFC5885] define BFD for MPLS. 1325 BFD can provide failure detection on any kind of path between 1326 systems, including direct physical links, virtual circuits, tunnels, 1327 MPLS Label Switched Paths (LSPs), multihop routed paths, and 1328 unidirectional links as long as there is some return path. 1330 The processing requirements for BFD are less than for LSP Ping, 1331 making BFD somewhat better suited for relatively high rate proactive 1332 monitoring. BFD does not verify that the data plane against the 1333 control plane, where LSP Ping does. LSP Ping somewhat better suited 1334 for on-demand monitoring including relatively low rate periodic 1335 verification of data plane and as a diagnostic tool. 1337 Hardware assistance is often provided for BFD response where BFD 1338 setup or parameter change is not involved and may be necessary for 1339 relatively high rate proactive monitoring. If both BFD and LSP Ping 1340 are recognized in filtering prior to passing traffic to a general 1341 purpose CPU, appropriate DoS protection can be applied (see 1342 Section 2.6.1). Failure to recognize BFD and LSP Ping and at least 1343 rate limit creates the potential for misconfiguration to cause 1344 outages rather than cause errors in the misconfigured OAM. 1346 2.6.3. Pseudowire OAM 1348 Pseudowire OAM makes use of the control channel provided by Virtual 1349 Circuit Connectivity Verification (VCCV) [RFC5085]. VCCV makes use 1350 of the Pseudowire Control Word. BFD support over VCCV is defined by 1351 [RFC5885]. [RFC5885] is updated by [RFC6478] in support of static 1352 pseudowires. [RFC4379] is updated by [RFC6829] supporting LSP Ping 1353 for Pseudowire FEC advertised over IPv6. 1355 G-ACh/GAL (defined in [RFC5586]) is the preferred MPLS-TP OAM control 1356 channel and applies to any MPLS-TP end points, including Pseudowire. 1357 See Section 2.6.4 for an overview of MPLS-TP OAM. 1359 2.6.4. MPLS-TP OAM 1361 [RFC6669] summarizes the MPLS-TP OAM toolset, the set of protocols 1362 supporting the MPLS-TP OAM requirements specified in [RFC5860] and 1363 supported by the MPLS-TP OAM framework defined in [RFC6371]. 1365 The MPLS-TP OAM toolset includes: 1367 CC-CV 1368 [RFC6428] defines BFD extensions to support proactive 1369 Connectivity Check and Connectivity Verification (CC-CV) 1370 applications. [RFC6426] provides LSP ping extensions that are 1371 used to implement on-demand connectivity verification. 1373 RDI 1374 Remote Defect Indication (RDI) is triggered by failure of 1375 proactive CC-CV, which is BFD based. For fast RDI initiation, 1376 RDI SHOULD be initiated and handled by hardware if BFD is handled 1377 in forwarding hardware. [RFC6428] provides an extension for BFD 1378 that includes the RDI indication in the BFD format and a 1379 specification of how this indication is to be used. 1381 Route Tracing 1382 [RFC6426] specifies that the LSP ping enhancements for MPLS-TP 1383 on-demand connectivity verification include information on the 1384 use of LSP ping for route tracing of an MPLS-TP path. 1386 Alarm Reporting 1387 [RFC6427] describes the details of a new protocol supporting 1388 Alarm Indication Signal (AIS), Link Down Indication, and fault 1389 management. Failure to support this functionality in forwarding 1390 hardware can potentially result in failure to meet protection 1391 recovery time requirements and is therefore strongly recommended. 1393 Lock Instruct 1394 Lock instruct is initiated on-demand and therefore need not be 1395 implemented in forwarding hardware. [RFC6435] defines a lock 1396 instruct protocol. 1398 Lock Reporting 1399 [RFC6427] covers lock reporting. Lock reporting need not be 1400 implemented in forwarding hardware. 1402 Diagnostic 1403 [RFC6435] defines protocol support for loopback. Loopback 1404 initiation is on-demand and therefore need not be implemented in 1405 forwarding hardware. Loopback of packet traffic SHOULD be 1406 implemented in forwarding hardware on high speed interfaces. 1408 Packet Loss and Delay Measurement 1409 [RFC6374] and [RFC6375] define a protocol and profile for packet 1410 loss measurement (LM) and delay measurement (DM). LM requires a 1411 very accurate capture and insertion of packet and byte counters 1412 when a packet is transmitted and capture of packet and byte 1413 counters when a packet is received. This capture and insertion 1414 MUST be implemented in forwarding hardware for LM OAM if high 1415 accuracy is needed. DM requires very accurate capture and 1416 insertion of a timestamp on transmission and capture of timestamp 1417 when a packet is received. This timestamp capture and insertion 1418 MUST be implemented in forwarding hardware for DM OAM if high 1419 accuracy is needed. 1421 See Section 2.6.2 for discussion of hardware support necessary for 1422 BFD and LSP Ping. 1424 CC-CV and alarm reporting is tied to protection and therefore SHOULD 1425 be supported in forwarding hardware in order to provide protection 1426 for a large number of affected LSP within target response intervals. 1427 Since CC-CV is supported by BFD, for MPLS-TP providing hardware 1428 assistance for BFD processing helps insure that protection recovery 1429 time requirements can be met even for faults affecting a large number 1430 of LSP. 1432 2.6.5. MPLS OAM and Layer-2 OAM Interworking 1434 [RFC6670] provides the reasons for selecting a single MPLS-TP OAM 1435 solution and examines the consequences were ITU-T to develop a second 1436 OAM solution that is based on Ethernet encodings and mechanisms. 1438 [RFC6310] and [I-D.ietf-pwe3-mpls-eth-oam-iwk] specifies the mapping 1439 of defect states between many types of hardware Attachment Circuits 1440 (ACs) and associated Pseudowires (PWs). This functionality SHOULD be 1441 supported in forwarding hardware. 1443 It is beneficial if an MPLS OAM implementation can interwork with the 1444 underlying server layer and provide a means to interwork with a 1445 client layer. For example, [RFC6427] specifies an inter-layer 1446 propogation of AIS and LDI from MPLS server layer to client MPLS 1447 layers. Where the server layer is a Layer-2, such as Ethernet, PPP 1448 over SONET/SDH, or GFP over OTN, interwork among layers is also 1449 beneficial. For high speed interfaces, supporting this interworking 1450 in forwarding hardware helps insure that protection based on this 1451 interworking can meet recovery time requirements even for faults 1452 affecting a large number of LSP. 1454 2.6.6. Extent of OAM Support by Hardware 1456 Where certain requirements must be met, such as relatively high CC-CV 1457 rates and a large number of interfaces, or strict protection recovery 1458 time requirements and a moderate number of affected LSP, some OAM 1459 functionality must be supported by forwarding hardware. In other 1460 cases, such as highly accurate LM and DM OAM or strict protection 1461 recovery time requirements with a large number of affected LSP, OAM 1462 functionality must be entirely implemented in forwarding hardware. 1464 Where possible, implementation in forwarding hardware should be in 1465 programmable hardware such that if standards are later changed or 1466 extended these changes are likely to be accommodated with hardware 1467 reprogramming rather than replacement. 1469 For some functionality there is a strong case for an implementation 1470 in dedicated forwarding hardware. Examples include packet and byte 1471 counters needed for LM OAM as well as needed for management 1472 protocols. Similarly the capture and insertion of packet and byte 1473 counts or timestamps needed for transmitted LM or DM or time 1474 synchronization packets MUST be implemented in forwarding hardware if 1475 high accuracy is required. 1477 For some functions there is a strong case to provide limited support 1478 in forwarding hardware but may make use of an external general 1479 purpose processor if performance criteria can be met. For example 1480 origination of RDI triggered by CC-CV, response to RDI, and PSC 1481 functionality may be supported by hardware, but expansion to a large 1482 number of client LSP and transmission of AIS or RDI to the client LSP 1483 may occur in a general purpose processor. Some forwarding hardware 1484 supports one or more on-chip general purpose processors which may be 1485 well suited for such a role. 1487 The customer (system supplier or provider) should not dictate design, 1488 but should independently validate target functionality and 1489 performance. However, it is not uncommon for service providers and 1490 system implementors to insist on reviewing design details (under NDA) 1491 due to past experiences with suppliers and to reject suppliers who 1492 are unwilling to provide details. 1494 2.7. Number and Size of Flows 1496 Service provider networks may carry up to hundreds of millions of 1497 flows on 10 Gb/s links. Most flows are very short lived, many under 1498 a second. A subset of the flows are low capacity and somewhat long 1499 lived. When Internet traffic dominates capacity a very small subset 1500 of flows are high capacity and/or very long lived. 1502 Two types of limitations with regard to number and size of flows have 1503 been observed. 1505 1. Some hardware cannot handle some very large flows because of 1506 internal paths which are limited, such as per packet backplane 1507 paths or paths internal or external to chips such as buffer 1508 memory paths. Such designs can handle aggregates of smaller 1509 flows. Some hardware with acknowledged limitations has been 1510 successfully deployed but may be increasingly problematic if the 1511 capacity of large microflows in deployed networks continues to 1512 grow. 1514 2. Some hardware approaches cannot handle a large number of flows, 1515 or a large number of large flows due to attempting to count per 1516 flow, rather than deal with aggregates of flows. Hash techniques 1517 scale with regard to number of flows due to a fixed hash size 1518 with many flows falling into the same hash bucket. Techniques 1519 that identify individual flows have been implemented but have 1520 never successfully deployed for Internet traffic. 1522 3. Questions for Suppliers 1524 The following questions should be asked of a supplier. These 1525 questions are grouped into broad categories. The questions 1526 themselves are intended to be an open ended question to the supplier. 1527 The tests in Section 4 are intended to verify whether the supplier 1528 disclosed any compliance or performance limitations completely and 1529 accurately. 1531 3.1. Basic Compliance 1533 Q#1 Can the implementation forward packets with an arbitrarily 1534 large stack depth? What limitations exist, and under what 1535 circumstances do further limitations come into play (such as 1536 high packet rate or specific features enabled or specific types 1537 of packet processing)? See Section 2.1. 1539 Q#2 Is the entire set of basic MPLS functionality described in 1540 Section 2.1 supported? 1542 Q#3 Are the set of MPLS reserved labels handled correctly and with 1543 adequate performance? See Section 2.1.1. 1545 Q#4 Are mappings of label value and TC to PHB handled correctly, 1546 including RFC3270 L-LSP mappings and RFC4124 CT mappings to 1547 PHB? See Section 2.1.2. 1549 Q#5 Is time synchronization adequately supported in forwarding 1550 hardware? 1552 A. Are both PTP and NTP formats supported? 1554 B. Is the accuracy of timestamp insertion and incoming 1555 stamping sufficient? 1557 See Section 2.1.3. 1559 Q#6 Is link bundling supported? 1561 A. Can LSP be pinned to specific components? 1563 B. Is the "all-ones" component link supported? 1565 See Section 2.1.5. 1567 Q#7 Is MPLS hierarchy supported? 1569 A. Are both PHP and UHP supported? What limitations exist on 1570 the number of pop operations with UHP? 1572 B. Are the pipe, short-pipe, and uniform models supported? 1573 Are TTL and TC values updated correctly at egress where 1574 applicable? 1576 See Section 2.1.6 1578 Q#8 Are pseudowire sequence numbers handled correctly? See 1579 Section 2.1.8.1. 1581 Q#9 Is VPN LER functionality handled correctly and without 1582 performance issues? See Section 2.1.9. 1584 Q#10 Is MPLS multicast (P2MP and MP2MP) handled correctly? 1586 A. Are packets dropped on uncongested outputs if some outputs 1587 are congested? 1589 B. Is performance limited in high fanout situations? 1591 See Section 2.2. 1593 3.2. Basic Performance 1595 Q#11 Can very small packets be forwarded at full line rate on all 1596 interfaces indefinitely? What limitations exist, and under 1597 what circumstances do further limitations come into play (such 1598 as specific features enabled or specific types of packet 1599 processing)? 1601 Q#12 Customers must decide whether to relax the prior requirement 1602 and to what extent. If the answer to the prior question 1603 indicates that limitations exist, then: 1605 A. What is the smallest packet size where full line rate 1606 forwarding can be supported? 1608 B. What is the longest burst of full rate small packets that 1609 can be supported? 1611 Specify circumstances (such as specific features enabled or 1612 specific types of packet processing) often impact these rates 1613 and burst sizes. 1615 Q#13 How many pop operations can be supported along with a swap 1616 operation at full line rate while maintaining per LSP packet 1617 and byte counts for each pop and swap? This requirement is 1618 particularly relevant for MPLS-TP. 1620 Q#14 How many label push operations can be supported. While this 1621 limitation is rarely an issue, it applies to both PHP and UHP, 1622 unlike the pop limit which applies to UHP. 1624 Q#15 For a worst case where all packets arrive on one LSP, what is 1625 the counter overflow time? Are any means provided to avoid 1626 polling all counters at short intervals? This applies to both 1627 MPLS and MPLS-TP. 1629 3.3. Multipath Capabilities and Performance 1631 Multipath capabilities and performance do not apply to MPLS-TP but 1632 apply to MPLS and apply if MPLS-TP is carried in MPLS. 1634 Q#16 How are large microflows accommodated? Is there active 1635 management of the hash space mapping to output ports? See 1636 Section 2.4.2. 1638 Q#17 How many MPLS labels can be included in a hash based on the 1639 MPLS label stack? 1641 Q#18 Is packet rate performance decreased beyond some number of 1642 labels? 1644 Q#19 Can the IP header and payload information below the MPLS stack 1645 be used in the hash? If so, which IP fields, payload types and 1646 payload fields are supported? 1648 Q#20 At what maximum MPLS label stack depth can Bottom of Stack and 1649 an IP header appear without impacting packet rate performance? 1651 Q#21 Are reserved labels excluded from the label stack hash? See 1652 Section 2.4.5.1. 1654 Q#22 How is multipath performance affected by very large flows or an 1655 extremely large number of flows, or by very short lived flows? 1656 See Section 2.7. 1658 3.4. Pseudowire Capabilities and Performance 1660 Q#23 Is the pseudowire control word supported? 1662 Q#24 What is the maximum rate of pseudowire encapsulation and 1663 decapsulation? Apply the same questions as in Base Performance 1664 for any packet based pseudowire such as IP VPN or Ethernet. 1666 Q#25 Does inclusion of a pseudowire control word impact performance? 1668 Q#26 Are flow labels supported? 1670 Q#27 If so, what fields are hashed on for the flow label for 1671 different types of pseudowires? 1673 Q#28 Does inclusion of a flow label impact performance? 1675 3.5. Entropy Label Support and Performance 1677 Q#29 Can an entropy label be added when acting as in ingress LER and 1678 can it be removed when acting as an egress LER? 1680 Q#30 If so, what fields are hashed on for the entropy label? 1682 Q#31 Does adding or removing an entropy label impact packet rate 1683 performance? 1685 Q#32 Can an entropy label be detected in the label stack, used in 1686 the hash, and properly terminate the search for further 1687 information to hash on? 1689 Q#33 Does using an entropy label have any negative impact on 1690 performance? It should have no impact or a positive impact. 1692 3.6. DoS Protection 1694 Q#34 For each control and management plane protocol in use, what 1695 measures are taken to provide DoS attack hardenning? 1697 Q#35 Have DoS attack tests been performed? 1699 Q#36 Can compromise of an internal computer on a management subnet 1700 be leveraged for any form of attack including DoS attack? 1702 3.7. OAM Capabilities and Performance 1704 Q#37 What OAM proactive and on-demand mechanisms are supported? 1706 Q#38 What performance limits exist under high proactive monitoring 1707 rates? 1709 Q#39 Can excessively high proactive monitoring rates impact control 1710 plane performance or cause control plane instability? 1712 Q#40 Ask the prior questions for each of the following. 1714 A. MPLS OAM 1716 B. Pseudowire OAM 1718 C. MPLS-TP OAM 1720 D. Layer-2 OAM Interworking 1722 See Section 2.6.2. 1724 4. Forwarding Compliance and Performance Testing 1726 Packet rate performance of equipment supporting a large number of 10 1727 Gb/s or 100 Gb/s links is not possible using desktop computers or 1728 workstations. The use of high end workstations as a source of test 1729 traffic was barely viable 20 years ago, but is no longer at all 1730 viable. Though custom microcode has been used on specialized router 1731 forwarding cards to serve the purpose of generating test traffic and 1732 measuring it, for the most part performance testing will require 1733 specialized test equipment. There are multiple sources of suitable 1734 equipment. 1736 The set of tests listed here do not correspond one-to-one to the set 1737 of questions in Section 3. The same categorization is used and these 1738 tests largely serve to validate answers provided to the prior 1739 questions, and can also provide answers where a supplier is unwilling 1740 to disclose compliance or performance. 1742 Performance testing is the domain of the IETF Benchmark Methodology 1743 Working Group (BMWG). Below are brief descriptions of conformance 1744 and performance tests. Some very basic tests are specified in 1745 [RFC5695] which partially cover only the basic performance test T#3. 1747 The following tests should be performed by the systems designer, or 1748 deployer, or performed by the supplier on their behalf if it is not 1749 practical for the potential customer to perform the tests directly. 1750 These tests are grouped into broad categories. 1752 The tests in Section 4.1 should be repeated under various conditions 1753 to retest basic performance when critical capabilities are enabled. 1754 Complete repetition of the performance tests enabling each capability 1755 and combinations of capabilities would be very time intensive, 1756 therefore a reduced set of performance tests can be used to gauge the 1757 impact of enabling specific capabilities. 1759 4.1. Basic Compliance 1761 T#1 Test forwarding at a high rate for packets with varying number 1762 of label entries. While packets with more than a dozen label 1763 entries are unlikely to be used in any practical scenario today, 1764 it is useful to know if limitations exists. 1766 T#2 For each of the questions listed under "Basic Compliance" in 1767 Section 3, verify the claimed compliance. For any functionality 1768 considered critical to a deployment, where applicable 1769 performance using each capability under load should be verified 1770 in addition to basic compliance. 1772 4.2. Basic Performance 1774 T#3 Test packet forwarding at full line rate with small packets. 1775 See [RFC5695]. The most likely case to fail is the smallest 1776 packet size. Also test with packet sizes in four byte 1777 increments ranging from payload sizes or 40 to 128 bytes. 1779 T#4 If the prior tests did not succeed for all packet sizes, then 1780 perform the following tests. 1782 A. Increase the packet size by 4 bytes until a size is found 1783 that can be forwarded at full rate. 1785 B. Inject bursts of consecutive small packets into a stream of 1786 larger packets. Allow some time for recovery between 1787 bursts. Increase the number of packets in the burst until 1788 packets are dropped. 1790 T#5 Send test traffic where a swap operation is required. Also set 1791 up multiple LSP carried over other LSP where the device under 1792 test (DUT) is the egress of these LSP. Create test packets such 1793 that the swap operation is performed after pop operations, 1794 increasing the number of pop operations until forwarding of 1795 small packets at full line rate can no longer be supported. 1796 Also check to see how many pop operations can be supported 1797 before the full set of counters can no longer be maintained. 1798 This requirement is particularly relevant for MPLS-TP. 1800 T#6 Send all traffic on one LSP and see if the counters become 1801 inaccurate. Often counters on silicon are much smaller than the 1802 64 bit packet and byte counters in IETF MIB. System developers 1803 should consider what counter polling rate is necessary to 1804 maintain accurate counters and whether those polling rates are 1805 practical. Relevant MIBs for MPLS are discussed in [RFC4221] 1806 and [RFC6639]. 1808 4.3. Multipath Capabilities and Performance 1810 Multipath capabilities do not apply to MPLS-TP but apply to MPLS and 1811 apply if MPLS-TP is carried in MPLS. 1813 T#7 Send traffic at a rate well exceeding the capacity of a single 1814 multipath component link, and where entropy exists only below 1815 the top of stack. If only the top label is used this test will 1816 fail immediately. 1818 T#8 Move the labels with entropy down in the stack until either the 1819 full forwarding rate can no longer be supported or most or all 1820 packets try to use the same component link. 1822 T#9 Repeat the two tests above with the entropy contained in IP 1823 headers or IP payload fields below the label stack rather than 1824 in the label stack. Test with the set of IP headers or IP 1825 payload fields considered relevant to the deployment or to the 1826 target market. 1828 T#10 Determine whether traffic that contains a pseudowire control 1829 word is interpreted as IP traffic. Information in the payload 1830 MUST NOT be used in the load balancing if the first nibble of 1831 the packet is not 4 or 6 (IPv4 or IPv6). 1833 T#11 Determine whether reserved labels are excluded from the label 1834 stack hash. They MUST be excluded. 1836 T#12 Perform testing in the presence of combinations of: 1838 A. Very large microflows. 1840 B. Relatively short lived high capacity flows. 1842 C. Extremely large numbers of flows. 1844 D. Very short lived small flows. 1846 4.4. Pseudowire Capabilities and Performance 1848 T#13 Ensure that pseudowire can be set up with a pseudowire label 1849 and pseudowire control word added at ingress and the pseudowire 1850 label and pseudowire control word removed at egress. 1852 T#14 For pseudowire that contains variable length payload packets, 1853 repeat performance tests listed under "Basic Performance" for 1854 pseudowire ingress and egress functions. 1856 T#15 Repeat pseudowire performance tests with and without a 1857 pseudowire control word. 1859 T#16 Determine whether pseudowire can be set up with a pseudowire 1860 label, flow label, and pseudowire control word added at ingress 1861 and the pseudowire label, flow label, and pseudowire control 1862 word removed at egress. 1864 T#17 Determine which payload fields are used to create the flow 1865 label and whether the set of fields and algorithm provide 1866 sufficient entropy for load balancing. 1868 T#18 Repeat pseudowire performance tests with flow labels included. 1870 4.5. Entropy Label Support and Performance 1871 T#19 Determine whether entropy labels can be added at ingress and 1872 removed at egress. 1874 T#20 Determine which fields are used to create an entropy label. 1875 Labels further down in the stack, including entropy labels 1876 further down and IP headers or IP payload fields where 1877 applicable should be used. Determine whether the set of fields 1878 and algorithm provide sufficient entropy for load balancing. 1880 T#21 Repeat performance tests under "Basic Performance" when entropy 1881 labels are used, where ingress or egress is the device under 1882 test (DUT). 1884 T#22 Determine whether an ELI is detected when acting as a midpoint 1885 LSR and whether the search for further information on which to 1886 base the load balancing is used. Information below the entropy 1887 label SHOULD NOT be used. 1889 T#23 Ensure that the entropy label indicator and entropy label (ELI 1890 and EL) are removed from the label stack during UHP and PHP 1891 operations. 1893 T#24 Insure that operations on the TC field when adding and removing 1894 entropy label are correctly carried out. If TC is changed 1895 during a swap operation, the ability to transfer that change 1896 MUST be provided. The ability to suppress the transfer of TC 1897 MUST also be provided. See "pipe", "short pipe", and "uniform" 1898 models in [RFC3443]. 1900 T#25 Repeat performance tests for midpoint LSR with entropy labels 1901 found at various label stack depths. 1903 4.6. DoS Protection 1905 T#26 Actively attack LSR under high protocol churn load and 1906 determine control plane performance impact or successful DoS 1907 under test conditions. Specifically test for the following. 1909 A. TCP SYN attack against control plane and management plane 1910 protocols using TCP, including CLI access (typically SSH 1911 protected login), NETCONF, etc. 1913 B. High traffic volume attack against control plane and 1914 management plane protocols not using TCP. 1916 C. Attacks which can be performed from a compromised 1917 management subnet computer, but not one with authentication 1918 keys. 1920 D. Attacks which can be performed from a compromised peer 1921 within the control plane (internal domain and external 1922 domain). Assume that per peering keys and per router ID 1923 keys rather than network wide keys are in use. 1925 See Section 2.6.1. 1927 4.7. OAM Capabilities and Performance 1929 T#27 Determine maximum sustainable rates of BFD traffic. If BFD 1930 requires CPU intervention, determine both maximum rates and CPU 1931 loading when multiple interfaces are active. 1933 T#28 Verify LSP Ping and LSP Traceroute capability. 1935 T#29 Determine maximum rates of MPLS-TP CC-CV traffic. If CC-CV 1936 requires CPU intervention, determine both maximum rates and CPU 1937 loading when multiple interfaces are active. 1939 T#30 Determine MPLS-TP DM precision. 1941 T#31 Determine MPLS-TP LM accuracy. 1943 T#32 Verify MPLS-TP AIS/RDI and PSC functionality, protection speed, 1944 and AIS/RDI notification speed when a large number of 1945 Management Entities (ME) must be notified with AIS/RDI. 1947 5. Acknowledgements 1949 Numerous very useful comments have been received in private email. 1950 Some of these contributions are acknowledged here, approximately in 1951 chronologic order. 1953 Paul Doolan provided a brief review resulting in a number of 1954 clarifications, most notably regarding on-chip vs. system buffering, 1955 100 Gb/s link speed assumptions in the 150 Mpps figure, and handling 1956 of large microflows. Pablo Frank reminded us of the sawtooth effect 1957 in PPS vs. packet size graphs, prompting the addition of a few 1958 paragraphs on this. Comments from Lou Berger at IETF-85 prompted the 1959 addition of Section 2.7. 1961 Valuable comments were received on the BMWG mailing list. Jay 1962 Karthik pointed out testing methodology hints that after discussion 1963 were deemed out of scope and were removed. 1965 Nabil Bitar pointed out the need to cover QoS (Differentiated 1966 Services), MPLS multicast (P2MP and MP2MP), and MPLS-TP OAM. Nabil 1967 also provided a number of clarifications to the questions and tests 1968 in Section 3 and Section 4. 1970 Gregory Mirsky and Thomas Beckhaus provided useful comments during 1971 the MPLS RT review. 1973 Tal Mizrahi provided comments that prompted clarifications regarding 1974 timestamp processing, local delivery of packets, and the need for 1975 hardware assistance in processing OAM traffic. 1977 6. IANA Considerations 1979 This memo includes no request to IANA. 1981 7. Security Considerations 1983 This document reviews forwarding behavior specified elsewhere and 1984 points out compliance and performance requirements. As such it 1985 introduces no new security requirements or concerns. 1987 Knowledge of potential performance shortcomings may serve to help new 1988 implementations avoid pitfalls. It is unlikely that such knowledge 1989 could be the basis of new denial of service as these pitfalls are 1990 already widely known in the service provider community and among 1991 leading equipment suppliers. In practice extreme data and packet 1992 rate are needed to affect existing equipment and networks that may be 1993 still vulnerable due to failure to implement adequate protection and 1994 make this type of denial of service unlikely and make undetectable 1995 denial of service of this type impossible. 1997 8. References 1999 8.1. Normative References 2001 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 2002 Requirement Levels", BCP 14, RFC 2119, March 1997. 2004 [RFC3032] Rosen, E., Tappan, D., Fedorkow, G., Rekhter, Y., 2005 Farinacci, D., Li, T., and A. Conta, "MPLS Label Stack 2006 Encoding", RFC 3032, January 2001. 2008 [RFC3209] Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V., 2009 and G. Swallow, "RSVP-TE: Extensions to RSVP for LSP 2010 Tunnels", RFC 3209, December 2001. 2012 [RFC3270] Le Faucheur, F., Wu, L., Davie, B., Davari, S., Vaananen, 2013 P., Krishnan, R., Cheval, P., and J. Heinanen, "Multi- 2014 Protocol Label Switching (MPLS) Support of Differentiated 2015 Services", RFC 3270, May 2002. 2017 [RFC3443] Agarwal, P. and B. Akyol, "Time To Live (TTL) Processing 2018 in Multi-Protocol Label Switching (MPLS) Networks", 2019 RFC 3443, January 2003. 2021 [RFC4090] Pan, P., Swallow, G., and A. Atlas, "Fast Reroute 2022 Extensions to RSVP-TE for LSP Tunnels", RFC 4090, 2023 May 2005. 2025 [RFC4182] Rosen, E., "Removing a Restriction on the use of MPLS 2026 Explicit NULL", RFC 4182, September 2005. 2028 [RFC4201] Kompella, K., Rekhter, Y., and L. Berger, "Link Bundling 2029 in MPLS Traffic Engineering (TE)", RFC 4201, October 2005. 2031 [RFC4385] Bryant, S., Swallow, G., Martini, L., and D. McPherson, 2032 "Pseudowire Emulation Edge-to-Edge (PWE3) Control Word for 2033 Use over an MPLS PSN", RFC 4385, February 2006. 2035 [RFC5129] Davie, B., Briscoe, B., and J. Tay, "Explicit Congestion 2036 Marking in MPLS", RFC 5129, January 2008. 2038 [RFC5586] Bocci, M., Vigoureux, M., and S. Bryant, "MPLS Generic 2039 Associated Channel", RFC 5586, June 2009. 2041 [RFC6391] Bryant, S., Filsfils, C., Drafz, U., Kompella, V., Regan, 2042 J., and S. Amante, "Flow-Aware Transport of Pseudowires 2043 over an MPLS Packet Switched Network", RFC 6391, 2044 November 2011. 2046 [RFC6790] Kompella, K., Drake, J., Amante, S., Henderickx, W., and 2047 L. Yong, "The Use of Entropy Labels in MPLS Forwarding", 2048 RFC 6790, November 2012. 2050 8.2. Informative References 2052 [ACK-compression] 2053 "Observations and Dynamics of a Congestion Control 2054 Algorithm: The Effects of Two-Way Traffic", Proc. ACM 2055 SIGCOMM, ACM Computer Communications Review (CCR) Vol 21, 2056 No 4, 1991, pp.133-147., 1991. 2058 [ATM-EPD-and-PPD] 2059 "Dynamics of TCP Traffic over ATM Networks", IEEE Journal 2060 of Special Areas of Communication Vol 13 No 4, May 1995, 2061 pp. 633-641., May 1995. 2063 [I-D.ietf-pwe3-mpls-eth-oam-iwk] 2064 Mohan, D., Bitar, N., and A. Sajassi, "MPLS and Ethernet 2065 OAM Interworking", draft-ietf-pwe3-mpls-eth-oam-iwk-07 2066 (work in progress), January 2013. 2068 [I-D.ietf-tictoc-1588overmpls] 2069 Davari, S., Oren, A., Bhatia, M., Roberts, P., and L. 2070 Montini, "Transporting Timing messages over MPLS 2071 Networks", draft-ietf-tictoc-1588overmpls-04 (work in 2072 progress), February 2013. 2074 [RFC0791] Postel, J., "Internet Protocol", STD 5, RFC 791, 2075 September 1981. 2077 [RFC2474] Nichols, K., Blake, S., Baker, F., and D. Black, 2078 "Definition of the Differentiated Services Field (DS 2079 Field) in the IPv4 and IPv6 Headers", RFC 2474, 2080 December 1998. 2082 [RFC2475] Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z., 2083 and W. Weiss, "An Architecture for Differentiated 2084 Services", RFC 2475, December 1998. 2086 [RFC2597] Heinanen, J., Baker, F., Weiss, W., and J. Wroclawski, 2087 "Assured Forwarding PHB Group", RFC 2597, June 1999. 2089 [RFC3031] Rosen, E., Viswanathan, A., and R. Callon, "Multiprotocol 2090 Label Switching Architecture", RFC 3031, January 2001. 2092 [RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition 2093 of Explicit Congestion Notification (ECN) to IP", 2094 RFC 3168, September 2001. 2096 [RFC3429] Ohta, H., "Assignment of the 'OAM Alert Label' for 2097 Multiprotocol Label Switching Architecture (MPLS) 2098 Operation and Maintenance (OAM) Functions", RFC 3429, 2099 November 2002. 2101 [RFC3471] Berger, L., "Generalized Multi-Protocol Label Switching 2102 (GMPLS) Signaling Functional Description", RFC 3471, 2103 January 2003. 2105 [RFC3985] Bryant, S. and P. Pate, "Pseudo Wire Emulation Edge-to- 2106 Edge (PWE3) Architecture", RFC 3985, March 2005. 2108 [RFC4110] Callon, R. and M. Suzuki, "A Framework for Layer 3 2109 Provider-Provisioned Virtual Private Networks (PPVPNs)", 2110 RFC 4110, July 2005. 2112 [RFC4124] Le Faucheur, F., "Protocol Extensions for Support of 2113 Diffserv-aware MPLS Traffic Engineering", RFC 4124, 2114 June 2005. 2116 [RFC4206] Kompella, K. and Y. Rekhter, "Label Switched Paths (LSP) 2117 Hierarchy with Generalized Multi-Protocol Label Switching 2118 (GMPLS) Traffic Engineering (TE)", RFC 4206, October 2005. 2120 [RFC4221] Nadeau, T., Srinivasan, C., and A. Farrel, "Multiprotocol 2121 Label Switching (MPLS) Management Overview", RFC 4221, 2122 November 2005. 2124 [RFC4377] Nadeau, T., Morrow, M., Swallow, G., Allan, D., and S. 2125 Matsushima, "Operations and Management (OAM) Requirements 2126 for Multi-Protocol Label Switched (MPLS) Networks", 2127 RFC 4377, February 2006. 2129 [RFC4379] Kompella, K. and G. Swallow, "Detecting Multi-Protocol 2130 Label Switched (MPLS) Data Plane Failures", RFC 4379, 2131 February 2006. 2133 [RFC4664] Andersson, L. and E. Rosen, "Framework for Layer 2 Virtual 2134 Private Networks (L2VPNs)", RFC 4664, September 2006. 2136 [RFC4875] Aggarwal, R., Papadimitriou, D., and S. Yasukawa, 2137 "Extensions to Resource Reservation Protocol - Traffic 2138 Engineering (RSVP-TE) for Point-to-Multipoint TE Label 2139 Switched Paths (LSPs)", RFC 4875, May 2007. 2141 [RFC4928] Swallow, G., Bryant, S., and L. Andersson, "Avoiding Equal 2142 Cost Multipath Treatment in MPLS Networks", BCP 128, 2143 RFC 4928, June 2007. 2145 [RFC4950] Bonica, R., Gan, D., Tappan, D., and C. Pignataro, "ICMP 2146 Extensions for Multiprotocol Label Switching", RFC 4950, 2147 August 2007. 2149 [RFC5036] Andersson, L., Minei, I., and B. Thomas, "LDP 2150 Specification", RFC 5036, October 2007. 2152 [RFC5082] Gill, V., Heasley, J., Meyer, D., Savola, P., and C. 2153 Pignataro, "The Generalized TTL Security Mechanism 2154 (GTSM)", RFC 5082, October 2007. 2156 [RFC5085] Nadeau, T. and C. Pignataro, "Pseudowire Virtual Circuit 2157 Connectivity Verification (VCCV): A Control Channel for 2158 Pseudowires", RFC 5085, December 2007. 2160 [RFC5317] Bryant, S. and L. Andersson, "Joint Working Team (JWT) 2161 Report on MPLS Architectural Considerations for a 2162 Transport Profile", RFC 5317, February 2009. 2164 [RFC5332] Eckert, T., Rosen, E., Aggarwal, R., and Y. Rekhter, "MPLS 2165 Multicast Encapsulations", RFC 5332, August 2008. 2167 [RFC5462] Andersson, L. and R. Asati, "Multiprotocol Label Switching 2168 (MPLS) Label Stack Entry: "EXP" Field Renamed to "Traffic 2169 Class" Field", RFC 5462, February 2009. 2171 [RFC5695] Akhter, A., Asati, R., and C. Pignataro, "MPLS Forwarding 2172 Benchmarking Methodology for IP Flows", RFC 5695, 2173 November 2009. 2175 [RFC5860] Vigoureux, M., Ward, D., and M. Betts, "Requirements for 2176 Operations, Administration, and Maintenance (OAM) in MPLS 2177 Transport Networks", RFC 5860, May 2010. 2179 [RFC5880] Katz, D. and D. Ward, "Bidirectional Forwarding Detection 2180 (BFD)", RFC 5880, June 2010. 2182 [RFC5884] Aggarwal, R., Kompella, K., Nadeau, T., and G. Swallow, 2183 "Bidirectional Forwarding Detection (BFD) for MPLS Label 2184 Switched Paths (LSPs)", RFC 5884, June 2010. 2186 [RFC5885] Nadeau, T. and C. Pignataro, "Bidirectional Forwarding 2187 Detection (BFD) for the Pseudowire Virtual Circuit 2188 Connectivity Verification (VCCV)", RFC 5885, June 2010. 2190 [RFC5905] Mills, D., Martin, J., Burbank, J., and W. Kasch, "Network 2191 Time Protocol Version 4: Protocol and Algorithms 2192 Specification", RFC 5905, June 2010. 2194 [RFC6291] Andersson, L., van Helvoort, H., Bonica, R., Romascanu, 2195 D., and S. Mansfield, "Guidelines for the Use of the "OAM" 2196 Acronym in the IETF", BCP 161, RFC 6291, June 2011. 2198 [RFC6310] Aissaoui, M., Busschbach, P., Martini, L., Morrow, M., 2199 Nadeau, T., and Y(J). Stein, "Pseudowire (PW) Operations, 2200 Administration, and Maintenance (OAM) Message Mapping", 2201 RFC 6310, July 2011. 2203 [RFC6371] Busi, I. and D. Allan, "Operations, Administration, and 2204 Maintenance Framework for MPLS-Based Transport Networks", 2205 RFC 6371, September 2011. 2207 [RFC6374] Frost, D. and S. Bryant, "Packet Loss and Delay 2208 Measurement for MPLS Networks", RFC 6374, September 2011. 2210 [RFC6375] Frost, D. and S. Bryant, "A Packet Loss and Delay 2211 Measurement Profile for MPLS-Based Transport Networks", 2212 RFC 6375, September 2011. 2214 [RFC6388] Wijnands, IJ., Minei, I., Kompella, K., and B. Thomas, 2215 "Label Distribution Protocol Extensions for Point-to- 2216 Multipoint and Multipoint-to-Multipoint Label Switched 2217 Paths", RFC 6388, November 2011. 2219 [RFC6424] Bahadur, N., Kompella, K., and G. Swallow, "Mechanism for 2220 Performing Label Switched Path Ping (LSP Ping) over MPLS 2221 Tunnels", RFC 6424, November 2011. 2223 [RFC6425] Saxena, S., Swallow, G., Ali, Z., Farrel, A., Yasukawa, 2224 S., and T. Nadeau, "Detecting Data-Plane Failures in 2225 Point-to-Multipoint MPLS - Extensions to LSP Ping", 2226 RFC 6425, November 2011. 2228 [RFC6426] Gray, E., Bahadur, N., Boutros, S., and R. Aggarwal, "MPLS 2229 On-Demand Connectivity Verification and Route Tracing", 2230 RFC 6426, November 2011. 2232 [RFC6427] Swallow, G., Fulignoli, A., Vigoureux, M., Boutros, S., 2233 and D. Ward, "MPLS Fault Management Operations, 2234 Administration, and Maintenance (OAM)", RFC 6427, 2235 November 2011. 2237 [RFC6428] Allan, D., Swallow Ed. , G., and J. Drake Ed. , "Proactive 2238 Connectivity Verification, Continuity Check, and Remote 2239 Defect Indication for the MPLS Transport Profile", 2240 RFC 6428, November 2011. 2242 [RFC6435] Boutros, S., Sivabalan, S., Aggarwal, R., Vigoureux, M., 2243 and X. Dai, "MPLS Transport Profile Lock Instruct and 2244 Loopback Functions", RFC 6435, November 2011. 2246 [RFC6438] Carpenter, B. and S. Amante, "Using the IPv6 Flow Label 2247 for Equal Cost Multipath Routing and Link Aggregation in 2248 Tunnels", RFC 6438, November 2011. 2250 [RFC6478] Martini, L., Swallow, G., Heron, G., and M. Bocci, 2251 "Pseudowire Status for Static Pseudowires", RFC 6478, 2252 May 2012. 2254 [RFC6639] King, D. and M. Venkatesan, "Multiprotocol Label Switching 2255 Transport Profile (MPLS-TP) MIB-Based Management 2256 Overview", RFC 6639, June 2012. 2258 [RFC6669] Sprecher, N. and L. Fang, "An Overview of the Operations, 2259 Administration, and Maintenance (OAM) Toolset for MPLS- 2260 Based Transport Networks", RFC 6669, July 2012. 2262 [RFC6670] Sprecher, N. and KY. Hong, "The Reasons for Selecting a 2263 Single Solution for MPLS Transport Profile (MPLS-TP) 2264 Operations, Administration, and Maintenance (OAM)", 2265 RFC 6670, July 2012. 2267 [RFC6720] Pignataro, C. and R. Asati, "The Generalized TTL Security 2268 Mechanism (GTSM) for the Label Distribution Protocol 2269 (LDP)", RFC 6720, August 2012. 2271 [RFC6829] Chen, M., Pan, P., Pignataro, C., and R. Asati, "Label 2272 Switched Path (LSP) Ping for Pseudowire Forwarding 2273 Equivalence Classes (FECs) Advertised over IPv6", 2274 RFC 6829, January 2013. 2276 Appendix A. Organization of References Section 2278 The References section is split into Normative and Informative 2279 subsections. References that directly specify forwarding 2280 encapsulations or behaviors are listed as normative. References 2281 which describe signaling only, though normative with respect to 2282 signaling, are listed as informative. They are informative with 2283 respect to MPLS forwarding. 2285 Authors' Addresses 2287 Curtis Villamizar (editor) 2288 Outer Cape Cod Network Consulting, LLC 2290 Email: curtis@occnc.com 2292 Kireeti Kompella 2293 Contrail Systems 2295 Email: kireeti.kompella@gmail.com 2296 Shane Amante 2297 Level 3 Communications, Inc. 2298 1025 Eldorado Blvd 2299 Broomfield, CO 80021 2301 Email: shane@level3.net 2303 Andrew Malis 2304 Verizon 2305 60 Sylvan Road 2306 Waltham, MA 02451 2308 Phone: +1 781-466-2362 2309 Email: andrew.g.malis@verizon.com 2311 Carlos Pignataro 2312 Cisco Systems 2313 7200-12 Kit Creek Road 2314 Research Triangle Park, NC 27709 2315 US 2317 Email: cpignata@cisco.com