idnits 2.17.1 draft-ietf-lsvr-l3dl-02.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (July 7, 2019) is 1745 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Outdated reference: A later version (-18) exists of draft-ietf-idr-bgp-ls-segment-routing-ext-16 == Outdated reference: A later version (-29) exists of draft-ietf-lsvr-bgp-spf-04 -- Possible downref: Non-RFC (?) normative reference: ref. 'IANA-PEN' -- Possible downref: Non-RFC (?) normative reference: ref. 'IEEE802-2014' ** Obsolete normative reference: RFC 5226 (Obsoleted by RFC 8126) ** Obsolete normative reference: RFC 7752 (Obsoleted by RFC 9552) Summary: 2 errors (**), 0 flaws (~~), 3 warnings (==), 3 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group R. Bush 3 Internet-Draft Arrcus & Internet Initiative Japan 4 Intended status: Standards Track R. Austein 5 Expires: January 8, 2020 K. Patel 6 Arrcus 7 July 7, 2019 9 Layer 3 Discovery and Liveness 10 draft-ietf-lsvr-l3dl-02 12 Abstract 14 In Massive Data Centers, BGP-SPF and similar routing protocols are 15 used to build topology and reachability databases. These protocols 16 need to discover IP Layer 3 attributes of links, such as logical link 17 IP encapsulation abilities, IP neighbor address discovery, and link 18 liveness. This Layer 3 Discovery and Liveness protocol collects 19 these data, which may then be disseminated using BGP-SPF and similar 20 protocols. 22 Requirements Language 24 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 25 "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and 26 "OPTIONAL" in this document are to be interpreted as described in 27 BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all 28 capitals, as shown here. 30 Status of This Memo 32 This Internet-Draft is submitted in full conformance with the 33 provisions of BCP 78 and BCP 79. 35 Internet-Drafts are working documents of the Internet Engineering 36 Task Force (IETF). Note that other groups may also distribute 37 working documents as Internet-Drafts. The list of current Internet- 38 Drafts is at https://datatracker.ietf.org/drafts/current/. 40 Internet-Drafts are draft documents valid for a maximum of six months 41 and may be updated, replaced, or obsoleted by other documents at any 42 time. It is inappropriate to use Internet-Drafts as reference 43 material or to cite them other than as "work in progress." 45 This Internet-Draft will expire on January 8, 2020. 47 Copyright Notice 49 Copyright (c) 2019 IETF Trust and the persons identified as the 50 document authors. All rights reserved. 52 This document is subject to BCP 78 and the IETF Trust's Legal 53 Provisions Relating to IETF Documents 54 (https://trustee.ietf.org/license-info) in effect on the date of 55 publication of this document. Please review these documents 56 carefully, as they describe your rights and restrictions with respect 57 to this document. Code Components extracted from this document must 58 include Simplified BSD License text as described in Section 4.e of 59 the Trust Legal Provisions and are provided without warranty as 60 described in the Simplified BSD License. 62 Table of Contents 64 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 65 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4 66 3. Background . . . . . . . . . . . . . . . . . . . . . . . . . 5 67 4. Top Level Overview . . . . . . . . . . . . . . . . . . . . . 5 68 5. Inter-Link Protocol Overview . . . . . . . . . . . . . . . . 6 69 5.1. L3DL Ladder Diagram . . . . . . . . . . . . . . . . . . . 7 70 6. Transport Layer . . . . . . . . . . . . . . . . . . . . . . . 8 71 7. The Checksum . . . . . . . . . . . . . . . . . . . . . . . . 10 72 8. TLV PDUs . . . . . . . . . . . . . . . . . . . . . . . . . . 12 73 9. Logical Link Endpoint Identifier . . . . . . . . . . . . . . 13 74 10. HELLO . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 75 11. OPEN . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 76 12. ACK . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 77 12.1. Retransmission . . . . . . . . . . . . . . . . . . . . . 18 78 13. The Encapsulations . . . . . . . . . . . . . . . . . . . . . 18 79 13.1. The Encapsulation PDU Skeleton . . . . . . . . . . . . . 19 80 13.2. Encapsulaion Flags . . . . . . . . . . . . . . . . . . . 20 81 13.3. IPv4 Encapsulation . . . . . . . . . . . . . . . . . . . 21 82 13.4. IPv6 Encapsulation . . . . . . . . . . . . . . . . . . . 21 83 13.5. MPLS Label List . . . . . . . . . . . . . . . . . . . . 22 84 13.6. MPLS IPv4 Encapsulation . . . . . . . . . . . . . . . . 22 85 13.7. MPLS IPv6 Encapsulation . . . . . . . . . . . . . . . . 23 86 14. VENDOR - Vendor Extensions . . . . . . . . . . . . . . . . . 24 87 15. KEEPALIVE - Layer 2 Liveness . . . . . . . . . . . . . . . . 25 88 16. Layers 2.5 and 3 Liveness . . . . . . . . . . . . . . . . . . 26 89 17. The North/South Protocol . . . . . . . . . . . . . . . . . . 26 90 17.1. Use BGP-LS as Much as Possible . . . . . . . . . . . . . 26 91 17.2. Extensions to BGP-LS . . . . . . . . . . . . . . . . . . 26 92 18. Discussion . . . . . . . . . . . . . . . . . . . . . . . . . 27 93 18.1. HELLO Discussion . . . . . . . . . . . . . . . . . . . . 27 94 18.2. HELLO versus KEEPALIVE . . . . . . . . . . . . . . . . . 27 96 19. VLANs/SVIs/Sub-interfaces . . . . . . . . . . . . . . . . . . 27 97 20. Implementation Considerations . . . . . . . . . . . . . . . . 28 98 21. Security Considerations . . . . . . . . . . . . . . . . . . . 28 99 22. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 29 100 22.1. PDU Types . . . . . . . . . . . . . . . . . . . . . . . 29 101 22.2. Signature Type . . . . . . . . . . . . . . . . . . . . . 29 102 22.3. Flag Bits . . . . . . . . . . . . . . . . . . . . . . . 30 103 22.4. Error Codes . . . . . . . . . . . . . . . . . . . . . . 30 104 23. IEEE Considerations . . . . . . . . . . . . . . . . . . . . . 30 105 24. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 30 106 25. References . . . . . . . . . . . . . . . . . . . . . . . . . 31 107 25.1. Normative References . . . . . . . . . . . . . . . . . . 31 108 25.2. Informative References . . . . . . . . . . . . . . . . . 32 109 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 33 111 1. Introduction 113 The Massive Data Center (MDC) environment presents unusual problems 114 of scale, e.g. O(10,000) forwarding devices, while its homogeneity 115 presents opportunities for simple approaches. Approaches such as 116 Jupiter Rising [JUPITER] use a central controller to deal with 117 scaling, while BGP-SPF [I-D.ietf-lsvr-bgp-spf] provides massive 118 scale-out without centralization using a tried and tested scalable 119 distributed control plane, offering a scalable routing solution in 120 Clos [Clos0][Clos1] and similar environments. But BGP-SPF and 121 similar higher level device-spanning protocols, e.g. 122 [I-D.malhotra-bess-evpn-lsoe], need logical link state and addressing 123 data from the network to build the routing topology. They also need 124 prompt but prudent reaction to (logical) link failure. 126 Layer 3 Discovery and Liveness (L3DL) provides brutally simple 127 mechanisms for devices to 129 o Discover each other's unique endpoint identification, 131 o Discover mutually supported layer 3 encapsulations, e.g. IP/MPLS, 133 o Discover Layer 3 IP and/or MPLS addressing of interfaces of the 134 encapsulations, 136 o Present these data, using a very restricted profile of a BGP-LS 137 [RFC7752] API, to BGP-SPF which computes the topology and builds 138 routing and forwarding tables, 140 o Enable layer 3 link liveness such as BFD, and finally 142 o Provide Layer 2 keep-alive messages for session continuity. 144 This protocol may be more widely applicable to a range of routing and 145 similar protocols which need layer 3 discovery and characterisation. 147 2. Terminology 149 Even though it concentrates on the inter-device layer, this document 150 relies heavily on routing terminology. The following attempts to 151 clarify the use of some possibly confusing terms: 153 ASN: Autonomous System Number [RFC4271], a BGP identifier for 154 an originator of Layer 3 routes, particularly BGP 155 announcements. 156 BGP-LS: A mechanism by which link-state and TE information can be 157 collected from networks and shared with external 158 components using the BGP routing protocol. See [RFC7752]. 159 BGP-SPF A hybrid protocol using BGP transport but a Dijkstra 160 Shortest Path First decision process. See 161 [I-D.ietf-lsvr-bgp-spf]. 162 Clos: A hierarchic subset of a crossbar switch topology commonly 163 used in data centers. 164 Datagram: The L3DL content of a single Layer 2 frame. A full L3DL 165 PDU may be packaged in multiple Datagrams. 166 Encapsulation: Address Family Indicator and Subsequent Address 167 Family Indicator (AFI/SAFI). I.e. classes of layer 2.5 168 and 3 addresses such as IPv4, IPv6, MPLS, etc. 169 Frame: A Layer 2 packet. 170 Link or Logical Link: A logical connection between two logical ports 171 on two devices. E.g. two VLANs between the same two ports 172 are two links. 173 LLEI: Logical Link Endpoint Identifier, the unique identifier of 174 one end of a logical link, see Section 9. 175 MAC Address: 48-bit Layer 2 addresses are assumed since they are 176 used by all widely deployed Layer 2 network technologies 177 of interest, especially Ethernet. See [IEEE.802_2001]. 178 MDC: Massive Data Center, commonly composed of thousands of Top 179 of Rack Switches (TORs). 180 MTU: Maximum Transmission Unit, the size in octets of the 181 largest packet that can be sent on a medium, see [RFC1122] 182 1.3.3. 183 PDU: Protocol Data Unit, an L3DL application layer message. A 184 PDU may need to be broken into multiple Datagrams to make 185 it through MTU or other restrictions. 186 RouterID: An 32-bit identifier unique in the current routing domain, 187 see [RFC6286]. 188 Session: An established, via OPEN PDUs, session between two L3DL 189 capable link end-points, 190 SPF: Shortest Path First, an algorithm for finding the shortest 191 paths between nodes in a graph; AKA Dijkstra's algorithm. 193 System Identifier: An eight octet ISO System Identifier a la 194 [RFC1629] System ID 195 TOR: Top Of Rack switch, aggregates the servers in a rack and 196 connects to aggregation layers of the Clos tree, AKA the 197 Clos spine. 198 ZTP: Zero Touch Provisioning gives devices initial addresses, 199 credentials, etc. on boot/restart. 201 3. Background 203 L3DL is primarily designed for a Clos type datacenter scale and 204 topology, but can accommodate richer topologies which contain 205 potential cycles. 207 While L3DL is designed for the MDC, there are no inherent reasons it 208 could not run on a WAN. The authentication and authorization needed 209 to run safely on a WAN need to be considered, and the appropriate 210 level of security options chosen. 212 L3DL assumes a new IEEE assigned EtherType (TBD). 214 The number of addresses of one Encapsulation type on an interface 215 link may be quite large given a TOR with tens of servers, each server 216 having a few hundred micro-services, resulting in an inordinate 217 number of addresses. And highly automated micro-service migration 218 can cause serious address prefix disaggregation, resulting in 219 interfaces with thousands of disaggregated prefixes. 221 Therefore the L3DL protocol is session oriented and uses incremental 222 announcement and widrawal with session restart, a la BGP ([RFC4271]). 224 4. Top Level Overview 226 o Devices discover each other on logical links 228 o Logical Link Endpoint Identifiers are exchanged 230 o Layer 2 Liveness Checks may be started 232 o Encapsulation data are exchanged and IP-Level Liveness Checks 233 enabled 235 o A BGP-like upper layer protocol is assumed to use these data to 236 discover and build a topology database 238 +-------------------+ +-------------------+ +-------------------+ 239 | Device | | Device | | Device | 240 | | | | | | 241 |+-----------------+| |+-----------------+| |+-----------------+| 242 || || || || || || 243 || BGP-SPF <+---+> BGP-SPF <+---+> BGP-SPF || 244 || || || || || || 245 |+--------^--------+| |+--------^--------+| |+--------^--------+| 246 | | | | | | | | | 247 | | | | | | | | | 248 |+--------+--------+| |+--------+--------+| |+--------+--------+| 249 || Encapsulations || || Encapsulations || || Encapsulations || 250 || Addresses || || Addresses || || Addresses || 251 || L2 Liveness || || L2 Liveness || || L2 Liveness || 252 |+--------^--------+| |+--------^--------+| |+--------^--------+| 253 | | | | | | | | | 254 | | | | | | | | | 255 |+--------v--------+| |+--------v--------+| |+--------v--------+| 256 || || || || || || 257 ||Inter-Device PDUs<+---+>Inter-Device PDUs<+---+>Inter-Device PDUs|| 258 || || || || || || 259 |+-----------------+| |+-----------------+| |+-----------------+| 260 +-------------------+ +-------------------+ +-------------------+ 262 There are two protocols, the inter-device per-link layer 3 discovery 263 and the API to the upper level BGP-like routing prototol: 265 o Inter-device PDUs are used to exchange device and logical link 266 identities and layer 2.5 and 3 identifiers (not payloads), e.g. 267 device IDs, port identities, VLAN IDs, Encapsulations, and IP 268 addresses. 270 o A Link Layer to BGP API presents these data up the stack to a BGP 271 protocol or an other device-spanning upper layer protocol, 272 presenting them using the BGP-LS BGP-like data format. 274 The upper layer BGP family routing protocols cross all the devices, 275 though they are not part of these L3DL protocols. 277 To simplify this document, Layer 2 framing is not shown. L3DL is 278 about layer 3. 280 5. Inter-Link Protocol Overview 282 Two devices discover each other and their respective identities by 283 sending multicast HELLO PDUs (Section 10). To assure discovery of 284 new devices coming up on a multi-link topology, devices on such a 285 topology send periodic HELLOs forever, see Section 18.1. 287 Once a new device is recognized, both devices attempt to negotiate 288 and establish a session by sending unicast OPEN PDUs (Section 11). 289 In an established session, the Encapsulations (Section 13) configured 290 on an end point may be announced and modified. Note that these are 291 only the encapsuation and addresses configured on the announcing 292 interface; though a device's loopback and overlay interface(s) may 293 also be announced. When two devices on a link have compatible 294 Encapsulations and addresses, i.e. the same AFI/SAFI and the same 295 subnet, the link is announced via the BGP-LS API. 297 5.1. L3DL Ladder Diagram 299 The HELLO, Section 10, is a priming message. It is a small L3DL PDU 300 encapsulated in an Ethernet multicast frame with the simple goal of 301 discovering the identities of logical link endpoint(s) reachable from 302 a Logical Link Endpoint, Section 9. 304 The HELLO and OPEN, Section 11, PDUs, which are used to discover and 305 exchange detailed Logical Link Endpoint Identifiers, LLEIs, and the 306 ACK/ERROR PDU, are mandatory; other PDUs are optional; though at 307 least one encapsulation SHOULD be agreed at some point. 309 The following is a ladder-style diagram of the L3DL protocol 310 exchanges: 312 | HELLO | Logical Link Peer discovery 313 |---------------------------->| 314 | HELLO | Mandatory 315 |<----------------------------| 316 | | 317 | | 318 | OPEN | MACs, IDs, etc. 319 |---------------------------->| 320 | ACK | 321 |<----------------------------| 322 | | 323 | OPEN | Mandatory 324 |<----------------------------| 325 | ACK | 326 |---------------------------->| 327 | | 328 | | 329 | Interface IPv4 Addresses | Interface IPv4 Addresses 330 |---------------------------->| Optional 331 | ACK | 332 |<----------------------------| 333 | | 334 | Interface IPv4 Addresses | 335 |<----------------------------| 336 | ACK | 337 |---------------------------->| 338 | | 339 | | 340 | Interface IPv6 Addresses | Interface IPv6 Addresses 341 |---------------------------->| Optional 342 | ACK | 343 |<----------------------------| 344 | | 345 | Interface IPv6 Addresses | 346 |<----------------------------| 347 | ACK | 348 |---------------------------->| 349 | | 350 | | 351 | Interface MPLSv4 Labels | Interface MPLSv4 Labels 352 |---------------------------->| Optional 353 | ACK | 354 |<----------------------------| 355 | | 356 | Interface MPLSv4 Labels | Interface MPLSv4 Labels 357 |<----------------------------| Optional 358 | ACK | 359 |---------------------------->| 360 | | 361 | | 362 | Interface MPLSv6 Labels | Interface MPLSv6 Labels 363 |---------------------------->| Optional 364 | ACK | 365 |<----------------------------| 366 | | 367 | Interface MPLSv6 Labels | Interface MPLSv6 Labels 368 |<----------------------------| Optional 369 | ACK | 370 |---------------------------->| 371 | | 372 | | 373 | L3DL KEEPALIVE | Layer 2 Liveness 374 |---------------------------->| Optional 375 | L3DL KEEPALIVE | 376 |<----------------------------| 378 6. Transport Layer 380 L3DL PDUs are carried by a simple transport layer which allows PDUs 381 to occupy many Ethernet frames. An L3DL Ethernet frame is referred 382 to as a Datagram. 384 The L3DL Transport Layer encapsulates each Datagram using a common 385 transport header. 387 If a PDU does not fit in a single datagram, it is broken into 388 multiple datagrams and reassembled by the receiver a la [RFC0791] 389 Section 2.3 Fragmentation. 391 0 1 2 3 392 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 393 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 394 | Version |L| Datagram Number | 395 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 396 | Datagram Length | Checksum ~ 397 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 398 ~ | Payload... ~ 399 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 401 The fields of the L3DL Transport Header are as follows: 403 Version: Seven-bit Version number of the protocol, currently 0. 404 Values other than 0 MUST BE treated as an error. The protocol 405 version nees to be in one and only one place, so it is in the 406 datagram as opposed to, for example, the PDU header. 408 L: A bit that set to one if this Datagram is the last Datagram of the 409 PDU. For a PDU which fits in only one Datagram, it is set to one. 410 Note that this is the inverse of the marking technique used by 411 [RFC0791]. 413 Datagram Number: A monotonically increasing 24-bit value which 414 starts at zero for each PDU. This is used to reassemble frames 415 into PDUs a la [RFC0791] Section 2.3. Note that this limits an 416 L3DL PDU to 2^24 frames. 418 Datagram Length: Total number of octets in the Datagram including 419 all payloads and fields. Note that this limits a datagram to 2^16 420 octets. 422 Checksum: A 32 bit hash over the Datagram to detect bit flips, see 423 Section 7. 425 Payload: The PDU being transported or a fragment thereof. 427 To avoid the need for a receiver to reassemble two PDUs at the same 428 time, a sender MUST NOT send a subsequent PDU when a PDU is already 429 in flight and not yet acknowledged if it is an ACKed PDU Type. 431 7. The Checksum 433 There is a reason conservative folk use a checksum in UDP. And as 434 many operators stretch to jumbo frames (over 1,500 octets) longer 435 checksums are the prudent approach. 437 For the purpose of computing a checksum, the checksum field itself is 438 assumed to be zero. 440 The following code describes the suggested algorithm. 442 Sum up 32-bit unsigned ints in a 64-bit long, then take the high- 443 order section, shift it right, rotate, add it in, repeat until zero. 445 446 #include 447 #include 449 /* The F table from Skipjack, and it would work for the S-Box. */ 450 static const uint8_t sbox[256] = { 451 0xa3,0xd7,0x09,0x83,0xf8,0x48,0xf6,0xf4,0xb3,0x21,0x15,0x78, 452 0x99,0xb1,0xaf,0xf9,0xe7,0x2d,0x4d,0x8a,0xce,0x4c,0xca,0x2e, 453 0x52,0x95,0xd9,0x1e,0x4e,0x38,0x44,0x28,0x0a,0xdf,0x02,0xa0, 454 0x17,0xf1,0x60,0x68,0x12,0xb7,0x7a,0xc3,0xe9,0xfa,0x3d,0x53, 455 0x96,0x84,0x6b,0xba,0xf2,0x63,0x9a,0x19,0x7c,0xae,0xe5,0xf5, 456 0xf7,0x16,0x6a,0xa2,0x39,0xb6,0x7b,0x0f,0xc1,0x93,0x81,0x1b, 457 0xee,0xb4,0x1a,0xea,0xd0,0x91,0x2f,0xb8,0x55,0xb9,0xda,0x85, 458 0x3f,0x41,0xbf,0xe0,0x5a,0x58,0x80,0x5f,0x66,0x0b,0xd8,0x90, 459 0x35,0xd5,0xc0,0xa7,0x33,0x06,0x65,0x69,0x45,0x00,0x94,0x56, 460 0x6d,0x98,0x9b,0x76,0x97,0xfc,0xb2,0xc2,0xb0,0xfe,0xdb,0x20, 461 0xe1,0xeb,0xd6,0xe4,0xdd,0x47,0x4a,0x1d,0x42,0xed,0x9e,0x6e, 462 0x49,0x3c,0xcd,0x43,0x27,0xd2,0x07,0xd4,0xde,0xc7,0x67,0x18, 463 0x89,0xcb,0x30,0x1f,0x8d,0xc6,0x8f,0xaa,0xc8,0x74,0xdc,0xc9, 464 0x5d,0x5c,0x31,0xa4,0x70,0x88,0x61,0x2c,0x9f,0x0d,0x2b,0x87, 465 0x50,0x82,0x54,0x64,0x26,0x7d,0x03,0x40,0x34,0x4b,0x1c,0x73, 466 0xd1,0xc4,0xfd,0x3b,0xcc,0xfb,0x7f,0xab,0xe6,0x3e,0x5b,0xa5, 467 0xad,0x04,0x23,0x9c,0x14,0x51,0x22,0xf0,0x29,0x79,0x71,0x7e, 468 0xff,0x8c,0x0e,0xe2,0x0c,0xef,0xbc,0x72,0x75,0x6f,0x37,0xa1, 469 0xec,0xd3,0x8e,0x62,0x8b,0x86,0x10,0xe8,0x08,0x77,0x11,0xbe, 470 0x92,0x4f,0x24,0xc5,0x32,0x36,0x9d,0xcf,0xf3,0xa6,0xbb,0xac, 471 0x5e,0x6c,0xa9,0x13,0x57,0x25,0xb5,0xe3,0xbd,0xa8,0x3a,0x01, 472 0x05,0x59,0x2a,0x46 473 }; 475 /* non-normative example C code, constant time even */ 477 uint32_t sbox_checksum_32(const uint8_t *b, const size_t n) 478 { 479 uint32_t sum[4] = {0, 0, 0, 0}; 480 uint64_t result = 0; 481 for (size_t i = 0; i < n; i++) 482 sum[i & 3] += sbox[*b++]; 483 for (int i = 0; i < sizeof(sum)/sizeof(*sum); i++) 484 result = (result << 8) + sum[i]; 485 result = (result >> 32) + (result & 0xFFFFFFFF); 486 result = (result >> 32) + (result & 0xFFFFFFFF); 487 return (uint32_t) result; 488 } 489 491 8. TLV PDUs 493 The basic L3DL application layer PDU is a typical TLV (Type Length 494 Value) PDU. It includes a signature to provide optional integrity 495 and authentication. It may be broken into multiple Datagrams, see 496 Section 6. 498 0 1 2 3 499 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 500 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 501 | PDU Type | Payload Length ~ 502 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 503 ~ | Payload ... | 504 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 505 | Sig Type | Signature Length | ~ 506 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + 507 ~ Signature ~ 508 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 510 The fields of the basic L3DL header are as follows: 512 PDU Type: An integer differentiating PDU payload types. See 513 Section 22.1. 515 Payload Length: Total number of octets in the Payload field. 517 Payload: The application layer content of the L3DL PDU. 519 Sig Type: The type of the Signature, see Section 22.2. Type 0, a 520 null signature, is defined in this document. 522 Sig Type 0 indicates a null Signature. For a trivial PDU such as 523 KEEPALIVE, the underlying Datagram checksum may be sufficient for 524 integrity, though it lacks authentication. 526 Other Sig Types may be defined in other documents. 528 Signature Length: The length of the Signature, possibly including 529 padding, in octets. If Sig Type is 0, Signature Length MUST BE 0. 531 Signature: The result of running the signature algorithm specified 532 in Sig Type over all octets of the PDU except for the Signature 533 itself. 535 9. Logical Link Endpoint Identifier 537 L3DL discovers neighbors on logical links and establishes sessions 538 between the two ends of all consenting discovered logical links. A 539 logical link is described by a pair of Logical Link Endpoint 540 Identifiers, LLEIs. 542 An LLEI is a variable length descriptor which could be an ASN, a 543 classic RouterID, a catenation of the two, an eight octet ISO System 544 Identifier [RFC1629], or any other identifier unique to a single 545 logical link endpoint in the topology. 547 An L3DL deployment will choose and define an LLEI which suits its 548 needs, simple or complex. Two extremes are as follows: 550 A simplistic view of a link between two devices is two ports, 551 identified by unique MAC addresses, carrying a layer 3 protocol 552 conversation. In this case, the MAC addresses might suffice for the 553 LLEIs. 555 Unfortunately, things can get more complex. Multiple VLANs can run 556 between those two MAC addresses. In practice, many real devices use 557 the same MAC address on multiple ports and/or sub-interfaces. 559 Therefore, in the general circumstance, a fully described LLEI might 560 be as follows: 562 0 1 2 3 563 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 564 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 565 | | 566 + System Identifier + 567 | | 568 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 569 | ifIndex | 570 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 572 System Identifier, a la [RFC1629], is an eight octet identifier 573 unique in the entire operational space. Routers and switches usually 574 have internal MAC Addresses which can be padded with high order zeros 575 and used if no System ID exists on the device. If no unique 576 identifier is burned into a device, the local L3DL configuration 577 SHOULD create and assign a unique one by configuration. 579 ifIndex is the SNMP identifier of the (sub-)interface, see [RFC1213]. 580 This uniquely identifies the port. 582 For a layer 3 tagged sub-interface or a VLAN/SVI interface, Ifindex 583 is that of the logical sub-interface, so no further disambiguation is 584 needed. 586 L3DL PDUs learned over VLAN-ports may be interpreted by upper layer-3 587 routing protocols as being learned on the corresponding layer-3 SVI 588 interface for the VLAN. 590 LLEIs are big-endian. 592 10. HELLO 594 The HELLO PDU is unique in that it is encapsulated in a multicast 595 Ethernet frame. It solicits response(s) from other LLEI(s) on the 596 link. See Section 18.1 for why multicast is used. The destination 597 multicast MAC Addressees to be used MUST be one of the following, See 598 Clause 9.2.2 of [IEEE802-2014]: 600 01-80-C2-00-00-0E: Nearest Bridge = Propagation constrained to a 601 single physical link; stopped by all types of bridges (including 602 MPRs (media converters)). This SHOULD BE used when the link is 603 known to be a simple point to point link. 604 To Be Assigned: When a switch receives a frame with a multicast 605 destination MAC it does not recognize, it forwards to all ports. 606 This destination MAC is to be sent when the interface is known to 607 be connected to a switch. See Section 23. This SHOULD BE used 608 when the link may be a multi-point link. 610 All other L3DL PDUs are encapsulated in unicast frames, as the peer's 611 destination MAC address is known after the HELLO exchange. 613 When an interface is turned up on a device, it SHOULD issue a HELLO 614 if it is to participate in L3DL sessions. 616 If a constrained Nearest Bridge destination address is configured for 617 a point-to-point interface, see above, then the HELLO SHOULD NOT be 618 repeated once a session has been created by an exchange of OPENs. 620 If the configured destination address is one that is propagated by 621 switches, the HELLO SHOULD be repeated at a configured interval, with 622 a default of 60 seconds. This allows discovery by new devices which 623 come up on the layer-2 mesh. 625 0 1 2 3 626 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 627 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 628 | PDU Type = 0 | Payload Length = 0 ~ 629 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 630 ~ | Sig Type = 0 | Signature Length = 0 | 631 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 633 If more than one device responds, one adjacency is formed for each 634 unique source LLEI response. L3DL treats each adjacency as a 635 separate logical link. 637 When a HELLO is received from a source MAC address with which there 638 is no established L3DL session, the receiver SHOULD respond with an 639 OPEN PDU. The two devices establish an L3DL session by exchanging 640 OPEN PDUs. 642 The Payload Length is zero as there is no payload. 644 HELLO PDUs can not be signed as keying material has yet to be 645 exchanged. Hence the signature MUST always be the null type. 647 11. OPEN 649 Each device has learned the other's MAC Address from the HELLO 650 exchange, see Section 10. Therefore the OPEN and subsequent PDUs 651 MUST BE unicast, as opposed to the HELLO's multicast frame. 653 0 1 2 3 654 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 655 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 656 | PDU Type = 1 | Payload Length ~ 657 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 658 ~ | Nonce ~ 659 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 660 ~ | LLEI Length | My LLEI | 661 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-~ 662 ~ | AttrCount | ~ 663 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 664 ~ Attribute List ... | Auth Type | Key Length ~ 665 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 666 ~ | Key ... | 667 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 668 | Serial Number | 669 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 670 | Sig Type | Signature Length | Signature ... | 671 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 672 The Payload Length is the number of octets in all fields of the PDU 673 from the Nonce through the Serial Number, not including the three 674 final signature fields. 676 The Nonce enables detection of a duplicate OPEN PDU. It SHOULD be 677 either a random number or a high resolution timestamp. It is needed 678 to prevent session closure due to a repeated OPEN caused by a race or 679 a dropped or delayed ACK. 681 My LLEI is the sender's LLEI, see Section 9. 683 AttrCount is the number of attributes in the Attribute List. 684 Attributes are single octets the semantics of which are operator- 685 defined. 687 A node may have zero or more operator-defined attributes, e.g.: 688 spine, leaf, backbone, route reflector, arabica, ... 690 Attribute syntax and semantics are local to an operator or 691 datacenter; hence there is no global registry. Nodes exchange their 692 attributes only in the OPEN PDU. 694 Auth Type is the Signature algorithm suite, see Section 8. 696 Key Length is a 16-bit field denoting the length in octets of the Key 697 itself, not including the Auth Type or the Key Length. If there is 698 no Key, the Auth Type and key Length MUST both be zero. 700 The Key is specific to the operational environment. A failure to 701 authenticate is a failure to start the L3DL session, an ERROR PDU 702 MUST BE sent (Error Code 2), and HELLOs MUST be restarted. 704 The Serial Number is that of the last received and processed PDU. 705 This allows a receiver sending an OPEN to tell the sender that the 706 receiver wants to resume a session and the sender only needs to send 707 data more recent than the Serial Number. If this OPEN is not trying 708 to restart a lost session, the Serial Number MUST BE set to zero. 710 The Signature fields are described in Section 8 and in an asymmetric 711 key environment serve as a proof of possession of the signing auth 712 data by the sender. 714 Once two logical link endpoints know each other, and have ACKed each 715 other's OPEN PDUs, Layer 2 KEEPALIVEs (see Section 15) MAY be started 716 to ensure Layer 2 liveness and keep the session semantics alive. The 717 timing and acceptable drop of KEEPALIVE PDUs are discussed in 718 Section 15. 720 If a sender of OPEN does not receive an ACK of the OPEN PDU, then 721 they MUST resend the same OPEN PDU, with the same Nonce. Resending 722 an unacknowledged OPEN PDU, like other ACKed PDUs, SHOULD use 723 exponential back-off, see [RFC1122]. 725 If a properly authenticated OPEN arrives with a new Nonce from an 726 LLEI with which the receiving logical link endpoint believes it 727 already has an L3DL session (OPENs have already been exchanged), and 728 the Serial Number in the OPEN is non-zero, the receiver SHOULD 729 establish a new session by sending an OPEN with the Serial Number of 730 the last data it received. Each party MUST resume sending 731 encapsulations etc. subsequent to the other party's Sequence Number. 732 And each MUST retain all previously discovered encapsulation and 733 other data. 735 If a properly authenticated OPEN arrives with a new Nonce from an 736 LLEI with which the receiving logical link endpoint believes it 737 already has an L3DL session (OPENs have already been exchanged), and 738 the Serial Number in the OPEN is zero, then the receiver MUST assume 739 that the sending LLEI or entire device has been reset. All 740 previously discovered encapsulation data MUST NOT be kept and MUST be 741 withdrawn via the BGP-LS API and the recipient MUST respond with a 742 new OPEN. 744 12. ACK 746 The ACK PDU acknowledges receipt of a PDU and reports any error 747 condition which might have been raised. 749 0 1 2 3 750 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 751 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 752 | PDU Type = 3 | Payload Length = 5 ~ 753 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 754 ~ | ACKed PDU | EType | Error Code | 755 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 756 | Error Hint | Sig Type |Signature Leng.~ 757 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 758 ~ | Signature ... | 759 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 761 The ACK acknowledges receipt of an OPEN, Encapsulation, VENDOR PDU, 762 etc. 764 The ACKed PDU is the PDU Type of the PDU being acknowledged, e.g., 765 OPEN, one of the Encapsulations, etc. 767 If there was an error processing the received PDU, then the EType is 768 non-zero. If the EType is zero, Error Code and Error Hint MUST also 769 be zero. 771 A non-zero EType is the receiver's way of telling the PDU's sender 772 that the receiver had problems processing the PDU. The Error Code 773 and Error Hint will tell the sender more detail about the error. 775 The decimal value of EType gives a strong hint how the receiver 776 sending the ACK believes things should proceed: 778 0 - No Error, Error Code and Error Hint MUST be zero 779 1 - Warning, something not too serious happened, continue 780 2 - Session should not be continued, try to restart 781 3 - Restart is hopeless, call the operator 782 4-15 - Reserved 784 The Error Codes, noting protocol failures listed in thi document, are 785 listed in Section 22.4. Someone stuck in the 1990s might think the 786 catenation of EType and Error Code as an echo of 0x1zzz, 0x2zzz, etc. 787 They might be right; or not. 789 The Error Hint is any additional data the sender of the error PDU 790 thinks will help the recipient or the debugger with the particular 791 error. 793 The Signature fields are described in Section 8. 795 12.1. Retransmission 797 If a PDU sender expects an ACK, e.g. for an OPEN, an Encapsulation, a 798 VENDOR PDU, etc., and does not receive the ACK for a configurable 799 time (default one second), and the interface is live at layer 2, the 800 sender resends the PDU using exponential back-off, see [RFC1122]. 801 This cycle MAY be repeated a configurable number of times (default 802 three) before it is considered a failure. The session MAY BE 803 considered closed in case of this ACK failure. 805 If the link is broken at layer 2, retransmission MAY BE retried when 806 the link is restored. 808 13. The Encapsulations 810 Once the devices know each other's LLEIs, know each other's upper 811 layer identities, have means to ensure link state, etc., the L3DL 812 session is considered established, and the devices SHOULD exchange L3 813 interface encapsulations, L3 addresses, and L2.5 labels. 815 The Encapsulation types the peers exchange may be IPv4 816 (Section 13.3), IPv6 (Section 13.4), MPLS IPv4 (Section 13.6), MPLS 817 IPv6 (Section 13.7), and/or possibly others not defined here. 819 The sender of an Encapsulation PDU MUST NOT assume that the peer is 820 capable of the same Encapsulation Type. An ACK (Section 12) merely 821 acknowledges receipt. Only if both peers have sent the same 822 Encapsulation Type is it safe to assume that they are compatible for 823 that type. 825 A receiver of an encapsulation might recognize an addressing 826 conflict, such as both ends of the link trying to use the same 827 address. In this case, the receiver SHOULD respond with an error 828 (Error Code 1) ACK. As there may be other usable addresses or 829 encapsulations, this error might log and continue, letting an upper 830 layer topology builder deal with what works. 832 Further, to consider a logical link of a type to formally be 833 established so that it may be pushed up to upper layer protocols, the 834 addressing for the type must be compatible, e.g. on the same IP 835 subnet. 837 13.1. The Encapsulation PDU Skeleton 839 The header for all encapsulation PDUs is as follows: 841 0 1 2 3 842 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 843 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 844 | PDU Type | Payload Length ~ 845 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 846 ~ | Count | 847 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 848 | Serial Number | 849 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 850 | Encapsulation List... | Sig Type | 851 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 852 | Signature Length | Signature ... | 853 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 855 An Encapsulation PDU describes zero or more addresses of the 856 encapsulation type. 858 The 24-bit Count is the number of Encapsulations in the Encapsulation 859 list. 861 The Serial Number is a monotonically increasing 32-bit value 862 representing the sender's state in time. It may be an integer, a 863 timestamp, etc. On session restart (new OPEN), a receiver MAY send 864 the last received Session Number to tell the sender to only send 865 newer data. 867 If a sender has multiple links on the same interface, separate state: 868 data, ACKs, etc. must be kept for each peer session. 870 Over time, multiple Encapsulation PDUs may be sent for an interface 871 as configuration changes. 873 If the length of an Encapsulation PDU exceeds the Datagram size limit 874 on media, the PDU is broken into multiple Datagrams. See Section 8. 876 The Signature fields are described in Section 8. 878 The Receiver MUST acknowledge the Encapsulation PDU with a Type=3, 879 ACK PDU (Section 12) with the Encapsulation Type being that of the 880 encapsulation being announced, see Section 12. 882 If the Sender does not receive an ACK in a configurable interval 883 (default one second), and the interface is live at layer 2, they 884 SHOULD retransmit. After a user configurable number of failures, the 885 L3DL session should be considered dead and the OPEN process SHOULD be 886 restarted. 888 If the link is broken at layer 2, retransmission MAY BE retried if 889 data have not changed in the interim. 891 13.2. Encapsulaion Flags 893 0 1 2 3 4 ... 7 894 +------------+------------+------------+------------+------------+ 895 | Ann/With | Primary | Under/Over | Loopback | Reserved ..| 896 +------------+------------+------------+------------+------------+ 898 Each encapsulation in an Encapsulation PDU of Type T may announce new 899 and/or withdraw old encapsulations of Type T. It indicates this with 900 the Ann/With Encapsulation Flag, Announce == 1, Withdraw == 0. 902 Each Encapsulation interface address in an Encapsulation PDU is 903 either a new encapsulation be announced (Ann/With == 1) (yes, a la 904 BGP) or requests one be withdrawn (Ann/With == 0). Adding an 905 encapsulation which already exists SHOULD raise an Announce/Withdraw 906 Error (see Section 22.4); the EType SHOULD be 2, suggesting a session 907 restart (see Section 12 so all encapsulations will be resent. 909 If an LLEI has multiple addresses for an encapsulation type, one and 910 only one address SHOULD be configured to be marked as primary 911 (Primary Flag == 1). Only one address on an interface MAY be marked 912 as primary for a particular encapsulation type. 914 An Encapsulation interface address in an Encapsulation PDU MAY be 915 marked as a loopback, in which case the Loopback bit is set. 916 Loopback addresses are generally not seen directly on an external 917 interface. One or more loopback addresses MAY be exposed by 918 configuration on one or more L3DL speaking external interfaces, e.g. 919 for iBGP peering. They SHOULD be marked as such, Loopback Flag == 1. 921 Each Encapsulation interface address in an Encapsulation PDU is that 922 of the direct 'underlay interface (Under/Over == 1), or an 'overlay' 923 address (Under/Over == 0), likely that of a VM or container guest 924 bridged or configured on to the interface already having an underlay 925 address. 927 13.3. IPv4 Encapsulation 929 The IPv4 Encapsulation describes a device's ability to exchange IPv4 930 packets on one or more subnets. It does so by stating the 931 interface's addresses and the corresponding prefix lengths. 933 0 1 2 3 934 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 935 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 936 | PDU Type = 4 | Payload Length ~ 937 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 938 ~ | Count | 939 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 940 | Serial Number | 941 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 942 | Encaps Flags | IPv4 Address ~ 943 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 944 ~ | PrefixLen | more ... | Sig Type | 945 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 946 | Signature Length | Signature ... | 947 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 949 The 24-bit Count is the number of IPv4 Encapsulations being announced 950 and/or withdrawn. 952 13.4. IPv6 Encapsulation 954 The IPv6 Encapsulation describes a logical link's ability to exchange 955 IPv6 packets on one or more subnets. It does so by stating the 956 interface's addresses and the corresponding prefix lengths. 958 0 1 2 3 959 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 960 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 961 | PDU Type = 5 | Payload Length ~ 962 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 963 ~ | Count | 964 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 965 | Serial Number | 966 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 967 | Encaps Flags | | 968 +-+-+-+-+-+-+-+-+ + 969 | | 970 + + 971 | | 972 + + 973 | IPv6 Address | 974 + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 975 | | PrefixLen | more ... | Sig Type | 976 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 977 | Signature Length | Signature ... | 978 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 980 The 24-bit Count is the number of IPv6 Encapsulations being announced 981 and/or withdrawn. 983 13.5. MPLS Label List 985 As an MPLS enabled interface may have a label stack, see [RFC3032], a 986 variable length list of labels is needed. These are the labels the 987 sender will accept for the prefix to which the list is attached. 989 0 1 2 3 990 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 991 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 992 | Label Count | Label | Exp |S| 993 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 994 | Label | Exp |S| more ... | 995 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 997 A Label Count of zero is an implicit withdraw of all labels for that 998 prefix on that interface. 1000 13.6. MPLS IPv4 Encapsulation 1002 The MPLS IPv4 Encapsulation describes a logical link's ability to 1003 exchange labeled IPv4 packets on one or more subnets. It does so by 1004 stating the interface's addresses the corresponding prefix lengths, 1005 and the corresponding labels which will be accepted fpr each address. 1007 0 1 2 3 1008 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1009 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1010 | PDU Type = 6 | Payload Length ~ 1011 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1012 ~ | Count | 1013 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1014 | Serial Number | 1015 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1016 | Encaps Flags | MPLS Label List ... | ~ 1017 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1018 ~ IPv4 Address | PrefixLen | 1019 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1020 | more ... | Sig Type | Signature Length | 1021 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1022 | Signature | 1023 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1025 The 24-bit Count is the number of MPLSv4 Encapsulation being 1026 announced and/or withdrawns. 1028 13.7. MPLS IPv6 Encapsulation 1030 The MPLS IPv4 Encapsulation describes a logical link's ability to 1031 exchange labeled IPv4 packets on one or more subnets. It does so by 1032 stating the interface's addresses, the corresponding prefix lengths, 1033 and the corresponding labels which will be accepted for each address. 1035 0 1 2 3 1036 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1037 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1038 | PDU Type = 7 | Payload Length ~ 1039 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1040 ~ | Count | 1041 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1042 | Serial Number | 1043 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1044 | Encaps Flags | MPLS Label List ... | | 1045 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + 1046 | | 1047 + + 1048 | | 1049 + + 1050 | IPv6 Address | 1051 + +-+-+-+-+-+-+-+-+ 1052 | | Prefix Len | 1053 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1054 | more ... | Sig Type | Signature Length | 1055 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1056 | Signature ... | 1057 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1059 The 24-bit Count is the number of MPLSv6 Encapsulations being 1060 announced and/or withdrawn. 1062 14. VENDOR - Vendor Extensions 1064 0 1 2 3 1065 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1066 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1067 | PDU Type = 255| Payload Length ~ 1068 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1069 ~ | Serial Number ~ 1070 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1071 ~ | Enterprise Number | 1072 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1073 | Ent Type | Enterprise Data ... ~ 1074 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1075 ~ | Sig Type | Signature Length | 1076 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1077 | Signature ... | 1078 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1080 Vendors or enterprises may define TLVs beyond the scope of L3DL 1081 standards. This is done using a Private Enterprise Number [IANA-PEN] 1082 followed by Enterprise Data in a format defined for that Enterprise 1083 Number and Ent Type. 1085 Ent Type allows a VENDOR PDU to be sub-typed in the event that the 1086 vendor/enterprise needs multiple PDU types. 1088 As with Encapsulation PDUs, a receiver of a VENDOR PDU MUST respond 1089 with an ACK or an ERROR PDU. Similarly, a VENDOR PDU MUST only be 1090 sent over an open session. 1092 15. KEEPALIVE - Layer 2 Liveness 1094 0 1 2 3 1095 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1096 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1097 | PDU Type = 2 | Payload Length = 0 ~ 1098 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1099 ~ | Sig Type = 0 | Signature Length = 0 | 1100 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1102 L3DL devices SHOULD beacon frequent Layer 2 KEEPALIVE PDUs to ensure 1103 session continuity. A receiver may choose to ignore KEEPALIVE PDUs. 1105 An operational deployment MUST BE configured whether to use 1106 KEEPALIVEs or not, either globally, or down to per-link granularity. 1107 Disagreement MAY result in repeated session break and 1108 reestablishment. 1110 KEEPALIVEs SHOULD be beaconed at a configured frequency. One per 1111 second is the default. Layer 3 liveness, such as BFD, may be more 1112 (or less) aggressive. 1114 When a sender transmits a PDU which is not a KEEPALIVE, the sender 1115 SHOULD reset the KEEPALIVE timer. I.e. sending any PDU acts as a 1116 keepalive. Once the last fragment has been sent, the KEEPALIVE timer 1117 SHOULD BE restarted. Do not wait for the ACK. 1119 If a KEEPALIVE or other PDUs have not been received from a peer with 1120 which a receiver has an open session for a configurable time (default 1121 30 seconds), the link SHOULD BE presumed down. The devices MAY keep 1122 configuration state and restore it without retransmission if no data 1123 have changed. Otherwise, a new session SHOULD BE established and new 1124 Encapsulation PDUs exchanged. 1126 16. Layers 2.5 and 3 Liveness 1128 Layer 2 liveness may be continuously tested by KEEPALIVE PDUs, see 1129 Section 15. As layer 2.5 or layer 3 connectivity could still break, 1130 liveness above layer 2 MAY be frequently tested using BFD ([RFC5880]) 1131 or a similar technique. 1133 This protocol assumes that one or more Encapsulation addresses may be 1134 used to ping, run BFD, or whatever the operator configures. 1136 17. The North/South Protocol 1138 Thus far, a one-hop point-to-point logical link discovery protocol 1139 has been defined. 1141 The devices know their unique LLEIs and know the unique peer LLEIs 1142 and Encapsulations on each logical link interface. 1144 Full topology discovery is not appropriate at the L3DL layer, so 1145 Dijkstra a la IS-IS etc. is assumed to be done by higher level 1146 protocols such as BGP-SPF. 1148 Therefore the LLEIs, link Encapsulations, and state changes are 1149 pushed North via a small subset of the BGP-LS API. The upper layer 1150 routing protocol(s), e.g. BGP-SPF, learn and maintain the topology, 1151 run Dijkstra, and build the routing database(s). 1153 For example, if a neighbor's IPv4 Encapsulation address changes, the 1154 devices seeing the change push that change Northbound. 1156 17.1. Use BGP-LS as Much as Possible 1158 BGP-LS [RFC7752] defines BGP-like Datagrams describing logical link 1159 state (links, nodes, link prefixes, and many other things), and a new 1160 BGP path attribute providing Northbound transport, all of which can 1161 be ingested by upper layer protocols such as BGP-SPF; see Section 4 1162 of [I-D.ietf-lsvr-bgp-spf]. 1164 For IPv4 links, TLVs 259 and 260 are used. For IPv6 links, TLVs 261 1165 and 262. If there are multiple addresses on a link, multiple TLV 1166 pairs are pushed North, having the same ID pairs. 1168 17.2. Extensions to BGP-LS 1170 The Northbound protocol needs a few minor extensions to BGP-LS. 1171 Luckily, others have needed the same extensions. 1173 Similarly to BGP-SPF, the BGP protocol is used in the Protocol-ID 1174 field specified in table 1 of 1175 [I-D.ietf-idr-bgpls-segment-routing-epe]. The local and remote node 1176 descriptors for all NLRI are the IDs described in Section 11. This 1177 is equivalent to an adjacency SID or a node SID if the address is a 1178 loopback address. 1180 Label Sub-TLVs from [I-D.ietf-idr-bgp-ls-segment-routing-ext] 1181 Section 2.1.1, are used to associate one or more MPLS Labels with a 1182 link. 1184 18. Discussion 1186 This section explores some trade-offs taken and some considerations. 1188 18.1. HELLO Discussion 1190 A device with multiple Layer 2 interfaces, traditionally called a 1191 switch, may be used to forward frames and therefore packets from 1192 multiple devices to one logical interface (LLEI), I, on an L3DL 1193 speaking device. Interface I could discover a peer J across the 1194 switch. Later, a prospective peer K could come up across the switch. 1195 If I was not still sending and listening for HELLOs, the potential 1196 peering with K could not be discovered. Therefore, on multi-link 1197 interfaces MUST continue to send HELLOs as long as they are turned 1198 up. 1200 18.2. HELLO versus KEEPALIVE 1202 Both HELLO and KEEPALIVE are periodic. KEEPALIVE might be eliminated 1203 in favor of keeping only HELLOs. But KEEPALIVEs are unicast, and 1204 thus less noisy on the network, especially if HELLO is configured to 1205 transit layer-2-only switches, see Section 18.1. 1207 19. VLANs/SVIs/Sub-interfaces 1209 One can think of the protocol as an instance (i.e. state machine) 1210 which runs on each logical link of a device. 1212 As the upper routing layer must view VLAN topologies as separate 1213 graphs, L3DL treats VLAN ports as separate links. 1215 L3DL PDUs learned over VLAN-ports may be interpreted by upper layer-3 1216 routing protocols as being learned on the corresponding layer-3 SVI 1217 interface for the VLAN. 1219 As Sub-Interfaces each have their own LLIEs, they act as separate 1220 interfaces, forming their own links. 1222 20. Implementation Considerations 1224 An implementation SHOULD provide the ability to configure a logical 1225 interface as L3DL speaking or not. 1227 An implementation SHOULD provide the ability to configure whether 1228 HELLOs on an L3DL enabled interface send Nearest Bridge or the MAC 1229 which is propagated by switches from that interface; see Section 10. 1231 An implementation SHOULD provide the ability to distribute one or 1232 more loopback addresses or interfaces into L3DL on an external L3DL 1233 speaking interface. 1235 An implementation SHOULD provide the ability to distribute one or 1236 more overlay and/or underlay addresses or interfaces into L3DL on an 1237 external L3DL speaking interface. 1239 An implementation SHOULD provide the ability to configure one of the 1240 addresses of an encapsulation as primary on an L3DL speaking 1241 interface. If there is only one address for a particular 1242 encapsulation, the implementation MAY mark it as primary by default. 1244 An implementation MAY allow optional configuration which updates the 1245 local forwarding table with overlay and underlay data both learned 1246 from L3DL peers and configured locally. 1248 21. Security Considerations 1250 The protocol as is MUST NOT be used outside a datacenter or similarly 1251 closed environment due to lack of formal definition of the 1252 authentication and authorization mechanism. Sufficient mechanisms 1253 may be described in separate documents. 1255 Many MDC operators have a strange belief that physical walls and 1256 firewalls provide sufficient security. This is not credible. All 1257 MDC protocols need to be examined for exposure and attack surface. 1258 In the case of L3DL, Authentication and Integrity as provided in 1259 [draft-ymbk-l3dl-signing] is strongly recommended. 1261 It is generally unwise to assume that on the wire Layer 2 is secure. 1262 Strange/unauthorized devices may plug into a port. Mis-wiring is 1263 very common in datacenter installations. A poisoned laptop might be 1264 plugged into a device's port, form malicious sessions, etc. to 1265 divert, intercept, or drop traffic. 1267 Similarly, malicious nodes/devices could mis-announce addressing. 1269 If OPENs are not being authenticated, an attacker could forge an OPEN 1270 for an existing session and cause the session to be reset. 1272 For these reasons, the OPEN PDU's authentication data exchange SHOULD 1273 be used. 1275 If the KEEPALIVE PDU is not signed (as suggested in Section 8) to 1276 save computation, then a MITM could fake a session being alive. 1278 22. IANA Considerations 1280 22.1. PDU Types 1282 This document requests the IANA create a registry for L3DL PDU Type, 1283 which may range from 0 to 255. The name of the registry should be 1284 L3DL-PDU-Type. The policy for adding to the registry is RFC Required 1285 per [RFC5226], either standards track or experimental. The initial 1286 entries should be the following: 1288 PDU 1289 Code PDU Name 1290 ---- ------------------- 1291 0 HELLO 1292 1 OPEN 1293 2 KEEPALIVE 1294 3 ACK 1295 4 IPv4 Announcement 1296 5 IPv6 Announcement 1297 6 MPLS IPv4 Announcement 1298 7 MPLS IPv6 Announcement 1299 8-254 Reserved 1300 255 VENDOR 1302 22.2. Signature Type 1304 This document requests the IANA create a registry for L3DL Signature 1305 Type, AKA Sig Type, which may range from 0 to 255. The name of the 1306 registry should be L3DL-Signature-Type. The policy for adding to the 1307 registry is RFC Required per [RFC5226], either standards track or 1308 experimental. The initial entries should be the following: 1310 Number Name 1311 ------ ------------------- 1312 0 Null 1313 1-255 Reserved 1315 22.3. Flag Bits 1317 This document requests the IANA create a registry for L3DL PL Flag 1318 Bits, which may range from 0 to 7. The name of the registry should 1319 be L3DL-PL-Flag-Bits. The policy for adding to the registry is RFC 1320 Required per [RFC5226], either standards track or experimental. The 1321 initial entries should be the following: 1323 Bit Bit Name 1324 ---- ------------------- 1325 0 Announce/Withdraw (ann == 0) 1326 1 Primary 1327 2 Underlay/Overlay (under == 0) 1328 3 Loopback 1329 4-7 Reserved 1331 22.4. Error Codes 1333 This document requests the IANA create a registry for L3DL Error 1334 Codes, a 16 bit integer. The name of the registry should be L3DL- 1335 Error-Codes. The policy for adding to the registry is RFC Required 1336 per [RFC5226], either standards track or experimental. The initial 1337 entries should be the following: 1339 Error 1340 Code Error Name 1341 ---- ------------------- 1342 0 No Error 1343 1 Logical Link Addressing Conflict 1344 2 Authorization Failure in OPEN 1345 3 Signature Failure in PDU 1346 4 Announce/Withdraw Error 1348 23. IEEE Considerations 1350 This document requires a new EtherType. 1352 This document requires a new multicast MAC address that will be 1353 broadcast through a switch. 1355 24. Acknowledgments 1357 The authors thank Cristel Pelsser for multiple reviews, Harsha Kovuru 1358 for comments during implementation, Jeff Haas for review and 1359 comments, Joe Clarke for a useful review, John Scudder for deeply 1360 serious review and comments, Larry Kreeger for a lot of layer 2 clue, 1361 Martijn Schmidt for his contribution, Neeraj Malhotra for review, 1362 Russ Housley for checksum discussion and sBox, and Steve Bellovin for 1363 checksum advice. 1365 25. References 1367 25.1. Normative References 1369 [I-D.ietf-idr-bgp-ls-segment-routing-ext] 1370 Previdi, S., Talaulikar, K., Filsfils, C., Gredler, H., 1371 and M. Chen, "BGP Link-State extensions for Segment 1372 Routing", draft-ietf-idr-bgp-ls-segment-routing-ext-16 1373 (work in progress), June 2019. 1375 [I-D.ietf-idr-bgpls-segment-routing-epe] 1376 Previdi, S., Talaulikar, K., Filsfils, C., Patel, K., Ray, 1377 S., and J. Dong, "BGP-LS extensions for Segment Routing 1378 BGP Egress Peer Engineering", draft-ietf-idr-bgpls- 1379 segment-routing-epe-19 (work in progress), May 2019. 1381 [I-D.ietf-lsvr-bgp-spf] 1382 Patel, K., Lindem, A., Zandi, S., and W. Henderickx, 1383 "Shortest Path Routing Extensions for BGP Protocol", 1384 draft-ietf-lsvr-bgp-spf-04 (work in progress), December 1385 2018. 1387 [IANA-PEN] 1388 "IANA Private Enterprise Numbers", 1389 . 1392 [IEEE.802_2001] 1393 IEEE, "IEEE Standard for Local and Metropolitan Area 1394 Networks: Overview and Architecture", IEEE 802-2001, 1395 DOI 10.1109/ieeestd.2002.93395, July 2002, 1396 . 1398 [IEEE802-2014] 1399 Institute of Electrical and Electronics Engineers, "Local 1400 and Metropolitan Area Networks: Overview and 1401 Architecture", IEEE Std 802-2014, 2014. 1403 [RFC1213] McCloghrie, K. and M. Rose, "Management Information Base 1404 for Network Management of TCP/IP-based internets: MIB-II", 1405 STD 17, RFC 1213, DOI 10.17487/RFC1213, March 1991, 1406 . 1408 [RFC1629] Colella, R., Callon, R., Gardner, E., and Y. Rekhter, 1409 "Guidelines for OSI NSAP Allocation in the Internet", 1410 RFC 1629, DOI 10.17487/RFC1629, May 1994, 1411 . 1413 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1414 Requirement Levels", BCP 14, RFC 2119, 1415 DOI 10.17487/RFC2119, March 1997, 1416 . 1418 [RFC3032] Rosen, E., Tappan, D., Fedorkow, G., Rekhter, Y., 1419 Farinacci, D., Li, T., and A. Conta, "MPLS Label Stack 1420 Encoding", RFC 3032, DOI 10.17487/RFC3032, January 2001, 1421 . 1423 [RFC4271] Rekhter, Y., Ed., Li, T., Ed., and S. Hares, Ed., "A 1424 Border Gateway Protocol 4 (BGP-4)", RFC 4271, 1425 DOI 10.17487/RFC4271, January 2006, 1426 . 1428 [RFC5226] Narten, T. and H. Alvestrand, "Guidelines for Writing an 1429 IANA Considerations Section in RFCs", RFC 5226, 1430 DOI 10.17487/RFC5226, May 2008, 1431 . 1433 [RFC5880] Katz, D. and D. Ward, "Bidirectional Forwarding Detection 1434 (BFD)", RFC 5880, DOI 10.17487/RFC5880, June 2010, 1435 . 1437 [RFC6286] Chen, E. and J. Yuan, "Autonomous-System-Wide Unique BGP 1438 Identifier for BGP-4", RFC 6286, DOI 10.17487/RFC6286, 1439 June 2011, . 1441 [RFC7752] Gredler, H., Ed., Medved, J., Previdi, S., Farrel, A., and 1442 S. Ray, "North-Bound Distribution of Link-State and 1443 Traffic Engineering (TE) Information Using BGP", RFC 7752, 1444 DOI 10.17487/RFC7752, March 2016, 1445 . 1447 [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 1448 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, 1449 May 2017, . 1451 25.2. Informative References 1453 [Clos0] Clos, C., "A study of non-blocking switching networks 1454 [PAYWALLED]", Bell System Technical Journal 32 (2), pp 1455 406-424, March 1953. 1457 [Clos1] "Clos Network", 1458 . 1460 [I-D.malhotra-bess-evpn-lsoe] 1461 Malhotra, N., Patel, K., and J. Rabadan, "LSoE-based PE-CE 1462 Control Plane for EVPN", draft-malhotra-bess-evpn-lsoe-00 1463 (work in progress), March 2019. 1465 [JUPITER] Singh, A., Germano, P., Kanagala, A., Liu, H., Provost, 1466 J., Simmons, J., Tanda, E., Wanderer, J., HAP.lzle, U., 1467 Stuart, S., Vahdat, A., Ong, J., Agarwal, A., Anderson, 1468 G., Armistead, A., Bannon, R., Boving, S., Desai, G., and 1469 B. Felderman, "Jupiter rising", Communications of the 1470 ACM Vol. 59, pp. 88-97, DOI 10.1145/2975159, August 2016. 1472 [RFC0791] Postel, J., "Internet Protocol", STD 5, RFC 791, 1473 DOI 10.17487/RFC0791, September 1981, 1474 . 1476 [RFC1122] Braden, R., Ed., "Requirements for Internet Hosts - 1477 Communication Layers", STD 3, RFC 1122, 1478 DOI 10.17487/RFC1122, October 1989, 1479 . 1481 Authors' Addresses 1483 Randy Bush 1484 Arrcus & Internet Initiative Japan 1485 5147 Crystal Springs 1486 Bainbridge Island, WA 98110 1487 US 1489 Email: randy@psg.com 1491 Rob Austein 1492 Arrcus, Inc 1494 Email: sra@hactrn.net 1496 Keyur Patel 1497 Arrcus 1498 2077 Gateway Place, Suite #400 1499 San Jose, CA 95119 1500 US 1502 Email: keyur@arrcus.com