idnits 2.17.1 draft-ietf-lsvr-l3dl-03.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (November 2, 2019) is 1630 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Outdated reference: A later version (-18) exists of draft-ietf-idr-bgp-ls-segment-routing-ext-16 == Outdated reference: A later version (-29) exists of draft-ietf-lsvr-bgp-spf-06 == Outdated reference: A later version (-01) exists of draft-ymbk-lsvr-l3dl-signing-00 -- Possible downref: Non-RFC (?) normative reference: ref. 'IANA-PEN' -- Possible downref: Non-RFC (?) normative reference: ref. 'IEEE802-2014' ** Obsolete normative reference: RFC 5226 (Obsoleted by RFC 8126) ** Obsolete normative reference: RFC 7752 (Obsoleted by RFC 9552) Summary: 2 errors (**), 0 flaws (~~), 4 warnings (==), 3 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group R. Bush 3 Internet-Draft Arrcus & Internet Initiative Japan 4 Intended status: Standards Track R. Austein 5 Expires: May 5, 2020 K. Patel 6 Arrcus 7 November 2, 2019 9 Layer 3 Discovery and Liveness 10 draft-ietf-lsvr-l3dl-03 12 Abstract 14 In Massive Data Centers, BGP-SPF and similar routing protocols are 15 used to build topology and reachability databases. These protocols 16 need to discover IP Layer 3 attributes of links, such as logical link 17 IP encapsulation abilities, IP neighbor address discovery, and link 18 liveness. This Layer 3 Discovery and Liveness protocol collects 19 these data, which may then be disseminated using BGP-SPF and similar 20 protocols. 22 Requirements Language 24 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 25 "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and 26 "OPTIONAL" in this document are to be interpreted as described in BCP 27 14 [RFC2119] [RFC8174] when, and only when, they appear in all 28 capitals, as shown here. 30 Status of This Memo 32 This Internet-Draft is submitted in full conformance with the 33 provisions of BCP 78 and BCP 79. 35 Internet-Drafts are working documents of the Internet Engineering 36 Task Force (IETF). Note that other groups may also distribute 37 working documents as Internet-Drafts. The list of current Internet- 38 Drafts is at https://datatracker.ietf.org/drafts/current/. 40 Internet-Drafts are draft documents valid for a maximum of six months 41 and may be updated, replaced, or obsoleted by other documents at any 42 time. It is inappropriate to use Internet-Drafts as reference 43 material or to cite them other than as "work in progress." 45 This Internet-Draft will expire on May 5, 2020. 47 Copyright Notice 49 Copyright (c) 2019 IETF Trust and the persons identified as the 50 document authors. All rights reserved. 52 This document is subject to BCP 78 and the IETF Trust's Legal 53 Provisions Relating to IETF Documents 54 (https://trustee.ietf.org/license-info) in effect on the date of 55 publication of this document. Please review these documents 56 carefully, as they describe your rights and restrictions with respect 57 to this document. Code Components extracted from this document must 58 include Simplified BSD License text as described in Section 4.e of 59 the Trust Legal Provisions and are provided without warranty as 60 described in the Simplified BSD License. 62 Table of Contents 64 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 65 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4 66 3. Background . . . . . . . . . . . . . . . . . . . . . . . . . 5 67 4. Top Level Overview . . . . . . . . . . . . . . . . . . . . . 5 68 5. Inter-Link Protocol Overview . . . . . . . . . . . . . . . . 7 69 5.1. L3DL Ladder Diagram . . . . . . . . . . . . . . . . . . . 7 70 6. Transport Layer . . . . . . . . . . . . . . . . . . . . . . . 9 71 7. The Checksum . . . . . . . . . . . . . . . . . . . . . . . . 10 72 8. TLV PDUs . . . . . . . . . . . . . . . . . . . . . . . . . . 12 73 9. Logical Link Endpoint Identifier . . . . . . . . . . . . . . 13 74 10. HELLO . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 75 11. OPEN . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 76 12. ACK . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 77 12.1. Retransmission . . . . . . . . . . . . . . . . . . . . . 19 78 13. The Encapsulations . . . . . . . . . . . . . . . . . . . . . 19 79 13.1. The Encapsulation PDU Skeleton . . . . . . . . . . . . . 20 80 13.2. Encapsulaion Flags . . . . . . . . . . . . . . . . . . . 21 81 13.3. IPv4 Encapsulation . . . . . . . . . . . . . . . . . . . 21 82 13.4. IPv6 Encapsulation . . . . . . . . . . . . . . . . . . . 22 83 13.5. MPLS Label List . . . . . . . . . . . . . . . . . . . . 23 84 13.6. MPLS IPv4 Encapsulation . . . . . . . . . . . . . . . . 23 85 13.7. MPLS IPv6 Encapsulation . . . . . . . . . . . . . . . . 24 86 14. VENDOR - Vendor Extensions . . . . . . . . . . . . . . . . . 24 87 15. KEEPALIVE - Layer 2 Liveness . . . . . . . . . . . . . . . . 25 88 16. Layers 2.5 and 3 Liveness . . . . . . . . . . . . . . . . . . 26 89 17. The North/South Protocol . . . . . . . . . . . . . . . . . . 26 90 17.1. Use BGP-LS as Much as Possible . . . . . . . . . . . . . 27 91 17.2. Extensions to BGP-LS . . . . . . . . . . . . . . . . . . 27 92 18. Discussion . . . . . . . . . . . . . . . . . . . . . . . . . 27 93 18.1. HELLO Discussion . . . . . . . . . . . . . . . . . . . . 27 94 18.2. HELLO versus KEEPALIVE . . . . . . . . . . . . . . . . . 28 96 19. VLANs/SVIs/Sub-interfaces . . . . . . . . . . . . . . . . . . 28 97 20. Implementation Considerations . . . . . . . . . . . . . . . . 28 98 21. Security Considerations . . . . . . . . . . . . . . . . . . . 29 99 22. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 29 100 22.1. PDU Types . . . . . . . . . . . . . . . . . . . . . . . 29 101 22.2. Signature Type . . . . . . . . . . . . . . . . . . . . . 30 102 22.3. Flag Bits . . . . . . . . . . . . . . . . . . . . . . . 30 103 22.4. Error Codes . . . . . . . . . . . . . . . . . . . . . . 30 104 23. IEEE Considerations . . . . . . . . . . . . . . . . . . . . . 31 105 24. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 31 106 25. References . . . . . . . . . . . . . . . . . . . . . . . . . 31 107 25.1. Normative References . . . . . . . . . . . . . . . . . . 31 108 25.2. Informative References . . . . . . . . . . . . . . . . . 33 109 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 34 111 1. Introduction 113 The Massive Data Center (MDC) environment presents unusual problems 114 of scale, e.g. O(10,000) forwarding devices, while its homogeneity 115 presents opportunities for simple approaches. Approaches such as 116 Jupiter Rising [JUPITER] use a central controller to deal with 117 scaling, while BGP-SPF [I-D.ietf-lsvr-bgp-spf] provides massive 118 scale-out without centralization using a tried and tested scalable 119 distributed control plane, offering a scalable routing solution in 120 Clos [Clos0][Clos1] and similar environments. But BGP-SPF and 121 similar higher level device-spanning protocols, e.g. 122 [I-D.malhotra-bess-evpn-lsoe], need logical link state and addressing 123 data from the network to build the routing topology. They also need 124 prompt but prudent reaction to (logical) link failure. 126 Layer 3 Discovery and Liveness (L3DL) provides brutally simple 127 mechanisms for devices to 129 o Discover each other's unique endpoint identification, 131 o Discover mutually supported layer 3 encapsulations, e.g. IP/MPLS, 133 o Discover Layer 3 IP and/or MPLS addressing of interfaces of the 134 encapsulations, 136 o Present these data, using a very restricted profile of a BGP-LS 137 [RFC7752] API, to BGP-SPF which computes the topology and builds 138 routing and forwarding tables, 140 o Enable Layer 3 link liveness such as BFD, 142 o Provide Layer 2 keep-alive messages for session continuity, and 143 finally 145 o Provide for authenticity verification of protocol messages. 147 This protocol may be more widely applicable to a range of routing and 148 similar protocols which need layer 3 discovery and characterisation. 150 2. Terminology 152 Even though it concentrates on the inter-device layer, this document 153 relies heavily on routing terminology. The following attempts to 154 clarify the use of some possibly confusing terms: 156 ASN: Autonomous System Number [RFC4271], a BGP identifier for 157 an originator of Layer 3 routes, particularly BGP 158 announcements. 159 BGP-LS: A mechanism by which link-state and TE information can be 160 collected from networks and shared with external 161 components using the BGP routing protocol. See [RFC7752]. 162 BGP-SPF A hybrid protocol using BGP transport but a Dijkstra 163 Shortest Path First decision process. See 164 [I-D.ietf-lsvr-bgp-spf]. 165 Clos: A hierarchic subset of a crossbar switch topology commonly 166 used in data centers. 167 Datagram: The L3DL content of a single Layer 2 frame, sans Ethernet 168 framing. A full L3DL PDU may be packaged in multiple 169 Datagrams. 170 Encapsulation: Address Family Indicator and Subsequent Address 171 Family Indicator (AFI/SAFI). I.e. classes of layer 2.5 172 and 3 addresses such as IPv4, IPv6, MPLS, etc. 173 Frame: A Layer 2 Ethernet packet. 174 Link or Logical Link: A logical connection between two logical ports 175 on two devices. E.g. two VLANs between the same two ports 176 are two links. 177 LLEI: Logical Link Endpoint Identifier, the unique identifier of 178 one end of a logical link, see Section 9. 179 MAC Address: 48-bit Layer 2 addresses are assumed since they are 180 used by all widely deployed Layer 2 network technologies 181 of interest, especially Ethernet. See [IEEE.802_2001]. 182 MDC: Massive Data Center, commonly composed of thousands of Top 183 of Rack Switches (TORs). 184 MTU: Maximum Transmission Unit, the size in octets of the 185 largest packet that can be sent on a medium, see [RFC1122] 186 1.3.3. 187 PDU: Protocol Data Unit, an L3DL application layer message. A 188 PDU's content may need to be broken into multiple 189 Datagrams to make it through MTU or other restrictions. 190 RouterID: An 32-bit identifier unique in the current routing domain, 191 see [RFC6286]. 193 Session: An established, via OPEN PDUs, session between two L3DL 194 capable link end-points, 195 SPF: Shortest Path First, an algorithm for finding the shortest 196 paths between nodes in a graph; AKA Dijkstra's algorithm. 197 System Identifier: An eight octet ISO System Identifier a la 198 [RFC1629] System ID 199 TOR: Top Of Rack switch, aggregates the servers in a rack and 200 connects to aggregation layers of the Clos tree, AKA the 201 Clos spine. 202 ZTP: Zero Touch Provisioning gives devices initial addresses, 203 credentials, etc. on boot/restart. 205 3. Background 207 L3DL is primarily designed for a Clos type datacenter scale and 208 topology, but can accommodate richer topologies which contain 209 potential cycles. 211 While L3DL is designed for the MDC, there are no inherent reasons it 212 could not run on a WAN. The authentication and authorization needed 213 to run safely on a WAN need to be considered, and the appropriate 214 level of security options chosen. 216 L3DL assumes a new IEEE assigned EtherType (TBD). 218 The number of addresses of one Encapsulation type on an interface 219 link may be quite large given a TOR with tens of servers, each server 220 having a few hundred micro-services, resulting in an inordinate 221 number of addresses. And highly automated micro-service migration 222 can cause serious address prefix disaggregation, resulting in 223 interfaces with thousands of disaggregated prefixes. 225 Therefore the L3DL protocol is session oriented and uses incremental 226 announcement and withdrawal with session restart, a la BGP 227 ([RFC4271]). 229 4. Top Level Overview 231 o Devices discover each other on logical links 233 o Logical Link Endpoint Identifiers (LLEIs) are exchanged 235 o Layer 2 Liveness checks may be started 237 o Encapsulation data are exchanged and IP-Level Liveness checks 238 enabled 240 o A BGP-like upper layer protocol is assumed to use the identiiers 241 and encapsulation data to discover and build a topology database 243 +-------------------+ +-------------------+ +-------------------+ 244 | Device | | Device | | Device | 245 | | | | | | 246 |+-----------------+| |+-----------------+| |+-----------------+| 247 || || || || || || 248 || BGP-SPF <+---+> BGP-SPF <+---+> BGP-SPF || 249 || || || || || || 250 |+--------^--------+| |+--------^--------+| |+--------^--------+| 251 | | | | | | | | | 252 | | | | | | | | | 253 |+--------+--------+| |+--------+--------+| |+--------+--------+| 254 || Encapsulations || || Encapsulations || || Encapsulations || 255 || Addresses || || Addresses || || Addresses || 256 || L2 Liveness || || L2 Liveness || || L2 Liveness || 257 |+--------^--------+| |+--------^--------+| |+--------^--------+| 258 | | | | | | | | | 259 | | | | | | | | | 260 |+--------v--------+| |+--------v--------+| |+--------v--------+| 261 || || || || || || 262 ||Inter-Device PDUs<+---+>Inter-Device PDUs<+---+>Inter-Device PDUs|| 263 || || || || || || 264 |+-----------------+| |+-----------------+| |+-----------------+| 265 +-------------------+ +-------------------+ +-------------------+ 267 There are two protocols, the inter-device (left-right in the diagram) 268 per-link layer 3 discovery and the API to the upper level BGP-like 269 routing prototol (up-down in the above diagram): 271 o Inter-device PDUs are used to exchange device and logical link 272 identities and layer 2.5 (MPLS) and 3 identifiers (not payloads), 273 e.g. device IDs, port identities, VLAN IDs, Encapsulations, and IP 274 addresses. 276 o A Link Layer to BGP API presents these data up the stack to a BGP 277 protocol or an other device-spanning upper layer protocol, 278 presenting them using the BGP-LS BGP-like data format. 280 The upper layer BGP family routing protocols cross all the devices, 281 though they are not part of these L3DL protocols. 283 To simplify this document, Layer 2 framing is not shown. L3DL is 284 about layer 3. 286 5. Inter-Link Protocol Overview 288 Two devices discover each other and their respective identities by 289 sending multicast HELLO PDUs (Section 10). To assure discovery of 290 new devices coming up on a multi-link topology, devices on such a 291 topology, and only on a multi-link topology, send periodic HELLOs 292 forever, see Section 18.1. 294 Once a new device is recognized, both devices attempt to negotiate 295 and establish a session by sending unicast OPEN PDUs (Section 11) to 296 the source MAC addresses (plus VIDs if VLANs) of the received HELLOs. 297 Once a session is established through the OPEN exchange, the 298 Encapsulations (Section 13) configured on an end point may be 299 announced and modified. Note that these are only the encapsuation 300 and addresses configured on the announcing interface; though a 301 device's loopback and overlay interface(s) may also be announced. 302 When two devices on a link have compatible Encapsulations and 303 addresses, i.e. the same AFI/SAFI and the same subnet, the link is 304 announced via the BGP-LS API. 306 5.1. L3DL Ladder Diagram 308 The HELLO, Section 10, is a priming message sent on all configured 309 logical links. It is a small L3DL PDU encapsulated in an Ethernet 310 multicast frame with the simple goal of discovering the identities of 311 logical link endpoint(s) reachable from a Logical Link Endpoint, 312 Section 9. 314 The HELLO and OPEN, Section 11, PDUs, which are used to discover and 315 exchange detailed Logical Link Endpoint Identifiers, LLEIs, and the 316 ACK/ERROR PDU, are mandatory; other PDUs are optional; though at 317 least one encapsulation SHOULD be agreed at some point. 319 The following is a ladder-style diagram of the L3DL protocol 320 exchanges: 322 | HELLO | Logical Link Peer discovery 323 |---------------------------->| 324 | HELLO | Mandatory 325 |<----------------------------| 326 | | 327 | | 328 | OPEN | MACs, IDs, etc. 329 |---------------------------->| 330 | ACK | 331 |<----------------------------| 332 | | 333 | OPEN | Mandatory 334 |<----------------------------| 335 | ACK | 336 |---------------------------->| 337 | | 338 | | 339 | Interface IPv4 Addresses | Interface IPv4 Addresses 340 |---------------------------->| Optional 341 | ACK | 342 |<----------------------------| 343 | | 344 | Interface IPv4 Addresses | 345 |<----------------------------| 346 | ACK | 347 |---------------------------->| 348 | | 349 | | 350 | Interface IPv6 Addresses | Interface IPv6 Addresses 351 |---------------------------->| Optional 352 | ACK | 353 |<----------------------------| 354 | | 355 | Interface IPv6 Addresses | 356 |<----------------------------| 357 | ACK | 358 |---------------------------->| 359 | | 360 | | 361 | Interface MPLSv4 Labels | Interface MPLSv4 Labels 362 |---------------------------->| Optional 363 | ACK | 364 |<----------------------------| 365 | | 366 | Interface MPLSv4 Labels | Interface MPLSv4 Labels 367 |<----------------------------| Optional 368 | ACK | 369 |---------------------------->| 370 | | 371 | | 372 | Interface MPLSv6 Labels | Interface MPLSv6 Labels 373 |---------------------------->| Optional 374 | ACK | 375 |<----------------------------| 376 | | 377 | Interface MPLSv6 Labels | Interface MPLSv6 Labels 378 |<----------------------------| Optional 379 | ACK | 380 |---------------------------->| 381 | | 382 | | 383 | L3DL KEEPALIVE | Layer 2 Liveness 384 |---------------------------->| Optional 385 | L3DL KEEPALIVE | 386 |<----------------------------| 388 6. Transport Layer 390 L3DL PDUs are carried by a simple transport layer which allows long 391 PDUs to occupy many Ethernet frames. The L3DL content of a single 392 Ethernet frame, exclusive of Ethernet framing data, is referred to as 393 a Datagram. 395 The L3DL Transport Layer encapsulates each Datagram using a common 396 transport header. 398 If a PDU does not fit in a single datagram, it is broken into 399 multiple Datagrams and reassembled by the receiver a la [RFC0791] 400 Section 2.3 Fragmentation. 402 Should a PDU need to be retransmitted, it MUST BE sent as the 403 identical Datagram set as the original transmission. The 404 Transmission Sequence Number informs the receiver that it is the same 405 PDU. 407 0 1 2 3 408 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 409 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 410 | Version | Transmission Sequence Number |L| ~ 411 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 412 ~ Datagram Number | Datagram Length | 413 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 414 | Checksum | 415 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 416 | Payload... | 417 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 419 The fields of the L3DL Transport Header are as follows: 421 Version: Seven-bit Version number of the protocol, currently 0. 422 Values other than 0 MUST BE treated as an error. The protocol 423 version needs to be in one and only one place, so it is in the 424 datagram as opposed to, for example, the PDU header. 426 L: A bit that set to one if this Datagram is the last Datagram of the 427 PDU. For a PDU which fits in only one Datagram, it is set to one. 428 Note that this is the inverse of the marking technique used by 429 [RFC0791]. 431 Transmission Sequence Number: A 16-bit strictly increasing unsigned 432 integer identifying this PDU, possibly across retransmissions, 433 that wraps from 2^16-1 to 0. The initial value is arbitrary. See 434 [RFC1982] on DNS Serial Number Arithmetic for too much detail on 435 comparison and incrementing a wrapping sequence number. 437 Datagram Number: A monotonically increasing 24-bit value which 438 starts at zero for each PDU. This is used to reassemble frames 439 into PDUs a la [RFC0791] Section 2.3. Note that this limits an 440 L3DL PDU to 2^24 frames. 442 Datagram Length: Total number of octets in the Datagram including 443 all payloads and fields. Note that this limits a datagram to 2^16 444 octets; though Ethernet framing is likely to impose a smaller 445 limit. 447 Checksum: A 32 bit hash over the Datagram to detect bit flips, see 448 Section 7. 450 If a Datagram fails checksum verification, the datagram is invalid 451 and should be silently discarded. The sender will retransmit the 452 PDU, and the receiver can assmble it. 454 Payload: The PDU being transported or a fragment thereof. 456 To avoid the need for a receiver to reassemble two PDUs at the same 457 time, a sender MUST NOT send a subsequent PDU when a PDU is already 458 in flight and not yet acknowledged; assuming it is an ACKed PDU Type. 460 7. The Checksum 462 There is a reason conservative folk use a checksum in UDP. And as 463 many operators stretch to jumbo frames (over 1,500 octets) longer 464 checksums are the prudent approach. 466 For the purpose of computing a checksum, the checksum field itself is 467 assumed to be zero. 469 The following code describes the suggested algorithm. 471 Sum up 32-bit unsigned ints in a 64-bit long, then take the high- 472 order section, shift it right, rotate, add it in, repeat until zero. 474 475 #include 476 #include 478 /* The F table from Skipjack, and it would work for the S-Box. */ 479 static const uint8_t sbox[256] = { 480 0xa3,0xd7,0x09,0x83,0xf8,0x48,0xf6,0xf4,0xb3,0x21,0x15,0x78, 481 0x99,0xb1,0xaf,0xf9,0xe7,0x2d,0x4d,0x8a,0xce,0x4c,0xca,0x2e, 482 0x52,0x95,0xd9,0x1e,0x4e,0x38,0x44,0x28,0x0a,0xdf,0x02,0xa0, 483 0x17,0xf1,0x60,0x68,0x12,0xb7,0x7a,0xc3,0xe9,0xfa,0x3d,0x53, 484 0x96,0x84,0x6b,0xba,0xf2,0x63,0x9a,0x19,0x7c,0xae,0xe5,0xf5, 485 0xf7,0x16,0x6a,0xa2,0x39,0xb6,0x7b,0x0f,0xc1,0x93,0x81,0x1b, 486 0xee,0xb4,0x1a,0xea,0xd0,0x91,0x2f,0xb8,0x55,0xb9,0xda,0x85, 487 0x3f,0x41,0xbf,0xe0,0x5a,0x58,0x80,0x5f,0x66,0x0b,0xd8,0x90, 488 0x35,0xd5,0xc0,0xa7,0x33,0x06,0x65,0x69,0x45,0x00,0x94,0x56, 489 0x6d,0x98,0x9b,0x76,0x97,0xfc,0xb2,0xc2,0xb0,0xfe,0xdb,0x20, 490 0xe1,0xeb,0xd6,0xe4,0xdd,0x47,0x4a,0x1d,0x42,0xed,0x9e,0x6e, 491 0x49,0x3c,0xcd,0x43,0x27,0xd2,0x07,0xd4,0xde,0xc7,0x67,0x18, 492 0x89,0xcb,0x30,0x1f,0x8d,0xc6,0x8f,0xaa,0xc8,0x74,0xdc,0xc9, 493 0x5d,0x5c,0x31,0xa4,0x70,0x88,0x61,0x2c,0x9f,0x0d,0x2b,0x87, 494 0x50,0x82,0x54,0x64,0x26,0x7d,0x03,0x40,0x34,0x4b,0x1c,0x73, 495 0xd1,0xc4,0xfd,0x3b,0xcc,0xfb,0x7f,0xab,0xe6,0x3e,0x5b,0xa5, 496 0xad,0x04,0x23,0x9c,0x14,0x51,0x22,0xf0,0x29,0x79,0x71,0x7e, 497 0xff,0x8c,0x0e,0xe2,0x0c,0xef,0xbc,0x72,0x75,0x6f,0x37,0xa1, 498 0xec,0xd3,0x8e,0x62,0x8b,0x86,0x10,0xe8,0x08,0x77,0x11,0xbe, 499 0x92,0x4f,0x24,0xc5,0x32,0x36,0x9d,0xcf,0xf3,0xa6,0xbb,0xac, 500 0x5e,0x6c,0xa9,0x13,0x57,0x25,0xb5,0xe3,0xbd,0xa8,0x3a,0x01, 501 0x05,0x59,0x2a,0x46 502 }; 504 /* non-normative example C code, constant time even */ 506 uint32_t sbox_checksum_32(const uint8_t *b, const size_t n) 507 { 508 uint32_t sum[4] = {0, 0, 0, 0}; 509 uint64_t result = 0; 510 for (size_t i = 0; i < n; i++) 511 sum[i & 3] += sbox[*b++]; 512 for (int i = 0; i < sizeof(sum)/sizeof(*sum); i++) 513 result = (result << 8) + sum[i]; 514 result = (result >> 32) + (result & 0xFFFFFFFF); 515 result = (result >> 32) + (result & 0xFFFFFFFF); 516 return (uint32_t) result; 517 } 518 520 8. TLV PDUs 522 The basic L3DL application layer PDU is a typical TLV (Type Length 523 Value) PDU. It includes a signature to provide optional integrity 524 and authentication. It may be broken into multiple Datagrams, see 525 Section 6. 527 0 1 2 3 528 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 529 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 530 | PDU Type | Payload Length ~ 531 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 532 ~ | Payload ... | 533 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 534 | Sig Type | Signature Length | ~ 535 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + 536 ~ Signature ~ 537 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 539 The fields of the basic L3DL header are as follows: 541 PDU Type: An integer differentiating PDU payload types. See 542 Section 22.1. 544 Payload Length: Total number of octets in the Payload field. 546 Payload: The application layer content of the L3DL PDU. 548 Sig Type: The type of the Signature, see Section 22.2. Type 0, a 549 null signature, is defined in this document. 551 Sig Type 0 indicates a null Signature. For a trivial PDU such as 552 KEEPALIVE, the underlying Datagram checksum may be sufficient for 553 integrity, though it lacks authenticity. 555 Other Sig Types may be defined in other documents, cf. 556 [I-D.ymbk-lsvr-l3dl-signing]. 558 Signature Length: The length of the Signature, possibly including 559 padding, in octets. If Sig Type is 0, Signature Length MUST BE 0. 561 Signature: The result of running the signature algorithm specified 562 in Sig Type over all octets of the PDU except for the Signature 563 itself. 565 9. Logical Link Endpoint Identifier 567 L3DL discovers neighbors on logical links and establishes sessions 568 between the two ends of all consenting discovered logical links. A 569 logical link is described by a pair of Logical Link Endpoint 570 Identifiers, LLEIs. 572 An LLEI is a variable length descriptor which could be an ASN, a 573 classic RouterID, a catenation of the two, an eight octet ISO System 574 Identifier [RFC1629], or any other identifier unique to a single 575 logical link endpoint in the topology. 577 An L3DL deployment will choose and define an LLEI which suits its 578 needs, simple or complex. Examples of two extremes follow: 580 A simplistic view of a link between two devices is two ports, 581 identified by unique MAC addresses, carrying a layer 3 protocol 582 conversation. In this case, the MAC addresses might suffice for the 583 LLEIs. 585 Unfortunately, things can get more complex. Multiple VLANs can run 586 between those two MAC addresses. In practice, many real devices use 587 the same MAC address on multiple ports and/or sub-interfaces. 589 Therefore, in the general circumstance, a fully described LLEI might 590 be as follows: 592 0 1 2 3 593 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 594 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 595 | | 596 + System Identifier + 597 | | 598 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 599 | ifIndex | 600 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 602 System Identifier, a la [RFC1629], is an eight octet identifier 603 unique in the entire operational space. Routers and switches usually 604 have internal MAC Addresses which can be padded with high order zeros 605 and used if no System ID exists on the device. If no unique 606 identifier is burned into a device, the local L3DL configuration 607 SHOULD create and assign a unique one, likely by configuration. 609 ifIndex is the SNMP identifier of the (sub-)interface, see [RFC1213]. 610 This uniquely identifies the port. 612 For a layer 3 tagged sub-interface or a VLAN/SVI interface, Ifindex 613 is that of the logical sub-interface, so no further disambiguation is 614 needed. 616 L3DL PDUs learned over VLAN-ports may be interpreted by upper layer-3 617 routing protocols as being learned on the corresponding layer-3 SVI 618 interface for the VLAN. 620 LLEIs are big-endian. 622 10. HELLO 624 The HELLO PDU is unique in that it is encapsulated in a multicast 625 Ethernet frame. It solicits response(s) from other LLEI(s) on the 626 link. See Section 18.1 for why multicast is used. The destination 627 multicast MAC Addressees to be used MUST be one of the following, See 628 Clause 9.2.2 of [IEEE802-2014]: 630 01-80-C2-00-00-0E: Nearest Bridge = Propagation constrained to a 631 single physical link; stopped by all types of bridges (including 632 MPRs (media converters)). This SHOULD BE used when the link is 633 known to be a simple point to point link. 634 To Be Assigned: When a switch receives a frame with a multicast 635 destination MAC it does not recognize, it forwards to all ports. 636 This destination MAC is to be sent when the interface is known to 637 be connected to a switch. See Section 23. This SHOULD BE used 638 when the link may be a multi-point link. 640 All other L3DL PDUs are encapsulated in unicast frames, as the peer's 641 destination MAC address is known after the HELLO exchange. 643 When an interface is turned up on a device, it SHOULD issue a HELLO 644 if it is to participate in L3DL sessions. 646 If a constrained Nearest Bridge destination address has been 647 configured for a point-to-point interface, see above, then the HELLO 648 SHOULD NOT be repeated once a session has been created by an exchange 649 of OPENs. 651 If the configured destination address is one that is propagated by 652 switches, the HELLO SHOULD be repeated at a configured interval, with 653 a default of 60 seconds. This allows discovery by new devices which 654 come up on the layer-2 mesh. 656 0 1 2 3 657 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 658 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 659 | PDU Type = 0 | Payload Length = 0 ~ 660 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 661 ~ | Sig Type = 0 | Signature Length = 0 | 662 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 664 If more than one device responds, one adjacency is formed for each 665 unique source LLEI response. L3DL treats each adjacency as a 666 separate logical link. 668 When a HELLO is received from a source MAC address (plus VID if VLAN) 669 with which there is no established L3DL session, the receiver SHOULD 670 respond by sending an OPEN PDU to the source MAC address (plus VID). 671 The two devices establish an L3DL session by exchanging OPEN PDUs. 673 If a HELLO is received from a MAC address with which there is an 674 established session, the HELLO should be dropped. 676 The Payload Length is zero as there is no payload. 678 HELLO PDUs can not be signed as keying material has yet to be 679 exchanged. Hence the signature MUST always be the null type. 681 11. OPEN 683 Each device has learned the other's MAC Address from the HELLO 684 exchange, see Section 10. Therefore the OPEN and all subsequent PDUs 685 MUST BE unicast, as opposed to the HELLO's multicast frame. 687 0 1 2 3 688 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 689 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 690 | PDU Type = 1 | Payload Length ~ 691 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 692 ~ | Nonce ~ 693 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 694 ~ | LLEI Length | My LLEI | 695 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-~ 696 ~ | AttrCount | ~ 697 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 698 ~ Attribute List ... | Auth Type | Key Length ~ 699 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 700 ~ | Key ... | 701 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 702 | Serial Number | 703 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 704 | Sig Type | Signature Length | Signature ... | 705 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 707 The Payload Length is the number of octets in all fields of the PDU 708 from the Nonce through the Serial Number, not including the three 709 final signature fields. 711 The Nonce enables detection of a duplicate OPEN PDU. It SHOULD be 712 either a random number or a high resolution timestamp. It is needed 713 to prevent session closure due to a repeated OPEN caused by a race or 714 a dropped or delayed ACK. 716 My LLEI is the sender's LLEI, see Section 9. 718 AttrCount is the number of attributes in the Attribute List. 719 Attributes are single octets the semantics of which are operator- 720 defined. 722 A node may have zero or more operator-defined attributes, e.g.: 723 spine, leaf, backbone, route reflector, arabica, ... 725 Attribute syntax and semantics are local to an operator or 726 datacenter; hence there is no global registry. Nodes exchange their 727 attributes only in the OPEN PDU. 729 Auth Type is the Signature algorithm suite, see Section 8. 731 Key Length is a 16-bit field denoting the length in octets of the Key 732 itself, not including the Auth Type or the Key Length. If the Auth 733 Type is zero, then the Key Length MUST also be zero, and there MUST 734 BE no Key data. 736 The Key is specific to the operational environment. A failure to 737 authenticate is a failure to start the L3DL session, an ERROR PDU 738 MUST BE sent (Error Code 3), and HELLOs MUST be restarted. 740 The Serial Number is that of the last received and processed PDU. 741 This allows a receiver sending an OPEN to tell the sender that the 742 receiver wants to resume a session and the sender only needs to send 743 data more recent than the Serial Number. If this OPEN is not trying 744 to restart a lost session, the Serial Number MUST BE set to zero. 746 The Signature fields are described in Section 8 and in an asymmetric 747 key environment serve as a proof of possession of the signing auth 748 data by the sender. 750 Once two logical link endpoints know each other, and have ACKed each 751 other's OPEN PDUs, Layer 2 KEEPALIVEs (see Section 15) MAY be started 752 to ensure Layer 2 liveness and keep the session semantics alive. The 753 timing and acceptable drop of KEEPALIVE PDUs are discussed in 754 Section 15. 756 If a sender of OPEN does not receive an ACK of the OPEN PDU, then 757 they MUST resend the same OPEN PDU, with the same Nonce. Resending 758 an unacknowledged OPEN PDU, like other ACKed PDUs, SHOULD use 759 exponential back-off, see [RFC1122]. 761 If a properly authenticated OPEN arrives with a new Nonce from an 762 LLEI with which the receiving logical link endpoint believes it 763 already has an L3DL session (OPENs have already been exchanged), and 764 the Serial Number in the OPEN is non-zero, the receiver SHOULD 765 establish a new session by sending an OPEN with the Serial Number of 766 the last data it received. Each party MUST resume sending 767 encapsulations etc. subsequent to the other party's Sequence Number. 768 And each MUST retain all previously discovered encapsulation and 769 other data. 771 If a properly authenticated OPEN arrives with a new Nonce from an 772 LLEI with which the receiving logical link endpoint believes it 773 already has an L3DL session (OPENs have already been exchanged), and 774 the Serial Number in the OPEN is zero, then the receiver MUST assume 775 that the sending LLEI or entire device has been reset. All 776 previously discovered encapsulation data MUST NOT be kept and MUST BE 777 withdrawn via the BGP-LS API and the recipient MUST respond with a 778 new OPEN. 780 12. ACK 782 The ACK PDU acknowledges receipt of a PDU and reports any error 783 condition which might have been raised. 785 0 1 2 3 786 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 787 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 788 | PDU Type = 3 | Payload Length = 5 ~ 789 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 790 ~ | ACKed PDU | EType | Error Code | 791 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 792 | Error Hint | Sig Type |Signature Leng.~ 793 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 794 ~ | Signature ... | 795 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 797 The ACK acknowledges receipt of an OPEN, Encapsulation, VENDOR PDU, 798 etc. 800 The ACKed PDU is the PDU Type of the PDU being acknowledged, e.g., 801 OPEN, one of the Encapsulations, etc. 803 If there was an error processing the received PDU, then the EType is 804 non-zero. If the EType is zero, Error Code and Error Hint MUST also 805 be zero. 807 A non-zero EType is the receiver's way of telling the PDU's sender 808 that the receiver had problems processing the PDU. The Error Code 809 and Error Hint will tell the sender more detail about the error. 811 The decimal value of EType gives a strong hint how the receiver 812 sending the ACK believes things should proceed: 814 0 - No Error, Error Code and Error Hint MUST be zero 815 1 - Warning, something not too serious happened, continue 816 2 - Session should not be continued, try to restart 817 3 - Restart is hopeless, call the operator 818 4-15 - Reserved 820 The Error Codes, noting protocol failures, are listed in 821 Section 22.4. Someone stuck in the 1990s might think the catenation 822 of EType and Error Code as an echo of 0x1zzz, 0x2zzz, etc. They 823 might be right; or not. 825 The Error Hint, an arbitrary 16 bits, is any additional data the 826 sender of the error PDU thinks will help the recipient or the 827 debugger with the particular error. 829 The Signature fields are described in Section 8. 831 12.1. Retransmission 833 If a PDU sender expects an ACK, e.g. for an OPEN, an Encapsulation, a 834 VENDOR PDU, etc., and does not receive the ACK for a configurable 835 time (default one second), and the interface is live at layer 2, the 836 sender resends the PDU using exponential back-off, see [RFC1122]. 837 This cycle MAY be repeated a configurable number of times (default 838 three) before it is considered a failure. The session MAY BE 839 considered closed in case of this ACK failure. 841 If the link is broken at layer 2, retransmission MAY BE retried when 842 the link is restored. 844 13. The Encapsulations 846 Once the devices know each other's LLEIs, know each other's upper 847 layer (L2.5 and L3) identities, have means to ensure link state, 848 etc., the L3DL session is considered established, and the devices 849 SHOULD exchange L3 interface encapsulations, L3 addresses, and L2.5 850 labels. 852 The Encapsulation types the peers exchange may be IPv4 853 (Section 13.3), IPv6 (Section 13.4), MPLS IPv4 (Section 13.6), MPLS 854 IPv6 (Section 13.7), and/or possibly others not defined here. 856 The sender of an Encapsulation PDU MUST NOT assume that the peer is 857 capable of the same Encapsulation Type. An ACK (Section 12) merely 858 acknowledges receipt. Only if both peers have sent the same 859 Encapsulation Type is it safe for Layer 3 protocols to assume that 860 they are compatible for that type. 862 A receiver of an encapsulation might recognize an addressing 863 conflict, such as both ends of the link trying to use the same 864 address. In this case, the receiver SHOULD respond with an error 865 (Error Code 2) ACK. As there may be other usable addresses or 866 encapsulations, this error might log and continue, letting an upper 867 layer topology builder deal with what works. 869 Further, to consider a logical link of a type to formally be 870 established so that it may be pushed up to upper layer protocols, the 871 addressing for the type must be compatible, e.g. on the same IP 872 subnet. 874 13.1. The Encapsulation PDU Skeleton 876 The header for all encapsulation PDUs is as follows: 878 0 1 2 3 879 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 880 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 881 | PDU Type | Payload Length ~ 882 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 883 ~ | Count | 884 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 885 | Serial Number | 886 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 887 | Encapsulation List... | Sig Type | 888 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 889 | Signature Length | Signature ... | 890 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 892 An Encapsulation PDU describes zero or more addresses of the 893 encapsulation type. 895 The 24-bit Count is the number of Encapsulations in the Encapsulation 896 list. 898 The Serial Number is a monotonically increasing 32-bit value 899 representing the sender's state in time. It may be an integer, a 900 timestamp, etc. On session restart (new OPEN), a receiver MAY send 901 the last received Session Number to tell the sender to only send 902 newer data. 904 If a sender has multiple links on the same interface, separate state: 905 data, ACKs, etc. must be kept for each peer session. 907 Over time, multiple Encapsulation PDUs may be sent for an interface 908 as configuration changes. 910 If the length of an Encapsulation PDU exceeds the Datagram size limit 911 on media, the PDU is broken into multiple Datagrams. See Section 8. 913 The Signature fields are described in Section 8. 915 The Receiver MUST acknowledge the Encapsulation PDU with a Type=3, 916 ACK PDU (Section 12) with the Encapsulation Type being that of the 917 encapsulation being announced, see Section 12. 919 If the Sender does not receive an ACK in a configurable interval 920 (default one second), and the interface is live at layer 2, they 921 SHOULD retransmit. After a user configurable number of failures 922 (default three), the L3DL session should be considered dead and the 923 OPEN process SHOULD be restarted. 925 If the link is broken at layer 2, retransmission MAY BE retried if 926 data have not changed in the interim. 928 13.2. Encapsulaion Flags 930 The Encapsulation Flags are a sequence of bit fields as follows: 932 0 1 2 3 4 ... 7 933 +------------+------------+------------+------------+------------+ 934 | Ann/With | Primary | Under/Over | Loopback | Reserved ..| 935 +------------+------------+------------+------------+------------+ 937 Each encapsulation in an Encapsulation PDU of Type T may announce new 938 and/or withdraw old encapsulations of Type T. It indicates this with 939 the Ann/With Encapsulation Flag, Announce == 1, Withdraw == 0. 941 Each Encapsulation interface address in an Encapsulation PDU is 942 either a new encapsulation be announced (Ann/With == 1) (yes, a la 943 BGP) or requests one be withdrawn (Ann/With == 0). Adding an 944 encapsulation which already exists SHOULD raise an Announce/Withdraw 945 Error (see Section 22.4); the EType SHOULD be 2, suggesting a session 946 restart (see Section 12 so all encapsulations will be resent. 948 If an LLEI has multiple addresses for an encapsulation type, one and 949 only one address MAY be marked as primary (Primary Flag == 1) for 950 that Encapsulation Type. 952 An Encapsulation interface address in an Encapsulation PDU MAY be 953 marked as a loopback, in which case the Loopback bit is set. 954 Loopback addresses are generally not seen directly on an external 955 interface. One or more loopback addresses MAY be exposed by 956 configuration on one or more L3DL speaking external interfaces, e.g. 957 for iBGP peering. They SHOULD be marked as such, Loopback Flag == 1. 959 Each Encapsulation interface address in an Encapsulation PDU is that 960 of the direct 'underlay interface (Under/Over == 1), or an 'overlay' 961 address (Under/Over == 0), likely that of a VM or container guest 962 bridged or configured on to the interface already having an underlay 963 address. 965 13.3. IPv4 Encapsulation 967 The IPv4 Encapsulation describes a device's ability to exchange IPv4 968 packets on one or more subnets. It does so by stating the 969 interface's addresses and the corresponding prefix lengths. 971 0 1 2 3 972 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 973 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 974 | PDU Type = 4 | Payload Length ~ 975 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 976 ~ | Count | 977 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 978 | Serial Number | 979 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 980 | Encaps Flags | IPv4 Address ~ 981 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 982 ~ | PrefixLen | more ... | Sig Type | 983 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 984 | Signature Length | Signature ... | 985 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 987 The 24-bit Count is the sum of the number of IPv4 Encapsulations 988 being announced and/or withdrawn. 990 13.4. IPv6 Encapsulation 992 The IPv6 Encapsulation describes a logical link's ability to exchange 993 IPv6 packets on one or more subnets. It does so by stating the 994 interface's addresses and the corresponding prefix lengths. 996 0 1 2 3 997 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 998 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 999 | PDU Type = 5 | Payload Length ~ 1000 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1001 ~ | Count | 1002 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1003 | Serial Number | 1004 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1005 | Encaps Flags | | 1006 +-+-+-+-+-+-+-+-+ + 1007 | | 1008 + + 1009 | | 1010 + + 1011 | IPv6 Address | 1012 + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1013 | | PrefixLen | more ... | Sig Type | 1014 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1015 | Signature Length | Signature ... | 1016 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1017 The 24-bit Count is the sum of the number of IPv6 Encapsulations 1018 being announced and/or withdrawn. 1020 13.5. MPLS Label List 1022 As an MPLS enabled interface may have a label stack, see [RFC3032], a 1023 variable length list of labels is needed. These are the labels the 1024 sender will accept for the prefix to which the list is attached. 1026 0 1 2 3 1027 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1028 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1029 | Label Count | Label | Exp |S| 1030 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1031 | Label | Exp |S| more ... | 1032 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1034 A Label Count of zero is an implicit withdraw of all labels for that 1035 prefix on that interface. 1037 13.6. MPLS IPv4 Encapsulation 1039 The MPLS IPv4 Encapsulation describes a logical link's ability to 1040 exchange labeled IPv4 packets on one or more subnets. It does so by 1041 stating the interface's addresses the corresponding prefix lengths, 1042 and the corresponding labels which will be accepted fpr each address. 1044 0 1 2 3 1045 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1046 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1047 | PDU Type = 6 | Payload Length ~ 1048 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1049 ~ | Count | 1050 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1051 | Serial Number | 1052 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1053 | Encaps Flags | MPLS Label List ... | ~ 1054 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1055 ~ IPv4 Address | PrefixLen | 1056 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1057 | more ... | Sig Type | Signature Length | 1058 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1059 | Signature | 1060 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1062 The 24-bit Count is the sum of the number of MPLSv4 Encapsulation 1063 being announced and/or withdrawns. 1065 13.7. MPLS IPv6 Encapsulation 1067 The MPLS IPv6 Encapsulation describes a logical link's ability to 1068 exchange labeled IPv6 packets on one or more subnets. It does so by 1069 stating the interface's addresses, the corresponding prefix lengths, 1070 and the corresponding labels which will be accepted for each address. 1072 0 1 2 3 1073 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1074 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1075 | PDU Type = 7 | Payload Length ~ 1076 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1077 ~ | Count | 1078 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1079 | Serial Number | 1080 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1081 | Encaps Flags | MPLS Label List ... | | 1082 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + 1083 | | 1084 + + 1085 | | 1086 + + 1087 | IPv6 Address | 1088 + +-+-+-+-+-+-+-+-+ 1089 | | Prefix Len | 1090 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1091 | more ... | Sig Type | Signature Length | 1092 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1093 | Signature ... | 1094 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1096 The 24-bit Count is the sum of the number of MPLSv6 Encapsulations 1097 being announced and/or withdrawn. 1099 14. VENDOR - Vendor Extensions 1100 0 1 2 3 1101 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1102 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1103 | PDU Type = 255| Payload Length ~ 1104 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1105 ~ | Serial Number ~ 1106 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1107 ~ | Enterprise Number | 1108 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1109 | Ent Type | Enterprise Data ... ~ 1110 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1111 ~ | Sig Type | Signature Length | 1112 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1113 | Signature ... | 1114 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1116 Vendors or enterprises may define TLVs beyond the scope of L3DL 1117 standards. This is done using a Private Enterprise Number [IANA-PEN] 1118 followed by Enterprise Data in a format defined for that Enterprise 1119 Number and Ent Type. 1121 Ent Type allows a VENDOR PDU to be sub-typed in the event that the 1122 vendor/enterprise needs multiple PDU types. 1124 As with Encapsulation PDUs, a receiver of a VENDOR PDU MUST respond 1125 with an ACK or an ERROR PDU. Similarly, a VENDOR PDU MUST only be 1126 sent over an open session. 1128 15. KEEPALIVE - Layer 2 Liveness 1130 0 1 2 3 1131 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1132 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1133 | PDU Type = 2 | Payload Length = 0 ~ 1134 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1135 ~ | Sig Type = 0 | Signature Length = 0 | 1136 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1138 L3DL devices SHOULD beacon frequent Layer 2 KEEPALIVE PDUs to ensure 1139 session continuity. The inter-KEEPALIVE interval is configurable, 1140 with a default of ten seconds. A receiver may choose to ignore 1141 KEEPALIVE PDUs. 1143 An operational deployment MUST BE configured whether to use 1144 KEEPALIVEs or not, either globally, or as finely as to per-link 1145 granularity. Disagreement MAY result in repeated session failure and 1146 reestablishment. 1148 KEEPALIVEs SHOULD be beaconed at a configured frequency. One per 1149 second is the default. Layer 3 liveness, such as BFD, may be more 1150 (or less) aggressive. 1152 When a sender transmits a PDU which is not a KEEPALIVE, the sender 1153 SHOULD reset the KEEPALIVE timer. I.e. sending any PDU acts as a 1154 keepalive. Once the last fragment has been sent, the KEEPALIVE timer 1155 SHOULD BE restarted. Do not wait for the ACK. 1157 If a KEEPALIVE or other PDUs have not been received from a peer with 1158 which a receiver has an open session for a configurable time (default 1159 30 seconds), the link SHOULD BE presumed down. The devices MAY keep 1160 configuration state and restore it without retransmission if no data 1161 have changed. Otherwise, a new session SHOULD BE established and new 1162 Encapsulation PDUs exchanged. 1164 16. Layers 2.5 and 3 Liveness 1166 Layer 2 liveness may be continuously tested by KEEPALIVE PDUs, see 1167 Section 15. As layer 2.5 or layer 3 connectivity could still break, 1168 liveness above layer 2 MAY be frequently tested using BFD ([RFC5880]) 1169 or a similar technique. 1171 This protocol assumes that one or more Encapsulation addresses may be 1172 used to ping, run BFD, or whatever the operator configures. 1174 17. The North/South Protocol 1176 Thus far, a one-hop point-to-point logical link discovery protocol 1177 has been defined. 1179 The devices know their unique LLEIs and know the unique peer LLEIs 1180 and Encapsulations on each logical link interface. 1182 Full topology discovery is not appropriate at the L3DL layer, so 1183 Dijkstra a la IS-IS etc. is assumed to be done by higher level 1184 protocols such as BGP-SPF. 1186 Therefore the LLEIs, link Encapsulations, and state changes are 1187 pushed North via a small subset of the BGP-LS API. The upper layer 1188 routing protocol(s), e.g. BGP-SPF, learn and maintain the topology, 1189 run Dijkstra, and build the routing database(s). 1191 For example, if a neighbor's IPv4 Encapsulation address changes, the 1192 devices seeing the change push that change Northbound. 1194 17.1. Use BGP-LS as Much as Possible 1196 BGP-LS [RFC7752] defines BGP-like Datagrams describing logical link 1197 state (links, nodes, link prefixes, and many other things), and a new 1198 BGP path attribute providing Northbound transport, all of which can 1199 be ingested by upper layer protocols such as BGP-SPF; see Section 4 1200 of [I-D.ietf-lsvr-bgp-spf]. 1202 For IPv4 links, TLVs 259 and 260 are used. For IPv6 links, TLVs 261 1203 and 262. If there are multiple addresses on a link, multiple TLV 1204 pairs are pushed North, having the same ID pairs. 1206 17.2. Extensions to BGP-LS 1208 The Northbound protocol needs a few minor extensions to BGP-LS. 1209 Luckily, others have needed the same extensions. 1211 Similarly to BGP-SPF, the BGP protocol is used in the Protocol-ID 1212 field specified in table 1 of 1213 [I-D.ietf-idr-bgpls-segment-routing-epe]. The local and remote node 1214 descriptors for all NLRI are the IDs described in Section 11. This 1215 is equivalent to an adjacency SID or a node SID if the address is a 1216 loopback address. 1218 Label Sub-TLVs from [I-D.ietf-idr-bgp-ls-segment-routing-ext] 1219 Section 2.1.1, are used to associate one or more MPLS Labels with a 1220 link. 1222 18. Discussion 1224 This section explores some trade-offs taken and some considerations. 1226 18.1. HELLO Discussion 1228 A device with multiple Layer 2 interfaces, traditionally called a 1229 switch, may be used to forward frames and therefore packets from 1230 multiple devices to one logical interface (LLEI), I, on an L3DL 1231 speaking device. Interface I could discover a peer J across the 1232 switch. Later, a prospective peer K could come up across the switch. 1233 If I was not still sending and listening for HELLOs, the potential 1234 peering with K could not be discovered. Therefore, on multi-link 1235 interfaces, L3DL MUST continue to send HELLOs as long as they are 1236 turned up. 1238 18.2. HELLO versus KEEPALIVE 1240 Both HELLO and KEEPALIVE are periodic. KEEPALIVE might be eliminated 1241 in favor of keeping only HELLOs. But KEEPALIVEs are unicast, and 1242 thus less noisy on the network, especially if HELLO is configured to 1243 transit layer-2-only switches, see Section 18.1. 1245 19. VLANs/SVIs/Sub-interfaces 1247 One can think of the protocol as an instance (i.e. state machine) 1248 which runs on each logical link of a device. 1250 As the upper routing layer must view VLAN topologies as separate 1251 graphs, L3DL treats VLAN ports as separate links. 1253 L3DL PDUs learned over VLAN-ports may be interpreted by upper layer-3 1254 routing protocols as being learned on the corresponding layer-3 SVI 1255 interface for the VLAN. 1257 As Sub-Interfaces each have their own LLIEs, they act as separate 1258 interfaces, forming their own links. 1260 20. Implementation Considerations 1262 An implementation SHOULD provide the ability to configure each 1263 logical interface as L3DL speaking or not. 1265 An implementation SHOULD provide the ability to configure whether 1266 HELLOs on an L3DL enabled interface send Nearest Bridge or the MAC 1267 which is propagated by switches from that interface; see Section 10. 1269 An implementation SHOULD provide the ability to distribute one or 1270 more loopback addresses or interfaces into L3DL on an external L3DL 1271 speaking interface. 1273 An implementation SHOULD provide the ability to distribute one or 1274 more overlay and/or underlay addresses or interfaces into L3DL on an 1275 external L3DL speaking interface. 1277 An implementation SHOULD provide the ability to configure one of the 1278 addresses of an encapsulation as primary on an L3DL speaking 1279 interface. If there is only one address for a particular 1280 encapsulation, the implementation MAY mark it as primary by default. 1282 An implementation MAY allow optional configuration which updates the 1283 local forwarding table with overlay and underlay data both learned 1284 from L3DL peers and configured locally. 1286 21. Security Considerations 1288 The protocol as is MUST NOT be used outside a datacenter or similarly 1289 closed environment without authentication ans authorisation 1290 mechanisms such as [I-D.ymbk-lsvr-l3dl-signing]. 1292 Many MDC operators have a strange belief that physical walls and 1293 firewalls provide sufficient security. This is not credible. All 1294 MDC protocols need to be examined for exposure and attack surface. 1295 In the case of L3DL, Authentication and Integrity as provided in 1296 [I-D.ymbk-lsvr-l3dl-signing] is strongly recommended. 1298 It is generally unwise to assume that on the wire Layer 2 is secure. 1299 Strange/unauthorized devices may plug into a port. Mis-wiring is 1300 very common in datacenter installations. A poisoned laptop might be 1301 plugged into a device's port, form malicious sessions, etc. to 1302 divert, intercept, or drop traffic. 1304 Similarly, malicious nodes/devices could mis-announce addressing. 1306 If OPENs are not being authenticated, an attacker could forge an OPEN 1307 for an existing session and cause the session to be reset. 1309 For these reasons, the OPEN PDU's authentication data exchange SHOULD 1310 be used. 1312 If the KEEPALIVE PDU is not signed (as suggested in Section 8) to 1313 save computation, then a MITM could fake a session being alive. 1315 22. IANA Considerations 1317 22.1. PDU Types 1319 This document requests the IANA create a registry for L3DL PDU Type, 1320 which may range from 0 to 255. The name of the registry should be 1321 L3DL-PDU-Type. The policy for adding to the registry is RFC Required 1322 per [RFC5226], either standards track or experimental. The initial 1323 entries should be the following: 1325 PDU 1326 Code PDU Name 1327 ---- ------------------- 1328 0 HELLO 1329 1 OPEN 1330 2 KEEPALIVE 1331 3 ACK 1332 4 IPv4 Announcement 1333 5 IPv6 Announcement 1334 6 MPLS IPv4 Announcement 1335 7 MPLS IPv6 Announcement 1336 8-254 Reserved 1337 255 VENDOR 1339 22.2. Signature Type 1341 This document requests the IANA create a registry for L3DL Signature 1342 Type, AKA Sig Type, which may range from 0 to 255. The name of the 1343 registry should be L3DL-Signature-Type. The policy for adding to the 1344 registry is RFC Required per [RFC5226], either standards track or 1345 experimental. The initial entries should be the following: 1347 Number Name 1348 ------ ------------------- 1349 0 Null 1350 1-255 Reserved 1352 22.3. Flag Bits 1354 This document requests the IANA create a registry for L3DL PL Flag 1355 Bits, which may range from 0 to 7. The name of the registry should 1356 be L3DL-PL-Flag-Bits. The policy for adding to the registry is RFC 1357 Required per [RFC5226], either standards track or experimental. The 1358 initial entries should be the following: 1360 Bit Bit Name 1361 ---- ------------------- 1362 0 Announce/Withdraw (ann == 0) 1363 1 Primary 1364 2 Underlay/Overlay (under == 0) 1365 3 Loopback 1366 4-7 Reserved 1368 22.4. Error Codes 1370 This document requests the IANA create a registry for L3DL Error 1371 Codes, a 16 bit integer. The name of the registry should be L3DL- 1372 Error-Codes. The policy for adding to the registry is RFC Required 1373 per [RFC5226], either standards track or experimental. The initial 1374 entries should be the following: 1376 Error 1377 Code Error Name 1378 ---- ------------------- 1379 0 No Error 1380 1 Checksum Error 1381 2 Logical Link Addressing Conflict 1382 3 Authorization Failure 1383 4 Announce/Withdraw Error 1385 23. IEEE Considerations 1387 This document requires a new EtherType. 1389 This document requires a new multicast MAC address that will be 1390 broadcast through a switch. 1392 24. Acknowledgments 1394 The authors thank Cristel Pelsser for multiple reviews, Harsha Kovuru 1395 for comments during implementation, Jeff Haas for review and 1396 comments, Joe Clarke for a useful review, John Scudder for deeply 1397 serious review and comments, Larry Kreeger for a lot of layer 2 clue, 1398 Martijn Schmidt for his contribution, Nalinaksh Pai for transport 1399 discussions, Neeraj Malhotra for review, Paul Congdon for Ethernet 1400 hints, Russ Housley for checksum discussion and sBox, and Steve 1401 Bellovin for checksum advice. 1403 25. References 1405 25.1. Normative References 1407 [I-D.ietf-idr-bgp-ls-segment-routing-ext] 1408 Previdi, S., Talaulikar, K., Filsfils, C., Gredler, H., 1409 and M. Chen, "BGP Link-State extensions for Segment 1410 Routing", draft-ietf-idr-bgp-ls-segment-routing-ext-16 1411 (work in progress), June 2019. 1413 [I-D.ietf-idr-bgpls-segment-routing-epe] 1414 Previdi, S., Talaulikar, K., Filsfils, C., Patel, K., Ray, 1415 S., and J. Dong, "BGP-LS extensions for Segment Routing 1416 BGP Egress Peer Engineering", draft-ietf-idr-bgpls- 1417 segment-routing-epe-19 (work in progress), May 2019. 1419 [I-D.ietf-lsvr-bgp-spf] 1420 Patel, K., Lindem, A., Zandi, S., and W. Henderickx, 1421 "Shortest Path Routing Extensions for BGP Protocol", 1422 draft-ietf-lsvr-bgp-spf-06 (work in progress), September 1423 2019. 1425 [I-D.ymbk-lsvr-l3dl-signing] 1426 Bush, R. and R. Austein, "Layer 3 Discovery and Liveness 1427 Signing", draft-ymbk-lsvr-l3dl-signing-00 (work in 1428 progress), October 2019. 1430 [IANA-PEN] 1431 "IANA Private Enterprise Numbers", 1432 . 1435 [IEEE.802_2001] 1436 IEEE, "IEEE Standard for Local and Metropolitan Area 1437 Networks: Overview and Architecture", IEEE 802-2001, 1438 DOI 10.1109/ieeestd.2002.93395, July 2002, 1439 . 1441 [IEEE802-2014] 1442 Institute of Electrical and Electronics Engineers, "Local 1443 and Metropolitan Area Networks: Overview and 1444 Architecture", IEEE Std 802-2014, 2014. 1446 [RFC1213] McCloghrie, K. and M. Rose, "Management Information Base 1447 for Network Management of TCP/IP-based internets: MIB-II", 1448 STD 17, RFC 1213, DOI 10.17487/RFC1213, March 1991, 1449 . 1451 [RFC1629] Colella, R., Callon, R., Gardner, E., and Y. Rekhter, 1452 "Guidelines for OSI NSAP Allocation in the Internet", 1453 RFC 1629, DOI 10.17487/RFC1629, May 1994, 1454 . 1456 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1457 Requirement Levels", BCP 14, RFC 2119, 1458 DOI 10.17487/RFC2119, March 1997, 1459 . 1461 [RFC3032] Rosen, E., Tappan, D., Fedorkow, G., Rekhter, Y., 1462 Farinacci, D., Li, T., and A. Conta, "MPLS Label Stack 1463 Encoding", RFC 3032, DOI 10.17487/RFC3032, January 2001, 1464 . 1466 [RFC4271] Rekhter, Y., Ed., Li, T., Ed., and S. Hares, Ed., "A 1467 Border Gateway Protocol 4 (BGP-4)", RFC 4271, 1468 DOI 10.17487/RFC4271, January 2006, 1469 . 1471 [RFC5226] Narten, T. and H. Alvestrand, "Guidelines for Writing an 1472 IANA Considerations Section in RFCs", RFC 5226, 1473 DOI 10.17487/RFC5226, May 2008, 1474 . 1476 [RFC5880] Katz, D. and D. Ward, "Bidirectional Forwarding Detection 1477 (BFD)", RFC 5880, DOI 10.17487/RFC5880, June 2010, 1478 . 1480 [RFC6286] Chen, E. and J. Yuan, "Autonomous-System-Wide Unique BGP 1481 Identifier for BGP-4", RFC 6286, DOI 10.17487/RFC6286, 1482 June 2011, . 1484 [RFC7752] Gredler, H., Ed., Medved, J., Previdi, S., Farrel, A., and 1485 S. Ray, "North-Bound Distribution of Link-State and 1486 Traffic Engineering (TE) Information Using BGP", RFC 7752, 1487 DOI 10.17487/RFC7752, March 2016, 1488 . 1490 [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 1491 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, 1492 May 2017, . 1494 25.2. Informative References 1496 [Clos0] Clos, C., "A study of non-blocking switching networks 1497 [PAYWALLED]", Bell System Technical Journal 32 (2), pp 1498 406-424, March 1953. 1500 [Clos1] "Clos Network", 1501 . 1503 [I-D.malhotra-bess-evpn-lsoe] 1504 Malhotra, N., Patel, K., and J. Rabadan, "LSoE-based PE-CE 1505 Control Plane for EVPN", draft-malhotra-bess-evpn-lsoe-00 1506 (work in progress), March 2019. 1508 [JUPITER] Singh, A., Germano, P., Kanagala, A., Liu, H., Provost, 1509 J., Simmons, J., Tanda, E., Wanderer, J., HAP.lzle, U., 1510 Stuart, S., Vahdat, A., Ong, J., Agarwal, A., Anderson, 1511 G., Armistead, A., Bannon, R., Boving, S., Desai, G., and 1512 B. Felderman, "Jupiter rising", Communications of the 1513 ACM Vol. 59, pp. 88-97, DOI 10.1145/2975159, August 2016. 1515 [RFC0791] Postel, J., "Internet Protocol", STD 5, RFC 791, 1516 DOI 10.17487/RFC0791, September 1981, 1517 . 1519 [RFC1122] Braden, R., Ed., "Requirements for Internet Hosts - 1520 Communication Layers", STD 3, RFC 1122, 1521 DOI 10.17487/RFC1122, October 1989, 1522 . 1524 [RFC1982] Elz, R. and R. Bush, "Serial Number Arithmetic", RFC 1982, 1525 DOI 10.17487/RFC1982, August 1996, 1526 . 1528 Authors' Addresses 1530 Randy Bush 1531 Arrcus & Internet Initiative Japan 1532 5147 Crystal Springs 1533 Bainbridge Island, WA 98110 1534 US 1536 Email: randy@psg.com 1538 Rob Austein 1539 Arrcus, Inc 1541 Email: sra@hactrn.net 1543 Keyur Patel 1544 Arrcus 1545 2077 Gateway Place, Suite #400 1546 San Jose, CA 95119 1547 US 1549 Email: keyur@arrcus.com