< draft-ietf-lsvr-l3dl-07.txt   draft-ietf-lsvr-l3dl-08.txt >
Network Working Group R. Bush Network Working Group R. Bush
Internet-Draft Arrcus & Internet Initiative Japan Internet-Draft Arrcus & Internet Initiative Japan
Intended status: Standards Track R. Austein Intended status: Standards Track R. Austein
Expires: July 30, 2021 K. Patel Expires: 17 April 2022 K. Patel
Arrcus Arrcus
January 26, 2021 14 October 2021
Layer 3 Discovery and Liveness Layer-3 Discovery and Liveness
draft-ietf-lsvr-l3dl-07 draft-ietf-lsvr-l3dl-08
Abstract Abstract
In Massive Data Centers, BGP-SPF and similar routing protocols are In Massive Data Centers, BGP-SPF and similar routing protocols are
used to build topology and reachability databases. These protocols used to build topology and reachability databases. These protocols
need to discover IP Layer 3 attributes of links, such as neighbor IP need to discover IP Layer-3 attributes of links, such as neighbor IP
addressing, logical link IP encapsulation abilities, and link addressing, logical link IP encapsulation abilities, and link
liveness. This Layer 3 Discovery and Liveness protocol collects liveness. This Layer-3 Discovery and Liveness protocol collects
these data, which may then be disseminated using BGP-SPF and similar these data, which may then be disseminated using BGP-SPF and similar
protocols. protocols.
Requirements Language Requirements Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described in BCP "OPTIONAL" in this document are to be interpreted as described in BCP
14 [RFC2119] [RFC8174] when, and only when, they appear in all 14 [RFC2119] [RFC8174] when, and only when, they appear in all
capitals, as shown here. capitals, as shown here.
skipping to change at page 1, line 46 skipping to change at page 1, line 46
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/. Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on July 30, 2021. This Internet-Draft will expire on 17 April 2022.
Copyright Notice Copyright Notice
Copyright (c) 2021 IETF Trust and the persons identified as the Copyright (c) 2021 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents (https://trustee.ietf.org/
(https://trustee.ietf.org/license-info) in effect on the date of license-info) in effect on the date of publication of this document.
publication of this document. Please review these documents Please review these documents carefully, as they describe your rights
carefully, as they describe your rights and restrictions with respect and restrictions with respect to this document. Code Components
to this document. Code Components extracted from this document must extracted from this document must include Simplified BSD License text
include Simplified BSD License text as described in Section 4.e of as described in Section 4.e of the Trust Legal Provisions and are
the Trust Legal Provisions and are provided without warranty as provided without warranty as described in the Simplified BSD License.
described in the Simplified BSD License.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3
2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4
3. Background . . . . . . . . . . . . . . . . . . . . . . . . . 5 3. Background . . . . . . . . . . . . . . . . . . . . . . . . . 5
4. Top Level Overview . . . . . . . . . . . . . . . . . . . . . 6 4. Top Level Overview . . . . . . . . . . . . . . . . . . . . . 6
5. Inter-Link Protocol Overview . . . . . . . . . . . . . . . . 7 5. Inter-Link Protocol Overview . . . . . . . . . . . . . . . . 8
5.1. L3DL Ladder Diagram . . . . . . . . . . . . . . . . . . . 7 5.1. L3DL Ladder Diagram . . . . . . . . . . . . . . . . . . . 8
6. Transport Layer . . . . . . . . . . . . . . . . . . . . . . . 9 6. Transport Layer . . . . . . . . . . . . . . . . . . . . . . . 10
7. The Checksum . . . . . . . . . . . . . . . . . . . . . . . . 11 7. The Checksum . . . . . . . . . . . . . . . . . . . . . . . . 12
8. TLV PDUs . . . . . . . . . . . . . . . . . . . . . . . . . . 13 8. TLV PDUs . . . . . . . . . . . . . . . . . . . . . . . . . . 14
9. Logical Link Endpoint Identifier . . . . . . . . . . . . . . 14 9. Logical Link Endpoint Identifier . . . . . . . . . . . . . . 15
10. HELLO . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 10. HELLO . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
11. OPEN . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 11. OPEN . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
12. ACK . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 12. ACK . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
12.1. Retransmission . . . . . . . . . . . . . . . . . . . . . 20 12.1. Retransmission . . . . . . . . . . . . . . . . . . . . . 21
13. The Encapsulations . . . . . . . . . . . . . . . . . . . . . 20 13. The Encapsulations . . . . . . . . . . . . . . . . . . . . . 22
13.1. The Encapsulation PDU Skeleton . . . . . . . . . . . . . 21 13.1. The Encapsulation PDU Skeleton . . . . . . . . . . . . . 22
13.2. Encapsulaion Flags . . . . . . . . . . . . . . . . . . . 22 13.2. Encapsulaion Flags . . . . . . . . . . . . . . . . . . . 24
13.3. IPv4 Encapsulation . . . . . . . . . . . . . . . . . . . 22 13.3. IPv4 Encapsulation . . . . . . . . . . . . . . . . . . . 24
13.4. IPv6 Encapsulation . . . . . . . . . . . . . . . . . . . 23 13.4. IPv6 Encapsulation . . . . . . . . . . . . . . . . . . . 25
13.5. MPLS Label List . . . . . . . . . . . . . . . . . . . . 24 13.5. MPLS Label List . . . . . . . . . . . . . . . . . . . . 26
13.6. MPLS IPv4 Encapsulation . . . . . . . . . . . . . . . . 24 13.6. MPLS IPv4 Encapsulation . . . . . . . . . . . . . . . . 26
13.7. MPLS IPv6 Encapsulation . . . . . . . . . . . . . . . . 25 13.7. MPLS IPv6 Encapsulation . . . . . . . . . . . . . . . . 27
14. VENDOR - Vendor Extensions . . . . . . . . . . . . . . . . . 25 14. VENDOR - Vendor Extensions . . . . . . . . . . . . . . . . . 27
15. KEEPALIVE - Layer 2 Liveness . . . . . . . . . . . . . . . . 26 15. KEEPALIVE - Layer-2 Liveness . . . . . . . . . . . . . . . . 28
16. Layers 2.5 and 3 Liveness . . . . . . . . . . . . . . . . . . 27 16. Layers-2.5 and 3 Liveness . . . . . . . . . . . . . . . . . . 29
17. The North/South Protocol . . . . . . . . . . . . . . . . . . 27 17. The North/South Protocol . . . . . . . . . . . . . . . . . . 29
17.1. Use BGP-LS as Much as Possible . . . . . . . . . . . . . 28 17.1. Use BGP-LS as Much as Possible . . . . . . . . . . . . . 30
17.2. Extensions to BGP-LS . . . . . . . . . . . . . . . . . . 28 17.2. Extensions to BGP-LS . . . . . . . . . . . . . . . . . . 30
18. Discussion . . . . . . . . . . . . . . . . . . . . . . . . . 28 18. Discussion . . . . . . . . . . . . . . . . . . . . . . . . . 30
18.1. HELLO Discussion . . . . . . . . . . . . . . . . . . . . 28 18.1. HELLO Discussion . . . . . . . . . . . . . . . . . . . . 30
18.2. HELLO versus KEEPALIVE . . . . . . . . . . . . . . . . . 29 18.2. HELLO versus KEEPALIVE . . . . . . . . . . . . . . . . . 31
19. VLANs/SVIs/Sub-interfaces . . . . . . . . . . . . . . . . . . 31
19. VLANs/SVIs/Sub-interfaces . . . . . . . . . . . . . . . . . . 29 20. Implementation Considerations . . . . . . . . . . . . . . . . 31
20. Implementation Considerations . . . . . . . . . . . . . . . . 29 21. Security Considerations . . . . . . . . . . . . . . . . . . . 32
21. Security Considerations . . . . . . . . . . . . . . . . . . . 30 22. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 32
22. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 30 22.1. PDU Types . . . . . . . . . . . . . . . . . . . . . . . 32
22.1. PDU Types . . . . . . . . . . . . . . . . . . . . . . . 30 22.2. Signature Type . . . . . . . . . . . . . . . . . . . . . 33
22.2. Signature Type . . . . . . . . . . . . . . . . . . . . . 31 22.3. Flag Bits . . . . . . . . . . . . . . . . . . . . . . . 33
22.3. Flag Bits . . . . . . . . . . . . . . . . . . . . . . . 31 22.4. Error Codes . . . . . . . . . . . . . . . . . . . . . . 34
22.4. Error Codes . . . . . . . . . . . . . . . . . . . . . . 31 23. IEEE Considerations . . . . . . . . . . . . . . . . . . . . . 34
23. IEEE Considerations . . . . . . . . . . . . . . . . . . . . . 32 24. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 34
24. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 32 25. References . . . . . . . . . . . . . . . . . . . . . . . . . 34
25. References . . . . . . . . . . . . . . . . . . . . . . . . . 32 25.1. Normative References . . . . . . . . . . . . . . . . . . 34
25.1. Normative References . . . . . . . . . . . . . . . . . . 32 25.2. Informative References . . . . . . . . . . . . . . . . . 36
25.2. Informative References . . . . . . . . . . . . . . . . . 34 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 37
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 35
1. Introduction 1. Introduction
The Massive Data Center (MDC) environment presents unusual problems The Massive Data Center (MDC) environment presents unusual problems
of scale, e.g. O(10,000) forwarding devices, while its homogeneity of scale, e.g. O(10,000) forwarding devices, while its homogeneity
presents opportunities for simple approaches. Approaches such as presents opportunities for simple approaches. Approaches such as
Jupiter Rising [JUPITER] use a central controller to deal with Jupiter Rising [JUPITER] use a central controller to deal with
scaling, while BGP-SPF [I-D.ietf-lsvr-bgp-spf] provides massive scaling, while BGP-SPF [I-D.ietf-lsvr-bgp-spf] provides massive
scale-out without centralization using a tried and tested scalable scale-out without centralization using a tried and tested scalable
distributed control plane, offering a scalable routing solution in distributed control plane, offering a scalable routing solution in
Clos [Clos0][Clos1] and similar environments. But BGP-SPF and Clos [Clos0][Clos1] and similar environments. But BGP-SPF and
similar higher level device-spanning protocols, e.g. similar higher level device-spanning protocols, e.g.
[I-D.malhotra-bess-evpn-lsoe], need logical link state and addressing [I-D.malhotra-bess-evpn-lsoe], need logical link state and addressing
data from the network to build the routing topology. They also need data from the network to build the routing topology. They also need
prompt but prudent reaction to (logical) link failure. prompt but prudent reaction to (logical) link failure.
Layer 3 Discovery and Liveness (L3DL) provides brutally simple Layer-3 Discovery and Liveness (L3DL) provides brutally simple
mechanisms for devices to mechanisms for devices to
o Discover each other's unique endpoint identification, * Discover each other's unique endpoint identification,
o Discover mutually supported layer 3 encapsulations, e.g. IP/MPLS, * Discover mutually supported layer-3 encapsulations, e.g. IP/MPLS,
o Discover Layer 3 IP and/or MPLS addressing of interfaces of the * Discover Layer-3 IP and/or MPLS addressing of interfaces of the
encapsulations, encapsulations,
o Present these data, using a very restricted profile of a BGP-LS * Present these data, using a very restricted profile of a BGP-LS
[RFC7752] API, to BGP-SPF which computes the topology and builds [RFC7752] API, to BGP-SPF which computes the topology and builds
routing and forwarding tables, routing and forwarding tables,
o Enable Layer 3 link liveness such as BFD, * Enable Layer-3 link liveness such as BFD,
o Provide Layer 2 keep-alive messages for session continuity, and * Provide Layer-2 keep-alive messages for session continuity, and
finally finally
o Provide for authenticity verification of protocol messages. * Provide for authenticity verification of protocol messages.
In this document, the use case for L3DL is for point to point links In this document, the use case for L3DL is for point to point links
in a datacenter Clos in order to exchange the data needed for BGP-SPF in a datacenter Clos in order to exchange the data needed for BGP-SPF
[I-D.ietf-lsvr-bgp-spf] bootstrap and continuity. Once layer two [I-D.ietf-lsvr-bgp-spf] bootstrap and continuity. Once layer-2
connectivity has been leveraged to get layer three addressability and connectivity has been leveraged to get layer-3 addressability and
forwarding capabilities, normal layer three forwarding and routing forwarding capabilities, normal layer-3 forwarding and routing can
can take over. take over.
L3DL might be found to be more widely applicable to a range of L3DL might be found to be more widely applicable to a range of
routing and similar protocols which need layer three discovery and routing and similar protocols which need layer-3 discovery and
characterisation. characterisation.
2. Terminology 2. Terminology
Even though it concentrates on the inter-device layer, this document Even though it concentrates on the inter-device layer, this document
relies heavily on routing terminology. The following attempts to relies heavily on routing terminology. The following attempts to
clarify the use of some possibly confusing terms: clarify the use of some possibly confusing terms:
ASN: Autonomous System Number [RFC4271], a BGP identifier for ASN: Autonomous System Number [RFC4271], a BGP identifier for
an originator of Layer 3 routes, particularly BGP an originator of Layer-3 routes, particularly BGP
announcements. announcements.
BGP-LS: A mechanism by which link-state and TE information can be BGP-LS: A mechanism by which link-state and TE information can be
collected from networks and shared with external collected from networks and shared with external
components using the BGP routing protocol. See [RFC7752]. components using the BGP routing protocol. See [RFC7752].
BGP-SPF A hybrid protocol using BGP transport but a Dijkstra BGP-SPF A hybrid protocol using BGP transport but a Dijkstra
Shortest Path First decision process. See Shortest Path First decision process. See
[I-D.ietf-lsvr-bgp-spf]. [I-D.ietf-lsvr-bgp-spf].
Clos: A hierarchic subset of a crossbar switch topology commonly Clos: A hierarchic subset of a crossbar switch topology commonly
used in data centers. used in data centers.
Datagram: The L3DL content of a single Layer 2 frame, sans Ethernet
Datagram: The L3DL content of a single Layer-2 frame, sans Ethernet
framing. A full L3DL PDU may be packaged in multiple framing. A full L3DL PDU may be packaged in multiple
Datagrams. Datagrams.
Encapsulation: Address Family Indicator and Subsequent Address Encapsulation: Address Family Indicator and Subsequent Address
Family Indicator (AFI/SAFI). I.e. classes of layer 2.5 Family Indicator (AFI/SAFI). I.e. classes of layer-2.5
and 3 addresses such as IPv4, IPv6, MPLS, etc. and 3 addresses such as IPv4, IPv6, MPLS, etc.
Frame: A Layer 2 Ethernet packet.
Frame: A Layer-2 Ethernet packet.
Link or Logical Link: A logical connection between two logical ports Link or Logical Link: A logical connection between two logical ports
on two devices. E.g. two VLANs between the same two ports on two devices. E.g. two VLANs between the same two ports
are two links. are two links.
LLEI: Logical Link Endpoint Identifier, the unique identifier of LLEI: Logical Link Endpoint Identifier, the unique identifier of
one end of a logical link, see Section 9. one end of a logical link, see Section 9.
MAC Address: 48-bit Layer 2 addresses are assumed since they are
used by all widely deployed Layer 2 network technologies MAC Address: 48-bit Layer-2 addresses are assumed since they are
used by all widely deployed Layer-2 network technologies
of interest, especially Ethernet. See [IEEE.802_2001]. of interest, especially Ethernet. See [IEEE.802_2001].
MDC: Massive Data Center, commonly composed of thousands of Top MDC: Massive Data Center, commonly composed of thousands of Top
of Rack Switches (TORs). of Rack Switches (TORs).
MTU: Maximum Transmission Unit, the size in octets of the MTU: Maximum Transmission Unit, the size in octets of the
largest packet that can be sent on a medium, see [RFC1122] largest packet that can be sent on a medium, see [RFC1122]
1.3.3. 1.3.3.
PDU: Protocol Data Unit, an L3DL application layer message. A PDU: Protocol Data Unit, an L3DL application layer message. A
PDU's content may need to be broken into multiple PDU's content may need to be broken into multiple
Datagrams to make it through MTU or other restrictions. Datagrams to make it through MTU or other restrictions.
RouterID: An 32-bit identifier unique in the current routing domain, RouterID: An 32-bit identifier unique in the current routing domain,
see [RFC6286]. see [RFC6286].
Session: An established, via OPEN PDUs, session between two L3DL Session: An established, via OPEN PDUs, session between two L3DL
capable link end-points, capable link end-points,
SPF: Shortest Path First, an algorithm for finding the shortest SPF: Shortest Path First, an algorithm for finding the shortest
paths between nodes in a graph; AKA Dijkstra's algorithm. paths between nodes in a graph; AKA Dijkstra's algorithm.
System Identifier: An eight octet ISO System Identifier a la System Identifier: An eight octet ISO System Identifier a la
[RFC1629] System ID [RFC1629] System ID
TOR: Top Of Rack switch, aggregates the servers in a rack and TOR: Top Of Rack switch, aggregates the servers in a rack and
connects to aggregation layers of the Clos tree, AKA the connects to aggregation layers of the Clos tree, AKA the
Clos spine. Clos spine.
ZTP: Zero Touch Provisioning gives devices initial addresses, ZTP: Zero Touch Provisioning gives devices initial addresses,
credentials, etc. on boot/restart. credentials, etc. on boot/restart.
3. Background 3. Background
L3DL is primarily designed for a Clos type datacenter scale and L3DL is primarily designed for a Clos type datacenter scale and
topology, but can accommodate richer topologies which contain topology, but can accommodate richer topologies which contain
potential cycles. potential cycles.
While L3DL is designed for the MDC, there are no inherent reasons it While L3DL is designed for the MDC, there are no inherent reasons it
skipping to change at page 6, line 7 skipping to change at page 6, line 20
number of addresses. And highly automated micro-service migration number of addresses. And highly automated micro-service migration
can cause serious address prefix disaggregation, resulting in can cause serious address prefix disaggregation, resulting in
interfaces with thousands of disaggregated prefixes. interfaces with thousands of disaggregated prefixes.
Therefore the L3DL protocol is session oriented and uses incremental Therefore the L3DL protocol is session oriented and uses incremental
announcement and withdrawal with session restart, a la BGP announcement and withdrawal with session restart, a la BGP
([RFC4271]). ([RFC4271]).
4. Top Level Overview 4. Top Level Overview
o Devices discover each other on logical links * Devices discover each other on logical links
o Logical Link Endpoint Identifiers (LLEIs) are exchanged * Logical Link Endpoint Identifiers (LLEIs) are exchanged
o Layer 2 Liveness checks may be started * Layer-2 Liveness checks may be started
o Encapsulation data are exchanged and IP-Level Liveness checks * Encapsulation data are exchanged and IP-Level Liveness checks
enabled enabled
o A BGP-like upper layer protocol is assumed to use the identifiers * A BGP-like upper layer protocol is assumed to use the identifiers
and encapsulation data to discover and build a topology database and encapsulation data to discover and build a topology database
+-------------------+ +-------------------+ +-------------------+ +-------------------+ +-------------------+ +-------------------+
| Device | | Device | | Device | | Device | | Device | | Device |
| | | | | | | | | | | |
|+-----------------+| |+-----------------+| |+-----------------+| |+-----------------+| |+-----------------+| |+-----------------+|
|| || || || || || || || || || || ||
|| BGP-SPF <+---+> BGP-SPF <+---+> BGP-SPF || || BGP-SPF <+---+> BGP-SPF <+---+> BGP-SPF ||
|| || || || || || || || || || || ||
|+--------^--------+| |+--------^--------+| |+--------^--------+| |+--------^--------+| |+--------^--------+| |+--------^--------+|
skipping to change at page 6, line 44 skipping to change at page 7, line 30
| | | | | | | | | | | | | | | | | |
| | | | | | | | | | | | | | | | | |
|+--------v--------+| |+--------v--------+| |+--------v--------+| |+--------v--------+| |+--------v--------+| |+--------v--------+|
|| || || || || || || || || || || ||
||Inter-Device PDUs<+---+>Inter-Device PDUs<+---+>Inter-Device PDUs|| ||Inter-Device PDUs<+---+>Inter-Device PDUs<+---+>Inter-Device PDUs||
|| || || || || || || || || || || ||
|+-----------------+| |+-----------------+| |+-----------------+| |+-----------------+| |+-----------------+| |+-----------------+|
+-------------------+ +-------------------+ +-------------------+ +-------------------+ +-------------------+ +-------------------+
There are two protocols, the inter-device (left-right in the diagram) There are two protocols, the inter-device (left-right in the diagram)
per-link layer 3 discovery and the API to the upper level BGP-like per-link layer-3 discovery and the API to the upper level BGP-like
routing protocol (up-down in the above diagram): routing protocol (up-down in the above diagram):
o Inter-device PDUs are used to exchange device and logical link * Inter-device PDUs are used to exchange device and logical link
identities and layer 2.5 (MPLS) and 3 identifiers (not payloads), identities and layer-2.5 (MPLS) and 3 identifiers (not payloads),
e.g. device IDs, port identities, VLAN IDs, Encapsulations, and IP e.g. device IDs, port identities, VLAN IDs, Encapsulations, and IP
addresses. addresses.
o A Link Layer to BGP API presents these data up the stack to a BGP * A Link Layer to BGP API presents these data up the stack to a BGP
protocol or an other device-spanning upper layer protocol, protocol or an other device-spanning upper layer protocol,
presenting them using the BGP-LS BGP-like data format. presenting them using the BGP-LS BGP-like data format.
The upper layer BGP family routing protocols cross all the devices, The upper layer BGP family routing protocols cross all the devices,
though they are not part of these L3DL protocols. though they are not part of these L3DL protocols.
To simplify this document, Layer 2 framing is not shown. L3DL is To simplify this document, Layer-2 framing is not shown. L3DL is
about layer 3. about layer-3.
5. Inter-Link Protocol Overview 5. Inter-Link Protocol Overview
Two devices discover each other and their respective identities by Two devices discover each other and their respective identities by
sending multicast HELLO PDUs (Section 10). To assure discovery of sending multicast HELLO PDUs (Section 10). To assure discovery of
new devices coming up on a multi-link topology, devices on such a new devices coming up on a multi-link topology, devices on such a
topology, and only on a multi-link topology, send periodic HELLOs topology, and only on a multi-link topology, send periodic HELLOs
forever, see Section 18.1. forever, see Section 18.1.
Once a new device is recognized, both devices attempt to negotiate Once a new device is recognized, both devices attempt to negotiate
skipping to change at page 9, line 15 skipping to change at page 10, line 5
|---------------------------->| Optional |---------------------------->| Optional
| ACK | | ACK |
|<----------------------------| |<----------------------------|
| | | |
| Interface MPLSv6 Labels | Interface MPLSv6 Labels | Interface MPLSv6 Labels | Interface MPLSv6 Labels
|<----------------------------| Optional |<----------------------------| Optional
| ACK | | ACK |
|---------------------------->| |---------------------------->|
| | | |
| | | |
| L3DL KEEPALIVE | Layer 2 Liveness | L3DL KEEPALIVE | Layer-2 Liveness
|---------------------------->| Optional |---------------------------->| Optional
| L3DL KEEPALIVE | | L3DL KEEPALIVE |
|<----------------------------| |<----------------------------|
6. Transport Layer 6. Transport Layer
L3DL PDUs are carried by a simple transport layer which allows long L3DL PDUs are carried by a simple transport layer which allows long
PDUs to occupy many Ethernet frames. The L3DL content of a single PDUs to occupy many Ethernet frames. The L3DL content of a single
Ethernet frame, exclusive of Ethernet framing data, is referred to as Ethernet frame, exclusive of Ethernet framing data, is referred to as
a Datagram. a Datagram.
skipping to change at page 9, line 43 skipping to change at page 10, line 33
This is not classic 'fragmentation', but rather decomposition at the This is not classic 'fragmentation', but rather decomposition at the
origin to allow PDU payloads larger than the frame allows. There are origin to allow PDU payloads larger than the frame allows. There are
no intermediate devices capable of further fragmentation or no intermediate devices capable of further fragmentation or
reassembly. reassembly.
A PDU might need a large number of frames to be sent. As fragments A PDU might need a large number of frames to be sent. As fragments
are not ACK paced (as PDUs are), to avoid overwhelming bursts, the are not ACK paced (as PDUs are), to avoid overwhelming bursts, the
sender should pace fragments of a large PDU. sender should pace fragments of a large PDU.
L3DL is carrying relatively small amounts of data on relatively high L3DL is carrying a relatively small amount of data on relatively high
bandwidth links, and at a time when the link is not active with other bandwidth links, and at a time when the link is not active with other
data as it does not yet have layer three connectivity. So congestion data as it does not yet have layer-3 connectivity. So congestion is
is not considered a sufficiently significant risk to warrant not considered a sufficiently significant risk to warrant additional
additional complexity. complexity.
Should a PDU need to be retransmitted, it MUST BE sent as the Should a PDU need to be retransmitted, it MUST BE sent as the
identical Datagram set as the original transmission. The identical Datagram set as the original transmission. The
Transmission Sequence Number informs the receiver that it is the same Transmission Sequence Number informs the receiver that it is the same
PDU. PDU.
0 1 2 3 0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Version | Transmission Sequence Number |L| Dtgm Number ~ | Version | Transmission Sequence Number |L| Dtgm Number ~
skipping to change at page 10, line 32 skipping to change at page 11, line 30
Values other than 0 MUST BE treated as an error. The protocol Values other than 0 MUST BE treated as an error. The protocol
version needs to be in one and only one place, so it is in the version needs to be in one and only one place, so it is in the
datagram as opposed to, for example, the PDU header. datagram as opposed to, for example, the PDU header.
Transmission Sequence Number: A 16-bit strictly increasing unsigned Transmission Sequence Number: A 16-bit strictly increasing unsigned
integer identifying this PDU, possibly across retransmissions, integer identifying this PDU, possibly across retransmissions,
that wraps from 2^16-1 to 0. The initial value is arbitrary. See that wraps from 2^16-1 to 0. The initial value is arbitrary. See
[RFC1982] on DNS Serial Number Arithmetic for too much detail on [RFC1982] on DNS Serial Number Arithmetic for too much detail on
comparing and incrementing a wrapping sequence number. comparing and incrementing a wrapping sequence number.
L: A bit that set to one if this Datagram is the last Datagram of the L: A bit that set to one if this Datagram is the last Datagram of
PDU. For a PDU which fits in only one Datagram, it is set to one. the PDU. For a PDU which fits in only one Datagram, it is set to
Note that this is the inverse of the marking technique used by one. Note that this is the inverse of the marking technique used
[RFC0791]. by [RFC0791].
Datagram Number: A monotonically increasing 23-bit value which Datagram Number: A monotonically increasing 23-bit value which
starts at zero for each PDU. This is used to reassemble frames starts at zero for each PDU. This is used to reassemble frames
into PDUs a la [RFC0791] Section 2.3. Note that this limits an into PDUs a la [RFC0791] Section 2.3. Note that this limits an
L3DL PDU to 2^24 frames. L3DL PDU to 2^24 frames.
Datagram Length: Total number of octets in the Datagram including Datagram Length: Total number of octets in the Datagram including
all payloads and fields. Note that this limits a datagram to 2^16 all payloads and fields. Note that this limits a datagram to 2^16
octets; though Ethernet framing is likely to impose a smaller octets; though Ethernet framing is likely to impose a smaller
limit. limit.
Checksum: A 32 bit hash over the Datagram to detect bit flips, see Checksum: A 32 bit hash over the Datagram to detect bit flips, see
Section 7. Section 7.
If a Datagram fails checksum verification, the datagram is invalid If a Datagram fails checksum verification, the datagram is invalid
and should be silently discarded. The sender will retransmit the and SHOULD be silently discarded. The sender will retransmit the
PDU, and the receiver can assemble it. PDU, and the receiver can assemble it.
Payload: The PDU being transported or a fragment thereof. Payload: The PDU being transported or a fragment thereof.
To avoid the need for a receiver to reassemble two PDUs at the same To avoid the need for a receiver to reassemble two PDUs at the same
time, a sender MUST NOT send a subsequent PDU when a PDU is already time, a sender MUST NOT send a subsequent PDU when a PDU is already
in flight and not yet acknowledged; assuming it is an ACKed PDU Type. in flight and not yet acknowledged; assuming it is an ACKed PDU Type.
7. The Checksum 7. The Checksum
skipping to change at page 14, line 21 skipping to change at page 15, line 21
An LLEI is a variable length descriptor which could be an ASN, a An LLEI is a variable length descriptor which could be an ASN, a
classic RouterID, a catenation of the two, an eight octet ISO System classic RouterID, a catenation of the two, an eight octet ISO System
Identifier [RFC1629], or any other identifier unique to a single Identifier [RFC1629], or any other identifier unique to a single
logical link endpoint in the topology. logical link endpoint in the topology.
An L3DL deployment will choose and define an LLEI which suits its An L3DL deployment will choose and define an LLEI which suits its
needs, simple or complex. Examples of two extremes follow: needs, simple or complex. Examples of two extremes follow:
A simplistic view of a link between two devices is two ports, A simplistic view of a link between two devices is two ports,
identified by unique MAC addresses, carrying a layer 3 protocol identified by unique MAC addresses, carrying a layer-3 protocol
conversation. In this case, the MAC addresses might suffice for the conversation. In this case, the MAC addresses might suffice for the
LLEIs. LLEIs.
Unfortunately, things can get more complex. Multiple VLANs can run Unfortunately, things can get more complex. Multiple VLANs can run
between those two MAC addresses. In practice, many real devices use between those two MAC addresses. In practice, many real devices use
the same MAC address on multiple ports and/or sub-interfaces. the same MAC address on multiple ports and/or sub-interfaces.
Therefore, in the general circumstance, a fully described LLEI might Therefore, in the general circumstance, a fully described LLEI might
be as follows: be as follows:
skipping to change at page 15, line 5 skipping to change at page 16, line 5
System Identifier, a la [RFC1629], is an eight octet identifier System Identifier, a la [RFC1629], is an eight octet identifier
unique in the entire operational space. Routers and switches usually unique in the entire operational space. Routers and switches usually
have internal MAC Addresses which can be padded with high order zeros have internal MAC Addresses which can be padded with high order zeros
and used if no System ID exists on the device. If no unique and used if no System ID exists on the device. If no unique
identifier is burned into a device, the local L3DL configuration identifier is burned into a device, the local L3DL configuration
SHOULD create and assign a unique one, likely by configuration. SHOULD create and assign a unique one, likely by configuration.
ifIndex is the SNMP identifier of the (sub-)interface, see [RFC1213]. ifIndex is the SNMP identifier of the (sub-)interface, see [RFC1213].
This uniquely identifies the port. This uniquely identifies the port.
For a layer 3 tagged sub-interface or a VLAN/SVI interface, Ifindex For a layer-3 tagged sub-interface or a VLAN/SVI interface, IfIndex
is that of the logical sub-interface, so no further disambiguation is is that of the logical sub-interface, so no further disambiguation is
needed. needed.
L3DL PDUs learned over VLAN-ports may be interpreted by upper layer-3 L3DL PDUs learned over VLAN-ports may be interpreted by upper layer-3
routing protocols as being learned on the corresponding layer-3 SVI routing protocols as being learned on the corresponding layer-3 SVI
interface for the VLAN. interface for the VLAN.
LLEIs are big-endian. LLEIs are big-endian.
10. HELLO 10. HELLO
The HELLO PDU is unique in that it is encapsulated in a multicast The HELLO PDU is unique in that it is encapsulated in a multicast
Ethernet frame. It solicits response(s) from other LLEI(s) on the Ethernet frame. It solicits response(s) from other LLEI(s) on the
link. See Section 18.1 for why multicast is used. The destination link. See Section 18.1 for why multicast is used. The destination
multicast MAC Addressees to be used MUST be one of the following, See multicast MAC Addressees to be used MUST be one of the following, See
Clause 9.2.2 of [IEEE802-2014]: Clause 9.2.2 of [IEEE802-2014]:
01-80-C2-00-00-0E: Nearest Bridge = Propagation constrained to a 01-80-C2-00-00-0E: Nearest Bridge = Propagation constrained to a
single physical link; stopped by all types of bridges (including single physical link; stopped by all types of bridges (including
MPRs (media converters)). This SHOULD BE used when the link is MPRs (media converters)). This SHOULD be used when the link is
known to be a simple point to point link. known to be a simple point to point link.
To Be Assigned: When a switch receives a frame with a multicast To Be Assigned: When a switch receives a frame with a multicast
destination MAC it does not recognize, it forwards to all ports. destination MAC it does not recognize, it forwards to all ports.
This destination MAC is to be sent when the interface is known to This destination MAC SHOULD be sent when the interface is known to
be connected to a switch. See Section 23. This SHOULD BE used be connected to a switch. See Section 23. This SHOULD be used
when the link may be a multi-point link. when the link may be a multi-point link.
All other L3DL PDUs are encapsulated in unicast frames, as the peer's All other L3DL PDUs are encapsulated in unicast frames, as the peer's
destination MAC address is known after the HELLO exchange. destination MAC address is known after the HELLO exchange.
When an interface is turned up on a device, it SHOULD issue a HELLO When an interface is turned up on a device, it SHOULD issue a HELLO
if it is to participate in L3DL sessions. if it is to participate in L3DL sessions.
If a constrained Nearest Bridge destination address has been If a constrained Nearest Bridge destination address has been
configured for a point-to-point interface, see above, then the HELLO configured for a point-to-point interface, see above, then the HELLO
skipping to change at page 16, line 24 skipping to change at page 17, line 24
unique source LLEI response. L3DL treats each adjacency as a unique source LLEI response. L3DL treats each adjacency as a
separate logical link. separate logical link.
When a HELLO is received from a source MAC address (plus VID if VLAN) When a HELLO is received from a source MAC address (plus VID if VLAN)
with which there is no established L3DL session, the receiver SHOULD with which there is no established L3DL session, the receiver SHOULD
respond by sending an OPEN PDU to the source MAC address (plus VID). respond by sending an OPEN PDU to the source MAC address (plus VID).
The two devices establish an L3DL session by exchanging OPEN PDUs. The two devices establish an L3DL session by exchanging OPEN PDUs.
To ameliorate possible load spikes during bootstrap or event To ameliorate possible load spikes during bootstrap or event
recovery, there SHOULD be a jittered delay between receipt of a HELLO recovery, there SHOULD be a jittered delay between receipt of a HELLO
and issue of the OPEN. The default delay range SHOULD BE zero to and issue of the OPEN. The default delay range SHOULD be zero to
five seconds, and MUST be configurable. five seconds, and MUST be configurable.
If a HELLO is received from a MAC address with which there is an If a HELLO is received from a MAC address with which there is an
established session, the HELLO should be dropped. established session, the HELLO should be dropped.
The Payload Length is zero as there is no payload. The Payload Length is zero as there is no payload.
HELLO PDUs can not be signed as keying material has yet to be HELLO PDUs can not be signed as keying material has yet to be
exchanged. Hence the signature MUST always be the null type. exchanged. Hence the signature MUST always be the null type.
11. OPEN 11. OPEN
Each device has learned the other's MAC Address from the HELLO Each device has learned the other's MAC Address from the HELLO
exchange, see Section 10. Therefore the OPEN and all subsequent PDUs exchange, see Section 10. Therefore the OPEN and all subsequent PDUs
MUST BE unicast, as opposed to the HELLO's multicast frame. MUST BE unicast, as opposed to the HELLO's multicast frame.
0 1 2 3 0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| PDU Type = 1 | Payload Length ~ | PDU Type = 1 | Payload Length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
~ | Nonce ~ | | Nonce |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
~ | LLEI Length | My LLEI | | | LLEI Length | My LLEI |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-~
~ | AttrCount | ~
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
~ Attribute List ... | Auth Type | Key Length ~ | | AttrCount | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
~ | Key ... | | Attribute List ... | Auth Type | Key Length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| | Key ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Serial Number | | Serial Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Sig Type | Signature Length | Signature ... | | Sig Type | Signature Length | Signature ... |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
The Payload Length is the number of octets in all fields of the PDU The Payload Length is the number of octets in all fields of the PDU
from the Nonce through the Serial Number, not including the three from the Nonce through the Serial Number, not including the three
final signature fields. final signature fields.
skipping to change at page 18, line 14 skipping to change at page 19, line 14
The Key is specific to the operational environment. A failure to The Key is specific to the operational environment. A failure to
authenticate is a failure to start the L3DL session, an ERROR PDU authenticate is a failure to start the L3DL session, an ERROR PDU
MUST BE sent (Error Code 3), and HELLOs MUST be restarted. MUST BE sent (Error Code 3), and HELLOs MUST be restarted.
Although delay and jitter in responding with an OPEN were specified Although delay and jitter in responding with an OPEN were specified
above, beware of load created by long strings of authentication above, beware of load created by long strings of authentication
failures and retries. A configurable failure count limit (default 8) failures and retries. A configurable failure count limit (default 8)
SHOULD result in giving up on the connection attempt. SHOULD result in giving up on the connection attempt.
The Serial Number is that of the last received and processed PDU. The Serial Number is a monotonically increasing 32-bit value
This allows a receiver sending an OPEN to tell the sender that the representing the sender's state at the time of sending the last PDU.
receiver wants to resume a session and the sender only needs to send It may be an integer, a timestamp, etc. If incrementing the Serial
data more recent than the Serial Number. If this OPEN is not trying Number would cause it to be zero, it should be incremented again.
to restart a lost session, the Serial Number MUST BE set to zero.
On session restart (new OPEN), a receiver MAY send the last received
Serial Number to tell the sender to only send data with a Serial
Number greater (in the [RFC1982] sense), or send a Serial Number of
zero to request all data.
The Serial Number supports session resumption in anticipation of
peers having a very large amount of state they would prefer not to
re-exchange because of some glitch. The Serial Number is not
expected to wrap for a considerable time, e.g. days or weeks. But to
address the rare case it does, [RFC1982] on DNS Serial Number
Arithmetic should be used as it is in the Transmission Sequence
Number.
This allows a sender of an OPEN to tell the receiver that the sender
would like to resume a session and that the receiver only needs to
send data starting with the PDU with the lowest Serial Number greater
(in the [RFC1982] sense) than the one sent in the OPEN. If the
sender is not trying to resume a dropped session, the Serial Number
MUST be zero.
If the receiver of an OPEN PDU with a non-zero Serial Number can not
resume from the requested point, it should return an ACK with an
Error Code of 2, Session could not be continued. The sender of the
failing OPEN PDU SHOULD then send an OPEN PDU with a Serial Number of
zero.
The Signature fields are described in Section 8 and in an asymmetric The Signature fields are described in Section 8 and in an asymmetric
key environment serve as a proof of possession of the signing auth key environment serve as a proof of possession of the signing auth
data by the sender. data by the sender.
Once two logical link endpoints know each other, and have ACKed each Once two logical link endpoints know each other, and have ACKed each
other's OPEN PDUs, Layer 2 KEEPALIVEs (see Section 15) MAY be started other's OPEN PDUs, Layer-2 KEEPALIVEs (see Section 15) MAY be started
to ensure Layer 2 liveness and keep the session semantics alive. The to ensure Layer-2 liveness and keep the session semantics alive. The
timing and acceptable drop of KEEPALIVE PDUs are discussed in timing and acceptable drop of KEEPALIVE PDUs are discussed in
Section 15. Section 15.
If a sender of OPEN does not receive an ACK of the OPEN PDU, then If a sender of OPEN does not receive an ACK of the OPEN PDU, then
they MUST resend the same OPEN PDU, with the same Nonce. Resending they MUST resend the same OPEN PDU, with the same Nonce. Resending
an unacknowledged OPEN PDU, like other ACKed PDUs, SHOULD use an unacknowledged OPEN PDU, like other ACKed PDUs, SHOULD use
exponential back-off, see [RFC1122]. exponential back-off, see [RFC1122].
If a properly authenticated OPEN arrives at L3DL speaker A with a new If a properly authenticated OPEN arrives at L3DL speaker A with a new
Nonce from an LLEI, speaker B, with which A believes it already has Nonce from an LLEI, speaker B, with which A believes it already has
an L3DL session (OPENs have already been exchanged), and the Serial an L3DL session (OPENs have already been exchanged), and the Serial
Number in the OPEN PDU is non-zero, speaker A SHOULD establish a new Number in the OPEN PDU is non-zero, speaker A SHOULD establish a new
session by sending an OPEN with the Serial Number being the same as sending session by sending an OPEN with the Serial Number being the
that of A's last sent and ACKed PDU. Each party MUST resume sending same as that of A's last sent and ACKed PDU. A MUST resume sending
encapsulations etc. subsequent to the other party's Sequence Number. encapsulations etc. subsequent to the requested Sequence Number. And
And each MUST retain all previously discovered encapsulation and B MUST retain all previously discovered encapsulation and other data
other data. received from A.
If a properly authenticated OPEN arrives with a new Nonce from an If a properly authenticated OPEN arrives with a new Nonce from an
LLEI with which the receiving logical link endpoint believes it LLEI with which the receiving logical link endpoint believes it
already has an L3DL session (OPENs have already been exchanged), and already has an L3DL session (OPENs have already been exchanged), and
the Serial Number in the OPEN is zero, then the receiver MUST assume the Serial Number in the OPEN is zero, then the receiver MUST assume
that the sending LLEI or entire device has been reset. All that the sending LLEI or entire device has been reset. All
previously discovered encapsulation data MUST NOT be kept and MUST BE Previously discovered encapsulation data MUST NOT be kept and MUST BE
withdrawn via the BGP-LS API and the recipient MUST respond with a withdrawn via the BGP-LS API and the recipient MUST respond with a
new OPEN. new OPEN.
12. ACK 12. ACK
The ACK PDU acknowledges receipt of a PDU and reports any error The ACK PDU acknowledges receipt of a PDU and reports any error
condition which might have been raised. condition which might have been raised.
0 1 2 3 0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
skipping to change at page 19, line 40 skipping to change at page 21, line 22
be zero. be zero.
A non-zero EType is the receiver's way of telling the PDU's sender A non-zero EType is the receiver's way of telling the PDU's sender
that the receiver had problems processing the PDU. The Error Code that the receiver had problems processing the PDU. The Error Code
and Error Hint will tell the sender more detail about the error. and Error Hint will tell the sender more detail about the error.
The decimal value of EType gives a strong hint how the receiver The decimal value of EType gives a strong hint how the receiver
sending the ACK believes things should proceed: sending the ACK believes things should proceed:
0 - No Error, Error Code and Error Hint MUST be zero 0 - No Error, Error Code and Error Hint MUST be zero
1 - Warning, something not too serious happened, continue 1 - Warning, something not too serious happened, continue
2 - Session should not be continued, try to restart 2 - Session should not be continued, try to restart
3 - Restart is hopeless, call the operator 3 - Restart is hopeless, call the operator
4-15 - Reserved 4-15 - Reserved
The Error Codes, noting protocol failures, are listed in The Error Codes, noting protocol failures, are listed in
Section 22.4. Someone stuck in the 1990s might think the catenation Section 22.4. Someone stuck in the 1990s might think the catenation
of EType and Error Code as an echo of 0x1zzz, 0x2zzz, etc. They of EType and Error Code as an echo of 0x1zzz, 0x2zzz, etc. They
might be right; or not. might be right; or not.
The Error Hint, an arbitrary 16 bits, is any additional data the The Error Hint, an arbitrary 16 bits, is any additional data the
sender of the error PDU thinks will help the recipient or the sender of the error PDU thinks will help the recipient or the
debugger with the particular error. debugger with the particular error.
The Signature fields are described in Section 8. The Signature fields are described in Section 8.
12.1. Retransmission 12.1. Retransmission
If a PDU sender expects an ACK, e.g. for an OPEN, an Encapsulation, a If a PDU sender expects an ACK, e.g. for an OPEN, an Encapsulation, a
VENDOR PDU, etc., and does not receive the ACK for a configurable VENDOR PDU, etc., and does not receive the ACK for a configurable
time (default one second), and the interface is live at layer 2, the time (default one second), and the interface is live at layer-2, the
sender resends the PDU using exponential back-off, see [RFC1122]. sender resends the PDU using exponential back-off, see [RFC1122].
This cycle MAY be repeated a configurable number of times (default This cycle MAY be repeated a configurable number of times (default
three) before it is considered a failure. The session MAY BE three) before it is considered a failure. The session MAY BE
considered closed in this case of this ACK failure. considered closed in this case of this ACK failure.
If the link is broken at layer 2, retransmission MAY BE retried when If the link is broken at layer-2, retransmission MAY BE retried when
the link is restored. the link is restored.
13. The Encapsulations 13. The Encapsulations
Once the devices know each other's LLEIs, know each other's upper Once the devices know each other's LLEIs, know each other's upper
layer (L2.5 and L3) identities, have means to ensure link state, layer (L2.5 and L3) identities, have means to ensure link state,
etc., the L3DL session is considered established, and the devices etc., the L3DL session is considered established, and the devices
SHOULD exchange L3 interface encapsulations, L3 addresses, and L2.5 SHOULD exchange L3 interface encapsulations, L3 addresses, and L2.5
labels. labels.
The Encapsulation types the peers exchange may be IPv4 The Encapsulation types the peers exchange may be IPv4
(Section 13.3), IPv6 (Section 13.4), MPLS IPv4 (Section 13.6), MPLS (Section 13.3), IPv6 (Section 13.4), MPLS IPv4 (Section 13.6), MPLS
IPv6 (Section 13.7), and/or possibly others not defined here. IPv6 (Section 13.7), and/or possibly others not defined here.
The sender of an Encapsulation PDU MUST NOT assume that the peer is The sender of an Encapsulation PDU MUST NOT assume that the peer is
capable of the same Encapsulation Type. An ACK (Section 12) merely capable of the same Encapsulation Type. An ACK (Section 12) merely
acknowledges receipt. Only if both peers have sent the same acknowledges receipt. Only if both peers have sent the same
Encapsulation Type is it safe for Layer 3 protocols to assume that Encapsulation Type is it safe for Layer-3 protocols to assume that
they are compatible for that type. they are compatible for that type.
A receiver of an encapsulation might recognize an addressing A receiver of an encapsulation might recognize an addressing
conflict, such as both ends of the link trying to use the same conflict, such as both ends of the link trying to use the same
address. In this case, the receiver SHOULD respond with an error address. In this case, the receiver SHOULD respond with an error
(Error Code 2) ACK. As there may be other usable addresses or (Error Code 2) ACK. As there may be other usable addresses or
encapsulations, this error might log and continue, letting an upper encapsulations, this error might log and continue, letting an upper
layer topology builder deal with what works. layer topology builder deal with what works.
Further, to consider a logical link of a type to formally be Further, to consider a logical link of a type to formally be
skipping to change at page 21, line 51 skipping to change at page 23, line 47
If the length of an Encapsulation PDU exceeds the Datagram size limit If the length of an Encapsulation PDU exceeds the Datagram size limit
on media, the PDU is broken into multiple Datagrams. See Section 8. on media, the PDU is broken into multiple Datagrams. See Section 8.
The Signature fields are described in Section 8. The Signature fields are described in Section 8.
The Receiver MUST acknowledge the Encapsulation PDU with a Type=3, The Receiver MUST acknowledge the Encapsulation PDU with a Type=3,
ACK PDU (Section 12) with the Encapsulation Type being that of the ACK PDU (Section 12) with the Encapsulation Type being that of the
encapsulation being announced, see Section 12. encapsulation being announced, see Section 12.
If the Sender does not receive an ACK in a configurable interval If the Sender does not receive an ACK in a configurable interval
(default one second), and the interface is live at layer 2, they (default one second), and the interface is live at layer-2, they
SHOULD retransmit. After a user configurable number of failures SHOULD retransmit. After a user configurable number of failures
(default three), the L3DL session should be considered dead and the (default three), the L3DL session should be considered dead and the
OPEN process SHOULD be restarted. OPEN process SHOULD be restarted.
If the link is broken at layer 2, retransmission MAY BE retried if If the link is broken at layer-2, retransmission MAY BE retried if
data have not changed in the interim. data have not changed in the interim.
13.2. Encapsulaion Flags 13.2. Encapsulaion Flags
The Encapsulation Flags are a sequence of bit fields as follows: The Encapsulation Flags are a sequence of bit fields as follows:
0 1 2 3 4 ... 7 0 1 2 3 4 ... 7
+------------+------------+------------+------------+------------+ +------------+------------+------------+------------+------------+
| Ann/With | Primary | Under/Over | Loopback | Reserved ..| | Ann/With | Primary | Under/Over | Loopback | Reserved ..|
+------------+------------+------------+------------+------------+ +------------+------------+------------+------------+------------+
skipping to change at page 26, line 32 skipping to change at page 28, line 32
followed by Enterprise Data in a format defined for that Enterprise followed by Enterprise Data in a format defined for that Enterprise
Number and Ent Type. Number and Ent Type.
Ent Type allows a VENDOR PDU to be sub-typed in the event that the Ent Type allows a VENDOR PDU to be sub-typed in the event that the
vendor/enterprise needs multiple PDU types. vendor/enterprise needs multiple PDU types.
As with Encapsulation PDUs, a receiver of a VENDOR PDU MUST respond As with Encapsulation PDUs, a receiver of a VENDOR PDU MUST respond
with an ACK or an ERROR PDU. Similarly, a VENDOR PDU MUST only be with an ACK or an ERROR PDU. Similarly, a VENDOR PDU MUST only be
sent over an open session. sent over an open session.
15. KEEPALIVE - Layer 2 Liveness 15. KEEPALIVE - Layer-2 Liveness
0 1 2 3 0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| PDU Type = 2 | Payload Length = 0 ~ | PDU Type = 2 | Payload Length = 0 ~
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
~ | Sig Type = 0 | Signature Length = 0 | ~ | Sig Type = 0 | Signature Length = 0 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
L3DL devices SHOULD beacon frequent Layer 2 KEEPALIVE PDUs to ensure L3DL devices SHOULD beacon frequent Layer-2 KEEPALIVE PDUs to ensure
session continuity. The inter-KEEPALIVE interval is configurable, session continuity. The inter-KEEPALIVE interval is configurable,
with a default of ten seconds. A receiver may choose to ignore with a default of ten seconds. A receiver may choose to ignore
KEEPALIVE PDUs. KEEPALIVE PDUs.
An operational deployment MUST BE configured whether to use An operational deployment MUST BE configured whether to use
KEEPALIVEs or not, either globally, or as finely as to per-link KEEPALIVEs or not, either globally, or as finely as to per-link
granularity. Disagreement MAY result in repeated session failure and granularity. Disagreement MAY result in repeated session failure and
reestablishment. reestablishment.
KEEPALIVEs SHOULD be beaconed at a configured frequency. One per KEEPALIVEs SHOULD be beaconed at a configured frequency. One per
second is the default. Layer 3 liveness, such as BFD, may be more second is the default. Layer-3 liveness, such as BFD, may be more
(or less) aggressive. (or less) aggressive.
When a sender transmits a PDU which is not a KEEPALIVE, the sender When a sender transmits a PDU which is not a KEEPALIVE, the sender
SHOULD reset the KEEPALIVE timer. I.e. sending any PDU acts as a SHOULD reset the KEEPALIVE timer. I.e. sending any PDU acts as a
keepalive. Once the last fragment has been sent, the KEEPALIVE timer keepalive. Once the last fragment has been sent, the KEEPALIVE timer
SHOULD BE restarted. Do not wait for the ACK. SHOULD be restarted. Do not wait for the ACK.
If a KEEPALIVE or other PDUs have not been received from a peer with If a KEEPALIVE or other PDUs have not been received from a peer with
which a receiver has an open session for a configurable time (default which a receiver has an open session for a configurable time (default
30 seconds), the link SHOULD BE presumed down. The devices MAY keep 30 seconds), the link SHOULD be presumed down. The devices MAY keep
configuration state and restore it without retransmission if no data configuration state and restore it without retransmission if no data
have changed. Otherwise, a new session SHOULD BE established and new have changed. Otherwise, a new session SHOULD be established and new
Encapsulation PDUs exchanged. Encapsulation PDUs exchanged.
16. Layers 2.5 and 3 Liveness 16. Layers-2.5 and 3 Liveness
Layer 2 liveness may be continuously tested by KEEPALIVE PDUs, see Layer-2 liveness may be continuously tested by KEEPALIVE PDUs, see
Section 15. As layer 2.5 or layer 3 connectivity could still break, Section 15. As layer-2.5 or layer-3 connectivity could still break,
liveness above layer 2 MAY be frequently tested using BFD ([RFC5880]) liveness above layer-2 MAY be frequently tested using BFD ([RFC5880])
or a similar technique. or a similar technique.
This protocol assumes that one or more Encapsulation addresses may be This protocol assumes that one or more Encapsulation addresses may be
used to ping, run BFD, or whatever the operator configures. used to ping, run BFD, or whatever the operator configures.
17. The North/South Protocol 17. The North/South Protocol
Thus far, a one-hop point-to-point logical link discovery protocol Thus far, a one-hop point-to-point logical link discovery protocol
has been defined. has been defined.
skipping to change at page 28, line 39 skipping to change at page 30, line 39
Label Sub-TLVs from [I-D.ietf-idr-bgp-ls-segment-routing-ext] Label Sub-TLVs from [I-D.ietf-idr-bgp-ls-segment-routing-ext]
Section 2.1.1, are used to associate one or more MPLS Labels with a Section 2.1.1, are used to associate one or more MPLS Labels with a
link. link.
18. Discussion 18. Discussion
This section explores some trade-offs taken and some considerations. This section explores some trade-offs taken and some considerations.
18.1. HELLO Discussion 18.1. HELLO Discussion
A device with multiple Layer 2 interfaces, traditionally called a A device with multiple Layer-2 interfaces, traditionally called a
switch, may be used to forward frames and therefore packets from switch, may be used to forward frames and therefore packets from
multiple devices to one logical interface (LLEI), I, on an L3DL multiple devices to one logical interface (LLEI), I, on an L3DL
speaking device. Interface I could discover a peer J across the speaking device. Interface I could discover a peer J across the
switch. Later, a prospective peer K could come up across the switch. switch. Later, a prospective peer K could come up across the switch.
If I was not still sending and listening for HELLOs, the potential If I was not still sending and listening for HELLOs, the potential
peering with K could not be discovered. Therefore, on multi-link peering with K could not be discovered. Therefore, on multi-link
interfaces, L3DL MUST continue to send HELLOs as long as they are interfaces, L3DL MUST continue to send HELLOs as long as they are
turned up. turned up.
18.2. HELLO versus KEEPALIVE 18.2. HELLO versus KEEPALIVE
skipping to change at page 30, line 17 skipping to change at page 32, line 17
The protocol as is MUST NOT be used outside a datacenter or similarly The protocol as is MUST NOT be used outside a datacenter or similarly
closed environment without authentication and authorization closed environment without authentication and authorization
mechanisms such as [I-D.ymbk-lsvr-l3dl-signing]. mechanisms such as [I-D.ymbk-lsvr-l3dl-signing].
Many MDC operators have a strange belief that physical walls and Many MDC operators have a strange belief that physical walls and
firewalls provide sufficient security. This is not credible. All firewalls provide sufficient security. This is not credible. All
MDC protocols need to be examined for exposure and attack surface. MDC protocols need to be examined for exposure and attack surface.
In the case of L3DL, Authentication and Integrity as provided in In the case of L3DL, Authentication and Integrity as provided in
[I-D.ymbk-lsvr-l3dl-signing] is strongly recommended. [I-D.ymbk-lsvr-l3dl-signing] is strongly recommended.
It is generally unwise to assume that on the wire Layer 2 is secure. It is generally unwise to assume that on the wire Layer-2 is secure.
Strange/unauthorized devices may plug into a port. Mis-wiring is Strange/unauthorized devices may plug into a port. Mis-wiring is
very common in datacenter installations. A poisoned laptop might be very common in datacenter installations. A poisoned laptop might be
plugged into a device's port, form malicious sessions, etc. to plugged into a device's port, form malicious sessions, etc. to
divert, intercept, or drop traffic. divert, intercept, or drop traffic.
Similarly, malicious nodes/devices could mis-announce addressing. Similarly, malicious nodes/devices could mis-announce addressing.
If OPENs are not being authenticated, an attacker could forge an OPEN If OPENs are not being authenticated, an attacker could forge an OPEN
for an existing session and cause the session to be reset. for an existing session and cause the session to be reset.
skipping to change at page 32, line 29 skipping to change at page 34, line 35
This document requires a new multicast MAC address that will be This document requires a new multicast MAC address that will be
broadcast through a switch. broadcast through a switch.
24. Acknowledgments 24. Acknowledgments
The authors thank Cristel Pelsser for multiple reviews, Harsha Kovuru The authors thank Cristel Pelsser for multiple reviews, Harsha Kovuru
for comments during implementation, Jeff Haas for review and for comments during implementation, Jeff Haas for review and
comments, Joerg Ott for an early but deep transport review, Joe comments, Joerg Ott for an early but deep transport review, Joe
Clarke for a useful review, John Scudder for deeply serious review Clarke for a useful review, John Scudder for deeply serious review
and comments, Larry Kreeger for a lot of layer 2 clue, Martijn and comments, Larry Kreeger for a lot of layer-2 clue, Martijn
Schmidt for his contribution, Nalinaksh Pai for transport Schmidt for his contribution, Nalinaksh Pai for transport
discussions, Neeraj Malhotra for review, Paul Congdon for Ethernet discussions, Neeraj Malhotra for review, Paul Congdon for Ethernet
hints, Russ Housley for checksum discussion and sBox, and Steve hints, Russ Housley for checksum discussion and sBox, and Steve
Bellovin for checksum advice. Bellovin for checksum advice.
25. References 25. References
25.1. Normative References 25.1. Normative References
[I-D.ietf-idr-bgp-ls-segment-routing-ext] [I-D.ietf-idr-bgp-ls-segment-routing-ext]
Previdi, S., Talaulikar, K., Filsfils, C., Gredler, H., Previdi, S., Talaulikar, K., Filsfils, C., Gredler, H.,
and M. Chen, "BGP Link-State extensions for Segment and M. Chen, "Border Gateway Protocol - Link State (BGP-
Routing", draft-ietf-idr-bgp-ls-segment-routing-ext-16 LS) Extensions for Segment Routing", Work in Progress,
(work in progress), June 2019. Internet-Draft, draft-ietf-idr-bgp-ls-segment-routing-ext-
18, 15 April 2021, <https://www.ietf.org/archive/id/draft-
ietf-idr-bgp-ls-segment-routing-ext-18.txt>.
[I-D.ietf-idr-bgpls-segment-routing-epe] [I-D.ietf-idr-bgpls-segment-routing-epe]
Previdi, S., Talaulikar, K., Filsfils, C., Patel, K., Ray, Previdi, S., Talaulikar, K., Filsfils, C., Patel, K., Ray,
S., and J. Dong, "BGP-LS extensions for Segment Routing S., and J. Dong, "Border Gateway Protocol - Link State
BGP Egress Peer Engineering", draft-ietf-idr-bgpls- (BGP-LS) Extensions for Segment Routing BGP Egress Peer
segment-routing-epe-19 (work in progress), May 2019. Engineering", Work in Progress, Internet-Draft, draft-
ietf-idr-bgpls-segment-routing-epe-19, 16 May 2019,
<https://www.ietf.org/archive/id/draft-ietf-idr-bgpls-
segment-routing-epe-19.txt>.
[I-D.ietf-lsvr-bgp-spf] [I-D.ietf-lsvr-bgp-spf]
Patel, K., Lindem, A., Zandi, S., and W. Henderickx, Patel, K., Lindem, A., Zandi, S., and W. Henderickx, "BGP
"Shortest Path Routing Extensions for BGP Protocol", Link-State Shortest Path First (SPF) Routing", Work in
draft-ietf-lsvr-bgp-spf-11 (work in progress), August Progress, Internet-Draft, draft-ietf-lsvr-bgp-spf-15, 1
2020. July 2021, <https://www.ietf.org/archive/id/draft-ietf-
lsvr-bgp-spf-15.txt>.
[I-D.ymbk-lsvr-l3dl-signing] [I-D.ymbk-lsvr-l3dl-signing]
Bush, R. and R. Austein, "Layer 3 Discovery and Liveness Bush, R. and R. Austein, "Layer 3 Discovery and Liveness
Signing", draft-ymbk-lsvr-l3dl-signing-01 (work in Signing", Work in Progress, Internet-Draft, draft-ymbk-
progress), May 2020. lsvr-l3dl-signing-01, 6 May 2020,
<https://www.ietf.org/archive/id/draft-ymbk-lsvr-l3dl-
signing-01.txt>.
[IANA-PEN] [IANA-PEN] "IANA Private Enterprise Numbers",
"IANA Private Enterprise Numbers",
<https://www.iana.org/assignments/enterprise-numbers/ <https://www.iana.org/assignments/enterprise-numbers/
enterprise-numbers>. enterprise-numbers>.
[IEEE.802_2001] [IEEE.802_2001]
IEEE, "IEEE Standard for Local and Metropolitan Area IEEE, "IEEE Standard for Local and Metropolitan Area
Networks: Overview and Architecture", IEEE 802-2001, Networks: Overview and Architecture", IEEE 802-2001,
DOI 10.1109/ieeestd.2002.93395, July 2002, DOI 10.1109/ieeestd.2002.93395, 27 July 2002,
<http://ieeexplore.ieee.org/servlet/opac?punumber=7732>. <http://ieeexplore.ieee.org/servlet/opac?punumber=7732>.
[IEEE802-2014] [IEEE802-2014]
Institute of Electrical and Electronics Engineers, "Local Institute of Electrical and Electronics Engineers, "Local
and Metropolitan Area Networks: Overview and and Metropolitan Area Networks: Overview and
Architecture", IEEE Std 802-2014, 2014. Architecture", IEEE Std 802-2014, 2014.
[RFC1213] McCloghrie, K. and M. Rose, "Management Information Base [RFC1213] McCloghrie, K. and M. Rose, "Management Information Base
for Network Management of TCP/IP-based internets: MIB-II", for Network Management of TCP/IP-based internets: MIB-II",
STD 17, RFC 1213, DOI 10.17487/RFC1213, March 1991, STD 17, RFC 1213, DOI 10.17487/RFC1213, March 1991,
skipping to change at page 34, line 44 skipping to change at page 37, line 7
[Clos0] Clos, C., "A study of non-blocking switching networks [Clos0] Clos, C., "A study of non-blocking switching networks
[PAYWALLED]", Bell System Technical Journal 32 (2), pp [PAYWALLED]", Bell System Technical Journal 32 (2), pp
406-424, March 1953. 406-424, March 1953.
[Clos1] "Clos Network", [Clos1] "Clos Network",
<https://en.wikipedia.org/wiki/Clos_network/>. <https://en.wikipedia.org/wiki/Clos_network/>.
[I-D.malhotra-bess-evpn-lsoe] [I-D.malhotra-bess-evpn-lsoe]
Malhotra, N., Patel, K., and J. Rabadan, "LSoE-based PE-CE Malhotra, N., Patel, K., and J. Rabadan, "LSoE-based PE-CE
Control Plane for EVPN", draft-malhotra-bess-evpn-lsoe-00 Control Plane for EVPN", Work in Progress, Internet-Draft,
(work in progress), March 2019. draft-malhotra-bess-evpn-lsoe-00, 11 March 2019,
<https://www.ietf.org/archive/id/draft-malhotra-bess-evpn-
lsoe-00.txt>.
[JUPITER] Singh, A., Ong, J., Agarwal, A., Anderson, G., Armistead, [JUPITER] Singh, A., Ong, J., Agarwal, A., Anderson, G., Armistead,
A., Bannon, R., Boving, S., Desai, G., Felderman, B., A., Bannon, R., Boving, S., Desai, G., Felderman, B.,
Germano, P., Kanagala, A., Liu, H., Provost, J., Simmons, Germano, P., Kanagala, A., Liu, H., Provost, J., Simmons,
J., Tanda, E., Wanderer, J., Hoelzle, U., Stuart, S., and J., Tanda, E., Wanderer, J., Hรถlzle, U., Stuart, S., and
A. Vahdat, "Jupiter rising: a decade of clos topologies A. Vahdat, "Jupiter rising: a decade of clos topologies
and centralized control in Google's datacenter network", and centralized control in Google's datacenter network",
Communications of the ACM Vol. 59, pp. 88-97, Communications of the ACM Vol. 59, pp. 88-97,
DOI 10.1145/2975159, August 2016. DOI 10.1145/2975159, August 2016,
<https://doi.org/10.1145/2975159>.
[RFC0791] Postel, J., "Internet Protocol", STD 5, RFC 791, [RFC0791] Postel, J., "Internet Protocol", STD 5, RFC 791,
DOI 10.17487/RFC0791, September 1981, DOI 10.17487/RFC0791, September 1981,
<https://www.rfc-editor.org/info/rfc791>. <https://www.rfc-editor.org/info/rfc791>.
[RFC1122] Braden, R., Ed., "Requirements for Internet Hosts - [RFC1122] Braden, R., Ed., "Requirements for Internet Hosts -
Communication Layers", STD 3, RFC 1122, Communication Layers", STD 3, RFC 1122,
DOI 10.17487/RFC1122, October 1989, DOI 10.17487/RFC1122, October 1989,
<https://www.rfc-editor.org/info/rfc1122>. <https://www.rfc-editor.org/info/rfc1122>.
[RFC1982] Elz, R. and R. Bush, "Serial Number Arithmetic", RFC 1982, [RFC1982] Elz, R. and R. Bush, "Serial Number Arithmetic", RFC 1982,
DOI 10.17487/RFC1982, August 1996, DOI 10.17487/RFC1982, August 1996,
<https://www.rfc-editor.org/info/rfc1982>. <https://www.rfc-editor.org/info/rfc1982>.
Authors' Addresses Authors' Addresses
Randy Bush Randy Bush
Arrcus & Internet Initiative Japan Arrcus & Internet Initiative Japan
5147 Crystal Springs 5147 Crystal Springs
Bainbridge Island, WA 98110 Bainbridge Island, WA 98110
US United States of America
Email: randy@psg.com Email: randy@psg.com
Rob Austein Rob Austein
Arrcus, Inc Arrcus, Inc
Email: sra@hactrn.net Email: sra@hactrn.net
Keyur Patel Keyur Patel
Arrcus Arrcus
2077 Gateway Place, Suite #400 2077 Gateway Place, Suite #400
San Jose, CA 95119 San Jose, CA 95119
US United States of America
Email: keyur@arrcus.com Email: keyur@arrcus.com
 End of changes. 97 change blocks. 
165 lines changed or deleted 219 lines changed or added

This html diff was produced by rfcdiff 1.48. The latest version is available from http://tools.ietf.org/tools/rfcdiff/