< draft-ymbk-idr-l3nd-02.txt		draft-ymbk-idr-l3nd-03.txt >
	Network Working Group R. Bush		Network Working Group R. Bush
	Internet-Draft Arrcus & Internet Initiative Japan		Internet-Draft Arrcus & Internet Initiative Japan
	Intended status: Standards Track R. Housley		Intended status: Standards Track R. Housley
	Expires: 8 September 2022 Vigil Security		Expires: 22 September 2022 Vigil Security
	R. Austein		R. Austein
	Arrcus		Arrcus
	S. Hares		S. Hares
	Hickory Hill Consulting		Hickory Hill Consulting
	K. Patel		K. Patel
	Arrcus		Arrcus
	7 March 2022		21 March 2022
	Layer-3 Neighbor Discovery		Layer-3 Neighbor Discovery
	draft-ymbk-idr-l3nd-02		draft-ymbk-idr-l3nd-03
	Abstract		Abstract
	Data Centers where the topology is BGP-based need to discover		Data Centers where the topology is BGP-based need to discover
	neighbor IP addressing, IP Layer-3 BGP neighbors, etc. This Layer-3		neighbor IP addressing, IP Layer-3 BGP neighbors, etc. This Layer-3
	Neighbor Discovery protocol identifies BGP neighbor candidates.		Neighbor Discovery protocol identifies BGP neighbor candidates.
	Requirements Language		Requirements Language
	The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",		The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
	skipping to change at page 1, line 47 ¶		skipping to change at page 1, line 47 ¶
	Internet-Drafts are working documents of the Internet Engineering		Internet-Drafts are working documents of the Internet Engineering
	Task Force (IETF). Note that other groups may also distribute		Task Force (IETF). Note that other groups may also distribute
	working documents as Internet-Drafts. The list of current Internet-		working documents as Internet-Drafts. The list of current Internet-
	Drafts is at https://datatracker.ietf.org/drafts/current/.		Drafts is at https://datatracker.ietf.org/drafts/current/.
	Internet-Drafts are draft documents valid for a maximum of six months		Internet-Drafts are draft documents valid for a maximum of six months
	and may be updated, replaced, or obsoleted by other documents at any		and may be updated, replaced, or obsoleted by other documents at any
	time. It is inappropriate to use Internet-Drafts as reference		time. It is inappropriate to use Internet-Drafts as reference
	material or to cite them other than as "work in progress."		material or to cite them other than as "work in progress."
	This Internet-Draft will expire on 8 September 2022.		This Internet-Draft will expire on 22 September 2022.
	Copyright Notice		Copyright Notice
	Copyright (c) 2022 IETF Trust and the persons identified as the		Copyright (c) 2022 IETF Trust and the persons identified as the
	document authors. All rights reserved.		document authors. All rights reserved.
	This document is subject to BCP 78 and the IETF Trust's Legal		This document is subject to BCP 78 and the IETF Trust's Legal
	Provisions Relating to IETF Documents (https://trustee.ietf.org/		Provisions Relating to IETF Documents (https://trustee.ietf.org/
	license-info) in effect on the date of publication of this document.		license-info) in effect on the date of publication of this document.
	Please review these documents carefully, as they describe your rights		Please review these documents carefully, as they describe your rights
	skipping to change at page 2, line 28 ¶		skipping to change at page 2, line 28 ¶
	Table of Contents		Table of Contents
	1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3		1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3
	2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4		2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4
	3. Background . . . . . . . . . . . . . . . . . . . . . . . . . 4		3. Background . . . . . . . . . . . . . . . . . . . . . . . . . 4
	4. Inter-Link Protocol Overview . . . . . . . . . . . . . . . . 5		4. Inter-Link Protocol Overview . . . . . . . . . . . . . . . . 5
	4.1. L3ND Ladder Diagram . . . . . . . . . . . . . . . . . . . 5		4.1. L3ND Ladder Diagram . . . . . . . . . . . . . . . . . . . 5
	5. TLV PDUs . . . . . . . . . . . . . . . . . . . . . . . . . . 7		5. TLV PDUs . . . . . . . . . . . . . . . . . . . . . . . . . . 7
	6. HELLO . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8		6. HELLO . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
	7. TCP Set-Up . . . . . . . . . . . . . . . . . . . . . . . . . 10		7. TCP Set-Up . . . . . . . . . . . . . . . . . . . . . . . . . 9
	8. OPEN . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11		8. OPEN . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
	9. ACK . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13		9. ACK . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
	9.1. Retransmission . . . . . . . . . . . . . . . . . . . . . 14		9.1. Retransmission . . . . . . . . . . . . . . . . . . . . . 14
	10. The Encapsulations . . . . . . . . . . . . . . . . . . . . . 14		10. The Encapsulations . . . . . . . . . . . . . . . . . . . . . 14
	10.1. The Encapsulation PDU Skeleton . . . . . . . . . . . . . 15		10.1. The Encapsulation PDU Skeleton . . . . . . . . . . . . . 14
	10.2. Encapsulaion Flags . . . . . . . . . . . . . . . . . . . 16		10.2. Encapsulaion Flags . . . . . . . . . . . . . . . . . . . 16
	10.3. IPv4 Encapsulation . . . . . . . . . . . . . . . . . . . 17		10.3. IPv4 Encapsulation . . . . . . . . . . . . . . . . . . . 16
	10.4. IPv6 Encapsulation . . . . . . . . . . . . . . . . . . . 17		10.4. IPv6 Encapsulation . . . . . . . . . . . . . . . . . . . 17
	10.5. MPLS Label List . . . . . . . . . . . . . . . . . . . . 18		10.5. MPLS Label List . . . . . . . . . . . . . . . . . . . . 18
	10.6. MPLS IPv4 Encapsulation . . . . . . . . . . . . . . . . 19		10.6. MPLS IPv4 Encapsulation . . . . . . . . . . . . . . . . 18
	10.7. MPLS IPv6 Encapsulation . . . . . . . . . . . . . . . . 19		10.7. MPLS IPv6 Encapsulation . . . . . . . . . . . . . . . . 19
	11. VENDOR - Vendor Extensions . . . . . . . . . . . . . . . . . 20		11. VENDOR - Vendor Extensions . . . . . . . . . . . . . . . . . 19
	12. Discussion . . . . . . . . . . . . . . . . . . . . . . . . . 21		12. Discussion . . . . . . . . . . . . . . . . . . . . . . . . . 20
	12.1. HELLO Discussion . . . . . . . . . . . . . . . . . . . . 21		12.1. HELLO Discussion . . . . . . . . . . . . . . . . . . . . 20
	13. VLANs/SVIs/Sub-interfaces . . . . . . . . . . . . . . . . . . 21		13. VLANs/SVIs/Sub-interfaces . . . . . . . . . . . . . . . . . . 20
	14. Implementation Considerations . . . . . . . . . . . . . . . . 21		14. Implementation Considerations . . . . . . . . . . . . . . . . 21
	15. Security Considerations . . . . . . . . . . . . . . . . . . . 22		15. Security Considerations . . . . . . . . . . . . . . . . . . . 21
	16. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 22		16. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 22
	16.1. Link Local Layer-3 Addresses . . . . . . . . . . . . . . 23		16.1. Link Local Layer-3 Addresses . . . . . . . . . . . . . . 22
	16.2. Ports for TLS/TCP . . . . . . . . . . . . . . . . . . . 23		16.2. Ports for TLS/TCP . . . . . . . . . . . . . . . . . . . 22
	16.3. PDU Types . . . . . . . . . . . . . . . . . . . . . . . 23		16.3. PDU Types . . . . . . . . . . . . . . . . . . . . . . . 22
	16.4. Flag Bits . . . . . . . . . . . . . . . . . . . . . . . 23		16.4. Flag Bits . . . . . . . . . . . . . . . . . . . . . . . 23
	16.5. Error Codes . . . . . . . . . . . . . . . . . . . . . . 24		16.5. Error Codes . . . . . . . . . . . . . . . . . . . . . . 23
	17. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 24		17. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 23
	18. References . . . . . . . . . . . . . . . . . . . . . . . . . 24		18. References . . . . . . . . . . . . . . . . . . . . . . . . . 23
	18.1. Normative References . . . . . . . . . . . . . . . . . . 24		18.1. Normative References . . . . . . . . . . . . . . . . . . 23
	18.2. Informative References . . . . . . . . . . . . . . . . . 25		18.2. Informative References . . . . . . . . . . . . . . . . . 24
	Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 26		Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 25
	1. Introduction		1. Introduction
	The Massive Data Center (MDC) environment presents unusual problems		The Massive Data Center (MDC) environment presents unusual problems
	of scale, e.g. O(10,000) forwarding devices, while its homogeneity		of scale, e.g. O(10,000) forwarding devices, while its homogeneity
	presents opportunities for simple approaches. Layer-3 Discovery and		presents opportunities for simple approaches. Layer-3 Discovery and
	Liveness (L3DL), [I-D.ietf-lsvr-l3dl], provides neighbor discovery at		Liveness (L3DL), [I-D.ietf-lsvr-l3dl], provides neighbor discovery at
	Layer-2. This document (set) provides a similar solution at Layer-3,		Layer-2. This document (set) provides a similar solution at Layer-3,
	attempting to be as similar as reasonable to L3DL.		attempting to be as similar as reasonable to L3DL.
	skipping to change at page 3, line 33 ¶		skipping to change at page 3, line 33 ¶
	* Security: Authorization and Integrity more than Confidentiality,		* Security: Authorization and Integrity more than Confidentiality,
	and		and
	* Massive Scalability		* Massive Scalability
	Layer-3 Neighbor Discovery (L3ND) provides brutally simple mechanisms		Layer-3 Neighbor Discovery (L3ND) provides brutally simple mechanisms
	for neighboring devices to		for neighboring devices to
	* Discover each other's IP Addresses,		* Discover each other's IP Addresses,
	* Discover mutually supported layer-3 encapsulations, e.g. IP/MPLS,		* Discover mutually supported layer-3 encapsulations, e.g. IPv4/
			IPv6//MPLS,
	* Discover Layer-3 IP and/or MPLS addressing of interfaces of the		* Discover Layer-3 IP and/or MPLS addressing of interfaces of the
	encapsulations,		encapsulations,
	* Provide authenticity, integrity, and verification of protocol		* Provide authenticity, integrity, and verification of protocol
	messages, and		messages, and
	* Accommodate scaling needed for EVPN etc.		* Accommodate scaling needed for EVPN etc.
	L3DN is intended for use within single IP subnets (IP over Ethernet		L3ND is intended for use within single IP subnets (IP over Ethernet
	or other point-to-point or multi-point IP link) in order to exchange		or other point-to-point or multi-point IP link) in order to exchange
	the data needed to bootstrap BGP-based peering, EVPN, etc.;		the data needed to bootstrap BGP-based peering, EVPN, etc.;
	especially in a datacenter Clos [Clos] environment. Once IP		especially in a datacenter Clos [Clos] environment. Once IP
	connectivity has been leveraged to discover1 layer-3 addressability		connectivity has been leveraged to discover layer-3 addressability
	and forwarding capabilities, normal IP forwarding and routing can		and forwarding capabilities, normal IP forwarding and routing can
	take over.		take over.
	L3ND might be found to be widely applicable to a range of routing and		L3ND might be more widely applicable to a range of routing and
	similar protocols which need Layer-3 neighbor discovery.		similar protocols which need Layer-3 neighbor discovery.
	2. Terminology		2. Terminology
	Even though it concentrates on the inter-device layer, this document		Even though it concentrates on the inter-device layer, this document
	relies heavily on routing terminology. The following attempts to		relies heavily on routing terminology. The following attempts to
	clarify the use of some possibly confusing terms:		clarify the use of some possibly confusing terms:
	Clos: A hierarchic subset of a crossbar switch topology commonly		Clos: A hierarchic subset of a crossbar switch topology commonly
	used in data centers [Clos].		used in data centers [Clos].
	skipping to change at page 4, line 34 ¶		skipping to change at page 4, line 34 ¶
	two different devices. E.g. two VLANs between the same		two different devices. E.g. two VLANs between the same
	two ports are two links.		two ports are two links.
	MDC: Massive Data Center, commonly composed of thousands of Top		MDC: Massive Data Center, commonly composed of thousands of Top
	of Rack Switches (TORs).		of Rack Switches (TORs).
	MTU: Maximum Transmission Unit, the size in octets of the		MTU: Maximum Transmission Unit, the size in octets of the
	largest packet that can be sent on a medium, see [RFC1122]		largest packet that can be sent on a medium, see [RFC1122]
	1.3.3.		1.3.3.
	PDU: Protocol Data Unit, an L3ND application layer message. A		PDU: Protocol Data Unit, an L3ND application layer message.
	PDU's content may need to be broken into multiple
	Datagrams to make it through MTU or other restrictions.
	Session: An established, via exchange of OPEN PDUs, session between		Session: An established, via exchange of OPEN PDUs, session between
	two L3ND capable IP interfaces on a link.		two L3ND capable IP interfaces on a link.
	TOR Switch: Top Of Rack switch, aggregates the servers in a rack and		TOR Switch: Top Of Rack switch, aggregates the servers in a rack and
	connects to aggregation layers of the Clos tree, AKA the		connects to aggregation layers of the Clos tree, AKA the
	Clos spine.		Clos spine.
	3. Background		3. Background
	skipping to change at page 10, line 33 ¶		skipping to change at page 10, line 16 ¶
	and authentication, both sides have agreed on an AFI for the		and authentication, both sides have agreed on an AFI for the
	transport and on each other's IP address in that AFI. This is		transport and on each other's IP address in that AFI. This is
	sufficient to open a TCP session between them, which will allow for		sufficient to open a TCP session between them, which will allow for
	reliable transport of very large data PDUs while obviating the need		reliable transport of very large data PDUs while obviating the need
	to invent complex transports.		to invent complex transports.
	The L3ND peer with the higher IP address MUST act as the TLS/TCP		The L3ND peer with the higher IP address MUST act as the TLS/TCP
	client and open the transport session (send SYN) toward the peer with		client and open the transport session (send SYN) toward the peer with
	the lower IP address.		the lower IP address.
	The server, the sender of the HELLO with from the lower IP address,		The server, the sender of the HELLO from the lower IP address,
	listens on the advertised port for the TLS/TCP session open. The		listens on the advertised port for the TLS/TCP session open. The
	receiver of the acceptable HELLO, the TLS/TCP client, initiates a TLS		receiver of the acceptable HELLO, the TLS/TCP client, initiates a TLS
	or raw TCP session toward the sender of the HELLO, the TLS/TCP		or raw TCP session toward the sender of the HELLO, the TLS/TCP
	server, preferably TLS, as advertised. If TLS, the server has chosen		server, preferably TLS, as advertised. If TLS, the server has chosen
	and signaled either a self-signed certificate or one configured from		and signaled either a self-signed certificate or one configured from
	the operational CA trusted by both parties, as negotiated in the		the operational CA trusted by both parties, as negotiated in the
	HELLO exchange.		HELLO exchange.
	Once the TLS/TCP session is established, if its interface is		Once the TLS/TCP session is established, if its interface is
	configured as point to point, the client side SHOULD stop listening		configured as point to point, the client side SHOULD stop listening
	on any port for which it has sent a HELLO. The server, if configured		on any port for which it has sent a HELLO. The server, if configured
	as a point to point interface SHOULD stop sending HELLOs.		as a point to point interface SHOULD stop sending HELLOs.
	If the TLS/TCP open fails, then this SHOULD be logged and the parties		If the TLS/TCP open fails, then this SHOULD be logged and the parties
	MUST go back to the initial state and try HELLO. Loggin SHOULD be		MUST go back to the initial state and try HELLO. Logging SHOULD be
	rate limited.		rate limited.
	Should an interface with an established TLS/TCP session be		Should an interface with an established TLS/TCP session be
	reconfigured changing the TLS/TCP parameters, the TLS/TCP session		reconfigured changing the TLS/TCP parameters, the TLS/TCP session
	should be closed or torn down and both parties should return to the		should be closed or torn down and both parties should return to the
	HELLO state.		HELLO state.
	Should the TLS/TCP session terminate for any reason, the devices		Should the TLS/TCP session terminate for any reason, the devices
	SHOULD restart/resume HELLOs. When the new TLS/TCP session is		SHOULD restart/resume HELLOs. When the new TLS/TCP session is
	started, if possible the OPEN PDU SHOULD try to resume the lost		started, if possible the OPEN PDU SHOULD try to resume the lost
	skipping to change at page 11, line 31 ¶		skipping to change at page 11, line 11 ¶
	Each device has learned the other's IP Address from the HELLO		Each device has learned the other's IP Address from the HELLO
	exchange, see Section 6 and established a TLS/TCP session over a		exchange, see Section 6 and established a TLS/TCP session over a
	particular AFI.		particular AFI.
	The first PDU each sends MUST be an OPEN, and the other side MUST		The first PDU each sends MUST be an OPEN, and the other side MUST
	respond with an ACK PDU.		respond with an ACK PDU.
	0 1 2 3		0 1 2 3
	0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1		0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
	+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+		+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
	| Version = 0 | PDU Type = 2 | Payload Length ~		| Version = 0 | PDU Type = 1 | Payload Length ~
	+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+		+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
	~ | Nonce ~		~ | Session ID ~
			+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
			~ | Serial Number ~
	+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+		+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
	~ | AttrCount | ~		~ | AttrCount | ~
	+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +		+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +
	~ Attribute List ... ~		~ Attribute List ... ~
	+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+		+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
	| Serial Number |
	+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
	The four octet Payload Length is the number of octets in all fields		The four octet Payload Length is the number of octets in all fields
	of the PDU from the Nonce through the Serial Number.		of the PDU from the Session ID through the Serial Number.
	The four octet Nonce enables detection of a duplicate OPEN PDU. It		The four octet Session ID is a nonce which uniquely identifies a
	SHOULD be either a random number or a high resolution timestamp. It		session. It enables detection of a duplicate OPEN PDU. It SHOULD be
	is needed to prevent session closure due to a repeated OPEN caused by		either a random number or a high resolution timestamp. It is needed
	a race or a dropped or delayed ACK. It can be used to resume a		to prevent session closure due to a repeated OPEN caused by a race or
	dropped logical session.		a dropped or delayed ACK. It can be used to resume a dropped logical
			session.
	The one octet AttrCount is the number of attributes in the Attribute		The one octet AttrCount is the number of attributes in the Attribute
	List. A node may send zero or more attributes.		List. A node may send zero or more attributes.
	Attributes are single octets the semantics of which are operator-		Attributes are single octets the semantics of which are operator-
	defined, e.g.: spine, leaf, backbone, route reflector, arabica, ...		defined, e.g.: spine, leaf, backbone, route reflector, arabica, ...
	Attribute syntax and semantics are local to an operator or		Attribute syntax and semantics are local to an operator or
	datacenter; hence there is no global registry. Nodes exchange their		datacenter; hence there is no global registry. Nodes exchange their
	attributes only in the OPEN PDU.		attributes only in the OPEN PDU.
	Unlike L3DL [I-D.ietf-lsvr-l3dl], there are no verifiable keys in the		Unlike L3DL [I-D.ietf-lsvr-l3dl], there are no verifiable keys in the
	PDUs. If the operator wants authentication, integrity,		PDUs. If the operator wants authentication, integrity,
	confidentiality, then TLS MUST have been requested by the HELLO and		confidentiality, then TLS MUST have been requested by the HELLO and
	agreed by the TLS session open.		agreed by the TLS session open.
	The Serial Number is a monotonically increasing four octet value		The Serial Number is a monotonically increasing four octet value
	representing the sender's state at the time of sending the last PDU.		representing the sender's state at the time of sending the last PDU.
	It may be an non-negative integer, a timestamp, etc. If incrementing		It may be a non-negative integer, a timestamp, etc. If incrementing
	the Serial Number would cause it to be zero, it should be incremented		the Serial Number would cause it to be zero, it should be incremented
	again.		again.
	On session restart (new OPEN, same Nonce), a receiver MAY send the		On session restart (new OPEN, same Session ID), a receiver MAY send
	last received Serial Number to tell the sender to only send data with		the last received Serial Number to tell the sender to only send data
	a Serial Number greater (in the [RFC1982] sense), or send a Serial		with a Serial Number greater (in the [RFC1982] sense), or send a
	Number of zero to request all data.		Serial Number of zero to request all data.
	This allows a sender of an OPEN to tell the receiver that the sender		This allows a sender of an OPEN to tell the receiver that the sender
	would like to resume a logical session and that the receiver of the		would like to resume a logical session and that the receiver of the
	OPEN PDU only needs to send data starting with the PDU with the		OPEN PDU only needs to send data starting with the PDU with the
	lowest Serial Number greater (in the [RFC1982] sense) than the one		lowest Serial Number greater (in the [RFC1982] sense) than the one
	sent in the OPEN. If the sender is not trying to resume a dropped		sent in the OPEN. If the sender is not trying to resume a dropped
	session, the Serial Number MUST be zero.		session, the Serial Number MUST be zero.
	If the receiver of an OPEN PDU with a non-zero Serial Number can not		If the receiver of an OPEN PDU with a non-zero Serial Number can not
	resume from the requested point, it should return an ACK with an		resume from the requested point, it should return an ACK with an
	skipping to change at page 13, line 19 ¶		skipping to change at page 12, line 42 ¶
	with the Serial Number being the same as that of A's last sent and		with the Serial Number being the same as that of A's last sent and
	ACKed PDU. A MUST resume sending encapsulations etc. subsequent to		ACKed PDU. A MUST resume sending encapsulations etc. subsequent to
	the requested Sequence Number. And B MUST retain all previously		the requested Sequence Number. And B MUST retain all previously
	discovered encapsulation and other data received from A.		discovered encapsulation and other data received from A.
	If an OPEN arrives at L3ND speaker A from B with which A believes it		If an OPEN arrives at L3ND speaker A from B with which A believes it
	already has an L3ND session (i.e. OPENs have already been		already has an L3ND session (i.e. OPENs have already been
	exchanged), and the Serial Number in B's OPEN is zero, then the A		exchanged), and the Serial Number in B's OPEN is zero, then the A
	MUST assume that B's internal state has been reset. All Previously		MUST assume that B's internal state has been reset. All Previously
	discovered encapsulation data MUST BE discarded; and A MUST respond		discovered encapsulation data MUST BE discarded; and A MUST respond
	with a new OPEN with a Serial Number of zero.		with a new OPEN PDU with a Serial Number of zero.
	TCP KeepAlives should be configured and tuned to meet local		TCP KeepAlives should be configured and tuned to meet local
	operational needs. Some defaults and recommendations are needed		operational needs. Some defaults and recommendations are needed
	here.		here.
	9. ACK		9. ACK
	The ACK PDU acknowledges receipt of a PDU and reports any error		The ACK PDU acknowledges receipt of a PDU and reports any error
	condition which might have been raised.		condition which might have been raised.
	0 1 2 3		0 1 2 3
	0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1		0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
	+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+		+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
	| Version = 0 | PDU Type = 3 | Payload Length = 6 |		| Version = 0 | PDU Type = 3 | Payload Length = 6 ~
	+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+		+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
	| | ACKed PDU | EType |		~ | ACKed PDU | EType |
	+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+		+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
	| Error Code | Error Hint |		| Error Code | Error Hint |
	+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+		+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
	The ACK PDU acknowledges receipt of an OPEN, Encapsulation, Vendor		The ACK PDU acknowledges receipt of an OPEN, Encapsulation, Vendor
	PDU, etc. and is used to return error codes if any.		PDU, etc. and is used to return error codes if any.
	The one octet ACKed PDU is the PDU Type of the PDU being		The one octet ACKed PDU is the PDU Type of the PDU being
	acknowledged, e.g., OPEN, one of the Encapsulations, etc.		acknowledged, e.g., OPEN, one of the Encapsulations, etc.
	skipping to change at page 16, line 12 ¶		skipping to change at page 15, line 36 ¶
	the last received Serial Number to request the sender to only send		the last received Serial Number to request the sender to only send
	newer data.		newer data.
	If a sender has multiple links on the same interface, separate state:		If a sender has multiple links on the same interface, separate state:
	data, ACKs, etc. must be kept for each peer session.		data, ACKs, etc. must be kept for each peer session.
	Over time, multiple Encapsulation PDUs may be sent for an interface		Over time, multiple Encapsulation PDUs may be sent for an interface
	in a session as configuration changes.		in a session as configuration changes.
	The Receiver MUST acknowledge the Encapsulation PDU with an ACK PDU		The Receiver MUST acknowledge the Encapsulation PDU with an ACK PDU
	(Section 9) with the PDU Type being that of the type PDU being		(Section 9) with the Type field being that of the Type of the
	announced, see Section 9.		Encapsulation PDU being announced, see Section 9.
	If the Sender does not receive an ACK in a configurable interval		If the Sender does not receive an ACK in a configurable interval
	(default five seconds), they SHOULD retransmit. After a user		(default five seconds), they SHOULD retransmit. After a user
	configurable number of failures (default three), the L3ND session		configurable number of failures (default three), the L3ND session
	should be considered dead, TLS/TCP torn down, and the HELLO process		should be considered dead, TLS/TCP torn down, and the HELLO process
	SHOULD be restarted.		SHOULD be restarted.
	If the link is broken below layer-3, retransmission MAY BE retried if		If the link is broken below layer-3, retransmission MAY BE retried if
	data have not changed in the interim and the TLS/TCP session is still		data have not changed in the interim and the TLS/TCP session is still
	alive.		alive.
	skipping to change at page 20, line 43 ¶		skipping to change at page 20, line 4 ¶
	0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1		0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
	+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+		+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
	| Version = 0 | PDU Type = 255| Payload Length ~		| Version = 0 | PDU Type = 255| Payload Length ~
	+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+		+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
	~ | Serial Number ~		~ | Serial Number ~
	+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+		+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
	~ | Enterprise Number ~		~ | Enterprise Number ~
	+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+		+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
	~ | Ent Type | Enterprise Data ... ~		~ | Ent Type | Enterprise Data ... ~
	+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+		+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
	Vendors or enterprises may define TLVs beyond the scope of L3ND		Vendors or enterprises may define TLVs beyond the scope of L3ND
	standards. This is done using a Private Enterprise Number [IANA-PEN]		standards. This is done using a Private Enterprise Number [IANA-PEN]
	followed by Enterprise Data in a format defined for that three octet		followed by Enterprise Data in a format defined for that three octet
	Enterprise Number and one octet Ent Type.		Enterprise Number and one octet Ent Type.
	Ent Type allows a Vendor PDU to be sub-typed in the event that the		Ent Type allows a Vendor PDU to be sub-typed in the event that the
	vendor/enterprise needs multiple PDU types.		vendor/enterprise needs multiple PDU types.
	As with Encapsulation PDUs, a receiver of a Vendor PDU MUST respond		As with Encapsulation PDUs, a receiver of a Vendor PDU MUST respond
	with an ACK or an ERROR PDU. Similarly, a Vendor PDU MUST only be		with an ACK PDU, possibly signalling an error. Similarly, a Vendor
	sent over an open session.		PDU MUST only be sent over an open session.
	12. Discussion		12. Discussion
	This section explores some trade-offs taken and some considerations.		This section explores some trade-offs taken and some considerations.
	12.1. HELLO Discussion		12.1. HELLO Discussion
	A device may send IP packets over a Layer-3 interface which transmits		A device may send IP packets over a Layer-3 interface which transmits
	data over a single Layer-2 interface or multiple Layer-2 interfaces.		data over a single Layer-2 interface or multiple Layer-2 interfaces.
	Packets sourced by one Layer-3 IP interface over multiple Layer-2		Packets sourced by one Layer-3 IP interface over multiple Layer-2
	skipping to change at page 22, line 24 ¶		skipping to change at page 21, line 29 ¶
	addresses of an encapsulation as primary on an L3ND speaking		addresses of an encapsulation as primary on an L3ND speaking
	interface. If there is only one address for a particular		interface. If there is only one address for a particular
	encapsulation, the implementation MAY mark it as primary by default.		encapsulation, the implementation MAY mark it as primary by default.
	An implementation MAY allow optional configuration which updates the		An implementation MAY allow optional configuration which updates the
	local forwarding table with overlay and underlay data both learned		local forwarding table with overlay and underlay data both learned
	from L3ND peers and configured locally.		from L3ND peers and configured locally.
	15. Security Considerations		15. Security Considerations
	For TLS, version 1.1 or higher MUST be used.		For TLS, versions greater than 1.1 MUST be used.
	The protocol as is MUST NOT be used outside a datacenter or similarly		The protocol as is MUST NOT be used outside a datacenter or similarly
	closed environment without using TLS encapsulation which is based on		closed environment without using TLS encapsulation which is based on
	a configured CA trust anchor.		a configured CA trust anchor.
	Many datacenter operators have a strange belief that physical walls		Many datacenter operators have a strange belief that physical walls
	and firewalls provide sufficient security. This is not credible.		and firewalls provide sufficient security. This is not credible.
	All DC protocols need to be examined for exposure and attack surface.		All DC protocols need to be examined for exposure and attack surface.
	In the case of L3ND, authentication and integrity as provided by TLS		In the case of L3ND, authentication and integrity as provided by TLS
	validated to a configured shared CA trust anchor is strongly		validated to a configured shared CA trust anchor is strongly
	skipping to change at page 25, line 46 ¶		skipping to change at page 25, line 4 ¶
	May 2017, <https://www.rfc-editor.org/info/rfc8174>.		May 2017, <https://www.rfc-editor.org/info/rfc8174>.
	18.2. Informative References		18.2. Informative References
	[Clos] "Clos Network",		[Clos] "Clos Network",
	<https://en.wikipedia.org/wiki/Clos_network/>.		<https://en.wikipedia.org/wiki/Clos_network/>.
	[I-D.ymbk-idr-l3nd-ulpc]		[I-D.ymbk-idr-l3nd-ulpc]
	Bush, R. and K. Patel, "L3ND Upper-Layer Protocol		Bush, R. and K. Patel, "L3ND Upper-Layer Protocol
	Configuration", Work in Progress, Internet-Draft, draft-		Configuration", Work in Progress, Internet-Draft, draft-
	ymbk-idr-l3nd-ulpc-02, 27 February 2022,		ymbk-idr-l3nd-ulpc-04, 21 March 2022,
	<https://www.ietf.org/archive/id/draft-ymbk-idr-l3nd-ulpc-		<https://www.ietf.org/archive/id/draft-ymbk-idr-l3nd-ulpc-
	02.txt>.		04.txt>.
	[RFC1122] Braden, R., Ed., "Requirements for Internet Hosts -		[RFC1122] Braden, R., Ed., "Requirements for Internet Hosts -
	Communication Layers", STD 3, RFC 1122,		Communication Layers", STD 3, RFC 1122,
	DOI 10.17487/RFC1122, October 1989,		DOI 10.17487/RFC1122, October 1989,
	<https://www.rfc-editor.org/info/rfc1122>.		<https://www.rfc-editor.org/info/rfc1122>.
	[RFC1982] Elz, R. and R. Bush, "Serial Number Arithmetic", RFC 1982,		[RFC1982] Elz, R. and R. Bush, "Serial Number Arithmetic", RFC 1982,
	DOI 10.17487/RFC1982, August 1996,		DOI 10.17487/RFC1982, August 1996,
	<https://www.rfc-editor.org/info/rfc1982>.		<https://www.rfc-editor.org/info/rfc1982>.
End of changes. 35 change blocks.
	59 lines changed or deleted		58 lines changed or added
This html diff was produced by rfcdiff 1.48. The latest version is available from http://tools.ietf.org/tools/rfcdiff/