< draft-ietf-sfc-nsh-ecn-support-04.txt		draft-ietf-sfc-nsh-ecn-support-05.txt >
	INTERNET-DRAFT Donald Eastlake		INTERNET-DRAFT D. Eastlake
	Intended status: Proposed Standard Futurewei Technologies		Intended status: Proposed Standard Futurewei Technologies
	Bob Briscoe		B. Briscoe
	Independent
	Andrew Malis
	Independent		Independent
	Expires: June 5, 2021 December 6, 2020		Y. Li
			Huawei Technologies
			A Malis
			Malis Consulting
			X. Wei
			Huawei Technologies
			Expires: October 1, 2021 April 2, 2021
	Explicit Congestion Notification (ECN) and Congestion Feedback		Explicit Congestion Notification (ECN) and Congestion Feedback
	Using the Network Service Header (NSH)		Using the Network Service Header (NSH) and IPFIX
	<draft-ietf-sfc-nsh-ecn-support-04.txt>		<draft-ietf-sfc-nsh-ecn-support-05.txt>
	Abstract		Abstract
	Explicit congestion notification (ECN) allows a forwarding element to		Explicit congestion notification (ECN) allows a forwarding element to
	notify downstream devices of the onset of congestion without having		notify downstream devices of the onset of congestion without having
	to drop packets. Coupled with a means to feed information about		to drop packets. Coupled with a means to feed information about
	congestion back to upstream nodes, this can improve network		congestion back to upstream nodes, this can improve network
	efficiency through better congestion control, frequently without		efficiency through better congestion control, frequently without
	packet drops. This document specifies ECN and congestion feedback		packet drops. This document specifies ECN and congestion feedback
	support within a Service Function Chaining (SFC) architecture domain		support within a Service Function Chaining (SFC) architecture domain
	through use of the Network Service Header (NSH, RFC 8300) and IP Flow		through use of the Network Service Header (NSH, RFC 8300) and IP Flow
	Information Export (IPFIX,		Information Export (IPFIX, RFC 7011).
	draft-ietf-tsvwg-tunnel-congestion-feedback).
	Status of This Memo		Status of This Memo
	This Internet-Draft is submitted in full conformance with the		This Internet-Draft is submitted in full conformance with the
	provisions of BCP 78 and BCP 79.		provisions of BCP 78 and BCP 79.
	Distribution of this document is unlimited. Comments should be sent		Distribution of this document is unlimited. Comments should be sent
	to the SFC Working Group mailing list <sfc@ietf.org> or to the		to the SFC Working Group mailing list <sfc@ietf.org> or to the
	authors.		authors.
	Internet-Drafts are working documents of the Internet Engineering		Internet-Drafts are working documents of the Internet Engineering
	Task Force (IETF), its areas, and its working groups. Note that		Task Force (IETF), its areas, and its working groups. Note that
	other groups may also distribute working documents as Internet-		other groups may also distribute working documents as Internet-
	Drafts.		Drafts.
	Internet-Drafts are draft documents valid for a maximum of six months		Internet-Drafts are draft documents valid for a maximum of six months
	and may be updated, replaced, or obsoleted by other documents at any		and may be updated, replaced, or obsoleted by other documents at any
	time. It is inappropriate to use Internet-Drafts as reference		time. It is inappropriate to use Internet-Drafts as reference
	material or to cite them other than as "work in progress."		material or to cite them other than as "work in progress."
			INTERNET-DRAFT NSH ECN & Congestion Feedback
	The list of current Internet-Drafts can be accessed at		The list of current Internet-Drafts can be accessed at
	http://www.ietf.org/1id-abstracts.html. The list of Internet-Draft		http://www.ietf.org/1id-abstracts.html. The list of Internet-Draft
	Shadow Directories can be accessed at		Shadow Directories can be accessed at
	http://www.ietf.org/shadow.html.		http://www.ietf.org/shadow.html.
	INTERNET-DRAFT NSH ECN & Congestion Feedback		INTERNET-DRAFT NSH ECN & Congestion Feedback
	Table of Contents		Table of Contents
	1. Introduction............................................3		1. Introduction............................................4
	1.1 NSH Background.........................................3		1.1 NSH Background.........................................4
	1.2 ECN Background.........................................5		1.2 ECN Background.........................................6
	1.3 Tunnel Congestion Feedback Background..................5		1.3 Tunnel Congestion Feedback Background..................6
	1.4 Conventions Used in This Document......................7		1.4 Conventions Used in This Document......................8
	2. The NSH ECN Field.......................................8		2. The NSH ECN Field......................................10
	3. ECN Support in the NSH.................................10		3. ECN Support in the NSH.................................12
	3.1 At The Ingress........................................11		3.1 At The Ingress........................................13
	3.2 At Transit Nodes......................................12		3.2 At Transit Nodes......................................14
	3.2.1 At NSH Transit Nodes................................12		3.2.1 At NSH Transit Nodes................................14
	3.2.2 At an SF/Proxy......................................13		3.2.2 At an SF/Proxy......................................15
	3.2.3 At Other Forwarding Nodes...........................13		3.2.3 At Other Forwarding Nodes...........................15
	3.3 At Exit/Egress........................................13		3.3 At Exit/Egress........................................16
	3.4 Conservation of Packets...............................14		3.4 Conservation of Packets...............................16
	4. Tunnel Congestion Feedback Support.....................15		4. Tunnel Congestion Feedback Support.....................18
			4.1 Congestion Level Measurement..........................18
			4.3 Congestion Information Delivery.......................19
			4.3 IPFIX Extensions......................................21
			4.3.1 nshServicePathID....................................21
			4.3.2 tunnelEcnCeCeByteTotalCount.........................21
			4.3.3 tunnelEcnEctNectBytetTotalCount.....................22
			4.3.4 tunnelEcnCeNectByteTotalCount.......................22
			4.3.5 tunnelEcnCeEctByteTotalCount........................22
			4.3.6 tunnelEcnEctEctByteTotalCount.......................23
			4.3.7 tunnelEcnCEMarkedRatio..............................23
	5. IANA Considerations....................................16		5. Example of Use.........................................24
	5.1 SFC NSH Header ECN Bits...............................16
	5.2 IPFIX Information Element ID..........................16
	6. Security Considerations................................17		6. IANA Considerations....................................27
			6.1 SFC NSH Header ECN Bits...............................27
			6.2 IPFIX Information Element IDs.........................27
	7. Acknowledgements.......................................17		7. Security Considerations................................29
			8. Acknowledgements.......................................29
	Normative References......................................18		Normative References......................................30
	Informative References....................................18		Informative References....................................31
	Authors' Addresses........................................20		Authors' Addresses........................................32
	INTERNET-DRAFT NSH ECN & Congestion Feedback		INTERNET-DRAFT NSH ECN & Congestion Feedback
	1. Introduction		1. Introduction
	Explicit congestion notification (ECN [RFC3168]) allows a forwarding		Explicit Congestion Notification (ECN [RFC3168]) allows a forwarding
	element to notify downstream devices of the onset of congestion		element to notify downstream devices of the onset of congestion
	without having to drop packets. Coupled with a means to feed		without having to drop packets. Coupled with a means to feed
	information about congestion back to upstream nodes, this can improve		information about congestion back to upstream nodes, this can improve
	network efficiency through better congestion control, frequently		network efficiency through better congestion control, frequently
	without packet drops. This document specifies ECN and congestion		without packet drops. This document specifies ECN and congestion
	feedback support within a Service Function Chaining (SFC [RFC7665])		feedback support within a Service Function Chaining (SFC [RFC7665])
	architecture domain through use of the Network Service Header (NSH		architecture domain through use of the Network Service Header (NSH
	[RFC8300]) and IP Flow Information Export (IPFIX		[RFC8300]) and IP Flow Information Export (IPFIX [RFC7011]).
	[TunnelCongFeedback]).
	It requires that all ingress and egress nodes of the SFC domain		It requires that all ingress and egress nodes of the SFC domain
	implement ECN. While congestion management will be the most effective		implement ECN. While congestion management will be the most effective
	if all interior nodes of the SFC domain implement ECN, some benefit		if all interior nodes of the SFC domain implement ECN, some benefit
	is obtained even if some interior nodes do not implement ECN.		is obtained even if some interior nodes do not implement ECN.
	Congestion at any interior bottleneck where ECN marking is not		Congestion at any interior bottleneck where ECN marking is not
	implemented will be unmanaged.		implemented will be unmanaged.
	The subsections below in this section provide background information		The subsections below in this section provide background information
	on NSH, ECN, congestion feedback, and terminology used in this		on NSH, ECN, congestion feedback, and terminology used in this
	skipping to change at page 5, line 26 ¶		skipping to change at page 6, line 26 ¶
	Service Function (SF)) to notify downstream devices of the onset of		Service Function (SF)) to notify downstream devices of the onset of
	congestion without having to drop packets. This can be used as an		congestion without having to drop packets. This can be used as an
	element in active queue management (AQM) [RFC7567] to improve network		element in active queue management (AQM) [RFC7567] to improve network
	efficiency through better traffic control without packet drops. The		efficiency through better traffic control without packet drops. The
	forwarding element can explicitly mark some packets in an ECN field		forwarding element can explicitly mark some packets in an ECN field
	instead of dropping the packet. For example, a two-bit field is		instead of dropping the packet. For example, a two-bit field is
	available for ECN marking in IP headers [RFC3168].		available for ECN marking in IP headers [RFC3168].
	1.3 Tunnel Congestion Feedback Background		1.3 Tunnel Congestion Feedback Background
	Tunnel Congestion Feedback [TunnelCongFeedback] is a building block		Tunnels are widely deployed in various networks including data center
	for various congestion mitigation methods. It supports feedback of		networks, enterprise network, and the public Internet. A tunnel
	congestion information from an egress node to an ingress node. This		consists of ingress, egress, and a set of intermediate nodes
	document treats the SFC domain as a tunnel with the initial		including routers. Tunnel Congestion Feedback (Section 4) is a
	Classifier node being the ingress.		building block for congestion mitigation methods. It supports
			feedback of congestion information from an egress node to an ingress
			node. This document treats the SFC domain as a tunnel with the
			initial Classifier node being the ingress; however, the Tunnel
			Congestion Feedback facilities specified in this document MAY be used
			in other contexts besides SFC domains.
	Examples of actions that can be taken by an ingress node when it has		Examples of actions that can be taken by an ingress node when it has
	knowledge of downstream congestion include those listed below.		knowledge of downstream congestion include those listed below.
	Details of implementing these traffic control methods, beyond those		Details of implementing these traffic control methods, beyond those
	given here, are outside the scope of this document.		given here, are outside the scope of this document.
	Any action by a tunnel ingress to reduce congestion needs to allow		Any action by a tunnel ingress to reduce congestion needs to allow
	sufficient time for the end-to-end congestion control loop to respond		sufficient time for the end-to-end congestion control loop to respond
	first, for instance by the ingress taking a smoothed average of the		first, otherwise the system could go unstable. For instance by the
	level of congestion signalled by feedback from the tunnel egress.		ingress taking a smoothed average of the level of congestion signaled
			by feedback from the tunnel egress or delaying any action for at
			least the worst case global round trip time (for example 100
			milliseconds).
	(1) Traffic throttling (policing), where the downstream traffic		(1) Traffic throttling (policing), where the downstream traffic
	flowing out of the ingress node is limited to reduce or eliminate		flowing out of the ingress node is limited to reduce or eliminate
	congestion.		congestion.
			INTERNET-DRAFT NSH ECN & Congestion Feedback
	(2) Upstream congestion feedback, where the ingress node sends		(2) Upstream congestion feedback, where the ingress node sends
	messages upstream to or towards the ultimate traffic source, a		messages upstream to or towards the ultimate traffic source, a
	function that can throttle traffic generation/transmission.		function that can throttle traffic generation/transmission.
	(3) Traffic re-direction, where the ingress node configures the NSH		(3) Traffic re-direction, where the ingress node configures the NSH
	of some future traffic so that it avoids congested paths. Great		of some future traffic so that it avoids congested paths. Great
	care must be taken with this option to avoid (a) significant re-		care must be taken with this option to avoid (a) significant re-
	INTERNET-DRAFT NSH ECN & Congestion Feedback
	ordering of traffic in flows that it is desirable to keep in		ordering of traffic in flows that it is desirable to keep in
	order and (b) oscillation/instability in traffic paths due to		order and (b) oscillation/instability in traffic paths due to
	alternate congestion of previously idle paths and the idling of		alternate congestion of previously idle paths and the idling of
	previously congested paths. For example, it is preferable to		previously congested paths. For example, it is preferable to
	classify traffic into flows of a sufficiently coarse granularity		classify traffic into flows of a sufficiently coarse granularity
	that the flows are long lived and then use a stable path per		that the flows are long lived and then use a stable path per
	flow, sending only newly appearing flows on apparently		flow, sending only newly appearing flows on apparently
	uncongested paths.		uncongested paths.
	Figure 2 shows an example path from an origin sender to a final		Figure 2 shows an example path from an original sender to a final
	receiver passing through an example chain of service functions		receiver passing through an example chain of service functions
	between the ingress and egress of an SFC domain. The path is also		between the ingress and egress of an SFC domain. The path is also
	likely to pass through other network nodes outside the SFC domain		likely to pass through other network nodes outside the SFC domain
	(not shown) before entering the SFC domain and after leaving the SFC		(not shown) before entering the SFC domain and after leaving the SFC
	domain.		domain.
	The figure shows typical congestion feedback that would be expected		The figure shows typical congestion feedback that would be expected
	from the final receiver to the origin sender, which controls the load		from the final receiver to the origin sender, which controls the load
	the origin sender applies to all elements on the path. The figure		the origin sender applies to all elements on the path. The figure
	also shows the congestion feedback from the egress to the ingress of		also shows the congestion feedback from the egress to the ingress of
	the SFC domain that is described in this document, to control or		the SFC domain that is described in this document, to control or
	balance load within the SFC domain.		balance load within the SFC domain.
			INTERNET-DRAFT NSH ECN & Congestion Feedback
	.:= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = :.		.:= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = :.
	_||_ End-to-End Congestion Feedback ||		_||_ End-to-End Congestion Feedback ||
	\ / ||		\ / ||
	\/ ||		\/ ||
	__ Inner Transport Header and Payload __		__ Inner Transport Header and Payload __
	| | ->- - - - - - - - - - - - - - ->- - - - - -- - - - - - ->- | |		| | ->- - - - - - - - - - - - - - ->- - - - - -- - - - - - ->- | |
	| | | |		| | | |
	| | .:= = = = = = = = = = = = = = = = = = = = = =:. | |		| | .:= = = = = = = = = = = = = = = = = = = = = =:. | |
	| | _||_ Tunnel Congestion Feedback || | |		| | _||_ Tunnel Congestion Feedback || | |
	| | \ / || | |		| | \ / || | |
	skipping to change at page 6, line 54 ¶		skipping to change at page 8, line 31 ¶
	| | | | OT1 | | OT4 | | . . . | | OTn | | | |		| | | | OT1 | | OT4 | | . . . | | OTn | | | |
	| | | |-->--|SFF|--->---|SFF| |SFF|-->--| | | |		| | | |-->--|SFF|--->---|SFF| |SFF|-->--| | | |
	|__| |__| |___| |___| |___| |__| |__|		|__| |__| |___| |___| |___| |__| |__|
	origin SFC | ^ | ^ SFC final		origin SFC | ^ | ^ SFC final
	sender domain OT2| |OT3 OT6| |OT7 domain rcvr		sender domain OT2| |OT3 OT6| |OT7 domain rcvr
	ingress v | v | egress		ingress v | v | egress
	+---+ +---+		+---+ +---+
	|SF | |SF |		|SF | |SF |
	+---+ +---+		+---+ +---+
	Figure 2: Congestion Feedback across an SFC Domain		Figure 2. Congestion Feedback across an SFC Domain
	INTERNET-DRAFT NSH ECN & Congestion Feedback
	SFC Domain congestion feedback in Figure 2 is shown within the		SFC Domain congestion feedback in Figure 2 is shown within the
	context of an end-to-end congestion feedback loop. Also shown is the		context of an end-to-end congestion feedback loop. Also shown is the
	encapsulated layering of NSH headers within a series of outer		encapsulated layering of NSH headers within a series of outer
	transport headers (OT1, OT2, ... OTn).		transport headers (OT1, OT2, ... OTn).
	1.4 Conventions Used in This Document		1.4 Conventions Used in This Document
	The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",		The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
	"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this		"SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
	document are to be interpreted as described in [RFC2119] [RFC8174]		"OPTIONAL" in this document are to be interpreted as described in BCP
	when, and only when, they appear in all capitals, as shown here.		14 [RFC2119] [RFC8174] when, and only when, they appear in all
			capitals, as shown here.
	Acronyms:		Acronyms:
	AQM - Active Queue Management [RFC7567]		AQM - Active Queue Management [RFC7567]
	CE - Congestion Experienced [RFC3168]		CE - Congestion Experienced [RFC3168]
	downstream - The direction from ingress to egress		downstream - The direction from ingress to egress
			INTERNET-DRAFT NSH ECN & Congestion Feedback
	ECN - Explicit Congestion Notification [RFC3168]		ECN - Explicit Congestion Notification [RFC3168]
	ECT - ECN Capable Transport [RFC3168]		ECT - ECN Capable Transport [RFC3168]
	IPFIX - IP Flow Information Export [RFC7011]		IPFIX - IP Flow Information Export [RFC7011]
	Not-ECT - Not ECN-Capable Transport [RFC3168]		Not-ECT - Not ECN-Capable Transport [RFC3168]
	NSH - Network Service Header [RFC8300]		NSH - Network Service Header [RFC8300]
	skipping to change at page 8, line 46 ¶		skipping to change at page 10, line 46 ¶
	+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+		+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
	|Ver|O|U| TTL | Length |U|U|U|U|MD Type| Next Protocol |		|Ver|O|U| TTL | Length |U|U|U|U|MD Type| Next Protocol |
	+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+		+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
	^ ^		^ ^
	| |		| |
	+-------+		+-------+
	|NSH ECN|		|NSH ECN|
	| field |		| field |
	+-------+		+-------+
	Figure 4: NSH Base Header		Figure 4. NSH Base Header
	Note to RFC Editor: The above figure should be adjusted based on the		Note to RFC Editor: The above figure should be adjusted based on the
	bits assigned by IANA (see Section 5) and this note deleted.		bits assigned by IANA (see Section 5) and this note deleted.
	Table 1 shows the meaning of the code points in the NSH ECN field.		Table 1 shows the meaning of the code points in the NSH ECN field.
	These have the same meaning as the ECN field code points in the IPv4		These have the same meaning as the ECN field code points in the IPv4
	or IPv6 header as defined in [RFC3168].		or IPv6 header as defined in [RFC3168].
	INTERNET-DRAFT NSH ECN & Congestion Feedback		INTERNET-DRAFT NSH ECN & Congestion Feedback
	skipping to change at page 13, line 18 ¶		skipping to change at page 15, line 18 ¶
	If the SF is NSH and ECN-aware, the processing is essentially the		If the SF is NSH and ECN-aware, the processing is essentially the
	same at the SF as at an SFF as discussed in Section 3.2.1.		same at the SF as at an SFF as discussed in Section 3.2.1.
	If the SF is NSH-aware but ECN-unaware, then the SFF transmitting the		If the SF is NSH-aware but ECN-unaware, then the SFF transmitting the
	packet to the SF will use Compatibility Mode. Congestion encountered		packet to the SF will use Compatibility Mode. Congestion encountered
	in the SFF to SF and SF to SFF paths will be unmanaged.		in the SFF to SF and SF to SFF paths will be unmanaged.
	If the SF is not NSH-aware, then an NSH proxy will be between the SFF		If the SF is not NSH-aware, then an NSH proxy will be between the SFF
	and the SF to avoid exposure of the NSH to the SF that does not		and the SF to avoid exposure of the NSH to the SF that does not
	understand NSHs. This is described in Section 4.6 of [RFC7665]. The		understand NSHs as shown in Figure 6. This is described in Section
	SF and proxy together look to the SFF like an NSH-aware SF. The		4.6 of [RFC7665]. The SF and proxy together look to the SFF like an
	behavior at the proxy and SF in this case is as below:		NSH-aware SF. The behavior at the proxy and SF in this case is as
			below:
	If such a proxy is not ECN-aware then congestion in the entire		If such a proxy is not ECN-aware then congestion in the entire
	path from SFF to proxy to SF back to proxy to SFF will be		path from SFF to proxy to SF back to proxy to SFF will be
	unmanaged.		unmanaged.
			|
			v
			+----------+ +---------+
			| | +-------+ | NSH |
			| SFF +---->| NSH +---->|un-aware |
			|(Service | | aware | | SF |
			| Function |<----+ proxy |<----+(Service |
			|Forwarder)| +-------+ |Function)|
			+----------+ +---------+
			|
			v
			Figure 6. Proxy for NSH Un-aware SFF
	If the proxy is ECN-aware, the proxy uses an AQM to indicate		If the proxy is ECN-aware, the proxy uses an AQM to indicate
	congestion within the proxy in the NSH that it returns to the SFF.		congestion within the proxy in the NSH that it returns to the SFF.
	The outer header used for the proxy to SF path uses Normal Mode.		The outer header used for the proxy-to-SF path uses Normal Mode.
	The outer head used for the proxy return to SFF path uses Normal		The outer head used for the proxy-to-SFF path uses Normal Mode
	Mode based copying of the NSH ECN field to the outer header. Thus		based copying of the NSH ECN field to the outer header. Thus
	congestion in the proxy will be managed. Congestion in the SF will		congestion in the proxy will be managed. Congestion in the SF will
	be managed only if the SF is ECN-aware implementing an AQM.		be managed only if the SF is ECN-aware and implements an AQM.
	3.2.3 At Other Forwarding Nodes		3.2.3 At Other Forwarding Nodes
	Other forwarding nodes, that is non-NSH forwarding nodes between NSH		Other forwarding nodes, that is non-NSH forwarding nodes between NSH
	forwarding nodes, such as IP or label switched routers, might also		forwarding nodes, such as IP or label switched routers, might also
	contain potential bottlenecks. If so, they SHOULD implement an AQM		contain potential bottlenecks. If so, they SHOULD implement an AQM
	algorithm to update the ECN marking in the outer transport header as		algorithm to update the ECN marking in the outer transport header as
			INTERNET-DRAFT NSH ECN & Congestion Feedback
	specified in [RFC3168].		specified in [RFC3168].
	3.3 At Exit/Egress		3.3 At Exit/Egress
	First, any actions are taken based on Congestion Experienced or other		At the SFC domain egress node, first any actions are taken based on
	values of ECN marking, such as accumulating statistics to forward		Congestion Experienced or other values of ECN marking, such as
	back to the ingress (see Section 4). If the packet being carried		accumulating statistics to send back to the ingress (see Section 4)
	inside the NSH is IP, when the NSH is removed the NSH ECN field MUST		or for other uses. If the packet being carried inside the NSH is IP,
	be combined with the IP ECN field as specified in Table 3 that was		when the NSH is removed the NSH ECN field MUST be combined with the
	extracted from [RFC6040]. This requirement applies to all egress		IP ECN field as specified in Table 3 that was extracted from
	nodes for the domain in which NSH is being used to route traffic.		[RFC6040]. This requirement applies to all egress nodes for the
			domain in which NSH is being used to route traffic.
	INTERNET-DRAFT NSH ECN & Congestion Feedback
	+---------+---------------------------------------------+		+---------+---------------------------------------------+
	|Arriving | Arriving Outer Header |		|Arriving | Arriving Outer Header |
	| Inner +---------+-----------+-----------+-----------+		| Inner +---------+-----------+-----------+-----------+
	| Header | Not-ECT | ECT(0) | ECT(1) | CE |		| Header | Not-ECT | ECT(0) | ECT(1) | CE |
	+---------+---------+-----------+-----------+-----------+		+---------+---------+-----------+-----------+-----------+
	| Not-ECT | Not-ECT |Not-ECT |Not-ECT | <drop> |		| Not-ECT | Not-ECT |Not-ECT |Not-ECT | <drop> |
	| ECT(0) | ECT(0) | ECT(0) | ECT(0) | CE |		| ECT(0) | ECT(0) | ECT(0) | ECT(0) | CE |
	| ECT(1) | ECT(1) | ECT(1) | ECT(1) | CE |		| ECT(1) | ECT(1) | ECT(1) | ECT(1) | CE |
	| CE | CE | CE | CE | CE |		| CE | CE | CE | CE | CE |
	skipping to change at page 14, line 33 ¶		skipping to change at page 16, line 46 ¶
	used.		used.
	3.4 Conservation of Packets		3.4 Conservation of Packets
	The SFC specification permits an SF to absorb packets and to generate		The SFC specification permits an SF to absorb packets and to generate
	new packets as well as simply processing and forwarding the packets		new packets as well as simply processing and forwarding the packets
	it receives. Such actions might appear to be packet loss due to		it receives. Such actions might appear to be packet loss due to
	congestion or might mask the loss of packets by generating additional		congestion or might mask the loss of packets by generating additional
	packets.		packets.
	The tunnel congestion feedback approach [TunnelCongFeedback] detects		The tunnel congestion feedback approach (Section 4) detects loss by
	loss by counting payload bytes in at the ingress and counting them		counting payload bytes in at the ingress and counting them out at the
	out at the egress. This does not work unless nodes conserve the		egress. This does not work unless nodes conserve the amount of
	amount of payload bytes. Therefore, it will not be possible to detect		payload bytes. Therefore, it will not be possible to detect loss
	loss using this technique if they are not conserved.		using this technique if they are not conserved.
			INTERNET-DRAFT NSH ECN & Congestion Feedback
	Nonetheless, if a bottleneck supports ECN marking, it will be		Nonetheless, if a bottleneck supports ECN marking, it will be
	possible to detect the very high level of CE markings that are		possible to detect the very high level of CE markings that are
	associated with congestion that is so excessive that it leads to		associated with congestion that is so excessive that it leads to
	loss. However, it will not be possible for the tunnel congestion		loss. However, it will not be possible for the tunnel congestion
	feedback approach to detect any congestion, whether slight or severe,		feedback approach to detect any congestion, whether slight or severe,
	if it occurs at a bottleneck that does not support ECN marking.		if it occurs at a bottleneck that does not support ECN marking.
	INTERNET-DRAFT NSH ECN & Congestion Feedback		INTERNET-DRAFT NSH ECN & Congestion Feedback
	4. Tunnel Congestion Feedback Support		4. Tunnel Congestion Feedback Support
	The collection and storage of congestion information at the egress		The collection and storage of congestion information at the egress
	may be useful for later analysis but, unless it can be fed back to a		may be useful for later analysis but, unless it can be fed back to a
	point which can take action to reduce congestion, it will not be		point which can take action to reduce congestion, it will not be
	useful in real time. Such congestion feedback to the ingress enables		useful in real time. Such congestion feedback to the ingress enables
	it to take actions such as those listed in Section 1.3.		it to take actions such as those listed in Section 1.3.
	IP Flow Information Export (IPFIX [RFC7011]) provides a standard for		IP Flow Information Export (IPFIX [RFC7011]) provides a standard for
	communicating traffic flow statistics. As extended by		communicating traffic flow statistics. As extended by this document,
	[TunnelCongFeedback] and herein, IPFIX messages from the egress to		IPFIX messages from the egress to the ingress are used to communicate
	the ingress are used to communicate the extent of congestion between		the extent of congestion between an ingress and egress based on ECN
	an ingress and egress based on ECN marking in the NSH. For example,		marking in the NSH.
	the tunnelEcnCEMarkedRatio field, specified in [TunnelCongFeedback],
	indicates tha fraction of traffic that has been marked in the ECN		4.1 Congestion Level Measurement
			The congestion level measurement is based on ECN marking in the NSH
			and packet drop, particularly the fraction of packets that are CE-
			marked packet and fraction that are dropped. If the congestion level
			is not high enough, the packets are marked as CE instead of being
			dropped, and then it is easy to calculate congestion level according
			to the ratio of CE-marked packets. If the congestion level is so high
			that ECT packets will be dropped, then the packet loss ratio could be
			calculated by comparing total packets entering ingress and total
			packets arriving at egress over the same span of packets. If packet
			loss is detected, it could be assumed that severe congestion has
			occurred in the tunnel.
			The egress calculates the CE-marked packet ratio by counting packets
			with different ECN markings. The CE-marked packet ratio will be used
			as an indication of tunnel load level. It is assumed that nodes
			between the ingress and egress will not drop packets biased towards
			certain ECN codepoints, so calculating of CE-marked packet ratio is
			not affect by packet drop.
			The calculation of volumes of packet drop is by comparing the traffic
			volumes between ingress and egress.
			Faked ECN-Capable Transport (ECT) is used at the ingress to defer
			packet loss to the egress. The basic idea of faked ECT is that, when
			encapsulating packets, the ingress first marks the tunnel outer
			header (NSH for an SFC domain) according to [RFC6040], and then
			remarks the outer header of Not-ECT packets as ECT. (ECT(0) and
			ECT(1) are treated as the same.) Thus, as transmitted by the ingress
			node, there will be one of three combinations of outer header ECN
			field and inner header ECN field: CE|CE, ECT|N-ECT, and ECT|ECT (in
			the format of outer-ECN|inner-ECN); when decapsulating packets at the
			egress, [RFC6040] defined decapsulation behavior is used, and
			INTERNET-DRAFT NSH ECN & Congestion Feedback
			according to [RFC6040], the packets marked as CE|N-ECT will be
			dropped. Faked-ECT is used to shift some drops to the egress in order
			to allow the egress to calculate the CE-marked packet ratio more
			precisely.
			The ingress encapsulates packets and marks their outer header
			according to faked ECT as described above. The ingress cumulatively
			counts packet bytes for three types of ECN combination (CE|CE, ECT|N-
			ECT, and ECT|ECT) and then the ingress regularly sends cumulative
			bytes counts message of each type of ECN combination to the egress.
			When each message arrives at the egress, (1) egress calculates the
			ratio of CE-marked packet; (2) the egress cumulatively counts packet
			bytes coming from the ingress and adds its own bytes counts of each
			type of ECN combination (CE|CE, ECT|N-ECT, CE|N-ECT, CE|ECT, and
			ECT|ECT) to the message for ingress to calculate packet loss. The
			egress feeds back CE-marked packet ratio and bytes counts information
			to the ingress for evaluating congestion level in the tunnel.
			The counting of bytes can be at the granularity of all traffic from
			the ingress to the egress to learn about the overall congestion
			status of the path between the ingress and the egress. The counting
			can also be at the granularity of individual customer's traffic or a
			specific set of flows to learn about their congestion contribution.
			For example, the tunnelEcnCEMarkedRatio field (specified below)
			indicates the fraction of traffic that has been marked in the ECN
	field of the NSH as Congestion Experienced (CE).		field of the NSH as Congestion Experienced (CE).
			4.3 Congestion Information Delivery
			As described above, the tunnel ingress needs to send a messages
			containing cumulative bytes counts of packets of each type of ECN
			combination to the tunnel egress, and the tunnel egress also needs to
			feed back messages with cumulative bytes counts of packets of each
			type of ECN combination and CE-marked packet ratio to the ingress.
			This section specifies how the messages should be conveyed.
			IPFIX recommends, but does not require, use of SCTP [RFC4960] in
			partial reliability mode [RFC3758] for the transport of its messages.
			This mode allows loss of some packets, which is tolerable because
			IPFIX communicates cumulative statistics. IPFIX over SCTP over IP
			SHOULD be used directly where there is IP connectivity between the
			ingress and egress; however, there might be different transport
			protocols or address spaces used in different regions of an SFC
			domain that make such direct IP connectivity problematic. The NSH
			provides the general method of routing traffic within an SFC domain
			so the encapsulation of the required IPFIX traffic in NSH MUST be
			INTERNET-DRAFT NSH ECN & Congestion Feedback
			implemented and, when IP connectivity is not available, IPFIX over
			NSH SHOULD be used along with configuration of appropriate SFC paths
			for the IPFIX over NSH traffic.
			Typically IPFIX messages could travel along the same path as network
			data traffic. In any case, an IPFIX message packet may get lost in
			case of network congestion. Even though the missing information could
			be recovered because of the use of cumulative counts, the message
			SHOULD be transmitted at a higher priority than users' traffic flows.
			The ingress node can do congestion management at different
			granularity which means both the overall aggregated inner tunnel
			congestion level and congestion level contributed by certain traffic
			flows could be measured for different congestion management purpose.
			For example, if the ingress only wants to limit congestion volume
			caused by certain traffic flows, such as UDP-based traffic, then
			congestion volume for that traffic will be fed back; or if the
			ingress is doing overall congestion management, the aggregated
			congestion volume will be fed back.
			When sending IPFIX messages from ingress to egress, the ingress acts
			as IPFIX exporter and egress acts as IPFIX collector; When feedback
			congestion level information from egress to ingress, then the egress
			acts as IPFIX exporter and ingress acts as IPFIX collector.
			The combination of congestion level measurement and congestion
			information delivery procedures should be as following:
			o The ingress node determines the IPFIX template record to be used.
			The template record can be pre-configured or determined at
			runtime, the content of template record will be determined
			according to the granularity of congestion management; if the
			ingress wants to limit congestion volume contributed by specific
			traffic flow then the elements such as source IP address,
			destination IP address, flow id and CE-marked packet volume of the
			flow, etc., will be included in the template record.
			o Metering on the ingress measures traffic volume according to
			template record chosen and then the measurement records are sent
			to the egress.
			o Metering on the egress measures congestion level information
			according to template record which should be the same as the
			template record sent by the ingress.
			o The egress sends measurement records together with the measurement
			records of ingress back to the ingress.
			INTERNET-DRAFT NSH ECN & Congestion Feedback
			4.3 IPFIX Extensions
			This section specifies new IPFIX Information Elements according to
			[RFC7013].
			4.3.1 nshServicePathID
	In order to identify SFC flows, so that congestion can be measured		In order to identify SFC flows, so that congestion can be measured
	and reported at that granularity, it is necessary for IPFIX to be		and reported at that granularity, it is necessary for IPFIX to be
	able to classify traffic based on the Service Path Identifier field		able to classify traffic based on the Service Path Identifier field
	of the NSH [RFC8300]. Thus an NSH Service Path Identifier		of the NSH [RFC8300]. Thus an NSH Service Path Identifier
	(nshServicePathID) IPFIX Information Element [RFC7012] is specified		(nshServicePathID) IPFIX Information Element [RFC7012] is specified.
	as follows for use in this application of IPFIX:
	Name: nshServicePathID		Name: nshServicePathID
	Description: Network Service Header [RFC8300] Service Path		Description: Network Service Header [RFC8300] Service Path
	Identifier. This is a 24-bit value which is left justified in		Identifier. This is a 24-bit value which is left justified in
	the Information Element. The low order byte MUST be sent as		the Information Element. The low order byte MUST be sent as
	zero and ignored on receipt.		zero and ignored on receipt.
	Abstract Data Type: unsigned32		Abstract Data Type: unsigned32
	Data Type Semantics: identifier		Data Type Semantics: identifier
	ElementID: tbd		ElementId: tbd0
	Status: current		Status: current
	IPFIX recommends, but does not require, use of SCTP [RFC4960] in		4.3.2 tunnelEcnCeCeByteTotalCount
	partial reliability mode for the transport of its messages. This mode
	allows loss of some packets, which is tolerable because IPFIX		Description: The total number of bytes of incoming packets with
	communicates cumulative statistics. IPFIX over SCTP SHOULD be used		CE|CE ECN marking combination at the Observation Point since
	directly where there is IP connectivity between the ingress and		the Metering Process (re-)initialization for this Observation
	egress; however, there might be different transport protocols or		Point.
	address spaces used in different regions of an SFC domain that make
	such direct IP connectivity problematic. The NSH provides the general		Abstract Data Type: unsigned64
	method of routing traffic within such an SFC domain so the IPFIX
	traffic MUST be encapsulated in NSH when IP connectivity is not		Data Type Semantics: totalCounter
	available.
			ElementId: TBD1
			Statues: current
			Units: bytes
	INTERNET-DRAFT NSH ECN & Congestion Feedback		INTERNET-DRAFT NSH ECN & Congestion Feedback
	5. IANA Considerations		4.3.3 tunnelEcnEctNectBytetTotalCount
	5.1 SFC NSH Header ECN Bits		Description: The total number of bytes of incoming packets with
			ECT|N-ECT ECN marking combination (ECT(0) and ECT(1) are
			treated as the same) at the Observation Point since the
			Metering Process (re-)initialization for this Observation
			Point.
			Abstract Data Type: unsigned64
			Data Type Semantics: totalCounter
			ElementId: TBD2
			Statues: current
			Units: bytes
			4.3.4 tunnelEcnCeNectByteTotalCount
			Description: The total number of bytes of incoming packets with
			CE|N-ECT ECN marking combination at the Observation Point since
			the Metering Process (re-)initialization for this Observation
			Point.
			Abstract Data Type: unsigned64
			Data Type Semantics: totalCounter
			ElementId: TBD3
			Statues: current
			Units: bytes
			4.3.5 tunnelEcnCeEctByteTotalCount
			Description: The total number of bytes of incoming packets with
			CE|ECT ECN marking combination (ECT(0) and ECT(1) are treated
			as the same) at the Observation Point since the Metering
			Process (re-)initialization for this Observation Point.
			Abstract Data Type: unsigned64
			Data Type Semantics: totalCounter
			INTERNET-DRAFT NSH ECN & Congestion Feedback
			ElementId: TBD4
			Statues: current
			Units: bytes
			4.3.6 tunnelEcnEctEctByteTotalCount
			Description: The total number of bytes of incoming packets with
			ECT|ECT ECN marking combination (ECT(0) and ECT(1) are treated
			as the same) at the Observation Point since the Metering
			Process (re-)initialization for this Observation Point.
			Abstract Data Type: unsigned64
			Data Type Semantics: totalCounter
			ElementId: TBD5
			Statues: current
			Units: bytes
			4.3.7 tunnelEcnCEMarkedRatio
			Description: The ratio of CE-marked Packet at the Observation
			Point.
			Abstract Data Type: float32
			ElementId: TBD6
			Statues: current
			INTERNET-DRAFT NSH ECN & Congestion Feedback
			5. Example of Use
			This subsection provides an example of how the solution described in
			this document could work.
			First of all, IPFIX template records are exchanged between ingress
			and egress to negotiate the format of data records. The example here
			is to measure the congestion level for the overall tunnel caused by
			all the traffic. After the negotiation is finished, the ingress sends
			in-band messages to egress containing the number of each kind of ECN-
			marked packets (i.e.. CE|CE, ECT|N-ECT and ECT|ECT) received until it
			sent the message.
			After the egress receives the message, the egress calculates the CE-
			marked packet ratio and counts the number of different kinds of ECN-
			marking packets received until it received the message. Then the
			egress sends a feedback message containing the counts together with
			the information in the ingress's message back to the ingress.
			Figures 7 to 10 below show the example procedure between ingress and
			egress.
			+---------------------------------+----------------------+
			|Set ID=2 Length=40 |
			|---------------------------------|----------------------|
			|Template ID=256 Field Count=8 |
			|---------------------------------|----------------------|
			|tunnelEcnCeCeByteTotalCount Field Length=8 |
			|---------------------------------|----------------------|
			|tunnelEcnEctNectByteTotalCount Field Length=8 |
			|---------------------------------|----------------------|
			|tunnelEcnEctEctByteTotalCount Field Length=8 |
			|---------------------------------|----------------------|
			|tunnelEcnCeNectByteTotalCount Field Length=8 |
			|---------------------------------|----------------------|
			|tunnelEcnCeEctByteTotalCount Field Length=8 |
			+---------------------------------|----------------------+
			|tunnelEcnCEMarkedRatio Field Length=4 |
			+---------------------------------+----------------------+
			Figure 7. Template Record Sent From Egress to Ingress
			INTERNET-DRAFT NSH ECN & Congestion Feedback
			+---------------------------------+----------------------+
			|Set ID=2 Length=28 |
			|---------------------------------|----------------------|
			|Template ID=257 Field Count=3 |
			|---------------------------------|----------------------|
			|tunnelEcnCeCeByteTotalCount Field Length=8 |
			|---------------------------------|----------------------|
			|tunnelEcnEctNectByteTotalCount Field Length=8 |
			|---------------------------------|----------------------|
			|tunnelEcnEctEctByteTotalCount Field Length=8 |
			|---------------------------------+----------------------|
			Figure 8. Template Record Sent From Ingress to Egress
			+-------+ +-+ +-+ +-+ +-+ +-+ +-+ +-+ +-------+
			| | |M| |P| |P| |P| |M| |P| |P| | |
			| | +-+ +-+ +-+ +-+ +-+ +-+ +-+ | |
			| |<---------------------------------------| |
			| | | |
			| | | |
			|egress | +-+ +-+ |ingress|
			| | |M| |M| | |
			| | +-+ +-+ | |
			| |--------------------------------------->| |
			| | | |
			| | | |
			+-------+ +-------+
			+-+
			|M| : Message Packet
			+-+
			+-+
			|P| : User Packet
			+-+
			Figure 9. Traffic flow Between Ingress and Egress
			INTERNET-DRAFT NSH ECN & Congestion Feedback
			Set ID=257, Length=28
			+------+ A1 +------+
			| | B1 | |
			| | C1 | |
			| | <----------------------------- | |
			| | | |
			| | | |
			| | SetID=256, Length=72 | |
			| | A1 | |
			| | B1 | |
			|egress| C1 ingress|
			| | A2 | |
			| | B2 | |
			| | C2 | |
			| | D | |
			| | E | |
			| | R | |
			| | ----------------------------> | |
			| | | |
			+------+ +------+
			Figure 10. Messages Between Ingress and Egress
			The following provides an example of how the tunnel congestion level
			could be calculated:
			The congestion Level could be divided into two categories: (1)
			slight congestion (no packets dropped); (2) serious congestion
			(packets are being dropped).
			For slight congestion, the congestion level is indicated as the
			ratio of CE-marked packet:
			ce_marked = R;
			For serious congestion, the congestion level is indicated as the
			number of volume loss:
			total_ingress = (A1 + B1 + C1)
			total_egress = (A2 + B2 + C2 + D + E)
			volume_loss = (total_ingress - total_egress)
			INTERNET-DRAFT NSH ECN & Congestion Feedback
			6. IANA Considerations
			The following subsections provide IANA assignment considerations.
			6.1 SFC NSH Header ECN Bits
	IANA is requested to assign two contiguous bits in the NSH Base		IANA is requested to assign two contiguous bits in the NSH Base
	Header Bits registry for ECN (bits 16 and 17 suggested) and note this		Header Bits registry for ECN (bits 16 and 17 suggested) and note this
	assignment as follows:		assignment as follows:
	Bit Description Reference		Bit Description Reference
	---------- ----------- ---------------		---------- ----------- -----------------
	tbd(16-17) NSH ECN [this document]		tbd(16-17) NSH ECN [this document]
	5.2 IPFIX Information Element ID		6.2 IPFIX Information Element IDs
	IANA is request to assign an IPFIX Information Element ID as follows:		IANA is requested to assign IPFIX Information Element IDs as follows:
	ElementID: tbd		ElementID: tbd0
	Name: nshServicePathID		Name: nshServicePathID
	Data Type: unsigned32		Data Type: unsigned32
	Data Type Semantics: identifier		Data Type Semantics: identifier
	Status: current		Status: current
	Description: The Network Service Header [RFC8300] Service Path		Description: The Network Service Header [RFC8300] Service Path
	Identifier.		Identifier.
			ElementID: TBD1
			Name: tunnelEcnCeCePacketTotalCount
			Data Type: unsigned64
			Data Type Semantics: totalCounter
			Status: current
			Description: The total number of bytes of incoming packets with
			CE|CE ECN marking combination at the Observation Point since
			the Metering Process (re-)initialization for this Observation
			Point.
			Units: octets
			ElementID: TBD2
			Name: tunnelEcnEctNectPacketTotalCount
			Data Type: unsigned64
			Data Type Semantics: totalCounter
			Status: current
			Description: The total number of bytes of incoming packets with
			ECT|N-ECT ECN marking combination at the Observation Point
			since the Metering Process (re-)initialization for this
			Observation Point.
	INTERNET-DRAFT NSH ECN & Congestion Feedback		INTERNET-DRAFT NSH ECN & Congestion Feedback
	6. Security Considerations		Units: octets
			ElementID: TBD3
			Name: tunnelEcnCeNectPacketTotalCount
			Data Type: unsigned64
			Data Type Semantics: totalCounter
			Status: current
			Description: The total number of bytes of incoming packets with
			CE|N-ECT ECN marking combination at the Observation Point since
			the Metering Process (re-)initialization for this Observation
			Point.
			Units: octets
			ElementID: TBD4
			Name: tunnelEcnCeEctPacketTotalCount
			Data Type: unsigned64
			Data Type Semantics: totalCounter
			Status: current
			Description: The total number of bytes of incoming packets with
			CE|ECT ECN marking combination at the Observation Point since
			the Metering Process (re-)initialization for this Observation
			Point.
			Units: octets
			ElementID: TBD5
			Name: tunnelEcnEctEctPacketTotalCount
			Data Type: unsigned64
			Data Type Semantics: totalCounter
			Status: current
			Description: The total number of bytes of incoming packets with
			CE|ECT(0) ECN marking combination at the Observation Point
			since the Metering Process (re-)initialization for this
			Observation Point.
			Units: octets
			ElementID: TBD6
			Name: tunnelEcnCEMarkedRatio
			Data Type: float32
			Status: current
			Description: The ratio of CE-marked Packet at the Observation
			Point.
			INTERNET-DRAFT NSH ECN & Congestion Feedback
			7. Security Considerations
	For general NSH security considerations, see [RFC8300].		For general NSH security considerations, see [RFC8300].
	For security considerations concerning tampering with ECN signaling,		For security considerations concerning tampering with ECN signaling,
	see [RFC3168]. For security considerations concerning ECN and		see [RFC3168]. For security considerations concerning ECN and
	encapsulation, see [RFC6040].		encapsulation, see [RFC6040].
	For general IPFIX security considerations, see [RFC7011]. If deployed		For general IPFIX security considerations, see [RFC7011]. If deployed
	in an untrusted environment, the signaling traffic between ingress		in an untrusted environment, the signaling traffic between ingress
	and egress can be protected utilizing the security mechanisms		and egress can be protected utilizing the security mechanisms
	provided by IPFIX (see Section 11 in [RFC7011]).		provided by IPFIX (see Section 11 in [RFC7011]).
			The tunnel endpoints (the ingress and egress for an SFC domain) are
			assumed to be in the same administrative domain, so they will trust
			each other.
	The solution in this document does not introduce any greater		The solution in this document does not introduce any greater
	potential to invade privacy than would have been available without		potential to invade privacy than would have been available without
	the solution.		the solution.
	7. Acknowledgements		8. Acknowledgements
	The authors wish to thank the following for their comments and		Most of the material on Tunnel Congestion Feedback was originally in
	suggestion:		draft-ietf-twvwg-tunnel-congestion-feedback. After discussion with
			the authors of that draft, the authors of this draft, and the Chairs
			of the TSVWG and SFC Working Groups, the Tunnel Congestion Feedback
			draft was merged into this draft.
	Joel Halpern, Tal Mizrahi, Xinpeng Wei		The authors wish to thank the following for their comments,
			suggestions, and reviews:
			David Black, Sami Boutros, Anthony Chan, Lingli Deng, Liang Geng,
			Joel Halpern, Jake Holland, John Kaippallimalil, Tal Mizrahi,
			Vincent Roca, Lei Zhu
	INTERNET-DRAFT NSH ECN & Congestion Feedback		INTERNET-DRAFT NSH ECN & Congestion Feedback
	Normative References		Normative References
	[RFC2119] - Bradner, S., "Key words for use in RFCs to Indicate		[RFC2119] - Bradner, S., "Key words for use in RFCs to Indicate
	Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119,		Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119,
	March 1997, <http://www.rfc-editor.org/info/rfc2119>.		March 1997, <http://www.rfc-editor.org/info/rfc2119>.
	[RFC3168] - Ramakrishnan, K., Floyd, S., and D. Black, "The Addition		[RFC3168] - Ramakrishnan, K., Floyd, S., and D. Black, "The Addition
	of Explicit Congestion Notification (ECN) to IP", RFC 3168, DOI		of Explicit Congestion Notification (ECN) to IP", RFC 3168, DOI
	10.17487/RFC3168, September 2001, <http://www.rfc-		10.17487/RFC3168, September 2001, <http://www.rfc-
	editor.org/info/rfc3168>.		editor.org/info/rfc3168>.
			[RFC3758] - Stewart, R., Ramalho, M., Xie, Q., Tuexen, M., and P.
			Conrad, "Stream Control Transmission Protocol (SCTP) Partial
			Reliability Extension", RFC 3758, DOI 10.17487/RFC3758, May
			2004, <https://www.rfc-editor.org/info/rfc3758>.
	[RFC5129] - Davie, B., Briscoe, B., and J. Tay, "Explicit Congestion		[RFC5129] - Davie, B., Briscoe, B., and J. Tay, "Explicit Congestion
	Marking in MPLS", RFC 5129, DOI 10.17487/RFC5129, January 2008,		Marking in MPLS", RFC 5129, DOI 10.17487/RFC5129, January 2008,
	<https://www.rfc-editor.org/info/rfc5129>.		<https://www.rfc-editor.org/info/rfc5129>.
	[RFC6040] - Briscoe, B., "Tunnelling of Explicit Congestion		[RFC6040] - Briscoe, B., "Tunnelling of Explicit Congestion
	Notification", RFC 6040, DOI 10.17487/RFC6040, November 2010,		Notification", RFC 6040, DOI 10.17487/RFC6040, November 2010,
	<http://www.rfc-editor.org/info/rfc6040>.		<http://www.rfc-editor.org/info/rfc6040>.
	[RFC7011] - Claise, B., Ed., Trammell, B., Ed., and P. Aitken,		[RFC7011] - Claise, B., Ed., Trammell, B., Ed., and P. Aitken,
	"Specification of the IP Flow Information Export (IPFIX)		"Specification of the IP Flow Information Export (IPFIX)
	Protocol for the Exchange of Flow Information", STD 77, RFC		Protocol for the Exchange of Flow Information", STD 77, RFC
	7011, DOI 10.17487/RFC7011, September 2013, <https://www.rfc-		7011, DOI 10.17487/RFC7011, September 2013, <https://www.rfc-
	editor.org/info/rfc7011>.		editor.org/info/rfc7011>.
			[RFC7013] - Trammell, B. and B. Claise, "Guidelines for Authors and
			Reviewers of IP Flow Information Export (IPFIX) Information
			Elements", BCP 184, RFC 7013, DOI 10.17487/RFC7013, September
			2013, <https://www.rfc-editor.org/info/rfc7013>.
	[RFC7567] - Baker, F., Ed., and G. Fairhurst, Ed., "IETF		[RFC7567] - Baker, F., Ed., and G. Fairhurst, Ed., "IETF
	Recommendations Regarding Active Queue Management", BCP 197,		Recommendations Regarding Active Queue Management", BCP 197,
	RFC 7567, DOI 10.17487/RFC7567, July 2015, <http://www.rfc-		RFC 7567, DOI 10.17487/RFC7567, July 2015, <http://www.rfc-
	editor.org/info/rfc7567>.		editor.org/info/rfc7567>.
	[RFC8174] - Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC		[RFC8174] - Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
	2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May		2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May
	2017, <http://www.rfc-editor.org/info/rfc8174>		2017, <http://www.rfc-editor.org/info/rfc8174>
	[RFC8300] - Quinn, P., Ed., Elzur, U., Ed., and C. Pignataro, Ed.,		[RFC8300] - Quinn, P., Ed., Elzur, U., Ed., and C. Pignataro, Ed.,
	"Network Service Header (NSH)", RFC 8300, DOI 10.17487/RFC8300,		"Network Service Header (NSH)", RFC 8300, DOI 10.17487/RFC8300,
	January 2018, <https://www.rfc-editor.org/info/rfc8300>.		January 2018, <https://www.rfc-editor.org/info/rfc8300>.
	[TunnelCongFeedback] - Wei, X., Li, Y., Boutros, S., and L. Deng,		INTERNET-DRAFT NSH ECN & Congestion Feedback
	"Tunnel Congestion Feedback", draft-ietf-tsvwg-tunnel-
	congestion-feedback, work in progress.
	Informative References		Informative References
	[RFC4301] - Kent, S. and K. Seo, "Security Architecture for the		[RFC4301] - Kent, S. and K. Seo, "Security Architecture for the
	Internet Protocol", RFC 4301, DOI 10.17487/RFC4301, December		Internet Protocol", RFC 4301, DOI 10.17487/RFC4301, December
	2005, <https://www.rfc-editor.org/info/rfc4301>.		2005, <https://www.rfc-editor.org/info/rfc4301>.
	INTERNET-DRAFT NSH ECN & Congestion Feedback
	[RFC4960] - Stewart, R., Ed., "Stream Control Transmission Protocol",		[RFC4960] - Stewart, R., Ed., "Stream Control Transmission Protocol",
	RFC 4960, DOI 10.17487/RFC4960, September 2007,		RFC 4960, DOI 10.17487/RFC4960, September 2007,
	<https://www.rfc-editor.org/info/rfc4960>.		<https://www.rfc-editor.org/info/rfc4960>.
	[RFC7665] - Halpern, J., Ed., and C. Pignataro, Ed., "Service
	Function Chaining (SFC) Architecture", RFC 7665, DOI
	10.17487/RFC7665, October 2015, <https://www.rfc-
	editor.org/info/rfc7665>.
	[RFC7012] - Claise, B., Ed., and B. Trammell, Ed., "Information Model		[RFC7012] - Claise, B., Ed., and B. Trammell, Ed., "Information Model
	for IP Flow Information Export (IPFIX)", RFC 7012, DOI		for IP Flow Information Export (IPFIX)", RFC 7012, DOI
	10.17487/RFC7012, September 2013, <https://www.rfc-		10.17487/RFC7012, September 2013, <https://www.rfc-
	editor.org/info/rfc7012>.		editor.org/info/rfc7012>.
			[RFC7665] - Halpern, J., Ed., and C. Pignataro, Ed., "Service
			Function Chaining (SFC) Architecture", RFC 7665, DOI
			10.17487/RFC7665, October 2015, <https://www.rfc-
			editor.org/info/rfc7665>.
	[RFC8311] - Black, D., "Relaxing Restrictions on Explicit Congestion		[RFC8311] - Black, D., "Relaxing Restrictions on Explicit Congestion
	Notification (ECN) Experimentation", RFC 8311, DOI		Notification (ECN) Experimentation", RFC 8311, DOI
	10.17487/RFC8311, January 2018, <https://www.rfc-		10.17487/RFC8311, January 2018, <https://www.rfc-
	editor.org/info/rfc8311>.		editor.org/info/rfc8311>.
	[ecnL4S] - De Schepper, K., and B. Briscoe, "Identifying Modified		[ecnL4S] - De Schepper, K., and B. Briscoe, "Identifying Modified
	Explicit Congestion Notification (ECN) Semantics for Ultra-Low		Explicit Congestion Notification (ECN) Semantics for Ultra-Low
	Queuing Delay (L4S)", draft-ietf-tsvwg-ecn-l4s-id, work in		Queuing Delay (L4S)", draft-ietf-tsvwg-ecn-l4s-id, work in
	progress.		progress.
	skipping to change at page 20, line 24 ¶		skipping to change at page 32, line 24 ¶
	Tel: +1-508-333-2270		Tel: +1-508-333-2270
	Email: d3e3e3@gmail.com		Email: d3e3e3@gmail.com
	Bob Briscoe		Bob Briscoe
	Independent		Independent
	UK		UK
	Email: ietf@bobbriscoe.net		Email: ietf@bobbriscoe.net
	URI: http://bobbriscoe.net/		URI: http://bobbriscoe.net/
			Yizhou Li
			Huawei Technologies
			101 Software Avenue,
			Nanjing 210012, P. R China
			Phone: +86-25-56624584
			EMail: liyizhou@huawei.com
	Andrew G. Malis		Andrew G. Malis
	Independent		Malis Consulting
	Email: agmalis@gmail.com		Email: agmalis@gmail.com
			Xinpeng Wei
			Huawei Technologies
			Beiqing Rd. Z-park No.156, Haidian District,
			Beijing, 100095, P. R. China
			EMail: weixinpeng@huawei.com
	INTERNET-DRAFT NSH ECN & Congestion Feedback		INTERNET-DRAFT NSH ECN & Congestion Feedback
	Copyright and IPR Provisions		Copyright and IPR Provisions
	Copyright (c) 2020 IETF Trust and the persons identified as the		Copyright (c) 2021 IETF Trust and the persons identified as the
	document authors. All rights reserved.		document authors. All rights reserved.
	This document is subject to BCP 78 and the IETF Trust's Legal		This document is subject to BCP 78 and the IETF Trust's Legal
	Provisions Relating to IETF Documents		Provisions Relating to IETF Documents
	(http://trustee.ietf.org/license-info) in effect on the date of		(http://trustee.ietf.org/license-info) in effect on the date of
	publication of this document. Please review these documents		publication of this document. Please review these documents
	carefully, as they describe your rights and restrictions with respect		carefully, as they describe your rights and restrictions with respect
	to this document. Code Components extracted from this document must		to this document. Code Components extracted from this document must
	include Simplified BSD License text as described in Section 4.e of		include Simplified BSD License text as described in Section 4.e of
	the Trust Legal Provisions and are provided without warranty as		the Trust Legal Provisions and are provided without warranty as
End of changes. 61 change blocks.
	119 lines changed or deleted		623 lines changed or added
This html diff was produced by rfcdiff 1.48. The latest version is available from http://tools.ietf.org/tools/rfcdiff/