< draft-dunbar-e2e-latency-arch-view-and-gaps-00.txt		draft-dunbar-e2e-latency-arch-view-and-gaps-01.txt >
	Network working group L. Dunbar		Network working group L. Dunbar
	Internet Draft		Internet Draft
	Category: Standards Track Huawei		Category: Informational Huawei
	Expires: November 2017		Expires: November 2017
	March 13, 2017		March 27, 2017
	Architectural View of E2E Latency and Gaps		Architectural View of E2E Latency and Gaps
	draft-dunbar-e2e-latency-arch-view-and-gaps-00.txt		draft-dunbar-e2e-latency-arch-view-and-gaps-01.txt
	Abstract		Abstract
	Ultra-Low Latency is a highly desired property for many types of		Ultra-Low Latency is a highly desired property for many types of
	services, such as 5G MTC (Machine Type Communication) requiring		services, such as 5G MTC (Machine Type Communication) requiring
	E2E connection for V2V to be less than 2ms, AR/VR requiring delay		E2E connection for V2V to be less than 2ms, AR/VR requiring delay
	less than 5ms, V2X less than 20ms, etc.		less than 5ms, V2X less than 20ms, etc.
	This draft examines the E2E latency from architectural		This draft examines the E2E latency from architectural
	perspective, from studying how different OSI layers contribute to		perspective, from studying how different OSI layers contribute to
	skipping to change at page 2, line 32 ¶		skipping to change at page 2, line 32 ¶
	documents at any time. It is inappropriate to use Internet-		documents at any time. It is inappropriate to use Internet-
	Drafts as reference material or to cite them other than as "work		Drafts as reference material or to cite them other than as "work
	in progress."		in progress."
	The list of current Internet-Drafts can be accessed at		The list of current Internet-Drafts can be accessed at
	http://www.ietf.org/ietf/1id-abstracts.txt		http://www.ietf.org/ietf/1id-abstracts.txt
	The list of Internet-Draft Shadow Directories can be accessed at		The list of Internet-Draft Shadow Directories can be accessed at
	http://www.ietf.org/shadow.html		http://www.ietf.org/shadow.html
	This Internet-Draft will expire on September 13, 2017.		This Internet-Draft will expire on September 27, 2017.
	Copyright Notice		Copyright Notice
	Copyright (c) 2017 IETF Trust and the persons identified as the		Copyright (c) 2017 IETF Trust and the persons identified as the
	document authors. All rights reserved.		document authors. All rights reserved.
	This document is subject to BCP 78 and the IETF Trust's Legal		This document is subject to BCP 78 and the IETF Trust's Legal
	Provisions Relating to IETF Documents		Provisions Relating to IETF Documents
	(http://trustee.ietf.org/license-info) in effect on the date of		(http://trustee.ietf.org/license-info) in effect on the date of
	publication of this document. Please review these documents		publication of this document. Please review these documents
	skipping to change at page 3, line 14 ¶		skipping to change at page 3, line 14 ¶
	Internet-Draft E2E Over Internet Latency Taxonomy		Internet-Draft E2E Over Internet Latency Taxonomy
	Section 4.e of the Trust Legal Provisions and are provided		Section 4.e of the Trust Legal Provisions and are provided
	without warranty as described in the Simplified BSD License.		without warranty as described in the Simplified BSD License.
	Table of Contents		Table of Contents
	1. Introduction................................................. 4		1. Introduction................................................. 4
	2. Terminology.................................................. 5		2. Terminology.................................................. 5
	3. Contributing Factors to E2E Latency.......................... 5		3. AR/VR Use Case............................................... 5
	4. Application Layer Initiative in reducing E2E latency......... 6		4. Contributing Factors to E2E Latency.......................... 6
	4.1. Content Placement mechanisms need visibility to Network. 6		5. Application Layer Initiative in reducing E2E latency......... 6
	5. Transport Layer Initiatives in reducing Latency and gaps..... 6		5.1. Content Placement mechanisms need visibility to Network. 7
	5.1. TCP Layer Latency Improvement Alone is not enough....... 7		6. Transport Layer Initiatives in reducing Latency and gaps..... 7
	5.2. LTE Latency Impact on TCP Performance................... 7		6.1. TCP Layer Latency Improvement Alone is not enough....... 7
	5.3. Low Latency via Multipath TCP Extension................. 8		6.2. LTE Latency Impact on TCP Performance................... 8
	6. Network and Link Layer Initiatives in reducing E2E Latency... 9		6.3. Low Latency via Multipath TCP Extension................. 9
	7. Radio Channel Quality Impact to flows with High QoS.......... 9		7. Network and Link Layer Initiatives in reducing E2E Latency... 9
	8. E2E Latency Contributed by multiple domains................. 10		8. Radio Channel Quality Impact to flows with High QoS......... 10
	9. Conclusion.................................................. 10		9. E2E Latency Contributed by multiple domains................. 10
	10. Security Considerations.................................... 10		10. Conclusion................................................. 11
	11. IANA Considerations........................................ 11		11. Security Considerations.................................... 11
	12. Acknowledgements........................................... 11		12. IANA Considerations........................................ 11
	13. References................................................. 11		13. Acknowledgements........................................... 11
	13.1. Normative References.................................. 11		14. References................................................. 11
	13.2. Informative References................................ 11		14.1. Normative References.................................. 11
	14. Appendix:.................................................. 11		14.2. Informative References................................ 12
	14.1. Example: multi-Segments Latency for services via		15. Appendix:.................................................. 12
	Cellular Access............................................. 11		15.1. Example: multi-Segments Latency for services via
	14.2. Latency contributed by multiple nodes................. 13		Cellular Access............................................. 12
	14.3. Latency through the Data Center that hosts S-GW & P-GW 13		15.2. Latency contributed by multiple nodes................. 14
	Authors' Addresses............................................. 14		15.3. Latency through the Data Center that hosts S-GW & P-GW 14
			Authors' Addresses............................................. 15
	Internet-Draft E2E Over Internet Latency Taxonomy		Internet-Draft E2E Over Internet Latency Taxonomy
	1. Introduction		1. Introduction
	Ultra-Low Latency is a highly desired property for many types of		Ultra-Low Latency is a highly desired property for many types of
	services, such as 5G MTC (Machine Type Communication) requiring		services, such as 5G MTC (Machine Type Communication) requiring
	E2E connection for V2V to be less than 2ms, AR/VR requiring delay		E2E connection for V2V to be less than 2ms, AR/VR requiring delay
	less than 5ms, V2X less than 20ms, etc.		less than 5ms, V2X less than 20ms, etc.
	skipping to change at page 4, line 29 ¶		skipping to change at page 4, line 29 ¶
	in reducing latency.		in reducing latency.
	The primary purpose of studying E2E Latency from architectural		The primary purpose of studying E2E Latency from architectural
	perspective is to help the IETF community identify potential work		perspective is to help the IETF community identify potential work
	areas for reducing E2E latency of services over the Internet.		areas for reducing E2E latency of services over the Internet.
	In recent years, the internet industry has been exploring		In recent years, the internet industry has been exploring
	technologies and innovations at all layers of the OSI stack to		technologies and innovations at all layers of the OSI stack to
	reduce latency. At the upper (application) layer, more contents		reduce latency. At the upper (application) layer, more contents
	are distributed to the edges closer to end points and more		are distributed to the edges closer to end points and more
	progress in Mobile Edge Computing (MEC). At Transport layer,		progress in Mobile Edge Computing (MEC) has been made. At the
	there are QUIC/L4S initiatives. At the network layer, there are		Transport layer, there are QUIC/L4S initiatives. At the network
	IP/MPLS Hardened pipe (RFC 7625), latency optimized router		layer, there are IP/MPLS Hardened pipe (RFC 7625), latency
	design, and BBF's Broadband Assured Services (BAS). At the link		optimized router design, and BBF's Broadband Assured Services
	layer, there are IETF DETNET, IEEE 802.1 TSN (Time Sensitive		(BAS). At the link layer, there are IETF DETNET, IEEE 802.1 TSN
	Networking), and Flex Ethernet (OIF).		(Time Sensitive Networking), and Flex Ethernet (OIF).
	By studying the contributing factors to E2E latency from various		By studying the contributing factors to E2E latency from various
	angles, the draft identifies some gaps of recent technology		angles, the draft identifies some gaps of recent technology
	advancement for E2E services traversing multiple domains and		advancement for E2E services traversing multiple domains and
	involving multiple layers, which can be the basis for IAB to		involving multiple layers, which can be the basis for IAB to
	organize more in-depth discussion, like workshop or cross Area		organize more in-depth discussion, like a workshop or cross Area
	(or industry wide) BOF, as the scope of the discussion will be		(or industry wide) BOF, as the scope of the discussion will be
	too wide for one IETF WG. The discussion might touch upon		too wide for one IETF WG. The discussion might touch upon
	multiple IETF areas.		multiple IETF areas.
	The goal of those further "deep-dive" workshop or cross area BOF		The goal of the further "deep-dive" workshop or cross area BOF is
	is to establish more comprehensive foundation to IESG and the		to establish more comprehensive foundation to IESG and the IETF
	IETF community in identifying potential work initiatives for		community in identifying potential work initiatives for reducing
	reducing E2E latency of services over the Internet.		E2E latency of services over the Internet.
	Internet-Draft E2E Over Internet Latency Taxonomy		Internet-Draft E2E Over Internet Latency Taxonomy
	2. Terminology		2. Terminology
	DA: Destination Address		DA: Destination Address
	DC: Data Center		DC: Data Center
	E2E: End To End		E2E: End To End
	skipping to change at page 5, line 31 ¶		skipping to change at page 5, line 31 ¶
	data.		data.
	LTE: Long Term Evolution		LTE: Long Term Evolution
	TS: Tenant System		TS: Tenant System
	VM: Virtual Machines		VM: Virtual Machines
	VN: Virtual Network		VN: Virtual Network
	3. Contributing Factors to E2E Latency		3. AR/VR Use Case
			The E-2-E delays of AR/VR system come from delay of multiple
			systems:
			- Tracking delay
			- Application delay
			- Rendering delay
			- Display delay
			For human beings not to feel dizzy viewing AR/VR images, the
			oculus delay should be less than 19.3ms, which includes display
			delay, computing delay, transport delay, and sensoring delay.
			That means the "Network Delay" budget is only 5ms at the most.
			Internet-Draft E2E Over Internet Latency Taxonomy
			4. Contributing Factors to E2E Latency
	Internet data is packaged and transported in small pieces of		Internet data is packaged and transported in small pieces of
	data. The flow of these small pieces of data directly affects a		data. The flow of these small pieces of data directly affects a
	user's internet experience. When data packets arrive in a smooth		user's internet experience. When data packets arrive in a smooth
	and timely manner, the user sees a continuous flow of data; if		and timely manner, the user sees a continuous flow of data; if
	data packets arrive with large and variable delays between		data packets arrive with large and variable delays between
	packets, the user's experience is degraded.		packets, the user's experience is degraded.
	Key contributing factors to E2E latency:		Key contributing factors to E2E latency:
	- Generation: delay between physical event and availability of		- Generation: delay between physical event and availability of
	data		data
	- Transmission: signal propagation, initial signal encoding		- Transmission: signal propagation, initial signal encoding
	- Processing: Forwarding, encap/decap, NAT, encryption,		- Processing: Forwarding, encap/decap, NAT, encryption,
	authentication, compress, error coding, signal translation		authentication, compress, error coding, signal translation
	- Multiplexing: Delays needed to support sharing; Shared channel		- Multiplexing: Delays needed to support sharing; Shared channel
	acquisition, output queuing, connection establishment		acquisition, output queuing, connection establishment
	- Grouping: Reduces frequency of control information and		- Grouping: Reduces frequency of control information and
	processing; Packetization, message aggregation		processing; Packetization, message aggregation
	Internet-Draft E2E Over Internet Latency Taxonomy
	The 2013 ISOC Workshop [Latency-ISOC] on Internet Latency		The 2013 ISOC Workshop [Latency-ISOC] on Internet Latency
	concluded that:		concluded that:
	o Bandwidth alone is not enough in reducing latency		o Bandwidth alone is not enough in reducing latency
	o Bufferbloat is one of the main causes for high latency in		o Bufferbloat is one of the main causes for high latency in
	the Internet.		the Internet.
	Figure 1 of the 2013 ISOC workshop report showed that the timing		Figure 1 of the 2013 ISOC workshop report showed that the timing
	of download of an apparently uncluttered example Web page		of download of an apparently uncluttered example Web page
	(ieeexplore.ieee.org), actually comprised of over one hundred		(ieeexplore.ieee.org), actually comprised of over one hundred
	objects, transferred over 23 connections needing 10 different DNS		objects, transferred over 23 connections needing 10 different DNS
	look-ups. This phenomenon just further proves that reducing E2E		look-ups. This phenomenon just further proves that reducing E2E
	latency will need multiple layers coordination and interaction.		latency will need multiple layers coordination and interaction.
	4. Application Layer Initiative in reducing E2E latency		5. Application Layer Initiative in reducing E2E latency
	More and more End to End services over internet are from end		More and more End to End services over internet are from end
	users/devices to applications hosted in data centers.		users/devices to applications hosted in data centers.
	As most content today is distributed, E2E services usually do not		As most content today is distributed, E2E services usually do not
	traverse the globe but rather more often than not, the network		traverse the globe but rather more often than not, the network
			Internet-Draft E2E Over Internet Latency Taxonomy
	segments that the E2E service traverses are from end users to		segments that the E2E service traverses are from end users to
	regional data centers. The practice of content distribution to		regional data centers. The practice of content distribution to
	the edge has transformed reaching low latency goals from fighting		the edge has transformed reaching low latency goals from fighting
	against the speed of light to optimizing communication between		against the speed of light to optimizing communication between
	end users and their desired content.		end users and their desired content.
	However, without awareness of latency characteristics of network		However, without awareness of latency characteristics of network
	segments, the content distribution mechanisms & algorithms might		segments, the content distribution mechanisms & algorithms might
	not achieve their intended optimal result.		not achieve their intended optimal result.
	4.1. Content Placement mechanisms need visibility to Network		5.1. Content Placement mechanisms need visibility to Network
	To be added.		To be added.
	5. Transport Layer Initiatives in reducing Latency and gaps		6. Transport Layer Initiatives in reducing Latency and gaps
	IETF QUIC, L4S are some of the initiatives in reducing E2E		IETF QUIC, L4S are some of the initiatives in reducing E2E
	latency at Transport Layer.		latency at the Transport Layer.
	Internet-Draft E2E Over Internet Latency Taxonomy
	IETF QUIC focus on the improvement from end points. It doesn't		IETF QUIC focus on the improvement from end points. It doesn't
	take into consideration of the network latency that the data		take into consideration of the network latency that the data
	packets traverse.		packets traverse.
	The IETF L4S uses AQM for network nodes to purposely drop packets		The IETF L4S uses AQM for network nodes to purposely drop packets
	or send indication to end points when their queues are above		or send indication to end points when their queues are above
	certain thresholds. The goal is for the end nodes to reduce		certain thresholds. The goal is for the end nodes to reduce
	transmission rate when intermediate nodes buffers are almost		transmission rate when intermediate nodes buffers are almost
	full. It has following issue:		full. It has following issues:
	As network aggregates many flows from many different end points		As network aggregates many flows from many different end points
	and most flows have variable data rate, a network node/port's		and most flows have variable data rate, an intermediate network
	buffer being almost full at one specific time doesn't mean that		node+port's buffer being almost full at one specific time
	the same amount of traffic will traverse the same port a few		doesn't mean that the same amount of traffic will traverse the
	microseconds later. If all end (source) points reduce		same port a few microseconds later. If all end (source) points
	transmission rate upon receiving the AQM indication (or		reduce transmission rate upon receiving the AQM indication (or
	experiencing packets drop), traffic through network will be		experiencing packets drop), traffic through the network can be
	greatly reduced. Then all end points will increase their rate,		greatly reduced (i.e. leaving no queue in the buffer). Then all
	causing traffic pattern oscillation and buffer congestion		end points can increase their rate, causing traffic pattern
	again.		oscillation and buffer congestion again.
	5.1. TCP Layer Latency Improvement Alone is not enough		6.1. TCP Layer Latency Improvement Alone is not enough
	The following example shows why simply optimizing transport layer		The following example shows why simply optimizing transport layer
	alone is not enough. More details can be found at		alone is not enough. More details can be found at
	https://www.w3.org/Protocols/HTTP/Performance/Pipeline.html.		https://www.w3.org/Protocols/HTTP/Performance/Pipeline.html.
			Internet-Draft E2E Over Internet Latency Taxonomy
	Typical web pages today contain a HyperText Markup Language		Typical web pages today contain a HyperText Markup Language
	(HTML) document, and many embedded images. Twenty or more		(HTML) document and many embedded images. Twenty or more
	embedded images are quite common. Each of these images is an		embedded images are quite common. Each of these images is an
	independent object in the Web, retrieved (or validated for		independent object in the Web, retrieved (or validated for
	change) separately. The common behavior for a web client,		change) separately. The common behavior for a web client,
	therefore, is to fetch the base HTML document, and then		therefore, is to fetch the base HTML document, and then
	immediately fetch the embedded objects, which are typically		immediately fetch the embedded objects, which are typically
	located on the same server.		located on the same server.
	The large number of embedded objects represents a change from		The large number of embedded objects represents a change from
	the environment in which the Web transfer protocol, the		the environment in which the Web transfer protocol, the
	Hypertext Transfer Protocol (HTTP) was designed. As a result,		Hypertext Transfer Protocol (HTTP), was designed. As a result,
	HTTP/1.0 handles multiple requests from the same server		HTTP/1.0 handles multiple requests from the same server
	inefficiently, creating a separate TCP connection for each		inefficiently, creating a separate TCP connection for each
	object.		object.
	5.2. LTE Latency Impact on TCP Performance		6.2. LTE Latency Impact on TCP Performance
	HTTP/TCP is the dominating application and transport layer		HTTP/TCP is the dominating application and transport layer
	protocol suite used on the internet today. According to HTTP		protocol suite used on the internet today. According to HTTP
	Internet-Draft E2E Over Internet Latency Taxonomy
	Archive (http://httparchive.org/trends.php), the typical size of		Archive (http://httparchive.org/trends.php), the typical size of
	HTTP based transactions over the internet are in the range of a		HTTP based transactions over the internet are in the range of a
	few 10's of Kbytes up to 1 Mbyte. In this size range, the TCP		few 10's of Kbytes up to 1 Mbyte. In this size range, the TCP
	slow start period is a significant part of the total transport		slow start period is a significant part of the total transport
	period of the packet stream.		period of the packet stream.
	During TCP slow start, TCP exponentially increases its congestion		During TCP slow start, TCP exponentially increases its congestion
	window, i.e. the number of segments it brings into flight, until		window, i.e. the number of segments it brings into flight, until
	it fully utilizes the throughput that LTE (Radio + EPC) can		it fully utilizes the throughput that LTE (Radio + EPC) can
	offer. The incremental increases are based on TCP ACKs which are		offer. The incremental increases are based on TCP ACKs which are
	received after one round trip delay in the LTE system. Thus, as		received after one round trip delay in the LTE system. Thus, as
	it turns out, during TCP slow start the performance is latency		it turns out, during TCP slow start the performance is latency
	limited in Radio Network (LTE). Hence, improved latency in LTE		limited in Radio Network (LTE). Hence, improved latency in LTE
	can improve the perceived data rate for TCP based data		can improve the perceived data rate for TCP based data
	transactions, which in its turn reduces the time it takes to		transactions, which in its turn reduces the time it takes to
	complete a data down-load or upload.		complete a data down-load or upload.
	Despite rather small (in terms of milliseconds) improvements that		Despite rather small (in terms of milliseconds) improvements that
	can be achieved over the radio round trip time, the total		can be achieved over the radio round trip time, the total
	increase in in the perceived throughput and delay savings of		increase in the perceived throughput and delay savings of
	downloading an item below 1MB is significant due to the additive		downloading an item below 1MB is significant due to the additive
	effect of LTE latency improvements in the TCP slow start[LTE-		effect of LTE latency improvements in the TCP slow start[LTE-
	Research].		Research].
	5.3. Low Latency via Multipath TCP Extension		Internet-Draft E2E Over Internet Latency Taxonomy
			6.3. Low Latency via Multipath TCP Extension
	There are some research work on how to use multi-path TCP to		There are some research work on how to use multi-path TCP to
	reduce E2E latency, such as		reduce E2E latency, such as
	http://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=7510787. The		http://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=7510787. The
	paper proposes an MPTCP extension that sends data redundantly		paper proposes an MPTCP extension that sends data redundantly
	over multiple paths in the network, which basically exchanges		over multiple paths in the network, which basically exchanges
	bandwidth for latency. The integration into the MPTCP protocol		bandwidth for latency. The integration into the MPTCP protocol
	provides benefits such as transparent end-to-end connection		provides benefits such as transparent end-to-end connection
	establishment, multipath-enabled congestion control, and the		establishment, multipath-enabled congestion control, and the
	prevention of head of line blocking. The research paper claims		prevention of head of line blocking. The research paper claims
	that their proposed Multipath TCP extension can halve the average		that their proposed Multipath TCP extension can halve the average
	round-trip time and reduce its standard deviation by a factor of		round-trip time and reduce its standard deviation by a factor of
	19 for a real world mobile scenario in a stressed environment.		19 for a real world mobile scenario in a stressed environment.
	Those kind of researchers should be invited to the "Reducing		Those kind of researchers should be invited to the "Reducing
	latency over Internet Deep-Dive" workshop or cross-area BOF (to		latency over Internet Deep-Dive" workshop or cross-area BOF (to
	be organized by IAB).		be organized by IAB).
	Internet-Draft E2E Over Internet Latency Taxonomy		7. Network and Link Layer Initiatives in reducing E2E Latency
	6. Network and Link Layer Initiatives in reducing E2E Latency
	Several industry initiatives already exist for improving latency		Several industry initiatives already exist for improving latency
	at the Link and Network layers:		at the Link and Network layers:
	- Link Layer: IEEE 802.1 TSN (Time Sensitive Networking), and		- Link Layer: IEEE 802.1 TSN (Time Sensitive Networking), and
	Flex Ethernet (OIF).		Flex Ethernet (OIF).
	- The network layer: IETF DETNET, IP/MPLS Hardened pipe (RFC		- The network layer: IETF DETNET, IP/MPLS Hardened pipe (RFC
	7625).		7625).
	Gaps:		Gaps:
	IEEE 802.1 TSN (Time Sensitive Networking) requires stringent		IEEE 802.1 TSN (Time Sensitive Networking) requires stringent
	synchronous timing among all the nodes, which is suitable for		synchronous timing among all the nodes, which is suitable for
	small scoped network, not suitable for the internet because most		small scoped network, but not suitable for the internet because
	routers/switches in the network don't support synchronous timing.		most routers/switches in the network don't support synchronous
			timing.
	IP/MPLS hardened pipe can guarantee no congestion and no		IP/MPLS hardened pipe can guarantee no congestion and no
	buffering on all nodes along the path, therefore, ensure the		buffering on all nodes along the path, therefore, ensure the
	lowest latency along the path. The hardened pipe is ideal for		lowest latency along the path. The hardened pipe is ideal for
	flows with steady bandwidth requirement.		flows with steady bandwidth requirement.
			Internet-Draft E2E Over Internet Latency Taxonomy
	But for applications that don't have steady flow size, the		But for applications that don't have steady flow size, the
	hardened pipe requires reserving the peak rate dedicated		hardened pipe requires reserving the peak rate dedicated
	channels, which, like TDM, will incur bandwidth waste when		channels, which, like TDM, will incur bandwidth waste when
	application traffic goes below peak rate.		application traffic goes below peak rate.
	Traffic Engineering is one of the most commonly used methods to		Traffic Engineering is one of the most commonly used methods to
	reduce congestion at the network layer. However, it doesn't		reduce congestion at the network layer. However, it doesn't
	completely prevent transient congestion. Depending on the tunnel		completely prevent transient congestion. Depending on the tunnel
	sizing, there could be momentary traffic bursts that exceed the		sizing, there could be momentary traffic bursts that exceed the
	tunnel size, thus causing congestion if there isn't adequate		tunnel size, thus causing congestion if there isn't adequate
	headroom on the trunk carrying the tunnel to absorb the burst. Or		headroom on the trunk carrying the tunnel to absorb the burst. Or
	a link or node outage, that reroutes the tunnel onto a secondary		a link or node outage, that reroutes the tunnel onto a secondary
	path that becomes overloaded, could cause congestion.		path that becomes overloaded, could cause congestion.
	7. Radio Channel Quality Impact to flows with High QoS.		8. Radio Channel Quality Impact to flows with High QoS.
	QoS is one of the key methods employed by fixed IP network to		QoS is one of the key methods employed by fixed IP network to
	reduce latency for some flows. However, in Radio network, if a		reduce latency for some flows. However, in Radio network, if a
	UE's channel condition is poor, the eNB may schedule more frames		UE's channel condition is poor, the eNB may schedule more frames
	to other UEs whose flow are marked with much lower QoS.		to other UEs whose flow are marked with much lower QoS.
	There are many studies showing how Radio quality negatively		There are many studies showing how Radio quality negatively
	impact to the TCP performance.		impact to the TCP performance.
	Internet-Draft E2E Over Internet Latency Taxonomy
	It is beneficial to the whole industry if there is a workshop to		It is beneficial to the whole industry if there is a workshop to
	get people or SDOs working different layers of Internet service		get people or SDOs working on different layers of Internet
	together to showcase their work or their pain points.		service together to showcase their work or their pain points.
	IESG can make much more informed decision on creating useful		IESG can make much more informed decision on creating useful
	initiatives when the community is aware of other work and		initiatives when the community is aware of other work and
	obstacles.		obstacles.
	8. E2E Latency Contributed by multiple domains		9. E2E Latency Contributed by multiple domains
	All of the latency improvement initiatives in the link layer have		All of the latency improvement initiatives in the link layer have
	been within a single domain, such as IETF DETNET, IEEE 802.1 TSN		been within a single domain, such as IETF DETNET, IEEE 802.1 TSN
	(Time Sensitive Networking), and Flex Ethernet (OIF). The network		(Time Sensitive Networking), and Flex Ethernet (OIF). The network
	layer latency improvement, such as IP/MPLS Hardened pipe (RFC		layer latency improvement, such as IP/MPLS Hardened pipe (RFC
	7625) is also within a single domain.		7625) is also within a single domain.
	But E2E services usually traverse more than one domain, which can		But E2E services usually traverse more than one domain, which can
	be administrative domains or multiple operators' networks.		be administrative domains or multiple operators' networks.
	Yet today, there is no interface between domains to:		Yet today, there is no interface between domains to:
			Internet-Draft E2E Over Internet Latency Taxonomy
	- Inquire about the latency characteristics or capabilities from		- Inquire about the latency characteristics or capabilities from
	another domain		another domain
	- Negotiate or reserve latency capabilities from another domain.		- Negotiate or reserve latency capabilities from another domain.
	- Have a standardized method to characterize latency		- Have a standardized method to characterize latency
	IETF/IAB is an ideal organization to tackle those issues because		IETF/IAB is an ideal organization to tackle those issues because
	IETF has the expertise.		IETF has the expertise.
	9. Conclusion		10. Conclusion
	As end to end services traverse multiple types of network		As end to end services traverse multiple types of network
	segments and domains, and involve multiple layers, more informed		segments and domains, and involve multiple layers, more informed
	decision in each layer technological improvement is important.		decision in each layer technological improvement is important.
	- Need across domain coordination		- Need across domain coordination
	- Need across layer coordination		- Need across layer coordination
	10. Security Considerations		11. Security Considerations
	As the trend is going more encryption, it is getting more		As the trend is going more encryption, it is getting more
	difficult for various network segments to detect applications		difficult for various network segments to detect applications
	Internet-Draft E2E Over Internet Latency Taxonomy
	sessions. Therefore, it is more important to create ways for		sessions. Therefore, it is more important to create ways for
	better coordination among different layers, for improved latency,		better coordination among different layers, for improved latency,
	trouble shooting, restoration, etc.		trouble shooting, restoration, etc.
	11. IANA Considerations		12. IANA Considerations
	This section gives IANA allocation and registry considerations.		This section gives IANA allocation and registry considerations.
	12. Acknowledgements		13. Acknowledgements
	Special thanks to Jari Arkko for encouraging writing this draft.		Special thanks to Jari Arkko for encouraging writing this draft.
	And many thanks to Andy Malis, Jim Guichard, Spenser Dawkins, and		And many thanks to Andy Malis, Jim Guichard, Spenser Dawkins, and
	Donald Eastlake for suggestions and comments to this draft.		Donald Eastlake for suggestions and comments to this draft.
	13. References		14. References
	13.1. Normative References		14.1. Normative References
	13.2. Informative References		Internet-Draft E2E Over Internet Latency Taxonomy
			14.2. Informative References
	[LTE-latency] https://www.ericsson.com/research-blog/lte/lte-		[LTE-latency] https://www.ericsson.com/research-blog/lte/lte-
	latency-improvement-gains/		latency-improvement-gains/
	[Latency-ISOC] 2013 ISOC organized Latency over Internet workshop		[Latency-ISOC] 2013 ISOC organized Latency over Internet workshop
	report		report
	14. Appendix:		15. Appendix:
	14.1. Example: multi-Segments Latency for services via Cellular		15.1. Example: multi-Segments Latency for services via Cellular
	Access		Access
	Via Cellular network, there are User Plane Latency and Control		Via Cellular network, there are User Plane Latency and Control
	Plane Latency. Control plane deals with signaling and control		Plane Latency. Control plane deals with signaling and control
	functions, while user plane deals with actual user data		functions, while user plane deals with actual user data
	transmission.		transmission.
	The User Plane latency can be measured by the time it takes for a		The User Plane latency can be measured by the time it takes for a
	small IP packet to travel from the terminal through the network		small IP packet to travel from the terminal through the network
	to the internet server, and back. The Control Plane latency is		to the internet server, and back. The Control Plane latency is
	measured as the time required for the UE (User Equipment) to		measured as the time required for the UE (User Equipment) to
	transit from idle state to active state.		transit from idle state to active state.
	Internet-Draft E2E Over Internet Latency Taxonomy
	User Plane latency is relevant for the performance of many		User Plane latency is relevant for the performance of many
	applications. This document mainly focuses on the User Plane		applications. This document mainly focuses on the User Plane
	Latency. The following diagram depicts a logical path from an end		Latency. The following diagram depicts a logical path from an end
	user (smart phone) application to the application controller		user (smart phone) application to the application controller
	hosted in a data center via 4G Mobile network, which utilize the		hosted in a data center via 4G Mobile network, which utilize the
	Evolved Packet Core (EPC).		Evolved Packet Core (EPC).
	+------+ +---------+		+------+ +---------+
	|DC | | EPC | +----+		|DC | | EPC | +----+
	|Apps |<----------->|P-GW/S-GW|< -------> | eNB|<---> UE		|Apps |<----------->|P-GW/S-GW|< -------> | eNB|<---> UE
	| | +---------+ Mobile +----+ Radio		| | +---------+ Mobile +----+ Radio
	+------+ Internet Backhaul Access		+------+ Internet Backhaul Access
	Mobility Management Entity (MME) is responsible for		Mobility Management Entity (MME) is responsible for
	authentication of the mobile device. MME retains location		authentication of the mobile device. MME retains location
	information for each user and then selects the Serving Gateway		information for each user and then selects the Serving Gateway
	(S-GW) for a UE at the initial attach and at time of intra-LTE		(S-GW) for a UE at the initial attach and at time of intra-LTE
	handover involving Core Network (CN) node relocation.		handover involving Core Network (CN) node relocation.
			Internet-Draft E2E Over Internet Latency Taxonomy
	The Serving Gateway (S-GW) resides in the user plane where it		The Serving Gateway (S-GW) resides in the user plane where it
	forwards and routes packets to and from the eNodeB (eNB)		forwards and routes packets to and from the eNodeB (eNB)
	and packet data network gateway (P-GW). The S-GW also serves as		and packet data network gateway (P-GW). The S-GW also serves as
	the local mobility anchor for inter-eNodeB handover and mobility		the local mobility anchor for inter-eNodeB handover and mobility
	between 3GPP networks.		between 3GPP networks.
	P-GW (Packet Data Network Gateway) provides connectivity from the		P-GW (Packet Data Network Gateway) provides connectivity from the
	UE to external packet data networks by being the point of exit		UE to external packet data networks by being the point of exit
	and entry of traffic for the UE. A UE may have simultaneous		and entry of traffic for the UE. A UE may have simultaneous
	connectivity with more than one P-GW for accessing multiple		connectivity with more than one P-GW for accessing multiple
	skipping to change at page 13, line 4 ¶		skipping to change at page 13, line 34 ¶
	eNB and S-GW is encapsulated by GTP-U.		eNB and S-GW is encapsulated by GTP-U.
	The figure above shows that the end to end services from/to UE		The figure above shows that the end to end services from/to UE
	consists of the following network segments:		consists of the following network segments:
	- Radio Access network - RAN		- Radio Access network - RAN
	- Mobile Backhaul network that connect eNB to S-GW.		- Mobile Backhaul network that connect eNB to S-GW.
	- Network within the DC that hosts S-GW & P-GW		- Network within the DC that hosts S-GW & P-GW
	- Packet Data Network, which can dedicated VPN, internet, or		- Packet Data Network, which can dedicated VPN, internet, or
	other data network.		other data network.
	Internet-Draft E2E Over Internet Latency Taxonomy
	- Network within the DC that hosts the App.		- Network within the DC that hosts the App.
	The RAN (Radio Access Network) is between UE (e.g. smart phone)		The RAN (Radio Access Network) is between UE (e.g. smart phone)
	and eNB. 3GPP has a group TSG RAN working on improving		and eNB. 3GPP has a group TSG RAN working on improving
	performance (including latency) of the Radio Access network.		performance (including latency) of the Radio Access network.
	There are many factors impacting the latency through RAN.		There are many factors impacting the latency through RAN.
	The Mobile Backhaul Network connects eNBs to S-GW/P-GW, with data		The Mobile Backhaul Network connects eNBs to S-GW/P-GW, with data
	traffic being encapsulated in GTP protocol. The number of UEs		traffic being encapsulated in GTP protocol. The number of UEs
	that one eNB can handle are in 100s. The number of UEs that one		that one eNB can handle are in 100s. The number of UEs that one
	S-GW/P-GW can handle are in millions. Therefore, the mobile		S-GW/P-GW can handle are in millions. Therefore, the mobile
	backhaul network connects 10s of thousands of eNBs to S-GW/P-GW.		backhaul network connects 10s of thousands of eNBs to S-GW/P-GW.
	Therefore, the number of network nodes in the Mobile Backhaul		Therefore, the number of network nodes in the Mobile Backhaul
	network can be very large. Therefore, any new protocol		network can be very large. Therefore, any new protocol
	improvement in reducing latency can play a big part in reducing		improvement in reducing latency can play a big part in reducing
	the overall latency for the end to end services.		the overall latency for the end to end services.
	14.2. Latency contributed by multiple nodes		Internet-Draft E2E Over Internet Latency Taxonomy
			15.2. Latency contributed by multiple nodes
	The variant of delay for data packets through network is caused		The variant of delay for data packets through network is caused
	by network nodes along the path as the transmission delay on		by network nodes along the path as the transmission delay on
	physical link is fixed. When there is no congestion, the latency		physical link is fixed. When there is no congestion, the latency
	across most routers and switches are very small, in the magnitude		across most routers and switches are very small, in the magnitude
	of ~20us (worst case in ~40us). When congestion occurs within a		of ~20us (worst case in ~40us). When congestion occurs within a
	node, i.e. with buffer/queues being used to avoid dropping		node, i.e. with buffer/queues being used to avoid dropping
	packets, latency across a node can be in the magnitude of micro-		packets, latency across a node can be in the magnitude of micro-
	seconds. The recent improvements made within router architecture		seconds. The recent improvements made within router architecture
	have greatly improved latency through a node. However, there is		have greatly improved latency through a node. However, there is
	no standard methods for routers to characterize and expose		no standard methods for routers to characterize and expose
	various latency characteristics through a network node.		various latency characteristics through a network node.
	Data packets also traverse through network functions, such as FW,		Data packets also traverse through network functions, such as FW,
	DPI, OPS, whose latency vary depending on the depth of the		DPI, OPS, whose latency vary depending on the depth of the
	processing and the equipment performance.		processing and the equipment performance.
	14.3. Latency through the Data Center that hosts S-GW & P-GW		15.3. Latency through the Data Center that hosts S-GW & P-GW
	S-GW and P-GW are hosted in Data center. There are typically 2~3		S-GW and P-GW are hosted in Data center. There are typically 2~3
	tiers of switches connecting the servers that hosts S-GW & P-GW		tiers of switches connecting the servers that hosts S-GW & P-GW
	to the external network, as depicted in the following:		to the external network, as depicted in the following:
	Internet-Draft E2E Over Internet Latency Taxonomy
	+---------+		+---------+
	| Gateway |		| Gateway |
	+---------+		+---------+
	\ +-------+ +------+ /		\ +-------+ +------+ /
	\ +/------+ | +/-----+ | /		\ +/------+ | +/-----+ | /
	\ | Aggr11| + ----- |AggrN1| + /		\ | Aggr11| + ----- |AggrN1| + /
	\ +---+---+/ +------+/ /		\ +---+---+/ +------+/ /
	\ / \ / \ /		\ / \ / \ /
	\ / \ / \ /		\ / \ / \ /
	skipping to change at page 14, line 29 ¶		skipping to change at page 15, line 5 ¶
	+---+ +---+ +---+ +---+		+---+ +---+ +---+ +---+
	| | | |		| | | |
	+-|-+ +-|-+ +-|-+ +-|-+ Servers		+-|-+ +-|-+ +-|-+ +-|-+ Servers
	| |... |SGW| | S | | S |<-		| |... |SGW| | S | | S |<-
	+---+ +---+ +---+ +---+		+---+ +---+ +---+ +---+
	| |... |PGW| | S | ... | S |		| |... |PGW| | S | ... | S |
	+---+ +---+ +---+ +---+		+---+ +---+ +---+ +---+
	| |... | S | | S | ... | S |		| |... | S | | S | ... | S |
	+---+ +---+ +---+ +---+		+---+ +---+ +---+ +---+
			Internet-Draft E2E Over Internet Latency Taxonomy
	As the distance within data center can be small, the transmission		As the distance within data center can be small, the transmission
	delay within data center can be negligent. The majority of		delay within data center can be negligent. The majority of
	latency within data center is caused by the switching within the		latency within data center is caused by the switching within the
	gateway routers, traffic traversing through middleware boxes such		gateway routers, traffic traversing through middleware boxes such
	as FW, DPI, IPS, value added services, the top of the rack		as FW, DPI, IPS, value added services, the top of the rack
	switches, and aggregation switches.		switches, and aggregation switches.
	If the S-GW and P-GW are hosted in large data center, there could		If the S-GW and P-GW are hosted in large data center, there could
	be latency contributed by the		be latency contributed by the
	encapsulation/decapsulation such as work specified by		encapsulation/decapsulation such as work specified by
End of changes. 50 change blocks.
	98 lines changed or deleted		115 lines changed or added
This html diff was produced by rfcdiff 1.48. The latest version is available from http://tools.ietf.org/tools/rfcdiff/