< draft-krishna-mops-ar-use-case-01.txt		draft-krishna-mops-ar-use-case-02.txt >
	MOPS R. Krishna		MOPS R. Krishna
	Internet-Draft InterDigital Europe Limited		Internet-Draft InterDigital Europe Limited
	Intended status: Informational A. Rahman		Intended status: Informational A. Rahman
	Expires: May 3, 2021 InterDigital Communications, LLC		Expires: August 21, 2021 InterDigital Communications, LLC
	October 30, 2020		February 17, 2021
	Media Operations Use Case for an Augmented Reality Application on Edge		Media Operations Use Case for an Augmented Reality Application on Edge
	Computing Infrastructure		Computing Infrastructure
	draft-krishna-mops-ar-use-case-01		draft-krishna-mops-ar-use-case-02
	Abstract		Abstract
	A use case describing transmission of an application on the Internet		A use case describing transmission of an application on the Internet
	that has several unique characteristics of Augmented Reality (AR)		that has several unique characteristics of Augmented Reality (AR)
	applications is presented for the consideration of the Media		applications is presented for the consideration of the Media
	Operations (MOPS) Working Group. One key requirement identified is		Operations (MOPS) Working Group. One key requirement identified is
	that the Adaptive-Bit-Rate (ABR) algorithms' current usage of		that the Adaptive-Bit-Rate (ABR) algorithms' current usage of
	policies based on heuristics and models is inadequate for AR		policies based on heuristics and models is inadequate for AR
	applications running on the Edge Computing infrastructure.		applications running on the Edge Computing infrastructure.
	skipping to change at page 1, line 38 ¶		skipping to change at page 1, line 38 ¶
	Internet-Drafts are working documents of the Internet Engineering		Internet-Drafts are working documents of the Internet Engineering
	Task Force (IETF). Note that other groups may also distribute		Task Force (IETF). Note that other groups may also distribute
	working documents as Internet-Drafts. The list of current Internet-		working documents as Internet-Drafts. The list of current Internet-
	Drafts is at https://datatracker.ietf.org/drafts/current/.		Drafts is at https://datatracker.ietf.org/drafts/current/.
	Internet-Drafts are draft documents valid for a maximum of six months		Internet-Drafts are draft documents valid for a maximum of six months
	and may be updated, replaced, or obsoleted by other documents at any		and may be updated, replaced, or obsoleted by other documents at any
	time. It is inappropriate to use Internet-Drafts as reference		time. It is inappropriate to use Internet-Drafts as reference
	material or to cite them other than as "work in progress."		material or to cite them other than as "work in progress."
	This Internet-Draft will expire on May 3, 2021.		This Internet-Draft will expire on August 21, 2021.
	Copyright Notice		Copyright Notice
	Copyright (c) 2020 IETF Trust and the persons identified as the		Copyright (c) 2021 IETF Trust and the persons identified as the
	document authors. All rights reserved.		document authors. All rights reserved.
	This document is subject to BCP 78 and the IETF Trust's Legal		This document is subject to BCP 78 and the IETF Trust's Legal
	Provisions Relating to IETF Documents		Provisions Relating to IETF Documents
	(https://trustee.ietf.org/license-info) in effect on the date of		(https://trustee.ietf.org/license-info) in effect on the date of
	publication of this document. Please review these documents		publication of this document. Please review these documents
	carefully, as they describe your rights and restrictions with respect		carefully, as they describe your rights and restrictions with respect
	to this document. Code Components extracted from this document must		to this document. Code Components extracted from this document must
	include Simplified BSD License text as described in Section 4.e of		include Simplified BSD License text as described in Section 4.e of
	the Trust Legal Provisions and are provided without warranty as		the Trust Legal Provisions and are provided without warranty as
	described in the Simplified BSD License.		described in the Simplified BSD License.
	Table of Contents		Table of Contents
	1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2		1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2
	2. Conventions used in this document . . . . . . . . . . . . . . 3		2. Conventions used in this document . . . . . . . . . . . . . . 3
	3. Use Case . . . . . . . . . . . . . . . . . . . . . . . . . . 3		3. Use Case . . . . . . . . . . . . . . . . . . . . . . . . . . 3
	4. Requirements . . . . . . . . . . . . . . . . . . . . . . . . 3		3.1. Processing of Scenes . . . . . . . . . . . . . . . . . . 3
			3.2. Generation of Images . . . . . . . . . . . . . . . . . . 4
			4. Requirements . . . . . . . . . . . . . . . . . . . . . . . . 4
	5. Informative References . . . . . . . . . . . . . . . . . . . 5		5. Informative References . . . . . . . . . . . . . . . . . . . 5
	Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 6		Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 8
	1. Introduction		1. Introduction
	The MOPS draft, [I-D.ietf-mops-streaming-opcons], provides an		The MOPS draft, [I-D.ietf-mops-streaming-opcons], provides an
	overview of operational networking issues that pertain to Quality of		overview of operational networking issues that pertain to Quality of
	Experience (QoE) in delivery of video and other high-bitrate media		Experience (QoE) in delivery of video and other high-bitrate media
	over the Internet. However, as it does not cover the increasingly		over the Internet. However, as it does not cover the increasingly
	large number of applications with Augmented Reality (AR)		large number of applications with Augmented Reality (AR)
	characteristics and their requirements on ABR algorithms, the		characteristics and their requirements on ABR algorithms, the
	discussion in this draft compliments the overview presented in that		discussion in this draft compliments the overview presented in that
	draft [I-D.ietf-mops-streaming-opcons].		draft [I-D.ietf-mops-streaming-opcons].
	Future AR applications will bring several requirements for the		Future AR applications will bring several requirements for the
	Internet and the mobile devices running these applications. AR		Internet and the mobile devices running these applications. AR
	applications require a real-time processing of video streams to		applications require a real-time processing of video streams to
	recognize specific objects. This is then used to overlay information		recognize specific objects. This is then used to overlay information
	on the video being displayed to the user. In addition some AR		on the video being displayed to the user. In addition some AR
	applications will also require generation of new video frames to be		applications will also require generation of new video frames to be
	played to the user. In order to run future applications with AR		played to the user. Both the real-time processing of video streams
			and the generation of overlay information are computationally
			intensive tasks that generate heat [DEV_HEAT_1], [DEV_HEAT_2] and
			drain battery power [BATT_DRAIN] on the AR mobile device.
			Consequently, in order to run future applications with AR
	characteristics on mobile devices, computationally intensive tasks		characteristics on mobile devices, computationally intensive tasks
	need to be offloaded to resources provided by Edge Computing.		need to be offloaded to resources provided by Edge Computing.
	Edge Computing is an emerging paradigm where computing resources and		Edge Computing is an emerging paradigm where computing resources and
	storage are made available in close network proximity at the edge of		storage are made available in close network proximity at the edge of
	the Internet to mobile devices and sensors [EDGE_1], [EDGE_2].		the Internet to mobile devices and sensors [EDGE_1], [EDGE_2].
	Adaptive-Bit-Rate (ABR) algorithms currently base their policy for		Adaptive-Bit-Rate (ABR) algorithms currently base their policy for
	bit-rate selection on heuristics or models of the deployment		bit-rate selection on heuristics or models of the deployment
	environment that do not account for the environment's dynamic nature		environment that do not account for the environment's dynamic nature
	skipping to change at page 3, line 22 ¶		skipping to change at page 3, line 25 ¶
	We now descibe a use case that involves an application with AR		We now descibe a use case that involves an application with AR
	systems' characteristics. Consider a group of tourists who are being		systems' characteristics. Consider a group of tourists who are being
	conducted in a tour around the historical site of the Tower of		conducted in a tour around the historical site of the Tower of
	London. As they move around the site and within the historical		London. As they move around the site and within the historical
	buildings, they can watch and listen to historical scenes in 3D that		buildings, they can watch and listen to historical scenes in 3D that
	are generated by the AR application and then overlaid by their AR		are generated by the AR application and then overlaid by their AR
	headsets onto their real-world view. The headset then continuously		headsets onto their real-world view. The headset then continuously
	updates their view as they move around.		updates their view as they move around.
	The AR application processes the scene that the walking tourist is		The AR application first processes the scene that the walking tourist
	watching in real-time and identifies objects that will be targeted		is watching in real-time and identifies objects that will be targeted
	for overlay of high resolution videos. It then generates high		for overlay of high resolution videos. It then generates high
	resolution 3D images of historical scenes related to the perspective		resolution 3D images of historical scenes related to the perspective
	of the tourist in real-time. These generated video images are then		of the tourist in real-time. These generated video images are then
	overlaid on the view of the real-world as seen by the tourist.		overlaid on the view of the real-world as seen by the tourist.
	Offloading to the remote Cloud is not feasible for applications with		We now discuss this processsing of scenes and generation of high
	AR characteristics as the end-to-end delays must be within the order		resolution images in greater detail.
	of a few milliseconds. In order to achieve such hard timing
	constraints, computationally intensive tasks can be offloaded to Edge		3.1. Processing of Scenes
	devices.
			The AR application that runs on the mobile device needs to first
			track the pose (coordinates and orientation) of the user's head, eyes
			and the objects that are in view.This requires tracking natural
			features and developing an annotated point cloud based model that is
			then stored in a database.To ensure that this database can be scaled
			up,techniques such as combining a client side simultaneous tracking
			and mapping and a server-side localization are used[SLAM_1],
			[SLAM_2], [SLAM_3], [SLAM_4]. Once the natural features are tracked,
			virtual objects are geometrically aligned with those features.This is
			followed by resolving occlusion that can occur between virtual and
			the real objects [OCCL_1], [OCCL_2].
			The next step for the AR apllication is to apply photometric
			registration [PHOTO_REG]. This requires aligning the brightness and
			color between the virtual and real objects.Additionally, algorithms
			that calculate global illumination of both the virtual and real
			objects [GLB_ILLUM_1], [GLB_ILLUM_2] are executed.Various algorithms
			to deal with artifacts generated by lens distortion [LENS_DIST], blur
			[BLUR], noise [NOISE] etc are also required.
			3.2. Generation of Images
			The AR application must generate a high-quality video that has the
			properties descibed in the previous step and overlay the video on the
			AR device's display- a step called situated visualization. This
			entails dealing with registration errors that may arise, esuring that
			there is no visual interference [VIS_INTERFERE], and finally
			maintaining temporal coherence by adapting to the movement of user's
			eyes and head.
	4. Requirements		4. Requirements
	As discussed above an AR application requires offloading of its		The components of AR applications perform tasks such as real-time
	components to resources provided by Edge Computing. These components		generation and processing of high-quality video content that are
	perform tasks such as real-time generation and processing of high-		computationally intensive. As a result,on AR devices such as AR
	quality video content that are too computationally intensive for the		glasses excessive heat is generated by the chip-sets that are
	mobile device.		involved in the computation [DEV_HEAT_1], [DEV_HEAT_2].
			Additionally, the battery on such devices discharges quickly when
			running such applications [BATT_DRAIN].
	In addition, such applications require high bandwidth and low jitter		A solution to the heat dissipation and battery drainge problem is to
	to provide a high QoE to the user. Another consequence of running		offload the processing and video generation tasks to the remote
	such computationally intensive applications on AR devices such as AR		cloud.However, running such tasks on the cloud is not feasible as the
	glasses is the excessive heat generated by the chip-sets that are		end-to-end delays must be within the order of a few milliseconds.
	involved in the computation [DEV_HEAT_1]. Finally, the battery on		Additionally,such applications require high bandwidth and low jitter
	such devices discharges quickly when running such applications if		to provide a high QoE to the user.In order to achieve such hard
	some processing is not off-loaded to the Edge Computing.		timing constraints, computationally intensive tasks can be offloaded
			to Edge devices.
	Note that the Edge device providing the computation and storage is		Note that the Edge device providing the computation and storage is
	itself limited in such resources compared to the Cloud. So, for		itself limited in such resources compared to the Cloud. So, for
	example, a sudden surge in demand from a large group of tourists can		example, a sudden surge in demand from a large group of tourists can
	overwhelm that device. This will result in a degraded user		overwhelm that device. This will result in a degraded user
	experience as their AR device experiences delays in receiving the		experience as their AR device experiences delays in receiving the
	video frames. In order to deal with this problem, the client AR		video frames. In order to deal with this problem, the client AR
	applications will need to use Adaptive Bit Rate (ABR) algorithms that		applications will need to use Adaptive Bit Rate (ABR) algorithms that
	choose bit-rates policies tailored in a fine-grained manner to the		choose bit-rates policies tailored in a fine-grained manner to the
	resource demands and playback the videos with appropriate QoE metrics		resource demands and playback the videos with appropriate QoE metrics
	skipping to change at page 5, line 14 ¶		skipping to change at page 5, line 50 ¶
	in the ABR algorithm not changing to a different rate so as to		in the ABR algorithm not changing to a different rate so as to
	ensure a non-fluctuating bit-rate and the resultant smoothness of		ensure a non-fluctuating bit-rate and the resultant smoothness of
	video quality . The ABR algorithm must be able to handle this		video quality . The ABR algorithm must be able to handle this
	situation.		situation.
	5. Informative References		5. Informative References
	[ABR_1] Mao, H., Netravali, R., and M. Alizadeh, "Neural Adaptive		[ABR_1] Mao, H., Netravali, R., and M. Alizadeh, "Neural Adaptive
	Video Streaming with Pensieve", In Proceedings of the		Video Streaming with Pensieve", In Proceedings of the
	Conference of the ACM Special Interest Group on Data		Conference of the ACM Special Interest Group on Data
	Communication, (pp. 197-210), 2017.		Communication, pp. 197-210, 2017.
	[ABR_2] Yan, F., Ayers, H., Zhu, C., Fouladi, S., Hong, J., Zhang,		[ABR_2] Yan, F., Ayers, H., Zhu, C., Fouladi, S., Hong, J., Zhang,
	K., Levis, P., and K. Winstein, "Learning in situ: a		K., Levis, P., and K. Winstein, "Learning in situ: a
	randomized experiment in video streaming", In 17th		randomized experiment in video streaming", In 17th
	{USENIX} Symposium on Networked Systems Design and		USENIX Symposium on Networked Systems Design and
	Implementation ({NSDI} 20), (pp. 495-511), 2020.		Implementation (NSDI 20), pp. 495-511, 2020.
			[BATT_DRAIN]
			Seneviratne, S., Hu, Y., Nguyen, T., Lan, G., Khalifa, S.,
			Thilakarathna, K., Hassan, M., and A. Seneviratne, "A
			survey of wearable devices and challenges.", In IEEE
			Communication Surveys and Tutorials, 19(4), p.2573-2620.,
			2017.
			[BLUR] Kan, P. and H. Kaufmann, "Physically-Based Depth of Field
			in Augmented Reality.", In Eurographics (Short Papers),
			pp. 89-92., 2012.
	[DEV_HEAT_1]		[DEV_HEAT_1]
	LiKamWa, R., Wang, Z., Carroll, A., Lin, F., and L. Zhong,		LiKamWa, R., Wang, Z., Carroll, A., Lin, F., and L. Zhong,
	"Draining our Glass: An Energy and Heat characterization		"Draining our Glass: An Energy and Heat characterization
	of Google Glass", In Proceedings of 5th Asia-Pacific		of Google Glass", In Proceedings of 5th Asia-Pacific
	Workshop on Systems (pp. 1-7), 2013.		Workshop on Systems pp. 1-7, 2013.
			[DEV_HEAT_2]
			Matsuhashi, K., Kanamoto, T., and A. Kurokawa, "Thermal
			model and countermeasures for future smart glasses.",
			In Sensors, 20(5), p.1446., 2020.
	[EDGE_1] Satyanarayanan, M., "The Emergence of Edge Computing",		[EDGE_1] Satyanarayanan, M., "The Emergence of Edge Computing",
	In Computer 50(1) (pp. 30-39), 2017.		In Computer 50(1) pp. 30-39, 2017.
	[EDGE_2] Satyanarayanan, M., Klas, G., Silva, M., and S. Mangiante,		[EDGE_2] Satyanarayanan, M., Klas, G., Silva, M., and S. Mangiante,
	"The Seminal Role of Edge-Native Applications", In IEEE		"The Seminal Role of Edge-Native Applications", In IEEE
	International Conference on Edge Computing (EDGE) (pp.		International Conference on Edge Computing (EDGE) pp.
	33-40), 2019.		33-40, 2019.
			[GLB_ILLUM_1]
			Kan, P. and H. Kaufmann, "Differential irradiance caching
			for fast high-quality light transport between virtual and
			real worlds.", In IEEE International Symposium on Mixed
			and Augmented Reality (ISMAR),pp. 133-141, 2013.
			[GLB_ILLUM_2]
			Franke, T., "Delta voxel cone tracing.", In IEEE
			International Symposium on Mixed and Augmented Reality
			(ISMAR), pp. 39-44, 2014.
	[HEAVY_TAIL_1]		[HEAVY_TAIL_1]
	Crovella, M. and B. Krishnamurthy, "Internet measurement:		Crovella, M. and B. Krishnamurthy, "Internet measurement:
	infrastructure, traffic and applications", John Wiley and		infrastructure, traffic and applications", John Wiley and
	Sons Inc., 2006.		Sons Inc., 2006.
	[HEAVY_TAIL_2]		[HEAVY_TAIL_2]
	Taleb, N., "The Statistical Consequences of Fat Tails",		Taleb, N., "The Statistical Consequences of Fat Tails",
	STEM Academic Press, 2020.		STEM Academic Press, 2020.
	[I-D.ietf-mops-streaming-opcons]		[I-D.ietf-mops-streaming-opcons]
	Holland, J., Begen, A., and S. Dawkins, "Operational		Holland, J., Begen, A., and S. Dawkins, "Operational
	Considerations for Streaming Media", draft-ietf-mops-		Considerations for Streaming Media", draft-ietf-mops-
	streaming-opcons-02 (work in progress), July 2020.		streaming-opcons-03 (work in progress), November 2020.
			[LENS_DIST]
			Fuhrmann, A. and D. Schmalstieg, "Practical calibration
			procedures for augmented reality.", In Virtual
			Environments 2000, pp. 3-12. Springer, Vienna, 2000.
			[NOISE] Fischer, J., Bartz, D., and W. Strasser, "Enhanced visual
			realism by incorporating camera image effects.",
			In IEEE/ACM International Symposium on Mixed and
			Augmented Reality, pp. 205-208., 2006.
			[OCCL_1] Breen, D., Whitaker, R., and M. Tuceryan, "Interactive
			Occlusion and automatic object placementfor augmented
			reality", In Computer Graphics Forum, vol. 15, no. 3 ,
			pp. 229-238,Edinburgh, UK: Blackwell Science Ltd, 1996.
			[OCCL_2] Zheng, F., Schmalstieg, D., and G. Welch, "Pixel-wise
			closed-loop registration in video-based augmented
			reality", In IEEE International Symposium on Mixed and
			Augmented Reality (ISMAR), pp. 135-143, 2014.
			[PHOTO_REG]
			Liu, Y. and X. Granier, "Online tracking of outdoor
			lighting variations for augmented reality with moving
			cameras", In IEEE Transactions on visualization and
			computer graphics, 18(4), pp.573-580, 2012.
	[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate		[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
	Requirement Levels", BCP 14, RFC 2119,		Requirement Levels", BCP 14, RFC 2119,
	DOI 10.17487/RFC2119, March 1997,		DOI 10.17487/RFC2119, March 1997,
	<https://www.rfc-editor.org/info/rfc2119>.		<https://www.rfc-editor.org/info/rfc2119>.
			[SLAM_1] Ventura, J., Arth, C., Reitmayr, G., and D. Schmalstieg,
			"A minimal solution to the generalized pose-and-scale
			problem", In Proceedings of the IEEE Conference on
			Computer Vision and Pattern Recognition, pp. 422-429,
			2014.
			[SLAM_2] Sweeny, C., Fragoso, V., Hollerer, T., and M. Turk, "A
			scalable solution to the generalized pose and scale
			problem", In European Conference on Computer Vision, pp.
			16-31, 2014.
			[SLAM_3] Gauglitz, S., Sweeny, C., Ventura, J., Turk, M., and T.
			Hollerer, "Model estimation and selection towards
			unconstrained real-time tracking and mapping", In IEEE
			transactions on visualization and computer graphics,
			20(6), pp. 825-838, 2013.
			[SLAM_4] Pirchheim, C., Schmalstieg, D., and G. Reitmayr, "Handling
			pure camera rotation in keyframe-based SLAM", In 2013
			IEEE international symposium on mixed and augmented
			reality (ISMAR), pp. 229-238, 2013.
	[UBICOMP] Bardram, J. and A. Friday, "Ubiquitous Computing Systems",		[UBICOMP] Bardram, J. and A. Friday, "Ubiquitous Computing Systems",
	In Ubiquitous Computing Fundamentals (pp. 37-94). CRC		In Ubiquitous Computing Fundamentals pp. 37-94. CRC
	Press, 2009.		Press, 2009.
			[VIS_INTERFERE]
			Kalkofen, D., Mendez, E., and D. Schmalstieg, "Interactive
			focus and context visualization for augmented reality.",
			In 6th IEEE and ACM International Symposium on Mixed and
			Augmented Reality, pp. 191-201., 2007.
	Authors' Addresses		Authors' Addresses
	Renan Krishna		Renan Krishna
	InterDigital Europe Limited		InterDigital Europe Limited
	64, Great Eastern Street		64, Great Eastern Street
	London EC2A 3QR		London EC2A 3QR
	United Kingdom		United Kingdom
	Email: renan.krishna@interdigital.com		Email: renan.krishna@interdigital.com
	Akbar Rahman		Akbar Rahman
	InterDigital Communications, LLC		InterDigital Communications, LLC
	1000 Sherbrooke Street West		1000 Sherbrooke Street West
	Montreal H3A 3G4		Montreal H3A 3G4
	Canada		Canada
	Email: Akbar.Rahman@InterDigital.com		Email: Akbar.Rahman@InterDigital.com
End of changes. 21 change blocks.
	37 lines changed or deleted		155 lines changed or added
This html diff was produced by rfcdiff 1.48. The latest version is available from http://tools.ietf.org/tools/rfcdiff/