idnits 2.17.1 draft-montpetit-coin-xr-01.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack a Security Considerations section. ** The document seems to lack an IANA Considerations section. (See Section 2.2 of https://www.ietf.org/id-info/checklist for how to handle the case when there are no actions for IANA.) Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document doesn't use any RFC 2119 keywords, yet seems to have RFC 2119 boilerplate text. -- The document date (October 23, 2018) is 2010 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Unused Reference: 'INTER' is defined on line 416, but no explicit reference was found in the text Summary: 2 errors (**), 0 flaws (~~), 3 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 COIN M. Montpetit 3 Internet-Draft Triangle Video 4 Intended status: Informational October 23, 2018 5 Expires: April 22, 2019 7 In Network Computing Enablers for Extended Reality 8 draft-montpetit-coin-xr-01 10 Abstract 12 Augmented Reality (AR) and Virtual Reality (VR), combined as Extended 13 Reality or XR, challenge networking technologies and protocols 14 because they combine the features of fast information display, image 15 processing, computing and forwarding. This document presents some of 16 these challenges and how adding computing in the network could 17 respond to them. 19 Status of This Memo 21 This Internet-Draft is submitted in full conformance with the 22 provisions of BCP 78 and BCP 79. 24 Internet-Drafts are working documents of the Internet Engineering 25 Task Force (IETF). Note that other groups may also distribute 26 working documents as Internet-Drafts. The list of current Internet- 27 Drafts is at https://datatracker.ietf.org/drafts/current/. 29 Internet-Drafts are draft documents valid for a maximum of six months 30 and may be updated, replaced, or obsoleted by other documents at any 31 time. It is inappropriate to use Internet-Drafts as reference 32 material or to cite them other than as "work in progress." 34 This Internet-Draft will expire on April 22, 2019. 36 Copyright Notice 38 Copyright (c) 2018 IETF Trust and the persons identified as the 39 document authors. All rights reserved. 41 This document is subject to BCP 78 and the IETF Trust's Legal 42 Provisions Relating to IETF Documents 43 (https://trustee.ietf.org/license-info) in effect on the date of 44 publication of this document. Please review these documents 45 carefully, as they describe your rights and restrictions with respect 46 to this document. Code Components extracted from this document must 47 include Simplified BSD License text as described in Section 4.e of 48 the Trust Legal Provisions and are provided without warranty as 49 described in the Simplified BSD License. 51 Table of Contents 53 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 54 1.1. Requirements Language . . . . . . . . . . . . . . . . . . 3 55 2. Definitions . . . . . . . . . . . . . . . . . . . . . . . . . 3 56 3. Extended Reality and In-Network Computing . . . . . . . . . . 4 57 3.1. XR Network Requirements . . . . . . . . . . . . . . . . . 4 58 3.2. In-Network Computing Advantages in XR . . . . . . . . . . 5 59 4. Enabling Technologies . . . . . . . . . . . . . . . . . . . . 6 60 4.1. Information Centric Networking (ICN) and Named Data 61 Networking (NDN) . . . . . . . . . . . . . . . . . . . . 7 62 4.2. Network Coding . . . . . . . . . . . . . . . . . . . . . 7 63 4.3. Blockchains and Distributed Trust . . . . . . . . . . . . 8 64 5. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . 9 65 6. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 9 66 7. References . . . . . . . . . . . . . . . . . . . . . . . . . 9 67 7.1. Normative References . . . . . . . . . . . . . . . . . . 9 68 7.2. Informative References . . . . . . . . . . . . . . . . . 9 69 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 10 71 1. Introduction 73 Augmented and Virtual reality target different applications 74 but they all share a number of stringent delay and bandwidth 75 requirements to prevent confusing the brain whenever information 76 about the virtual environment is not wholly consistent causing motion 77 sickness symptoms [VRSICK]. Hence to now XR has been delivered 78 mostly locally via combinations of computers and headsets with some 79 cloud implementations being limited to time invariant imaging in one 80 direction. 82 But with the emergence of the edge and the programmability of network 83 elements all the way from the data center to the users the 84 possibility of creating networked, multiparty/multisource and 85 interacting XR comes closer to reality. This document wants to 86 review what is necessary for the current localized and cloud 87 supported XR to evolve to a more distributed and edge centric 88 architecture to support advanced immersive application and services. 89 It assumes that network programmability will enable to tailor the 90 network to the XR requirements. This document is about requirements 91 not solutions per se but will mention work that has already been done 92 towards a more networked XR including Information Centric 93 architectures, Artificial Intelligence and in network coding. The 94 networked functionality should enable to supplement local XR services 95 and devices while keeping the very low latency and the very high data 96 rates that are required by XR. 98 This document is intended as informative to both the networking and 99 application research community. It does not address a specific 100 network layer or protocol but provides architecture and system level 101 specifications and guidelines. For example: 103 Latency: the physical distance between the XR content cloud of AR/ 104 VR and users are short enough to limit the propagation delay to 105 the 20 ms usually cited for XR applications [ref] mixed for 106 example with IoT devices and sensors delay reduction for range of 107 interest (RoI) detection. 109 Applications: better coding and use of compression algorithms, 110 pre-fetching and pre-caching and movement prediction. 112 Network access: push some networking functions in the data plane 113 into the user plane to enable the deployment of stream specific 114 algorithms for congestion control and application-based load 115 balancing based on machine learning and user data patterns. 117 1.1. Requirements Language 119 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 120 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 121 document are to be interpreted as described in RFC 2119 [RFC2119]. 123 2. Definitions 125 AR: Augmented Reality (AR) is a live direct or indirect view of a 126 physical, real-world environment whose elements are augmented by 127 computer-generated input such as sound, video, location or 128 graphical data. It is related to a more general concept called 129 mediated reality [MEDIA], in which reality is modified (diminished 130 or augmented) by computer-generated imagery. 132 VR Virtual Reality (VR): uses software-generated realistic 133 imaging, sounds and other sensor inputs to replicate a real or 134 imaginary setting, to simulates a user's physical presence in this 135 environment and provide an immersive experience that enable the 136 user to interact with objects and move within this space. 138 360-degree video: 360-degree videos, also known as immersive 139 videos or spherical videos, are video recordings where a view in 140 every direction is recorded at the same time using an 141 omnidirectional camera or a collection of cameras. 360o video is 142 outside the scope of this document. 144 XR: extended reality is used to address both AR and VR together. 146 3. Extended Reality and In-Network Computing 148 XR is an example of the Multisource Multidestination Problem that 149 combines video, haptics and tactile experiences as well, in 150 interactive or networked mode multiparty and social interactions. 151 Thus, XR is difficult to deliver to deliver with a client-server 152 strictly cloud-based solution as it requires a combination of stream 153 synchronization, lows delay and delay variations as mentioned above 154 as well as means to cover from losses and provide optimized caching 155 in the cloud and rendering as close as possible to the user at the 156 network edge. 158 3.1. XR Network Requirements 160 In order to deliver the XR experience, there is a need to achieve 161 complete 6 degrees of freedom meaning the 3 axes for body movement 162 (x,y,z) plus pitch, yaw, rotation of the head all of which must be 163 fulfilled in real time again focusing on the low delay, low loss and 164 low delay variation to avoid sea sickness symptoms if the image does 165 not follow the movement [CABLE]. But this is not the only 166 difficulty, as there is also the need to provide real-time 167 interactivity for immersive sports, mobile immersive applications 168 with tactile and time-sensitive data and high bandwidth for high 169 resolution images. Since XR deals with personal information and 170 potentially protected content (in entertainment and gaming) XR must 171 also provide a secure environment and ensure user privacy. And of 172 course, the sheer amount data needed for and generated by the XR 173 applications will use recent trend analysis and mechanisms, including 174 machine learning to find these trends and reduce the size of the data 175 sets. 177 Shared and global immersive experiences require interconnected, 178 distributed and federated XR nodes. The requirements can be 179 summarized as: 181 - Allow joint collaboration in VR. 183 - Provide multi-view AR. 185 - Add extra streams (IoT) to AR and VR experiences across 186 services. 188 - Provide "Social Television" experiences and global viewing and 189 experience rooms. 191 - Enable multistream, multidevice, multidestination applications. 193 - Use new Internet Architectures at the edge for improved 194 performance. 196 - Integrate with holography, 3D displays and image processing 197 systems [CABLE]. 199 3.2. In-Network Computing Advantages in XR 201 One aspect of the push of XR to the edge is of course to provide 202 cloud-based services with much lower latency. While this is very 203 promising the question of the localization of the networking 204 resources in order to provide the service becomes an essential 205 component of the overall architecture. But it is not only finding 206 the best geographical location but also providing the right level of 207 reliability when one or more location is not available especially for 208 mission critical services in medicine or manufacturing. And it does 209 not mean only data laid distribution but also ensuring the 210 availability of the right computational capabilities. The 211 optimization of the location and type of the required resources for 212 the multisouce, multidestination, mutiparty, multi-input XR 213 applications can use AI and ML, and advanced load balancing and 214 distributed network principles. There is a need for more research in 215 such resource allocation problems at the edge to enable autonomous 216 node operation and quality of experience [SOL]. These are of course 217 multi-variate and heterogeneous goal optimization problems requiring 218 advanced analysis with fast converging algorithms [MULTI][PACKET]. 219 This is essential for the federation of nodes to provide the required 220 experience. 222 Of course, image rendering and video processing in XR leverages 223 different HW capabilities combinations of CPU and GPU. Current 224 programmable network entities need to be evaluated to see if they can 225 be sufficient to provide the speed required to provide real-time 226 rendering and execute complex analytics: P4 for example does not 227 support the floating-point operations necessary for advanced 228 graphics. 230 Finally, dynamic network programmability could enable the use of 231 joint learning algorithms across both data center, edge computers and 232 goggle or glasses to allocate functionality and the creation of semi 233 permanent datasets and analytics for usage trending. In the end, the 234 use of computing or networked XR will enable the allocation of 235 control, forwarding and storage resources and related usage models 236 when needed by the application. This may mean re-evaluating the 237 distribution of functionalities between datacenter and edge with less 238 critical elements rendered in the cloud combined with a better 239 understanding of the operational decomposition of the XR experience 240 to allow the use of novel data structures, three-dimensional modeling 241 and image processing algorithms. 243 Other advantages of adding computing to networked XR include: 245 - Multicast distribution and processing as well as peer to peer 246 distribution in bandwidth constrained environments. 248 - Evaluation of local caching and micro datacenters with local or 249 cloud-based pre-rendering. 251 - Trend or ML based congestion control to manage XR sessions 252 quality of service. 254 - Higher layer protocols optimization to reduce latency. 256 - Trust, including blockchains and smart-contracts to enable 257 secure community building across domains. 259 - Support for nomadicity and mobility (link to mobile edge). 261 - Use of 5G slicing to create independent session-driven 262 processing/rendering. 264 - Performance optimization by tunneling, session virtualization 265 and loss protection. 267 4. Enabling Technologies 269 This section presents some salient research that will lead to in- 270 network computing becoming a major enabler of networked XR. 272 NOTE: more information and added sub-sections will be added in future 273 versions of the draft with the collaboration of co-authors in the 274 specific research areas. 276 4.1. Information Centric Networking (ICN) and Named Data Networking 277 (NDN) 279 The Named Data Networking (NDN) architecture, one architecture of 280 ICN, is particularly well suited for the multisource multi- 281 destination architecture of XR because it allows to create the 282 content experiences based on their components names not a location or 283 pointer to a location hence provides a natural functional 284 decomposition. ICN allows content delivery to evolve from single, 285 context-independent streams to context-dependent Information 286 components that can adapt dynamically to the changes necessary to 287 maintain the immersive nature of the experience and be delivered 288 efficiently. The combination of interest messages to signal what 289 content is needed combined with the data responses help to coordinate 290 the different streams and multiple users (pull mechanisms). The ICE- 291 AR [ICE] project already mentions a concept of acceleration as a 292 service: the exploration of the design and the usage of computation 293 at the edge including the wireless edge. 295 For XR, ICN also allows to develop robust and resilient networking 296 while allowing application developer to continue using known 297 programming model [RICE]. This is important for the XR developers 298 community that come from the entertainment, gaming or other non 299 network specific industries and could enable ICN and XR to coexist in 300 user devices (the ultimate edge). NDN concepts are already 301 integrated to distributed video distribution with trust mechanisms 302 (see section below) such as smart contracts on the blockchain to 303 proof of origin and destination sent along with interest messages 304 [HUITX]. 306 4.2. Network Coding 308 Networked XR requires the synchronization of multiple streams but 309 with its delay sensitivity the use of buffering schemes to achieve 310 this synchronization is impractical. At the same time the need to 311 maintain high image quality means that packet losses also need to be 312 limited. Network coding has proven very useful to achieve both these 313 goals in commercial streaming services like Netflix, is being added 314 to protocols like QUIC and in another multi-stream service namely 315 Social Television [SOCIAL] avoiding the reliance on complex 316 synchronization algorithms. The main difference between XR and 317 Social Television is that the former is even more constrained in 318 latency and loss budgets hence even the delay due to encoding and 319 decoding operations needs to be minimized. Hence the idea of in- 320 network coding and re-encoding to adapt to dynamic network 321 conditions, not just end to end, can be used to ensure on time packet 322 delivery with loss recovery. In network encoding needs the type of 323 programmability that COIN provides. 325 4.3. Blockchains and Distributed Trust 327 If XR is to be integrated at the edge of the network to provide the 328 required delay and loss guarantees, then relying on centralized 329 mechanisms for trust is non-realistic. Traditional centralized 330 mechanisms to discover and admit nodes to the network, to provide 331 access right and name resolution need to be updated to be used in the 332 dynamic XR environment. Blockchain technology, with operation 333 performed at the edge and in a decentralized way is fast becoming a 334 major scalable means of providing trust and validate provenance in a 335 large number of applications including those on the XR portfolio. 336 Smart contracts (on the blockchain) supply a mechanism to provide the 337 trust and validation for XR edge nodes. 339 A new XR participant node is admitted after it has committed to a 340 smart contract that contains the rules and mechanisms to distribute 341 content via this node in a trusted and secure way. This constitutes 342 its proof of validity. After a node is admitted, it will can then 343 provisioned with the appropriate software to become fully operational 344 to provide the XR experience. Newly admitted nodes will be inserted 345 in the general ledger on the blockchain enabling other nodes to 346 discover them, and hence, to form a trusted network. A name 347 resolution authority can also be provided by the blockchain to manage 348 and validate the origin of the content, the proof of origin, and to 349 provide the ability to search such content. The proof of origin can 350 also be used to prevent some content from reaching one or more nodes 351 and implement content filtering based on trusted authorities. This 352 is useful not only for content packets but also for packets capable 353 of modifying the node operations. Finally, when some content reaches 354 a specific destination, it can be verified against the content rules 355 of the reached node even and before it is sent to the application; 356 this allows to provide a proof of delivery for the content and enable 357 to generate statistics, performance metrics and enable the nodes to 358 adapt to the XR requirements. 360 All of the above assumes that the nodes can implement the functions 361 needed by the blockchain hence once again infers that there is enough 362 computing power in the nodes to perform these operations. At this 363 point both proof of concept and proof of every are limited due to the 364 added overhead and the size of the blockchain. As distributed 365 blockchain and COIN continue to evolve this should continue to be a 366 field of interest for the development of secure and private XR 367 experiences. 369 5. Conclusion 371 More and more applications and service are being developed and 372 deployed that use or will use combinations of AR and VR, XR, along 373 with extra stream from sensors and IoT devices. And many of these 374 applications require to be deployed over a network because of their 375 interactive or multiparty nature. In that context, it not uniquely 376 necessary to move functionality to the network but to carefully 377 evaluate which elements to locate in network nodes, where these nodes 378 are and what computational support they need to support the XR 379 experience. Hence, it is believed that a great enabler of networked 380 XR is the capability to co-locate programmable elements in the XR 381 network node to respond to the dynamics of the services in an 382 efficient, resilient and secure manner. 384 6. Acknowledgements 386 The author would like to thank Jeffrey He, Dirk Kutscher, Cedric 387 Westphal and Weiguang Wang for their contributions to the 388 presentation that lead to this draft. 390 7. References 392 7.1. Normative References 394 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 395 Requirement Levels", BCP 14, RFC 2119, 396 DOI 10.17487/RFC2119, March 1997, 397 . 399 7.2. Informative References 401 [CABLE] Hinds, A., "The Near Future of Immersive Experiences: 402 Where We Are on the Journey, What Lies Ahead, and What It 403 Takes to Get There", SIGCOMM 2018 Workshop on AR/VR 404 http://conferences.sigcomm.org/sigcomm/2018/workshop- 405 arvr.html, August 2018. 407 [HUITX] "8X: ICN Based Video Distribution", 2018, 408 . 410 [ICE] Burke., J., "ICN-Enabled Secure Edge Networking with 411 Augmented Reality: ICE-AR", ICE-AR Presentation at NDNCOM 412 September 2018 https://www.nist.gov/news- 413 events/events/2018/09/named-data-networking-community- 414 meeting-2018, 2018, . 416 [INTER] Bastug et al., E., "Towards Interconnected Virtual 417 Reality:Opportunities, Challenges and Enablers", IEEE 418 Communications Magazine, Volume 55 , Issue: 6 , 419 2017 https://arxiv.org/pdf/1611.05356.pdf, June 2017. 421 [MEDIA] "Mediated Reality", Wikipedia.org 422 https://en.wikipedia.org/wiki/Computer-mediated_reality, 423 2018. 425 [MULTI] Batalla, J., "Evolutionary Multiobjective optimization 426 algorithm for multimedia delivery in critical applications 427 through Content-Aware Networks", The Journal of 428 Supercomputing, Volume 73, Issue 3, pp. 993-1016 429 https://link.springer.com/article/10.1007/ 430 s11227-016-1731-x, March 2017. 432 [PACKET] Jeyakumar et al., V., "Millions of Little Minions: Using 433 Packets for Low Latency Network Programming and 434 Visibility", Proceedings of SIGCOMM 2014 435 http://conferences.sigcomm.org/sigcomm/2014/program.php, 436 August 2018. 438 [RICE] Krol et al., M., "RICE: Remote Method Invocation in ICN", 439 Proceedings of the ACM Conference on Information-Centric 440 Networking 2018 http://conferences.sigcomm.org/acm- 441 icn/2018/proceedings/icn18-final9.pdf, September 2018. 443 [SOCIAL] Montpetit, M. and M. Medard, "Social Television: Enabling 444 Technologies and Architectures", Proceedings of the IEEE, 445 Volume 100, pp. 446 1395-1399 http://proceedingsoftheieee.ieee.org, May 2012. 448 [SOL] Heorhiadi et al., V., "Simplifying Software-Defined 449 Network Optimization Using SOL", 13th USENIX Symposium on 450 Networked Systems Design and Implementation 451 https://www.usenix.org/system/files/conference/nsdi16/ 452 nsdi16-paper-heorhiadi.pdf, March 2016. 454 [VRSICK] LaViola, J., "A Discussion of Cybersickness in Virtual 455 Environments", ACM SIGCHI Bulletin 32(1):47-56 456 http://www.eecs.ucf.edu/~jjl/pubs/cybersick.pdf, January 457 2000. 459 Author's Address 460 Marie-Jose Montpetit 461 Triangle Video 462 Boston, MA 463 US 465 Email: marie@mjmontpetit.com