idnits 2.17.1 draft-montpetit-coin-xr-03.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack a Security Considerations section. ** The document seems to lack an IANA Considerations section. (See Section 2.2 of https://www.ietf.org/id-info/checklist for how to handle the case when there are no actions for IANA.) Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document doesn't use any RFC 2119 keywords, yet seems to have RFC 2119 boilerplate text. -- The document date (July 8, 2019) is 1753 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Unused Reference: 'VRSICK' is defined on line 468, but no explicit reference was found in the text Summary: 2 errors (**), 0 flaws (~~), 3 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 COIN M. Montpetit 3 Internet-Draft Triangle Video 4 Intended status: Informational July 8, 2019 5 Expires: January 9, 2020 7 In Network Computing Enablers for Extended Reality 8 draft-montpetit-coin-xr-03 10 Abstract 12 Augmented Reality (AR) and Virtual Reality (VR), combined as Extended 13 Reality or XR, challenge networking technologies and protocols 14 because they combine the features of fast information display, image 15 processing, computing and forwarding. This document presents some of 16 these challenges and how adding computing in the network could 17 respond to them. 19 Status of This Memo 21 This Internet-Draft is submitted in full conformance with the 22 provisions of BCP 78 and BCP 79. 24 Internet-Drafts are working documents of the Internet Engineering 25 Task Force (IETF). Note that other groups may also distribute 26 working documents as Internet-Drafts. The list of current Internet- 27 Drafts is at https://datatracker.ietf.org/drafts/current/. 29 Internet-Drafts are draft documents valid for a maximum of six months 30 and may be updated, replaced, or obsoleted by other documents at any 31 time. It is inappropriate to use Internet-Drafts as reference 32 material or to cite them other than as "work in progress." 34 This Internet-Draft will expire on January 9, 2020. 36 Copyright Notice 38 Copyright (c) 2019 IETF Trust and the persons identified as the 39 document authors. All rights reserved. 41 This document is subject to BCP 78 and the IETF Trust's Legal 42 Provisions Relating to IETF Documents 43 (https://trustee.ietf.org/license-info) in effect on the date of 44 publication of this document. Please review these documents 45 carefully, as they describe your rights and restrictions with respect 46 to this document. Code Components extracted from this document must 47 include Simplified BSD License text as described in Section 4.e of 48 the Trust Legal Provisions and are provided without warranty as 49 described in the Simplified BSD License. 51 Table of Contents 53 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 54 1.1. Requirements Language . . . . . . . . . . . . . . . . . . 3 55 2. Definitions . . . . . . . . . . . . . . . . . . . . . . . . . 3 56 3. Extended Reality and In-Network Computing . . . . . . . . . . 4 57 3.1. XR Network Requirements . . . . . . . . . . . . . . . . . 4 58 3.2. In-Network Computing Advantages in XR . . . . . . . . . . 5 59 4. XR in Data Intensive Services and Applications . . . . . . . 6 60 5. Enabling Technologies . . . . . . . . . . . . . . . . . . . . 6 61 5.1. Information Centric Networking (ICN) and Named Data 62 Networking (NDN) . . . . . . . . . . . . . . . . . . . . 7 63 5.2. Network Coding . . . . . . . . . . . . . . . . . . . . . 7 64 5.3. Blockchains and Distributed Trust . . . . . . . . . . . . 8 65 6. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . 9 66 7. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 9 67 8. References . . . . . . . . . . . . . . . . . . . . . . . . . 9 68 8.1. Normative References . . . . . . . . . . . . . . . . . . 9 69 8.2. Informative References . . . . . . . . . . . . . . . . . 9 70 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 10 72 1. Introduction 74 Virtual Reality (VR) and Augmented Reality (AR) taken together as 75 Extended Reality or XR are at the center of a number of technological 76 advances in many different fields, including not only gaming and 77 entertainment but immersive journalism, remote diagnosis and 78 maintenance, telemedicine, manufacturing and assembly and smart 79 cities. 81 But with the emergence of the edge and the programmability of network 82 elements all the way from the data center to the users the 83 possibility of creating networked, multiparty/multisource and 84 interacting XR comes closer to reality. This document wants to 85 review what is necessary for the current localized and cloud enhanced 86 XR to evolve to a more distributed and edge centric architecture to 87 support advanced immersive application and services. It assumes that 88 network programmability will enable to tailor the network to the XR 89 requirements. This document is about requirements not solutions per 90 se but will mention work that has already been done towards a more 91 networked XR including Information Centric architectures, Artificial 92 Intelligence and in network coding. The networked functionality 93 should enable to supplement local XR services and devices while 94 keeping the very low latency and the very high data rates that are 95 required by XR. 97 This document is intended as informative to both the networking and 98 application research community. It does not address a specific 99 network layer or protocol but provides architecture and system level 100 specifications and guidelines. For example: 102 Latency: the physical distance between the XR content cloud of AR/ 103 VR and users are short enough to limit the propagation delay to 104 the 20 ms usually cited for XR applications mixed for example with 105 IoT devices and sensors delay reduction for range of interest 106 (RoI) detection. 108 Applications: better transcoding and use of compression 109 algorithms, pre-fetching and pre-caching and movement prediction. 111 Monitoring: implement management and metrics gathering closer to 112 where the XR services are consumed. 114 Network access: push some networking functions in the kernel space 115 into the user space to enable the deployment of stream specific 116 algorithms for congestion control and application-based load 117 balancing based on machine learning and user data patterns. 119 1.1. Requirements Language 121 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 122 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 123 document are to be interpreted as described in RFC 2119 [RFC2119]. 125 2. Definitions 127 - 360-degree video: 360-degree videos, also known as immersive 128 videos or spherical videos, are video recordings where a view in 129 every direction is recorded at the same time using an 130 omnidirectional camera or a collection of cameras. 360o video is 131 outside the scope of this document. 133 - AR: Augmented Reality (AR) is a live direct or indirect view of 134 a physical, real-world environment whose elements are augmented by 135 computer-generated input such as sound, video, location or 136 graphical data. It is related to a more general concept called 137 mediated reality [MEDIA], in which reality is modified (diminished 138 or augmented) by computer-generated imagery. 140 - VR Virtual Reality (VR): uses software-generated realistic 141 imaging, sounds and other sensor inputs to replicate a real or 142 imaginary setting, to simulates a user's physical presence in this 143 environment and provide an immersive experience that enable the 144 user to interact with objects and move within this space. 146 - XR: extended reality is used to address both AR and VR together. 148 3. Extended Reality and In-Network Computing 150 XR is an example of the Multisource Multidestination Problem that 151 combines video, haptics and tactile experiences as well, in 152 interactive or networked mode multiparty and social interactions. 153 Thus, XR is difficult to deliver with a client-server cloud-based 154 solution as it requires a combination of stream synchronization, lows 155 delay and delay variations as mentioned above as well as means to 156 cover from losses and provide optimized caching in the cloud and 157 rendering as close as possible to the user at the network edge. 159 3.1. XR Network Requirements 161 To deliver the XR experience, there is a need to achieve complete 6 162 degrees of freedom meaning the 3 axes for body movement (x,y,z) plus 163 pitch, yaw, rotation of the head all of which must be fulfilled in 164 real time. Low delay, low loss and low delay variation are needed to 165 avoid sea sickness symptoms if the image does not follow the movement 166 [CABLE]. 168 But this is not the only difficulty, as there is also the need to 169 provide real-time interactivity for immersive sports, mobile 170 immersive applications with tactile and time-sensitive data and high 171 bandwidth for high resolution images. Since XR deals with personal 172 information and potentially protected content (in entertainment and 173 gaming) XR must also provide a secure environment and ensure user 174 privacy. And of course, the sheer amount data needed for and 175 generated by the XR applications will use recent trend analysis and 176 mechanisms, including machine learning to find these trends and 177 reduce the size of the data sets [INTER]. 179 Shared and global immersive experiences require interconnected, 180 distributed and federated XR nodes. The requirements can be 181 summarized as: 183 - Allow joint collaboration in VR. 185 - Provide multi-view AR. 187 - Add extra streams (IoT) to AR and VR experiences across data 188 intensive services, manufacturing and industrial processes. 190 - Provide "Social Television" experiences and global viewing and 191 experience rooms. 193 - Enable multistream, multidevice, multidestination applications. 195 - Use new Internet Architectures at the edge for improved 196 performance and perfromance management. 198 - Integrate with holography, 3D displays and image processing 199 systems [CABLE]. 201 3.2. In-Network Computing Advantages in XR 203 One aspect of the push of XR to the edge is of course to take 204 advantage of the cloud/edge continuum and functional decomposition to 205 provide lower latency services and enable the development of new 206 applications . While this is very promising the question of the 207 localization of the networking resources in order to provide the 208 service becomes an essential component of the overall architecture. 209 But it is not only finding the best geographical location but also 210 providing the right level of reliability when one or more location is 211 not available especially for mission critical services in medicine or 212 manufacturing. And it does not mean only data laid distribution but 213 also ensuring the availability of the right computational 214 capabilities. The optimization of the location and type of the 215 required resources for the multisouce, multidestination, mutiparty, 216 multi-input XR applications can use AI and ML, and advanced load 217 balancing and distributed network principles. There is a need for 218 more research in such resource allocation problems at the edge to 219 enable autonomous node operation and quality of experience 220 [SOL].These are of course multi-variate and heterogeneous goal 221 optimization problems requiring advanced analysis with fast 222 converging algorithms [MULTI][PACKET]. This is essential for the 223 federation of nodes to provide the required experience. 225 Of course, image rendering and video processing in XR leverages 226 different HW capabilities combinations of CPU and GPU. Current 227 programmable network entities need to be evaluated to see if they can 228 be sufficient to provide the speed required to provide real-time 229 rendering and execute complex analytics: P4 for example does not 230 support the floating-point operations necessary for advanced 231 graphics. 233 Finally, dynamic network programmability could enable the use of 234 joint learning algorithms across both data center, edge computers and 235 goggles or glasses to allocate functionality and the creation of semi 236 permanent datasets and analytics for usage trending. In the end, the 237 use of computing or networked XR will enable the allocation of 238 control, forwarding and storage resources and related usage models 239 when needed by the application. This may mean re-evaluating the 240 distribution of functionalities between datacenter and edge with less 241 critical elements rendered in the cloud combined with a better 242 understanding of the operational decomposition of the XR experience 243 to allow the use of novel data structures, three-dimensional modeling 244 and image processing algorithms. 246 Other advantages of adding computing to networked XR include: 248 - Multicast distribution and processing as well as peer to peer 249 distribution in bandwidth and capacity constrained environments. 251 - Evaluation of local caching and micro datacenters with local or 252 cloud-based pre-rendering. 254 - Trend or ML based congestion control to manage XR sessions 255 quality of service. 257 - Higher layer protocols optimization to reduce latency especially 258 in data intensive applications at the edge. 260 - Trust, including blockchains and smart-contracts to enable 261 secure community building across domains. 263 - Support for nomadicity and mobility (link to mobile edge). 265 - Use of 5G slicing to create independent session-driven 266 processing/rendering. 268 - Performance optimization by tunneling, session virtualization 269 and loss protection. 271 4. XR in Data Intensive Services and Applications 273 In-network computing is essential for data reduction and multistream 274 low latency services at the edge where moving the data to the cloud 275 is either requiring too much bandwidth or adding unacceptable 276 latency. Examples of these services included industrial processes 277 monitoring, AR-aided design and fabrication and AR/VR supported 278 medical interventions. 280 5. Enabling Technologies 282 This section presents some salient research that will lead to in- 283 network computing becoming a major enabler of networked XR. 285 NOTE: more information and added sub-sections will be added in future 286 versions of the draft with the collaboration of co-authors in the 287 specific research areas. 289 5.1. Information Centric Networking (ICN) and Named Data Networking 290 (NDN) 292 The Named Data Networking (NDN) architecture, one architecture of 293 ICN, is particularly well suited for the multisource multi- 294 destination architecture of XR because it allows to create the 295 content experiences based on their components names not a location or 296 pointer to a location hence provides a natural functional 297 decomposition. ICN allows content delivery to evolve from single, 298 context-independent streams to context-dependent Information 299 components that can adapt dynamically to the changes necessary to 300 maintain the immersive nature of the experience and be delivered 301 efficiently. The combination of interest messages to signal what 302 content is needed combined with the data responses help to coordinate 303 the different streams and multiple users (pull mechanisms). The ICE- 304 AR [ICE] project already mentions a concept of acceleration as a 305 service: the exploration of the design and the usage of computation 306 at the edge including the wireless edge. 308 For XR, ICN also allows to develop robust and resilient networking 309 while allowing application developer to continue using known 310 programming model [RICE]. This is important for the XR developers 311 community that come from the entertainment, gaming or other non 312 networking specific industries and could enable ICN and XR to coexist 313 in user devices (the ultimate edge). NDN concepts are already 314 integrated to distributed video distribution with trust mechanisms 315 (see section below) such as smart contracts on the blockchain to 316 proof of origin and destination sent along with interest messages 317 [HUITX]. 319 5.2. Network Coding 321 Networked XR requires the synchronization of multiple streams but 322 with its delay sensitivity the use of buffering schemes to achieve 323 this synchronization is impractical. At the same time the need to 324 maintain high data integrity means that packet losses also need to be 325 limited. Network coding has proven very useful to achieve both these 326 goals in commercial streaming services like Netflix, is being added 327 to protocols like QUIC, multi-stream services such as Social 328 Television [SOCIAL] and other data-centric low latency applications. 329 This avoids being reliant on complex synchronization algorithms. The 330 many XR servcies are constrained in latency and loss budgets 331 especially in mission critical applications hence even the delay due 332 to encoding and decoding operations needs to be minimized. Hence the 333 idea of in-network coding and re-encoding to adapt to dynamic network 334 conditions, not just end to end, can be used to ensure on time packet 335 delivery with loss recovery. In network encoding needs the type of 336 programmability that COIN provides and the developement of 337 programmable switches and the P4 language may allow to create very 338 efficient in-network coding even considering the limitations of the 339 language Note: references to be added. 341 5.3. Blockchains and Distributed Trust 343 If XR is to be integrated at the edge of the network to provide the 344 required delay and loss guarantees, then relying on centralized 345 mechanisms for trust is non-realistic. Traditional centralized 346 mechanisms to discover and admit nodes to the network, to provide 347 access right and name resolution need to be updated to be used in the 348 dynamic XR environment.Blockchain technology, with operations 349 performed at the edge and in a decentralized way is fast becoming a 350 major scalable means of providing trust and validate provenance in a 351 large number of applications including those on the XR portfolio. 353 Smart contracts (on the blockchain) supply a mechanism to provide the 354 trust and validation for XR edge nodes. A new XR participant node is 355 admitted after it has committed to a smart contract that contains the 356 rules and mechanisms to distribute content via this node in a trusted 357 and secure way. This constitutes its proof of validity. After a 358 node is admitted, it will can then provisioned with the appropriate 359 software to become fully operational to provide the XR experience. 360 Newly admitted nodes will be inserted in the general ledger on the 361 blockchain enabling other nodes to discover them, and hence, to form 362 a trusted network. A name resolution authority can also be provided 363 by the blockchain to manage and validate the origin of the content, 364 the proof of origin, and to provide the ability to search such 365 content. The proof of origin can also be used to prevent some 366 content from reaching one or more nodes and implement content 367 filtering based on trusted authorities. This is useful not only for 368 content packets but also for packets capable of modifying the node 369 operations. Finally, when some content reaches a specific 370 destination, it can be verified against the content rules of the 371 reached node even and before it is sent to the application; this 372 allows to provide a proof of delivery for the content and enable to 373 generate statistics, performance metrics and enable the nodes to 374 adapt to the XR requirements. 376 All of the above assumes that the nodes can implement the functions 377 needed by the blockchain hence once again infers that there is enough 378 computing power in the nodes to perform these operations. At this 379 point both proof of concept and proof of every are limited due to the 380 added overhead and the size of the blockchain. As distributed 381 blockchains and COIN continue to evolve this should continue to be a 382 field of interest for the development of secure and private XR 383 experiences. 385 6. Conclusion 387 More and more applications and service are being developed and 388 deployed that use or will use combinations of AR and VR, XR, along 389 with extra stream from sensors and IoT devices. And many of these 390 applications require to be deployed over a network because of their 391 interactive or multiparty nature. In that context, it not uniquely 392 necessary to move functionality to the network but to carefully 393 evaluate which elements to locate in network nodes, where these nodes 394 are and what computational support they need to support the XR 395 experience. Hence, it is believed that a great enabler of networked 396 XR is the capability to co-locate programmable elements in the XR 397 network node to respond to the dynamics of the services in an 398 efficient, resilient and secure manner. 400 7. Acknowledgements 402 The author would like to thank Jeffrey He, Dirk Kutscher, Cedric 403 Westphal and Weiguang Wang for their contributions to the 404 presentation that lead to this draft. 406 8. References 408 8.1. Normative References 410 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 411 Requirement Levels", BCP 14, RFC 2119, 412 DOI 10.17487/RFC2119, March 1997, 413 . 415 8.2. Informative References 417 [CABLE] Hinds, A., "The Near Future of Immersive Experiences: 418 Where We Are on the Journey, What Lies Ahead, and What It 419 Takes to Get There.", SIGCOMM 2018 Workshop on AR/VR. 420 http://conferences.sigcomm.org/sigcomm/2018/workshop- 421 arvr.html, August 2018. 423 [HUITX] "8X: ICN Based Video Distribution.", 8XLab Web 424 Site. https://www.8xlabs.com, 2018. 426 [ICE] Burke, J., "ICN-Enabled Secure Edge Networking with 427 Augmented Reality: ICE-AR.", ICE-AR Presentation at 428 NDNCOM. https://www.nist.gov/news- 429 events/events/2018/09/named-data-networking-community- 430 meeting-2018, September 2018. 432 [INTER] Bastug, E., "Towards Interconnected Virtual 433 Reality:Opportunities, Challenges and Enablers.", IEEE 434 Communications Magazine, Volume 55 , Issue: 435 6. https://arxiv.org/pdf/1611.05356.pdf, June 2017. 437 [MEDIA] Wikipedia.org, "Mediated Reality.", 2018, 438 . 440 [MULTI] Batalla, J., "Evolutionary Multi-objective optimization 441 algorithm for multimedia delivery in critical applications 442 through Content-Aware Networks.", The Journal of 443 Supercomputing. https://link.springer.com/article/10.1007/ 444 s11227-016-1731-x, March 2017. 446 [PACKET] Jeyakumar, V., "Millions of Little Minions: Using Packets 447 for Low Latency Network Programming and Visibility.", 448 Proceedings of SIGCOMM 2014. 449 http://conferences.sigcomm.org/sigcomm/2014/program.php, 450 August 2014. 452 [RICE] Krol, M., "RICE: Remote Method Invocation in ICN.", 453 Proceedings of the ACM Conference on Information-Centric 454 Networking 2018. http://conferences.sigcomm.org/acm- 455 icn/2018/proceedings/icn18-final9.pdf, September 2018. 457 [SOCIAL] Montpetit, M. and M. Medard, "Social Television: Enabling 458 Technologies and Architectures.", Proceedings of the IEEE, 459 Volume 100, pp. 460 1395-1399. http://proceedingsoftheieee.ieee.org, May 2012. 462 [SOL] Heorhiadi, V., "Simplifying Software-Defined Network 463 Optimization Using SOL.", 13th USENIX Symposium on 464 Networked Systems Design and Implementation. 465 https://www.usenix.org/system/files/conference/nsdi16/ 466 nsdi16-paper-heorhiadi.pdf, March 2016. 468 [VRSICK] LaViola, J., "SA Discussion of Cybersickness in Virtual 469 Environments.", ACM SIGCHI Bulletin 32(1):47-56. 470 http://www.eecs.ucf.edu/~jjl/pubs/cybersick.pdf, January 471 2000. 473 Author's Address 475 Marie-Jose 476 Triangle Video 478 Email: marie@mjmontpetit.com