idnits 2.17.1 draft-bryant-arch-fwd-layer-ps-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (March 09, 2020) is 1507 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Outdated reference: A later version (-31) exists of draft-bonica-6man-comp-rtg-hdr-13 == Outdated reference: A later version (-16) exists of draft-filsfils-spring-net-pgm-extension-srv6-usid-04 == Outdated reference: A later version (-03) exists of draft-herbert-6man-eh-attrib-00 == Outdated reference: A later version (-07) exists of draft-ietf-detnet-ip-05 == Outdated reference: A later version (-13) exists of draft-ietf-detnet-mpls-05 == Outdated reference: A later version (-16) exists of draft-ietf-detnet-security-08 == Outdated reference: A later version (-17) exists of draft-ietf-ippm-ioam-data-08 -- Obsolete informational reference (is this intentional?): RFC 2460 (Obsoleted by RFC 8200) Summary: 0 errors (**), 0 flaws (~~), 8 warnings (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 INTAREA Working Group S. Bryant 3 Internet-Draft U. Chunduri 4 Intended status: Informational T. Eckert 5 Expires: September 10, 2020 A. Clemm 6 Futurewei Technologies Inc. 7 March 09, 2020 9 Forwarding Layer Problem Statement 10 draft-bryant-arch-fwd-layer-ps-00 12 Abstract 14 This document considers the new use cases for IP together with the 15 network capabilities and services that will be needed to address 16 those use cases. It then looks at the underlying packet requirements 17 and considers the changing deployment models and the issues with 18 existing packet designs that need to be addressed. It concludes by 19 looking at some parameters of a solution. 21 Status of This Memo 23 This Internet-Draft is submitted in full conformance with the 24 provisions of BCP 78 and BCP 79. 26 Internet-Drafts are working documents of the Internet Engineering 27 Task Force (IETF). Note that other groups may also distribute 28 working documents as Internet-Drafts. The list of current Internet- 29 Drafts is at https://datatracker.ietf.org/drafts/current/. 31 Internet-Drafts are draft documents valid for a maximum of six months 32 and may be updated, replaced, or obsoleted by other documents at any 33 time. It is inappropriate to use Internet-Drafts as reference 34 material or to cite them other than as "work in progress." 36 This Internet-Draft will expire on September 10, 2020. 38 Copyright Notice 40 Copyright (c) 2020 IETF Trust and the persons identified as the 41 document authors. All rights reserved. 43 This document is subject to BCP 78 and the IETF Trust's Legal 44 Provisions Relating to IETF Documents 45 (https://trustee.ietf.org/license-info) in effect on the date of 46 publication of this document. Please review these documents 47 carefully, as they describe your rights and restrictions with respect 48 to this document. Code Components extracted from this document must 49 include Simplified BSD License text as described in Section 4.e of 50 the Trust Legal Provisions and are provided without warranty as 51 described in the Simplified BSD License. 53 Table of Contents 55 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 56 1.1. Forwarding Layer . . . . . . . . . . . . . . . . . . . . 4 57 2. New Use Cases for packet networks . . . . . . . . . . . . . . 5 58 2.1. Video and AR/VR . . . . . . . . . . . . . . . . . . . . . 5 59 2.2. Role of Fixed Networks in 5G and Beyond 5G . . . . . . . 6 60 2.3. ITU-T Focus Group Network-2030 . . . . . . . . . . . . . 6 61 3. New Network Capabilities and Services . . . . . . . . . . . . 8 62 3.1. New Services . . . . . . . . . . . . . . . . . . . . . . 8 63 3.2. New Capabilities . . . . . . . . . . . . . . . . . . . . 9 64 4. Underlying New Requirements . . . . . . . . . . . . . . . . . 10 65 4.1. Better than Best Effort . . . . . . . . . . . . . . . . . 10 66 4.2. Efficient Packet Design . . . . . . . . . . . . . . . . . 10 67 4.3. Forwarding Identifiers . . . . . . . . . . . . . . . . . 11 68 4.4. Operational visibility . . . . . . . . . . . . . . . . . 12 69 4.5. Holistic solution . . . . . . . . . . . . . . . . . . . . 12 70 5. Deployment Models . . . . . . . . . . . . . . . . . . . . . . 13 71 5.1. Edge-2-Edge Model . . . . . . . . . . . . . . . . . . . . 13 72 5.2. End-2-End Model Single Provider . . . . . . . . . . . . . 13 73 5.3. End-2-End Model with multiple Providers . . . . . . . . . 14 74 5.4. Embedded Service . . . . . . . . . . . . . . . . . . . . 15 75 5.5. Embedded Global Service . . . . . . . . . . . . . . . . . 16 76 6. Existing Protocol and Layering Challenges and Gaps . . . . . 17 77 6.1. Challenges with IPv6 . . . . . . . . . . . . . . . . . . 17 78 6.1.1. The End-to-End Model . . . . . . . . . . . . . . . . 17 79 6.1.2. Fixed Address Length . . . . . . . . . . . . . . . . 21 80 6.2. Better Than Best Effort E2E Network Services . . . . . . 22 81 6.3. Adaptive Bit-rate Video streaming . . . . . . . . . . . . 23 82 6.4. DetNet and Higher Precision Networking Service . . . . . 24 83 6.5. Forwarding Plane vs. Control Plane . . . . . . . . . . . 24 84 6.6. User-Network/Network-User Interface Signaling . . . . . . 26 85 7. Candidate Solution Directions . . . . . . . . . . . . . . . . 26 86 7.1. Variable Length Addresses . . . . . . . . . . . . . . . . 27 87 7.2. Address Semantics . . . . . . . . . . . . . . . . . . . . 27 88 7.3. Multiple Instructions . . . . . . . . . . . . . . . . . . 28 89 7.4. Node and Path Specific Processing Instructions . . . . . 28 90 7.5. Integrated Assurance and Verification . . . . . . . . . . 28 91 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 29 92 9. Security Considerations . . . . . . . . . . . . . . . . . . . 29 93 10. Appendix 1: Expanded Summary of Sub-G1 Use Cases . . . . . . 29 94 10.1. Holographic-type communications . . . . . . . . . . . . 29 95 10.2. Tactile Internet for Remote Operations . . . . . . . . . 30 96 10.3. Space-Terrestrial Integrated Networks . . . . . . . . . 30 97 10.4. ManyNets . . . . . . . . . . . . . . . . . . . . . . . . 31 98 11. Appendix 2: Expanded Summary of Sub-G2 New Network 99 Capabilities and Services . . . . . . . . . . . . . . . . . . 32 100 11.1. New Services . . . . . . . . . . . . . . . . . . . . . . 32 101 11.1.1. High-Precision Communications Services . . . . . . . 32 102 11.1.2. In-time Services . . . . . . . . . . . . . . . . . . 33 103 11.1.3. On-time Services . . . . . . . . . . . . . . . . . . 33 104 11.1.4. Coordinated Services . . . . . . . . . . . . . . . . 34 105 11.1.5. Qualitative Communication Services . . . . . . . . . 34 106 11.2. New Capabilities . . . . . . . . . . . . . . . . . . . . 34 107 11.2.1. Manageability . . . . . . . . . . . . . . . . . . . 34 108 11.2.2. High Programmability and Agile Lifecycle . . . . . . 35 109 11.2.3. Security . . . . . . . . . . . . . . . . . . . . . . 36 110 11.2.4. Trustworthiness . . . . . . . . . . . . . . . . . . 37 111 11.2.5. Resilience . . . . . . . . . . . . . . . . . . . . . 38 112 11.2.6. Privacy-Sensitive . . . . . . . . . . . . . . . . . 38 113 11.2.7. Accountability and Verifiability . . . . . . . . . . 39 114 12. References . . . . . . . . . . . . . . . . . . . . . . . . . 40 115 12.1. Normative References . . . . . . . . . . . . . . . . . . 40 116 12.2. Informative References . . . . . . . . . . . . . . . . . 40 117 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 45 119 1. Introduction 121 There is an emerging set of new requirements that exceed the network 122 and transport services of the current Internet, which only delivers 123 "best effort" service. While many controlled or private networks 124 include further services, such as other DiffServ QoS in addition to 125 best effort and traffic engineering with bandwidth guarantees, the 126 solutions used today only support walled gardens and are thus not 127 available to application service providers and consumers across the 128 Internet. 130 The purpose of this document is to look at current, evolving and 131 future use cases and to examine the shortcomings that the existing 132 network and transport layer protocols a well as their associated 133 control plane need to overcome to meet these needs. 135 The IETF is the body responsible for the long term evolution of the 136 IP protocol suit, but is missing a work track to discuss the long- 137 term Internet network architecture evolution. In particular it lacks 138 a programme for the long term evolution of IP itself. 140 Approximately 30 years ago, the IETF started a process to 141 revolutionize the IPv4 [RFC0791] Internet Protocol. In this process, 142 researchers, industry, and service providers got together, and 143 brought up a number of new proposals, and worked toward a successor 144 to IPv4, which became IPv6 [RFC2460] and later [RFC8200]. 146 30 years later, there is heavy resistance to anything more than minor 147 incremental evolutions to IPv6. There are a number of reasons for 148 this ranging from opinions that all future IP needs can be met 149 through minor incremental evolutions to fears that major proposals 150 for innovation at the IP would be an unwelcome disrupter to the 151 current business of the vendors or the service providers. 153 The authors take no position on the scale of the problem or the 154 difficulties of deploying any solutions at scale in the Internet. 155 What we seek to do is to establish the scope and nature of the 156 problem. A decision on which aspects of the problem are economically 157 tractable is out of scope of this text, but technologies to support 158 monetization are not. 160 As a problem statement, this documents goal is to not propose or 161 promote specific solutions to the problems raised. Instead it uses 162 references to not Internet adopted, but proposed or existing 163 solutions only as example evidence that the described problem can 164 actually be solved. 166 Because the document does not propose specific solutions, it also 167 does not attempt to structure the problem description in a way that 168 would identify sub-set of problems to be resolved by specific 169 solution components. 171 The purpose of this text is thus to stimulate discussion on the 172 emerging needs of the forwarding layer and to start the process of 173 determining how they are best satisfied within the IETF protocol 174 suite. 176 1.1. Forwarding Layer 178 The term "forwarding layer" is used in this document because none of 179 the standard terms encompass the parts of the network stack that need 180 attention to address the needs of the applications that are foreseen. 182 It is possible that development work will need to reach down to layer 183 2.5 in order to ensure that packets are handled correctly down to the 184 physical layer. The MAC layer is quite sophisticated and includes 185 its own switching function so we need to be sure that the good work 186 done in the network layer is not undone lower down the stack. 187 Equally it is possible that development work will need to reach into 188 the transport layer to address new approaches to congestion and to 189 ensure that the network layer understands the requirements placed on 190 it by the application. An open mind is needed on the boundaries of 191 the layers as they exist today when analyzing the consequential 192 network changes needed to support the evolving application space. 194 In the network layer itself, this document is only concerned with the 195 forwarding component, not path selection or the other components of 196 routing. 198 Thus we use the term forwarding layer to describe the scope of the 199 stack that this document addresses. 201 2. New Use Cases for packet networks 203 This section summarizes the use case areas that have been observed by 204 the authors and are considered relevant to the following analysis of 205 gaps. 207 This section is structured into sub-sections discussing either group 208 of use cases directly or the work of specific groups that are 209 identifying use cases and that may also work on identifying issues 210 and or proposed architectures or solutions for them. 212 Subsections are ordered from what might be considered to be the most 213 near-term use cases to the potentially most far reaching ones. 215 2.1. Video and AR/VR 217 Audio/Video streaming for production, entertainment, surveillance and 218 other purposes, and interactive audio/video are the most ubiquitous 219 applications on the Internet and private IP networks after web- 220 services. They have grown primarily through an evolution of the 221 applications to work with the constraints of todays Internet and 222 adopting pre-existing infrastructure such as content caches: best- 223 effort streaming with adaptive video, no service guaranteees for most 224 services, and co-location of caches with large user communities. In 225 environments where more than best-effort services for these 226 applications are required and deployment of current technologies to 227 support them is feasible, it is done. Examples include DiffServ or 228 even on or offpath bandwidth reservations in controlled networks. 230 Networked AR/VR is a very near term set of use cases, where solution 231 models are very much attempting to use and expand existing solution 232 approaches for video network streaming but where the limits of above 233 current best practices are also amplified by the larger bandwidth 234 requirements and stricter latency and jitter requirements of AR/VR. 236 To ensure a good user experience, for live Virtual Reality (VR), a 237 much higher resolution than 8K video is required. In addition to the 238 high bandwidth requirements of VR, there needs to be a supporting 239 transmission network to provide a communications path with bounded 240 low latency as well. This stringent VR latency requirement is a 241 challenge to existing networks. 243 In cellular networks, even though the the air interface link latency 244 needed is significantly reduced e.g. with New Radio (5GNR), the end- 245 to-end (E2E) requirements for live VR is harder to meet. This is 246 because of the fixed L2/IP/MPLS networks in front/mid/backhaul 247 components, and because of the best effort nature of the packet 248 delivery systems in these networks. 250 2.2. Role of Fixed Networks in 5G and Beyond 5G 252 The 5G and beyond 5G (B5G) services are not meant to be limited to 253 the 5G-NR (new-radio). In fact for those services relating to uRLLC, 254 mMTC and eMBB packet networks have evolve along with the radio 255 technologies. While 5G-NR protocol stack has evolved to provide per- 256 frame reliability and latency guarantees, the IP/MPLS transport 257 network by and large remains best-effort. It is no longer possible 258 to solve network problems simply by increasing the capacity. The 259 expectations 5G devices have of 5G networks, can not be met without 260 improving IP/MPLS based backhaul networks. For example, the 5G based 261 systems involve machine to machine communications, generally using 262 command-based smaller payloads. In this case the overheads of packet 263 headers and overlays become apparent when computing latency budget of 264 such packets. 266 The IETF has produced a large body of work on the deterministic needs 267 of network applications [RFC8578]. These range from refinements and 268 expansions of above summarized Audio/Video and AR/VR use cases over 269 gaming into many more "industrial" use cases. Industrial use cases 270 generally involve industrial controllers for high-precision machinery 271 and equipment, such as robotic arms, centrifuges, or manufacturing 272 equipment for the assembly of electronic components. 273 These use cases have in common that they require delivery of packets 274 with very precise and "deterministic" performance characteristics, as 275 the controlled equipment and the control loops involved have very 276 exact timing requirements and are not tolerant of any latency 277 variations, as otherwise control loop issues and other undesired 278 effects may occur. 279 Specifically, the use cases involve curtailing maximum latency that 280 could be incurred. However, deterministic networking, by itself, 281 does not appear to be sufficient to meet all of the emerging needs. 283 2.3. ITU-T Focus Group Network-2030 285 The ITU-T has been running a Focus Group (FG) Network-2030 286 [FGNETWORK2030] to analyze the needs of networks in the period post 287 2030. This work started in July 2018 and has been an open process 288 with contribution by a cross-section of the networking industry. 289 Because this is non-IETF work, this section summarizes the currently 290 finalied key findings of the ITU-T Focus Group Network-2030 to make 291 it easier for the reader to better undersstand the work. Note that 292 this work is still ongoing and additional findings may be published. 294 The Focus Group Network 2030 considered a number of use cases that it 295 was postulated would need to be addressed in the 2030 time-frame and 296 the technology gaps that need to be bridged in order to address these 297 needs. It then considered a number of new network services that 298 would be needed to support these services. 300 An ongoing piece of work on the architecture of the network post 2030 301 has not yet been completed at the time of writing and is only 302 partially discussed in this document. 304 The reader is referred to [WP], [NET2030SubG2], [UC] for information 305 beyond that provided in this summary. 307 ITU-T FG NET2030 Sub-group Sub-G1 (Sub-G1) considered a number of use 308 cases that it considered to be representative of the network needs 309 post 2030. These needs are legitimate needs in their own right, but 310 as is always the case act as poster-children for new applications 311 that will inevitable conceived in the light of the new network 312 capabilities that we postulate to be necessary. 314 o Holographic-type communications (HCT) 316 o Tactile Internet for Remote Operations (TIRO) 318 o Network and Computing Convergence (NCC) 320 o Digital Twin (DT) 322 o Space-Terrestrial Integrated Networks (STIN) 324 o ManyNets 326 o Industrial IoT (IIoT) with cloudification. 328 Further information on these use cases is provided in Section 10, and 329 in the ITU documents [UC] and [WP]. 331 Note to the reader: Unlike ITU-T Study Groups which are restricted to 332 members, ITU-T Focus Groups are open to anyone without payment. At 333 the time of writing, ITU-T Focus Group Network-2030 material that is 334 not available for anonymous download, is accessible for free by 335 joining the Study Group. 337 3. New Network Capabilities and Services 339 In order to support the use cases presented in Section 2, a number of 340 new network services will be needed. Likewise, a number of 341 additional more general network capabilities will becoming 342 increasingly important. Neither services nor capabilities are 343 sufficiently supported to the degree that will be required by 344 Internet technology in use today. 346 This section describes these services and capabilities at a high 347 level. It builds on a corresponding analysis that was conducted at 348 ITU-T FG-NET2030; readers are referred Section 11 for further detail 349 and, of course, to output produced by that group [NET2030SubG2] for a 350 more complete explanation of their considerations. 352 3.1. New Services 354 [NET2030SubG2] identifies a number of network services that will be 355 needed to support many of the new use cases. These network services 356 are divided into two categories: 358 o Foundational Services (FS) require which dedicated support on some 359 or all network system nodes which are delivering the service 360 between two or more application system nodes. 362 o Compound Services (CS) are composed of one or more foundational 363 services, and are used to make network services easier to consume 364 by certain applications or categories of use cases. An example of 365 a CS would be a Tactile Internet Service which consisted of 366 tactile control channel and a haptic feedback channel. 368 The following are a set of Foundational Services : 370 o High-Precision Communications Services: services with precisely 371 defined service level objectives related to end-to-end latency. 372 Three high-precision communications services that have so far been 373 proposed: 375 * In-time Services: services that require end-to-end latency 376 within a quantifiable limit. This service is similar to the 377 service provided by DetNet [RFC8655] but with more demanding 378 applications which need to be satisfied over IP. 380 * On-time Services: services require end-to-end-latency to be of 381 an exact duration. 383 * Coordinated Services: Coordinated services require multiple 384 interdependent flows to be delivered with the same end-to-end 385 latency, regardless of any (potential additional) service level 386 objective. 388 o Qualitative Communication Services: services that are able to 389 suppress retransmission of less relevant portions of the payload 390 in order to meet requirements on latency by applications that are 391 tolerant to this. 393 These are described in more detain in Section 11.1. 395 3.2. New Capabilities 397 [NET2030SubG2] identifies also a number of network capabilities that 398 will become increasingly important going forward, in addition to the 399 support for any particular services. 400 A number of those need to be taken into consideration from the very 401 beginning when thinking about how future data-planes need to evolve. 402 These capabilities are described in more detail in Section 11.2. 404 o Manageability: Many of the services that need to be supported in 405 the future will require advances in measurements and telemetry 406 will be required in order to monitor and validate that promised 407 service levels are indeed being delivered. These will requires 408 advanced instrumentation that is ideally built. 410 o High Programmability and Agile Life-cycle: Methods to provide 411 operators need to be able to rapidly nd easily introduce new 412 network services and adapt to new contexts and application needs. 414 o Security and Trustworthiness: New mechanisms are needed to 415 authorize packets to enter the network from a host or from another 416 network, and for them to then receive the required premium service 417 that can operate. This must operate without impacting the latency 418 and MTU requirements. This security mechanism has to protect both 419 the network, the user data and the user privacy, but still expose 420 sufficient information to the network that the correct premium 421 service can be delivered. 423 o Resilience: Ultra-low-latency requirements and the huge increase 424 of bandwidth demands of new services such as holographic type 425 communication services make retransmission as a mechanism to 426 recover data that was lost in transit increasingly less feasible. 427 Therefore, network resilience and avoidance of loss becomes more 428 importance that it is for best effort networks. 430 o Privacy-Sensitive: There is a growing awareness of the lack of 431 privacy in the Internet and its implications. 433 New network services have to be sensitive to and comply with 434 heightened user privacy expectations. 435 At the same time, the need for privacy needs to be balanced with 436 legitimate needs of network providers to operate and maintain 437 their networks, which requires some visibility into what is 438 happening on the network and how it is being used. There are a 439 variety of privacy-related requirements that ensue, such as: 441 * Anonymization 443 * Opaque User data 445 * Secured Storage 447 * Flow anonymization 449 o Accountability and Verifiability: Provision of the methods to 450 account for an verify delivery of premium services. 452 4. Underlying New Requirements 454 4.1. Better than Best Effort 456 The current Internet is essentially of best-effort system, but future 457 applications require high-precision KPIs on throughput, latency and 458 packet loss for industrial manufacturing, control, automation, and 459 machine-to-machine communications. 461 With upcoming Cellular technologies (5G/B5G) there is a need for 462 Service Providers to expand the type of customers for metropolitan 463 size networks to address their better than best-effort traffic needs. 465 DetNet has been proposed to support this, however: 467 o Only some aspects of DetNet currently only run on top of current 468 IP/IPv6. 470 o DetNet service is too constrained: It only supports constant bit 471 rate (CBR), reserved bandwidth. It does not support flexible 472 bandwidth. The notion of contracts in a future development of the 473 forwarding layer will support more flexible managed bandwidth and 474 managed latency contracts for traffic. 476 4.2. Efficient Packet Design 478 The ratio of useful data in the payload to overhead has a direct 479 financial impact on communication links; these links are of finite 480 capacity and hence have a finite cost-per-unit-data that can be 481 calculated. The capacity used to transport information as compared 482 to the overhead which is unavailable for use by a customer, but 483 required to transmit is often expresses as a good-put efficiency and 484 can be related to cost to transmit payload data. 486 o There is a need to support large number of low power user 487 equipment (UE) devices (low-power IoTs) connecting through various 488 radio networks (LTE/5G/B5G) where spectral efficiency is needed. 489 This needs to be achieved without header compression techniques 490 like as [RFC6282] since, compression can result in additional 491 processing and energy consumption overhead. 493 o The handling network protocol headers, requires that portions of 494 each packet be held in memory or buffer structures; the more 495 levels of information which need to be held for processing by 496 network nodes, the more memory space will be required, and this 497 directly effects the cost of operation and cost of manufacture/ 498 provision of such equipment. 500 On the other hand, in various non-constrained environments where 501 various network layer functionalities are desired, there are 502 different set of requirements. For example: 504 o Segment Routing over IPv6 (SRv6) parameter encoding 505 [I-D.filsfils-spring-srv6-network-programming] in the SRv6 SID 506 [I-D.ietf-6man-segment-routing-header] is limited by the prefix 507 portion of the IPv6 address. 509 o In Identifier Locator Addressing (ILA), the identifier (ID) 510 portion of the address length is limited because of 128 bits 511 limit. 513 4.3. Forwarding Identifiers 515 Developments in IPv6 [I-D.filsfils-spring-srv6-network-programming] 516 formalize a trend that has been happening for a long time: the 517 morphing of network layer addresses into forwarding identifiers (FI). 518 However, constraining FIs to a fixed size ill serves the development 519 of the forwarding layer. There are clear cases as illustrated above 520 where it would be useful to have shorter network layer addresses. 521 Equally we can see that there will be future cases where 128 bits may 522 be insufficient to specify a forwarding operation. The requirement 523 is thus to formally introduce the concept of forwarding identifiers 524 in place of network layer addresses, and use a forwarding identifier 525 construct that supports multiple semantics and multiple, possibly 526 fully variable, lengths. 528 4.4. Operational visibility 530 Network operators crave facilities that let them better understand 531 and fine tune detailed network behavior, which are hard to retrofit 532 with current IP/IPv6. 534 The rise of machine learning has led to the expectation of being able 535 to better optimize networks This in turn leads to the increase of 536 network telemetry as a source of data to base these systems on. In- 537 Situ OAM (IOAM) [I-D.ietf-ippm-ioam-data] represents one of the 538 latest developments in that space, allowing the data plane to piggy- 539 back telemetry data onto individual packets in order to diagnose and 540 fine-tune service levels such as latency or jitter. However, there 541 are several issues with this approach: 543 o MTU issues limit amount of data that can be obtained. With IOAM 544 packet size increases with number of data items and number of 545 hops. 547 o The data that can be obtained is very limited. 549 o The OAM data volume can easily exceeds that of production traffic 550 which is wasteful 552 o There is no ability to aggregate OAM data, or make context 553 dependent OAM collection. 555 o Integration with other solutions such as DetNet is unclear. 557 While useful, IOAM exposes the limits of what add-on solutions can 558 provide. Solutions that provide visibility at the level of flows or 559 that provide automatic verification of Service Level Objectives are 560 missing entirely. 562 4.5. Holistic solution 564 It needs to be also recognized that it will not be sufficient for 565 solutions to support new services and capabilities one at a time and 566 independently from one another. Instead, solutions need to be 567 holistic and be able to support new services and capabilities in 568 integrated fashion and simultaneously. 569 For example, better-than-best-effort, operational visibility, and 570 efficient packet design should go together, without leading to 571 additional integration problems ore requiring users to to make a 572 choice. 574 This is in contrast to the current piecemeal approach, in which 575 solutions for any one particular problem may well be developed but 576 emerge one at a time, resulting in fragmented solutions that are may 577 be hard to integrate. 579 5. Deployment Models 581 Service requirements from networks and security implications vastly 582 differ in various deployment models as categorized below. 584 5.1. Edge-2-Edge Model 586 This is the traditional service provider deployment where various 587 network services (VPN, security, Bandwidth..) are offered to the 588 endpoints of the communication and other providers. Such 589 capabilities are purchased through contract with the service provider 590 and are typically expensive. 592 These networks predominantly use MPLS technology though native IP 593 (IPv4/IPv6) with GRE and IPv6 with routing extension headers with 594 SRv6 are being deployed recently. 596 .................................. 597 +---+ . +---+ Single +---+ . +---+ 598 |CE1|---|PE1|---.. Provider ..---|PE2|---|CE2| 599 +---+ . +---+ Network +---+ . +---+ 600 .................................. 602 Figure 1: An Edge-2-Edge Network 604 5.2. End-2-End Model Single Provider 606 In this case there is a single provider network in which E2E 607 offerings and host session are initiated and terminated with in the 608 single provider network. 610 1. OTT Provider Networks: Endpoints of the communication (virtual or 611 physical hosts) consuming services through with in the OTT 612 provider network servers (Cloud and Data Center (DC) networks); 613 where the other endpoint can be in the same server form or on the 614 DC Gateway or on the other end of the DC Server Farm connected 615 through Data Center Interconnect (DCI). 617 2. Wireless and Wire-line Networks: Endpoints (UE's) connecting to 618 the provider wireless or wired networks, where service is 619 terminated inside the provider network end points. Based on the 620 service offerings connection termination can happen close to the 621 Radio/access nodes with multi-access edge computing (MEC) clouds 622 or in the provider core network (core-cloud) before going to the 623 Internet eventually. Example of these deployments include BNG, 624 4G and 5G wireless access/RAN/backhaul networks. 626 There are two sub cases: 628 a) Where the host is physically (wired/wireless) connected to the 629 Provider Edge (PE) 631 ............................................. 632 +---+ .+----+ +----+. +---+ 633 | H +--+ PE |--- 1 Provider Network ---| PE +--+ H | 634 +---+ .+----+ (Single/Multiple domains) +----+. +---+ 635 ............................................. 637 Figure 2: An Edge-2-Edge Network Direct 639 In this case the provider controls the whole path and can certify the 640 correct operation of the service according to contract. 642 b) Where the host is connected via its own network to the PE 644 +---+ +---+ 645 | H | | H | 646 +-+-+ +-+-+ 647 | | 648 | ............................................. | 649 +-+--+ .+----+ +----+. +-+-+ 650 | CE +--+ PE |--- 1 Provider Network ---| PE +--+ H | 651 +----+ .+----+ (Single/Multiple domains) +----+. +---+ 652 .............................................. 654 Figure 3: An Edge-2-Edge Network Indirect 656 In this case the provider controls only the path to the CE and can 657 certify the correct operation of the service according to contract 658 from that point but the user is responsible for providing the 659 required service characteristics into their own network. 661 5.3. End-2-End Model with multiple Providers 663 There are two cases to consider: 665 1. Multiple provider with Transit Networks: These are traditional 666 E2E deployments where communication endpoints of the data traffic 667 on different provider networks with regional, transit network 668 providers through Internet Exchange Providers (IXPs) providing 669 the global inter connection. 671 2. Two Providers with no Internet Transit Network: Another variant 672 of the E2E connectivity can be seen as evolving comprises only 673 endpoints provider (access) network and receiver access provider 674 network with global transit provided by one ISP. 676 The first case is very difficult to support since it is unlikely that 677 the whole path is know to support extended capabilities in the 678 forwarding plane. It is not infeasible, and it would be possible to 679 set up such paths in principle given suitable enhancements to the 680 routing system. However such a scenario must be considered 681 infeasible for the foreseeable future. 683 The second case is more tractable provided there is co-operation 684 between the providers. 686 5.4. Embedded Service 688 The industry move is towards content and application service 689 providers embedding themselves within the edge network. This is 690 currently done to save bandwidth and improve response time. As the 691 need for high precision low latency networking develops the need for 692 edge computing rises since the closer the client and the server the 693 less the scope for network induced performance degradation. 695 +---+ 696 | H | 697 +-+-+ 698 | 699 | ..................................... 700 +-+--+ .+----+ +---+ . 701 | CE +--+ PE |--------+ S | . 702 +----+ .+----+ +---+ Provider 1 . 703 ..................................... 705 Figure 4: An Edge-2-Provider 707 In this network the server S (owned by the content and applications 708 provider) has a contractual relationship with provider 1 and is thus 709 able to negotiate the network characteristics needed to meet its 710 service requirement. This model in which the server brokers the user 711 to network interface (UNI) requirements removes many of the 712 objections to the classical UNI model in which the client requests 713 the service requirements. In this model the host authenticates 714 itself with the server, having formed a previous business 715 relationship (for example by purchasing a holographic conferencing 716 service). The server has a relationship with Provider1, and thus is 717 a trusted party able to request that the service be set up between 718 itself and and its client, paying as necessary. As this is a 719 requested paid service traversing a limited distance over a defined 720 network, a bespoke packet protocol can, if necessary, be used with in 721 a contained and constrained way. 723 How the server communicates with any other part of the application 724 domain is out of scope for this document and possibly out of scope 725 for Provider 1. 727 This takes us to consider the embedded global service described in 728 {#EGS}. 730 5.5. Embedded Global Service 732 +---+ 733 | H1| 734 +-+-+ 735 | 736 | ...................................... 737 +-+--+ . +----+ +---+ . 738 | CE +---+ PE |--------+ S1| . 739 +----+ . +----+ +-+-+ Provider 1 . 740 ..................|................... 741 | 742 | 743 ..................|................... 744 +----+ . +----+ +-+-+ . 745 | CE +---+ PE |--------+ S2| . 746 +----+ . +----+ +---+ Provider 2 . 747 | ...................................... 748 | 749 +-+-+ 750 | H2| 751 +-+-+ 753 Figure 5: Edge-2-Edge via Provider 755 In this network model, the server S1 (owned by the content and 756 applications provider) has a contractual relationship with provider 1 757 and is thus able to negotiate the network characteristics needed to 758 meet its service requirement. It is servicing the needs of host H1. 760 Similarly that same provider has a contractual relationship with 761 provider 2 where it is servicing the needs of host H2. 763 By a method outside the scope of this document and outside the scope 764 of the global Internet the contents and applications provider has a 765 private path between S1 and S2. 767 This scenario shown in Figure 5 is important because it removes the 768 overwhelming issues associated with providing enhanced service across 769 the global Internet. Furthermore it describes a model where there is 770 commercial incentive, at scale, for the edge providers (Provider 1 771 and 2 above) to invest in providing and enhanced access service. 773 6. Existing Protocol and Layering Challenges and Gaps 775 Despite IPv4 still having a large user base, and having a number of 776 useful properties the IETF has abandoned future development of IPv4 777 as a way to force the deployment of IPv6. For example, in terms of 778 traffic steering the segment routing could have usefully been applied 779 to IPv4 to support network operators that wished to retain IPv4 as 780 their preferred internal protocol. 782 Given the gaps in each of the existing network layer protocols the 783 IETF may wish to look at the design of a protocol that both fills the 784 gaps and unifies its three existing network layer protocols. 786 Additionally there is a clear need for a more sophisticated approach 787 to indicating the required quality of service that a packet, or flow, 788 needs in an IP network. 790 6.1. Challenges with IPv6 792 6.1.1. The End-to-End Model 794 IPv6 and specifically [RFC8200] was designed to fit within an 795 Internet architecture centered around the end-to-end model with 796 "Internet Paths" potentially passing through one or more networks 797 without any relationship to the endpoints of a communication such as 798 most so-called transit-AS. As history already from IPv4 had shown, 799 anything more than the most simple per-hop processing options can 800 cause interoperability issues. In result, [RFC8200] has drastically 801 limited such per-hop processing options. 803 Two core restrictions of RFC8200 are the following: 805 o Restrictions on extension headers (EH): EHs must never be deleted 806 or changed in size by any node on the path the packet takes. 807 Intermediate nodes are only expected to examine these headers (if 808 they are configured to do so). Implementations cannot expect 809 intermediate nodes to examine, or act on, except for hop-by-hop 810 header (section 4.8 of [RFC8200]). 812 At the time of writing this is an area of considerable active 813 discussion in the IETF 6MAN and SPRING WGs. The issues that 814 arrise from allowing unrestricted insertion, deletion or 815 modification of EHs are for example: 817 * Breakage of path MTU discovery 819 * Impact on the Authetication Header protocol 821 * Inability to return ICMP error messages to the correct node. 823 See Section 6.1.1.1 for further discussion. 825 o No new hop-by-hop headers (HBH) in IPV6: No new EHs that require 826 hop-by-hop behavior should be defined (section 4 of [RFC8200]) - 827 the only EH that has hop-by-hop behavior is the Hop-by-Hop Options 828 header. The only alternative available to the designer is instead 829 to use destination headers (section 6.8 of [RFC8200]). 831 6.1.1.1. IPv6 For Controlled Networks 833 While [RFC8200] is a conservative set of requirements to enable 834 proliferation of the target use case of "Internet Paths", the same 835 set of requirements limit the flexibility of IPv6 unnecessarily when 836 it is used in controlled networks where the constraints and 837 interoperability issues for "Internet Paths" do not equally apply, 838 for example the deployment scenarios shown in Section 5.4 and 839 Section 5.5. 841 One typical type of controlled networks are service providers (SP) 842 where SRv6 is used as the architecture within the SP network. 844 o IPv6 extension headers can not be added on a midpoint. Any 845 addition/change requires an encapsulation where another IPv6 846 header with optional SRH extension header is prepended to the 847 carried IPv6 packet. This is expensive in terms of packet MTU, 848 and in terms of packet buffer requirements at the ends of the 849 provider path which can be an economic issue in cost sensitive 850 network segments. 852 o The requirement to encapsulate instead of being allowed to add an 853 EH along the path stems from the desire to isolate any header 854 changes from Path MTU Discovery (PMTUD). This is a necessary 855 complexity when traversing uncontrolled hops across the Internet, 856 but it is unnecessary overhead when only passing through 857 controlled hops. In MPLS and SR-MPLS, the MPLS header size is not 858 included in the MTU available to the MPLS payload, instead the 859 network is managed such that the maximum MPLS header size plus the 860 available payload MTU is always smaller that the encapsulating L2 861 frame MTU. In IPv6 instead, the encapsulating and decapsulating 862 would logically have to perform signaling for PMTUD 863 (unnecessarily). 865 o Because of the authorization header (AH) [RFC4302] and OAM 866 concerns, [RFC8200] likewise prohibits removing extension headers 867 or fields thereof on hops along the path, requiring for example 868 more complex packet parsers. In SR-MPLS it is possible to simply 869 remove the top SID on a node that has processed it, in SRv6 it is 870 instead necessary to look up an offset field in the SRH and, read 871 the appropriate SID (which may be deep in the packet), and then 872 increment the offset field. 874 o Even though the number of identifiers required within a controlled 875 network is often less than 16 bit, and almost always 32 bits, 876 carrying the overhead of 128 bits per SID in SRv6 can be seen as a 877 significant unnecessary overhead, and workarounds such a proposed 878 micro programs [I-D.bonica-6man-comp-rtg-hdr], 879 [I-D.bonica-spring-srv6-plus], 880 [I-D.filsfils-spring-net-pgm-extension-srv6-usid] require complex 881 forwarding plane processing and SRv6 programmability in the lower 882 64 bit is not required in the majority of use-cases for SIDs on 883 midpoints. 885 For use-cases like this, it would be a lot easier to innovate IPv6 by 886 clone & modify: E.g.: defining (say) IPv7 to be similar to IPv6, but 887 without the constraints that are not useful for the controlled 888 network use-case. A better alternative would be to create different 889 profiles of IPv6 with [RFC8200] being one. However, there is, as 890 yet, no concept of "profiles" in IPv6. 892 The issue of IP protocol operation in limited domains is discussed in 893 [I-D.carpenter-limited-domains]. 895 Some possible solutions are described in 896 [I-D.herbert-6man-eh-attrib]. This will be considered further in a 897 future version of this text. 899 6.1.1.2. IPv6 for Edge-Compute 901 Today, the majority of end-to-end connections already do not pass via 902 the traditional "Internet-Path" but instead toward a server in data 903 center co-located with the access service provider Figure 4[DOT]. In 904 this case, there is no transit service provider, but there is a well- 905 established commercial relationship between either end of the 906 communications and the access service provider. 908 Today, the majority of traffic consists of video-streaming/TV 909 services, but in the future, Edge-Compute will enable ever more 910 applications to operate in such a controlled environment. 912 The difference between the aforementioned use-case of IPv6 within an 913 service provider, and this use-case is that enhanced services in this 914 would naturally operate end-to-end between a Data Center application 915 server and the subscriber endpoints. 917 In the case of SRv6, it is not necessary to incur the overhead of an 918 IPv6 in IPv6 encapsulation, the SRH can be inserted by the endpoint 919 and removed by the endpoint on the other side. Nevertheless, the 920 [RFC8200] limitations of not being able to add/remove or freely 921 change the content of the SRH payload or any other EH on a midpoint 922 router still exists. This seriously limits the usage and evolution 923 of of IPv6 to the edge-to-edge model. 925 6.1.1.3. Hop-by-Hop Extension Header processing 927 Hop-by-hop IPv6 extension headers caused interoperability and 928 performance issues and as a result caused resistance to further 929 leverage and extend them except for SRv6-SRH RPL-SRH [RFC6554]. In 930 the authors opinions, this regression on hop-by-hop extension headers 931 is because of a combination of insufficient specifications and 932 resulting implementation issues. Both could be solved in future work 933 with new hop-by-hop processing specifications. 935 For example, router alert (RA) was (and still maybe) implemented in 936 routers so that all router alert packets are punted from the fast- 937 path to the slow-path even when the "value" field identifies a 938 protocol that the router can not process. As a result, protocols 939 that rely on RA such as RSVP [RFC2205] or even more so Pragmatic 940 General Multicast (PGM) [RFC3208] where filtered in networks because 941 they caused high control plane load on routers that did not support 942 either protocols but still unnecessarily punted their packets with 943 RA. 945 There are no normative statements about the need that fast-path 946 forwarding planes "MUST" be able to ignore unsupported/not-enabled EH 947 features at a speed such that such a packet can be forward at the 948 same speed as the same packet without the EH. For example, for RA, 949 there is only a "SHOULD" requirement to do this in [RFC6398], a BCP 950 published a decade after IPv6 router alert [RFC2711]. With such a 951 gap in time between the specification and the BCP, it is impossible 952 to rely on the existing RA and expect safe deployment across the 953 Internet without still running into performance issues. 955 6.1.1.4. Segment Routing Header Constraints 957 The same design paradigm could have been used for the Segment Routing 958 Header (SRH) [I-D.ietf-6man-segment-routing-header], but there is no 959 distinction possible for IPv6 instances running in such a controlled 960 network or running as an Internetwork instance to form the Internet. 961 This is particularly unfortunate as we are evolving to a model where, 962 as noted earlier in this document, in most cases the packet will only 963 travel through two well-known networks: the hosts network and the 964 service provider network hosting the server to which the client is 965 interacting. 967 6.1.2. Fixed Address Length 969 When IPv6 was designed, the key focus was on solving the problem of 970 growth of the Internet and resulting growth of global Internet 971 address space. Variable length and a hetrogenious address approach 972 were proposed [RFC1347] however, these were rejected partially for 973 political reasons and partially out of a concern over the difficulty 974 of parsing the packet and doing a fast address lookup. 976 There was seemingly no focus on better supporting the now millions of 977 often network-layer isolated TCP/IP networks in industrial, defense, 978 research, embedded, industrial or other commercial environments. 980 One key problems with with 128 bit addresses is the overhead on low- 981 speed radio/IoT-wire networks. This is especially the case when 982 using source-routing, where multiple of these addresses have to be 983 included in the header. Current solutions are only able to resolve 984 these issues with CPU expensive IETF standardized header compression 985 techniques [RFC2507], [RFC3095], [RFC5795]. Even though these 986 approaches are feasible in many of todays IoT networks, there is a 987 strong desire to reduce power consumption in such devices. This is 988 particularly the case where they are powered by a single-for-life- 989 battery, or are self-powering through automatic replenished energy 990 sources. As a result of this CPU performance in future IoT network 991 should not be expected to increase but whenever feasible is more 992 likely to decrease. 994 Another, often overlooked, problem of the 128 bit IPv6 addresses is 995 that global address prefix allocation is a a big up-front burden on 996 many IoT networks, but also isolated networks (industrial, defense, 997 research, industrial). Often, this leads to the use of Unique Local 998 Addresses (ULA) [RFC4193], which have the risk of conflicts when 999 those previously isolated networks need to interconnect with other 1000 networks. 1002 While solutions to these problems may look easier enough, it should 1003 be noted that in the time when IPv6 was designed, variable length 1004 addresses in the fastest forwarding planes were not seen as feasible, 1005 and there was also a lack of experience with the impact of 1006 interconnecting heterogeneous address spaces other than as ships-in- 1007 the-night parallel operation of protocols. A lot of that experience 1008 came later through 14++ IPv4/IPv6 transition solutions designed in 1009 the past 20 years and respective work on address discovery in IETF 1010 frameworks such as SIP/STUN/ICE. 1012 Another issue with the fixed length homogeneous address approach is 1013 the constraints this places on the current practice of overloading 1014 addresses with other functionality for example 1015 [I-D.filsfils-spring-srv6-network-programming]. 1017 6.2. Better Than Best Effort E2E Network Services 1019 Some of the fastest growing network segments where new services are 1020 being introduced in an End-2-End manner belong to deployment models 1021 as described in Section 5.2. The requirements here for service 1022 delivery involves stringent E2E latency with no retransmission and no 1023 packet loss. Not all scenarios need "lower" latency but bounded to a 1024 particular value/range. Example use cases involving an user 1025 equipment (UE) consuming service from the provider cloud network or 1026 another UE (e.g. Vehicular device, IIoT) in the same network. Here 1027 the service endpoints could be connected over wire or wireless (LTE/ 1028 NR) and the service termination happens in the provider network 1029 either close to the access network or provider core network as 1030 illustrated in Figure 2, Figure 3. The existing network layer and 1031 best-effort model simply cannot guarantee needed service level 1032 objectives in these scenarios. 1034 Some specific needs and requirements from cellular fixed transport 1035 networks are: 1037 o Need for determinism on E2E throughput and latency. The current 1038 TCP/IP is hence not-suitable for Mission-critical and real-time 1039 E2E applications. 1041 o Need for E2E QoS for ultra-reliable-low-latency communications 1042 (uRLLC). 1044 o Efficient use of protocols in the network by minimizing tunnels 1045 over tunnels and duplicate header fields. 1047 o Efficient deployment of network slicing 1049 6.3. Adaptive Bit-rate Video streaming 1051 Even without going to future application requirements as described 1052 elsewhere in this document, even the majority of existing Internet 1053 traffic is lacking competitively usable and standardized service to 1054 support quality of service. 1056 The majority of traffic today is Adaptive Bit-rate (ABR) based audio/ 1057 video streaming. The primary benefit of this approach is that it can 1058 adjust itself to much lower bandwidth than the bandwidth to offer the 1059 ideal/target experience quality to the user. It therefore enabled 1060 Over The Top (OTT) services to offer streaming media. Nevertheless, 1061 ABR itself does not provide any actual quality guarantees. 1063 Service providers that use ABR streaming to their subscribers do 1064 therefore combine ABR with IP developments, some non-published, which 1065 are often out-of-band bandwidth reservation schemes. These allow ABR 1066 video streams to have their ideal/target experience bandwidth within 1067 the SP's network and only need to degrade if there was bandwidth 1068 contention in the subscribers (home) network. 1070 If a subscriber, or a content provider which is not the access 1071 service provider wanted to get the same type of bandwidth guarantees 1072 for other content across the access providers network, they could do 1073 so with existing IETF standards via RSVP [RFC2205] which is widely 1074 implemented, or NSIS [RFC4080], which was to the knowledge of the 1075 authors never implemented in widely used router products (because it 1076 does not offer sufficient benefits over RSVP). In either case, the 1077 per-flow control-plane based signaling architecture including the 1078 aforementioned router-alert issues make these protocols a difficult, 1079 likely not future-proof solution. 1081 Even more fundamentally, ABR has shown that media streaming can 1082 easily support elastic adjustment between a range of bandwidth limits 1083 in which the quality is between acceptable and ideal, but there is as 1084 of today no standardized mechanisms by which to express relative 1085 bandwidth allocations when streams compete against each other that 1086 goes beyond the very loosely defined "internet fairness". For 1087 example, more intelligent congestion management could defend 1088 bandwidth the more the bandwidth approaches the minimum acceptable 1089 bandwidth, or admission control of bandwidth could be elastic. Some 1090 work in these direction exists in [RFC8698] with its ability for 1091 weighted congestion control or 1092 [I-D.ietf-tsvwg-intserv-multiple-tspec] for (limited) elastic 1093 admission control management. 1095 6.4. DetNet and Higher Precision Networking Service 1097 Time Critical (TC), Ultra-Reliable, Low Latency (URLLC), Internet-of- 1098 Things is another important use case scenario-set that highlights 1099 requirements that are difficult to satisfy with existing Internet 1100 connectivity paths where a part of that path includes a radio access 1101 link. These kind of close-loop control systems borne over 1102 heterogeneous communications networks have very precision and bounded 1103 latency requirements for the E2E network connecting the sensor and 1104 actuator. 1106 Deterministic networking within the IETF is focused on only one 1107 dimension of the URLLC problem. 1109 DetNet is also far from attempting to identify currently if/how the 1110 services it plans to introduce could be made to operate over the 1111 Internet in general, instead, it focusses mostly on the shorter term 1112 goal to enable them in controlled networks within a limited domains. 1114 Currently, the requirements for a DetNet forwarding plane have been 1115 reasonably mapped out for an MPLS based forwarding layer. 1116 Nevertheless, in addressing these needs within an IP network 1117 [I-D.ietf-detnet-ip] the solution has of necessity been limited to 1118 the capabilities of the IP as it exists today. It has not, for 1119 example, been possible to add the packet replication elimination and 1120 reordering function (PREOF)which allows multiple concurrent packet 1121 delivery attempts in an MPLS network [I-D.ietf-detnet-mpls]. The 1122 DETNET body of requirements needs to be revisited in the light of any 1123 development to network forwarding capabilities. 1125 6.5. Forwarding Plane vs. Control Plane 1127 High-end hardware with accelerated forwarding plane devices, can 1128 support a significant number of forwarding states including 1129 destination entries (IP destination/mask, MPLS label, SR SID) as well 1130 as 2, 3 or 5 tuple IP/IPv6 "flow" entries. Nevertheless, the control 1131 plane that builds and changes these entries often limits their 1132 usability because the control plane does not even scale to the number 1133 of hardware accelerated forwarding entries possible, or because the 1134 supported rate of changes is slow. 1136 The root of this problem is that with the increase of speed and scale 1137 of hardware accelerated forwarding hardware, control plane had 1138 challenges to keep up in performance. The performance of 1139 appropriately priced control plane CPUs (relative to the cost of the 1140 forwarding plane) has not grown at the same speed as that of hardware 1141 accelerated forwarding plane chips. 1143 One of the directions to overcome these challenges is invisible 1144 outside these forwarder devices and it is to optimize the control- 1145 plane to forwarding plane interactions, such as programming the 1146 building of forwarding state directly on the accelerated forwarding 1147 infrastructure (e.g. NPU), but using otherwise existing control 1148 plane protocols. 1150 A more fundamental approach is to redesign control plane protocols 1151 such that they are lighter weight in their signaling and state 1152 machinery, and can therefore be completely implemented in the 1153 hardware accelerated forwarding plane. Effectively turning a control 1154 plane protocol into an advanced forwarding plane protocol function. 1156 This approach is logically most easily applicable to on-path per-flow 1157 signaling mechanisms such as RSVP or RSVP-TE, both of which are quite 1158 complex with their signaling messaging and state keeping and 1159 therefore directly infeasible to become hardware accelerated 1160 forwarding implementations. An example approach to provide similar 1161 functionality to RSVP with signaling light-weight enough to allow 1162 hardware accelerated implementation are the in-band signaling 1163 mechanisms (e.g. for TCP or UDP) described in [DIP1] 1164 [I-D.han-tsvwg-ip-transport-qos] [I-D.han-tsvwg-enhanced-diffserv]. 1166 Signaling that is feasible to become part of a complete in- 1167 forwarding-plane signaling solution is not limited to in-band on-path 1168 flow signaling, but would likely also be applied to other signaling 1169 options. Of the aforementioned existing signaling protocols, IGPs 1170 are likely the ones whose signaling could most easily be processed in 1171 an NPU compute elements except that the SPF calculation itself 1172 introduces a complexity that would make this very complex. One 1173 example of a solution that solves this problem by signaling the 1174 actual per-hop adjacencies in IGP and therefore eases NPU 1175 implementation can be found in 1176 [I-D.chunduri-isis-preferred-path-routing]. 1178 In summary: The scope of what should be considered forwarding plane 1179 today is defined by decade historic architectures, but should for the 1180 future be scoped by the realities of the new, different "layers" of 1181 hardware and their capabilities. Hence also the use of the term 1182 forwarding plane, because it can span not only across classical 1183 bridging (L2), label/tag/SIG switching (L2.5), network/internetwork 1184 (L3) and transport (L4) layers, but also across the classical "data 1185 plane" and "control plane" components of each such layer. 1187 6.6. User-Network/Network-User Interface Signaling 1189 Some of the deployment models as described in Section 5.2, needs 1190 specific signaling mechanism from user/applications. These are 1191 needed for E2E service offering for better than best effort 1192 Section 6.2 or high-precision networking Section 6.4. These may 1193 involve new transport mechanisms at hosts, middle-boxes and routers 1194 to meet the E2E service requirements in these limited domain 1195 deployments. 1197 Here one of the functional requirements is to signal the service 1198 level objectives (SLOs) dynamically for a particular service from the 1199 network. This signaling includes the service description, the 1200 service negotiation with the network, the service setup or 1201 modification, or the need to execute some functions at network device 1202 and send the results back to the sender. However, the current IP was 1203 not designed for this. For example, the result of SLO negotiation at 1204 any hop needs to be updated in the IP packet at the router and 1205 returned back to the sender (originating host or gateway device for a 1206 Service Provider). 1208 There are some attempts to achieve the above as described in 1209 [I-D.han-tsvwg-ip-transport-qos], which describes general in-band 1210 signaling for QoS control with IPv6 protocol and 1211 [I-D.han-tsvwg-enhanced-diffserv], which proposes a backward 1212 compatible class-based queuing and scheduling schema for hybrid 1213 service to support guaranteed service from the network (e.g. for 1214 latency and bandwidth). 1216 In summary, it is difficult to do better than best effort or High 1217 Precision Services described in Section Section 6.4, in closed 1218 domains with current IP given the best effort congestion control 1219 (TCP/QUIC) and explicit congestion notification (ECN) framework. A 1220 comprehensive mechanism needs to be explored as the limitations in 1221 silicon technologies or deployment models 30 years ago are not 1222 relevant with respect to security, scalability, packet size change, 1223 MSS or FCS recalculation, etc. 1225 7. Candidate Solution Directions 1227 This section describes a number of solution considerations, but is 1228 not prescriptive about any specific approach or technical solution, 1229 and is provided to stimulate thought on the subject. 1231 7.1. Variable Length Addresses 1233 When private networks are set up, they only need to use an address 1234 length that allows the construction of networks sufficiently large to 1235 meet the expected service requirements. If a future network layer 1236 protocol could support address length of e.g.: 16, 24, 32, 48, 64 and 1237 128 bits (or maybe more), it would be easy for such networks to pick 1238 a right size. This would allow them to have as efficient packets 1239 without compression as possible, and it would also avoid for them to 1240 have to think about allocation procedures for "global" addresses. 1242 Whenever networks with a smaller address size would later on have to 1243 interconnect to other networks, the shorter length address would have 1244 to be interpreted as the suffix of a sufficiently larger address 1245 space through which those connecting networks could achieve unique, 1246 non-overlapping addresses. At the border between these networks, 1247 high speed forwarding planes could easily perform per-packet 1248 stateless prefix addition/deletion transformations of addresses in 1249 the packet header when the interconnection should be free of further 1250 policy. When such an interconnection is desired to employ specific 1251 traffic control policies, mapping of addresses in a stateful manner 1252 is a convenient way to enforce and support such policies through the 1253 forwarding plane. 1255 7.2. Address Semantics 1257 Classically IP unicast addresses identify an interface. There is the 1258 special case of a loop-back address, but this is normally modeled as 1259 an internal interface. Addresses are often silently mapped to 1260 include other semantics and this is most developed in the IP network 1261 programming concept [I-D.filsfils-spring-srv6-network-programming]. 1263 MPLS is more general. It defines the concept of a Forwarding 1264 Equivalence Class in which a Label which can be visualized as an 1265 offset into a specific table with up to 2^20 entries, with the table 1266 containing the instruction to be executed. Thus a single identifier 1267 is able to specify: forward towards an egress, forward along a 1268 specific path, decapsulate and sent to an interface, decapsulate and 1269 forward via an IP lookup in a label specific address table etc. 1271 The semantics of the MPLS label and the size of the label are such 1272 that it is not possible to include any instruction parameters in the 1273 label and very inefficient to include those parameters in one or more 1274 further labels. The only example of doing this is the Entropy Label 1275 indicator [RFC6790] which uses two Label Stack Entries (LSEs). Any 1276 future development along these lines will need at least three LSEs. 1278 Whilst an IPv6 is larger there is still limited space to add 1279 parameters within the address. In the current work on this the size 1280 is limited to 16 bits, and there is a fundamental limit of 64 bits. 1282 It is clear that move is towards a multiplicity of semantic for the 1283 network layer address, and indeed a formal recognition that the 1284 address is in reality an instruction with a specific scope. 1286 7.3. Multiple Instructions 1288 What we have learned from MPLS and then from SRv6 is that it is often 1289 desirable for a node (be that the originating host or a router) to 1290 impose on a packet a set of instructions to be executed in sequence 1291 by one or more entities in the network. An development of IP or any 1292 successor needs to recognize this and provide a simple and efficient 1293 way to incorporate a list (or stack) of instructions within the 1294 packet header. 1296 7.4. Node and Path Specific Processing Instructions 1298 There is an established need to do node specific instructions as is 1299 indicated by the design of MPLS and Segment Routing (SR). Any 1300 development of the forwarding system needs to retain this feature and 1301 ideally develop a method that is simultaneously both general and 1302 efficient. 1304 References to efficiency include efficiency in packet size and 1305 efficiency decoding and and executing the instruction. The 1306 efficiency of encoding is not simply a matter of on the wire 1307 bandwidth, but is also a matter of the size of the forwarder packet 1308 header cache. This cache has to operate at wire speed can be an 1309 expensive silicon element. 1311 There is also a need to do path specific operations as are done in 1312 RSVP-TE. However RSVP has a significant path set-up and path 1313 maintenance cost. Clearly a per path instruction can be specified as 1314 a set of N per node instructions where N is the number of hops along 1315 the path, for example by using SR, but that is not an efficient 1316 encoding where N is large. It is thus a useful optimization to 1317 include the ability to include per path instructions, and this is the 1318 subject of further study. 1320 7.5. Integrated Assurance and Verification 1322 Being best effort in nature, assurance for services provided using IP 1323 is left to add-on solutions built after the fact. How to perform 1324 tasks such as verifying of service levels is left as an exercise for 1325 network providers, often approached using statistical approaches that 1326 are themselves "best effort" in nature. This will be no longer 1327 sufficient for mission-critical services such as tele-driving or 1328 tele-operations that demand guarantees, where failure to meet those 1329 guarantees may expose providers and users exposed to liability 1330 demands and call the feasibility of applications relying on those 1331 services into question. 1333 Moving forward, network protocols suitable to deliver high-precision 1334 services for mission critical applications need to address assurance 1335 as an intrinsic property, not left to afterthoughts. 1337 8. IANA Considerations 1339 This document does not request any allocations from IANA. 1341 9. Security Considerations 1343 Security is likely to be more significant with the applications being 1344 considered in this work. With interest in tightly controlled access 1345 and latency, and contractual terms of business it is going to be 1346 necessary to have provable right of access to network resources. 1347 However heavyweight security is a contra-requirement to the light- 1348 weight process needed for power efficiency, fast forwarding and low 1349 latency. Addressing this will require new insights into network 1350 security. 1352 Further information on the issue of providing security in latency 1353 sensitive environments can be found in [I-D.ietf-detnet-security] 1354 which are a sub-set of the considerations applicable to the new use 1355 cases considered in this text. 1357 10. Appendix 1: Expanded Summary of Sub-G1 Use Cases 1359 10.1. Holographic-type communications 1361 This work projects that we will move towards a holographic society 1362 where users remotely interact with the physical world over the 1363 network. In industry the digital twin model will enable the control 1364 of real objects through digital replicas. Telepresence will move to 1365 a new level with multi-site collaborations becoming much closer to 1366 physical meetings that can take place without the time and 1367 environmental cost of physical travel. 3D medical scans will become 1368 full 3D views rather than the body/ organ slices that too many of us 1369 are regrettably familiar with. It is easy to imagine that this 1370 technology will take message delivery to a completely new level. 1372 Analysis of these concepts results in the conclusion that the 1373 following key network requirements are necessary: 1375 o Ultra-high bandwidth (Tbps class) 1377 o Ultra-low latency (sub-ms) 1379 o Multi-stream synchronization 1381 o Enhanced network security 1383 o Enhanced network reliability 1385 o Edge computation 1387 10.2. Tactile Internet for Remote Operations 1389 Two cases were proposed as examples of this class of application. 1390 The first is remote industrial management which involves the real- 1391 time monitoring and control of industrial infrastructure operations. 1392 The second involves remote robotic surgery. Remote robotic surgery 1393 within an operating suite complex is a standard practice today, 1394 however there are cases where it would be desirable to extend the 1395 range of this facility. 1397 Analysis of these concepts results in the conclusion that the 1398 following key network requirements are necessary: 1400 o Ultra-high bandwidth (Tbps class) 1402 o Ultra-low latency (sub-ms) 1404 o Sensory input synchronization 1406 o Enhanced network security 1408 o Enhanced network reliability 1410 o Differentiated prioritization levels 1412 10.3. Space-Terrestrial Integrated Networks 1414 The game-changer in the area of space-terrestrial networking is the 1415 active deployment of huge clusters of cheap Low Earth Orbit (LEO) 1416 satellite constellations. These LEOs have a number of properties 1417 that make them attractive, but arguably the most important is that 1418 they combine global coverage with low latency. Studies [Handley] 1419 show that for distances over 3000Km latency via a LEO cluster is 1420 lower than the latency of terrestrial networks. The up-link to a LEO 1421 cluster has to constantly change the point of attachment to the 1422 cluster as the satellites that form the cluster rapidly move across 1423 the sky relative to both the ground and relative to the satellites in 1424 other orbits. In this scenario a number of access and connection 1425 models need to be considered. 1427 Analysis of these concepts results in the conclusion that the 1428 following key network requirements are necessary: 1430 o A suitable addressing and routing mechanism to deal with a network 1431 that is constantly in flux. 1433 o Sufficient bandwidth capacity on the satellite side to support the 1434 new application needs 1436 o A suitable satellite admission system 1438 o Edge computation and storage 1440 10.4. ManyNets 1442 There is evidence that there is a change in direction from the 1443 Internet as a single hetrogenious network back to a true Internet, 1444 that is an interconnection of a number of networks each optimized for 1445 its local use but capable of inter-working. 1447 For example, satellite and the terrestrial networks adopt different 1448 protocol architecture, which causes the difficulty to internetwork 1449 between them, yet the common goal is to provide access to the 1450 Internet. Secondly, there will be a massive number of IoT-type 1451 devices connecting to the networks but the current interconnection 1452 schemes are too complex for these services. There are further trends 1453 in 5G/B5G back-haul infrastructure, requiring diverse set of resource 1454 guarantees in networks to support different industry verticals. The 1455 collection of such special purpose networks, existing together and 1456 requiring interconnection among themselves are called ManyNets. 1458 Much closer the traditional Internet model is the move to edge 1459 computing services in which the client traffic is terminated at a 1460 compute node very close to access edge. [DOT] Any resultant 1461 application traffic is a private matter between the application on 1462 the edge server and the servers it communicates with in the 1463 fulfillment of those needs. Furthermore the application on the 1464 client may be using a tunnel to the edge compute server. In such a 1465 network the protocol used inside the tunnel and the protocol used 1466 between the servers executing the service is a private matter. 1468 The ManyNets concept aims to support flexible methods to support the 1469 communication among such heterogeneous devices and their networks. 1471 11. Appendix 2: Expanded Summary of Sub-G2 New Network Capabilities and 1472 Services 1474 This appendix expands on the ITU-T Sub-G2 new network capabilities 1475 and services introcuced in Section 3 It builds upon the analysis that 1476 was conducted at ITU-T FG-NET2030; readers are also referred to 1477 output produced by that group [NET2030SubG2] for more detail. 1479 11.1. New Services 1481 [NET2030SubG2] identifies a number of network services that will be 1482 needed to support many of the new use cases. These network services 1483 are divided into two categories: 1485 o Foundational Services (FS) require which dedicated support on some 1486 or all network system nodes which are delivering the service 1487 between two or more application system nodes. FS cannot be 1488 decomposed into other services. For example, IP packet routing 1489 and forwarding are is a (pre-existing) foundational network 1490 services. 1492 o Compound Services (CS) are composed of one or more foundational 1493 services. CS are "convenience services" that make network 1494 services easier to consume by certain applications or categories 1495 of use cases, but do not by themselves introduce new network 1496 services or requirements into network system nodes. One example 1497 would be a Tactile Internet Service which consists of two 1498 communications channels, one for tactile control and the other for 1499 haptic feedback. 1501 The following sections focus on Foundational Services only, as these 1502 are the ones that provide the basic building blocks with which the 1503 needs of all other services can be addressed, and which are the ones 1504 that potentially introduce new foundational requirements on network 1505 system nodes. 1507 11.1.1. High-Precision Communications Services 1509 High-Precision Communications Services refers to services that have 1510 precisely defined service level objectives related to end-to-end 1511 latency, in many cases coupled with stringent requirements regarding 1512 to packet loss and to bandwidth needs. These requirements are in 1513 stark contrast to the best effort nature with related to existing 1514 network services. 1515 Of course, existing services often go to great lengths in order to 1516 optimize service levels and minimize latency, and QoS techniques aim 1517 to mitigate adverse effects of e.g. congestion by applying various 1518 forms of prioritization and admission control. However, 1519 fundamentally all of these techniques still constitute patches that, 1520 while alleviating the symptoms of the underlying best-effort nature, 1521 do not address the underlying cause and fall short of providing 1522 service level guarantees that will not be just of a statistical 1523 nature but that will be met by design. 1525 The high-precision communications services that have been identified 1526 are described in the following three sub-sections. 1528 11.1.2. In-time Services 1530 In-time services require end-to-end latency within a quantifiable 1531 limit. They specific a service level objective that is not to be 1532 exceeded, such as a maximum acceptable latency (putting a hard 1533 boundary on the worst case). In-time services are required by 1534 applications and use cases that have clear bounds on acceptable 1535 latency, beyond which the Quality of Experience would deteriorate 1536 rapidly, rendering the application unusable. An example concerns use 1537 cases that involve providing tactile feedback to users. Creating an 1538 illusion of touch requires a control loop with a hard-bounded round- 1539 trip time that is determined by human / biological factors, beyond 1540 which the sense of touch is lost and with it the ability to e.g. 1541 operate a piece of machinery from remote. Because many such use 1542 cases are mission-critical (such as tele-driving or remote surgery), 1543 in addition any loss or need for retransmission is unacceptable. 1545 This service is similar to the service provided by DetNet [RFC8655] 1546 but with more demanding applications which need to be satisfied over 1547 IP. 1549 11.1.3. On-time Services 1551 On-time services require end-to-end-latency to be of an exact 1552 duration, with the possibility of a small quantifiable variance as 1553 can be specified either by an acceptable window around the targeted 1554 latency or by a lower bound in addition to an upper bound. Examples 1555 of use cases include applications that require synchronization 1556 between multiple flows that have the same in-time latency target, or 1557 applications requiring fairness between multiple participants 1558 regardless of path lengths, such as gaming or market exchanges when 1559 required by regulatory authorities. The concept of a lowest 1560 acceptable latency imposes new requirements on networks to 1561 potentially slow down packets by buffering or other means, which 1562 introduces challenges due to high data rates and the cost e.g. of 1563 associated memory. 1565 11.1.4. Coordinated Services 1567 Coordinated services require multiple interdependent flows to be 1568 delivered with the same end-to-end latency, regardless of any 1569 (potential additional) service level objective. Use cases and 1570 applications include applications that require synchronization 1571 between multiple flows, such as use cases involving data streams from 1572 multiple cameras and telemetry sources. In the special case where an 1573 on-time service is required, no additional service is needed (as 1574 synchronization occurs by virtue of the fact that each flow adheres 1575 to the same SLO), but coordination may also be required in cases 1576 where no specific end-to-end latency is required, as long as all 1577 flows are serviced with service levels that are identical. 1579 11.1.5. Qualitative Communication Services 1581 Qualitative communication services (QCS) are able to suppress 1582 retransmission of portions of the payload that are deemed less 1583 relevant when necessary in order to meet requirements on latency by 1584 applications that are tolerant of certain quality degradation. They 1585 may involve the application of network coding schemes. 1587 QCS is a new service type that is needed to support AR/VR, 1588 holographic-type communications Industrial Internet and services such 1589 as autonomous driving. This needs the support of a new network 1590 capability that is as yet to be developed. 1592 11.2. New Capabilities 1594 [NET2030SubG2] identifies also a number of network capabilities that 1595 will become increasingly important going forward, in addition to the 1596 support for any particular services. These were introduced in 1597 Section 3.2. A number of these capabilities need to be taken into 1598 consideration from the very beginning when thinking about how future 1599 dataplanes need to evolve. 1600 While many of those capabilities are well known, the past has shown 1601 that retrofitting dataplanes with such capabilities after the fact 1602 and in a way that is adequate to the problem at hand is very hard. 1604 11.2.1. Manageability 1606 Many of the services that need to be supported in the future have in 1607 common that they place very high demands on latency and precision 1608 that need to be supported at very high scales, coupled with 1609 expectations of zero packet loss and much higher availability than 1610 today. 1612 In order to assure in-time and on-time services with high levels of 1613 accuracy, advances in measurements and telemetry will be required in 1614 order to monitor and validate that promised service levels are indeed 1615 being delivered. This requires advanced instrumentation that is 1616 ideally built-in all the way to the protocol level. 1618 For example, the ability to identify and automatically eliminate 1619 potential sources of service-level degradations and fluctuations will 1620 become of increasing importance. This requires the ability to 1621 generate corresponding telemetry data and the ability to observe the 1622 performance of packets as they traverse the network. Some of the 1623 challenges that need to be addressed include the very high volume of 1624 data that gets generated and needs to be assessed, and the effects of 1625 the collection itself on performance. In general, greater emphasis 1626 will need to be placed on the ability to monitor, observe, and 1627 validate packet performance and behavior than is the case today. For 1628 seamless support, these capabilities will be inherently integrated 1629 with the forwarding function itself, for example delivered together 1630 with the packets. Today's solution approach, IOAM, is a promising 1631 technology currently that points in the right direction, and that 1632 also highlights some of the challenges - from MTU considerations due 1633 to extending packet sizes to the ability to customize and obtain the 1634 "right" data. It will therefore be not sufficient by itself. Data 1635 to be generated from the network will need to be "smarter", i.e. more 1636 insightful and action-able. This will require additional abilities 1637 to process data "on-device". In additional, the need for new 1638 management functions may arise, such as functions that allow to 1639 validate adherence with agreed-upon service levels for a flow as a 1640 whole, and to prevent data or privacy leakage as well as provide 1641 evidence for the possibility or absence of such leakage. 1643 11.2.2. High Programmability and Agile Lifecycle 1645 Operators need to be able to rapidly introduce new network services 1646 and adapt to new contexts and application needs. This will require 1647 advances in network programmability. Today's model of vendor-defined 1648 (supporting service features via new firmware or hardware-based 1649 networking features) or operator-defined (supporting service features 1650 via programmable software-defined networking (SDN) controllers, 1651 virtualized network functions (VNF) and Network Function 1652 Virtualization (NFV), and service function chaining (SFC) will no 1653 longer be sufficient. 1655 Software Defined Networking and Network Function Virtualization (NFV) 1656 have opened up the possibility to accelerate development life-cycles 1657 and enable network providers to develop new networking features on 1658 their own if needed. Segment Routing is being evolved for that 1659 purpose as well. Furthermore, network slicing promises more agility 1660 in the introduction of new network services. However, the complexity 1661 of the associated controller software results in its own challenges 1662 with software development cycles that, while more agile than life- 1663 cycles before, are still prohibitive and that can only be undertaken 1664 by network providers, not by their customers. Rapid customization of 1665 networking services for specific needs or adaptation to unique 1666 deployments are out of reach for network provider customers. What is 1667 lacking is the ability for applications to rapidly introduce and 1668 customize novel behavior at the network flow level, without need to 1669 introduce application-level over-the-top (OTT) overlays. Such a 1670 capability would be analogous to server-less computing that is 1671 revolutionizing cloud services today. In addition, it should be 1672 noted that softwarized networks are built on relatively stable (and 1673 slowly evolving) underlying physical commodity hardware network 1674 infrastructure. This is insufficient to deliver on new high- 1675 precision network services, which require hardware advances at many 1676 levels to provide programmable flow and QoS behavior at line rate, 1677 affecting everything from queuing and scheduling to packet processing 1678 pipelines. 1680 The evolution of forwarding planes should allow development life- 1681 cycles that are much more agile than today and move from "Dev Ops" to 1682 "Flow Ops" (i.e. dynamic programmability of networks at the flow 1683 level). 1684 This requires support of novel network and data-plane programming 1685 models which can possibly be delivered and effected via the 1686 forwarding plane itself. 1688 11.2.3. Security 1690 The possibility of security threats increases with complexity of 1691 networks, the potential ramifications of attacks are growing more 1692 serious with increasing mission-criticality of networking services 1693 and applications. 1694 The forwarding plane plays a large role in the ability to thwart 1695 attacks. 1696 For example, the fact that source addresses are not authenticated in 1697 existing IP is at the root of a wide range security problems from 1698 phishing and fraudulent impersonation designed to compromise and 1699 steal user assets to amplification attacks designed to bring down 1700 services. 1701 Going forward, it is absolutely critical, then, to minimize the 1702 attack surface of the forwarding plane as it evolves. 1704 A key security aspects needed from the network point of view includes 1705 to verify if the packet is authorized to enter into the network and 1706 if it is sufficiently integrity protected. However, when packets are 1707 emitted from the host for these new communication services, the 1708 network portion of the packet (e.g., an extension header or an 1709 overlay header) should not be encrypted because network nodes may 1710 need to interpret the header and provide the desired service. 1711 Lack of encryption and integrity validation, of course, would at the 1712 same time increase the threat surface and open up the possibility for 1713 attacks. 1714 Mechanisms for authorization and integrity protection must be 1715 developed to meet the line rate performance as services delivered can 1716 be time sensitive. At the same time, the size of packets should not 1717 be significantly increased to avoid negative impact on utilization 1718 and overhead tax. 1719 This limits the options for additional security collateral that can 1720 be included with packets. 1722 Homomorphic forms of encryption may need to be devised in which 1723 network operations can be performed in privacy-preserving manner on 1724 encrypted packet headers and tunneled packets without exposing any of 1725 their contents. 1727 Another dimension to security arises when the end to end service that 1728 needs to be delivered crosses the administrative boundary of the 1729 originating host. For those cases, additional mechanisms need to be 1730 specified to sufficiently ensure the privacy and confidentiality of 1731 the network layer information. While there are lot of avenues to 1732 tackle these issues and some aspects are being looked into by various 1733 Standards Development Organizations, e.g. IRTF PANRG on Path-Aware 1734 Networking, comprehensive solutions are yet to be worked out. 1736 Any mechanisms specified for authorization, integrity protection, and 1737 network header confidentiality should be orthogonal to the transport 1738 layer and above transport layer security mechanisms set in place by 1739 the end host/user. Regardless of whether or not the latest security 1740 advances in transport and layers above (e.g. TLS1.3, QUIC or HTTPSx) 1741 are applied on the payload, network nodes should not have to act on 1742 this information to deliver new services to avoid layer violations. 1744 11.2.4. Trustworthiness 1746 As future network services are deployed, deployment scenarios will 1747 include cases in which packets need to traverse trust boundaries 1748 which are under different administrative domains. As the forwarding 1749 plane evolves, it should do so in such a way that trustworthiness of 1750 packets is maintained - i.e. integrity of data is protected, 1751 tampering with packet meta-data (such as source authentication or 1752 service level telemetry) would be evident, and privacy of users is 1753 guarded. 1755 11.2.5. Resilience 1757 Ultra-low-latency requirements and the huge increase of bandwidth 1758 demands of new services such as holographic type communication 1759 services make retransmission as a mechanism to recover data that was 1760 lost in transit increasingly less feasible. Therefore, network 1761 resilience and avoidance of loss becomes of paramount importance. 1763 There are many methods for providing network resilience. This 1764 includes providing redundancy and diversity of both physical (e.g. 1765 ports, routers, line cards) and logical (e.g. shapers, policers, 1766 classifiers) entities. It also includes the use of protocols that 1767 provide quick re-convergence and maintain high availability of 1768 existing connections after a failure event occurs in the network. 1769 Other techniques include packet replication or network coding and 1770 error correction techniques to overcome packet loss. 1771 As the forwarding plane evolves, mechanisms to provide network 1772 resilience should be inherently supported. 1774 11.2.6. Privacy-Sensitive 1776 Today, there is a growing awareness of the lack of privacy in the 1777 Internet and its implications. 1778 New network services have to be sensitive to and comply with 1779 heightened user privacy expectations. 1780 At the same time, the need for privacy needs to be balanced with 1781 legitimate needs of network providers to operate and maintain their 1782 networks, which requires some visibility into what is happening on 1783 the network and how it is being used. 1784 Likewise, mechanisms to provide privacy must be provided in such a 1785 way to not compromise security, such as allowing anonymous attackers 1786 to prey on other users. 1788 An evolved forwarding plane must provide mechanisms that ensure users 1789 privacy by design and prevent illegitimate exposing of personally- 1790 identifiable information (PII), while preventing abuse of those 1791 mechanisms by attack exploits and while affording network providers 1792 with legitimate visibility into use of their network and services. 1794 There are a variety of privacy-related requirements that ensue, such 1795 as: 1797 o Anonymization: To prevent tracking by eavesdropper by packet 1798 capture, visible information in packets such as source and 1799 destination addresses should be difficult (ideally: impossible) to 1800 directly correlate to PII. 1802 o Opaque User data: Networks must not rely on the user data to 1803 provide or improve the service. However, network providers may 1804 use specific service-visible data in packets. 1806 o Secured Storage: Some services may require the network to slow 1807 down the delivery of the packets, implying the possibility that 1808 packets are temporarily buffered on the router. The storage of 1809 those packets must be secured and prevented from extraction for 1810 deep inspection or analysis. 1812 o Flow anonymization: Flows of information should be randomized in a 1813 dynamic manner so that it is difficult through traffic analysis to 1814 deduce patterns and identify the type of traffic. 1816 Potential mechanisms to consider include (but are not limited to) 1817 avoiding the need for long-lived addresses (to prevent trackablity) 1818 and the use of homomorphic encryption for packet headers and tunneled 1819 packets (in addition to traditional payload encryption) that allow to 1820 perform network operations in privacy-preserving manner without 1821 exposing meta-data carried in headers. 1823 11.2.7. Accountability and Verifiability 1825 Many new services demand guarantees instead of being accepting of 1826 "best effort". 1827 As a result, today's "best effort" accounting may no longer be 1828 sufficient. 1830 Today's accounting technology largely relies on interface statistics 1831 and flow records. 1832 Those statistics and records may not be entirely accurate. 1833 For example, in many cases their generation involves sampling and is 1834 thus subject to sampling inaccuracies. 1835 In addition, this data largely accounts for volume but not so much 1836 for actual service levels (e.g. latencies, let alone coordination 1837 across flows) that are delivered. 1838 Service level measurements can be used to complement other statistics 1839 but come with significant overhead and also have various limitations, 1840 from sampling to the consumption of network and edge node processing 1841 bandwidth. 1842 Techniques that rely on passive measurements are infeasible in many 1843 network deployments and hampered by encryption as well as issues 1844 relating to privacy. 1846 Guarantees demand their price. This makes it increasingly important 1847 both for providers and users of services to be able to validate that 1848 promised service levels were delivered on. 1850 For example, proof of service delivery (including proof of service 1851 level delivery) may need to be provided to account and charge for 1852 network services. 1853 This will require advances in accounting technology that should be 1854 considered as forwarding technology evolves, possibly providing 1855 accounting as a function that is intrinsically coupled with 1856 forwarding itself. 1858 12. References 1860 12.1. Normative References 1862 [RFC0791] Postel, J., "Internet Protocol", STD 5, RFC 791, 1863 DOI 10.17487/RFC0791, September 1981, 1864 . 1866 [RFC8200] Deering, S. and R. Hinden, "Internet Protocol, Version 6 1867 (IPv6) Specification", STD 86, RFC 8200, 1868 DOI 10.17487/RFC8200, July 2017, 1869 . 1871 12.2. Informative References 1873 [DIP1] ETSI, "Recommendation for New Transport Technologies, GR 1874 NGP 010", September 2018, 1875 . 1878 [DOT] Huston, G., "The Death of Transit and Beyond", n.d., 1879 . 1882 [FGNETWORK2030] 1883 "Focus Group on Technologies for Network 2030", n.d., 1884 . 1887 [Handley] Handley, M., "Delay is Not an Option: Low Latency Routing 1888 in Space", n.d., 1889 . 1891 [I-D.bonica-6man-comp-rtg-hdr] 1892 Bonica, R., Kamite, Y., Niwa, T., Alston, A., and L. 1893 Jalil, "The IPv6 Compressed Routing Header (CRH)", draft- 1894 bonica-6man-comp-rtg-hdr-13 (work in progress), March 1895 2020. 1897 [I-D.bonica-spring-srv6-plus] 1898 Bonica, R., Hegde, S., Kamite, Y., Alston, A., Henriques, 1899 D., Jalil, L., Halpern, J., Linkova, J., and G. Chen, 1900 "Segment Routing Mapped To IPv6 (SRm6)", draft-bonica- 1901 spring-srv6-plus-06 (work in progress), October 2019. 1903 [I-D.carpenter-limited-domains] 1904 Carpenter, B. and B. Liu, "Limited Domains and Internet 1905 Protocols", draft-carpenter-limited-domains-13 (work in 1906 progress), February 2020. 1908 [I-D.chunduri-isis-preferred-path-routing] 1909 Chunduri, U., Li, R., White, R., Tantsura, J., Contreras, 1910 L., and Y. Qu, "Preferred Path Routing (PPR) in IS-IS", 1911 draft-chunduri-isis-preferred-path-routing-00 (work in 1912 progress), June 2018. 1914 [I-D.filsfils-spring-net-pgm-extension-srv6-usid] 1915 Filsfils, C., Camarillo, P., Cai, D., Voyer, D., Meilik, 1916 I., Patel, K., Henderickx, W., Jonnalagadda, P., and D. 1917 Melman, "Network Programming extension: SRv6 uSID 1918 instruction", draft-filsfils-spring-net-pgm-extension- 1919 srv6-usid-04 (work in progress), February 2020. 1921 [I-D.filsfils-spring-srv6-network-programming] 1922 Filsfils, C., Camarillo, P., Leddy, J., 1923 daniel.voyer@bell.ca, d., Matsushima, S., and Z. Li, "SRv6 1924 Network Programming", draft-filsfils-spring-srv6-network- 1925 programming-07 (work in progress), February 2019. 1927 [I-D.han-tsvwg-enhanced-diffserv] 1928 Han, L., Qu, Y., and R. Li, "Enhanced DiffServ by In-band 1929 Signaling", draft-han-tsvwg-enhanced-diffserv-00 (work in 1930 progress), November 2019. 1932 [I-D.han-tsvwg-ip-transport-qos] 1933 Han, L., Qu, Y., Dong, L., Li, R., Nadeau, T., Smith, K., 1934 and J. Tantsura, "Resource Reservation Protocol for IP 1935 Transport QoS", draft-han-tsvwg-ip-transport-qos-03 (work 1936 in progress), October 2019. 1938 [I-D.herbert-6man-eh-attrib] 1939 Herbert, T., "Attribution Option for Extension Header 1940 Insertion", draft-herbert-6man-eh-attrib-00 (work in 1941 progress), December 2019. 1943 [I-D.ietf-6man-segment-routing-header] 1944 Filsfils, C., Dukes, D., Previdi, S., Leddy, J., 1945 Matsushima, S., and D. Voyer, "IPv6 Segment Routing Header 1946 (SRH)", draft-ietf-6man-segment-routing-header-26 (work in 1947 progress), October 2019. 1949 [I-D.ietf-detnet-ip] 1950 Varga, B., Farkas, J., Berger, L., Fedyk, D., Malis, A., 1951 and S. Bryant, "DetNet Data Plane: IP", draft-ietf-detnet- 1952 ip-05 (work in progress), February 2020. 1954 [I-D.ietf-detnet-mpls] 1955 Varga, B., Farkas, J., Berger, L., Fedyk, D., Malis, A., 1956 Bryant, S., and J. Korhonen, "DetNet Data Plane: MPLS", 1957 draft-ietf-detnet-mpls-05 (work in progress), February 1958 2020. 1960 [I-D.ietf-detnet-security] 1961 Mizrahi, T., Grossman, E., Hacker, A., Das, S., Dowdell, 1962 J., Austad, H., and N. Finn, "Deterministic Networking 1963 (DetNet) Security Considerations", draft-ietf-detnet- 1964 security-08 (work in progress), February 2020. 1966 [I-D.ietf-ippm-ioam-data] 1967 Brockners, F., Bhandari, S., Pignataro, C., Gredler, H., 1968 Leddy, J., Youell, S., Mizrahi, T., Mozes, D., Lapukhov, 1969 P., remy@barefootnetworks.com, r., daniel.bernier@bell.ca, 1970 d., and J. Lemon, "Data Fields for In-situ OAM", draft- 1971 ietf-ippm-ioam-data-08 (work in progress), October 2019. 1973 [I-D.ietf-tsvwg-intserv-multiple-tspec] 1974 Polk, J. and S. Dhesikan, "Integrated Services (IntServ) 1975 Extension to Allow Signaling of Multiple Traffic 1976 Specifications and Multiple Flow Specifications in 1977 RSVPv1", draft-ietf-tsvwg-intserv-multiple-tspec-02 (work 1978 in progress), February 2013. 1980 [NET2030SubG2] 1981 ITU-T FGNET2030, "New Services and Capabilities for 1982 Network 2030: Description, Technical Gap and Performance 1983 Target Analysis", October 2019, . 1987 [RFC1347] Callon, R., "TCP and UDP with Bigger Addresses (TUBA), A 1988 Simple Proposal for Internet Addressing and Routing", 1989 RFC 1347, DOI 10.17487/RFC1347, June 1992, 1990 . 1992 [RFC2205] Braden, R., Ed., Zhang, L., Berson, S., Herzog, S., and S. 1993 Jamin, "Resource ReSerVation Protocol (RSVP) -- Version 1 1994 Functional Specification", RFC 2205, DOI 10.17487/RFC2205, 1995 September 1997, . 1997 [RFC2460] Deering, S. and R. Hinden, "Internet Protocol, Version 6 1998 (IPv6) Specification", RFC 2460, DOI 10.17487/RFC2460, 1999 December 1998, . 2001 [RFC2507] Degermark, M., Nordgren, B., and S. Pink, "IP Header 2002 Compression", RFC 2507, DOI 10.17487/RFC2507, February 2003 1999, . 2005 [RFC2711] Partridge, C. and A. Jackson, "IPv6 Router Alert Option", 2006 RFC 2711, DOI 10.17487/RFC2711, October 1999, 2007 . 2009 [RFC3095] Bormann, C., Burmeister, C., Degermark, M., Fukushima, H., 2010 Hannu, H., Jonsson, L-E., Hakenberg, R., Koren, T., Le, 2011 K., Liu, Z., Martensson, A., Miyazaki, A., Svanbro, K., 2012 Wiebke, T., Yoshimura, T., and H. Zheng, "RObust Header 2013 Compression (ROHC): Framework and four profiles: RTP, UDP, 2014 ESP, and uncompressed", RFC 3095, DOI 10.17487/RFC3095, 2015 July 2001, . 2017 [RFC3208] Speakman, T., Crowcroft, J., Gemmell, J., Farinacci, D., 2018 Lin, S., Leshchiner, D., Luby, M., Montgomery, T., Rizzo, 2019 L., Tweedly, A., Bhaskar, N., Edmonstone, R., 2020 Sumanasekera, R., and L. Vicisano, "PGM Reliable Transport 2021 Protocol Specification", RFC 3208, DOI 10.17487/RFC3208, 2022 December 2001, . 2024 [RFC4080] Hancock, R., Karagiannis, G., Loughney, J., and S. Van den 2025 Bosch, "Next Steps in Signaling (NSIS): Framework", 2026 RFC 4080, DOI 10.17487/RFC4080, June 2005, 2027 . 2029 [RFC4193] Hinden, R. and B. Haberman, "Unique Local IPv6 Unicast 2030 Addresses", RFC 4193, DOI 10.17487/RFC4193, October 2005, 2031 . 2033 [RFC4302] Kent, S., "IP Authentication Header", RFC 4302, 2034 DOI 10.17487/RFC4302, December 2005, 2035 . 2037 [RFC5795] Sandlund, K., Pelletier, G., and L-E. Jonsson, "The RObust 2038 Header Compression (ROHC) Framework", RFC 5795, 2039 DOI 10.17487/RFC5795, March 2010, 2040 . 2042 [RFC6282] Hui, J., Ed. and P. Thubert, "Compression Format for IPv6 2043 Datagrams over IEEE 802.15.4-Based Networks", RFC 6282, 2044 DOI 10.17487/RFC6282, September 2011, 2045 . 2047 [RFC6398] Le Faucheur, F., Ed., "IP Router Alert Considerations and 2048 Usage", BCP 168, RFC 6398, DOI 10.17487/RFC6398, October 2049 2011, . 2051 [RFC6554] Hui, J., Vasseur, JP., Culler, D., and V. Manral, "An IPv6 2052 Routing Header for Source Routes with the Routing Protocol 2053 for Low-Power and Lossy Networks (RPL)", RFC 6554, 2054 DOI 10.17487/RFC6554, March 2012, 2055 . 2057 [RFC6790] Kompella, K., Drake, J., Amante, S., Henderickx, W., and 2058 L. Yong, "The Use of Entropy Labels in MPLS Forwarding", 2059 RFC 6790, DOI 10.17487/RFC6790, November 2012, 2060 . 2062 [RFC8578] Grossman, E., Ed., "Deterministic Networking Use Cases", 2063 RFC 8578, DOI 10.17487/RFC8578, May 2019, 2064 . 2066 [RFC8655] Finn, N., Thubert, P., Varga, B., and J. Farkas, 2067 "Deterministic Networking Architecture", RFC 8655, 2068 DOI 10.17487/RFC8655, October 2019, 2069 . 2071 [RFC8698] Zhu, X., Pan, R., Ramalho, M., and S. Mena, "Network- 2072 Assisted Dynamic Adaptation (NADA): A Unified Congestion 2073 Control Scheme for Real-Time Media", RFC 8698, 2074 DOI 10.17487/RFC8698, February 2020, 2075 . 2077 [UC] ITU-T FGNET2030, "Use Cases and Requirements for Network 2078 2030 Summary report "Representative use cases and key 2079 network requirements for Network 2030"", January 2020, 2080 . 2083 [WP] "Network 2030 - A Blueprint of Technology, Applications, 2084 and Market Drivers towards the Year 2030 and Beyond, a 2085 White Paper on Network 2030, ITU-T", May 2019, 2086 . 2089 Authors' Addresses 2091 Stewart Bryant 2092 Futurewei Technologies Inc. 2094 Email: sb@stewartbryant.com 2096 Uma Chunduri 2097 Futurewei Technologies Inc. 2099 Email: uma.chunduri@futurewei.com 2101 Toerless Eckert 2102 Futurewei Technologies Inc. 2104 Email: tte@cs.fau.de 2106 Alexander Clemm 2107 Futurewei Technologies Inc. 2109 Email: ludwig@clemm.org