idnits 2.17.1 draft-ietf-raw-architecture-01.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 1017 has weird spacing: '...-- Node z-- ...' == Line 1022 has weird spacing: '... Node z-- ...' -- The document date (28 July 2021) is 996 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Missing Reference: 'PCE' is mentioned on line 757, but not defined == Outdated reference: A later version (-08) exists of draft-ietf-raw-technologies-02 == Outdated reference: A later version (-11) exists of draft-ietf-raw-use-cases-02 -- Obsolete informational reference (is this intentional?): RFC 3272 (ref. 'TE') (Obsoleted by RFC 9522) == Outdated reference: A later version (-13) exists of draft-ietf-detnet-ip-oam-02 == Outdated reference: A later version (-13) exists of draft-ietf-bier-te-arch-10 == Outdated reference: A later version (-15) exists of draft-thubert-6man-ipv6-over-wireless-09 == Outdated reference: A later version (-06) exists of draft-ietf-raw-oam-support-02 == Outdated reference: A later version (-11) exists of draft-ietf-ippm-ioam-direct-export-05 == Outdated reference: A later version (-11) exists of draft-ietf-detnet-oam-framework-03 == Outdated reference: A later version (-15) exists of draft-mirsky-ippm-hybrid-two-step-11 == Outdated reference: A later version (-04) exists of draft-mirsky-ippm-epm-03 == Outdated reference: A later version (-15) exists of draft-mirsky-bfd-mpls-demand-09 == Outdated reference: A later version (-17) exists of draft-ietf-ippm-ioam-data-14 Summary: 0 errors (**), 0 flaws (~~), 16 warnings (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 RAW P. Thubert, Ed. 3 Internet-Draft Cisco Systems 4 Intended status: Informational G.Z. Papadopoulos 5 Expires: 29 January 2022 IMT Atlantique 6 L. Berger 7 LabN Consulting, L.L.C. 8 28 July 2021 10 Reliable and Available Wireless Architecture/Framework 11 draft-ietf-raw-architecture-01 13 Abstract 15 Reliable and Available Wireless (RAW) provides for high reliability 16 and availability for IP connectivity over a wireless medium. The 17 wireless medium presents significant challenges to achieve 18 deterministic properties such as low packet error rate, bounded 19 consecutive losses, and bounded latency. This document defines the 20 RAW Architecture following an OODA loop that involves OAM, PCE, PSE 21 and PAREO functions. It builds on the DetNet Architecture and 22 discusses specific challenges and technology considerations needed to 23 deliver DetNet service utilizing scheduled wireless segments and 24 other media, e.g., frequency/time-sharing physical media resources 25 with stochastic traffic. 27 Status of This Memo 29 This Internet-Draft is submitted in full conformance with the 30 provisions of BCP 78 and BCP 79. 32 Internet-Drafts are working documents of the Internet Engineering 33 Task Force (IETF). Note that other groups may also distribute 34 working documents as Internet-Drafts. The list of current Internet- 35 Drafts is at https://datatracker.ietf.org/drafts/current/. 37 Internet-Drafts are draft documents valid for a maximum of six months 38 and may be updated, replaced, or obsoleted by other documents at any 39 time. It is inappropriate to use Internet-Drafts as reference 40 material or to cite them other than as "work in progress." 42 This Internet-Draft will expire on 29 January 2022. 44 Copyright Notice 46 Copyright (c) 2021 IETF Trust and the persons identified as the 47 document authors. All rights reserved. 49 This document is subject to BCP 78 and the IETF Trust's Legal 50 Provisions Relating to IETF Documents (https://trustee.ietf.org/ 51 license-info) in effect on the date of publication of this document. 52 Please review these documents carefully, as they describe your rights 53 and restrictions with respect to this document. Code Components 54 extracted from this document must include Simplified BSD License text 55 as described in Section 4.e of the Trust Legal Provisions and are 56 provided without warranty as described in the Simplified BSD License. 58 Table of Contents 60 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 61 2. The RAW problem . . . . . . . . . . . . . . . . . . . . . . . 5 62 2.1. Terminology . . . . . . . . . . . . . . . . . . . . . . . 5 63 2.2. Reliability and Availability . . . . . . . . . . . . . . 8 64 2.2.1. High Availability Engineering Principles . . . . . . 8 65 2.2.2. Applying Reliability Concepts to Networking . . . . . 11 66 2.2.3. Reliability in the Context of RAW . . . . . . . . . . 11 67 2.3. Use Cases and Requirements Served . . . . . . . . . . . . 13 68 2.3.1. Radio Access Protection . . . . . . . . . . . . . . . 13 69 2.3.2. End-to-End Protection in a Wireless Mesh . . . . . . 14 70 2.4. Related Work at The IETF . . . . . . . . . . . . . . . . 14 71 3. The RAW Framework . . . . . . . . . . . . . . . . . . . . . . 15 72 3.1. Scope and Prerequisites . . . . . . . . . . . . . . . . . 16 73 3.2. Routing Time Scale vs. Forwarding Time Scale . . . . . . 16 74 3.3. Wireless Tracks . . . . . . . . . . . . . . . . . . . . . 18 75 3.4. Flow Identification vs. Path Identification . . . . . . . 19 76 3.5. Source-Routed vs. Distributed Forwarding Decision . . . . 21 77 3.6. Encapsulation and Decapsulation . . . . . . . . . . . . . 22 78 4. The RAW Architecture . . . . . . . . . . . . . . . . . . . . 22 79 4.1. The RAW Conceptual Model . . . . . . . . . . . . . . . . 22 80 4.2. The OODA Loop . . . . . . . . . . . . . . . . . . . . . . 24 81 4.3. Observe: The RAW OAM . . . . . . . . . . . . . . . . . . 25 82 4.3.1. DetNet OAM . . . . . . . . . . . . . . . . . . . . . 26 83 4.3.2. RAW Extensions . . . . . . . . . . . . . . . . . . . 27 84 4.3.3. Observed Metrics . . . . . . . . . . . . . . . . . . 27 85 4.4. Orient: The Path Computation Engine . . . . . . . . . . . 28 86 4.5. Decide: The Path Selection Engine . . . . . . . . . . . . 28 87 4.6. Act: The PAREO Functions . . . . . . . . . . . . . . . . 30 88 4.6.1. Packet Replication . . . . . . . . . . . . . . . . . 31 89 4.6.2. Packet Elimination . . . . . . . . . . . . . . . . . 32 90 4.6.3. Promiscuous Overhearing . . . . . . . . . . . . . . . 32 91 4.6.4. Constructive Interference . . . . . . . . . . . . . . 33 92 5. Security Considerations . . . . . . . . . . . . . . . . . . . 33 93 5.1. Forced Access . . . . . . . . . . . . . . . . . . . . . . 33 94 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 33 95 7. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 33 96 8. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 34 97 9. References . . . . . . . . . . . . . . . . . . . . . . . . . 34 98 9.1. Normative References . . . . . . . . . . . . . . . . . . 34 99 9.2. Informative References . . . . . . . . . . . . . . . . . 36 100 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 39 102 1. Introduction 104 Deterministic Networking is an attempt to emulate the properties of a 105 serial link over a switched fabric, by providing a bounded latency 106 and eliminating congestion loss, even when co-existing with best- 107 effort traffic. It is getting traction in various industries 108 including professional A/V, manufacturing, online gaming, and 109 smartgrid automation, enabling cost and performance optimizations 110 (e.g., vs. loads of P2P cables). 112 Bringing determinism in a packet network means eliminating the 113 statistical effects of multiplexing that result in probabilistic 114 jitter and loss. This can be approached with a tight control of the 115 physical resources to maintain the amount of traffic within a 116 budgetted volume of data per unit of time that fits the physical 117 capabilities of the underlying network, and the use of time-shared 118 resources (bandwidth and buffers) per circuit, and/or by shaping and/ 119 or scheduling the packets at every hop. 121 This innovation was initially introduced on wired networks, with IEEE 122 802.1 Time Sensitive networking (TSN) - for Ethernet LANs - and IETF 123 DetNet. But the wired and the wireless media are fundamentally 124 different at the physical level and in the possible abstractions that 125 can be built for IP [IPoWIRELESS]. Wireless networks operate on a 126 shared medium where uncontrolled interference, including the self- 127 induced multipath fading, cause random transmission losses and add 128 new dimensions to the statistical effects that affect reachability 129 and packet delivery. 131 To defeat those additional causes of transmission delay and loss, 132 Reliable and Available Wireless (RAW) leverages scheduled 133 transmissions with redundancy and diversity in the spatial, time, 134 code, and frequency domains. The challenge is to provide enough 135 diversity and redundancy to ensure the timely packet delivery while 136 preserving energy and optimizing the use of the shared spectrum. 138 While the generic "Deterministic Networking Problem Statement" 139 [RFC8557] applies to both the wired and the wireless media, the 140 methods to achieve RAW must extend those used to support time- 141 sensitive networking over wires, as a RAW solution has to address 142 less consistent transmissions, energy conservation and shared 143 spectrum efficiency. 145 Uncontrolled interference and transmission obstacles may impede the 146 wireless transmission, causing rapid variations of the throughput and 147 packet delivery ratio (PDR) of the link. This uncertainty limits the 148 volume and/or duration of traffic that can be safely transmitted on 149 the same link while conforming to a RAW Service Level Agreement 150 (SLA). 152 This increased complexity explains why the development of 153 deterministic wireless technologies has been lagging behind the 154 similar efforts for wired systems, both at the IEEE and the IETF. 155 But recent progress on scheduled radios such as TSCH and OFDMA 156 indicates that wireless is finally catching up at the lower layers. 157 Sitting at the layer above, RAW takes up the challenge of providing 158 highly available and reliable end-to-end performances in a network 159 with scheduled wireless segments. 161 RAW provides DetNet elements that are specialized for short range 162 radios. From this inheritance, RAW stays agnostic to the radio layer 163 underneath though the capability to schedule transmissions is 164 assumed. How the PHY is programmed to do so, and whether the radio 165 is single-hop or meshed, are unknown at the IP layer and not part of 166 the RAW abstraction. 168 The "Deterministic Networking Architecture" [RFC8655] is composed of 169 three planes: the Application (User) Plane, the Controller Plane, and 170 the Network Plane. The RAW Architecture extends the DetNet Network 171 Plane, to accommodate one or multiple hops of homogeneous or 172 heterogeneous wireless technologies, e.g. a Wi-Fi6 Mesh or parallel 173 CBRS access links federated by a 5G backhaul. 175 The establishment of a path is not in-scope for RAW. It may be the 176 product of a centralized Controller Plane as described for DetNet. 177 As opposed to wired networks, the action of installing a path over a 178 set of wireless links may be very slow relative to the speed at which 179 the radio conditions vary, and it makes sense in the wireless case to 180 provide redundant forwarding solutions along a complex path and to 181 leave it to the Network Plane to select which of those forwarding 182 solutions are to be used for a given packet based on the current 183 conditions. 185 RAW distinguishes the longer time scale at which routes are computed 186 from the the shorter forwarding time scale where per-packet decisions 187 are made. RAW operates within the Network Plane at the forwarding 188 time scale on one DetNet flow over a complex path called a Track. 189 The Track is preestablished and installed by means outside of the 190 scope of RAW; it may be strict or loose depending on whether each or 191 just a subset of the hops are observed and controlled by RAW. 193 The RAW Architecture is structured as an OODA Loop (Observe, Orient, 194 Decide, Act). It involves: 196 1. Network Plane measurement protocols for Operations, 197 Administration and Maintenance (OAM) to Observe some or all hops 198 along a Track as well as the end-to-end packet delivery 200 2. Controller plane elements to reports the links statistics to a 201 Path computation Element (PCE) in a centralized controller that 202 computes and installs the Tracks and provides meta data to Orient 203 the routing decision 205 3. A Runtime distributed Path Selection Engine (PSE) thar Decides 206 which subTrack to use for the next packet(s) that are routed 207 along the Track 209 4. Packet (hybrid) ARQ, Replication, Elimination and Ordering 210 Dataplane actions that operate at the DetNet Service Layer to 211 increase the reliability of the end-to-end transmission. The RAW 212 architecture also covers in-situ signalling when the decision is 213 Acted by a node that down the Track from the PSE. 215 The overall OODA Loop optimizes the use of redundancy to achieve the 216 required reliability and availability Service Level Agreement (SLA) 217 while minimizing the use of constrained resources such as spectrum 218 and battery. 220 2. The RAW problem 222 2.1. Terminology 224 RAW reuses terminology defined for DetNet in the "Deterministic 225 Networking Architecture" [RFC8655], e.g., PREOF for Packet 226 Replication, Elimination and Ordering Functions. 228 RAW also reuses terminology defined for 6TiSCH in [6TiSCH-ARCHI] such 229 as the term Track. A Track as a complex path with associated PAREO 230 operations. The concept is abstract to the underlaying technology 231 and applies to any fully or partially wireless mesh, including, e.g., 232 a Wi-Fi mesh. RAW specifies strict and loose Tracks depending on 233 whether the path is fully controlled by RAW or traverses an opaque 234 network where RAW cannot observe and control the individual hops. 236 RAW uses the following terminology: 238 OODA: Observe, Orient, Decide, Act. The OODA Loop is a conceptual 239 cyclic model developed by USAF Colonel John Boyd, and that is 240 applicable in multiple domains where agility can provide benefits 241 against brute force. 243 PAREO: Packet (hybrid) ARQ, Replication, Elimination and Ordering. 244 PAREO is a superset Of DetNet's PREOF that includes radio-specific 245 techniques such as short range broadcast, MUMIMO, constructive 246 interference and overhearing, which can be leveraged separately or 247 combined to increase the reliability. 249 Flow: A collection of consecutive packets that must be placed on the 250 same Track to receive an equivalent treatment from Ingress to 251 Egress within the Track. Multiple flows may be transported along 252 the same Track. The subTrack that is selected for the flow may 253 change over time under the control of the PSE. 255 Track: A networking graph that can be used as a "path" to transport 256 RAW packets with equivalent treatment; as opposed to the usual 257 understanding of a path (see for instance the definition of "path" 258 in section 1.1 of [RFC9049]), a Track may fork and rejoin to 259 enable the PAREO operations. 261 In DetNet [RFC8655] terms, a Track has the following properties: 263 * A Track has one Ingress and one Egress nodes, which operate as 264 DetNet Edge nodes. 266 * A Track is reversible, meaning that packets can be routed 267 against the flow of data packets, e.g., to carry OAM 268 measurements or control messages back to the Ingress. 270 * The vertices of the Track are DetNet Relay nodes that operate 271 at the DetNet Service sublayer and provide the PAREO functions. 273 * The topological edges of the graph are serial sequences of 274 DetNet Transit nodes that operate at the DetNet Forwarding 275 sublayer. 277 SubTrack: A Track within a Track. The RAW PSE selects a subTrack on 278 a per-packet or a per-collection of packets basis to provide the 279 desired reliability for the transported flows. 281 Segment: A serial path formed by a topological edge of a Track. 282 East-West Segments are oriented from Ingress (East) to Egress 283 (West). North/South Segments can be bidirectional; to avoid 284 loops, measures must be taken to ensure that a given packet flows 285 either Northwards or Southwards along a bidirectional Segment, but 286 never bounces back. 288 Flapping: In the context of RAW, a link flaps when the reliability 289 of the wireless connectivity drops abruptly for a short period of 290 time, typically of a subsecond to seconds duration. 292 OAM: OAM stands for Operations, Administration, and Maintenance, and 293 covers the processes, activities, tools, and standards involved 294 with operating, administering, managing and maintaining any 295 system. This document uses the terms Operations, Administration, 296 and Maintenance, in conformance with the 'Guidelines for the Use 297 of the "OAM" Acronym in the IETF' [RFC6291] and the system 298 observed by the RAW OAM is the Track. 300 Active OAM: See [RFC7799]. In the context of RAW, Active OAM is 301 used to observe a particular Track, subTrack, or Segment of a 302 Track regardless of whether it is used for traffic at that time. 304 In-Band OAM: An active OAM packet is considered in-band for the 305 monitored Track when it traverses the same set of links and 306 interfaces and if the OAM packet receives the same QoS and PAREO 307 treatment as the packets of the data flows that are injected in 308 the Track. 310 Out-of-Band OAM: Out-of-band OAM is an active OAM whose path is not 311 topologically congruent to the Track, or its test packets receive 312 a QoS and/or PAREO treatment that is different from that of the 313 packets of the data flows that are injected in the Track, or both. 315 Limited OAM: An active OAM packet is a Limited OAM packet when it 316 observes the RAW operation over a node, a segment, or a subTrack 317 of the Track, though not from Ingress to Egress. It is injected 318 in the datapath and extracted from the datapath around the 319 particular function or subnetwork (e.g., around a relay providing 320 a service layer replication point) that is being tested. 322 Upstream OAM: An upstream OAM packet is an Out-of-Band OAM packet 323 that traverses the Track from egress to ingress on the reverse 324 direction, to capture and report OAM measurements upstream. The 325 collection may capture all information along the whole Track, or 326 it may only learn select data across all, or only a particular 327 subTrack, or Segment of a Track. 329 [DetNet-OAM] provides additional terminology related to OAM in the 330 context of DetNet and by extension of RAW, whereas [RFC7799] defines 331 the Active, Passive, and Hybrid OAM methods. 333 In the context of the RAW work, Reliability and Availability are 334 defined as follows: 336 Reliability: Reliability is a measure of the probability that an 337 item will perform its intended function for a specified interval 338 under stated conditions. For RAW, the service that is expected is 339 delivery within a bounded latency and a failure is when the packet 340 is either lost or delivered too late. RAW expresses reliability 341 in terms of Mean Time Between Failure (MTBF) and Maximum 342 Consecutive Failures (MCF). More in [NASA]. 344 Availability: Availability is a measure of the relative amount of 345 time where a path operates in stated condition, in other words 346 (uptime)/(uptime+downtime). Because a serial wireless path may 347 not be good enough to provide the required reliability, and even 2 348 parallel paths may not be over a longer period of time, the RAW 349 availability implies a path that is a lot more complex than what 350 DetNet typically envisages (a Track). 352 Residence Time: A residence time (RT) is defined as the time period 353 between the reception of a packet starts and the transmission of 354 the packet begins. In the context of RAW, RT is useful for a 355 transit node, not ingress or egress. 357 2.2. Reliability and Availability 359 2.2.1. High Availability Engineering Principles 361 The reliability criteria of a critical system pervade through its 362 elements, and if the system comprises a data network then the data 363 network is also subject to the inherited reliability and availability 364 criteria. It is only natural to consider the art of high 365 availability engineering and apply it to wireless communications in 366 the context of RAW. 368 There are three principles [pillars] of high availability 369 engineering: 371 1. elimination of single points of failure 372 2. reliable crossover 373 3. prompt detection of failures as they occur. 375 These principles are common to all high availability systems, not 376 just ones with Internet technology at the center. Examples of both 377 non-Internet and Internet are included. 379 2.2.1.1. Elimination of Single Points of Failure 381 Physical and logical components in a system happen to fail, either as 382 the effect of wear and tear, when used beyond acceptable limits, or 383 due to a software bug. It is necessary to decouple component failure 384 from system failure to avoid the latter. This allows failed 385 components to be restored while the rest of the system continues to 386 function. 388 IP Routers leverage routing protocols to compute alternate routes in 389 case of a failure. There is a rather open-ended issue over alternate 390 routes -- for example, when links are cabled through the same 391 conduit, they form a shared risk link group (SRLG), and will share 392 the same fate if the bundle is cut. The same effect can happen with 393 virtual links that end up in a same physical transport through the 394 games of encapsulation. In a same fashion, an interferer or an 395 obstacle may affect multiple wireless transmissions at the same time, 396 even between different sets of peers. 398 Intermediate network Nodes such as routers, switches and APs, wire 399 bundles and the air medium itself can become single points of 400 failure. For High Availability, it is thus required to use 401 physically link- and Node-disjoint paths; in the wireless space, it 402 is also required to use the highest possible degree of diversity in 403 the transmissions over the air to combat the additional causes of 404 transmission loss. 406 From an economics standpoint, executing this principle properly 407 generally increases capitalization expense because of the redundant 408 equipment. In a constrained network where the waste of energy and 409 bandwidth should be minimized, an excessive use of redundant links 410 must be avoided; for RAW this means that the extra bandwidth must be 411 used wisely and with parcimony. 413 2.2.1.2. Reliable Crossover 415 Having a backup equipment has a limited value unless it can be 416 reliably switched into use within the down-time parameters. IP 417 Routers execute reliable crossover continuously because the routers 418 will use any alternate routes that are available [RFC0791]. This is 419 due to the stateless nature of IP datagrams and the dissociation of 420 the datagrams from the forwarding routes they take. The "IP Fast 421 Reroute Framework" [FRR] analyzes mechanisms for fast failure 422 detection and path repair for IP Fast-Reroute, and discusses the case 423 of multiple failures and SRLG. Examples of FRR techniques include 424 Remote Loop-Free Alternate [RLFA-FRR] and backup label-switched path 425 (LSP) tunnels for the local repair of LSP tunnels using RSVP-TE 426 [RFC4090]. 428 Deterministic flows, on the contrary, are attached to specific paths 429 where dedicated resources are reserved for each flow. This is why 430 each DetNet path must inherently provide sufficient redundancy to 431 provide the guaranteed SLA at all times. The DetNet PREOF typically 432 leverages 1+1 redundancy whereby a packet is sent twice, over non- 433 congruent paths. This avoids the gap during the fast reroute 434 operation, but doubles the traffic in the network. 436 In the case of RAW, the expectation is that multiple transient faults 437 may happen in overlapping time windows, in which case the 1+1 438 redundancy with delayed reestablishment of the second path will not 439 provide the required guarantees. The Data Plane must be configured 440 with a sufficient degree of redundancy to select an alternate 441 redundant path immediately upon a fault, without the need for a slow 442 intervention from the controller plane. 444 2.2.1.3. Prompt Notification of Failures 446 The execution of the two above principles is likely to render a 447 system where the user will rarely see a failure. But someone needs 448 to in order to direct maintenance. 450 There are many reasons for system monitoring (FCAPS for fault, 451 configuration, accounting, performance, security is a handy mental 452 checklist) but fault monitoring is sufficient reason. 454 "An Architecture for Describing Simple Network Management Protocol 455 (SNMP) Management Frameworks" [STD 62] describes how to use SNMP to 456 observe and correct long-term faults. 458 "Overview and Principles of Internet Traffic Engineering" [TE] 459 discusses the importance of measurement for network protection, and 460 provides abstract an method for network survivability with the 461 analysis of a traffic matrix as observed by SNMP, probing techniques, 462 FTP, IGP link state advertisements, and more. 464 Those measurements are needed in the context of RAW to inform the 465 controller and make the long term reactive decision to rebuild a 466 complex path. But RAW itself operates in the Network Plane at a 467 faster time scale. To act on the Data Plane, RAW needs live 468 information from the Operational Plane , e.g., using Bidirectional 469 Forwarding Detection [BFD] and its variants (bidirectional and remote 470 BFD) to protect a link, and OAM techniques to protect a path. 472 2.2.2. Applying Reliability Concepts to Networking 474 The terms Reliability and Availability are defined for use in RAW in 475 Section 2.1 and the reader is invited to read [NASA] for more details 476 on the general definition of Reliability. Practically speaking a 477 number of nines is often used to indicate the reliability of a data 478 link, e.g., 5 nines indicate a Packet Delivery Ratio (PDR) of 479 99.999%. 481 This number is typical in a wired environment where the loss is due 482 to a random event such as a solar particle that affects the 483 transmission of a particular frame, but does not affect the previous 484 or next frame, nor frames transmitted on other links. Note that the 485 QoS requirements in RAW may include a bounded latency, and a packet 486 that arrives too late is a fault and not considered as delivered. 488 For a periodic networking pattern such as an automation control loop, 489 this number is proportional to the Mean Time Between Failures (MTBF). 490 When a single fault can have dramatic consequences, the MTBF 491 expresses the chances that the unwanted fault event occurs. In data 492 networks, this is rarely the case. Packet loss cannot never be fully 493 avoided and the systems are built to resist to one loss, e.g., using 494 redundancy with Retries (HARQ) or Packet Replication and Elimination 495 (PRE), or, in a typical control loop, by linear interpolation from 496 the previous measurements. 498 But the linear interpolation method cannot resist multiple 499 consecutive losses, and a high MTBF is desired as a guarantee that 500 this will not happen, IOW that the number of losses-in-a-row can be 501 bounded. In that case, what is really desired is a Maximum 502 Consecutive Failures (MCF). If the number of losses in a row passes 503 the MCF, the control loop has to abort and the system, e.g., the 504 production line, may need to enter an emergency stop condition. 506 Engineers that build automated processes may use the network 507 reliability expressed in nines or as an MTBF as a proxy to indicate 508 an MCF, e.g., as described in section 7.4 of the "Deterministic 509 Networking Use Cases" [RFC8578]. 511 2.2.3. Reliability in the Context of RAW 513 In contrast with wired networks, errors in transmission are the 514 predominant source of packet loss in wireless networks. 516 The root cause for the loss may be of multiple origins, calling for 517 the use of different forms of diversity: 519 Multipath Fading: A destructive interference by a reflection of the 520 original signal. 522 A radio signal may be received directly (line-of-sight) and/or as 523 a reflection on a physical structure (echo). The reflections take 524 a longer path and are delayed by the extra distance divided by the 525 speed of light in the medium. Depending on the frequency, the 526 echo lands with a different phase which may add up to 527 (constructive interference) or cancel the direct signal 528 (destructive interference). 530 The affected frequencies depend on the relative position of the 531 sender, the receiver, and all the reflecting objects in the 532 environment. A given hop will suffer from multipath fading for 533 multiple packets in a row till the something moves that changes 534 the reflection patterns. 536 Co-channel Interference: Energy in the spectrum used for the 537 transmission confuses the receiver. 539 The wireless medium itself is a Shared Risk Link Group (SRLG) for 540 nearby users of the same spectrum, as an interference may affect 541 multiple co-channel transmissions between different peers within 542 the interference domain of the interferer, possibly even when they 543 use different technologies. 545 Obstacle in Fresnel Zone: The optimal transmission happens when the 546 Fresnel Zone between the sender and the receiver is free of 547 obstacles. 549 As long as a physical object (e.g., a metallic trolley between 550 peers) that affects the transmission is not removed, the quality 551 of the link is affected. 553 In an environment that is rich of metallic structures and mobile 554 objects, a single radio link will provide a fuzzy service, meaning 555 that it cannot be trusted to transport the traffic reliably over a 556 long period of time. 558 Transmission losses are typically not independent, and their nature 559 and duration are unpredictable; as long as a physical object (e.g., a 560 metallic trolley between peers) that affects the transmission is not 561 removed, or as long as the interferer (e.g., a radar) keeps 562 transmitting, a continuous stream of packets will be affected. 564 The key technique to combat those unpredictable losses is diversity. 565 Different forms of diversity are necessary to combat different causes 566 of loss and the use of diversity must be maximized to optimize the 567 PDR. 569 A single packet may be sent at different times (time diversity) over 570 diverse paths (spatial diversity) that rely on diverse radio channels 571 (frequency diversity) and diverse PHY technologies, e.g., narrowband 572 vs. spread spectrum, or diverse codes. Using time diversity will 573 defeat short-term interferences; spatial diversity combats very local 574 causes such as multipath fading; narrowband and spread spectrum are 575 relatively innocuous to one another and can be used for diversity in 576 the presence of the other. 578 2.3. Use Cases and Requirements Served 580 In order to focus on real-worlds issues and assert the feasibility of 581 the proposed capabilities, RAW focuses on selected technologies that 582 can be scheduled at the lower layers: IEEE Std. 802.15.4 timeslotted 583 channel hopping (TSCH), 3GPP 5G ultra-reliable low latency 584 communications (URLLC), IEEE 802.11ax/be where 802.11be is extreme 585 high throughput (EHT), and L-band Digital Aeronautical Communications 586 System (LDACS). See [RAW-TECHNOS] for more. 588 "Deterministic Networking Use Cases" [RFC8578] presents a number of 589 wireless use cases including Wireless, such as application to 590 Industrial Applications, Pro-Audio, and SmartGrid Automation. 591 [RAW-USE-CASES] adds a number of use cases that demonstrate the need 592 for RAW capabilities for new applications such as Pro-Gaming and 593 drones. The use cases can be abstracted in two families, Loose 594 Protection, e.g., protecting the first hop in Radio Access Protection 595 and Strict Protection, e.g., providing End-to-End Protection in a 596 wireless mesh. 598 2.3.1. Radio Access Protection 600 To maintain the required SLA at all times, a wireless Host may use 601 more than one Radio Access Network (RAN) in parallel. 603 ... .. 604 RAN 1 ----- ... .. ... 605 / . .. .... 606 +--------+ / . .... +-----------+ 607 |Wireless|- . ..... | Service | 608 | Device |-***-- RAN 2 -- . Internet ....---| / | 609 |(STA/UE)|- .. ..... |Application| 610 +--------+ $$$ . ....... +-----------+ 611 \ ... ... ..... 612 RAN n -------- ... ..... 614 *** = flapping at this time $$$ expensive 616 Figure 1: Radio Access Protection 618 The RANs may be heterogeneous, e.g., 3GPP 5G [RAW-5G] and Wi-Fi 619 [RAW-TECHNOS] for high-speed communication, in which case a Layer-3 620 abstraction becomes useful to select which of the RANs are used at a 621 particular point of time, and the amount of traffic that is 622 distributed over each RAN. 624 The idea is that the rest of the path to the destination(s) is 625 protected separately (e.g., uses non-congruent paths, leverages 626 DetNet / TSN, etc...) and is a lot more reliable, e.g., wired. In 627 that case, RAW observes the reliability of the end-to-end operation 628 through each of the RANs but only observes and controls the wireless 629 operation the first hop. 631 A variation of that use case has a pair of wireless Hosts connected 632 over a wired core / backbone network. In that case, RAW observes and 633 controls the Ingress and Egress RANs, while neglecting the hops in 634 the core. The resulting loose Track may be instantiated, e.g., using 635 tunneling or loose source routing between the RANs. 637 2.3.2. End-to-End Protection in a Wireless Mesh 639 In radio technologies that support mesh networking (e.g., Wi-Fi and 640 TSCH), a Track is a complex path with distributed PAREO capabilities. 641 In that case, RAW operates through the multipath and makes decisions 642 either at the Ingress or at every hop (more in Section 3.3). 644 A-------B-------C-----D 645 / \ / / \ 646 Ingress ----M-------N--zzzzz--- Egress 647 \ \ / / 648 P--zzz--Q-------------R 650 zzz = flapping now 652 Figure 2: End-to-End Protection 654 The Protection may be imposed by the source based on end-to-end OAM, 655 or performed hop-by-hop, in which case the OAM must enables the 656 intermediate Nodes to estimate the quality of the rest of the 657 feasible paths in the remainder of the Track to the destination. 659 2.4. Related Work at The IETF 661 RAW intersects with protocols or practices in development at the IETF 662 as follows: 664 * The Dynamic Link Exchange Protocol (DLEP) [RFC8175] from [MANET] 665 can be leveraged at each hop to derive generic radio metrics 666 (e.g., based on LQI, RSSI, queueing delays and ETX) on individual 667 hops. 669 * [detnet] provides an OAM framework with [DetNet-OAM] that applies 670 within the DetNet dataplane described in [DetNet-DP],which is 671 typically based on MPLS or IPv6 pseudowires. 673 * [BFD] detect faults in the path between an Ingress and an Egress 674 forwarding engines, but is unaware of the complexity of a path 675 with replication, and expects bidirectionality. BFD asynchronous 676 mode considers delivery as success whereas with DetNet and RAW, 677 the bounded latency can be as important as the delivery itself, 678 and delivering too late is actually a failure. Note that the BFD 679 Demand mode with unsolicited notifications may be more suitable 680 then the Asynchronous BFD mode. The use of the Demand mode in 681 MPLS is analyzed in [I-D.mirsky-bfd-mpls-demand] and similar 682 considerations could apply to IP as well. 684 * [SPRING] and [BIER] define in-band signaling that influences the 685 routing when decided at the head-end on the path. There's already 686 one RAW-related draft at BIER [BIER-PREF] more may follow. RAW 687 will need new in-band signaling when the decision is distributed, 688 e.g., required chances of reliable delivery to destination within 689 latency. This signaling enables relays to tune retries and 690 replication to meet the required SLA. 692 * [CCAMP] defines protocol-independent metrics and parameters 693 (measurement attributes) for describing links and paths that are 694 required for routing and signaling in technology-specific 695 networks. RAW would be a source of requirements for CCAMP to 696 define metrics that are significant to the focus radios. 698 * [IPPM] develops and maintains standard metrics that can be applied 699 to the quality, performance, and reliability of Internet data 700 delivery services and applications running over transport layer 701 protocols (e.g. TCP, UDP) over IP. 703 3. The RAW Framework 704 3.1. Scope and Prerequisites 706 A prerequisite to the RAW operation is that an end-to-end routing 707 function computes a complex sub-topology along which forwarding can 708 happen between a source and one or more destinations. The concept of 709 Track is specified in the 6TiSCH Architecture [6TiSCH-ARCHI] to 710 represent that complex sub-topology. Tracks provide a high degree of 711 redundancy and diversity and enable the DetNet PREOF, network coding, 712 and possibly RAW specific techniques such as PAREO, leveraging 713 frequency diversity, time diversity, and possibly other forms of 714 diversity as well. 716 How the routing operation (e.g., PCE) in the Controller Plane 717 computes the Track is out of scope for RAW. The scope of the RAW 718 operation is one Track, and the goal of the RAW operation is to 719 optimize the use of the Track at the forwarding timescale to maintain 720 the expected SLA while optimizing the usage of constrained resources 721 such as energy and spectrum. 723 Another prerequisite is that an IP link can be established over the 724 radio with some guarantees in terms of service reliability, e.g., it 725 can be relied upon to transmit a packet within a bounded latency and 726 provides a guaranteed BER/PDR outside rare but existing transient 727 outage windows that can last from split seconds to minutes. The 728 radio layer can be programmed with abstract parameters, and can 729 return an abstract view of the state of the Link to help the Network 730 Layer forwarding decision (think DLEP from MANET). 732 How the radio interface manages its lower layers is out of control 733 and out of scope for RAW. In the same fashion, the non-RAW portion 734 along a loose Track is by definition out of control and out of scope 735 for RAW. Whether it is a single hop or a mesh is also unknown and 736 out of scope. 738 3.2. Routing Time Scale vs. Forwarding Time Scale 740 With DetNet, the Controller Plane Function that handles the routing 741 computation and maintenance (the PCE) can be centralized and can 742 reside outside the network. In a wireless mesh, the path to the PCE 743 can be expensive and slow, possibly going across the whole mesh and 744 back. Reaching to the PCE can also be slow in regards to the speed 745 of events that affect the forwarding operation at the radio layer. 747 Due to that cost and latency, the Controller Plane is not expected to 748 be sensitive/reactive to transient changes. The abstraction of a 749 link at the routing level is expected to use statistical metrics that 750 aggregate the behavior of a link over long periods of time, and 751 represent its properties as shades of gray as opposed to numerical 752 values such as a link quality indicator, or a boolean value for 753 either up or down. 755 +----------------+ 756 | Controller | 757 | [PCE] | 758 +----------------+ 759 ^ 760 | 761 Slow 762 | 763 _-._-._-._-._-._-. | ._-._-._-._-._-._-._-._-._-._-._-._- 764 _-._-._-._-._-._-._-. | _-._-._-._-._-._-._-._-._-._-._-._- 765 | 766 Expensive 767 | 768 .... | ....... 769 .... . | . ....... 770 .... v ... 771 .. A-------B-------C---D .. 772 ... / \ / / \ .. 773 . I ----M-------N--***-- E .. 774 .. \ \ / / ... 775 .. P--***--Q----------R .... 776 .. .... 777 . <----- Fast -------> .... 778 ....... .... 779 ................. 781 *** = flapping at this time 783 Figure 3: Time Scales 785 In the case of wireless, the changes that affect the forwarding 786 decision can happen frequently and often for short durations, e.g., a 787 mobile object moves between a transmitter and a receiver, and will 788 cancel the line of sight transmission for a few seconds, or a radar 789 measures the depth of a pool and interferes on a particular channel 790 for a split second. 792 There is thus a desire to separate the long term computation of the 793 route and the short term forwarding decision. In that model, the 794 routing operation computes a complex Track that enables multiple Non- 795 Equal Cost Multi-Path (N-ECMP) forwarding solutions, and leaves it to 796 the Data Plane to make the per-packet decision of which of these 797 possibilities should be used. 799 In the wired world, and more specifically in the context of Traffic 800 Engineering (TE), an alternate path can be used upon the detection of 801 a failure in the main path, e.g., using OAM in MPLS-TP or BFD over a 802 collection of SD-WAN tunnels. RAW formalizes a forwarding time scale 803 that is an order(s) of magnitude shorter than the controller plane 804 routing time scale, and separates the protocols and metrics that are 805 used at both scales. Routing can operate on long term statistics 806 such as delivery ratio over minutes to hours, but as a first 807 approximation can ignore flapping. On the other hand, the RAW 808 forwarding decision is made at the scale of the packet rate, and uses 809 information that must be pertinent at the present time for the 810 current transmission(s). 812 3.3. Wireless Tracks 814 The "6TiSCH Architecture" [6TiSCH-ARCHI] introduces the concept of 815 Track. RAW extends the concept to any wireless mesh technology, 816 including, e.g., Wi-Fi. A simple Track is composed of a direct 817 sequence of reserved hops to ensure the transmission of a single 818 packet from a source Node to a destination Node across a multihop 819 path. 821 A Complex Track provides multiple N-ECMP forwarding solutions. The 822 Complex Track enables to support multi-path redundant forwarding by 823 employing PRE functions [RFC8655] and the ingress and within the 824 Track. For example, a Complex Track may branch off and rejoin over 825 non-congruent segments. 827 In the context of RAW, some links or segments in the Track may be 828 reversible, meaning that they can be used in either direction. In 829 that case, an indication in the packet signals the direction of the 830 reversible links or segments that the packet traverses and thus 831 places a constraint that prevents loops from occurring. An 832 individual packet follows a destination-oriented directed acyclic 833 graph (DODAG) towards a destination Node inside the Complex Track. 835 3.4. Flow Identification vs. Path Identification 837 Section 4.7 of the DetNet Architecture [RFC8655] ties the app-flow 838 identification which is an application-layer concept with the network 839 path identification that depends on the networking technology by 840 "exporting of flow identification", e.g., to a MPLS label. 842 With RAW, this exporting operation is injective but not bijective. 843 e.g., a flow is fully placed within one RAW Track, but not all 844 packets along that Track are necessarily part of the same flow. For 845 instance, out-of-band OAM packets must circulate in the exact same 846 fashion as the flows that they observe. It results that the flow 847 identification that maps to an application layer flowat the network 848 layer must be separate from the path identification that is used to 849 forward a packet. 851 Section 3.4 of the DetNet data-plane framework [DetNet-DP] indicates 852 that for a DetNet IP Data Plane, a flow is identified by an IPv6 853 6-tuple. With RAW, that 6-tuple is not what indicates the Track, in 854 other words, the flow ID is not the Track ID. 856 For instance, the 6TiSCH Architecture [6TiSCH-ARCHI] uses a 857 combination of the address of the Egress End System and an instance 858 identifier in a Hop-by-hop option to indicate a Track. This way, if 859 a packet "escapes" the Track, it will reach the Track Egress point 860 through normal routing and be treated at the service layer through, 861 say, elimination and reordering. 863 The RAW service includes forwarding over a subset of the Links that 864 form the Track (a subTrack). Packets from the same or a different 865 flow that are routed through the same Track will not necessarily 866 traverse the same Links. The PSE selects a subTrack for a packet 867 based on the links that are preferred and those that should be 868 avoided at this time. 870 Each packet is forwarded within the subTrack that provides the best 871 adequation with the SLA of the flow and the energy and bandwidth 872 constraints of the network. 874 Flow 1 (6-tuple) ----+ 875 | 876 Flow 2 (6-tuple) ---+ | 877 | | 878 OAM -----------+ | | 879 | | | 880 | | | 881 | | | | | 882 | v v v | 883 | | 884 +---------+---------+ 885 | 886 | 887 Track i (Ingress IP Address, RPLinstanceId) 888 | 889 | 890 | 891 +---------+-----+--....-------+ 892 | | | 893 | | | 894 subTrack 1 subTrack 2 subTrack n 895 | | | 896 | | | 897 V V V 898 +-----------------------------------+ 899 | | 900 | Destination | 901 | | 902 +-----------------------------------+ 904 Figure 4: Flow Injection 906 With 6TiSCH, packets are tagged with the same (destination address, 907 instance ID) will experience the same RAW service regardless of the 908 IPv6 6-tuple that indicates the flow. The forwarding does not depend 909 on whether the packets transport application flows or OAM. In the 910 generic case, the Track or the subTrack can be signaled in the packet 911 through other means, e.g., encoded in the suffix of the destination 912 address as a Segment Routing Service Instruction [SR-ARCHI], or 913 leveraging Bit Index Explicit Replication [BIER] Traffic Engineering 914 [BIER-TE]. 916 3.5. Source-Routed vs. Distributed Forwarding Decision 918 Within a large routed topology, the route-over mesh operation builds 919 a particular complex Track with one source and one or more 920 destinations; within the Track, packets may follow different paths 921 and may be subject to RAW forwarding operations that include 922 replication, elimination, retries, overhearing and reordering. 924 The RAW forwarding decisions include the selection of points of 925 replication and elimination, how many retries can take place, and a 926 limit of validity for the packet beyond which the packet should be 927 destroyed rather than forwarded uselessly further down the Track. 929 The decision to apply the RAW techniques must be done quickly, and 930 depends on a very recent and precise knowledge of the forwarding 931 conditions within the complex Track. There is a need for an 932 observation method to provide the RAW Data Plane with the specific 933 knowledge of the state of the Track for the type of flow of interest 934 (e.g., for a QoS level of interest). To observe the whole Track in 935 quasi real time, RAW considers existing tools such as L2-triggers, 936 DLEP, BFD and leverages in-band and out-of-band OAM to capture and 937 report that information to the PSE. 939 One possible way of making the RAW forwarding decisions within a 940 Track is to position a unique PSE at the Ingress and express its 941 decision in-band in the packet, which requires the explicit signaling 942 of the subTrack within the Track. In that case, the RAW forwarding 943 operation along the Track is encoded by the source, e.g., by 944 indicating the subTrack in the Segment Routing (SRv6) Service 945 Instruction, or by leveraging BIER-TE such as done with [BIER-PREF]. 947 The alternate way is to operate the PSE in each forwarding Node, 948 which makes the RAW forwarding decisions for a packet on its own, 949 based on its knowledge of the expectation (timeliness and 950 reliability) for that packet and a recent observation of the rest of 951 the way across the possible paths based on OAM. Information about 952 the desired service should be placed in the packet and matched with 953 the forwarding Node's capabilities and policies. 955 In either case, a per-track/subTrack state is installed in all the 956 intermediate Nodes to recognize the packets that are following a 957 Track and determine the forwarding operation to be applied. 959 3.6. Encapsulation and Decapsulation 961 In the generic case where the Track Ingress Node is not the source of 962 the Packet, the Ingress Node needs to encapsulate IP-in-IP to ensure 963 that the Destination IP Address is that of the Egress Node and that 964 the necessary Headers (Routing Header, Segment Routing Header and/or 965 Hop-By-Hop Header) can be added to the packet to signal the Track or 966 the subTrack, conforming [IPv6] that discourages the insertion of a 967 Header on the fly. 969 In the specific case where the Ingress Node is the source of the 970 packet, the encapsulation can be avoided, provided that the source 971 adds the necessary headers and that the destination is set to the 972 Egress Node. Forwarding to a final destination beyond the Egress 973 Node is possible, e.g., with a Segment Routing Header that signals 974 the rest of the way. In that case a Hop-by-Hop Header is not 975 recommmended since its validity is within the Track only. 977 4. The RAW Architecture 979 4.1. The RAW Conceptual Model 981 RAW inherits the conceptual model described in section 4 of the 982 DetNet Architecture [RFC8655]. RAW extends the DetNet service layer 983 to provide additional agility against transmission loss. 985 A RAW Network Plane may be strict or loose, depending on whether RAW 986 observes and takes actions on all hops or not. For instance, the 987 packets between two wireless entities may be relayed over a wired 988 infrastructure such as a Wi-Fi extended service set (ESS) or a 5G 989 Core; in that case, RAW observes and control the transmission over 990 the wireless first and last hops, as well as end-to-end metrics such 991 as latency, jitter, and delivery ratio. This operation is loose 992 since the structure and properties of the wired infrastructure are 993 ignored, and may be either controlled by other means such as DetNet/ 994 TSN, or neglected in the face of the wireless hops. 996 A Controller Plane Function (CPF) called the Path Computation Element 997 (PCE) [RFC4655] interacts with RAW Nodes over a Southbound API. The 998 RAW Nodes are DetNet relays that are capable of additional diversity 999 mechanisms and measurement functions related to the radio interface, 1000 in particular the PAREO diversity mechanisms. 1002 The PCE defines a complex Track between an Ingress End System and an 1003 Egress End System, and indicates to the RAW Nodes where the PAREO 1004 operations may be actionned in the Network Plane. The Track may be 1005 expressed loosely to enable traversing a non-RAW subnetwork. In that 1006 case, the expectation is that the non-RAW subnetwork can be neglected 1007 in the RAW computation, that is, considered infinitely fast, reliable 1008 and/or available in comparison with the links between RAW nodes. 1010 CPF CPF CPF CPF 1012 Southbound API 1013 _-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._- 1014 _-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._- 1016 RAW --z RAW --z RAW --z RAW 1017 z-- Node z-- Node z-- Node z-- Node --z 1018 Ingress --z / / z-- Egress 1019 End \ \ .. . End 1020 Node ---z / / .. .. . z-- Node 1021 z-- RAW --z RAW ( non-RAW ) -- RAW --z 1022 Node z-- Node --- ( Nodes ) Node 1023 ... . 1024 --z wireless wired 1025 z-- link --- link 1027 Figure 5: RAW Nodes 1029 The Link-Layer metrics are reported to the PCE in a time-aggregated, 1030 e.g., statistical fashion. Example Link-Layer metrics include 1031 typical Link bandwidth (the medium speed depends dynamically on the 1032 PHY mode and the number of users sharing the spectrum) and average 1033 and mean squared deviation of availability and reliability figures 1034 such as Packet Delivery Ratio (PDR) over long periods of time. 1036 Based on those metrics, the PCE installs the Track with enough 1037 redundant forwarding solutions to ensure that the Network Plane can 1038 reliably deliver the packets within a System Level Agreement (SLA) 1039 associated to the flows that it transports. The SLA defines end-to- 1040 end reliability and availability requirements, where reliability may 1041 be expressed as a successful delivery in order and within a bounded 1042 delay of at least one copy of a packet. 1044 Depending on the use case and the SLA, the Track may comprise non-RAW 1045 segments, either interleaved inside the Track, or all the way to the 1046 Egress End Node (e.g., a server in the Internet). RAW observes the 1047 Lower-Layer Links between RAW nodes (typically, radio links) and the 1048 end-to-end Network Layer operation to decide at all times which of 1049 the PAREO diversity schemes is actioned by which RAW Nodes. 1051 Once a Track is established, per-segment and end-to-end reliability 1052 and availability statistics are periodically reported to the PCE to 1053 assure that the SLA can be met or have it recompute the Track if not. 1055 4.2. The OODA Loop 1057 The RAW Architecture is structured as an OODA Loop (Observe, Orient, 1058 Decide, Act). It involves: 1060 1. Network Plane measurement protocols for Operations, 1061 Administration and Maintenance (OAM) to Observe some or all hops 1062 along a Track as well as the end-to-end packet delivery, more in 1063 Section 4.3; 1065 2. Controller plane elements to reports the links statistics to a 1066 Path computation Element (PCE) in a centralized controller that 1067 computes and installs the Tracks and provides meta data to Orient 1068 the routing decision, more in Section 4.4; 1070 3. A Runtime distributed Path Selection Engine (PSE) thar Decides 1071 which subTrack to use for the next packet(s) that are routed 1072 along the Track, more in Section 4.5; 1074 4. Packet (hybrid) ARQ, Replication, Elimination and Ordering 1075 Dataplane actions that operate at the DetNet Service Layer to 1076 increase the reliability o fthe end-to-end transmission. The RAW 1077 architecture also covers in-situ signalling when the decision is 1078 Acted by a node that down the Track from the PSE, more in 1079 Section 4.6. 1081 +-------> Orient (PCE) --------+ 1082 | link stats, | 1083 | pre-trained model | 1084 | ... | 1085 | | 1086 | v 1087 Observe (OAM) Decide (PSE) 1088 ^ | 1089 | | 1090 | | 1091 +-------- Act (PAREO) <--------+ 1092 At DetNet 1093 Service sublayer 1095 Figure 6: The RAW OODA Loop 1097 The overall OODA Loop optimizes the use of redundancy to achieve the 1098 required reliability and availability Service Level Agreement (SLA) 1099 while minimizing the use of constrained resources such as spectrum 1100 and battery. 1102 4.3. Observe: The RAW OAM 1104 RAW In-situ OAM operation in the Network Plane may observe either a 1105 full Track or subTracks that are being used at this time. Active RAW 1106 OAM may be needed to observe the unused segments and evaluate the 1107 desirability of a rerouting decision. Finally, the RAW Service Layer 1108 Assurance may observe the individual PAREO operation of a relay node 1109 to ensure that it is conforming; this might require injecting an OAM 1110 packet at an upstream point inside the Track and extracting that 1111 packet at another point downstream before it reaches the egress. 1113 This observation feeds the RAW PSE that makes the decision on which 1114 PAREO function in actioned at which RAW Node, for one a small 1115 continuous series of packets. 1117 ... .. 1118 RAN 1 ----- ... .. ... 1119 / . .. .... 1120 +-------+ / . .. .... +------+ 1121 |Ingress|- . ..... |Egress| 1122 | End |------ RAN 2 -- . Internet ....---| End | 1123 |System |- .. ..... |System| 1124 +-------+ \ . ...... +------+ 1125 \ ... ... ..... 1126 RAN n -------- ... ..... 1128 <------------------> <--------------------> 1129 Observed by OAM Opaque to OAM 1131 Figure 7: Observed Links in Radio Access Protection 1133 In the case of a End-to-End Protection in a Wireless Mesh, the Track 1134 is strict and congruent with the path so all links are observed. 1135 Conversely, in the case of Radio Access Protection, the Track is 1136 Loose and in that case only the first hop is observed; the rest of 1137 the path is abstracted and considered infinitely reliable. 1139 In the case of the Radio Access Protection, only the first hop is 1140 protected; the loss of a packet that was sent over one of the 1141 possible first hops is attributed to that first hop, even if a 1142 particular loss effectively happens farther down the path. 1144 The Links that are not observed by OAM are opaque to it, meaning that 1145 the OAM information is carried across and possibly echoed as data, 1146 but there is no information capture in intermediate nodes. In the 1147 example above, the Internet is opaque and not controlled by RAW; 1148 still the RAW OAM measures the end-to-end latency and delivery ratio 1149 for packets sent via each if RAN 1, RAN 2 and RAN 3, and determines 1150 whether a packet should be sent over either or a collection of those 1151 access links. 1153 4.3.1. DetNet OAM 1155 [detnet] provides an OAM framework with [DetNet-OAM] that applies 1156 within the DetNet dataplane described in [DetNet-DP],which is 1157 typically based on MPLS or IPv6 pseudowires. How the framework 1158 applies to IPv6 is detailed in [DetNet-IP-OAM]. Within that 1159 framework, OAM messages follow the same forward path as the data 1160 packets and gather information about their individual treatment at 1161 each hop. When the destination receives an OAM message, it gets a 1162 view on the full path or at least of a segment of the path from the 1163 source of the flow. 1165 In-situ OAM (IOAM) adds telemetry information about the experience of 1166 one packet within the packet itself [I-D.ietf-ippm-ioam-data], with 1167 the caveats that the measurement and the consecutive update of the 1168 packet interfere with the operation being observed, e.g., may 1169 increase the latency of the packet for which it is measured and into 1170 which it is stamped. 1172 Note: IOAM and analogous on-path telemetry methods are capable of 1173 facilitating collection of useful telemetry information that 1174 characterizes the state of a system as experienced by the packet. 1175 But because of statistical character of a packet network, these 1176 methods may not be used to monitor the continuity of a path (Track) 1177 or proper connectivity of the Track (no leaking packets across 1178 Tracks). 1180 This effect can be alleviated by measuring on the fly but reporting 1181 later, e.g., by exporting the data as a separate management packet 1182 [I-D.ietf-ippm-ioam-direct-export]. 1183 [I-D.mirsky-ippm-hybrid-two-step] proposes an hybrid two-steps method 1184 (HTS) where a trigger message starts the measurement and a follow up 1185 along the Track packet gathers the measured data. 1187 "Error Performance Measurement" [I-D.mirsky-ippm-epm] uses Fault 1188 Management (FM) and Performance Management (PM) OAM mechanisms to 1189 determine availability/unavailability of a path according to 1190 predefined SLA. 1192 4.3.2. RAW Extensions 1194 Classical OAM typically measures information at the transmitter, 1195 e.g., residence time in the node or transmit queue size. With RAW, 1196 there is a need to combine information at the sender (number of 1197 retries) with that at the receiver (LQI, RSSI). This doubles the 1198 operating cost of an IAOM processing that would gather the experience 1199 of a single packet. 1201 The RAW PSE may be centralized at the Track Ingress, or distributed 1202 long the Track. Either way, the PSE needs instant information about 1203 the rest of the way to the destination over the possible next-hop 1204 adjacencies along the Track in order to decide how to perform simple 1205 forwarding, load balancing, and/or replication, as well as 1206 determining how much latency credit is available for ARQ. 1208 To provide that information timely, it makes sense that the OAM 1209 packets that gather instantaneous values from the radio senders and 1210 receivers at each hop flow on the reverse path and inform the PSE at 1211 the source and/or the PAREO relays about the state of the rest of the 1212 way. This is achieved using Reverse OAM packets that flow along the 1213 Reversed Track, West to East. 1215 Because the quality of transmission over a wireless medium varies 1216 continuously, it is important that RAW OAM captures the state of the 1217 medium across an adjacency over multiple transmission and over a 1218 recent period of time, whether the transmitted packets belong to this 1219 flow or another. Some of the measured information relates to the 1220 medium itself. In other words, the captured information does not 1221 only relate to the experience of one packet as is the case for IOAM, 1222 but also to the medium itself. This makes an approach like HTS more 1223 suitable as it can trigger the capture of multiple measurements over 1224 a short period of time. On the other hand, the PSE needs a 1225 continuous measurement stream where a single trigger is followed by a 1226 periodic follow up capture. 1228 In other words, the best suited OAM method to enable the PSE make 1229 accurate PAREO forwarding decisions is a periodic variation of the 1230 two-steps method flowing along the reverse Track, as an upstream OAM 1231 technique. [RAW-OAM] provides more information on the RAW OAM 1232 problem and solution approaches. 1234 4.3.3. Observed Metrics 1236 The Dynamic Link Exchange Protocol (DLEP) [RFC8175] from [MANET] can 1237 be leveraged at each hop to derive generic radio metrics (e.g., based 1238 on LQI, RSSI, queueing delays and ETX) on individual hops. 1240 Those lower-layer metrics are aggregated along a multihop segment 1241 into abstract layer 3 information that reflect the instant 1242 reliability and latency of the observed path. 1244 4.4. Orient: The Path Computation Engine 1246 RAW separates the path computation time scale at which a complex path 1247 is recomputed from the path selection time scale at which the 1248 forwarding decision is taken for one or a few packets (see in 1249 Section 3.2). 1251 The path computation is out of scope, but RAW expects that the 1252 Controller plane protocol that installs the Track also provides 1253 related knowledge in the form of meta data about the links, segments 1254 and possible subTracks. That meta data can be a pre-digested 1255 statistical model, and may include prediction of future flaps and 1256 packet loss, as well as recommended actions when that happens. 1258 The meta data may include: 1260 * Pre-Determined subTracks to match predictable error profiles 1262 * Pre-Trained models 1264 * Link Quality Statistics and their projected evolution 1266 The Track is installed with measurable objectives that are computed 1267 by the PCE to achieve the RAW SLA. The objectives can be expressed 1268 as any of maximum number of packet lost in a row, bounded latency, 1269 maximal jitter, maximum nmuber of interleaved out of order packets, 1270 average number of copies received at the elimination point, and 1271 maximal delay between the first and the last received copy of the 1272 same packet. 1274 4.5. Decide: The Path Selection Engine 1276 The RAW OODA Loop operates at the path selection time scale to 1277 provide agility vs. the brute force approach of flooding the whole 1278 Track. The OODA Loop controls, within the redundant solutions that 1279 are proposed by the PCE, which will be used for each packet to 1280 provide a Reliable and Available service while minimizing the waste 1281 of constrained resources. 1283 To that effect, RAW defines the Path Selection Engine (PSE) that is 1284 the counterpart of the PCE to perform rapid local adjustments of the 1285 forwarding tables within the diversity that the PCE has selected for 1286 the Track. The PSE enables to exploit the richer forwarding 1287 capabilities with PAREO and scheduled transmissions at a faster time 1288 scale over the smaller domain that is the Track, in either a loose or 1289 a strict fashion. 1291 Compared to the PCE, the PSE operates on metrics that evolve faster, 1292 but that needs to be advertised at a fast rate but only locally, 1293 within the Track. The forwarding decision may also change rapidly, 1294 but with a scope that is also contained within the Track, with no 1295 visibility to the other Tracks and flows in the network. This is as 1296 opposed to the PCE that needs to observe the whole network, and 1297 optimize all the Tracks globally, which can only be done at a slow 1298 pace and using long-term statistical metrics, as presented in 1299 Table 1. 1301 +===============+========================+===================+ 1302 | | PCE (Not in Scope) | PSE (In Scope) | 1303 +===============+========================+===================+ 1304 | Operation | Centralized | Source-Routed or | 1305 | | | Distributed | 1306 +---------------+------------------------+-------------------+ 1307 | Communication | Slow, expensive | Fast, local | 1308 +---------------+------------------------+-------------------+ 1309 | Time Scale | hours and above | seconds and below | 1310 +---------------+------------------------+-------------------+ 1311 | Network Size | Large, many Tracks to | Small, within one | 1312 | | optimize globally | Track | 1313 +---------------+------------------------+-------------------+ 1314 | Considered | Averaged, Statistical, | Instant values / | 1315 | Metrics | Shade of grey | boolean condition | 1316 +---------------+------------------------+-------------------+ 1318 Table 1: PCE vs. PSE 1320 The PSE sits in the DetNet Service sub-Layer of Edge and Relay Nodes. 1321 On the one hand, it operates on the packet flow, learning the Track 1322 and path selection information from the packet, possibly making local 1323 decision and retagging the packet to indicate so. On the other hand, 1324 the PSE interacts with the lower layers and with its peers to obtain 1325 up-to-date information about its radio links and the quality of the 1326 overall Track, respectively, as illustrated in Figure 8. 1328 | 1329 packet | going 1330 down the | stack 1331 +==========v==========+=====================+=====================+ 1332 | (iOAM + iCTRL) | (L2 Triggers, DLEP) | (oOAM) | 1333 +==========v==========+=====================+=====================+ 1334 | Learn from Learn from | 1335 | packet tagging Maintain end-to-end | 1336 +----------v----------+ Forwarding OAM packets | 1337 | Forwarding decision < State +---------^-----------| 1338 +----------v----------+ | Enrich or | 1339 + Retag Packet | Learn abstracted > Regenerate | 1340 | and Forward | metrics about Links | OAM packets | 1341 +..........v..........+..........^..........+.........^.v.........+ 1342 | Lower layers | 1343 +..........v.....................^....................^.v.........+ 1344 frame | sent Frame | L2 Ack oOAM | | packet 1345 over | wireless In | In | | and out 1346 v | | v 1348 Figure 8: PSE 1350 4.6. Act: The PAREO Functions 1352 RAW may control whether and how to use packet replication and 1353 elimination (PRE), Automatic Repeat reQuest (ARQ), Hybrid ARQ (HARQ) 1354 that includes Forward Error Correction (FEC) and coding, and other 1355 wireless-specific techniques such as overhearing and constructive 1356 interferences, in order to increase the reliabiility and availability 1357 of the end-to-end transmission. 1359 Collectively, those function are called PAREO for Packet (hybrid) 1360 ARQ, Replication, Elimination and Ordering. By tuning dynamically 1361 the use of PAREO functions, RAW avoids the waste of critical 1362 resources such as spectrum and energy while providing that the 1363 guaranteed SLA, e.g., by adding redundancy only when a spike of loss 1364 is observed. 1366 In a nutshell, PAREO establishes several paths in a network to 1367 provide redundancy and parallel transmissions to bound the end-to-end 1368 delay to traverse the network. Optionally, promiscuous listening 1369 between paths is possible, such that the Nodes on one path may 1370 overhear transmissions along the other path. Considering the 1371 scenario shown in Figure 9, many different paths are possible for to 1372 traverse the network from ingress to egress. A simple way to benefit 1373 from this topology could be to use the two independent paths via 1374 Nodes A, C, E and via B, D, F. But more complex paths are possible 1375 by interleaving transmissions from the lower level of the path to the 1376 upper level. 1378 (A) -- (C) -- (E) 1379 / \ 1380 Ingress = | | | = Egress 1381 \ / 1382 (B) -- (D) -- (F) 1384 Figure 9: A Ladder Shape with Two Parallel Paths 1386 PAREO may also take advantage of the shared properties of the 1387 wireless medium to compensate for the potential loss that is incurred 1388 with radio transmissions. 1390 For instance, when the source sends to Node A, Node B may listen 1391 promiscuously and get a second chance to receive the frame without an 1392 additional transmission. Note that B would not have to listen if it 1393 already received that particular frame at an earlier timeslot in a 1394 dedicated transmission towards B. 1396 The PAREO model can be implemented in both centralized and 1397 distributed scheduling approaches. In the centralized approach, a 1398 Path Computation Element (PCE) scheduler calculates a Track and 1399 schedules the communication. In the distributed approach, the Track 1400 is computed within the network, and signaled in the packets, e.g., 1401 using BIER-TE, Segment Routing, or a Source Routing Header. 1403 4.6.1. Packet Replication 1405 By employing a Packet Replication procedure, a Node forwards a copy 1406 of each data packet to more than one successor. To do so, each Node 1407 (i.e., Ingress and intermediate Node) sends the data packet multiple 1408 times as separate unicast transmissions. For instance, in Figure 10, 1409 the Ingress Node is transmitting the packet to both successors, nodes 1410 A and B, at two different times. 1412 ===> (A) => (C) => (E) === 1413 // \\// \\// \\ 1414 Ingress //\\ //\\ Egress 1415 \\ // \\ // \\ // 1416 ===> (B) => (D) => (F) === 1418 Figure 10: Packet Replication 1420 An example schedule is shown in Table 2. This way, the transmission 1421 leverages with the time and spatial forms of diversity. 1423 +=========+======+======+======+======+======+======+======+ 1424 | Channel | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 1425 +=========+======+======+======+======+======+======+======+ 1426 | 0 | S->A | S->B | B->C | B->D | C->F | E->R | F->R | 1427 +---------+------+------+------+------+------+------+------+ 1428 | 1 | | A->C | A->D | C->E | D->E | D->F | | 1429 +---------+------+------+------+------+------+------+------+ 1431 Table 2: Packet Replication: Sample schedule 1433 4.6.2. Packet Elimination 1435 The replication operation increases the traffic load in the network, 1436 due to packet duplications. This may occur at several stages inside 1437 the Track, and to avoid an explosion of the number of copies, a 1438 Packet Elimination procedure must be applied as well. To this aim, 1439 once a Node receives the first copy of a data packet, it discards the 1440 subsequent copies. 1442 The logical functions of Replication and Elimination may be 1443 collocated in an intermediate Node, the Node first eliminating the 1444 redundant copies and then sending the packet exactly once to each of 1445 the selected successors. 1447 4.6.3. Promiscuous Overhearing 1449 Considering that the wireless medium is broadcast by nature, any 1450 neighbor of a transmitter may overhear a transmission. By employing 1451 the Promiscuous Overhearing operation, the next hops have additional 1452 opportunities to capture the data packets. In Figure 11, when Node A 1453 is transmitting to its DP (Node C), the AP (Node D) and its sibling 1454 (Node B) may decode this data packet as well. As a result, by 1455 employing corellated paths, a Node may have multiple opportunities to 1456 receive a given data packet. 1458 ===> (A) ====> (C) ====> (E) ==== 1459 // ^ | \\ \\ 1460 Ingress | | \\ Egress 1461 \\ | v \\ // 1462 ===> (B) ====> (D) ====> (F) ==== 1464 Figure 11: Unicast with Overhearing 1466 4.6.4. Constructive Interference 1468 Constructive Interference can be seen as the reverse of Promiscuous 1469 Overhearing, and refers to the case where two senders transmit the 1470 exact same signal in a fashion that the emitted symbols add up at the 1471 receiver and permit a reception that would not be possible with a 1472 single sender at the same PHY mode and the same power level. 1474 Constructive Interference was proposed on 5G, Wi-Fi7 and even tested 1475 on IEEE Std 802.14.5. The hard piece is to synchronize the senders 1476 to the point that the signals are emitted at slightly different time 1477 to offset the difference of propagation delay that corresponds to the 1478 difference of distance of the transmitters to the receiver at the 1479 speed of light to the point that the symbols are superposed long 1480 enough to be recognizable. 1482 5. Security Considerations 1484 RAW uses all forms of diversity including radio technology and 1485 physical path to increase the reliability and availability in the 1486 face of unpredictable conditions. While this is not done 1487 specifically to defeat an attacker, the amount of diversity used in 1488 RAW makes an attack harder to achieve. 1490 5.1. Forced Access 1492 RAW will typically select the cheapest collection of links that 1493 matches the requested SLA, for instance, leverage free WI-Fi vs. paid 1494 3GPP access. By defeating the cheap connectivity (e.g., PHY-layer 1495 interference) the attacker can force an End System to use the paid 1496 access and increase the cost of the transmission for the user. 1498 6. IANA Considerations 1500 This document has no IANA actions. 1502 7. Contributors 1504 The editor wishes to thank: 1506 Xavi Vilajosana: Wireless Networks Research Lab, Universitat Oberta 1507 de Catalunya 1509 Remous-Aris Koutsiamanis: IMT Atlantique 1511 Nicolas Montavont: IMT Atlantique 1513 Rex Buddenberg: Individual contributor 1515 Greg Mirsky: ZTE 1517 for their contributions to the text and ideas exposed in this 1518 document. 1520 8. Acknowledgments 1522 TBD 1524 9. References 1526 9.1. Normative References 1528 [6TiSCH-ARCHI] 1529 Thubert, P., Ed., "An Architecture for IPv6 over the Time- 1530 Slotted Channel Hopping Mode of IEEE 802.15.4 (6TiSCH)", 1531 RFC 9030, DOI 10.17487/RFC9030, May 2021, 1532 . 1534 [RAW-TECHNOS] 1535 Thubert, P., Cavalcanti, D., Vilajosana, X., Schmitt, C., 1536 and J. Farkas, "Reliable and Available Wireless 1537 Technologies", Work in Progress, Internet-Draft, draft- 1538 ietf-raw-technologies-02, 7 June 2021, 1539 . 1542 [RAW-USE-CASES] 1543 Papadopoulos, G. Z., Thubert, P., Theoleyre, F., and C. J. 1544 Bernardos, "RAW use cases", Work in Progress, Internet- 1545 Draft, draft-ietf-raw-use-cases-02, 12 July 2021, 1546 . 1549 [RFC4655] Farrel, A., Vasseur, J.-P., and J. Ash, "A Path 1550 Computation Element (PCE)-Based Architecture", RFC 4655, 1551 DOI 10.17487/RFC4655, August 2006, 1552 . 1554 [BFD] Katz, D. and D. Ward, "Bidirectional Forwarding Detection 1555 (BFD)", RFC 5880, DOI 10.17487/RFC5880, June 2010, 1556 . 1558 [RFC6291] Andersson, L., van Helvoort, H., Bonica, R., Romascanu, 1559 D., and S. Mansfield, "Guidelines for the Use of the "OAM" 1560 Acronym in the IETF", BCP 161, RFC 6291, 1561 DOI 10.17487/RFC6291, June 2011, 1562 . 1564 [RFC7799] Morton, A., "Active and Passive Metrics and Methods (with 1565 Hybrid Types In-Between)", RFC 7799, DOI 10.17487/RFC7799, 1566 May 2016, . 1568 [RFC8578] Grossman, E., Ed., "Deterministic Networking Use Cases", 1569 RFC 8578, DOI 10.17487/RFC8578, May 2019, 1570 . 1572 [IPv6] Deering, S. and R. Hinden, "Internet Protocol, Version 6 1573 (IPv6) Specification", STD 86, RFC 8200, 1574 DOI 10.17487/RFC8200, July 2017, 1575 . 1577 [SR-ARCHI] Filsfils, C., Ed., Previdi, S., Ed., Ginsberg, L., 1578 Decraene, B., Litkowski, S., and R. Shakir, "Segment 1579 Routing Architecture", RFC 8402, DOI 10.17487/RFC8402, 1580 July 2018, . 1582 [BIER] Wijnands, IJ., Ed., Rosen, E., Ed., Dolganow, A., 1583 Przygienda, T., and S. Aldrin, "Multicast Using Bit Index 1584 Explicit Replication (BIER)", RFC 8279, 1585 DOI 10.17487/RFC8279, November 2017, 1586 . 1588 [RFC8175] Ratliff, S., Jury, S., Satterwhite, D., Taylor, R., and B. 1589 Berry, "Dynamic Link Exchange Protocol (DLEP)", RFC 8175, 1590 DOI 10.17487/RFC8175, June 2017, 1591 . 1593 [RFC8557] Finn, N. and P. Thubert, "Deterministic Networking Problem 1594 Statement", RFC 8557, DOI 10.17487/RFC8557, May 2019, 1595 . 1597 [RFC8655] Finn, N., Thubert, P., Varga, B., and J. Farkas, 1598 "Deterministic Networking Architecture", RFC 8655, 1599 DOI 10.17487/RFC8655, October 2019, 1600 . 1602 [RFC9049] Dawkins, S., Ed., "Path Aware Networking: Obstacles to 1603 Deployment (A Bestiary of Roads Not Taken)", RFC 9049, 1604 DOI 10.17487/RFC9049, June 2021, 1605 . 1607 9.2. Informative References 1609 [RFC0791] Postel, J., "Internet Protocol", STD 5, RFC 791, 1610 DOI 10.17487/RFC0791, September 1981, 1611 . 1613 [TE] Awduche, D., Chiu, A., Elwalid, A., Widjaja, I., and X. 1614 Xiao, "Overview and Principles of Internet Traffic 1615 Engineering", RFC 3272, DOI 10.17487/RFC3272, May 2002, 1616 . 1618 [STD 62] Harrington, D., Presuhn, R., and B. Wijnen, "An 1619 Architecture for Describing Simple Network Management 1620 Protocol (SNMP) Management Frameworks", STD 62, RFC 3411, 1621 DOI 10.17487/RFC3411, December 2002, 1622 . 1624 [RFC4090] Pan, P., Ed., Swallow, G., Ed., and A. Atlas, Ed., "Fast 1625 Reroute Extensions to RSVP-TE for LSP Tunnels", RFC 4090, 1626 DOI 10.17487/RFC4090, May 2005, 1627 . 1629 [FRR] Shand, M. and S. Bryant, "IP Fast Reroute Framework", 1630 RFC 5714, DOI 10.17487/RFC5714, January 2010, 1631 . 1633 [RLFA-FRR] Bryant, S., Filsfils, C., Previdi, S., Shand, M., and N. 1634 So, "Remote Loop-Free Alternate (LFA) Fast Reroute (FRR)", 1635 RFC 7490, DOI 10.17487/RFC7490, April 2015, 1636 . 1638 [DetNet-DP] 1639 Varga, B., Ed., Farkas, J., Berger, L., Malis, A., and S. 1640 Bryant, "Deterministic Networking (DetNet) Data Plane 1641 Framework", RFC 8938, DOI 10.17487/RFC8938, November 2020, 1642 . 1644 [BIER-PREF] 1645 Thubert, P., Eckert, T., Brodard, Z., and H. Jiang, "BIER- 1646 TE extensions for Packet Replication and Elimination 1647 Function (PREF) and OAM", Work in Progress, Internet- 1648 Draft, draft-thubert-bier-replication-elimination-03, 3 1649 March 2018, . 1652 [DetNet-IP-OAM] 1653 Mirsky, G., Chen, M., and D. Black, "Operations, 1654 Administration and Maintenance (OAM) for Deterministic 1655 Networks (DetNet) with IP Data Plane", Work in Progress, 1656 Internet-Draft, draft-ietf-detnet-ip-oam-02, 30 March 1657 2021, . 1660 [RAW-5G] Farkas, J., Dudda, T., Shapin, A., and S. Sandberg, "5G - 1661 Ultra-Reliable Wireless Technology with Low Latency", Work 1662 in Progress, Internet-Draft, draft-farkas-raw-5g-00, 1 1663 April 2020, . 1666 [BIER-TE] Eckert, T., Cauchie, G., and M. Menth, "Tree Engineering 1667 for Bit Index Explicit Replication (BIER-TE)", Work in 1668 Progress, Internet-Draft, draft-ietf-bier-te-arch-10, 9 1669 July 2021, . 1672 [IPoWIRELESS] 1673 Thubert, P., "IPv6 Neighbor Discovery on Wireless 1674 Networks", Work in Progress, Internet-Draft, draft- 1675 thubert-6man-ipv6-over-wireless-09, 17 May 2021, 1676 . 1679 [RAW-OAM] Theoleyre, F., Papadopoulos, G. Z., Mirsky, G., and C. J. 1680 Bernardos, "Operations, Administration and Maintenance 1681 (OAM) features for RAW", Work in Progress, Internet-Draft, 1682 draft-ietf-raw-oam-support-02, 3 June 2021, 1683 . 1686 [I-D.ietf-ippm-ioam-direct-export] 1687 Song, H., Gafni, B., Zhou, T., Li, Z., Brockners, F., 1688 Bhandari, S., Sivakolundu, R., and T. Mizrahi, "In-situ 1689 OAM Direct Exporting", Work in Progress, Internet-Draft, 1690 draft-ietf-ippm-ioam-direct-export-05, 12 July 2021, 1691 . 1694 [DetNet-OAM] 1695 Mirsky, G., Theoleyre, F., Papadopoulos, G. Z., and C. J. 1696 Bernardos, "Framework of Operations, Administration and 1697 Maintenance (OAM) for Deterministic Networking (DetNet)", 1698 Work in Progress, Internet-Draft, draft-ietf-detnet-oam- 1699 framework-03, 6 July 2021, 1700 . 1703 [I-D.mirsky-ippm-hybrid-two-step] 1704 Mirsky, G., Lingqiang, W., Zhui, G., and H. Song, "Hybrid 1705 Two-Step Performance Measurement Method", Work in 1706 Progress, Internet-Draft, draft-mirsky-ippm-hybrid-two- 1707 step-11, 8 July 2021, 1708 . 1711 [I-D.mirsky-ippm-epm] 1712 Mirsky, G., Min, X., and L. Han, "Error Performance 1713 Measurement in Packet-switched Networks", Work in 1714 Progress, Internet-Draft, draft-mirsky-ippm-epm-03, 26 1715 March 2021, . 1718 [I-D.mirsky-bfd-mpls-demand] 1719 Mirsky, G., "BFD in Demand Mode over Point-to-Point MPLS 1720 LSP", Work in Progress, Internet-Draft, draft-mirsky-bfd- 1721 mpls-demand-09, 30 March 2021, 1722 . 1725 [I-D.ietf-ippm-ioam-data] 1726 Brockners, F., Bhandari, S., and T. Mizrahi, "Data Fields 1727 for In-situ OAM", Work in Progress, Internet-Draft, draft- 1728 ietf-ippm-ioam-data-14, 24 June 2021, 1729 . 1732 [NASA] Adams, T., "RELIABILITY: Definition & Quantitative 1733 Illustration", . 1736 [MANET] IETF, "Mobile Ad hoc Networking", 1737 . 1739 [detnet] IETF, "Deterministic Networking", 1740 . 1742 [SPRING] IETF, "Source Packet Routing in Networking", 1743 . 1745 [BIER] IETF, "Bit Indexed Explicit Replication", 1746 . 1748 [BFD] IETF, "Bidirectional Forwarding Detection", 1749 . 1751 [CCAMP] IETF, "Common Control and Measurement Plane", 1752 . 1754 [IPPM] IETF, "IP Performance Measurement", 1755 . 1757 Authors' Addresses 1759 Pascal Thubert (editor) 1760 Cisco Systems, Inc 1761 Building D 1762 45 Allee des Ormes - BP1200 1763 06254 MOUGINS - Sophia Antipolis 1764 France 1766 Phone: +33 497 23 26 34 1767 Email: pthubert@cisco.com 1769 Georgios Z. Papadopoulos 1770 IMT Atlantique 1771 Office B00 - 114A 1772 2 Rue de la Chataigneraie 1773 35510 Cesson-Sevigne - Rennes 1774 France 1776 Phone: +33 299 12 70 04 1777 Email: georgios.papadopoulos@imt-atlantique.fr 1778 Lou Berger 1779 LabN Consulting, L.L.C. 1780 United States of America 1782 Email: lberger@labn.net