idnits 2.17.1 draft-pthubert-raw-architecture-09.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 978 has weird spacing: '...-- Node z-- ...' == Line 983 has weird spacing: '... Node z-- ...' -- The document date (7 July 2021) is 1024 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Missing Reference: 'PCE' is mentioned on line 728, but not defined == Outdated reference: A later version (-08) exists of draft-ietf-raw-technologies-01 == Outdated reference: A later version (-11) exists of draft-ietf-raw-use-cases-01 -- Obsolete informational reference (is this intentional?): RFC 3272 (ref. 'TE') (Obsoleted by RFC 9522) == Outdated reference: A later version (-13) exists of draft-ietf-detnet-ip-oam-02 == Outdated reference: A later version (-13) exists of draft-ietf-bier-te-arch-09 == Outdated reference: A later version (-15) exists of draft-thubert-6man-ipv6-over-wireless-09 == Outdated reference: A later version (-06) exists of draft-ietf-raw-oam-support-02 == Outdated reference: A later version (-11) exists of draft-ietf-ippm-ioam-direct-export-03 == Outdated reference: A later version (-11) exists of draft-ietf-detnet-oam-framework-01 == Outdated reference: A later version (-15) exists of draft-mirsky-ippm-hybrid-two-step-09 == Outdated reference: A later version (-04) exists of draft-mirsky-ippm-epm-03 == Outdated reference: A later version (-15) exists of draft-mirsky-bfd-mpls-demand-09 == Outdated reference: A later version (-17) exists of draft-ietf-ippm-ioam-data-12 Summary: 0 errors (**), 0 flaws (~~), 16 warnings (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 RAW P. Thubert, Ed. 3 Internet-Draft Cisco Systems 4 Intended status: Informational G.Z. Papadopoulos 5 Expires: 8 January 2022 IMT Atlantique 6 L. Berger 7 LabN Consulting, L.L.C. 8 7 July 2021 10 Reliable and Available Wireless Architecture/Framework 11 draft-pthubert-raw-architecture-09 13 Abstract 15 Reliable and Available Wireless (RAW) provides for high reliability 16 and availability for IP connectivity over a wireless medium. The 17 wireless medium presents significant challenges to achieve 18 deterministic properties such as low packet error rate, bounded 19 consecutive losses, and bounded latency. This document defines the 20 RAW Architecture. It builds on the DetNet Architecture and discusses 21 specific challenges and technology considerations needed to deliver 22 DetNet service utilizing scheduled wireless segments and other media, 23 e.g., frequency/time-sharing physical media resources with stochastic 24 traffic. 26 Status of This Memo 28 This Internet-Draft is submitted in full conformance with the 29 provisions of BCP 78 and BCP 79. 31 Internet-Drafts are working documents of the Internet Engineering 32 Task Force (IETF). Note that other groups may also distribute 33 working documents as Internet-Drafts. The list of current Internet- 34 Drafts is at https://datatracker.ietf.org/drafts/current/. 36 Internet-Drafts are draft documents valid for a maximum of six months 37 and may be updated, replaced, or obsoleted by other documents at any 38 time. It is inappropriate to use Internet-Drafts as reference 39 material or to cite them other than as "work in progress." 41 This Internet-Draft will expire on 8 January 2022. 43 Copyright Notice 45 Copyright (c) 2021 IETF Trust and the persons identified as the 46 document authors. All rights reserved. 48 This document is subject to BCP 78 and the IETF Trust's Legal 49 Provisions Relating to IETF Documents (https://trustee.ietf.org/ 50 license-info) in effect on the date of publication of this document. 51 Please review these documents carefully, as they describe your rights 52 and restrictions with respect to this document. Code Components 53 extracted from this document must include Simplified BSD License text 54 as described in Section 4.e of the Trust Legal Provisions and are 55 provided without warranty as described in the Simplified BSD License. 57 Table of Contents 59 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 60 2. The RAW problem . . . . . . . . . . . . . . . . . . . . . . . 5 61 2.1. Terminology . . . . . . . . . . . . . . . . . . . . . . . 5 62 2.2. Reliability and Availability . . . . . . . . . . . . . . 7 63 2.2.1. High Availability Engineering Principles . . . . . . 8 64 2.2.2. Applying Reliability Concepts to Networking . . . . . 10 65 2.2.3. Reliability in the Context of RAW . . . . . . . . . . 11 66 2.3. Use Cases and Requirements Served . . . . . . . . . . . . 12 67 2.3.1. Radio Access Protection . . . . . . . . . . . . . . . 13 68 2.3.2. End-to-End Protection in a Wireless Mesh . . . . . . 13 69 2.4. Related Work at The IETF . . . . . . . . . . . . . . . . 14 70 3. The RAW Framework . . . . . . . . . . . . . . . . . . . . . . 15 71 3.1. Scope and Prerequisites . . . . . . . . . . . . . . . . . 15 72 3.2. Routing Time Scale vs. Forwarding Time Scale . . . . . . 16 73 3.3. Wireless Tracks . . . . . . . . . . . . . . . . . . . . . 17 74 3.4. PAREO Functions . . . . . . . . . . . . . . . . . . . . . 18 75 3.4.1. Packet Replication . . . . . . . . . . . . . . . . . 19 76 3.4.2. Packet Elimination . . . . . . . . . . . . . . . . . 20 77 3.4.3. Promiscuous Overhearing . . . . . . . . . . . . . . . 20 78 3.4.4. Constructive Interference . . . . . . . . . . . . . . 20 79 4. The RAW Architecture . . . . . . . . . . . . . . . . . . . . 21 80 4.1. The RAW Conceptual Model . . . . . . . . . . . . . . . . 21 81 4.2. The Path Selection Engine . . . . . . . . . . . . . . . . 23 82 4.3. RAW OAM . . . . . . . . . . . . . . . . . . . . . . . . . 24 83 4.3.1. DetNet OAM . . . . . . . . . . . . . . . . . . . . . 25 84 4.3.2. RAW Extensions . . . . . . . . . . . . . . . . . . . 26 85 4.3.3. Observed Metrics . . . . . . . . . . . . . . . . . . 27 86 4.4. Flow Identification vs. Path Identification . . . . . . . 27 87 4.5. Source-Routed vs. Distributed Forwarding Decision . . . . 30 88 4.6. Encapsulation and Decapsulation . . . . . . . . . . . . . 31 89 5. Security Considerations . . . . . . . . . . . . . . . . . . . 31 90 5.1. Forced Access . . . . . . . . . . . . . . . . . . . . . . 31 91 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 31 92 7. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 31 93 8. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 32 94 9. References . . . . . . . . . . . . . . . . . . . . . . . . . 32 95 9.1. Normative References . . . . . . . . . . . . . . . . . . 32 96 9.2. Informative References . . . . . . . . . . . . . . . . . 34 97 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 37 99 1. Introduction 101 Deterministic Networking is an attempt to emulate the properties of a 102 serial link over a switched fabric, by providing a bounded latency 103 and eliminating congestion loss, even when co-existing with best- 104 effort traffic. It is getting traction in various industries 105 including professional A/V, manufacturing, online gaming, and 106 smartgrid automation, enabling cost and performance optimizations 107 (e.g., vs. loads of P2P cables). 109 Bringing determinism in a packet network means eliminating the 110 statistical effects of multiplexing that result in probabilistic 111 jitter and loss. This can be approached with a tight control of the 112 physical resources to maintain the amount of traffic within a 113 budgetted volume of data per unit of time that fits the physical 114 capabilities of the underlying network, and the use of time-shared 115 resources (bandwidth and buffers) per circuit, and/or by shaping and/ 116 or scheduling the packets at every hop. 118 This innovation was initially introduced on wired networks, with IEEE 119 802.1 Time Sensitive networking (TSN) - for Ethernet LANs - and IETF 120 DetNet. But the wired and the wireless media are fundamentally 121 different at the physical level and in the possible abstractions that 122 can be built for IP [IPoWIRELESS]. Wireless networks operate on a 123 shared medium where uncontrolled interference, including the self- 124 induced multipath fading, cause random transmission losses and add 125 new dimensions to the statistical effects that affect reachability 126 and packet delivery. 128 To defeat those additional causes of transmission delay and loss, 129 Reliable and Available Wireless (RAW) leverages scheduled 130 transmissions with redundancy and diversity in the spatial, time, 131 code, and frequency domains. The challenge is to provide enough 132 diversity and redundancy to ensure the timely packet delivery while 133 preserving energy and optimizing the use of the shared spectrum. 135 While the generic "Deterministic Networking Problem Statement" 136 [RFC8557] applies to both the wired and the wireless media, the 137 methods to achieve RAW must extend those used to support time- 138 sensitive networking over wires, as a RAW solution has to address 139 less consistent transmissions, energy conservation and shared 140 spectrum efficiency. 142 Uncontrolled interference and transmission obstacles may impede the 143 wireless transmission, causing rapid variations of the throughput and 144 packet delivery ratio (PDR) of the link. This uncertainty limits the 145 volume and/or duration of traffic that can be safely transmitted on 146 the same link while conforming to a RAW Service Level Agreement 147 (SLA). 149 This increased complexity explains why the development of 150 deterministic wireless technologies has been lagging behind the 151 similar efforts for wired systems, both at the IEEE and the IETF. 152 But recent progress on scheduled radios such as TSCH and OFDMA 153 indicates that wireless is finally catching up at the lower layers. 154 Sitting at the layer above, RAW takes up the challenge of providing 155 highly available and reliable end-to-end performances in a network 156 with scheduled wireless segments. 158 RAW provides DetNet elements that are specialized for short range 159 radios. From this inheritance, RAW stays agnostic to the radio layer 160 underneath though the capability to schedule transmissions is 161 assumed. How the PHY is programmed to do so, and whether the radio 162 is single-hop or meshed, are unknown at the IP layer and not part of 163 the RAW abstraction. 165 The "Deterministic Networking Architecture" [RFC8655] is composed of 166 three planes: the Application (User) Plane, the Controller Plane, and 167 the Network Plane. The RAW Architecture extends the DetNet Network 168 Plane, to accomodate one or multiple hops of homogeneous or 169 heterogeneous wireless technologies, e.g. a Wi-Fi6 Mesh or parallel 170 CBRS access links federated by a 5G backhaul. 172 The establishment of a path is not in-scope for RAW. It may be the 173 product of a centralized Controller Plane as described for DetNet. 174 As opposed to wired networks, the action of installing a path over a 175 set of wireless links may be very slow relative to the speed at which 176 the radio conditions vary, and it makes sense in the wireless case to 177 provide redundant forwarding solutions along a complex path and to 178 leave it to the Network Plane to select which of those forwarding 179 solutions are to be used for a given packet based on the current 180 conditions. 182 RAW distinguishes the longer time scale at which routes are computed 183 from the the shorter forwarding time scale where per-packet decisions 184 are made. RAW operates within the Network Plane at the forwarding 185 time scale on one DetNet flow over a complex path called a Track. 186 The Track is preestablished and installed by means outside of the 187 scope of RAW; it may be strict or loose depending on whether each or 188 just a subset of the hops are observed and controlled by RAW. 190 The RAW Architecture covers Network Plane protocol elements such as 191 Operations, Administration and Maintenance (OAM) to observe some or 192 all hops along a Track as well as the end-to-end packet delivery, and 193 in-band control to optimize the use of redundancy to achieve the 194 required SLA with minimal use of constrained resources. 196 2. The RAW problem 198 2.1. Terminology 200 RAW reuses terminology defined for DetNet in the "Deterministic 201 Networking Architecture" [RFC8655], e.g., PREOF for Packet 202 Replication, Elimination and Ordering Functions. 204 RAW also reuses terminology defined for 6TiSCH in [6TiSCH-ARCHI] such 205 as the term Track. A Track as a complex path with associated PAREO 206 operations. The concept is abstract to the underlaying technology 207 and applies to any fully or partially wireless mesh, including, e.g., 208 a Wi-Fi mesh. RAW specifies strict and loose Tracks depending on 209 whether the path is fully controlled by RAW or traverses an opaque 210 network where RAW cannot observe and control the individual hops. 212 RAW uses the following terminology: 214 PAREO: Packet (hybrid) ARQ, Replication, Elimination and Ordering. 215 PAREO is a superset Of DetNet's PREOF that includes radio-specific 216 techniques such as short range broadcast, MUMIMO, constructive 217 interference and overhearing, which can be leveraged separately or 218 combined to increase the reliability. 220 Flow: A collection of consecutive packets that must be placed on the 221 same Track to receive an equivalent treatment from Ingress to 222 Egress within the Track. Multiple flows may be transported along 223 the same Track. The subTrack that is selected for the flow may 224 change over time under the control of the PSE. 226 Track: A networking graph that can be used as a "path" to transport 227 RAW packets with equivalent treatment; as opposed to the usual 228 understanding of a path (see for instance the definition of "path" 229 in section 1.1 of [RFC9049]), a Track may fork and rejoin to 230 enable the PAREO operations. 232 In DetNet [RFC8655] terms, a Track has the following properties: 234 * A Track has one Ingress and one Egress nodes, which operate as 235 DetNet Edge nodes. 237 * A Track is reversible, meaning that packets can be routed 238 against the flow of data packets, e.g., to carry OAM 239 measurements or control messages back to the Ingress. 241 * The vertices of the Track are DetNet Relay nodes that operate 242 at the DetNet Service sublayer and provide the PAREO functions. 244 * The topological edges of the graph are serial sequences of 245 DetNet Transit nodes that operate at the DetNet Forwarding 246 sublayer. 248 SubTrack: A Track within a Track. The RAW PSE selects a subTrack on 249 a per-packet or a per-collection of packets basis to provide the 250 desired reliability for the transported flows. 252 Segment: A serial path formed by a topological edge of a Track. 253 East-West Segments are oriented from Ingress (East) to Egress 254 (West). North/South Segments can be bidirectional; to avoid 255 loops, measures must be taken to ensure that a given packet flows 256 either Northwards or Southwards along a bidirectional Segment, but 257 never bounces back. 259 Flapping: In the context of RAW, a link flaps when the reliability 260 of the wireless connectivity drops abruptly for a short period of 261 time, typically of a subsecond to seconds duration. 263 OAM: OAM stands for Operations, Administration, and Maintenance, and 264 covers the processes, activities, tools, and standards involved 265 with operating, administering, managing and maintaining any 266 system. This document uses the terms Operations, Administration, 267 and Maintenance, in conformance with the 'Guidelines for the Use 268 of the "OAM" Acronym in the IETF' [RFC6291] and the system 269 observed by the RAW OAM is the Track. 271 Active OAM: See [RFC7799]. In the context of RAXW, Active OAM is 272 used to observe a particular Track, subTrack, or Segment of a 273 Track regardless of whether it is used for traffic at that time. 275 In-Band OAM: An active OAM packet is considered in-band for the 276 monitored Track when it traverses the same set of links and 277 interfaces and if the OAM packet receives the same QoS and PAREO 278 treatment as the packets of the data flows that are injected in 279 the Track. 281 Out-of-Band OAM: Out-of-band OAM is an active OAM whose path is not 282 topologically congruent to the Track, or its test packets receive 283 a QoS and/or PAREO treatment that is different from that of the 284 packets of the data flows that are injected in the Track, or both. 286 Limited OAM: An active OAM packet is a Limited OAM packet when it 287 observes the RAW operation over a node, a segment, or a subTrack 288 of the Track, though not from Ingress to Egress. It is injected 289 in the datapath and extracted from the datapath around the 290 particular function or subnetwork (e.g., around a relay providing 291 a service layer replication point) that is being tested. 293 Reverse OAM: A Reverse OAM packet is an Out-of-Band OAM packet that 294 traverses the Track from egress to ingress on the reverse 295 direction, to capture and report OAM measurements upstream. The 296 collection may capture all information along the whole Track, or 297 it may only learn select data across all, or only a particular 298 subTrack, or Segment of a Track. 300 [DetNet-OAM] provides additional terminology related to OAM in the 301 context of DetNet and by extension of RAW, whereas [RFC7799] defines 302 the Active, Passive, and Hybrid OAM methods. 304 In the context of the RAW work, Reliability and Availability are 305 defined as follows: 307 Reliability: Reliability is a measure of the probability that an 308 item will perform its intended function for a specified interval 309 under stated conditions. For RAW, the service that is expected is 310 delivery within a bounded latency and a failure is when the packet 311 is either lost or delivered too late. RAW expresses reliability 312 in terms of Mean Time Between Failure (MTBF) and Maximum 313 Consecutive Failures (MCF). More in [NASA]. 315 Availability: Availability is a measure of the relative amount of 316 time where a path operates in stated condition, in other words 317 (uptime)/(uptime+downtime). Because a serial wireless path may 318 not be good enough to provide the required reliability, and even 2 319 parallel paths may not be over a longer period of time, the RAW 320 availability implies a path that is a lot more complex than what 321 DetNet typically envisages (a Track). 323 Residence Time: A residence time (RT) is defined as the time period 324 between the reception of a packet starts and the transmission of 325 the packet begins. In the context of RAW, RT is useful for a 326 transit node, not ingress or egress. 328 2.2. Reliability and Availability 329 2.2.1. High Availability Engineering Principles 331 The reliability criteria of a critical system pervade through its 332 elements, and if the system comprises a data network then the data 333 network is also subject to the inherited reliability and availability 334 criteria. It is only natural to consider the art of high 335 availability engineering and apply it to wireless communications in 336 the context of RAW. 338 There are three principles [pillars] of high availability 339 engineering: 341 1. elimination of single points of failure 342 2. reliable crossover 343 3. prompt detection of failures as they occur. 345 These principles are common to all high availability systems, not 346 just ones with Internet technology at the center. Examples of both 347 non-Internet and Internet are included. 349 2.2.1.1. Elimination of Single Points of Failure 351 Physical and logical components in a system happen to fail, either as 352 the effect of wear and tear, when used beyond acceptable limits, or 353 due to a software bug. It is necessary to decouple component failure 354 from system failure to avoid the latter. This allows failed 355 components to be restored while the rest of the system continues to 356 function. 358 IP Routers leverage routing protocols to compute alternate routes in 359 case of a failure. There is a rather open-ended issue over alternate 360 routes -- for example, when links are cabled through the same 361 conduit, they form a shared risk link group (SRLG), and will share 362 the same fate if the bundle is cut. The same effect can happen with 363 virtual links that end up in a same physical transport through the 364 games of encapsulation. In a same fashion, an interferer or an 365 obstacle may affect multiple wireless transmissions at the same time, 366 even between different sets of peers. 368 Intermediate network Nodes such as routers, switches and APs, wire 369 bundles and the air medium itself can become single points of 370 failure. For High Availability, it is thus required to use 371 physically link- and Node-disjoint paths; in the wireless space, it 372 is also required to use the highest possible degree of diversity in 373 the transmissions over the air to combat the additional causes of 374 transmission loss. 376 From an economics standpoint, executing this principle properly 377 generally increases capitalization expense because of the redundant 378 equipment. In a constrained network where the waste of energy and 379 bandwidth should be minimized, an excessive use of redundant links 380 must be avoided; for RAW this means that the extra bandwidth must be 381 used wisely and with parcimony. 383 2.2.1.2. Reliable Crossover 385 Having a backup equipment has a limited value unless it can be 386 reliably switched into use within the down-time parameters. IP 387 Routers execute reliable crossover continuously because the routers 388 will use any alternate routes that are available [RFC0791]. This is 389 due to the stateless nature of IP datagrams and the dissociation of 390 the datagrams from the forwarding routes they take. The "IP Fast 391 Reroute Framework" [FRR] analyzes mechanisms for fast failure 392 detection and path repair for IP Fast-Reroute, and discusses the case 393 of multiple failures and SRLG. Examples of FRR techniques include 394 Remote Loop-Free Alternate [RLFA-FRR] and backup label-switched path 395 (LSP) tunnels for the local repair of LSP tunnels using RSVP-TE 396 [RFC4090]. 398 Deterministic flows, on the contrary, are attached to specific paths 399 where dedicated resources are reserved for each flow. This is why 400 each DetNet path must inherently provide sufficient redundancy to 401 provide the guaranteed SLA at all times. The DetNet PREOF typically 402 leverages 1+1 redundancy whereby a packet is sent twice, over non- 403 congruent paths. This avoids the gap during the fast reroute 404 operation, but doubles the traffic in the network. 406 In the case of RAW, the expectation is that multiple transient faults 407 may happen in overlapping time windows, in which case the 1+1 408 redundancy with delayed reestablishment of the second path will not 409 provide the required guarantees. The Data Plane must be configured 410 with a sufficient degree of redundancy to select an alternate 411 redundant path immediately upon a fault, without the need for a slow 412 intervention from the controller plane. 414 2.2.1.3. Prompt Notification of Failures 416 The execution of the two above principles is likely to render a 417 system where the user will rarely see a failure. But someone needs 418 to in order to direct maintenance. 420 There are many reasons for system monitoring (FCAPS for fault, 421 configuration, accounting, performance, security is a handy mental 422 checklist) but fault monitoring is sufficient reason. 424 "An Architecture for Describing Simple Network Management Protocol 425 (SNMP) Management Frameworks" [STD 62] describes how to use SNMP to 426 observe and correct long-term faults. 428 "Overview and Principles of Internet Traffic Engineering" [TE] 429 discusses the importance of measurement for network protection, and 430 provides abstract an method for network survivability with the 431 analysis of a traffic matrix as observed by SNMP, probing techniques, 432 FTP, IGP link state advertisements, and more. 434 Those measurements are needed in the context of RAW to inform the 435 controller and make the long term reactive decision to rebuild a 436 complex path. But RAW itself operates in the Network Plane at a 437 faster time scale. To act on the Data Plane, RAW needs live 438 information from the Operational Plane , e.g., using Bidirectional 439 Forwarding Detection [BFD] and its variants (bidirectional and remote 440 BFD) to protect a link, and OAM techniques to protect a path. 442 2.2.2. Applying Reliability Concepts to Networking 444 The terms Reliability and Availability are defined for use in RAW in 445 Section 2.1 and the reader is invited to read [NASA] for more details 446 on the general definition of Reliability. Practically speaking a 447 number of nines is often used to indicate the reliability of a data 448 link, e.g., 5 nines indicate a Packet Delivery Ratio (PDR) of 449 99.999%. 451 This number is typical in a wired environment where the loss is due 452 to a random event such as a solar particle that affects the 453 transmission of a particular frame, but does not affect the previous 454 or next frame, nor frames transmitted on other links. Note that the 455 QoS requirements in RAW may include a bounded latency, and a packet 456 that arrives too late is a fault and not considered as delivered. 458 For a periodic networking pattern such as an automation control loop, 459 this number is proportional to the Mean Time Between Failures (MTBF). 460 When a single fault can have dramatic consequences, the MTBF 461 expresses the chances that the unwanted fault event occurs. In data 462 networks, this is rarely the case. Packet loss cannot never be fully 463 avoided and the systems are built to resist to one loss, e.g., using 464 redundancy with Retries (HARQ) or Packet Replication and Elimination 465 (PRE), or, in a typical control loop, by linear interpolation from 466 the previous measurements. 468 But the linear interpolation method cannot resist multiple 469 consecutive losses, and a high MTBF is desired as a guarantee that 470 this will not happen, IOW that the number of losses-in-a-row can be 471 bounded. In that case, what is really desired is a Maximum 472 Consecutive Failures (MCF). If the number of losses in a row passes 473 the MCF, the control loop has to abort and the system, e.g., the 474 production line, may need to enter an emergency stop condition. 476 Engineers that build automated processes may use the network 477 reliability expressed in nines or as an MTBF as a proxy to indicate 478 an MCF, e.g., as described in section 7.4 of the "Deterministic 479 Networking Use Cases" [RFC8578]. 481 2.2.3. Reliability in the Context of RAW 483 In contrast with wired networks, errors in transmission are the 484 predominant source of packet loss in wireless networks. 486 The root cause for the loss may be of multiple origins, calling for 487 the use of different forms of diversity: 489 Multipath Fading: A destructive interference by a reflection of the 490 original signal. 492 A radio signal may be received directly (line-of-sight) and/or as 493 a reflection on a physical structure (echo). The reflections take 494 a longer path and are delayed by the extra distance divided by the 495 speed of light in the medium. Depending on the frequency, the 496 echo lands with a different phase which may add up to 497 (constructive interference) or cancel the direct signal 498 (destructive interference). 500 The affected frequencies depend on the relative position of the 501 sender, the receiver, and all the reflecting objects in the 502 environment. A given hop will suffer from multipath fading for 503 multiple packets in a row till the something moves that changes 504 the reflection patterns. 506 Co-channel Interference: Energy in the spectrum used for the 507 transmission confuses the receiver. 509 The wireless medium itself is a Shared Risk Link Group (SRLG) for 510 nearby users of the same spectrum, as an interference may affect 511 multiple co-channel transmissions between different peers within 512 the interference domain of the interferer, possibly even when they 513 use different technologies. 515 Obstacle in Fresnel Zone: The optimal transmission happens when the 516 Fresnel Zone between the sender and the receiver is free of 517 obstacles. 519 As long as a physical object (e.g., a metallic trolley between 520 peers) that affects the transmission is not removed, the quality 521 of the link is affected. 523 In an environment that is rich of metallic structures and mobile 524 objects, a single radio link will provide a fuzzy service, meaning 525 that it cannot be trusted to transport the traffic reliably over a 526 long period of time. 528 Transmission losses are typically not independent, and their nature 529 and duration are unpredictable; as long as a physical object (e.g., a 530 metallic trolley between peers) that affects the transmission is not 531 removed, or as long as the interferer (e.g., a radar) keeps 532 transmitting, a continuous stream of packets will be affected. 534 The key technique to combat those unpredictable losses is diversity. 535 Different forms of diversity are necessary to combat different causes 536 of loss and the use of diversity must be maximised to optimize the 537 PDR. 539 A single packet may be sent at different times (time diversity) over 540 diverse paths (spatial diversity) that rely on diverse radio channels 541 (frequency diversity) and diverse PHY technologies, e.g., narrowband 542 vs. spread spectrum, or diverse codes. Using time diversity will 543 defeat short-term interferences; spatial diversity combats very local 544 causes such as multipath fading; narrowband and spread spectrum are 545 relatively innocuous to one another and can be used for diversity in 546 the presence of the other. 548 2.3. Use Cases and Requirements Served 550 In order to focus on real-worlds issues and assert the feasibility of 551 the proposed capabilities, RAW focuses on selected technologies that 552 can be scheduled at the lower layers: IEEE Std. 802.15.4 timeslotted 553 channel hopping (TSCH), 3GPP 5G ultra-reliable low latency 554 communications (URLLC), IEEE 802.11ax/be where 802.11be is extreme 555 high throughput (EHT), and L-band Digital Aeronautical Communications 556 System (LDACS). See [RAW-TECHNOS] for more. 558 "Deterministic Networking Use Cases" [RFC8578] presents a number of 559 wireless use cases including Wireless, such as application to 560 Industrial Applications, Pro-Audio, and SmartGrid Automation. 561 [RAW-USE-CASES] adds a number of use cases that demonstrate the need 562 for RAW capabilities for new applications such as Pro-Gaming and 563 drones. The use cases can be abstracted in two families, Loose 564 Protection, e.g., protecting the first hop in Radio Access Protection 565 and Strict Protection, e.g., providing End-to-End Protection in a 566 wireless mesh. 568 2.3.1. Radio Access Protection 570 To maintain the required SLA at all times, a wireless Host may use 571 more than one Radio Access Network (RAN) in parallel. 573 ... .. 574 RAN 1 ----- ... .. ... 575 / . .. .... 576 +--------+ / . .... +-----------+ 577 |Wireless|- . ..... | Service | 578 | Device |-***-- RAN 2 -- . Internet ....---| / | 579 |(STA/UE)|- .. ..... |Application| 580 +--------+ $$$ . ....... +-----------+ 581 \ ... ... ..... 582 RAN n -------- ... ..... 584 *** = flapping at this time $$$ expensive 586 Figure 1: Radio Access Protection 588 The RANs may be heterogeneous, e.g., 3GPP 5G [RAW-5G] and Wi-Fi 589 [RAW-TECHNOS] for high-speed communication, in which case a Layer-3 590 abstraction becomes useful to select which of the RANs are used at a 591 particular point of time, and the amount of traffic that is 592 distributed over each RAN. 594 The idea is that the rest of the path to the destination(s) is 595 protected separately (e.g., uses non-congruent paths, leverages 596 DetNet / TSN, etc...) and is a lot more reliable, e.g., wired. In 597 that case, RAW observes the reliability of the end-to-end operation 598 through each of the RANs but only observes and controls the wireless 599 operation the first hop. 601 A variation of that use case has a pair of wireless Hosts connected 602 over a wired core / backbone network. In that case, RAW observes and 603 controls the Ingress and Egress RANs, while neglecting the hops in 604 the core. The resulting loose Track may be instantiated, e.g., using 605 tunneling or loose source routing between the RANs. 607 2.3.2. End-to-End Protection in a Wireless Mesh 609 In radio technologies that support mesh networking (e.g., Wi-Fi and 610 TSCH), a Track is a complex path with distributed PAREO capabilities. 611 In that case, RAW operates through the multipath and makes decisions 612 either at the Ingress or at every hop (more in Section 3.3). 614 A-------B-------C-----D 615 / \ / / \ 616 Ingress ----M-------N--zzzzz--- Egress 617 \ \ / / 618 P--zzz--Q-------------R 620 zzz = flapping now 622 Figure 2: End-to-End Protection 624 The Protection may be imposed by the source based on end-to-end OAM, 625 or performed hop-by-hop, in which case the OAM must enables the 626 intermediate Nodes to estimate the quality of the rest of the 627 feasible paths in the remainder of the Track to the destination. 629 2.4. Related Work at The IETF 631 RAW intersects with protocols or practices in development at the IETF 632 as follows: 634 * The Dynamic Link Exchange Protocol (DLEP) [RFC8175] from [MANET] 635 can be leveraged at each hop to derive generic radio metrics 636 (e.g., based on LQI, RSSI, queueing delays and ETX) on individual 637 hops. 639 * [detnet] provides an OAM framework with [DetNet-OAM] that applies 640 within the DetNet dataplane described in [DetNet-DP],which is 641 typically based on MPLS or IPv6 pseudowires. 643 * [BFD] detect faults in the path between an Ingress and an Egress 644 forwarding engines, but is unaware of the complexity of a path 645 with replication, and expects bidirectionality. BFD asynchronous 646 mode considers delivery as success whereas with DetNet and RAW, 647 the bounded latency can be as important as the delivery itself, 648 and delivering too late is actually a failure. Note that the BFD 649 Demand mode with unsolicited notifications may be more suitable 650 then the Asynchronous BFD mode. The use of the Demand mode in 651 MPLS is analyzed in [I-D.mirsky-bfd-mpls-demand] and similar 652 considerations could apply to IP as well. 654 * [SPRING] and [BIER] define in-band signaling that influences the 655 routing when decided at the head-end on the path. There's already 656 one RAW-related draft at BIER [BIER-PREF] more may follow. RAW 657 will need new in-band signaling when the decision is distributed, 658 e.g., required chances of reliable delivery to destination within 659 latency. This signaling enables relays to tune retries and 660 replication to meet the required SLA. 662 * [CCAMP] defines protocol-independent metrics and parameters 663 (measurement attributes) for describing links and paths that are 664 required for routing and signaling in technology-specific 665 networks. RAW would be a source of requirements for CCAMP to 666 define metrics that are significant to the focus radios. 668 * [IPPM] develops and maintains standard metrics that can be applied 669 to the quality, performance, and reliability of Internet data 670 delivery services and applications running over transport layer 671 protocols (e.g. TCP, UDP) over IP. 673 3. The RAW Framework 675 3.1. Scope and Prerequisites 677 A prerequisite to the RAW operation is that an end-to-end routing 678 function computes a complex sub-topology along which forwarding can 679 happen between a source and one or more destinations. The concept of 680 Track is specified in the 6TiSCH Architecture [6TiSCH-ARCHI] to 681 represent that complex sub-topology. Tracks provide a high degree of 682 redundancy and diversity and enable the DetNet PREOF, network coding, 683 and possibly RAW specific techniques such as PAREO, leveraging 684 frequency diversity, time diversity, and possibly other forms of 685 diversity as well. 687 How the routing operation (e.g., PCE) in the Controller Plane 688 computes the Track is out of scope for RAW. The scope of the RAW 689 operation is one Track, and the goal of the RAW operation is to 690 optimize the use of the Track at the forwarding timescale to maintain 691 the expected SLA while optimizing the usage of constrained resources 692 such as energy and spectrum. 694 Another prerequisite is that an IP link can be established over the 695 radio with some guarantees in terms of service reliability, e.g., it 696 can be relied upon to transmit a packet within a bounded latency and 697 provides a guaranteed BER/PDR outside rare but existing transient 698 outage windows that can last from split seconds to minutes. The 699 radio layer can be programmed with abstract parameters, and can 700 return an abstract view of the state of the Link to help the Network 701 Layer forwarding decision (think DLEP from MANET). 703 How the radio interface manages its lower layers is out of control 704 and out of scope for RAW. In the same fashion, the non-RAW portion 705 along a loose Track is by definition out of control and out of scope 706 for RAW. Whether it is a single hop or a mesh is also unknown and 707 out of scope. 709 3.2. Routing Time Scale vs. Forwarding Time Scale 711 With DetNet, the Controller Plane Function that handles the routing 712 computation and maintenance (the PCE) can be centralized and can 713 reside outside the network. In a wireless mesh, the path to the PCE 714 can be expensive and slow, possibly going across the whole mesh and 715 back. Reaching to the PCE can also be slow in regards to the speed 716 of events that affect the forwarding operation at the radio layer. 718 Due to that cost and latency, the Controller Plane is not expected to 719 be sensitive/reactive to transient changes. The abstraction of a 720 link at the routing level is expected to use statistical metrics that 721 aggregate the behavior of a link over long periods of time, and 722 represent its properties as shades of gray as opposed to numerical 723 values such as a link quality indicator, or a boolean value for 724 either up or down. 726 +----------------+ 727 | Controller | 728 | [PCE] | 729 +----------------+ 730 ^ 731 | 732 Slow 733 | 734 _-._-._-._-._-._-. | ._-._-._-._-._-._-._-._-._-._-._-._- 735 _-._-._-._-._-._-._-. | _-._-._-._-._-._-._-._-._-._-._-._- 736 | 737 Expensive 738 | 739 .... | ....... 740 .... . | . ....... 741 .... v ... 742 .. A-------B-------C---D .. 743 ... / \ / / \ .. 744 . I ----M-------N--***-- E .. 745 .. \ \ / / ... 746 .. P--***--Q----------R .... 747 .. .... 748 . <----- Fast -------> .... 749 ....... .... 750 ................. 752 *** = flapping at this time 754 Figure 3: Time Scales 756 In the case of wireless, the changes that affect the forwarding 757 decision can happen frequently and often for short durations, e.g., a 758 mobile object moves between a transmitter and a receiver, and will 759 cancel the line of sight transmission for a few seconds, or a radar 760 measures the depth of a pool and interferes on a particular channel 761 for a split second. 763 There is thus a desire to separate the long term computation of the 764 route and the short term forwarding decision. In that model, the 765 routing operation computes a complex Track that enables multiple Non- 766 Equal Cost Multi-Path (N-ECMP) forwarding solutions, and leaves it to 767 the Data Plane to make the per-packet decision of which of these 768 possibilities should be used. 770 In the wired world, and more specifically in the context of Traffic 771 Engineering (TE), an alternate path can be used upon the detection of 772 a failure in the main path, e.g., using OAM in MPLS-TP or BFD over a 773 collection of SD-WAN tunnels. RAW formalizes a forwarding time scale 774 that is an order(s) of magnitude shorter than the controller plane 775 routing time scale, and separates the protocols and metrics that are 776 used at both scales. Routing can operate on long term statistics 777 such as delivery ratio over minutes to hours, but as a first 778 approximation can ignore flapping. On the other hand, the RAW 779 forwarding decision is made at the scale of the packet rate, and uses 780 information that must be pertinent at the present time for the 781 current transmission(s). 783 3.3. Wireless Tracks 785 The "6TiSCH Architecture" [6TiSCH-ARCHI] introduces the concept of 786 Track. RAW extends the concept to any wireless mesh technology, 787 including, e.g., Wi-Fi. A simple Track is composed of a direct 788 sequence of reserved hops to ensure the transmission of a single 789 packet from a source Node to a destination Node across a multihop 790 path. 792 A Complex Track provides multiple N-ECMP forwarding solutions. The 793 Complex Track enables to support multi-path redundant forwarding by 794 employing PRE functions [RFC8655] and the ingress and within the 795 Track. For example, a Complex Track may branch off and rejoin over 796 non-congruent segments. 798 In the context of RAW, some links or segments in the Track may be 799 reversible, meaning that they can be used in either direction. In 800 that case, an indication in the packet signals the direction of the 801 reversible links or segments that the packet traverses and thus 802 places a constraint that prevents loops from occuring. An indidual 803 packet follows a destination-oriented directed acyclic graph (DODAG) 804 towards a destination Node inside the Complex Track. 806 3.4. PAREO Functions 808 RAW may control whether and how to use packet replication and 809 elimination (PRE), Automatic Repeat reQuest (ARQ), Hybrid ARQ (HARQ) 810 that includes Forward Error Correction (FEC) and coding, and other 811 wireless-specific techniques such as overhearing and constructive 812 interferences, in order to increase the reliabiility and availability 813 of the end-to-end transmission. 815 Collectively, those function are called PAREO for Packet (hybrid) 816 ARQ, Replication, Elimination and Ordering. By tuning dynamically 817 the use of PAREO functions, RAW avoids the waste of critical 818 resources such as spectrum and energy while providing that the 819 guaranteed SLA, e.g., by adding redundancy only when a spike of loss 820 is observed. 822 In a nutshell, PAREO establishes several paths in a network to 823 provide redundancy and parallel transmissions to bound the end-to-end 824 delay to traverse the network. Optionally, promiscuous listening 825 between paths is possible, such that the Nodes on one path may 826 overhear transmissions along the other path. Considering the 827 scenario shown in Figure 4, many different paths are possible for to 828 traverse the network from ingress to egress. A simple way to benefit 829 from this topology could be to use the two independent paths via 830 Nodes A, C, E and via B, D, F. But more complex paths are possible 831 by interleaving transmissions from the lower level of the path to the 832 upper level. 834 (A) -- (C) -- (E) 835 / \ 836 Ingress = | | | = Egress 837 \ / 838 (B) -- (D) -- (F) 840 Figure 4: A Ladder Shape with Two Parallel Paths 842 PAREO may also take advantage of the shared properties of the 843 wireless medium to compensate for the potential loss that is incurred 844 with radio transmissions. 846 For instance, when the source sends to Node A, Node B may listen 847 promiscuously and get a second chance to receive the frame without an 848 additional transmission. Note that B would not have to listen if it 849 already received that particular frame at an earlier timeslot in a 850 dedicated transmission towards B. 852 The PAREO model can be implemented in both centralized and 853 distributed scheduling approaches. In the centralized approach, a 854 Path Computation Element (PCE) scheduler calculates a Track and 855 schedules the communication. In the distributed approach, the Track 856 is computed within the network, and signaled in the packets, e.g., 857 using BIER-TE, Segment Routing, or a Source Routing Header. 859 3.4.1. Packet Replication 861 By employing a Packet Replication procedure, a Node forwards a copy 862 of each data packet to more than one successor. To do so, each Node 863 (i.e., Ingress and intermediate Node) sends the data packet multiple 864 times as separate unicast transmissions. For instance, in Figure 5, 865 the Ingress Node is transmitting the packet to both successors, nodes 866 A and B, at two different times. 868 ===> (A) => (C) => (E) === 869 // \\// \\// \\ 870 Ingress //\\ //\\ Egress 871 \\ // \\ // \\ // 872 ===> (B) => (D) => (F) === 874 Figure 5: Packet Replication 876 An example schedule is shown in Table 1. This way, the transmission 877 leverages with the time and spatial forms of diversity. 879 +=========+======+======+======+======+======+======+======+ 880 | Channel | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 881 +=========+======+======+======+======+======+======+======+ 882 | 0 | S->A | S->B | B->C | B->D | C->F | E->R | F->R | 883 +---------+------+------+------+------+------+------+------+ 884 | 1 | | A->C | A->D | C->E | D->E | D->F | | 885 +---------+------+------+------+------+------+------+------+ 887 Table 1: Packet Replication: Sample schedule 889 3.4.2. Packet Elimination 891 The replication operation increases the traffic load in the network, 892 due to packet duplications. This may occur at several stages inside 893 the Track, and to avoid an explosion of the number of copies, a 894 Packet Elimination procedure must be applied as well. To this aim, 895 once a Node receives the first copy of a data packet, it discards the 896 subsequent copies. 898 The logical functions of Replication and Elimination may be 899 collocated in an intermediate Node, the Node first eliminating the 900 redundant copies and then sending the packet exactly once to each of 901 the selected successors. 903 3.4.3. Promiscuous Overhearing 905 Considering that the wireless medium is broadcast by nature, any 906 neighbor of a transmitter may overhear a transmission. By employing 907 the Promiscuous Overhearing operation, the next hops have additional 908 opportunities to capture the data packets. In Figure 6, when Node A 909 is transmitting to its DP (Node C), the AP (Node D) and its sibling 910 (Node B) may decode this data packet as well. As a result, by 911 employing corellated paths, a Node may have multiple opportunities to 912 receive a given data packet. 914 ===> (A) ====> (C) ====> (E) ==== 915 // ^ | \\ \\ 916 Ingress | | \\ Egress 917 \\ | v \\ // 918 ===> (B) ====> (D) ====> (F) ==== 920 Figure 6: Unicast with Overhearing 922 3.4.4. Constructive Interference 924 Constructive Interference can be seen as the reverse of Promiscuous 925 Overhearing, and refers to the case where two senders transmit the 926 exact same signal in a fashion that the emitted symbols add up at the 927 receiver and permit a reception that would not be possible with a 928 single sender at the same PHY mode and the same power level. 930 Constructive Interference was proposed on 5G, Wi-Fi7 and even tested 931 on IEEE Std 802.14.5. The hard piece is to synchronize the senders 932 to the point that the signals are emitted at slightly different time 933 to offset the difference of propagation delay that corresponds to the 934 difference of distance of the transmitters to the receiver at the 935 speed of light to the point that the symbols are superposed long 936 enough to be recognizable. 938 4. The RAW Architecture 940 4.1. The RAW Conceptual Model 942 RAW inherits the conceptual model described in section 4 of the 943 DetNet Architecture [RFC8655]. RAW extends the DetNet service layer 944 to provide additional agility against transmission loss. 946 A RAW Network Plane may be strict or loose, depending on whether RAW 947 observes and takes actions on all hops or not. For instance, the 948 packets between two wireless entities may be relayed over a wired 949 infrastructure such as a Wi-Fi extended service set (ESS) or a 5G 950 Core; in that case, RAW observes and control the transmission over 951 the wireless first and last hops, as well as end-to-end metrics such 952 as latency, jitter, and delivery ratio. This operation is loose 953 since the structure and properties of the wired infrastructure are 954 ignored, and may be either controlled by other means such as DetNet/ 955 TSN, or neglected in the face of the wireless hops. 957 A Controller Plane Function (CPF) called the Path Computation Element 958 (PCE) [RFC4655] interacts with RAW Nodes over a Southbound API. The 959 RAW Nodes are DetNet relays that are capable of additional diversity 960 mechanisms and measurement functions related to the radio interface, 961 in particular the PAREO diversity mechanisms. 963 The PCE defines a complex Track between an Ingress End System and an 964 Egress End System, and indicates to the RAW Nodes where the PAREO 965 operations may be actionned in the Network Plane. The Track may be 966 expressed loosely to enable traversing a non-RAW subnetwork. In that 967 case, the expectation is that the non-RAW subnetwork can be neglected 968 in the RAW computation, that is, considered infinitely fast, reliable 969 and/or available in comparison with the links between RAW nodes. 971 CPF CPF CPF CPF 973 Southbound API 974 _-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._- 975 _-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._- 977 RAW --z RAW --z RAW --z RAW 978 z-- Node z-- Node z-- Node z-- Node --z 979 Ingress --z / / z-- Egress 980 End \ \ .. . End 981 Node ---z / / .. .. . z-- Node 982 z-- RAW --z RAW ( non-RAW ) -- RAW --z 983 Node z-- Node --- ( Nodes ) Node 984 ... . 985 --z wireless wired 986 z-- link --- link 988 Figure 7: RAW Nodes 990 The Link-Layer metrics are reported to the PCE in a time-aggregated, 991 e.g., statistical fashion. Example Link-Layer metrics include 992 typical Link bandwidth (the medium speed depends dynamically on the 993 PHY mode and the number of users sharing the spectrum) and average 994 and mean squared deviation of availability and reliability figures 995 such as Packet Delivery Ratio (PDR) over long periods of time. 997 Based on those metrics, the PCE installs the Track with enough 998 redundant forwarding solutions to ensure that the Network Plane can 999 reliably deliver the packets within a System Level Agreement (SLA) 1000 associated to the flows that it transports. The SLA defines end-to- 1001 end reliability and availability requirements, where reliability may 1002 be expressed as a successful delivery in order and within a bounded 1003 delay of at least one copy of a packet. 1005 Depending on the use case and the SLA, the Track may comprise non-RAW 1006 segments, either interleaved inside the Track, or all the way to the 1007 Egress End Node (e.g., a server in the Internet). RAW observes the 1008 Lower-Layer Links between RAW nodes (typically, radio links) and the 1009 end-to-end Network Layer operation to decide at all times which of 1010 the PAREO diversity schemes is actioned by which RAW Nodes. 1012 Once a Track is established, per-segment and end-to-end reliability 1013 and availability statistics are periodically reported to the PCE to 1014 assure that the SLA can be met or have it recompute the Track if not. 1016 4.2. The Path Selection Engine 1018 RAW separates the path computation time scale at which a complex path 1019 is recomputed from the path selection time scale at which the 1020 forwarding decision is taken for one or a few packets (more in 1021 Section 3.2). RAW operates at the path selection time scale. The 1022 RAW problem is to decide, within the redundant solutions that are 1023 proposed by the PCE, which will be used for each packet to provide a 1024 Reliable and Available service while minimizing the waste of 1025 constrained resources. 1027 To that effect, RAW defines the Path Selection Engine (PSE) that is 1028 the counter-part of the PCE to perform rapid local adjustments of the 1029 forwarding tables within the diversity that the PCE has selected for 1030 the Track. The PSE enables to exploit the richer forwarding 1031 capabilities with PAREO and scheduled transmissions at a faster time 1032 scale over the smaller domain that is the Track, in either a loose or 1033 a strict fashion. 1035 Compared to the PCE, the PSE operates on metrics that evolve faster, 1036 but that needs to be advertised at a fast rate but only locally, 1037 within the Track. The forwarding decision may also change rapidly, 1038 but wiht a scope that is also contained within the Track, with no 1039 visibility to the other Tracks and flows in the network. This is as 1040 opposed to the PCE that needs to observe the whole network, and 1041 optimize all the Tracks globally, which can only be done at a slow 1042 pace and using long-term statistical metrics, as presented in 1043 Table 2. 1045 +===============+========================+===================+ 1046 | | PCE (Not in Scope) | PSE (In Scope) | 1047 +===============+========================+===================+ 1048 | Operation | Centralized | Source-Routed or | 1049 | | | Distributed | 1050 +---------------+------------------------+-------------------+ 1051 | Communication | Slow, expensive | Fast, local | 1052 +---------------+------------------------+-------------------+ 1053 | Time Scale | hours and above | seconds and below | 1054 +---------------+------------------------+-------------------+ 1055 | Network Size | Large, many Tracks to | Small, within one | 1056 | | optimize globally | Track | 1057 +---------------+------------------------+-------------------+ 1058 | Considered | Averaged, Statistical, | Instant values / | 1059 | Metrics | Shade of grey | boolean condition | 1060 +---------------+------------------------+-------------------+ 1062 Table 2: PCE vs. PSE 1064 The PSE sits in the DetNet Service sub-Layer of Edge and Relay Nodes. 1065 On the one hand, it operates on the packet flow, learning the Track 1066 and path selection information from the packet, possibly making local 1067 decision and retagging the packet to indicate so. On the other hand, 1068 the PSE interacts with the lower layers and with its peers to obtain 1069 up-to-date information about its radio links and the quality of the 1070 overall Track, respectively, as illustrated in Figure 8. 1072 | 1073 packet | going 1074 down the | stack 1075 +==========v==========+=====================+=====================+ 1076 | (iOAM + iCTRL) | (L2 Triggers, DLEP) | (oOAM) | 1077 +==========v==========+=====================+=====================+ 1078 | Learn from Learn from | 1079 | packet tagging Maintain end-to-end | 1080 +----------v----------+ Forwarding OAM packets | 1081 | Forwarding decision < State +---------^-----------| 1082 +----------v----------+ | Enrich or | 1083 + Retag Packet | Learn abstracted > Regenerate | 1084 | and Forward | metrics about Links | OAM packets | 1085 +..........v..........+..........^..........+.........^.v.........+ 1086 | Lower layers | 1087 +..........v.....................^....................^.v.........+ 1088 frame | sent Frame | L2 Ack oOAM | | packet 1089 over | wireless In | In | | and out 1090 v | | v 1092 Figure 8: PSE 1094 4.3. RAW OAM 1096 RAW In-situ OAM operation in the Network Plane may observe either a 1097 full Track or subTracks that are being used at this time. Active RAW 1098 OAM may be needed to observe the unused segments and evaluate the 1099 desirability of a rerouting decision. Finally, the RAW Service Layer 1100 Assurance may observe the individual PAREO operation of a relay node 1101 to ensure that it is conforming; this might require injecting an OAM 1102 packet at an upstream point inside the Track and extracting that 1103 packet at another point downstream before it reaches the egress. 1105 This observation feeds the RAW PSE that makes the decision on which 1106 PAREO function in actioned at which RAW Node, for one a small 1107 continuous series of packets. 1109 ... .. 1110 RAN 1 ----- ... .. ... 1111 / . .. .... 1112 +-------+ / . .. .... +------+ 1113 |Ingress|- . ..... |Egress| 1114 | End |------ RAN 2 -- . Internet ....---| End | 1115 |System |- .. ..... |System| 1116 +-------+ \ . ...... +------+ 1117 \ ... ... ..... 1118 RAN n -------- ... ..... 1120 <------------------> <--------------------> 1121 Observed by OAM Opaque to OAM 1123 Figure 9: Observed Links in Radio Access Protection 1125 In the case of a End-to-End Protection in a Wireless Mesh, the Track 1126 is strict and congruent with the path so all links are observed. 1127 Conversely, in the case of Radio Access Protection, the Track is 1128 Loose and in that case only the first hop is observed; the rest of 1129 the path is abstracted and considered infinitely reliable. 1131 In the case of the Radio Access Protection, only the first hop is 1132 protected; the loss of a packet that was sent over one of the 1133 possible first hops is attributed to that first hop, even if a 1134 particular loss effectively happens farther down the path. 1136 The Links that are not observed by OAM are opaque to it, meaning that 1137 the OAM information is carried across and possibly echoed as data, 1138 but there is no information capture in intermediate nodes. In the 1139 example above, the Internet is opaque and not controlled by RAW; 1140 still the RAW OAM measures the end-to-end latency and delivery ratio 1141 for packets sent via each if RAN 1, RAN 2 and RAN 3, and determines 1142 whether a packet should be sent over either or a collection of those 1143 access links. 1145 4.3.1. DetNet OAM 1147 [detnet] provides an OAM framework with [DetNet-OAM] that applies 1148 within the DetNet dataplane described in [DetNet-DP],which is 1149 typically based on MPLS or IPv6 pseudowires. How the framework 1150 applies to IPv6 is detailed in [DetNet-IP-OAM]. Within that 1151 framework, OAM messages follow the same forward path as the data 1152 packets and gather information about their individual treatment at 1153 each hop. When the destination receives an OAM message, it gets a 1154 view on the full path or at least of a segment of the path from the 1155 source of the flow. 1157 In-situ OAM (IOAM) adds telemetry information about the experience of 1158 one packet within the packet itself [I-D.ietf-ippm-ioam-data], with 1159 the caveats that the measurement and the consecutive update of the 1160 packet interfere with the operation being observed, e.g., may 1161 increase the latency of the packet for which it is measured and into 1162 which it is stamped. 1164 Note: IOAM and analogous on-path telemetry methods are capable of 1165 facilitating collection of useful telemetry information that 1166 characterizes the state of a system as experienced by the packet. 1167 But because of statistical character of a packet network, these 1168 methods may not be used to monitor the continuity of a path (Track) 1169 or proper connectivity of the Track (no leaking packets across 1170 Tracks). 1172 This effect can be alleviated by measuring on the fly but reporting 1173 later, e.g., by exporting the data as a separate management packet 1174 [I-D.ietf-ippm-ioam-direct-export]. 1175 [I-D.mirsky-ippm-hybrid-two-step] proposes an hybrid two-steps method 1176 (HTS) where a trigger message starts the measurement and a follow up 1177 along the Track packet gathers the measured data. 1179 "Error Performance Measurement" [I-D.mirsky-ippm-epm] uses Fault 1180 Management (FM) and Performance Management (PM) OAM mechanisms to 1181 determine availability/unavailability of a path according to 1182 predefined SLA. 1184 4.3.2. RAW Extensions 1186 Classical OAM typically measures information at the transmitter, 1187 e.g., residence time in the node or transmit queue size. With RAW, 1188 there is a need to combine information at the sender (number of 1189 retries) with that at the receiver (LQI, RSSI). This doubles the 1190 operating cost of an IAOM processing that would gather the experience 1191 of a single packet. 1193 The RAW PSE may be centralized at the Track Ingress, or distributed 1194 long the Track. Either way, the PSE needs instant information about 1195 the rest of the way to the destination over the possible next-hop 1196 adjacencies along the Track in order to decide how to perform simple 1197 forwarding, load balancing, and/or replication, as well as 1198 determining how much latency credit is available for ARQ. 1200 To provide that information timely, it makes sense that the OAM 1201 packets that gather instantaneous values from the radio senders and 1202 receivers at each hop flow on the reverse path and inform the PSE at 1203 the source and/or the PAREO relays about the state of the rest of the 1204 way. This is achieved using Reverse OAM packets that flow along the 1205 Reversed Track, West to East. 1207 Because the quality of transmission over a wireless medium varies 1208 continuously, it is important that RAW OAM captures the state of the 1209 medium across an adjacency over multiple transmission and over a 1210 recent period of time, whether the transmitted packets belong to this 1211 flow or another. Some of the measured information relates to the 1212 medium itself. In other words, the captured information does not 1213 only relate to the experience of one packet as is the case for IOAM, 1214 but also to the medium itself. This makes an approach like HTS more 1215 suitable as it can trigger the capture of multiple measurements over 1216 a short period of time. On the other hand, the PSE needs a 1217 continuous measurement stream where a single trigger is followed by a 1218 periodic follow up capture. 1220 In other words, the best suited OAM method to enable the PSE make 1221 accurate PAREO forwarding decisions is a periodic variation of the 1222 two-steps method flowing along the reverse Track, as a Reverse OAM 1223 technique. [RAW-OAM] provides more information on the RAW OAM 1224 problem and solution approaches. 1226 4.3.3. Observed Metrics 1228 The Dynamic Link Exchange Protocol (DLEP) [RFC8175] from [MANET] can 1229 be leveraged at each hop to derive generic radio metrics (e.g., based 1230 on LQI, RSSI, queueing delays and ETX) on individual hops. 1232 Those lower-layer metrics are aggregated along a multihop segment 1233 into abstract layer 3 information that reflect the instant 1234 reliability and latency of the observed path. 1236 4.4. Flow Identification vs. Path Identification 1238 Section 4.7 of the DetNet Architecture [RFC8655] ties the app-flow 1239 identification which is an appliation layer concept with the network 1240 path identification that depends on the networking technology by 1241 "exporting of flow identification", e.g., to a MPLS label. 1243 With RAW, this exporting operation is injective but not bijective. 1244 e.g., a flow is fully placed within one RAW Track, but not all 1245 packets along that Track are necessarily part of the same flow. For 1246 instance, out-of-band OAM packets must circulate in the exact same 1247 fashion as the flows that they observe. It results that the flow 1248 identification that maps to to app-flow at the network layer must be 1249 separate from the path identification that is used to forward a 1250 packet. 1252 Section 3.4 of the DetNet data-plane framework [DetNet-DP] indicates 1253 that for a DetNet IP Data Plane, a flow is identified by an IPv6 1254 6-tuple. With RAW, that 6-tuple is not what indicates the Track, in 1255 other words, the flow ID is not the Track ID. 1257 For instance, the 6TiSCH Architecture [6TiSCH-ARCHI] uses a 1258 combination of the address of the Egress End System and an instance 1259 identifier in a Hop-by-hop option to indicate a Track. This way, if 1260 a packet "escapes" the Track, it will reach the Track Egress point 1261 through normal routing and be treated at the service layer through, 1262 say, elimination and reordering. 1264 The RAW service includes forwarding over a subset of the Links that 1265 form the Track (a subTrack). Packets from the same or a different 1266 flow that are routed through the same Track will not necessarily 1267 traverse the same Links. The PSE selects a subTrack for a packet 1268 based on the links that are preferred and those that should be 1269 avoided at this time. 1271 Each packet is forwarded within the subTrack that provides the best 1272 adequation with the SLA of the flow and the energy and bandwidth 1273 constraints of the network. 1275 Flow 1 (6-tuple) ----+ 1276 | 1277 Flow 2 (6-tuple) ---+ | 1278 | | 1279 OAM -----------+ | | 1280 | | | 1281 | | | 1282 | | | | | 1283 | v v v | 1284 | | 1285 +---------+---------+ 1286 | 1287 | 1288 Track i (Ingress IP Address, RPLinstanceId) 1289 | 1290 | 1291 | 1292 +---------+-----+--....-------+ 1293 | | | 1294 | | | 1295 subTrack 1 subTrack 2 subTrack n 1296 | | | 1297 | | | 1298 V V V 1299 +-----------------------------------+ 1300 | | 1301 | Destination | 1302 | | 1303 +-----------------------------------+ 1305 Figure 10: Flow Injection 1307 With 6TiSCH, packets are tagged with the same (destination address, 1308 instance ID) will experience the same RAW service regardless of the 1309 IPv6 6-tuple that indicates the flow. The forwarding does not depend 1310 on whether the packets transport application flows or OAM. In the 1311 generic case, the Track or the subTrack can be signaled in the packet 1312 through other means, e.g., encoded in the suffix of the destination 1313 address as a Segment Routing Service Instruction [SR-ARCHI], or 1314 leveraging Bit Index Explicit Replication [BIER] Traffic Engineering 1315 [BIER-TE]. 1317 4.5. Source-Routed vs. Distributed Forwarding Decision 1319 Within a large routed topology, the route-over mesh operation builds 1320 a particular complex Track with one source and one or more 1321 destinations; within the Track, packets may follow different paths 1322 and may be subject to RAW forwarding operations that include 1323 replication, elimination, retries, overhearing and reordering. 1325 The RAW forwarding decisions include the selection of points of 1326 replication and elimination, how many retries can take place, and a 1327 limit of validity for the packet beyond which the packet should be 1328 destroyed rather than forwarded uselessly further down the Track. 1330 The decision to apply the RAW techniques must be done quickly, and 1331 depends on a very recent and precise knowledge of the forwarding 1332 conditions within the complex Track. There is a need for an 1333 observation method to provide the RAW Data Plane with the specific 1334 knowledge of the state of the Track for the type of flow of interest 1335 (e.g., for a QoS level of interest). To observe the whole Track in 1336 quasi real time, RAW considers existing tools such as L2-triggers, 1337 DLEP, BFD and leverages in-band and out-of-band OAM to capture and 1338 report that information to the PSE. 1340 One possible way of making the RAW forwarding decisions within a 1341 Track is to position a unique PSE at the Ingress and express its 1342 decision in-band in the packet, which requires the explicit signaling 1343 of the subTrack within the Track. In that case, the RAW forwarding 1344 operation along the Track is encoded by the source, e.g., by 1345 indicating the subTrack in the Segment Routing (SRv6) Service 1346 Instruction, or by leveraging BIER-TE such as done with [BIER-PREF]. 1348 The alternate way is to operate the PSE in each forwarding Node, 1349 which makes the RAW forwarding decisions for a packet on its own, 1350 based on its knowledge of the expectation (timeliness and 1351 reliability) for that packet and a recent observation of the rest of 1352 the way across the possible paths based on OAM. Information about 1353 the desired service should be placed in the packet and matched with 1354 the forwarding Node's capabilities and policies. 1356 In either case, a per-track/subTrack state is installed in all the 1357 intermediate Nodes to recognize the packets that are following a 1358 Track and determine the forwarding operation to be applied. 1360 4.6. Encapsulation and Decapsulation 1362 In the generic case where the Track Ingress Node is not the source of 1363 the Packet, the Ingress Node needs to encapsulate IP-in-IP to ensure 1364 that the Destination IP Address is that of the Egress Node and that 1365 the necessary Headers (Routing Header, Segment Routing Header and/or 1366 Hop-By-Hop Header) can be added to the packet to signal the Track or 1367 the subTrack, conforming [IPv6] that discourages the insertion of a 1368 Header on the fly. 1370 In the specific case where the Ingress Node is the source of the 1371 packet, the encapsulation can be avoided, provided that the source 1372 adds the necessary headers and that the destination is set to the 1373 Egress Node. Forwarding to a final destination beyond the Egress 1374 Node is possible, e.g., with a Segment Routing Header that signals 1375 the rest of the way. In that case a Hop-by-Hop Header is not 1376 recommmended since its validity is within the Track only. 1378 5. Security Considerations 1380 RAW uses all forms of diversity including radio technology and 1381 physical path to increase the reliability and availability in the 1382 face of unpredictable conditions. While this is not done 1383 specifically to defeat an attacker, the amount of diversity used in 1384 RAW makes an attack harder to achieve. 1386 5.1. Forced Access 1388 RAW will typically select the cheapest collection of links that 1389 matches the requested SLA, for instance, leverage free WI-Fi vs. paid 1390 3GPP access. By defeating the cheap connectivity (e.g., PHY-layer 1391 interference) the attacker can force an End System to use the paid 1392 access and increase the cost of the transmission for the user. 1394 6. IANA Considerations 1396 This document has no IANA actions. 1398 7. Contributors 1400 The editor wishes to thank: 1402 Xavi Vilajosana: Wireless Networks Research Lab, Universitat Oberta 1403 de Catalunya 1405 Remous-Aris Koutsiamanis: IMT Atlantique 1406 Nicolas Montavont: IMT Atlantique 1408 Rex Buddenberg: Individual contributor 1410 Greg Mirsky: ZTE 1412 for their contributions to the text and ideas exposed in this 1413 document. 1415 8. Acknowledgments 1417 TBD 1419 9. References 1421 9.1. Normative References 1423 [6TiSCH-ARCHI] 1424 Thubert, P., Ed., "An Architecture for IPv6 over the Time- 1425 Slotted Channel Hopping Mode of IEEE 802.15.4 (6TiSCH)", 1426 RFC 9030, DOI 10.17487/RFC9030, May 2021, 1427 . 1429 [RAW-TECHNOS] 1430 Thubert, P., Cavalcanti, D., Vilajosana, X., Schmitt, C., 1431 and J. Farkas, "Reliable and Available Wireless 1432 Technologies", Work in Progress, Internet-Draft, draft- 1433 ietf-raw-technologies-01, 19 February 2021, 1434 . 1437 [RAW-USE-CASES] 1438 Papadopoulos, G. Z., Thubert, P., Theoleyre, F., and C. J. 1439 Bernardos, "RAW use cases", Work in Progress, Internet- 1440 Draft, draft-ietf-raw-use-cases-01, 21 February 2021, 1441 . 1444 [RFC4655] Farrel, A., Vasseur, J.-P., and J. Ash, "A Path 1445 Computation Element (PCE)-Based Architecture", RFC 4655, 1446 DOI 10.17487/RFC4655, August 2006, 1447 . 1449 [BFD] Katz, D. and D. Ward, "Bidirectional Forwarding Detection 1450 (BFD)", RFC 5880, DOI 10.17487/RFC5880, June 2010, 1451 . 1453 [RFC6291] Andersson, L., van Helvoort, H., Bonica, R., Romascanu, 1454 D., and S. Mansfield, "Guidelines for the Use of the "OAM" 1455 Acronym in the IETF", BCP 161, RFC 6291, 1456 DOI 10.17487/RFC6291, June 2011, 1457 . 1459 [RFC7799] Morton, A., "Active and Passive Metrics and Methods (with 1460 Hybrid Types In-Between)", RFC 7799, DOI 10.17487/RFC7799, 1461 May 2016, . 1463 [RFC8578] Grossman, E., Ed., "Deterministic Networking Use Cases", 1464 RFC 8578, DOI 10.17487/RFC8578, May 2019, 1465 . 1467 [IPv6] Deering, S. and R. Hinden, "Internet Protocol, Version 6 1468 (IPv6) Specification", STD 86, RFC 8200, 1469 DOI 10.17487/RFC8200, July 2017, 1470 . 1472 [SR-ARCHI] Filsfils, C., Ed., Previdi, S., Ed., Ginsberg, L., 1473 Decraene, B., Litkowski, S., and R. Shakir, "Segment 1474 Routing Architecture", RFC 8402, DOI 10.17487/RFC8402, 1475 July 2018, . 1477 [BIER] Wijnands, IJ., Ed., Rosen, E., Ed., Dolganow, A., 1478 Przygienda, T., and S. Aldrin, "Multicast Using Bit Index 1479 Explicit Replication (BIER)", RFC 8279, 1480 DOI 10.17487/RFC8279, November 2017, 1481 . 1483 [RFC8175] Ratliff, S., Jury, S., Satterwhite, D., Taylor, R., and B. 1484 Berry, "Dynamic Link Exchange Protocol (DLEP)", RFC 8175, 1485 DOI 10.17487/RFC8175, June 2017, 1486 . 1488 [RFC8557] Finn, N. and P. Thubert, "Deterministic Networking Problem 1489 Statement", RFC 8557, DOI 10.17487/RFC8557, May 2019, 1490 . 1492 [RFC8655] Finn, N., Thubert, P., Varga, B., and J. Farkas, 1493 "Deterministic Networking Architecture", RFC 8655, 1494 DOI 10.17487/RFC8655, October 2019, 1495 . 1497 [RFC9049] Dawkins, S., Ed., "Path Aware Networking: Obstacles to 1498 Deployment (A Bestiary of Roads Not Taken)", RFC 9049, 1499 DOI 10.17487/RFC9049, June 2021, 1500 . 1502 9.2. Informative References 1504 [RFC0791] Postel, J., "Internet Protocol", STD 5, RFC 791, 1505 DOI 10.17487/RFC0791, September 1981, 1506 . 1508 [TE] Awduche, D., Chiu, A., Elwalid, A., Widjaja, I., and X. 1509 Xiao, "Overview and Principles of Internet Traffic 1510 Engineering", RFC 3272, DOI 10.17487/RFC3272, May 2002, 1511 . 1513 [STD 62] Harrington, D., Presuhn, R., and B. Wijnen, "An 1514 Architecture for Describing Simple Network Management 1515 Protocol (SNMP) Management Frameworks", STD 62, RFC 3411, 1516 DOI 10.17487/RFC3411, December 2002, 1517 . 1519 [RFC4090] Pan, P., Ed., Swallow, G., Ed., and A. Atlas, Ed., "Fast 1520 Reroute Extensions to RSVP-TE for LSP Tunnels", RFC 4090, 1521 DOI 10.17487/RFC4090, May 2005, 1522 . 1524 [FRR] Shand, M. and S. Bryant, "IP Fast Reroute Framework", 1525 RFC 5714, DOI 10.17487/RFC5714, January 2010, 1526 . 1528 [RLFA-FRR] Bryant, S., Filsfils, C., Previdi, S., Shand, M., and N. 1529 So, "Remote Loop-Free Alternate (LFA) Fast Reroute (FRR)", 1530 RFC 7490, DOI 10.17487/RFC7490, April 2015, 1531 . 1533 [DetNet-DP] 1534 Varga, B., Ed., Farkas, J., Berger, L., Malis, A., and S. 1535 Bryant, "Deterministic Networking (DetNet) Data Plane 1536 Framework", RFC 8938, DOI 10.17487/RFC8938, November 2020, 1537 . 1539 [BIER-PREF] 1540 Thubert, P., Eckert, T., Brodard, Z., and H. Jiang, "BIER- 1541 TE extensions for Packet Replication and Elimination 1542 Function (PREF) and OAM", Work in Progress, Internet- 1543 Draft, draft-thubert-bier-replication-elimination-03, 3 1544 March 2018, . 1547 [DetNet-IP-OAM] 1548 Mirsky, G., Chen, M., and D. Black, "Operations, 1549 Administration and Maintenance (OAM) for Deterministic 1550 Networks (DetNet) with IP Data Plane", Work in Progress, 1551 Internet-Draft, draft-ietf-detnet-ip-oam-02, 30 March 1552 2021, . 1555 [RAW-5G] Farkas, J., Dudda, T., Shapin, A., and S. Sandberg, "5G - 1556 Ultra-Reliable Wireless Technology with Low Latency", Work 1557 in Progress, Internet-Draft, draft-farkas-raw-5g-00, 1 1558 April 2020, . 1561 [BIER-TE] Eckert, T., Cauchie, G., and M. Menth, "Tree Engineering 1562 for Bit Index Explicit Replication (BIER-TE)", Work in 1563 Progress, Internet-Draft, draft-ietf-bier-te-arch-09, 30 1564 October 2020, . 1567 [IPoWIRELESS] 1568 Thubert, P., "IPv6 Neighbor Discovery on Wireless 1569 Networks", Work in Progress, Internet-Draft, draft- 1570 thubert-6man-ipv6-over-wireless-09, 17 May 2021, 1571 . 1574 [RAW-OAM] Theoleyre, F., Papadopoulos, G. Z., Mirsky, G., and C. J. 1575 Bernardos, "Operations, Administration and Maintenance 1576 (OAM) features for RAW", Work in Progress, Internet-Draft, 1577 draft-ietf-raw-oam-support-02, 3 June 2021, 1578 . 1581 [I-D.ietf-ippm-ioam-direct-export] 1582 Song, H., Gafni, B., Zhou, T., Li, Z., Brockners, F., 1583 Bhandari, S., Sivakolundu, R., and T. Mizrahi, "In-situ 1584 OAM Direct Exporting", Work in Progress, Internet-Draft, 1585 draft-ietf-ippm-ioam-direct-export-03, 17 February 2021, 1586 . 1589 [DetNet-OAM] 1590 Mirsky, G., Theoleyre, F., Papadopoulos, G. Z., and C. J. 1591 Bernardos, "Framework of Operations, Administration and 1592 Maintenance (OAM) for Deterministic Networking (DetNet)", 1593 Work in Progress, Internet-Draft, draft-ietf-detnet-oam- 1594 framework-01, 19 May 2021, 1595 . 1598 [I-D.mirsky-ippm-hybrid-two-step] 1599 Mirsky, G., Lingqiang, W., Zhui, G., and H. Song, "Hybrid 1600 Two-Step Performance Measurement Method", Work in 1601 Progress, Internet-Draft, draft-mirsky-ippm-hybrid-two- 1602 step-09, 30 March 2021, 1603 . 1606 [I-D.mirsky-ippm-epm] 1607 Mirsky, G., Min, X., and L. Han, "Error Performance 1608 Measurement in Packet-switched Networks", Work in 1609 Progress, Internet-Draft, draft-mirsky-ippm-epm-03, 26 1610 March 2021, . 1613 [I-D.mirsky-bfd-mpls-demand] 1614 Mirsky, G., "BFD in Demand Mode over Point-to-Point MPLS 1615 LSP", Work in Progress, Internet-Draft, draft-mirsky-bfd- 1616 mpls-demand-09, 30 March 2021, 1617 . 1620 [I-D.ietf-ippm-ioam-data] 1621 Brockners, F., Bhandari, S., and T. Mizrahi, "Data Fields 1622 for In-situ OAM", Work in Progress, Internet-Draft, draft- 1623 ietf-ippm-ioam-data-12, 21 February 2021, 1624 . 1627 [NASA] Adams, T., "RELIABILITY: Definition & Quantitative 1628 Illustration", . 1631 [MANET] IETF, "Mobile Ad hoc Networking", 1632 . 1634 [detnet] IETF, "Deterministic Networking", 1635 . 1637 [SPRING] IETF, "Source Packet Routing in Networking", 1638 . 1640 [BIER] IETF, "Bit Indexed Explicit Replication", 1641 . 1643 [BFD] IETF, "Bidirectional Forwarding Detection", 1644 . 1646 [CCAMP] IETF, "Common Control and Measurement Plane", 1647 . 1649 [IPPM] IETF, "IP Performance Measurement", 1650 . 1652 Authors' Addresses 1654 Pascal Thubert (editor) 1655 Cisco Systems, Inc 1656 Building D 1657 45 Allee des Ormes - BP1200 1658 06254 MOUGINS - Sophia Antipolis 1659 France 1661 Phone: +33 497 23 26 34 1662 Email: pthubert@cisco.com 1664 Georgios Z. Papadopoulos 1665 IMT Atlantique 1666 Office B00 - 114A 1667 2 Rue de la Chataigneraie 1668 35510 Cesson-Sevigne - Rennes 1669 France 1671 Phone: +33 299 12 70 04 1672 Email: georgios.papadopoulos@imt-atlantique.fr 1674 Lou Berger 1675 LabN Consulting, L.L.C. 1676 United States of America 1678 Email: lberger@labn.net