idnits 2.17.1 draft-ietf-ippm-ioam-data-16.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (November 8, 2021) is 871 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) -- Looks like a reference, but probably isn't: '0' on line 622 -- Looks like a reference, but probably isn't: '1' on line 626 -- Possible downref: Non-RFC (?) normative reference: ref. 'POSIX' == Outdated reference: A later version (-07) exists of draft-ietf-ippm-ioam-data-integrity-00 == Outdated reference: A later version (-05) exists of draft-ietf-ippm-ioam-deployment-00 == Outdated reference: A later version (-13) exists of draft-ietf-nvo3-vxlan-gpe-12 == Outdated reference: A later version (-07) exists of draft-spiegel-ippm-ioam-rawexport-05 Summary: 0 errors (**), 0 flaws (~~), 5 warnings (==), 4 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 ippm F. Brockners, Ed. 3 Internet-Draft Cisco 4 Intended status: Standards Track S. Bhandari, Ed. 5 Expires: May 12, 2022 Thoughtspot 6 T. Mizrahi, Ed. 7 Huawei 8 November 8, 2021 10 Data Fields for In-situ OAM 11 draft-ietf-ippm-ioam-data-16 13 Abstract 15 In-situ Operations, Administration, and Maintenance (IOAM) records 16 operational and telemetry information in the packet while the packet 17 traverses a path in the network. This document discusses the data 18 fields and associated data types for in-situ OAM. In-situ OAM data 19 fields can be encapsulated into a variety of protocols such as NSH, 20 Segment Routing, Geneve, or IPv6. In-situ OAM can be used to 21 complement OAM mechanisms based on, e.g., ICMP or other types of 22 probe packets. 24 Status of This Memo 26 This Internet-Draft is submitted in full conformance with the 27 provisions of BCP 78 and BCP 79. 29 Internet-Drafts are working documents of the Internet Engineering 30 Task Force (IETF). Note that other groups may also distribute 31 working documents as Internet-Drafts. The list of current Internet- 32 Drafts is at https://datatracker.ietf.org/drafts/current/. 34 Internet-Drafts are draft documents valid for a maximum of six months 35 and may be updated, replaced, or obsoleted by other documents at any 36 time. It is inappropriate to use Internet-Drafts as reference 37 material or to cite them other than as "work in progress." 39 This Internet-Draft will expire on May 12, 2022. 41 Copyright Notice 43 Copyright (c) 2021 IETF Trust and the persons identified as the 44 document authors. All rights reserved. 46 This document is subject to BCP 78 and the IETF Trust's Legal 47 Provisions Relating to IETF Documents 48 (https://trustee.ietf.org/license-info) in effect on the date of 49 publication of this document. Please review these documents 50 carefully, as they describe your rights and restrictions with respect 51 to this document. Code Components extracted from this document must 52 include Simplified BSD License text as described in Section 4.e of 53 the Trust Legal Provisions and are provided without warranty as 54 described in the Simplified BSD License. 56 Table of Contents 58 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 59 2. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 3 60 3. Conventions . . . . . . . . . . . . . . . . . . . . . . . . . 4 61 4. Scope, Applicability, and Assumptions . . . . . . . . . . . . 5 62 5. IOAM Data-Fields, Types, Nodes . . . . . . . . . . . . . . . 6 63 5.1. IOAM Data-Fields and Option-Types . . . . . . . . . . . . 7 64 5.2. IOAM-Domains and types of IOAM Nodes . . . . . . . . . . 7 65 5.3. IOAM-Namespaces . . . . . . . . . . . . . . . . . . . . . 8 66 5.4. IOAM Trace Option-Types . . . . . . . . . . . . . . . . . 11 67 5.4.1. Pre-allocated and Incremental Trace Option-Types . . 13 68 5.4.2. IOAM node data fields and associated formats . . . . 17 69 5.4.2.1. Hop_Lim and node_id short format . . . . . . . . 18 70 5.4.2.2. ingress_if_id and egress_if_id . . . . . . . . . 19 71 5.4.2.3. timestamp seconds . . . . . . . . . . . . . . . . 19 72 5.4.2.4. timestamp fraction . . . . . . . . . . . . . . . 20 73 5.4.2.5. transit delay . . . . . . . . . . . . . . . . . . 20 74 5.4.2.6. namespace specific data . . . . . . . . . . . . . 20 75 5.4.2.7. queue depth . . . . . . . . . . . . . . . . . . . 21 76 5.4.2.8. Checksum Complement . . . . . . . . . . . . . . . 21 77 5.4.2.9. Hop_Lim and node_id wide . . . . . . . . . . . . 22 78 5.4.2.10. ingress_if_id and egress_if_id wide . . . . . . . 22 79 5.4.2.11. namespace specific data wide . . . . . . . . . . 22 80 5.4.2.12. buffer occupancy . . . . . . . . . . . . . . . . 23 81 5.4.2.13. Opaque State Snapshot . . . . . . . . . . . . . . 23 82 5.4.3. Examples of IOAM node data . . . . . . . . . . . . . 24 83 5.5. IOAM Proof of Transit Option-Type . . . . . . . . . . . . 26 84 5.5.1. IOAM Proof of Transit Type 0 . . . . . . . . . . . . 28 85 5.6. IOAM Edge-to-Edge Option-Type . . . . . . . . . . . . . . 28 86 6. Timestamp Formats . . . . . . . . . . . . . . . . . . . . . . 31 87 6.1. PTP Truncated Timestamp Format . . . . . . . . . . . . . 31 88 6.2. NTP 64-bit Timestamp Format . . . . . . . . . . . . . . . 32 89 6.3. POSIX-based Timestamp Format . . . . . . . . . . . . . . 33 90 7. IOAM Data Export . . . . . . . . . . . . . . . . . . . . . . 35 91 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 35 92 8.1. IOAM Option-Type Registry . . . . . . . . . . . . . . . . 35 93 8.2. IOAM Trace-Type Registry . . . . . . . . . . . . . . . . 36 94 8.3. IOAM Trace-Flags Registry . . . . . . . . . . . . . . . . 37 95 8.4. IOAM POT-Type Registry . . . . . . . . . . . . . . . . . 38 96 8.5. IOAM POT-Flags Registry . . . . . . . . . . . . . . . . . 38 97 8.6. IOAM E2E-Type Registry . . . . . . . . . . . . . . . . . 38 98 8.7. IOAM Namespace-ID Registry . . . . . . . . . . . . . . . 39 99 9. Management and Deployment Considerations . . . . . . . . . . 40 100 10. Security Considerations . . . . . . . . . . . . . . . . . . . 41 101 11. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 43 102 12. References . . . . . . . . . . . . . . . . . . . . . . . . . 43 103 12.1. Normative References . . . . . . . . . . . . . . . . . . 43 104 12.2. Informative References . . . . . . . . . . . . . . . . . 44 105 Contributors' Addresses . . . . . . . . . . . . . . . . . . . . . 46 106 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 47 108 1. Introduction 110 This document defines data fields for "in-situ" Operations, 111 Administration, and Maintenance (IOAM). In-situ OAM records OAM 112 information within the packet while the packet traverses a particular 113 network domain. The term "in-situ" refers to the fact that the OAM 114 data is added to the data packets rather than being sent within 115 packets specifically dedicated to OAM. IOAM is to complement 116 mechanisms such as Ping or Traceroute. In terms of "active" or 117 "passive" OAM, "in-situ" OAM can be considered a hybrid OAM type. 118 "In-situ" mechanisms do not require extra packets to be sent. IOAM 119 adds information to the already available data packets and therefore 120 cannot be considered passive. In terms of the classification given 121 in [RFC7799], IOAM could be portrayed as Hybrid Type I. IOAM 122 mechanisms can be leveraged where mechanisms using, e.g., ICMP do not 123 apply or do not offer the desired results, such as proving that a 124 certain traffic flow takes a pre-defined path, SLA verification for 125 the data traffic, detailed statistics on traffic distribution paths 126 in networks that distribute traffic across multiple paths, or 127 scenarios in which probe traffic is potentially handled differently 128 from regular data traffic by the network devices. 130 The term "in situ OAM" was originally motivated by the use of OAM 131 related mechanisms that add information into a packet. This document 132 uses IOAM as a term defining the IOAM technology. IOAM includes "in- 133 situ" mechanisms, but also mechanisms that could trigger the creation 134 of additional packets dedicated to OAM. 136 2. Contributors 138 This document was the collective effort of several authors. The text 139 and content were contributed by the editors and the co-authors listed 140 below. The contact information of the co-authors appears at the end 141 of this document. 143 o Carlos Pignataro 144 o Mickey Spiegel 146 o Barak Gafni 148 o Jennifer Lemon 150 o Hannes Gredler 152 o John Leddy 154 o Stephen Youell 156 o David Mozes 158 o Petr Lapukhov 160 o Remy Chang 162 o Daniel Bernier 164 3. Conventions 166 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 167 "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and 168 "OPTIONAL" in this document are to be interpreted as described in BCP 169 14 [RFC2119] [RFC8174] when, and only when, they appear in all 170 capitals, as shown here. 172 Abbreviations and definitions used in this document: 174 E2E: Edge to Edge 176 Geneve: Generic Network Virtualization Encapsulation [RFC8926] 178 IOAM: In-situ Operations, Administration, and Maintenance 180 MTU: Maximum Transmit Unit 182 NSH: Network Service Header [RFC8300] 184 OAM: Operations, Administration, and Maintenance 186 PMTU: Path MTU 188 POT: Proof of Transit 190 Short format: "Short format" refers to an IOAM-Data-Field which 191 comprises 4 octets. 193 SID: Segment Identifier 195 SR: Segment Routing 197 VXLAN-GPE: Virtual eXtensible Local Area Network, Generic Protocol 198 Extension [I-D.ietf-nvo3-vxlan-gpe] 200 Wide format: "Wide format" refers to an IOAM-Data-Field which 201 comprises 8 octets. 203 4. Scope, Applicability, and Assumptions 205 IOAM assumes a set of constraints as well as guiding principles and 206 concepts that go hand in hand with the definition of the IOAM data 207 fields. These constraints, guiding principles, and concepts are 208 described in this section. A discussion of how IOAM data fields and 209 the associated concepts are applied to an IOAM deployment are out of 210 scope for this document. Please refer to 211 [I-D.ietf-ippm-ioam-deployment] for IOAM deployment considerations. 213 Scope: This document defines the data fields and associated data 214 types for in-situ OAM. The in-situ OAM data fields can be 215 encapsulated in a variety of protocols, including NSH, Segment 216 Routing, Geneve, and IPv6. Specification details for these different 217 protocols are outside the scope of this document. It is expected 218 that each such encapsulation would be specified by an RFC, jointly 219 designed by the working group that develops or maintains the 220 encapsulation protocol and the IETF IPPM working group. 222 Deployment domain (or scope) of in-situ OAM deployment: IOAM is 223 focused on "limited domains" as defined in [RFC8799]. For IOAM, a 224 limited domain could for example be an enterprise campus using 225 physical connections between devices or an overlay network using 226 virtual connections / tunnels for connectivity between said devices. 227 A limited domain which uses IOAM may constitute one or multiple 228 "IOAM-domains", each disambiguated through separate namespace 229 identifiers. An IOAM-domain is bounded by its perimeter or edge. 230 IOAM-domains may overlap inside the limited domain. Designers of 231 protocol encapsulations for IOAM specify mechanisms to ensure that 232 IOAM data stays within an IOAM-domain. In addition, the operator of 233 such a domain is expected to put provisions in place to ensure that 234 IOAM data does not leak beyond the edge of an IOAM-domain using, for 235 example, packet filtering methods. The operator SHOULD consider the 236 potential operational impact of IOAM to mechanisms such as ECMP 237 processing (e.g., load-balancing schemes based on packet length could 238 be impacted by the increased packet size due to IOAM), path MTU 239 (i.e., ensure that the MTU of all links within a domain is 240 sufficiently large to support the increased packet size due to IOAM) 241 and ICMP message handling (i.e., in case of IPv6, IOAM support for 242 ICMPv6 Echo Request/Reply is desired which would translate into 243 ICMPv6 extensions to enable IOAM-Data-Fields to be copied from an 244 Echo Request message to an Echo Reply message). 246 IOAM control points: IOAM-Data-Fields are added to or removed from 247 the user traffic by the devices which form the edge of a domain. 248 Devices which form an IOAM-Domain can add, update or remove IOAM- 249 Data-Fields. Edge devices of an IOAM-Domain can be hosts or network 250 devices. 252 Traffic-sets that IOAM is applied to: IOAM can be deployed on all or 253 only on subsets of the user traffic. Using IOAM on a selected set of 254 traffic (e.g., per interface, based on an access control list or flow 255 specification defining a specific set of traffic, etc.) could be 256 useful in deployments where the cost of processing IOAM-Data-Fields 257 by encapsulating, transit, or decapsulating node(s) might be a 258 concern from a performance or operational perspective. Thus limiting 259 the amount of traffic IOAM is applied to could be beneficial in some 260 deployments. 262 Encapsulation independence: The definition of IOAM-Data-Fields is 263 independent from the protocols the IOAM-Data-Fields are encapsulated 264 into. IOAM-Data-Fields can be encapsulated into several 265 encapsulating protocols. 267 Layering: If several encapsulation protocols (e.g., in case of 268 tunneling) are stacked on top of each other, IOAM-Data-Fields could 269 be present at multiple layers. The behavior follows the ships-in- 270 the-night model, i.e., IOAM-Data-Fields in one layer are independent 271 from IOAM-Data-Fields in another layer. Layering allows operators to 272 instrument the protocol layer they want to measure. The different 273 layers could, but do not have to, share the same IOAM encapsulation 274 mechanisms. 276 IOAM implementation: The definition of the IOAM-Data-Fields take the 277 specifics of devices with hardware data planes and software data 278 planes into account. 280 5. IOAM Data-Fields, Types, Nodes 282 This section details IOAM-related nomenclature and describes data 283 types such as IOAM-Data-Fields, IOAM-Types, IOAM-Namespaces as well 284 as the different types of IOAM nodes. 286 5.1. IOAM Data-Fields and Option-Types 288 An IOAM-Data-Field is a set of bits with a defined format and 289 meaning, which can be stored at a certain place in a packet for the 290 purpose of IOAM. 292 To accommodate the different uses of IOAM, IOAM-Data-Fields fall into 293 different categories. In IOAM, these categories are referred to as 294 IOAM-Option-Types. A common registry is maintained for IOAM-Option- 295 Types, see Section 8.1 for details. Corresponding to these IOAM- 296 Option-Types, different IOAM-Data-Fields are defined. 298 This document defines four IOAM-Option-Types: 300 o Pre-allocated Trace Option-Type 302 o Incremental Trace Option-Type 304 o Proof of Transit (POT) Option-Type 306 o Edge-to-Edge (E2E) Option-Type 308 Future IOAM-Option-Types can be allocated by IANA, as described in 309 Section 8.1. 311 5.2. IOAM-Domains and types of IOAM Nodes 313 Section 4 already mentioned that IOAM is expected to be deployed in a 314 limited domain [RFC8799]. One or more IOAM-Option-Types are added to 315 a packet upon entering an IOAM-Domain and are removed from the packet 316 when exiting the domain. Within the IOAM-Domain, the IOAM-Data- 317 Fields MAY be updated by network nodes that the packet traverses. An 318 IOAM-Domain consists of "IOAM encapsulating nodes", "IOAM 319 decapsulating nodes" and "IOAM transit nodes". The role of a node 320 (i.e., encapsulating, transit, decapsulating) is defined within an 321 IOAM-Namespace (see below). A node can have different roles in 322 different IOAM-Namespaces. 324 A device which adds at least one IOAM-Option-Type to the packet is 325 called an "IOAM encapsulating node", whereas a device which removes 326 an IOAM-Option-Type is referred to as an "IOAM decapsulating node". 327 Nodes within the domain which are aware of IOAM data and read and/or 328 write and/or process IOAM data are called "IOAM transit nodes". IOAM 329 nodes which add or remove the IOAM-Data-Fields can also update the 330 IOAM-Data-Fields at the same time. Or in other words, IOAM 331 encapsulating or decapsulating nodes can also serve as IOAM transit 332 nodes at the same time. Note that not every node in an IOAM-domain 333 needs to be an IOAM transit node. For example, a deployment might 334 require that packets traverse a set of firewalls which support IOAM. 335 In that case, only the set of firewall nodes would be IOAM transit 336 nodes rather than all nodes. 338 An "IOAM encapsulating node" incorporates one or more IOAM-Option- 339 Types (from the list of IOAM-Types, see Section 8.1) into packets 340 that IOAM is enabled for. If IOAM is enabled for a selected subset 341 of the traffic, the IOAM encapsulating node is responsible for 342 applying the IOAM functionality to the selected subset. 344 An "IOAM transit node" reads and/or writes and/or processes one or 345 more of the IOAM-Data-Fields. If both the Pre-allocated and the 346 Incremental Trace Option-Types are present in the packet, each IOAM 347 transit node based on configuration and available implementation of 348 IOAM might populate IOAM trace data in either Pre-allocated or 349 Incremental Trace Option-Type but not both. Note that not populating 350 any of the Trace Option-Types is also valid behavior for an IOAM 351 transit node. A transit node MUST ignore IOAM-Option-Types that it 352 does not understand. A transit node MUST NOT add new IOAM-Option- 353 Types to a packet, MUST NOT remove IOAM-Option-Types from a packet, 354 and MUST NOT change the IOAM-Data-Fields of an IOAM Edge-to-Edge 355 Option-Type. 357 An "IOAM decapsulating node" removes IOAM-Option-Type(s) from 358 packets. 360 The role of an IOAM-encapsulating, IOAM-transit or IOAM-decapsulating 361 node is always performed within a specific IOAM-Namespace. This 362 means that an IOAM node which is, e.g., an IOAM-decapsulating node 363 for IOAM-Namespace "A" but not for IOAM-Namespace "B" will only 364 remove the IOAM-Option-Types for IOAM-Namespace "A" from the packet. 365 Note that this applies even for IOAM-Option-Types that the node does 366 not understand, for example an IOAM-Option-Type other than the four 367 described above, that is added in a future revision. 369 IOAM-Namespaces allow for a namespace-specific definition and 370 interpretation of IOAM-Data-Fields. An interface-id could for 371 example point to a physical interface (e.g., to understand which 372 physical interface of an aggregated link is used when receiving or 373 transmitting a packet) whereas in another case it could refer to a 374 logical interface (e.g., in case of tunnels). Please refer to 375 Section 5.3 for details on IOAM-Namespaces. 377 5.3. IOAM-Namespaces 379 IOAM-Namespaces add further context to IOAM-Option-Types and 380 associated IOAM-Data-Fields. The IOAM-Option-Types and associated 381 IOAM-Data-Fields are interpreted as defined in this document, 382 regardless of the value of the IOAM-Namespace. However, IOAM- 383 Namespaces provide a way to group nodes to support different 384 deployment approaches of IOAM (see a few example use-cases below). 385 IOAM-Namespaces also help to resolve potential issues which can occur 386 due to IOAM-Data-Fields not being globally unique (e.g., IOAM node 387 identifiers do not have to be globally unique). IOAM-Data-Fields 388 significance is always within a particular IOAM-Namespace. Given 389 that IOAM-Data-Fields are always interpreted the context of a 390 specific namespace, the namespace-id field always needs to be carried 391 along with the IOAM data-fields themselves. 393 An IOAM-Namespace is identified by a 16-bit namespace identifier 394 (Namespace-ID). The IOAM-Namespace field is included in all the 395 IOAM-Option-Types defined in this document, and MUST be included in 396 all future IOAM-Option-Types. The Namespace-ID value is divided into 397 two sub-ranges: 399 o An operator-assigned range from 0x0001 to 0x7FFF 401 o An IANA-assigned range from 0x8000 to 0xFFFF 403 The IANA-assigned range is intended to allow future extensions to 404 have new and interoperable IOAM functionality, while the operator- 405 assigned range is intended to be domain-specific, and managed by the 406 network operator. The Namespace-ID value of 0x0000 is the "Default- 407 Namespace-ID". The Default-Namespace-ID indicates that no specific 408 namespace is associated with the IOAM data fields in the packet. The 409 Default-Namespace-ID MUST be supported by all nodes implementing 410 IOAM. A use-case for the Default-Namespace-ID are deployments which 411 do not leverage specific namespaces for some or all of their packets 412 that carry IOAM data fields. 414 Namespace identifiers allow devices which are IOAM capable to 415 determine: 417 o whether IOAM-Option-Type(s) need to be processed by a device: If 418 the Namespace-ID contained in a packet does not match any 419 Namespace-ID the node is configured to operate on, then the node 420 MUST NOT change the contents of the IOAM-Data-Fields. 422 o which IOAM-Option-Type needs to be processed/updated in case there 423 are multiple IOAM-Option-Types present in the packet. Multiple 424 IOAM-Option-Types can be present in a packet in case of 425 overlapping IOAM-Domains or in case of a layered IOAM deployment. 427 o whether IOAM-Option-Type(s) have to be removed from the packet, 428 e.g., at a domain edge or domain boundary. 430 IOAM-Namespaces support several different uses: 432 o IOAM-Namespaces can be used by an operator to distinguish 433 different IOAM-domains. Devices at edges of an IOAM-domain can 434 filter on Namespace-IDs to provide for proper IOAM-domain 435 isolation. 437 o IOAM-Namespaces provide additional context for IOAM-Data-Fields 438 and thus can be used to ensure that IOAM-Data-Fields are unique 439 and are interpreted properly by management stations or network 440 controllers. The node identifier field (node_id, see below) does 441 not need to be unique in a deployment. This could be the case if 442 an operator wishes to use different node identifiers for different 443 IOAM layers, even within the same device or node identifiers might 444 not be unique for other organizational reasons, such as after a 445 merger of two formerly separated organizations. The Namespace-ID 446 can be used as a context identifier, such that the combination of 447 node_id and Namespace-ID will always be unique. 449 o Similarly, IOAM-Namespaces can be used to define how certain IOAM- 450 Data-Fields are interpreted: IOAM offers three different timestamp 451 format options. The Namespace-ID can be used to determine the 452 timestamp format. IOAM-Data-Fields (e.g., buffer occupancy) which 453 do not have a unit associated are to be interpreted within the 454 context of a IOAM-Namespace. 456 o IOAM-Namespaces can be used to identify different sets of devices 457 (e.g., different types of devices) in a deployment: If an operator 458 desires to insert different IOAM-Data-Fields based on the device, 459 the devices could be grouped into multiple IOAM-Namespaces. This 460 could be due to the fact that the IOAM feature set differs between 461 different sets of devices, or it could be for reasons of optimized 462 space usage in the packet header. It could also stem from 463 hardware or operational limitations on the size of the trace data 464 that can be added and processed, preventing collection of a full 465 trace for a flow. 467 o By assigning different IOAM Namespace-IDs to different sets of 468 nodes or network partitions and using a separate instance of an 469 IOAM-Option-Type for each Namespace-ID, a full trace for a flow 470 could be collected and constructed via partial traces from each 471 IOAM-Option-Type in each of the packets in the flow. Example: An 472 operator could choose to group the devices of a domain into two 473 IOAM-Namespaces, in a way that each IOAM-Namespace is represented 474 by one of two IOAM-Option-Types in the packet. Each node would 475 record data only for the IOAM-Namespace that it belongs to, 476 ignoring the other IOAM-Option-Type with a IOAM-Namespace to which 477 it doesn't belong. To retrieve a full view of the deployment, the 478 captured IOAM-Data-Fields of the two IOAM-Namespaces need to be 479 correlated. 481 5.4. IOAM Trace Option-Types 483 In a typical deployment, all nodes in an IOAM-Domain would 484 participate in IOAM and thus be IOAM transit nodes, IOAM 485 encapsulating or IOAM decapsulating nodes. If not all nodes within a 486 domain support IOAM functionality as defined in this document, IOAM 487 tracing information (i.e., node data, see below) can only be 488 collected on those nodes which support IOAM functionality as defined 489 in this document. Nodes which do not support IOAM functionality as 490 defined in this document will forward the packet without any changes 491 to the IOAM-Data-Fields. The maximum number of hops and the minimum 492 path MTU of the IOAM-domain is assumed to be known. An overflow 493 indicator (O-bit) is defined as one of the ways to deal with 494 situations where the PMTU was underestimated, i.e., where the number 495 of hops which are IOAM capable exceeds the available space in the 496 packet. 498 To optimize hardware and software implementations, IOAM tracing is 499 defined as two separate options. A deployment can choose to 500 configure and support one or both of the following options. 502 Pre-allocated Trace-Option: This trace option is defined as a 503 container of node data fields (see below) with pre-allocated space 504 for each node to populate its information. This option is useful 505 for implementations where it is efficient to allocate the space 506 once and index into the array to populate the data during transit 507 (e.g., software forwarders often fall into this class). The IOAM 508 encapsulating node allocates space for Pre-allocated Trace Option- 509 Type in the packet and sets corresponding fields in this IOAM- 510 Option-Type. The IOAM encapsulating node allocates an array which 511 is used to store operational data retrieved from every node while 512 the packet traverses the domain. IOAM transit nodes update the 513 content of the array, and possibly update the checksums of outer 514 headers. A pointer which is part of the IOAM trace data, points 515 to the next empty slot in the array. An IOAM transit node that 516 updates the content of the pre-allocated option also updates the 517 value of the pointer, which specifies where the next IOAM transit 518 node fills in its data. The "node data list" array (see below) in 519 the packet is populated iteratively as the packet traverses the 520 network, starting with the last entry of the array, i.e., "node 521 data list [n]" is the first entry to be populated, "node data list 522 [n-1]" is the second one, etc. 524 Incremental Trace-Option: This trace option is defined as a 525 container of node data fields where each node allocates and pushes 526 its node data immediately following the option header. This type 527 of trace recording is useful for some of the hardware 528 implementations as it eliminates the need for the transit network 529 elements to read the full array in the option and allows for 530 arbitrarily long packets as the MTU allows. The IOAM 531 encapsulating node allocates space for the Incremental Trace 532 Option-Type. Based on operational state and configuration, the 533 IOAM encapsulating node sets the fields in the Option-Type that 534 control what IOAM-Data-Fields have to be collected and how large 535 the node data list can grow. IOAM transit nodes push their node 536 data to the node data list subject to any protocol constraints of 537 the encapsulating layer. They then decrease the remaining length 538 available to subsequent nodes and adjust the lengths and possibly 539 checksums in outer headers. 541 IOAM encapsulating nodes and IOAM decapsulating nodes which support 542 tracing MUST support both Trace-Option-Types. For IOAM transit nodes 543 it is sufficient to support one of the Trace-Option-Types. In the 544 event that both options are utilized in a deployment at the same 545 time, the Incremental Trace-Option MUST be placed before the Pre- 546 allocated Trace-Option. Deployments which mix devices with either 547 the Incremental Trace-Option or the Pre-allocated Trace-Option could 548 result in both Option-Types being present in a packet. Given that 549 the operator knows which equipment is deployed in a particular IOAM- 550 domain, the operator will decide by means of configuration which 551 type(s) of trace options will be used for a particular domain. 553 Every node data entry holds information for a particular IOAM transit 554 node that is traversed by a packet. The IOAM decapsulating node 555 removes the IOAM-Option-Type(s) and processes and/or exports the 556 associated data. Like all IOAM-Data-Fields, the IOAM-Data-Fields of 557 the IOAM-Trace-Option-Types are defined in the context of an IOAM- 558 Namespace. 560 IOAM tracing can collect the following types of information: 562 o Identification of the IOAM node. An IOAM node identifier can 563 match to a device identifier or a particular control point or 564 subsystem within a device. 566 o Identification of the interface that a packet was received on, 567 i.e., ingress interface. 569 o Identification of the interface that a packet was sent out on, 570 i.e., egress interface. 572 o Time of day when the packet was processed by the node as well as 573 the transit delay. Different definitions of processing time are 574 feasible and expected, though it is important that all devices of 575 an IOAM-domain follow the same definition. 577 o Generic data: Format-free information where syntax and semantic of 578 the information is defined by the operator in a specific 579 deployment. For a specific IOAM-Namespace, all IOAM nodes have to 580 interpret the generic data the same way. Examples for generic 581 IOAM data include geo-location information (location of the node 582 at the time the packet was processed), buffer queue fill level or 583 cache fill level at the time the packet was processed, or even a 584 battery charge level. 586 o Information to detect whether IOAM trace data was added at every 587 hop or whether certain hops in the domain weren't IOAM transit 588 nodes. 590 It should be noted that the semantics of some of the node data fields 591 that are defined below, such as the queue depth and buffer occupancy, 592 are implementation specific. This approach is intended to allow IOAM 593 nodes with various different architectures. 595 5.4.1. Pre-allocated and Incremental Trace Option-Types 597 The IOAM Pre-allocated Trace-Option and the IOAM Incremental Trace- 598 Option have similar formats. Except where noted below, the internal 599 formats and fields of the two trace options are identical. Both 600 Trace-Options consist of a fixed size "trace option header" and a 601 variable data space to store gathered data, the "node data list". An 602 IOAM transit node (that is not an IOAM encapsulating node or IOAM 603 decapsulating node) MUST NOT modify any of the fields in the fixed 604 size "trace option header", other than "flags" and "RemainingLen", 605 i.e., an IOAM transit node MUST NOT modify the Namespace-ID, NodeLen, 606 IOAM-Trace-Type, or Reserved fields. 608 Pre-allocated and incremental trace option headers: 610 0 1 2 3 611 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 612 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 613 | Namespace-ID |NodeLen | Flags | RemainingLen| 614 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 615 | IOAM-Trace-Type | Reserved | 616 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 618 The trace option data MUST be 4-octet aligned: 620 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+<-+ 621 | | | 622 | node data list [0] | | 623 | | | 624 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ D 625 | | a 626 | node data list [1] | t 627 | | a 628 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 629 ~ ... ~ S 630 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ p 631 | | a 632 | node data list [n-1] | c 633 | | e 634 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 635 | | | 636 | node data list [n] | | 637 | | | 638 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+<-+ 640 Namespace-ID: 16-bit identifier of an IOAM-Namespace. The 641 Namespace-ID value of 0x0000 is defined as the "Default-Namespace- 642 ID" (see Section 5.3) and MUST be known to all the nodes 643 implementing IOAM. For any other Namespace-ID value that does not 644 match any Namespace-ID the node is configured to operate on, the 645 node MUST NOT change the contents of the IOAM-Data-Fields. 647 NodeLen: 5-bit unsigned integer. This field specifies the length of 648 data added by each node in multiples of 4-octets, excluding the 649 length of the "Opaque State Snapshot" field. 651 If IOAM-Trace-Type bit 22 is not set, then NodeLen specifies the 652 actual length added by each node. If IOAM-Trace-Type bit 22 is 653 set, then the actual length added by a node would be (NodeLen + 654 length of the "Opaque State Snapshot" field) in 4 octet units. 656 For example, if 3 IOAM-Trace-Type bits are set and none of them 657 are in wide format, then NodeLen would be 3. If 3 IOAM-Trace-Type 658 bits are set and 2 of them are wide, then NodeLen would be 5. 660 An IOAM encapsulating node MUST set NodeLen. 662 A node receiving an IOAM Pre-allocated or Incremental Trace-Option 663 relies on the NodeLen value. 665 Flags 4-bit field. Flags are allocated by IANA, as specified in 666 Section 8.3. This document allocates a single flag as follows: 668 Bit 0 "Overflow" (O-bit) (most significant bit). In case a 669 network element is supposed to add node data to a packet, but 670 detects that there are not enough octets left to record the 671 node data, the network element MUST NOT add any fields and MUST 672 set the overflow "O-bit" to "1" in the IOAM-Trace-Option 673 header. This is useful for transit nodes to ignore further 674 processing of the option. 676 RemainingLen: 7-bit unsigned integer. This field specifies the data 677 space in multiples of 4-octets remaining for recording the node 678 data, before the node data list is considered to have overflowed. 679 The sender MUST assign the initial value of the RemainingLen 680 field. The sender MAY calculate the value of the RemainingLen 681 field by computing the number of node data bytes allowed before 682 exceeding the path MTU (PMTU), given that the PMTU is known to the 683 sender. Subsequent nodes can carry out a simple comparison 684 between RemainingLen and NodeLen, along with the length of the 685 "Opaque State Snapshot" if applicable, to determine whether or not 686 data can be added by this node. When node data is added, the node 687 MUST decrease RemainingLen by the amount of data added. In the 688 pre-allocated trace option, RemainingLen is used to derive the 689 offset in data space to record the node data element. 690 Specifically, the recording of the node data element would start 691 from RemainingLen - NodeLen - sizeof(opaque snapshot) in 4 octet 692 units. If RemainingLen in a pre-allocated trace option exceeds 693 the length of the option, as specified in the lower layer header 694 (which is not within the scope of this document), then the node 695 MUST NOT add any fields. 697 IOAM-Trace-Type: A 24-bit identifier which specifies which data 698 types are used in this node data list. 700 The IOAM-Trace-Type value is a bit field. The following bits are 701 defined in this document, with details on each bit described in 702 the Section 5.4.2. The order of packing the data fields in each 703 node data element follows the bit order of the IOAM-Trace-Type 704 field, as follows: 706 Bit 0 (Most significant bit) When set, indicates presence of 707 Hop_Lim and node_id (short format) in the node data. 709 Bit 1 When set, indicates presence of ingress_if_id and 710 egress_if_id (short format) in the node data. 712 Bit 2 When set, indicates presence of timestamp seconds in the 713 node data. 715 Bit 3 When set, indicates presence of timestamp fraction in the 716 node data. 718 Bit 4 When set, indicates presence of transit delay in the node 719 data. 721 Bit 5 When set, indicates presence of IOAM-Namespace specific 722 data (short format) in the node data. 724 Bit 6 When set, indicates presence of queue depth in the node 725 data. 727 Bit 7 When set, indicates presence of the Checksum Complement 728 node data. 730 Bit 8 When set, indicates presence of Hop_Lim and node_id in 731 wide format in the node data. 733 Bit 9 When set, indicates presence of ingress_if_id and 734 egress_if_id in wide format in the node data. 736 Bit 10 When set, indicates presence of IOAM-Namespace specific 737 data in wide format in the node data. 739 Bit 11 When set, indicates presence of buffer occupancy in the 740 node data. 742 Bit 12-21 Undefined. These values are available for future 743 assignment in the IOAM Trace-Type Registry (Section 8.2). 744 Every future node data field corresponding to one of 745 these bits MUST be 4-octets long. An IOAM encapsulating 746 node MUST set the value of each undefined bit to 0. If 747 an IOAM transit node receives a packet with one or more 748 of these bits set to 1, it MUST either: 750 1. Add corresponding node data filled with the reserved 751 value 0xFFFFFFFF, after the node data fields for the 752 IOAM-Trace-Type bits defined above, such that the 753 total node data added by this node in units of 754 4-octets is equal to NodeLen, or 756 2. Not add any node data fields to the packet, even for 757 the IOAM-Trace-Type bits defined above. 759 Bit 22 When set, indicates presence of variable length Opaque 760 State Snapshot field. 762 Bit 23 Reserved: MUST be set to zero upon transmission and 763 ignored upon receipt. This bit is reserved to allow for 764 future extensions of the IOAM-Trace-Type bit field. 766 Section 5.4.2 describes the IOAM-Data-Types and their formats. 767 Within an IOAM-Domain possible combinations of these bits making 768 the IOAM-Trace-Type can be restricted by configuration knobs. 770 Reserved: 8-bits. An IOAM encapsulating node MUST set the value to 771 zero upon transmission. IOAM transit nodes MUST ignore the 772 received value. 774 Node data List [n]: Variable-length field. This is a list of node 775 data elements where the content of each node data element is 776 determined by the IOAM-Trace-Type. The order of packing the data 777 fields in each node data element follows the bit order of the 778 IOAM-Trace-Type field. Each node MUST prepend its node data 779 element in front of the node data elements that it received, such 780 that the transmitted node data list begins with this node's data 781 element as the first populated element in the list. The last node 782 data element in this list is the node data of the first IOAM 783 capable node in the path. Populating the node data list in this 784 way ensures that the order of node data list is the same for 785 incremental and pre-allocated trace options. In the pre-allocated 786 trace option, the index contained in RemainingLen identifies the 787 offset for current active node data to be populated. 789 5.4.2. IOAM node data fields and associated formats 791 All the IOAM-Data-Fields MUST be 4-octet aligned. If a node which is 792 supposed to update an IOAM-Data-Field is not capable of populating 793 the value of a field set in the IOAM-Trace-Type, the field value MUST 794 be set to 0xFFFFFFFF for 4-octet fields or 0xFFFFFFFFFFFFFFFF for 795 8-octet fields, indicating that the value is not populated, except 796 when explicitly specified in the field description below. 798 Some IOAM-Data-Fields defined below, such as interface identifiers or 799 IOAM-Namespace specific data, are defined in both "short format" as 800 well as "wide format". The use of "short format" or "wide format" is 801 not mutually exclusive. A deployment could choose to leverage both. 802 For example, ingress_if_id_(short format) could be an identifier for 803 the physical interface, whereas ingress_if_id_(wide format) could be 804 an identifier for a logical sub-interface of that physical interface. 806 Data fields and associated data types for each of the IOAM-Data- 807 Fields are specified in the following sections. The definition of 808 IOAM-Data-Fields focuses on the syntax of the data-fields and avoids 809 specifying the semantics where feasible. This is why no units are 810 defined for data-fields like e.g., "buffer occupancy" or "queue 811 depth". With this approach, nodes can supply the information in 812 their native format and are not required to perform unit or format 813 conversions. Systems that further process IOAM information, like 814 e.g., a network management system are assumed to also handle unit 815 conversions as part of their IOAM data-fields processing. The 816 combination of a particular data-field and the namespace-id provides 817 for the context to interpret the provided data appropriately. 819 5.4.2.1. Hop_Lim and node_id short format 821 The "Hop_Lim and node_id short format" field is a 4-octet field that 822 is defined as follows: 824 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 825 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 826 | Hop_Lim | node_id | 827 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 829 Hop_Lim: 1-octet unsigned integer. It is set to the Hop Limit value 830 in the packet at egress from the node that records this data. Hop 831 Limit information is used to identify the location of the node in 832 the communication path. This is copied from the lower layer, 833 e.g., TTL value in IPv4 header or hop limit field from IPv6 header 834 of the packet when the packet is ready for transmission. The 835 semantics of the Hop_Lim field depend on the lower layer protocol 836 that IOAM is encapsulated into, and therefore its specific 837 semantics are outside the scope of this memo. The value of this 838 field MUST be set to 0xff when the lower level does not have a 839 TTL/Hop limit equivalent field. 841 node_id: 3-octet unsigned integer. Node identifier field to 842 uniquely identify a node within the IOAM-Namespace and associated 843 IOAM-Domain. The procedure to allocate, manage and map the 844 node_ids is beyond the scope of this document. See 845 [I-D.ietf-ippm-ioam-deployment] for a discussion of deployment 846 related aspects of the node_id. 848 5.4.2.2. ingress_if_id and egress_if_id 850 The "ingress_if_id and egress_if_id" field is a 4-octet field that is 851 defined as follows: 853 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 854 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 855 | ingress_if_id | egress_if_id | 856 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 858 ingress_if_id: 2-octet unsigned integer. Interface identifier to 859 record the ingress interface the packet was received on. 861 egress_if_id: 2-octet unsigned integer. Interface identifier to 862 record the egress interface the packet is forwarded out of. 864 Note that due to the fact that IOAM uses its own IOAM-Namespaces for 865 IOAM-Data-Fields, data fields like interface identifiers can be used 866 in a flexible way to represent system resources that are associated 867 with ingressing or egressing packets, i.e., ingress_if_id could 868 represent a physical interface, a virtual or logical interface, or 869 even a queue. 871 5.4.2.3. timestamp seconds 873 The "timestamp seconds" field is a 4-octet unsigned integer field. 874 It contains the absolute timestamp in seconds that specifies the time 875 at which the packet was received by the node. This field has three 876 possible formats; based on either PTP (see e.g., [RFC8877]), NTP 877 [RFC5905], or POSIX [POSIX]. The three timestamp formats are 878 specified in Section 6. In all three cases, the Timestamp Seconds 879 field contains the 32 most significant bits of the timestamp format 880 that is specified in Section 6. If a node is not capable of 881 populating this field, it assigns the value 0xFFFFFFFF. Note that 882 this is a legitimate value that is valid for 1 second in 883 approximately 136 years; the analyzer has to correlate several 884 packets or compare the timestamp value to its own time-of-day in 885 order to detect the error indication. 887 5.4.2.4. timestamp fraction 889 The "timestamp fraction" field is a 4-octet unsigned integer field. 890 Fraction specifies the fractional portion of the number of seconds 891 since the NTP epoch [RFC8877]. The field specifies the time at which 892 the packet was received by the node. This field has three possible 893 formats; based on either PTP (see e.g., [RFC8877]), NTP [RFC5905], or 894 POSIX [POSIX]. The three timestamp formats are specified in 895 Section 6. In all three cases, the Timestamp fraction field contains 896 the 32 least significant bits of the timestamp format that is 897 specified in Section 6. If a node is not capable of populating this 898 field, it assigns the value 0xFFFFFFFF. Note that this is a 899 legitimate value in the NTP format, valid for approximately 233 900 picoseconds in every second. If the NTP format is used the analyzer 901 has to correlate several packets in order to detect the error 902 indication. 904 5.4.2.5. transit delay 906 The "transit delay" field is a 4-octet unsigned integer in the range 907 0 to 2^31-1. It is the time in nanoseconds the packet spent in the 908 transit node. This can serve as an indication of the queuing delay 909 at the node. If the transit delay exceeds 2^31-1 nanoseconds then 910 the top bit 'O' is set to indicate overflow and value set to 911 0x80000000. When this field is part of the data field but a node 912 populating the field is not able to fill it, the field position in 913 the field MUST be filled with value 0xFFFFFFFF to mean not populated. 915 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 916 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 917 |O| transit delay | 918 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 920 5.4.2.6. namespace specific data 922 The "namespace specific data" field is a 4-octet field which can be 923 used by the node to add IOAM-Namespace specific data. This 924 represents a "free-format" 4-octet bit field with its semantics 925 defined in the context of a specific IOAM-Namespace. 927 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 928 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 929 | namespace specific data | 930 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 932 5.4.2.7. queue depth 934 The "queue depth" field is a 4-octet unsigned integer field. This 935 field indicates the current length of the egress interface queue of 936 the interface from where the packet is forwarded out. The queue 937 depth is expressed as the current amount of memory buffers used by 938 the queue (a packet could consume one or more memory buffers, 939 depending on its size). 941 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 942 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 943 | queue depth | 944 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 946 5.4.2.8. Checksum Complement 948 The "Checksum Complement" field is a 4-octet node data which contains 949 a 4-octet Checksum Complement field. The Checksum Complement is 950 useful when IOAM is transported over encapsulations that make use of 951 a UDP transport, such as VXLAN-GPE or Geneve. Without the Checksum 952 Complement, nodes adding IOAM node data update the UDP Checksum field 953 following the recommendation of the encapsulation protocols. When 954 the Checksum Complement is present, an IOAM encapsulating node or 955 IOAM transit node adding node data MUST carry out one of the 956 following two alternatives in order to maintain the correctness of 957 the UDP Checksum value: 959 1. Recompute the UDP Checksum field. 961 2. Use the Checksum Complement to make a checksum-neutral update in 962 the UDP payload; the Checksum Complement is assigned a value that 963 complements the rest of the node data fields that were added by 964 the current node, causing the existing UDP Checksum field to 965 remain correct. 967 IOAM decapsulating nodes MUST recompute the UDP Checksum field, since 968 they do not know whether previous hops modified the UDP Checksum 969 field or the Checksum Complement field. 971 Checksum Complement fields are used in a similar manner in [RFC7820] 972 and [RFC7821]. 974 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 975 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 976 | Checksum Complement | 977 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 979 5.4.2.9. Hop_Lim and node_id wide 981 The "Hop_Lim and node_id wide" field is an 8-octet field defined as 982 follows: 984 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 985 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 986 | Hop_Lim | node_id ~ 987 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 988 ~ node_id (contd) | 989 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 991 Hop_Lim: 1-octet unsigned integer. See Section 5.4.2.1 for the 992 definition of the field. 994 node_id: 7-octet unsigned integer. Node identifier field to 995 uniquely identify a node within the IOAM-Namespace and associated 996 IOAM-Domain. The procedure to allocate, manage and map the 997 node_ids is beyond the scope of this document. 999 5.4.2.10. ingress_if_id and egress_if_id wide 1001 The "ingress_if_id and egress_if_id wide" field is an 8-octet field 1002 which is defined as follows: 1004 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1005 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1006 | ingress_if_id | 1007 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1008 | egress_if_id | 1009 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1011 ingress_if_id: 4-octet unsigned integer. Interface identifier to 1012 record the ingress interface the packet was received on. 1014 egress_if_id: 4-octet unsigned integer. Interface identifier to 1015 record the egress interface the packet is forwarded out of. 1017 5.4.2.11. namespace specific data wide 1019 The "namespace specific data wide" field is an 8-octet field which 1020 can be used by the node to add IOAM-Namespace specific data. This 1021 represents a "free-format" 8-octet bit field with its semantics 1022 defined in the context of a specific IOAM-Namespace. 1024 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1025 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1026 | namespace specific data ~ 1027 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1028 ~ namespace specific data (contd) | 1029 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1031 5.4.2.12. buffer occupancy 1033 The "buffer occupancy" field is a 4-octet unsigned integer field. 1034 This field indicates the current status of the occupancy of the 1035 common buffer pool used by a set of queues. The units of this field 1036 are implementation specific. Hence, the units are interpreted within 1037 the context of an IOAM-Namespace and/or node-id if used. The authors 1038 acknowledge that in some operational cases there is a need for the 1039 units to be consistent across a packet path through the network, 1040 hence it is recommended for implementations to use standard units 1041 such as Bytes. 1043 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1044 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1045 | buffer occupancy | 1046 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1048 5.4.2.13. Opaque State Snapshot 1050 The "Opaque State Snapshot" is a variable length field and follows 1051 the fixed length IOAM-Data-Fields defined above. It allows the 1052 network element to store an arbitrary state in the node data field, 1053 without a pre-defined schema. The schema is to be defined within the 1054 context of an IOAM-Namespace. The schema needs to be made known to 1055 the analyzer by some out-of-band mechanism. The specification of 1056 this mechanism is beyond the scope of this document. A 24-bit 1057 "Schema Id" field, interpreted within the context of an IOAM- 1058 Namespace, indicates which particular schema is used, and has to be 1059 configured on the network element by the operator. 1061 0 1 2 3 1062 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1063 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1064 | Length | Schema ID | 1065 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1066 | | 1067 | | 1068 | Opaque data | 1069 ~ ~ 1070 . . 1071 . . 1072 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1074 Length: 1-octet unsigned integer. It is the length in multiples of 1075 4-octets of the Opaque data field that follows Schema Id. 1077 Schema ID: 3-octet unsigned integer identifying the schema of Opaque 1078 data. 1080 Opaque data: Variable length field. This field is interpreted as 1081 specified by the schema identified by the Schema ID. 1083 When this field is part of the data field but a node populating the 1084 field has no opaque state data to report, the Length MUST be set to 0 1085 and the Schema ID MUST be set to 0xFFFFFF to mean no schema. 1087 5.4.3. Examples of IOAM node data 1089 The format used for the entries in a packet's "node data list" array 1090 can vary from packet to packet and deployment to deployment". Some 1091 deployments might only be interested in recording the node 1092 identifiers, whereas others might be interested in recording node 1093 identifiers and timestamps. This section provides example entries of 1094 the "node data list". 1096 0xD40000: IOAM-Trace-Type is 0xD40000 (0b110101000000000000000000) 1097 then the format of node data is: 1099 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1100 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1101 | Hop_Lim | node_id | 1102 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1103 | ingress_if_id | egress_if_id | 1104 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1105 | timestamp fraction | 1106 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1107 | namespace specific data | 1108 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1110 0xC00000: IOAM-Trace-Type is 0xC00000 (0b110000000000000000000000) 1111 then the format is: 1113 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1114 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1115 | Hop_Lim | node_id | 1116 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1117 | ingress_if_id | egress_if_id | 1118 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1120 0x900000: IOAM-Trace-Type is 0x900000 (0b100100000000000000000000) 1121 then the format is: 1123 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1124 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1125 | Hop_Lim | node_id | 1126 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1127 | timestamp fraction | 1128 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1130 0x840000: IOAM-Trace-Type is 0x840000 (0b100001000000000000000000) 1131 then the format is: 1133 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1134 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1135 | Hop_Lim | node_id | 1136 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1137 | namespace specific data | 1138 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1140 0x940000: IOAM-Trace-Type is 0x940000 (0b100101000000000000000000) 1141 then the format is: 1143 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1144 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1145 | Hop_Lim | node_id | 1146 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1147 | timestamp fraction | 1148 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1149 | namespace specific data | 1150 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1152 0x308002: IOAM-Trace-Type is 0x308002 (0b001100001000000000000010) 1153 then the format is: 1155 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1156 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1157 | timestamp seconds | 1158 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1159 | timestamp fraction | 1160 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1161 | Hop_Lim | node_id | 1162 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1163 | node_id(contd) | 1164 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1165 | Length | Schema Id | 1166 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1167 | | 1168 | | 1169 | Opaque data | 1170 ~ ~ 1171 . . 1172 . . 1173 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1175 5.5. IOAM Proof of Transit Option-Type 1177 IOAM Proof of Transit Option-Type is used to support path or service 1178 function chain [RFC7665] verification use cases, i.e., prove that 1179 traffic transited a defined path. While details on how the IOAM data 1180 for the Proof-of-transit option is processed at IOAM encapsulating, 1181 decapsulating and transit nodes are outside the scope of the 1182 document, proof of transit approaches share the need to uniquely 1183 identify a packet as well as iteratively operate on a set of 1184 information that is handed from node to node. Correspondingly, two 1185 pieces of information are added as IOAM-Data-Fields to the packet: 1187 o PktID: Unique identifier for the packet. 1189 o Cumulative: Information which is handed from node to node and 1190 updated by every node according to a verification algorithm. 1192 The IOAM Proof-of-Transit Option-Type consist of a fixed size "IOAM 1193 proof of transit option header" and "IOAM proof of transit option 1194 data fields": 1196 IOAM proof of transit option header: 1198 0 1 2 3 1199 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1200 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1201 | Namespace-ID |IOAM POT Type | IOAM POT flags| 1202 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1204 IOAM proof of transit Option-Type IOAM-Data-Fields MUST be 1205 4-octet aligned: 1207 0 1 2 3 1208 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1209 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1210 | POT Option data field determined by IOAM-POT-Type | 1211 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1213 Namespace-ID: 16-bit identifier of an IOAM-Namespace. The 1214 Namespace-ID value of 0x0000 is defined as the "Default-Namespace- 1215 ID" (see Section 5.3) and MUST be known to all the nodes 1216 implementing IOAM. For any other Namespace-ID value that does not 1217 match any Namespace-ID the node is configured to operate on, the 1218 node MUST NOT change the contents of the IOAM-Data-Fields. 1220 IOAM POT Type: 8-bit identifier of a particular POT variant that 1221 specifies the POT data that is included. This document defines 1222 POT Type 0: 1224 0: POT data is a 16 Octet field to carry data associated to POT 1225 procedures. 1227 If a node receives an IOAM POT Type value that it does not 1228 understand, the node MUST NOT change, add to, or remove the 1229 contents of the OAM-Data-Fields. 1231 IOAM POT flags: 8-bit. This document does not define any flags. 1232 Bits 0-7 These bits are available for assignment, see Section 8.5. 1233 Bits which have not been assigned MUST be set to zero upon 1234 transmission and ignored upon receipt. 1236 POT Option data: Variable-length field. The type of which is 1237 determined by the IOAM-POT-Type. 1239 5.5.1. IOAM Proof of Transit Type 0 1241 IOAM proof of transit option of IOAM POT Type 0: 1243 0 1 2 3 1244 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1245 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1246 | Namespace-ID |IOAM POT Type=0|R R R R R R R R| 1247 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+<-+ 1248 | PktID | | 1249 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ P 1250 | PktID (contd) | O 1251 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ T 1252 | Cumulative | | 1253 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 1254 | Cumulative (contd) | | 1255 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+<-+ 1257 Namespace-ID: 16-bit identifier of an IOAM-Namespace (see 1258 Section 5.5 above). 1260 IOAM POT Type: 8-bit identifier of a particular POT variant that 1261 specifies the POT data that is included (see Section 5.5 above). 1262 For this case here, IOAM POT Type is set to the value 0. 1264 Bit 0-7: Undefined (see Section 5.5 above). 1266 PktID: 64-bit packet identifier. 1268 Cumulative: 64-bit Cumulative that is updated at specific nodes by 1269 processing per packet PktID field and configured parameters. 1271 Note: Larger or smaller sizes of "PktID" and "Cumulative" data are 1272 feasible and could be required for certain deployments, e.g., in case 1273 of space constraints in the encapsulation protocols used. Future 1274 documents could introduce different sizes of data for "proof of 1275 transit". 1277 5.6. IOAM Edge-to-Edge Option-Type 1279 The IOAM Edge-to-Edge Option-Type is to carry data that is added by 1280 the IOAM encapsulating node and interpreted by IOAM decapsulating 1281 node. The IOAM transit nodes MAY process the data but MUST NOT 1282 modify it. 1284 The IOAM Edge-to-Edge Option-Type consist of a fixed size "IOAM Edge- 1285 to-Edge Option-Type header" and "IOAM Edge-to-Edge Option-Type data 1286 fields": 1288 IOAM Edge-to-Edge Option-Type header: 1290 0 1 2 3 1291 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1292 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1293 | Namespace-ID | IOAM-E2E-Type | 1294 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1296 IOAM Edge-to-Edge Option-Type IOAM-Data-Fields MUST 1297 be 4-octet aligned: 1299 0 1 2 3 1300 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1301 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1302 | E2E Option data field determined by IOAM-E2E-Type | 1303 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1305 Namespace-ID: 16-bit identifier of an IOAM-Namespace. The 1306 Namespace-ID value of 0x0000 is defined as the "Default-Namespace- 1307 ID" (see Section 5.3) and MUST be known to all the nodes 1308 implementing IOAM. For any other Namespace-ID value that does not 1309 match any Namespace-ID the node is configured to operate on, then 1310 the node MUST NOT change the contents of the IOAM-Data-Fields. 1312 IOAM-E2E-Type: A 16-bit identifier which specifies which data types 1313 are used in the E2E option data. The IOAM-E2E-Type value is a bit 1314 field. The order of packing the E2E option data field elements 1315 follows the bit order of the IOAM-E2E-Type field, as follows: 1317 Bit 0 (Most significant bit) When set indicates presence of a 1318 64-bit sequence number added to a specific "packet group" 1319 which is used to detect packet loss, packet reordering, 1320 or packet duplication within the group. The "packet 1321 group" is deployment dependent and defined at the IOAM 1322 encapsulating node, e.g., by n-tuple based classification 1323 of packets. When this bit is set, "Bit 1" (for 32-bit 1324 sequence number, see below) MUST be zero. 1326 Bit 1 When set indicates presence of a 32-bit sequence number 1327 added to a specific "packet group" which is used to 1328 detect packet loss, packet reordering, or packet 1329 duplication within that group. The "packet group" is 1330 deployment dependent and defined at the IOAM 1331 encapsulating node, e.g., by n-tuple based classification 1332 of packets. When this bit is set, "Bit 0" (for 64-bit 1333 sequence number, see above) MUST be zero. 1335 Bit 2 When set indicates presence of timestamp seconds, 1336 representing the time at which the packet entered the 1337 IOAM-domain. Within the IOAM encapsulating node, the 1338 time that the timestamp is retrieved can depend on the 1339 implementation. Some possibilities are: 1) the time at 1340 which the packet was received by the node, 2) the time at 1341 which the packet was transmitted by the node, 3) when a 1342 tunnel encapsulation is used, the point at which the 1343 packet is encapsulated into the tunnel. Each 1344 implementation has to document when the E2E timestamp 1345 that is going to be put in the packet is retrieved. This 1346 4-octet field has three possible formats; based on either 1347 PTP (see e.g., [RFC8877]), NTP [RFC5905], or POSIX 1348 [POSIX]. The three timestamp formats are specified in 1349 Section 6. In all three cases, the Timestamp Seconds 1350 field contains the 32 most significant bits of the 1351 timestamp format that is specified in Section 6. If a 1352 node is not capable of populating this field, it assigns 1353 the value 0xFFFFFFFF. Note that this is a legitimate 1354 value that is valid for 1 second in approximately 136 1355 years; the analyzer has to correlate several packets or 1356 compare the timestamp value to its own time-of-day in 1357 order to detect the error indication. 1359 Bit 3 When set indicates presence of timestamp fraction, 1360 representing the time at which the packet entered the 1361 IOAM-domain. This 4-octet field has three possible 1362 formats; based on either PTP (see e.g., [RFC8877]), NTP 1363 [RFC5905], or POSIX [POSIX]. The three timestamp formats 1364 are specified in Section 6. In all three cases, the 1365 Timestamp fraction field contains the 32 least 1366 significant bits of the timestamp format that is 1367 specified in Section 6. If a node is not capable of 1368 populating this field, it assigns the value 0xFFFFFFFF. 1369 Note that this is a legitimate value in the NTP format, 1370 valid for approximately 233 picoseconds in every second. 1371 If the NTP format is used the analyzer has to correlate 1372 several packets in order to detect the error indication. 1374 Bit 4-15 Undefined. An IOAM encapsulating node MUST set the value 1375 of these bits to zero upon transmission and ignore upon 1376 receipt. 1378 E2E Option data: Variable-length field. The type of which is 1379 determined by the IOAM-E2E-Type. 1381 6. Timestamp Formats 1383 The IOAM-Data-Fields include a timestamp field which is represented 1384 in one of three possible timestamp formats. It is assumed that the 1385 management plane is responsible for determining which timestamp 1386 format is used. 1388 6.1. PTP Truncated Timestamp Format 1390 The Precision Time Protocol (PTP) uses an 80-bit timestamp format. 1391 The truncated timestamp format is a 64-bit field, which is the 64 1392 least significant bits of the 80-bit PTP timestamp. The PTP 1393 truncated format is specified in Section 4.3 of [RFC8877], and the 1394 details are presented below for the sake of completeness. 1396 0 1 2 3 1397 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1398 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1399 | Seconds | 1400 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1401 | Nanoseconds | 1402 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1404 Figure 1: PTP Truncated Timestamp Format 1406 Timestamp field format: 1408 Seconds: specifies the integer portion of the number of seconds 1409 since the PTP epoch. 1411 + Size: 32 bits. 1413 + Units: seconds. 1415 Nanoseconds: specifies the fractional portion of the number of 1416 seconds since the PTP epoch. 1418 + Size: 32 bits. 1420 + Units: nanoseconds. The value of this field is in the range 0 1421 to (10^9)-1. 1423 Epoch: 1425 PTP epoch. For details see e.g., [RFC8877]. 1427 Resolution: 1429 The resolution is 1 nanosecond. 1431 Wraparound: 1433 This time format wraps around every 2^32 seconds, which is roughly 1434 136 years. The next wraparound will occur in the year 2106. 1436 Synchronization Aspects: 1438 It is assumed that nodes that run this protocol are synchronized 1439 among themselves. Nodes MAY be synchronized to a global reference 1440 time. Note that if PTP is used for synchronization, the timestamp 1441 MAY be derived from the PTP-synchronized clock, allowing the 1442 timestamp to be measured with respect to the clock of an PTP 1443 Grandmaster clock. 1445 The PTP truncated timestamp format is not affected by leap 1446 seconds. 1448 6.2. NTP 64-bit Timestamp Format 1450 The Network Time Protocol (NTP) [RFC5905] timestamp format is 64 bits 1451 long. This format is specified in Section 4.2.1 of [RFC8877], and 1452 the details are presented below for the sake of completeness. 1454 0 1 2 3 1455 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1456 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1457 | Seconds | 1458 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1459 | Fraction | 1460 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1462 Figure 2: NTP [RFC5905] 64-bit Timestamp Format 1464 Timestamp field format: 1466 Seconds: specifies the integer portion of the number of seconds 1467 since the NTP epoch. 1469 + Size: 32 bits. 1471 + Units: seconds. 1473 Fraction: specifies the fractional portion of the number of 1474 seconds since the NTP epoch. 1476 + Size: 32 bits. 1478 + Units: the unit is 2^(-32) seconds, which is roughly equal to 1479 233 picoseconds. 1481 Epoch: 1483 NTP Epoch. For details see [RFC5905]. 1485 Resolution: 1487 The resolution is 2^(-32) seconds. 1489 Wraparound: 1491 This time format wraps around every 2^32 seconds, which is roughly 1492 136 years. The next wraparound will occur in the year 2036. 1494 Synchronization Aspects: 1496 Nodes that use this timestamp format will typically be 1497 synchronized to UTC using NTP [RFC5905]. Thus, the timestamp MAY 1498 be derived from the NTP-synchronized clock, allowing the timestamp 1499 to be measured with respect to the clock of an NTP server. 1501 The NTP timestamp format is affected by leap seconds; it 1502 represents the number of seconds since the epoch minus the number 1503 of leap seconds that have occurred since the epoch. The value of 1504 a timestamp during or slightly after a leap second could be 1505 temporarily inaccurate. 1507 6.3. POSIX-based Timestamp Format 1509 This timestamp format is based on the POSIX time format [POSIX]. The 1510 detailed specification of the timestamp format used in this document 1511 is presented below. 1513 0 1 2 3 1514 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1515 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1516 | Seconds | 1517 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1518 | Microseconds | 1519 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1521 Figure 3: POSIX-based Timestamp Format 1523 Timestamp field format: 1525 Seconds: specifies the integer portion of the number of seconds 1526 since the POSIX epoch. 1528 + Size: 32 bits. 1530 + Units: seconds. 1532 Microseconds: specifies the fractional portion of the number of 1533 seconds since the POSIX epoch. 1535 + Size: 32 bits. 1537 + Units: the unit is microseconds. The value of this field is in 1538 the range 0 to (10^6)-1. 1540 Epoch: 1542 POSIX epoch. For details, see [POSIX], appendix A.4.16. 1544 Resolution: 1546 The resolution is 1 microsecond. 1548 Wraparound: 1550 This time format wraps around every 2^32 seconds, which is roughly 1551 136 years. The next wraparound will occur in the year 2106. 1553 Synchronization Aspects: 1555 It is assumed that nodes that use this timestamp format run the 1556 Linux operating system, and hence use the POSIX time. In some 1557 cases nodes MAY be synchronized to UTC using a synchronization 1558 mechanism that is outside the scope of this document, such as NTP 1559 [RFC5905]. Thus, the timestamp MAY be derived from the NTP- 1560 synchronized clock, allowing the timestamp to be measured with 1561 respect to the clock of an NTP server. 1563 7. IOAM Data Export 1565 IOAM nodes collect information for packets traversing a domain that 1566 supports IOAM. IOAM decapsulating nodes as well as IOAM transit 1567 nodes can choose to retrieve IOAM information from the packet, 1568 process the information further and export the information using 1569 e.g., IPFIX. The mechanisms and associated data formats for 1570 exporting IOAM data is outside the scope of this document. 1572 A way to perform raw data export of IOAM data using IPFIX is 1573 discussed in [I-D.spiegel-ippm-ioam-rawexport]. 1575 8. IANA Considerations 1577 This document requests the following IANA Actions. 1579 IANA is requested to define a registry group named "In-Situ OAM 1580 (IOAM) Protocol Parameters". 1582 This group will include the following registries: 1584 IOAM Option-Type 1586 IOAM Trace-Type 1588 IOAM Trace-Flags 1590 IOAM POT-Type 1592 IOAM POT-Flags 1594 IOAM E2E-Type 1596 IOAM Namespace-ID 1598 The subsequent sub-sections detail the registries herein contained. 1600 8.1. IOAM Option-Type Registry 1602 This registry defines 128 code points for the IOAM Option-Type field 1603 for identifying IOAM Option-Types as explained in Section 5. The 1604 following code points are defined in this draft: 1606 0 IOAM Pre-allocated Trace Option-Type 1607 1 IOAM Incremental Trace Option-Type 1609 2 IOAM POT Option-Type 1611 3 IOAM E2E Option-Type 1613 4 - 127 are available for assignment via "IETF Review" process as per 1614 [RFC8126]. 1616 New registration requests MUST use the following template: 1618 Name: Name of the newly registered Option-Type. 1620 Code point: Desired value of the requested code point. 1622 Description: Brief description of the newly registered Option-Type. 1624 Reference: Reference to the document that defines the new Option- 1625 Type. 1627 The evaluation of a new registration request MUST also include 1628 checking whether the new IOAM Option-Type includes an IOAM-Namespace 1629 field and that the IOAM-Namespace field is the first field in the 1630 newly defined header of the new Option-Type. 1632 8.2. IOAM Trace-Type Registry 1634 This registry defines code point for each bit in the 24-bit IOAM- 1635 Trace-Type field for Pre-allocated Trace-Option-Type and Incremental 1636 Trace-Option-Type defined in Section 5.4. The meaning of Bits 0 - 11 1637 is defined in this document in Paragraph 5 of Section 5.4.1: 1639 Bit 0 hop_Lim and node_id in short format 1641 Bit 1 ingress_if_id and egress_if_id in short format 1643 Bit 2 timestamp seconds 1645 Bit 3 timestamp fraction 1647 Bit 4 transit delay 1649 Bit 5 namespace specific data in short format 1651 Bit 6 queue depth 1653 Bit 7 checksum complement 1654 Bit 8 hop_Lim and node_id in wide format 1656 Bit 9 ingress_if_id and egress_if_id in wide format 1658 Bit 10 namespace specific data in wide format 1660 Bit 11 buffer occupancy 1662 Bit 22 variable length Opaque State Snapshot 1664 Bit 23 reserved 1666 The meaning for Bits 12 - 21 are available for assignment via "IETF 1667 Review" process as per [RFC8126]. 1669 New registration requests MUST use the following template: 1671 Bit: Desired bit to be allocated in the 24-bit IOAM Trace-Option- 1672 Type field for Pre-allocated Trace-Option-Type and Incremental 1673 Trace-Option-Type. 1675 Description: Brief description of the newly registered bit. 1677 Reference: Reference to the document that defines the new bit. 1679 8.3. IOAM Trace-Flags Registry 1681 This registry defines code points for each bit in the 4 bit flags for 1682 the Pre-allocated trace option and for the Incremental trace option 1683 defined in Section 5.4. The meaning of Bit 0 (the most significant 1684 bit) for trace flags is defined in this document in Paragraph 3 of 1685 Section 5.4.1: 1687 Bit 0 "Overflow" (O-bit) 1689 Bit 1 - 3 are available for assignment via "IETF Review" process as 1690 per [RFC8126]. 1692 New registration requests MUST use the following template: 1694 Bit: Desired bit to be allocated in the 8 bit flags field of the 1695 Pre-allocated Trace-Option-Type and for the Incremental Trace- 1696 Option-Type. 1698 Description: Brief description of the newly registered bit. 1700 Reference: Reference to the document that defines the new bit. 1702 8.4. IOAM POT-Type Registry 1704 This registry defines 256 code points to define IOAM POT Type for 1705 IOAM proof of transit option Section 5.5. The code point value 0 is 1706 defined in this document: 1708 0: 16 Octet POT data 1710 1 - 255 are available for assignment via "IETF Review" process as per 1711 [RFC8126]. 1713 New registration requests MUST use the following template: 1715 Name: Name of the newly registered POT-Type. 1717 Code point: Desired value of the requested code point. 1719 Description: Brief description of the newly registered POT-Type. 1721 Reference: Reference to the document that defines the new POT-Type. 1723 8.5. IOAM POT-Flags Registry 1725 This registry defines code points for each bit in the 8 bit flags for 1726 IOAM POT Option-Type defined in Section 5.5. 1728 The meaning for Bits 0 - 7 are available for assignment via "IETF 1729 Review" process as per [RFC8126]. 1731 New registration requests MUST use the following template: 1733 Bit: Desired bit to be allocated in the 8 bit flags field of the 1734 IOAM POT Option-Type. 1736 Description: Brief description of the newly registered bit. 1738 Reference: Reference to the document that defines the new bit. 1740 8.6. IOAM E2E-Type Registry 1742 This registry defines code points for each bit in the 16 bit IOAM- 1743 E2E-Type field for IOAM E2E option Section 5.6. The meaning of Bit 0 1744 - 3 are defined in this document: 1746 Bit 0 64-bit sequence number 1748 Bit 1 32-bit sequence number 1749 Bit 2 timestamp seconds 1751 Bit 3 timestamp fraction 1753 The meaning of Bits 4 - 15 are available for assignment via "IETF 1754 Review" process as per [RFC8126]. 1756 New registration requests MUST use the following template: 1758 Bit: Desired bit to be allocated in the 16 bit IOAM-E2E-Type field. 1760 Description: Brief description of the newly registered bit. 1762 Reference: Reference to the document that defines the new bit. 1764 8.7. IOAM Namespace-ID Registry 1766 IANA is requested to set up an "IOAM Namespace-ID Registry", 1767 containing 16-bit values and following the template for requests 1768 shown below. The meaning of 0x0000 is defined in this document. 1769 IANA is requested to reserve the values 0x0001 to 0x7FFF for private 1770 use (managed by operators), as specified in Section 5.3 of the 1771 current document. Registry entries for the values 0x8000 to 0xFFFF 1772 are to be assigned via the "Expert Review" policy defined in 1773 [RFC8126]. 1775 Upon receiving a new allocation request, a designated expert will 1776 perform the following: 1778 o Review whether the request is complete, i.e., the registration 1779 template has been filled in. The expert will send incomplete 1780 requests back to the requestor. 1782 o Check whether the request is neither a duplicate of nor 1783 conflicting with either an already existing allocation or a 1784 pending allocation. In case of duplicates or conflicts, the 1785 expert will ask the requestor to update the allocation request 1786 accordingly. 1788 o Solicit feedback from relevant working groups and communities to 1789 ensure that the new allocation request has been properly reviewed 1790 and that rough consensus on the request exists. At a minimum, the 1791 expert will solicit feedback from the IPPM working group in the 1792 IETF by posting the request to the ippm@ietf.org mailing list. 1793 The expert will allow for a 3-week review period on the mailing 1794 lists. If the feedback received from the relevant working groups 1795 and communities within the review period indicates rough consensus 1796 on the request, the expert will approve the request and ask IANA 1797 for allocating the new Namespace-ID. In case the expert senses a 1798 lack of consensus from the feedback received, the expert will ask 1799 the requestor to engage with the corresponding working groups and 1800 communities to further review and refine the request. 1802 It is intended that any allocation will be accompanied by a published 1803 RFC. In order to allow for the allocation of code points prior to 1804 the RFC being approved for publication, the designated expert can 1805 approve allocations once it seems clear that an RFC will be 1806 published. 1808 0x0000: default namespace (known to all IOAM nodes) 1810 0x0001 - 0x7FFF: reserved for private use 1812 0x8000 - 0xFFFF: unassigned 1814 New registration requests MUST use the following template: 1816 Name: Name of the newly registered Namespace-ID. 1818 Code point: Desired value of the requested Namespace-ID. 1820 Description: Brief description of the newly registered Namespace-ID. 1822 Reference: Reference to the document that defines the new Namespace- 1823 ID. 1825 Status of the registration: Status can be either "permanent" or 1826 "provisional". Namespace-ID registrations following a successful 1827 expert review will have the status "provisional". Once the RFC, 1828 which defines the new Namespace-ID is published, the status is 1829 changed to "permanent". 1831 9. Management and Deployment Considerations 1833 This document defines the structure and use of IOAM data fields. 1834 This document does not define the encapsulation of IOAM data fields 1835 into different protocols. Management and deployment aspects for IOAM 1836 have to be considered within the context of the protocol IOAM data 1837 fields are encapsulated into and as such, are out of scope for this 1838 document. For a discussion of IOAM deployment, please also refer to 1839 [I-D.ietf-ippm-ioam-deployment], which outlines a framework for IOAM 1840 deployment and provides best current practices. 1842 10. Security Considerations 1844 As discussed in [RFC7276], a successful attack on an OAM protocol in 1845 general, and specifically on IOAM, can prevent the detection of 1846 failures or anomalies, or create a false illusion of nonexistent 1847 ones. In particular, these threats are applicable by compromising 1848 the integrity of IOAM data, either by maliciously modifying IOAM 1849 options in transit, or by injecting packets with maliciously 1850 generated IOAM options. All nodes in the path of a IOAM carrying 1851 packet can perform such an attack. 1853 The Proof of Transit Option-Type (see Section 5.5) is used for 1854 verifying the path of data packets, i.e., proving that packets 1855 transited through a defined set of nodes. 1857 In case an attacker gains access to several nodes in a network and 1858 would be able to change the system software of these nodes, IOAM data 1859 fields could be misused and repurposed for a use different from what 1860 is specified in this document. One type of misuse is the 1861 implementation of a covert channel between network nodes. 1863 From a confidentiality perspective, although IOAM options are not 1864 expected to contain user data, they can be used for network 1865 reconnaissance, allowing attackers to collect information about 1866 network paths, performance, queue states, buffer occupancy and other 1867 information. Moreover, if IOAM data leaks from the IOAM-domain it 1868 could enable reconnaissance beyond the scope of the IOAM-domain. One 1869 possible application of such reconnaissance is to gauge the 1870 effectiveness of an ongoing attack, e.g., if buffers and queues are 1871 overflowing. 1873 IOAM can be used as a means for implementing Denial of Service (DoS) 1874 attacks, or for amplifying them. For example, a malicious attacker 1875 can add an IOAM header to packets in order to consume the resources 1876 of network devices that take part in IOAM or entities that receive, 1877 collect or analyze the IOAM data. Another example is a packet length 1878 attack, in which an attacker pushes headers associated with IOAM 1879 Option-Types into data packets, causing these packets to be increased 1880 beyond the MTU size, resulting in fragmentation or in packet drops. 1881 In case POT is used, an attacker could corrupt the POT data fields in 1882 the packet, resulting in a verification failure of the POT data, even 1883 if the packet followed the correct path. 1885 Since IOAM options can include timestamps, if network devices use 1886 synchronization protocols then any attack on the time protocol 1887 [RFC7384] can compromise the integrity of the timestamp-related data 1888 fields. 1890 At the management plane, attacks can be set up by misconfiguring or 1891 by maliciously configuring IOAM-enabled nodes in a way that enables 1892 other attacks. IOAM configuration should only managed by authorized 1893 processes or users. 1895 IETF protocols require features to ensure their security. While IOAM 1896 data fields don't represent a protocol by themselves, the IOAM data 1897 fields add to the protocol that the IOAM data fields are encapsulated 1898 into. Any specification that defines how IOAM data fields carried in 1899 an encapsulating protocol MUST provide for a mechanism for 1900 cryptographic integrity protection of the IOAM data fields. 1901 Cryptographic integrity protection could be either achieved through a 1902 mechanism of the encapsulating protocol or it could incorporate the 1903 mechanisms specified in [I-D.ietf-ippm-ioam-data-integrity]. 1905 The current document does not define a specific IOAM encapsulation. 1906 It has to be noted that some IOAM encapsulation types can introduce 1907 specific security considerations. A specification that defines an 1908 IOAM encapsulation is expected to address the respective 1909 encapsulation-specific security considerations. 1911 Notably, IOAM is expected to be deployed in limited domains, thus 1912 confining the potential attack vectors to within the limited domain. 1913 A limited administrative domain provides the operator with the means 1914 to select, monitor, and control the access of all the network 1915 devices, making these devices trusted by the operator. Indeed, in 1916 order to limit the scope of threats mentioned above to within the 1917 current limited domain the network operator is expected to enforce 1918 policies that prevent IOAM traffic from leaking outside of the IOAM 1919 domain, and prevent IOAM data from outside the domain to be processed 1920 and used within the domain. 1922 This document does not define the data contents of custom fields like 1923 "Opaque State Snapshot" and "namespace specific data" IOAM data 1924 fields. These custom data fields will have security considerations 1925 corresponding to their defined data contents that need to be 1926 described where those formats are defined. 1928 IOAM deployments which leverage both IOAM Trace Option-Types, i.e., 1929 the Pre-allocated Trace Option-Type and Incremental Trace Option-Type 1930 can suffer from incomplete visibility if the information gathered via 1931 the two Trace Option-Types is not correlated and aggregated 1932 appropriately. If IOAM transit nodes leverage the IOAM data fields 1933 in the packet for further actions or insights, then IOAM transit 1934 nodes which only support one IOAM Trace Option-Type in an IOAM 1935 deployment which leverages both Trace Option-Types, have limited 1936 visibility and thus can draw inappropriate conclusions or take wrong 1937 actions. 1939 The security considerations of a system that deploys IOAM, much like 1940 any system, has to be reviewed on a per-deployment-scenario basis, 1941 based on a systems-specific threat analysis, which can lead to 1942 specific security solutions that are beyond the scope of the current 1943 document. Specifically, in an IOAM deployment that is not confined 1944 to a single LAN, but spans multiple inter-connected sites (for 1945 example, using an overlay network), the inter-site links can be 1946 secured (e.g., by IPsec) in order to avoid external threats. 1948 IOAM deployment considerations, including approaches to mitigate the 1949 above discussed threads and potential attacks are outside the scope 1950 of this document. IOAM deployment considerations are discussed in 1951 [I-D.ietf-ippm-ioam-deployment]. 1953 11. Acknowledgements 1955 The authors would like to thank Eric Vyncke, Nalini Elkins, Srihari 1956 Raghavan, Ranganathan T S, Karthik Babu Harichandra Babu, Akshaya 1957 Nadahalli, LJ Wobker, Erik Nordmark, Vengada Prasad Govindan, Andrew 1958 Yourtchenko, Aviv Kfir, Tianran Zhou, Zhenbin (Robin) and Greg Mirsky 1959 for the comments and advice. 1961 This document leverages and builds on top of several concepts 1962 described in [I-D.kitamura-ipv6-record-route]. The authors would 1963 like to acknowledge the work done by the author Hiroshi Kitamura and 1964 people involved in writing it. 1966 The authors would like to gracefully acknowledge useful review and 1967 insightful comments received from Joe Clarke, Al Morton, Tom Herbert, 1968 Carlos Bernardos, Haoyu Song, Mickey Spiegel, Roman Danyliw, Benjamin 1969 Kaduk, Murray S. Kucherawy, Ian Swett, Martin Duke, Francesca 1970 Palombini, Lars Eggert, Alvaro Retana, Erik Kline, Robert Wilton, 1971 Zaheduzzaman Sarker, Dan Romascanu and Barak Gafni. 1973 12. References 1975 12.1. Normative References 1977 [POSIX] Institute of Electrical and Electronics Engineers, "IEEE 1978 Std 1003.1-2017 (Revision of IEEE Std 1003.1-2017) - IEEE 1979 Standard for Information Technology - Portable Operating 1980 System Interface (POSIX(TM) Base Specifications, Issue 1981 7)", IEEE Std 1003.1-2017, 2017, 1982 . 1985 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1986 Requirement Levels", BCP 14, RFC 2119, 1987 DOI 10.17487/RFC2119, March 1997, 1988 . 1990 [RFC5905] Mills, D., Martin, J., Ed., Burbank, J., and W. Kasch, 1991 "Network Time Protocol Version 4: Protocol and Algorithms 1992 Specification", RFC 5905, DOI 10.17487/RFC5905, June 2010, 1993 . 1995 [RFC8126] Cotton, M., Leiba, B., and T. Narten, "Guidelines for 1996 Writing an IANA Considerations Section in RFCs", BCP 26, 1997 RFC 8126, DOI 10.17487/RFC8126, June 2017, 1998 . 2000 [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2001 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, 2002 May 2017, . 2004 12.2. Informative References 2006 [I-D.ietf-ippm-ioam-data-integrity] 2007 Brockners, F., Bhandari, S., and T. Mizrahi, "Integrity of 2008 In-situ OAM Data Fields", draft-ietf-ippm-ioam-data- 2009 integrity-00 (work in progress), October 2021. 2011 [I-D.ietf-ippm-ioam-deployment] 2012 Brockners, F., Bhandari, S., Bernier, D., and T. Mizrahi, 2013 "In-situ OAM Deployment", draft-ietf-ippm-ioam- 2014 deployment-00 (work in progress), October 2021. 2016 [I-D.ietf-nvo3-vxlan-gpe] 2017 (Editor), F. M., (editor), L. K., and U. E. (editor), 2018 "Generic Protocol Extension for VXLAN (VXLAN-GPE)", draft- 2019 ietf-nvo3-vxlan-gpe-12 (work in progress), September 2021. 2021 [I-D.kitamura-ipv6-record-route] 2022 Kitamura, H., "Record Route for IPv6 (PR6) Hop-by-Hop 2023 Option Extension", draft-kitamura-ipv6-record-route-00 2024 (work in progress), November 2000. 2026 [I-D.spiegel-ippm-ioam-rawexport] 2027 Spiegel, M., Brockners, F., Bhandari, S., and R. 2028 Sivakolundu, "In-situ OAM raw data export with IPFIX", 2029 draft-spiegel-ippm-ioam-rawexport-05 (work in progress), 2030 July 2021. 2032 [RFC7276] Mizrahi, T., Sprecher, N., Bellagamba, E., and Y. 2033 Weingarten, "An Overview of Operations, Administration, 2034 and Maintenance (OAM) Tools", RFC 7276, 2035 DOI 10.17487/RFC7276, June 2014, 2036 . 2038 [RFC7384] Mizrahi, T., "Security Requirements of Time Protocols in 2039 Packet Switched Networks", RFC 7384, DOI 10.17487/RFC7384, 2040 October 2014, . 2042 [RFC7665] Halpern, J., Ed. and C. Pignataro, Ed., "Service Function 2043 Chaining (SFC) Architecture", RFC 7665, 2044 DOI 10.17487/RFC7665, October 2015, 2045 . 2047 [RFC7799] Morton, A., "Active and Passive Metrics and Methods (with 2048 Hybrid Types In-Between)", RFC 7799, DOI 10.17487/RFC7799, 2049 May 2016, . 2051 [RFC7820] Mizrahi, T., "UDP Checksum Complement in the One-Way 2052 Active Measurement Protocol (OWAMP) and Two-Way Active 2053 Measurement Protocol (TWAMP)", RFC 7820, 2054 DOI 10.17487/RFC7820, March 2016, 2055 . 2057 [RFC7821] Mizrahi, T., "UDP Checksum Complement in the Network Time 2058 Protocol (NTP)", RFC 7821, DOI 10.17487/RFC7821, March 2059 2016, . 2061 [RFC8300] Quinn, P., Ed., Elzur, U., Ed., and C. Pignataro, Ed., 2062 "Network Service Header (NSH)", RFC 8300, 2063 DOI 10.17487/RFC8300, January 2018, 2064 . 2066 [RFC8799] Carpenter, B. and B. Liu, "Limited Domains and Internet 2067 Protocols", RFC 8799, DOI 10.17487/RFC8799, July 2020, 2068 . 2070 [RFC8877] Mizrahi, T., Fabini, J., and A. Morton, "Guidelines for 2071 Defining Packet Timestamps", RFC 8877, 2072 DOI 10.17487/RFC8877, September 2020, 2073 . 2075 [RFC8926] Gross, J., Ed., Ganga, I., Ed., and T. Sridhar, Ed., 2076 "Geneve: Generic Network Virtualization Encapsulation", 2077 RFC 8926, DOI 10.17487/RFC8926, November 2020, 2078 . 2080 Contributors' Addresses 2082 Carlos Pignataro 2083 Cisco Systems, Inc. 2084 7200-11 Kit Creek Road 2085 Research Triangle Park, NC 27709 2086 United States 2088 Email: cpignata@cisco.com 2090 Mickey Spiegel 2091 Barefoot Networks, an Intel company 2092 4750 Patrick Henry Drive 2093 Santa Clara, CA 95054 2094 US 2096 Email: mickey.spiegel@intel.com 2098 Barak Gafni 2099 Nvidia 2100 350 Oakmead Parkway, Suite 100 2101 Sunnyvale, CA 94085 2102 U.S.A. 2104 Email: gbarak@nvidia.com 2106 Jennifer Lemon 2107 Broadcom 2108 270 Innovation Drive 2109 San Jose, CA 95134 2110 US 2112 Email: jennifer.lemon@broadcom.com 2114 Hannes Gredler 2115 RtBrick Inc. 2117 Email: hannes@rtbrick.com 2119 John Leddy 2120 United States 2122 Email: john@leddy.net 2123 Stephen Youell 2124 JP Morgan Chase 2125 25 Bank Street 2126 London E14 5JP 2127 United Kingdom 2129 Email: stephen.youell@jpmorgan.com 2131 David Mozes 2133 Email: mosesster@gmail.com 2135 Petr Lapukhov 2136 Facebook 2137 1 Hacker Way 2138 Menlo Park, CA 94025 2139 US 2141 Email: petr@fb.com 2143 Remy Chang 2144 Barefoot Networks 2145 4750 Patrick Henry Drive 2146 Santa Clara, CA 95054 2147 US 2149 Email: remy@barefootnetworks.com 2151 Daniel Bernier 2152 Bell Canada 2153 Canada 2155 Email: daniel.bernier@bell.ca 2157 Authors' Addresses 2159 Frank Brockners (editor) 2160 Cisco Systems, Inc. 2161 Hansaallee 249, 3rd Floor 2162 DUESSELDORF, NORDRHEIN-WESTFALEN 40549 2163 Germany 2165 Email: fbrockne@cisco.com 2166 Shwetha Bhandari (editor) 2167 Thoughtspot 2168 3rd Floor, Indiqube Orion, 24th Main Rd, Garden Layout, HSR Layout 2169 Bangalore, KARNATAKA 560 102 2170 India 2172 Email: shwetha.bhandari@thoughtspot.com 2174 Tal Mizrahi (editor) 2175 Huawei 2176 8-2 Matam 2177 Haifa 3190501 2178 Israel 2180 Email: tal.mizrahi.phd@gmail.com