idnits 2.17.1 draft-ietf-ippm-ioam-data-15.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (October 3, 2021) is 935 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) -- Looks like a reference, but probably isn't: '0' on line 637 -- Looks like a reference, but probably isn't: '1' on line 641 -- Possible downref: Non-RFC (?) normative reference: ref. 'POSIX' == Outdated reference: A later version (-13) exists of draft-ietf-nvo3-vxlan-gpe-12 == Outdated reference: A later version (-07) exists of draft-spiegel-ippm-ioam-rawexport-05 Summary: 0 errors (**), 0 flaws (~~), 3 warnings (==), 4 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 ippm F. Brockners, Ed. 3 Internet-Draft Cisco 4 Intended status: Standards Track S. Bhandari, Ed. 5 Expires: April 6, 2022 Thoughtspot 6 T. Mizrahi, Ed. 7 Huawei 8 October 3, 2021 10 Data Fields for In-situ OAM 11 draft-ietf-ippm-ioam-data-15 13 Abstract 15 In-situ Operations, Administration, and Maintenance (IOAM) records 16 operational and telemetry information in the packet while the packet 17 traverses a path in the network. This document discusses the data 18 fields and associated data types for in-situ OAM. In-situ OAM data 19 fields can be encapsulated into a variety of protocols such as NSH, 20 Segment Routing, Geneve, or IPv6. In-situ OAM can be used to 21 complement OAM mechanisms based on, e.g., ICMP or other types of 22 probe packets. 24 Status of This Memo 26 This Internet-Draft is submitted in full conformance with the 27 provisions of BCP 78 and BCP 79. 29 Internet-Drafts are working documents of the Internet Engineering 30 Task Force (IETF). Note that other groups may also distribute 31 working documents as Internet-Drafts. The list of current Internet- 32 Drafts is at https://datatracker.ietf.org/drafts/current/. 34 Internet-Drafts are draft documents valid for a maximum of six months 35 and may be updated, replaced, or obsoleted by other documents at any 36 time. It is inappropriate to use Internet-Drafts as reference 37 material or to cite them other than as "work in progress." 39 This Internet-Draft will expire on April 6, 2022. 41 Copyright Notice 43 Copyright (c) 2021 IETF Trust and the persons identified as the 44 document authors. All rights reserved. 46 This document is subject to BCP 78 and the IETF Trust's Legal 47 Provisions Relating to IETF Documents 48 (https://trustee.ietf.org/license-info) in effect on the date of 49 publication of this document. Please review these documents 50 carefully, as they describe your rights and restrictions with respect 51 to this document. Code Components extracted from this document must 52 include Simplified BSD License text as described in Section 4.e of 53 the Trust Legal Provisions and are provided without warranty as 54 described in the Simplified BSD License. 56 Table of Contents 58 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 59 2. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 3 60 3. Conventions . . . . . . . . . . . . . . . . . . . . . . . . . 4 61 4. Scope, Applicability, and Assumptions . . . . . . . . . . . . 5 62 5. IOAM Data-Fields, Types, Nodes . . . . . . . . . . . . . . . 6 63 5.1. IOAM Data-Fields and Option-Types . . . . . . . . . . . . 7 64 5.2. IOAM-Domains and types of IOAM Nodes . . . . . . . . . . 7 65 5.3. IOAM-Namespaces . . . . . . . . . . . . . . . . . . . . . 9 66 5.4. IOAM Trace Option-Types . . . . . . . . . . . . . . . . . 11 67 5.4.1. Pre-allocated and Incremental Trace Option-Types . . 13 68 5.4.2. IOAM node data fields and associated formats . . . . 18 69 5.4.2.1. Hop_Lim and node_id short format . . . . . . . . 18 70 5.4.2.2. ingress_if_id and egress_if_id . . . . . . . . . 19 71 5.4.2.3. timestamp seconds . . . . . . . . . . . . . . . . 19 72 5.4.2.4. timestamp fraction . . . . . . . . . . . . . . . 20 73 5.4.2.5. transit delay . . . . . . . . . . . . . . . . . . 20 74 5.4.2.6. namespace specific data . . . . . . . . . . . . . 20 75 5.4.2.7. queue depth . . . . . . . . . . . . . . . . . . . 21 76 5.4.2.8. Checksum Complement . . . . . . . . . . . . . . . 21 77 5.4.2.9. Hop_Lim and node_id wide . . . . . . . . . . . . 22 78 5.4.2.10. ingress_if_id and egress_if_id wide . . . . . . . 22 79 5.4.2.11. namespace specific data wide . . . . . . . . . . 22 80 5.4.2.12. buffer occupancy . . . . . . . . . . . . . . . . 23 81 5.4.2.13. Opaque State Snapshot . . . . . . . . . . . . . . 23 82 5.4.3. Examples of IOAM node data . . . . . . . . . . . . . 24 83 5.5. IOAM Proof of Transit Option-Type . . . . . . . . . . . . 26 84 5.5.1. IOAM Proof of Transit Type 0 . . . . . . . . . . . . 28 85 5.6. IOAM Edge-to-Edge Option-Type . . . . . . . . . . . . . . 29 86 6. Timestamp Formats . . . . . . . . . . . . . . . . . . . . . . 31 87 6.1. PTP Truncated Timestamp Format . . . . . . . . . . . . . 31 88 6.2. NTP 64-bit Timestamp Format . . . . . . . . . . . . . . . 32 89 6.3. POSIX-based Timestamp Format . . . . . . . . . . . . . . 34 90 7. IOAM Data Export . . . . . . . . . . . . . . . . . . . . . . 35 91 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 35 92 8.1. IOAM Option-Type Registry . . . . . . . . . . . . . . . . 36 93 8.2. IOAM Trace-Type Registry . . . . . . . . . . . . . . . . 36 94 8.3. IOAM Trace-Flags Registry . . . . . . . . . . . . . . . . 37 95 8.4. IOAM POT-Type Registry . . . . . . . . . . . . . . . . . 38 96 8.5. IOAM POT-Flags Registry . . . . . . . . . . . . . . . . . 38 97 8.6. IOAM E2E-Type Registry . . . . . . . . . . . . . . . . . 39 98 8.7. IOAM Namespace-ID Registry . . . . . . . . . . . . . . . 39 99 9. Management and Deployment Considerations . . . . . . . . . . 40 100 10. Security Considerations . . . . . . . . . . . . . . . . . . . 41 101 11. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 43 102 12. References . . . . . . . . . . . . . . . . . . . . . . . . . 43 103 12.1. Normative References . . . . . . . . . . . . . . . . . . 43 104 12.2. Informative References . . . . . . . . . . . . . . . . . 44 105 Contributors' Addresses . . . . . . . . . . . . . . . . . . . . . 45 106 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 47 108 1. Introduction 110 This document defines data fields for "in-situ" Operations, 111 Administration, and Maintenance (IOAM). In-situ OAM records OAM 112 information within the packet while the packet traverses a particular 113 network domain. The term "in-situ" refers to the fact that the OAM 114 data is added to the data packets rather than being sent within 115 packets specifically dedicated to OAM. IOAM is to complement 116 mechanisms such as Ping or Traceroute. In terms of "active" or 117 "passive" OAM, "in-situ" OAM can be considered a hybrid OAM type. 118 "In-situ" mechanisms do not require extra packets to be sent. IOAM 119 adds information to the already available data packets and therefore 120 cannot be considered passive. In terms of the classification given 121 in [RFC7799], IOAM could be portrayed as Hybrid Type I. IOAM 122 mechanisms can be leveraged where mechanisms using, e.g., ICMP do not 123 apply or do not offer the desired results, such as proving that a 124 certain traffic flow takes a pre-defined path, SLA verification for 125 the data traffic, detailed statistics on traffic distribution paths 126 in networks that distribute traffic across multiple paths, or 127 scenarios in which probe traffic is potentially handled differently 128 from regular data traffic by the network devices. 130 The term "in situ OAM" was originally motivated by the use of OAM 131 related mechanisms that add information into a packet. This document 132 uses IOAM as a term defining the IOAM technology. IOAM includes "in- 133 situ" mechanisms, but also mechanisms that could trigger the creation 134 of additional packets dedicated to OAM. 136 2. Contributors 138 This document was the collective effort of several authors. The text 139 and content were contributed by the editors and the co-authors listed 140 below. The contact information of the co-authors appears at the end 141 of this document. 143 o Carlos Pignataro 144 o Mickey Spiegel 146 o Barak Gafni 148 o Jennifer Lemon 150 o Hannes Gredler 152 o John Leddy 154 o Stephen Youell 156 o David Mozes 158 o Petr Lapukhov 160 o Remy Chang 162 o Daniel Bernier 164 3. Conventions 166 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 167 "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and 168 "OPTIONAL" in this document are to be interpreted as described in BCP 169 14 [RFC2119] [RFC8174] when, and only when, they appear in all 170 capitals, as shown here. 172 Abbreviations and definitions used in this document: 174 E2E: Edge to Edge 176 Geneve: Generic Network Virtualization Encapsulation [RFC8926] 178 IOAM: In-situ Operations, Administration, and Maintenance 180 MTU: Maximum Transmit Unit 182 NSH: Network Service Header [RFC8300] 184 OAM: Operations, Administration, and Maintenance 186 PMTU: Path MTU 188 POT: Proof of Transit 190 Short format: "Short format" refers to an IOAM-Data-Field which 191 comprises 4 octets. 193 SID: Segment Identifier 195 SR: Segment Routing 197 VXLAN-GPE: Virtual eXtensible Local Area Network, Generic Protocol 198 Extension [I-D.ietf-nvo3-vxlan-gpe] 200 Wide format: "Wide format" refers to an IOAM-Data-Field which 201 comprises 8 octets. 203 4. Scope, Applicability, and Assumptions 205 IOAM assumes a set of constraints as well as guiding principles and 206 concepts that go hand in hand with the definition of the IOAM data 207 fields. These constraints, guiding principles, and concepts are 208 described in this section. A discussion of how IOAM data fields and 209 the associated concepts are applied to an IOAM deployment are out of 210 scope for this document. Please refer to 211 [I-D.brockners-opsawg-ioam-deployment] for IOAM deployment 212 considerations. 214 Scope: This document defines the data fields and associated data 215 types for in-situ OAM. The in-situ OAM data fields can be 216 encapsulated in a variety of protocols, including NSH, Segment 217 Routing, Geneve, and IPv6. Specification details for these different 218 protocols are outside the scope of this document. It is expected 219 that each such encapsulation would be specified by an RFC, jointly 220 designed by the working group that develops or maintains the 221 encapsulation protocol and the IETF IPPM working group. 223 Deployment domain (or scope) of in-situ OAM deployment: IOAM is 224 focused on "limited domains" as defined in [RFC8799]. For IOAM, a 225 limited domain could for example be an enterprise campus using 226 physical connections between devices or an overlay network using 227 virtual connections / tunnels for connectivity between said devices. 228 A limited domain which uses IOAM may constitute one or multiple "IOAM 229 domains", each disambiguated through separate namespace identifiers. 230 An IOAM domain is bounded by its perimeter or edge. IOAM domains may 231 overlap inside the limited domain. Designers of protocol 232 encapsulations for IOAM specify mechanisms to ensure that IOAM data 233 stays within an IOAM domain. In addition, the operator of such a 234 domain is expected to put provisions in place to ensure that IOAM 235 data does not leak beyond the edge of an IOAM domain using, for 236 example, packet filtering methods. The operator SHOULD consider the 237 potential operational impact of IOAM to mechanisms such as ECMP 238 processing (e.g., load-balancing schemes based on packet length could 239 be impacted by the increased packet size due to IOAM), path MTU 240 (i.e., ensure that the MTU of all links within a domain is 241 sufficiently large to support the increased packet size due to IOAM) 242 and ICMP message handling (i.e., in case of IPv6, IOAM support for 243 ICMPv6 Echo Request/Reply is desired which would translate into 244 ICMPv6 extensions to enable IOAM-Data-Fields to be copied from an 245 Echo Request message to an Echo Reply message). 247 IOAM control points: IOAM-Data-Fields are added to or removed from 248 the user traffic by the devices which form the edge of a domain. 249 Devices which form an IOAM-Domain can add, update or remove IOAM- 250 Data-Fields. Edge devices of an IOAM-Domain can be hosts or network 251 devices. 253 Traffic-sets that IOAM is applied to: IOAM can be deployed on all or 254 only on subsets of the user traffic. Using IOAM on a selected set of 255 traffic (e.g., per interface, based on an access control list or flow 256 specification defining a specific set of traffic, etc.) could be 257 useful in deployments where the cost of processing IOAM-Data-Fields 258 by encapsulating, transit, or decapsulating node(s) might be a 259 concern from a performance or operational perspective. Thus limiting 260 the amount of traffic IOAM is applied to could be beneficial in some 261 deployments. 263 Encapsulation independence: The definition of IOAM-Data-Fields is 264 independent from the protocols the IOAM-Data-Fields are encapsulated 265 into. IOAM-Data-Fields can be encapsulated into several 266 encapsulating protocols. 268 Layering: If several encapsulation protocols (e.g., in case of 269 tunneling) are stacked on top of each other, IOAM-Data-Fields could 270 be present at multiple layers. The behavior follows the ships-in- 271 the-night model, i.e., IOAM-Data-Fields in one layer are independent 272 from IOAM-Data-Fields in another layer. Layering allows operators to 273 instrument the protocol layer they want to measure. The different 274 layers could, but do not have to, share the same IOAM encapsulation 275 mechanisms. 277 IOAM implementation: The definition of the IOAM-Data-Fields take the 278 specifics of devices with hardware data planes and software data 279 planes into account. 281 5. IOAM Data-Fields, Types, Nodes 283 This section details IOAM-related nomenclature and describes data 284 types such as IOAM-Data-Fields, IOAM-Types, IOAM-Namespaces as well 285 as the different types of IOAM nodes. 287 5.1. IOAM Data-Fields and Option-Types 289 An IOAM-Data-Field is a set of bits with a defined format and 290 meaning, which can be stored at a certain place in a packet for the 291 purpose of IOAM. 293 To accommodate the different uses of IOAM, IOAM-Data-Fields fall into 294 different categories. In IOAM, these categories are referred to as 295 IOAM-Option-Types. A common registry is maintained for IOAM-Option- 296 Types, see Section 8.1 for details. Corresponding to these IOAM- 297 Option-Types, different IOAM-Data-Fields are defined. 299 This document defines four IOAM-Option-Types: 301 o Pre-allocated Trace Option-Type 303 o Incremental Trace Option-Type 305 o Proof of Transit (POT) Option-Type 307 o Edge-to-Edge (E2E) Option-Type 309 Future IOAM-Option-Types can be allocated by IANA, as described in 310 Section 8.1. 312 5.2. IOAM-Domains and types of IOAM Nodes 314 IOAM is expected to be deployed in a specific domain. The part of 315 the network which employs IOAM is referred to as the "IOAM-Domain". 316 One or more IOAM-Option-Types are added to a packet upon entering the 317 IOAM-Domain and are removed from the packet when exiting the domain. 318 Within the IOAM-Domain, the IOAM-Data-Fields MAY be updated by 319 network nodes that the packet traverses. An IOAM-Domain consists of 320 "IOAM encapsulating nodes", "IOAM decapsulating nodes" and "IOAM 321 transit nodes". The role of a node (i.e., encapsulating, transit, 322 decapsulating) is defined within an IOAM-Namespace (see below). A 323 node can have different roles in different IOAM-Namespaces. 325 A device which adds at least one IOAM-Option-Type to the packet is 326 called an "IOAM encapsulating node", whereas a device which removes 327 an IOAM-Option-Type is referred to as an "IOAM decapsulating node". 328 Nodes within the domain which are aware of IOAM data and read and/or 329 write and/or process IOAM data are called "IOAM transit nodes". IOAM 330 nodes which add or remove the IOAM-Data-Fields can also update the 331 IOAM-Data-Fields at the same time. Or in other words, IOAM 332 encapsulating or decapsulating nodes can also serve as IOAM transit 333 nodes at the same time. Note that not every node in an IOAM domain 334 needs to be an IOAM transit node. For example, a deployment might 335 require that packets traverse a set of firewalls which support IOAM. 336 In that case, only the set of firewall nodes would be IOAM transit 337 nodes rather than all nodes. 339 An "IOAM encapsulating node" incorporates one or more IOAM-Option- 340 Types (from the list of IOAM-Types, see Section 8.1) into packets 341 that IOAM is enabled for. If IOAM is enabled for a selected subset 342 of the traffic, the IOAM encapsulating node is responsible for 343 applying the IOAM functionality to the selected subset. 345 An "IOAM transit node" read and/or write and/or process one or more 346 of the IOAM-Data-Fields. If both the Pre-allocated and the 347 Incremental Trace Option-Types are present in the packet, each IOAM 348 transit node based on configuration and available implementation of 349 IOAM populates IOAM trace data in either Pre-allocated or Incremental 350 Trace Option-Type but not both. A transit node MUST ignore IOAM- 351 Option-Types that it does not understand. A transit node MUST NOT 352 add new IOAM-Option-Types to a packet, MUST NOT remove IOAM-Option- 353 Types from a packet, and MUST NOT change the IOAM-Data-Fields of an 354 IOAM Edge-to-Edge Option-Type. 356 An "IOAM decapsulating node" removes IOAM-Option-Type(s) from 357 packets. 359 The role of an IOAM-encapsulating, IOAM-transit or IOAM-decapsulating 360 node is always performed within a specific IOAM-Namespace. This 361 means that an IOAM node which is, e.g., an IOAM-decapsulating node 362 for IOAM-Namespace "A" but not for IOAM-Namespace "B" will only 363 remove the IOAM-Option-Types for IOAM-Namespace "A" from the packet. 364 Note that this applies even for IOAM-Option-Types that the node does 365 not understand, for example an IOAM-Option-Type other than the four 366 described above, that is added in a future revision. An IOAM 367 decapsulating node situated at the edge of an IOAM domain MUST remove 368 all IOAM-Option-Types and associated encapsulation headers for all 369 IOAM-Namespaces from the packet. 371 IOAM-Namespaces allow for a namespace-specific definition and 372 interpretation of IOAM-Data-Fields. An interface-id could for 373 example point to a physical interface (e.g., to understand which 374 physical interface of an aggregated link is used when receiving or 375 transmitting a packet) whereas in another case it could refer to a 376 logical interface (e.g., in case of tunnels). Please refer to 377 Section 5.3 for details on IOAM-Namespaces. 379 5.3. IOAM-Namespaces 381 An IOAM-Namespace can be associated to a subset or all of the the 382 IOAM-Option-Types and their corresponding IOAM-Data-Fields. IOAM- 383 Namespaces add further context to IOAM-Option-Types and associated 384 IOAM-Data-Fields. The IOAM-Option-Types and associated IOAM-Data- 385 Fields are interpreted as defined in this document, regardless of the 386 value of the IOAM-Namespace. However, IOAM-Namespaces provide a way 387 to group nodes to support different deployment approaches of IOAM 388 (see a few example use-cases below). IOAM-Namespaces also help to 389 resolve potential issues which can occur due to IOAM-Data-Fields not 390 being globally unique (e.g., IOAM node identifiers do not have to be 391 globally unique). IOAM-Data-Fields significance is always within a 392 particular IOAM-Namespace. Given that IOAM-Data-Fields are always 393 interpreted the context of a specific namespace, the namespace-id 394 field always needs to be carried along with the IOAM data-fields 395 themselves. 397 An IOAM-Namespace is identified by a 16-bit namespace identifier 398 (Namespace-ID). The IOAM-Namespace field is included in all the 399 IOAM-Option-Types defined in this document, and MUST be included in 400 all future IOAM-Option-Types. The Namespace-ID value is divided into 401 two sub-ranges: 403 o An operator-assigned range from 0x0001 to 0x7FFF 405 o An IANA-assigned range from 0x8000 to 0xFFFF 407 The IANA-assigned range is intended to allow future extensions to 408 have new and interoperable IOAM functionality, while the operator- 409 assigned range is intended to be domain-specific, and managed by the 410 network operator. The Namespace-ID value of 0x0000 is the "Default- 411 Namespace-ID". The Default-Namespace-ID indicates that no specific 412 namespace is associated with the IOAM data fields in the packet. The 413 Default-Namespace-ID MUST be supported by all nodes implementing 414 IOAM. A use-case for the Default-Namespace-ID are deployments which 415 do not leverage specific namespaces for some or all of their packets 416 that carry IOAM data fields. 418 Namespace identifiers allow devices which are IOAM capable to 419 determine: 421 o whether IOAM-Option-Type(s) need to be processed by a device: If 422 the Namespace-ID contained in a packet does not match any 423 Namespace-ID the node is configured to operate on, then the node 424 MUST NOT change the contents of the IOAM-Data-Fields. 426 o which IOAM-Option-Type needs to be processed/updated in case there 427 are multiple IOAM-Option-Types present in the packet. Multiple 428 IOAM-Option-Types can be present in a packet in case of 429 overlapping IOAM-Domains or in case of a layered IOAM deployment. 431 o whether IOAM-Option-Type(s) have to be removed from the packet, 432 e.g., at a domain edge or domain boundary. 434 IOAM-Namespaces support several different uses: 436 o IOAM-Namespaces can be used by an operator to distinguish 437 different operational domains. Devices at domain edges can filter 438 on Namespace-IDs to provide for proper IOAM-Domain isolation. 440 o IOAM-Namespaces provide additional context for IOAM-Data-Fields 441 and thus can be used to ensure that IOAM-Data-Fields are unique 442 and are interpreted properly by management stations or network 443 controllers. For example, the node identifier field (node_id, see 444 below) does not need to be unique in a deployment (e.g., if an 445 operator wishes to use different node identifiers for different 446 IOAM layers, even within the same device; or node identifiers 447 might not be unique for other organizational reasons, such as 448 after a merger of two formerly separated organizations), the 449 Namespace-ID can be used as a context identifier, such that the 450 combination of node_id and Namespace-ID will always be unique. 451 Similarly, IOAM-Namespaces can be used to define how certain IOAM- 452 Data-Fields are interpreted: IOAM offers three different timestamp 453 format options. The Namespace-ID can be used to determine the 454 timestamp format. IOAM-Data-Fields (e.g., buffer occupancy) which 455 do not have a unit associated are to be interpreted within the 456 context of a IOAM-Namespace. 458 o IOAM-Namespaces can be used to identify different sets of devices 459 (e.g., different types of devices) in a deployment: If an operator 460 desires to insert different IOAM-Data-Fields based on the device, 461 the devices could be grouped into multiple IOAM-Namespaces. This 462 could be due to the fact that the IOAM feature set differs between 463 different sets of devices, or it could be for reasons of optimized 464 space usage in the packet header. It could also stem from 465 hardware or operational limitations on the size of the trace data 466 that can be added and processed, preventing collection of a full 467 trace for a flow. 469 * Assigning different IOAM Namespace-IDs to different sets of 470 nodes or network partitions and using the Namespace-ID as a 471 selector at the IOAM encapsulating node, a full trace for a 472 flow could be collected and constructed via partial traces in 473 different packets of the same flow. Example: An operator could 474 choose to group the devices of a domain into two IOAM- 475 Namespaces, in a way that on average, only every second hop 476 would be recorded by any device. To retrieve a full view of 477 the deployment, the captured IOAM-Data-Fields of the two IOAM- 478 Namespaces need to be correlated. 480 * Assigning different IOAM Namespace-IDs to different sets of 481 nodes or network partitions and using a separate instance of an 482 IOAM-Option-Type for each Namespace-ID, a full trace for a flow 483 could be collected and constructed via partial traces from each 484 IOAM-Option-Type in each of the packets in the flow. Example: 485 An operator could choose to group the devices of a domain into 486 two IOAM-Namespaces, in a way that each IOAM-Namespace is 487 represented by one of two IOAM-Option-Types in the packet. 488 Each node would record data only for the IOAM-Namespace that it 489 belongs to, ignoring the other IOAM-Option-Type with a IOAM- 490 Namespace to which it doesn't belong. To retrieve a full view 491 of the deployment, the captured IOAM-Data-Fields of the two 492 IOAM-Namespaces need to be correlated. 494 5.4. IOAM Trace Option-Types 496 "IOAM tracing data" is expected to be collected at every IOAM transit 497 node that a packet traverses to ensure visibility into the entire 498 path a packet takes within an IOAM-Domain. I.e., in a typical 499 deployment, all nodes in an IOAM-Domain would participate in IOAM and 500 thus be IOAM transit nodes, IOAM encapsulating or IOAM decapsulating 501 nodes. If not all nodes within a domain support IOAM functionality 502 as defined in this document, IOAM tracing information (i.e., node 503 data, see below) will only be collected on those nodes which support 504 IOAM functionality as defined in this document. Nodes which do not 505 support IOAM functionality as defined in this document will forward 506 the packet without any changes to the IOAM-Data-Fields. The maximum 507 number of hops and the minimum path MTU of the IOAM domain is assumed 508 to be known. An overflow indicator (O-bit) is defined as one of the 509 ways to deal with situations where the PMTU was underestimated, i.e., 510 where the number of hops which are IOAM capable exceeds the available 511 space in the packet. 513 To optimize hardware and software implementations, IOAM tracing is 514 defined as two separate options. A deployment can choose to 515 configure and support one or both of the following options. 517 Pre-allocated Trace-Option: This trace option is defined as a 518 container of node data fields (see below) with pre-allocated space 519 for each node to populate its information. This option is useful 520 for implementations where it is efficient to allocate the space 521 once and index into the array to populate the data during transit 522 (e.g., software forwarders often fall into this class). The IOAM 523 encapsulating node allocates space for Pre-allocated Trace Option- 524 Type in the packet and sets corresponding fields in this IOAM- 525 Option-Type. The IOAM encapsulating node allocates an array which 526 is used to store operational data retrieved from every node while 527 the packet traverses the domain. IOAM transit nodes update the 528 content of the array, and possibly update the checksums of outer 529 headers. A pointer which is part of the IOAM trace data, points 530 to the next empty slot in the array. An IOAM transit node that 531 updates the content of the pre-allocated option also updates the 532 value of the pointer, which specifies where the next IOAM transit 533 node fills in its data. The "node data list" array (see below) in 534 the packet is populated iteratively as the packet traverses the 535 network, starting with the last entry of the array, i.e., "node 536 data list [n]" is the first entry to be populated, "node data list 537 [n-1]" is the second one, etc. 539 Incremental Trace-Option: This trace option is defined as a 540 container of node data fields where each node allocates and pushes 541 its node data immediately following the option header. This type 542 of trace recording is useful for some of the hardware 543 implementations as it eliminates the need for the transit network 544 elements to read the full array in the option and allows for 545 arbitrarily long packets as the MTU allows. The IOAM 546 encapsulating node allocates space for the Incremental Trace 547 Option-Type. Based on operational state and configuration, the 548 IOAM encapsulating node sets the fields in the Option-Type that 549 control what IOAM-Data-Fields have to be collected and how large 550 the node data list can grow. IOAM transit nodes push their node 551 data to the node data list subject to any protocol constraints of 552 the encapsulating layer. They then decrease the remaining length 553 available to subsequent nodes and adjust the lengths and possibly 554 checksums in outer headers. 556 IOAM encapsulating nodes and IOAM decapsulating nodes which support 557 tracing MUST support both Trace-Option-Types. For IOAM transit nodes 558 it is sufficient to support one of the Trace-Option-Types. In the 559 event that both options are utilized in a deployment at the same 560 time, the Incremental Trace-Option MUST be placed before the Pre- 561 allocated Trace-Option. Deployments which mix devices with either 562 the Incremental Trace-Option or the Pre-allocated Trace-Option could 563 result in both Option-Types being present in a packet. Given that 564 the operator knows which equipment is deployed in a particular IOAM 565 domain, the operator will decide by means of configuration which 566 type(s) of trace options will be used for a particular domain. 568 Every node data entry holds information for a particular IOAM transit 569 node that is traversed by a packet. The IOAM decapsulating node 570 removes the IOAM-Option-Type(s) and processes and/or exports the 571 associated data. Like all IOAM-Data-Fields, the IOAM-Data-Fields of 572 the IOAM-Trace-Option-Types are defined in the context of an IOAM- 573 Namespace. 575 IOAM tracing can collect the following types of information: 577 o Identification of the IOAM node. An IOAM node identifier can 578 match to a device identifier or a particular control point or 579 subsystem within a device. 581 o Identification of the interface that a packet was received on, 582 i.e., ingress interface. 584 o Identification of the interface that a packet was sent out on, 585 i.e., egress interface. 587 o Time of day when the packet was processed by the node as well as 588 the transit delay. Different definitions of processing time are 589 feasible and expected, though it is important that all devices of 590 an in-situ OAM domain follow the same definition. 592 o Generic data: Format-free information where syntax and semantic of 593 the information is defined by the operator in a specific 594 deployment. For a specific IOAM-Namespace, all IOAM nodes have to 595 interpret the generic data the same way. Examples for generic 596 IOAM data include geo-location information (location of the node 597 at the time the packet was processed), buffer queue fill level or 598 cache fill level at the time the packet was processed, or even a 599 battery charge level. 601 o Information to detect whether IOAM trace data was added at every 602 hop or whether certain hops in the domain weren't IOAM transit 603 nodes. 605 It should be noted that the semantics of some of the node data fields 606 that are defined below, such as the queue depth and buffer occupancy, 607 are implementation specific. This approach is intended to allow IOAM 608 nodes with various different architectures. 610 5.4.1. Pre-allocated and Incremental Trace Option-Types 612 The IOAM Pre-allocated Trace-Option and the IOAM Incremental Trace- 613 Option have similar formats. Except where noted below, the internal 614 formats and fields of the two trace options are identical. Both 615 Trace-Options consist of a fixed size "trace option header" and a 616 variable data space to store gathered data, the "node data list". An 617 IOAM transit node (that is not an IOAM encapsulating node or IOAM 618 decapsulating node) MUST NOT modify any of the fields in the fixed 619 size "trace option header", other than "flags" and "RemainingLen", 620 i.e., an IOAM transit node MUST NOT modify the Namespace-ID, NodeLen, 621 IOAM-Trace-Type, or Reserved fields. 623 Pre-allocated and incremental trace option headers: 625 0 1 2 3 626 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 627 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 628 | Namespace-ID |NodeLen | Flags | RemainingLen| 629 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 630 | IOAM-Trace-Type | Reserved | 631 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 633 The trace option data MUST be 4-octet aligned: 635 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+<-+ 636 | | | 637 | node data list [0] | | 638 | | | 639 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ D 640 | | a 641 | node data list [1] | t 642 | | a 643 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 644 ~ ... ~ S 645 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ p 646 | | a 647 | node data list [n-1] | c 648 | | e 649 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 650 | | | 651 | node data list [n] | | 652 | | | 653 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+<-+ 655 Namespace-ID: 16-bit identifier of an IOAM-Namespace. The 656 Namespace-ID value of 0x0000 is defined as the "Default-Namespace- 657 ID" (see Section 5.3) and MUST be known to all the nodes 658 implementing IOAM. For any other Namespace-ID value that does not 659 match any Namespace-ID the node is configured to operate on, the 660 node MUST NOT change the contents of the IOAM-Data-Fields. 662 NodeLen: 5-bit unsigned integer. This field specifies the length of 663 data added by each node in multiples of 4-octets, excluding the 664 length of the "Opaque State Snapshot" field. 666 If IOAM-Trace-Type bit 22 is not set, then NodeLen specifies the 667 actual length added by each node. If IOAM-Trace-Type bit 22 is 668 set, then the actual length added by a node would be (NodeLen + 669 length of the "Opaque State Snapshot" field) in 4 octet units. 671 For example, if 3 IOAM-Trace-Type bits are set and none of them 672 are in wide format, then NodeLen would be 3. If 3 IOAM-Trace-Type 673 bits are set and 2 of them are wide, then NodeLen would be 5. 675 An IOAM encapsulating node MUST set NodeLen. 677 A node receiving an IOAM Pre-allocated or Incremental Trace-Option 678 relies on the NodeLen value. 680 Flags 4-bit field. Flags are allocated by IANA, as specified in 681 Section 8.3. This document allocates a single flag as follows: 683 Bit 0 "Overflow" (O-bit) (most significant bit). In case a 684 network element is supposed to add node data to a packet, but 685 detects that there are not enough octets left to record the 686 node data, the network element MUST NOT add any fields and MUST 687 set the overflow "O-bit" to "1" in the IOAM-Trace-Option 688 header. This is useful for transit nodes to ignore further 689 processing of the option. 691 RemainingLen: 7-bit unsigned integer. This field specifies the data 692 space in multiples of 4-octets remaining for recording the node 693 data, before the node data list is considered to have overflowed. 694 The sender MUST assign the initial value of the RemainingLen 695 field. The sender MAY calculate the value of the RemainingLen 696 field by computing the number of node data bytes allowed before 697 exceeding the path MTU (PMTU), given that the PMTU is known to the 698 sender. Subsequent nodes can carry out a simple comparison 699 between RemainingLen and NodeLen, along with the length of the 700 "Opaque State Snapshot" if applicable, to determine whether or not 701 data can be added by this node. When node data is added, the node 702 MUST decrease RemainingLen by the amount of data added. In the 703 pre-allocated trace option, RemainingLen is used to derive the 704 offset in data space to record the node data element. 705 Specifically, the recording of the node data element would start 706 from RemainingLen - NodeLen - sizeof(opaque snapshot) in 4 octet 707 units. If RemainingLen in a pre-allocated trace option exceeds 708 the length of the option, as specified in the lower layer header 709 (which is not within the scope of this document), then the node 710 MUST NOT add any fields. 712 IOAM-Trace-Type: A 24-bit identifier which specifies which data 713 types are used in this node data list. 715 The IOAM-Trace-Type value is a bit field. The following bits are 716 defined in this document, with details on each bit described in 717 the Section 5.4.2. The order of packing the data fields in each 718 node data element follows the bit order of the IOAM-Trace-Type 719 field, as follows: 721 Bit 0 (Most significant bit) When set, indicates presence of 722 Hop_Lim and node_id (short format) in the node data. 724 Bit 1 When set, indicates presence of ingress_if_id and 725 egress_if_id (short format) in the node data. 727 Bit 2 When set, indicates presence of timestamp seconds in the 728 node data. 730 Bit 3 When set, indicates presence of timestamp fraction in the 731 node data. 733 Bit 4 When set, indicates presence of transit delay in the node 734 data. 736 Bit 5 When set, indicates presence of IOAM-Namespace specific 737 data (short format) in the node data. 739 Bit 6 When set, indicates presence of queue depth in the node 740 data. 742 Bit 7 When set, indicates presence of the Checksum Complement 743 node data. 745 Bit 8 When set, indicates presence of Hop_Lim and node_id in 746 wide format in the node data. 748 Bit 9 When set, indicates presence of ingress_if_id and 749 egress_if_id in wide format in the node data. 751 Bit 10 When set, indicates presence of IOAM-Namespace specific 752 data in wide format in the node data. 754 Bit 11 When set, indicates presence of buffer occupancy in the 755 node data. 757 Bit 12-21 Undefined. These values are available for future 758 assignment in the IOAM Trace-Type Registry (Section 8.2). 759 Every future node data field corresponding to one of 760 these bits MUST be 4-octets long. An IOAM encapsulating 761 node MUST set the value of each undefined bit to 0. If 762 an IOAM transit node receives a packet with one or more 763 of these bits set to 1, it MUST either: 765 1. Add corresponding node data filled with the reserved 766 value 0xFFFFFFFF, after the node data fields for the 767 IOAM-Trace-Type bits defined above, such that the 768 total node data added by this node in units of 769 4-octets is equal to NodeLen, or 771 2. Not add any node data fields to the packet, even for 772 the IOAM-Trace-Type bits defined above. 774 Bit 22 When set, indicates presence of variable length Opaque 775 State Snapshot field. 777 Bit 23 Reserved: MUST be set to zero upon transmission and 778 ignored upon receipt. This bit is reserved to allow for 779 future extensions of the IOAM-Trace-Type bit field. 781 Section 5.4.2 describes the IOAM-Data-Types and their formats. 782 Within an IOAM-Domain possible combinations of these bits making 783 the IOAM-Trace-Type can be restricted by configuration knobs. 785 Reserved: 8-bits. An IOAM encapsulating node MUST set the value to 786 zero upon transmission. IOAM transit nodes MUST ignore the 787 received value. 789 Node data List [n]: Variable-length field. This is a list of node 790 data elements where the content of each node data element is 791 determined by the IOAM-Trace-Type. The order of packing the data 792 fields in each node data element follows the bit order of the 793 IOAM-Trace-Type field. Each node MUST prepend its node data 794 element in front of the node data elements that it received, such 795 that the transmitted node data list begins with this node's data 796 element as the first populated element in the list. The last node 797 data element in this list is the node data of the first IOAM 798 capable node in the path. Populating the node data list in this 799 way ensures that the order of node data list is the same for 800 incremental and pre-allocated trace options. In the pre-allocated 801 trace option, the index contained in RemainingLen identifies the 802 offset for current active node data to be populated. 804 5.4.2. IOAM node data fields and associated formats 806 All the IOAM-Data-Fields MUST be 4-octet aligned. If a node which is 807 supposed to update an IOAM-Data-Field is not capable of populating 808 the value of a field set in the IOAM-Trace-Type, the field value MUST 809 be set to 0xFFFFFFFF for 4-octet fields or 0xFFFFFFFFFFFFFFFF for 810 8-octet fields, indicating that the value is not populated, except 811 when explicitly specified in the field description below. 813 Some IOAM-Data-Fields defined below, such as interface identifiers or 814 IOAM-Namespace specific data, are defined in both "short format" as 815 well as "wide format". "Short format" refers to an IOAM-Data-Field 816 which comprises 4 octets. "Wide format" refers to an IOAM-Data-Field 817 which comprises 8 octets. The use of "short format" or "wide format" 818 is not mutually exclusive. A deployment could choose to leverage 819 both. For example, ingress_if_id_(short format) could be an 820 identifier for the physical interface, whereas ingress_if_id_(wide 821 format) could be an identifier for a logical sub-interface of that 822 physical interface. 824 Data fields and associated data types for each of the IOAM-Data- 825 Fields are specified in the following sections. The definition of 826 IOAM-Data-Fields focuses on the syntax of the data-fields and avoids 827 specifying the semantics where feasible. This is why no units are 828 defined for data-fields like e.g., "buffer occupancy" or "queue 829 depth". With this approach, nodes can supply the information in 830 their native format and are not required to perform unit or format 831 conversions. Systems that further process IOAM information, like 832 e.g., a network management system are assumed to also handle unit 833 conversions as part of their IOAM data-fields processing. The 834 combination of a particular data-field and the namespace-id provides 835 for the context to interpret the provided data appropriately. 837 5.4.2.1. Hop_Lim and node_id short format 839 The "Hop_Lim and node_id short format" field is a 4-octet field that 840 is defined as follows: 842 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 843 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 844 | Hop_Lim | node_id | 845 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 847 Hop_Lim: 1-octet unsigned integer. It is set to the Hop Limit value 848 in the packet at egress from the node that records this data. Hop 849 Limit information is used to identify the location of the node in 850 the communication path. This is copied from the lower layer, 851 e.g., TTL value in IPv4 header or hop limit field from IPv6 header 852 of the packet when the packet is ready for transmission. The 853 semantics of the Hop_Lim field depend on the lower layer protocol 854 that IOAM is encapsulated into, and therefore its specific 855 semantics are outside the scope of this memo. The value of this 856 field MUST be set to 0xff when the lower level does not have a 857 TTL/Hop limit equivalent field. 859 node_id: 3-octet unsigned integer. Node identifier field to 860 uniquely identify a node within the IOAM-Namespace and associated 861 IOAM-Domain. The procedure to allocate, manage and map the 862 node_ids is beyond the scope of this document. See 863 [I-D.brockners-opsawg-ioam-deployment] for a discussion of 864 deployment related aspects of the node_id. 866 5.4.2.2. ingress_if_id and egress_if_id 868 The "ingress_if_id and egress_if_id" field is a 4-octet field that is 869 defined as follows: 871 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 872 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 873 | ingress_if_id | egress_if_id | 874 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 876 ingress_if_id: 2-octet unsigned integer. Interface identifier to 877 record the ingress interface the packet was received on. 879 egress_if_id: 2-octet unsigned integer. Interface identifier to 880 record the egress interface the packet is forwarded out of. 882 Note that due to the fact that IOAM uses its own IOAM-Namespaces for 883 IOAM-Data-Fields, data fields like interface identifiers can be used 884 in a flexible way to represent system resources that are associated 885 with ingressing or egressing packets, i.e., ingress_if_id could 886 represent a physical interface, a virtual or logical interface, or 887 even a queue. 889 5.4.2.3. timestamp seconds 891 The "timestamp seconds" field is a 4-octet unsigned integer field. 892 It contains the absolute timestamp in seconds that specifies the time 893 at which the packet was received by the node. This field has three 894 possible formats; based on either PTP (see e.g., [RFC8877]), NTP 895 [RFC5905], or POSIX [POSIX]. The three timestamp formats are 896 specified in Section 6. In all three cases, the Timestamp Seconds 897 field contains the 32 most significant bits of the timestamp format 898 that is specified in Section 6. If a node is not capable of 899 populating this field, it assigns the value 0xFFFFFFFF. Note that 900 this is a legitimate value that is valid for 1 second in 901 approximately 136 years; the analyzer has to correlate several 902 packets or compare the timestamp value to its own time-of-day in 903 order to detect the error indication. 905 5.4.2.4. timestamp fraction 907 The "timestamp fraction" field is a 4-octet unsigned integer field. 908 Fraction specifies the fractional portion of the number of seconds 909 since the NTP epoch [RFC8877]. The field specifies the time at which 910 the packet was received by the node. This field has three possible 911 formats; based on either PTP (see e.g., [RFC8877]), NTP [RFC5905], or 912 POSIX [POSIX]. The three timestamp formats are specified in 913 Section 6. In all three cases, the Timestamp fraction field contains 914 the 32 least significant bits of the timestamp format that is 915 specified in Section 6. If a node is not capable of populating this 916 field, it assigns the value 0xFFFFFFFF. Note that this is a 917 legitimate value in the NTP format, valid for approximately 233 918 picoseconds in every second. If the NTP format is used the analyzer 919 has to correlate several packets in order to detect the error 920 indication. 922 5.4.2.5. transit delay 924 The "transit delay" field is a 4-octet unsigned integer in the range 925 0 to 2^31-1. It is the time in nanoseconds the packet spent in the 926 transit node. This can serve as an indication of the queuing delay 927 at the node. If the transit delay exceeds 2^31-1 nanoseconds then 928 the top bit 'O' is set to indicate overflow and value set to 929 0x80000000. When this field is part of the data field but a node 930 populating the field is not able to fill it, the field position in 931 the field MUST be filled with value 0xFFFFFFFF to mean not populated. 933 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 934 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 935 |O| transit delay | 936 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 938 5.4.2.6. namespace specific data 940 The "namespace specific data" field is a 4-octet field which can be 941 used by the node to add IOAM-Namespace specific data. This 942 represents a "free-format" 4-octet bit field with its semantics 943 defined in the context of a specific IOAM-Namespace. 945 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 946 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 947 | namespace specific data | 948 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 950 5.4.2.7. queue depth 952 The "queue depth" field is a 4-octet unsigned integer field. This 953 field indicates the current length of the egress interface queue of 954 the interface from where the packet is forwarded out. The queue 955 depth is expressed as the current amount of memory buffers used by 956 the queue (a packet could consume one or more memory buffers, 957 depending on its size). 959 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 960 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 961 | queue depth | 962 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 964 5.4.2.8. Checksum Complement 966 The "Checksum Complement" field is a 4-octet node data which contains 967 a 4-octet Checksum Complement field. The Checksum Complement is 968 useful when IOAM is transported over encapsulations that make use of 969 a UDP transport, such as VXLAN-GPE or Geneve. Without the Checksum 970 Complement, nodes adding IOAM node data update the UDP Checksum field 971 following the recommendation of the encapsulation protocols. When 972 the Checksum Complement is present, an IOAM encapsulating node or 973 IOAM transit node adding node data MUST carry out one of the 974 following two alternatives in order to maintain the correctness of 975 the UDP Checksum value: 977 1. Recompute the UDP Checksum field. 979 2. Use the Checksum Complement to make a checksum-neutral update in 980 the UDP payload; the Checksum Complement is assigned a value that 981 complements the rest of the node data fields that were added by 982 the current node, causing the existing UDP Checksum field to 983 remain correct. 985 IOAM decapsulating nodes MUST recompute the UDP Checksum field, since 986 they do not know whether previous hops modified the UDP Checksum 987 field or the Checksum Complement field. 989 Checksum Complement fields are used in a similar manner in [RFC7820] 990 and [RFC7821]. 992 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 993 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 994 | Checksum Complement | 995 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 997 5.4.2.9. Hop_Lim and node_id wide 999 The "Hop_Lim and node_id wide" field is an 8-octet field defined as 1000 follows: 1002 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1003 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1004 | Hop_Lim | node_id ~ 1005 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1006 ~ node_id (contd) | 1007 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1009 Hop_Lim: 1-octet unsigned integer. See Section 5.4.2.1 for the 1010 definition of the field. 1012 node_id: 7-octet unsigned integer. Node identifier field to 1013 uniquely identify a node within the IOAM-Namespace and associated 1014 IOAM-Domain. The procedure to allocate, manage and map the 1015 node_ids is beyond the scope of this document. 1017 5.4.2.10. ingress_if_id and egress_if_id wide 1019 The "ingress_if_id and egress_if_id wide" field is an 8-octet field 1020 which is defined as follows: 1022 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1023 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1024 | ingress_if_id | 1025 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1026 | egress_if_id | 1027 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1029 ingress_if_id: 4-octet unsigned integer. Interface identifier to 1030 record the ingress interface the packet was received on. 1032 egress_if_id: 4-octet unsigned integer. Interface identifier to 1033 record the egress interface the packet is forwarded out of. 1035 5.4.2.11. namespace specific data wide 1037 The "namespace specific data wide" field is an 8-octet field which 1038 can be used by the node to add IOAM-Namespace specific data. This 1039 represents a "free-format" 8-octet bit field with its semantics 1040 defined in the context of a specific IOAM-Namespace. 1042 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1043 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1044 | namespace specific data ~ 1045 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1046 ~ namespace specific data (contd) | 1047 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1049 5.4.2.12. buffer occupancy 1051 The "buffer occupancy" field is a 4-octet unsigned integer field. 1052 This field indicates the current status of the occupancy of the 1053 common buffer pool used by a set of queues. The units of this field 1054 are implementation specific. Hence, the units are interpreted within 1055 the context of an IOAM-Namespace and/or node-id if used. The authors 1056 acknowledge that in some operational cases there is a need for the 1057 units to be consistent across a packet path through the network, 1058 hence it is recommended for implementations to use standard units 1059 such as Bytes. 1061 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1062 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1063 | buffer occupancy | 1064 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1066 5.4.2.13. Opaque State Snapshot 1068 The "Opaque State Snapshot" is a variable length field and follows 1069 the fixed length IOAM-Data-Fields defined above. It allows the 1070 network element to store an arbitrary state in the node data field, 1071 without a pre-defined schema. The schema is to be defined within the 1072 context of an IOAM-Namespace. The schema needs to be made known to 1073 the analyzer by some out-of-band mechanism. The specification of 1074 this mechanism is beyond the scope of this document. A 24-bit 1075 "Schema Id" field, interpreted within the context of an IOAM- 1076 Namespace, indicates which particular schema is used, and has to be 1077 configured on the network element by the operator. 1079 0 1 2 3 1080 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1081 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1082 | Length | Schema ID | 1083 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1084 | | 1085 | | 1086 | Opaque data | 1087 ~ ~ 1088 . . 1089 . . 1090 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1092 Length: 1-octet unsigned integer. It is the length in multiples of 1093 4-octets of the Opaque data field that follows Schema Id. 1095 Schema ID: 3-octet unsigned integer identifying the schema of Opaque 1096 data. 1098 Opaque data: Variable length field. This field is interpreted as 1099 specified by the schema identified by the Schema ID. 1101 When this field is part of the data field but a node populating the 1102 field has no opaque state data to report, the Length MUST be set to 0 1103 and the Schema ID MUST be set to 0xFFFFFF to mean no schema. 1105 5.4.3. Examples of IOAM node data 1107 The format used for the entries in a packet's "node data list" array 1108 can vary from packet to packet and deployment to deployment". Some 1109 deployments might only be interested in recording the node 1110 identifiers, whereas others might be interested in recording node 1111 identifiers and timestamps. This section provides example entries of 1112 the "node data list". 1114 0xD40000: IOAM-Trace-Type is 0xD40000 (0b110101000000000000000000) 1115 then the format of node data is: 1117 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1118 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1119 | Hop_Lim | node_id | 1120 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1121 | ingress_if_id | egress_if_id | 1122 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1123 | timestamp fraction | 1124 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1125 | namespace specific data | 1126 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1128 0xC00000: IOAM-Trace-Type is 0xC00000 (0b110000000000000000000000) 1129 then the format is: 1131 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1132 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1133 | Hop_Lim | node_id | 1134 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1135 | ingress_if_id | egress_if_id | 1136 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1138 0x900000: IOAM-Trace-Type is 0x900000 (0b100100000000000000000000) 1139 then the format is: 1141 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1142 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1143 | Hop_Lim | node_id | 1144 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1145 | timestamp fraction | 1146 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1148 0x840000: IOAM-Trace-Type is 0x840000 (0b100001000000000000000000) 1149 then the format is: 1151 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1152 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1153 | Hop_Lim | node_id | 1154 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1155 | namespace specific data | 1156 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1158 0x940000: IOAM-Trace-Type is 0x940000 (0b100101000000000000000000) 1159 then the format is: 1161 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1162 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1163 | Hop_Lim | node_id | 1164 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1165 | timestamp fraction | 1166 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1167 | namespace specific data | 1168 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1170 0x308002: IOAM-Trace-Type is 0x308002 (0b001100001000000000000010) 1171 then the format is: 1173 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1174 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1175 | timestamp seconds | 1176 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1177 | timestamp fraction | 1178 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1179 | Hop_Lim | node_id | 1180 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1181 | node_id(contd) | 1182 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1183 | Length | Schema Id | 1184 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1185 | | 1186 | | 1187 | Opaque data | 1188 ~ ~ 1189 . . 1190 . . 1191 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1193 5.5. IOAM Proof of Transit Option-Type 1195 IOAM Proof of Transit Option-Type is used to support path or service 1196 function chain [RFC7665] verification use cases. For further 1197 information on Proof-of-transit, please refer to 1198 [I-D.ietf-sfc-proof-of-transit]. While details on how the IOAM data 1199 for the Proof-of-transit option is processed at IOAM encapsulating, 1200 decapsulating and transit nodes are outside the scope of the 1201 document, all of these approaches share the need to uniquely identify 1202 a packet as well as iteratively operate on a set of information that 1203 is handed from node to node. Correspondingly, two pieces of 1204 information are added as IOAM-Data-Fields to the packet: 1206 o PktID: Unique identifier for the packet. 1208 o Cumulative: Information which is handed from node to node and 1209 updated by every node according to a verification algorithm. 1211 The IOAM Proof-of-Transit Option-Type consist of a fixed size "IOAM 1212 proof of transit option header" and "IOAM proof of transit option 1213 data fields": 1215 IOAM proof of transit option header: 1217 0 1 2 3 1218 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1219 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1220 | Namespace-ID |IOAM POT Type | IOAM POT flags| 1221 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1223 IOAM proof of transit Option-Type IOAM-Data-Fields MUST be 1224 4-octet aligned: 1226 0 1 2 3 1227 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1228 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1229 | POT Option data field determined by IOAM-POT-Type | 1230 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1232 Namespace-ID: 16-bit identifier of an IOAM-Namespace. The 1233 Namespace-ID value of 0x0000 is defined as the "Default-Namespace- 1234 ID" (see Section 5.3) and MUST be known to all the nodes 1235 implementing IOAM. For any other Namespace-ID value that does not 1236 match any Namespace-ID the node is configured to operate on, the 1237 node MUST NOT change the contents of the IOAM-Data-Fields. 1239 IOAM POT Type: 8-bit identifier of a particular POT variant that 1240 specifies the POT data that is included. This document defines 1241 POT Type 0: 1243 0: POT data is a 16 Octet field to carry data associated to POT 1244 procedures. [I-D.ietf-sfc-proof-of-transit] describes an 1245 implementation of POT and provides details on the data carried 1246 in POT data. 1248 If a node receives an IOAM POT Type value that it does not 1249 understand, the node MUST NOT change, add to, or remove the 1250 contents of the OAM-Data-Fields. 1252 IOAM POT flags: 8-bit. Following flags are defined: 1254 Bit 0 "Profile-to-use" (P-bit) (most significant bit). For IOAM 1255 POT types that use a maximum of two profiles to drive 1256 computation, indicates which POT-profile (see 1257 [I-D.ietf-sfc-proof-of-transit] for details) is used. The two 1258 profiles are numbered 0, 1. This bit conveys whether profile 0 1259 or profile 1 is used to compute the Cumulative. 1261 Bit 1-7 Undefined: These bits are available for assignment, see 1262 Section 8.5. Bits which have not been assigned MUST be set to 1263 zero upon transmission and ignored upon receipt. 1265 POT Option data: Variable-length field. The type of which is 1266 determined by the IOAM-POT-Type. 1268 5.5.1. IOAM Proof of Transit Type 0 1270 IOAM proof of transit option of IOAM POT Type 0: 1272 0 1 2 3 1273 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1274 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1275 | Namespace-ID |IOAM POT Type=0|P|R R R R R R R| 1276 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+<-+ 1277 | PktID | | 1278 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ P 1279 | PktID (contd) | O 1280 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ T 1281 | Cumulative | | 1282 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 1283 | Cumulative (contd) | | 1284 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+<-+ 1286 Namespace-ID: 16-bit identifier of an IOAM-Namespace (see 1287 Section 5.5 above). 1289 IOAM POT Type: 8-bit identifier of a particular POT variant that 1290 specifies the POT data that is included (see Section 5.5 above). 1291 For this case here, IOAM POT Type is set to the value 0. 1293 Bit 0: 1-bit. "Profile-to-use" (P-bit) (most significant bit), see 1294 Section 5.5 above. 1296 Bit 1-7: Undefined (see Section 5.5 above). 1298 PktID: 64-bit packet identifier. 1300 Cumulative: 64-bit Cumulative that is updated at specific nodes by 1301 processing per packet PktID field and configured parameters. 1303 Note: Larger or smaller sizes of "PktID" and "Cumulative" data are 1304 feasible and could be required for certain deployments, e.g., in case 1305 of space constraints in the encapsulation protocols used. Future 1306 documents could introduce different sizes of data for "proof of 1307 transit". 1309 5.6. IOAM Edge-to-Edge Option-Type 1311 The IOAM Edge-to-Edge Option-Type is to carry data that is added by 1312 the IOAM encapsulating node and interpreted by IOAM decapsulating 1313 node. The IOAM transit nodes MAY process the data but MUST NOT 1314 modify it. 1316 The IOAM Edge-to-Edge Option-Type consist of a fixed size "IOAM Edge- 1317 to-Edge Option-Type header" and "IOAM Edge-to-Edge Option-Type data 1318 fields": 1320 IOAM Edge-to-Edge Option-Type header: 1322 0 1 2 3 1323 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1324 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1325 | Namespace-ID | IOAM-E2E-Type | 1326 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1328 IOAM Edge-to-Edge Option-Type IOAM-Data-Fields MUST 1329 be 4-octet aligned: 1331 0 1 2 3 1332 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1333 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1334 | E2E Option data field determined by IOAM-E2E-Type | 1335 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1337 Namespace-ID: 16-bit identifier of an IOAM-Namespace. The 1338 Namespace-ID value of 0x0000 is defined as the "Default-Namespace- 1339 ID" (see Section 5.3) and MUST be known to all the nodes 1340 implementing IOAM. For any other Namespace-ID value that does not 1341 match any Namespace-ID the node is configured to operate on, then 1342 the node MUST NOT change the contents of the IOAM-Data-Fields. 1344 IOAM-E2E-Type: A 16-bit identifier which specifies which data types 1345 are used in the E2E option data. The IOAM-E2E-Type value is a bit 1346 field. The order of packing the E2E option data field elements 1347 follows the bit order of the IOAM-E2E-Type field, as follows: 1349 Bit 0 (Most significant bit) When set indicates presence of a 1350 64-bit sequence number added to a specific "packet group" 1351 which is used to detect packet loss, packet reordering, 1352 or packet duplication within the group. The "packet 1353 group" is deployment dependent and defined at the IOAM 1354 encapsulating node, e.g., by n-tuple based classification 1355 of packets. When this bit is set, "Bit 1" (for 32-bit 1356 sequence number, see below) MUST be zero. 1358 Bit 1 When set indicates presence of a 32-bit sequence number 1359 added to a specific "packet group" which is used to 1360 detect packet loss, packet reordering, or packet 1361 duplication within that group. The "packet group" is 1362 deployment dependent and defined at the IOAM 1363 encapsulating node, e.g., by n-tuple based classification 1364 of packets. When this bit is set, "Bit 0" (for 64-bit 1365 sequence number, see above) MUST be zero. 1367 Bit 2 When set indicates presence of timestamp seconds, 1368 representing the time at which the packet entered the 1369 IOAM domain. Within the IOAM encapsulating node, the 1370 time that the timestamp is retrieved can depend on the 1371 implementation. Some possibilities are: 1) the time at 1372 which the packet was received by the node, 2) the time at 1373 which the packet was transmitted by the node, 3) when a 1374 tunnel encapsulation is used, the point at which the 1375 packet is encapsulated into the tunnel. Each 1376 implementation has to document when the E2E timestamp 1377 that is going to be put in the packet is retrieved. This 1378 4-octet field has three possible formats; based on either 1379 PTP (see e.g., [RFC8877]), NTP [RFC5905], or POSIX 1380 [POSIX]. The three timestamp formats are specified in 1381 Section 6. In all three cases, the Timestamp Seconds 1382 field contains the 32 most significant bits of the 1383 timestamp format that is specified in Section 6. If a 1384 node is not capable of populating this field, it assigns 1385 the value 0xFFFFFFFF. Note that this is a legitimate 1386 value that is valid for 1 second in approximately 136 1387 years; the analyzer has to correlate several packets or 1388 compare the timestamp value to its own time-of-day in 1389 order to detect the error indication. 1391 Bit 3 When set indicates presence of timestamp fraction, 1392 representing the time at which the packet entered the 1393 IOAM domain. This 4-octet field has three possible 1394 formats; based on either PTP (see e.g., [RFC8877]), NTP 1395 [RFC5905], or POSIX [POSIX]. The three timestamp formats 1396 are specified in Section 6. In all three cases, the 1397 Timestamp fraction field contains the 32 least 1398 significant bits of the timestamp format that is 1399 specified in Section 6. If a node is not capable of 1400 populating this field, it assigns the value 0xFFFFFFFF. 1401 Note that this is a legitimate value in the NTP format, 1402 valid for approximately 233 picoseconds in every second. 1403 If the NTP format is used the analyzer has to correlate 1404 several packets in order to detect the error indication. 1406 Bit 4-15 Undefined. An IOAM encapsulating node MUST set the value 1407 of these bits to zero upon transmission and ignore upon 1408 receipt. 1410 E2E Option data: Variable-length field. The type of which is 1411 determined by the IOAM-E2E-Type. 1413 6. Timestamp Formats 1415 The IOAM-Data-Fields include a timestamp field which is represented 1416 in one of three possible timestamp formats. It is assumed that the 1417 management plane is responsible for determining which timestamp 1418 format is used. 1420 6.1. PTP Truncated Timestamp Format 1422 The Precision Time Protocol (PTP) uses an 80-bit timestamp format. 1423 The truncated timestamp format is a 64-bit field, which is the 64 1424 least significant bits of the 80-bit PTP timestamp. The PTP 1425 truncated format is specified in Section 4.3 of [RFC8877], and the 1426 details are presented below for the sake of completeness. 1428 0 1 2 3 1429 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1430 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1431 | Seconds | 1432 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1433 | Nanoseconds | 1434 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1436 Figure 1: PTP Truncated Timestamp Format 1438 Timestamp field format: 1440 Seconds: specifies the integer portion of the number of seconds 1441 since the PTP epoch. 1443 + Size: 32 bits. 1445 + Units: seconds. 1447 Nanoseconds: specifies the fractional portion of the number of 1448 seconds since the PTP epoch. 1450 + Size: 32 bits. 1452 + Units: nanoseconds. The value of this field is in the range 0 1453 to (10^9)-1. 1455 Epoch: 1457 PTP epoch. For details see e.g., [RFC8877]. 1459 Resolution: 1461 The resolution is 1 nanosecond. 1463 Wraparound: 1465 This time format wraps around every 2^32 seconds, which is roughly 1466 136 years. The next wraparound will occur in the year 2106. 1468 Synchronization Aspects: 1470 It is assumed that nodes that run this protocol are synchronized 1471 among themselves. Nodes MAY be synchronized to a global reference 1472 time. Note that if PTP is used for synchronization, the timestamp 1473 MAY be derived from the PTP-synchronized clock, allowing the 1474 timestamp to be measured with respect to the clock of an PTP 1475 Grandmaster clock. 1477 The PTP truncated timestamp format is not affected by leap 1478 seconds. 1480 6.2. NTP 64-bit Timestamp Format 1482 The Network Time Protocol (NTP) [RFC5905] timestamp format is 64 bits 1483 long. This format is specified in Section 4.2.1 of [RFC8877], and 1484 the details are presented below for the sake of completeness. 1486 0 1 2 3 1487 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1488 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1489 | Seconds | 1490 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1491 | Fraction | 1492 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1494 Figure 2: NTP [RFC5905] 64-bit Timestamp Format 1496 Timestamp field format: 1498 Seconds: specifies the integer portion of the number of seconds 1499 since the NTP epoch. 1501 + Size: 32 bits. 1503 + Units: seconds. 1505 Fraction: specifies the fractional portion of the number of 1506 seconds since the NTP epoch. 1508 + Size: 32 bits. 1510 + Units: the unit is 2^(-32) seconds, which is roughly equal to 1511 233 picoseconds. 1513 Epoch: 1515 NTP Epoch. For details see [RFC5905]. 1517 Resolution: 1519 The resolution is 2^(-32) seconds. 1521 Wraparound: 1523 This time format wraps around every 2^32 seconds, which is roughly 1524 136 years. The next wraparound will occur in the year 2036. 1526 Synchronization Aspects: 1528 Nodes that use this timestamp format will typically be 1529 synchronized to UTC using NTP [RFC5905]. Thus, the timestamp MAY 1530 be derived from the NTP-synchronized clock, allowing the timestamp 1531 to be measured with respect to the clock of an NTP server. 1533 The NTP timestamp format is affected by leap seconds; it 1534 represents the number of seconds since the epoch minus the number 1535 of leap seconds that have occurred since the epoch. The value of 1536 a timestamp during or slightly after a leap second could be 1537 temporarily inaccurate. 1539 6.3. POSIX-based Timestamp Format 1541 This timestamp format is based on the POSIX time format [POSIX]. The 1542 detailed specification of the timestamp format used in this document 1543 is presented below. 1545 0 1 2 3 1546 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1547 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1548 | Seconds | 1549 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1550 | Microseconds | 1551 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1553 Figure 3: POSIX-based Timestamp Format 1555 Timestamp field format: 1557 Seconds: specifies the integer portion of the number of seconds 1558 since the POSIX epoch. 1560 + Size: 32 bits. 1562 + Units: seconds. 1564 Microseconds: specifies the fractional portion of the number of 1565 seconds since the POSIX epoch. 1567 + Size: 32 bits. 1569 + Units: the unit is microseconds. The value of this field is in 1570 the range 0 to (10^6)-1. 1572 Epoch: 1574 POSIX epoch. For details, see [POSIX], appendix A.4.16. 1576 Resolution: 1578 The resolution is 1 microsecond. 1580 Wraparound: 1582 This time format wraps around every 2^32 seconds, which is roughly 1583 136 years. The next wraparound will occur in the year 2106. 1585 Synchronization Aspects: 1587 It is assumed that nodes that use this timestamp format run the 1588 Linux operating system, and hence use the POSIX time. In some 1589 cases nodes MAY be synchronized to UTC using a synchronization 1590 mechanism that is outside the scope of this document, such as NTP 1591 [RFC5905]. Thus, the timestamp MAY be derived from the NTP- 1592 synchronized clock, allowing the timestamp to be measured with 1593 respect to the clock of an NTP server. 1595 7. IOAM Data Export 1597 IOAM nodes collect information for packets traversing a domain that 1598 supports IOAM. IOAM decapsulating nodes as well as IOAM transit 1599 nodes can choose to retrieve IOAM information from the packet, 1600 process the information further and export the information using 1601 e.g., IPFIX. The mechanisms and associated data formats for 1602 exporting IOAM data is outside the scope of this document. 1604 Raw data export of IOAM data using IPFIX is discussed in 1605 [I-D.spiegel-ippm-ioam-rawexport]. 1607 8. IANA Considerations 1609 This document requests the following IANA Actions. 1611 IANA is requested to define a registry group named "In-Situ OAM 1612 (IOAM) Protocol Parameters". 1614 This group will include the following registries: 1616 IOAM Option-Type 1618 IOAM Trace-Type 1620 IOAM Trace-Flags 1622 IOAM POT-Type 1624 IOAM POT-Flags 1626 IOAM E2E-Type 1627 IOAM Namespace-ID 1629 The subsequent sub-sections detail the registries herein contained. 1631 8.1. IOAM Option-Type Registry 1633 This registry defines 128 code points for the IOAM Option-Type field 1634 for identifying IOAM Option-Types as explained in Section 5. The 1635 following code points are defined in this draft: 1637 0 IOAM Pre-allocated Trace Option-Type 1639 1 IOAM Incremental Trace Option-Type 1641 2 IOAM POT Option-Type 1643 3 IOAM E2E Option-Type 1645 4 - 127 are available for assignment via "IETF Review" process as per 1646 [RFC8126]. 1648 New registration requests MUST use the following template: 1650 Name: Name of the newly registered Option-Type. 1652 Code point: Desired value of the requested code point. 1654 Description: Brief description of the newly registered Option-Type. 1656 Reference: Reference to the document that defines the new Option- 1657 Type. 1659 8.2. IOAM Trace-Type Registry 1661 This registry defines code point for each bit in the 24-bit IOAM- 1662 Trace-Type field for Pre-allocated Trace-Option-Type and Incremental 1663 Trace-Option-Type defined in Section 5.4. The meaning of Bits 0 - 11 1664 is defined in this document in Paragraph 5 of Section 5.4.1: 1666 Bit 0 hop_Lim and node_id in short format 1668 Bit 1 ingress_if_id and egress_if_id in short format 1670 Bit 2 timestamp seconds 1672 Bit 3 timestamp fraction 1674 Bit 4 transit delay 1675 Bit 5 namespace specific data in short format 1677 Bit 6 queue depth 1679 Bit 7 checksum complement 1681 Bit 8 hop_Lim and node_id in wide format 1683 Bit 9 ingress_if_id and egress_if_id in wide format 1685 Bit 10 namespace specific data in wide format 1687 Bit 11 buffer occupancy 1689 Bit 22 variable length Opaque State Snapshot 1691 Bit 23 reserved 1693 The meaning for Bits 12 - 21 are available for assignment via "IETF 1694 Review" process as per [RFC8126]. 1696 New registration requests MUST use the following template: 1698 Bit: Desired bit to be allocated in the 24-bit IOAM Trace-Option- 1699 Type field for Pre-allocated Trace-Option-Type and Incremental 1700 Trace-Option-Type. 1702 Description: Brief description of the newly registered bit. 1704 Reference: Reference to the document that defines the new bit. 1706 8.3. IOAM Trace-Flags Registry 1708 This registry defines code points for each bit in the 4 bit flags for 1709 the Pre-allocated trace option and for the Incremental trace option 1710 defined in Section 5.4. The meaning of Bit 0 (the most significant 1711 bit) for trace flags is defined in this document in Paragraph 3 of 1712 Section 5.4.1: 1714 Bit 0 "Overflow" (O-bit) 1716 Bit 1 - 3 are available for assignment via "IETF Review" process as 1717 per [RFC8126]. 1719 New registration requests MUST use the following template: 1721 Bit: Desired bit to be allocated in the 8 bit flags field of the 1722 Pre-allocated Trace-Option-Type and for the Incremental Trace- 1723 Option-Type. 1725 Description: Brief description of the newly registered bit. 1727 Reference: Reference to the document that defines the new bit. 1729 8.4. IOAM POT-Type Registry 1731 This registry defines 256 code points to define IOAM POT Type for 1732 IOAM proof of transit option Section 5.5. The code point value 0 is 1733 defined in this document: 1735 0: 16 Octet POT data 1737 1 - 255 are available for assignment via "IETF Review" process as per 1738 [RFC8126]. 1740 New registration requests MUST use the following template: 1742 Name: Name of the newly registered POT-Type. 1744 Code point: Desired value of the requested code point. 1746 Description: Brief description of the newly registered POT-Type. 1748 Reference: Reference to the document that defines the new POT-Type. 1750 8.5. IOAM POT-Flags Registry 1752 This registry defines code points for each bit in the 8 bit flags for 1753 IOAM POT Option-Type defined in Section 5.5. The meaning of Bit 0 1754 for IOAM POT flags is defined in this document in Section 5.5: 1756 Bit 0 "Profile-to-use" (P-bit) 1758 The meaning for Bits 1 - 7 are available for assignment via "IETF 1759 Review" process as per [RFC8126]. 1761 New registration requests MUST use the following template: 1763 Bit: Desired bit to be allocated in the 8 bit flags field of the 1764 IOAM POT Option-Type. 1766 Description: Brief description of the newly registered bit. 1768 Reference: Reference to the document that defines the new bit. 1770 8.6. IOAM E2E-Type Registry 1772 This registry defines code points for each bit in the 16 bit IOAM- 1773 E2E-Type field for IOAM E2E option Section 5.6. The meaning of Bit 0 1774 - 3 are defined in this document: 1776 Bit 0 64-bit sequence number 1778 Bit 1 32-bit sequence number 1780 Bit 2 timestamp seconds 1782 Bit 3 timestamp fraction 1784 The meaning of Bits 4 - 15 are available for assignment via "IETF 1785 Review" process as per [RFC8126]. 1787 New registration requests MUST use the following template: 1789 Bit: Desired bit to be allocated in the 16 bit IOAM-E2E-Type field. 1791 Description: Brief description of the newly registered bit. 1793 Reference: Reference to the document that defines the new bit. 1795 8.7. IOAM Namespace-ID Registry 1797 IANA is requested to set up an "IOAM Namespace-ID Registry", 1798 containing 16-bit values and following the template for requests 1799 shown below. The meaning of 0x0000 is defined in this document. 1800 IANA is requested to reserve the values 0x0001 to 0x7FFF for private 1801 use (managed by operators), as specified in Section 5.3 of the 1802 current document. Registry entries for the values 0x8000 to 0xFFFF 1803 are to be assigned via the "Expert Review" policy defined in 1804 [RFC8126]. 1806 Upon receiving a new allocation request, a designated expert will 1807 perform the following: 1809 o Review whether the request is complete, i.e., the registration 1810 template has been filled in. The expert will send incomplete 1811 requests back to the requestor. 1813 o Check whether the request is neither a duplicate of nor 1814 conflicting with either an already existing allocation or a 1815 pending allocation. In case of duplicates or conflicts, the 1816 expert will ask the requestor to update the allocation request 1817 accordingly. 1819 o Solicit feedback from relevant working groups and communities to 1820 ensure that the new allocation request has been properly reviewed 1821 and that rough consensus on the request exists. At a minumum, the 1822 expert will solicit feedback from the IPPM working group in the 1823 IETF by posting the request to the ippm@ietf.org mailing list. 1824 The expert will allow for a 3-week review period on the mailing 1825 lists. If the feedback received from the relevant working groups 1826 and communities within the review period indicates rough consensus 1827 on the request, the expert will approve the request and ask IANA 1828 for allocating the new Namespace-ID. In case the expert senses a 1829 lack of consensus from the feedback received, the expert will ask 1830 the requestor to engage with the corresponding working groups and 1831 communities to further review and refine the request. 1833 It is intended that any allocation will be accompanied by a published 1834 RFC. In order to allow for the allocation of code points prior to 1835 the RFC being approved for publication, the designated expert can 1836 approve allocations once it seems clear that an RFC will be 1837 published. 1839 0x0000: default namespace (known to all IOAM nodes) 1841 0x0001 - 0x7FFF: reserved for private use 1843 0x8000 - 0xFFFF: unassigned 1845 New registration requests MUST use the following template: 1847 Name: Name of the newly registered Namespace-ID. 1849 Code point: Desired value of the requested Namespace-ID. 1851 Description: Brief description of the newly registered Namespace-ID. 1853 Reference: Reference to the document that defines the new Namespace- 1854 ID. 1856 Status of the registration: Status can be either "permanent" or 1857 "provisional". Namespace-ID registrations following a successful 1858 expert review will have the status "provisional". Once the RFC, 1859 which defines the new Namespace-ID is published, the status is 1860 changed to "permanent". 1862 9. Management and Deployment Considerations 1864 This document defines the structure and use of IOAM data fields. 1865 This document does not define the encapsulation of IOAM data fields 1866 into different protocols. Management and deployment aspects for IOAM 1867 have to be considered within the context of the protocol IOAM data 1868 fields are encapsulated into and as such, are out of scope for this 1869 document. For a discussion of IOAM deployment, please also refer to 1870 [I-D.brockners-opsawg-ioam-deployment], which outlines a framework 1871 for IOAM deployment and provides best current practices. 1873 10. Security Considerations 1875 As discussed in [RFC7276], a successful attack on an OAM protocol in 1876 general, and specifically on IOAM, can prevent the detection of 1877 failures or anomalies, or create a false illusion of nonexistent 1878 ones. In particular, these threats are applicable by compromising 1879 the integrity of IOAM data, either by maliciously modifying IOAM 1880 options in transit, or by injecting packets with maliciously 1881 generated IOAM options. All nodes in the path of a IOAM carrying 1882 packet can perform such an attack. 1884 The Proof of Transit Option-Type (see Section 5.5) is used for 1885 verifying the path of data packets. The security considerations of 1886 POT are further discussed in [I-D.ietf-sfc-proof-of-transit]. 1888 In case an attacker gains access to several nodes in a network and 1889 would be able to change the system software of these nodes, IOAM data 1890 fields could be misused and repurposed for a use different from what 1891 is specified in this document. One type of misuse is the 1892 implementation of a covert channel between network nodes. 1894 From a confidentiality perspective, although IOAM options are not 1895 expected to contain user data, they can be used for network 1896 reconnaissance, allowing attackers to collect information about 1897 network paths, performance, queue states, buffer occupancy and other 1898 information. Moreover, if IOAM data leaks from the IOAM domain it 1899 could enable reconnaissance beyond the scope of the IOAM domain. 1901 IOAM can be used as a means for implementing Denial of Service (DoS) 1902 attacks, or for amplifying them. For example, a malicious attacker 1903 can add an IOAM header to packets in order to consume the resources 1904 of network devices that take part in IOAM or entities that receive, 1905 collect or analyze the IOAM data. Another example is a packet length 1906 attack, in which an attacker pushes headers associated with IOAM 1907 Option-Types into data packets, causing these packets to be increased 1908 beyond the MTU size, resulting in fragmentation or in packet drops. 1909 In case POT is used, an attacker could corrupt the POT data fields in 1910 the packet, resulting in a verification failure of the POT data, even 1911 if the packet followed the correct path. 1913 Since IOAM options can include timestamps, if network devices use 1914 synchronization protocols then any attack on the time protocol 1916 [RFC7384] can compromise the integrity of the timestamp-related data 1917 fields. 1919 At the management plane, attacks can be set up by misconfiguring or 1920 by maliciously configuring IOAM-enabled nodes in a way that enables 1921 other attacks. IOAM configuration should only managed by authorized 1922 processes or users. 1924 Solutions to ensure the integrity of IOAM data fields are outside the 1925 scope of this document. [I-D.brockners-ippm-ioam-data-integrity] 1926 discusses several methods to ensure the integrity of IOAM data fields 1927 for those deployments that have a need to protect the integrity of 1928 IOAM data fields. 1930 The current document does not define a specific IOAM encapsulation. 1931 It has to be noted that some IOAM encapsulation types can introduce 1932 specific security considerations. A specification that defines an 1933 IOAM encapsulation is expected to address the respective 1934 encapsulation-specific security considerations. 1936 Notably, IOAM is expected to be deployed in limited domains, thus 1937 confining the potential attack vectors to within the limited domain. 1938 A limited administrative domain provides the operator with the means 1939 to select, monitor, and control the access of all the network 1940 devices, making these devices trusted by the operator. Indeed, in 1941 order to limit the scope of threats mentioned above to within the 1942 current limited domain the network operator is expected to enforce 1943 policies that prevent IOAM traffic from leaking outside of the IOAM 1944 domain, and prevent IOAM data from outside the domain to be processed 1945 and used within the domain. 1947 The security considerations of a system that deploys IOAM, much like 1948 any system, has to be reviewed on a per-deployment-scenario basis, 1949 based on a systems-specific threat analysis, which can lead to 1950 specific security solutions that are beyond the scope of the current 1951 document. Specifically, in an IOAM deployment that is not confined 1952 to a single LAN, but spans multiple inter-connected sites (for 1953 example, using an overlay network), the inter-site links can be 1954 secured (e.g., by IPsec) in order to avoid external threats. 1956 IOAM deployment considerations, including approaches to mitigate the 1957 above discussed threads and potential attacks are outside the scope 1958 of this document. IOAM deployment considerations are discussed in 1959 [I-D.brockners-opsawg-ioam-deployment]. 1961 11. Acknowledgements 1963 The authors would like to thank Eric Vyncke, Nalini Elkins, Srihari 1964 Raghavan, Ranganathan T S, Karthik Babu Harichandra Babu, Akshaya 1965 Nadahalli, LJ Wobker, Erik Nordmark, Vengada Prasad Govindan, Andrew 1966 Yourtchenko, Aviv Kfir, Tianran Zhou, Zhenbin (Robin) and Greg Mirsky 1967 for the comments and advice. 1969 This document leverages and builds on top of several concepts 1970 described in [I-D.kitamura-ipv6-record-route]. The authors would 1971 like to acknowledge the work done by the author Hiroshi Kitamura and 1972 people involved in writing it. 1974 The authors would like to gracefully acknowledge useful review and 1975 insightful comments received from Joe Clarke, Al Morton, Tom Herbert, 1976 Carlos Bernardos, Haoyu Song, Mickey Spiegel, Roman Danyliw, Benjamin 1977 Kaduk, Murray S. Kucherawy, Ian Swett, Martin Duke, Francesca 1978 Palombini, Lars Eggert, Alvaro Retana, Erik Kline, Robert Wilton, 1979 Zaheduzzaman Sarker, Dan Romascanu and Barak Gafni. 1981 12. References 1983 12.1. Normative References 1985 [POSIX] Institute of Electrical and Electronics Engineers, "IEEE 1986 Std 1003.1-2017 (Revision of IEEE Std 1003.1-2017) - IEEE 1987 Standard for Information Technology - Portable Operating 1988 System Interface (POSIX(TM) Base Specifications, Issue 1989 7)", IEEE Std 1003.1-2017, 2017, 1990 . 1993 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1994 Requirement Levels", BCP 14, RFC 2119, 1995 DOI 10.17487/RFC2119, March 1997, 1996 . 1998 [RFC5905] Mills, D., Martin, J., Ed., Burbank, J., and W. Kasch, 1999 "Network Time Protocol Version 4: Protocol and Algorithms 2000 Specification", RFC 5905, DOI 10.17487/RFC5905, June 2010, 2001 . 2003 [RFC8126] Cotton, M., Leiba, B., and T. Narten, "Guidelines for 2004 Writing an IANA Considerations Section in RFCs", BCP 26, 2005 RFC 8126, DOI 10.17487/RFC8126, June 2017, 2006 . 2008 [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2009 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, 2010 May 2017, . 2012 12.2. Informative References 2014 [I-D.brockners-ippm-ioam-data-integrity] 2015 Brockners, F., Bhandari, S., and T. Mizrahi, "Integrity of 2016 In-situ OAM Data Fields", draft-brockners-ippm-ioam-data- 2017 integrity-03 (work in progress), July 2021. 2019 [I-D.brockners-opsawg-ioam-deployment] 2020 Brockners, F., Bhandari, S., Bernier, D., and T. Mizrahi, 2021 "In-situ OAM Deployment", draft-brockners-opsawg-ioam- 2022 deployment-03 (work in progress), June 2021. 2024 [I-D.ietf-nvo3-vxlan-gpe] 2025 (Editor), F. M., (editor), L. K., and U. E. (editor), 2026 "Generic Protocol Extension for VXLAN (VXLAN-GPE)", draft- 2027 ietf-nvo3-vxlan-gpe-12 (work in progress), September 2021. 2029 [I-D.ietf-sfc-proof-of-transit] 2030 Brockners, F., Bhandari, S., Mizrahi, T., Dara, S., and S. 2031 Youell, "Proof of Transit", draft-ietf-sfc-proof-of- 2032 transit-08 (work in progress), November 2020. 2034 [I-D.kitamura-ipv6-record-route] 2035 Kitamura, H., "Record Route for IPv6 (PR6) Hop-by-Hop 2036 Option Extension", draft-kitamura-ipv6-record-route-00 2037 (work in progress), November 2000. 2039 [I-D.spiegel-ippm-ioam-rawexport] 2040 Spiegel, M., Brockners, F., Bhandari, S., and R. 2041 Sivakolundu, "In-situ OAM raw data export with IPFIX", 2042 draft-spiegel-ippm-ioam-rawexport-05 (work in progress), 2043 July 2021. 2045 [RFC7276] Mizrahi, T., Sprecher, N., Bellagamba, E., and Y. 2046 Weingarten, "An Overview of Operations, Administration, 2047 and Maintenance (OAM) Tools", RFC 7276, 2048 DOI 10.17487/RFC7276, June 2014, 2049 . 2051 [RFC7384] Mizrahi, T., "Security Requirements of Time Protocols in 2052 Packet Switched Networks", RFC 7384, DOI 10.17487/RFC7384, 2053 October 2014, . 2055 [RFC7665] Halpern, J., Ed. and C. Pignataro, Ed., "Service Function 2056 Chaining (SFC) Architecture", RFC 7665, 2057 DOI 10.17487/RFC7665, October 2015, 2058 . 2060 [RFC7799] Morton, A., "Active and Passive Metrics and Methods (with 2061 Hybrid Types In-Between)", RFC 7799, DOI 10.17487/RFC7799, 2062 May 2016, . 2064 [RFC7820] Mizrahi, T., "UDP Checksum Complement in the One-Way 2065 Active Measurement Protocol (OWAMP) and Two-Way Active 2066 Measurement Protocol (TWAMP)", RFC 7820, 2067 DOI 10.17487/RFC7820, March 2016, 2068 . 2070 [RFC7821] Mizrahi, T., "UDP Checksum Complement in the Network Time 2071 Protocol (NTP)", RFC 7821, DOI 10.17487/RFC7821, March 2072 2016, . 2074 [RFC8300] Quinn, P., Ed., Elzur, U., Ed., and C. Pignataro, Ed., 2075 "Network Service Header (NSH)", RFC 8300, 2076 DOI 10.17487/RFC8300, January 2018, 2077 . 2079 [RFC8799] Carpenter, B. and B. Liu, "Limited Domains and Internet 2080 Protocols", RFC 8799, DOI 10.17487/RFC8799, July 2020, 2081 . 2083 [RFC8877] Mizrahi, T., Fabini, J., and A. Morton, "Guidelines for 2084 Defining Packet Timestamps", RFC 8877, 2085 DOI 10.17487/RFC8877, September 2020, 2086 . 2088 [RFC8926] Gross, J., Ed., Ganga, I., Ed., and T. Sridhar, Ed., 2089 "Geneve: Generic Network Virtualization Encapsulation", 2090 RFC 8926, DOI 10.17487/RFC8926, November 2020, 2091 . 2093 Contributors' Addresses 2095 Carlos Pignataro 2096 Cisco Systems, Inc. 2097 7200-11 Kit Creek Road 2098 Research Triangle Park, NC 27709 2099 United States 2101 Email: cpignata@cisco.com 2102 Mickey Spiegel 2103 Barefoot Networks, an Intel company 2104 4750 Patrick Henry Drive 2105 Santa Clara, CA 95054 2106 US 2108 Email: mickey.spiegel@intel.com 2110 Barak Gafni 2111 Nvidia 2112 350 Oakmead Parkway, Suite 100 2113 Sunnyvale, CA 94085 2114 U.S.A. 2116 Email: gbarak@nvidia.com 2118 Jennifer Lemon 2119 Broadcom 2120 270 Innovation Drive 2121 San Jose, CA 95134 2122 US 2124 Email: jennifer.lemon@broadcom.com 2126 Hannes Gredler 2127 RtBrick Inc. 2129 Email: hannes@rtbrick.com 2131 John Leddy 2132 United States 2134 Email: john@leddy.net 2136 Stephen Youell 2137 JP Morgan Chase 2138 25 Bank Street 2139 London E14 5JP 2140 United Kingdom 2142 Email: stephen.youell@jpmorgan.com 2143 David Mozes 2145 Email: mosesster@gmail.com 2147 Petr Lapukhov 2148 Facebook 2149 1 Hacker Way 2150 Menlo Park, CA 94025 2151 US 2153 Email: petr@fb.com 2155 Remy Chang 2156 Barefoot Networks 2157 4750 Patrick Henry Drive 2158 Santa Clara, CA 95054 2159 US 2161 Email: remy@barefootnetworks.com 2163 Daniel Bernier 2164 Bell Canada 2165 Canada 2167 Email: daniel.bernier@bell.ca 2169 Authors' Addresses 2171 Frank Brockners (editor) 2172 Cisco Systems, Inc. 2173 Hansaallee 249, 3rd Floor 2174 DUESSELDORF, NORDRHEIN-WESTFALEN 40549 2175 Germany 2177 Email: fbrockne@cisco.com 2179 Shwetha Bhandari (editor) 2180 Thoughtspot 2181 3rd Floor, Indiqube Orion, 24th Main Rd, Garden Layout, HSR Layout 2182 Bangalore, KARNATAKA 560 102 2183 India 2185 Email: shwetha.bhandari@thoughtspot.com 2186 Tal Mizrahi (editor) 2187 Huawei 2188 8-2 Matam 2189 Haifa 3190501 2190 Israel 2192 Email: tal.mizrahi.phd@gmail.com