idnits 2.17.1 draft-ietf-ippm-ioam-data-14.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (June 24, 2021) is 1038 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) -- Looks like a reference, but probably isn't: '0' on line 636 -- Looks like a reference, but probably isn't: '1' on line 640 -- Possible downref: Non-RFC (?) normative reference: ref. 'POSIX' == Outdated reference: A later version (-03) exists of draft-brockners-ippm-ioam-data-integrity-01 == Outdated reference: A later version (-03) exists of draft-brockners-opsawg-ioam-deployment-02 == Outdated reference: A later version (-13) exists of draft-ietf-nvo3-vxlan-gpe-11 == Outdated reference: A later version (-07) exists of draft-spiegel-ippm-ioam-rawexport-04 Summary: 0 errors (**), 0 flaws (~~), 5 warnings (==), 4 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 ippm F. Brockners, Ed. 3 Internet-Draft Cisco 4 Intended status: Standards Track S. Bhandari, Ed. 5 Expires: December 26, 2021 Thoughtspot 6 T. Mizrahi, Ed. 7 Huawei 8 June 24, 2021 10 Data Fields for In-situ OAM 11 draft-ietf-ippm-ioam-data-14 13 Abstract 15 In-situ Operations, Administration, and Maintenance (IOAM) records 16 operational and telemetry information in the packet while the packet 17 traverses a path in the network. This document discusses the data 18 fields and associated data types for in-situ OAM. In-situ OAM data 19 fields can be encapsulated into a variety of protocols such as NSH, 20 Segment Routing, Geneve, or IPv6. In-situ OAM can be used to 21 complement OAM mechanisms based on, e.g., ICMP or other types of 22 probe packets. 24 Status of This Memo 26 This Internet-Draft is submitted in full conformance with the 27 provisions of BCP 78 and BCP 79. 29 Internet-Drafts are working documents of the Internet Engineering 30 Task Force (IETF). Note that other groups may also distribute 31 working documents as Internet-Drafts. The list of current Internet- 32 Drafts is at https://datatracker.ietf.org/drafts/current/. 34 Internet-Drafts are draft documents valid for a maximum of six months 35 and may be updated, replaced, or obsoleted by other documents at any 36 time. It is inappropriate to use Internet-Drafts as reference 37 material or to cite them other than as "work in progress." 39 This Internet-Draft will expire on December 26, 2021. 41 Copyright Notice 43 Copyright (c) 2021 IETF Trust and the persons identified as the 44 document authors. All rights reserved. 46 This document is subject to BCP 78 and the IETF Trust's Legal 47 Provisions Relating to IETF Documents 48 (https://trustee.ietf.org/license-info) in effect on the date of 49 publication of this document. Please review these documents 50 carefully, as they describe your rights and restrictions with respect 51 to this document. Code Components extracted from this document must 52 include Simplified BSD License text as described in Section 4.e of 53 the Trust Legal Provisions and are provided without warranty as 54 described in the Simplified BSD License. 56 Table of Contents 58 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 59 2. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 3 60 3. Conventions . . . . . . . . . . . . . . . . . . . . . . . . . 4 61 4. Scope, Applicability, and Assumptions . . . . . . . . . . . . 5 62 5. IOAM Data-Fields, Types, Nodes . . . . . . . . . . . . . . . 6 63 5.1. IOAM Data-Fields and Option-Types . . . . . . . . . . . . 7 64 5.2. IOAM-Domains and types of IOAM Nodes . . . . . . . . . . 7 65 5.3. IOAM-Namespaces . . . . . . . . . . . . . . . . . . . . . 9 66 5.4. IOAM Trace Option-Types . . . . . . . . . . . . . . . . . 11 67 5.4.1. Pre-allocated and Incremental Trace Option-Types . . 13 68 5.4.2. IOAM node data fields and associated formats . . . . 18 69 5.4.2.1. Hop_Lim and node_id short format . . . . . . . . 18 70 5.4.2.2. ingress_if_id and egress_if_id . . . . . . . . . 19 71 5.4.2.3. timestamp seconds . . . . . . . . . . . . . . . . 19 72 5.4.2.4. timestamp faction . . . . . . . . . . . . . . . . 20 73 5.4.2.5. transit delay . . . . . . . . . . . . . . . . . . 20 74 5.4.2.6. namespace specific data . . . . . . . . . . . . . 20 75 5.4.2.7. queue depth . . . . . . . . . . . . . . . . . . . 21 76 5.4.2.8. Checksum Complement . . . . . . . . . . . . . . . 21 77 5.4.2.9. Hop_Lim and node_id wide . . . . . . . . . . . . 22 78 5.4.2.10. ingress_if_id and egress_if_id wide . . . . . . . 22 79 5.4.2.11. namespace specific data wide . . . . . . . . . . 22 80 5.4.2.12. buffer occupancy . . . . . . . . . . . . . . . . 23 81 5.4.2.13. Opaque State Snapshot . . . . . . . . . . . . . . 23 82 5.4.3. Examples of IOAM node data . . . . . . . . . . . . . 24 83 5.5. IOAM Proof of Transit Option-Type . . . . . . . . . . . . 26 84 5.5.1. IOAM Proof of Transit Type 0 . . . . . . . . . . . . 28 85 5.6. IOAM Edge-to-Edge Option-Type . . . . . . . . . . . . . . 29 86 6. Timestamp Formats . . . . . . . . . . . . . . . . . . . . . . 31 87 6.1. PTP Truncated Timestamp Format . . . . . . . . . . . . . 31 88 6.2. NTP 64-bit Timestamp Format . . . . . . . . . . . . . . . 32 89 6.3. POSIX-based Timestamp Format . . . . . . . . . . . . . . 34 90 7. IOAM Data Export . . . . . . . . . . . . . . . . . . . . . . 35 91 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 35 92 8.1. IOAM Option-Type Registry . . . . . . . . . . . . . . . . 36 93 8.2. IOAM Trace-Type Registry . . . . . . . . . . . . . . . . 36 94 8.3. IOAM Trace-Flags Registry . . . . . . . . . . . . . . . . 37 95 8.4. IOAM POT-Type Registry . . . . . . . . . . . . . . . . . 38 96 8.5. IOAM POT-Flags Registry . . . . . . . . . . . . . . . . . 38 97 8.6. IOAM E2E-Type Registry . . . . . . . . . . . . . . . . . 39 98 8.7. IOAM Namespace-ID Registry . . . . . . . . . . . . . . . 39 99 9. Management and Deployment Considerations . . . . . . . . . . 41 100 10. Security Considerations . . . . . . . . . . . . . . . . . . . 41 101 11. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 43 102 12. References . . . . . . . . . . . . . . . . . . . . . . . . . 43 103 12.1. Normative References . . . . . . . . . . . . . . . . . . 43 104 12.2. Informative References . . . . . . . . . . . . . . . . . 44 105 Contributors' Addresses . . . . . . . . . . . . . . . . . . . . . 45 106 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 47 108 1. Introduction 110 This document defines data fields for "in-situ" Operations, 111 Administration, and Maintenance (IOAM). In-situ OAM records OAM 112 information within the packet while the packet traverses a particular 113 network domain. The term "in-situ" refers to the fact that the OAM 114 data is added to the data packets rather than being sent within 115 packets specifically dedicated to OAM. IOAM is to complement 116 mechanisms such as Ping or Traceroute. In terms of "active" or 117 "passive" OAM, "in-situ" OAM can be considered a hybrid OAM type. 118 "In-situ" mechanisms do not require extra packets to be sent. IOAM 119 adds information to the already available data packets and therefore 120 cannot be considered passive. In terms of the classification given 121 in [RFC7799], IOAM could be portrayed as Hybrid Type I. IOAM 122 mechanisms can be leveraged where mechanisms using, e.g., ICMP do not 123 apply or do not offer the desired results, such as proving that a 124 certain traffic flow takes a pre-defined path, SLA verification for 125 the data traffic, detailed statistics on traffic distribution paths 126 in networks that distribute traffic across multiple paths, or 127 scenarios in which probe traffic is potentially handled differently 128 from regular data traffic by the network devices. 130 The term "in situ OAM" was originally motivated by the use of OAM 131 related mechanisms that add information into a packet. This document 132 uses IOAM as a term defining the IOAM technology. IOAM includes "in- 133 situ" mechanisms, but also mechanisms that could trigger the creation 134 of additional packets dedicated to OAM. 136 2. Contributors 138 This document was the collective effort of several authors. The text 139 and content were contributed by the editors and the co-authors listed 140 below. The contact information of the co-authors appears at the end 141 of this document. 143 o Carlos Pignataro 144 o Mickey Spiegel 146 o Barak Gafni 148 o Jennifer Lemon 150 o Hannes Gredler 152 o John Leddy 154 o Stephen Youell 156 o David Mozes 158 o Petr Lapukhov 160 o Remy Chang 162 o Daniel Bernier 164 3. Conventions 166 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 167 "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and 168 "OPTIONAL" in this document are to be interpreted as described in BCP 169 14 [RFC2119] [RFC8174] when, and only when, they appear in all 170 capitals, as shown here. 172 Abbreviations and definitions used in this document: 174 E2E: Edge to Edge 176 Geneve: Generic Network Virtualization Encapsulation [RFC8926] 178 IOAM: In-situ Operations, Administration, and Maintenance 180 MTU: Maximum Transmit Unit 182 NSH: Network Service Header [RFC8300] 184 OAM: Operations, Administration, and Maintenance 186 PMTU Path MTU 188 POT: Proof of Transit 190 SFC: Service Function Chain 191 Short format: "Short format" refers to an IOAM-Data-Field which 192 comprises 4 octets. 194 SID: Segment Identifier 196 SR: Segment Routing 198 VXLAN-GPE: Virtual eXtensible Local Area Network, Generic Protocol 199 Extension [I-D.ietf-nvo3-vxlan-gpe] 201 Wide format: "Wide format" refers to an IOAM-Data-Field which 202 comprises 8 octets. 204 4. Scope, Applicability, and Assumptions 206 IOAM assumes a set of constraints as well as guiding principles and 207 concepts that go hand in hand with the definition of the IOAM data 208 fields. These constraints, guiding principles, and concepts are 209 described in this section. A discussion of how IOAM data fields and 210 the associated concepts are applied to an IOAM deployment are out of 211 scope for this document. Please refer to 212 [I-D.brockners-opsawg-ioam-deployment] for IOAM deployment 213 considerations. 215 Scope: This document defines the data fields and associated data 216 types for in-situ OAM. The in-situ OAM data fields can be 217 encapsulated in a variety of protocols, including NSH, Segment 218 Routing, Geneve, and IPv6. Specification details for these different 219 protocols are outside the scope of this document. It is expected 220 that each such encapsulation would be specified by an RFC, jointly 221 designed by the working group that develops or maintains the 222 encapsulation protocol and the IETF IPPM working group. 224 Deployment domain (or scope) of in-situ OAM deployment: IOAM is 225 focused on "limited domains" as defined in [RFC8799]. For IOAM, a 226 limited domain could for example be an enterprise campus using 227 physical connections between devices or an overlay network using 228 virtual connections / tunnels for connectivity between said devices. 229 A limited domain which uses IOAM is called an "IOAM domain". An IOAM 230 domain is bounded by its perimeter or edge. Designers of protocol 231 encapsulations for IOAM specify mechanisms to ensure that IOAM data 232 stays within an IOAM domain. In addition, the operator of such a 233 domain is expected to put provisions in place to ensure that IOAM 234 data does not leak beyond the edge of an IOAM domain using, for 235 example, packet filtering methods. The operator has to consider the 236 potential operational impact of IOAM to mechanisms such as ECMP 237 processing (e.g. load-balancing schemes based on packet length could 238 be impacted by the increased packet size due to IOAM), path MTU 239 (i.e., ensure that the MTU of all links within a domain is 240 sufficiently large to support the increased packet size due to IOAM) 241 and ICMP message handling (i.e., in case of IPv6, IOAM support for 242 ICMPv6 Echo Request/Reply is desired which would translate into 243 ICMPv6 extensions to enable IOAM-Data-Fields to be copied from an 244 Echo Request message to an Echo Reply message). 246 IOAM control points: IOAM-Data-Fields are added to or removed from 247 the user traffic by the devices which form the edge of a domain. 248 Devices which form an IOAM-Domain can add, update or remove IOAM- 249 Data-Fields. Edge devices of an IOAM-Domain can be hosts or network 250 devices. 252 Traffic-sets that IOAM is applied to: IOAM can be deployed on all or 253 only on subsets of the user traffic. Using IOAM on a selected set of 254 traffic (e.g., per interface, based on an access control list or flow 255 specification defining a specific set of traffic, etc.) could be 256 useful in deployments where the cost of processing IOAM-Data-Fields 257 by encapsulating, transit, or decapsulating node(s) might be a 258 concern from a performance or operational perspective. Thus limiting 259 the amount of traffic IOAM is applied to could be beneficial in some 260 deployments. 262 Encapsulation independence: The definition of IOAM-Data-Fields is 263 independent from the protocols the IOAM-Data-Fields are encapsulated 264 into. IOAM-Data-Fields can be encapsulated into several 265 encapsulating protocols. 267 Layering: If several encapsulation protocols (e.g., in case of 268 tunneling) are stacked on top of each other, IOAM-Data-Fields could 269 be present at multiple layers. The behavior follows the ships-in- 270 the-night model, i.e., IOAM-Data-Fields in one layer are independent 271 from IOAM-Data-Fields in another layer. Layering allows operators to 272 instrument the protocol layer they want to measure. The different 273 layers could, but do not have to, share the same IOAM encapsulation 274 mechanisms. 276 IOAM implementation: The definition of the IOAM-Data-Fields take the 277 specifics of devices with hardware data planes and software data 278 planes into account. 280 5. IOAM Data-Fields, Types, Nodes 282 This section details IOAM-related nomenclature and describes data 283 types such as IOAM-Data-Fields, IOAM-Types, IOAM-Namespaces as well 284 as the different types of IOAM nodes. 286 5.1. IOAM Data-Fields and Option-Types 288 An IOAM-Data-Field is a set of bits with a defined format and 289 meaning, which can be stored at a certain place in a packet for the 290 purpose of IOAM. 292 To accommodate the different uses of IOAM, IOAM-Data-Fields fall into 293 different categories. In IOAM, these categories are referred to as 294 IOAM-Option-Types. A common registry is maintained for IOAM-Option- 295 Types, see Section 8.1 for details. Corresponding to these IOAM- 296 Option-Types, different IOAM-Data-Fields are defined. 298 This document defines four IOAM-Option-Types: 300 o Pre-allocated Trace Option-Type 302 o Incremental Trace Option-Type 304 o Proof of Transit (POT) Option-Type 306 o Edge-to-Edge (E2E) Option-Type 308 Future IOAM-Option-Types can be allocated by IANA, as described in 309 Section 8.1. 311 5.2. IOAM-Domains and types of IOAM Nodes 313 IOAM is expected to be deployed in a specific domain. The part of 314 the network which employs IOAM is referred to as the "IOAM-Domain". 315 One or more IOAM-Option-Types are added to a packet upon entering the 316 IOAM-Domain and are removed from the packet when exiting the domain. 317 Within the IOAM-Domain, the IOAM-Data-Fields MAY be updated by 318 network nodes that the packet traverses. An IOAM-Domain consists of 319 "IOAM encapsulating nodes", "IOAM decapsulating nodes" and "IOAM 320 transit nodes". The role of a node (i.e., encapsulating, transit, 321 decapsulating) is defined within an IOAM-Namespace (see below). A 322 node can have different roles in different IOAM-Namespaces. 324 A device which adds at least one IOAM-Option-Type to the packet is 325 called an "IOAM encapsulating node", whereas a device which removes 326 an IOAM-Option-Type is referred to as an "IOAM decapsulating node". 327 Nodes within the domain which are aware of IOAM data and read and/or 328 write and/or process IOAM data are called "IOAM transit nodes". IOAM 329 nodes which add or remove the IOAM-Data-Fields can also update the 330 IOAM-Data-Fields at the same time. Or in other words, IOAM 331 encapsulating or decapsulating nodes can also serve as IOAM transit 332 nodes at the same time. Note that not every node in an IOAM domain 333 needs to be an IOAM transit node. For example, a deployment might 334 require that packets traverse a set of firewalls which support IOAM. 335 In that case, only the set of firewall nodes would be IOAM transit 336 nodes rather than all nodes. 338 An "IOAM encapsulating node" incorporates one or more IOAM-Option- 339 Types (from the list of IOAM-Types, see Section 8.1) into packets 340 that IOAM is enabled for. If IOAM is enabled for a selected subset 341 of the traffic, the IOAM encapsulating node is responsible for 342 applying the IOAM functionality to the selected subset. 344 An "IOAM transit node" read and/or write and/or process one or more 345 of the IOAM-Data-Fields. If both the Pre-allocated and the 346 Incremental Trace Option-Types are present in the packet, each IOAM 347 transit node based on configuration and available implementation of 348 IOAM populates IOAM trace data in either Pre-allocated or Incremental 349 Trace Option-Type but not both. A transit node MUST ignore IOAM- 350 Option-Types that it does not understand. A transit node MUST NOT 351 add new IOAM-Option-Types to a packet, MUST NOT remove IOAM-Option- 352 Types from a packet, and MUST NOT change the IOAM-Data-Fields of an 353 IOAM Edge-to-Edge Option-Type. 355 An "IOAM decapsulating node" removes IOAM-Option-Type(s) from 356 packets. 358 The role of an IOAM-encapsulating, IOAM-transit or IOAM-decapsulating 359 node is always performed within a specific IOAM-Namespace. This 360 means that an IOAM node which is, e.g., an IOAM-decapsulating node 361 for IOAM-Namespace "A" but not for IOAM-Namespace "B" will only 362 remove the IOAM-Option-Types for IOAM-Namespace "A" from the packet. 363 Note that this applies even for IOAM-Option-Types that the node does 364 not understand, for example an IOAM-Option-Type other than the four 365 described above, that is added in a future revision. An IOAM 366 decapsulating node situated at the edge of an IOAM domain MUST remove 367 all IOAM-Option-Types and associated encapsulation headers for all 368 IOAM-Namespaces from the packet. 370 IOAM-Namespaces allow for a namespace-specific definition and 371 interpretation of IOAM-Data-Fields. An interface-id could for 372 example point to a physical interface (e.g., to understand which 373 physical interface of an aggregated link is used when receiving or 374 transmitting a packet) whereas in another case it could refer to a 375 logical interface (e.g., in case of tunnels). Please refer to 376 Section 5.3 for details on IOAM-Namespaces. 378 5.3. IOAM-Namespaces 380 An IOAM-Namespace can be associated to a subset or all of the the 381 IOAM-Option-Types and their corresponding IOAM-Data-Fields. IOAM- 382 Namespaces add further context to IOAM-Option-Types and associated 383 IOAM-Data-Fields. The IOAM-Option-Types and associated IOAM-Data- 384 Fields are interpreted as defined in this document, regardless of the 385 value of the IOAM-Namespace. However, IOAM-Namespaces provide a way 386 to group nodes to support different deployment approaches of IOAM 387 (see a few example use-cases below). IOAM-Namespaces also help to 388 resolve potential issues which can occur due to IOAM-Data-Fields not 389 being globally unique (e.g., IOAM node identifiers do not have to be 390 globally unique). IOAM-Data-Fields significance is always within a 391 particular IOAM-Namespace. Given that IOAM-Data-Fields are always 392 interpreted the context of a specific namespace, the namespace-id 393 field always needs to be carried along with the IOAM data-fields 394 themselves. 396 An IOAM-Namespace is identified by a 16-bit namespace identifier 397 (Namespace-ID). The IOAM-Namespace field is included in all the 398 IOAM-Option-Types defined in this document, and MUST be included in 399 all future IOAM-Option-Types. The Namespace-ID value is divided into 400 two sub-ranges: 402 o An operator-assigned range from 0x0001 to 0x7FFF 404 o An IANA-assigned range from 0x8000 to 0xFFFF 406 The IANA-assigned range is intended to allow future extensions to 407 have new and interoperable IOAM functionality, while the operator- 408 assigned range is intended to be domain-specific, and managed by the 409 network operator. The Namespace-ID value of 0x0000 is the "Default- 410 Namespace-ID". The Default-Namespace-ID indicates that no specific 411 namespace is associated with the IOAM data fields in the packet. The 412 Default-Namespace-ID MUST be supported by all nodes implementing 413 IOAM. A use-case for the Default-Namespace-ID are deployments which 414 do not leverage specific namespaces for some or all of their packets 415 that carry IOAM data fields. 417 Namespace identifiers allow devices which are IOAM capable to 418 determine: 420 o whether IOAM-Option-Type(s) need to be processed by a device: If 421 the Namespace-ID contained in a packet does not match any 422 Namespace-ID the node is configured to operate on, then the node 423 MUST NOT change the contents of the IOAM-Data-Fields. 425 o which IOAM-Option-Type needs to be processed/updated in case there 426 are multiple IOAM-Option-Types present in the packet. Multiple 427 IOAM-Option-Types can be present in a packet in case of 428 overlapping IOAM-Domains or in case of a layered IOAM deployment. 430 o whether IOAM-Option-Type(s) have to be removed from the packet, 431 e.g., at a domain edge or domain boundary. 433 IOAM-Namespaces support several different uses: 435 o IOAM-Namespaces can be used by an operator to distinguish 436 different operational domains. Devices at domain edges can filter 437 on Namespace-IDs to provide for proper IOAM-Domain isolation. 439 o IOAM-Namespaces provide additional context for IOAM-Data-Fields 440 and thus can be used to ensure that IOAM-Data-Fields are unique 441 and are interpreted properly by management stations or network 442 controllers. For example, the node identifier field (node_id, see 443 below) does not need to be unique in a deployment (e.g., if an 444 operator wishes to use different node identifiers for different 445 IOAM layers, even within the same device; or node identifiers 446 might not be unique for other organizational reasons, such as 447 after a merger of two formerly separated organizations), the 448 Namespace-ID can be used as a context identifier, such that the 449 combination of node_id and Namespace-ID will always be unique. 450 Similarly, IOAM-Namespaces can be used to define how certain IOAM- 451 Data-Fields are interpreted: IOAM offers three different timestamp 452 format options. The Namespace-ID can be used to determine the 453 timestamp format. IOAM-Data-Fields (e.g., buffer occupancy) which 454 do not have a unit associated are to be interpreted within the 455 context of a IOAM-Namespace. 457 o IOAM-Namespaces can be used to identify different sets of devices 458 (e.g., different types of devices) in a deployment: If an operator 459 desires to insert different IOAM-Data-Fields based on the device, 460 the devices could be grouped into multiple IOAM-Namespaces. This 461 could be due to the fact that the IOAM feature set differs between 462 different sets of devices, or it could be for reasons of optimized 463 space usage in the packet header. It could also stem from 464 hardware or operational limitations on the size of the trace data 465 that can be added and processed, preventing collection of a full 466 trace for a flow. 468 * Assigning different IOAM Namespace-IDs to different sets of 469 nodes or network partitions and using the Namespace-ID as a 470 selector at the IOAM encapsulating node, a full trace for a 471 flow could be collected and constructed via partial traces in 472 different packets of the same flow. Example: An operator could 473 choose to group the devices of a domain into two IOAM- 474 Namespaces, in a way that on average, only every second hop 475 would be recorded by any device. To retrieve a full view of 476 the deployment, the captured IOAM-Data-Fields of the two IOAM- 477 Namespaces need to be correlated. 479 * Assigning different IOAM Namespace-IDs to different sets of 480 nodes or network partitions and using a separate instance of an 481 IOAM-Option-Type for each Namespace-ID, a full trace for a flow 482 could be collected and constructed via partial traces from each 483 IOAM-Option-Type in each of the packets in the flow. Example: 484 An operator could choose to group the devices of a domain into 485 two IOAM-Namespaces, in a way that each IOAM-Namespace is 486 represented by one of two IOAM-Option-Types in the packet. 487 Each node would record data only for the IOAM-Namespace that it 488 belongs to, ignoring the other IOAM-Option-Type with a IOAM- 489 Namespace to which it doesn't belong. To retrieve a full view 490 of the deployment, the captured IOAM-Data-Fields of the two 491 IOAM-Namespaces need to be correlated. 493 5.4. IOAM Trace Option-Types 495 "IOAM tracing data" is expected to be collected at every IOAM transit 496 node that a packet traverses to ensure visibility into the entire 497 path a packet takes within an IOAM-Domain. I.e., in a typical 498 deployment, all nodes in an IOAM-Domain would participate in IOAM and 499 thus be IOAM transit nodes, IOAM encapsulating or IOAM decapsulating 500 nodes. If not all nodes within a domain support IOAM functionality 501 as defined in this document, IOAM tracing information (i.e., node 502 data, see below) will only be collected on those nodes which support 503 IOAM functionality as defined in this document. Nodes which do not 504 support IOAM functionality as defined in this document will forward 505 the packet without any changes to the IOAM-Data-Fields. The maximum 506 number of hops and the minimum path MTU of the IOAM domain is assumed 507 to be known. An overflow indicator (O-bit) is defined as one of the 508 ways to deal with situations where the PMTU was underestimated, i.e., 509 where the number of hops which are IOAM capable exceeds the available 510 space in the packet. 512 To optimize hardware and software implementations, IOAM tracing is 513 defined as two separate options. A deployment can choose to 514 configure and support one or both of the following options. 516 Pre-allocated Trace-Option: This trace option is defined as a 517 container of node data fields (see below) with pre-allocated space 518 for each node to populate its information. This option is useful 519 for implementations where it is efficient to allocate the space 520 once and index into the array to populate the data during transit 521 (e.g., software forwarders often fall into this class). The IOAM 522 encapsulating node allocates space for Pre-allocated Trace Option- 523 Type in the packet and sets corresponding fields in this IOAM- 524 Option-Type. The IOAM encapsulating node allocates an array which 525 is used to store operational data retrieved from every node while 526 the packet traverses the domain. IOAM transit nodes update the 527 content of the array, and possibly update the checksums of outer 528 headers. A pointer which is part of the IOAM trace data, points 529 to the next empty slot in the array. An IOAM transit node that 530 updates the content of the pre-allocated option also updates the 531 value of the pointer, which specifies where the next IOAM transit 532 node fills in its data. The "node data list" array (see below) in 533 the packet is populated iteratively as the packet traverses the 534 network, starting with the last entry of the array, i.e., "node 535 data list [n]" is the first entry to be populated, "node data list 536 [n-1]" is the second one, etc. 538 Incremental Trace-Option: This trace option is defined as a 539 container of node data fields where each node allocates and pushes 540 its node data immediately following the option header. This type 541 of trace recording is useful for some of the hardware 542 implementations as it eliminates the need for the transit network 543 elements to read the full array in the option and allows for 544 arbitrarily long packets as the MTU allows. The IOAM 545 encapsulating node allocates space for the Incremental Trace 546 Option-Type. Based on operational state and configuration, the 547 IOAM encapsulating node sets the fields in the Option-Type that 548 control what IOAM-Data-Fields have to be collected and how large 549 the node data list can grow. IOAM transit nodes push their node 550 data to the node data list subject to any protocol constraints of 551 the encapsulating layer. They then decrease the remaining length 552 available to subsequent nodes and adjust the lengths and possibly 553 checksums in outer headers. 555 IOAM encapsulating nodes and IOAM decapsulating nodes which support 556 tracing MUST support both Trace-Option-Types. For IOAM transit nodes 557 it is sufficient to support one of the Trace-Option-Types. In the 558 event that both options are utilized in a deployment at the same 559 time, the Incremental Trace-Option MUST be placed before the Pre- 560 allocated Trace-Option. Deployments which mix devices with either 561 the Incremental Trace-Option or the Pre-allocated Trace-Option could 562 result in both Option-Types being present in a packet. Given that 563 the operator knows which equipment is deployed in a particular IOAM 564 domain, the operator will decide by means of configuration which 565 type(s) of trace options will be used for a particular domain. 567 Every node data entry holds information for a particular IOAM transit 568 node that is traversed by a packet. The IOAM decapsulating node 569 removes the IOAM-Option-Type(s) and processes and/or exports the 570 associated data. Like all IOAM-Data-Fields, the IOAM-Data-Fields of 571 the IOAM-Trace-Option-Types are defined in the context of an IOAM- 572 Namespace. 574 IOAM tracing can collect the following types of information: 576 o Identification of the IOAM node. An IOAM node identifier can 577 match to a device identifier or a particular control point or 578 subsystem within a device. 580 o Identification of the interface that a packet was received on, 581 i.e., ingress interface. 583 o Identification of the interface that a packet was sent out on, 584 i.e., egress interface. 586 o Time of day when the packet was processed by the node as well as 587 the transit delay. Different definitions of processing time are 588 feasible and expected, though it is important that all devices of 589 an in-situ OAM domain follow the same definition. 591 o Generic data: Format-free information where syntax and semantic of 592 the information is defined by the operator in a specific 593 deployment. For a specific IOAM-Namespace, all IOAM nodes have to 594 interpret the generic data the same way. Examples for generic 595 IOAM data include geo-location information (location of the node 596 at the time the packet was processed), buffer queue fill level or 597 cache fill level at the time the packet was processed, or even a 598 battery charge level. 600 o Information to detect whether IOAM trace data was added at every 601 hop or whether certain hops in the domain weren't IOAM transit 602 nodes. 604 It should be noted that the semantics of some of the node data fields 605 that are defined below, such as the queue depth and buffer occupancy, 606 are implementation specific. This approach is intended to allow IOAM 607 nodes with various different architectures. 609 5.4.1. Pre-allocated and Incremental Trace Option-Types 611 The IOAM Pre-allocated Trace-Option and the IOAM Incremental Trace- 612 Option have similar formats. Except where noted below, the internal 613 formats and fields of the two trace options are identical. Both 614 Trace-Options consist of a fixed size "trace option header" and a 615 variable data space to store gathered data, the "node data list". An 616 IOAM transit node (that is not an IOAM encapsulating node or IOAM 617 decapsulating node) MUST NOT modify any of the fields in the fixed 618 size "trace option header", other than "flags" and "RemainingLen", 619 i.e., an IOAM transit node MUST NOT modify the Namespace-ID, NodeLen, 620 IOAM-Trace-Type, or Reserved fields. 622 Pre-allocated and incremental trace option headers: 624 0 1 2 3 625 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 626 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 627 | Namespace-ID |NodeLen | Flags | RemainingLen| 628 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 629 | IOAM-Trace-Type | Reserved | 630 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 632 The trace option data MUST be 4-octet aligned: 634 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+<-+ 635 | | | 636 | node data list [0] | | 637 | | | 638 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ D 639 | | a 640 | node data list [1] | t 641 | | a 642 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 643 ~ ... ~ S 644 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ p 645 | | a 646 | node data list [n-1] | c 647 | | e 648 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 649 | | | 650 | node data list [n] | | 651 | | | 652 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+<-+ 654 Namespace-ID: 16-bit identifier of an IOAM-Namespace. The 655 Namespace-ID value of 0x0000 is defined as the "Default-Namespace- 656 ID" (see Section 5.3) and MUST be known to all the nodes 657 implementing IOAM. For any other Namespace-ID value that does not 658 match any Namespace-ID the node is configured to operate on, the 659 node MUST NOT change the contents of the IOAM-Data-Fields. 661 NodeLen: 5-bit unsigned integer. This field specifies the length of 662 data added by each node in multiples of 4-octets, excluding the 663 length of the "Opaque State Snapshot" field. 665 If IOAM-Trace-Type bit 22 is not set, then NodeLen specifies the 666 actual length added by each node. If IOAM-Trace-Type bit 22 is 667 set, then the actual length added by a node would be (NodeLen + 668 length of the "Opaque State Snapshot" field) in 4 octet units. 670 For example, if 3 IOAM-Trace-Type bits are set and none of them 671 are in wide format, then NodeLen would be 3. If 3 IOAM-Trace-Type 672 bits are set and 2 of them are wide, then NodeLen would be 5. 674 An IOAM encapsulating node MUST set NodeLen. 676 A node receiving an IOAM Pre-allocated or Incremental Trace-Option 677 relies on the NodeLen value. 679 Flags 4-bit field. Flags are allocated by IANA, as specified in 680 Section 8.3. This document allocates a single flag as follows: 682 Bit 0 "Overflow" (O-bit) (most significant bit). In case a 683 network element is supposed to add node data to a packet, but 684 detects that there are not enough octets left to record the 685 node data, the network element MUST NOT add any fields and MUST 686 set the overflow "O-bit" to "1" in the IOAM-Trace-Option 687 header. This is useful for transit nodes to ignore further 688 processing of the option. 690 RemainingLen: 7-bit unsigned integer. This field specifies the data 691 space in multiples of 4-octets remaining for recording the node 692 data, before the node data list is considered to have overflowed. 693 The sender MUST assign the initial value of the RemainingLen 694 field. The sender MAY calculate the value of the RemainingLen 695 field by computing the number of node data bytes allowed before 696 exceeding the path MTU (PMTU), given that the PMTU is known to the 697 sender. Subsequent nodes can carry out a simple comparison 698 between RemainingLen and NodeLen, along with the length of the 699 "Opaque State Snapshot" if applicable, to determine whether or not 700 data can be added by this node. When node data is added, the node 701 MUST decrease RemainingLen by the amount of data added. In the 702 pre-allocated trace option, RemainingLen is used to derive the 703 offset in data space to record the node data element. 704 Specifically, the recording of the node data element would start 705 from RemainingLen - NodeLen - sizeof(opaque snapshot) in 4 octet 706 units. If RemainingLen in a pre-allocated trace option exceeds 707 the length of the option, as specified in the lower layer header 708 (which is not within the scope of this document), then the node 709 MUST NOT add any fields. 711 IOAM-Trace-Type: A 24-bit identifier which specifies which data 712 types are used in this node data list. 714 The IOAM-Trace-Type value is a bit field. The following bits are 715 defined in this document, with details on each bit described in 716 the Section 5.4.2. The order of packing the data fields in each 717 node data element follows the bit order of the IOAM-Trace-Type 718 field, as follows: 720 Bit 0 (Most significant bit) When set, indicates presence of 721 Hop_Lim and node_id (short format) in the node data. 723 Bit 1 When set, indicates presence of ingress_if_id and 724 egress_if_id (short format) in the node data. 726 Bit 2 When set, indicates presence of timestamp seconds in the 727 node data. 729 Bit 3 When set, indicates presence of timestamp fraction in the 730 node data. 732 Bit 4 When set, indicates presence of transit delay in the node 733 data. 735 Bit 5 When set, indicates presence of IOAM-Namespace specific 736 data (short format) in the node data. 738 Bit 6 When set, indicates presence of queue depth in the node 739 data. 741 Bit 7 When set, indicates presence of the Checksum Complement 742 node data. 744 Bit 8 When set, indicates presence of Hop_Lim and node_id in 745 wide format in the node data. 747 Bit 9 When set, indicates presence of ingress_if_id and 748 egress_if_id in wide format in the node data. 750 Bit 10 When set, indicates presence of IOAM-Namespace specific 751 data in wide format in the node data. 753 Bit 11 When set, indicates presence of buffer occupancy in the 754 node data. 756 Bit 12-21 Undefined. These values are available for future 757 assignment in the IOAM Trace-Type Registry (Section 8.2). 758 Every future node data field corresponding to one of 759 these bits MUST be 4-octets long. An IOAM encapsulating 760 node MUST set the value of each undefined bit to 0. If 761 an IOAM transit node receives a packet with one or more 762 of these bits set to 1, it MUST either: 764 1. Add corresponding node data filled with the reserved 765 value 0xFFFFFFFF, after the node data fields for the 766 IOAM-Trace-Type bits defined above, such that the 767 total node data added by this node in units of 768 4-octets is equal to NodeLen, or 770 2. Not add any node data fields to the packet, even for 771 the IOAM-Trace-Type bits defined above. 773 Bit 22 When set, indicates presence of variable length Opaque 774 State Snapshot field. 776 Bit 23 Reserved: MUST be set to zero upon transmission and 777 ignored upon receipt. This bit is reserved to allow for 778 future extensions of the IOAM-Trace-Type bit field. 780 Section 5.4.2 describes the IOAM-Data-Types and their formats. 781 Within an IOAM-Domain possible combinations of these bits making 782 the IOAM-Trace-Type can be restricted by configuration knobs. 784 Reserved: 8-bits. An IOAM encapsulating node MUST set the value to 785 zero upon transmission. IOAM transit nodes MUST ignore the 786 received value. 788 Node data List [n]: Variable-length field. This is a list of node 789 data elements where the content of each node data element is 790 determined by the IOAM-Trace-Type. The order of packing the data 791 fields in each node data element follows the bit order of the 792 IOAM-Trace-Type field. Each node MUST prepend its node data 793 element in front of the node data elements that it received, such 794 that the transmitted node data list begins with this node's data 795 element as the first populated element in the list. The last node 796 data element in this list is the node data of the first IOAM 797 capable node in the path. Populating the node data list in this 798 way ensures that the order of node data list is the same for 799 incremental and pre-allocated trace options. In the pre-allocated 800 trace option, the index contained in RemainingLen identifies the 801 offset for current active node data to be populated. 803 5.4.2. IOAM node data fields and associated formats 805 All the IOAM-Data-Fields MUST be 4-octet aligned. If a node which is 806 supposed to update an IOAM-Data-Field is not capable of populating 807 the value of a field set in the IOAM-Trace-Type, the field value MUST 808 be set to 0xFFFFFFFF for 4-octet fields or 0xFFFFFFFFFFFFFFFF for 809 8-octet fields, indicating that the value is not populated, except 810 when explicitly specified in the field description below. 812 Some IOAM-Data-Fields defined below, such as interface identifiers or 813 IOAM-Namespace specific data, are defined in both "short format" as 814 well as "wide format". "Short format" refers to an IOAM-Data-Field 815 which comprises 4 octets. "Wide format" refers to an IOAM-Data-Field 816 which comprises 8 octets. The use of "short format" or "wide format" 817 is not mutually exclusive. A deployment could choose to leverage 818 both. For example, ingress_if_id_(short format) could be an 819 identifier for the physical interface, whereas ingress_if_id_(wide 820 format) could be an identifier for a logical sub-interface of that 821 physical interface. 823 Data fields and associated data types for each of the IOAM-Data- 824 Fields are specified in the following sections. The definition of 825 IOAM-Data-Fields focuses on the syntax of the data-fields and avoids 826 specifying the semantics where feasible. This is why no units are 827 defined for data-fields like e.g., "buffer occupancy" or "queue 828 depth". With this approach, nodes can supply the information in 829 their native format and are not required to perform unit or format 830 conversions. Systems that further process IOAM information, like 831 e.g., a network management system are assumed to also handle unit 832 conversions as part of their IOAM data-fields processing. The 833 combination of a particular data-field and the namespace-id provides 834 for the context to interpret the provided data appropriately. 836 5.4.2.1. Hop_Lim and node_id short format 838 The "Hop_Lim and node_id short format" field is a 4-octet field that 839 is defined as follows: 841 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 842 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 843 | Hop_Lim | node_id | 844 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 846 Hop_Lim: 1-octet unsigned integer. It is set to the Hop Limit value 847 in the packet at egress from the node that records this data. Hop 848 Limit information is used to identify the location of the node in 849 the communication path. This is copied from the lower layer, 850 e.g., TTL value in IPv4 header or hop limit field from IPv6 header 851 of the packet when the packet is ready for transmission. The 852 semantics of the Hop_Lim field depend on the lower layer protocol 853 that IOAM is encapsulated into, and therefore its specific 854 semantics are outside the scope of this memo. The value of this 855 field MUST be set to 0xff when the lower level does not have a 856 TTL/Hop limit equivalent field. 858 node_id: 3-octet unsigned integer. Node identifier field to 859 uniquely identify a node within the IOAM-Namespace and associated 860 IOAM-Domain. The procedure to allocate, manage and map the 861 node_ids is beyond the scope of this document. See 862 [I-D.brockners-opsawg-ioam-deployment] for a discussion of 863 deployment related aspects of the node_id. 865 5.4.2.2. ingress_if_id and egress_if_id 867 The "ingress_if_id and egress_if_id" field is a 4-octet field that is 868 defined as follows: 870 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 871 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 872 | ingress_if_id | egress_if_id | 873 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 875 ingress_if_id: 2-octet unsigned integer. Interface identifier to 876 record the ingress interface the packet was received on. 878 egress_if_id: 2-octet unsigned integer. Interface identifier to 879 record the egress interface the packet is forwarded out of. 881 Note that due to the fact that IOAM uses its own IOAM-Namespaces for 882 IOAM-Data-Fields, data fields like interface identifiers can be used 883 in a flexible way to represent system resources that are associated 884 with ingressing or egressing packets, i.e., ingress_if_id could 885 represent a physical interface, a virtual or logical interface, or 886 even a queue. 888 5.4.2.3. timestamp seconds 890 The "timestamp seconds" field is a 4-octet unsigned integer field. 891 It contains the absolute timestamp in seconds that specifies the time 892 at which the packet was received by the node. This field has three 893 possible formats; based on either PTP (see e.g., [RFC8877]), NTP 894 [RFC5905], or POSIX [POSIX]. The three timestamp formats are 895 specified in Section 6. In all three cases, the Timestamp Seconds 896 field contains the 32 most significant bits of the timestamp format 897 that is specified in Section 6. If a node is not capable of 898 populating this field, it assigns the value 0xFFFFFFFF. Note that 899 this is a legitimate value that is valid for 1 second in 900 approximately 136 years; the analyzer has to correlate several 901 packets or compare the timestamp value to its own time-of-day in 902 order to detect the error indication. 904 5.4.2.4. timestamp faction 906 The "timestamp fraction" field is a 4-octet unsigned integer field. 907 Fraction specifies the fractional portion of the number of seconds 908 since the NTP epoch [RFC8877]. The field specifies the time at which 909 the packet was received by the node. This field has three possible 910 formats; based on either PTP (see e.g., [RFC8877]), NTP [RFC5905], or 911 POSIX [POSIX]. The three timestamp formats are specified in 912 Section 6. In all three cases, the Timestamp fraction field contains 913 the 32 least significant bits of the timestamp format that is 914 specified in Section 6. If a node is not capable of populating this 915 field, it assigns the value 0xFFFFFFFF. Note that this is a 916 legitimate value in the NTP format, valid for approximately 233 917 picoseconds in every second. If the NTP format is used the analyzer 918 has to correlate several packets in order to detect the error 919 indication. 921 5.4.2.5. transit delay 923 The "transit delay" field is a 4-octet unsigned integer in the range 924 0 to 2^31-1. It is the time in nanoseconds the packet spent in the 925 transit node. This can serve as an indication of the queuing delay 926 at the node. If the transit delay exceeds 2^31-1 nanoseconds then 927 the top bit 'O' is set to indicate overflow and value set to 928 0x80000000. When this field is part of the data field but a node 929 populating the field is not able to fill it, the field position in 930 the field MUST be filled with value 0xFFFFFFFF to mean not populated. 932 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 933 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 934 |O| transit delay | 935 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 937 5.4.2.6. namespace specific data 939 The "namespace specific data" field is a 4-octet field which can be 940 used by the node to add IOAM-Namespace specific data. This 941 represents a "free-format" 4-octet bit field with its semantics 942 defined in the context of a specific IOAM-Namespace. 944 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 945 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 946 | namespace specific data | 947 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 949 5.4.2.7. queue depth 951 The "queue depth" field is a 4-octet unsigned integer field. This 952 field indicates the current length of the egress interface queue of 953 the interface from where the packet is forwarded out. The queue 954 depth is expressed as the current amount of memory buffers used by 955 the queue (a packet could consume one or more memory buffers, 956 depending on its size). 958 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 959 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 960 | queue depth | 961 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 963 5.4.2.8. Checksum Complement 965 The "Checksum Complement" field is a 4-octet node data which contains 966 a 4-octet Checksum Complement field. The Checksum Complement is 967 useful when IOAM is transported over encapsulations that make use of 968 a UDP transport, such as VXLAN-GPE or Geneve. Without the Checksum 969 Complement, nodes adding IOAM node data update the UDP Checksum field 970 following the recommendation of the encapsulation protocols. When 971 the Checksum Complement is present, an IOAM encapsulating node or 972 IOAM transit node adding node data MUST carry out one of the 973 following two alternatives in order to maintain the correctness of 974 the UDP Checksum value: 976 1. Recompute the UDP Checksum field. 978 2. Use the Checksum Complement to make a checksum-neutral update in 979 the UDP payload; the Checksum Complement is assigned a value that 980 complements the rest of the node data fields that were added by 981 the current node, causing the existing UDP Checksum field to 982 remain correct. 984 IOAM decapsulating nodes MUST recompute the UDP Checksum field, since 985 they do not know whether previous hops modified the UDP Checksum 986 field or the Checksum Complement field. 988 Checksum Complement fields are used in a similar manner in [RFC7820] 989 and [RFC7821]. 991 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 992 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 993 | Checksum Complement | 994 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 996 5.4.2.9. Hop_Lim and node_id wide 998 The "Hop_Lim and node_id wide" field is an 8-octet field defined as 999 follows: 1001 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1002 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1003 | Hop_Lim | node_id ~ 1004 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1005 ~ node_id (contd) | 1006 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1008 Hop_Lim: 1-octet unsigned integer. See Section 5.4.2.1 for the 1009 definition of the field. 1011 node_id: 7-octet unsigned integer. Node identifier field to 1012 uniquely identify a node within the IOAM-Namespace and associated 1013 IOAM-Domain. The procedure to allocate, manage and map the 1014 node_ids is beyond the scope of this document. 1016 5.4.2.10. ingress_if_id and egress_if_id wide 1018 The "ingress_if_id and egress_if_id wide" field is an 8-octet field 1019 which is defined as follows: 1021 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1022 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1023 | ingress_if_id | 1024 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1025 | egress_if_id | 1026 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1028 ingress_if_id: 4-octet unsigned integer. Interface identifier to 1029 record the ingress interface the packet was received on. 1031 egress_if_id: 4-octet unsigned integer. Interface identifier to 1032 record the egress interface the packet is forwarded out of. 1034 5.4.2.11. namespace specific data wide 1036 The "namespace specific data wide" field is an 8-octet field which 1037 can be used by the node to add IOAM-Namespace specific data. This 1038 represents a "free-format" 8-octet bit field with its semantics 1039 defined in the context of a specific IOAM-Namespace. 1041 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1042 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1043 | namespace specific data ~ 1044 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1045 ~ namespace specific data (contd) | 1046 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1048 5.4.2.12. buffer occupancy 1050 The "buffer occupancy" field is a 4-octet unsigned integer field. 1051 This field indicates the current status of the occupancy of the 1052 common buffer pool used by a set of queues. The units of this field 1053 are implementation specific. Hence, the units are interpreted within 1054 the context of an IOAM-Namespace and/or node-id if used. The authors 1055 acknowledge that in some operational cases there is a need for the 1056 units to be consistent across a packet path through the network, 1057 hence it is recommended for implementations to use standard units 1058 such as Bytes. 1060 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1061 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1062 | buffer occupancy | 1063 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1065 5.4.2.13. Opaque State Snapshot 1067 The "Opaque State Snapshot" is a variable length field and follows 1068 the fixed length IOAM-Data-Fields defined above. It allows the 1069 network element to store an arbitrary state in the node data field, 1070 without a pre-defined schema. The schema is to be defined within the 1071 context of an IOAM-Namespace. The schema needs to be made known to 1072 the analyzer by some out-of-band mechanism. The specification of 1073 this mechanism is beyond the scope of this document. A 24-bit 1074 "Schema Id" field, interpreted within the context of an IOAM- 1075 Namespace, indicates which particular schema is used, and has to be 1076 configured on the network element by the operator. 1078 0 1 2 3 1079 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1080 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1081 | Length | Schema ID | 1082 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1083 | | 1084 | | 1085 | Opaque data | 1086 ~ ~ 1087 . . 1088 . . 1089 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1091 Length: 1-octet unsigned integer. It is the length in multiples of 1092 4-octets of the Opaque data field that follows Schema Id. 1094 Schema ID: 3-octet unsigned integer identifying the schema of Opaque 1095 data. 1097 Opaque data: Variable length field. This field is interpreted as 1098 specified by the schema identified by the Schema ID. 1100 When this field is part of the data field but a node populating the 1101 field has no opaque state data to report, the Length MUST be set to 0 1102 and the Schema ID MUST be set to 0xFFFFFF to mean no schema. 1104 5.4.3. Examples of IOAM node data 1106 The format used for the entries in a packet's "node data list" array 1107 can vary from packet to packet and deployment to deployment". Some 1108 deployments might only be interested in recording the node 1109 identifiers, whereas others might be interested in recording node 1110 identifiers and timestamps. This section provides example entries of 1111 the "node data list". 1113 0xD40000: IOAM-Trace-Type is 0xD40000 (0b110101000000000000000000) 1114 then the format of node data is: 1116 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1117 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1118 | Hop_Lim | node_id | 1119 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1120 | ingress_if_id | egress_if_id | 1121 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1122 | timestamp fraction | 1123 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1124 | namespace specific data | 1125 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1127 0xC00000: IOAM-Trace-Type is 0xC00000 (0b110000000000000000000000) 1128 then the format is: 1130 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1131 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1132 | Hop_Lim | node_id | 1133 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1134 | ingress_if_id | egress_if_id | 1135 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1137 0x900000: IOAM-Trace-Type is 0x900000 (0b100100000000000000000000) 1138 then the format is: 1140 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1141 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1142 | Hop_Lim | node_id | 1143 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1144 | timestamp fraction | 1145 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1147 0x840000: IOAM-Trace-Type is 0x840000 (0b100001000000000000000000) 1148 then the format is: 1150 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1151 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1152 | Hop_Lim | node_id | 1153 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1154 | namespace specific data | 1155 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1157 0x940000: IOAM-Trace-Type is 0x940000 (0b100101000000000000000000) 1158 then the format is: 1160 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1161 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1162 | Hop_Lim | node_id | 1163 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1164 | timestamp fraction | 1165 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1166 | namespace specific data | 1167 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1169 0x308002: IOAM-Trace-Type is 0x308002 (0b001100001000000000000010) 1170 then the format is: 1172 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1173 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1174 | timestamp seconds | 1175 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1176 | timestamp fraction | 1177 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1178 | Hop_Lim | node_id | 1179 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1180 | node_id(contd) | 1181 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1182 | Length | Schema Id | 1183 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1184 | | 1185 | | 1186 | Opaque data | 1187 ~ ~ 1188 . . 1189 . . 1190 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1192 5.5. IOAM Proof of Transit Option-Type 1194 IOAM Proof of Transit Option-Type is used to support path or service 1195 function chain [RFC7665] verification use cases. Proof-of-transit 1196 leverages mechanisms like Shamir's Secret Sharing Schema (SSSS) 1197 [SSS]. For further information on Proof-of-transit, please refer to 1198 [I-D.ietf-sfc-proof-of-transit]. While details on how the IOAM data 1199 for the Proof-of-transit option is processed at IOAM encapsulating, 1200 decapsulating and transit nodes are outside the scope of the 1201 document, all of these approaches share the need to uniquely identify 1202 a packet as well as iteratively operate on a set of information that 1203 is handed from node to node. Correspondingly, two pieces of 1204 information are added as IOAM-Data-Fields to the packet: 1206 o PktID: Unique identifier for the packet. 1208 o Cumulative: Information which is handed from node to node and 1209 updated by every node according to a verification algorithm. 1211 The IOAM Proof-of-Transit Option-Type consist of a fixed size "IOAM 1212 proof of transit option header" and "IOAM proof of transit option 1213 data fields": 1215 IOAM proof of transit option header: 1217 0 1 2 3 1218 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1219 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1220 | Namespace-ID |IOAM POT Type | IOAM POT flags| 1221 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1223 IOAM proof of transit Option-Type IOAM-Data-Fields MUST be 1224 4-octet aligned: 1226 0 1 2 3 1227 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1228 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1229 | POT Option data field determined by IOAM-POT-Type | 1230 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1232 Namespace-ID: 16-bit identifier of an IOAM-Namespace. The 1233 Namespace-ID value of 0x0000 is defined as the "Default-Namespace- 1234 ID" (see Section 5.3) and MUST be known to all the nodes 1235 implementing IOAM. For any other Namespace-ID value that does not 1236 match any Namespace-ID the node is configured to operate on, the 1237 node MUST NOT change the contents of the IOAM-Data-Fields. 1239 IOAM POT Type: 8-bit identifier of a particular POT variant that 1240 specifies the POT data that is included. This document defines 1241 POT Type 0: 1243 0: POT data is a 16 Octet field to carry data associated to POT 1244 procedures. [I-D.ietf-sfc-proof-of-transit] describes an 1245 implementation of POT and provides details on the data carried 1246 in POT data. 1248 If a node receives an IOAM POT Type value that it does not 1249 understand, the node MUST NOT change, add to, or remove the 1250 contents of the OAM-Data-Fields. 1252 IOAM POT flags: 8-bit. Following flags are defined: 1254 Bit 0 "Profile-to-use" (P-bit) (most significant bit). For IOAM 1255 POT types that use a maximum of two profiles to drive 1256 computation, indicates which POT-profile (see 1257 [I-D.ietf-sfc-proof-of-transit] for details) is used. The two 1258 profiles are numbered 0, 1. This bit conveys whether profile 0 1259 or profile 1 is used to compute the Cumulative. 1261 Bit 1-7 Undefined: These bits are available for assignment, see 1262 Section 8.5. Bits which have not been assigned MUST be set to 1263 zero upon transmission and ignored upon receipt. 1265 POT Option data: Variable-length field. The type of which is 1266 determined by the IOAM-POT-Type. 1268 5.5.1. IOAM Proof of Transit Type 0 1270 IOAM proof of transit option of IOAM POT Type 0: 1272 0 1 2 3 1273 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1274 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1275 | Namespace-ID |IOAM POT Type=0|P|R R R R R R R| 1276 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+<-+ 1277 | PktID | | 1278 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ P 1279 | PktID (contd) | O 1280 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ T 1281 | Cumulative | | 1282 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 1283 | Cumulative (contd) | | 1284 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+<-+ 1286 Namespace-ID: 16-bit identifier of an IOAM-Namespace (see 1287 Section 5.5 above). 1289 IOAM POT Type: 8-bit identifier of a particular POT variant that 1290 specifies the POT data that is included (see Section 5.5 above). 1291 For this case here, IOAM POT Type is set to the value 0. 1293 Bit 0: 1-bit. "Profile-to-use" (P-bit) (most significant bit), see 1294 Section 5.5 above. 1296 Bit 1-7: Undefined (see Section 5.5 above). 1298 PktID: 64-bit packet identifier. 1300 Cumulative: 64-bit Cumulative that is updated at specific nodes by 1301 processing per packet PktID field and configured parameters. 1303 Note: Larger or smaller sizes of "PktID" and "Cumulative" data are 1304 feasible and could be required for certain deployments, e.g., in case 1305 of space constraints in the encapsulation protocols used. Future 1306 documents could introduce different sizes of data for "proof of 1307 transit". 1309 5.6. IOAM Edge-to-Edge Option-Type 1311 The IOAM Edge-to-Edge Option-Type is to carry data that is added by 1312 the IOAM encapsulating node and interpreted by IOAM decapsulating 1313 node. The IOAM transit nodes MAY process the data but MUST NOT 1314 modify it. 1316 The IOAM Edge-to-Edge Option-Type consist of a fixed size "IOAM Edge- 1317 to-Edge Option-Type header" and "IOAM Edge-to-Edge Option-Type data 1318 fields": 1320 IOAM Edge-to-Edge Option-Type header: 1322 0 1 2 3 1323 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1324 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1325 | Namespace-ID | IOAM-E2E-Type | 1326 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1328 IOAM Edge-to-Edge Option-Type IOAM-Data-Fields MUST 1329 be 4-octet aligned: 1331 0 1 2 3 1332 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1333 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1334 | E2E Option data field determined by IOAM-E2E-Type | 1335 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1337 Namespace-ID: 16-bit identifier of an IOAM-Namespace. The 1338 Namespace-ID value of 0x0000 is defined as the "Default-Namespace- 1339 ID" (see Section 5.3) and MUST be known to all the nodes 1340 implementing IOAM. For any other Namespace-ID value that does not 1341 match any Namespace-ID the node is configured to operate on, then 1342 the node MUST NOT change the contents of the IOAM-Data-Fields. 1344 IOAM-E2E-Type: A 16-bit identifier which specifies which data types 1345 are used in the E2E option data. The IOAM-E2E-Type value is a bit 1346 field. The order of packing the E2E option data field elements 1347 follows the bit order of the IOAM-E2E-Type field, as follows: 1349 Bit 0 (Most significant bit) When set indicates presence of a 1350 64-bit sequence number added to a specific "packet group" 1351 which is used to detect packet loss, packet reordering, 1352 or packet duplication within the group. The "packet 1353 group" is deployment dependent and defined at the IOAM 1354 encapsulating node, e.g., by n-tuple based classification 1355 of packets. When this bit is set, "Bit 1" (for 32-bit 1356 sequence number, see below) MUST be zero. 1358 Bit 1 When set indicates presence of a 32-bit sequence number 1359 added to a specific "packet group" which is used to 1360 detect packet loss, packet reordering, or packet 1361 duplication within that group. The "packet group" is 1362 deployment dependent and defined at the IOAM 1363 encapsulating node, e.g., by n-tuple based classification 1364 of packets. When this bit is set, "Bit 0" (for 64-bit 1365 sequence number, see above) MUST be zero. 1367 Bit 2 When set indicates presence of timestamp seconds, 1368 representing the time at which the packet entered the 1369 IOAM domain. Within the IOAM encapsulating node, the 1370 time that the timestamp is retrieved can depend on the 1371 implementation. Some possibilities are: 1) the time at 1372 which the packet was received by the node, 2) the time at 1373 which the packet was transmitted by the node, 3) when a 1374 tunnel encapsulation is used, the point at which the 1375 packet is encapsulated into the tunnel. Each 1376 implementation has to document when the E2E timestamp 1377 that is going to be put in the packet is retrieved. This 1378 4-octet field has three possible formats; based on either 1379 PTP (see e.g., [RFC8877]), NTP [RFC5905], or POSIX 1380 [POSIX]. The three timestamp formats are specified in 1381 Section 6. In all three cases, the Timestamp Seconds 1382 field contains the 32 most significant bits of the 1383 timestamp format that is specified in Section 6. If a 1384 node is not capable of populating this field, it assigns 1385 the value 0xFFFFFFFF. Note that this is a legitimate 1386 value that is valid for 1 second in approximately 136 1387 years; the analyzer has to correlate several packets or 1388 compare the timestamp value to its own time-of-day in 1389 order to detect the error indication. 1391 Bit 3 When set indicates presence of timestamp fraction, 1392 representing the time at which the packet entered the 1393 IOAM domain. This 4-octet field has three possible 1394 formats; based on either PTP (see e.g., [RFC8877]), NTP 1395 [RFC5905], or POSIX [POSIX]. The three timestamp formats 1396 are specified in Section 6. In all three cases, the 1397 Timestamp fraction field contains the 32 least 1398 significant bits of the timestamp format that is 1399 specified in Section 6. If a node is not capable of 1400 populating this field, it assigns the value 0xFFFFFFFF. 1401 Note that this is a legitimate value in the NTP format, 1402 valid for approximately 233 picoseconds in every second. 1403 If the NTP format is used the analyzer has to correlate 1404 several packets in order to detect the error indication. 1406 Bit 4-15 Undefined. An IOAM encapsulating node MUST set the value 1407 of these bits to zero upon transmission and ignore upon 1408 receipt. 1410 E2E Option data: Variable-length field. The type of which is 1411 determined by the IOAM-E2E-Type. 1413 6. Timestamp Formats 1415 The IOAM-Data-Fields include a timestamp field which is represented 1416 in one of three possible timestamp formats. It is assumed that the 1417 management plane is responsible for determining which timestamp 1418 format is used. 1420 6.1. PTP Truncated Timestamp Format 1422 The Precision Time Protocol (PTP) uses an 80-bit timestamp format. 1423 The truncated timestamp format is a 64-bit field, which is the 64 1424 least significant bits of the 80-bit PTP timestamp. The PTP 1425 truncated format is specified in Section 4.3 of [RFC8877], and the 1426 details are presented below for the sake of completeness. 1428 0 1 2 3 1429 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1430 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1431 | Seconds | 1432 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1433 | Nanoseconds | 1434 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1436 Figure 1: PTP Truncated Timestamp Format 1438 Timestamp field format: 1440 Seconds: specifies the integer portion of the number of seconds 1441 since the PTP epoch. 1443 + Size: 32 bits. 1445 + Units: seconds. 1447 Nanoseconds: specifies the fractional portion of the number of 1448 seconds since the PTP epoch. 1450 + Size: 32 bits. 1452 + Units: nanoseconds. The value of this field is in the range 0 1453 to (10^9)-1. 1455 Epoch: 1457 PTP epoch. For details see e.g., [RFC8877]. 1459 Resolution: 1461 The resolution is 1 nanosecond. 1463 Wraparound: 1465 This time format wraps around every 2^32 seconds, which is roughly 1466 136 years. The next wraparound will occur in the year 2106. 1468 Synchronization Aspects: 1470 It is assumed that nodes that run this protocol are synchronized 1471 among themselves. Nodes MAY be synchronized to a global reference 1472 time. Note that if PTP is used for synchronization, the timestamp 1473 MAY be derived from the PTP-synchronized clock, allowing the 1474 timestamp to be measured with respect to the clock of an PTP 1475 Grandmaster clock. 1477 The PTP truncated timestamp format is not affected by leap 1478 seconds. 1480 6.2. NTP 64-bit Timestamp Format 1482 The Network Time Protocol (NTP) [RFC5905] timestamp format is 64 bits 1483 long. This format is specified in Section 4.2.1 of [RFC8877], and 1484 the details are presented below for the sake of completeness. 1486 0 1 2 3 1487 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1488 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1489 | Seconds | 1490 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1491 | Fraction | 1492 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1494 Figure 2: NTP [RFC5905] 64-bit Timestamp Format 1496 Timestamp field format: 1498 Seconds: specifies the integer portion of the number of seconds 1499 since the NTP epoch. 1501 + Size: 32 bits. 1503 + Units: seconds. 1505 Fraction: specifies the fractional portion of the number of 1506 seconds since the NTP epoch. 1508 + Size: 32 bits. 1510 + Units: the unit is 2^(-32) seconds, which is roughly equal to 1511 233 picoseconds. 1513 Epoch: 1515 NTP Epoch. For details see [RFC5905]. 1517 Resolution: 1519 The resolution is 2^(-32) seconds. 1521 Wraparound: 1523 This time format wraps around every 2^32 seconds, which is roughly 1524 136 years. The next wraparound will occur in the year 2036. 1526 Synchronization Aspects: 1528 Nodes that use this timestamp format will typically be 1529 synchronized to UTC using NTP [RFC5905]. Thus, the timestamp MAY 1530 be derived from the NTP-synchronized clock, allowing the timestamp 1531 to be measured with respect to the clock of an NTP server. 1533 The NTP timestamp format is affected by leap seconds; it 1534 represents the number of seconds since the epoch minus the number 1535 of leap seconds that have occurred since the epoch. The value of 1536 a timestamp during or slightly after a leap second could be 1537 temporarily inaccurate. 1539 6.3. POSIX-based Timestamp Format 1541 This timestamp format is based on the POSIX time format [POSIX]. The 1542 detailed specification of the timestamp format used in this document 1543 is presented below. 1545 0 1 2 3 1546 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1547 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1548 | Seconds | 1549 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1550 | Microseconds | 1551 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1553 Figure 3: POSIX-based Timestamp Format 1555 Timestamp field format: 1557 Seconds: specifies the integer portion of the number of seconds 1558 since the POSIX epoch. 1560 + Size: 32 bits. 1562 + Units: seconds. 1564 Microseconds: specifies the fractional portion of the number of 1565 seconds since the POSIX epoch. 1567 + Size: 32 bits. 1569 + Units: the unit is microseconds. The value of this field is in 1570 the range 0 to (10^6)-1. 1572 Epoch: 1574 POSIX epoch. For details, see [POSIX], appendix A.4.16. 1576 Resolution: 1578 The resolution is 1 microsecond. 1580 Wraparound: 1582 This time format wraps around every 2^32 seconds, which is roughly 1583 136 years. The next wraparound will occur in the year 2106. 1585 Synchronization Aspects: 1587 It is assumed that nodes that use this timestamp format run the 1588 Linux operating system, and hence use the POSIX time. In some 1589 cases nodes MAY be synchronized to UTC using a synchronization 1590 mechanism that is outside the scope of this document, such as NTP 1591 [RFC5905]. Thus, the timestamp MAY be derived from the NTP- 1592 synchronized clock, allowing the timestamp to be measured with 1593 respect to the clock of an NTP server. 1595 7. IOAM Data Export 1597 IOAM nodes collect information for packets traversing a domain that 1598 supports IOAM. IOAM decapsulating nodes as well as IOAM transit 1599 nodes can choose to retrieve IOAM information from the packet, 1600 process the information further and export the information using 1601 e.g., IPFIX. The mechanisms and associated data formats for 1602 exporting IOAM data is outside the scope of this document. 1604 Raw data export of IOAM data using IPFIX is discussed in 1605 [I-D.spiegel-ippm-ioam-rawexport]. 1607 8. IANA Considerations 1609 This document requests the following IANA Actions. 1611 IANA is requested to define a registry group named "In-Situ OAM 1612 (IOAM) Protocol Parameters". 1614 This group will include the following registries: 1616 IOAM Option-Type 1618 IOAM Trace-Type 1620 IOAM Trace-Flags 1622 IOAM POT-Type 1624 IOAM POT-Flags 1626 IOAM E2E-Type 1627 IOAM Namespace-ID 1629 New registries in this group can be created via RFC Required process 1630 as per [RFC8126]. 1632 The subsequent sub-sections detail the registries herein contained. 1634 8.1. IOAM Option-Type Registry 1636 This registry defines 128 code points for the IOAM Option-Type field 1637 for identifying IOAM Option-Types as explained in Section 5. The 1638 following code points are defined in this draft: 1640 0 IOAM Pre-allocated Trace Option-Type 1642 1 IOAM Incremental Trace Option-Type 1644 2 IOAM POT Option-Type 1646 3 IOAM E2E Option-Type 1648 4 - 127 are available for assignment via "IETF Review" process as per 1649 [RFC8126]. 1651 New registration requests MUST use the following template: 1653 Name: Name of the newly registered Option-Type. 1655 Code point: Desired value of the requested code point. 1657 Description: Brief description of the newly registered Option-Type. 1659 Reference: Reference to the document that defines the new Option- 1660 Type. 1662 8.2. IOAM Trace-Type Registry 1664 This registry defines code point for each bit in the 24-bit IOAM- 1665 Trace-Type field for Pre-allocated Trace-Option-Type and Incremental 1666 Trace-Option-Type defined in Section 5.4. The meaning of Bits 0 - 11 1667 is defined in this document in Paragraph 5 of Section 5.4.1: 1669 Bit 0 hop_Lim and node_id in short format 1671 Bit 1 ingress_if_id and egress_if_id in short format 1673 Bit 2 timestamp seconds 1674 Bit 3 timestamp fraction 1676 Bit 4 transit delay 1678 Bit 5 namespace specific data in short format 1680 Bit 6 queue depth 1682 Bit 7 checksum complement 1684 Bit 8 hop_Lim and node_id in wide format 1686 Bit 9 ingress_if_id and egress_if_id in wide format 1688 Bit 10 namespace specific data in wide format 1690 Bit 11 buffer occupancy 1692 Bit 22 variable length Opaque State Snapshot 1694 Bit 23 reserved 1696 The meaning for Bits 12 - 21 are available for assignment via "IETF 1697 Review" process as per [RFC8126]. 1699 New registration requests MUST use the following template: 1701 Bit: Desired bit to be allocated in the 24-bit IOAM Trace-Option- 1702 Type field for Pre-allocated Trace-Option-Type and Incremental 1703 Trace-Option-Type. 1705 Description: Brief description of the newly registered bit. 1707 Reference: Reference to the document that defines the new bit. 1709 8.3. IOAM Trace-Flags Registry 1711 This registry defines code points for each bit in the 4 bit flags for 1712 the Pre-allocated trace option and for the Incremental trace option 1713 defined in Section 5.4. The meaning of Bit 0 (the most significant 1714 bit) for trace flags is defined in this document in Paragraph 3 of 1715 Section 5.4.1: 1717 Bit 0 "Overflow" (O-bit) 1719 Bit 1 - 3 are available for assignment via "IETF Review" process as 1720 per [RFC8126]. 1722 New registration requests MUST use the following template: 1724 Bit: Desired bit to be allocated in the 8 bit flags field of the 1725 Pre-allocated Trace-Option-Type and for the Incremental Trace- 1726 Option-Type. 1728 Description: Brief description of the newly registered bit. 1730 Reference: Reference to the document that defines the new bit. 1732 8.4. IOAM POT-Type Registry 1734 This registry defines 256 code points to define IOAM POT Type for 1735 IOAM proof of transit option Section 5.5. The code point value 0 is 1736 defined in this document: 1738 0: 16 Octet POT data 1740 1 - 255 are available for assignment via "IETF Review" process as per 1741 [RFC8126]. 1743 New registration requests MUST use the following template: 1745 Name: Name of the newly registered POT-Type. 1747 Code point: Desired value of the requested code point. 1749 Description: Brief description of the newly registered POT-Type. 1751 Reference: Reference to the document that defines the new POT-Type. 1753 8.5. IOAM POT-Flags Registry 1755 This registry defines code points for each bit in the 8 bit flags for 1756 IOAM POT Option-Type defined in Section 5.5. The meaning of Bit 0 1757 for IOAM POT flags is defined in this document in Section 5.5: 1759 Bit 0 "Profile-to-use" (P-bit) 1761 The meaning for Bits 1 - 7 are available for assignment via "IETF 1762 Review" process as per [RFC8126]. 1764 New registration requests MUST use the following template: 1766 Bit: Desired bit to be allocated in the 8 bit flags field of the 1767 IOAM POT Option-Type. 1769 Description: Brief description of the newly registered bit. 1771 Reference: Reference to the document that defines the new bit. 1773 8.6. IOAM E2E-Type Registry 1775 This registry defines code points for each bit in the 16 bit IOAM- 1776 E2E-Type field for IOAM E2E option Section 5.6. The meaning of Bit 0 1777 - 3 are defined in this document: 1779 Bit 0 64-bit sequence number 1781 Bit 1 32-bit sequence number 1783 Bit 2 timestamp seconds 1785 Bit 3 timestamp fraction 1787 The meaning of Bits 4 - 15 are available for assignment via "IETF 1788 Review" process as per [RFC8126]. 1790 New registration requests MUST use the following template: 1792 Bit: Desired bit to be allocated in the 16 bit IOAM-E2E-Type field. 1794 Description: Brief description of the newly registered bit. 1796 Reference: Reference to the document that defines the new bit. 1798 8.7. IOAM Namespace-ID Registry 1800 IANA is requested to set up an "IOAM Namespace-ID Registry", 1801 containing 16-bit values and following the template for requests 1802 shown below. The meaning of 0x0000 is defined in this document. 1803 IANA is requested to reserve the values 0x0001 to 0x7FFF for private 1804 use (managed by operators), as specified in Section 5.3 of the 1805 current document. Registry entries for the values 0x8000 to 0xFFFF 1806 are to be assigned via the "Expert Review" policy defined in 1807 [RFC8126]. 1809 Upon receiving a new allocation request, a designated expert will 1810 perform the following: 1812 o Review whether the request is complete, i.e., the registration 1813 template has been filled in. The expert will send incomplete 1814 requests back to the requestor. 1816 o Check whether the request is neither a duplicate of nor 1817 conflicting with either an already existing allocation or a 1818 pending allocation. In case of duplicates or conflicts, the 1819 expert will ask the requestor to update the allocation request 1820 accordingly. 1822 o Solicit feedback from relevant working groups and communities to 1823 ensure that the new allocation request has been properly reviewed 1824 and that rough consensus on the request exists. At a minumum, the 1825 expert will solicit feedback from the IPPM working group in the 1826 IETF by posting the request to the ippm@ietf.org mailing list. 1827 The expert will allow for a 3-week review period on the mailing 1828 lists. If the feedback received from the relevant working groups 1829 and communities within the review period indicates rough consensus 1830 on the request, the expert will approve the request and ask IANA 1831 for allocating the new Namespace-ID. In case the expert senses a 1832 lack of consensus from the feedback received, the expert will ask 1833 the requestor to engage with the corresponding working groups and 1834 communities to further review and refine the request. 1836 It is intended that any allocation will be accompanied by a published 1837 RFC. In order to allow for the allocation of code points prior to 1838 the RFC being approved for publication, the designated expert can 1839 approve allocations once it seems clear that an RFC will be 1840 published. 1842 0x0000: default namespace (known to all IOAM nodes) 1844 0x0001 - 0x7FFF: reserved for private use 1846 0x8000 - 0xFFFF: unassigned 1848 New registration requests MUST use the following template: 1850 Name: Name of the newly registered Namespace-ID. 1852 Code point: Desired value of the requested Namespace-ID. 1854 Description: Brief description of the newly registered Namespace-ID. 1856 Reference: Reference to the document that defines the new Namespace- 1857 ID. 1859 Status of the registration: Status can be either "permanent" or 1860 "provisional". Namespace-ID registrations following a successful 1861 expert review will have the status "provisional". Once the RFC, 1862 which defines the new Namespace-ID is published, the status is 1863 changed to "permanent". 1865 9. Management and Deployment Considerations 1867 This document defines the structure and use of IOAM data fields. 1868 This document does not define the encapsulation of IOAM data fields 1869 into different protocols. Management and deployment aspects for IOAM 1870 have to be considered within the context of the protocol IOAM data 1871 fields are encapsulated into and as such, are out of scope for this 1872 document. For a discussion of IOAM deployment, please also refer to 1873 [I-D.brockners-opsawg-ioam-deployment], which outlines a framework 1874 for IOAM deployment and provides best current practices. 1876 10. Security Considerations 1878 As discussed in [RFC7276], a successful attack on an OAM protocol in 1879 general, and specifically on IOAM, can prevent the detection of 1880 failures or anomalies, or create a false illusion of nonexistent 1881 ones. In particular, these threats are applicable by compromising 1882 the integrity of IOAM data, either by maliciously modifying IOAM 1883 options in transit, or by injecting packets with maliciously 1884 generated IOAM options. All nodes in the path of a IOAM carrying 1885 packet can perform such an attack. 1887 The Proof of Transit Option-Type (see Section 5.5) is used for 1888 verifying the path of data packets. The security considerations of 1889 POT are further discussed in [I-D.ietf-sfc-proof-of-transit]. 1891 In case an attacker gains access to several nodes in a network and 1892 would be able to change the system software of these nodes, IOAM data 1893 fields could be misused and repurposed for a use different from what 1894 is specified in this document. One type of misuse is the 1895 implementation of a covert channel between network nodes. 1897 From a confidentiality perspective, although IOAM options are not 1898 expected to contain user data, they can be used for network 1899 reconnaissance, allowing attackers to collect information about 1900 network paths, performance, queue states, buffer occupancy and other 1901 information. Moreover, if IOAM data leaks from the IOAM domain it 1902 could enable reconnaissance beyond the scope of the IOAM domain. 1904 IOAM can be used as a means for implementing Denial of Service (DoS) 1905 attacks, or for amplifying them. For example, a malicious attacker 1906 can add an IOAM header to packets in order to consume the resources 1907 of network devices that take part in IOAM or entities that receive, 1908 collect or analyze the IOAM data. Another example is a packet length 1909 attack, in which an attacker pushes headers associated with IOAM 1910 Option-Types into data packets, causing these packets to be increased 1911 beyond the MTU size, resulting in fragmentation or in packet drops. 1912 In case POT is used, an attacker could corrupt the POT data fields in 1913 the packet, resulting in a verification failure of the POT data, even 1914 if the packet followed the correct path. 1916 Since IOAM options can include timestamps, if network devices use 1917 synchronization protocols then any attack on the time protocol 1918 [RFC7384] can compromise the integrity of the timestamp-related data 1919 fields. 1921 At the management plane, attacks can be set up by misconfiguring or 1922 by maliciously configuring IOAM-enabled nodes in a way that enables 1923 other attacks. IOAM configuration should only managed by authorized 1924 processes or users. 1926 Solutions to ensure the integrity of IOAM data fields are outside the 1927 scope of this document. [I-D.brockners-ippm-ioam-data-integrity] 1928 discusses several methods to ensure the integrity of IOAM data fields 1929 for those deployments that have a need to protect the integrity of 1930 IOAM data fields. 1932 The current document does not define a specific IOAM encapsulation. 1933 It has to be noted that some IOAM encapsulation types can introduce 1934 specific security considerations. A specification that defines an 1935 IOAM encapsulation is expected to address the respective 1936 encapsulation-specific security considerations. 1938 Notably, IOAM is expected to be deployed in limited domains, thus 1939 confining the potential attack vectors to within the limited domain. 1940 A limited administrative domain provides the operator with the means 1941 to select, monitor, and control the access of all the network 1942 devices, making these devices trusted by the operator. Indeed, in 1943 order to limit the scope of threats mentioned above to within the 1944 current limited domain the network operator is expected to enforce 1945 policies that prevent IOAM traffic from leaking outside of the IOAM 1946 domain, and prevent IOAM data from outside the domain to be processed 1947 and used within the domain. 1949 The security considerations of a system that deploys IOAM, much like 1950 any system, has to be reviewed on a per-deployment-scenario basis, 1951 based on a systems-specific threat analysis, which can lead to 1952 specific security solutions that are beyond the scope of the current 1953 document. Specifically, in an IOAM deployment that is not confined 1954 to a single LAN, but spans multiple inter-connected sites (for 1955 example, using an overlay network), the inter-site links can be 1956 secured (e.g., by IPsec) in order to avoid external threats. 1958 IOAM deployment considerations, including approaches to mitigate the 1959 above discussed threads and potential attacks are outside the scope 1960 of this document. IOAM deployment considerations are discussed in 1961 [I-D.brockners-opsawg-ioam-deployment]. 1963 11. Acknowledgements 1965 The authors would like to thank Eric Vyncke, Nalini Elkins, Srihari 1966 Raghavan, Ranganathan T S, Karthik Babu Harichandra Babu, Akshaya 1967 Nadahalli, LJ Wobker, Erik Nordmark, Vengada Prasad Govindan, Andrew 1968 Yourtchenko, Aviv Kfir, Tianran Zhou, Zhenbin (Robin) and Greg Mirsky 1969 for the comments and advice. 1971 This document leverages and builds on top of several concepts 1972 described in [I-D.kitamura-ipv6-record-route]. The authors would 1973 like to acknowledge the work done by the author Hiroshi Kitamura and 1974 people involved in writing it. 1976 The authors would like to gracefully acknowledge useful review and 1977 insightful comments received from Joe Clarke, Al Morton, Tom Herbert, 1978 Haoyu Song, Mickey Spiegel, Roman Danyliw, Benjamin Kaduk, Ian Swett, 1979 Martin Duke, Francesca Palombini, Lars Eggert, Dan Romascanu and 1980 Barak Gafni. 1982 12. References 1984 12.1. Normative References 1986 [POSIX] Institute of Electrical and Electronics Engineers, "IEEE 1987 Std 1003.1-2017 (Revision of IEEE Std 1003.1-2017) - IEEE 1988 Standard for Information Technology - Portable Operating 1989 System Interface (POSIX(TM) Base Specifications, Issue 1990 7)", IEEE Std 1003.1-2017, 2017, 1991 . 1994 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1995 Requirement Levels", BCP 14, RFC 2119, 1996 DOI 10.17487/RFC2119, March 1997, 1997 . 1999 [RFC5905] Mills, D., Martin, J., Ed., Burbank, J., and W. Kasch, 2000 "Network Time Protocol Version 4: Protocol and Algorithms 2001 Specification", RFC 5905, DOI 10.17487/RFC5905, June 2010, 2002 . 2004 [RFC8126] Cotton, M., Leiba, B., and T. Narten, "Guidelines for 2005 Writing an IANA Considerations Section in RFCs", BCP 26, 2006 RFC 8126, DOI 10.17487/RFC8126, June 2017, 2007 . 2009 [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2010 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, 2011 May 2017, . 2013 12.2. Informative References 2015 [I-D.brockners-ippm-ioam-data-integrity] 2016 Brockners, F., Bhandari, S., and T. Mizrahi, "Integrity of 2017 In-situ OAM Data Fields", draft-brockners-ippm-ioam-data- 2018 integrity-01 (work in progress), February 2021. 2020 [I-D.brockners-opsawg-ioam-deployment] 2021 Brockners, F., Bhandari, S., and D. Bernier, "In-situ OAM 2022 Deployment", draft-brockners-opsawg-ioam-deployment-02 2023 (work in progress), September 2020. 2025 [I-D.ietf-nvo3-vxlan-gpe] 2026 (Editor), F. M., (editor), L. K., and U. E. (editor), 2027 "Generic Protocol Extension for VXLAN (VXLAN-GPE)", draft- 2028 ietf-nvo3-vxlan-gpe-11 (work in progress), March 2021. 2030 [I-D.ietf-sfc-proof-of-transit] 2031 Brockners, F., Bhandari, S., Mizrahi, T., Dara, S., and S. 2032 Youell, "Proof of Transit", draft-ietf-sfc-proof-of- 2033 transit-08 (work in progress), November 2020. 2035 [I-D.kitamura-ipv6-record-route] 2036 Kitamura, H., "Record Route for IPv6 (PR6) Hop-by-Hop 2037 Option Extension", draft-kitamura-ipv6-record-route-00 2038 (work in progress), November 2000. 2040 [I-D.spiegel-ippm-ioam-rawexport] 2041 Spiegel, M., Brockners, F., Bhandari, S., and R. 2042 Sivakolundu, "In-situ OAM raw data export with IPFIX", 2043 draft-spiegel-ippm-ioam-rawexport-04 (work in progress), 2044 November 2020. 2046 [RFC7276] Mizrahi, T., Sprecher, N., Bellagamba, E., and Y. 2047 Weingarten, "An Overview of Operations, Administration, 2048 and Maintenance (OAM) Tools", RFC 7276, 2049 DOI 10.17487/RFC7276, June 2014, 2050 . 2052 [RFC7384] Mizrahi, T., "Security Requirements of Time Protocols in 2053 Packet Switched Networks", RFC 7384, DOI 10.17487/RFC7384, 2054 October 2014, . 2056 [RFC7665] Halpern, J., Ed. and C. Pignataro, Ed., "Service Function 2057 Chaining (SFC) Architecture", RFC 7665, 2058 DOI 10.17487/RFC7665, October 2015, 2059 . 2061 [RFC7799] Morton, A., "Active and Passive Metrics and Methods (with 2062 Hybrid Types In-Between)", RFC 7799, DOI 10.17487/RFC7799, 2063 May 2016, . 2065 [RFC7820] Mizrahi, T., "UDP Checksum Complement in the One-Way 2066 Active Measurement Protocol (OWAMP) and Two-Way Active 2067 Measurement Protocol (TWAMP)", RFC 7820, 2068 DOI 10.17487/RFC7820, March 2016, 2069 . 2071 [RFC7821] Mizrahi, T., "UDP Checksum Complement in the Network Time 2072 Protocol (NTP)", RFC 7821, DOI 10.17487/RFC7821, March 2073 2016, . 2075 [RFC8300] Quinn, P., Ed., Elzur, U., Ed., and C. Pignataro, Ed., 2076 "Network Service Header (NSH)", RFC 8300, 2077 DOI 10.17487/RFC8300, January 2018, 2078 . 2080 [RFC8799] Carpenter, B. and B. Liu, "Limited Domains and Internet 2081 Protocols", RFC 8799, DOI 10.17487/RFC8799, July 2020, 2082 . 2084 [RFC8877] Mizrahi, T., Fabini, J., and A. Morton, "Guidelines for 2085 Defining Packet Timestamps", RFC 8877, 2086 DOI 10.17487/RFC8877, September 2020, 2087 . 2089 [RFC8926] Gross, J., Ed., Ganga, I., Ed., and T. Sridhar, Ed., 2090 "Geneve: Generic Network Virtualization Encapsulation", 2091 RFC 8926, DOI 10.17487/RFC8926, November 2020, 2092 . 2094 [SSS] Wikipedia, "Shamir's Secret Sharing", 2095 . 2097 Contributors' Addresses 2099 Carlos Pignataro 2100 Cisco Systems, Inc. 2101 7200-11 Kit Creek Road 2102 Research Triangle Park, NC 27709 2103 United States 2104 Email: cpignata@cisco.com 2106 Mickey Spiegel 2107 Barefoot Networks, an Intel company 2108 4750 Patrick Henry Drive 2109 Santa Clara, CA 95054 2110 US 2112 Email: mickey.spiegel@intel.com 2114 Barak Gafni 2115 Nvidia 2116 350 Oakmead Parkway, Suite 100 2117 Sunnyvale, CA 94085 2118 U.S.A. 2120 Email: gbarak@nvidia.com 2122 Jennifer Lemon 2123 Broadcom 2124 270 Innovation Drive 2125 San Jose, CA 95134 2126 US 2128 Email: jennifer.lemon@broadcom.com 2130 Hannes Gredler 2131 RtBrick Inc. 2133 Email: hannes@rtbrick.com 2135 John Leddy 2136 United States 2138 Email: john@leddy.net 2140 Stephen Youell 2141 JP Morgan Chase 2142 25 Bank Street 2143 London E14 5JP 2144 United Kingdom 2145 Email: stephen.youell@jpmorgan.com 2147 David Mozes 2149 Email: mosesster@gmail.com 2151 Petr Lapukhov 2152 Facebook 2153 1 Hacker Way 2154 Menlo Park, CA 94025 2155 US 2157 Email: petr@fb.com 2159 Remy Chang 2160 Barefoot Networks 2161 4750 Patrick Henry Drive 2162 Santa Clara, CA 95054 2163 US 2165 Email: remy@barefootnetworks.com 2167 Daniel Bernier 2168 Bell Canada 2169 Canada 2171 Email: daniel.bernier@bell.ca 2173 Authors' Addresses 2175 Frank Brockners (editor) 2176 Cisco Systems, Inc. 2177 Hansaallee 249, 3rd Floor 2178 DUESSELDORF, NORDRHEIN-WESTFALEN 40549 2179 Germany 2181 Email: fbrockne@cisco.com 2182 Shwetha Bhandari (editor) 2183 Thoughtspot 2184 3rd Floor, Indiqube Orion, 24th Main Rd, Garden Layout, HSR Layout 2185 Bangalore, KARNATAKA 560 102 2186 India 2188 Email: shwetha.bhandari@thoughtspot.com 2190 Tal Mizrahi (editor) 2191 Huawei 2192 8-2 Matam 2193 Haifa 3190501 2194 Israel 2196 Email: tal.mizrahi.phd@gmail.com