idnits 2.17.1 draft-ietf-ippm-ioam-data-07.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (September 11, 2019) is 1689 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) -- Looks like a reference, but probably isn't: '0' on line 688 -- Looks like a reference, but probably isn't: '1' on line 544 -- Possible downref: Non-RFC (?) normative reference: ref. 'IEEE1588v2' -- Possible downref: Non-RFC (?) normative reference: ref. 'POSIX' == Outdated reference: A later version (-09) exists of draft-ietf-ntp-packet-timestamps-07 == Outdated reference: A later version (-16) exists of draft-ietf-nvo3-geneve-13 == Outdated reference: A later version (-13) exists of draft-ietf-nvo3-vxlan-gpe-07 == Outdated reference: A later version (-08) exists of draft-ietf-sfc-proof-of-transit-02 == Outdated reference: A later version (-07) exists of draft-spiegel-ippm-ioam-rawexport-02 Summary: 0 errors (**), 0 flaws (~~), 6 warnings (==), 5 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 ippm F. Brockners 3 Internet-Draft S. Bhandari 4 Intended status: Standards Track C. Pignataro 5 Expires: March 14, 2020 Cisco 6 H. Gredler 7 RtBrick Inc. 8 J. Leddy 10 S. Youell 11 JPMC 12 T. Mizrahi 13 Huawei Network.IO Innovation Lab 14 D. Mozes 16 P. Lapukhov 17 Facebook 18 R. Chang 19 Barefoot Networks 20 D. Bernier 21 Bell Canada 22 J. Lemon 23 Broadcom 24 September 11, 2019 26 Data Fields for In-situ OAM 27 draft-ietf-ippm-ioam-data-07 29 Abstract 31 In-situ Operations, Administration, and Maintenance (IOAM) records 32 operational and telemetry information in the packet while the packet 33 traverses a path between two points in the network. This document 34 discusses the data fields and associated data types for in-situ OAM. 35 In-situ OAM data fields can be embedded into a variety of transports 36 such as NSH, Segment Routing, Geneve, native IPv6 (via extension 37 header), or IPv4. In-situ OAM can be used to complement OAM 38 mechanisms based on e.g. ICMP or other types of probe packets. 40 Status of This Memo 42 This Internet-Draft is submitted in full conformance with the 43 provisions of BCP 78 and BCP 79. 45 Internet-Drafts are working documents of the Internet Engineering 46 Task Force (IETF). Note that other groups may also distribute 47 working documents as Internet-Drafts. The list of current Internet- 48 Drafts is at https://datatracker.ietf.org/drafts/current/. 50 Internet-Drafts are draft documents valid for a maximum of six months 51 and may be updated, replaced, or obsoleted by other documents at any 52 time. It is inappropriate to use Internet-Drafts as reference 53 material or to cite them other than as "work in progress." 55 This Internet-Draft will expire on March 14, 2020. 57 Copyright Notice 59 Copyright (c) 2019 IETF Trust and the persons identified as the 60 document authors. All rights reserved. 62 This document is subject to BCP 78 and the IETF Trust's Legal 63 Provisions Relating to IETF Documents 64 (https://trustee.ietf.org/license-info) in effect on the date of 65 publication of this document. Please review these documents 66 carefully, as they describe your rights and restrictions with respect 67 to this document. Code Components extracted from this document must 68 include Simplified BSD License text as described in Section 4.e of 69 the Trust Legal Provisions and are provided without warranty as 70 described in the Simplified BSD License. 72 Table of Contents 74 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 75 2. Conventions . . . . . . . . . . . . . . . . . . . . . . . . . 3 76 3. Scope, Applicability, and Assumptions . . . . . . . . . . . . 4 77 4. IOAM Data-Fields, Types, Nodes . . . . . . . . . . . . . . . 5 78 4.1. IOAM Data-Fields and Option-Types . . . . . . . . . . . . 5 79 4.2. IOAM-Domains and types of IOAM Nodes . . . . . . . . . . 6 80 4.3. IOAM-Namespaces . . . . . . . . . . . . . . . . . . . . . 7 81 4.4. IOAM Trace Option-Types . . . . . . . . . . . . . . . . . 9 82 4.4.1. Pre-allocated and Incremental Trace Option-Types . . 11 83 4.4.2. IOAM node data fields and associated formats . . . . 15 84 4.4.3. Examples of IOAM node data . . . . . . . . . . . . . 21 85 4.5. IOAM Proof of Transit Option-Type . . . . . . . . . . . . 23 86 4.5.1. IOAM Proof of Transit Type 0 . . . . . . . . . . . . 25 87 4.6. IOAM Edge-to-Edge Option-Type . . . . . . . . . . . . . . 26 88 5. Timestamp Formats . . . . . . . . . . . . . . . . . . . . . . 28 89 5.1. PTP Truncated Timestamp Format . . . . . . . . . . . . . 28 90 5.2. NTP 64-bit Timestamp Format . . . . . . . . . . . . . . . 29 91 5.3. POSIX-based Timestamp Format . . . . . . . . . . . . . . 30 92 6. IOAM Data Export . . . . . . . . . . . . . . . . . . . . . . 32 93 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 32 94 7.1. Creation of a new In-Situ OAM Protocol Parameters 95 Registry (IOAM) Protocol Parameters IANA registry . . . . 32 96 7.2. IOAM Option-Type Registry . . . . . . . . . . . . . . . . 33 97 7.3. IOAM Trace-Type Registry . . . . . . . . . . . . . . . . 33 98 7.4. IOAM Trace-Flags Registry . . . . . . . . . . . . . . . . 34 99 7.5. IOAM POT-Type Registry . . . . . . . . . . . . . . . . . 34 100 7.6. IOAM POT-Flags Registry . . . . . . . . . . . . . . . . . 34 101 7.7. IOAM E2E-Type Registry . . . . . . . . . . . . . . . . . 35 102 7.8. IOAM Namespace-ID Registry . . . . . . . . . . . . . . . 35 103 8. Security Considerations . . . . . . . . . . . . . . . . . . . 35 104 9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 36 105 10. References . . . . . . . . . . . . . . . . . . . . . . . . . 37 106 10.1. Normative References . . . . . . . . . . . . . . . . . . 37 107 10.2. Informative References . . . . . . . . . . . . . . . . . 37 108 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 39 110 1. Introduction 112 This document defines data fields for "in-situ" Operations, 113 Administration, and Maintenance (IOAM). In-situ OAM records OAM 114 information within the packet while the packet traverses a particular 115 network domain. The term "in-situ" refers to the fact that the OAM 116 data is added to the data packets rather than is being sent within 117 packets specifically dedicated to OAM. IOAM is to complement 118 mechanisms such as Ping or Traceroute, or more recent active probing 119 mechanisms as described in [I-D.lapukhov-dataplane-probe]. In terms 120 of "active" or "passive" OAM, "in-situ" OAM can be considered a 121 hybrid OAM type. "In-situ" mechanisms do not require extra packets 122 to be sent. IOAM adds information to the already available data 123 packets and therefore cannot be considered passive. In terms of the 124 classification given in [RFC7799] IOAM could be portrayed as Hybrid 125 Type 1. IOAM mechanisms can be leveraged where mechanisms using e.g. 126 ICMP do not apply or do not offer the desired results, such as 127 proving that a certain traffic flow takes a pre-defined path, SLA 128 verification for the live data traffic, detailed statistics on 129 traffic distribution paths in networks that distribute traffic across 130 multiple paths, or scenarios in which probe traffic is potentially 131 handled differently from regular data traffic by the network devices. 133 2. Conventions 135 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 136 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 137 document are to be interpreted as described in [RFC2119]. 139 Abbreviations used in this document: 141 E2E Edge to Edge 143 Geneve: Generic Network Virtualization Encapsulation 144 [I-D.ietf-nvo3-geneve] 146 IOAM: In-situ Operations, Administration, and Maintenance 148 MTU: Maximum Transmit Unit 150 NSH: Network Service Header [RFC8300] 152 OAM: Operations, Administration, and Maintenance 154 POT: Proof of Transit 156 SFC: Service Function Chain 158 SID: Segment Identifier 160 SR: Segment Routing 162 VXLAN-GPE: Virtual eXtensible Local Area Network, Generic Protocol 163 Extension [I-D.ietf-nvo3-vxlan-gpe] 165 3. Scope, Applicability, and Assumptions 167 IOAM deployment assumes a set of constraints, requirements, and 168 guiding principles which are described in this section. 170 Scope: This document defines the data fields and associated data 171 types for in-situ OAM. The in-situ OAM data field can be transported 172 by a variety of transport protocols, including NSH, Segment Routing, 173 Geneve, IPv6, or IPv4. Specification details for these different 174 transport protocols are outside the scope of this document. 176 Deployment domain (or scope) of in-situ OAM deployment: IOAM is a 177 network domain focused feature, with "network domain" being a set of 178 network devices or entities within a single administration. For 179 example, a network domain can include an enterprise campus using 180 physical connections between devices or an overlay network using 181 virtual connections / tunnels for connectivity between said devices. 182 A network domain is defined by its perimeter or edge. Designers of 183 protocol encapsulations for IOAM must specify mechanisms to ensure 184 that IOAM data stays within an IOAM domain. In addition, the 185 operator of such a domain is expected to put provisions in place to 186 ensure that IOAM data does not leak beyond the edge of an IOAM 187 domain, e.g. using for example packet filtering methods. The 188 operator should consider the potential operational impact of IOAM to 189 mechanisms such as ECMP processing (e.g. load-balancing schemes 190 based on packet length could be impacted by the increased packet size 191 due to IOAM), path MTU (i.e. ensure that the MTU of all links within 192 a domain is sufficiently large to support the increased packet size 193 due to IOAM) and ICMP message handling (i.e. in case of a native IPv6 194 transport, IOAM support for ICMPv6 Echo Request/Reply could desired 195 which would translate into ICMPv6 extensions to enable IOAM-Data- 196 Fields to be copied from an Echo Request message to an Echo Reply 197 message). 199 IOAM control points: IOAM-Data-Fields are added to or removed from 200 the live user traffic by the devices which form the edge of a domain. 201 Devices which form an IOAM-Domain can add, update or remove IOAM- 202 Data-Fields. Edge devices of an IOAM-Domain can be hosts or network 203 devices. 205 Traffic-sets that IOAM is applied to: IOAM can be deployed on all or 206 only on subsets of the live user traffic. It SHOULD be possible to 207 enable IOAM on a selected set of traffic (e.g., per interface, based 208 on an access control list or flow specification defining a specific 209 set of traffic, etc.) The selected set of traffic can also be all 210 traffic. 212 Encapsulation independence: Data formats for IOAM SHOULD be defined 213 in a transport-independent manner. IOAM applies to a variety of 214 encapsulating protocols. A definition of how IOAM-Data-Fields are 215 encapsulated into "parent" protocols, like NSH or IPv6 is outside the 216 scope of this document. 218 Layering: If several encapsulation protocols (e.g., in case of 219 tunneling) are stacked on top of each other, IOAM-Data-Fields could 220 be present at multiple layers. The behavior follows the ships-in- 221 the-night model, i.e. IOAM-Data-Fields in one layer are independent 222 from IOAM-Data-Fields in another layer. Layering allows operators to 223 instrument the protocol layer they want to measure. The different 224 layers could, but do not have to share the same IOAM encapsulation 225 mechanisms. 227 IOAM implementation: The definition of the IOAM-Data-Fields take the 228 specifics of devices with hardware data-plane and software data-plane 229 into account. 231 4. IOAM Data-Fields, Types, Nodes 233 This section details IOAM-related nomenclature and describes data 234 types such as IOAM-Data-Fields, IOAM-Types, IOAM-Namespaces as well 235 as the different types of IOAM nodes. 237 4.1. IOAM Data-Fields and Option-Types 239 An IOAM-Data-Field is a set of bits with a defined format and 240 meaning, which can be stored at a certain place in a packet for the 241 purpose of IOAM. 243 To accommodate the different uses of IOAM, IOAM-Data-Fields fall into 244 different categories. In IOAM these categories are referred to as 245 IOAM-Option-Types. A common registry is maintained for IOAM-Option- 246 Types, see Section 7.2 for details. Corresponding to these IOAM- 247 Option-Types, different IOAM-Data-Fields are defined. IOAM-Data- 248 Fields can be encapsulated into a variety of protocols, such as NSH, 249 Geneve, IPv6, etc. The definition of how IOAM-Data-Fields are 250 encapsulated into other protocols is outside the scope of this 251 document. 253 This document defines four IOAM-Option-Types: 255 o Pre-allocated Trace Option-Type 257 o Incremental Trace Option-Type 259 o Proof of Transit (POT) Option-Type 261 o Edge-to-Edge (E2E) Option-Type 263 4.2. IOAM-Domains and types of IOAM Nodes 265 IOAM is expected to be deployed in a specific domain. The part of 266 the network which employs IOAM is referred to as the "IOAM-Domain". 267 One or more IOAM-Option-Types are added to a packet upon entering the 268 IOAM-Domain and are removed from the packet when exiting the domain. 269 Within the IOAM-Domain, the IOAM-Data-Fields MAY be updated by 270 network nodes that the packet traverses. An IOAM-Domain consists of 271 "IOAM encapsulating nodes", "IOAM decapsulating nodes" and "IOAM 272 transit nodes". The role of a node (i.e. encapsulating, transit, 273 decapsulating) is defined within an IOAM-Namespace (see below). A 274 node can have different roles in different IOAM-Namespaces. 276 A device which adds at least one IOAM-Option-Type to the packet is 277 called the "IOAM encapsulating node", whereas a device which removes 278 an IOAM-Option-Type is referred to as the "IOAM decapsulating node". 279 Nodes within the domain which are aware of IOAM data and read and/or 280 write or process the IOAM data are called "IOAM transit nodes". IOAM 281 nodes which add or remove the IOAM-Data-Fields can also update the 282 IOAM-Data-Fields at the same time. Or in other words, IOAM 283 encapsulating or decapsulating nodes can also serve as IOAM transit 284 nodes at the same time. Note that not every node in an IOAM domain 285 needs to be an IOAM transit node. For example, a deployment might 286 require that packets traverse a set of firewalls which support IOAM. 287 In that case, only the set of firewall nodes would be IOAM transit 288 nodes rather than all nodes. 290 An "IOAM encapsulating node" incorporates one or more IOAM-Option- 291 Types (from the list of IOAM-Types, see Section 7.2) into packets 292 that IOAM is enabled for. If IOAM is enabled for a selected subset 293 of the traffic, the IOAM encapsulating node is responsible for 294 applying the IOAM functionality to the selected subset. 296 An "IOAM transit node" updates one or more of the IOAM-Data-Fields. 297 If both the Pre-allocated and the Incremental Trace Option-Types are 298 present in the packet, each IOAM transit node will update at most one 299 of these Option-Types. A transit node MUST NOT add new IOAM-Option- 300 Types to a packet, and MUST NOT change the IOAM-Data-Fields of an 301 IOAM Edge-to-Edge Option-Type. 303 An "IOAM decapsulating node" removes IOAM-Option-Type(s) from 304 packets. 306 4.3. IOAM-Namespaces 308 A subset or all of the IOAM-Option-Types and their corresponding 309 IOAM-Data-Fields can be associated to an IOAM-Namespace. IOAM- 310 Namespaces add further context to IOAM-Option-Types and associated 311 IOAM-Data-Fields. Any IOAM-Namespace MUST interpret the IOAM-Option- 312 Types and associated IOAM-Data-Fields per the definition in this 313 document. IOAM-Namespaces group nodes to support different 314 deployment approaches of IOAM (see a few example use-cases below) as 315 well as resolve issues which can occur due to IOAM-Data-Fields not 316 being globally unique (e.g. IOAM node identifiers do not have to be 317 globally unique). IOAM-Data-Fields significance is always within a 318 particular IOAM-Namespace. 320 An IOAM-Namespace is identified by a 16-bit namespace identifier 321 (Namespace-ID). IOAM-Namespace identifiers MUST be present and 322 populated in all IOAM-Option-Types. The Namespace-ID value is 323 divided into two sub-ranges: 325 o An operator-assigned range from 0x0001 to 0x7FFF 327 o An IANA-assigned range from 0x8000 to 0xFFFF 329 The IANA-assigned range is intended to allow future extensions to 330 have new and interoperable IOAM functionality, while the operator- 331 assigned range is intended to be domain specific, and managed by the 332 network operator. The Namespace-ID value of 0x0000 is default and 333 known to all the nodes implementing IOAM. 335 Namespace identifiers allow devices which are IOAM capable to 336 determine: 338 o whether IOAM-Option-Type(s) need to be processed by a device: If 339 the Namespace-ID contained in a packet does not match any 340 Namespace-ID the node is configured to operate on, then the node 341 MUST NOT change the contents of the IOAM-Data-Fields. 343 o which IOAM-Option-Type needs to be processed/updated in case there 344 are multiple IOAM-Option-Types present in the packet. Multiple 345 IOAM-Option-Types can be present in a packet in case of 346 overlapping IOAM-Domains or in case of a layered IOAM deployment. 348 o whether IOAM-Option-Type(s) should be removed from the packet, 349 e.g. at a domain edge or domain boundary. 351 IOAM-Namespaces support several different uses: 353 o IOAM-Namespaces can be used by an operator to distinguish 354 different operational domains. Devices at domain edges can filter 355 on Namespace-IDs to provide for proper IOAM-Domain isolation. 357 o IOAM-Namespaces provide additional context for IOAM-Data-Fields 358 and thus ensure that IOAM-Data-Fields are unique and can be 359 interpreted properly by management stations or network 360 controllers. While, for example, the node identifier field 361 (node_id, see below) does not need to be unique in a deployment 362 (e.g. an operator may wish to use different node identifiers for 363 different IOAM layers, even within the same device; or node 364 identifiers might not be unique for other organizational reasons, 365 such as after a merger of two formerly separated organizations), 366 the combination of node_id and Namespace-ID will always be unique. 367 Similarly, IOAM-Namespaces can be used to define how certain IOAM- 368 Data-Fields are interpreted: IOAM offers three different timestamp 369 format options. The Namespace-ID can be used to determine the 370 timestamp format. IOAM-Data-Fields (e.g. buffer occupancy) which 371 do not have a unit associated are to be interpreted within the 372 context of a IOAM-Namespace. 374 o IOAM-Namespaces can be used to identify different sets of devices 375 (e.g., different types of devices) in a deployment: If an operator 376 desires to insert different IOAM-Data-Fields based on the device, 377 the devices could be grouped into multiple IOAM-Namespaces. This 378 could be due to the fact that the IOAM feature set differs between 379 different sets of devices, or it could be for reasons of optimized 380 space usage in the packet header. It could also stem from 381 hardware or operational limitations on the size of the trace data 382 that can be added and processed, preventing collection of a full 383 trace for a flow. 385 * Assigning different IOAM Namespace-IDs to different sets of 386 nodes or network partitions and using the Namespace-ID as a 387 selector at the IOAM encapsulating node, a full trace for a 388 flow could be collected and constructed via partial traces in 389 different packets of the same flow. Example: An operator could 390 choose to group the devices of a domain into two IOAM- 391 Namespaces, in a way that at average, only every second hop 392 would be recorded by any device. To retrieve a full view of 393 the deployment, the captured IOAM-Data-Fields of the two IOAM- 394 Namespaces need to be correlated. 396 * Assigning different IOAM Namespace-IDs to different sets of 397 nodes or network partitions and using a separate instance of an 398 IOAM-Option-Type for each Namespace-ID, a full trace for a flow 399 could be collected and constructed via partial traces from each 400 IOAM-Option-Type in each of the packets in the flow. Example: 401 An operator could choose to group the devices of a domain into 402 two IOAM-Namespaces, in a way that each IOAM-Namespace is 403 represented by one of two IOAM-Option-Types in the packet. 404 Each node would record data only for the IOAM-Namespace that it 405 belongs to, ignoring the other IOAM-Option-Type with a IOAM- 406 Namespace to which it doesn't belong. To retrieve a full view 407 of the deployment, the captured IOAM-Data-Fields of the two 408 IOAM-Namespaces need to be correlated. 410 4.4. IOAM Trace Option-Types 412 "IOAM tracing data" is expected to be collected at every IOAM transit 413 node that a packet traverses to ensure visibility into the entire 414 path a packet takes within an IOAM-Domain. I.e., in a typical 415 deployment all nodes in an IOAM-Domain would participate in IOAM and 416 thus be IOAM transit nodes, IOAM encapsulating or IOAM decapsulating 417 nodes. If not all nodes within a domain are IOAM capable, IOAM 418 tracing information (i.e., node data, see below) will only be 419 collected on those nodes which are IOAM capable. Nodes which are not 420 IOAM capable will forward the packet without any changes to the IOAM- 421 Data-Fields. The maximum number of hops and the minimum path MTU of 422 the IOAM domain is assumed to be known. 424 To optimize hardware and software implementations IOAM tracing is 425 defined as two separate options. Any deployment MAY choose to 426 configure and support one or both of the following options. 428 Pre-allocated Trace-Option: This trace option is defined as a 429 container of node data fields (see below) with pre-allocated space 430 for each node to populate its information. This option is useful 431 for implementations where it is efficient to allocate the space 432 once and index into the array to populate the data during transit 433 (e.g., software forwarders often fall into this class). The IOAM 434 encapsulating node allocates space for Pre-allocated Trace Option- 435 Type in the packet and sets corresponding fields in this IOAM- 436 Option-Type. The IOAM encapsulating node allocates an array which 437 is used to store operational data retrieved from every node while 438 the packet traverses the domain. IOAM transit nodes update the 439 content of the array, and possibly update the checksums of outer 440 headers. A pointer which is part of the IOAM trace data, points 441 to the next empty slot in the array. An IOAM transit node that 442 updates the content of the pre-allocated option also updates the 443 value of the pointer, which specifies where the next IOAM transit 444 node fills in its data.The "node data list" array (see below) in 445 the packet is populated iteratively as the packet traverses the 446 network, starting with the last entry of the array, i.e., "node 447 data list [n]" is the first entry to be populated, "node data list 448 [n-1]" is the second one, etc. 450 Incremental Trace-Option: This trace option is defined as a 451 container of node data fields where each node allocates and pushes 452 its node data immediately following the option header. This type 453 of trace recording is useful for some of the hardware 454 implementations as it eliminates the need for the transit network 455 elements to read the full array in the option and allows for 456 arbitrarily long packets as the MTU allows. The IOAM 457 encapsulating node allocates space for the Incremental Trace 458 Option-Type. Based on operational state and configuration, the 459 IOAM encapsulating node sets the fields in the Option-Type that 460 control what IOAM-Data-Fields should be collected and how large 461 the node data list can grow. IOAM transit nodes push their node 462 data to the node data list, decrease the remaining length 463 available to subsequent nodes and adjust the lengths and possibly 464 checksums in outer headers. 466 A particular implementation of IOAM MAY choose to support only one of 467 the two trace option types. In the event that both options are 468 utilized at the same time, the Incremental Trace-Option MUST be 469 placed before the Pre-allocated Trace-Option. Deployments which mix 470 devices which either the Incremental Trace-Option or the Pre- 471 allocated Trace-Option could result in both Option-Types being 472 present in a packet. Given that the operator knows which equipment 473 is deployed in a particular IOAM, the operator will decide by means 474 of configuration which type(s) of trace options will be used for a 475 particular domain. 477 Every node data entry holds information for a particular IOAM transit 478 node that is traversed by a packet. The IOAM decapsulating node 479 removes the IOAM-Option-Type(s) and processes and/or exports the 480 associated data. IOAM-Data-Fields are defined in the context of an 481 IOAM-Namespace. This allows for a namespace-specific definition and 482 interpretation. For example: In one case an interface-id could point 483 to a physical interface (e.g., to understand which physical interface 484 of an aggregated link is used when receiving or transmitting a 485 packet) whereas in another case it could refer to a logical interface 486 (e.g., in case of tunnels). 488 IOAM tracing can collect the following types of information: 490 o Identification of the IOAM node. An IOAM node identifier can 491 match to a device identifier or a particular control point or 492 subsystem within a device. 494 o Identification of the interface that a packet was received on, 495 i.e. ingress interface. 497 o Identification of the interface that a packet was sent out on, 498 i.e. egress interface. 500 o Time of day when the packet was processed by the node as well as 501 the transit delay. Different definitions of processing time are 502 feasible and expected, though it is important that all devices of 503 an in-situ OAM domain follow the same definition. 505 o Generic data: Format-free information where syntax and semantic of 506 the information is defined by the operator in a specific 507 deployment. For a specific IOAM-Namespace, all IOAM nodes should 508 interpret the generic data the same way. Examples for generic 509 IOAM data include geo-location information (location of the node 510 at the time the packet was processed), buffer queue fill level or 511 cache fill level at the time the packet was processed, or even a 512 battery charge level. 514 o Information to detect whether IOAM trace data was added at every 515 hop or whether certain hops in the domain weren't IOAM transit 516 nodes. 518 4.4.1. Pre-allocated and Incremental Trace Option-Types 520 The IOAM Pre-allocated Trace-Option and the IOAM Incremental Trace- 521 Option have similar formats. Except where noted below, the internal 522 formats and fields of the two trace options are identical. Both 523 Trace-Options consist of a fixed size "trace option header" and a 524 variable data space to store gathered data, the "node data list": 526 Pre-allocated and incremental trace option headers: 528 0 1 2 3 529 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 530 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 531 | Namespace-ID |NodeLen | Flags | RemainingLen| 532 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 533 | IOAM-Trace-Type | Reserved | 534 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 536 The trace option data MUST be 4-octet aligned: 538 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+<-+ 539 | | | 540 | node data list [0] | | 541 | | | 542 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ D 543 | | a 544 | node data list [1] | t 545 | | a 546 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 547 ~ ... ~ S 548 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ p 549 | | a 550 | node data list [n-1] | c 551 | | e 552 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 553 | | | 554 | node data list [n] | | 555 | | | 556 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+<-+ 558 Namespace-ID: 16-bit identifier of an IOAM-Namespace. The 559 Namespace-ID value of 0x0000 is defined as the default value and 560 MUST be known to all the nodes implementing IOAM. For any other 561 Namespace-ID value that does not match any Namespace-ID the node 562 is configured to operate on, the node MUST NOT change the contents 563 of the IOAM-Data-Fields. 565 NodeLen: 5-bit unsigned integer. This field specifies the length of 566 data added by each node in multiples of 4-octets, excluding the 567 length of the "Opaque State Snapshot" field. 569 If IOAM-Trace-Type bit 22 is not set, then NodeLen specifies the 570 actual length added by each node. If IOAM-Trace-Type bit 22 is 571 set, then the actual length added by a node would be (NodeLen + 572 Opaque Data Length). 574 For example, if 3 IOAM-Trace-Type bits are set and none of them 575 are wide, then NodeLen would be 3. If 3 IOAM-Trace-Type bits are 576 set and 2 of them are wide, then NodeLen would be 5. 578 An IOAM encapsulating node must set NodeLen. 580 A node receiving an IOAM Pre-allocated or Incremental Trace-Option 581 may rely on the NodeLen value, or it may ignore the NodeLen value 582 and calculate the node length from the IOAM-Trace-Type bits (see 583 below). 585 Flags 4-bit field. Flags are allocated by IANA, as specified in 586 Section 7.4. This document allocates a single flag as follows: 588 Bit 0 "Overflow" (O-bit) (most significant bit). This bit is set 589 by the network element if there are not enough octets left to 590 record node data, no field is added and the overflow "O-bit" 591 must be set to "1" in the IOAM-Trace-Option header. This is 592 useful for transit nodes to ignore further processing of the 593 option. 595 RemainingLen: 7-bit unsigned integer. This field specifies the data 596 space in multiples of 4-octets remaining for recording the node 597 data, before the node data list is considered to have overflowed. 598 When RemainingLen reaches 0, nodes are no longer allowed to add 599 node data. Given that the sender knows the minimum path MTU, the 600 sender MAY set the initial value of RemainingLen according to the 601 number of node data bytes allowed before exceeding the MTU. 602 Subsequent nodes can carry out a simple comparison between 603 RemainingLen and NodeLen, along with the length of the "Opaque 604 State Snapshot" if applicable, to determine whether or not data 605 can be added by this node. When node data is added, the node MUST 606 decrease RemainingLen by the amount of data added. In the pre- 607 allocated trace option, RemainingLength is used to derive the 608 offset in data space to record the node data element. 609 Specifically, the recording of the node data element would start 610 from RemainingLen - NodeLen - sizeof(opaque snapshot) in 4 octet 611 units. 613 IOAM-Trace-Type: A 24-bit identifier which specifies which data 614 types are used in this node data list. 616 The IOAM-Trace-Type value is a bit field. The following bits are 617 defined in this document, with details on each bit described in 618 the Section 4.4.2. The order of packing the data fields in each 619 node data element follows the bit order of the IOAM-Trace-Type 620 field, as follows: 622 Bit 0 (Most significant bit) When set indicates presence of 623 Hop_Lim and node_id in the node data. 625 Bit 1 When set indicates presence of ingress_if_id and 626 egress_if_id (short format) in the node data. 628 Bit 2 When set indicates presence of timestamp seconds in the 629 node data. 631 Bit 3 When set indicates presence of timestamp subseconds in 632 the node data. 634 Bit 4 When set indicates presence of transit delay in the node 635 data. 637 Bit 5 When set indicates presence of IOAM-Namespace specific 638 data (short format) in the node data. 640 Bit 6 When set indicates presence of queue depth in the node 641 data. 643 Bit 7 When set indicates presence of the Checksum Complement 644 node data. 646 Bit 8 When set indicates presence of Hop_Lim and node_id in 647 wide format in the node data. 649 Bit 9 When set indicates presence of ingress_if_id and 650 egress_if_id in wide format in the node data. 652 Bit 10 When set indicates presence of IOAM-Namespace specific 653 data in wide format in the node data. 655 Bit 11 When set indicates presence of buffer occupancy in the 656 node data. 658 Bit 12-21 Undefined. An IOAM encapsulating node MUST set the 659 value of each of these bits to 0. If an IOAM transit 660 node receives a packet with one or more of these bits set 661 to 1, it must either: 663 1. Add corresponding node data filled with the reserved 664 value 0xFFFFFFFF, after the node data fields for the 665 IOAM-Trace-Type bits defined above, such that the 666 total node data added by this node in units of 667 4-octets is equal to NodeLen, or 669 2. Not add any node data fields to the packet, even for 670 the IOAM-Trace-Type bits defined above. 672 Bit 22 When set indicates presence of variable length Opaque 673 State Snapshot field. 675 Bit 23 Reserved: Must be set to zero upon transmission and 676 ignored upon receipt. 678 Section 4.4.2 describes the IOAM-Data-Types and their formats. 679 Within an IOAM-Domain possible combinations of these bits making 680 the IOAM-Trace-Type can be restricted by configuration knobs. 682 Reserved: 8-bits. Must be zero. 684 Node data List [n]: Variable-length field. The type of which is 685 determined by the IOAM-Trace-Type bit representing the n-th node 686 data in the node data list. The node data list is encoded 687 starting from the last node data of the path. The first element 688 of the node data list (node data list [0]) contains the last node 689 of the path while the last node data of the node data list (node 690 data list[n]) contains the first node data of the path traced. 691 Populating the node data list in this way ensures that the order 692 of node data list is the same for incremental and pre-allocated 693 trace options. In the pre-allocated trace option, the index 694 contained in RemainingLen identifies the offset for current active 695 node data to be populated. 697 4.4.2. IOAM node data fields and associated formats 699 All the IOAM-Data-Fields MUST be 4-octet aligned. If a node which is 700 supposed to update an IOAM-Data-Field is not capable of populating 701 the value of a field set in the IOAM-Trace-Type, the field value MUST 702 be set to 0xFFFFFFFF for 4-octet fields or 0xFFFFFFFFFFFFFFFF for 703 8-octet fields, indicating that the value is not populated, except 704 when explicitly specified in the field description below. 706 Some IOAM-Data-Fields defined below, such as interface identifiers or 707 IOAM-Namespace specific data, are defined in both "short format" as 708 well as "wide format". Their use is not exclusive. A deployment 709 could choose to leverage both. For example, ingress_if_id_(short 710 format) could be an identifier for the physical interface, whereas 711 ingress_if_id_(wide format) could be an identifier for a logical sub- 712 interface of that physical interface. 714 Data field and associated data type for each of the IOAM-Data-Fields 715 is shown below: 717 Hop_Lim and node_id: 4-octet field defined as follows: 719 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 720 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 721 | Hop_Lim | node_id | 722 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 724 Hop_Lim: 1-octet unsigned integer. It is set to the Hop Limit 725 value in the packet at the node that records this data. Hop 726 Limit information is used to identify the location of the node 727 in the communication path. This is copied from the lower 728 layer, e.g., TTL value in IPv4 header or hop limit field from 729 IPv6 header of the packet when the packet is ready for 730 transmission. The semantics of the Hop_Lim field depend on the 731 lower layer protocol that IOAM is encapsulated over, and 732 therefore its specific semantics are outside the scope of this 733 memo. 735 node_id: 3-octet unsigned integer. Node identifier field to 736 uniquely identify a node within the IOAM-Namespace and 737 associated IOAM-Domain. The procedure to allocate, manage and 738 map the node_ids is beyond the scope of this document. 740 ingress_if_id and egress_if_id: 4-octet field defined as follows: 742 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 743 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 744 | ingress_if_id | egress_if_id | 745 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 747 ingress_if_id: 2-octet unsigned integer. Interface identifier to 748 record the ingress interface the packet was received on. 750 egress_if_id: 2-octet unsigned integer. Interface identifier to 751 record the egress interface the packet is forwarded out of. 753 Note that due to the fact that IOAM uses its own IOAM-Namespaces 754 for IOAM-Data-Fields, data fields like interface identifiers can 755 be used in a flexible way to represent system resources that are 756 associated with ingressing or egressing packets, i.e. 757 ingress_if_id could represent a physical interface, a virtual or 758 logical interface, or even a queue. 760 timestamp seconds: 4-octet unsigned integer. Absolute timestamp in 761 seconds that specifies the time at which the packet was received 762 by the node. This field has three possible formats; based on 763 either PTP [IEEE1588v2], NTP [RFC5905], or POSIX [POSIX]. The 764 three timestamp formats are specified in Section 5. In all three 765 cases, the Timestamp Seconds field contains the 32 most 766 significant bits of the timestamp format that is specified in 767 Section 5. If a node is not capable of populating this field, it 768 assigns the value 0xFFFFFFFF. Note that this is a legitimate 769 value that is valid for 1 second in approximately 136 years; the 770 analyzer should correlate several packets or compare the timestamp 771 value to its own time-of-day in order to detect the error 772 indication. 774 timestamp subseconds: 4-octet unsigned integer. Absolute timestamp 775 in subseconds that specifies the time at which the packet was 776 received by the node. This field has three possible formats; 777 based on either PTP [IEEE1588v2], NTP [RFC5905], or POSIX [POSIX]. 778 The three timestamp formats are specified in Section 5. In all 779 three cases, the Timestamp Subseconds field contains the 32 least 780 significant bits of the timestamp format that is specified in 781 Section 5. If a node is not capable of populating this field, it 782 assigns the value 0xFFFFFFFF. Note that this is a legitimate 783 value in the NTP format, valid for approximately 233 picoseconds 784 in every second. If the NTP format is used the analyzer should 785 correlate several packets in order to detect the error indication. 787 transit delay: 4-octet unsigned integer in the range 0 to 2^31-1. 788 It is the time in nanoseconds the packet spent in the transit 789 node. This can serve as an indication of the queuing delay at the 790 node. If the transit delay exceeds 2^31-1 nanoseconds then the 791 top bit 'O' is set to indicate overflow and value set to 792 0x80000000. When this field is part of the data field but a node 793 populating the field is not able to fill it, the field position in 794 the field must be filled with value 0xFFFFFFFF to mean not 795 populated. 797 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 798 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 799 |O| transit delay | 800 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 802 namespace specific data: 4-octet field which can be used by the node 803 to add IOAM-Namespace specific data. This represents a "free- 804 format" 4-octet bit field with its semantics defined in the 805 context of a specific IOAM-Namespace. 807 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 808 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 809 | namespace specific data | 810 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 812 queue depth: 4-octet unsigned integer field. This field indicates 813 the current length of the egress interface queue of the interface 814 from where the packet is forwarded out. The queue depth is 815 expressed as the current number of memory buffers used by the 816 queue (a packet may consume one or more memory buffers, depending 817 on its size). 819 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 820 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 821 | queue depth | 822 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 824 Hop_Lim and node_id wide: 8-octet field defined as follows: 826 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 827 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 828 | Hop_Lim | node_id ~ 829 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 830 ~ node_id (contd) | 831 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 833 Hop_Lim: 1-octet unsigned integer. It is set to the Hop Limit 834 value in the packet at the node that records this data. Hop 835 Limit information is used to identify the location of the node 836 in the communication path. This is copied from the lower layer 837 for e.g. TTL value in IPv4 header or hop limit field from IPv6 838 header of the packet. The semantics of the Hop_Lim field 839 depend on the lower layer protocol that IOAM is encapsulated 840 over, and therefore its specific semantics are outside the 841 scope of this memo. 843 node_id: 7-octet unsigned integer. Node identifier field to 844 uniquely identify a node within the IOAM-Namespace and 845 associated IOAM-Domain. The procedure to allocate, manage and 846 map the node_ids is beyond the scope of this document. 848 ingress_if_id and egress_if_id wide: 8-octet field defined as 849 follows: 851 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 852 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 853 | ingress_if_id | 854 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 855 | egress_if_id | 856 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 858 ingress_if_id: 4-octet unsigned integer. Interface identifier to 859 record the ingress interface the packet was received on. 861 egress_if_id: 4-octet unsigned integer. Interface identifier to 862 record the egress interface the packet is forwarded out of. 864 namespace specific data wide: 8-octet field which can be used by the 865 node to add IOAM-Namespace specific data. This represents a 866 "free-format" 8-octet bit field with its semantics defined in the 867 context of a specific IOAM-Namespace. 869 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 870 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 871 | namespace specific data ~ 872 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 873 ~ namespace specific data (contd) | 874 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 876 buffer occupancy: 4-octet unsigned integer field. This field 877 indicates the current status of the occupancy of the common buffer 878 pool used by a set of queues. The units of this field depend on 879 the equipment type and deployment and has to be interpreted within 880 the context of an IOAM-Namespace and/or node-id if used. 882 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 883 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 884 | buffer occupancy | 885 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 887 Checksum Complement: 4-octet node data which contains a 4-octet 888 Checksum Complement field. The Checksum Complement is useful when 889 IOAM is transported over encapsulations that make use of a UDP 890 transport, such as VXLAN-GPE or Geneve. Without the Checksum 891 Complement, nodes adding IOAM node data must update the UDP 892 Checksum field. When the Checksum Complement is present, an IOAM 893 encapsulating node or IOAM transit node adding node data MUST 894 carry out one of the following two alternatives in order to 895 maintain the correctness of the UDP Checksum value: 897 1. Recompute the UDP Checksum field. 899 2. Use the Checksum Complement to make a checksum-neutral update 900 in the UDP payload; the Checksum Complement is assigned a 901 value that complements the rest of the node data fields that 902 were added by the current node, causing the existing UDP 903 Checksum field to remain correct. 905 IOAM decapsulating nodes MUST recompute the UDP Checksum field, 906 since they do not know whether previous hops modified the UDP 907 Checksum field or the Checksum Complement field. 909 Checksum Complement fields are used in a similar manner in 910 [RFC7820] and [RFC7821]. 912 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 913 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 914 | Checksum Complement | 915 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 917 Opaque State Snapshot: Opaque State Snapshot is a variable length 918 field and immediately follows the fixed length IOAM-Data-Fields 919 defined above. It allows the network element to store an 920 arbitrary state in the node data field, without a pre-defined 921 schema. The schema is to be defined within the context of an 922 IOAM-Namespace. The schema needs to be made known to the analyzer 923 by some out-of-band mechanism. The specification of this 924 mechanism is beyond the scope of this document. A 24-bit "Schema 925 Id" field, interpreted within the context of an IOAM-Namespace, 926 indicates which particular schema is used, and should be 927 configured on the network element by the operator. 929 0 1 2 3 930 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 931 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 932 | Length | Schema ID | 933 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 934 | | 935 | | 936 | Opaque data | 937 ~ ~ 938 . . 939 . . 940 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 942 Length: 1-octet unsigned integer. It is the length in multiples 943 of 4-octets of the Opaque data field that follows Schema Id. 945 Schema ID: 3-octet unsigned integer identifying the schema of 946 Opaque data. 948 Opaque data: Variable length field. This field is interpreted as 949 specified by the schema identified by the Schema ID. 951 When this field is part of the data field but a node populating 952 the field has no opaque state data to report, the Length must be 953 set to 0 and the Schema ID must be set to 0xFFFFFF to mean no 954 schema. 956 4.4.3. Examples of IOAM node data 958 An entry in the "node data list" array can have different formats, 959 following the needs of the deployment. Some deployments might only 960 be interested in recording the node identifiers, whereas others might 961 be interested in recording node identifier and timestamp. The 962 section provides example entries of the "node data list". 964 0xD40000: IOAM-Trace-Type is 0xD40000 (0b110101000000000000000000) 965 then the format of node data is: 967 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 968 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 969 | Hop_Lim | node_id | 970 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 971 | ingress_if_id | egress_if_id | 972 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 973 | timestamp subseconds | 974 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 975 | namespace specific data | 976 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 978 0xC00000: IOAM-Trace-Type is 0xC00000 (0b110000000000000000000000) 979 then the format is: 981 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 982 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 983 | Hop_Lim | node_id | 984 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 985 | ingress_if_id | egress_if_id | 986 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 988 0x900000: IOAM-Trace-Type is 0x900000 (0b100100000000000000000000) 989 then the format is: 991 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 992 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 993 | Hop_Lim | node_id | 994 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 995 | timestamp subseconds | 996 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 998 0x840000: IOAM-Trace-Type is 0x840000 (0b100001000000000000000000) 999 then the format is: 1001 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1002 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1003 | Hop_Lim | node_id | 1004 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1005 | namespace specific data | 1006 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1008 0x940000: IOAM-Trace-Type is 0x940000 (0b100101000000000000000000) 1009 then the format is: 1011 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1012 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1013 | Hop_Lim | node_id | 1014 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1015 | timestamp subseconds | 1016 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1017 | namespace specific data | 1018 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1020 0x308002: IOAM-Trace-Type is 0x308002 (0b001100001000000000000010) 1021 then the format is: 1023 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1024 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1025 | timestamp seconds | 1026 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1027 | timestamp subseconds | 1028 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1029 | Hop_Lim | node_id | 1030 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1031 | node_id(contd) | 1032 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1033 | Length | Schema Id | 1034 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1035 | | 1036 | | 1037 | Opaque data | 1038 ~ ~ 1039 . . 1040 . . 1041 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1043 4.5. IOAM Proof of Transit Option-Type 1045 IOAM Proof of Transit Option-Type is to support path or service 1046 function chain [RFC7665] verification use cases. Proof-of-transit 1047 uses methods like nested hashing or nested encryption of the IOAM 1048 data or mechanisms such as Shamir's Secret Sharing Schema (SSSS). 1049 While details on how the IOAM data for the proof of transit option is 1050 processed at IOAM encapsulating, decapsulating and transit nodes are 1051 outside the scope of the document, all of these approaches share the 1052 need to uniquely identify a packet as well as iteratively operate on 1053 a set of information that is handed from node to node. 1054 Correspondingly, two pieces of information are added as IOAM-Data- 1055 Fields to the packet: 1057 o Random: Unique identifier for the packet (e.g., 64-bits allow for 1058 the unique identification of 2^64 packets). 1060 o Cumulative: Information which is handed from node to node and 1061 updated by every node according to a verification algorithm. 1063 The IOAM Proof of Transit Option-Type consist of a fixed size "IOAM 1064 proof of transit option header" and "IOAM proof of transit option 1065 data fields": 1067 IOAM proof of transit option header: 1069 0 1 2 3 1070 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1071 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1072 | Namespace-ID |IOAM POT Type | IOAM POT flags| 1073 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1075 IOAM proof of transit Option-Type IOAM-Data-Fields MUST be 1076 4-octet aligned: 1078 0 1 2 3 1079 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1080 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1081 | POT Option data field determined by IOAM-POT-Type | 1082 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1084 Namespace-ID: 16-bit identifier of an IOAM-Namespace. The 1085 Namespace-ID value of 0x0000 is defined as the default value and 1086 MUST be known to all the nodes implementing IOAM. For any other 1087 Namespace-ID value that does not match any Namespace-ID the node 1088 is configured to operate on, the node MUST NOT change the contents 1089 of the IOAM-Data-Fields. 1091 IOAM POT Type: 8-bit identifier of a particular POT variant that 1092 specifies the POT data that is included. This document defines 1093 POT Type 0: 1095 0: POT data is a 16 Octet field as described below. 1097 IOAM POT flags: 8-bit. Following flags are defined: 1099 Bit 0 "Profile-to-use" (P-bit) (most significant bit). For IOAM 1100 POT types that use a maximum of two profiles to drive 1101 computation, indicates which POT-profile is used. The two 1102 profiles are numbered 0, 1. 1104 Bit 1-7 Reserved: Must be set to zero upon transmission and 1105 ignored upon receipt. 1107 POT Option data: Variable-length field. The type of which is 1108 determined by the IOAM-POT-Type. 1110 4.5.1. IOAM Proof of Transit Type 0 1112 IOAM proof of transit option of IOAM POT Type 0: 1114 0 1 2 3 1115 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1116 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1117 | Namespace-ID |IOAM POT Type=0|P|R R R R R R R| 1118 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+<-+ 1119 | Random | | 1120 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ P 1121 | Random(contd) | O 1122 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ T 1123 | Cumulative | | 1124 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 1125 | Cumulative (contd) | | 1126 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+<-+ 1128 Namespace-ID: 16-bit identifier of an IOAM-Namespace. The 1129 Namespace-ID value of 0x0000 is defined as the default value and 1130 MUST be known to all the nodes implementing IOAM. For any other 1131 Namespace-ID value that does not match any Namespace-ID the node 1132 is configured to operate on, the node MUST NOT change the contents 1133 of the IOAM-Data-Fields. 1135 IOAM POT Type: 8-bit identifier of a particular POT variant that 1136 specifies the POT data that is included. This section defines the 1137 POT data when the IOAM POT Type is set to the value 0. 1139 P bit: 1-bit. "Profile-to-use" (P-bit) (most significant bit). 1140 Indicates which POT-profile is used to generate the Cumulative. 1141 Any node participating in POT will have a maximum of 2 profiles 1142 configured that drive the computation of cumulative. The two 1143 profiles are numbered 0, 1. This bit conveys whether profile 0 or 1144 profile 1 is used to compute the Cumulative. 1146 R (7 bits): 7-bit IOAM POT flags for future use. MUST be set to 1147 zero upon transmission and ignored upon receipt. 1149 Random: 64-bit Per packet Random number. 1151 Cumulative: 64-bit Cumulative that is updated at specific nodes by 1152 processing per packet Random number field and configured 1153 parameters. 1155 Note: Larger or smaller sizes of "Random" and "Cumulative" data are 1156 feasible and could be required for certain deployments (e.g. in case 1157 of space constraints in the transport protocol used). Future 1158 versions of this document will address different sizes of data for 1159 "proof of transit". 1161 4.6. IOAM Edge-to-Edge Option-Type 1163 The IOAM Edge-to-Edge Option-Type is to carry data that is added by 1164 the IOAM encapsulating node and interpreted by IOAM decapsulating 1165 node. The IOAM transit nodes MAY process the data but MUST NOT 1166 modify it. 1168 The IOAM Edge-to-Edge Option-Type consist of a fixed size "IOAM Edge- 1169 to-Edge Option-Type header" and "IOAM Edge-to-Edge Option-Type data 1170 fields": 1172 IOAM Edge-to-Edge Option-Type header: 1174 0 1 2 3 1175 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1176 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1177 | Namespace-ID | IOAM-E2E-Type | 1178 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1180 IOAM Edge-to-Edge Option-Type IOAM-Data-Fields MUST 1181 be 4-octet aligned: 1183 0 1 2 3 1184 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1185 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1186 | E2E Option data field determined by IOAM-E2E-Type | 1187 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1189 Namespace-ID: 16-bit identifier of an IOAM-Namespace. The 1190 Namespace-ID value of 0x0000 is defined as the default value and 1191 MUST be known to all the nodes implementing IOAM. For any other 1192 Namespace-ID value that does not match any Namespace-ID the node 1193 is configured to operate on, then the node MUST NOT change the 1194 contents of the IOAM-Data-Fields. 1196 IOAM-E2E-Type: A 16-bit identifier which specifies which data types 1197 are used in the E2E option data. The IOAM-E2E-Type value is a bit 1198 field. The order of packing the E2E option data field elements 1199 follows the bit order of the IOAM-E2E-Type field, as follows: 1201 Bit 0 (Most significant bit) When set indicates presence of a 1202 64-bit sequence number added to a specific "packet group" 1203 which is used to detect packet loss, packet reordering, 1204 or packet duplication within the group. The "packet 1205 group" is deployment dependent and defined at the IOAM 1206 encapsulating node e.g. by n-tuple based classification 1207 of packets. 1209 Bit 1 When set indicates presence of a 32-bit sequence number 1210 added to a specific "packet group" which is used to 1211 detect packet loss, packet reordering, or packet 1212 duplication within that group. The "packet group" is 1213 deployment dependent and defined at the IOAM 1214 encapsulating node e.g. by n-tuple based classification 1215 of packets. 1217 Bit 2 When set indicates presence of timestamp seconds for the 1218 transmission of the frame. This 4-octet field has three 1219 possible formats; based on either PTP [IEEE1588v2], NTP 1220 [RFC5905], or POSIX [POSIX]. The three timestamp formats 1221 are specified in Section 5. In all three cases, the 1222 Timestamp Seconds field contains the 32 most significant 1223 bits of the timestamp format that is specified in 1224 Section 5. If a node is not capable of populating this 1225 field, it assigns the value 0xFFFFFFFF. Note that this 1226 is a legitimate value that is valid for 1 second in 1227 approximately 136 years; the analyzer should correlate 1228 several packets or compare the timestamp value to its own 1229 time-of-day in order to detect the error indication. 1231 Bit 3 When set indicates presence of timestamp subseconds for 1232 the transmission of the frame. This 4-octet field has 1233 three possible formats; based on either PTP [IEEE1588v2], 1234 NTP [RFC5905], or POSIX [POSIX]. The three timestamp 1235 formats are specified in Section 5. In all three cases, 1236 the Timestamp Subseconds field contains the 32 least 1237 significant bits of the timestamp format that is 1238 specified in Section 5. If a node is not capable of 1239 populating this field, it assigns the value 0xFFFFFFFF. 1240 Note that this is a legitimate value in the NTP format, 1241 valid for approximately 233 picoseconds in every second. 1242 If the NTP format is used the analyzer should correlate 1243 several packets in order to detect the error indication. 1245 Bit 4-15 Undefined. An IOAM encapsulating node Must set the value 1246 of these bits to zero upon transmission and ignore upon 1247 receipt. 1249 E2E Option data: Variable-length field. The type of which is 1250 determined by the IOAM-E2E-Type. 1252 5. Timestamp Formats 1254 The IOAM-Data-Fields include a timestamp field which is represented 1255 in one of three possible timestamp formats. It is assumed that the 1256 management plane is responsible for determining which timestamp 1257 format is used. 1259 5.1. PTP Truncated Timestamp Format 1261 The Precision Time Protocol (PTP) [IEEE1588v2] uses an 80-bit 1262 timestamp format. The truncated timestamp format is a 64-bit field, 1263 which is the 64 least significant bits of the 80-bit PTP timestamp. 1264 The PTP truncated format is specified in Section 4.3 of 1265 [I-D.ietf-ntp-packet-timestamps], and the details are presented below 1266 for the sake of completeness. 1268 0 1 2 3 1269 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1270 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1271 | Seconds | 1272 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1273 | Nanoseconds | 1274 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1276 Figure 1: PTP [IEEE1588v2] Truncated Timestamp Format 1278 Timestamp field format: 1280 Seconds: specifies the integer portion of the number of seconds 1281 since the epoch. 1283 + Size: 32 bits. 1285 + Units: seconds. 1287 Nanoseconds: specifies the fractional portion of the number of 1288 seconds since the epoch. 1290 + Size: 32 bits. 1292 + Units: nanoseconds. The value of this field is in the range 0 1293 to (10^9)-1. 1295 Epoch: 1297 The PTP [IEEE1588v2] epoch is 1 January 1970 00:00:00 TAI, which 1298 is 31 December 1969 23:59:51.999918 UTC. 1300 Resolution: 1302 The resolution is 1 nanosecond. 1304 Wraparound: 1306 This time format wraps around every 2^32 seconds, which is roughly 1307 136 years. The next wraparound will occur in the year 2106. 1309 Synchronization Aspects: 1311 It is assumed that nodes that run this protocol are synchronized 1312 among themselves. Nodes may be synchronized to a global reference 1313 time. Note that if PTP [IEEE1588v2] is used for synchronization, 1314 the timestamp may be derived from the PTP-synchronized clock, 1315 allowing the timestamp to be measured with respect to the clock of 1316 an PTP Grandmaster clock. 1318 The PTP truncated timestamp format is not affected by leap 1319 seconds. 1321 5.2. NTP 64-bit Timestamp Format 1323 The Network Time Protocol (NTP) [RFC5905] timestamp format is 64 bits 1324 long. This format is specified in Section 4.2.1 of 1325 [I-D.ietf-ntp-packet-timestamps], and the details are presented below 1326 for the sake of completeness. 1328 0 1 2 3 1329 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1330 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1331 | Seconds | 1332 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1333 | Fraction | 1334 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1336 Figure 2: NTP [RFC5905] 64-bit Timestamp Format 1338 Timestamp field format: 1340 Seconds: specifies the integer portion of the number of seconds 1341 since the epoch. 1343 + Size: 32 bits. 1345 + Units: seconds. 1347 Fraction: specifies the fractional portion of the number of 1348 seconds since the epoch. 1350 + Size: 32 bits. 1352 + Units: the unit is 2^(-32) seconds, which is roughly equal to 1353 233 picoseconds. 1355 Epoch: 1357 The epoch is 1 January 1900 at 00:00 UTC. 1359 Resolution: 1361 The resolution is 2^(-32) seconds. 1363 Wraparound: 1365 This time format wraps around every 2^32 seconds, which is roughly 1366 136 years. The next wraparound will occur in the year 2036. 1368 Synchronization Aspects: 1370 Nodes that use this timestamp format will typically be 1371 synchronized to UTC using NTP [RFC5905]. Thus, the timestamp may 1372 be derived from the NTP-synchronized clock, allowing the timestamp 1373 to be measured with respect to the clock of an NTP server. 1375 The NTP timestamp format is affected by leap seconds; it 1376 represents the number of seconds since the epoch minus the number 1377 of leap seconds that have occurred since the epoch. The value of 1378 a timestamp during or slightly after a leap second may be 1379 temporarily inaccurate. 1381 5.3. POSIX-based Timestamp Format 1383 This timestamp format is based on the POSIX time format [POSIX]. The 1384 detailed specification of the timestamp format used in this document 1385 is presented below. 1387 0 1 2 3 1388 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1389 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1390 | Seconds | 1391 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1392 | Microseconds | 1393 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1395 Figure 3: POSIX-based Timestamp Format 1397 Timestamp field format: 1399 Seconds: specifies the integer portion of the number of seconds 1400 since the epoch. 1402 + Size: 32 bits. 1404 + Units: seconds. 1406 Microseconds: specifies the fractional portion of the number of 1407 seconds since the epoch. 1409 + Size: 32 bits. 1411 + Units: the unit is microseconds. The value of this field is in 1412 the range 0 to (10^6)-1. 1414 Epoch: 1416 The epoch is 1 January 1970 00:00:00 TAI, which is 31 December 1417 1969 23:59:51.999918 UTC. 1419 Resolution: 1421 The resolution is 1 microsecond. 1423 Wraparound: 1425 This time format wraps around every 2^32 seconds, which is roughly 1426 136 years. The next wraparound will occur in the year 2106. 1428 Synchronization Aspects: 1430 It is assumed that nodes that use this timestamp format run Linux 1431 operating system, and hence use the POSIX time. In some cases 1432 nodes may be synchronized to UTC using a synchronization mechanism 1433 that is outside the scope of this document, such as NTP [RFC5905]. 1434 Thus, the timestamp may be derived from the NTP-synchronized 1435 clock, allowing the timestamp to be measured with respect to the 1436 clock of an NTP server. 1438 The POSIX-based timestamp format is affected by leap seconds; it 1439 represents the number of seconds since the epoch minus the number 1440 of leap seconds that have occurred since the epoch. The value of 1441 a timestamp during or slightly after a leap second may be 1442 temporarily inaccurate. 1444 6. IOAM Data Export 1446 IOAM nodes collect information for packets traversing a domain that 1447 supports IOAM. IOAM decapsulating nodes as well as IOAM transit 1448 nodes can choose to retrieve IOAM information from the packet, 1449 process the information further and export the information using 1450 e.g., IPFIX. The mechanisms and associated data formats for 1451 exporting IOAM data is outside the scope of this document. 1453 Raw data export of IOAM data using IPFIX is discussed in 1454 [I-D.spiegel-ippm-ioam-rawexport]. 1456 7. IANA Considerations 1458 This document requests the following IANA Actions. 1460 7.1. Creation of a new In-Situ OAM Protocol Parameters Registry (IOAM) 1461 Protocol Parameters IANA registry 1463 IANA is requested to create a new protocol registry for "In-Situ OAM 1464 (IOAM) Protocol Parameters". This is the common registry that will 1465 include registrations for all IOAM-Namespaces. Each Registry, whose 1466 names are listed below: 1468 IOAM Option-Type 1470 IOAM Trace-Type 1472 IOAM Trace-Flags 1474 IOAM POT-Type 1476 IOAM POT-Flags 1478 IOAM E2E-Type 1480 IOAM Namespace-ID 1482 will contain the current set of possibilities defined in this 1483 document. New registries in this name space are created via RFC 1484 Required process as per [RFC8126]. 1486 The subsequent sub-sections detail the registries herein contained. 1488 7.2. IOAM Option-Type Registry 1490 This registry defines 128 code points for the IOAM Option-Type field 1491 for identifying IOAM Option-Types as explained in Section 4. The 1492 following code points are defined in this draft: 1494 0 IOAM Pre-allocated Trace Option-Type 1496 1 IOAM Incremental Trace Option-Type 1498 2 IOAM POT Option-Type 1500 3 IOAM E2E Option-Type 1502 4 - 127 are available for assignment via RFC Required process as per 1503 [RFC8126]. 1505 7.3. IOAM Trace-Type Registry 1507 This registry defines code point for each bit in the 24-bit IOAM- 1508 Trace-Type field for Pre-allocated trace option and Incremental trace 1509 option defined in Section 4.4. The meaning of Bits 0 - 11 for trace 1510 type are defined in this document in Paragraph 5 of Section 4.4.1: 1512 Bit 0 hop_Lim and node_id in short format 1514 Bit 1 ingress_if_id and egress_if_id in short format 1516 Bit 2 timestamp seconds 1518 Bit 3 timestamp subseconds 1520 Bit 4 transit delay 1522 Bit 5 namespace specific data in short format 1524 Bit 6 queue depth 1526 Bit 7 checksum complement 1528 Bit 8 hop_Lim and node_id in wide format 1529 Bit 9 ingress_if_id and egress_if_id in wide format 1531 Bit 10 namespace specific data in wide format 1533 Bit 11 buffer occupancy 1535 Bit 22 variable length Opaque State Snapshot 1537 Bit 23 reserved 1539 The meaning for Bits 12 - 21 are available for assignment via RFC 1540 Required process as per [RFC8126]. 1542 7.4. IOAM Trace-Flags Registry 1544 This registry defines code points for each bit in the 4 bit flags for 1545 the Pre-allocated trace option and for the Incremental trace option 1546 defined in Section 4.4. The meaning of Bit 0 (the most significant 1547 bit) for trace flags is defined in this document in Paragraph 3 of 1548 Section 4.4.1: 1550 Bit 0 "Overflow" (O-bit) 1552 7.5. IOAM POT-Type Registry 1554 This registry defines 256 code points to define IOAM POT Type for 1555 IOAM proof of transit option Section 4.5. The code point value 0 is 1556 defined in this document: 1558 0: 16 Octet POT data 1560 1 - 255 are available for assignment via RFC Required process as per 1561 [RFC8126]. 1563 7.6. IOAM POT-Flags Registry 1565 This registry defines code points for each bit in the 8 bit flags for 1566 IOAM POT option defined in Section 4.5. The meaning of Bit 0 for 1567 IOAM POT flags is defined in this document in Section 4.5: 1569 Bit 0 "Profile-to-use" (P-bit) 1571 The meaning for Bits 1 - 7 are available for assignment via RFC 1572 Required process as per [RFC8126]. 1574 7.7. IOAM E2E-Type Registry 1576 This registry defines code points for each bit in the 16 bit IOAM- 1577 E2E-Type field for IOAM E2E option Section 4.6. The meaning of Bit 0 1578 - 3 are defined in this document: 1580 Bit 0 64-bit sequence number 1582 Bit 1 32-bit sequence number 1584 Bit 2 timestamp seconds 1586 Bit 3 timestamp subseconds 1588 The meaning of Bits 4 - 15 are available for assignment via RFC 1589 Required process as per [RFC8126]. 1591 7.8. IOAM Namespace-ID Registry 1593 IANA is requested to set up an "IOAM Namespace-ID Registry", 1594 containing 16-bit values. The meaning of Bit 0 is defined in this 1595 document. IANA is requested to reserve the values 0x0001 to 0x7FFF 1596 for private use (managed by operators), as specified in Section 4.3 1597 of the current document. Registry entries for the values 0x8000 to 1598 0xFFFF are to be assigned via the "Expert Review" policy defined in 1599 [RFC8126]. 1601 0: default namespace (known to all IOAM nodes) 1603 0x0001 - 0x7FFF: reserved for private use 1605 0x8000 - 0xFFFF: unassigned 1607 8. Security Considerations 1609 As discussed in [RFC7276], a successful attack on an OAM protocol in 1610 general, and specifically on IOAM, can prevent the detection of 1611 failures or anomalies, or create a false illusion of nonexistent 1612 ones. 1614 The Proof of Transit Option-Type (Section Section 4.5) is used for 1615 verifying the path of data packets. The security considerations of 1616 POT are further discussed in [I-D.ietf-sfc-proof-of-transit]. 1618 The data elements of IOAM can be used for network reconnaissance, 1619 allowing attackers to collect information about network paths, 1620 performance, queue states, buffer occupancy and other information. 1621 Note that in case IOAM is used in "immediate export" mode (reference 1622 to be added in a future revision), the IOAM related trace information 1623 would not be available in the customer data packets, but would 1624 trigger export of packet related IOAM information at every node. 1625 IOAM data export and securing IOAM data export is outside the scope 1626 of this document. 1628 IOAM can be used as a means for implementing Denial of Service (DoS) 1629 attacks, or for amplifying them. For example, a malicious attacker 1630 can add an IOAM header to packets in order to consume the resources 1631 of network devices that take part in IOAM or collectors that analyze 1632 the IOAM data. Another example is a packet length attack, in which 1633 an attacker pushes headers associated with IOAM Option-Types into 1634 data packets, causing these packets to be increased beyond the MTU 1635 size, resulting in fragmentation or in packet drops. 1637 Since IOAM options may include timestamps, if network devices use 1638 synchronization protocols then any attack on the time protocol 1639 [RFC7384] can compromise the integrity of the timestamp-related data 1640 fields. 1642 At the management plane, attacks may be implemented by misconfiguring 1643 or by maliciously configuring IOAM-enabled nodes in a way that 1644 enables other attacks. Thus, IOAM configuration should be secured in 1645 a way that authenticates authorized users and verifies the integrity 1646 of configuration procedures. 1648 Notably, IOAM is expected to be deployed in specific network domains, 1649 thus confining the potential attack vectors to within the network 1650 domain. Indeed, in order to limit the scope of threats to within the 1651 current network domain the network operator is expected to enforce 1652 policies that prevent IOAM traffic from leaking outside of the IOAM 1653 domain, and prevent IOAM data from outside the domain to be processed 1654 and used within the domain. Note that the Immediate Export mode 1655 (reference to be added in a future revision) can mitigate the 1656 potential threat of IOAM data leaking through data packets. 1658 9. Acknowledgements 1660 The authors would like to thank Eric Vyncke, Nalini Elkins, Srihari 1661 Raghavan, Ranganathan T S, Karthik Babu Harichandra Babu, Akshaya 1662 Nadahalli, LJ Wobker, Erik Nordmark, Vengada Prasad Govindan, Andrew 1663 Yourtchenko, Aviv Kfir, Tianran Zhou and Zhenbin (Robin) for the 1664 comments and advice. 1666 This document leverages and builds on top of several concepts 1667 described in [I-D.kitamura-ipv6-record-route]. The authors would 1668 like to acknowledge the work done by the author Hiroshi Kitamura and 1669 people involved in writing it. 1671 The authors would like to gracefully acknowledge useful review and 1672 insightful comments received from Joe Clarke, Al Morton, Tom Herbet, 1673 Haoyu song, and Mickey Spiegel. 1675 The authors would like to acknowledge the contribution of "Immediate 1676 export" of IOAM trace by Barak Gafni. 1678 10. References 1680 10.1. Normative References 1682 [IEEE1588v2] 1683 Institute of Electrical and Electronics Engineers, "IEEE 1684 Std 1588-2008 - IEEE Standard for a Precision Clock 1685 Synchronization Protocol for Networked Measurement and 1686 Control Systems", IEEE Std 1588-2008, 2008, 1687 . 1690 [POSIX] Institute of Electrical and Electronics Engineers, "IEEE 1691 Std 1003.1-2008 (Revision of IEEE Std 1003.1-2004) - IEEE 1692 Standard for Information Technology - Portable Operating 1693 System Interface (POSIX(R))", IEEE Std 1003.1-2008, 2008, 1694 . 1697 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1698 Requirement Levels", BCP 14, RFC 2119, 1699 DOI 10.17487/RFC2119, March 1997, 1700 . 1702 [RFC5905] Mills, D., Martin, J., Ed., Burbank, J., and W. Kasch, 1703 "Network Time Protocol Version 4: Protocol and Algorithms 1704 Specification", RFC 5905, DOI 10.17487/RFC5905, June 2010, 1705 . 1707 [RFC8126] Cotton, M., Leiba, B., and T. Narten, "Guidelines for 1708 Writing an IANA Considerations Section in RFCs", BCP 26, 1709 RFC 8126, DOI 10.17487/RFC8126, June 2017, 1710 . 1712 10.2. Informative References 1714 [I-D.ietf-ntp-packet-timestamps] 1715 Mizrahi, T., Fabini, J., and A. Morton, "Guidelines for 1716 Defining Packet Timestamps", draft-ietf-ntp-packet- 1717 timestamps-07 (work in progress), August 2019. 1719 [I-D.ietf-nvo3-geneve] 1720 Gross, J., Ganga, I., and T. Sridhar, "Geneve: Generic 1721 Network Virtualization Encapsulation", draft-ietf- 1722 nvo3-geneve-13 (work in progress), March 2019. 1724 [I-D.ietf-nvo3-vxlan-gpe] 1725 Maino, F., Kreeger, L., and U. Elzur, "Generic Protocol 1726 Extension for VXLAN", draft-ietf-nvo3-vxlan-gpe-07 (work 1727 in progress), April 2019. 1729 [I-D.ietf-sfc-proof-of-transit] 1730 Brockners, F., Bhandari, S., Dara, S., Pignataro, C., 1731 Leddy, J., Youell, S., Mozes, D., Mizrahi, T., Aguado, A., 1732 and D. Lopez, "Proof of Transit", draft-ietf-sfc-proof-of- 1733 transit-02 (work in progress), March 2019. 1735 [I-D.kitamura-ipv6-record-route] 1736 Kitamura, H., "Record Route for IPv6 (PR6) Hop-by-Hop 1737 Option Extension", draft-kitamura-ipv6-record-route-00 1738 (work in progress), November 2000. 1740 [I-D.lapukhov-dataplane-probe] 1741 Lapukhov, P. and r. remy@barefootnetworks.com, "Data-plane 1742 probe for in-band telemetry collection", draft-lapukhov- 1743 dataplane-probe-01 (work in progress), June 2016. 1745 [I-D.spiegel-ippm-ioam-rawexport] 1746 Spiegel, M., Brockners, F., Bhandari, S., and R. 1747 Sivakolundu, "In-situ OAM raw data export with IPFIX", 1748 draft-spiegel-ippm-ioam-rawexport-02 (work in progress), 1749 July 2019. 1751 [RFC7276] Mizrahi, T., Sprecher, N., Bellagamba, E., and Y. 1752 Weingarten, "An Overview of Operations, Administration, 1753 and Maintenance (OAM) Tools", RFC 7276, 1754 DOI 10.17487/RFC7276, June 2014, 1755 . 1757 [RFC7384] Mizrahi, T., "Security Requirements of Time Protocols in 1758 Packet Switched Networks", RFC 7384, DOI 10.17487/RFC7384, 1759 October 2014, . 1761 [RFC7665] Halpern, J., Ed. and C. Pignataro, Ed., "Service Function 1762 Chaining (SFC) Architecture", RFC 7665, 1763 DOI 10.17487/RFC7665, October 2015, 1764 . 1766 [RFC7799] Morton, A., "Active and Passive Metrics and Methods (with 1767 Hybrid Types In-Between)", RFC 7799, DOI 10.17487/RFC7799, 1768 May 2016, . 1770 [RFC7820] Mizrahi, T., "UDP Checksum Complement in the One-Way 1771 Active Measurement Protocol (OWAMP) and Two-Way Active 1772 Measurement Protocol (TWAMP)", RFC 7820, 1773 DOI 10.17487/RFC7820, March 2016, 1774 . 1776 [RFC7821] Mizrahi, T., "UDP Checksum Complement in the Network Time 1777 Protocol (NTP)", RFC 7821, DOI 10.17487/RFC7821, March 1778 2016, . 1780 [RFC8300] Quinn, P., Ed., Elzur, U., Ed., and C. Pignataro, Ed., 1781 "Network Service Header (NSH)", RFC 8300, 1782 DOI 10.17487/RFC8300, January 2018, 1783 . 1785 Authors' Addresses 1787 Frank Brockners 1788 Cisco Systems, Inc. 1789 Hansaallee 249, 3rd Floor 1790 DUESSELDORF, NORDRHEIN-WESTFALEN 40549 1791 Germany 1793 Email: fbrockne@cisco.com 1795 Shwetha Bhandari 1796 Cisco Systems, Inc. 1797 Cessna Business Park, Sarjapura Marathalli Outer Ring Road 1798 Bangalore, KARNATAKA 560 087 1799 India 1801 Email: shwethab@cisco.com 1803 Carlos Pignataro 1804 Cisco Systems, Inc. 1805 7200-11 Kit Creek Road 1806 Research Triangle Park, NC 27709 1807 United States 1809 Email: cpignata@cisco.com 1810 Hannes Gredler 1811 RtBrick Inc. 1813 Email: hannes@rtbrick.com 1815 John Leddy 1816 United States 1818 Email: john@leddy.net 1820 Stephen Youell 1821 JP Morgan Chase 1822 25 Bank Street 1823 London E14 5JP 1824 United Kingdom 1826 Email: stephen.youell@jpmorgan.com 1828 Tal Mizrahi 1829 Huawei Network.IO Innovation Lab 1830 Israel 1832 Email: tal.mizrahi.phd@gmail.com 1834 David Mozes 1836 Email: mosesster@gmail.com 1838 Petr Lapukhov 1839 Facebook 1840 1 Hacker Way 1841 Menlo Park, CA 94025 1842 US 1844 Email: petr@fb.com 1846 Remy Chang 1847 Barefoot Networks 1848 4750 Patrick Henry Drive 1849 Santa Clara, CA 95054 1850 US 1851 Daniel Bernier 1852 Bell Canada 1853 Canada 1855 Email: daniel.bernier@bell.ca 1857 John Lemon 1858 Broadcom 1859 270 Innovation Drive 1860 San Jose, CA 95134 1861 US 1863 Email: john.lemon@broadcom.com