idnits 2.17.1 draft-ietf-ippm-ioam-data-08.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (October 24, 2019) is 1639 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) -- Looks like a reference, but probably isn't: '0' on line 700 -- Looks like a reference, but probably isn't: '1' on line 557 -- Possible downref: Non-RFC (?) normative reference: ref. 'IEEE1588v2' -- Possible downref: Non-RFC (?) normative reference: ref. 'POSIX' == Outdated reference: A later version (-09) exists of draft-ietf-ntp-packet-timestamps-07 == Outdated reference: A later version (-16) exists of draft-ietf-nvo3-geneve-14 == Outdated reference: A later version (-13) exists of draft-ietf-nvo3-vxlan-gpe-07 == Outdated reference: A later version (-08) exists of draft-ietf-sfc-proof-of-transit-03 == Outdated reference: A later version (-07) exists of draft-spiegel-ippm-ioam-rawexport-02 Summary: 0 errors (**), 0 flaws (~~), 6 warnings (==), 5 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 ippm F. Brockners 3 Internet-Draft S. Bhandari 4 Intended status: Standards Track C. Pignataro 5 Expires: April 26, 2020 Cisco 6 H. Gredler 7 RtBrick Inc. 8 J. Leddy 10 S. Youell 11 JPMC 12 T. Mizrahi 13 Huawei Network.IO Innovation Lab 14 D. Mozes 16 P. Lapukhov 17 Facebook 18 R. Chang 19 Barefoot Networks 20 D. Bernier 21 Bell Canada 22 J. Lemon 23 Broadcom 24 October 24, 2019 26 Data Fields for In-situ OAM 27 draft-ietf-ippm-ioam-data-08 29 Abstract 31 In-situ Operations, Administration, and Maintenance (IOAM) records 32 operational and telemetry information in the packet while the packet 33 traverses a path between two points in the network. This document 34 discusses the data fields and associated data types for in-situ OAM. 35 In-situ OAM data fields can be embedded into a variety of transports 36 such as NSH, Segment Routing, Geneve, native IPv6 (via extension 37 header), or IPv4. In-situ OAM can be used to complement OAM 38 mechanisms based on e.g. ICMP or other types of probe packets. 40 Status of This Memo 42 This Internet-Draft is submitted in full conformance with the 43 provisions of BCP 78 and BCP 79. 45 Internet-Drafts are working documents of the Internet Engineering 46 Task Force (IETF). Note that other groups may also distribute 47 working documents as Internet-Drafts. The list of current Internet- 48 Drafts is at https://datatracker.ietf.org/drafts/current/. 50 Internet-Drafts are draft documents valid for a maximum of six months 51 and may be updated, replaced, or obsoleted by other documents at any 52 time. It is inappropriate to use Internet-Drafts as reference 53 material or to cite them other than as "work in progress." 55 This Internet-Draft will expire on April 26, 2020. 57 Copyright Notice 59 Copyright (c) 2019 IETF Trust and the persons identified as the 60 document authors. All rights reserved. 62 This document is subject to BCP 78 and the IETF Trust's Legal 63 Provisions Relating to IETF Documents 64 (https://trustee.ietf.org/license-info) in effect on the date of 65 publication of this document. Please review these documents 66 carefully, as they describe your rights and restrictions with respect 67 to this document. Code Components extracted from this document must 68 include Simplified BSD License text as described in Section 4.e of 69 the Trust Legal Provisions and are provided without warranty as 70 described in the Simplified BSD License. 72 Table of Contents 74 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 75 2. Conventions . . . . . . . . . . . . . . . . . . . . . . . . . 3 76 3. Scope, Applicability, and Assumptions . . . . . . . . . . . . 4 77 4. IOAM Data-Fields, Types, Nodes . . . . . . . . . . . . . . . 5 78 4.1. IOAM Data-Fields and Option-Types . . . . . . . . . . . . 5 79 4.2. IOAM-Domains and types of IOAM Nodes . . . . . . . . . . 6 80 4.3. IOAM-Namespaces . . . . . . . . . . . . . . . . . . . . . 7 81 4.4. IOAM Trace Option-Types . . . . . . . . . . . . . . . . . 9 82 4.4.1. Pre-allocated and Incremental Trace Option-Types . . 12 83 4.4.2. IOAM node data fields and associated formats . . . . 15 84 4.4.3. Examples of IOAM node data . . . . . . . . . . . . . 21 85 4.5. IOAM Proof of Transit Option-Type . . . . . . . . . . . . 23 86 4.5.1. IOAM Proof of Transit Type 0 . . . . . . . . . . . . 25 87 4.6. IOAM Edge-to-Edge Option-Type . . . . . . . . . . . . . . 26 88 5. Timestamp Formats . . . . . . . . . . . . . . . . . . . . . . 28 89 5.1. PTP Truncated Timestamp Format . . . . . . . . . . . . . 28 90 5.2. NTP 64-bit Timestamp Format . . . . . . . . . . . . . . . 29 91 5.3. POSIX-based Timestamp Format . . . . . . . . . . . . . . 30 92 6. IOAM Data Export . . . . . . . . . . . . . . . . . . . . . . 32 93 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 32 94 7.1. Creation of a new In-Situ OAM Protocol Parameters 95 Registry (IOAM) Protocol Parameters IANA registry . . . . 32 96 7.2. IOAM Option-Type Registry . . . . . . . . . . . . . . . . 33 97 7.3. IOAM Trace-Type Registry . . . . . . . . . . . . . . . . 33 98 7.4. IOAM Trace-Flags Registry . . . . . . . . . . . . . . . . 34 99 7.5. IOAM POT-Type Registry . . . . . . . . . . . . . . . . . 34 100 7.6. IOAM POT-Flags Registry . . . . . . . . . . . . . . . . . 34 101 7.7. IOAM E2E-Type Registry . . . . . . . . . . . . . . . . . 35 102 7.8. IOAM Namespace-ID Registry . . . . . . . . . . . . . . . 35 103 8. Security Considerations . . . . . . . . . . . . . . . . . . . 35 104 9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 36 105 10. References . . . . . . . . . . . . . . . . . . . . . . . . . 37 106 10.1. Normative References . . . . . . . . . . . . . . . . . . 37 107 10.2. Informative References . . . . . . . . . . . . . . . . . 37 108 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 39 110 1. Introduction 112 This document defines data fields for "in-situ" Operations, 113 Administration, and Maintenance (IOAM). In-situ OAM records OAM 114 information within the packet while the packet traverses a particular 115 network domain. The term "in-situ" refers to the fact that the OAM 116 data is added to the data packets rather than is being sent within 117 packets specifically dedicated to OAM. IOAM is to complement 118 mechanisms such as Ping or Traceroute, or more recent active probing 119 mechanisms as described in [I-D.lapukhov-dataplane-probe]. In terms 120 of "active" or "passive" OAM, "in-situ" OAM can be considered a 121 hybrid OAM type. "In-situ" mechanisms do not require extra packets 122 to be sent. IOAM adds information to the already available data 123 packets and therefore cannot be considered passive. In terms of the 124 classification given in [RFC7799] IOAM could be portrayed as Hybrid 125 Type 1. IOAM mechanisms can be leveraged where mechanisms using e.g. 126 ICMP do not apply or do not offer the desired results, such as 127 proving that a certain traffic flow takes a pre-defined path, SLA 128 verification for the live data traffic, detailed statistics on 129 traffic distribution paths in networks that distribute traffic across 130 multiple paths, or scenarios in which probe traffic is potentially 131 handled differently from regular data traffic by the network devices. 133 2. Conventions 135 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 136 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 137 document are to be interpreted as described in [RFC2119]. 139 Abbreviations used in this document: 141 E2E Edge to Edge 143 Geneve: Generic Network Virtualization Encapsulation 144 [I-D.ietf-nvo3-geneve] 146 IOAM: In-situ Operations, Administration, and Maintenance 148 MTU: Maximum Transmit Unit 150 NSH: Network Service Header [RFC8300] 152 OAM: Operations, Administration, and Maintenance 154 POT: Proof of Transit 156 SFC: Service Function Chain 158 SID: Segment Identifier 160 SR: Segment Routing 162 VXLAN-GPE: Virtual eXtensible Local Area Network, Generic Protocol 163 Extension [I-D.ietf-nvo3-vxlan-gpe] 165 3. Scope, Applicability, and Assumptions 167 IOAM deployment assumes a set of constraints, requirements, and 168 guiding principles which are described in this section. 170 Scope: This document defines the data fields and associated data 171 types for in-situ OAM. The in-situ OAM data field can be transported 172 by a variety of transport protocols, including NSH, Segment Routing, 173 Geneve, IPv6, or IPv4. Specification details for these different 174 transport protocols are outside the scope of this document. 176 Deployment domain (or scope) of in-situ OAM deployment: IOAM is a 177 network domain focused feature, with "network domain" being a set of 178 network devices or entities within a single administration. For 179 example, a network domain can include an enterprise campus using 180 physical connections between devices or an overlay network using 181 virtual connections / tunnels for connectivity between said devices. 182 A network domain is defined by its perimeter or edge. Designers of 183 protocol encapsulations for IOAM must specify mechanisms to ensure 184 that IOAM data stays within an IOAM domain. In addition, the 185 operator of such a domain is expected to put provisions in place to 186 ensure that IOAM data does not leak beyond the edge of an IOAM 187 domain, e.g. using for example packet filtering methods. The 188 operator should consider the potential operational impact of IOAM to 189 mechanisms such as ECMP processing (e.g. load-balancing schemes 190 based on packet length could be impacted by the increased packet size 191 due to IOAM), path MTU (i.e. ensure that the MTU of all links within 192 a domain is sufficiently large to support the increased packet size 193 due to IOAM) and ICMP message handling (i.e. in case of a native IPv6 194 transport, IOAM support for ICMPv6 Echo Request/Reply could desired 195 which would translate into ICMPv6 extensions to enable IOAM-Data- 196 Fields to be copied from an Echo Request message to an Echo Reply 197 message). 199 IOAM control points: IOAM-Data-Fields are added to or removed from 200 the live user traffic by the devices which form the edge of a domain. 201 Devices which form an IOAM-Domain can add, update or remove IOAM- 202 Data-Fields. Edge devices of an IOAM-Domain can be hosts or network 203 devices. 205 Traffic-sets that IOAM is applied to: IOAM can be deployed on all or 206 only on subsets of the live user traffic. It SHOULD be possible to 207 enable IOAM on a selected set of traffic (e.g., per interface, based 208 on an access control list or flow specification defining a specific 209 set of traffic, etc.) The selected set of traffic can also be all 210 traffic. 212 Encapsulation independence: Data formats for IOAM SHOULD be defined 213 in a transport-independent manner. IOAM applies to a variety of 214 encapsulating protocols. A definition of how IOAM-Data-Fields are 215 encapsulated into "parent" protocols, like NSH or IPv6 is outside the 216 scope of this document. 218 Layering: If several encapsulation protocols (e.g., in case of 219 tunneling) are stacked on top of each other, IOAM-Data-Fields could 220 be present at multiple layers. The behavior follows the ships-in- 221 the-night model, i.e. IOAM-Data-Fields in one layer are independent 222 from IOAM-Data-Fields in another layer. Layering allows operators to 223 instrument the protocol layer they want to measure. The different 224 layers could, but do not have to share the same IOAM encapsulation 225 mechanisms. 227 IOAM implementation: The definition of the IOAM-Data-Fields take the 228 specifics of devices with hardware data-plane and software data-plane 229 into account. 231 4. IOAM Data-Fields, Types, Nodes 233 This section details IOAM-related nomenclature and describes data 234 types such as IOAM-Data-Fields, IOAM-Types, IOAM-Namespaces as well 235 as the different types of IOAM nodes. 237 4.1. IOAM Data-Fields and Option-Types 239 An IOAM-Data-Field is a set of bits with a defined format and 240 meaning, which can be stored at a certain place in a packet for the 241 purpose of IOAM. 243 To accommodate the different uses of IOAM, IOAM-Data-Fields fall into 244 different categories. In IOAM these categories are referred to as 245 IOAM-Option-Types. A common registry is maintained for IOAM-Option- 246 Types, see Section 7.2 for details. Corresponding to these IOAM- 247 Option-Types, different IOAM-Data-Fields are defined. IOAM-Data- 248 Fields can be encapsulated into a variety of protocols, such as NSH, 249 Geneve, IPv6, etc. The definition of how IOAM-Data-Fields are 250 encapsulated into other protocols is outside the scope of this 251 document. 253 This document defines four IOAM-Option-Types: 255 o Pre-allocated Trace Option-Type 257 o Incremental Trace Option-Type 259 o Proof of Transit (POT) Option-Type 261 o Edge-to-Edge (E2E) Option-Type 263 4.2. IOAM-Domains and types of IOAM Nodes 265 IOAM is expected to be deployed in a specific domain. The part of 266 the network which employs IOAM is referred to as the "IOAM-Domain". 267 One or more IOAM-Option-Types are added to a packet upon entering the 268 IOAM-Domain and are removed from the packet when exiting the domain. 269 Within the IOAM-Domain, the IOAM-Data-Fields MAY be updated by 270 network nodes that the packet traverses. An IOAM-Domain consists of 271 "IOAM encapsulating nodes", "IOAM decapsulating nodes" and "IOAM 272 transit nodes". The role of a node (i.e. encapsulating, transit, 273 decapsulating) is defined within an IOAM-Namespace (see below). A 274 node can have different roles in different IOAM-Namespaces. 276 A device which adds at least one IOAM-Option-Type to the packet is 277 called the "IOAM encapsulating node", whereas a device which removes 278 an IOAM-Option-Type is referred to as the "IOAM decapsulating node". 279 Nodes within the domain which are aware of IOAM data and read and/or 280 write or process the IOAM data are called "IOAM transit nodes". IOAM 281 nodes which add or remove the IOAM-Data-Fields can also update the 282 IOAM-Data-Fields at the same time. Or in other words, IOAM 283 encapsulating or decapsulating nodes can also serve as IOAM transit 284 nodes at the same time. Note that not every node in an IOAM domain 285 needs to be an IOAM transit node. For example, a deployment might 286 require that packets traverse a set of firewalls which support IOAM. 287 In that case, only the set of firewall nodes would be IOAM transit 288 nodes rather than all nodes. 290 An "IOAM encapsulating node" incorporates one or more IOAM-Option- 291 Types (from the list of IOAM-Types, see Section 7.2) into packets 292 that IOAM is enabled for. If IOAM is enabled for a selected subset 293 of the traffic, the IOAM encapsulating node is responsible for 294 applying the IOAM functionality to the selected subset. 296 An "IOAM transit node" updates one or more of the IOAM-Data-Fields. 297 If both the Pre-allocated and the Incremental Trace Option-Types are 298 present in the packet, each IOAM transit node will update at most one 299 of these Option-Types. A transit node MUST NOT add new IOAM-Option- 300 Types to a packet, and MUST NOT change the IOAM-Data-Fields of an 301 IOAM Edge-to-Edge Option-Type. 303 An "IOAM decapsulating node" removes IOAM-Option-Type(s) from 304 packets. 306 The role of an IOAM-encapsulating, IOAM-transit or IOAM-decapsulating 307 node is always performed within a specific IOAM-Namespace. This 308 means that an IOAM node which is e.g. an IOAM-decapsulating node for 309 IOAM-Namespace "A" but not for IOAM-Namespace "B" will only remove 310 the IOAM-Option-Types for IOAM-Namespace "A" from the packet. An 311 IOAM decapsulating node situated at the edge of an IOAM domain MUST 312 remove all IOAM-Option-Types and associated encapsulation headers for 313 all IOAM-Namespaces from the packet. 315 IOAM-Namespaces allow for a namespace-specific definition and 316 interpretation of IOAM-Data-Fields. An interface-id could for 317 example point to a physical interface (e.g., to understand which 318 physical interface of an aggregated link is used when receiving or 319 transmitting a packet) whereas in another case it could refer to a 320 logical interface (e.g., in case of tunnels). Please refer to 321 Section 4.3 for details on IOAM-Namespaces. 323 4.3. IOAM-Namespaces 325 A subset or all of the IOAM-Option-Types and their corresponding 326 IOAM-Data-Fields can be associated to an IOAM-Namespace. IOAM- 327 Namespaces add further context to IOAM-Option-Types and associated 328 IOAM-Data-Fields. Any IOAM-Namespace MUST interpret the IOAM-Option- 329 Types and associated IOAM-Data-Fields per the definition in this 330 document. IOAM-Namespaces group nodes to support different 331 deployment approaches of IOAM (see a few example use-cases below) as 332 well as resolve issues which can occur due to IOAM-Data-Fields not 333 being globally unique (e.g. IOAM node identifiers do not have to be 334 globally unique). IOAM-Data-Fields significance is always within a 335 particular IOAM-Namespace. 337 An IOAM-Namespace is identified by a 16-bit namespace identifier 338 (Namespace-ID). IOAM-Namespace identifiers MUST be present and 339 populated in all IOAM-Option-Types. The Namespace-ID value is 340 divided into two sub-ranges: 342 o An operator-assigned range from 0x0001 to 0x7FFF 344 o An IANA-assigned range from 0x8000 to 0xFFFF 346 The IANA-assigned range is intended to allow future extensions to 347 have new and interoperable IOAM functionality, while the operator- 348 assigned range is intended to be domain specific, and managed by the 349 network operator. The Namespace-ID value of 0x0000 is default and 350 known to all the nodes implementing IOAM. 352 Namespace identifiers allow devices which are IOAM capable to 353 determine: 355 o whether IOAM-Option-Type(s) need to be processed by a device: If 356 the Namespace-ID contained in a packet does not match any 357 Namespace-ID the node is configured to operate on, then the node 358 MUST NOT change the contents of the IOAM-Data-Fields. 360 o which IOAM-Option-Type needs to be processed/updated in case there 361 are multiple IOAM-Option-Types present in the packet. Multiple 362 IOAM-Option-Types can be present in a packet in case of 363 overlapping IOAM-Domains or in case of a layered IOAM deployment. 365 o whether IOAM-Option-Type(s) should be removed from the packet, 366 e.g. at a domain edge or domain boundary. 368 IOAM-Namespaces support several different uses: 370 o IOAM-Namespaces can be used by an operator to distinguish 371 different operational domains. Devices at domain edges can filter 372 on Namespace-IDs to provide for proper IOAM-Domain isolation. 374 o IOAM-Namespaces provide additional context for IOAM-Data-Fields 375 and thus ensure that IOAM-Data-Fields are unique and can be 376 interpreted properly by management stations or network 377 controllers. While, for example, the node identifier field 378 (node_id, see below) does not need to be unique in a deployment 379 (e.g. an operator may wish to use different node identifiers for 380 different IOAM layers, even within the same device; or node 381 identifiers might not be unique for other organizational reasons, 382 such as after a merger of two formerly separated organizations), 383 the combination of node_id and Namespace-ID will always be unique. 384 Similarly, IOAM-Namespaces can be used to define how certain IOAM- 385 Data-Fields are interpreted: IOAM offers three different timestamp 386 format options. The Namespace-ID can be used to determine the 387 timestamp format. IOAM-Data-Fields (e.g. buffer occupancy) which 388 do not have a unit associated are to be interpreted within the 389 context of a IOAM-Namespace. 391 o IOAM-Namespaces can be used to identify different sets of devices 392 (e.g., different types of devices) in a deployment: If an operator 393 desires to insert different IOAM-Data-Fields based on the device, 394 the devices could be grouped into multiple IOAM-Namespaces. This 395 could be due to the fact that the IOAM feature set differs between 396 different sets of devices, or it could be for reasons of optimized 397 space usage in the packet header. It could also stem from 398 hardware or operational limitations on the size of the trace data 399 that can be added and processed, preventing collection of a full 400 trace for a flow. 402 * Assigning different IOAM Namespace-IDs to different sets of 403 nodes or network partitions and using the Namespace-ID as a 404 selector at the IOAM encapsulating node, a full trace for a 405 flow could be collected and constructed via partial traces in 406 different packets of the same flow. Example: An operator could 407 choose to group the devices of a domain into two IOAM- 408 Namespaces, in a way that at average, only every second hop 409 would be recorded by any device. To retrieve a full view of 410 the deployment, the captured IOAM-Data-Fields of the two IOAM- 411 Namespaces need to be correlated. 413 * Assigning different IOAM Namespace-IDs to different sets of 414 nodes or network partitions and using a separate instance of an 415 IOAM-Option-Type for each Namespace-ID, a full trace for a flow 416 could be collected and constructed via partial traces from each 417 IOAM-Option-Type in each of the packets in the flow. Example: 418 An operator could choose to group the devices of a domain into 419 two IOAM-Namespaces, in a way that each IOAM-Namespace is 420 represented by one of two IOAM-Option-Types in the packet. 421 Each node would record data only for the IOAM-Namespace that it 422 belongs to, ignoring the other IOAM-Option-Type with a IOAM- 423 Namespace to which it doesn't belong. To retrieve a full view 424 of the deployment, the captured IOAM-Data-Fields of the two 425 IOAM-Namespaces need to be correlated. 427 4.4. IOAM Trace Option-Types 429 "IOAM tracing data" is expected to be collected at every IOAM transit 430 node that a packet traverses to ensure visibility into the entire 431 path a packet takes within an IOAM-Domain. I.e., in a typical 432 deployment all nodes in an IOAM-Domain would participate in IOAM and 433 thus be IOAM transit nodes, IOAM encapsulating or IOAM decapsulating 434 nodes. If not all nodes within a domain are IOAM capable, IOAM 435 tracing information (i.e., node data, see below) will only be 436 collected on those nodes which are IOAM capable. Nodes which are not 437 IOAM capable will forward the packet without any changes to the IOAM- 438 Data-Fields. The maximum number of hops and the minimum path MTU of 439 the IOAM domain is assumed to be known. 441 To optimize hardware and software implementations IOAM tracing is 442 defined as two separate options. Any deployment MAY choose to 443 configure and support one or both of the following options. 445 Pre-allocated Trace-Option: This trace option is defined as a 446 container of node data fields (see below) with pre-allocated space 447 for each node to populate its information. This option is useful 448 for implementations where it is efficient to allocate the space 449 once and index into the array to populate the data during transit 450 (e.g., software forwarders often fall into this class). The IOAM 451 encapsulating node allocates space for Pre-allocated Trace Option- 452 Type in the packet and sets corresponding fields in this IOAM- 453 Option-Type. The IOAM encapsulating node allocates an array which 454 is used to store operational data retrieved from every node while 455 the packet traverses the domain. IOAM transit nodes update the 456 content of the array, and possibly update the checksums of outer 457 headers. A pointer which is part of the IOAM trace data, points 458 to the next empty slot in the array. An IOAM transit node that 459 updates the content of the pre-allocated option also updates the 460 value of the pointer, which specifies where the next IOAM transit 461 node fills in its data.The "node data list" array (see below) in 462 the packet is populated iteratively as the packet traverses the 463 network, starting with the last entry of the array, i.e., "node 464 data list [n]" is the first entry to be populated, "node data list 465 [n-1]" is the second one, etc. 467 Incremental Trace-Option: This trace option is defined as a 468 container of node data fields where each node allocates and pushes 469 its node data immediately following the option header. This type 470 of trace recording is useful for some of the hardware 471 implementations as it eliminates the need for the transit network 472 elements to read the full array in the option and allows for 473 arbitrarily long packets as the MTU allows. The IOAM 474 encapsulating node allocates space for the Incremental Trace 475 Option-Type. Based on operational state and configuration, the 476 IOAM encapsulating node sets the fields in the Option-Type that 477 control what IOAM-Data-Fields should be collected and how large 478 the node data list can grow. IOAM transit nodes push their node 479 data to the node data list, decrease the remaining length 480 available to subsequent nodes and adjust the lengths and possibly 481 checksums in outer headers. 483 A particular implementation of IOAM MAY choose to support only one of 484 the two trace option types. In the event that both options are 485 utilized at the same time, the Incremental Trace-Option MUST be 486 placed before the Pre-allocated Trace-Option. Deployments which mix 487 devices which either the Incremental Trace-Option or the Pre- 488 allocated Trace-Option could result in both Option-Types being 489 present in a packet. Given that the operator knows which equipment 490 is deployed in a particular IOAM, the operator will decide by means 491 of configuration which type(s) of trace options will be used for a 492 particular domain. 494 Every node data entry holds information for a particular IOAM transit 495 node that is traversed by a packet. The IOAM decapsulating node 496 removes the IOAM-Option-Type(s) and processes and/or exports the 497 associated data. Like all IOAM-Data-Fields, the IOAM-Data-Fields of 498 the IOAM-Trace-Option-Types are defined in the context of an IOAM- 499 Namespace. 501 IOAM tracing can collect the following types of information: 503 o Identification of the IOAM node. An IOAM node identifier can 504 match to a device identifier or a particular control point or 505 subsystem within a device. 507 o Identification of the interface that a packet was received on, 508 i.e. ingress interface. 510 o Identification of the interface that a packet was sent out on, 511 i.e. egress interface. 513 o Time of day when the packet was processed by the node as well as 514 the transit delay. Different definitions of processing time are 515 feasible and expected, though it is important that all devices of 516 an in-situ OAM domain follow the same definition. 518 o Generic data: Format-free information where syntax and semantic of 519 the information is defined by the operator in a specific 520 deployment. For a specific IOAM-Namespace, all IOAM nodes should 521 interpret the generic data the same way. Examples for generic 522 IOAM data include geo-location information (location of the node 523 at the time the packet was processed), buffer queue fill level or 524 cache fill level at the time the packet was processed, or even a 525 battery charge level. 527 o Information to detect whether IOAM trace data was added at every 528 hop or whether certain hops in the domain weren't IOAM transit 529 nodes. 531 4.4.1. Pre-allocated and Incremental Trace Option-Types 533 The IOAM Pre-allocated Trace-Option and the IOAM Incremental Trace- 534 Option have similar formats. Except where noted below, the internal 535 formats and fields of the two trace options are identical. Both 536 Trace-Options consist of a fixed size "trace option header" and a 537 variable data space to store gathered data, the "node data list": 539 Pre-allocated and incremental trace option headers: 541 0 1 2 3 542 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 543 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 544 | Namespace-ID |NodeLen | Flags | RemainingLen| 545 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 546 | IOAM-Trace-Type | Reserved | 547 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 549 The trace option data MUST be 4-octet aligned: 551 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+<-+ 552 | | | 553 | node data list [0] | | 554 | | | 555 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ D 556 | | a 557 | node data list [1] | t 558 | | a 559 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 560 ~ ... ~ S 561 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ p 562 | | a 563 | node data list [n-1] | c 564 | | e 565 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 566 | | | 567 | node data list [n] | | 568 | | | 569 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+<-+ 570 Namespace-ID: 16-bit identifier of an IOAM-Namespace. The 571 Namespace-ID value of 0x0000 is defined as the default value and 572 MUST be known to all the nodes implementing IOAM. For any other 573 Namespace-ID value that does not match any Namespace-ID the node 574 is configured to operate on, the node MUST NOT change the contents 575 of the IOAM-Data-Fields. 577 NodeLen: 5-bit unsigned integer. This field specifies the length of 578 data added by each node in multiples of 4-octets, excluding the 579 length of the "Opaque State Snapshot" field. 581 If IOAM-Trace-Type bit 22 is not set, then NodeLen specifies the 582 actual length added by each node. If IOAM-Trace-Type bit 22 is 583 set, then the actual length added by a node would be (NodeLen + 584 Opaque Data Length). 586 For example, if 3 IOAM-Trace-Type bits are set and none of them 587 are wide, then NodeLen would be 3. If 3 IOAM-Trace-Type bits are 588 set and 2 of them are wide, then NodeLen would be 5. 590 An IOAM encapsulating node must set NodeLen. 592 A node receiving an IOAM Pre-allocated or Incremental Trace-Option 593 may rely on the NodeLen value, or it may ignore the NodeLen value 594 and calculate the node length from the IOAM-Trace-Type bits (see 595 below). 597 Flags 4-bit field. Flags are allocated by IANA, as specified in 598 Section 7.4. This document allocates a single flag as follows: 600 Bit 0 "Overflow" (O-bit) (most significant bit). This bit is set 601 by the network element if there are not enough octets left to 602 record node data, no field is added and the overflow "O-bit" 603 must be set to "1" in the IOAM-Trace-Option header. This is 604 useful for transit nodes to ignore further processing of the 605 option. 607 RemainingLen: 7-bit unsigned integer. This field specifies the data 608 space in multiples of 4-octets remaining for recording the node 609 data, before the node data list is considered to have overflowed. 610 When RemainingLen reaches 0, nodes are no longer allowed to add 611 node data. Given that the sender knows the minimum path MTU, the 612 sender MAY set the initial value of RemainingLen according to the 613 number of node data bytes allowed before exceeding the MTU. 614 Subsequent nodes can carry out a simple comparison between 615 RemainingLen and NodeLen, along with the length of the "Opaque 616 State Snapshot" if applicable, to determine whether or not data 617 can be added by this node. When node data is added, the node MUST 618 decrease RemainingLen by the amount of data added. In the pre- 619 allocated trace option, RemainingLength is used to derive the 620 offset in data space to record the node data element. 621 Specifically, the recording of the node data element would start 622 from RemainingLen - NodeLen - sizeof(opaque snapshot) in 4 octet 623 units. 625 IOAM-Trace-Type: A 24-bit identifier which specifies which data 626 types are used in this node data list. 628 The IOAM-Trace-Type value is a bit field. The following bits are 629 defined in this document, with details on each bit described in 630 the Section 4.4.2. The order of packing the data fields in each 631 node data element follows the bit order of the IOAM-Trace-Type 632 field, as follows: 634 Bit 0 (Most significant bit) When set indicates presence of 635 Hop_Lim and node_id in the node data. 637 Bit 1 When set indicates presence of ingress_if_id and 638 egress_if_id (short format) in the node data. 640 Bit 2 When set indicates presence of timestamp seconds in the 641 node data. 643 Bit 3 When set indicates presence of timestamp subseconds in 644 the node data. 646 Bit 4 When set indicates presence of transit delay in the node 647 data. 649 Bit 5 When set indicates presence of IOAM-Namespace specific 650 data (short format) in the node data. 652 Bit 6 When set indicates presence of queue depth in the node 653 data. 655 Bit 7 When set indicates presence of the Checksum Complement 656 node data. 658 Bit 8 When set indicates presence of Hop_Lim and node_id in 659 wide format in the node data. 661 Bit 9 When set indicates presence of ingress_if_id and 662 egress_if_id in wide format in the node data. 664 Bit 10 When set indicates presence of IOAM-Namespace specific 665 data in wide format in the node data. 667 Bit 11 When set indicates presence of buffer occupancy in the 668 node data. 670 Bit 12-21 Undefined. An IOAM encapsulating node MUST set the 671 value of each of these bits to 0. If an IOAM transit 672 node receives a packet with one or more of these bits set 673 to 1, it must either: 675 1. Add corresponding node data filled with the reserved 676 value 0xFFFFFFFF, after the node data fields for the 677 IOAM-Trace-Type bits defined above, such that the 678 total node data added by this node in units of 679 4-octets is equal to NodeLen, or 681 2. Not add any node data fields to the packet, even for 682 the IOAM-Trace-Type bits defined above. 684 Bit 22 When set indicates presence of variable length Opaque 685 State Snapshot field. 687 Bit 23 Reserved: Must be set to zero upon transmission and 688 ignored upon receipt. 690 Section 4.4.2 describes the IOAM-Data-Types and their formats. 691 Within an IOAM-Domain possible combinations of these bits making 692 the IOAM-Trace-Type can be restricted by configuration knobs. 694 Reserved: 8-bits. Must be zero. 696 Node data List [n]: Variable-length field. The type of which is 697 determined by the IOAM-Trace-Type bit representing the n-th node 698 data in the node data list. The node data list is encoded 699 starting from the last node data of the path. The first element 700 of the node data list (node data list [0]) contains the last node 701 of the path while the last node data of the node data list (node 702 data list[n]) contains the first node data of the path traced. 703 Populating the node data list in this way ensures that the order 704 of node data list is the same for incremental and pre-allocated 705 trace options. In the pre-allocated trace option, the index 706 contained in RemainingLen identifies the offset for current active 707 node data to be populated. 709 4.4.2. IOAM node data fields and associated formats 711 All the IOAM-Data-Fields MUST be 4-octet aligned. If a node which is 712 supposed to update an IOAM-Data-Field is not capable of populating 713 the value of a field set in the IOAM-Trace-Type, the field value MUST 714 be set to 0xFFFFFFFF for 4-octet fields or 0xFFFFFFFFFFFFFFFF for 715 8-octet fields, indicating that the value is not populated, except 716 when explicitly specified in the field description below. 718 Some IOAM-Data-Fields defined below, such as interface identifiers or 719 IOAM-Namespace specific data, are defined in both "short format" as 720 well as "wide format". Their use is not exclusive. A deployment 721 could choose to leverage both. For example, ingress_if_id_(short 722 format) could be an identifier for the physical interface, whereas 723 ingress_if_id_(wide format) could be an identifier for a logical sub- 724 interface of that physical interface. 726 Data field and associated data type for each of the IOAM-Data-Fields 727 is shown below: 729 Hop_Lim and node_id: 4-octet field defined as follows: 731 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 732 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 733 | Hop_Lim | node_id | 734 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 736 Hop_Lim: 1-octet unsigned integer. It is set to the Hop Limit 737 value in the packet at the node that records this data. Hop 738 Limit information is used to identify the location of the node 739 in the communication path. This is copied from the lower 740 layer, e.g., TTL value in IPv4 header or hop limit field from 741 IPv6 header of the packet when the packet is ready for 742 transmission. The semantics of the Hop_Lim field depend on the 743 lower layer protocol that IOAM is encapsulated over, and 744 therefore its specific semantics are outside the scope of this 745 memo. 747 node_id: 3-octet unsigned integer. Node identifier field to 748 uniquely identify a node within the IOAM-Namespace and 749 associated IOAM-Domain. The procedure to allocate, manage and 750 map the node_ids is beyond the scope of this document. 752 ingress_if_id and egress_if_id: 4-octet field defined as follows: 754 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 755 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 756 | ingress_if_id | egress_if_id | 757 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 759 ingress_if_id: 2-octet unsigned integer. Interface identifier to 760 record the ingress interface the packet was received on. 762 egress_if_id: 2-octet unsigned integer. Interface identifier to 763 record the egress interface the packet is forwarded out of. 765 Note that due to the fact that IOAM uses its own IOAM-Namespaces 766 for IOAM-Data-Fields, data fields like interface identifiers can 767 be used in a flexible way to represent system resources that are 768 associated with ingressing or egressing packets, i.e. 769 ingress_if_id could represent a physical interface, a virtual or 770 logical interface, or even a queue. 772 timestamp seconds: 4-octet unsigned integer. Absolute timestamp in 773 seconds that specifies the time at which the packet was received 774 by the node. This field has three possible formats; based on 775 either PTP [IEEE1588v2], NTP [RFC5905], or POSIX [POSIX]. The 776 three timestamp formats are specified in Section 5. In all three 777 cases, the Timestamp Seconds field contains the 32 most 778 significant bits of the timestamp format that is specified in 779 Section 5. If a node is not capable of populating this field, it 780 assigns the value 0xFFFFFFFF. Note that this is a legitimate 781 value that is valid for 1 second in approximately 136 years; the 782 analyzer should correlate several packets or compare the timestamp 783 value to its own time-of-day in order to detect the error 784 indication. 786 timestamp subseconds: 4-octet unsigned integer. Absolute timestamp 787 in subseconds that specifies the time at which the packet was 788 received by the node. This field has three possible formats; 789 based on either PTP [IEEE1588v2], NTP [RFC5905], or POSIX [POSIX]. 790 The three timestamp formats are specified in Section 5. In all 791 three cases, the Timestamp Subseconds field contains the 32 least 792 significant bits of the timestamp format that is specified in 793 Section 5. If a node is not capable of populating this field, it 794 assigns the value 0xFFFFFFFF. Note that this is a legitimate 795 value in the NTP format, valid for approximately 233 picoseconds 796 in every second. If the NTP format is used the analyzer should 797 correlate several packets in order to detect the error indication. 799 transit delay: 4-octet unsigned integer in the range 0 to 2^31-1. 800 It is the time in nanoseconds the packet spent in the transit 801 node. This can serve as an indication of the queuing delay at the 802 node. If the transit delay exceeds 2^31-1 nanoseconds then the 803 top bit 'O' is set to indicate overflow and value set to 804 0x80000000. When this field is part of the data field but a node 805 populating the field is not able to fill it, the field position in 806 the field must be filled with value 0xFFFFFFFF to mean not 807 populated. 809 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 810 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 811 |O| transit delay | 812 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 814 namespace specific data: 4-octet field which can be used by the node 815 to add IOAM-Namespace specific data. This represents a "free- 816 format" 4-octet bit field with its semantics defined in the 817 context of a specific IOAM-Namespace. 819 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 820 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 821 | namespace specific data | 822 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 824 queue depth: 4-octet unsigned integer field. This field indicates 825 the current length of the egress interface queue of the interface 826 from where the packet is forwarded out. The queue depth is 827 expressed as the current number of memory buffers used by the 828 queue (a packet may consume one or more memory buffers, depending 829 on its size). 831 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 832 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 833 | queue depth | 834 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 836 Hop_Lim and node_id wide: 8-octet field defined as follows: 838 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 839 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 840 | Hop_Lim | node_id ~ 841 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 842 ~ node_id (contd) | 843 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 845 Hop_Lim: 1-octet unsigned integer. It is set to the Hop Limit 846 value in the packet at the node that records this data. Hop 847 Limit information is used to identify the location of the node 848 in the communication path. This is copied from the lower layer 849 for e.g. TTL value in IPv4 header or hop limit field from IPv6 850 header of the packet. The semantics of the Hop_Lim field 851 depend on the lower layer protocol that IOAM is encapsulated 852 over, and therefore its specific semantics are outside the 853 scope of this memo. 855 node_id: 7-octet unsigned integer. Node identifier field to 856 uniquely identify a node within the IOAM-Namespace and 857 associated IOAM-Domain. The procedure to allocate, manage and 858 map the node_ids is beyond the scope of this document. 860 ingress_if_id and egress_if_id wide: 8-octet field defined as 861 follows: 863 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 864 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 865 | ingress_if_id | 866 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 867 | egress_if_id | 868 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 870 ingress_if_id: 4-octet unsigned integer. Interface identifier to 871 record the ingress interface the packet was received on. 873 egress_if_id: 4-octet unsigned integer. Interface identifier to 874 record the egress interface the packet is forwarded out of. 876 namespace specific data wide: 8-octet field which can be used by the 877 node to add IOAM-Namespace specific data. This represents a 878 "free-format" 8-octet bit field with its semantics defined in the 879 context of a specific IOAM-Namespace. 881 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 882 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 883 | namespace specific data ~ 884 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 885 ~ namespace specific data (contd) | 886 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 888 buffer occupancy: 4-octet unsigned integer field. This field 889 indicates the current status of the occupancy of the common buffer 890 pool used by a set of queues. The units of this field depend on 891 the equipment type and deployment and has to be interpreted within 892 the context of an IOAM-Namespace and/or node-id if used. 894 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 895 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 896 | buffer occupancy | 897 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 899 Checksum Complement: 4-octet node data which contains a 4-octet 900 Checksum Complement field. The Checksum Complement is useful when 901 IOAM is transported over encapsulations that make use of a UDP 902 transport, such as VXLAN-GPE or Geneve. Without the Checksum 903 Complement, nodes adding IOAM node data must update the UDP 904 Checksum field. When the Checksum Complement is present, an IOAM 905 encapsulating node or IOAM transit node adding node data MUST 906 carry out one of the following two alternatives in order to 907 maintain the correctness of the UDP Checksum value: 909 1. Recompute the UDP Checksum field. 911 2. Use the Checksum Complement to make a checksum-neutral update 912 in the UDP payload; the Checksum Complement is assigned a 913 value that complements the rest of the node data fields that 914 were added by the current node, causing the existing UDP 915 Checksum field to remain correct. 917 IOAM decapsulating nodes MUST recompute the UDP Checksum field, 918 since they do not know whether previous hops modified the UDP 919 Checksum field or the Checksum Complement field. 921 Checksum Complement fields are used in a similar manner in 922 [RFC7820] and [RFC7821]. 924 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 925 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 926 | Checksum Complement | 927 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 929 Opaque State Snapshot: Opaque State Snapshot is a variable length 930 field and immediately follows the fixed length IOAM-Data-Fields 931 defined above. It allows the network element to store an 932 arbitrary state in the node data field, without a pre-defined 933 schema. The schema is to be defined within the context of an 934 IOAM-Namespace. The schema needs to be made known to the analyzer 935 by some out-of-band mechanism. The specification of this 936 mechanism is beyond the scope of this document. A 24-bit "Schema 937 Id" field, interpreted within the context of an IOAM-Namespace, 938 indicates which particular schema is used, and should be 939 configured on the network element by the operator. 941 0 1 2 3 942 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 943 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 944 | Length | Schema ID | 945 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 946 | | 947 | | 948 | Opaque data | 949 ~ ~ 950 . . 951 . . 952 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 953 Length: 1-octet unsigned integer. It is the length in multiples 954 of 4-octets of the Opaque data field that follows Schema Id. 956 Schema ID: 3-octet unsigned integer identifying the schema of 957 Opaque data. 959 Opaque data: Variable length field. This field is interpreted as 960 specified by the schema identified by the Schema ID. 962 When this field is part of the data field but a node populating 963 the field has no opaque state data to report, the Length must be 964 set to 0 and the Schema ID must be set to 0xFFFFFF to mean no 965 schema. 967 4.4.3. Examples of IOAM node data 969 An entry in the "node data list" array can have different formats, 970 following the needs of the deployment. Some deployments might only 971 be interested in recording the node identifiers, whereas others might 972 be interested in recording node identifier and timestamp. The 973 section provides example entries of the "node data list". 975 0xD40000: IOAM-Trace-Type is 0xD40000 (0b110101000000000000000000) 976 then the format of node data is: 978 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 979 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 980 | Hop_Lim | node_id | 981 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 982 | ingress_if_id | egress_if_id | 983 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 984 | timestamp subseconds | 985 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 986 | namespace specific data | 987 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 989 0xC00000: IOAM-Trace-Type is 0xC00000 (0b110000000000000000000000) 990 then the format is: 992 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 993 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 994 | Hop_Lim | node_id | 995 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 996 | ingress_if_id | egress_if_id | 997 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 999 0x900000: IOAM-Trace-Type is 0x900000 (0b100100000000000000000000) 1000 then the format is: 1002 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1003 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1004 | Hop_Lim | node_id | 1005 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1006 | timestamp subseconds | 1007 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1009 0x840000: IOAM-Trace-Type is 0x840000 (0b100001000000000000000000) 1010 then the format is: 1012 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1013 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1014 | Hop_Lim | node_id | 1015 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1016 | namespace specific data | 1017 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1019 0x940000: IOAM-Trace-Type is 0x940000 (0b100101000000000000000000) 1020 then the format is: 1022 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1023 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1024 | Hop_Lim | node_id | 1025 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1026 | timestamp subseconds | 1027 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1028 | namespace specific data | 1029 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1031 0x308002: IOAM-Trace-Type is 0x308002 (0b001100001000000000000010) 1032 then the format is: 1034 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1035 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1036 | timestamp seconds | 1037 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1038 | timestamp subseconds | 1039 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1040 | Hop_Lim | node_id | 1041 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1042 | node_id(contd) | 1043 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1044 | Length | Schema Id | 1045 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1046 | | 1047 | | 1048 | Opaque data | 1049 ~ ~ 1050 . . 1051 . . 1052 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1054 4.5. IOAM Proof of Transit Option-Type 1056 IOAM Proof of Transit Option-Type is to support path or service 1057 function chain [RFC7665] verification use cases. Proof-of-transit 1058 uses methods like nested hashing or nested encryption of the IOAM 1059 data or mechanisms such as Shamir's Secret Sharing Schema (SSSS). 1060 While details on how the IOAM data for the proof of transit option is 1061 processed at IOAM encapsulating, decapsulating and transit nodes are 1062 outside the scope of the document, all of these approaches share the 1063 need to uniquely identify a packet as well as iteratively operate on 1064 a set of information that is handed from node to node. 1065 Correspondingly, two pieces of information are added as IOAM-Data- 1066 Fields to the packet: 1068 o Random: Unique identifier for the packet (e.g., 64-bits allow for 1069 the unique identification of 2^64 packets). 1071 o Cumulative: Information which is handed from node to node and 1072 updated by every node according to a verification algorithm. 1074 The IOAM Proof of Transit Option-Type consist of a fixed size "IOAM 1075 proof of transit option header" and "IOAM proof of transit option 1076 data fields": 1078 IOAM proof of transit option header: 1080 0 1 2 3 1081 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1082 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1083 | Namespace-ID |IOAM POT Type | IOAM POT flags| 1084 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1086 IOAM proof of transit Option-Type IOAM-Data-Fields MUST be 1087 4-octet aligned: 1089 0 1 2 3 1090 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1091 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1092 | POT Option data field determined by IOAM-POT-Type | 1093 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1095 Namespace-ID: 16-bit identifier of an IOAM-Namespace. The 1096 Namespace-ID value of 0x0000 is defined as the default value and 1097 MUST be known to all the nodes implementing IOAM. For any other 1098 Namespace-ID value that does not match any Namespace-ID the node 1099 is configured to operate on, the node MUST NOT change the contents 1100 of the IOAM-Data-Fields. 1102 IOAM POT Type: 8-bit identifier of a particular POT variant that 1103 specifies the POT data that is included. This document defines 1104 POT Type 0: 1106 0: POT data is a 16 Octet field as described below. 1108 IOAM POT flags: 8-bit. Following flags are defined: 1110 Bit 0 "Profile-to-use" (P-bit) (most significant bit). For IOAM 1111 POT types that use a maximum of two profiles to drive 1112 computation, indicates which POT-profile is used. The two 1113 profiles are numbered 0, 1. 1115 Bit 1-7 Reserved: Must be set to zero upon transmission and 1116 ignored upon receipt. 1118 POT Option data: Variable-length field. The type of which is 1119 determined by the IOAM-POT-Type. 1121 4.5.1. IOAM Proof of Transit Type 0 1123 IOAM proof of transit option of IOAM POT Type 0: 1125 0 1 2 3 1126 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1127 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1128 | Namespace-ID |IOAM POT Type=0|P|R R R R R R R| 1129 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+<-+ 1130 | Random | | 1131 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ P 1132 | Random(contd) | O 1133 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ T 1134 | Cumulative | | 1135 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 1136 | Cumulative (contd) | | 1137 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+<-+ 1139 Namespace-ID: 16-bit identifier of an IOAM-Namespace. The 1140 Namespace-ID value of 0x0000 is defined as the default value and 1141 MUST be known to all the nodes implementing IOAM. For any other 1142 Namespace-ID value that does not match any Namespace-ID the node 1143 is configured to operate on, the node MUST NOT change the contents 1144 of the IOAM-Data-Fields. 1146 IOAM POT Type: 8-bit identifier of a particular POT variant that 1147 specifies the POT data that is included. This section defines the 1148 POT data when the IOAM POT Type is set to the value 0. 1150 P bit: 1-bit. "Profile-to-use" (P-bit) (most significant bit). 1151 Indicates which POT-profile is used to generate the Cumulative. 1152 Any node participating in POT will have a maximum of 2 profiles 1153 configured that drive the computation of cumulative. The two 1154 profiles are numbered 0, 1. This bit conveys whether profile 0 or 1155 profile 1 is used to compute the Cumulative. 1157 R (7 bits): 7-bit IOAM POT flags for future use. MUST be set to 1158 zero upon transmission and ignored upon receipt. 1160 Random: 64-bit Per packet Random number. 1162 Cumulative: 64-bit Cumulative that is updated at specific nodes by 1163 processing per packet Random number field and configured 1164 parameters. 1166 Note: Larger or smaller sizes of "Random" and "Cumulative" data are 1167 feasible and could be required for certain deployments (e.g. in case 1168 of space constraints in the transport protocol used). Future 1169 versions of this document will address different sizes of data for 1170 "proof of transit". 1172 4.6. IOAM Edge-to-Edge Option-Type 1174 The IOAM Edge-to-Edge Option-Type is to carry data that is added by 1175 the IOAM encapsulating node and interpreted by IOAM decapsulating 1176 node. The IOAM transit nodes MAY process the data but MUST NOT 1177 modify it. 1179 The IOAM Edge-to-Edge Option-Type consist of a fixed size "IOAM Edge- 1180 to-Edge Option-Type header" and "IOAM Edge-to-Edge Option-Type data 1181 fields": 1183 IOAM Edge-to-Edge Option-Type header: 1185 0 1 2 3 1186 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1187 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1188 | Namespace-ID | IOAM-E2E-Type | 1189 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1191 IOAM Edge-to-Edge Option-Type IOAM-Data-Fields MUST 1192 be 4-octet aligned: 1194 0 1 2 3 1195 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1196 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1197 | E2E Option data field determined by IOAM-E2E-Type | 1198 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1200 Namespace-ID: 16-bit identifier of an IOAM-Namespace. The 1201 Namespace-ID value of 0x0000 is defined as the default value and 1202 MUST be known to all the nodes implementing IOAM. For any other 1203 Namespace-ID value that does not match any Namespace-ID the node 1204 is configured to operate on, then the node MUST NOT change the 1205 contents of the IOAM-Data-Fields. 1207 IOAM-E2E-Type: A 16-bit identifier which specifies which data types 1208 are used in the E2E option data. The IOAM-E2E-Type value is a bit 1209 field. The order of packing the E2E option data field elements 1210 follows the bit order of the IOAM-E2E-Type field, as follows: 1212 Bit 0 (Most significant bit) When set indicates presence of a 1213 64-bit sequence number added to a specific "packet group" 1214 which is used to detect packet loss, packet reordering, 1215 or packet duplication within the group. The "packet 1216 group" is deployment dependent and defined at the IOAM 1217 encapsulating node e.g. by n-tuple based classification 1218 of packets. 1220 Bit 1 When set indicates presence of a 32-bit sequence number 1221 added to a specific "packet group" which is used to 1222 detect packet loss, packet reordering, or packet 1223 duplication within that group. The "packet group" is 1224 deployment dependent and defined at the IOAM 1225 encapsulating node e.g. by n-tuple based classification 1226 of packets. 1228 Bit 2 When set indicates presence of timestamp seconds for the 1229 transmission of the frame. This 4-octet field has three 1230 possible formats; based on either PTP [IEEE1588v2], NTP 1231 [RFC5905], or POSIX [POSIX]. The three timestamp formats 1232 are specified in Section 5. In all three cases, the 1233 Timestamp Seconds field contains the 32 most significant 1234 bits of the timestamp format that is specified in 1235 Section 5. If a node is not capable of populating this 1236 field, it assigns the value 0xFFFFFFFF. Note that this 1237 is a legitimate value that is valid for 1 second in 1238 approximately 136 years; the analyzer should correlate 1239 several packets or compare the timestamp value to its own 1240 time-of-day in order to detect the error indication. 1242 Bit 3 When set indicates presence of timestamp subseconds for 1243 the transmission of the frame. This 4-octet field has 1244 three possible formats; based on either PTP [IEEE1588v2], 1245 NTP [RFC5905], or POSIX [POSIX]. The three timestamp 1246 formats are specified in Section 5. In all three cases, 1247 the Timestamp Subseconds field contains the 32 least 1248 significant bits of the timestamp format that is 1249 specified in Section 5. If a node is not capable of 1250 populating this field, it assigns the value 0xFFFFFFFF. 1251 Note that this is a legitimate value in the NTP format, 1252 valid for approximately 233 picoseconds in every second. 1253 If the NTP format is used the analyzer should correlate 1254 several packets in order to detect the error indication. 1256 Bit 4-15 Undefined. An IOAM encapsulating node Must set the value 1257 of these bits to zero upon transmission and ignore upon 1258 receipt. 1260 E2E Option data: Variable-length field. The type of which is 1261 determined by the IOAM-E2E-Type. 1263 5. Timestamp Formats 1265 The IOAM-Data-Fields include a timestamp field which is represented 1266 in one of three possible timestamp formats. It is assumed that the 1267 management plane is responsible for determining which timestamp 1268 format is used. 1270 5.1. PTP Truncated Timestamp Format 1272 The Precision Time Protocol (PTP) [IEEE1588v2] uses an 80-bit 1273 timestamp format. The truncated timestamp format is a 64-bit field, 1274 which is the 64 least significant bits of the 80-bit PTP timestamp. 1275 The PTP truncated format is specified in Section 4.3 of 1276 [I-D.ietf-ntp-packet-timestamps], and the details are presented below 1277 for the sake of completeness. 1279 0 1 2 3 1280 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1281 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1282 | Seconds | 1283 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1284 | Nanoseconds | 1285 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1287 Figure 1: PTP [IEEE1588v2] Truncated Timestamp Format 1289 Timestamp field format: 1291 Seconds: specifies the integer portion of the number of seconds 1292 since the epoch. 1294 + Size: 32 bits. 1296 + Units: seconds. 1298 Nanoseconds: specifies the fractional portion of the number of 1299 seconds since the epoch. 1301 + Size: 32 bits. 1303 + Units: nanoseconds. The value of this field is in the range 0 1304 to (10^9)-1. 1306 Epoch: 1308 The PTP [IEEE1588v2] epoch is 1 January 1970 00:00:00 TAI, which 1309 is 31 December 1969 23:59:51.999918 UTC. 1311 Resolution: 1313 The resolution is 1 nanosecond. 1315 Wraparound: 1317 This time format wraps around every 2^32 seconds, which is roughly 1318 136 years. The next wraparound will occur in the year 2106. 1320 Synchronization Aspects: 1322 It is assumed that nodes that run this protocol are synchronized 1323 among themselves. Nodes may be synchronized to a global reference 1324 time. Note that if PTP [IEEE1588v2] is used for synchronization, 1325 the timestamp may be derived from the PTP-synchronized clock, 1326 allowing the timestamp to be measured with respect to the clock of 1327 an PTP Grandmaster clock. 1329 The PTP truncated timestamp format is not affected by leap 1330 seconds. 1332 5.2. NTP 64-bit Timestamp Format 1334 The Network Time Protocol (NTP) [RFC5905] timestamp format is 64 bits 1335 long. This format is specified in Section 4.2.1 of 1336 [I-D.ietf-ntp-packet-timestamps], and the details are presented below 1337 for the sake of completeness. 1339 0 1 2 3 1340 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1341 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1342 | Seconds | 1343 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1344 | Fraction | 1345 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1347 Figure 2: NTP [RFC5905] 64-bit Timestamp Format 1349 Timestamp field format: 1351 Seconds: specifies the integer portion of the number of seconds 1352 since the epoch. 1354 + Size: 32 bits. 1356 + Units: seconds. 1358 Fraction: specifies the fractional portion of the number of 1359 seconds since the epoch. 1361 + Size: 32 bits. 1363 + Units: the unit is 2^(-32) seconds, which is roughly equal to 1364 233 picoseconds. 1366 Epoch: 1368 The epoch is 1 January 1900 at 00:00 UTC. 1370 Resolution: 1372 The resolution is 2^(-32) seconds. 1374 Wraparound: 1376 This time format wraps around every 2^32 seconds, which is roughly 1377 136 years. The next wraparound will occur in the year 2036. 1379 Synchronization Aspects: 1381 Nodes that use this timestamp format will typically be 1382 synchronized to UTC using NTP [RFC5905]. Thus, the timestamp may 1383 be derived from the NTP-synchronized clock, allowing the timestamp 1384 to be measured with respect to the clock of an NTP server. 1386 The NTP timestamp format is affected by leap seconds; it 1387 represents the number of seconds since the epoch minus the number 1388 of leap seconds that have occurred since the epoch. The value of 1389 a timestamp during or slightly after a leap second may be 1390 temporarily inaccurate. 1392 5.3. POSIX-based Timestamp Format 1394 This timestamp format is based on the POSIX time format [POSIX]. The 1395 detailed specification of the timestamp format used in this document 1396 is presented below. 1398 0 1 2 3 1399 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1400 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1401 | Seconds | 1402 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1403 | Microseconds | 1404 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1406 Figure 3: POSIX-based Timestamp Format 1408 Timestamp field format: 1410 Seconds: specifies the integer portion of the number of seconds 1411 since the epoch. 1413 + Size: 32 bits. 1415 + Units: seconds. 1417 Microseconds: specifies the fractional portion of the number of 1418 seconds since the epoch. 1420 + Size: 32 bits. 1422 + Units: the unit is microseconds. The value of this field is in 1423 the range 0 to (10^6)-1. 1425 Epoch: 1427 The epoch is 1 January 1970 00:00:00 TAI, which is 31 December 1428 1969 23:59:51.999918 UTC. 1430 Resolution: 1432 The resolution is 1 microsecond. 1434 Wraparound: 1436 This time format wraps around every 2^32 seconds, which is roughly 1437 136 years. The next wraparound will occur in the year 2106. 1439 Synchronization Aspects: 1441 It is assumed that nodes that use this timestamp format run Linux 1442 operating system, and hence use the POSIX time. In some cases 1443 nodes may be synchronized to UTC using a synchronization mechanism 1444 that is outside the scope of this document, such as NTP [RFC5905]. 1445 Thus, the timestamp may be derived from the NTP-synchronized 1446 clock, allowing the timestamp to be measured with respect to the 1447 clock of an NTP server. 1449 The POSIX-based timestamp format is affected by leap seconds; it 1450 represents the number of seconds since the epoch minus the number 1451 of leap seconds that have occurred since the epoch. The value of 1452 a timestamp during or slightly after a leap second may be 1453 temporarily inaccurate. 1455 6. IOAM Data Export 1457 IOAM nodes collect information for packets traversing a domain that 1458 supports IOAM. IOAM decapsulating nodes as well as IOAM transit 1459 nodes can choose to retrieve IOAM information from the packet, 1460 process the information further and export the information using 1461 e.g., IPFIX. The mechanisms and associated data formats for 1462 exporting IOAM data is outside the scope of this document. 1464 Raw data export of IOAM data using IPFIX is discussed in 1465 [I-D.spiegel-ippm-ioam-rawexport]. 1467 7. IANA Considerations 1469 This document requests the following IANA Actions. 1471 7.1. Creation of a new In-Situ OAM Protocol Parameters Registry (IOAM) 1472 Protocol Parameters IANA registry 1474 IANA is requested to create a new protocol registry for "In-Situ OAM 1475 (IOAM) Protocol Parameters". This is the common registry that will 1476 include registrations for all IOAM-Namespaces. Each Registry, whose 1477 names are listed below: 1479 IOAM Option-Type 1481 IOAM Trace-Type 1483 IOAM Trace-Flags 1485 IOAM POT-Type 1487 IOAM POT-Flags 1489 IOAM E2E-Type 1491 IOAM Namespace-ID 1493 will contain the current set of possibilities defined in this 1494 document. New registries in this name space are created via RFC 1495 Required process as per [RFC8126]. 1497 The subsequent sub-sections detail the registries herein contained. 1499 7.2. IOAM Option-Type Registry 1501 This registry defines 128 code points for the IOAM Option-Type field 1502 for identifying IOAM Option-Types as explained in Section 4. The 1503 following code points are defined in this draft: 1505 0 IOAM Pre-allocated Trace Option-Type 1507 1 IOAM Incremental Trace Option-Type 1509 2 IOAM POT Option-Type 1511 3 IOAM E2E Option-Type 1513 4 - 127 are available for assignment via RFC Required process as per 1514 [RFC8126]. 1516 7.3. IOAM Trace-Type Registry 1518 This registry defines code point for each bit in the 24-bit IOAM- 1519 Trace-Type field for Pre-allocated trace option and Incremental trace 1520 option defined in Section 4.4. The meaning of Bits 0 - 11 for trace 1521 type are defined in this document in Paragraph 5 of Section 4.4.1: 1523 Bit 0 hop_Lim and node_id in short format 1525 Bit 1 ingress_if_id and egress_if_id in short format 1527 Bit 2 timestamp seconds 1529 Bit 3 timestamp subseconds 1531 Bit 4 transit delay 1533 Bit 5 namespace specific data in short format 1535 Bit 6 queue depth 1537 Bit 7 checksum complement 1539 Bit 8 hop_Lim and node_id in wide format 1540 Bit 9 ingress_if_id and egress_if_id in wide format 1542 Bit 10 namespace specific data in wide format 1544 Bit 11 buffer occupancy 1546 Bit 22 variable length Opaque State Snapshot 1548 Bit 23 reserved 1550 The meaning for Bits 12 - 21 are available for assignment via RFC 1551 Required process as per [RFC8126]. 1553 7.4. IOAM Trace-Flags Registry 1555 This registry defines code points for each bit in the 4 bit flags for 1556 the Pre-allocated trace option and for the Incremental trace option 1557 defined in Section 4.4. The meaning of Bit 0 (the most significant 1558 bit) for trace flags is defined in this document in Paragraph 3 of 1559 Section 4.4.1: 1561 Bit 0 "Overflow" (O-bit) 1563 7.5. IOAM POT-Type Registry 1565 This registry defines 256 code points to define IOAM POT Type for 1566 IOAM proof of transit option Section 4.5. The code point value 0 is 1567 defined in this document: 1569 0: 16 Octet POT data 1571 1 - 255 are available for assignment via RFC Required process as per 1572 [RFC8126]. 1574 7.6. IOAM POT-Flags Registry 1576 This registry defines code points for each bit in the 8 bit flags for 1577 IOAM POT option defined in Section 4.5. The meaning of Bit 0 for 1578 IOAM POT flags is defined in this document in Section 4.5: 1580 Bit 0 "Profile-to-use" (P-bit) 1582 The meaning for Bits 1 - 7 are available for assignment via RFC 1583 Required process as per [RFC8126]. 1585 7.7. IOAM E2E-Type Registry 1587 This registry defines code points for each bit in the 16 bit IOAM- 1588 E2E-Type field for IOAM E2E option Section 4.6. The meaning of Bit 0 1589 - 3 are defined in this document: 1591 Bit 0 64-bit sequence number 1593 Bit 1 32-bit sequence number 1595 Bit 2 timestamp seconds 1597 Bit 3 timestamp subseconds 1599 The meaning of Bits 4 - 15 are available for assignment via RFC 1600 Required process as per [RFC8126]. 1602 7.8. IOAM Namespace-ID Registry 1604 IANA is requested to set up an "IOAM Namespace-ID Registry", 1605 containing 16-bit values. The meaning of Bit 0 is defined in this 1606 document. IANA is requested to reserve the values 0x0001 to 0x7FFF 1607 for private use (managed by operators), as specified in Section 4.3 1608 of the current document. Registry entries for the values 0x8000 to 1609 0xFFFF are to be assigned via the "Expert Review" policy defined in 1610 [RFC8126]. 1612 0: default namespace (known to all IOAM nodes) 1614 0x0001 - 0x7FFF: reserved for private use 1616 0x8000 - 0xFFFF: unassigned 1618 8. Security Considerations 1620 As discussed in [RFC7276], a successful attack on an OAM protocol in 1621 general, and specifically on IOAM, can prevent the detection of 1622 failures or anomalies, or create a false illusion of nonexistent 1623 ones. 1625 The Proof of Transit Option-Type (Section Section 4.5) is used for 1626 verifying the path of data packets. The security considerations of 1627 POT are further discussed in [I-D.ietf-sfc-proof-of-transit]. 1629 The data elements of IOAM can be used for network reconnaissance, 1630 allowing attackers to collect information about network paths, 1631 performance, queue states, buffer occupancy and other information. 1632 Note that in case IOAM is used in "immediate export" mode (reference 1633 to be added in a future revision), the IOAM related trace information 1634 would not be available in the customer data packets, but would 1635 trigger export of packet related IOAM information at every node. 1636 IOAM data export and securing IOAM data export is outside the scope 1637 of this document. 1639 IOAM can be used as a means for implementing Denial of Service (DoS) 1640 attacks, or for amplifying them. For example, a malicious attacker 1641 can add an IOAM header to packets in order to consume the resources 1642 of network devices that take part in IOAM or collectors that analyze 1643 the IOAM data. Another example is a packet length attack, in which 1644 an attacker pushes headers associated with IOAM Option-Types into 1645 data packets, causing these packets to be increased beyond the MTU 1646 size, resulting in fragmentation or in packet drops. 1648 Since IOAM options may include timestamps, if network devices use 1649 synchronization protocols then any attack on the time protocol 1650 [RFC7384] can compromise the integrity of the timestamp-related data 1651 fields. 1653 At the management plane, attacks may be implemented by misconfiguring 1654 or by maliciously configuring IOAM-enabled nodes in a way that 1655 enables other attacks. Thus, IOAM configuration should be secured in 1656 a way that authenticates authorized users and verifies the integrity 1657 of configuration procedures. 1659 Notably, IOAM is expected to be deployed in specific network domains, 1660 thus confining the potential attack vectors to within the network 1661 domain. Indeed, in order to limit the scope of threats to within the 1662 current network domain the network operator is expected to enforce 1663 policies that prevent IOAM traffic from leaking outside of the IOAM 1664 domain, and prevent IOAM data from outside the domain to be processed 1665 and used within the domain. Note that the Immediate Export mode 1666 (reference to be added in a future revision) can mitigate the 1667 potential threat of IOAM data leaking through data packets. 1669 9. Acknowledgements 1671 The authors would like to thank Eric Vyncke, Nalini Elkins, Srihari 1672 Raghavan, Ranganathan T S, Karthik Babu Harichandra Babu, Akshaya 1673 Nadahalli, LJ Wobker, Erik Nordmark, Vengada Prasad Govindan, Andrew 1674 Yourtchenko, Aviv Kfir, Tianran Zhou and Zhenbin (Robin) for the 1675 comments and advice. 1677 This document leverages and builds on top of several concepts 1678 described in [I-D.kitamura-ipv6-record-route]. The authors would 1679 like to acknowledge the work done by the author Hiroshi Kitamura and 1680 people involved in writing it. 1682 The authors would like to gracefully acknowledge useful review and 1683 insightful comments received from Joe Clarke, Al Morton, Tom Herbet, 1684 Haoyu song, and Mickey Spiegel. 1686 The authors would like to acknowledge the contribution of "Immediate 1687 export" of IOAM trace by Barak Gafni. 1689 10. References 1691 10.1. Normative References 1693 [IEEE1588v2] 1694 Institute of Electrical and Electronics Engineers, "IEEE 1695 Std 1588-2008 - IEEE Standard for a Precision Clock 1696 Synchronization Protocol for Networked Measurement and 1697 Control Systems", IEEE Std 1588-2008, 2008, 1698 . 1701 [POSIX] Institute of Electrical and Electronics Engineers, "IEEE 1702 Std 1003.1-2008 (Revision of IEEE Std 1003.1-2004) - IEEE 1703 Standard for Information Technology - Portable Operating 1704 System Interface (POSIX(R))", IEEE Std 1003.1-2008, 2008, 1705 . 1708 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1709 Requirement Levels", BCP 14, RFC 2119, 1710 DOI 10.17487/RFC2119, March 1997, 1711 . 1713 [RFC5905] Mills, D., Martin, J., Ed., Burbank, J., and W. Kasch, 1714 "Network Time Protocol Version 4: Protocol and Algorithms 1715 Specification", RFC 5905, DOI 10.17487/RFC5905, June 2010, 1716 . 1718 [RFC8126] Cotton, M., Leiba, B., and T. Narten, "Guidelines for 1719 Writing an IANA Considerations Section in RFCs", BCP 26, 1720 RFC 8126, DOI 10.17487/RFC8126, June 2017, 1721 . 1723 10.2. Informative References 1725 [I-D.ietf-ntp-packet-timestamps] 1726 Mizrahi, T., Fabini, J., and A. Morton, "Guidelines for 1727 Defining Packet Timestamps", draft-ietf-ntp-packet- 1728 timestamps-07 (work in progress), August 2019. 1730 [I-D.ietf-nvo3-geneve] 1731 Gross, J., Ganga, I., and T. Sridhar, "Geneve: Generic 1732 Network Virtualization Encapsulation", draft-ietf- 1733 nvo3-geneve-14 (work in progress), September 2019. 1735 [I-D.ietf-nvo3-vxlan-gpe] 1736 Maino, F., Kreeger, L., and U. Elzur, "Generic Protocol 1737 Extension for VXLAN", draft-ietf-nvo3-vxlan-gpe-07 (work 1738 in progress), April 2019. 1740 [I-D.ietf-sfc-proof-of-transit] 1741 Brockners, F., Bhandari, S., Mizrahi, T., Dara, S., and S. 1742 Youell, "Proof of Transit", draft-ietf-sfc-proof-of- 1743 transit-03 (work in progress), September 2019. 1745 [I-D.kitamura-ipv6-record-route] 1746 Kitamura, H., "Record Route for IPv6 (PR6) Hop-by-Hop 1747 Option Extension", draft-kitamura-ipv6-record-route-00 1748 (work in progress), November 2000. 1750 [I-D.lapukhov-dataplane-probe] 1751 Lapukhov, P. and r. remy@barefootnetworks.com, "Data-plane 1752 probe for in-band telemetry collection", draft-lapukhov- 1753 dataplane-probe-01 (work in progress), June 2016. 1755 [I-D.spiegel-ippm-ioam-rawexport] 1756 Spiegel, M., Brockners, F., Bhandari, S., and R. 1757 Sivakolundu, "In-situ OAM raw data export with IPFIX", 1758 draft-spiegel-ippm-ioam-rawexport-02 (work in progress), 1759 July 2019. 1761 [RFC7276] Mizrahi, T., Sprecher, N., Bellagamba, E., and Y. 1762 Weingarten, "An Overview of Operations, Administration, 1763 and Maintenance (OAM) Tools", RFC 7276, 1764 DOI 10.17487/RFC7276, June 2014, 1765 . 1767 [RFC7384] Mizrahi, T., "Security Requirements of Time Protocols in 1768 Packet Switched Networks", RFC 7384, DOI 10.17487/RFC7384, 1769 October 2014, . 1771 [RFC7665] Halpern, J., Ed. and C. Pignataro, Ed., "Service Function 1772 Chaining (SFC) Architecture", RFC 7665, 1773 DOI 10.17487/RFC7665, October 2015, 1774 . 1776 [RFC7799] Morton, A., "Active and Passive Metrics and Methods (with 1777 Hybrid Types In-Between)", RFC 7799, DOI 10.17487/RFC7799, 1778 May 2016, . 1780 [RFC7820] Mizrahi, T., "UDP Checksum Complement in the One-Way 1781 Active Measurement Protocol (OWAMP) and Two-Way Active 1782 Measurement Protocol (TWAMP)", RFC 7820, 1783 DOI 10.17487/RFC7820, March 2016, 1784 . 1786 [RFC7821] Mizrahi, T., "UDP Checksum Complement in the Network Time 1787 Protocol (NTP)", RFC 7821, DOI 10.17487/RFC7821, March 1788 2016, . 1790 [RFC8300] Quinn, P., Ed., Elzur, U., Ed., and C. Pignataro, Ed., 1791 "Network Service Header (NSH)", RFC 8300, 1792 DOI 10.17487/RFC8300, January 2018, 1793 . 1795 Authors' Addresses 1797 Frank Brockners 1798 Cisco Systems, Inc. 1799 Hansaallee 249, 3rd Floor 1800 DUESSELDORF, NORDRHEIN-WESTFALEN 40549 1801 Germany 1803 Email: fbrockne@cisco.com 1805 Shwetha Bhandari 1806 Cisco Systems, Inc. 1807 Cessna Business Park, Sarjapura Marathalli Outer Ring Road 1808 Bangalore, KARNATAKA 560 087 1809 India 1811 Email: shwethab@cisco.com 1813 Carlos Pignataro 1814 Cisco Systems, Inc. 1815 7200-11 Kit Creek Road 1816 Research Triangle Park, NC 27709 1817 United States 1819 Email: cpignata@cisco.com 1820 Hannes Gredler 1821 RtBrick Inc. 1823 Email: hannes@rtbrick.com 1825 John Leddy 1826 United States 1828 Email: john@leddy.net 1830 Stephen Youell 1831 JP Morgan Chase 1832 25 Bank Street 1833 London E14 5JP 1834 United Kingdom 1836 Email: stephen.youell@jpmorgan.com 1838 Tal Mizrahi 1839 Huawei Network.IO Innovation Lab 1840 Israel 1842 Email: tal.mizrahi.phd@gmail.com 1844 David Mozes 1846 Email: mosesster@gmail.com 1848 Petr Lapukhov 1849 Facebook 1850 1 Hacker Way 1851 Menlo Park, CA 94025 1852 US 1854 Email: petr@fb.com 1855 Remy Chang 1856 Barefoot Networks 1857 4750 Patrick Henry Drive 1858 Santa Clara, CA 95054 1859 US 1861 Email: remy@barefootnetworks.com 1863 Daniel Bernier 1864 Bell Canada 1865 Canada 1867 Email: daniel.bernier@bell.ca 1869 Jennifer Lemon 1870 Broadcom 1871 270 Innovation Drive 1872 San Jose, CA 95134 1873 US 1875 Email: jennifer.lemon@broadcom.com