idnits 2.17.1 draft-ietf-ippm-ioam-data-05.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (March 10, 2019) is 1868 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) -- Looks like a reference, but probably isn't: '0' on line 729 -- Looks like a reference, but probably isn't: '1' on line 518 -- Possible downref: Non-RFC (?) normative reference: ref. 'IEEE1588v2' -- Possible downref: Non-RFC (?) normative reference: ref. 'POSIX' == Outdated reference: A later version (-09) exists of draft-ietf-ntp-packet-timestamps-06 == Outdated reference: A later version (-16) exists of draft-ietf-nvo3-geneve-11 == Outdated reference: A later version (-13) exists of draft-ietf-nvo3-vxlan-gpe-06 == Outdated reference: A later version (-07) exists of draft-spiegel-ippm-ioam-rawexport-01 Summary: 0 errors (**), 0 flaws (~~), 5 warnings (==), 5 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 ippm F. Brockners 3 Internet-Draft S. Bhandari 4 Intended status: Standards Track C. Pignataro 5 Expires: September 11, 2019 Cisco 6 H. Gredler 7 RtBrick Inc. 8 J. Leddy 9 Comcast 10 S. Youell 11 JPMC 12 T. Mizrahi 13 Huawei Network.IO Innovation Lab 14 D. Mozes 16 P. Lapukhov 17 Facebook 18 R. Chang 19 Barefoot Networks 20 D. Bernier 21 Bell Canada 22 J. Lemon 23 Broadcom 24 March 10, 2019 26 Data Fields for In-situ OAM 27 draft-ietf-ippm-ioam-data-05 29 Abstract 31 In-situ Operations, Administration, and Maintenance (IOAM) records 32 operational and telemetry information in the packet while the packet 33 traverses a path between two points in the network. This document 34 discusses the data fields and associated data types for in-situ OAM. 35 In-situ OAM data fields can be embedded into a variety of transports 36 such as NSH, Segment Routing, Geneve, native IPv6 (via extension 37 header), or IPv4. In-situ OAM can be used to complement OAM 38 mechanisms based on e.g. ICMP or other types of probe packets. 40 Status of This Memo 42 This Internet-Draft is submitted in full conformance with the 43 provisions of BCP 78 and BCP 79. 45 Internet-Drafts are working documents of the Internet Engineering 46 Task Force (IETF). Note that other groups may also distribute 47 working documents as Internet-Drafts. The list of current Internet- 48 Drafts is at https://datatracker.ietf.org/drafts/current/. 50 Internet-Drafts are draft documents valid for a maximum of six months 51 and may be updated, replaced, or obsoleted by other documents at any 52 time. It is inappropriate to use Internet-Drafts as reference 53 material or to cite them other than as "work in progress." 55 This Internet-Draft will expire on September 11, 2019. 57 Copyright Notice 59 Copyright (c) 2019 IETF Trust and the persons identified as the 60 document authors. All rights reserved. 62 This document is subject to BCP 78 and the IETF Trust's Legal 63 Provisions Relating to IETF Documents 64 (https://trustee.ietf.org/license-info) in effect on the date of 65 publication of this document. Please review these documents 66 carefully, as they describe your rights and restrictions with respect 67 to this document. Code Components extracted from this document must 68 include Simplified BSD License text as described in Section 4.e of 69 the Trust Legal Provisions and are provided without warranty as 70 described in the Simplified BSD License. 72 Table of Contents 74 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 75 2. Conventions . . . . . . . . . . . . . . . . . . . . . . . . . 3 76 3. Scope, Applicability, and Assumptions . . . . . . . . . . . . 4 77 4. IOAM Data Types and Formats . . . . . . . . . . . . . . . . . 5 78 4.1. IOAM Namespaces . . . . . . . . . . . . . . . . . . . . . 7 79 4.2. IOAM Tracing Options . . . . . . . . . . . . . . . . . . 9 80 4.2.1. Pre-allocated and Incremental Trace Options . . . . . 11 81 4.2.2. IOAM node data fields and associated formats . . . . 17 82 4.2.3. Examples of IOAM node data . . . . . . . . . . . . . 22 83 4.3. IOAM Proof of Transit Option . . . . . . . . . . . . . . 24 84 4.3.1. IOAM Proof of Transit Type 0 . . . . . . . . . . . . 26 85 4.4. IOAM Edge-to-Edge Option . . . . . . . . . . . . . . . . 27 86 5. Timestamp Formats . . . . . . . . . . . . . . . . . . . . . . 29 87 5.1. PTP Truncated Timestamp Format . . . . . . . . . . . . . 29 88 5.2. NTP 64-bit Timestamp Format . . . . . . . . . . . . . . . 30 89 5.3. POSIX-based Timestamp Format . . . . . . . . . . . . . . 31 90 6. IOAM Data Export . . . . . . . . . . . . . . . . . . . . . . 33 91 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 33 92 7.1. Creation of a new In-Situ OAM Protocol Parameters 93 Registry (IOAM) Protocol Parameters IANA registry . . . . 33 94 7.2. IOAM Type Registry . . . . . . . . . . . . . . . . . . . 34 95 7.3. IOAM Trace Type Registry . . . . . . . . . . . . . . . . 34 96 7.4. IOAM Trace Flags Registry . . . . . . . . . . . . . . . . 35 97 7.5. IOAM POT Type Registry . . . . . . . . . . . . . . . . . 35 98 7.6. IOAM POT Flags Registry . . . . . . . . . . . . . . . . . 35 99 7.7. IOAM E2E Type Registry . . . . . . . . . . . . . . . . . 36 100 7.8. IOAM Namespace-ID Registry . . . . . . . . . . . . . . . 36 101 8. Security Considerations . . . . . . . . . . . . . . . . . . . 36 102 9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 37 103 10. References . . . . . . . . . . . . . . . . . . . . . . . . . 38 104 10.1. Normative References . . . . . . . . . . . . . . . . . . 38 105 10.2. Informative References . . . . . . . . . . . . . . . . . 38 106 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 40 108 1. Introduction 110 This document defines data fields for "in-situ" Operations, 111 Administration, and Maintenance (IOAM). In-situ OAM records OAM 112 information within the packet while the packet traverses a particular 113 network domain. The term "in-situ" refers to the fact that the OAM 114 data is added to the data packets rather than is being sent within 115 packets specifically dedicated to OAM. IOAM is to complement 116 mechanisms such as Ping or Traceroute, or more recent active probing 117 mechanisms as described in [I-D.lapukhov-dataplane-probe]. In terms 118 of "active" or "passive" OAM, "in-situ" OAM can be considered a 119 hybrid OAM type. While no extra packets are sent, IOAM adds 120 information to the packets therefore cannot be considered passive. 121 In terms of the classification given in [RFC7799] IOAM could be 122 portrayed as Hybrid Type 1. "In-situ" mechanisms do not require 123 extra packets to be sent and hence don't change the packet traffic 124 mix within the network. IOAM mechanisms can be leveraged where 125 mechanisms using e.g. ICMP do not apply or do not offer the desired 126 results, such as proving that a certain traffic flow takes a pre- 127 defined path, SLA verification for the live data traffic, detailed 128 statistics on traffic distribution paths in networks that distribute 129 traffic across multiple paths, or scenarios in which probe traffic is 130 potentially handled differently from regular data traffic by the 131 network devices. 133 2. Conventions 135 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 136 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 137 document are to be interpreted as described in [RFC2119]. 139 Abbreviations used in this document: 141 E2E Edge to Edge 143 Geneve: Generic Network Virtualization Encapsulation 144 [I-D.ietf-nvo3-geneve] 146 IOAM: In-situ Operations, Administration, and Maintenance 148 MTU: Maximum Transmit Unit 150 NSH: Network Service Header [RFC8300] 152 OAM: Operations, Administration, and Maintenance 154 POT: Proof of Transit 156 SFC: Service Function Chain 158 SID: Segment Identifier 160 SR: Segment Routing 162 VXLAN-GPE: Virtual eXtensible Local Area Network, Generic Protocol 163 Extension [I-D.ietf-nvo3-vxlan-gpe] 165 3. Scope, Applicability, and Assumptions 167 IOAM deployment assumes a set of constraints, requirements, and 168 guiding principles which are described in this section. 170 Scope: This document defines the data fields and associated data 171 types for in-situ OAM. The in-situ OAM data field can be transported 172 by a variety of transport protocols, including NSH, Segment Routing, 173 Geneve, IPv6, or IPv4. Specification details for these different 174 transport protocols are outside the scope of this document. 176 Deployment domain (or scope) of in-situ OAM deployment: IOAM is a 177 network domain focused feature, with "network domain" being a set of 178 network devices or entities within a single administration. For 179 example, a network domain can include an enterprise campus using 180 physical connections between devices or an overlay network using 181 virtual connections / tunnels for connectivity between said devices. 182 A network domain is defined by its perimeter or edge. Designers of 183 carrier protocols for IOAM must specify mechanisms to ensure that 184 IOAM data stays within an IOAM domain. In addition, the operator of 185 such a domain is expected to put provisions in place to ensure that 186 IOAM data does not leak beyond the edge of an IOAM domain, e.g. using 187 for example packet filtering methods. The operator should consider 188 potential operational impact of IOAM to mechanisms such as ECMP 189 processing (e.g. load-balancing schemes based on packet length could 190 be impacted by the increased packet size due to IOAM), path MTU (i.e. 191 ensure that the MTU of all links within a domain is sufficiently 192 large to support the increased packet size due to IOAM) and ICMP 193 message handling (i.e. in case of a native IPv6 transport, IOAM 194 support for ICMPv6 Echo Request/Reply could desired which would 195 translate into ICMPv6 extensions to enable IOAM data fields to be 196 copied from an Echo Request message to an Echo Reply message). 198 IOAM control points: IOAM data fields are added to or removed from 199 the live user traffic by the devices which form the edge of a domain. 200 Devices within an IOAM domain can update and/or add IOAM data-fields. 201 Domain edge devices can be hosts or network devices. 203 Traffic-sets that IOAM is applied to: IOAM can be deployed on all or 204 only on subsets of the live user traffic. It SHOULD be possible to 205 enable IOAM on a selected set of traffic (e.g., per interface, based 206 on an access control list or flow specification defining a specific 207 set of traffic, etc.) The selected set of traffic can also be all 208 traffic. 210 Encapsulation independence: Data formats for IOAM SHOULD be defined 211 in a transport-independent manner. IOAM applies to a variety of 212 encapsulating protocols. A definition of how IOAM data fields are 213 carried by different transport protocols is outside the scope of this 214 document. 216 Layering: If several encapsulation protocols (e.g., in case of 217 tunneling) are stacked on top of each other, IOAM data-records could 218 be present at every layer. The behavior follows the ships-in-the- 219 night model, i.e. IOAM data in one layer is independent from IOAM 220 data in another layer. Layering allows operators to instrument the 221 protocol layer they want to measure. The different layers could, but 222 do not have to share the same IOAM encapsulation and decapsulation. 224 Combination with active OAM mechanisms: IOAM should be usable for 225 active network probing, enabling for example a customized version of 226 traceroute. IOAM may also be carried out on cloned or sampled copies 227 of data packets, when the operator prefers not to directly modify 228 data packets for IOAM purposes. Decapsulating IOAM nodes must have 229 the ability to discard active IOAM packets, potentially in addition 230 to retrieving the IOAM information. 232 IOAM implementation: The IOAM data-field definitions take the 233 specifics of devices with hardware data-plane and software data-plane 234 into account. 236 4. IOAM Data Types and Formats 238 This section defines IOAM data types and data fields and associated 239 data types required for IOAM. 241 To accommodate the different uses of IOAM, IOAM data fields fall into 242 different categories, as specified below. In IOAM these categories 243 are referred to as IOAM-Types. A common registry is maintained for 244 IOAM-Types, see Section 7.2 for details. Corresponding to these 245 IOAM-Types, different IOAM data fields are defined. IOAM data fields 246 can be encapsulated into a variety of protocols, such as NSH, Geneve, 247 IPv6, etc. The definition of how IOAM data fields are encapsulated 248 into other protocols is outside the scope of this document. 250 This document defines four IOAM-Types, as specified in this section: 252 o Pre-allocated Trace Option 254 o Incremental Trace Option 256 o Proof of Transit (POT) Option 258 o Edge-to-Edge (E2E) Option 260 IOAM is expected to be deployed in a specific domain rather than on 261 the overall Internet. The part of the network which employs IOAM is 262 referred to as the "IOAM-domain". IOAM data is added to a packet 263 upon entering the IOAM-domain and is removed from the packet when 264 exiting the domain. Within the IOAM-domain, the IOAM data may be 265 updated by network nodes that the packet traverses. The device which 266 adds an IOAM data container to the packet to capture IOAM data is 267 called the "IOAM encapsulating node", whereas the device which 268 removes the IOAM data container is referred to as the "IOAM 269 decapsulating node". Nodes within the domain which are aware of IOAM 270 data and read and/or write or process the IOAM data are called "IOAM 271 transit nodes". IOAM nodes which add or remove the IOAM data fields 272 can also update the IOAM data fields at the same time. Or in other 273 words, IOAM encapsulating or decapsulating nodes can also serve as 274 IOAM transit nodes at the same time. Note that not every node in an 275 IOAM domain needs to be an IOAM transit node. For example, a 276 deployment might require that packets traverse a set of firewalls. 277 In that case, only the set of firewall nodes would be IOAM transit 278 nodes rather than all nodes. 280 An IOAM encapsulating node incorporates one or more IOAM-Types (from 281 the list of four IOAM-Types above) into packets that IOAM is enabled 282 for. If IOAM is enabled for a selected subset of the traffic, the 283 encapsulating node is responsible for applying the IOAM functionality 284 to the selected subset. 286 An IOAM transit node updates one or more of the IOAM data fields. If 287 both the pre-allocated and the incremental trace options are present 288 in the packet, each IOAM transit node will update at most one of 289 these options. A transit node cannot add new IOAM options to a 290 packet, and cannot change an IOAM Edge-to-Edge Option. 292 An IOAM decapsulating node removes all the IOAM-Types from packets. 294 The role of a node (i.e. encapsulating, transit, decapsulating) is 295 defined within an IOAM namespace (see below). A node can have 296 different roles in different IOAM namespaces. 298 4.1. IOAM Namespaces 300 IOAM data fields are defined within an IOAM namespace. An IOAM 301 namespace is identified by a 16-bit namespace identifier (Namespace- 302 ID). Namespace identifiers MUST be present and populated in all IOAM 303 option headers. The Namespace-ID value is divided into two sub- 304 ranges: 306 o An operator-assigned range from 0x0001 to 0x7FFF 308 o An IANA-assigned range from 0x8000 to 0xFFFF 310 The IANA-assigned range is intended to allow future extensions to 311 have new and interoperable IOAM functionality, while the operator- 312 assigned range is intended to be domain specific, and managed by the 313 network operator. The Namespace-ID value of 0x0000 is default and 314 known to all the nodes implementing IOAM. 316 Namespace identifiers allow devices which are IOAM capable to 317 determine: 319 o whether IOAM option header(s) need to be processed by a device: If 320 the Namespace-ID contained in a packet does not match any 321 Namespace-ID the node is configured to operate on, then the node 322 MUST NOT change the contents of the IOAM data fields. 324 o which IOAM option headers need to be processed/updated in case 325 there are multiple IOAM option headers present in the packet. 326 Multiple option headers can be present in a packet in case of 327 overlapping IOAM domains or in case of a layered IOAM deployment. 329 o whether IOAM option header(s) should be removed from the packet, 330 e.g. at a domain edge or domain boundary. 332 IOAM namespaces support several different uses: 334 o Namespaces can be used by an operator to distinguish different 335 operational domains. Devices at domain edges can filter on 336 Namespace-IDs to provide for proper IOAM domain isolation. 338 o Namespaces provide additional context for IOAM data fields and 339 thus ensure that IOAM data is unique and can be interpreted 340 properly by management stations or network controllers. While, 341 for example, the IOAM node identifier (Node-ID) does not need to 342 be unique in a deployment (e.g. an operator may wish to use 343 different Node-IDs for different IOAM layers, even within the same 344 device; or Node-IDs might not be unique for other organizational 345 reasons, such as after a merger of two formerly separated 346 organizations), the combination of Node-ID and Namespace-ID will 347 always be unique. Similarly, namespaces can be used to define how 348 certain IOAM data fields are interpreted: IOAM offers three 349 different timestamp format options. The Namespace-ID can be used 350 to determine the timestamp format. IOAM data fields (e.g. buffer 351 occupancy) which do not have a unit associated are to be 352 interpreted within the context of a namespace. 354 o Namespaces can be used to identify different sets of devices 355 (e.g., different types of devices) in a deployment: If an operator 356 desires to insert different IOAM data based on the device, the 357 devices could be grouped into multiple namespaces. This could be 358 due to the fact that the IOAM feature set differs between 359 different sets of devices, or it could be for reasons of optimized 360 space usage in the packet header. This could also stem from 361 hardware or operational limitations on the size of the trace data 362 that can be added and processed, preventing collection of a full 363 trace for a flow. 365 * Assigning different Namespace-IDs to different sets of nodes or 366 network partitions and using the Namespace-ID as a selector at 367 the IOAM encapsulating node, a full trace for a flow could be 368 collected and constructed via partial traces in different 369 packets of the same flow. Example: An operator could choose to 370 group the devices of a domain into two namespaces, in a way 371 that at average, only every second hop would be recorded by any 372 device. To retrieve a full view of the deployment, the 373 captured IOAM data fields of the two namespaces need to be 374 correlated. 376 * Assigning different Namespace-IDs to different sets of nodes or 377 network partitions and using a separate IOAM header for each 378 Namespace-ID, a full trace for a flow could be collected and 379 constructed via partial traces from each IOAM header in each of 380 the packets in the flow. Example: An operator could choose to 381 group the devices of a domain into two namespaces, in a way 382 that each namespace is represented by one of two IOAM headers 383 in the packet. Each node would record data only for the IOAM 384 namespace that it belongs to, ignoring the other IOAM header 385 with a namespace to which it doesn't belong. To retrieve a 386 full view of the deployment, the captured IOAM data fields of 387 the two namespaces need to be correlated. 389 4.2. IOAM Tracing Options 391 "IOAM tracing data" is expected to be collected at every node that a 392 packet traverses to ensure visibility into the entire path a packet 393 takes within an IOAM domain, i.e., in a typical deployment all nodes 394 in an in-situ OAM-domain would participate in IOAM and thus be IOAM 395 transit nodes, IOAM encapsulating or IOAM decapsulating nodes. If 396 not all nodes within a domain are IOAM capable, IOAM tracing 397 information will only be collected on those nodes which are IOAM 398 capable. Nodes which are not IOAM capable will forward the packet 399 without any changes to the IOAM data fields. The maximum number of 400 hops and the minimum path MTU of the IOAM domain is assumed to be 401 known. 403 To optimize hardware and software implementations tracing is defined 404 as two separate options. Any deployment MAY choose to configure and 405 support one or both of the following options. An implementation of 406 the transport protocol that carries these in-situ OAM data MAY choose 407 to support only one of the options. In the event that both options 408 are utilized at the same time, the Incremental Trace Option MUST be 409 placed before the Pre-allocated Trace Option. Given that the 410 operator knows which equipment is deployed in a particular IOAM, the 411 operator will decide by means of configuration which type(s) of trace 412 options will be enabled for a particular domain. 414 Pre-allocated Trace Option: This trace option is defined as a 415 container of node data fields with pre-allocated space for each 416 node to populate its information. This option is useful for 417 software implementations where it is efficient to allocate the 418 space once and index into the array to populate the data during 419 transit. The IOAM encapsulating node allocates the option header 420 and sets the fields in the option header. The in situ OAM 421 encapsulating node allocates an array which is used to store 422 operational data retrieved from every node while the packet 423 traverses the domain. IOAM transit nodes update the content of 424 the array, and possibly update the checksums of outer headers. A 425 pointer which is part of the IOAM trace data points to the next 426 empty slot in the array. An IOAM transit node that updates the 427 content of the pre-allocated option also updates the value of the 428 pointer, which specifies where the next IOAM transit node fills in 429 its data. 431 Incremental Trace Option: This trace option is defined as a 432 container of node data fields where each node allocates and pushes 433 its node data immediately following the option header. This type 434 of trace recording is useful for some of the hardware 435 implementations as this eliminates the need for the transit 436 network elements to read the full array in the option and allows 437 for arbitrarily long packets as the MTU allows. The in-situ OAM 438 encapsulating node allocates the option header. The in-situ OAM 439 encapsulating node based on operational state and configuration 440 sets the fields in the header that control what node data fields 441 should be collected, and how large the node data list can grow. 442 The in-situ OAM transit nodes push their node data to the node 443 data list, decrease the remaining length available to subsequent 444 nodes, and adjust the lengths and possibly checksums in outer 445 headers. 447 Every node data entry is to hold information for a particular IOAM 448 transit node that is traversed by a packet. The in-situ OAM 449 decapsulating node removes the IOAM data and processes and/or exports 450 the metadata. IOAM data uses its own name-space for information such 451 as node identifier or interface identifier. This allows for a 452 domain-specific definition and interpretation. For example: In one 453 case an interface-id could point to a physical interface (e.g., to 454 understand which physical interface of an aggregated link is used 455 when receiving or transmitting a packet) whereas in another case it 456 could refer to a logical interface (e.g., in case of tunnels). 458 The following IOAM data is defined for IOAM tracing: 460 o Identification of the IOAM node. An IOAM node identifier can 461 match to a device identifier or a particular control point or 462 subsystem within a device. 464 o Identification of the interface that a packet was received on, 465 i.e. ingress interface. 467 o Identification of the interface that a packet was sent out on, 468 i.e. egress interface. 470 o Time of day when the packet was processed by the node. Different 471 definitions of processing time are feasible and expected, though 472 it is important that all devices of an in-situ OAM domain follow 473 the same definition. 475 o Generic data: Format-free information where syntax and semantic of 476 the information is defined by the operator in a specific 477 deployment. For a specific deployment, all IOAM nodes should 478 interpret the generic data the same way. Examples for generic 479 IOAM data include geo-location information (location of the node 480 at the time the packet was processed), buffer queue fill level or 481 cache fill level at the time the packet was processed, or even a 482 battery charge level. 484 o A mechanism to detect whether IOAM trace data was added at every 485 hop or whether certain hops in the domain weren't in-situ OAM 486 transit nodes. 488 The "node data list" array in the packet is populated iteratively as 489 the packet traverses the network, starting with the last entry of the 490 array, i.e., "node data list [n]" is the first entry to be populated, 491 "node data list [n-1]" is the second one, etc. 493 4.2.1. Pre-allocated and Incremental Trace Options 495 The in-situ OAM pre-allocated trace option and the in-situ OAM 496 incremental trace option have similar formats. Except where noted 497 below, the internal formats and fields of the two trace options are 498 identical. 500 Pre-allocated and incremental trace option headers: 502 0 1 2 3 503 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 504 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 505 | Namespace-ID |NodeLen | Flags | RemainingLen| 506 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 507 | IOAM-Trace-Type | Reserved | 508 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 510 The trace option data MUST be 4-octet aligned: 512 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+<-+ 513 | | | 514 | node data list [0] | | 515 | | | 516 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ D 517 | | a 518 | node data list [1] | t 519 | | a 520 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 521 ~ ... ~ S 522 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ p 523 | | a 524 | node data list [n-1] | c 525 | | e 526 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 527 | | | 528 | node data list [n] | | 529 | | | 530 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+<-+ 532 Namespace-ID: 16-bit identifier of an IOAM namespace. The 533 Namespace-ID value of 0x0000 is defined as the default value and 534 MUST be known to all the nodes implementing IOAM. For any other 535 Namespace-ID value that does not match any Namespace-ID the node 536 is configured to operate on, the node MUST NOT change the contents 537 of the IOAM data fields. 539 NodeLen: 5-bit unsigned integer. This field specifies the length of 540 data added by each node in multiples of 4-octets, excluding the 541 length of the "Opaque State Snapshot" field. 543 If IOAM-Trace-Type bit 7 is not set, then NodeLen specifies the 544 actual length added by each node. If IOAM-Trace-Type bit 7 is 545 set, then the actual length added by a node would be (NodeLen + 546 Opaque Data Length). 548 For example, if 3 IOAM-Trace-Type bits are set and none of them 549 are wide, then NodeLen would be 3. If 3 IOAM-Trace-Type bits are 550 set and 2 of them are wide, then NodeLen would be 5. 552 An IOAM encapsulating node must set NodeLen. 554 A node receiving an IOAM Pre-allocated or Incremental Trace Option 555 may rely on the NodeLen value, or it may ignore the NodeLen value 556 and calculate the node length from the IOAM-Trace-Type bits. 558 Flags 4-bit field. The following flags are defined: 560 Bit 0 "Overflow" (O-bit) (most significant bit). This bit is set 561 by the network element if there is not enough number of octets 562 left to record node data, no field is added and the overflow 563 "O-bit" must be set to "1" in the header. This is useful for 564 transit nodes to ignore further processing of the option. 566 Bit 1 "Loopback" (L-bit). Loopback mode is used to send a copy 567 of a packet back towards the source. Loopback mode assumes 568 that a return path from transit nodes and destination nodes 569 towards the source exists. The encapsulating node decides 570 (e.g., using a filter) which packets loopback mode is enabled 571 for by setting the loopback bit. The encapsulating node also 572 needs to ensure that sufficient space is available in the IOAM 573 header for loopback operation, which includes intermediate 574 nodes adding trace data on the original path and then again on 575 the return path. A loopback bit that is set indicates to the 576 transit nodes processing this option that they are to create a 577 copy of the received packet and send the copy back to the 578 source of the packet. The copy has its metadata added after 579 being copied in order to allow any egress-dependent information 580 to be set based on the egress of the copy rather than the 581 original. The original packet continues towards its 582 destination. The source address of the original packet is used 583 as the destination address in the copied packet. The address 584 of the node performing the copy operation is used as the source 585 address. The L-bit MUST be cleared in the copy of the packet 586 that a node sends back towards the source. On its way back 587 towards the source, the copied packet is processed like any 588 other packet with IOAM information, including adding any 589 requested data at each transit node (assuming there is 590 sufficient space). Once the return packet reaches the IOAM 591 domain boundary, IOAM decapsulation occurs as with any other 592 packet containing IOAM information. Because any intermediate 593 node receiving such a packet would not know how to process the 594 original packet, and because there would be a risk of the 595 original packet leaking past the initiator of the IOAM 596 loopback, the initiator of an IOAM loopback MUST be the 597 initiator of the packet. Once a loopback packet is received 598 back at the initiator, it is a local matter how it is 599 recognized as a loopback packet. 601 Bit 2 "Active" (A-bit). When set, this indicates that this is an 602 active OAM packet, where "active" is used in the sense defined 603 in [RFC7799], rather than a data packet. For example, the 604 packet may be a probe, or it may be a (possibly truncated) copy 605 of a data packet. At the IOAM decapsulating node, in addition 606 to processing and/or exporting the metadata, the packet must be 607 discarded rather than forwarded. If this bit is not set, then 608 the decapsulating node should attempt to forward the packet 609 after IOAM decapsulation. 611 Bit 3 "Immediate Export" (I-bit). Immediate export mode is used 612 to export IOAM data fields immediately at every IOAM supported 613 network node, instead of adding the IOAM data fields to the 614 packet traversing the network. The various types of IOAM nodes 615 MUST process packets with the I-bit set as follows: 617 1. An encapsulating IOAM node configured to set the I-bit 618 encapsulates the packet with the IOAM header and sets the 619 I-bit, leaving the IOAM header without locally collected 620 data, and exports the requested IOAM data immediately. The 621 encapsulating IOAM node is the only type of node allowed to 622 set the I-bit. 624 2. A transit node that processes a packet with the I-bit set 625 is expected to export the requested IOAM data, and not 626 incorporate it into the IOAM header. 628 3. A decapsulating IOAM node that processes a packet with the 629 I-bit set is expected to export the requested IOAM data, 630 and decapsulate the IOAM header. 632 Note that in case of "Immediate Export" being employed, no IOAM 633 trace data is added to the packets traversing the network. As 634 a means to support correlation of exported IOAM data different 635 nodes in the network, a deployment could consider attaching an 636 IOAM E2E option in addition to the trace option, that includes 637 a sequence number. See Bit 1 in the IOAM-E2E-Types. Please 638 refer to Section 6 for a discussion of IOAM data export and 639 associated formats. 641 RemainingLen: 7-bit unsigned integer. This field specifies the data 642 space in multiples of 4-octets remaining for recording the node 643 data, before the node data list is considered to have overflowed. 644 When RemainingLen reaches 0, nodes are no longer allowed to add 645 node data. Given that the sender knows the minimum path MTU, the 646 sender MAY set the initial value of RemainingLen according to the 647 number of node data bytes allowed before exceeding the MTU. 648 Subsequent nodes can carry out a simple comparison between 649 RemainingLen and NodeLen, along with the length of the "Opaque 650 State Snapshot" if applicable, to determine whether or not data 651 can be added by this node. When node data is added, the node MUST 652 decrease RemainingLen by the amount of data added. In the pre- 653 allocated trace option, this is used as an offset in data space to 654 record the node data element. 656 IOAM-Trace-Type: A 24-bit identifier which specifies which data 657 types are used in this node data list. 659 The IOAM-Trace-Type value is a bit field. The following bit 660 fields are defined in this document, with details on each field 661 described in the Section 4.2.2. The order of packing the data 662 fields in each node data element follows the bit order of the 663 IOAM-Trace-Type field, as follows: 665 Bit 0 (Most significant bit) When set indicates presence of 666 Hop_Lim and node_id in the node data. 668 Bit 1 When set indicates presence of ingress_if_id and 669 egress_if_id (short format) in the node data. 671 Bit 2 When set indicates presence of timestamp seconds in the 672 node data. 674 Bit 3 When set indicates presence of timestamp subseconds in 675 the node data. 677 Bit 4 When set indicates presence of transit delay in the node 678 data. 680 Bit 5 When set indicates presence of namespace specific data 681 (short format) in the node data. 683 Bit 6 When set indicates presence of queue depth in the node 684 data. 686 Bit 7 When set indicates presence of variable length Opaque 687 State Snapshot field. 689 Bit 8 When set indicates presence of Hop_Lim and node_id in 690 wide format in the node data. 692 Bit 9 When set indicates presence of ingress_if_id and 693 egress_if_id in wide format in the node data. 695 Bit 10 When set indicates presence of namespace specific data in 696 wide format in the node data. 698 Bit 11 When set indicates presence of buffer occupancy in the 699 node data. 701 Bit 12-22 Undefined. An IOAM encapsulating node MUST set the 702 value of each of these bits to 0. If an IOAM transit 703 node receives a packet with one or more of these bits set 704 to 1, it must either: 706 1. Add corresponding node data filled with the reserved 707 value 0xFFFFFFFF, after the node data fields for the 708 IOAM-Trace-Type bits defined above, such that the 709 total node data added by this node in units of 710 4-octets is equal to NodeLen, or 712 2. Not add any node data fields to the packet, even for 713 the IOAM-Trace-Type bits defined above. 715 Bit 23 When set indicates presence of the Checksum Complement 716 node data. 718 Section 4.2.2 describes the IOAM data types and their formats. 719 Within an in-situ OAM domain possible combinations of these bits 720 making the IOAM-Trace-Type can be restricted by configuration 721 knobs. 723 Reserved: 8-bits. Must be zero. 725 Node data List [n]: Variable-length field. The type of which is 726 determined by the IOAM-Trace-Type bit representing the n-th node 727 data in the node data list. The node data list is encoded 728 starting from the last node data of the path. The first element 729 of the node data list (node data list [0]) contains the last node 730 of the path while the last node data of the node data list (node 731 data list[n]) contains the first node data of the path traced. 732 Populating the node data list in this way ensures that the order 733 of node data list is the same for incremental and pre-allocated 734 trace options. In the pre-allocated trace option, the index 735 contained in RemainingLen identifies the offset for current active 736 node data to be populated. 738 4.2.2. IOAM node data fields and associated formats 740 All the data fields MUST be 4-octet aligned. If a node which is 741 supposed to update an IOAM data field is not capable of populating 742 the value of a field set in the IOAM-Trace-Type, the field value MUST 743 be set to 0xFFFFFFFF for 4-octet fields or 0xFFFFFFFFFFFFFFFF for 744 8-octet fields, indicating that the value is not populated, except 745 when explicitly specified in the field description below. 747 Data field and associated data type for each of the data field is 748 shown below: 750 Hop_Lim and node_id: 4-octet field defined as follows: 752 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 753 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 754 | Hop_Lim | node_id | 755 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 757 Hop_Lim: 1-octet unsigned integer. It is set to the Hop Limit 758 value in the packet at the node that records this data. Hop 759 Limit information is used to identify the location of the node 760 in the communication path. This is copied from the lower 761 layer, e.g., TTL value in IPv4 header or hop limit field from 762 IPv6 header of the packet when the packet is ready for 763 transmission. The semantics of the Hop_Lim field depend on the 764 lower layer protocol that IOAM is encapsulated over, and 765 therefore its specific semantics are outside the scope of this 766 memo. 768 node_id: 3-octet unsigned integer. Node identifier field to 769 uniquely identify a node within in-situ OAM domain. The 770 procedure to allocate, manage and map the node_ids is beyond 771 the scope of this document. 773 ingress_if_id and egress_if_id: 4-octet field defined as follows: 775 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 776 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 777 | ingress_if_id | egress_if_id | 778 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 780 ingress_if_id: 2-octet unsigned integer. Interface identifier to 781 record the ingress interface the packet was received on. 783 egress_if_id: 2-octet unsigned integer. Interface identifier to 784 record the egress interface the packet is forwarded out of. 786 Note that due to the fact that IOAM uses its own namespaces for 787 IOAM data fields, data fields like interface identifiers can be 788 used in a flexible way to represent system resources that are 789 associated with ingressing or egressing packets, i.e. an IOAM 790 interface ID could represent a physical interface, a virtual or 791 logical interface, or even a queue. 793 timestamp seconds: 4-octet unsigned integer. Absolute timestamp in 794 seconds that specifies the time at which the packet was received 795 by the node. This field has three possible formats; based on 796 either PTP [IEEE1588v2], NTP [RFC5905], or POSIX [POSIX]. The 797 three timestamp formats are specified in Section 5. In all three 798 cases, the Timestamp Seconds field contains the 32 most 799 significant bits of the timestamp format that is specified in 800 Section 5. If a node is not capable of populating this field, it 801 assigns the value 0xFFFFFFFF. Note that this is a legitimate 802 value that is valid for 1 second in approximately 136 years; the 803 analyzer should correlate several packets or compare the timestamp 804 value to its own time-of-day in order to detect the error 805 indication. 807 timestamp subseconds: 4-octet unsigned integer. Absolute timestamp 808 in subseconds that specifies the time at which the packet was 809 received by the node. This field has three possible formats; 810 based on either PTP [IEEE1588v2], NTP [RFC5905], or POSIX [POSIX]. 811 The three timestamp formats are specified in Section 5. In all 812 three cases, the Timestamp Subseconds field contains the 32 least 813 significant bits of the timestamp format that is specified in 814 Section 5. If a node is not capable of populating this field, it 815 assigns the value 0xFFFFFFFF. Note that this is a legitimate 816 value in the NTP format, valid for approximately 233 picoseconds 817 in every second. If the NTP format is used the analyzer should 818 correlate several packets in order to detect the error indication. 820 transit delay: 4-octet unsigned integer in the range 0 to 2^31-1. 821 It is the time in nanoseconds the packet spent in the transit 822 node. This can serve as an indication of the queuing delay at the 823 node. If the transit delay exceeds 2^31-1 nanoseconds then the 824 top bit 'O' is set to indicate overflow and value set to 825 0x80000000. When this field is part of the data field but a node 826 populating the field is not able to fill it, the field position in 827 the field must be filled with value 0xFFFFFFFF to mean not 828 populated. 830 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 831 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 832 |O| transit delay | 833 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 835 namespace specific data: 4-octet field which can be used by the node 836 to add namespace specific data. This represents a "free-format" 837 4-octet bit field with its semantics defined in the context of a 838 specific namespace. 840 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 841 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 842 | namespace specific data | 843 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 845 queue depth: 4-octet unsigned integer field. This field indicates 846 the current length of the egress interface queue of the interface 847 from where the packet is forwarded out. The queue depth is 848 expressed as the current number of memory buffers used by the 849 queue (a packet may consume one or more memory buffers, depending 850 on its size). 852 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 853 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 854 | queue depth | 855 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 857 Opaque State Snapshot: Variable length field. It allows the network 858 element to store an arbitrary state in the node data field, 859 without a pre-defined schema. The schema is to be defined within 860 the context of a namespace. The schema needs to be made known to 861 the analyzer by some out-of-band mechanism. The specification of 862 this mechanism is beyond the scope of this document. A 24-bit 863 "Schema Id" field, interpreted within the context of a namespace, 864 indicates which particular schema is used, and should be 865 configured on the network element by the operator. 867 0 1 2 3 868 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 869 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 870 | Length | Schema ID | 871 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 872 | | 873 | | 874 | Opaque data | 875 ~ ~ 876 . . 877 . . 878 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 880 Length: 1-octet unsigned integer. It is the length in multiples 881 of 4-octets of the Opaque data field that follows Schema Id. 883 Schema ID: 3-octet unsigned integer identifying the schema of 884 Opaque data. 886 Opaque data: Variable length field. This field is interpreted as 887 specified by the schema identified by the Schema ID. 889 When this field is part of the data field but a node populating 890 the field has no opaque state data to report, the Length must be 891 set to 0 and the Schema ID must be set to 0xFFFFFF to mean no 892 schema. 894 Hop_Lim and node_id wide: 8-octet field defined as follows: 896 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 897 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 898 | Hop_Lim | node_id ~ 899 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 900 ~ node_id (contd) | 901 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 903 Hop_Lim: 1-octet unsigned integer. It is set to the Hop Limit 904 value in the packet at the node that records this data. Hop 905 Limit information is used to identify the location of the node 906 in the communication path. This is copied from the lower layer 907 for e.g. TTL value in IPv4 header or hop limit field from IPv6 908 header of the packet. The semantics of the Hop_Lim field 909 depend on the lower layer protocol that IOAM is encapsulated 910 over, and therefore its specific semantics are outside the 911 scope of this memo. 913 node_id: 7-octet unsigned integer. Node identifier field to 914 uniquely identify a node within in-situ OAM domain. The 915 procedure to allocate, manage and map the node_ids is beyond 916 the scope of this document. 918 ingress_if_id and egress_if_id wide: 8-octet field defined as 919 follows: 921 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 922 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 923 | ingress_if_id | 924 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 925 | egress_if_id | 926 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 928 ingress_if_id: 4-octet unsigned integer. Interface identifier to 929 record the ingress interface the packet was received on. 931 egress_if_id: 4-octet unsigned integer. Interface identifier to 932 record the egress interface the packet is forwarded out of. 934 namespace specific data wide: 8-octet field which can be used by the 935 node to add namespace specific data. This represents a "free- 936 format" 8-octet bit field with its semantics defined in the 937 context of a specific namespace. 939 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 940 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 941 | namespace specific data ~ 942 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 943 ~ namespace specific data (contd) | 944 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 946 buffer occupancy: 4-octet unsigned integer field. This field 947 indicates the current status of the occupancy of the common buffer 948 pool used by a set of queues. The units of this field depend on 949 the equipment type and deployment and has to be interpreted within 950 the context of a namespace and/or node-id if used. 952 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 953 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 954 | buffer occupancy | 955 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 957 Checksum Complement: 4-octet node data which contains a two-octet 958 Checksum Complement field, and a 2-octet reserved field. The 959 Checksum Complement is useful when IOAM is transported over 960 encapsulations that make use of a UDP transport, such as VXLAN-GPE 961 or Geneve. Without the Checksum Complement, nodes adding IOAM 962 node data must update the UDP Checksum field. When the Checksum 963 Complement is present, an IOAM encapsulating node or IOAM transit 964 node adding node data MUST carry out one of the following two 965 alternatives in order to maintain the correctness of the UDP 966 Checksum value: 968 1. Recompute the UDP Checksum field. 970 2. Use the Checksum Complement to make a checksum-neutral update 971 in the UDP payload; the Checksum Complement is assigned a 972 value that complements the rest of the node data fields that 973 were added by the current node, causing the existing UDP 974 Checksum field to remain correct. 976 IOAM decapsulating nodes MUST recompute the UDP Checksum field, 977 since they do not know whether previous hops modified the UDP 978 Checksum field or the Checksum Complement field. 980 Checksum Complement fields are used in a similar manner in 981 [RFC7820] and [RFC7821]. 983 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 984 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 985 | Checksum Complement | Reserved | 986 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 988 4.2.3. Examples of IOAM node data 990 An entry in the "node data list" array can have different formats, 991 following the needs of the deployment. Some deployments might only 992 be interested in recording the node identifiers, whereas others might 993 be interested in recording node identifier and timestamp. The 994 section defines different types that an entry in "node data list" can 995 take. 997 0xD40000: IOAM-Trace-Type is 0xD40000 then the format of node data 998 is: 1000 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1001 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1002 | Hop_Lim | node_id | 1003 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1004 | ingress_if_id | egress_if_id | 1005 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1006 | timestamp subseconds | 1007 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1008 | namespace specific data | 1009 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1011 0xC00000: IOAM-Trace-Type is 0xC00000 then the format is: 1013 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1014 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1015 | Hop_Lim | node_id | 1016 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1017 | ingress_if_id | egress_if_id | 1018 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1020 0x900000: IOAM-Trace-Type is 0x900000 then the format is: 1022 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1023 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1024 | Hop_Lim | node_id | 1025 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1026 | timestamp subseconds | 1027 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1029 0x840000: IOAM-Trace-Type is 0x840000 then the format is: 1031 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1032 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1033 | Hop_Lim | node_id | 1034 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1035 | namespace specific data | 1036 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1038 0x940000: IOAM-Trace-Type is 0x940000 then the format is: 1040 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1041 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1042 | Hop_Lim | node_id | 1043 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1044 | timestamp subseconds | 1045 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1046 | namespace specific data | 1047 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1049 0x318000: IOAM-Trace-Type is 0x318000 then the format is: 1051 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1052 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1053 | timestamp seconds | 1054 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1055 | timestamp subseconds | 1056 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1057 | Length | Schema Id | 1058 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1059 | | 1060 | | 1061 | Opaque data | 1062 ~ ~ 1063 . . 1064 . . 1065 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1066 | Hop_Lim | node_id | 1067 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1068 | node_id(contd) | 1069 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1071 4.3. IOAM Proof of Transit Option 1073 IOAM Proof of Transit data is to support the path or service function 1074 chain [RFC7665] verification use cases. Proof-of-transit uses 1075 methods like nested hashing or nested encryption of the IOAM data or 1076 mechanisms such as Shamir's Secret Sharing Schema (SSSS). While 1077 details on how the IOAM data for the proof of transit option is 1078 processed at IOAM encapsulating, decapsulating and transit nodes are 1079 outside the scope of the document, all of these approaches share the 1080 need to uniquely identify a packet as well as iteratively operate on 1081 a set of information that is handed from node to node. 1082 Correspondingly, two pieces of information are added as IOAM data to 1083 the packet: 1085 o Random: Unique identifier for the packet (e.g., 64-bits allow for 1086 the unique identification of 2^64 packets). 1088 o Cumulative: Information which is handed from node to node and 1089 updated by every node according to a verification algorithm. 1091 IOAM proof of transit option header: 1093 0 1 2 3 1094 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1095 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1096 | Namespace-ID |IOAM POT Type | IOAM POT flags| 1097 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1099 IOAM proof of transit option data MUST be 4-octet aligned.: 1101 0 1 2 3 1102 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1103 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1104 | POT Option data field determined by IOAM-POT-Type | 1105 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1107 Namespace-ID: 16-bit identifier of an IOAM namespace. The 1108 Namespace-ID value of 0x0000 is defined as the default value and 1109 MUST be known to all the nodes implementing IOAM. For any other 1110 Namespace-ID value that does not match any Namespace-ID the node 1111 is configured to operate on, the node MUST NOT change the contents 1112 of the IOAM data fields. 1114 IOAM POT Type: 8-bit identifier of a particular POT variant that 1115 specifies the POT data that is included. This document defines 1116 POT Type 0: 1118 0: POT data is a 16 Octet field as described below. 1120 IOAM POT flags: 8-bit. Following flags are defined: 1122 Bit 0 "Profile-to-use" (P-bit) (most significant bit). For IOAM 1123 POT types that use a maximum of two profiles to drive 1124 computation, indicates which POT-profile is used. The two 1125 profiles are numbered 0, 1. 1127 Bit 1-7 Reserved: Must be set to zero upon transmission and 1128 ignored upon receipt. 1130 POT Option data: Variable-length field. The type of which is 1131 determined by the IOAM-POT-Type. 1133 4.3.1. IOAM Proof of Transit Type 0 1135 IOAM proof of transit option of IOAM POT Type 0: 1137 0 1 2 3 1138 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1139 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1140 | Namespace-ID |IOAM POT Type=0|P|R R R R R R R| 1141 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+<-+ 1142 | Random | | 1143 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ P 1144 | Random(contd) | O 1145 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ T 1146 | Cumulative | | 1147 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 1148 | Cumulative (contd) | | 1149 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+<-+ 1151 Namespace-ID: 16-bit identifier of an IOAM namespace. The 1152 Namespace-ID value of 0x0000 is defined as the default value and 1153 MUST be known to all the nodes implementing IOAM. For any other 1154 Namespace-ID value that does not match any Namespace-ID the node 1155 is configured to operate on, the node MUST NOT change the contents 1156 of the IOAM data fields. 1158 IOAM POT Type: 8-bit identifier of a particular POT variant that 1159 specifies the POT data that is included. This section defines the 1160 POT data when the IOAM POT Type is set to the value 0. 1162 P bit: 1-bit. "Profile-to-use" (P-bit) (most significant bit). 1163 Indicates which POT-profile is used to generate the Cumulative. 1164 Any node participating in POT will have a maximum of 2 profiles 1165 configured that drive the computation of cumulative. The two 1166 profiles are numbered 0, 1. This bit conveys whether profile 0 or 1167 profile 1 is used to compute the Cumulative. 1169 R (7 bits): 7-bit IOAM POT flags for future use. MUST be set to 1170 zero upon transmission and ignored upon receipt. 1172 Random: 64-bit Per packet Random number. 1174 Cumulative: 64-bit Cumulative that is updated at specific nodes by 1175 processing per packet Random number field and configured 1176 parameters. 1178 Note: Larger or smaller sizes of "Random" and "Cumulative" data are 1179 feasible and could be required for certain deployments (e.g. in case 1180 of space constraints in the transport protocol used). Future 1181 versions of this document will address different sizes of data for 1182 "proof of transit". 1184 4.4. IOAM Edge-to-Edge Option 1186 The IOAM edge-to-edge option is to carry data that is added by the 1187 IOAM encapsulating node and interpreted by IOAM decapsulating node. 1188 The IOAM transit nodes MAY process the data without modifying it. 1190 IOAM edge-to-edge option header: 1192 0 1 2 3 1193 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1194 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1195 | Namespace-ID | IOAM-E2E-Type | 1196 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1198 IOAM edge-to-edge option data MUST be 4-octet aligned: 1200 0 1 2 3 1201 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1202 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1203 | E2E Option data field determined by IOAM-E2E-Type | 1204 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1206 Namespace-ID: 16-bit identifier of an IOAM namespace. The 1207 Namespace-ID value of 0x0000 is defined as the default value and 1208 MUST be known to all the nodes implementing IOAM. For any other 1209 Namespace-ID value that does not match any Namespace-ID the node 1210 is configured to operate on, then the node MUST NOT change the 1211 contents of the IOAM data fields. 1213 IOAM-E2E-Type: A 16-bit identifier which specifies which data types 1214 are used in the E2E option data. The IOAM-E2E-Type value is a bit 1215 field. The order of packing the E2E option data field elements 1216 follows the bit order of the IOAM-E2E-Type field, as follows: 1218 Bit 0 (Most significant bit) When set indicates presence of a 1219 64-bit sequence number added to a specific "packet group" 1220 which is used to detect packet loss, packet reordering, 1221 or packet duplication within the group. The "packet 1222 group" is deployment dependent and defined at the IOAM 1223 encapsulating node e.g. by n-tuple based classification 1224 of packets. 1226 Bit 1 When set indicates presence of a 32-bit sequence number 1227 added to a specific "packet group" which is used to 1228 detect packet loss, packet reordering, or packet 1229 duplication within that group. The "packet group" is 1230 deployment dependent and defined at the IOAM 1231 encapsulating node e.g. by n-tuple based classification 1232 of packets. 1234 Bit 2 When set indicates presence of timestamp seconds for the 1235 transmission of the frame. This 4-octet field has three 1236 possible formats; based on either PTP [IEEE1588v2], NTP 1237 [RFC5905], or POSIX [POSIX]. The three timestamp formats 1238 are specified in Section 5. In all three cases, the 1239 Timestamp Seconds field contains the 32 most significant 1240 bits of the timestamp format that is specified in 1241 Section 5. If a node is not capable of populating this 1242 field, it assigns the value 0xFFFFFFFF. Note that this 1243 is a legitimate value that is valid for 1 second in 1244 approximately 136 years; the analyzer should correlate 1245 several packets or compare the timestamp value to its own 1246 time-of-day in order to detect the error indication. 1248 Bit 3 When set indicates presence of timestamp subseconds for 1249 the transmission of the frame. This 4-octet field has 1250 three possible formats; based on either PTP [IEEE1588v2], 1251 NTP [RFC5905], or POSIX [POSIX]. The three timestamp 1252 formats are specified in Section 5. In all three cases, 1253 the Timestamp Subseconds field contains the 32 least 1254 significant bits of the timestamp format that is 1255 specified in Section 5. If a node is not capable of 1256 populating this field, it assigns the value 0xFFFFFFFF. 1257 Note that this is a legitimate value in the NTP format, 1258 valid for approximately 233 picoseconds in every second. 1259 If the NTP format is used the analyzer should correlate 1260 several packets in order to detect the error indication. 1262 Bit 4-15 Undefined. An IOAM encapsulating node Must set the value 1263 of these bits to zero upon transmission and ignore upon 1264 receipt. 1266 E2E Option data: Variable-length field. The type of which is 1267 determined by the IOAM-E2E-Type. 1269 5. Timestamp Formats 1271 The IOAM data fields include a timestamp field which is represented 1272 in one of three possible timestamp formats. It is assumed that the 1273 management plane is responsible for determining which timestamp 1274 format is used. 1276 5.1. PTP Truncated Timestamp Format 1278 The Precision Time Protocol (PTP) [IEEE1588v2] uses an 80-bit 1279 timestamp format. The truncated timestamp format is a 64-bit field, 1280 which is the 64 least significant bits of the 80-bit PTP timestamp. 1281 The PTP truncated format is specified in Section 4.3 of 1282 [I-D.ietf-ntp-packet-timestamps], and the details are presented below 1283 for the sake of completeness. 1285 0 1 2 3 1286 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1287 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1288 | Seconds | 1289 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1290 | Nanoseconds | 1291 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1293 Figure 1: PTP [IEEE1588v2] Truncated Timestamp Format 1295 Timestamp field format: 1297 Seconds: specifies the integer portion of the number of seconds 1298 since the epoch. 1300 + Size: 32 bits. 1302 + Units: seconds. 1304 Nanoseconds: specifies the fractional portion of the number of 1305 seconds since the epoch. 1307 + Size: 32 bits. 1309 + Units: nanoseconds. The value of this field is in the range 0 1310 to (10^9)-1. 1312 Epoch: 1314 The PTP [IEEE1588v2] epoch is 1 January 1970 00:00:00 TAI, which 1315 is 31 December 1969 23:59:51.999918 UTC. 1317 Resolution: 1319 The resolution is 1 nanosecond. 1321 Wraparound: 1323 This time format wraps around every 2^32 seconds, which is roughly 1324 136 years. The next wraparound will occur in the year 2106. 1326 Synchronization Aspects: 1328 It is assumed that nodes that run this protocol are synchronized 1329 among themselves. Nodes may be synchronized to a global reference 1330 time. Note that if PTP [IEEE1588v2] is used for synchronization, 1331 the timestamp may be derived from the PTP-synchronized clock, 1332 allowing the timestamp to be measured with respect to the clock of 1333 an PTP Grandmaster clock. 1335 The PTP truncated timestamp format is not affected by leap 1336 seconds. 1338 5.2. NTP 64-bit Timestamp Format 1340 The Network Time Protocol (NTP) [RFC5905] timestamp format is 64 bits 1341 long. This format is specified in Section 4.2.1 of 1342 [I-D.ietf-ntp-packet-timestamps], and the details are presented below 1343 for the sake of completeness. 1345 0 1 2 3 1346 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1347 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1348 | Seconds | 1349 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1350 | Fraction | 1351 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1353 Figure 2: NTP [RFC5905] 64-bit Timestamp Format 1355 Timestamp field format: 1357 Seconds: specifies the integer portion of the number of seconds 1358 since the epoch. 1360 + Size: 32 bits. 1362 + Units: seconds. 1364 Fraction: specifies the fractional portion of the number of 1365 seconds since the epoch. 1367 + Size: 32 bits. 1369 + Units: the unit is 2^(-32) seconds, which is roughly equal to 1370 233 picoseconds. 1372 Epoch: 1374 The epoch is 1 January 1900 at 00:00 UTC. 1376 Resolution: 1378 The resolution is 2^(-32) seconds. 1380 Wraparound: 1382 This time format wraps around every 2^32 seconds, which is roughly 1383 136 years. The next wraparound will occur in the year 2036. 1385 Synchronization Aspects: 1387 Nodes that use this timestamp format will typically be 1388 synchronized to UTC using NTP [RFC5905]. Thus, the timestamp may 1389 be derived from the NTP-synchronized clock, allowing the timestamp 1390 to be measured with respect to the clock of an NTP server. 1392 The NTP timestamp format is affected by leap seconds; it 1393 represents the number of seconds since the epoch minus the number 1394 of leap seconds that have occurred since the epoch. The value of 1395 a timestamp during or slightly after a leap second may be 1396 temporarily inaccurate. 1398 5.3. POSIX-based Timestamp Format 1400 This timestamp format is based on the POSIX time format [POSIX]. The 1401 detailed specification of the timestamp format used in this document 1402 is presented below. 1404 0 1 2 3 1405 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1406 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1407 | Seconds | 1408 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1409 | Microseconds | 1410 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1412 Figure 3: POSIX-based Timestamp Format 1414 Timestamp field format: 1416 Seconds: specifies the integer portion of the number of seconds 1417 since the epoch. 1419 + Size: 32 bits. 1421 + Units: seconds. 1423 Microseconds: specifies the fractional portion of the number of 1424 seconds since the epoch. 1426 + Size: 32 bits. 1428 + Units: the unit is microseconds. The value of this field is in 1429 the range 0 to (10^6)-1. 1431 Epoch: 1433 The epoch is 1 January 1970 00:00:00 TAI, which is 31 December 1434 1969 23:59:51.999918 UTC. 1436 Resolution: 1438 The resolution is 1 microsecond. 1440 Wraparound: 1442 This time format wraps around every 2^32 seconds, which is roughly 1443 136 years. The next wraparound will occur in the year 2106. 1445 Synchronization Aspects: 1447 It is assumed that nodes that use this timestamp format run Linux 1448 operating system, and hence use the POSIX time. In some cases 1449 nodes may be synchronized to UTC using a synchronization mechanism 1450 that is outside the scope of this document, such as NTP [RFC5905]. 1451 Thus, the timestamp may be derived from the NTP-synchronized 1452 clock, allowing the timestamp to be measured with respect to the 1453 clock of an NTP server. 1455 The POSIX-based timestamp format is affected by leap seconds; it 1456 represents the number of seconds since the epoch minus the number 1457 of leap seconds that have occurred since the epoch. The value of 1458 a timestamp during or slightly after a leap second may be 1459 temporarily inaccurate. 1461 6. IOAM Data Export 1463 IOAM nodes collect information for packets traversing a domain that 1464 supports IOAM. IOAM decapsulating nodes as well as IOAM transit 1465 nodes can choose to retrieve IOAM information from the packet, 1466 process the information further and export the information using 1467 e.g., IPFIX. The mechanisms and associated data formats for 1468 exporting IOAM data is outside the scope of this document. 1470 Raw data export of IOAM data using IPFIX is discussed in 1471 [I-D.spiegel-ippm-ioam-rawexport]. 1473 7. IANA Considerations 1475 This document requests the following IANA Actions. 1477 7.1. Creation of a new In-Situ OAM Protocol Parameters Registry (IOAM) 1478 Protocol Parameters IANA registry 1480 IANA is requested to create a new protocol registry for "In-Situ OAM 1481 (IOAM) Protocol Parameters". This is the common registry that will 1482 include registrations for all IOAM namespaces. Each Registry, whose 1483 names are listed below: 1485 IOAM Type 1487 IOAM Trace Type 1489 IOAM Trace flags 1491 IOAM POT Type 1493 IOAM POT flags 1495 IOAM E2E Type 1497 IOAM Namespace-ID 1499 will contain the current set of possibilities defined in this 1500 document. New registries in this name space are created via RFC 1501 Required process as per [RFC8126]. 1503 The subsequent sub-sections detail the registries herein contained. 1505 7.2. IOAM Type Registry 1507 This registry defines 128 code points for the IOAM-Type field for 1508 identifying IOAM options as explained in Section 4. The following 1509 code points are defined in this draft: 1511 0 IOAM Pre-allocated Trace Option Type 1513 1 IOAM Incremental Trace Option Type 1515 2 IOAM POT Option Type 1517 3 IOAM E2E Option Type 1519 4 - 127 are available for assignment via RFC Required process as per 1520 [RFC8126]. 1522 7.3. IOAM Trace Type Registry 1524 This registry defines code point for each bit in the 24-bit IOAM- 1525 Trace-Type field for Pre-allocated trace option and Incremental trace 1526 option defined in Section 4.2. The meaning of Bits 0 - 11 for trace 1527 type are defined in this document in Paragraph 5 of Section 4.2.1: 1529 Bit 0 hop_Lim and node_id in short format 1531 Bit 1 ingress_if_id and egress_if_id in short format 1533 Bit 2 timestamp seconds 1535 Bit 3 timestamp subseconds 1537 Bit 4 transit delay 1539 Bit 5 namespace specific data in short format 1541 Bit 6 queue depth 1543 Bit 7 variable length Opaque State Snapshot 1545 Bit 8 hop_Lim and node_id in wide format 1546 Bit 9 ingress_if_id and egress_if_id in wide format 1548 Bit 10 namespace specific data in wide format 1550 Bit 11 buffer occupancy 1552 Bit 23 checksum complement 1554 The meaning for Bits 12 - 22 are available for assignment via RFC 1555 Required process as per [RFC8126]. 1557 7.4. IOAM Trace Flags Registry 1559 This registry defines code points for each bit in the 4 bit flags for 1560 Pre-allocated trace option and Incremental trace option defined in 1561 Section 4.2. The meaning of Bit 0 - 2 for trace flags are defined in 1562 this document in Paragraph 3 of Section 4.2.1: 1564 Bit 0 "Overflow" (O-bit) 1566 Bit 1 "Loopback" (L-bit) 1568 Bit 2 "Active" (A-bit) 1570 Bit 3 "Immediate Export" (I-bit) 1572 7.5. IOAM POT Type Registry 1574 This registry defines 256 code points to define IOAM POT Type for 1575 IOAM proof of transit option Section 4.3. The code point value 0 is 1576 defined in this document: 1578 0: 16 Octet POT data 1580 1 - 255 are available for assignment via RFC Required process as per 1581 [RFC8126]. 1583 7.6. IOAM POT Flags Registry 1585 This registry defines code points for each bit in the 8 bit flags for 1586 IOAM POT option defined in Section 4.3. The meaning of Bit 0 for 1587 IOAM POT flags is defined in this document in Section 4.3: 1589 Bit 0 "Profile-to-use" (P-bit) 1591 The meaning for Bits 1 - 7 are available for assignment via RFC 1592 Required process as per [RFC8126]. 1594 7.7. IOAM E2E Type Registry 1596 This registry defines code points for each bit in the 16 bit IOAM- 1597 E2E-Type field for IOAM E2E option Section 4.4. The meaning of Bit 0 1598 - 3 are defined in this document: 1600 Bit 0 64-bit sequence number 1602 Bit 1 32-bit sequence number 1604 Bit 2 timestamp seconds 1606 Bit 3 timestamp subseconds 1608 The meaning of Bits 4 - 15 are available for assignment via RFC 1609 Required process as per [RFC8126]. 1611 7.8. IOAM Namespace-ID Registry 1613 IANA is requested to set up an "IOAM Namespace-ID Registry", 1614 containing 16-bit values. The meaning of Bit 0 is defined in this 1615 document. IANA is requested to reserve the values 0x0001 to 0x7FFF 1616 for private use (managed by operators), as specified in Section 4.1 1617 of the current document. Registry entries for the values 0x8000 to 1618 0xFFFF are to be assigned via the "Expert Review" policy defined in 1619 [RFC8126]. 1621 0: default namespace (known to all IOAM nodes) 1623 0x0001 - 0x7FFF: reserved for private use 1625 0x8000 - 0xFFFF: unassigned 1627 8. Security Considerations 1629 As discussed in [RFC7276], a successful attack on an OAM protocol in 1630 general, and specifically on IOAM, can prevent the detection of 1631 failures or anomalies, or create a false illusion of nonexistent 1632 ones. 1634 The Proof of Transit option (Section Section 4.3) is used for 1635 verifying the path of data packets. The security considerations of 1636 POT are further discussed in [I-D.brockners-proof-of-transit]. 1638 The data elements of IOAM can be used for network reconnaissance, 1639 allowing attackers to collect information about network paths, 1640 performance, queue states, buffer occupancy and other information. 1641 Note that in case IOAM is used in "immediate export" mode, the IOAM 1642 related trace information would not be available in the customer data 1643 packets, but would be exported by every IOAM node. IOAM data export 1644 and securing IOAM data export is outside the scope of this document. 1646 IOAM can be used as a means for implementing Denial of Service (DoS) 1647 attacks, or for amplifying them. For example, a malicious attacker 1648 can add an IOAM header to packets in order to consume the resources 1649 of network devices that take part in IOAM or collectors that analyze 1650 the IOAM data. Another example is a packet length attack, in which 1651 an attacker pushes IOAM headers into data packets, causing these 1652 packets to be increased beyond the MTU size, resulting in 1653 fragmentation or in packet drops. 1655 Since IOAM options may include timestamps, if network devices use 1656 synchronization protocols then any attack on the time protocol 1657 [RFC7384] can compromise the integrity of the timestamp-related data 1658 fields. 1660 At the management plane, attacks may be implemented by misconfiguring 1661 or by maliciously configuring IOAM-enabled nodes in a way that 1662 enables other attacks. Thus, IOAM configuration should be secured in 1663 a way that authenticates authorized users and verifies the integrity 1664 of configuration procedures. 1666 Notably, IOAM is expected to be deployed in specific network domains, 1667 thus confining the potential attack vectors to within the network 1668 domain. Indeed, in order to limit the scope of threats to within the 1669 current network domain the network operator is expected to enforce 1670 policies that prevent IOAM traffic from leaking outside of the IOAM 1671 domain, and prevent IOAM data from outside the domain to be processed 1672 and used within the domain. Another way to prevent the data to get 1673 leaked is using the Immediate Export mode of the trace option. 1675 9. Acknowledgements 1677 The authors would like to thank Eric Vyncke, Nalini Elkins, Srihari 1678 Raghavan, Ranganathan T S, Karthik Babu Harichandra Babu, Akshaya 1679 Nadahalli, LJ Wobker, Erik Nordmark, Vengada Prasad Govindan, Andrew 1680 Yourtchenko, Aviv Kfir, Tianran Zhou, Haoyu song and Robin 1681 for the comments and advice. 1683 This document leverages and builds on top of several concepts 1684 described in [I-D.kitamura-ipv6-record-route]. The authors would 1685 like to acknowledge the work done by the author Hiroshi Kitamura and 1686 people involved in writing it. 1688 The authors would like to gracefully acknowledge useful review and 1689 insightful comments received from Joe Clarke, Al Morton, and Mickey 1690 Spiegel. 1692 The authors would like to acknowledge the contribution of "Immediate 1693 export" of IOAM trace by Barak Gafni. 1695 10. References 1697 10.1. Normative References 1699 [IEEE1588v2] 1700 Institute of Electrical and Electronics Engineers, "IEEE 1701 Std 1588-2008 - IEEE Standard for a Precision Clock 1702 Synchronization Protocol for Networked Measurement and 1703 Control Systems", IEEE Std 1588-2008, 2008, 1704 . 1707 [POSIX] Institute of Electrical and Electronics Engineers, "IEEE 1708 Std 1003.1-2008 (Revision of IEEE Std 1003.1-2004) - IEEE 1709 Standard for Information Technology - Portable Operating 1710 System Interface (POSIX(R))", IEEE Std 1003.1-2008, 2008, 1711 . 1714 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1715 Requirement Levels", BCP 14, RFC 2119, 1716 DOI 10.17487/RFC2119, March 1997, 1717 . 1719 [RFC5905] Mills, D., Martin, J., Ed., Burbank, J., and W. Kasch, 1720 "Network Time Protocol Version 4: Protocol and Algorithms 1721 Specification", RFC 5905, DOI 10.17487/RFC5905, June 2010, 1722 . 1724 [RFC8126] Cotton, M., Leiba, B., and T. Narten, "Guidelines for 1725 Writing an IANA Considerations Section in RFCs", BCP 26, 1726 RFC 8126, DOI 10.17487/RFC8126, June 2017, 1727 . 1729 10.2. Informative References 1731 [I-D.brockners-proof-of-transit] 1732 Brockners, F., Bhandari, S., Dara, S., Pignataro, C., 1733 Leddy, J., Youell, S., Mozes, D., and T. Mizrahi, "Proof 1734 of Transit", draft-brockners-proof-of-transit-05 (work in 1735 progress), May 2018. 1737 [I-D.ietf-ntp-packet-timestamps] 1738 Mizrahi, T., Fabini, J., and A. Morton, "Guidelines for 1739 Defining Packet Timestamps", draft-ietf-ntp-packet- 1740 timestamps-06 (work in progress), February 2019. 1742 [I-D.ietf-nvo3-geneve] 1743 Gross, J., Ganga, I., and T. Sridhar, "Geneve: Generic 1744 Network Virtualization Encapsulation", draft-ietf- 1745 nvo3-geneve-11 (work in progress), March 2019. 1747 [I-D.ietf-nvo3-vxlan-gpe] 1748 Maino, F., Kreeger, L., and U. Elzur, "Generic Protocol 1749 Extension for VXLAN", draft-ietf-nvo3-vxlan-gpe-06 (work 1750 in progress), April 2018. 1752 [I-D.kitamura-ipv6-record-route] 1753 Kitamura, H., "Record Route for IPv6 (PR6) Hop-by-Hop 1754 Option Extension", draft-kitamura-ipv6-record-route-00 1755 (work in progress), November 2000. 1757 [I-D.lapukhov-dataplane-probe] 1758 Lapukhov, P. and r. remy@barefootnetworks.com, "Data-plane 1759 probe for in-band telemetry collection", draft-lapukhov- 1760 dataplane-probe-01 (work in progress), June 2016. 1762 [I-D.spiegel-ippm-ioam-rawexport] 1763 Spiegel, M., Brockners, F., Bhandari, S., and R. 1764 Sivakolundu, "In-situ OAM raw data export with IPFIX", 1765 draft-spiegel-ippm-ioam-rawexport-01 (work in progress), 1766 October 2018. 1768 [RFC7276] Mizrahi, T., Sprecher, N., Bellagamba, E., and Y. 1769 Weingarten, "An Overview of Operations, Administration, 1770 and Maintenance (OAM) Tools", RFC 7276, 1771 DOI 10.17487/RFC7276, June 2014, 1772 . 1774 [RFC7384] Mizrahi, T., "Security Requirements of Time Protocols in 1775 Packet Switched Networks", RFC 7384, DOI 10.17487/RFC7384, 1776 October 2014, . 1778 [RFC7665] Halpern, J., Ed. and C. Pignataro, Ed., "Service Function 1779 Chaining (SFC) Architecture", RFC 7665, 1780 DOI 10.17487/RFC7665, October 2015, 1781 . 1783 [RFC7799] Morton, A., "Active and Passive Metrics and Methods (with 1784 Hybrid Types In-Between)", RFC 7799, DOI 10.17487/RFC7799, 1785 May 2016, . 1787 [RFC7820] Mizrahi, T., "UDP Checksum Complement in the One-Way 1788 Active Measurement Protocol (OWAMP) and Two-Way Active 1789 Measurement Protocol (TWAMP)", RFC 7820, 1790 DOI 10.17487/RFC7820, March 2016, 1791 . 1793 [RFC7821] Mizrahi, T., "UDP Checksum Complement in the Network Time 1794 Protocol (NTP)", RFC 7821, DOI 10.17487/RFC7821, March 1795 2016, . 1797 [RFC8300] Quinn, P., Ed., Elzur, U., Ed., and C. Pignataro, Ed., 1798 "Network Service Header (NSH)", RFC 8300, 1799 DOI 10.17487/RFC8300, January 2018, 1800 . 1802 Authors' Addresses 1804 Frank Brockners 1805 Cisco Systems, Inc. 1806 Hansaallee 249, 3rd Floor 1807 DUESSELDORF, NORDRHEIN-WESTFALEN 40549 1808 Germany 1810 Email: fbrockne@cisco.com 1812 Shwetha Bhandari 1813 Cisco Systems, Inc. 1814 Cessna Business Park, Sarjapura Marathalli Outer Ring Road 1815 Bangalore, KARNATAKA 560 087 1816 India 1818 Email: shwethab@cisco.com 1820 Carlos Pignataro 1821 Cisco Systems, Inc. 1822 7200-11 Kit Creek Road 1823 Research Triangle Park, NC 27709 1824 United States 1826 Email: cpignata@cisco.com 1827 Hannes Gredler 1828 RtBrick Inc. 1830 Email: hannes@rtbrick.com 1832 John Leddy 1833 Comcast 1834 United States 1836 Email: John_Leddy@cable.comcast.com 1838 Stephen Youell 1839 JP Morgan Chase 1840 25 Bank Street 1841 London E14 5JP 1842 United Kingdom 1844 Email: stephen.youell@jpmorgan.com 1846 Tal Mizrahi 1847 Huawei Network.IO Innovation Lab 1848 Israel 1850 Email: tal.mizrahi.phd@gmail.com 1852 David Mozes 1854 Email: mosesster@gmail.com 1856 Petr Lapukhov 1857 Facebook 1858 1 Hacker Way 1859 Menlo Park, CA 94025 1860 US 1862 Email: petr@fb.com 1864 Remy Chang 1865 Barefoot Networks 1866 4750 Patrick Henry Drive 1867 Santa Clara, CA 95054 1868 US 1869 Daniel Bernier 1870 Bell Canada 1871 Canada 1873 Email: daniel.bernier@bell.ca 1875 John Lemon 1876 Broadcom 1877 270 Innovation Drive 1878 San Jose, CA 95134 1879 US 1881 Email: john.lemon@broadcom.com