idnits 2.17.1 draft-ietf-ippm-ioam-data-12.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (February 21, 2021) is 1160 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) -- Looks like a reference, but probably isn't: '0' on line 610 -- Looks like a reference, but probably isn't: '1' on line 614 -- Possible downref: Non-RFC (?) normative reference: ref. 'IEEE1588v2' -- Possible downref: Non-RFC (?) normative reference: ref. 'POSIX' == Outdated reference: A later version (-03) exists of draft-brockners-ippm-ioam-data-integrity-00 == Outdated reference: A later version (-03) exists of draft-brockners-opsawg-ioam-deployment-02 == Outdated reference: A later version (-13) exists of draft-ietf-nvo3-vxlan-gpe-10 == Outdated reference: A later version (-07) exists of draft-spiegel-ippm-ioam-rawexport-04 Summary: 0 errors (**), 0 flaws (~~), 5 warnings (==), 5 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 ippm F. Brockners, Ed. 3 Internet-Draft Cisco 4 Intended status: Standards Track S. Bhandari, Ed. 5 Expires: August 25, 2021 Thoughtspot 6 T. Mizrahi, Ed. 7 Huawei 8 February 21, 2021 10 Data Fields for In-situ OAM 11 draft-ietf-ippm-ioam-data-12 13 Abstract 15 In-situ Operations, Administration, and Maintenance (IOAM) records 16 operational and telemetry information in the packet while the packet 17 traverses a path between two points in the network. This document 18 discusses the data fields and associated data types for in-situ OAM. 19 In-situ OAM data fields can be encapsulated into a variety of 20 protocols such as NSH, Segment Routing, Geneve, IPv6 (via extension 21 header), or IPv4. In-situ OAM can be used to complement OAM 22 mechanisms based on e.g. ICMP or other types of probe packets. 24 Status of This Memo 26 This Internet-Draft is submitted in full conformance with the 27 provisions of BCP 78 and BCP 79. 29 Internet-Drafts are working documents of the Internet Engineering 30 Task Force (IETF). Note that other groups may also distribute 31 working documents as Internet-Drafts. The list of current Internet- 32 Drafts is at https://datatracker.ietf.org/drafts/current/. 34 Internet-Drafts are draft documents valid for a maximum of six months 35 and may be updated, replaced, or obsoleted by other documents at any 36 time. It is inappropriate to use Internet-Drafts as reference 37 material or to cite them other than as "work in progress." 39 This Internet-Draft will expire on August 25, 2021. 41 Copyright Notice 43 Copyright (c) 2021 IETF Trust and the persons identified as the 44 document authors. All rights reserved. 46 This document is subject to BCP 78 and the IETF Trust's Legal 47 Provisions Relating to IETF Documents 48 (https://trustee.ietf.org/license-info) in effect on the date of 49 publication of this document. Please review these documents 50 carefully, as they describe your rights and restrictions with respect 51 to this document. Code Components extracted from this document must 52 include Simplified BSD License text as described in Section 4.e of 53 the Trust Legal Provisions and are provided without warranty as 54 described in the Simplified BSD License. 56 Table of Contents 58 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 59 2. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 3 60 3. Conventions . . . . . . . . . . . . . . . . . . . . . . . . . 4 61 4. Scope, Applicability, and Assumptions . . . . . . . . . . . . 5 62 5. IOAM Data-Fields, Types, Nodes . . . . . . . . . . . . . . . 6 63 5.1. IOAM Data-Fields and Option-Types . . . . . . . . . . . . 6 64 5.2. IOAM-Domains and types of IOAM Nodes . . . . . . . . . . 7 65 5.3. IOAM-Namespaces . . . . . . . . . . . . . . . . . . . . . 8 66 5.4. IOAM Trace Option-Types . . . . . . . . . . . . . . . . . 11 67 5.4.1. Pre-allocated and Incremental Trace Option-Types . . 13 68 5.4.2. IOAM node data fields and associated formats . . . . 17 69 5.4.2.1. Hop_Lim and node_id short format . . . . . . . . 18 70 5.4.2.2. ingress_if_id and egress_if_id . . . . . . . . . 18 71 5.4.2.3. timestamp seconds . . . . . . . . . . . . . . . . 19 72 5.4.2.4. timestamp subseconds . . . . . . . . . . . . . . 19 73 5.4.2.5. transit delay . . . . . . . . . . . . . . . . . . 19 74 5.4.2.6. namespace specific data . . . . . . . . . . . . . 20 75 5.4.2.7. queue depth . . . . . . . . . . . . . . . . . . . 20 76 5.4.2.8. Checksum Complement . . . . . . . . . . . . . . . 20 77 5.4.2.9. Hop_Lim and node_id wide . . . . . . . . . . . . 21 78 5.4.2.10. ingress_if_id and egress_if_id wide . . . . . . . 22 79 5.4.2.11. namespace specific data wide . . . . . . . . . . 22 80 5.4.2.12. buffer occupancy . . . . . . . . . . . . . . . . 22 81 5.4.2.13. Opaque State Snapshot . . . . . . . . . . . . . . 23 82 5.4.3. Examples of IOAM node data . . . . . . . . . . . . . 23 83 5.5. IOAM Proof of Transit Option-Type . . . . . . . . . . . . 25 84 5.5.1. IOAM Proof of Transit Type 0 . . . . . . . . . . . . 27 85 5.6. IOAM Edge-to-Edge Option-Type . . . . . . . . . . . . . . 28 86 6. Timestamp Formats . . . . . . . . . . . . . . . . . . . . . . 30 87 6.1. PTP Truncated Timestamp Format . . . . . . . . . . . . . 30 88 6.2. NTP 64-bit Timestamp Format . . . . . . . . . . . . . . . 32 89 6.3. POSIX-based Timestamp Format . . . . . . . . . . . . . . 33 90 7. IOAM Data Export . . . . . . . . . . . . . . . . . . . . . . 34 91 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 35 92 8.1. IOAM Option-Type Registry . . . . . . . . . . . . . . . . 35 93 8.2. IOAM Trace-Type Registry . . . . . . . . . . . . . . . . 36 94 8.3. IOAM Trace-Flags Registry . . . . . . . . . . . . . . . . 36 95 8.4. IOAM POT-Type Registry . . . . . . . . . . . . . . . . . 37 96 8.5. IOAM POT-Flags Registry . . . . . . . . . . . . . . . . . 37 97 8.6. IOAM E2E-Type Registry . . . . . . . . . . . . . . . . . 37 98 8.7. IOAM Namespace-ID Registry . . . . . . . . . . . . . . . 37 99 9. Management and Deployment Considerations . . . . . . . . . . 38 100 10. Security Considerations . . . . . . . . . . . . . . . . . . . 38 101 11. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 40 102 12. References . . . . . . . . . . . . . . . . . . . . . . . . . 40 103 12.1. Normative References . . . . . . . . . . . . . . . . . . 40 104 12.2. Informative References . . . . . . . . . . . . . . . . . 41 105 Contributors' Addresses . . . . . . . . . . . . . . . . . . . . . 43 106 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 45 108 1. Introduction 110 This document defines data fields for "in-situ" Operations, 111 Administration, and Maintenance (IOAM). In-situ OAM records OAM 112 information within the packet while the packet traverses a particular 113 network domain. The term "in-situ" refers to the fact that the OAM 114 data is added to the data packets rather than being sent within 115 packets specifically dedicated to OAM. IOAM is to complement 116 mechanisms such as Ping or Traceroute. In terms of "active" or 117 "passive" OAM, "in-situ" OAM can be considered a hybrid OAM type. 118 "In-situ" mechanisms do not require extra packets to be sent. IOAM 119 adds information to the already available data packets and therefore 120 cannot be considered passive. In terms of the classification given 121 in [RFC7799] IOAM could be portrayed as Hybrid Type 1. IOAM 122 mechanisms can be leveraged where mechanisms using e.g. ICMP do not 123 apply or do not offer the desired results, such as proving that a 124 certain traffic flow takes a pre-defined path, SLA verification for 125 the live data traffic, detailed statistics on traffic distribution 126 paths in networks that distribute traffic across multiple paths, or 127 scenarios in which probe traffic is potentially handled differently 128 from regular data traffic by the network devices. 130 IOAM use cases and mechanisms have expanded as this document matured, 131 resulting in additional flags and options that could trigger creation 132 of additional packets dedicated to OAM. The term IOAM continues to 133 be used for such mechanisms, in addition to the "in-situ" mechanisms 134 that motivated this terminology. 136 2. Contributors 138 This document was the collective effort of several authors. The text 139 and content were contributed by the editors and the co-authors listed 140 below. The contact information of the co-authors appears at the end 141 of this document. 143 o Carlos Pignataro 144 o Mickey Spiegel 146 o Barak Gafni 148 o Jennifer Lemon 150 o Hannes Gredler 152 o John Leddy 154 o Stephen Youell 156 o David Mozes 158 o Petr Lapukhov 160 o Remy Chang 162 o Daniel Bernier 164 3. Conventions 166 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 167 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 168 document are to be interpreted as described in [RFC2119]. 170 Abbreviations used in this document: 172 E2E Edge to Edge 174 Geneve: Generic Network Virtualization Encapsulation 175 [I-D.ietf-nvo3-geneve] 177 IOAM: In-situ Operations, Administration, and Maintenance 179 MTU: Maximum Transmit Unit 181 NSH: Network Service Header [RFC8300] 183 OAM: Operations, Administration, and Maintenance 185 PMTU Path MTU 187 POT: Proof of Transit 189 SFC: Service Function Chain 191 SID: Segment Identifier 192 SR: Segment Routing 194 VXLAN-GPE: Virtual eXtensible Local Area Network, Generic Protocol 195 Extension [I-D.ietf-nvo3-vxlan-gpe] 197 4. Scope, Applicability, and Assumptions 199 IOAM deployment assumes a set of constraints, requirements, and 200 guiding principles which are described in this section. 202 Scope: This document defines the data fields and associated data 203 types for in-situ OAM. The in-situ OAM data field can be 204 encapsulated in a variety of protocols, including NSH, Segment 205 Routing, Geneve, IPv6, or IPv4. Specification details for these 206 different protocols are outside the scope of this document. It is 207 expected that each such encapsulation will be defined in the relevant 208 working group in the IETF. 210 Deployment domain (or scope) of in-situ OAM deployment: IOAM is a 211 network domain focused feature, with "network domain" being a set of 212 network devices or entities within a single administration. For 213 example, a network domain can include an enterprise campus using 214 physical connections between devices or an overlay network using 215 virtual connections / tunnels for connectivity between said devices. 216 A network domain is defined by its perimeter or edge. Designers of 217 protocol encapsulations for IOAM specify mechanisms to ensure that 218 IOAM data stays within an IOAM domain. In addition, the operator of 219 such a domain is expected to put provisions in place to ensure that 220 IOAM data does not leak beyond the edge of an IOAM domain using,for 221 example, packet filtering methods. The operator has to consider the 222 potential operational impact of IOAM to mechanisms such as ECMP 223 processing (e.g. load-balancing schemes based on packet length could 224 be impacted by the increased packet size due to IOAM), path MTU (i.e. 225 ensure that the MTU of all links within a domain is sufficiently 226 large to support the increased packet size due to IOAM) and ICMP 227 message handling (i.e. in case of IPv6, IOAM support for ICMPv6 Echo 228 Request/Reply is desired which would translate into ICMPv6 extensions 229 to enable IOAM-Data-Fields to be copied from an Echo Request message 230 to an Echo Reply message). 232 IOAM control points: IOAM-Data-Fields are added to or removed from 233 the live user traffic by the devices which form the edge of a domain. 234 Devices which form an IOAM-Domain can add, update or remove IOAM- 235 Data-Fields. Edge devices of an IOAM-Domain can be hosts or network 236 devices. 238 Traffic-sets that IOAM is applied to: IOAM can be deployed on all or 239 only on subsets of the live user traffic. Using IOAM on a selected 240 set of traffic (e.g., per interface, based on an access control list 241 or flow specification defining a specific set of traffic, etc.) could 242 be useful in deployments where the cost of processing IOAM-Data- 243 Fields by encapsulating, transit, or decapsulating node(s) might be a 244 concern from a performance or operational perspective. Thus limiting 245 the amount of traffic IOAM is applied to could be beneficial in some 246 deployments. 248 Encapsulation independence: The definition of IOAM-Data-Fields is 249 independent from the protocols the IOAM-Data-Fields are encapsulated 250 into. IOAM-Data-Fields can be encapsulated into several 251 encapsulating protocols. The specification of how IOAM-Data-Fields 252 are encapsulated into "parent" protocols, like e.g., NSH or IPv6 is 253 outside the scope of this document. 255 Layering: If several encapsulation protocols (e.g., in case of 256 tunneling) are stacked on top of each other, IOAM-Data-Fields could 257 be present at multiple layers. The behavior follows the ships-in- 258 the-night model, i.e. IOAM-Data-Fields in one layer are independent 259 from IOAM-Data-Fields in another layer. Layering allows operators to 260 instrument the protocol layer they want to measure. The different 261 layers could, but do not have to, share the same IOAM encapsulation 262 mechanisms. 264 IOAM implementation: The definition of the IOAM-Data-Fields take the 265 specifics of devices with hardware data planes and software data 266 planes into account. 268 5. IOAM Data-Fields, Types, Nodes 270 This section details IOAM-related nomenclature and describes data 271 types such as IOAM-Data-Fields, IOAM-Types, IOAM-Namespaces as well 272 as the different types of IOAM nodes. 274 5.1. IOAM Data-Fields and Option-Types 276 An IOAM-Data-Field is a set of bits with a defined format and 277 meaning, which can be stored at a certain place in a packet for the 278 purpose of IOAM. 280 To accommodate the different uses of IOAM, IOAM-Data-Fields fall into 281 different categories. In IOAM these categories are referred to as 282 IOAM-Option-Types. A common registry is maintained for IOAM-Option- 283 Types, see Section 8.1 for details. Corresponding to these IOAM- 284 Option-Types, different IOAM-Data-Fields are defined. IOAM-Data- 285 Fields can be encapsulated into a variety of protocols, such as NSH, 286 Geneve, IPv6, etc. The definition of how IOAM-Data-Fields are 287 encapsulated into other protocols is outside the scope of this 288 document. 290 This document defines four IOAM-Option-Types: 292 o Pre-allocated Trace Option-Type 294 o Incremental Trace Option-Type 296 o Proof of Transit (POT) Option-Type 298 o Edge-to-Edge (E2E) Option-Type 300 5.2. IOAM-Domains and types of IOAM Nodes 302 IOAM is expected to be deployed in a specific domain. The part of 303 the network which employs IOAM is referred to as the "IOAM-Domain". 304 One or more IOAM-Option-Types are added to a packet upon entering the 305 IOAM-Domain and are removed from the packet when exiting the domain. 306 Within the IOAM-Domain, the IOAM-Data-Fields MAY be updated by 307 network nodes that the packet traverses. An IOAM-Domain consists of 308 "IOAM encapsulating nodes", "IOAM decapsulating nodes" and "IOAM 309 transit nodes". The role of a node (i.e. encapsulating, transit, 310 decapsulating) is defined within an IOAM-Namespace (see below). A 311 node can have different roles in different IOAM-Namespaces. 313 A device which adds at least one IOAM-Option-Type to the packet is 314 called the "IOAM encapsulating node", whereas a device which removes 315 an IOAM-Option-Type is referred to as the "IOAM decapsulating node". 316 Nodes within the domain which are aware of IOAM data and read and/or 317 write or process the IOAM data are called "IOAM transit nodes". IOAM 318 nodes which add or remove the IOAM-Data-Fields can also update the 319 IOAM-Data-Fields at the same time. Or in other words, IOAM 320 encapsulating or decapsulating nodes can also serve as IOAM transit 321 nodes at the same time. Note that not every node in an IOAM domain 322 needs to be an IOAM transit node. For example, a deployment might 323 require that packets traverse a set of firewalls which support IOAM. 324 In that case, only the set of firewall nodes would be IOAM transit 325 nodes rather than all nodes. 327 An "IOAM encapsulating node" incorporates one or more IOAM-Option- 328 Types (from the list of IOAM-Types, see Section 8.1) into packets 329 that IOAM is enabled for. If IOAM is enabled for a selected subset 330 of the traffic, the IOAM encapsulating node is responsible for 331 applying the IOAM functionality to the selected subset. 333 An "IOAM transit node" updates one or more of the IOAM-Data-Fields. 334 If both the Pre-allocated and the Incremental Trace Option-Types are 335 present in the packet, each IOAM transit node based on configuration 336 and available implementation of IOAM populates IOAM trace data in 337 either Pre-allocated or Incremental Trace Option-Type but not both. 338 A transit node MUST ignore IOAM-Option-Types that it does not 339 understand. A transit node MUST NOT add new IOAM-Option-Types to a 340 packet, MUST NOT remove IOAM-Option-Types from a packet, and MUST NOT 341 change the IOAM-Data-Fields of an IOAM Edge-to-Edge Option-Type. 343 An "IOAM decapsulating node" removes IOAM-Option-Type(s) from 344 packets. 346 The role of an IOAM-encapsulating, IOAM-transit or IOAM-decapsulating 347 node is always performed within a specific IOAM-Namespace. This 348 means that an IOAM node which is e.g. an IOAM-decapsulating node for 349 IOAM-Namespace "A" but not for IOAM-Namespace "B" will only remove 350 the IOAM-Option-Types for IOAM-Namespace "A" from the packet. Note 351 that this applies even for IOAM-Option-Types that the node does not 352 understand, for example an IOAM-Option-Type other than the four 353 described above, that is added in a future revision. An IOAM 354 decapsulating node situated at the edge of an IOAM domain MUST remove 355 all IOAM-Option-Types and associated encapsulation headers for all 356 IOAM-Namespaces from the packet. 358 IOAM-Namespaces allow for a namespace-specific definition and 359 interpretation of IOAM-Data-Fields. An interface-id could for 360 example point to a physical interface (e.g., to understand which 361 physical interface of an aggregated link is used when receiving or 362 transmitting a packet) whereas in another case it could refer to a 363 logical interface (e.g., in case of tunnels). Please refer to 364 Section 5.3 for details on IOAM-Namespaces. 366 5.3. IOAM-Namespaces 368 A subset or all of the IOAM-Option-Types and their corresponding 369 IOAM-Data-Fields can be associated to an IOAM-Namespace. IOAM- 370 Namespaces add further context to IOAM-Option-Types and associated 371 IOAM-Data-Fields. Any IOAM-Namespace MUST interpret the IOAM-Option- 372 Types and associated IOAM-Data-Fields per the definition in this 373 document. IOAM-Namespaces group nodes to support different 374 deployment approaches of IOAM (see a few example use-cases below) as 375 well as resolve issues which can occur due to IOAM-Data-Fields not 376 being globally unique (e.g. IOAM node identifiers do not have to be 377 globally unique). IOAM-Data-Fields significance is always within a 378 particular IOAM-Namespace. 380 An IOAM-Namespace is identified by a 16-bit namespace identifier 381 (Namespace-ID). IOAM-Namespace identifiers MUST be present and 382 populated in all IOAM-Option-Types. The Namespace-ID value is 383 divided into two sub-ranges: 385 o An operator-assigned range from 0x0001 to 0x7FFF 387 o An IANA-assigned range from 0x8000 to 0xFFFF 389 The IANA-assigned range is intended to allow future extensions to 390 have new and interoperable IOAM functionality, while the operator- 391 assigned range is intended to be domain specific, and managed by the 392 network operator. The Namespace-ID value of 0x0000 is the "Default- 393 Namespace-ID". The Default-Namespace-ID indicates that no specific 394 namespace is associated with the IOAM data fields in the packet. The 395 Default-Namespace-ID MUST be supported by all nodes implementing 396 IOAM. A use-case for the Default-Namespace-ID are deployments which 397 do not leverage specific namespaces for some or all of their packets 398 that carry IOAM data fields. 400 Namespace identifiers allow devices which are IOAM capable to 401 determine: 403 o whether IOAM-Option-Type(s) need to be processed by a device: If 404 the Namespace-ID contained in a packet does not match any 405 Namespace-ID the node is configured to operate on, then the node 406 MUST NOT change the contents of the IOAM-Data-Fields. 408 o which IOAM-Option-Type needs to be processed/updated in case there 409 are multiple IOAM-Option-Types present in the packet. Multiple 410 IOAM-Option-Types can be present in a packet in case of 411 overlapping IOAM-Domains or in case of a layered IOAM deployment. 413 o whether IOAM-Option-Type(s) has to be removed from the packet, 414 e.g. at a domain edge or domain boundary. 416 IOAM-Namespaces support several different uses: 418 o IOAM-Namespaces can be used by an operator to distinguish 419 different operational domains. Devices at domain edges can filter 420 on Namespace-IDs to provide for proper IOAM-Domain isolation. 422 o IOAM-Namespaces provide additional context for IOAM-Data-Fields 423 and thus ensure that IOAM-Data-Fields are unique and can be 424 interpreted properly by management stations or network 425 controllers. While, for example, the node identifier field 426 (node_id, see below) does not need to be unique in a deployment 427 (e.g. if an operator wishes to use different node identifiers for 428 different IOAM layers, even within the same device; or node 429 identifiers might not be unique for other organizational reasons, 430 such as after a merger of two formerly separated organizations), 431 the combination of node_id and Namespace-ID will always be unique. 432 Similarly, IOAM-Namespaces can be used to define how certain IOAM- 433 Data-Fields are interpreted: IOAM offers three different timestamp 434 format options. The Namespace-ID can be used to determine the 435 timestamp format. IOAM-Data-Fields (e.g. buffer occupancy) which 436 do not have a unit associated are to be interpreted within the 437 context of a IOAM-Namespace. 439 o IOAM-Namespaces can be used to identify different sets of devices 440 (e.g., different types of devices) in a deployment: If an operator 441 desires to insert different IOAM-Data-Fields based on the device, 442 the devices could be grouped into multiple IOAM-Namespaces. This 443 could be due to the fact that the IOAM feature set differs between 444 different sets of devices, or it could be for reasons of optimized 445 space usage in the packet header. It could also stem from 446 hardware or operational limitations on the size of the trace data 447 that can be added and processed, preventing collection of a full 448 trace for a flow. 450 * Assigning different IOAM Namespace-IDs to different sets of 451 nodes or network partitions and using the Namespace-ID as a 452 selector at the IOAM encapsulating node, a full trace for a 453 flow could be collected and constructed via partial traces in 454 different packets of the same flow. Example: An operator could 455 choose to group the devices of a domain into two IOAM- 456 Namespaces, in a way that on average, only every second hop 457 would be recorded by any device. To retrieve a full view of 458 the deployment, the captured IOAM-Data-Fields of the two IOAM- 459 Namespaces need to be correlated. 461 * Assigning different IOAM Namespace-IDs to different sets of 462 nodes or network partitions and using a separate instance of an 463 IOAM-Option-Type for each Namespace-ID, a full trace for a flow 464 could be collected and constructed via partial traces from each 465 IOAM-Option-Type in each of the packets in the flow. Example: 466 An operator could choose to group the devices of a domain into 467 two IOAM-Namespaces, in a way that each IOAM-Namespace is 468 represented by one of two IOAM-Option-Types in the packet. 469 Each node would record data only for the IOAM-Namespace that it 470 belongs to, ignoring the other IOAM-Option-Type with a IOAM- 471 Namespace to which it doesn't belong. To retrieve a full view 472 of the deployment, the captured IOAM-Data-Fields of the two 473 IOAM-Namespaces need to be correlated. 475 5.4. IOAM Trace Option-Types 477 "IOAM tracing data" is expected to be collected at every IOAM transit 478 node that a packet traverses to ensure visibility into the entire 479 path a packet takes within an IOAM-Domain. I.e., in a typical 480 deployment all nodes in an IOAM-Domain would participate in IOAM and 481 thus be IOAM transit nodes, IOAM encapsulating or IOAM decapsulating 482 nodes. If not all nodes within a domain support IOAM functionality 483 as defined in this document, IOAM tracing information (i.e., node 484 data, see below) will only be collected on those nodes which support 485 IOAM functionality as defined in this document. Nodes which do not 486 support IOAM functionality as defined in this document will forward 487 the packet without any changes to the IOAM-Data-Fields. The maximum 488 number of hops and the minimum path MTU of the IOAM domain is assumed 489 to be known. An overflow indicator (O-bit) is defined as one of the 490 ways to deal with situations where the PMTU was underestimated, i.e. 491 where the number of hops which are IOAM capable exceeds the available 492 space in the packet. 494 To optimize hardware and software implementations, IOAM tracing is 495 defined as two separate options. Any deployment MAY choose to 496 configure and support one or both of the following options. 498 Pre-allocated Trace-Option: This trace option is defined as a 499 container of node data fields (see below) with pre-allocated space 500 for each node to populate its information. This option is useful 501 for implementations where it is efficient to allocate the space 502 once and index into the array to populate the data during transit 503 (e.g., software forwarders often fall into this class). The IOAM 504 encapsulating node allocates space for Pre-allocated Trace Option- 505 Type in the packet and sets corresponding fields in this IOAM- 506 Option-Type. The IOAM encapsulating node allocates an array which 507 is used to store operational data retrieved from every node while 508 the packet traverses the domain. IOAM transit nodes update the 509 content of the array, and possibly update the checksums of outer 510 headers. A pointer which is part of the IOAM trace data, points 511 to the next empty slot in the array. An IOAM transit node that 512 updates the content of the pre-allocated option also updates the 513 value of the pointer, which specifies where the next IOAM transit 514 node fills in its data. The "node data list" array (see below) in 515 the packet is populated iteratively as the packet traverses the 516 network, starting with the last entry of the array, i.e., "node 517 data list [n]" is the first entry to be populated, "node data list 518 [n-1]" is the second one, etc. 520 Incremental Trace-Option: This trace option is defined as a 521 container of node data fields where each node allocates and pushes 522 its node data immediately following the option header. This type 523 of trace recording is useful for some of the hardware 524 implementations as it eliminates the need for the transit network 525 elements to read the full array in the option and allows for 526 arbitrarily long packets as the MTU allows. The IOAM 527 encapsulating node allocates space for the Incremental Trace 528 Option-Type. Based on operational state and configuration, the 529 IOAM encapsulating node sets the fields in the Option-Type that 530 control what IOAM-Data-Fields have to be collected and how large 531 the node data list can grow. IOAM transit nodes push their node 532 data to the node data list, decrease the remaining length 533 available to subsequent nodes and adjust the lengths and possibly 534 checksums in outer headers. 536 A particular implementation of IOAM MAY choose to support only one of 537 the two trace option types. In the event that both options are 538 utilized at the same time, the Incremental Trace-Option MUST be 539 placed before the Pre-allocated Trace-Option. Deployments which mix 540 devices with either the Incremental Trace-Option or the Pre-allocated 541 Trace-Option could result in both Option-Types being present in a 542 packet. Given that the operator knows which equipment is deployed in 543 a particular IOAM, the operator will decide by means of configuration 544 which type(s) of trace options will be used for a particular domain. 546 Every node data entry holds information for a particular IOAM transit 547 node that is traversed by a packet. The IOAM decapsulating node 548 removes the IOAM-Option-Type(s) and processes and/or exports the 549 associated data. Like all IOAM-Data-Fields, the IOAM-Data-Fields of 550 the IOAM-Trace-Option-Types are defined in the context of an IOAM- 551 Namespace. 553 IOAM tracing can collect the following types of information: 555 o Identification of the IOAM node. An IOAM node identifier can 556 match to a device identifier or a particular control point or 557 subsystem within a device. 559 o Identification of the interface that a packet was received on, 560 i.e. ingress interface. 562 o Identification of the interface that a packet was sent out on, 563 i.e. egress interface. 565 o Time of day when the packet was processed by the node as well as 566 the transit delay. Different definitions of processing time are 567 feasible and expected, though it is important that all devices of 568 an in-situ OAM domain follow the same definition. 570 o Generic data: Format-free information where syntax and semantic of 571 the information is defined by the operator in a specific 572 deployment. For a specific IOAM-Namespace, all IOAM nodes have to 573 interpret the generic data the same way. Examples for generic 574 IOAM data include geo-location information (location of the node 575 at the time the packet was processed), buffer queue fill level or 576 cache fill level at the time the packet was processed, or even a 577 battery charge level. 579 o Information to detect whether IOAM trace data was added at every 580 hop or whether certain hops in the domain weren't IOAM transit 581 nodes. 583 5.4.1. Pre-allocated and Incremental Trace Option-Types 585 The IOAM Pre-allocated Trace-Option and the IOAM Incremental Trace- 586 Option have similar formats. Except where noted below, the internal 587 formats and fields of the two trace options are identical. Both 588 Trace-Options consist of a fixed size "trace option header" and a 589 variable data space to store gathered data, the "node data list". An 590 IOAM transit node (that is not an IOAM encapsulating node or IOAM 591 decapsulating node) MUST NOT modify any of the fields in the fixed 592 size "trace option header", other than "flags" and "RemainingLen", 593 i.e. an IOAM transit node MUST NOT modify the Namespace-ID, NodeLen, 594 IOAM-Trace-Type, or Reserved fields. 596 Pre-allocated and incremental trace option headers: 598 0 1 2 3 599 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 600 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 601 | Namespace-ID |NodeLen | Flags | RemainingLen| 602 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 603 | IOAM-Trace-Type | Reserved | 604 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 606 The trace option data MUST be 4-octet aligned: 608 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+<-+ 609 | | | 610 | node data list [0] | | 611 | | | 612 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ D 613 | | a 614 | node data list [1] | t 615 | | a 616 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 617 ~ ... ~ S 618 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ p 619 | | a 620 | node data list [n-1] | c 621 | | e 622 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 623 | | | 624 | node data list [n] | | 625 | | | 626 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+<-+ 628 Namespace-ID: 16-bit identifier of an IOAM-Namespace. The 629 Namespace-ID value of 0x0000 is defined as the "Default-Namespace- 630 ID" (see Section 5.3) and MUST be known to all the nodes 631 implementing IOAM. For any other Namespace-ID value that does not 632 match any Namespace-ID the node is configured to operate on, the 633 node MUST NOT change the contents of the IOAM-Data-Fields. 635 NodeLen: 5-bit unsigned integer. This field specifies the length of 636 data added by each node in multiples of 4-octets, excluding the 637 length of the "Opaque State Snapshot" field. 639 If IOAM-Trace-Type bit 22 is not set, then NodeLen specifies the 640 actual length added by each node. If IOAM-Trace-Type bit 22 is 641 set, then the actual length added by a node would be (NodeLen + 642 length of the "Opaque State Snapshot" field) in 4 octet units. 644 For example, if 3 IOAM-Trace-Type bits are set and none of them 645 are wide, then NodeLen would be 3. If 3 IOAM-Trace-Type bits are 646 set and 2 of them are wide, then NodeLen would be 5. 648 An IOAM encapsulating node MUST set NodeLen. 650 A node receiving an IOAM Pre-allocated or Incremental Trace-Option 651 relies on the NodeLen value, or it can ignore the NodeLen value 652 and calculate the node length from the IOAM-Trace-Type bits (see 653 below). 655 Flags 4-bit field. Flags are allocated by IANA, as specified in 656 Section 8.3. This document allocates a single flag as follows: 658 Bit 0 "Overflow" (O-bit) (most significant bit). If there are 659 not enough octets left to record node data, the network element 660 MUST NOT add any fields and MUST set the overflow "O-bit" to 661 "1" in the IOAM-Trace-Option header. This is useful for 662 transit nodes to ignore further processing of the option. 664 RemainingLen: 7-bit unsigned integer. This field specifies the data 665 space in multiples of 4-octets remaining for recording the node 666 data, before the node data list is considered to have overflowed. 667 Given that the sender knows the path MTU (PMTU), the sender MAY 668 set the initial value of RemainingLen according to the number of 669 node data bytes allowed before exceeding the MTU. Subsequent 670 nodes can carry out a simple comparison between RemainingLen and 671 NodeLen, along with the length of the "Opaque State Snapshot" if 672 applicable, to determine whether or not data can be added by this 673 node. When node data is added, the node MUST decrease 674 RemainingLen by the amount of data added. In the pre-allocated 675 trace option, RemainingLen is used to derive the offset in data 676 space to record the node data element. Specifically, the 677 recording of the node data element would start from RemainingLen - 678 NodeLen - sizeof(opaque snapshot) in 4 octet units. If 679 RemainingLen in a pre-allocated trace option exceeds the length of 680 the option, as specified in the preceding header, then the node 681 MUST NOT add any fields. 683 IOAM-Trace-Type: A 24-bit identifier which specifies which data 684 types are used in this node data list. 686 The IOAM-Trace-Type value is a bit field. The following bits are 687 defined in this document, with details on each bit described in 688 the Section 5.4.2. The order of packing the data fields in each 689 node data element follows the bit order of the IOAM-Trace-Type 690 field, as follows: 692 Bit 0 (Most significant bit) When set, indicates presence of 693 Hop_Lim and node_id (short format) in the node data. 695 Bit 1 When set, indicates presence of ingress_if_id and 696 egress_if_id (short format) in the node data. 698 Bit 2 When set, indicates presence of timestamp seconds in the 699 node data. 701 Bit 3 When set, indicates presence of timestamp subseconds in 702 the node data. 704 Bit 4 When set, indicates presence of transit delay in the node 705 data. 707 Bit 5 When set, indicates presence of IOAM-Namespace specific 708 data (short format) in the node data. 710 Bit 6 When set, indicates presence of queue depth in the node 711 data. 713 Bit 7 When set, indicates presence of the Checksum Complement 714 node data. 716 Bit 8 When set, indicates presence of Hop_Lim and node_id in 717 wide format in the node data. 719 Bit 9 When set, indicates presence of ingress_if_id and 720 egress_if_id in wide format in the node data. 722 Bit 10 When set, indicates presence of IOAM-Namespace specific 723 data in wide format in the node data. 725 Bit 11 When set, indicates presence of buffer occupancy in the 726 node data. 728 Bit 12-21 Undefined. An IOAM encapsulating node MUST set the 729 value of each of these bits to 0. If an IOAM transit 730 node receives a packet with one or more of these bits set 731 to 1, it MUST either: 733 1. Add corresponding node data filled with the reserved 734 value 0xFFFFFFFF, after the node data fields for the 735 IOAM-Trace-Type bits defined above, such that the 736 total node data added by this node in units of 737 4-octets is equal to NodeLen, or 739 2. Not add any node data fields to the packet, even for 740 the IOAM-Trace-Type bits defined above. 742 Bit 22 When set, indicates presence of variable length Opaque 743 State Snapshot field. 745 Bit 23 Reserved: MUST be set to zero upon transmission and 746 ignored upon receipt. 748 Section 5.4.2 describes the IOAM-Data-Types and their formats. 749 Within an IOAM-Domain possible combinations of these bits making 750 the IOAM-Trace-Type can be restricted by configuration knobs. 752 Reserved: 8-bits. An IOAM encapsulating node MUST set the value to 753 zero upon transmission. IOAM transit nodes MUST ignore the 754 received value. 756 Node data List [n]: Variable-length field. This is a list of node 757 data elements where the content of each node data element is 758 determined by the IOAM-Trace-Type. The order of packing the data 759 fields in each node data element follows the bit order of the 760 IOAM-Trace-Type field. Each node MUST prepend its node data 761 element in front of the node data elements that it received, such 762 that the transmitted node data list begins with this node's data 763 element as the first populated element in the list. The last node 764 data element in this list is the node data of the first IOAM 765 capable node in the path. Populating the node data list in this 766 way ensures that the order of node data list is the same for 767 incremental and pre-allocated trace options. In the pre-allocated 768 trace option, the index contained in RemainingLen identifies the 769 offset for current active node data to be populated. 771 5.4.2. IOAM node data fields and associated formats 773 All the IOAM-Data-Fields MUST be 4-octet aligned. If a node which is 774 supposed to update an IOAM-Data-Field is not capable of populating 775 the value of a field set in the IOAM-Trace-Type, the field value MUST 776 be set to 0xFFFFFFFF for 4-octet fields or 0xFFFFFFFFFFFFFFFF for 777 8-octet fields, indicating that the value is not populated, except 778 when explicitly specified in the field description below. 780 Some IOAM-Data-Fields defined below, such as interface identifiers or 781 IOAM-Namespace specific data, are defined in both "short format" as 782 well as "wide format". Their use is not exclusive. A deployment 783 could choose to leverage both. For example, ingress_if_id_(short 784 format) could be an identifier for the physical interface, whereas 785 ingress_if_id_(wide format) could be an identifier for a logical sub- 786 interface of that physical interface. 788 Data fields and associated data types for each of the IOAM-Data- 789 Fields are specified in the following sections. 791 5.4.2.1. Hop_Lim and node_id short format 793 The "Hop_Lim and node_id short format" field is a 4-octet field that 794 is defined as follows: 796 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 797 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 798 | Hop_Lim | node_id | 799 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 801 Hop_Lim: 1-octet unsigned integer. It is set to the Hop Limit value 802 in the packet at the node that records this data. Hop Limit 803 information is used to identify the location of the node in the 804 communication path. This is copied from the lower layer, e.g., 805 TTL value in IPv4 header or hop limit field from IPv6 header of 806 the packet when the packet is ready for transmission. The 807 semantics of the Hop_Lim field depend on the lower layer protocol 808 that IOAM is encapsulated into, and therefore its specific 809 semantics are outside the scope of this memo. The value of this 810 field MUST be set to 0xff when the lower level does not have a 811 TTL/Hop limit equivalent field. 813 node_id: 3-octet unsigned integer. Node identifier field to 814 uniquely identify a node within the IOAM-Namespace and associated 815 IOAM-Domain. The procedure to allocate, manage and map the 816 node_ids is beyond the scope of this document. 818 5.4.2.2. ingress_if_id and egress_if_id 820 The "ingress_if_id and egress_if_id" field is a 4-octet field that is 821 defined as follows: 823 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 824 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 825 | ingress_if_id | egress_if_id | 826 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 828 ingress_if_id: 2-octet unsigned integer. Interface identifier to 829 record the ingress interface the packet was received on. 831 egress_if_id: 2-octet unsigned integer. Interface identifier to 832 record the egress interface the packet is forwarded out of. 834 Note that due to the fact that IOAM uses its own IOAM-Namespaces for 835 IOAM-Data-Fields, data fields like interface identifiers can be used 836 in a flexible way to represent system resources that are associated 837 with ingressing or egressing packets, i.e. ingress_if_id could 838 represent a physical interface, a virtual or logical interface, or 839 even a queue. 841 5.4.2.3. timestamp seconds 843 The "timestamp seconds" field is a 4-octet unsigned integer field. 844 Absolute timestamp in seconds that specifies the time at which the 845 packet was received by the node. This field has three possible 846 formats; based on either PTP [IEEE1588v2], NTP [RFC5905], or POSIX 847 [POSIX]. The three timestamp formats are specified in Section 6. In 848 all three cases, the Timestamp Seconds field contains the 32 most 849 significant bits of the timestamp format that is specified in 850 Section 6. If a node is not capable of populating this field, it 851 assigns the value 0xFFFFFFFF. Note that this is a legitimate value 852 that is valid for 1 second in approximately 136 years; the analyzer 853 has to correlate several packets or compare the timestamp value to 854 its own time-of-day in order to detect the error indication. 856 5.4.2.4. timestamp subseconds 858 The "timestamp subseconds" field is a 4-octet unsigned integer field. 859 Absolute timestamp in subseconds that specifies the time at which the 860 packet was received by the node. This field has three possible 861 formats; based on either PTP [IEEE1588v2], NTP [RFC5905], or POSIX 862 [POSIX]. The three timestamp formats are specified in Section 6. In 863 all three cases, the Timestamp Subseconds field contains the 32 least 864 significant bits of the timestamp format that is specified in 865 Section 6. If a node is not capable of populating this field, it 866 assigns the value 0xFFFFFFFF. Note that this is a legitimate value 867 in the NTP format, valid for approximately 233 picoseconds in every 868 second. If the NTP format is used the analyzer has to correlate 869 several packets in order to detect the error indication. 871 5.4.2.5. transit delay 873 The "transit delay" field is a 4-octet unsigned integer in the range 874 0 to 2^31-1. It is the time in nanoseconds the packet spent in the 875 transit node. This can serve as an indication of the queuing delay 876 at the node. If the transit delay exceeds 2^31-1 nanoseconds then 877 the top bit 'O' is set to indicate overflow and value set to 878 0x80000000. When this field is part of the data field but a node 879 populating the field is not able to fill it, the field position in 880 the field MUST be filled with value 0xFFFFFFFF to mean not populated. 882 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 883 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 884 |O| transit delay | 885 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 887 5.4.2.6. namespace specific data 889 The "namespace specific data" field is a 4-octet field which can be 890 used by the node to add IOAM-Namespace specific data. This 891 represents a "free-format" 4-octet bit field with its semantics 892 defined in the context of a specific IOAM-Namespace. 894 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 895 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 896 | namespace specific data | 897 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 899 5.4.2.7. queue depth 901 The "queue depth" field is a 4-octet unsigned integer field. This 902 field indicates the current length of the egress interface queue of 903 the interface from where the packet is forwarded out. The queue 904 depth is expressed as the current amount of memory buffers used by 905 the queue (a packet could consume one or more memory buffers, 906 depending on its size). 908 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 909 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 910 | queue depth | 911 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 913 5.4.2.8. Checksum Complement 915 The "Checksum Complement" field is a 4-octet node data which contains 916 a 4-octet Checksum Complement field. The Checksum Complement is 917 useful when IOAM is transported over encapsulations that make use of 918 a UDP transport, such as VXLAN-GPE or Geneve. Without the Checksum 919 Complement, nodes adding IOAM node data update the UDP Checksum field 920 following the recommendation of the encapsulation protocols. When 921 the Checksum Complement is present, an IOAM encapsulating node or 922 IOAM transit node adding node data MUST carry out one of the 923 following two alternatives in order to maintain the correctness of 924 the UDP Checksum value: 926 1. Recompute the UDP Checksum field. 928 2. Use the Checksum Complement to make a checksum-neutral update in 929 the UDP payload; the Checksum Complement is assigned a value that 930 complements the rest of the node data fields that were added by 931 the current node, causing the existing UDP Checksum field to 932 remain correct. 934 IOAM decapsulating nodes MUST recompute the UDP Checksum field, since 935 they do not know whether previous hops modified the UDP Checksum 936 field or the Checksum Complement field. 938 Checksum Complement fields are used in a similar manner in [RFC7820] 939 and [RFC7821]. 941 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 942 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 943 | Checksum Complement | 944 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 946 5.4.2.9. Hop_Lim and node_id wide 948 The "Hop_Lim and node_id wide" field is an 8-octet field defined as 949 follows: 951 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 952 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 953 | Hop_Lim | node_id ~ 954 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 955 ~ node_id (contd) | 956 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 958 Hop_Lim: 1-octet unsigned integer. It is set to the Hop Limit value 959 in the packet at the node that records this data. Hop Limit 960 information is used to identify the location of the node in the 961 communication path. This is copied from the lower layer for e.g. 962 TTL value in IPv4 header or hop limit field from IPv6 header of 963 the packet. The semantics of the Hop_Lim field depend on the 964 lower layer protocol that IOAM is encapsulated into, and therefore 965 its specific semantics are outside the scope of this memo. The 966 value of this field MUST be set to 0xff when the lower level does 967 not have a TTL/Hop limit equivalent field. 969 node_id: 7-octet unsigned integer. Node identifier field to 970 uniquely identify a node within the IOAM-Namespace and associated 971 IOAM-Domain. The procedure to allocate, manage and map the 972 node_ids is beyond the scope of this document. 974 5.4.2.10. ingress_if_id and egress_if_id wide 976 The "ingress_if_id and egress_if_id wide" field is an 8-octet field 977 which is defined as follows: 979 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 980 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 981 | ingress_if_id | 982 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 983 | egress_if_id | 984 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 986 ingress_if_id: 4-octet unsigned integer. Interface identifier to 987 record the ingress interface the packet was received on. 989 egress_if_id: 4-octet unsigned integer. Interface identifier to 990 record the egress interface the packet is forwarded out of. 992 5.4.2.11. namespace specific data wide 994 The "namespace specific data wide" field is an 8-octet field which 995 can be used by the node to add IOAM-Namespace specific data. This 996 represents a "free-format" 8-octet bit field with its semantics 997 defined in the context of a specific IOAM-Namespace. 999 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1000 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1001 | namespace specific data ~ 1002 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1003 ~ namespace specific data (contd) | 1004 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1006 5.4.2.12. buffer occupancy 1008 The "buffer occupancy" field is a 4-octet unsigned integer field. 1009 This field indicates the current status of the occupancy of the 1010 common buffer pool used by a set of queues. The units of this field 1011 are implementation specific. Hence, the units are interpreted within 1012 the context of an IOAM-Namespace and/or node-id if used. The authors 1013 acknowledge that in some operational cases there is a need for the 1014 units to be consistent across a packet path through the network, 1015 hence it is RECOMMENDED for implementations to use standard units 1016 such as Bytes. 1018 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1019 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1020 | buffer occupancy | 1021 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1023 5.4.2.13. Opaque State Snapshot 1025 The "Opaque State Snapshot" is a variable length field and follows 1026 the fixed length IOAM-Data-Fields defined above. It allows the 1027 network element to store an arbitrary state in the node data field, 1028 without a pre-defined schema. The schema is to be defined within the 1029 context of an IOAM-Namespace. The schema needs to be made known to 1030 the analyzer by some out-of-band mechanism. The specification of 1031 this mechanism is beyond the scope of this document. A 24-bit 1032 "Schema Id" field, interpreted within the context of an IOAM- 1033 Namespace, indicates which particular schema is used, and has to be 1034 configured on the network element by the operator. 1036 0 1 2 3 1037 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1038 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1039 | Length | Schema ID | 1040 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1041 | | 1042 | | 1043 | Opaque data | 1044 ~ ~ 1045 . . 1046 . . 1047 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1049 Length: 1-octet unsigned integer. It is the length in multiples of 1050 4-octets of the Opaque data field that follows Schema Id. 1052 Schema ID: 3-octet unsigned integer identifying the schema of Opaque 1053 data. 1055 Opaque data: Variable length field. This field is interpreted as 1056 specified by the schema identified by the Schema ID. 1058 When this field is part of the data field but a node populating the 1059 field has no opaque state data to report, the Length MUST be set to 0 1060 and the Schema ID MUST be set to 0xFFFFFF to mean no schema. 1062 5.4.3. Examples of IOAM node data 1064 An entry in the "node data list" array can have different formats, 1065 following the needs of the deployment. Some deployments might only 1066 be interested in recording the node identifiers, whereas others might 1067 be interested in recording node identifier and timestamp. The 1068 section provides example entries of the "node data list". 1070 0xD40000: IOAM-Trace-Type is 0xD40000 (0b110101000000000000000000) 1071 then the format of node data is: 1073 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1074 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1075 | Hop_Lim | node_id | 1076 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1077 | ingress_if_id | egress_if_id | 1078 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1079 | timestamp subseconds | 1080 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1081 | namespace specific data | 1082 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1084 0xC00000: IOAM-Trace-Type is 0xC00000 (0b110000000000000000000000) 1085 then the format is: 1087 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1088 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1089 | Hop_Lim | node_id | 1090 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1091 | ingress_if_id | egress_if_id | 1092 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1094 0x900000: IOAM-Trace-Type is 0x900000 (0b100100000000000000000000) 1095 then the format is: 1097 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1098 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1099 | Hop_Lim | node_id | 1100 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1101 | timestamp subseconds | 1102 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1104 0x840000: IOAM-Trace-Type is 0x840000 (0b100001000000000000000000) 1105 then the format is: 1107 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1108 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1109 | Hop_Lim | node_id | 1110 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1111 | namespace specific data | 1112 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1114 0x940000: IOAM-Trace-Type is 0x940000 (0b100101000000000000000000) 1115 then the format is: 1117 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1118 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1119 | Hop_Lim | node_id | 1120 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1121 | timestamp subseconds | 1122 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1123 | namespace specific data | 1124 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1126 0x308002: IOAM-Trace-Type is 0x308002 (0b001100001000000000000010) 1127 then the format is: 1129 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1130 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1131 | timestamp seconds | 1132 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1133 | timestamp subseconds | 1134 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1135 | Hop_Lim | node_id | 1136 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1137 | node_id(contd) | 1138 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1139 | Length | Schema Id | 1140 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1141 | | 1142 | | 1143 | Opaque data | 1144 ~ ~ 1145 . . 1146 . . 1147 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1149 5.5. IOAM Proof of Transit Option-Type 1151 IOAM Proof of Transit Option-Type is to support path or service 1152 function chain [RFC7665] verification use cases. Proof-of-transit 1153 leverages mechanisms like Shamir's Secret Sharing Schema (SSSS) 1154 [SSS]. For further information on Proof-of-transit, please refer to 1155 [I-D.ietf-sfc-proof-of-transit]. While details on how the IOAM data 1156 for the Proof-of-transit option is processed at IOAM encapsulating, 1157 decapsulating and transit nodes are outside the scope of the 1158 document, all of these approaches share the need to uniquely identify 1159 a packet as well as iteratively operate on a set of information that 1160 is handed from node to node. Correspondingly, two pieces of 1161 information are added as IOAM-Data-Fields to the packet: 1163 o Random: Unique identifier for the packet (e.g., 64-bits allow for 1164 the unique identification of 2^64 packets). 1166 o Cumulative: Information which is handed from node to node and 1167 updated by every node according to a verification algorithm. 1169 The IOAM Proof-of-Transit Option-Type consist of a fixed size "IOAM 1170 proof of transit option header" and "IOAM proof of transit option 1171 data fields": 1173 IOAM proof of transit option header: 1175 0 1 2 3 1176 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1177 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1178 | Namespace-ID |IOAM POT Type | IOAM POT flags| 1179 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1181 IOAM proof of transit Option-Type IOAM-Data-Fields MUST be 1182 4-octet aligned: 1184 0 1 2 3 1185 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1186 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1187 | POT Option data field determined by IOAM-POT-Type | 1188 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1190 Namespace-ID: 16-bit identifier of an IOAM-Namespace. The 1191 Namespace-ID value of 0x0000 is defined as the "Default-Namespace- 1192 ID" (see Section 5.3) and MUST be known to all the nodes 1193 implementing IOAM. For any other Namespace-ID value that does not 1194 match any Namespace-ID the node is configured to operate on, the 1195 node MUST NOT change the contents of the IOAM-Data-Fields. 1197 IOAM POT Type: 8-bit identifier of a particular POT variant that 1198 specifies the POT data that is included. This document defines 1199 POT Type 0: 1201 0: POT data is a 16 Octet field as described below. 1203 If a node receives an IOAM POT Type value that it does not 1204 understand, the node MUST NOT change the contents of the IOAM- 1205 Data-Fields. 1207 IOAM POT flags: 8-bit. Following flags are defined: 1209 Bit 0 "Profile-to-use" (P-bit) (most significant bit). For IOAM 1210 POT types that use a maximum of two profiles to drive 1211 computation, indicates which POT-profile is used. The two 1212 profiles are numbered 0, 1. 1214 Bit 1-7 Reserved: MUST be set to zero upon transmission and 1215 ignored upon receipt. 1217 POT Option data: Variable-length field. The type of which is 1218 determined by the IOAM-POT-Type. 1220 5.5.1. IOAM Proof of Transit Type 0 1222 IOAM proof of transit option of IOAM POT Type 0: 1224 0 1 2 3 1225 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1226 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1227 | Namespace-ID |IOAM POT Type=0|P|R R R R R R R| 1228 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+<-+ 1229 | Random | | 1230 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ P 1231 | Random(contd) | O 1232 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ T 1233 | Cumulative | | 1234 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 1235 | Cumulative (contd) | | 1236 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+<-+ 1238 Namespace-ID: 16-bit identifier of an IOAM-Namespace. The 1239 Namespace-ID value of 0x0000 is defined as the "Default-Namespace- 1240 ID" (see Section 5.3) and MUST be known to all the nodes 1241 implementing IOAM. For any other Namespace-ID value that does not 1242 match any Namespace-ID the node is configured to operate on, the 1243 node MUST NOT change the contents of the IOAM-Data-Fields. 1245 IOAM POT Type: 8-bit identifier of a particular POT variant that 1246 specifies the POT data that is included. This section defines the 1247 POT data when the IOAM POT Type is set to the value 0. 1249 P bit: 1-bit. "Profile-to-use" (P-bit) (most significant bit). 1250 Indicates which POT-profile is used to generate the Cumulative. 1251 Any node participating in POT will have a maximum of 2 profiles 1252 configured that drive the computation of cumulative. The two 1253 profiles are numbered 0, 1. This bit conveys whether profile 0 or 1254 profile 1 is used to compute the Cumulative. 1256 R (7 bits): 7-bit IOAM POT flags for future use. MUST be set to 1257 zero upon transmission and ignored upon receipt. 1259 Random: 64-bit Per packet Random number. 1261 Cumulative: 64-bit Cumulative that is updated at specific nodes by 1262 processing per packet Random number field and configured 1263 parameters. 1265 Note: Larger or smaller sizes of "Random" and "Cumulative" data are 1266 feasible and could be required for certain deployments (e.g. in case 1267 of space constraints in the encapsulation protocols used). Future 1268 documents could introduce different sizes of data for "proof of 1269 transit". 1271 5.6. IOAM Edge-to-Edge Option-Type 1273 The IOAM Edge-to-Edge Option-Type is to carry data that is added by 1274 the IOAM encapsulating node and interpreted by IOAM decapsulating 1275 node. The IOAM transit nodes MAY process the data but MUST NOT 1276 modify it. 1278 The IOAM Edge-to-Edge Option-Type consist of a fixed size "IOAM Edge- 1279 to-Edge Option-Type header" and "IOAM Edge-to-Edge Option-Type data 1280 fields": 1282 IOAM Edge-to-Edge Option-Type header: 1284 0 1 2 3 1285 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1286 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1287 | Namespace-ID | IOAM-E2E-Type | 1288 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1290 IOAM Edge-to-Edge Option-Type IOAM-Data-Fields MUST 1291 be 4-octet aligned: 1293 0 1 2 3 1294 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1295 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1296 | E2E Option data field determined by IOAM-E2E-Type | 1297 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1298 Namespace-ID: 16-bit identifier of an IOAM-Namespace. The 1299 Namespace-ID value of 0x0000 is defined as the "Default-Namespace- 1300 ID" (see Section 5.3) and MUST be known to all the nodes 1301 implementing IOAM. For any other Namespace-ID value that does not 1302 match any Namespace-ID the node is configured to operate on, then 1303 the node MUST NOT change the contents of the IOAM-Data-Fields. 1305 IOAM-E2E-Type: A 16-bit identifier which specifies which data types 1306 are used in the E2E option data. The IOAM-E2E-Type value is a bit 1307 field. The order of packing the E2E option data field elements 1308 follows the bit order of the IOAM-E2E-Type field, as follows: 1310 Bit 0 (Most significant bit) When set indicates presence of a 1311 64-bit sequence number added to a specific "packet group" 1312 which is used to detect packet loss, packet reordering, 1313 or packet duplication within the group. The "packet 1314 group" is deployment dependent and defined at the IOAM 1315 encapsulating node e.g. by n-tuple based classification 1316 of packets. 1318 Bit 1 When set indicates presence of a 32-bit sequence number 1319 added to a specific "packet group" which is used to 1320 detect packet loss, packet reordering, or packet 1321 duplication within that group. The "packet group" is 1322 deployment dependent and defined at the IOAM 1323 encapsulating node e.g. by n-tuple based classification 1324 of packets. 1326 Bit 2 When set indicates presence of timestamp seconds, 1327 representing the time at which the packet entered the 1328 IOAM domain. Within the IOAM encapsulating node, the 1329 time that the timestamp is retrieved can depend on the 1330 implementation. Some possibilities are: 1) the time at 1331 which the packet was received by the node, 2) the time at 1332 which the packet was transmitted by the node, 3) when a 1333 tunnel encapsulation is used, the point at which the 1334 packet is encapsulated into the tunnel. Each 1335 implementation has to document when the E2E timestamp 1336 that is going to be put in the packet is retrieved. This 1337 4-octet field has three possible formats; based on either 1338 PTP [IEEE1588v2], NTP [RFC5905], or POSIX [POSIX]. The 1339 three timestamp formats are specified in Section 6. In 1340 all three cases, the Timestamp Seconds field contains the 1341 32 most significant bits of the timestamp format that is 1342 specified in Section 6. If a node is not capable of 1343 populating this field, it assigns the value 0xFFFFFFFF. 1344 Note that this is a legitimate value that is valid for 1 1345 second in approximately 136 years; the analyzer has to 1346 correlate several packets or compare the timestamp value 1347 to its own time-of-day in order to detect the error 1348 indication. 1350 Bit 3 When set indicates presence of timestamp subseconds, 1351 representing the time at which the packet entered the 1352 IOAM domain. This 4-octet field has three possible 1353 formats; based on either PTP [IEEE1588v2], NTP [RFC5905], 1354 or POSIX [POSIX]. The three timestamp formats are 1355 specified in Section 6. In all three cases, the 1356 Timestamp Subseconds field contains the 32 least 1357 significant bits of the timestamp format that is 1358 specified in Section 6. If a node is not capable of 1359 populating this field, it assigns the value 0xFFFFFFFF. 1360 Note that this is a legitimate value in the NTP format, 1361 valid for approximately 233 picoseconds in every second. 1362 If the NTP format is used the analyzer has to correlate 1363 several packets in order to detect the error indication. 1365 Bit 4-15 Undefined. An IOAM encapsulating node MUST set the value 1366 of these bits to zero upon transmission and ignore upon 1367 receipt. 1369 E2E Option data: Variable-length field. The type of which is 1370 determined by the IOAM-E2E-Type. 1372 6. Timestamp Formats 1374 The IOAM-Data-Fields include a timestamp field which is represented 1375 in one of three possible timestamp formats. It is assumed that the 1376 management plane is responsible for determining which timestamp 1377 format is used. 1379 6.1. PTP Truncated Timestamp Format 1381 The Precision Time Protocol (PTP) [IEEE1588v2] uses an 80-bit 1382 timestamp format. The truncated timestamp format is a 64-bit field, 1383 which is the 64 least significant bits of the 80-bit PTP timestamp. 1384 The PTP truncated format is specified in Section 4.3 of [RFC8877], 1385 and the details are presented below for the sake of completeness. 1387 0 1 2 3 1388 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1389 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1390 | Seconds | 1391 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1392 | Nanoseconds | 1393 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1395 Figure 1: PTP [IEEE1588v2] Truncated Timestamp Format 1397 Timestamp field format: 1399 Seconds: specifies the integer portion of the number of seconds 1400 since the epoch. 1402 + Size: 32 bits. 1404 + Units: seconds. 1406 Nanoseconds: specifies the fractional portion of the number of 1407 seconds since the epoch. 1409 + Size: 32 bits. 1411 + Units: nanoseconds. The value of this field is in the range 0 1412 to (10^9)-1. 1414 Epoch: 1416 The PTP [IEEE1588v2] epoch is 1 January 1970 00:00:00 TAI, which 1417 is 31 December 1969 23:59:51.999918 UTC. 1419 Resolution: 1421 The resolution is 1 nanosecond. 1423 Wraparound: 1425 This time format wraps around every 2^32 seconds, which is roughly 1426 136 years. The next wraparound will occur in the year 2106. 1428 Synchronization Aspects: 1430 It is assumed that nodes that run this protocol are synchronized 1431 among themselves. Nodes MAY be synchronized to a global reference 1432 time. Note that if PTP [IEEE1588v2] is used for synchronization, 1433 the timestamp MAY be derived from the PTP-synchronized clock, 1434 allowing the timestamp to be measured with respect to the clock of 1435 an PTP Grandmaster clock. 1437 The PTP truncated timestamp format is not affected by leap 1438 seconds. 1440 6.2. NTP 64-bit Timestamp Format 1442 The Network Time Protocol (NTP) [RFC5905] timestamp format is 64 bits 1443 long. This format is specified in Section 4.2.1 of [RFC8877], and 1444 the details are presented below for the sake of completeness. 1446 0 1 2 3 1447 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1448 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1449 | Seconds | 1450 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1451 | Fraction | 1452 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1454 Figure 2: NTP [RFC5905] 64-bit Timestamp Format 1456 Timestamp field format: 1458 Seconds: specifies the integer portion of the number of seconds 1459 since the epoch. 1461 + Size: 32 bits. 1463 + Units: seconds. 1465 Fraction: specifies the fractional portion of the number of 1466 seconds since the epoch. 1468 + Size: 32 bits. 1470 + Units: the unit is 2^(-32) seconds, which is roughly equal to 1471 233 picoseconds. 1473 Epoch: 1475 The epoch is 1 January 1900 at 00:00 UTC. 1477 Resolution: 1479 The resolution is 2^(-32) seconds. 1481 Wraparound: 1483 This time format wraps around every 2^32 seconds, which is roughly 1484 136 years. The next wraparound will occur in the year 2036. 1486 Synchronization Aspects: 1488 Nodes that use this timestamp format will typically be 1489 synchronized to UTC using NTP [RFC5905]. Thus, the timestamp MAY 1490 be derived from the NTP-synchronized clock, allowing the timestamp 1491 to be measured with respect to the clock of an NTP server. 1493 The NTP timestamp format is affected by leap seconds; it 1494 represents the number of seconds since the epoch minus the number 1495 of leap seconds that have occurred since the epoch. The value of 1496 a timestamp during or slightly after a leap second could be 1497 temporarily inaccurate. 1499 6.3. POSIX-based Timestamp Format 1501 This timestamp format is based on the POSIX time format [POSIX]. The 1502 detailed specification of the timestamp format used in this document 1503 is presented below. 1505 0 1 2 3 1506 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1507 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1508 | Seconds | 1509 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1510 | Microseconds | 1511 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1513 Figure 3: POSIX-based Timestamp Format 1515 Timestamp field format: 1517 Seconds: specifies the integer portion of the number of seconds 1518 since the epoch. 1520 + Size: 32 bits. 1522 + Units: seconds. 1524 Microseconds: specifies the fractional portion of the number of 1525 seconds since the epoch. 1527 + Size: 32 bits. 1529 + Units: the unit is microseconds. The value of this field is in 1530 the range 0 to (10^6)-1. 1532 Epoch: 1534 The epoch is 1 January 1970 00:00:00 TAI, which is 31 December 1535 1969 23:59:51.999918 UTC. 1537 Resolution: 1539 The resolution is 1 microsecond. 1541 Wraparound: 1543 This time format wraps around every 2^32 seconds, which is roughly 1544 136 years. The next wraparound will occur in the year 2106. 1546 Synchronization Aspects: 1548 It is assumed that nodes that use this timestamp format run the 1549 Linux operating system, and hence use the POSIX time. In some 1550 cases nodes MAY be synchronized to UTC using a synchronization 1551 mechanism that is outside the scope of this document, such as NTP 1552 [RFC5905]. Thus, the timestamp MAY be derived from the NTP- 1553 synchronized clock, allowing the timestamp to be measured with 1554 respect to the clock of an NTP server. 1556 The POSIX-based timestamp format is affected by leap seconds; it 1557 represents the number of seconds since the epoch minus the number 1558 of leap seconds that have occurred since the epoch. The value of 1559 a timestamp during or slightly after a leap second could be 1560 temporarily inaccurate. 1562 7. IOAM Data Export 1564 IOAM nodes collect information for packets traversing a domain that 1565 supports IOAM. IOAM decapsulating nodes as well as IOAM transit 1566 nodes can choose to retrieve IOAM information from the packet, 1567 process the information further and export the information using 1568 e.g., IPFIX. The mechanisms and associated data formats for 1569 exporting IOAM data is outside the scope of this document. 1571 Raw data export of IOAM data using IPFIX is discussed in 1572 [I-D.spiegel-ippm-ioam-rawexport]. 1574 8. IANA Considerations 1576 This document requests the following IANA Actions. 1578 IANA is requested to define a registry group named "In-Situ OAM 1579 (IOAM) Protocol Parameters". 1581 This group will include the following registries: 1583 IOAM Option-Type 1585 IOAM Trace-Type 1587 IOAM Trace-Flags 1589 IOAM POT-Type 1591 IOAM POT-Flags 1593 IOAM E2E-Type 1595 IOAM Namespace-ID 1597 New registries in this group can be created via RFC Required process 1598 as per [RFC8126]. 1600 The subsequent sub-sections detail the registries herein contained. 1602 8.1. IOAM Option-Type Registry 1604 This registry defines 128 code points for the IOAM Option-Type field 1605 for identifying IOAM Option-Types as explained in Section 5. The 1606 following code points are defined in this draft: 1608 0 IOAM Pre-allocated Trace Option-Type 1610 1 IOAM Incremental Trace Option-Type 1612 2 IOAM POT Option-Type 1614 3 IOAM E2E Option-Type 1616 4 - 127 are available for assignment via RFC Required process as per 1617 [RFC8126]. 1619 8.2. IOAM Trace-Type Registry 1621 This registry defines code point for each bit in the 24-bit IOAM- 1622 Trace-Type field for Pre-allocated trace option and Incremental trace 1623 option defined in Section 5.4. The meaning of Bits 0 - 11 for trace 1624 type are defined in this document in Paragraph 5 of Section 5.4.1: 1626 Bit 0 hop_Lim and node_id in short format 1628 Bit 1 ingress_if_id and egress_if_id in short format 1630 Bit 2 timestamp seconds 1632 Bit 3 timestamp subseconds 1634 Bit 4 transit delay 1636 Bit 5 namespace specific data in short format 1638 Bit 6 queue depth 1640 Bit 7 checksum complement 1642 Bit 8 hop_Lim and node_id in wide format 1644 Bit 9 ingress_if_id and egress_if_id in wide format 1646 Bit 10 namespace specific data in wide format 1648 Bit 11 buffer occupancy 1650 Bit 22 variable length Opaque State Snapshot 1652 Bit 23 reserved 1654 The meaning for Bits 12 - 21 are available for assignment via RFC 1655 Required process as per [RFC8126]. 1657 8.3. IOAM Trace-Flags Registry 1659 This registry defines code points for each bit in the 4 bit flags for 1660 the Pre-allocated trace option and for the Incremental trace option 1661 defined in Section 5.4. The meaning of Bit 0 (the most significant 1662 bit) for trace flags is defined in this document in Paragraph 3 of 1663 Section 5.4.1: 1665 Bit 0 "Overflow" (O-bit) 1666 Bit 1 - 3 are available for assignment via RFC Required process as 1667 per [RFC8126]. 1669 8.4. IOAM POT-Type Registry 1671 This registry defines 256 code points to define IOAM POT Type for 1672 IOAM proof of transit option Section 5.5. The code point value 0 is 1673 defined in this document: 1675 0: 16 Octet POT data 1677 1 - 255 are available for assignment via RFC Required process as per 1678 [RFC8126]. 1680 8.5. IOAM POT-Flags Registry 1682 This registry defines code points for each bit in the 8 bit flags for 1683 IOAM POT option defined in Section 5.5. The meaning of Bit 0 for 1684 IOAM POT flags is defined in this document in Section 5.5: 1686 Bit 0 "Profile-to-use" (P-bit) 1688 The meaning for Bits 1 - 7 are available for assignment via RFC 1689 Required process as per [RFC8126]. 1691 8.6. IOAM E2E-Type Registry 1693 This registry defines code points for each bit in the 16 bit IOAM- 1694 E2E-Type field for IOAM E2E option Section 5.6. The meaning of Bit 0 1695 - 3 are defined in this document: 1697 Bit 0 64-bit sequence number 1699 Bit 1 32-bit sequence number 1701 Bit 2 timestamp seconds 1703 Bit 3 timestamp subseconds 1705 The meaning of Bits 4 - 15 are available for assignment via RFC 1706 Required process as per [RFC8126]. 1708 8.7. IOAM Namespace-ID Registry 1710 IANA is requested to set up an "IOAM Namespace-ID Registry", 1711 containing 16-bit values. The meaning of Bit 0 is defined in this 1712 document. IANA is requested to reserve the values 0x0001 to 0x7FFF 1713 for private use (managed by operators), as specified in Section 5.3 1714 of the current document. Registry entries for the values 0x8000 to 1715 0xFFFF are to be assigned via the "Expert Review" policy defined in 1716 [RFC8126]. Upon a new allocation request, the responsible AD will 1717 appoint a designated expert, who will review the allocation request. 1718 The expert will post the request on the mailing list of the IPPM 1719 working group in the IETF (ippm@ietf.org), and possibly on other 1720 relevant mailing lists, to allow for community feedback. Based on 1721 the review, the expert will either approve or deny the request. The 1722 intention is that any allocation will be accompanied by a published 1723 RFC. But in order to allow for the allocation of values prior to the 1724 RFC being approved for publication, the designated expert can approve 1725 allocations once it seems clear that an RFC will be published. 1727 0: default namespace (known to all IOAM nodes) 1729 0x0001 - 0x7FFF: reserved for private use 1731 0x8000 - 0xFFFF: unassigned 1733 9. Management and Deployment Considerations 1735 This document defines the structure and use of IOAM data fields. 1736 This document does not define the encapsulation of IOAM data fields 1737 into different protocols. Management and deployment aspects for IOAM 1738 have to be considered within the context of the protocol IOAM data 1739 fields are encapsulated into and as such, are out of scope for this 1740 document. For a discussion of IOAM deployment, please also refer to 1741 [I-D.brockners-opsawg-ioam-deployment], which outlines a framework 1742 for IOAM deployment and provides best current practices. 1744 10. Security Considerations 1746 As discussed in [RFC7276], a successful attack on an OAM protocol in 1747 general, and specifically on IOAM, can prevent the detection of 1748 failures or anomalies, or create a false illusion of nonexistent 1749 ones. In particular, these threats are applicable by compromising 1750 the integrity of IOAM data, either by maliciously modifying IOAM 1751 options in transit, or by injecting packets with maliciously 1752 generated IOAM options 1754 The Proof of Transit Option-Type (Section Section 5.5) is used for 1755 verifying the path of data packets. The security considerations of 1756 POT are further discussed in [I-D.ietf-sfc-proof-of-transit]. 1758 From a confidentiality perspective, although IOAM options do not 1759 contain user data, they can be used for network reconnaissance, 1760 allowing attackers to collect information about network paths, 1761 performance, queue states, buffer occupancy and other information. 1763 Moreover, if IOAM data leaks from the IOAM domain it could enable 1764 reconnaissance beyond the scope of the IOAM domain. Note that in 1765 case IOAM is used in "Direct Exporting" mode 1766 [I-D.ioamteam-ippm-ioam-direct-export], the IOAM related trace 1767 information would not be available in the customer data packets, but 1768 would trigger export of packet related IOAM information at every 1769 node, thus restricting the potential threat to the management plane 1770 and mitigating the leakage threat. IOAM data exporting and the way 1771 it is secured is outside the scope of this document. 1773 IOAM can be used as a means for implementing Denial of Service (DoS) 1774 attacks, or for amplifying them. For example, a malicious attacker 1775 can add an IOAM header to packets in order to consume the resources 1776 of network devices that take part in IOAM or entities that receive, 1777 collect or analyze the IOAM data. Another example is a packet length 1778 attack, in which an attacker pushes headers associated with IOAM 1779 Option-Types into data packets, causing these packets to be increased 1780 beyond the MTU size, resulting in fragmentation or in packet drops. 1782 Since IOAM options can include timestamps, if network devices use 1783 synchronization protocols then any attack on the time protocol 1784 [RFC7384] can compromise the integrity of the timestamp-related data 1785 fields. 1787 At the management plane, attacks can be set up by misconfiguring or 1788 by maliciously configuring IOAM-enabled nodes in a way that enables 1789 other attacks. Thus, IOAM configuration has to be secured in a way 1790 that authenticates authorized users and verifies the integrity of 1791 configuration procedures. 1793 Solutions to ensure the integrity of IOAM data fields are outside the 1794 scope of this document. [I-D.brockners-ippm-ioam-data-integrity] 1795 discusses several methods to ensure the integrity of IOAM data fields 1796 for those deployments that have a need to protect the integrity of 1797 IOAM data fields. 1799 The current document does not define a specific IOAM encapsulation. 1800 It has to be noted that some IOAM encapsulation types can introduce 1801 specific security considerations. A specification that defines an 1802 IOAM encapsulation is expected to address the respective 1803 encapsulation-specific security considerations. 1805 Notably, in most cases IOAM is expected to be deployed in specific 1806 network domains, thus confining the potential attack vectors to 1807 within the network domain. A limited administrative domain provides 1808 the operator with the means to select, monitor, and control the 1809 access of all the network devices, making these devices trusted by 1810 the operator. Indeed, in order to limit the scope of threats 1811 mentioned above to within the current network domain the network 1812 operator is expected to enforce policies that prevent IOAM traffic 1813 from leaking outside of the IOAM domain, and prevent IOAM data from 1814 outside the domain to be processed and used within the domain. 1816 The security considerations of a system that deploys IOAM, much like 1817 any system, has to be reviewed on a per-deployment-scenario basis, 1818 based on a systems-specific threat analysis, which can lead to 1819 specific security solutions that are beyond the scope of the current 1820 document. Specifically, in an IOAM deployment that is not confined 1821 to a single LAN, but spans multiple inter-connected sites (for 1822 example, using an overlay network), the inter-site links can be 1823 secured (e.g., by IPsec) in order to avoid external threats. 1825 IOAM deployment considerations, including approaches to mitigate the 1826 above discussed threads and potential attacks are outside the scope 1827 of this document. IOAM deployment considerations are discussed in 1828 [I-D.brockners-opsawg-ioam-deployment]. 1830 11. Acknowledgements 1832 The authors would like to thank Eric Vyncke, Nalini Elkins, Srihari 1833 Raghavan, Ranganathan T S, Karthik Babu Harichandra Babu, Akshaya 1834 Nadahalli, LJ Wobker, Erik Nordmark, Vengada Prasad Govindan, Andrew 1835 Yourtchenko, Aviv Kfir, Tianran Zhou, Zhenbin (Robin) and Greg Mirsky 1836 for the comments and advice. 1838 This document leverages and builds on top of several concepts 1839 described in [I-D.kitamura-ipv6-record-route]. The authors would 1840 like to acknowledge the work done by the author Hiroshi Kitamura and 1841 people involved in writing it. 1843 The authors would like to gracefully acknowledge useful review and 1844 insightful comments received from Joe Clarke, Al Morton, Tom Herbert, 1845 Haoyu Song, Mickey Spiegel and Barak Gafni. 1847 12. References 1849 12.1. Normative References 1851 [IEEE1588v2] 1852 Institute of Electrical and Electronics Engineers, "IEEE 1853 Std 1588-2008 - IEEE Standard for a Precision Clock 1854 Synchronization Protocol for Networked Measurement and 1855 Control Systems", IEEE Std 1588-2008, 2008, 1856 . 1859 [POSIX] Institute of Electrical and Electronics Engineers, "IEEE 1860 Std 1003.1-2008 (Revision of IEEE Std 1003.1-2004) - IEEE 1861 Standard for Information Technology - Portable Operating 1862 System Interface (POSIX(R))", IEEE Std 1003.1-2008, 2008, 1863 . 1866 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1867 Requirement Levels", BCP 14, RFC 2119, 1868 DOI 10.17487/RFC2119, March 1997, 1869 . 1871 [RFC5905] Mills, D., Martin, J., Ed., Burbank, J., and W. Kasch, 1872 "Network Time Protocol Version 4: Protocol and Algorithms 1873 Specification", RFC 5905, DOI 10.17487/RFC5905, June 2010, 1874 . 1876 [RFC8126] Cotton, M., Leiba, B., and T. Narten, "Guidelines for 1877 Writing an IANA Considerations Section in RFCs", BCP 26, 1878 RFC 8126, DOI 10.17487/RFC8126, June 2017, 1879 . 1881 12.2. Informative References 1883 [I-D.brockners-ippm-ioam-data-integrity] 1884 Brockners, F., Bhandari, S., and T. Mizrahi, "Integrity of 1885 In-situ OAM Data Fields", draft-brockners-ippm-ioam-data- 1886 integrity-00 (work in progress), January 2021. 1888 [I-D.brockners-opsawg-ioam-deployment] 1889 Brockners, F., Bhandari, S., and d. 1890 daniel.bernier@bell.ca, "In-situ OAM Deployment", draft- 1891 brockners-opsawg-ioam-deployment-02 (work in progress), 1892 September 2020. 1894 [I-D.ietf-nvo3-geneve] 1895 Gross, J., Ganga, I., and T. Sridhar, "Geneve: Generic 1896 Network Virtualization Encapsulation", draft-ietf- 1897 nvo3-geneve-16 (work in progress), March 2020. 1899 [I-D.ietf-nvo3-vxlan-gpe] 1900 Maino, F., Kreeger, L., and U. Elzur, "Generic Protocol 1901 Extension for VXLAN (VXLAN-GPE)", draft-ietf-nvo3-vxlan- 1902 gpe-10 (work in progress), July 2020. 1904 [I-D.ietf-sfc-proof-of-transit] 1905 Brockners, F., Bhandari, S., Mizrahi, T., Dara, S., and S. 1906 Youell, "Proof of Transit", draft-ietf-sfc-proof-of- 1907 transit-08 (work in progress), November 2020. 1909 [I-D.ioamteam-ippm-ioam-direct-export] 1910 Song, H., Gafni, B., Zhou, T., Li, Z., Brockners, F., 1911 Bhandari, S., Sivakolundu, R., and T. Mizrahi, "In-situ 1912 OAM Direct Exporting", draft-ioamteam-ippm-ioam-direct- 1913 export-00 (work in progress), October 2019. 1915 [I-D.kitamura-ipv6-record-route] 1916 Kitamura, H., "Record Route for IPv6 (PR6) Hop-by-Hop 1917 Option Extension", draft-kitamura-ipv6-record-route-00 1918 (work in progress), November 2000. 1920 [I-D.spiegel-ippm-ioam-rawexport] 1921 Spiegel, M., Brockners, F., Bhandari, S., and R. 1922 Sivakolundu, "In-situ OAM raw data export with IPFIX", 1923 draft-spiegel-ippm-ioam-rawexport-04 (work in progress), 1924 November 2020. 1926 [RFC7276] Mizrahi, T., Sprecher, N., Bellagamba, E., and Y. 1927 Weingarten, "An Overview of Operations, Administration, 1928 and Maintenance (OAM) Tools", RFC 7276, 1929 DOI 10.17487/RFC7276, June 2014, 1930 . 1932 [RFC7384] Mizrahi, T., "Security Requirements of Time Protocols in 1933 Packet Switched Networks", RFC 7384, DOI 10.17487/RFC7384, 1934 October 2014, . 1936 [RFC7665] Halpern, J., Ed. and C. Pignataro, Ed., "Service Function 1937 Chaining (SFC) Architecture", RFC 7665, 1938 DOI 10.17487/RFC7665, October 2015, 1939 . 1941 [RFC7799] Morton, A., "Active and Passive Metrics and Methods (with 1942 Hybrid Types In-Between)", RFC 7799, DOI 10.17487/RFC7799, 1943 May 2016, . 1945 [RFC7820] Mizrahi, T., "UDP Checksum Complement in the One-Way 1946 Active Measurement Protocol (OWAMP) and Two-Way Active 1947 Measurement Protocol (TWAMP)", RFC 7820, 1948 DOI 10.17487/RFC7820, March 2016, 1949 . 1951 [RFC7821] Mizrahi, T., "UDP Checksum Complement in the Network Time 1952 Protocol (NTP)", RFC 7821, DOI 10.17487/RFC7821, March 1953 2016, . 1955 [RFC8300] Quinn, P., Ed., Elzur, U., Ed., and C. Pignataro, Ed., 1956 "Network Service Header (NSH)", RFC 8300, 1957 DOI 10.17487/RFC8300, January 2018, 1958 . 1960 [RFC8877] Mizrahi, T., Fabini, J., and A. Morton, "Guidelines for 1961 Defining Packet Timestamps", RFC 8877, 1962 DOI 10.17487/RFC8877, September 2020, 1963 . 1965 [SSS] "Shamir's Secret Sharing", 1966 . 1968 Contributors' Addresses 1970 Carlos Pignataro 1971 Cisco Systems, Inc. 1972 7200-11 Kit Creek Road 1973 Research Triangle Park, NC 27709 1974 United States 1976 Email: cpignata@cisco.com 1978 Mickey Spiegel 1979 Barefoot Networks, an Intel company 1980 4750 Patrick Henry Drive 1981 Santa Clara, CA 95054 1982 US 1984 Email: mickey.spiegel@intel.com 1986 Barak Gafni 1987 Nvidia 1988 350 Oakmead Parkway, Suite 100 1989 Sunnyvale, CA 94085 1990 U.S.A. 1992 Email: gbarak@nvidia.com 1994 Jennifer Lemon 1995 Broadcom 1996 270 Innovation Drive 1997 San Jose, CA 95134 1998 US 2000 Email: jennifer.lemon@broadcom.com 2002 Hannes Gredler 2003 RtBrick Inc. 2005 Email: hannes@rtbrick.com 2007 John Leddy 2008 United States 2010 Email: john@leddy.net 2012 Stephen Youell 2013 JP Morgan Chase 2014 25 Bank Street 2015 London E14 5JP 2016 United Kingdom 2018 Email: stephen.youell@jpmorgan.com 2020 David Mozes 2022 Email: mosesster@gmail.com 2024 Petr Lapukhov 2025 Facebook 2026 1 Hacker Way 2027 Menlo Park, CA 94025 2028 US 2030 Email: petr@fb.com 2032 Remy Chang 2033 Barefoot Networks 2034 4750 Patrick Henry Drive 2035 Santa Clara, CA 95054 2036 US 2037 Email: remy@barefootnetworks.com 2039 Daniel Bernier 2040 Bell Canada 2041 Canada 2043 Email: daniel.bernier@bell.ca 2045 Authors' Addresses 2047 Frank Brockners (editor) 2048 Cisco Systems, Inc. 2049 Hansaallee 249, 3rd Floor 2050 DUESSELDORF, NORDRHEIN-WESTFALEN 40549 2051 Germany 2053 Email: fbrockne@cisco.com 2055 Shwetha Bhandari (editor) 2056 Thoughtspot 2057 3rd Floor, Indiqube Orion, 24th Main Rd, Garden Layout, HSR Layout 2058 Bangalore, KARNATAKA 560 102 2059 India 2061 Email: shwetha.bhandari@thoughtspot.com 2063 Tal Mizrahi (editor) 2064 Huawei 2065 8-2 Matam 2066 Haifa 3190501 2067 Israel 2069 Email: tal.mizrahi.phd@gmail.com