idnits 2.17.1 draft-ietf-ippm-ioam-data-11.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (November 22, 2020) is 1251 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) -- Looks like a reference, but probably isn't: '0' on line 606 -- Looks like a reference, but probably isn't: '1' on line 610 -- Possible downref: Non-RFC (?) normative reference: ref. 'IEEE1588v2' -- Possible downref: Non-RFC (?) normative reference: ref. 'POSIX' == Outdated reference: A later version (-03) exists of draft-brockners-opsawg-ioam-deployment-02 == Outdated reference: A later version (-13) exists of draft-ietf-nvo3-vxlan-gpe-10 == Outdated reference: A later version (-07) exists of draft-spiegel-ippm-ioam-rawexport-04 Summary: 0 errors (**), 0 flaws (~~), 4 warnings (==), 5 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 ippm F. Brockners, Ed. 3 Internet-Draft S. Bhandari, Ed. 4 Intended status: Standards Track Cisco 5 Expires: May 26, 2021 T. Mizrahi, Ed. 6 Huawei 7 November 22, 2020 9 Data Fields for In-situ OAM 10 draft-ietf-ippm-ioam-data-11 12 Abstract 14 In-situ Operations, Administration, and Maintenance (IOAM) records 15 operational and telemetry information in the packet while the packet 16 traverses a path between two points in the network. This document 17 discusses the data fields and associated data types for in-situ OAM. 18 In-situ OAM data fields can be encapsulated into a variety of 19 protocols such as NSH, Segment Routing, Geneve, IPv6 (via extension 20 header), or IPv4. In-situ OAM can be used to complement OAM 21 mechanisms based on e.g. ICMP or other types of probe packets. 23 Status of This Memo 25 This Internet-Draft is submitted in full conformance with the 26 provisions of BCP 78 and BCP 79. 28 Internet-Drafts are working documents of the Internet Engineering 29 Task Force (IETF). Note that other groups may also distribute 30 working documents as Internet-Drafts. The list of current Internet- 31 Drafts is at https://datatracker.ietf.org/drafts/current/. 33 Internet-Drafts are draft documents valid for a maximum of six months 34 and may be updated, replaced, or obsoleted by other documents at any 35 time. It is inappropriate to use Internet-Drafts as reference 36 material or to cite them other than as "work in progress." 38 This Internet-Draft will expire on May 26, 2021. 40 Copyright Notice 42 Copyright (c) 2020 IETF Trust and the persons identified as the 43 document authors. All rights reserved. 45 This document is subject to BCP 78 and the IETF Trust's Legal 46 Provisions Relating to IETF Documents 47 (https://trustee.ietf.org/license-info) in effect on the date of 48 publication of this document. Please review these documents 49 carefully, as they describe your rights and restrictions with respect 50 to this document. Code Components extracted from this document must 51 include Simplified BSD License text as described in Section 4.e of 52 the Trust Legal Provisions and are provided without warranty as 53 described in the Simplified BSD License. 55 Table of Contents 57 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 58 2. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 3 59 3. Conventions . . . . . . . . . . . . . . . . . . . . . . . . . 4 60 4. Scope, Applicability, and Assumptions . . . . . . . . . . . . 5 61 5. IOAM Data-Fields, Types, Nodes . . . . . . . . . . . . . . . 6 62 5.1. IOAM Data-Fields and Option-Types . . . . . . . . . . . . 6 63 5.2. IOAM-Domains and types of IOAM Nodes . . . . . . . . . . 7 64 5.3. IOAM-Namespaces . . . . . . . . . . . . . . . . . . . . . 8 65 5.4. IOAM Trace Option-Types . . . . . . . . . . . . . . . . . 10 66 5.4.1. Pre-allocated and Incremental Trace Option-Types . . 13 67 5.4.2. IOAM node data fields and associated formats . . . . 17 68 5.4.2.1. Hop_Lim and node_id short format . . . . . . . . 18 69 5.4.2.2. ingress_if_id and egress_if_id . . . . . . . . . 18 70 5.4.2.3. timestamp seconds . . . . . . . . . . . . . . . . 19 71 5.4.2.4. timestamp subseconds . . . . . . . . . . . . . . 19 72 5.4.2.5. transit delay . . . . . . . . . . . . . . . . . . 19 73 5.4.2.6. namespace specific data . . . . . . . . . . . . . 20 74 5.4.2.7. queue depth . . . . . . . . . . . . . . . . . . . 20 75 5.4.2.8. Checksum Complement . . . . . . . . . . . . . . . 20 76 5.4.2.9. Hop_Lim and node_id wide . . . . . . . . . . . . 21 77 5.4.2.10. ingress_if_id and egress_if_id wide . . . . . . . 22 78 5.4.2.11. namespace specific data wide . . . . . . . . . . 22 79 5.4.2.12. buffer occupancy . . . . . . . . . . . . . . . . 22 80 5.4.2.13. Opaque State Snapshot . . . . . . . . . . . . . . 23 81 5.4.3. Examples of IOAM node data . . . . . . . . . . . . . 23 82 5.5. IOAM Proof of Transit Option-Type . . . . . . . . . . . . 25 83 5.5.1. IOAM Proof of Transit Type 0 . . . . . . . . . . . . 27 84 5.6. IOAM Edge-to-Edge Option-Type . . . . . . . . . . . . . . 28 85 6. Timestamp Formats . . . . . . . . . . . . . . . . . . . . . . 30 86 6.1. PTP Truncated Timestamp Format . . . . . . . . . . . . . 30 87 6.2. NTP 64-bit Timestamp Format . . . . . . . . . . . . . . . 32 88 6.3. POSIX-based Timestamp Format . . . . . . . . . . . . . . 33 89 7. IOAM Data Export . . . . . . . . . . . . . . . . . . . . . . 34 90 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 35 91 8.1. IOAM Option-Type Registry . . . . . . . . . . . . . . . . 35 92 8.2. IOAM Trace-Type Registry . . . . . . . . . . . . . . . . 36 93 8.3. IOAM Trace-Flags Registry . . . . . . . . . . . . . . . . 36 94 8.4. IOAM POT-Type Registry . . . . . . . . . . . . . . . . . 37 95 8.5. IOAM POT-Flags Registry . . . . . . . . . . . . . . . . . 37 96 8.6. IOAM E2E-Type Registry . . . . . . . . . . . . . . . . . 37 97 8.7. IOAM Namespace-ID Registry . . . . . . . . . . . . . . . 37 98 9. Management and Deployment Considerations . . . . . . . . . . 38 99 10. Security Considerations . . . . . . . . . . . . . . . . . . . 38 100 11. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 40 101 12. References . . . . . . . . . . . . . . . . . . . . . . . . . 40 102 12.1. Normative References . . . . . . . . . . . . . . . . . . 40 103 12.2. Informative References . . . . . . . . . . . . . . . . . 41 104 Contributors' Addresses . . . . . . . . . . . . . . . . . . . . . 43 105 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 44 107 1. Introduction 109 This document defines data fields for "in-situ" Operations, 110 Administration, and Maintenance (IOAM). In-situ OAM records OAM 111 information within the packet while the packet traverses a particular 112 network domain. The term "in-situ" refers to the fact that the OAM 113 data is added to the data packets rather than being sent within 114 packets specifically dedicated to OAM. IOAM is to complement 115 mechanisms such as Ping or Traceroute. In terms of "active" or 116 "passive" OAM, "in-situ" OAM can be considered a hybrid OAM type. 117 "In-situ" mechanisms do not require extra packets to be sent. IOAM 118 adds information to the already available data packets and therefore 119 cannot be considered passive. In terms of the classification given 120 in [RFC7799] IOAM could be portrayed as Hybrid Type 1. IOAM 121 mechanisms can be leveraged where mechanisms using e.g. ICMP do not 122 apply or do not offer the desired results, such as proving that a 123 certain traffic flow takes a pre-defined path, SLA verification for 124 the live data traffic, detailed statistics on traffic distribution 125 paths in networks that distribute traffic across multiple paths, or 126 scenarios in which probe traffic is potentially handled differently 127 from regular data traffic by the network devices. 129 IOAM use cases and mechanisms have expanded as this document matured, 130 resulting in additional flags and options that could trigger creation 131 of additional packets dedicated to OAM. The term IOAM continues to 132 be used for such mechanisms, in addition to the "in-situ" mechanisms 133 that motivated this terminology. 135 2. Contributors 137 This document was the collective effort of several authors. The text 138 and content were contributed by the editors and the co-authors listed 139 below. The contact information of the co-authors appears at the end 140 of this document. 142 o Carlos Pignataro 144 o Mickey Spiegel 145 o Barak Gafni 147 o Jennifer Lemon 149 o Hannes Gredler 151 o John Leddy 153 o Stephen Youell 155 o David Mozes 157 o Petr Lapukhov 159 o Remy Chang 161 o Daniel Bernier 163 3. Conventions 165 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 166 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 167 document are to be interpreted as described in [RFC2119]. 169 Abbreviations used in this document: 171 E2E Edge to Edge 173 Geneve: Generic Network Virtualization Encapsulation 174 [I-D.ietf-nvo3-geneve] 176 IOAM: In-situ Operations, Administration, and Maintenance 178 MTU: Maximum Transmit Unit 180 NSH: Network Service Header [RFC8300] 182 OAM: Operations, Administration, and Maintenance 184 PMTU Path MTU 186 POT: Proof of Transit 188 SFC: Service Function Chain 190 SID: Segment Identifier 192 SR: Segment Routing 193 VXLAN-GPE: Virtual eXtensible Local Area Network, Generic Protocol 194 Extension [I-D.ietf-nvo3-vxlan-gpe] 196 4. Scope, Applicability, and Assumptions 198 IOAM deployment assumes a set of constraints, requirements, and 199 guiding principles which are described in this section. 201 Scope: This document defines the data fields and associated data 202 types for in-situ OAM. The in-situ OAM data field can be 203 encapsulated in a variety of protocols, including NSH, Segment 204 Routing, Geneve, IPv6, or IPv4. Specification details for these 205 different protocols are outside the scope of this document. 207 Deployment domain (or scope) of in-situ OAM deployment: IOAM is a 208 network domain focused feature, with "network domain" being a set of 209 network devices or entities within a single administration. For 210 example, a network domain can include an enterprise campus using 211 physical connections between devices or an overlay network using 212 virtual connections / tunnels for connectivity between said devices. 213 A network domain is defined by its perimeter or edge. Designers of 214 protocol encapsulations for IOAM specify mechanisms to ensure that 215 IOAM data stays within an IOAM domain. In addition, the operator of 216 such a domain is expected to put provisions in place to ensure that 217 IOAM data does not leak beyond the edge of an IOAM domain using,for 218 example, packet filtering methods. The operator has to consider the 219 potential operational impact of IOAM to mechanisms such as ECMP 220 processing (e.g. load-balancing schemes based on packet length could 221 be impacted by the increased packet size due to IOAM), path MTU (i.e. 222 ensure that the MTU of all links within a domain is sufficiently 223 large to support the increased packet size due to IOAM) and ICMP 224 message handling (i.e. in case of IPv6, IOAM support for ICMPv6 Echo 225 Request/Reply is desired which would translate into ICMPv6 extensions 226 to enable IOAM-Data-Fields to be copied from an Echo Request message 227 to an Echo Reply message). 229 IOAM control points: IOAM-Data-Fields are added to or removed from 230 the live user traffic by the devices which form the edge of a domain. 231 Devices which form an IOAM-Domain can add, update or remove IOAM- 232 Data-Fields. Edge devices of an IOAM-Domain can be hosts or network 233 devices. 235 Traffic-sets that IOAM is applied to: IOAM can be deployed on all or 236 only on subsets of the live user traffic. Using IOAM on a selected 237 set of traffic (e.g., per interface, based on an access control list 238 or flow specification defining a specific set of traffic, etc.) could 239 be useful in deployments where the cost of processing IOAM-Data- 240 Fields by encapsulating, transit, or decapsulating node(s) might be a 241 concern from a performance or operational perspective. Thus limiting 242 the amount of traffic IOAM is applied to could be beneficial in some 243 deployments. 245 Encapsulation independence: The definition of IOAM-Data-Fields is 246 independent from the protocols the IOAM-Data-Fields are encapsulated 247 into. IOAM-Data-Fields can be encapsulated into several 248 encapsulating protocols. The specification of how IOAM-Data-Fields 249 are encapsulated into "parent" protocols, like e.g., NSH or IPv6 is 250 outside the scope of this document. 252 Layering: If several encapsulation protocols (e.g., in case of 253 tunneling) are stacked on top of each other, IOAM-Data-Fields could 254 be present at multiple layers. The behavior follows the ships-in- 255 the-night model, i.e. IOAM-Data-Fields in one layer are independent 256 from IOAM-Data-Fields in another layer. Layering allows operators to 257 instrument the protocol layer they want to measure. The different 258 layers could, but do not have to, share the same IOAM encapsulation 259 mechanisms. 261 IOAM implementation: The definition of the IOAM-Data-Fields take the 262 specifics of devices with hardware data planes and software data 263 planes into account. 265 5. IOAM Data-Fields, Types, Nodes 267 This section details IOAM-related nomenclature and describes data 268 types such as IOAM-Data-Fields, IOAM-Types, IOAM-Namespaces as well 269 as the different types of IOAM nodes. 271 5.1. IOAM Data-Fields and Option-Types 273 An IOAM-Data-Field is a set of bits with a defined format and 274 meaning, which can be stored at a certain place in a packet for the 275 purpose of IOAM. 277 To accommodate the different uses of IOAM, IOAM-Data-Fields fall into 278 different categories. In IOAM these categories are referred to as 279 IOAM-Option-Types. A common registry is maintained for IOAM-Option- 280 Types, see Section 8.1 for details. Corresponding to these IOAM- 281 Option-Types, different IOAM-Data-Fields are defined. IOAM-Data- 282 Fields can be encapsulated into a variety of protocols, such as NSH, 283 Geneve, IPv6, etc. The definition of how IOAM-Data-Fields are 284 encapsulated into other protocols is outside the scope of this 285 document. 287 This document defines four IOAM-Option-Types: 289 o Pre-allocated Trace Option-Type 291 o Incremental Trace Option-Type 293 o Proof of Transit (POT) Option-Type 295 o Edge-to-Edge (E2E) Option-Type 297 5.2. IOAM-Domains and types of IOAM Nodes 299 IOAM is expected to be deployed in a specific domain. The part of 300 the network which employs IOAM is referred to as the "IOAM-Domain". 301 One or more IOAM-Option-Types are added to a packet upon entering the 302 IOAM-Domain and are removed from the packet when exiting the domain. 303 Within the IOAM-Domain, the IOAM-Data-Fields MAY be updated by 304 network nodes that the packet traverses. An IOAM-Domain consists of 305 "IOAM encapsulating nodes", "IOAM decapsulating nodes" and "IOAM 306 transit nodes". The role of a node (i.e. encapsulating, transit, 307 decapsulating) is defined within an IOAM-Namespace (see below). A 308 node can have different roles in different IOAM-Namespaces. 310 A device which adds at least one IOAM-Option-Type to the packet is 311 called the "IOAM encapsulating node", whereas a device which removes 312 an IOAM-Option-Type is referred to as the "IOAM decapsulating node". 313 Nodes within the domain which are aware of IOAM data and read and/or 314 write or process the IOAM data are called "IOAM transit nodes". IOAM 315 nodes which add or remove the IOAM-Data-Fields can also update the 316 IOAM-Data-Fields at the same time. Or in other words, IOAM 317 encapsulating or decapsulating nodes can also serve as IOAM transit 318 nodes at the same time. Note that not every node in an IOAM domain 319 needs to be an IOAM transit node. For example, a deployment might 320 require that packets traverse a set of firewalls which support IOAM. 321 In that case, only the set of firewall nodes would be IOAM transit 322 nodes rather than all nodes. 324 An "IOAM encapsulating node" incorporates one or more IOAM-Option- 325 Types (from the list of IOAM-Types, see Section 8.1) into packets 326 that IOAM is enabled for. If IOAM is enabled for a selected subset 327 of the traffic, the IOAM encapsulating node is responsible for 328 applying the IOAM functionality to the selected subset. 330 An "IOAM transit node" updates one or more of the IOAM-Data-Fields. 331 If both the Pre-allocated and the Incremental Trace Option-Types are 332 present in the packet, each IOAM transit node based on configuration 333 and available implementation of IOAM populates IOAM trace data in 334 either Pre-allocated or Incremental Trace Option-Type but not both. 335 A transit node MUST ignore IOAM-Option-Types that it does not 336 understand. A transit node MUST NOT add new IOAM-Option-Types to a 337 packet, MUST NOT remove IOAM-Option-Types from a packet, and MUST NOT 338 change the IOAM-Data-Fields of an IOAM Edge-to-Edge Option-Type. 340 An "IOAM decapsulating node" removes IOAM-Option-Type(s) from 341 packets. 343 The role of an IOAM-encapsulating, IOAM-transit or IOAM-decapsulating 344 node is always performed within a specific IOAM-Namespace. This 345 means that an IOAM node which is e.g. an IOAM-decapsulating node for 346 IOAM-Namespace "A" but not for IOAM-Namespace "B" will only remove 347 the IOAM-Option-Types for IOAM-Namespace "A" from the packet. Note 348 that this applies even for IOAM-Option-Types that the node does not 349 understand, for example an IOAM-Option-Type other than the four 350 described above, that is added in a future revision. An IOAM 351 decapsulating node situated at the edge of an IOAM domain MUST remove 352 all IOAM-Option-Types and associated encapsulation headers for all 353 IOAM-Namespaces from the packet. 355 IOAM-Namespaces allow for a namespace-specific definition and 356 interpretation of IOAM-Data-Fields. An interface-id could for 357 example point to a physical interface (e.g., to understand which 358 physical interface of an aggregated link is used when receiving or 359 transmitting a packet) whereas in another case it could refer to a 360 logical interface (e.g., in case of tunnels). Please refer to 361 Section 5.3 for details on IOAM-Namespaces. 363 5.3. IOAM-Namespaces 365 A subset or all of the IOAM-Option-Types and their corresponding 366 IOAM-Data-Fields can be associated to an IOAM-Namespace. IOAM- 367 Namespaces add further context to IOAM-Option-Types and associated 368 IOAM-Data-Fields. Any IOAM-Namespace MUST interpret the IOAM-Option- 369 Types and associated IOAM-Data-Fields per the definition in this 370 document. IOAM-Namespaces group nodes to support different 371 deployment approaches of IOAM (see a few example use-cases below) as 372 well as resolve issues which can occur due to IOAM-Data-Fields not 373 being globally unique (e.g. IOAM node identifiers do not have to be 374 globally unique). IOAM-Data-Fields significance is always within a 375 particular IOAM-Namespace. 377 An IOAM-Namespace is identified by a 16-bit namespace identifier 378 (Namespace-ID). IOAM-Namespace identifiers MUST be present and 379 populated in all IOAM-Option-Types. The Namespace-ID value is 380 divided into two sub-ranges: 382 o An operator-assigned range from 0x0001 to 0x7FFF 384 o An IANA-assigned range from 0x8000 to 0xFFFF 385 The IANA-assigned range is intended to allow future extensions to 386 have new and interoperable IOAM functionality, while the operator- 387 assigned range is intended to be domain specific, and managed by the 388 network operator. The Namespace-ID value of 0x0000 is the "Default- 389 Namespace-ID". The Default-Namespace-ID indicates that no specific 390 namespace is associated with the IOAM data fields in the packet. The 391 Default-Namespace-ID MUST be supported by all nodes implementing 392 IOAM. A use-case for the Default-Namespace-ID are deployments which 393 do not leverage specific namespaces for some or all of their packets 394 that carry IOAM data fields. 396 Namespace identifiers allow devices which are IOAM capable to 397 determine: 399 o whether IOAM-Option-Type(s) need to be processed by a device: If 400 the Namespace-ID contained in a packet does not match any 401 Namespace-ID the node is configured to operate on, then the node 402 MUST NOT change the contents of the IOAM-Data-Fields. 404 o which IOAM-Option-Type needs to be processed/updated in case there 405 are multiple IOAM-Option-Types present in the packet. Multiple 406 IOAM-Option-Types can be present in a packet in case of 407 overlapping IOAM-Domains or in case of a layered IOAM deployment. 409 o whether IOAM-Option-Type(s) has to be removed from the packet, 410 e.g. at a domain edge or domain boundary. 412 IOAM-Namespaces support several different uses: 414 o IOAM-Namespaces can be used by an operator to distinguish 415 different operational domains. Devices at domain edges can filter 416 on Namespace-IDs to provide for proper IOAM-Domain isolation. 418 o IOAM-Namespaces provide additional context for IOAM-Data-Fields 419 and thus ensure that IOAM-Data-Fields are unique and can be 420 interpreted properly by management stations or network 421 controllers. While, for example, the node identifier field 422 (node_id, see below) does not need to be unique in a deployment 423 (e.g. if an operator wishes to use different node identifiers for 424 different IOAM layers, even within the same device; or node 425 identifiers might not be unique for other organizational reasons, 426 such as after a merger of two formerly separated organizations), 427 the combination of node_id and Namespace-ID will always be unique. 428 Similarly, IOAM-Namespaces can be used to define how certain IOAM- 429 Data-Fields are interpreted: IOAM offers three different timestamp 430 format options. The Namespace-ID can be used to determine the 431 timestamp format. IOAM-Data-Fields (e.g. buffer occupancy) which 432 do not have a unit associated are to be interpreted within the 433 context of a IOAM-Namespace. 435 o IOAM-Namespaces can be used to identify different sets of devices 436 (e.g., different types of devices) in a deployment: If an operator 437 desires to insert different IOAM-Data-Fields based on the device, 438 the devices could be grouped into multiple IOAM-Namespaces. This 439 could be due to the fact that the IOAM feature set differs between 440 different sets of devices, or it could be for reasons of optimized 441 space usage in the packet header. It could also stem from 442 hardware or operational limitations on the size of the trace data 443 that can be added and processed, preventing collection of a full 444 trace for a flow. 446 * Assigning different IOAM Namespace-IDs to different sets of 447 nodes or network partitions and using the Namespace-ID as a 448 selector at the IOAM encapsulating node, a full trace for a 449 flow could be collected and constructed via partial traces in 450 different packets of the same flow. Example: An operator could 451 choose to group the devices of a domain into two IOAM- 452 Namespaces, in a way that on average, only every second hop 453 would be recorded by any device. To retrieve a full view of 454 the deployment, the captured IOAM-Data-Fields of the two IOAM- 455 Namespaces need to be correlated. 457 * Assigning different IOAM Namespace-IDs to different sets of 458 nodes or network partitions and using a separate instance of an 459 IOAM-Option-Type for each Namespace-ID, a full trace for a flow 460 could be collected and constructed via partial traces from each 461 IOAM-Option-Type in each of the packets in the flow. Example: 462 An operator could choose to group the devices of a domain into 463 two IOAM-Namespaces, in a way that each IOAM-Namespace is 464 represented by one of two IOAM-Option-Types in the packet. 465 Each node would record data only for the IOAM-Namespace that it 466 belongs to, ignoring the other IOAM-Option-Type with a IOAM- 467 Namespace to which it doesn't belong. To retrieve a full view 468 of the deployment, the captured IOAM-Data-Fields of the two 469 IOAM-Namespaces need to be correlated. 471 5.4. IOAM Trace Option-Types 473 "IOAM tracing data" is expected to be collected at every IOAM transit 474 node that a packet traverses to ensure visibility into the entire 475 path a packet takes within an IOAM-Domain. I.e., in a typical 476 deployment all nodes in an IOAM-Domain would participate in IOAM and 477 thus be IOAM transit nodes, IOAM encapsulating or IOAM decapsulating 478 nodes. If not all nodes within a domain support IOAM functionality 479 as defined in this document, IOAM tracing information (i.e., node 480 data, see below) will only be collected on those nodes which support 481 IOAM functionality as defined in this document. Nodes which do not 482 support IOAM functionality as defined in this document will forward 483 the packet without any changes to the IOAM-Data-Fields. The maximum 484 number of hops and the minimum path MTU of the IOAM domain is assumed 485 to be known. An overflow indicator (O-bit) is defined as one of the 486 ways to deal with situations where the PMTU was underestimated, i.e. 487 where the number of hops which are IOAM capable exceeds the available 488 space in the packet. 490 To optimize hardware and software implementations, IOAM tracing is 491 defined as two separate options. Any deployment MAY choose to 492 configure and support one or both of the following options. 494 Pre-allocated Trace-Option: This trace option is defined as a 495 container of node data fields (see below) with pre-allocated space 496 for each node to populate its information. This option is useful 497 for implementations where it is efficient to allocate the space 498 once and index into the array to populate the data during transit 499 (e.g., software forwarders often fall into this class). The IOAM 500 encapsulating node allocates space for Pre-allocated Trace Option- 501 Type in the packet and sets corresponding fields in this IOAM- 502 Option-Type. The IOAM encapsulating node allocates an array which 503 is used to store operational data retrieved from every node while 504 the packet traverses the domain. IOAM transit nodes update the 505 content of the array, and possibly update the checksums of outer 506 headers. A pointer which is part of the IOAM trace data, points 507 to the next empty slot in the array. An IOAM transit node that 508 updates the content of the pre-allocated option also updates the 509 value of the pointer, which specifies where the next IOAM transit 510 node fills in its data. The "node data list" array (see below) in 511 the packet is populated iteratively as the packet traverses the 512 network, starting with the last entry of the array, i.e., "node 513 data list [n]" is the first entry to be populated, "node data list 514 [n-1]" is the second one, etc. 516 Incremental Trace-Option: This trace option is defined as a 517 container of node data fields where each node allocates and pushes 518 its node data immediately following the option header. This type 519 of trace recording is useful for some of the hardware 520 implementations as it eliminates the need for the transit network 521 elements to read the full array in the option and allows for 522 arbitrarily long packets as the MTU allows. The IOAM 523 encapsulating node allocates space for the Incremental Trace 524 Option-Type. Based on operational state and configuration, the 525 IOAM encapsulating node sets the fields in the Option-Type that 526 control what IOAM-Data-Fields have to be collected and how large 527 the node data list can grow. IOAM transit nodes push their node 528 data to the node data list, decrease the remaining length 529 available to subsequent nodes and adjust the lengths and possibly 530 checksums in outer headers. 532 A particular implementation of IOAM MAY choose to support only one of 533 the two trace option types. In the event that both options are 534 utilized at the same time, the Incremental Trace-Option MUST be 535 placed before the Pre-allocated Trace-Option. Deployments which mix 536 devices with either the Incremental Trace-Option or the Pre-allocated 537 Trace-Option could result in both Option-Types being present in a 538 packet. Given that the operator knows which equipment is deployed in 539 a particular IOAM, the operator will decide by means of configuration 540 which type(s) of trace options will be used for a particular domain. 542 Every node data entry holds information for a particular IOAM transit 543 node that is traversed by a packet. The IOAM decapsulating node 544 removes the IOAM-Option-Type(s) and processes and/or exports the 545 associated data. Like all IOAM-Data-Fields, the IOAM-Data-Fields of 546 the IOAM-Trace-Option-Types are defined in the context of an IOAM- 547 Namespace. 549 IOAM tracing can collect the following types of information: 551 o Identification of the IOAM node. An IOAM node identifier can 552 match to a device identifier or a particular control point or 553 subsystem within a device. 555 o Identification of the interface that a packet was received on, 556 i.e. ingress interface. 558 o Identification of the interface that a packet was sent out on, 559 i.e. egress interface. 561 o Time of day when the packet was processed by the node as well as 562 the transit delay. Different definitions of processing time are 563 feasible and expected, though it is important that all devices of 564 an in-situ OAM domain follow the same definition. 566 o Generic data: Format-free information where syntax and semantic of 567 the information is defined by the operator in a specific 568 deployment. For a specific IOAM-Namespace, all IOAM nodes have to 569 interpret the generic data the same way. Examples for generic 570 IOAM data include geo-location information (location of the node 571 at the time the packet was processed), buffer queue fill level or 572 cache fill level at the time the packet was processed, or even a 573 battery charge level. 575 o Information to detect whether IOAM trace data was added at every 576 hop or whether certain hops in the domain weren't IOAM transit 577 nodes. 579 5.4.1. Pre-allocated and Incremental Trace Option-Types 581 The IOAM Pre-allocated Trace-Option and the IOAM Incremental Trace- 582 Option have similar formats. Except where noted below, the internal 583 formats and fields of the two trace options are identical. Both 584 Trace-Options consist of a fixed size "trace option header" and a 585 variable data space to store gathered data, the "node data list". An 586 IOAM transit node (that is not an IOAM encapsulating node or IOAM 587 decapsulating node) MUST NOT modify any of the fields in the fixed 588 size "trace option header", other than "flags" and "RemainingLen", 589 i.e. an IOAM transit node MUST NOT modify the Namespace-ID, NodeLen, 590 IOAM-Trace-Type, or Reserved fields. 592 Pre-allocated and incremental trace option headers: 594 0 1 2 3 595 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 596 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 597 | Namespace-ID |NodeLen | Flags | RemainingLen| 598 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 599 | IOAM-Trace-Type | Reserved | 600 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 602 The trace option data MUST be 4-octet aligned: 604 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+<-+ 605 | | | 606 | node data list [0] | | 607 | | | 608 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ D 609 | | a 610 | node data list [1] | t 611 | | a 612 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 613 ~ ... ~ S 614 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ p 615 | | a 616 | node data list [n-1] | c 617 | | e 618 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 619 | | | 620 | node data list [n] | | 621 | | | 622 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+<-+ 624 Namespace-ID: 16-bit identifier of an IOAM-Namespace. The 625 Namespace-ID value of 0x0000 is defined as the "Default-Namespace- 626 ID" (see Section 5.3) and MUST be known to all the nodes 627 implementing IOAM. For any other Namespace-ID value that does not 628 match any Namespace-ID the node is configured to operate on, the 629 node MUST NOT change the contents of the IOAM-Data-Fields. 631 NodeLen: 5-bit unsigned integer. This field specifies the length of 632 data added by each node in multiples of 4-octets, excluding the 633 length of the "Opaque State Snapshot" field. 635 If IOAM-Trace-Type bit 22 is not set, then NodeLen specifies the 636 actual length added by each node. If IOAM-Trace-Type bit 22 is 637 set, then the actual length added by a node would be (NodeLen + 638 length of the "Opaque State Snapshot" field) in 4 octet units. 640 For example, if 3 IOAM-Trace-Type bits are set and none of them 641 are wide, then NodeLen would be 3. If 3 IOAM-Trace-Type bits are 642 set and 2 of them are wide, then NodeLen would be 5. 644 An IOAM encapsulating node MUST set NodeLen. 646 A node receiving an IOAM Pre-allocated or Incremental Trace-Option 647 relies on the NodeLen value, or it can ignore the NodeLen value 648 and calculate the node length from the IOAM-Trace-Type bits (see 649 below). 651 Flags 4-bit field. Flags are allocated by IANA, as specified in 652 Section 8.3. This document allocates a single flag as follows: 654 Bit 0 "Overflow" (O-bit) (most significant bit). If there are 655 not enough octets left to record node data, the network element 656 MUST NOT add any fields and MUST set the overflow "O-bit" to 657 "1" in the IOAM-Trace-Option header. This is useful for 658 transit nodes to ignore further processing of the option. 660 RemainingLen: 7-bit unsigned integer. This field specifies the data 661 space in multiples of 4-octets remaining for recording the node 662 data, before the node data list is considered to have overflowed. 663 Given that the sender knows the path MTU (PMTU), the sender MAY 664 set the initial value of RemainingLen according to the number of 665 node data bytes allowed before exceeding the MTU. Subsequent 666 nodes can carry out a simple comparison between RemainingLen and 667 NodeLen, along with the length of the "Opaque State Snapshot" if 668 applicable, to determine whether or not data can be added by this 669 node. When node data is added, the node MUST decrease 670 RemainingLen by the amount of data added. In the pre-allocated 671 trace option, RemainingLen is used to derive the offset in data 672 space to record the node data element. Specifically, the 673 recording of the node data element would start from RemainingLen - 674 NodeLen - sizeof(opaque snapshot) in 4 octet units. If 675 RemainingLen in a pre-allocated trace option exceeds the length of 676 the option, as specified in the preceding header, then the node 677 MUST NOT add any fields. 679 IOAM-Trace-Type: A 24-bit identifier which specifies which data 680 types are used in this node data list. 682 The IOAM-Trace-Type value is a bit field. The following bits are 683 defined in this document, with details on each bit described in 684 the Section 5.4.2. The order of packing the data fields in each 685 node data element follows the bit order of the IOAM-Trace-Type 686 field, as follows: 688 Bit 0 (Most significant bit) When set, indicates presence of 689 Hop_Lim and node_id (short format) in the node data. 691 Bit 1 When set, indicates presence of ingress_if_id and 692 egress_if_id (short format) in the node data. 694 Bit 2 When set, indicates presence of timestamp seconds in the 695 node data. 697 Bit 3 When set, indicates presence of timestamp subseconds in 698 the node data. 700 Bit 4 When set, indicates presence of transit delay in the node 701 data. 703 Bit 5 When set, indicates presence of IOAM-Namespace specific 704 data (short format) in the node data. 706 Bit 6 When set, indicates presence of queue depth in the node 707 data. 709 Bit 7 When set, indicates presence of the Checksum Complement 710 node data. 712 Bit 8 When set, indicates presence of Hop_Lim and node_id in 713 wide format in the node data. 715 Bit 9 When set, indicates presence of ingress_if_id and 716 egress_if_id in wide format in the node data. 718 Bit 10 When set, indicates presence of IOAM-Namespace specific 719 data in wide format in the node data. 721 Bit 11 When set, indicates presence of buffer occupancy in the 722 node data. 724 Bit 12-21 Undefined. An IOAM encapsulating node MUST set the 725 value of each of these bits to 0. If an IOAM transit 726 node receives a packet with one or more of these bits set 727 to 1, it MUST either: 729 1. Add corresponding node data filled with the reserved 730 value 0xFFFFFFFF, after the node data fields for the 731 IOAM-Trace-Type bits defined above, such that the 732 total node data added by this node in units of 733 4-octets is equal to NodeLen, or 735 2. Not add any node data fields to the packet, even for 736 the IOAM-Trace-Type bits defined above. 738 Bit 22 When set, indicates presence of variable length Opaque 739 State Snapshot field. 741 Bit 23 Reserved: MUST be set to zero upon transmission and 742 ignored upon receipt. 744 Section 5.4.2 describes the IOAM-Data-Types and their formats. 745 Within an IOAM-Domain possible combinations of these bits making 746 the IOAM-Trace-Type can be restricted by configuration knobs. 748 Reserved: 8-bits. An IOAM encapsulating node MUST set the value to 749 zero upon transmission. IOAM transit nodes MUST ignore the 750 received value. 752 Node data List [n]: Variable-length field. This is a list of node 753 data elements where the content of each node data element is 754 determined by the IOAM-Trace-Type. The order of packing the data 755 fields in each node data element follows the bit order of the 756 IOAM-Trace-Type field. Each node MUST prepend its node data 757 element in front of the node data elements that it received, such 758 that the transmitted node data list begins with this node's data 759 element as the first populated element in the list. The last node 760 data element in this list is the node data of the first IOAM 761 capable node in the path. Populating the node data list in this 762 way ensures that the order of node data list is the same for 763 incremental and pre-allocated trace options. In the pre-allocated 764 trace option, the index contained in RemainingLen identifies the 765 offset for current active node data to be populated. 767 5.4.2. IOAM node data fields and associated formats 769 All the IOAM-Data-Fields MUST be 4-octet aligned. If a node which is 770 supposed to update an IOAM-Data-Field is not capable of populating 771 the value of a field set in the IOAM-Trace-Type, the field value MUST 772 be set to 0xFFFFFFFF for 4-octet fields or 0xFFFFFFFFFFFFFFFF for 773 8-octet fields, indicating that the value is not populated, except 774 when explicitly specified in the field description below. 776 Some IOAM-Data-Fields defined below, such as interface identifiers or 777 IOAM-Namespace specific data, are defined in both "short format" as 778 well as "wide format". Their use is not exclusive. A deployment 779 could choose to leverage both. For example, ingress_if_id_(short 780 format) could be an identifier for the physical interface, whereas 781 ingress_if_id_(wide format) could be an identifier for a logical sub- 782 interface of that physical interface. 784 Data fields and associated data types for each of the IOAM-Data- 785 Fields are specified in the following sections. 787 5.4.2.1. Hop_Lim and node_id short format 789 The "Hop_Lim and node_id short format" field is a 4-octet field that 790 is defined as follows: 792 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 793 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 794 | Hop_Lim | node_id | 795 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 797 Hop_Lim: 1-octet unsigned integer. It is set to the Hop Limit value 798 in the packet at the node that records this data. Hop Limit 799 information is used to identify the location of the node in the 800 communication path. This is copied from the lower layer, e.g., 801 TTL value in IPv4 header or hop limit field from IPv6 header of 802 the packet when the packet is ready for transmission. The 803 semantics of the Hop_Lim field depend on the lower layer protocol 804 that IOAM is encapsulated into, and therefore its specific 805 semantics are outside the scope of this memo. The value of this 806 field MUST be set to 0xff when the lower level does not have a 807 TTL/Hop limit equivalent field. 809 node_id: 3-octet unsigned integer. Node identifier field to 810 uniquely identify a node within the IOAM-Namespace and associated 811 IOAM-Domain. The procedure to allocate, manage and map the 812 node_ids is beyond the scope of this document. 814 5.4.2.2. ingress_if_id and egress_if_id 816 The "ingress_if_id and egress_if_id" field is a 4-octet field that is 817 defined as follows: 819 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 820 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 821 | ingress_if_id | egress_if_id | 822 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 824 ingress_if_id: 2-octet unsigned integer. Interface identifier to 825 record the ingress interface the packet was received on. 827 egress_if_id: 2-octet unsigned integer. Interface identifier to 828 record the egress interface the packet is forwarded out of. 830 Note that due to the fact that IOAM uses its own IOAM-Namespaces for 831 IOAM-Data-Fields, data fields like interface identifiers can be used 832 in a flexible way to represent system resources that are associated 833 with ingressing or egressing packets, i.e. ingress_if_id could 834 represent a physical interface, a virtual or logical interface, or 835 even a queue. 837 5.4.2.3. timestamp seconds 839 The "timestamp seconds" field is a 4-octet unsigned integer field. 840 Absolute timestamp in seconds that specifies the time at which the 841 packet was received by the node. This field has three possible 842 formats; based on either PTP [IEEE1588v2], NTP [RFC5905], or POSIX 843 [POSIX]. The three timestamp formats are specified in Section 6. In 844 all three cases, the Timestamp Seconds field contains the 32 most 845 significant bits of the timestamp format that is specified in 846 Section 6. If a node is not capable of populating this field, it 847 assigns the value 0xFFFFFFFF. Note that this is a legitimate value 848 that is valid for 1 second in approximately 136 years; the analyzer 849 has to correlate several packets or compare the timestamp value to 850 its own time-of-day in order to detect the error indication. 852 5.4.2.4. timestamp subseconds 854 The "timestamp subseconds" field is a 4-octet unsigned integer field. 855 Absolute timestamp in subseconds that specifies the time at which the 856 packet was received by the node. This field has three possible 857 formats; based on either PTP [IEEE1588v2], NTP [RFC5905], or POSIX 858 [POSIX]. The three timestamp formats are specified in Section 6. In 859 all three cases, the Timestamp Subseconds field contains the 32 least 860 significant bits of the timestamp format that is specified in 861 Section 6. If a node is not capable of populating this field, it 862 assigns the value 0xFFFFFFFF. Note that this is a legitimate value 863 in the NTP format, valid for approximately 233 picoseconds in every 864 second. If the NTP format is used the analyzer has to correlate 865 several packets in order to detect the error indication. 867 5.4.2.5. transit delay 869 The "transit delay" field is a 4-octet unsigned integer in the range 870 0 to 2^31-1. It is the time in nanoseconds the packet spent in the 871 transit node. This can serve as an indication of the queuing delay 872 at the node. If the transit delay exceeds 2^31-1 nanoseconds then 873 the top bit 'O' is set to indicate overflow and value set to 874 0x80000000. When this field is part of the data field but a node 875 populating the field is not able to fill it, the field position in 876 the field MUST be filled with value 0xFFFFFFFF to mean not populated. 878 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 879 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 880 |O| transit delay | 881 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 883 5.4.2.6. namespace specific data 885 The "namespace specific data" field is a 4-octet field which can be 886 used by the node to add IOAM-Namespace specific data. This 887 represents a "free-format" 4-octet bit field with its semantics 888 defined in the context of a specific IOAM-Namespace. 890 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 891 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 892 | namespace specific data | 893 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 895 5.4.2.7. queue depth 897 The "queue depth" field is a 4-octet unsigned integer field. This 898 field indicates the current length of the egress interface queue of 899 the interface from where the packet is forwarded out. The queue 900 depth is expressed as the current amount of memory buffers used by 901 the queue (a packet could consume one or more memory buffers, 902 depending on its size). 904 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 905 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 906 | queue depth | 907 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 909 5.4.2.8. Checksum Complement 911 The "Checksum Complement" field is a 4-octet node data which contains 912 a 4-octet Checksum Complement field. The Checksum Complement is 913 useful when IOAM is transported over encapsulations that make use of 914 a UDP transport, such as VXLAN-GPE or Geneve. Without the Checksum 915 Complement, nodes adding IOAM node data update the UDP Checksum field 916 following the recommendation of the encapsulation protocols. When 917 the Checksum Complement is present, an IOAM encapsulating node or 918 IOAM transit node adding node data MUST carry out one of the 919 following two alternatives in order to maintain the correctness of 920 the UDP Checksum value: 922 1. Recompute the UDP Checksum field. 924 2. Use the Checksum Complement to make a checksum-neutral update in 925 the UDP payload; the Checksum Complement is assigned a value that 926 complements the rest of the node data fields that were added by 927 the current node, causing the existing UDP Checksum field to 928 remain correct. 930 IOAM decapsulating nodes MUST recompute the UDP Checksum field, since 931 they do not know whether previous hops modified the UDP Checksum 932 field or the Checksum Complement field. 934 Checksum Complement fields are used in a similar manner in [RFC7820] 935 and [RFC7821]. 937 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 938 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 939 | Checksum Complement | 940 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 942 5.4.2.9. Hop_Lim and node_id wide 944 The "Hop_Lim and node_id wide" field is an 8-octet field defined as 945 follows: 947 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 948 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 949 | Hop_Lim | node_id ~ 950 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 951 ~ node_id (contd) | 952 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 954 Hop_Lim: 1-octet unsigned integer. It is set to the Hop Limit value 955 in the packet at the node that records this data. Hop Limit 956 information is used to identify the location of the node in the 957 communication path. This is copied from the lower layer for e.g. 958 TTL value in IPv4 header or hop limit field from IPv6 header of 959 the packet. The semantics of the Hop_Lim field depend on the 960 lower layer protocol that IOAM is encapsulated into, and therefore 961 its specific semantics are outside the scope of this memo. The 962 value of this field MUST be set to 0xff when the lower level does 963 not have a TTL/Hop limit equivalent field. 965 node_id: 7-octet unsigned integer. Node identifier field to 966 uniquely identify a node within the IOAM-Namespace and associated 967 IOAM-Domain. The procedure to allocate, manage and map the 968 node_ids is beyond the scope of this document. 970 5.4.2.10. ingress_if_id and egress_if_id wide 972 The "ingress_if_id and egress_if_id wide" field is an 8-octet field 973 which is defined as follows: 975 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 976 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 977 | ingress_if_id | 978 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 979 | egress_if_id | 980 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 982 ingress_if_id: 4-octet unsigned integer. Interface identifier to 983 record the ingress interface the packet was received on. 985 egress_if_id: 4-octet unsigned integer. Interface identifier to 986 record the egress interface the packet is forwarded out of. 988 5.4.2.11. namespace specific data wide 990 The "namespace specific data wide" field is an 8-octet field which 991 can be used by the node to add IOAM-Namespace specific data. This 992 represents a "free-format" 8-octet bit field with its semantics 993 defined in the context of a specific IOAM-Namespace. 995 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 996 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 997 | namespace specific data ~ 998 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 999 ~ namespace specific data (contd) | 1000 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1002 5.4.2.12. buffer occupancy 1004 The "buffer occupancy" field is a 4-octet unsigned integer field. 1005 This field indicates the current status of the occupancy of the 1006 common buffer pool used by a set of queues. The units of this field 1007 are implementation specific. Hence, the units are interpreted within 1008 the context of an IOAM-Namespace and/or node-id if used. The authors 1009 acknowledge that in some operational cases there is a need for the 1010 units to be consistent across a packet path through the network, 1011 hence RECOMMEND the implementations to use standard units such as 1012 Bytes. 1014 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1015 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1016 | buffer occupancy | 1017 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1019 5.4.2.13. Opaque State Snapshot 1021 The "Opaque State Snapshot" is a variable length field and follows 1022 the fixed length IOAM-Data-Fields defined above. It allows the 1023 network element to store an arbitrary state in the node data field, 1024 without a pre-defined schema. The schema is to be defined within the 1025 context of an IOAM-Namespace. The schema needs to be made known to 1026 the analyzer by some out-of-band mechanism. The specification of 1027 this mechanism is beyond the scope of this document. A 24-bit 1028 "Schema Id" field, interpreted within the context of an IOAM- 1029 Namespace, indicates which particular schema is used, and has to be 1030 configured on the network element by the operator. 1032 0 1 2 3 1033 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1034 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1035 | Length | Schema ID | 1036 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1037 | | 1038 | | 1039 | Opaque data | 1040 ~ ~ 1041 . . 1042 . . 1043 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1045 Length: 1-octet unsigned integer. It is the length in multiples of 1046 4-octets of the Opaque data field that follows Schema Id. 1048 Schema ID: 3-octet unsigned integer identifying the schema of Opaque 1049 data. 1051 Opaque data: Variable length field. This field is interpreted as 1052 specified by the schema identified by the Schema ID. 1054 When this field is part of the data field but a node populating the 1055 field has no opaque state data to report, the Length MUST be set to 0 1056 and the Schema ID MUST be set to 0xFFFFFF to mean no schema. 1058 5.4.3. Examples of IOAM node data 1060 An entry in the "node data list" array can have different formats, 1061 following the needs of the deployment. Some deployments might only 1062 be interested in recording the node identifiers, whereas others might 1063 be interested in recording node identifier and timestamp. The 1064 section provides example entries of the "node data list". 1066 0xD40000: IOAM-Trace-Type is 0xD40000 (0b110101000000000000000000) 1067 then the format of node data is: 1069 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1070 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1071 | Hop_Lim | node_id | 1072 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1073 | ingress_if_id | egress_if_id | 1074 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1075 | timestamp subseconds | 1076 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1077 | namespace specific data | 1078 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1080 0xC00000: IOAM-Trace-Type is 0xC00000 (0b110000000000000000000000) 1081 then the format is: 1083 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1084 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1085 | Hop_Lim | node_id | 1086 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1087 | ingress_if_id | egress_if_id | 1088 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1090 0x900000: IOAM-Trace-Type is 0x900000 (0b100100000000000000000000) 1091 then the format is: 1093 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1094 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1095 | Hop_Lim | node_id | 1096 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1097 | timestamp subseconds | 1098 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1100 0x840000: IOAM-Trace-Type is 0x840000 (0b100001000000000000000000) 1101 then the format is: 1103 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1104 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1105 | Hop_Lim | node_id | 1106 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1107 | namespace specific data | 1108 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1110 0x940000: IOAM-Trace-Type is 0x940000 (0b100101000000000000000000) 1111 then the format is: 1113 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1114 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1115 | Hop_Lim | node_id | 1116 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1117 | timestamp subseconds | 1118 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1119 | namespace specific data | 1120 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1122 0x308002: IOAM-Trace-Type is 0x308002 (0b001100001000000000000010) 1123 then the format is: 1125 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1126 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1127 | timestamp seconds | 1128 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1129 | timestamp subseconds | 1130 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1131 | Hop_Lim | node_id | 1132 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1133 | node_id(contd) | 1134 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1135 | Length | Schema Id | 1136 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1137 | | 1138 | | 1139 | Opaque data | 1140 ~ ~ 1141 . . 1142 . . 1143 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1145 5.5. IOAM Proof of Transit Option-Type 1147 IOAM Proof of Transit Option-Type is to support path or service 1148 function chain [RFC7665] verification use cases. Proof-of-transit 1149 leverages mechanisms like Shamir's Secret Sharing Schema (SSSS) 1150 [SSS]. For further information on Proof-of-transit, please refer to 1151 [I-D.ietf-sfc-proof-of-transit]. While details on how the IOAM data 1152 for the Proof-of-transit option is processed at IOAM encapsulating, 1153 decapsulating and transit nodes are outside the scope of the 1154 document, all of these approaches share the need to uniquely identify 1155 a packet as well as iteratively operate on a set of information that 1156 is handed from node to node. Correspondingly, two pieces of 1157 information are added as IOAM-Data-Fields to the packet: 1159 o Random: Unique identifier for the packet (e.g., 64-bits allow for 1160 the unique identification of 2^64 packets). 1162 o Cumulative: Information which is handed from node to node and 1163 updated by every node according to a verification algorithm. 1165 The IOAM Proof-of-Transit Option-Type consist of a fixed size "IOAM 1166 proof of transit option header" and "IOAM proof of transit option 1167 data fields": 1169 IOAM proof of transit option header: 1171 0 1 2 3 1172 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1173 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1174 | Namespace-ID |IOAM POT Type | IOAM POT flags| 1175 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1177 IOAM proof of transit Option-Type IOAM-Data-Fields MUST be 1178 4-octet aligned: 1180 0 1 2 3 1181 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1182 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1183 | POT Option data field determined by IOAM-POT-Type | 1184 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1186 Namespace-ID: 16-bit identifier of an IOAM-Namespace. The 1187 Namespace-ID value of 0x0000 is defined as the "Default-Namespace- 1188 ID" (see Section 5.3) and MUST be known to all the nodes 1189 implementing IOAM. For any other Namespace-ID value that does not 1190 match any Namespace-ID the node is configured to operate on, the 1191 node MUST NOT change the contents of the IOAM-Data-Fields. 1193 IOAM POT Type: 8-bit identifier of a particular POT variant that 1194 specifies the POT data that is included. This document defines 1195 POT Type 0: 1197 0: POT data is a 16 Octet field as described below. 1199 If a node receives an IOAM POT Type value that it does not 1200 understand, the node MUST NOT change the contents of the IOAM- 1201 Data-Fields. 1203 IOAM POT flags: 8-bit. Following flags are defined: 1205 Bit 0 "Profile-to-use" (P-bit) (most significant bit). For IOAM 1206 POT types that use a maximum of two profiles to drive 1207 computation, indicates which POT-profile is used. The two 1208 profiles are numbered 0, 1. 1210 Bit 1-7 Reserved: MUST be set to zero upon transmission and 1211 ignored upon receipt. 1213 POT Option data: Variable-length field. The type of which is 1214 determined by the IOAM-POT-Type. 1216 5.5.1. IOAM Proof of Transit Type 0 1218 IOAM proof of transit option of IOAM POT Type 0: 1220 0 1 2 3 1221 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1222 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1223 | Namespace-ID |IOAM POT Type=0|P|R R R R R R R| 1224 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+<-+ 1225 | Random | | 1226 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ P 1227 | Random(contd) | O 1228 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ T 1229 | Cumulative | | 1230 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 1231 | Cumulative (contd) | | 1232 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+<-+ 1234 Namespace-ID: 16-bit identifier of an IOAM-Namespace. The 1235 Namespace-ID value of 0x0000 is defined as the "Default-Namespace- 1236 ID" (see Section 5.3) and MUST be known to all the nodes 1237 implementing IOAM. For any other Namespace-ID value that does not 1238 match any Namespace-ID the node is configured to operate on, the 1239 node MUST NOT change the contents of the IOAM-Data-Fields. 1241 IOAM POT Type: 8-bit identifier of a particular POT variant that 1242 specifies the POT data that is included. This section defines the 1243 POT data when the IOAM POT Type is set to the value 0. 1245 P bit: 1-bit. "Profile-to-use" (P-bit) (most significant bit). 1246 Indicates which POT-profile is used to generate the Cumulative. 1247 Any node participating in POT will have a maximum of 2 profiles 1248 configured that drive the computation of cumulative. The two 1249 profiles are numbered 0, 1. This bit conveys whether profile 0 or 1250 profile 1 is used to compute the Cumulative. 1252 R (7 bits): 7-bit IOAM POT flags for future use. MUST be set to 1253 zero upon transmission and ignored upon receipt. 1255 Random: 64-bit Per packet Random number. 1257 Cumulative: 64-bit Cumulative that is updated at specific nodes by 1258 processing per packet Random number field and configured 1259 parameters. 1261 Note: Larger or smaller sizes of "Random" and "Cumulative" data are 1262 feasible and could be required for certain deployments (e.g. in case 1263 of space constraints in the encapsulation protocols used). Future 1264 documents could introduce different sizes of data for "proof of 1265 transit". 1267 5.6. IOAM Edge-to-Edge Option-Type 1269 The IOAM Edge-to-Edge Option-Type is to carry data that is added by 1270 the IOAM encapsulating node and interpreted by IOAM decapsulating 1271 node. The IOAM transit nodes MAY process the data but MUST NOT 1272 modify it. 1274 The IOAM Edge-to-Edge Option-Type consist of a fixed size "IOAM Edge- 1275 to-Edge Option-Type header" and "IOAM Edge-to-Edge Option-Type data 1276 fields": 1278 IOAM Edge-to-Edge Option-Type header: 1280 0 1 2 3 1281 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1282 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1283 | Namespace-ID | IOAM-E2E-Type | 1284 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1286 IOAM Edge-to-Edge Option-Type IOAM-Data-Fields MUST 1287 be 4-octet aligned: 1289 0 1 2 3 1290 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1291 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1292 | E2E Option data field determined by IOAM-E2E-Type | 1293 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1294 Namespace-ID: 16-bit identifier of an IOAM-Namespace. The 1295 Namespace-ID value of 0x0000 is defined as the "Default-Namespace- 1296 ID" (see Section 5.3) and MUST be known to all the nodes 1297 implementing IOAM. For any other Namespace-ID value that does not 1298 match any Namespace-ID the node is configured to operate on, then 1299 the node MUST NOT change the contents of the IOAM-Data-Fields. 1301 IOAM-E2E-Type: A 16-bit identifier which specifies which data types 1302 are used in the E2E option data. The IOAM-E2E-Type value is a bit 1303 field. The order of packing the E2E option data field elements 1304 follows the bit order of the IOAM-E2E-Type field, as follows: 1306 Bit 0 (Most significant bit) When set indicates presence of a 1307 64-bit sequence number added to a specific "packet group" 1308 which is used to detect packet loss, packet reordering, 1309 or packet duplication within the group. The "packet 1310 group" is deployment dependent and defined at the IOAM 1311 encapsulating node e.g. by n-tuple based classification 1312 of packets. 1314 Bit 1 When set indicates presence of a 32-bit sequence number 1315 added to a specific "packet group" which is used to 1316 detect packet loss, packet reordering, or packet 1317 duplication within that group. The "packet group" is 1318 deployment dependent and defined at the IOAM 1319 encapsulating node e.g. by n-tuple based classification 1320 of packets. 1322 Bit 2 When set indicates presence of timestamp seconds, 1323 representing the time at which the packet entered the 1324 IOAM domain. Within the IOAM encapsulating node, the 1325 time that the timestamp is retrieved can depend on the 1326 implementation. Some possibilities are: 1) the time at 1327 which the packet was received by the node, 2) the time at 1328 which the packet was transmitted by the node, 3) when a 1329 tunnel encapsulation is used, the point at which the 1330 packet is encapsulated into the tunnel. Each 1331 implementation has to document when the E2E timestamp 1332 that is going to be put in the packet is retrieved. This 1333 4-octet field has three possible formats; based on either 1334 PTP [IEEE1588v2], NTP [RFC5905], or POSIX [POSIX]. The 1335 three timestamp formats are specified in Section 6. In 1336 all three cases, the Timestamp Seconds field contains the 1337 32 most significant bits of the timestamp format that is 1338 specified in Section 6. If a node is not capable of 1339 populating this field, it assigns the value 0xFFFFFFFF. 1340 Note that this is a legitimate value that is valid for 1 1341 second in approximately 136 years; the analyzer has to 1342 correlate several packets or compare the timestamp value 1343 to its own time-of-day in order to detect the error 1344 indication. 1346 Bit 3 When set indicates presence of timestamp subseconds, 1347 representing the time at which the packet entered the 1348 IOAM domain. This 4-octet field has three possible 1349 formats; based on either PTP [IEEE1588v2], NTP [RFC5905], 1350 or POSIX [POSIX]. The three timestamp formats are 1351 specified in Section 6. In all three cases, the 1352 Timestamp Subseconds field contains the 32 least 1353 significant bits of the timestamp format that is 1354 specified in Section 6. If a node is not capable of 1355 populating this field, it assigns the value 0xFFFFFFFF. 1356 Note that this is a legitimate value in the NTP format, 1357 valid for approximately 233 picoseconds in every second. 1358 If the NTP format is used the analyzer has to correlate 1359 several packets in order to detect the error indication. 1361 Bit 4-15 Undefined. An IOAM encapsulating node MUST set the value 1362 of these bits to zero upon transmission and ignore upon 1363 receipt. 1365 E2E Option data: Variable-length field. The type of which is 1366 determined by the IOAM-E2E-Type. 1368 6. Timestamp Formats 1370 The IOAM-Data-Fields include a timestamp field which is represented 1371 in one of three possible timestamp formats. It is assumed that the 1372 management plane is responsible for determining which timestamp 1373 format is used. 1375 6.1. PTP Truncated Timestamp Format 1377 The Precision Time Protocol (PTP) [IEEE1588v2] uses an 80-bit 1378 timestamp format. The truncated timestamp format is a 64-bit field, 1379 which is the 64 least significant bits of the 80-bit PTP timestamp. 1380 The PTP truncated format is specified in Section 4.3 of [RFC8877], 1381 and the details are presented below for the sake of completeness. 1383 0 1 2 3 1384 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1385 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1386 | Seconds | 1387 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1388 | Nanoseconds | 1389 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1391 Figure 1: PTP [IEEE1588v2] Truncated Timestamp Format 1393 Timestamp field format: 1395 Seconds: specifies the integer portion of the number of seconds 1396 since the epoch. 1398 + Size: 32 bits. 1400 + Units: seconds. 1402 Nanoseconds: specifies the fractional portion of the number of 1403 seconds since the epoch. 1405 + Size: 32 bits. 1407 + Units: nanoseconds. The value of this field is in the range 0 1408 to (10^9)-1. 1410 Epoch: 1412 The PTP [IEEE1588v2] epoch is 1 January 1970 00:00:00 TAI, which 1413 is 31 December 1969 23:59:51.999918 UTC. 1415 Resolution: 1417 The resolution is 1 nanosecond. 1419 Wraparound: 1421 This time format wraps around every 2^32 seconds, which is roughly 1422 136 years. The next wraparound will occur in the year 2106. 1424 Synchronization Aspects: 1426 It is assumed that nodes that run this protocol are synchronized 1427 among themselves. Nodes MAY be synchronized to a global reference 1428 time. Note that if PTP [IEEE1588v2] is used for synchronization, 1429 the timestamp MAY be derived from the PTP-synchronized clock, 1430 allowing the timestamp to be measured with respect to the clock of 1431 an PTP Grandmaster clock. 1433 The PTP truncated timestamp format is not affected by leap 1434 seconds. 1436 6.2. NTP 64-bit Timestamp Format 1438 The Network Time Protocol (NTP) [RFC5905] timestamp format is 64 bits 1439 long. This format is specified in Section 4.2.1 of [RFC8877], and 1440 the details are presented below for the sake of completeness. 1442 0 1 2 3 1443 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1444 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1445 | Seconds | 1446 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1447 | Fraction | 1448 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1450 Figure 2: NTP [RFC5905] 64-bit Timestamp Format 1452 Timestamp field format: 1454 Seconds: specifies the integer portion of the number of seconds 1455 since the epoch. 1457 + Size: 32 bits. 1459 + Units: seconds. 1461 Fraction: specifies the fractional portion of the number of 1462 seconds since the epoch. 1464 + Size: 32 bits. 1466 + Units: the unit is 2^(-32) seconds, which is roughly equal to 1467 233 picoseconds. 1469 Epoch: 1471 The epoch is 1 January 1900 at 00:00 UTC. 1473 Resolution: 1475 The resolution is 2^(-32) seconds. 1477 Wraparound: 1479 This time format wraps around every 2^32 seconds, which is roughly 1480 136 years. The next wraparound will occur in the year 2036. 1482 Synchronization Aspects: 1484 Nodes that use this timestamp format will typically be 1485 synchronized to UTC using NTP [RFC5905]. Thus, the timestamp MAY 1486 be derived from the NTP-synchronized clock, allowing the timestamp 1487 to be measured with respect to the clock of an NTP server. 1489 The NTP timestamp format is affected by leap seconds; it 1490 represents the number of seconds since the epoch minus the number 1491 of leap seconds that have occurred since the epoch. The value of 1492 a timestamp during or slightly after a leap second could be 1493 temporarily inaccurate. 1495 6.3. POSIX-based Timestamp Format 1497 This timestamp format is based on the POSIX time format [POSIX]. The 1498 detailed specification of the timestamp format used in this document 1499 is presented below. 1501 0 1 2 3 1502 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1503 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1504 | Seconds | 1505 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1506 | Microseconds | 1507 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1509 Figure 3: POSIX-based Timestamp Format 1511 Timestamp field format: 1513 Seconds: specifies the integer portion of the number of seconds 1514 since the epoch. 1516 + Size: 32 bits. 1518 + Units: seconds. 1520 Microseconds: specifies the fractional portion of the number of 1521 seconds since the epoch. 1523 + Size: 32 bits. 1525 + Units: the unit is microseconds. The value of this field is in 1526 the range 0 to (10^6)-1. 1528 Epoch: 1530 The epoch is 1 January 1970 00:00:00 TAI, which is 31 December 1531 1969 23:59:51.999918 UTC. 1533 Resolution: 1535 The resolution is 1 microsecond. 1537 Wraparound: 1539 This time format wraps around every 2^32 seconds, which is roughly 1540 136 years. The next wraparound will occur in the year 2106. 1542 Synchronization Aspects: 1544 It is assumed that nodes that use this timestamp format run the 1545 Linux operating system, and hence use the POSIX time. In some 1546 cases nodes MAY be synchronized to UTC using a synchronization 1547 mechanism that is outside the scope of this document, such as NTP 1548 [RFC5905]. Thus, the timestamp MAY be derived from the NTP- 1549 synchronized clock, allowing the timestamp to be measured with 1550 respect to the clock of an NTP server. 1552 The POSIX-based timestamp format is affected by leap seconds; it 1553 represents the number of seconds since the epoch minus the number 1554 of leap seconds that have occurred since the epoch. The value of 1555 a timestamp during or slightly after a leap second could be 1556 temporarily inaccurate. 1558 7. IOAM Data Export 1560 IOAM nodes collect information for packets traversing a domain that 1561 supports IOAM. IOAM decapsulating nodes as well as IOAM transit 1562 nodes can choose to retrieve IOAM information from the packet, 1563 process the information further and export the information using 1564 e.g., IPFIX. The mechanisms and associated data formats for 1565 exporting IOAM data is outside the scope of this document. 1567 Raw data export of IOAM data using IPFIX is discussed in 1568 [I-D.spiegel-ippm-ioam-rawexport]. 1570 8. IANA Considerations 1572 This document requests the following IANA Actions. 1574 IANA is requested to define a registry group named "In-Situ OAM 1575 (IOAM) Protocol Parameters". 1577 This group will include the following registries: 1579 IOAM Option-Type 1581 IOAM Trace-Type 1583 IOAM Trace-Flags 1585 IOAM POT-Type 1587 IOAM POT-Flags 1589 IOAM E2E-Type 1591 IOAM Namespace-ID 1593 New registries in this group can be created via RFC Required process 1594 as per [RFC8126]. 1596 The subsequent sub-sections detail the registries herein contained. 1598 8.1. IOAM Option-Type Registry 1600 This registry defines 128 code points for the IOAM Option-Type field 1601 for identifying IOAM Option-Types as explained in Section 5. The 1602 following code points are defined in this draft: 1604 0 IOAM Pre-allocated Trace Option-Type 1606 1 IOAM Incremental Trace Option-Type 1608 2 IOAM POT Option-Type 1610 3 IOAM E2E Option-Type 1612 4 - 127 are available for assignment via RFC Required process as per 1613 [RFC8126]. 1615 8.2. IOAM Trace-Type Registry 1617 This registry defines code point for each bit in the 24-bit IOAM- 1618 Trace-Type field for Pre-allocated trace option and Incremental trace 1619 option defined in Section 5.4. The meaning of Bits 0 - 11 for trace 1620 type are defined in this document in Paragraph 5 of Section 5.4.1: 1622 Bit 0 hop_Lim and node_id in short format 1624 Bit 1 ingress_if_id and egress_if_id in short format 1626 Bit 2 timestamp seconds 1628 Bit 3 timestamp subseconds 1630 Bit 4 transit delay 1632 Bit 5 namespace specific data in short format 1634 Bit 6 queue depth 1636 Bit 7 checksum complement 1638 Bit 8 hop_Lim and node_id in wide format 1640 Bit 9 ingress_if_id and egress_if_id in wide format 1642 Bit 10 namespace specific data in wide format 1644 Bit 11 buffer occupancy 1646 Bit 22 variable length Opaque State Snapshot 1648 Bit 23 reserved 1650 The meaning for Bits 12 - 21 are available for assignment via RFC 1651 Required process as per [RFC8126]. 1653 8.3. IOAM Trace-Flags Registry 1655 This registry defines code points for each bit in the 4 bit flags for 1656 the Pre-allocated trace option and for the Incremental trace option 1657 defined in Section 5.4. The meaning of Bit 0 (the most significant 1658 bit) for trace flags is defined in this document in Paragraph 3 of 1659 Section 5.4.1: 1661 Bit 0 "Overflow" (O-bit) 1662 Bit 1 - 3 are available for assignment via RFC Required process as 1663 per [RFC8126]. 1665 8.4. IOAM POT-Type Registry 1667 This registry defines 256 code points to define IOAM POT Type for 1668 IOAM proof of transit option Section 5.5. The code point value 0 is 1669 defined in this document: 1671 0: 16 Octet POT data 1673 1 - 255 are available for assignment via RFC Required process as per 1674 [RFC8126]. 1676 8.5. IOAM POT-Flags Registry 1678 This registry defines code points for each bit in the 8 bit flags for 1679 IOAM POT option defined in Section 5.5. The meaning of Bit 0 for 1680 IOAM POT flags is defined in this document in Section 5.5: 1682 Bit 0 "Profile-to-use" (P-bit) 1684 The meaning for Bits 1 - 7 are available for assignment via RFC 1685 Required process as per [RFC8126]. 1687 8.6. IOAM E2E-Type Registry 1689 This registry defines code points for each bit in the 16 bit IOAM- 1690 E2E-Type field for IOAM E2E option Section 5.6. The meaning of Bit 0 1691 - 3 are defined in this document: 1693 Bit 0 64-bit sequence number 1695 Bit 1 32-bit sequence number 1697 Bit 2 timestamp seconds 1699 Bit 3 timestamp subseconds 1701 The meaning of Bits 4 - 15 are available for assignment via RFC 1702 Required process as per [RFC8126]. 1704 8.7. IOAM Namespace-ID Registry 1706 IANA is requested to set up an "IOAM Namespace-ID Registry", 1707 containing 16-bit values. The meaning of Bit 0 is defined in this 1708 document. IANA is requested to reserve the values 0x0001 to 0x7FFF 1709 for private use (managed by operators), as specified in Section 5.3 1710 of the current document. Registry entries for the values 0x8000 to 1711 0xFFFF are to be assigned via the "Expert Review" policy defined in 1712 [RFC8126]. Upon a new allocation request, the responsible AD will 1713 appoint a designated expert, who will review the allocation request. 1714 The expert will post the request on the IPPM mailing list, and 1715 possibly on other relevant mailing lists, to allow for community 1716 feedback. Based on the review, the expert will either approve or 1717 deny the request. The intention is that any allocation will be 1718 accompanied by a published RFC. But in order to allow for the 1719 allocation of values prior to the RFC being approved for publication, 1720 the designated expert can approve allocations once it seems clear 1721 that an RFC will be published. 1723 0: default namespace (known to all IOAM nodes) 1725 0x0001 - 0x7FFF: reserved for private use 1727 0x8000 - 0xFFFF: unassigned 1729 9. Management and Deployment Considerations 1731 This document defines the structure and use of IOAM data fields. 1732 This document does not define the encapsulation of IOAM data fields 1733 into different protocols. Management and deployment aspects for IOAM 1734 have to be considered within the context of the protocol IOAM data 1735 fields are encapsulated into and as such, are out of scope for this 1736 document. For a discussion of IOAM deployment, please also refer to 1737 [I-D.brockners-opsawg-ioam-deployment], which outlines a framework 1738 for IOAM deployment and provides best current practices. 1740 10. Security Considerations 1742 As discussed in [RFC7276], a successful attack on an OAM protocol in 1743 general, and specifically on IOAM, can prevent the detection of 1744 failures or anomalies, or create a false illusion of nonexistent 1745 ones. In particular, these threats are applicable by compromising 1746 the integrity of IOAM data, either by maliciously modifying IOAM 1747 options in transit, or by injecting packets with maliciously 1748 generated IOAM options 1750 The Proof of Transit Option-Type (Section Section 5.5) is used for 1751 verifying the path of data packets. The security considerations of 1752 POT are further discussed in [I-D.ietf-sfc-proof-of-transit]. 1754 From a confidentiality perspective, although IOAM options do not 1755 contain user data, they can be used for network reconnaissance, 1756 allowing attackers to collect information about network paths, 1757 performance, queue states, buffer occupancy and other information. 1759 Moreover, if IOAM data leaks from the IOAM domain it could enable 1760 reconnaissance beyond the scope of the IOAM domain. Note that in 1761 case IOAM is used in "Direct Exporting" mode 1762 [I-D.ioamteam-ippm-ioam-direct-export], the IOAM related trace 1763 information would not be available in the customer data packets, but 1764 would trigger export of packet related IOAM information at every 1765 node, thus restricting the potential threat to the management plane 1766 and mitigating the leakage threat. IOAM data exporting and the way 1767 it is secured is outside the scope of this document. 1769 IOAM can be used as a means for implementing Denial of Service (DoS) 1770 attacks, or for amplifying them. For example, a malicious attacker 1771 can add an IOAM header to packets in order to consume the resources 1772 of network devices that take part in IOAM or entities that receive, 1773 collect or analyze the IOAM data. Another example is a packet length 1774 attack, in which an attacker pushes headers associated with IOAM 1775 Option-Types into data packets, causing these packets to be increased 1776 beyond the MTU size, resulting in fragmentation or in packet drops. 1778 Since IOAM options can include timestamps, if network devices use 1779 synchronization protocols then any attack on the time protocol 1780 [RFC7384] can compromise the integrity of the timestamp-related data 1781 fields. 1783 At the management plane, attacks can be set up by misconfiguring or 1784 by maliciously configuring IOAM-enabled nodes in a way that enables 1785 other attacks. Thus, IOAM configuration has to be secured in a way 1786 that authenticates authorized users and verifies the integrity of 1787 configuration procedures. 1789 The current document does not define a specific IOAM encapsulation. 1790 It has to be noted that some IOAM encapsulation types can introduce 1791 specific security considerations. A specification that defines an 1792 IOAM encapsulation is expected to address the respective 1793 encapsulation-specific security considerations. 1795 Notably, in most cases IOAM is expected to be deployed in specific 1796 network domains, thus confining the potential attack vectors to 1797 within the network domain. A limited administrative domain provides 1798 the operator with the means to select, monitor, and control the 1799 access of all the network devices, making these devices trusted by 1800 the operator. Indeed, in order to limit the scope of threats 1801 mentioned above to within the current network domain the network 1802 operator is expected to enforce policies that prevent IOAM traffic 1803 from leaking outside of the IOAM domain, and prevent IOAM data from 1804 outside the domain to be processed and used within the domain. 1806 The security considerations of a system that deploys IOAM, much like 1807 any system, has to be reviewed on a per-deployment-scenario basis, 1808 based on a systems-specific threat analysis, which can lead to 1809 specific security solutions that are beyond the scope of the current 1810 document. Specifically, in an IOAM deployment that is not confined 1811 to a single LAN, but spans multiple inter-connected sites (for 1812 example, using an overlay network), the inter-site links can be 1813 secured (e.g., by IPsec) in order to avoid external threats. 1815 11. Acknowledgements 1817 The authors would like to thank Eric Vyncke, Nalini Elkins, Srihari 1818 Raghavan, Ranganathan T S, Karthik Babu Harichandra Babu, Akshaya 1819 Nadahalli, LJ Wobker, Erik Nordmark, Vengada Prasad Govindan, Andrew 1820 Yourtchenko, Aviv Kfir, Tianran Zhou and Zhenbin (Robin) for the 1821 comments and advice. 1823 This document leverages and builds on top of several concepts 1824 described in [I-D.kitamura-ipv6-record-route]. The authors would 1825 like to acknowledge the work done by the author Hiroshi Kitamura and 1826 people involved in writing it. 1828 The authors would like to gracefully acknowledge useful review and 1829 insightful comments received from Joe Clarke, Al Morton, Tom Herbert, 1830 Haoyu Song, Mickey Spiegel and Barak Gafni. 1832 12. References 1834 12.1. Normative References 1836 [IEEE1588v2] 1837 Institute of Electrical and Electronics Engineers, "IEEE 1838 Std 1588-2008 - IEEE Standard for a Precision Clock 1839 Synchronization Protocol for Networked Measurement and 1840 Control Systems", IEEE Std 1588-2008, 2008, 1841 . 1844 [POSIX] Institute of Electrical and Electronics Engineers, "IEEE 1845 Std 1003.1-2008 (Revision of IEEE Std 1003.1-2004) - IEEE 1846 Standard for Information Technology - Portable Operating 1847 System Interface (POSIX(R))", IEEE Std 1003.1-2008, 2008, 1848 . 1851 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1852 Requirement Levels", BCP 14, RFC 2119, 1853 DOI 10.17487/RFC2119, March 1997, 1854 . 1856 [RFC5905] Mills, D., Martin, J., Ed., Burbank, J., and W. Kasch, 1857 "Network Time Protocol Version 4: Protocol and Algorithms 1858 Specification", RFC 5905, DOI 10.17487/RFC5905, June 2010, 1859 . 1861 [RFC8126] Cotton, M., Leiba, B., and T. Narten, "Guidelines for 1862 Writing an IANA Considerations Section in RFCs", BCP 26, 1863 RFC 8126, DOI 10.17487/RFC8126, June 2017, 1864 . 1866 12.2. Informative References 1868 [I-D.brockners-opsawg-ioam-deployment] 1869 Brockners, F., Bhandari, S., and d. 1870 daniel.bernier@bell.ca, "In-situ OAM Deployment", draft- 1871 brockners-opsawg-ioam-deployment-02 (work in progress), 1872 September 2020. 1874 [I-D.ietf-nvo3-geneve] 1875 Gross, J., Ganga, I., and T. Sridhar, "Geneve: Generic 1876 Network Virtualization Encapsulation", draft-ietf- 1877 nvo3-geneve-16 (work in progress), March 2020. 1879 [I-D.ietf-nvo3-vxlan-gpe] 1880 Maino, F., Kreeger, L., and U. Elzur, "Generic Protocol 1881 Extension for VXLAN (VXLAN-GPE)", draft-ietf-nvo3-vxlan- 1882 gpe-10 (work in progress), July 2020. 1884 [I-D.ietf-sfc-proof-of-transit] 1885 Brockners, F., Bhandari, S., Mizrahi, T., Dara, S., and S. 1886 Youell, "Proof of Transit", draft-ietf-sfc-proof-of- 1887 transit-08 (work in progress), November 2020. 1889 [I-D.ioamteam-ippm-ioam-direct-export] 1890 Song, H., Gafni, B., Zhou, T., Li, Z., Brockners, F., 1891 Bhandari, S., Sivakolundu, R., and T. Mizrahi, "In-situ 1892 OAM Direct Exporting", draft-ioamteam-ippm-ioam-direct- 1893 export-00 (work in progress), October 2019. 1895 [I-D.kitamura-ipv6-record-route] 1896 Kitamura, H., "Record Route for IPv6 (PR6) Hop-by-Hop 1897 Option Extension", draft-kitamura-ipv6-record-route-00 1898 (work in progress), November 2000. 1900 [I-D.spiegel-ippm-ioam-rawexport] 1901 Spiegel, M., Brockners, F., Bhandari, S., and R. 1902 Sivakolundu, "In-situ OAM raw data export with IPFIX", 1903 draft-spiegel-ippm-ioam-rawexport-04 (work in progress), 1904 November 2020. 1906 [RFC7276] Mizrahi, T., Sprecher, N., Bellagamba, E., and Y. 1907 Weingarten, "An Overview of Operations, Administration, 1908 and Maintenance (OAM) Tools", RFC 7276, 1909 DOI 10.17487/RFC7276, June 2014, 1910 . 1912 [RFC7384] Mizrahi, T., "Security Requirements of Time Protocols in 1913 Packet Switched Networks", RFC 7384, DOI 10.17487/RFC7384, 1914 October 2014, . 1916 [RFC7665] Halpern, J., Ed. and C. Pignataro, Ed., "Service Function 1917 Chaining (SFC) Architecture", RFC 7665, 1918 DOI 10.17487/RFC7665, October 2015, 1919 . 1921 [RFC7799] Morton, A., "Active and Passive Metrics and Methods (with 1922 Hybrid Types In-Between)", RFC 7799, DOI 10.17487/RFC7799, 1923 May 2016, . 1925 [RFC7820] Mizrahi, T., "UDP Checksum Complement in the One-Way 1926 Active Measurement Protocol (OWAMP) and Two-Way Active 1927 Measurement Protocol (TWAMP)", RFC 7820, 1928 DOI 10.17487/RFC7820, March 2016, 1929 . 1931 [RFC7821] Mizrahi, T., "UDP Checksum Complement in the Network Time 1932 Protocol (NTP)", RFC 7821, DOI 10.17487/RFC7821, March 1933 2016, . 1935 [RFC8300] Quinn, P., Ed., Elzur, U., Ed., and C. Pignataro, Ed., 1936 "Network Service Header (NSH)", RFC 8300, 1937 DOI 10.17487/RFC8300, January 2018, 1938 . 1940 [RFC8877] Mizrahi, T., Fabini, J., and A. Morton, "Guidelines for 1941 Defining Packet Timestamps", RFC 8877, 1942 DOI 10.17487/RFC8877, September 2020, 1943 . 1945 [SSS] Wikipedia, "Shamir's Secret Sharing", 1946 . 1948 Contributors' Addresses 1950 Carlos Pignataro 1951 Cisco Systems, Inc. 1952 7200-11 Kit Creek Road 1953 Research Triangle Park, NC 27709 1954 United States 1956 Email: cpignata@cisco.com 1958 Mickey Spiegel 1959 Barefoot Networks, an Intel company 1960 4750 Patrick Henry Drive 1961 Santa Clara, CA 95054 1962 US 1964 Email: mickey.spiegel@intel.com 1966 Barak Gafni 1967 Mellanox Technologies, Inc. 1968 350 Oakmead Parkway, Suite 100 1969 Sunnyvale, CA 94085 1970 U.S.A. 1972 Email: gbarak@mellanox.com 1974 Jennifer Lemon 1975 Broadcom 1976 270 Innovation Drive 1977 San Jose, CA 95134 1978 US 1980 Email: jennifer.lemon@broadcom.com 1982 Hannes Gredler 1983 RtBrick Inc. 1985 Email: hannes@rtbrick.com 1987 John Leddy 1988 United States 1990 Email: john@leddy.net 1991 Stephen Youell 1992 JP Morgan Chase 1993 25 Bank Street 1994 London E14 5JP 1995 United Kingdom 1997 Email: stephen.youell@jpmorgan.com 1999 David Mozes 2001 Email: mosesster@gmail.com 2003 Petr Lapukhov 2004 Facebook 2005 1 Hacker Way 2006 Menlo Park, CA 94025 2007 US 2009 Email: petr@fb.com 2011 Remy Chang 2012 Barefoot Networks 2013 4750 Patrick Henry Drive 2014 Santa Clara, CA 95054 2015 US 2017 Email: remy@barefootnetworks.com 2019 Daniel Bernier 2020 Bell Canada 2021 Canada 2023 Email: daniel.bernier@bell.ca 2025 Authors' Addresses 2027 Frank Brockners (editor) 2028 Cisco Systems, Inc. 2029 Hansaallee 249, 3rd Floor 2030 DUESSELDORF, NORDRHEIN-WESTFALEN 40549 2031 Germany 2033 Email: fbrockne@cisco.com 2034 Shwetha Bhandari (editor) 2035 Cisco Systems, Inc. 2036 Cessna Business Park, Sarjapura Marathalli Outer Ring Road 2037 Bangalore, KARNATAKA 560 087 2038 India 2040 Email: shwethab@cisco.com 2042 Tal Mizrahi (editor) 2043 Huawei 2044 8-2 Matam 2045 Haifa 3190501 2046 Israel 2048 Email: tal.mizrahi.phd@gmail.com