idnits 2.17.1 draft-browne-sfc-nsh-kpi-stamp-07.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** There is 1 instance of too long lines in the document, the longest one being 1 character in excess of 72. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document seems to lack the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords -- however, there's a paragraph with a matching beginning. Boilerplate error? (The document does seem to have the reference to RFC 2119 which the ID-Checklist requires). -- The document date (February 25, 2019) is 1859 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Unused Reference: 'RFC5226' is defined on line 1079, but no explicit reference was found in the text == Unused Reference: 'TS' is defined on line 1090, but no explicit reference was found in the text -- Obsolete informational reference (is this intentional?): RFC 5226 (Obsoleted by RFC 8126) Summary: 1 error (**), 0 flaws (~~), 4 warnings (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 Network Working Group R. Browne 2 Internet Draft A. Chilikin 3 Intended status: Informational Intel 4 Expires: August 2019 T. Mizrahi 5 Huawei Network.IO Innovation Lab 6 February 25, 2019 8 A Key Performance Indicators (KPI) 9 Stamping for the Network Service Header (NSH) 10 draft-browne-sfc-nsh-kpi-stamp-07 12 Abstract 14 This document describes a method of carrying Key Performance 15 Indicators (KPIs) using the Network Service Header (NSH). This method 16 may be used, for example, to monitor latency and QoS marking to 17 identify problems on some links or service functions. 19 Status of this Memo 21 This Internet-Draft is submitted to IETF in full conformance with the 22 provisions of BCP 78 and BCP 79. 24 Internet-Drafts are working documents of the Internet Engineering 25 Task Force (IETF), its areas, and its working groups. Note that 26 other groups may also distribute working documents as Internet- 27 Drafts. 29 Internet-Drafts are draft documents valid for a maximum of six months 30 and may be updated, replaced, or obsoleted by other documents at any 31 time. It is inappropriate to use Internet-Drafts as reference 32 material or to cite them other than as "work in progress." 34 The list of current Internet-Drafts can be accessed at 35 http://www.ietf.org/ietf/1id-abstracts.txt. 37 The list of Internet-Draft Shadow Directories can be accessed at 38 http://www.ietf.org/shadow.html. 40 This Internet-Draft will expire on August 25, 2019. 42 Copyright Notice 44 Copyright (c) 2019 IETF Trust and the persons identified as the 45 document authors. All rights reserved. 47 This document is subject to BCP 78 and the IETF Trust's Legal 48 Provisions Relating to IETF Documents 49 (http://trustee.ietf.org/license-info) in effect on the date of 50 publication of this document. Please review these documents 51 carefully, as they describe your rights and restrictions with respect 52 to this document. Code Components extracted from this document must 53 include Simplified BSD License text as described in Section 4.e of 54 the Trust Legal Provisions and are provided without warranty as 55 described in the Simplified BSD License. 57 Table of Contents 59 1. Introduction ................................................ 2 60 2. Terminology ................................................. 3 61 2.1. Requirement Language ................................... 3 62 2.2. Definition of Terms .................................... 3 63 2.2.1. Terms Defined in this Document .................... 4 64 2.3. Abbreviations .......................................... 4 65 3. NSH KPI Stamping: An Overview ............................... 6 66 3.1. Prerequisites .......................................... 7 67 3.2. Operation ............................................. 10 68 3.2.1. Flow Selection ................................... 10 69 3.2.2. SCP Interface .................................... 10 70 3.3. Performance Considerations ............................ 11 71 4. NSH KPI-stamping Encapsulation ............................. 12 72 4.1. KPI-stamping Extended Encapsulation ................... 13 73 4.1.1. NSH Timestamping Encapsulation (Extended Mode) ... 15 74 4.1.2. NSH QoS-stamping Encapsulation (Extended Mode) ... 17 75 4.2. KPI-stamping Encapsulation (Detection Mode) ........... 20 76 5. Hybrid Models .............................................. 22 77 5.1. Targeted VNF Stamp .................................... 23 78 6. Fragmentation Considerations ............................... 23 79 7. Security Considerations .................................... 24 80 8. IANA Considerations ........................................ 25 81 9. Contributors ............................................... 25 82 10. Acknowledgments ........................................... 25 83 11. References ................................................ 25 84 11.1. Normative References ................................. 25 85 11.2. Informative References ............................... 26 87 1. Introduction 89 Network Service Header (NSH), as defined by [RFC8300], specifies a 90 method for steering the traffic among an ordered set of Service 91 Functions (SFs) using an extensible service header. This allows for 92 flexibility and programmability in the forwarding plane to invoke the 93 appropriate SFs for specific flows. 95 NSH promises a compelling vista of operational flexibility. However, 96 many service providers are concerned about service and configuration 97 visibility. This concern increases when considering that many service 98 providers wish to run their networks seamlessly in 'hybrid' mode, 99 whereby they wish to mix physical and virtual SFs and run services 100 seamlessly between the two domains. 102 This document describes generic methods to monitor and debug service 103 function chains in terms of latency and QoS marking of the flows 104 within a service function chain. This is refered to detection mode 105 and extended mode and explained in section 4. 107 The method described in the document is compliant with hybrid 108 architectures in which Virtual Network Functions (VNFs) and Physical 109 Network Functions (PNFs) are freely mixed in the service function 110 chain. This method also provides flexibility to monitor the 111 performance and configuration of an entire chain or part thereof as 112 desired. This method is extensible to monitoring other KPIs. Please 113 refer to [RFC7665] for an architectural context for this document. 115 The method described in this document is not an OAM protocol such as 116 [Y.1731] or [Y.1564]. As such it does not define new OAM packet types 117 or operation. Rather it monitors the service function chain 118 performance and configuration for subscriber payloads and indicates 119 subscriber QoE rather than out-of-band infrastructure metrics. This 120 document differs from to [I-D.ippm.ioam] in the sense that it is 121 specifically tied to NSH operation and not generic in nature. 123 2. Terminology 125 2.1. Requirement Language 127 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 128 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 129 document are to be interpreted as described in BCP 14 130 [RFC2119][RFC8174] when, and only when, they appear in all capitals, 131 as shown here. 133 2.2. Definition of Terms 135 This section presents the main terms used in this document. This 136 document makes use of the terms defined in [RFC7665] and [RFC8300]. 138 2.2.1. Terms Defined in this Document 140 First Stamping Node (FSN): The first node along a service function 141 chain that stamps packets using KPI stamping. The FSN matches each 142 packet with a Stamping Controller flow based on a stamping 143 classification criterion such as transport 5-tuple coordinates, but 144 not limited to. 146 Last Stamping Node (LSN): The last node along a service function 147 chain that stamps packets using KPI stamping. From forwarding point 148 of view the LSN removes the NSH header and forwards the raw IP packet 149 to the next hop. From a control plane point of view the LSN reads all 150 the metadata and exports it to a system performance statistics agent 151 or repository. The LSN should use the NSH Service Index (SI) to 152 indicate if a SF was at the end of the chain. The LSN may change the 153 Service Path Identifier (SPI) to preconfigured value in order that 154 the network underlay forwards the metadata back directly to the KPI 155 database (KPIDB) based on this value. 157 Key Performance Indicator Database (KPIDB): denotes the external 158 storage of metadata for reporting, trend analysis, etc. 160 KPI-stamping: The insertion of latency-related and/or QoS-related 161 information into a packet using NSH metadata. 163 Flow ID: The Flow ID is a unique 16 bit identifier written into the 164 header by the classifier. This allows 65536 flows to be concurrently 165 stamped on any given NSH service chain (SPI). 167 QoS-stamping: The insertion of QoS-related information into a packet 168 using NSH metadata. 170 Stamping Controller (SC): The SC is the central logic that decides 171 what packets to stamp and how. The SC instructs the classifier on how 172 to build the KPI stamp specific parts of the NSH. 174 Stamp Control Plane (SCP): the control plane between the FSN and the 175 SC. 177 2.3. Abbreviations 179 DEI Drop Eligible Indicator 181 DSCP Differentiated Services Code Point 183 FSN First Stamping Node 184 KPI Key Performance Indicator 186 KPIDB Key Performance Indicator Database 188 LSN Last Stamping Node 190 MD Metadata 192 NFV Network Function Virtualization 194 NFVI-PoP NFV Infrastructure Point of Presence 196 NIC Network Interface Card 198 NSH Network Service Header 200 OAM Operations, Administration, and Maintenance 202 PCP Priority Code Point 204 PNF Physical Network Function 206 PNFN Physical Network Function Node 208 QoE Quality of Experience 210 QoS Quality of Service 212 QS QoS Stamp 214 RSP Rendered Service Path 216 SC Stamping Controller 218 SCL Service Classifier 220 SCP Stamp Control Plane 222 SI Service Index 224 SF Service Function 226 SFC Service Function Chain 228 SFP Service Function Path 230 SSI Stamp Service Index 231 TC Traffic Class 233 TS Timestamp 235 VLAN Virtual Local Area Network 237 VNF Virtual Network Function 239 vSwitch Virtual Switch 241 3. NSH KPI Stamping: An Overview 243 A typical KPI stamping architecture is presented in Figure 1. 245 Stamping 246 Controller 247 | KPIDB 248 | SCP Interface | 249 ,---. ,---. ,---. ,---. 250 / \ / \ / \ / \ 251 ( SCL )-------->( SF1 )--------->( SF2 )--------->( SFn ) 252 \ FSN / \ / \ / \ LSN / 253 `---' `---' `---' `---' 254 Figure 1: Logical roles in NSH KPI Stamping 256 The Stamping Controller (SC) will be part of the SFC control plane 257 architecture, but it is described separately in this document for 258 clarity. 260 The SC is responsible for initiating start/stop stamp requests to the 261 SCL or First Stamp Node (FSN), and also for distributing NSH stamping 262 policy into the service chain via the Stamping Control Plane (SCP) 263 interface. 265 The FSN will typically be part of the SCL, but again is called out as 266 separate logical entity for clarity. 268 The FSN is responsible for marking NSH MD fields which tells nodes in 269 the service chain how to behave in terms of stamping at SF ingress, 270 egress or both, or ignoring the stamp NSH metadata completely. 272 The FSN also writes the Reference Time value, a (possibly inaccurate) 273 estimate of the current time-of-day, into the header, allowing the 274 {SPI:Flow ID} performance to be compared to previous samples for 275 offline analysis. 277 The FSN should return an error to the SC if not synchronized to the 278 current time-of-day and forward the packet along the service-chain 279 unchanged. The code and format of the error is specific to the 280 protocol used between the FSN and SC; these considerations are out of 281 scope. 283 SF1 and SF2 stamp the packets as dictated by the FSN and process the 284 payload as per normal. 286 Note 1: The exact location of the stamp creation may not be in 287 the SF itself, as discussed in Section 3.3. 289 Note 2: Special cases exist where some of the SFs are 290 NSH-unaware. This is covered in Section 5. 292 The Last Stamp Node (LSN) should strip the entire NSH header and 293 forward the raw packet to the IP next hop as per [RFC8300]. The LSN 294 also exports NSH stamp information to the KPI Database (KPIDB) for 295 offline analysis; the LSN may either export the stamping information 296 of all packets, or a subset based on packet sampling. 298 In fully virtualized environments the LSN is likely to be co-located 299 with the SF that decrements the NSH Service Index (SI) to zero. 300 Corner cases exist whereby this is not the case and is covered in 301 Section 5. 303 3.1. Prerequisites 305 Timestamping presents a set of prerequisites not required to QoS- 306 Stamp. In order to guarantee metadata accuracy, all servers hosting 307 VNFs should be synchronized from a centralized stable clock. As it is 308 assumed that PNFs do not timestamp (as this would involve a software 309 change and probable throughput performance impact) there is no need 310 for them to synchronize. There are two possible levels of 311 synchronization: 313 Level A: Low accuracy time-of-day synchronization, based on 314 NTP [RFC5905]. 316 Level B: High accuracy synchronization (typically on the order of 317 microseconds), based on [IEEE1588]. 319 Each SF SHOULD have a level A synchronization, and MAY have a level B 320 synchronization. 322 Level A requires each platform (including the Stamp Controller) to 323 synchronize its system real-time-clock to an NTP server. This is used 324 to mark the metadata in the chain, using the field 325 in the NSH KPI-stamp header (Section 4.2). This timestamp is inserted 326 to the NSH by the first SF in the chain. NTP accuracy can vary by 327 several milliseconds between locations. This is not an issue as the 328 Reference Time is merely being used as a time-of-day reference 329 inserted into the KPIDB for performance monitoring and metadata 330 retrieval. 332 Level B synchronization requires each platform to be synchronized to 333 a Primary Reference Time Clock (PRTC) using the Precision Time 334 Protocol [IEEE1588]. A platform MAY also use Synchronous Ethernet 335 ([G.8261], [G.8262], [G.8264]), allowing more accurate frequency 336 synchronization. 338 If an SF is not synchronized at the moment of timestamping, it should 339 indicate its synchronization status in the NSH. This is described in 340 more detail in Section 4. 342 By synchronizing the network in this way, the timestamping operation 343 is independent of the current Rendered Service Path (RSP). Indeed the 344 timestamp metadata can indicate where a chain has been moved due to a 345 resource starvation event as indicated in Figure 2, between VNF 3 and 346 VNF 4 at time B. 348 Delay 349 | v 350 | v 351 | x 352 | x x = reference time A 353 | xv v = reference time B 354 | xv 355 | xv 356 |______|______|______|______|______|_____ 357 VNF1 VNF2 VNF3 VNF4 VNF5 359 Figure 2: Flow performance in a service chain 361 For QoS-stamping it is desired that the SCL or FSN be synchronized in 362 order to provide reference time for offline analysis, but this is not 363 a hard requirement (they may be in holdover or free-run state, for 364 example). Other SFs in the service chain do not need to be 365 synchronized for QoS-stamping operation as described below. 367 QoS-stamping can be used to check consistency of configuration across 368 the entire chain or part thereof. By adding all potential layer 2 and 369 layer 3 QoS fields into a QoS sum at SF ingress or egress, this 370 allows quick identification of QoS mismatches across multiple L2/L3 371 fields which otherwise is a manual, expert-led consuming process. 373 | 374 | 375 | xy 376 | xy x = ingress QoS sum 377 | xv v = egress QoS sum 378 | xv y = egress QoS sum miss 379 | xv 380 |______|______|______|______|______|_____ 381 SF1 SF2 SF3 SF4 SF5 383 Figure 3: Flow QoS Consistency in a service chain 385 Referring to Figure 3, x, v, and y are notional sum values of the QoS 386 marking configuration of the flow within a given chain. As the 387 encapsulation of the flow can change from hop to hop in terms of VLAN 388 header(s), MPLS labels, DSCP(s) these values are used to compare 389 consistency of configuration from for example payload DSCP through 390 overlay and underlay QoS settings in VLAN IEEE 802.1Q bits, MPLS bits 391 and infrastructure DSCPs. 393 Figure 3 indicates that at SF4 in the chain, the egress QoS marking 394 is inconsistent. That is, the ingress QoS settings do not match the 395 egress. The method described here will indicate which QoS field(s) is 396 inconsistent, and whether this is ingress (whereby the underlay has 397 incorrectly marked and queued the packet) or egress (where the SF has 398 incorrectly marked and queued the packet. 400 Note that the SC must be aware of when a SF remarks QoS fields 401 deliberately and thus does not flag an issue for desired behavior. 403 3.2. Operation 405 KPI-stamping detection mode uses MD type 2 defined in [RFC8300]. This 406 involves the SFC classifier stamping the flow at chain ingress, and 407 no subsequent stamps being applied, rather each SF upstream can 408 compare its local condition with the ingress value and take 409 appropriate action. Therefore detection mode is very efficient in 410 terms of header size that does not grow after the classification. 411 This is further explained in Section 4.1. 413 3.2.1. Flow Selection 415 The SC should maintain a list of flows within each service chain to 416 be monitored. This flow table should be in the format 'SPI:FlowID'. 417 The SC should map these pairs to unique values presented as Flow IDs 418 per service chain within the NSH TLV specified in this document. The 419 SC should instruct the FSN to initiate timestamping on flow table 420 match. The SC may also tell the classifier the duration of the 421 timestamping operation, either by a number of packets in the flow or 422 by a time duration. 424 In this way the system can monitor the performance of the all en- 425 route traffic, or an individual subscriber in a chain, or just a 426 specific application or a QoS class that is used in the network. 428 The SC should write the list of monitored flows into the KPIDB for 429 correlation of performance and configuration data. Thus, when the 430 KPIDB receives data from the LSN it understands to which flow the 431 data pertains. 433 The association of source IP to subscriber identity is outside the 434 scope of this document and will vary by network application. For 435 example, the method of association of a source IP to IMSI will be 436 different to how a CPE with NAT function may be chained in an 437 enterprise NFV application. 439 3.2.2. SCP Interface 441 A Stamp Control Plane (SCP) interface is required between the SC and 442 the FSN or classifier. This interface is used to: 444 o Query the SFC classifier for a list of active chains and flows. 446 o Communicate which chains and flows to stamp. This can be a 447 specific {SPI:Flow ID} combination or include wildcards for 448 monitoring subscribers across multiple chains or multiple flows 449 within one chain. 451 o Instruct how the stamp should be applied (ingress, egress, both 452 or specific). 454 o Indicate when to stop stamping, either after a certain number 455 of packets or duration. 457 Typically SCP timestamps flows for a certain duration for trend 458 analysis, but only stamps one packet of each QoS class in a chain 459 periodically (perhaps once per day or after a network change). 460 Therefore, timestamping is generally applied to a much larger set of 461 packets than QoS-stamping. 463 Exact specification of SCP is for further study. 465 3.3. Performance Considerations 467 This document does not mandate a specific stamping implementation 468 method, and thus NSH KPI stamping can either be performed by hardware 469 mechanisms, or by software. 471 If software-based stamping is used, applying and operating on the 472 stamps themselves incur an additional small delay in the service 473 chain. However, it can be assumed that these additional delays are 474 all relative for the flow in question. This is only pertinent for 475 timestamping mode, and not for QoS-stamping mode. Thus, whilst the 476 absolute timestamps may not be fully accurate for normal non- 477 timestamped traffic they can be assumed to be relative. 479 It is assumed that the method described in this document would only 480 operate on a small percentage of user flows. 482 The service provider may choose a flexible policy in the SC to 483 timestamp a selection of user-plane every minute for example to 484 highlight any performance issues. Alternatively, the LSN may 485 selectively export a subset of the KPI-stamps it receives, based on a 486 predefined sampling method. Of course the SC can stress test an 487 individual flow or chain should a deeper analysis be required. We can 488 expect that this type of deep analysis has an impact on the 489 performance of the chain itself whilst under investigation. The 490 impact will be dependent on vendor implementation and outside the 491 scope of this document. 493 For QoS-stamping the method described here is even less intrusive, as 494 typically multiple packets in a flow are QoS stamped periodically 495 (perhaps once per day) check one packet in a chain per QoS class. 497 4. NSH KPI-stamping Encapsulation 499 KPI stamping uses NSH MD type 0x2 for detection of anomalies and 500 extended mode for root cause analysis of KPI violations. These are 501 further explained in this section. 503 The generic NSH MD type 2 TLV for KPI Stamping is shown below. 505 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 506 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 507 |Ver|O|U| TTL | Length |U|U|U|U|Type=2 | Next Protocol | 508 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 509 | Service Path Identifier | Service Index | 510 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 511 | Metadata Class | Type |U| Length | 512 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 513 | Variable-length KPI Metadata header and TLV(s) | 514 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 515 Figure 4: Generic NSH KPI Encapsulation 517 Relevant fields in header that the FSN must implement: 519 o The O bit must not be set. 521 o The MD type must be set to 0x2 523 o The Metadata Class must be set to a value from the experimental 524 range 0xfff6 to 0xfffe according to an agreement by all parties to 525 the experiment. 527 o Unassigned bits: all fields, marked U, are unassigned and 528 available for future use [RFC8300] 530 o The Type field may have one of the following values; the 531 content of "KPI metadata" depends on the type value: 533 o Type = 0x01 Det: Detection 535 o Type = 0x02 TS: Timestamp Extended 537 o Type = 0x03 QoS: QoS-stamp Extended 539 The Type field determines the type of KPI-stamping format. The 540 supported formats are presented in the following subsections. 542 4.1. KPI-stamping Extended Encapsulation 544 The generic NSH MD type 2 KPI-stamping header extended mode is shown 545 in Figure 5. This is the format for performance monitoring of service 546 chain issues with respect to QoS configuration and latency. 548 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 549 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 550 |Ver|O|U| TTL | Length |U|U|U|U|Type=2 | Next Protocol | 551 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 552 | Service Path Identifier | Service Index | 553 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 554 | Metadata Class | Type |U| Length | 555 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 556 | Variable Length KPI Configuration Header | 557 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 558 | Variable Length KPI Value (LSN) | 559 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 560 \ \ 561 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 562 | Variable Length KPI Value (FSN) | 563 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 564 Figure 5: Generic KPI Encapsulation (Extended Mode) 566 As mentioned above, two types are defined under the experimental MD 567 class to indicate extended KPI MD: a timestamp type and a QoS-stamp 568 type. 570 The KPI Encapsulation Configuration Header format is shown below. 572 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 573 |K|K|T|K|K|K|K|K| Stamping SI | Flow ID | 574 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 575 | Reference Time | 576 | | 577 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 578 Figure 6: KPI Encapsulation Configuration Header 580 The bits marked as 'K' are reserved for specific KPI type use and 581 described in the corresponding subsections below. 583 The T bit should be set if Reference Time follows KPI Encapsulation 584 Configuration Header. 586 Stamping Service Index (Stamping SI) contains the Service Index used 587 for KPI stamping and described in the corresponding subsections 588 below. 590 The Flow ID is a unique 16 bit identifier written into the header by 591 the classifier. This allows 65536 flows to be concurrently stamped on 592 any given NSH service chain (SPI). Flow IDs are not written by 593 subsequent SFs in the chain. The FSN may export monitored flow IDs to 594 the KPIDB for correlation. 596 Reference Time is the wall clock of the FSN, and may be used for 597 historical comparison of SC performance. If the FSN is not Level A 598 synchronized (see Section 3.1) it should inform the SC over the SCP 599 interface. The Reference Time is represented in 64-bit NTP format 600 [RFC5905] presented in Figure 7: 602 0 1 2 3 603 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 604 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 605 | Seconds | 606 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 607 | Fraction | 608 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 609 Figure 7: NTP [RFC5905] 64-bit Timestamp Format 611 4.1.1. NSH Timestamping Encapsulation (Extended Mode) 613 The NSH timestamping extended encapsulation is shown below. 615 0 1 2 3 616 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 617 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 618 |Ver|O|C|U|U|U|U|U|U| Length |U|U|U|U|Type=2 | NextProto | 619 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 620 | Service Path ID | Service Index | 621 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 622 | Metadata Class | Type=TS(2) |U| Len | 623 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 624 |I|E|T|U|U|U|SSI| Stamping SI | Flow ID | 625 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-| 626 | Reference Time (T bit is set) | 627 | | 628 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 629 |I|E|U|U|U| SYN | Stamping SI | Unassigned | 630 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-| 631 | Ingress Timestamp (I bit is set)(LSN) | 632 | | 633 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 634 | Egress Timestamp (E bit is set)(LSN) | 635 | | 636 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 637 . . 638 . . 639 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 640 |I|E|U|U|U| SYN | Stamping SI | Unassigned | 641 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-| 642 | Ingress Timestamp (I bit is set) (FSN) | 643 | | 644 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 645 | Egress Timestamp (E bit is set) (FSN) | 646 | | 647 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 648 Figure 8: NSH Timestamp Encapsulation (Extended Mode) 650 The FSN KPI-stamp metadata starts with Stamping Configuration Header. 651 This header contains the I, E, T bits and Stamp Service Index (SSI). 653 The I bit should be set if Ingress stamp is requested. 655 The E bit should be set if Egress stamp is requested. 657 SSI field must be set to one of the following values: 659 o 0x0 KPI-stamp mode, no Service index specified in the Stamp 660 Service Index field. 662 o 0x1 KPI-stamp Hybrid mode is selected, Stamp Service Index 663 contains LSN Service index. This is used when PNFs or NSH-unaware 664 SFs are used at the tail of the chain. If SSI=0x1, then the value 665 in the type field informs the chain which SF should act as the 666 LSN. 668 o 0x2 KPI-stamp Specific mode is selected, Stamp Service Index 669 contains the targeted Service Index. In this case the Stamp 670 Service Index field indicates which SF is to be stamped. Both 671 ingress and egress stamps are performed when the SI=SSI on the 672 chain. For timestamping mode, the FSN will also apply the 673 Reference Time and Ingress Timestamp. This will indicate the delay 674 along the entire service chain to the targeted SF. This method may 675 also be used as a light implementation to monitor end-to-end 676 service chain performance whereby the targeted SF is the LSN. This 677 is not applicable to QoSStamping mode. 679 Each stamping Node adds stamping metadata which consist of Stamping 680 Reporting Header and timestamps. 682 The E bit should be set if Egress stamp is reported. 684 The I bit should be set if Ingress stamp is reported. 686 With respect to timestamping mode, the SYN bits are an indication of 687 the synchronization status of the node performing the timestamp and 688 must be set to one of the following values: 690 o In Synch: 0x00 692 o In holdover: 0x01 694 o In free run: 0x02 696 o Out of Synch: 0x03 697 If the platform hosting the SF is out of synch or in free run no 698 timestamp is applied by the node and the packet is processed 699 normally. 701 If FSN is out of synch or in free run timestamp request rejected and 702 not propagated though the chain. The FSN should inform the SC in such 703 an event over the SCP interface. Similarly if KPIDB receives 704 timestamps that are out of order (i.e. a time stamp of a 'N+1' SF is 705 in the past of a 'N' SF) it should notify SC of this condition over 706 the SCP interface. 708 The outer service index value is copied into the stamp metadata as 709 Stamping SI to help cater for hybrid chains that are a mix of VNFs 710 and PNFs or through NSH-unaware SFs. Thus, if a flow transits through 711 a PNF or an NSH-unaware node the delta in the inner service index 712 between timestamps will indicate this. 714 The Ingress Timestamp and Egress Timestamp are represented in 64-bit 715 NTP format. The corresponding bits (I and E) reported in the Stamping 716 Reporting Header of the node's metadata. 718 4.1.2. NSH QoS-stamping Encapsulation (Extended Mode) 720 Packets have a variable QoS stack. That is for example the same 721 payload IP can have a very different stack in the access part of the 722 network to the core. This is most apparent in mobile networks where 723 for example in an access circuit we would have 2 layers of 724 infrastructure IP header (DSCP) - one transport-based and the other 725 IPsec-based, in addition to multiple MPLS and VLAN tags. The same 726 packet as it leaves the PDN Gateway Gi egress interface may be very 727 much simplified in terms of overhead and related QoS fields. 729 Because of this variability we need to build extra meaning into the 730 QoS headers - they are not for example all PTP timestamps of a fixed 731 length as in the case of timestamping, rather they are variable 732 lengths and types. Also they can be changed on the underlay at any 733 time without knowledge by the SFC system. Therefore each SF must be 734 able to ascertain and record its ingress and egress QoS configuration 735 on the fly. 737 The suggested QoS type, lengths are as below. The type is 4 bits 738 long. 740 QoS Type(QT)Value Length Comment 742 IVLAN 0x01 4 Bits Ingress VLAN (PCP + DEI) 744 EVLAN 0x02 4 Bits Egress VLAN 746 IQINQ 0x03 8 Bits Ingress QinQ (2x PCP+DEI) 748 EQINQ 0x04 8 Bits Egress QinQ 750 IMPLS 0x05 3 Bits Ingress Label 752 EMPLS 0x06 3 Bits Egress Label 754 IMPLS 0x07 6 Bits 2 Ingress Labels (2x EXP) 756 EMPLS 0x08 6 Bits 2 Egress Labels 758 IDSCP 0x09 8 Bits Ingress DSCP 760 EDSCP 0x0A 8 Bits Egress DSCP 762 For stacked headers such as MPLS and 802.1ad, we extract the QoS 763 relevant data from the header and insert into one QoS value in order to 764 be more efficient on packet size. Thus for MPLS, we represent both EXP 765 fields in one QoS value, and both 802.1p priority and drop precedence in 766 one QoS value as indicated above. 768 For stack types not listed here, for example, 3 or more MPLS tags, SF 769 would insert IMPLS/EMPLS several times with each layer in the stack 770 indicating EXP Qos for that layer. 772 0 1 2 3 773 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 774 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 775 |Ver|O|C|U|U|U|U|U|U| Length |U|U|U|U|Type=2 | NextProto=0x0 | 776 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 777 | Service Path ID | Service Index | 778 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 779 | Metadata Class | Type=QoS(3) |U| Len | 780 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 781 |U|U|T|U|U|U|SSI| Stamping SI | Flow ID | 782 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-| 783 | Reference Time (T bit is set) | 784 | | 785 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 786 |U|U|U|U|U|U|U|U| Stamping SI | Unassigned | 787 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-| 788 | QT | QoS Value |U|U|U|E| QT | QoS Value |U|U|U|E| 789 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 790 . . 791 . . 792 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 793 |U|U|U|U|U|U|U|U| Stamping SI | Unassigned | 794 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-| 795 | QT | QoS Value |U|U|U|E| QT | QoS Value |U|U|U|E| 796 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 797 Figure 9: NSH QoS Configuration Encapsulation (Extended Mode) 799 The encapsulation in Figure 10 is very similar to that detailed in 800 Section 4.1 with the following exceptions: 802 - I and E bits are not required as we wish to examine the full QoS 803 stack at ingress and egress at every SF. 805 - Syn status bits are not required. 807 - The QT (QoS Type) and QoS value are as outlined in the table 808 above. 810 - The E bit at the tail of each QoS context field indicates if this 811 is the last egress QoS-stamp for a given SF. This should coincide 812 with SI=0 at the LSN, whereby the packet is truncated and the NSH 813 MD sent to the KPIDB and the subscriber raw IP packet forwarded to 814 the underlay next hop. 816 Note: It is possible to compress the frame structure to better 817 utilize the header, but this would come at the expense of crossing 818 byte boundaries. For ease of implementation, and that QoS-stamping is 819 applied on an extremely small subset of user plane traffic, we 820 believe the above structure is a pragmatic compromise between header 821 efficiency and ease of implementation. 823 4.2. KPI-stamping Encapsulation (Detection Mode) 825 The format of the NSH MD type 2 KPI-stamping TLV (detection mode) is 826 shown in Figure 11. 828 This TLV is used for KPI anomaly detection. Upon detecting a problem 829 or an anomaly it will be possible to enable the use of KPI-stamping 830 extended encapsulations, which will provide more detailed analysis. 832 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 833 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 834 |Ver|O|U| TTL | Length |U|U|U|U|Type=2 | Next Protocol | 835 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 836 | Service Path Identifier | Service Index | 837 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 838 | Metadata Class | Type=Det(1) |U| Length | 839 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 840 | KPI Type | Stamping SI | Flow ID | 841 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 842 | Threshold KPI Value | 843 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 844 | Ingress KPI-stamp | 845 | | 846 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 847 Figure 10: Generic NSH KPI Encapsulation (Detection Mode) 849 The following fields are defined in the KPI TSD metadata: 851 o KPI Type: determines the type of KPI-stamp that is included in 852 this metadata field. 853 If a receiver along the path does not understand the KPI Type it 854 will pass the packet transparently and not drop. 855 The supported values of the KPI Type are: 856 0x0 Timestamp 857 0x1 QoS-stamp 859 o Threshold KPI Value: In the first header the SFC classifier may 860 program a KPI threshold value. This is a value that when exceeded, 861 requires the SF to insert the current SI value into the SI field. 862 The KPI type is the type of KPI stamp inserted into the header as 863 per section 9. 865 o Stamping SI: Service Identifier of the SF when the Threshold 866 above is exceeded. 868 o Flow ID: The flow ID is inserted into the header by the SFC 869 classifier in order to correlate flow data in the KPIDB for 870 offline analysis. 872 o Ingress KPI-stamp: The last 8 octets are reserved for the KPI- 873 stamp. This is the KPI value at the chain ingress at the SFC 874 classifier. Depending on the KPI Type, the KPI-stamp either 875 includes a timestamp or a QoS-stamp. 876 If the KPI Type is Timestamp, then the Ingress KPI-stamp field 877 contains a timestamp in 64-bit NTP timestamp format. If the KPI 878 Type is QoS-stamp, then the format of the 64-bit Ingress KPI-stamp 879 is as follows. 881 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 882 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 883 | QT | QoS Value | Unassigned | 884 +-+-+-+-+-+-+-+-+-+-+-+-+ + 885 | | 886 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 887 Figure 11: QoS-stamp Format (Detection Mode) 889 As an example operation, say we are using KPI type 0x01 (timestamp) 890 when a service function (SFn) receives the packet it can compare 891 current local timestamp (it first checks that it is synchronized to 892 network PRC) with chain ingress timestamp to calculate the latency in 893 the chain. If this value exceeds the timestamp threshold, it then 894 inserts its SI and returns the NSH to the KPIDB. This effectively 895 tells the system that at SFn the packet violated the KPI threshold. 896 Please refer to figure 9 for timestamp format. 898 When this occurs the SFC control plane system would then invoke the 899 KPI extended mode, which uses a more sophisticated (and intrusive) 900 method to isolate KPI violation root cause as described below. 902 Note: Whilst detection mode is a valuable tool for latency actions, 903 the authors feel that it is not justified to build the logic into the 904 KPI system for QoS configuration. As QoS-stamping is done 905 infrequently and on a tiny percentage of user plane, it is more 906 practical to use extended mode only for service chain QoS 907 verification. 909 5. Hybrid Models 911 A hybrid chain may be defined as a chain whereby there is a mix of 912 NSH-aware and NSH-unaware SFs. 914 Example 1. PNF in the middle 916 Stamp 917 Controller 918 | KPIDB 919 | SCP Interface | 920 ,---. ,---. ,---. ,---. 921 / \ / \ / \ / \ 922 ( SCL )-------->( SF1 )--------->( SF2 )--------->( SFn ) 923 \ FSN / \ / \ PNF1/ \ LSN / 924 `---' `---' `---' `---' 925 Figure 12: Hybrid chain with PNF in middle 927 In this example the FSN begins operation and sets the SI to 3, SF1 928 decrements this to 2 and passes the packet to an SFC proxy (not 929 shown). 931 The SFC proxy strips the NSH and passes to the PNF. On receipt back 932 from the PNF, the proxy decrements the SI and passes the packet onto 933 the LSN with a SI=1. 935 After the LSN processes the traffic it knows it is the last node on 936 the chain from the SI value and exports the entire NSH and all 937 metadata to the KPIDB. The payload is forwarded to the next hop on 938 the underlay minus the NSH. The stamping information packet may be 939 given a new SPI to act as a homing tag to transport the stamp data 940 back to the KPIDB. 942 Example 2. PNF at the end 944 Stamp 945 Controller 946 | KPIDB 947 | SCP Interface | 948 ,---. ,---. ,---. ,---. 949 / \ / \ / \ / \ 950 ( SCL )-------->( SF1 )--------->( SF2 )--------->( PNFN ) 951 \ FSN / \ / \ LSN / \ / 952 `---' `---' `---' `---' 953 Figure 13: Hybrid Chain with PNF at end 955 In this example the FSN begins operation and sets the SI to 3, the 956 SSI field set to 0x1, and the type to 1. Thus, when SF2 receives the 957 packet with SI=1, it understands that it is expected to take on the 958 role of the LSN as it is the last NSH-aware node in the chain. 960 5.1. Targeted VNF Stamp 962 For the majority of flows within the service chain, stamps (ingress, 963 egress or both) will be carried out at each hop until the SI 964 decrements to zero and the NSH and Stamp MD is exported to the KPIDB. 965 There may exist however the need to just test a particular VNF 966 (perhaps after a scale out operation, software upgrade or underlay 967 change for example). In this case the FSN should mark the NSH as 968 follows: 970 SSI field is set to 0x2. Type is set to the expected SI at the SF in 971 question. When outer SI is equal to the SSI, stamps are applied at SF 972 ingress and egress, and the NSH and MD are exported to the KPIDB. 974 6. Fragmentation Considerations 976 The method described in this document does not support fragmentation. 977 The SC should return an error should a stamping request from an 978 external system exceed MTU limits and require fragmentation. 980 Depending on the length of the payload and the type of KPI-stamp and 981 chain length, this will vary for each packet. 983 In most service provider architectures we would expect a SI << 10, 984 and that may include some PNFs in the chain which do not add 985 overhead. Thus for typical IMIX packet sizes we expect to able to 986 perform timestamping on the vast majority of flows without 987 fragmenting. Thus the classifier can have a simple rule to only allow 988 KPI-stamping on packet sizes less than 1200 bytes for example. 990 7. Security Considerations 992 The security considerations of NSH in general are discussed in 993 [RFC8300]. 995 The use of in-band timestamping, as defined in this document, can be 996 used as a means for network reconnaissance. By passively 997 eavesdropping to timestamped traffic, an attacker can gather 998 information about network delays and performance bottlenecks. 1000 The NSH timestamp is intended to be used by various applications to 1001 monitor the network performance and to detect anomalies. Thus, a man- 1002 in-the-middle attacker can maliciously modify timestamps in order to 1003 attack applications that use the timestamp values. For example, an 1004 attacker could manipulate the SFC classifier operation, such that it 1005 forwards traffic through 'better' behaving chains. Furthermore, if 1006 timestamping is performed on a fraction of the traffic, an attacker 1007 can selectively induce synthetic delay only to timestamped packets, 1008 causing systematic error in the measurements. 1010 Similarly, if an attacker can modify QoS stamps, erroneous values may 1011 be imported into the KPIDB, resulting is further misconfiguration and 1012 subscriber QoE impairment. 1014 An attacker that gains access to the SCP can enable time and QoS- 1015 stamping for all subscriber flows, thereby causing performance 1016 bottlenecks, fragmentation, or outages. 1018 As discussed in previous sections, NSH timestamping relies on an 1019 underlying time synchronization protocol. Thus, by attacking the time 1020 protocol an attack can potentially compromise the integrity of the 1021 NSH timestamp. A detailed discussion about the threats against time 1022 protocols and how to mitigate them is presented in [RFC7384]. 1024 8. IANA Considerations 1026 This document makes no requests for IANA action, 1028 9. Contributors 1030 This document originated as draft-browne-sfc-nsh-timestamp-00 and had 1031 the following co-authors and contributors. We would like to thank and 1032 recognize them and their contributions. 1034 Yoram Moses 1036 Technion 1038 moses@ee.technion.ac.il 1040 Brendan Ryan 1042 Intel Corporation 1044 brendan.ryan@intel.com 1046 10. Acknowledgments 1048 This document was prepared using 2-Word-v2.0.template.dot. 1050 The authors gratefully acknowledge Mohamed Boucadair, Martin 1051 Vigoureux and Adrian Farrel for their thorough reviews and helpful 1052 comments. 1054 11. References 1056 11.1. Normative References 1058 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1059 Requirement Levels", BCP 14, RFC 2119, March 1997. 1061 [RFC7665] Halpern, J., Ed., and C. Pignataro, Ed., "Service 1062 Function Chaining (SFC) Architecture", RFC 7665, DOI 1063 10.17487/RFC7665, October 2015, . 1066 [RFC8174] B. Leiba, "Ambiguity of Uppercase vs Lowercase in RFC 1067 2119 Key Words", RFC8174, May 2017. 1069 [RFC8300] Quinn, P., Elzur, U., Pignataro, C., "Network Service 1070 Header (NSH)", RFC 8300, 2018. 1072 11.2. Informative References 1074 [IEEE1588] IEEE TC 9 Instrumentation and Measurement Society, 1075 "1588 IEEE Standard for a Precision Clock 1076 Synchronization Protocol for Networked Measurement and 1077 Control Systems Version 2", IEEE Standard, 2008. 1079 [RFC5226] Narten, T. and H. Alvestrand, "Guidelines for Writing 1080 an IANA Considerations Section in RFCs", BCP 26, RFC 1081 5226, May 2008. 1083 [RFC5905] Mills, D., Martin, J., Burbank, J., Kasch, W., 1084 "Network Time Protocol Version 4: Protocol and 1085 Algorithms Specification", RFC 5905, June 2010. 1087 [RFC7384] Mizrahi, T., "Security Requirements of Time Protocols 1088 in Packet Switched Networks", RFC 7384, October 2014. 1090 [TS] Mizrahi, T., Fabini, J., and A. Morton, "Guidelines 1091 for Defining Packet Timestamps", draft-ietf-ntp- 1092 packet-timestamps (work in progress), 2018. 1094 [Y.1731] ITU-T Recommendation G.8013/Y.1731, "OAM Functions and 1095 Mechanisms for Ethernet-based Networks", August 2015. 1097 [Y.1564] ITU-T Recommendation Y.1564, "Ethernet service 1098 activation test methodology", March 2011. 1100 [G.8261] ITU-T Recommendation G.8261/Y.1361, "Timing and 1101 synchronization aspects in packet networks", August 1102 2013. 1104 [G.8262] ITU-T Recommendation G.8262/Y.1362, "Timing 1105 characteristics of a synchronous Ethernet equipment 1106 slave clock", January 2015. 1108 [G.8264] ITU-T Recommendation G.8264/Y.1364, "Distribution of 1109 timing information through packet networks", May 2014. 1111 [I-D.ippm.ioam] 1112 Brockners, Bhandari et al. "Data Fields for In-situ OAM" 1113 draft-ietf-ippm-ioam-data-03 (work in progress), June 1114 2018 1116 Authors' Addresses 1118 Rory Browne 1119 Intel 1120 Dromore House 1121 Shannon 1122 Co.Clare 1123 Ireland 1125 Email: rory.browne@intel.com 1127 Andrey Chilikin 1128 Intel 1129 Dromore House 1130 Shannon 1131 Co.Clare 1132 Ireland 1134 Email: andrey.chilikin@intel.com 1136 Tal Mizrahi 1137 Huawei Network.IO Innovation Lab 1138 Israel 1140 Email: tal.mizrahi.phd@gmail.com