idnits 2.17.1 draft-bcx-rtgwg-tcr-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document seems to lack the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords. (The document does seem to have the reference to RFC 2119 which the ID-Checklist requires). -- The document date (April 12, 2021) is 1103 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Unused Reference: 'RFC2119' is defined on line 1702, but no explicit reference was found in the text == Outdated reference: A later version (-17) exists of draft-ietf-ippm-ioam-data-11 Summary: 0 errors (**), 0 flaws (~~), 4 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 rtgwg S. Bryant 3 Internet-Draft A. Clemm 4 Intended status: Informational Futurewei Technologies, Inc. 5 Expires: October 14, 2021 April 12, 2021 7 Token Cell Routing Data Plane Concepts 8 draft-bcx-rtgwg-tcr-00 10 Abstract 12 Token Cell Routing is a powerful yet hardware friendly method of 13 constructing data plane packets to meet the needs of new 14 applications. It is based on the use of token cells (special kinds 15 of lightly structured tokens) to provide pointers to procedures pre- 16 positioned in the forwarding layer together with the parameters 17 needed to provide the required processing context. A packet can be 18 composed from multiple token cells as needed to result in new new 19 network processing and forwarding semantics. 21 Status of This Memo 23 This Internet-Draft is submitted in full conformance with the 24 provisions of BCP 78 and BCP 79. 26 Internet-Drafts are working documents of the Internet Engineering 27 Task Force (IETF). Note that other groups may also distribute 28 working documents as Internet-Drafts. The list of current Internet- 29 Drafts is at https://datatracker.ietf.org/drafts/current/. 31 Internet-Drafts are draft documents valid for a maximum of six months 32 and may be updated, replaced, or obsoleted by other documents at any 33 time. It is inappropriate to use Internet-Drafts as reference 34 material or to cite them other than as "work in progress." 36 This Internet-Draft will expire on October 14, 2021. 38 Copyright Notice 40 Copyright (c) 2021 IETF Trust and the persons identified as the 41 document authors. All rights reserved. 43 This document is subject to BCP 78 and the IETF Trust's Legal 44 Provisions Relating to IETF Documents 45 (https://trustee.ietf.org/license-info) in effect on the date of 46 publication of this document. Please review these documents 47 carefully, as they describe your rights and restrictions with respect 48 to this document. Code Components extracted from this document must 49 include Simplified BSD License text as described in Section 4.e of 50 the Trust Legal Provisions and are provided without warranty as 51 described in the Simplified BSD License. 53 Table of Contents 55 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 56 2. The TCR Concept . . . . . . . . . . . . . . . . . . . . . . . 4 57 3. Relationship to prior work . . . . . . . . . . . . . . . . . 5 58 4. TCR Packet Structure . . . . . . . . . . . . . . . . . . . . 6 59 5. Token Cell types and categories . . . . . . . . . . . . . . . 7 60 6. Token Cell Structure . . . . . . . . . . . . . . . . . . . . 9 61 7. Token Cell Processing Model . . . . . . . . . . . . . . . . . 10 62 8. Token Cell Processing Order . . . . . . . . . . . . . . . . . 12 63 8.1. Serial Token Cell Processing . . . . . . . . . . . . . . 13 64 8.2. Parallel Token Cell Processing . . . . . . . . . . . . . 14 65 8.3. Combined Serial and Parallel Token Cell Processing . . . 16 66 9. Token Cell Pushing and Token Cell Popping . . . . . . . . . . 18 67 10. Selected Token Cell Type Categories . . . . . . . . . . . . . 19 68 10.1. Disposition Token Cells . . . . . . . . . . . . . . . . 19 69 10.2. Scratchpad and Metadata Token Cells . . . . . . . . . . 19 70 10.3. Conditional and Directive Token Cells . . . . . . . . . 21 71 10.4. Security Token Cells . . . . . . . . . . . . . . . . . . 21 72 11. Example applications of TCR . . . . . . . . . . . . . . . . . 23 73 11.1. Basic Tunneling of Payload . . . . . . . . . . . . . . . 24 74 11.2. Latency-Based Forwarding . . . . . . . . . . . . . . . . 25 75 11.3. Forwarding with Flexible Addressing . . . . . . . . . . 26 76 11.4. Forwarding with iOAM analytics . . . . . . . . . . . . . 27 77 11.5. FRR with Latency-Based Forwarding . . . . . . . . . . . 30 78 11.6. Segment Routing with Latency-Based Forwarding . . . . . 32 79 11.7. Enhanced Segment Routing with Latency-Based Forwarding . 32 80 11.8. Enhanced Segment Routing with Differentiated iOAM . . . 33 81 12. Items for further discussion . . . . . . . . . . . . . . . . 35 82 13. Security Considerations . . . . . . . . . . . . . . . . . . . 36 83 14. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 38 84 15. References . . . . . . . . . . . . . . . . . . . . . . . . . 38 85 15.1. Normative References . . . . . . . . . . . . . . . . . . 38 86 15.2. Informative References . . . . . . . . . . . . . . . . . 38 87 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 39 89 1. Introduction 91 Advances in data plane protocols are needed to address new network 92 requirements that stretch existing protocols (including MPLS, IPv6 93 and Segment Routing) to their limits. Token Cell Routing (TCR) is a 94 new network layer data plane technology that provides the ability to 95 program the data plane to meet the needs of many new operational 96 scenarios. TCR is based on token cells which provide a flexible 97 method describing the required packet action to the forwarder as well 98 as carrying any parameters and other data necessary to correctly 99 execute the required action. 101 Packet actions are not limited to forwarding actions, and it is 102 possible perform multiple packet actions at any given node. For 103 example, packet actions can include application of special QoS 104 algorithms, collection of telemetry, even assessment of dynamic 105 conditions before the performing of other actions. Token Cells can 106 be differentiated by type depending on the type of packet action that 107 they represent. 109 TCR is thus analogous to each packet carrying a stack of pointers to 110 procedures together with the data needed by those procedures. The 111 structure used allows token cells to be sequenced and parallelized in 112 groups, and permits the use of pointers to information in other token 113 cells. This results in a powerful method constructing advanced new 114 packet types from token cells to meet network needs of new 115 applications. 117 TCR therefore supports customizable semantics with which packets are 118 to be processed by nodes encountered along a path, it accommodates 119 flexible addressing semantics that do not necessarily depend on a 120 single addressing format. TCR accommodates custom "guidance" beyond 121 forwarding (such as the definition of QoS treatments that are to be 122 applied, the ability to differentiate behavior depending on dynamic 123 context encountered at a node, and the ability to collect and pre- 124 process telemetry in support of manageability applications). TCR 125 enables the direct processing via a "scaffold" that explicitly 126 indicates serialization, parallelization, disposition rules. It that 127 enables a "lego-esque" composition of packet processing behavior / 128 features while at the same time being hardware-friendly, and easy to 129 optimize for performance at line rate general in nature and easy to 130 extend. 132 A new TCR features is added by introducing a new procedure in the 133 forwarder and then including in the packet a pointer to that 134 procedure. The procedure knows how to interpret the other 135 information carried in the token cell. In many ways this is similar 136 to the way new FECs are introduced into MPLS and new instructions 137 introduced to segment routing. 139 TCR thus provides a general, highly extensible data plane that 140 supports custom semantics which is hardware friendly. It thus takes 141 the programmability of both MPLS and Segment Routing to a new level 142 of capability. 144 A key differentiator from earlier protocols is the ability to process 145 a variable number of processing actions at each hop, at directed by 146 the token cell structure. Furthermore token cells do not need to be 147 processed in the order in which they are placed in the packet, and 148 may be explicitly programmed for a flow. 150 We would like to note that at the time of writing the current -00 151 version of the draft, this represents a sketch of an idea that 152 neither we or anyone else has built. Thus, it is likely to have bugs 153 and certainly has many aspects that can be improved on. We would be 154 delighted to work with others who are interested in exploring this 155 idea and developing it further. A starter set of discussion items is 156 included in Section 12 towards the end of the document. 158 2. The TCR Concept 160 The foundation of TCR is the construction of the packet from a set of 161 token cells. A token cell is an extended length, type, value 162 construct. The type and part of the value is processed by a longest 163 match engine, which operates much like an IP address lookup engine, 164 but operates on arbitrary constructs rather than being confined to 165 address lookup. As part of its value, the token cell may also carry 166 parameters specific to the token cell type that are needed to process 167 the token cell. Depending on the token cell type, different code 168 points can be invoked that process the token cell. The token cell 169 type determines the semantics, i.e. the function to be applied. It 170 also defines any structure that may be contained in the token cell 171 value. 173 Token Cell processing can have generalizable packet processing 174 semantics: forwarding is a common semantic, but other semantics can 175 be applied, in a similar manner to the way in which an MPLS label has 176 generalized semantics. The processing of a token cell is: Input - 177 Match - Effect, where "effect" is one of forwarding, token cell 178 disposition, or something else (such as, conditional directives or 179 QoS treatment). Packet processing based on the first n bits of the 180 token cell at a known position, which is matchable on prefix, in 181 which case less significant bits serve as input. 183 The TCR approach to packet design is differentiated from common 184 protocol designs in that it allow for processing of variable number 185 of token cells per hop, as directed by token cell structure. Which 186 token cells to process, and whether token cells have 187 interdependencies that require them to be processed in serial order 188 or whether they allow for parallelization, is indicated by the token 189 cells themselves, as the token cell structure includes length and 190 serialization indicators. Processing can be serialized per "next 191 token cell" indicator. Processing can be parallelized per "manifest 192 token cell" that refers to parallel token cells in order to allow for 193 optimization. Note that there is no requirement to "must" 194 parallelize. Instead, parallelization is an optimization that nodes 195 with support for parallel packet processing stages may take advantage 196 of to reduce packet latency due to packet forwarding time. 198 Token Cells do not necessarily need to be processed in stack order 199 but can be located wherever is most efficient. Some token cells may 200 not be processed at all but can be used to carry meta-data (read-only 201 or writable) that can be referred to from other token cells. 203 The TCR approach allow for extensibility and programmability through: 205 o Level 1: Combining different token cells 207 o Level 2: Parameterizing token cells 209 o Level 3: Introducing new types of token cells 211 3. Relationship to prior work 213 In general data plane packets have been designed with a fixed 214 structure plus some variable number of TLVs to provide additional 215 instructions/advice to the forwarder. In general any address (for 216 example IP address) or instruction (MPLS label of SR SID) is of a 217 fixed length although may be structured into a prefix and suffix 218 arrangement to support aggregation and is processed using a longest 219 match lookup. Where TLVs are used there is their inclusion in the 220 packet is normally free form and thus it is necessary search the 221 packet for the set of TLVs that need to be processed. MPLS has a 222 very primitive parameter system in which one label may be used to 223 provide context for the label that follows. Examples of this are the 224 use of the context label and the use of the ELI/EL pair. 226 In deigning TCR we noted the generality and simplicity of the MPLS 227 label stack model and the effectiveness of the longest match 228 technique used in IP lookups. Putting these two together we 229 concluded that an LTV approach allowed the creation of a simple, 230 powerful, extensible, hardware friendly packet design. 232 As we will see later in the document, concatenating the T(ype) and 233 the V(alue) allows a single longest match look-up to resolve which 234 table to look up the action in, and then to absorb as much of the V 235 as is necessary to determine the specific action to take on the T. 236 The stack and pointer structure means that it is not necessary to 237 search for the applicable TLVs, they the forwarder is led to them 238 through the token cell structure pushed onto the packet. 240 The general method of processing is thus input - match - effect via a 241 match processor. The normal effect is to process one token cell at a 242 time in series, but an effect might be to process multiple other 243 token cells in parallel where the forwarder supports multiple 244 concurrent operations (where parallization is not supported serial 245 processing results in the same effect). Token Cells can have 246 interdependencies and they can allow for cross-referencing (e.g. 247 meta-data, scratch-pad) 249 4. TCR Packet Structure 251 A TCR packet Figure 1 is a series of token cells, each token cell 252 carrying a component of the packet delivery system or the payload 253 itself. 255 256 257 258 259 ... 260 262 Figure 1: Structure of a TCR Packet 264 Token Cells are a unit of packet processing that may include 265 parameters and/or data. The semantics, structure and processing of 266 the token cell is determined by the token cell type. A Token Cell 267 can be thought of as a type of "stem cell" that can be morphed into 268 any packet component, including components not yet designed, 269 resulting in a highly programmable packet design. 271 The preamble contains a small number of packet elements that are 272 always present in all packets, such as version identification and 273 TTL. It has yet to be decided if the preamble should be carried 274 conventionally or be carried within a token cell. 276 There is no separate payload portion of the packet. Instead, the 277 payload is carried within a token cell, generally located at the tail 278 of the packet. This allows for different payload delivery semantics 279 at the destination, including simply stripping the payload off or 280 applying a special type of codec. It also allows for the possibility 281 of payload-less packets that can be used for signaling and control 282 purposes. 284 A token cell MAY contain a complete TCR packet permitting 285 hierarchical encapsulation. 287 5. Token Cell types and categories 289 Token Cells have a type. Types themselves can be categorized, 290 depending on the purpose which token cells of that type serve. 292 The following token cell categories are provided as part of the TCR 293 design: 295 o Forwarding: A forwarding token cell specifies the destination 296 address and method of delivery of the packet. It may also include 297 the source address as a parameter, but this could also be 298 specified in a separate token cell. Different types within this 299 category may differentiate between address types such as IPv6 or 300 IPv4. 302 o SLO: A Service Level Objective (SLO) token cell specifies the 303 target quality of delivery, such as latency, delivery time, 304 required discard properties etc., each differentiated by 305 corresponding IDs as separate types. This allows intermediate 306 nodes on the path to apply special treatment to the packet, such 307 as scheduling algorithms, resource reservation, or prioritization, 308 in order meet SLOs as requirement. 310 o Metadata: These token cells carry metadata that can be referenced 311 and accessed as other token cells are being processed. Metadata 312 can thus be decoupled from token cells that access it, allowing 313 for their independent disposal, not interfering with pushing and 314 popping of other token cells. 316 o Scratchpad: Scratchpad token cell are in effect writeable metadata 317 token cells, a category of token cell in which the network takes 318 "notes" during the packet transit. Examples of this include 319 recording the route, adding proofs-of-transit, telemetry data, or 320 packet transit time of particular nodes that were traversed. 322 o Security: A security token cell signs parts of the packet with an 323 agreed cryptographic signature. It includes a signature mask that 324 specifies which token cells and/or portions thereof are covered by 325 the signature. This allows for the possibility of not only the 326 sender, but also nodes in transit being able to sign portions of 327 the packet. One example use would be for telemetry data that is 328 added to a scratchpad by a node being traversed while leaving 329 other parts of the scratchpad open to modification by other nodes. 331 o Conditional: A conditional token cell is able to test one of more 332 conditions and make invokation of the next token cell (NT) 333 dependent on the conditions evaluating to true. This allows to 334 define more complex behavior, such as the invocation of a 335 particular function depending on a dynamic condition encountered 336 at a node. 338 o Directive: A directive token cell specifies some type of action 339 that should be performed. An example would be a directive to 340 collect telemetry or OAM data. 342 o Manifest: A manifest token cell provides a method of specifying 343 which token cells may be processed in parallel. Parallel 344 processing is optional, and the token cells can also be correctly 345 processed serially. It is up to the entity that specifies the 346 manifest to ensure that the parallelism is safe. 348 o Rendezvous: A rendezvous token cell is a token cell used to ensure 349 that all parallel operations have completed and that it is hence 350 safe to resume serial operation of the forwarder. A rendezvous 351 token cell may specify the first serial operation to execute after 352 the rendezvous, or it may simply hand off to a new token cell. 354 o Disposition: A disposition token cell describes what is to be done 355 when the packet leaves the TCR domain. Such a token cell might, 356 for example specify a pseudowire [RFC3985] action (strip the TCR 357 header and send the payload to interface X), or a VPN action 358 (lookup the payload IP address in VRF Y). However, the mechanism 359 introduces the opportunity to attach a more sophisticated 360 disposition action, for example "if the packet arrives before time 361 T, forward using VRF V, otherwise drop the packet". 363 o Payload: A payload token cell simply carries the payload as its 364 value. 366 o Other: This is a catch-all category to allow for token cell types 367 that do not fit any of the other categories. 369 Some of the mentioned categories will be described in greater detail 370 in Section 10. In addition to the mentioned categories, it is 371 expected that other categories would be introduced as needed. 373 The grouping of types into categories may prove useful for various 374 purposes. For example, it will allow for the articulation of packet 375 grammars and "best packet practices", such as mandating that a packet 376 contain at least one token cell of the forwarding category, or that 377 the first token cell in a TCR packet must not be a token cell of 378 categories metadata, scratchpad, or security. Types allow to further 379 differentiate token cells within a given category. 381 It should be noted that some of the token categories should be 382 considered experimental. Which token types and even which token 383 categories to support will in depend on the needs of actual 384 deployments. 386 6. Token Cell Structure 388 The structure of a token cell is shown in Figure 2: 390 391 392 -+ 393 | 394 +- Match Zone 395 | 396 -+ 397 399 Figure 2: Structure of a TCR Token Cell 401 Token Cells will vary in length depending on the token cell type and 402 the contents of the token cell blob, although in practice a given 403 token cell type MAY be a fixed size. 405 Next token cell is a relative pointer (offset) to the next token cell 406 to be processed as part of the group of token cells to be 407 sequentially processed as a part of the packet action at the node. 408 If there is no next token cell to process, its value is null. 409 Processing of a packet begins with processing the first token cell. 410 Whether or not any additional token cells are processed depends on 411 the guidance provided via the Next Token field. 413 The token cell type is used to identify the category or purpose of 414 the token cell (Cat/Purpose) and the sub-type (ID) within that class. 415 An example of a purpose might be "Forwarding", indicating that the 416 token cell represents an instruction to forward the packet closer to 417 the destination. An example of a sub-type within that category might 418 be "IPv6", indicating tht the destination is identified by the 419 following IPv6 address. 421 The set of IDs consists of a set of well known IDs and a set of user 422 specified IDs. This provides both an extensible, and a programmable 423 mechanism for enhancing the protocol over time and within 424 deployments. Token cell type, category, and sub-type ID are not 425 fixed-length fields but represent conventions. 427 The token cell blob carries the information needed to process the 428 explicit packet that carries it, and/or is a place to record 429 information about the packet for later use. Within the token cell 430 blob there is a prefix and a suffix, which may themselves each be 431 structured. The structure of the token cell blob depends on the 432 token cell type, some of which may themselves be subjected to 433 standardization. 435 The token cell blob prefix, which is neither fixed nor a fixed length 436 field is used to qualify the type in the lookup. For example, if the 437 sub-type was IPv6 destination address the prefix would be an IPv6 438 address. However this is a more general device than just a 439 destination address and allows for token cell type categorization. 440 This may prove useful for various purposes (e.g. packet grammars and 441 Best Packet Practices). 443 The match zone is the portion of the token cell that is subject to 444 look up (see processing model section). The model is that the lookup 445 will be a longest match of the whole match zone. The token cell 446 design does not specify the length of the match zone or the length of 447 either the ID or the prefix. It is a property of longest match 448 lookup that it will either consume all the bits it needs, or reject 449 the lookup. If a result is found the result can specify the 450 structure of the token cell blob. Note that this is a model, and 451 there is much scope for implementor optimization without sacrificing 452 the generality of the design. 454 The number of bytes sent to the lookup engine is implementation 455 specific. If the attempted match is longer than needed, the longest 456 match will ignore the overspill. If more bytes are needed, it is a 457 property of longest match that the lookup can be restarted from where 458 it left off. 460 Within the blob, and in particular within the suffix, any structure 461 can be carried such as subfields, parameters. The definition of what 462 the suffix contains and how it is structured is part of the token 463 cell type definition. 465 7. Token Cell Processing Model 467 The TCR processing model is shown in Figure 3. 469 ______________________________ 470 |< Token Cell Match Zone >| 471 || 472 |______________________________| | 473 || || | 474 || || | 475 \ / | 476 +---v---+ | 477 | | 478 V | 479 Lookup | 480 Engine | 481 | | 482 | | 483 V | 484 Lookup o/p parameters | Pipeline 485 Code Pointer | state register 486 | | ! ^ 487 | | ! ! 488 V | ! ! 489 Code <-----------------+ ! ! 490 "callback" <~~~~~~~~~~~~~~~~~~~~~' ! 491 | ! 492 | ! 493 V ! 494 Effect ~~~~~~~~~~~~~~~~~~~~~~~~~~~~' 496 Figure 3: TCR Processing Model 498 This operates as follows. First the token cell match zone is fed 499 into the lookup engine. The lookup engine performs a longest match 500 and returns a parameter block which includes the address of the code 501 to be executed on the token cell by the forwarder. That code knows 502 how to interpret the token cell. Readers will recognize the genesis 503 of this is a hybrid between an IP address lookup which performs a 504 longest match lookup and returns a parameter block to the IP 505 forwarding code, and an MPLS label lookup which performs a fixed size 506 look-up logically returns a pointer to the executable code (MPLS 507 forward packet, pseudowire, VPN etc) and a parameter block. 509 Where it cannot be clear that what the length of the blog prefix is 510 from the lookup result, for example where the address is an IPv6 511 address, that length needs to be encoded in the prefix in some 512 convenient way, such as as a prefix to the prefix. 514 From the above, it is clear that there is no constraint on the type 515 and structure of the prefix and thus any address type or other 516 construct may be submitted to the lookup engine. Token Cell types 517 and prefix sets may be added to the forwarder by adding appropriate 518 data to the database queried by the lookup engine, and providing the 519 corresponding callback code to the network processor, in a manner 520 similar to that used in MPLS and in Network Programming. In this 521 regard the approach is compatible with proven hardware forwarding 522 models. 524 The callback code has access to any other element of the token cell 525 that it needs, and indeed to other elements of the packet as 526 required. Implementations MAY impose a reasonable limitation on the 527 number of processing cycles within which callback code needs to 528 complete. 530 As part of the effect of processing a token cell, a pipeline state 531 register can be written which can serve as input for the processing 532 of subsequent token cells of the same packet. This allows to compose 533 a pipeline of token cells being processed in which the output of one 534 function serve as input to the next function. In the special case of 535 rendezvous token cells that have multiple predecessors, their 536 respective outputs are merged as part of the specific rendezvous 537 semantics. 539 8. Token Cell Processing Order 541 It is entirely possible to require several token cells to be 542 processed at any given node. For example, one token cell may contain 543 a forwarding directive, whereas another token cell might contain a 544 directive related to QoS treatment of the packet for meeting a 545 Service Level Objective (SLO), while a third token cell indicates 546 that certain telemetry data from traversed nodes should be collected. 548 In the simple case that token cells can or should be serially 549 processed, the processing of token cells will simply be chained, as 550 directed by the token cells' respective NT fields. After processing 551 of a token cell concludes, the reference in the NT field is resolved 552 and processing continues with that token cell. If the NT contains no 553 reference (or in the special case of a conditional token cell 554 evaluating to false), the processing of the packet concludes. In 555 that case, the processing of a packet will require n stages, n being 556 the number of chained token cells. 558 A packet processing pipeline needs to support a depth that is 559 equivalent to the maximum number of token cells that can be chained. 561 In some cases, optimization is possible by exploiting 562 parallelization. In the earlier example, it may be possible to 563 perform some tasks in parallel, such as the application of QoS 564 treatment and collection of telemetry data, while other tasks may 565 still need to be serialized, such as determining which outgoing 566 interface to use as a forwarding decision before performing the QoS 567 actions against that interface. The fact that certain token cells 568 may be processed concurrently can be indicated through a special 569 manifest token cell that references the token cells that follow. 570 Each of those token cells can in turn have their own successors, in 571 effect resulting in separate packet processing "threads" (all 572 processing different token cells of the same packet). While the 573 processing of the manifest adds an additional cycle, depending on the 574 complexity of the workflow this may be more than offset by the 575 parallelization that ensues. 577 While full exploitation of the optimization potential may require 578 advances in hardware pipeline design, it should be emphasized any 579 such optimization is optional and not required. Also, to maintain 580 packet ordering, the packet will generally still be required to pass 581 through all n pipeline stages. That said, the option of 582 parallelization does allow for hardware pipeline designs able to 583 exploit concurrency of multiple threads and accommodating a larger 584 number of token cells than would otherwise be supported by pipelines 585 of a given depth, respectively reduce the pipeline depth that needs 586 to be supported. 588 8.1. Serial Token Cell Processing 590 All packets have an element of serial processing in that the preamble 591 and then the token cell that follows are always processed. 593 A serial group of token cells is constructed by using the next 594 pointer to point to the token cell to processed on completion of the 595 token cell. The end of a serial token cell group is logically 596 indicated using a NULL next token cell pointer, although none of the 597 foregoing should be taken as dictating the wire format which will be 598 the subject of another text. 600 Token Cell T1 601 T2 602 603 604 Token Cell T2 605 T4 606 607 608 Token Cell T3 609 - 610 611 612 Token Cell T4 613 - 614 615 617 Figure 4: TCR Serial Token Cell Processing of Token Cells 619 Figure 4 shows four token cells. The serial processing instructed by 620 this construct is that token cell 1 is to be processed, followed by 621 token cell 2 and then followed by token cell 4. There are many 622 reasons why this construct is interesting and these are discussed 623 later in this document. 625 In order to simplify the graphical annotation, references to token 626 cell cells are in the following simply indicated by a numeric 627 identifier instead of being depicted by arrows. 629 8.2. Parallel Token Cell Processing 631 In some cases it is desirable to introduce parallel processing to the 632 packet where there are token cells have no result interdependencies. 633 We do this through the introduction of a manifest token cell that 634 contains a set of pointers or offsets to other token cells. 636 637 638 -+ 639 | 640 +- Match Zone 641 | 642 -+ -+ 643 | 644 .. +- Manifest 645 | 646 -+ 648 Figure 5: TCR Parallel Processing of Token Cells 650 The structure of the manifest token cell is shown in Figure 5. The 651 initial part of the token cell is standard to all token cells. There 652 follows a list of token cell pointers to the set of token cells to be 653 processed in parallel. The end of the set of token cells to be 654 processed in parallel is determined when the end of the token cell is 655 reached as indicted by the token cell length. When all the child 656 token cells have completes execution the token cell pointed to by the 657 next token cell field is executed. 659 Note that the match zone overlaps the manifest. The token cells in 660 the manifest will however be ignored by the longest match which will 661 complete with the ID. 663 Also note that, as previously mentioned, parallelism is an efficiency 664 issue, not a correctness issue. A forwarder that does not support 665 the parallel dispatch of token cells or that supports less 666 parallelism than specified in the manifest can choose to execute 667 individual token cells (respectively groups of token cells) serially. 668 It is up to the sender to construct the packet as needed and make any 669 token cell interdependencies explit, without requiring the network to 670 second-guess whether or not there are any such interdependencies. In 671 other words, when the order in which token cells are processed might 672 result in different behavior, it is the responsibility of the sender 673 to specify any required serialization as needed. 675 In most cases where it is used, a manifest token cell will typically 676 be the first token cell after the preamble, to exploit the 677 possibility of concurrent processing threads for multiple token cells 678 in the same packet from the onset. However, this is not an 679 architectural requirement and manifest token cells could also occur 680 later in the packet. 682 It is possible to remerge concurrent token cell threads using a 683 special rendezvous token cell. A rendezvous token cell awaits for 684 each of its predecessors to have completed before resuming 685 processing. In addition, output from predecessors (i.e. pipeline 686 state register content) can be aggregated as needed. 688 8.3. Combined Serial and Parallel Token Cell Processing 690 Figure 6 illustrates the concept of parallel and serialized 691 processing further. Please note that this is an admittedly complex 692 scenario and most scenarios in practice will be simpler. 694 T1 695 / | \ 696 / | \ 697 T2 T3 T4 698 | / | \ 699 T5 T6 T7 T8 700 | 701 T9 703 Figure 6: TCR Combined Serial and Parallel Processing of Token Cells 705 After processing T1, three tokens (T2, T3, T4) can be processed 706 concurrently. T5 has a serialization dependency on T2. T6, T7, T8 707 can be procesed concurrently once T4 has been processed. Finally, T9 708 has a serialization dependency on T7. 710 A corresponding packet is shown in Figure 7. Manifest token cells 711 are introduced to represent the fact that T2, T3, T4 respectively T6, 712 T7, T8 can be processed in parallel. A Next Token field of "-" 713 indicates there is no next token. 715 Token Cell T1 716 M1 717 718 719 Token Cell M1 720 - 721 722 723 T2 724 T3 725 T4 726 Token Cell T2 727 T5 728 729 730 Token Cell T3 731 - 732 733 734 Token Cell T4 735 M2 736 737 738 Token Cell M2 739 - 740 741 T6 742 T7 743 T8 744 Token Cell T5 745 - 746 747 748 Token Cell T6 749 - 750 751 752 Token Cell T7 753 T9 754 755 756 Token Cell T8 757 - 758 759 760 Token Cell T9 761 - 762 763 765 Figure 7: TCR Combined Serial and Parallel Processing of Token Cells 767 Token Cell M1 is a manifest that indicates three children (at the 768 protocol level, the number of potential children is subject only to 769 packet size constraints). From a protocol perspective all three 770 children: token cell 2, token cell 3 and token cell 4 can execute 771 immediately and concurrently. When token cell 2 completes token cell 772 5 runs, when token cell 5 completes that that processing branch is 773 completed. Token Cell 3 runs and when it completes that token cell 774 branch is completed. Token Cell M2 is also a manifest with three 775 children: token cell 6, token cell 7 and token cell 8. When token 776 cell 6 completes that processing branch is completed. When token 777 cell 7 is completed token cell 9 runs. When token cell 9 completes 778 that processing branch is completed. When token cell 8 is completed 779 that processing branch is completed. When token cells 5, 3, 6, 9 and 780 8 are completed the token cell group is completed. 782 It should be noted that the current design of the manifest token cell 783 type includes all pointers to subsequent token cells as part of the 784 token cell blob. Alternative designs are conceivable in which (for 785 example) the next token field would be populated with the first of 786 the concurrent successors, with only additional token cells (beyond 787 the first) to be referenced from the token cell blob. 789 It should also be noted packet permutations can be accommodated, i.e. 790 the order of the token cells in the token cell group can be any order 791 convenient to the network application designer. For example, the 792 same token cells from Figure 7 could have been arranged also in the 793 following sequence: T1 - M1 - T2 - T5 - T4 - T3 - M2 - T6 - T7 - T9 - 794 T8. The token cells do not need to be arranged in a stack in the way 795 that MPLS or SRv6 arrange their labels or SIDs respectively; the 796 order in which to process token cells is always resolved by the NT 797 reference (respectively any manifest token cell references). However 798 it is RECOMMENDED that backward references are avoided as a packet 799 with no backward references will not form a processing loop. 801 9. Token Cell Pushing and Token Cell Popping 803 Token Cell groups can be stacked by pushing a group of token cells 804 onto a packet to encode a hierarchical set of operations to be 805 executed on the packet as it journeys to its destination. This is 806 similar to IP tunneling or the pushing and popping of MPLS labels. 808 In a manner similar to the pipe model described in [RFC3270] it is a 809 matter for the protocol designer and the operator whether the pushed 810 token cells are able to understand and interact with existing the 811 tokens cells. This will be discussed further in a future version of 812 this document. 814 Similarly when a token cell group has accomplished its purpose on the 815 packet journey it is popped so that the forwarded can gain access to 816 the next element of processing. 818 Again considerations similar to [RFC3270] may apply. 820 For the purposes of this discussion a Token Cell group may consist of 821 a single token cell. 823 Considerations regarding penultimate hop popping will be included in 824 a future version of this text. 826 10. Selected Token Cell Type Categories 828 An overview of token cell type categories was given in Section 5. In 829 this section, some of these categories will be explained in further 830 detail. 832 10.1. Disposition Token Cells 834 A disposition token cell describes to the network what actions are to 835 be performed as a packet egresses the TCR domain. A forwarding token 836 cell may for this purpose include a reference to a disposition token 837 cell as a parameter. 839 All existing IETF network protocols include a disposition semantic 840 although sometimes this is implicit. For example when IPv6 has a 841 next header of "TCP" the disposition instruction is to dispose of the 842 IPv6 header and hand the packet to the TCP handler. Similarly 843 pseudowires have a disposition instruction in the PW label 844 instructing the forwarder how to dispose of the MPLS header and to 845 reconstruct the packet in its original format. 847 However as the metadata carried in the packet become more 848 sophisticated there is a requirement for the disposition to become 849 more sophisticated. 851 Disposition token cells thus formalize the description of the egress 852 behavior of the network on the packet and allow a richer egress 853 semantic to be described. 855 It is conceivable to define TCR such that in the absence of a 856 reference to a disposition token cell, TCR will revert to implicit 857 disposition behavior when the destination address of a forwarding 858 token cell is reached. That will involve popping the token cells up 859 to the forwarding token cell and resuming processing with the 860 subsequent token cell. In other words, a disposition token cell is 861 in that case used to specify any special semantics that would go 862 beyond vanilla implicit disposition. 864 10.2. Scratchpad and Metadata Token Cells 866 Metadata, provided by a sender of a packet or by an ingress node, can 867 provide important guidance to nodes along a path to guide processing 868 of the packet. Examples include SLOs that should be taken into 869 account for QoS treatment, profiles to apply towards proessing a 870 packet, and more. Other uses include the carrying of security 871 material as well as the tagging of packets for classification 872 purposes. 874 Scratchpads refer to writeable metadata, i.e., metadata that can not 875 only be accessed but also modified or added by nodes "in flight", 876 i.e. during transit. Example uses include collection of telemetry 877 and iOAM data [I-D.ietf-ippm-ioam-data], auxiliary data used for 878 measurements such as intermediate time stamps, or proof-of-transit 879 data and data verifying certain properties of nodes being traversed 880 (such as whether from a trusted vendor or located in a certain 881 region). 883 Metadata and scratchpad token cells allow to carry such metadata as 884 part of a packet independent of the token cells that access it. This 885 decoupling allows to dispose of metadata and scratchpads independent 886 of token cells that process it. For example, it makes it conceivable 887 to collect different iOAM data along different segments of a path, as 888 directed by different token cells, and to export the data only once 889 an exporter or egress node is reached. This way, scratchpad token 890 cells enable applications that rely on sharing of node-specific data 891 along a path to do so without the complications of having to 892 introduce piggyback extensions to the underlying protocol. In 893 addition to adding and updating data items in scratchpad token cells 894 along a path, also additional scratchpads can be added. It also 895 avoids the need for the same metadata to be copied across token 896 cells, for example in cases where the same SLOs are applicable across 897 different segments, each governed by their own respective token 898 cells. 900 Metadata and scratchpad items can be referenced by other token cells. 901 The specific format of those references is to be determined by the 902 rules governing the content of the respective token cell blobs for 903 the token cell type; in general the reference format will involve an 904 offset from the referring token cell. 906 Disposition semantics need to include defining what is to happen with 907 metadata or scratchpad carried in corresponding token cells. 908 Possible disposition actions include: 910 o Discard 912 o Export (possibly parameterized to specificy export mechanism as 913 well as export target) 915 o Log (again, possibly parametrized with a logging target) 917 Metadata and scratchpad data can also be independently authenticated 918 and secured. This allows, for example, to ensure that scratchpad 919 data that is added or modified by intermediate nodes cannot be 920 tampered with. It also allows for the implementation of operations 921 that authenticate both metadata and scratchpad data before processing 922 it further. 924 Metadata, let alone scratchpad data, is a concept that is not 925 directly supported by token-based protocols such as SR or MPLS today. 926 While it would be possible to encode such as part of a token, its 927 processing would require the token itself be processed (e.g. as part 928 of a forwarding operation), tied to its being pushed or popped on a 929 stack. This would make it harder to e.g. update scratchpad data 930 (that should be protected from popping and may be buried underneath a 931 token or label stack), to access data without effectively copying it 932 across tokens, and generally to accommodate metadata and scratchpad 933 processing where the lifecycle of data items does not directly 934 correspond to that of segments. With TCR, there is no need to 935 process metadata in "stacked order" nor need for complex token cell 936 rewrite rules in order to preserve data. 938 10.3. Conditional and Directive Token Cells 940 Directive Token Cells allow the specification of some type of action 941 or function that should be performed as a result of the token cell 942 being processed. An example of a directive would be a request to 943 collect telemetry or in-situ OAM data and record it as part of 944 scratchpad. It is expected that there can potentially be a large 945 number of possible directives, each distinguished by its own ID 946 respectively token cell type. Different token cell types may impose 947 their own respective structure on the token cell blob to represent 948 different parameters. 950 Conditional Token Cells are similar to Directive Token Cells in that 951 they allow the specification of an operation to be performed. 952 However, it is different from other categories of tokens in that the 953 outcome of the operation determines whether the NT field will be 954 processed, i.e. whether processing will subsequently resume with the 955 token cell referenced by the NT field or whether it should terminate. 956 This allows for the definition of functionality that should be 957 performed depending on certain conditions that are being encountered. 958 For example, a conditional token cell might allow for a different 959 paths to be selected depending on dynamic circumstances such as load 960 conditions. Similarly, the collection of certain telemetry data 961 might be made dependent on certain conditions being encountered. 963 10.4. Security Token Cells 965 Security token cells enable a security mechanism that allows to sign 966 the invariant elements of the packet whilst avoiding signing the 967 packet elements that are modified during the passage of the packet 968 through the network, such as scratchpad token cells. 970 Likewise, they allow for differentiated signing of different parts of 971 the packet by different signers, such as in cases where nodes add 972 data items to a scratchpad that should be independently signed. 974 A security token cell allows to carry signature material pertaining 975 to elements of the packet. It includes the following items: 977 o An identification of the signer 979 o A mask indicating the parts of the packet respectively token 980 cell(s) being signed 982 o The signature material itself 984 The general structure of the security token cell is as follows 986 987 988 -+ 989 |- Match Zone 990 -+ 991 992 993 994 995 996 997 998 999 1000 1002 Figure 8: Structure of a TCR Security Token Cell 1004 The token purpose and ID specify that this is a security token, the 1005 exact structure of the security token, and the type of signature that 1006 has been used. The structure used here is illustrative and used to 1007 explain the concept. 1009 The token cell blob contains the security material and the mask. A 1010 possible structure is as follows: 1012 As described in [RFC8754] the Key ID is used to identify the 1013 preshared key and the algorithm, though the algorithm may be 1014 indicated by the token ID (this is a matter for the token designer). 1016 The Hashed Message Authentication Code (HMAC) is the hash of the 1017 complete TCR security token and the N TCR tokens, and within those 1018 tokens the token elements specified by the token mask. 1020 For efficiency the mask operates at a number of levels: 1022 o The next N token cells 1024 o The marked tokens within the next N token cells 1026 o Octets or words within a specific token cell 1028 N specifies the number of tokens covered by this security token in 1029 addition to itself. If present, Token Cell Mask is a structure that 1030 specifies which tokens are actually protected. Token_Protected.NP is 1031 the number of tokens within the token mask. 1032 Token_Protected.Full_Token is a bit mask of Token_Protected.NP bits 1033 indicating which complete tokens are to be included in the signature. 1034 Token_Protected.Byte_Protected is a bit mask of Token_Protected.NP 1035 bits indicating which complete tokens are to be included in the 1036 signature. 1038 11. Example applications of TCR 1040 This section contains a number of examples illustrating how the TCR 1041 token system is used to create or "program" packets. The examples 1042 are illustrative and not exhaustive. Only the essential features of 1043 the packet are shown. 1045 In order to simplify depiction of TCR packets, specifically the order 1046 in which tokens are processed and the cross-dependencies between 1047 tokens, we will introduce a syntax which identifies individual tokens 1048 by a numeric token identifier, and that allows to reference tokens 1049 using this identifier as opposed to packet offsets depicted using 1050 ASCII art pointers. This is merely done for convenience of textually 1051 representing TCR packets in this draft. It is independent of the 1052 actual TCR packet and token structure, in which tokens do not have 1053 separate identifiers and are simply referenced by offset. This 1054 applies to Next Token fields, as well as for disposition. A 1055 disposition token cell is referenced through a field in token cells 1056 whose type is of the forwarding category. 1058 A packet is simply represented by a sequence of tokens. Each token 1059 is represented by a set of fields and their values. In addition, a 1060 "pseudo field" is introduced for the numeric token identifier. A 1061 reference to another token (from the field, from a 1062 manifest token, or from a rendezvous token) contains as value the 1063 respective token identifier. 1065 11.1. Basic Tunneling of Payload 1067 In this example we consider a the case where a packet is to be 1068 tunneled across a network as one might do in a sub-IP network. 1070 For illustrative purposes we show in Figure 9 an IPv6 address being 1071 used as a destination address, but any other address family can be 1072 used with the corresponding forwarding token type. 1074 1075 Token Cell T1 1076 - 1077 1078 1079 1080 Token Cell T2 1081 1082 1083 1084 Token Cell T3 1085 1086 1087 1089 Figure 9: Basic Tunnelling Over TCR 1091 The packet starts with a preamble followed by token T1 which is IPv6 1092 forwarding token, a token with the semantic that it is to deliver the 1093 packet as close to the target address as it can reach. There is no 1094 next token cell to be processed until arrival at the destination and 1095 so Next Token is NULL. 1097 When the packet arrives at its destination the token that is pointed 1098 to by the disposition pointer is noted and all tokens up to the 1099 disposition token are popped (in this case only token 1 is popped). 1101 The disposition token T2 is processed and the disposition parameters 1102 say how the payload is to be processed. This may be as simple as 1103 providing the protocol type of the payload, but it may also provide 1104 information other information such as the identity of a VRF table to 1105 use, or an interface to dispatch the packet to in the case of a 1106 pseudowire. It also provides a pointer to the payload. When the 1107 parser knows how to dispose of the packet, it pops Token 2 and 1108 processes token 3. Token 3, in the case of containing a tunneled IP 1109 packet, just requires the length to be noted and corrected for the 1110 token overhead, the type confirmed as payload and payload to be 1111 forwarded as an IP packet. 1113 11.2. Latency-Based Forwarding 1115 In this example we consider the previous case (basic tunneling of 1116 payload) Section 11.1 and extend it to provide support for on-time 1117 delivery according to an end-to-end latency objective. The support 1118 is provided using a Latency-Based Forwarding (LBF). LBF bases the 1119 decision on when to sent to packet on how urgently or at what exact 1120 time it needs to arrive. To do so, LBF involves an algorithm that 1121 determines at any given node whether the packet is "on track" to meet 1122 its latency objective, and matches the QoS treatment and scheduling 1123 of the packet against a latency "budget" that is determined from 1124 latency objective, the latency that was already incurred, and the 1125 expected remaining latency towards the destination. For further 1126 details, please refer to [DOI.10.1109_NOMS47738.2020.9110431]. 1128 To indicate that a packet should undergo LBF treatment, a 1129 corresponding token cell type of category "SLO" is introduced. When 1130 it is processed, the packet is subject to LBF. Parameters for LBF 1131 include the end-to-end latency objectives and a helper parameter to 1132 assess the latency being incurred. An additional input is the egress 1133 interface that is obtained from processing of the fowarding token 1134 cell that precedes it and that can be passed using TCR's pipeline 1135 state register. 1137 A packet that enables LBF in conjunction with the preivious case is 1138 depicted in Figure 10. It adds one additional token cell over the 1139 previous use case, in essence adding the LBF functionality as a 1140 "module" that is combined with the earlier functionality. The 1141 ability to compose functionality by simply combining corresponding 1142 token cells into a packet is part of what makes TCR quite powerful. 1144 Processing of the packet is similar to the processing of the basic 1145 tunneling case except that Token 1 the IPv6fwd token contains a 1146 pointer to a further token that is to be sequentially processed at 1147 every hop, including arrival at the destination. The Next Token 1148 pointer in Token 1 points to token 2 which specifies that the packet 1149 must arrive at a specified time, and contains information needed to 1150 record the time taken and hence the time at which it should optimally 1151 leave its current node. 1153 When the packet arrives at its destination a final check is made to 1154 see if the arrival time requirement was met and it is processed 1155 according to the failure to arrive on time instructions in either the 1156 LBF_ontime token or the disposition token. All further processing is 1157 as per the above basic tunneling case. 1159 1160 Token Cell T1 1161 T2 1162 1163 1164 1165 Token Cell T2 1166 - 1167 1168 1169 Token Cell T3 1170 1171 1172 1173 Token Cell T4 1174 1175 1176 1178 Figure 10: LBF Using TCR 1180 11.3. Forwarding with Flexible Addressing 1182 The is an emerging need to support multiple address technologies. 1183 There are two cases in point, firstly where an operator wants to use 1184 a short address to to address the infrastructure nodes in their 1185 network. This is particularly the case in segment routing where 1186 there is competition amongst the vendors to introduce address 1187 compression due to the extended size of SRv6 data plane headers. 1188 There is a secondary benefit to this in that if an address system 1189 other than IP is used within the provider network there are security 1190 benefits as has been found in operating MPLS. Finally it has been 1191 speculated that some application environments would prefer to use 1192 their native addresses rather than manage the mapping between those 1193 addresses and IP addresses. 1195 This is achieved by creating a new token cell type, populating the 1196 FIB with the corresponding addresses and installing in the forwarder 1197 the necessary forwarding code. That forwarding code may be generic 1198 since the action will almost certainly be address family independent. 1200 In the example shown in Figure 11, the address family is encoded in 1201 the address itself. In an alternative model, the token cell type 1202 would be specific to the address family. 1204 1205 Token Cell T1 1206 - 1207 1208 1209 T2 1210 Token Cell T2 1211 - 1212 1213 1214 Token Cell T3 1215 1216 1217 1219 Figure 11: Indirect LBF UsingUsing Flexible Addresses in TCR 1221 11.4. Forwarding with iOAM analytics 1223 In many cases, there is desire to collect in-situ OAM data 1224 [I-D.ietf-ippm-ioam-data] as a packet traverses a path. There are 1225 multiple applications for this, including but not limited to 1226 diagnosing performance, identification of bottlenecks, and generation 1227 of data to feed machine-learning algorithms for service optimization. 1229 Using TCR, it is possible to indicate that iOAM data should be 1230 collected using a corresponding token cell. The token cell can 1231 contain in-situ parameters, such as which data items to collect. In- 1232 situ data itself can be added to a scratchpad, allowing for its 1233 export using a variety of means upon disposition of packet. 1235 One scenario is depicted in Figure 12. In this particular example, 1236 we assume that the iOAM data can be collected per T3 in parallel with 1237 the forwarding decision being made per T2 in order to show also use 1238 of a manifest (T1). iOAM Data items are written to the scratchpad in 1239 T5. T4 indicates disposition, T6 contains the payload. 1241 It should be noted that rather than simply collecting iOAM data, 1242 other operations could be applied to aggregate that data and result 1243 in more refined behavior. In the interest of brevity, the example 1244 does not feature security tokens used by intermediate nodes to sign 1245 the scratchpad data items that they add. 1247 1248 Token Cell T1 1249 - 1250 1251 T2 1252 T3 1253 Token Cell T2 1254 - 1255 1256 1257 T4 1258 Token Cell T3 1259 - 1260 1261 1262 T5 1263 Token Cell T4 1264 - 1265 1266 1267 Token Cell T5 1268 - 1269 1270 1271 Token Cell T6 1272 1273 1274 1276 Figure 12: iOAM in TCR 1278 If iOAM data to be collected includes telemetry about the egress 1279 interface, the manifest cell can be omitted as the forwarding 1280 decision needs to be made prior to collecting the iOAM data. The 1281 resulting packet becomes even simpler, as depicted in Figure 13. 1283 1284 Token Cell T1 1285 T2 1286 1287 1288 T3 1289 Token Cell T2 1290 - 1291 1292 1293 T4 1294 Token Cell T3 1295 - 1296 1297 1298 Token Cell T4 1299 - 1300 1301 1302 Token Cell T5 1303 1304 1305 1307 Figure 13: iOAM in TCR - serialized 1309 To show the modularity that TCR enables, the third scenario shows 1310 iOAM data to be collected while at the same time LBF is being applied 1311 to the same packet (Figure 14). Note that in this case, LBF and iOAM 1312 could have also been applied in parallel, in which case a manifest 1313 token cell would be added. T1 would then point to the manifest token 1314 cell, which in turn would point to T2 and T3 as successors. 1316 1317 Token Cell T1 1318 T2 1319 1320 1321 T4 1322 Token Cell T2 1323 T3 1324 1325 1326 Token Cell T3 1327 - 1328 1329 1330 T6 1331 Token Cell T4 1332 - 1333 1334 1335 Token Cell T5 1336 - 1337 1338 1339 Token Cell T6 1340 - 1341 1342 1344 Figure 14: iOAM and LBF in TCR 1346 11.5. FRR with Latency-Based Forwarding 1348 This example depicted in Figure 15 extends the earlier LBF example to 1349 include a fast re-route diversion. We show only a single token cell 1350 (T0) added at the point of local repair (PLR), but of course the 1351 repair might be more complex and need multiple intermediate staging 1352 counts to successfully be repaired. 1354 1355 Token Cell T0 1356 T2 1357 1358 1359 Token Cell T1 1360 T2 1361 1362 1363 1364 Token Cell T2 1365 - 1366 1367 1368 Token Cell T3 1369 1370 1371 1372 Token Cell T4 1373 1374 1375 1377 Figure 15: FRR with LBF Using TCR 1379 The PLR was expecting to forward the packet normally and so is aware 1380 that the packet is latency sensitive and understands the semantics 1381 and hence importance of token 2. To maintain the expected path 1382 quality the PLR MUST use an FRR path while also ensuring the SLO is 1383 still being adhered to per the LBF token cell. The FRR path 1384 therefore needs to feature nodes able to support LBF token cells, and 1385 not incur a latency penalty that would physically prohibit being able 1386 to meet the SLO. This path can be selected by the SDN controller, or 1387 locally through the use of path attributes applied to the normal IP- 1388 FRR path selection process. 1390 On detecting a local repairable failure of the next hop or the link 1391 to the next hop the PLR pushes one or more tokens as is necessary to 1392 deliver the packet to the destination. In each case the next token 1393 pointer points to Token 2 the LBF token. When the packet arrives at 1394 the intermediate node chosen by the PLR for the next stage of the 1395 repair, the token (in this case Token 0) at the top of the token 1396 stack is popped and forwarding proceeds as dictated by token 1. The 1397 above operation can clearly be carried out as many times as necessary 1398 on the packet to provide a repair, by pushing as many tokens as are 1399 needed. 1401 Note that the scheme in this example uses an implicit disposition 1402 operation by intermediate nodes as described above in Section 10.1. 1404 11.6. Segment Routing with Latency-Based Forwarding 1406 The operation described in Section 11.5 in which repair tokens are 1407 pushed onto the packet is identical to the operation that happens 1408 when a packet is configured for segment routing (SR). Technically 1409 the packet depicted in Figure 15 IS a segment routed packet. 1411 It therefore follows that implementing segement routing and segment 1412 routing enhanced by features such as enhanced QoS is trivial in TCR. 1413 As we shall see in the next sections, TCR is capable of enhancing SR 1414 significantly beyond that. 1416 11.7. Enhanced Segment Routing with Latency-Based Forwarding 1418 This example illustrates how TCR can be used to add an enhanced QoS 1419 capability such as latency based forwarding to segment routing. We 1420 saw previously Section 11.6 how all segments can be made to execute a 1421 common policy but it might be desirable to execute a different policy 1422 in different segments. For example, some segments might underly 1423 their own latency objectives (just for that segment), without 1424 affecting the overall end-to-end objective. 1426 A packet that achieves this is depicted below (Figure 16. The packet 1427 in the example has three segments. Segments 1 and 3 apply LBF for 1428 the end-to-end latency objective, per T5. Segment 2 applies a "sub- 1429 SLO" just for that particular segment, per T4. Disposition of T1 and 1430 T2 is implicit (popping the token on reaching the segment 1431 destination), whereas disposition of T3 is explicit per T6. 1433 1434 Token Cell T1 1435 T5 1436 1437 1438 Token Cell T2 1439 T4 1440 1441 1442 Token Cell T3 1443 T5 1444 1445 1446 1448 - 1449 1450 1451 Token Cell T5 1452 - 1453 1454 1455 Token Cell T6 1456 1457 1458 1459 Token Cell T7 1460 1461 1462 1464 Figure 16: Enhanced Segment Routing Using TCR 1466 11.8. Enhanced Segment Routing with Differentiated iOAM 1468 The final example shows a variation of the previous example. Instead 1469 of applying a specific latency objectives for particular segments, it 1470 is possible to also invoke other functionality, such as collecting 1471 certain iOAM data only for particular segments, or collecting 1472 additional iOAM data for one of the segments, or even for collecting 1473 different sets of iOAM data along different segments. The particular 1474 example depicted (Figure 17) shows a packet with three segments. One 1475 set of parameters is collected for segments 1 and 3 using scratchpad 1476 T6, another set of parameters is collected for segment 2 using 1477 scratchpad T7. 1479 If instead, segment 2 should collect additional parameters beyond 1480 those collected for segments 1 and 2, this could be easily 1481 accommodated by simply setting "Next Token" of T4 to T5 instead of 1482 null. (This does not include a possible optimization for 1483 parallelization, but attests to the flexibility of the approach.) 1485 1486 Token Cell T1 1487 T4 1488 1489 1490 Token Cell T2 1491 T5 1492 1493 1494 Token Cell T3 1495 T4 1496 1497 1498 1500 - 1501 1502 1503 T6 1504 Token Cell T5 1505 - 1506 1507 1508 T7 1509 Token Cell T6 1510 - 1511 1512 1513 Token Cell T7 1514 - 1515 1516 1517 Token Cell T8 1518 1519 1520 1521 Token Cell T9 1522 1523 1524 1526 Figure 17: Enhanced Segment Routing Using TCR 1528 12. Items for further discussion 1530 This document does not constitute a finalized design and there are 1531 many design decisions that will require further discussion. The 1532 following is a partial list: 1534 o Preamble. The preamble needs further study. The goal is that it 1535 contains only the minimum of information as it needs to be popped 1536 and pushed when a token cell is popped or pushed. We need to 1537 understand if there is a need for a number of token cell's 1538 indicator, and/or a last child indicator. 1540 o Token Cell identification. As an alternative to referencing other 1541 token cells by offset, it is conceivable to introduce token cell 1542 identifiers and refer to token cells by their ID. 1544 o Manifest token cells. Rather to require processing of a full 1545 token cell, it is conceivable to express manifests as part of a 1546 token cell preamble, at least as long as the number of token cells 1547 to process in parallel are limited. This could save on stage in a 1548 token cell processing pipeline. However, it would result in a 1549 slightly longer or more complex preamble. 1551 o Manifest token cells (cont'd.) It should be noted that the 1552 current design of the manifest token cell type includes all 1553 pointers to subsequent token cells as part of the token cell blob. 1554 Alternative designs are conceivable. For example, the design 1555 might be altered to allow for the next token to be populated and 1556 only require any additional token cells to be referenced from the 1557 token cell blob. 1559 o Rendezvous token cells. As an optimization, it is conceivable to 1560 combine manifests and rendezvous points into a single token cell. 1562 o Security token cells. Further investigation is needed to 1563 determine the most effective structure, specifically compact yet 1564 efficient encodings for the token cell mask that is used to 1565 identify the portions of the packet being signed. 1567 o Disposition token cells. There are different ways that 1568 disposition token cells could be referred to for processing, 1569 including through a DT parameter (Disposition Token Cell) as part 1570 of forwarding token cells, through a separate field as part of the 1571 token cell structure, or indirectly by virtue of popping a 1572 forwarding token cell when the destination is reached and 1573 processing the token cell behind it. 1575 o Disposition token cells for processing by intermediate nodes. The 1576 example in Section 11.5 uses an implicit disposition operation of 1577 forwarding token cells by intermediate nodes, in which a 1578 forwarding token cell is simply popped and processing simply 1579 resumes with the subsequent token cell. It is conceivable to 1580 apply explicit disposition operations instead for the sake of more 1581 consistent semantics. Explicit semantics of "pop and resume 1582 processing with subsequent token cell" could be incorporated into 1583 the semantics of a forwarding token cell itself, or potentially 1584 indicated by a corresponding flag in the prefix. 1586 o Implementation profiles. Implementations may impose a reasonable 1587 limit on the number of token cells that can be serially processed 1588 at any one node. This will facilitate mapping to packet 1589 processing pipelines, which then support a predefined number of 1590 token cell processing stages per packet at line rate while 1591 maintaining constant packet processing delay at any given node. 1592 Determination of reasonable constraints is for further study. 1593 Analogous limitations may apply to the number of processing cycles 1594 that can be performed by callback functions, within which the 1595 processing of any given token cell needs to complete. 1597 o Implementation profiles (contd.). For further study are any 1598 mechanisms for the discovery of constraints as mentioned in the 1599 previous list item, as well as any capabilities for negotiation of 1600 corresponding profiles and the definition of node behavior when 1601 such limitations were to be breached (in all likelihood resulting 1602 in aborting the processing of the packet, dropping it with 1603 corresponding error code). 1605 o The design only requires forward token cell pointers and this 1606 prevents processing loops. We need to understand if this is too 1607 significant a restriction. If this restriction is removed some 1608 form of maximum tokens to be processed limit, similar in concept 1609 to TTL may be needed. 1611 13. Security Considerations 1613 This section concerns itself with the security of the TCR dataplane. 1615 The security of the control and management plane is a matter for the 1616 designers of those aspects of the solution. However, it is not 1617 anticipated that the securing of those components will be any more 1618 onerous than securing the control and management plane of other IETF 1619 designed dataplanes. 1621 Security of entry to the dataplane will depend on what entities have 1622 access to the dataplane. If TCR is used as a single domain sub-IP 1623 layer in the way that MPLS is used, then it will have the same 1624 security properties as MPLS in that it is extremely difficult for an 1625 unauthorized party to inject traffic into such a network because with 1626 TCR such traffic is easily recognized at network ingress and dropped. 1627 If the traffic is a TCR packet that is to be carried across the TCR 1628 network it will be encapsulated and so, in the absence of a TCR 1629 network error, will not be able to escape the encapsulation and cause 1630 harm. Only if a TCR network (or node) were to peer with another TCR 1631 network would there be a security concern with that third party 1632 having the ability to control the actions taken on the packet. Such 1633 a case is for further study. It should be noted that similar 1634 situations have been satisfactorily addressed in MPLS. 1636 We now need to consider the security of the contents of the packet. 1637 Clearly we could craft a token that signed the payload, and where the 1638 payload was a token, we have the option of including that signature 1639 in the token itself. However, securing the tokens themselves is more 1640 interesting because we need to authenticate selected components of 1641 the packet header, such as single tokens, groups of tokens or even 1642 components/portions of tokens. This is needed to allow the 1643 differentiation of metadata that must not be altered from scratchpad 1644 data that may be modified during packet transit. In addition, we 1645 need to allow intermediate nodes along a path to authenticate data 1646 that they modify, e.g. scratchpad data items that they create. 1648 The approach used in TCR allows the authentication of tokens and 1649 processing guidance they contain for additional security. 1650 Furthermore there is flexibility for intermediate hops to provide 1651 their own authentication, to secure scratchpad-type data added to 1652 packets along a path. This also allows for "authenticity chains" in 1653 which nodes verify the authenticity of data items they operate on 1654 before modifying and signing the update. 1656 It is clear that the above approach is different from earlier 1657 protocols where payloads are generally signed in their entirety and 1658 do not include support for differentiated signing, accommodating 1659 multiple signers of different packet aspects along a path. 1661 Most protocols secure their payload in its entirety, exposing only 1662 the packet header for processing (unless that is tunneled as well). 1663 TCR is a protocols that include additional packet components that may 1664 require more differentiated securing. Specifically, it includes 1665 guidance for how to process packets, including tokens, and metadata. 1666 In addition, TCR includes some packet components that can be modified 1667 or added by intermediate nodes in transit, specifically scratchpad 1668 data. This includes telemetry and iOAM data as well as data to 1669 indicate and verify certain properties of nodes that were traversed. 1670 While some data must not be modified, other data items might be added 1671 and/or be subject to modification. An example would be data that 1672 aggregates or analyzes telemetry data encountered in transit, for 1673 example an indicator of a minimum or maximum (queue length, 1674 utilization, latency) encountered along a path. TCR thus includes a 1675 mechanism that allows the operator to ensure the authenticity of 1676 packet data beyond the payload, but that allows them to do this in a 1677 way that exempts certain data items which are allowed to be modified 1678 along the way, with the option to allow the corresponding nodes to 1679 secure these modifications. This allows receiving applications to 1680 (for example) verify the authenticity of scratchpad data, and allow 1681 for the modification of data items where such modification is 1682 permitted without compromising the authenticity of the remaining 1683 portions of the packet. 1685 The design of the security token is described in Section 10.4 . This 1686 can only be used to sign itself and tokens or token contents after 1687 the security token. By including in the security token a mask 1688 structure it is possible to select what is to be signed. The 1689 efficiency of this method is described in Section 10.4. 1691 Matters related to inter-domain security will be considered in a 1692 future version of this text. 1694 14. IANA Considerations 1696 This document makes no IANA requests. 1698 15. References 1700 15.1. Normative References 1702 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1703 Requirement Levels", BCP 14, RFC 2119, 1704 DOI 10.17487/RFC2119, March 1997, 1705 . 1707 15.2. Informative References 1709 [DOI.10.1109_NOMS47738.2020.9110431] 1710 Clemm, A. and T. Eckert, "High-Precision Latency 1711 Forwarding over Packet-Programmable Networks", NOMS 2020 - 1712 2020 IEEE/IFIP Network Operations and 1713 Management Symposium, DOI 10.1109/noms47738.2020.9110431, 1714 April 2020. 1716 [I-D.ietf-ippm-ioam-data] 1717 Brockners, F., Bhandari, S., and T. Mizrahi, "Data Fields 1718 for In-situ OAM", draft-ietf-ippm-ioam-data-11 (work in 1719 progress), November 2020. 1721 [RFC3270] Le Faucheur, F., Wu, L., Davie, B., Davari, S., Vaananen, 1722 P., Krishnan, R., Cheval, P., and J. Heinanen, "Multi- 1723 Protocol Label Switching (MPLS) Support of Differentiated 1724 Services", RFC 3270, DOI 10.17487/RFC3270, May 2002, 1725 . 1727 [RFC3985] Bryant, S., Ed. and P. Pate, Ed., "Pseudo Wire Emulation 1728 Edge-to-Edge (PWE3) Architecture", RFC 3985, 1729 DOI 10.17487/RFC3985, March 2005, 1730 . 1732 [RFC8754] Filsfils, C., Ed., Dukes, D., Ed., Previdi, S., Leddy, J., 1733 Matsushima, S., and D. Voyer, "IPv6 Segment Routing Header 1734 (SRH)", RFC 8754, DOI 10.17487/RFC8754, March 2020, 1735 . 1737 Authors' Addresses 1739 Stewart Bryant 1740 Futurewei Technologies, Inc. 1742 Email: sb@stewartbryant.com 1744 Alexander Clemm 1745 Futurewei Technologies, Inc. 1747 Email: ludwig@clemm.org