idnits 2.17.1 draft-bcx-rtgwg-tcr-01.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document seems to lack the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords -- however, there's a paragraph with a matching beginning. Boilerplate error? (The document does seem to have the reference to RFC 2119 which the ID-Checklist requires). -- The document date (October 19, 2021) is 892 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Outdated reference: A later version (-17) exists of draft-ietf-ippm-ioam-data-15 Summary: 0 errors (**), 0 flaws (~~), 3 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 rtgwg S. Bryant 3 Internet-Draft University of Surrey 5GIC 4 Intended status: Informational A. Clemm 5 Expires: April 22, 2022 Futurewei Technologies, Inc. 6 October 19, 2021 8 Token Cell Routing Data Plane Concepts 9 draft-bcx-rtgwg-tcr-01 11 Abstract 13 Token Cell Routing is a powerful yet hardware friendly method of 14 constructing data plane packets to meet the needs of new 15 applications. It is based on the use of token cells (special kinds 16 of lightly structured tokens) to provide pointers to procedures pre- 17 positioned in the forwarding layer together with the parameters 18 needed to provide the required processing context. A packet can be 19 composed from multiple token cells as needed to result in new new 20 network processing and forwarding semantics. 22 Status of This Memo 24 This Internet-Draft is submitted in full conformance with the 25 provisions of BCP 78 and BCP 79. 27 Internet-Drafts are working documents of the Internet Engineering 28 Task Force (IETF). Note that other groups may also distribute 29 working documents as Internet-Drafts. The list of current Internet- 30 Drafts is at https://datatracker.ietf.org/drafts/current/. 32 Internet-Drafts are draft documents valid for a maximum of six months 33 and may be updated, replaced, or obsoleted by other documents at any 34 time. It is inappropriate to use Internet-Drafts as reference 35 material or to cite them other than as "work in progress." 37 This Internet-Draft will expire on April 22, 2022. 39 Copyright Notice 41 Copyright (c) 2021 IETF Trust and the persons identified as the 42 document authors. All rights reserved. 44 This document is subject to BCP 78 and the IETF Trust's Legal 45 Provisions Relating to IETF Documents 46 (https://trustee.ietf.org/license-info) in effect on the date of 47 publication of this document. Please review these documents 48 carefully, as they describe your rights and restrictions with respect 49 to this document. Code Components extracted from this document must 50 include Simplified BSD License text as described in Section 4.e of 51 the Trust Legal Provisions and are provided without warranty as 52 described in the Simplified BSD License. 54 Table of Contents 56 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 57 1.1. Requirements Language . . . . . . . . . . . . . . . . . . 4 58 2. The TCR Concept . . . . . . . . . . . . . . . . . . . . . . . 4 59 3. Relationship to prior work . . . . . . . . . . . . . . . . . 5 60 4. TCR Packet Structure . . . . . . . . . . . . . . . . . . . . 6 61 5. Token Cell types and categories . . . . . . . . . . . . . . . 7 62 6. Token Cell Structure . . . . . . . . . . . . . . . . . . . . 9 63 7. Token Cell Processing Model . . . . . . . . . . . . . . . . . 10 64 8. Token Cell Processing Order . . . . . . . . . . . . . . . . . 12 65 8.1. Serial Token Cell Processing . . . . . . . . . . . . . . 13 66 8.2. Parallel Token Cell Processing . . . . . . . . . . . . . 14 67 8.3. Combined Serial and Parallel Token Cell Processing . . . 16 68 9. Token Cell Pushing and Token Cell Popping . . . . . . . . . . 18 69 10. Selected Token Cell Type Categories . . . . . . . . . . . . . 19 70 10.1. Disposition Token Cells . . . . . . . . . . . . . . . . 19 71 10.2. Scratchpad and Metadata Token Cells . . . . . . . . . . 19 72 10.3. Conditional and Directive Token Cells . . . . . . . . . 21 73 10.4. Security Token Cells . . . . . . . . . . . . . . . . . . 21 74 11. Example applications of TCR . . . . . . . . . . . . . . . . . 23 75 11.1. Basic Tunneling of Payload . . . . . . . . . . . . . . . 24 76 11.2. Latency-Based Forwarding . . . . . . . . . . . . . . . . 25 77 11.3. Forwarding with Flexible Addressing . . . . . . . . . . 26 78 11.4. Forwarding with iOAM analytics . . . . . . . . . . . . . 27 79 11.5. FRR with Latency-Based Forwarding . . . . . . . . . . . 30 80 11.6. Segment Routing with Latency-Based Forwarding . . . . . 32 81 11.7. Enhanced Segment Routing with Latency-Based Forwarding . 32 82 11.8. Enhanced Segment Routing with Differentiated iOAM . . . 33 83 12. Items for further discussion . . . . . . . . . . . . . . . . 35 84 13. Security Considerations . . . . . . . . . . . . . . . . . . . 36 85 14. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 38 86 15. References . . . . . . . . . . . . . . . . . . . . . . . . . 38 87 15.1. Normative References . . . . . . . . . . . . . . . . . . 38 88 15.2. Informative References . . . . . . . . . . . . . . . . . 38 89 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 39 91 1. Introduction 93 Advances in data plane protocols are needed to address new network 94 requirements that stretch existing protocols (including MPLS, IPv6 95 and Segment Routing) to their limits. Token Cell Routing (TCR) is a 96 new network layer data plane technology that provides the ability to 97 program the data plane to meet the needs of many new operational 98 scenarios. TCR is based on token cells which provide a flexible 99 method describing the required packet action to the forwarder as well 100 as carrying any parameters and other data necessary to correctly 101 execute the required action. 103 Packet actions are not limited to forwarding actions, and it is 104 possible perform multiple packet actions at any given node. For 105 example, packet actions can include application of special QoS 106 algorithms, collection of telemetry, even assessment of dynamic 107 conditions before the performing of other actions. Token Cells can 108 be differentiated by type depending on the type of packet action that 109 they represent. 111 TCR is thus analogous to each packet carrying a stack of pointers to 112 procedures together with the data needed by those procedures. The 113 structure used allows token cells to be sequenced and parallelized in 114 groups, and permits the use of pointers to information in other token 115 cells. This results in a powerful method constructing advanced new 116 packet types from token cells to meet network needs of new 117 applications. 119 TCR therefore supports customizable semantics with which packets are 120 to be processed by nodes encountered along a path, it accommodates 121 flexible addressing semantics that do not necessarily depend on a 122 single addressing format. TCR accommodates custom "guidance" beyond 123 forwarding (such as the definition of QoS treatments that are to be 124 applied, the ability to differentiate behavior depending on dynamic 125 context encountered at a node, and the ability to collect and pre- 126 process telemetry in support of manageability applications). TCR 127 enables the direct processing via a "scaffold" that explicitly 128 indicates serialization, parallelization, disposition rules. It that 129 enables a "lego-esque" composition of packet processing behavior / 130 features while at the same time being hardware-friendly, and easy to 131 optimize for performance at line rate general in nature and easy to 132 extend. 134 A new TCR features is added by introducing a new procedure in the 135 forwarder and then including in the packet a pointer to that 136 procedure. The procedure knows how to interpret the other 137 information carried in the token cell. In many ways this is similar 138 to the way new FECs are introduced into MPLS and new instructions 139 introduced to segment routing. 141 TCR thus provides a general, highly extensible data plane that 142 supports custom semantics which is hardware friendly. It thus takes 143 the programmability of both MPLS and Segment Routing to a new level 144 of capability. 146 A key differentiator from earlier protocols is the ability to process 147 a variable number of processing actions at each hop, at directed by 148 the token cell structure. Furthermore token cells do not need to be 149 processed in the order in which they are placed in the packet, and 150 may be explicitly programmed for a flow. 152 We would like to note that at the time of writing the current -00 153 version of the draft, this represents a sketch of an idea that 154 neither we or anyone else has built. Thus, it is likely to have bugs 155 and certainly has many aspects that can be improved on. We would be 156 delighted to work with others who are interested in exploring this 157 idea and developing it further. A starter set of discussion items is 158 included in Section 12 towards the end of the document. 160 1.1. Requirements Language 162 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 163 "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and 164 "OPTIONAL" in this document are to be interpreted as described in 165 BCP14 [RFC2119] [RFC8174] when, and only when, they appear in all 166 capitals, as shown here. 168 2. The TCR Concept 170 The foundation of TCR is the construction of the packet from a set of 171 token cells. A token cell is an extended length, type, value 172 construct. The type and part of the value is processed by a longest 173 match engine, which operates much like an IP address lookup engine, 174 but operates on arbitrary constructs rather than being confined to 175 address lookup. As part of its value, the token cell may also carry 176 parameters specific to the token cell type that are needed to process 177 the token cell. Depending on the token cell type, different code 178 points can be invoked that process the token cell. The token cell 179 type determines the semantics, i.e. the function to be applied. It 180 also defines any structure that may be contained in the token cell 181 value. 183 Token Cell processing can have generalizable packet processing 184 semantics: forwarding is a common semantic, but other semantics can 185 be applied, in a similar manner to the way in which an MPLS label has 186 generalized semantics. The processing of a token cell is: Input - 187 Match - Effect, where "effect" is one of forwarding, token cell 188 disposition, or something else (such as, conditional directives or 189 QoS treatment). Packet processing based on the first n bits of the 190 token cell at a known position, which is matchable on prefix, in 191 which case less significant bits serve as input. 193 The TCR approach to packet design is differentiated from common 194 protocol designs in that it allow for processing of variable number 195 of token cells per hop, as directed by token cell structure. Which 196 token cells to process, and whether token cells have 197 interdependencies that require them to be processed in serial order 198 or whether they allow for parallelization, is indicated by the token 199 cells themselves, as the token cell structure includes length and 200 serialization indicators. Processing can be serialized per "next 201 token cell" indicator. Processing can be parallelized per "manifest 202 token cell" that refers to parallel token cells in order to allow for 203 optimization. Note that there is no requirement to "must" 204 parallelize. Instead, parallelization is an optimization that nodes 205 with support for parallel packet processing stages may take advantage 206 of to reduce packet latency due to packet forwarding time. 208 Token Cells do not necessarily need to be processed in stack order 209 but can be located wherever is most efficient. Some token cells may 210 not be processed at all but can be used to carry meta-data (read-only 211 or writable) that can be referred to from other token cells. 213 The TCR approach allow for extensibility and programmability through: 215 o Level 1: Combining different token cells 217 o Level 2: Parameterizing token cells 219 o Level 3: Introducing new types of token cells 221 3. Relationship to prior work 223 In general data plane packets have been designed with a fixed 224 structure plus some variable number of TLVs to provide additional 225 instructions/advice to the forwarder. In general any address (for 226 example IP address) or instruction (MPLS label of SR SID) is of a 227 fixed length although may be structured into a prefix and suffix 228 arrangement to support aggregation and is processed using a longest 229 match lookup. Where TLVs are used there is their inclusion in the 230 packet is normally free form and thus it is necessary search the 231 packet for the set of TLVs that need to be processed. MPLS has a 232 very primitive parameter system in which one label may be used to 233 provide context for the label that follows. Examples of this are the 234 use of the context label and the use of the ELI/EL pair. 236 In deigning TCR we noted the generality and simplicity of the MPLS 237 label stack model and the effectiveness of the longest match 238 technique used in IP lookups. Putting these two together we 239 concluded that an LTV approach allowed the creation of a simple, 240 powerful, extensible, hardware friendly packet design. 242 As we will see later in the document, concatenating the T(ype) and 243 the V(alue) allows a single longest match look-up to resolve which 244 table to look up the action in, and then to absorb as much of the V 245 as is necessary to determine the specific action to take on the T. 246 The stack and pointer structure means that it is not necessary to 247 search for the applicable TLVs, they the forwarder is led to them 248 through the token cell structure pushed onto the packet. 250 The general method of processing is thus input - match - effect via a 251 match processor. The normal effect is to process one token cell at a 252 time in series, but an effect might be to process multiple other 253 token cells in parallel where the forwarder supports multiple 254 concurrent operations (where parallization is not supported serial 255 processing results in the same effect). Token Cells can have 256 interdependencies and they can allow for cross-referencing (e.g. 257 meta-data, scratch-pad) 259 4. TCR Packet Structure 261 A TCR packet Figure 1 is a series of token cells, each token cell 262 carrying a component of the packet delivery system or the payload 263 itself. 265 266 267 268 269 ... 270 272 Figure 1: Structure of a TCR Packet 274 Token Cells are a unit of packet processing that may include 275 parameters and/or data. The semantics, structure and processing of 276 the token cell is determined by the token cell type. A Token Cell 277 can be thought of as a type of "stem cell" that can be morphed into 278 any packet component, including components not yet designed, 279 resulting in a highly programmable packet design. 281 The preamble contains a small number of packet elements that are 282 always present in all packets, such as version identification and 283 TTL. It has yet to be decided if the preamble should be carried 284 conventionally or be carried within a token cell. 286 There is no separate payload portion of the packet. Instead, the 287 payload is carried within a token cell, generally located at the tail 288 of the packet. This allows for different payload delivery semantics 289 at the destination, including simply stripping the payload off or 290 applying a special type of codec. It also allows for the possibility 291 of payload-less packets that can be used for signaling and control 292 purposes. 294 A token cell MAY contain a complete TCR packet permitting 295 hierarchical encapsulation. 297 5. Token Cell types and categories 299 Token Cells have a type. Types themselves can be categorized, 300 depending on the purpose which token cells of that type serve. 302 The following token cell categories are provided as part of the TCR 303 design: 305 o Forwarding: A forwarding token cell specifies the destination 306 address and method of delivery of the packet. It may also include 307 the source address as a parameter, but this could also be 308 specified in a separate token cell. Different types within this 309 category may differentiate between address types such as IPv6 or 310 IPv4. 312 o SLO: A Service Level Objective (SLO) token cell specifies the 313 target quality of delivery, such as latency, delivery time, 314 required discard properties etc., each differentiated by 315 corresponding IDs as separate types. This allows intermediate 316 nodes on the path to apply special treatment to the packet, such 317 as scheduling algorithms, resource reservation, or prioritization, 318 in order meet SLOs as requirement. 320 o Metadata: These token cells carry metadata that can be referenced 321 and accessed as other token cells are being processed. Metadata 322 can thus be decoupled from token cells that access it, allowing 323 for their independent disposal, not interfering with pushing and 324 popping of other token cells. 326 o Scratchpad: Scratchpad token cell are in effect writeable metadata 327 token cells, a category of token cell in which the network takes 328 "notes" during the packet transit. Examples of this include 329 recording the route, adding proofs-of-transit, telemetry data, or 330 packet transit time of particular nodes that were traversed. 332 o Security: A security token cell signs parts of the packet with an 333 agreed cryptographic signature. It includes a signature mask that 334 specifies which token cells and/or portions thereof are covered by 335 the signature. This allows for the possibility of not only the 336 sender, but also nodes in transit being able to sign portions of 337 the packet. One example use would be for telemetry data that is 338 added to a scratchpad by a node being traversed while leaving 339 other parts of the scratchpad open to modification by other nodes. 341 o Conditional: A conditional token cell is able to test one of more 342 conditions and make invokation of the next token cell (NT) 343 dependent on the conditions evaluating to true. This allows to 344 define more complex behavior, such as the invocation of a 345 particular function depending on a dynamic condition encountered 346 at a node. 348 o Directive: A directive token cell specifies some type of action 349 that should be performed. An example would be a directive to 350 collect telemetry or OAM data. 352 o Manifest: A manifest token cell provides a method of specifying 353 which token cells may be processed in parallel. Parallel 354 processing is optional, and the token cells can also be correctly 355 processed serially. It is up to the entity that specifies the 356 manifest to ensure that the parallelism is safe. 358 o Rendezvous: A rendezvous token cell is a token cell used to ensure 359 that all parallel operations have completed and that it is hence 360 safe to resume serial operation of the forwarder. A rendezvous 361 token cell may specify the first serial operation to execute after 362 the rendezvous, or it may simply hand off to a new token cell. 364 o Disposition: A disposition token cell describes what is to be done 365 when the packet leaves the TCR domain. Such a token cell might, 366 for example specify a pseudowire [RFC3985] action (strip the TCR 367 header and send the payload to interface X), or a VPN action 368 (lookup the payload IP address in VRF Y). However, the mechanism 369 introduces the opportunity to attach a more sophisticated 370 disposition action, for example "if the packet arrives before time 371 T, forward using VRF V, otherwise drop the packet". 373 o Payload: A payload token cell simply carries the payload as its 374 value. 376 o Other: This is a catch-all category to allow for token cell types 377 that do not fit any of the other categories. 379 Some of the mentioned categories will be described in greater detail 380 in Section 10. In addition to the mentioned categories, it is 381 expected that other categories would be introduced as needed. 383 The grouping of types into categories may prove useful for various 384 purposes. For example, it will allow for the articulation of packet 385 grammars and "best packet practices", such as mandating that a packet 386 contain at least one token cell of the forwarding category, or that 387 the first token cell in a TCR packet must not be a token cell of 388 categories metadata, scratchpad, or security. Types allow to further 389 differentiate token cells within a given category. 391 It should be noted that some of the token categories should be 392 considered experimental. Which token types and even which token 393 categories to support will in depend on the needs of actual 394 deployments. 396 6. Token Cell Structure 398 The structure of a token cell is shown in Figure 2: 400 401 402 -+ 403 | 404 +- Match Zone 405 | 406 -+ 407 409 Figure 2: Structure of a TCR Token Cell 411 Token Cells will vary in length depending on the token cell type and 412 the contents of the token cell blob, although in practice a given 413 token cell type MAY be a fixed size. 415 Next token cell is a relative pointer (offset) to the next token cell 416 to be processed as part of the group of token cells to be 417 sequentially processed as a part of the packet action at the node. 418 If there is no next token cell to process, its value is null. 419 Processing of a packet begins with processing the first token cell. 420 Whether or not any additional token cells are processed depends on 421 the guidance provided via the Next Token field. 423 The token cell type is used to identify the category or purpose of 424 the token cell (Cat/Purpose) and the sub-type (ID) within that class. 425 An example of a purpose might be "Forwarding", indicating that the 426 token cell represents an instruction to forward the packet closer to 427 the destination. An example of a sub-type within that category might 428 be "IPv6", indicating tht the destination is identified by the 429 following IPv6 address. 431 The set of IDs consists of a set of well known IDs and a set of user 432 specified IDs. This provides both an extensible, and a programmable 433 mechanism for enhancing the protocol over time and within 434 deployments. Token cell type, category, and sub-type ID are not 435 fixed-length fields but represent conventions. 437 The token cell blob carries the information needed to process the 438 explicit packet that carries it, and/or is a place to record 439 information about the packet for later use. Within the token cell 440 blob there is a prefix and a suffix, which may themselves each be 441 structured. The structure of the token cell blob depends on the 442 token cell type, some of which may themselves be subjected to 443 standardization. 445 The token cell blob prefix, which is neither fixed nor a fixed length 446 field is used to qualify the type in the lookup. For example, if the 447 sub-type was IPv6 destination address the prefix would be an IPv6 448 address. However this is a more general device than just a 449 destination address and allows for token cell type categorization. 450 This may prove useful for various purposes (e.g. packet grammars and 451 Best Packet Practices). 453 The match zone is the portion of the token cell that is subject to 454 look up (see processing model section). The model is that the lookup 455 will be a longest match of the whole match zone. The token cell 456 design does not specify the length of the match zone or the length of 457 either the ID or the prefix. It is a property of longest match 458 lookup that it will either consume all the bits it needs, or reject 459 the lookup. If a result is found the result can specify the 460 structure of the token cell blob. Note that this is a model, and 461 there is much scope for implementor optimization without sacrificing 462 the generality of the design. 464 The number of bytes sent to the lookup engine is implementation 465 specific. If the attempted match is longer than needed, the longest 466 match will ignore the overspill. If more bytes are needed, it is a 467 property of longest match that the lookup can be restarted from where 468 it left off. 470 Within the blob, and in particular within the suffix, any structure 471 can be carried such as subfields, parameters. The definition of what 472 the suffix contains and how it is structured is part of the token 473 cell type definition. 475 7. Token Cell Processing Model 477 The TCR processing model is shown in Figure 3. 479 ______________________________ 480 |< Token Cell Match Zone >| 481 || 482 |______________________________| | 483 || || | 484 || || | 485 \ / | 486 +---v---+ | 487 | | 488 V | 489 Lookup | 490 Engine | 491 | | 492 | | 493 V | 494 Lookup o/p parameters | Pipeline 495 Code Pointer | state register 496 | | ! ^ 497 | | ! ! 498 V | ! ! 499 Code <-----------------+ ! ! 500 "callback" <~~~~~~~~~~~~~~~~~~~~~' ! 501 | ! 502 | ! 503 V ! 504 Effect ~~~~~~~~~~~~~~~~~~~~~~~~~~~~' 506 Figure 3: TCR Processing Model 508 This operates as follows. First the token cell match zone is fed 509 into the lookup engine. The lookup engine performs a longest match 510 and returns a parameter block which includes the address of the code 511 to be executed on the token cell by the forwarder. That code knows 512 how to interpret the token cell. Readers will recognize the genesis 513 of this is a hybrid between an IP address lookup which performs a 514 longest match lookup and returns a parameter block to the IP 515 forwarding code, and an MPLS label lookup which performs a fixed size 516 look-up logically returns a pointer to the executable code (MPLS 517 forward packet, pseudowire, VPN etc) and a parameter block. 519 Where it cannot be clear that what the length of the blog prefix is 520 from the lookup result, for example where the address is an IPv6 521 address, that length needs to be encoded in the prefix in some 522 convenient way, such as as a prefix to the prefix. 524 From the above, it is clear that there is no constraint on the type 525 and structure of the prefix and thus any address type or other 526 construct may be submitted to the lookup engine. Token Cell types 527 and prefix sets may be added to the forwarder by adding appropriate 528 data to the database queried by the lookup engine, and providing the 529 corresponding callback code to the network processor, in a manner 530 similar to that used in MPLS and in Network Programming. In this 531 regard the approach is compatible with proven hardware forwarding 532 models. 534 The callback code has access to any other element of the token cell 535 that it needs, and indeed to other elements of the packet as 536 required. Implementations MAY impose a reasonable limitation on the 537 number of processing cycles within which callback code needs to 538 complete. 540 As part of the effect of processing a token cell, a pipeline state 541 register can be written which can serve as input for the processing 542 of subsequent token cells of the same packet. This allows to compose 543 a pipeline of token cells being processed in which the output of one 544 function serve as input to the next function. In the special case of 545 rendezvous token cells that have multiple predecessors, their 546 respective outputs are merged as part of the specific rendezvous 547 semantics. 549 8. Token Cell Processing Order 551 It is entirely possible to require several token cells to be 552 processed at any given node. For example, one token cell may contain 553 a forwarding directive, whereas another token cell might contain a 554 directive related to QoS treatment of the packet for meeting a 555 Service Level Objective (SLO), while a third token cell indicates 556 that certain telemetry data from traversed nodes should be collected. 558 In the simple case that token cells can or should be serially 559 processed, the processing of token cells will simply be chained, as 560 directed by the token cells' respective NT fields. After processing 561 of a token cell concludes, the reference in the NT field is resolved 562 and processing continues with that token cell. If the NT contains no 563 reference (or in the special case of a conditional token cell 564 evaluating to false), the processing of the packet concludes. In 565 that case, the processing of a packet will require n stages, n being 566 the number of chained token cells. 568 A packet processing pipeline needs to support a depth that is 569 equivalent to the maximum number of token cells that can be chained. 571 In some cases, optimization is possible by exploiting 572 parallelization. In the earlier example, it may be possible to 573 perform some tasks in parallel, such as the application of QoS 574 treatment and collection of telemetry data, while other tasks may 575 still need to be serialized, such as determining which outgoing 576 interface to use as a forwarding decision before performing the QoS 577 actions against that interface. The fact that certain token cells 578 may be processed concurrently can be indicated through a special 579 manifest token cell that references the token cells that follow. 580 Each of those token cells can in turn have their own successors, in 581 effect resulting in separate packet processing "threads" (all 582 processing different token cells of the same packet). While the 583 processing of the manifest adds an additional cycle, depending on the 584 complexity of the workflow this may be more than offset by the 585 parallelization that ensues. 587 While full exploitation of the optimization potential may require 588 advances in hardware pipeline design, it should be emphasized any 589 such optimization is optional and not required. Also, to maintain 590 packet ordering, the packet will generally still be required to pass 591 through all n pipeline stages. That said, the option of 592 parallelization does allow for hardware pipeline designs able to 593 exploit concurrency of multiple threads and accommodating a larger 594 number of token cells than would otherwise be supported by pipelines 595 of a given depth, respectively reduce the pipeline depth that needs 596 to be supported. 598 8.1. Serial Token Cell Processing 600 All packets have an element of serial processing in that the preamble 601 and then the token cell that follows are always processed. 603 A serial group of token cells is constructed by using the next 604 pointer to point to the token cell to processed on completion of the 605 token cell. The end of a serial token cell group is logically 606 indicated using a NULL next token cell pointer, although none of the 607 foregoing should be taken as dictating the wire format which will be 608 the subject of another text. 610 Token Cell T1 611 T2 612 613 614 Token Cell T2 615 T4 616 617 618 Token Cell T3 619 - 620 621 622 Token Cell T4 623 - 624 625 627 Figure 4: TCR Serial Token Cell Processing of Token Cells 629 Figure 4 shows four token cells. The serial processing instructed by 630 this construct is that token cell 1 is to be processed, followed by 631 token cell 2 and then followed by token cell 4. There are many 632 reasons why this construct is interesting and these are discussed 633 later in this document. 635 In order to simplify the graphical annotation, references to token 636 cell cells are in the following simply indicated by a numeric 637 identifier instead of being depicted by arrows. 639 8.2. Parallel Token Cell Processing 641 In some cases it is desirable to introduce parallel processing to the 642 packet where there are token cells have no result interdependencies. 643 We do this through the introduction of a manifest token cell that 644 contains a set of pointers or offsets to other token cells. 646 647 648 -+ 649 | 650 +- Match Zone 651 | 652 -+ -+ 653 | 654 .. +- Manifest 655 | 656 -+ 658 Figure 5: TCR Parallel Processing of Token Cells 660 The structure of the manifest token cell is shown in Figure 5. The 661 initial part of the token cell is standard to all token cells. There 662 follows a list of token cell pointers to the set of token cells to be 663 processed in parallel. The end of the set of token cells to be 664 processed in parallel is determined when the end of the token cell is 665 reached as indicted by the token cell length. When all the child 666 token cells have completes execution the token cell pointed to by the 667 next token cell field is executed. 669 Note that the match zone overlaps the manifest. The token cells in 670 the manifest will however be ignored by the longest match which will 671 complete with the ID. 673 Also note that, as previously mentioned, parallelism is an efficiency 674 issue, not a correctness issue. A forwarder that does not support 675 the parallel dispatch of token cells or that supports less 676 parallelism than specified in the manifest can choose to execute 677 individual token cells (respectively groups of token cells) serially. 678 It is up to the sender to construct the packet as needed and make any 679 token cell interdependencies explit, without requiring the network to 680 second-guess whether or not there are any such interdependencies. In 681 other words, when the order in which token cells are processed might 682 result in different behavior, it is the responsibility of the sender 683 to specify any required serialization as needed. 685 In most cases where it is used, a manifest token cell will typically 686 be the first token cell after the preamble, to exploit the 687 possibility of concurrent processing threads for multiple token cells 688 in the same packet from the onset. However, this is not an 689 architectural requirement and manifest token cells could also occur 690 later in the packet. 692 It is possible to remerge concurrent token cell threads using a 693 special rendezvous token cell. A rendezvous token cell awaits for 694 each of its predecessors to have completed before resuming 695 processing. In addition, output from predecessors (i.e. pipeline 696 state register content) can be aggregated as needed. 698 8.3. Combined Serial and Parallel Token Cell Processing 700 Figure 6 illustrates the concept of parallel and serialized 701 processing further. Please note that this is an admittedly complex 702 scenario and most scenarios in practice will be simpler. 704 T1 705 / | \ 706 / | \ 707 T2 T3 T4 708 | / | \ 709 T5 T6 T7 T8 710 | 711 T9 713 Figure 6: TCR Combined Serial and Parallel Processing of Token Cells 715 After processing T1, three tokens (T2, T3, T4) can be processed 716 concurrently. T5 has a serialization dependency on T2. T6, T7, T8 717 can be procesed concurrently once T4 has been processed. Finally, T9 718 has a serialization dependency on T7. 720 A corresponding packet is shown in Figure 7. Manifest token cells 721 are introduced to represent the fact that T2, T3, T4 respectively T6, 722 T7, T8 can be processed in parallel. A Next Token field of "-" 723 indicates there is no next token. 725 Token Cell T1 726 M1 727 728 729 Token Cell M1 730 - 731 732 733 T2 734 T3 735 T4 736 Token Cell T2 737 T5 738 739 740 Token Cell T3 741 - 742 743 744 Token Cell T4 745 M2 746 747 748 Token Cell M2 749 - 750 751 T6 752 T7 753 T8 754 Token Cell T5 755 - 756 757 758 Token Cell T6 759 - 760 761 762 Token Cell T7 763 T9 764 765 766 Token Cell T8 767 - 768 769 770 Token Cell T9 771 - 772 773 775 Figure 7: TCR Combined Serial and Parallel Processing of Token Cells 777 Token Cell M1 is a manifest that indicates three children (at the 778 protocol level, the number of potential children is subject only to 779 packet size constraints). From a protocol perspective all three 780 children: token cell 2, token cell 3 and token cell 4 can execute 781 immediately and concurrently. When token cell 2 completes token cell 782 5 runs, when token cell 5 completes that that processing branch is 783 completed. Token Cell 3 runs and when it completes that token cell 784 branch is completed. Token Cell M2 is also a manifest with three 785 children: token cell 6, token cell 7 and token cell 8. When token 786 cell 6 completes that processing branch is completed. When token 787 cell 7 is completed token cell 9 runs. When token cell 9 completes 788 that processing branch is completed. When token cell 8 is completed 789 that processing branch is completed. When token cells 5, 3, 6, 9 and 790 8 are completed the token cell group is completed. 792 It should be noted that the current design of the manifest token cell 793 type includes all pointers to subsequent token cells as part of the 794 token cell blob. Alternative designs are conceivable in which (for 795 example) the next token field would be populated with the first of 796 the concurrent successors, with only additional token cells (beyond 797 the first) to be referenced from the token cell blob. 799 It should also be noted packet permutations can be accommodated, i.e. 800 the order of the token cells in the token cell group can be any order 801 convenient to the network application designer. For example, the 802 same token cells from Figure 7 could have been arranged also in the 803 following sequence: T1 - M1 - T2 - T5 - T4 - T3 - M2 - T6 - T7 - T9 - 804 T8. The token cells do not need to be arranged in a stack in the way 805 that MPLS or SRv6 arrange their labels or SIDs respectively; the 806 order in which to process token cells is always resolved by the NT 807 reference (respectively any manifest token cell references). However 808 it is RECOMMENDED that backward references are avoided as a packet 809 with no backward references will not form a processing loop. 811 9. Token Cell Pushing and Token Cell Popping 813 Token Cell groups can be stacked by pushing a group of token cells 814 onto a packet to encode a hierarchical set of operations to be 815 executed on the packet as it journeys to its destination. This is 816 similar to IP tunneling or the pushing and popping of MPLS labels. 818 In a manner similar to the pipe model described in [RFC3270] it is a 819 matter for the protocol designer and the operator whether the pushed 820 token cells are able to understand and interact with existing the 821 tokens cells. This will be discussed further in a future version of 822 this document. 824 Similarly when a token cell group has accomplished its purpose on the 825 packet journey it is popped so that the forwarded can gain access to 826 the next element of processing. 828 Again considerations similar to [RFC3270] may apply. 830 For the purposes of this discussion a Token Cell group may consist of 831 a single token cell. 833 Considerations regarding penultimate hop popping will be included in 834 a future version of this text. 836 10. Selected Token Cell Type Categories 838 An overview of token cell type categories was given in Section 5. In 839 this section, some of these categories will be explained in further 840 detail. 842 10.1. Disposition Token Cells 844 A disposition token cell describes to the network what actions are to 845 be performed as a packet egresses the TCR domain. A forwarding token 846 cell may for this purpose include a reference to a disposition token 847 cell as a parameter. 849 All existing IETF network protocols include a disposition semantic 850 although sometimes this is implicit. For example when IPv6 has a 851 next header of "TCP" the disposition instruction is to dispose of the 852 IPv6 header and hand the packet to the TCP handler. Similarly 853 pseudowires have a disposition instruction in the PW label 854 instructing the forwarder how to dispose of the MPLS header and to 855 reconstruct the packet in its original format. 857 However as the metadata carried in the packet become more 858 sophisticated there is a requirement for the disposition to become 859 more sophisticated. 861 Disposition token cells thus formalize the description of the egress 862 behavior of the network on the packet and allow a richer egress 863 semantic to be described. 865 It is conceivable to define TCR such that in the absence of a 866 reference to a disposition token cell, TCR will revert to implicit 867 disposition behavior when the destination address of a forwarding 868 token cell is reached. That will involve popping the token cells up 869 to the forwarding token cell and resuming processing with the 870 subsequent token cell. In other words, a disposition token cell is 871 in that case used to specify any special semantics that would go 872 beyond vanilla implicit disposition. 874 10.2. Scratchpad and Metadata Token Cells 876 Metadata, provided by a sender of a packet or by an ingress node, can 877 provide important guidance to nodes along a path to guide processing 878 of the packet. Examples include SLOs that should be taken into 879 account for QoS treatment, profiles to apply towards proessing a 880 packet, and more. Other uses include the carrying of security 881 material as well as the tagging of packets for classification 882 purposes. 884 Scratchpads refer to writeable metadata, i.e., metadata that can not 885 only be accessed but also modified or added by nodes "in flight", 886 i.e. during transit. Example uses include collection of telemetry 887 and iOAM data [I-D.ietf-ippm-ioam-data], auxiliary data used for 888 measurements such as intermediate time stamps, or proof-of-transit 889 data and data verifying certain properties of nodes being traversed 890 (such as whether from a trusted vendor or located in a certain 891 region). 893 Metadata and scratchpad token cells allow to carry such metadata as 894 part of a packet independent of the token cells that access it. This 895 decoupling allows to dispose of metadata and scratchpads independent 896 of token cells that process it. For example, it makes it conceivable 897 to collect different iOAM data along different segments of a path, as 898 directed by different token cells, and to export the data only once 899 an exporter or egress node is reached. This way, scratchpad token 900 cells enable applications that rely on sharing of node-specific data 901 along a path to do so without the complications of having to 902 introduce piggyback extensions to the underlying protocol. In 903 addition to adding and updating data items in scratchpad token cells 904 along a path, also additional scratchpads can be added. It also 905 avoids the need for the same metadata to be copied across token 906 cells, for example in cases where the same SLOs are applicable across 907 different segments, each governed by their own respective token 908 cells. 910 Metadata and scratchpad items can be referenced by other token cells. 911 The specific format of those references is to be determined by the 912 rules governing the content of the respective token cell blobs for 913 the token cell type; in general the reference format will involve an 914 offset from the referring token cell. 916 Disposition semantics need to include defining what is to happen with 917 metadata or scratchpad carried in corresponding token cells. 918 Possible disposition actions include: 920 o Discard 922 o Export (possibly parameterized to specificy export mechanism as 923 well as export target) 925 o Log (again, possibly parametrized with a logging target) 927 Metadata and scratchpad data can also be independently authenticated 928 and secured. This allows, for example, to ensure that scratchpad 929 data that is added or modified by intermediate nodes cannot be 930 tampered with. It also allows for the implementation of operations 931 that authenticate both metadata and scratchpad data before processing 932 it further. 934 Metadata, let alone scratchpad data, is a concept that is not 935 directly supported by token-based protocols such as SR or MPLS today. 936 While it would be possible to encode such as part of a token, its 937 processing would require the token itself be processed (e.g. as part 938 of a forwarding operation), tied to its being pushed or popped on a 939 stack. This would make it harder to e.g. update scratchpad data 940 (that should be protected from popping and may be buried underneath a 941 token or label stack), to access data without effectively copying it 942 across tokens, and generally to accommodate metadata and scratchpad 943 processing where the lifecycle of data items does not directly 944 correspond to that of segments. With TCR, there is no need to 945 process metadata in "stacked order" nor need for complex token cell 946 rewrite rules in order to preserve data. 948 10.3. Conditional and Directive Token Cells 950 Directive Token Cells allow the specification of some type of action 951 or function that should be performed as a result of the token cell 952 being processed. An example of a directive would be a request to 953 collect telemetry or in-situ OAM data and record it as part of 954 scratchpad. It is expected that there can potentially be a large 955 number of possible directives, each distinguished by its own ID 956 respectively token cell type. Different token cell types may impose 957 their own respective structure on the token cell blob to represent 958 different parameters. 960 Conditional Token Cells are similar to Directive Token Cells in that 961 they allow the specification of an operation to be performed. 962 However, it is different from other categories of tokens in that the 963 outcome of the operation determines whether the NT field will be 964 processed, i.e. whether processing will subsequently resume with the 965 token cell referenced by the NT field or whether it should terminate. 966 This allows for the definition of functionality that should be 967 performed depending on certain conditions that are being encountered. 968 For example, a conditional token cell might allow for a different 969 paths to be selected depending on dynamic circumstances such as load 970 conditions. Similarly, the collection of certain telemetry data 971 might be made dependent on certain conditions being encountered. 973 10.4. Security Token Cells 975 Security token cells enable a security mechanism that allows to sign 976 the invariant elements of the packet whilst avoiding signing the 977 packet elements that are modified during the passage of the packet 978 through the network, such as scratchpad token cells. 980 Likewise, they allow for differentiated signing of different parts of 981 the packet by different signers, such as in cases where nodes add 982 data items to a scratchpad that should be independently signed. 984 A security token cell allows to carry signature material pertaining 985 to elements of the packet. It includes the following items: 987 o An identification of the signer 989 o A mask indicating the parts of the packet respectively token 990 cell(s) being signed 992 o The signature material itself 994 The general structure of the security token cell is as follows 996 997 998 -+ 999 |- Match Zone 1000 -+ 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1012 Figure 8: Structure of a TCR Security Token Cell 1014 The token purpose and ID specify that this is a security token, the 1015 exact structure of the security token, and the type of signature that 1016 has been used. The structure used here is illustrative and used to 1017 explain the concept. 1019 The token cell blob contains the security material and the mask. A 1020 possible structure is as follows: 1022 As described in [RFC8754] the Key ID is used to identify the 1023 preshared key and the algorithm, though the algorithm may be 1024 indicated by the token ID (this is a matter for the token designer). 1026 The Hashed Message Authentication Code (HMAC) is the hash of the 1027 complete TCR security token and the N TCR tokens, and within those 1028 tokens the token elements specified by the token mask. 1030 For efficiency the mask operates at a number of levels: 1032 o The next N token cells 1034 o The marked tokens within the next N token cells 1036 o Octets or words within a specific token cell 1038 N specifies the number of tokens covered by this security token in 1039 addition to itself. If present, Token Cell Mask is a structure that 1040 specifies which tokens are actually protected. Token_Protected.NP is 1041 the number of tokens within the token mask. 1042 Token_Protected.Full_Token is a bit mask of Token_Protected.NP bits 1043 indicating which complete tokens are to be included in the signature. 1044 Token_Protected.Byte_Protected is a bit mask of Token_Protected.NP 1045 bits indicating which complete tokens are to be included in the 1046 signature. 1048 11. Example applications of TCR 1050 This section contains a number of examples illustrating how the TCR 1051 token system is used to create or "program" packets. The examples 1052 are illustrative and not exhaustive. Only the essential features of 1053 the packet are shown. 1055 In order to simplify depiction of TCR packets, specifically the order 1056 in which tokens are processed and the cross-dependencies between 1057 tokens, we will introduce a syntax which identifies individual tokens 1058 by a numeric token identifier, and that allows to reference tokens 1059 using this identifier as opposed to packet offsets depicted using 1060 ASCII art pointers. This is merely done for convenience of textually 1061 representing TCR packets in this draft. It is independent of the 1062 actual TCR packet and token structure, in which tokens do not have 1063 separate identifiers and are simply referenced by offset. This 1064 applies to Next Token fields, as well as for disposition. A 1065 disposition token cell is referenced through a field in token cells 1066 whose type is of the forwarding category. 1068 A packet is simply represented by a sequence of tokens. Each token 1069 is represented by a set of fields and their values. In addition, a 1070 "pseudo field" is introduced for the numeric token identifier. A 1071 reference to another token (from the field, from a 1072 manifest token, or from a rendezvous token) contains as value the 1073 respective token identifier. 1075 11.1. Basic Tunneling of Payload 1077 In this example we consider a the case where a packet is to be 1078 tunneled across a network as one might do in a sub-IP network. 1080 For illustrative purposes we show in Figure 9 an IPv6 address being 1081 used as a destination address, but any other address family can be 1082 used with the corresponding forwarding token type. 1084 1085 Token Cell T1 1086 - 1087 1088 1089 1090 Token Cell T2 1091 1092 1093 1094 Token Cell T3 1095 1096 1097 1099 Figure 9: Basic Tunnelling Over TCR 1101 The packet starts with a preamble followed by token T1 which is IPv6 1102 forwarding token, a token with the semantic that it is to deliver the 1103 packet as close to the target address as it can reach. There is no 1104 next token cell to be processed until arrival at the destination and 1105 so Next Token is NULL. 1107 When the packet arrives at its destination the token that is pointed 1108 to by the disposition pointer is noted and all tokens up to the 1109 disposition token are popped (in this case only token 1 is popped). 1111 The disposition token T2 is processed and the disposition parameters 1112 say how the payload is to be processed. This may be as simple as 1113 providing the protocol type of the payload, but it may also provide 1114 information other information such as the identity of a VRF table to 1115 use, or an interface to dispatch the packet to in the case of a 1116 pseudowire. It also provides a pointer to the payload. When the 1117 parser knows how to dispose of the packet, it pops Token 2 and 1118 processes token 3. Token 3, in the case of containing a tunneled IP 1119 packet, just requires the length to be noted and corrected for the 1120 token overhead, the type confirmed as payload and payload to be 1121 forwarded as an IP packet. 1123 11.2. Latency-Based Forwarding 1125 In this example we consider the previous case (basic tunneling of 1126 payload) Section 11.1 and extend it to provide support for on-time 1127 delivery according to an end-to-end latency objective. The support 1128 is provided using a Latency-Based Forwarding (LBF). LBF bases the 1129 decision on when to sent to packet on how urgently or at what exact 1130 time it needs to arrive. To do so, LBF involves an algorithm that 1131 determines at any given node whether the packet is "on track" to meet 1132 its latency objective, and matches the QoS treatment and scheduling 1133 of the packet against a latency "budget" that is determined from 1134 latency objective, the latency that was already incurred, and the 1135 expected remaining latency towards the destination. For further 1136 details, please refer to [DOI.10.1109_NOMS47738.2020.9110431]. 1138 To indicate that a packet should undergo LBF treatment, a 1139 corresponding token cell type of category "SLO" is introduced. When 1140 it is processed, the packet is subject to LBF. Parameters for LBF 1141 include the end-to-end latency objectives and a helper parameter to 1142 assess the latency being incurred. An additional input is the egress 1143 interface that is obtained from processing of the fowarding token 1144 cell that precedes it and that can be passed using TCR's pipeline 1145 state register. 1147 A packet that enables LBF in conjunction with the preivious case is 1148 depicted in Figure 10. It adds one additional token cell over the 1149 previous use case, in essence adding the LBF functionality as a 1150 "module" that is combined with the earlier functionality. The 1151 ability to compose functionality by simply combining corresponding 1152 token cells into a packet is part of what makes TCR quite powerful. 1154 Processing of the packet is similar to the processing of the basic 1155 tunneling case except that Token 1 the IPv6fwd token contains a 1156 pointer to a further token that is to be sequentially processed at 1157 every hop, including arrival at the destination. The Next Token 1158 pointer in Token 1 points to token 2 which specifies that the packet 1159 must arrive at a specified time, and contains information needed to 1160 record the time taken and hence the time at which it should optimally 1161 leave its current node. 1163 When the packet arrives at its destination a final check is made to 1164 see if the arrival time requirement was met and it is processed 1165 according to the failure to arrive on time instructions in either the 1166 LBF_ontime token or the disposition token. All further processing is 1167 as per the above basic tunneling case. 1169 1170 Token Cell T1 1171 T2 1172 1173 1174 1175 Token Cell T2 1176 - 1177 1178 1179 Token Cell T3 1180 1181 1182 1183 Token Cell T4 1184 1185 1186 1188 Figure 10: LBF Using TCR 1190 11.3. Forwarding with Flexible Addressing 1192 The is an emerging need to support multiple address technologies. 1193 There are two cases in point, firstly where an operator wants to use 1194 a short address to to address the infrastructure nodes in their 1195 network. This is particularly the case in segment routing where 1196 there is competition amongst the vendors to introduce address 1197 compression due to the extended size of SRv6 data plane headers. 1198 There is a secondary benefit to this in that if an address system 1199 other than IP is used within the provider network there are security 1200 benefits as has been found in operating MPLS. Finally it has been 1201 speculated that some application environments would prefer to use 1202 their native addresses rather than manage the mapping between those 1203 addresses and IP addresses. 1205 This is achieved by creating a new token cell type, populating the 1206 FIB with the corresponding addresses and installing in the forwarder 1207 the necessary forwarding code. That forwarding code may be generic 1208 since the action will almost certainly be address family independent. 1210 In the example shown in Figure 11, the address family is encoded in 1211 the address itself. In an alternative model, the token cell type 1212 would be specific to the address family. 1214 1215 Token Cell T1 1216 - 1217 1218 1219 T2 1220 Token Cell T2 1221 - 1222 1223 1224 Token Cell T3 1225 1226 1227 1229 Figure 11: Indirect LBF UsingUsing Flexible Addresses in TCR 1231 11.4. Forwarding with iOAM analytics 1233 In many cases, there is desire to collect in-situ OAM data 1234 [I-D.ietf-ippm-ioam-data] as a packet traverses a path. There are 1235 multiple applications for this, including but not limited to 1236 diagnosing performance, identification of bottlenecks, and generation 1237 of data to feed machine-learning algorithms for service optimization. 1239 Using TCR, it is possible to indicate that iOAM data should be 1240 collected using a corresponding token cell. The token cell can 1241 contain in-situ parameters, such as which data items to collect. In- 1242 situ data itself can be added to a scratchpad, allowing for its 1243 export using a variety of means upon disposition of packet. 1245 One scenario is depicted in Figure 12. In this particular example, 1246 we assume that the iOAM data can be collected per T3 in parallel with 1247 the forwarding decision being made per T2 in order to show also use 1248 of a manifest (T1). iOAM Data items are written to the scratchpad in 1249 T5. T4 indicates disposition, T6 contains the payload. 1251 It should be noted that rather than simply collecting iOAM data, 1252 other operations could be applied to aggregate that data and result 1253 in more refined behavior. In the interest of brevity, the example 1254 does not feature security tokens used by intermediate nodes to sign 1255 the scratchpad data items that they add. 1257 1258 Token Cell T1 1259 - 1260 1261 T2 1262 T3 1263 Token Cell T2 1264 - 1265 1266 1267 T4 1268 Token Cell T3 1269 - 1270 1271 1272 T5 1273 Token Cell T4 1274 - 1275 1276 1277 Token Cell T5 1278 - 1279 1280 1281 Token Cell T6 1282 1283 1284 1286 Figure 12: iOAM in TCR 1288 If iOAM data to be collected includes telemetry about the egress 1289 interface, the manifest cell can be omitted as the forwarding 1290 decision needs to be made prior to collecting the iOAM data. The 1291 resulting packet becomes even simpler, as depicted in Figure 13. 1293 1294 Token Cell T1 1295 T2 1296 1297 1298 T3 1299 Token Cell T2 1300 - 1301 1302 1303 T4 1304 Token Cell T3 1305 - 1306 1307 1308 Token Cell T4 1309 - 1310 1311 1312 Token Cell T5 1313 1314 1315 1317 Figure 13: iOAM in TCR - serialized 1319 To show the modularity that TCR enables, the third scenario shows 1320 iOAM data to be collected while at the same time LBF is being applied 1321 to the same packet (Figure 14). Note that in this case, LBF and iOAM 1322 could have also been applied in parallel, in which case a manifest 1323 token cell would be added. T1 would then point to the manifest token 1324 cell, which in turn would point to T2 and T3 as successors. 1326 1327 Token Cell T1 1328 T2 1329 1330 1331 T4 1332 Token Cell T2 1333 T3 1334 1335 1336 Token Cell T3 1337 - 1338 1339 1340 T6 1341 Token Cell T4 1342 - 1343 1344 1345 Token Cell T5 1346 - 1347 1348 1349 Token Cell T6 1350 - 1351 1352 1354 Figure 14: iOAM and LBF in TCR 1356 11.5. FRR with Latency-Based Forwarding 1358 This example depicted in Figure 15 extends the earlier LBF example to 1359 include a fast re-route diversion. We show only a single token cell 1360 (T0) added at the point of local repair (PLR), but of course the 1361 repair might be more complex and need multiple intermediate staging 1362 counts to successfully be repaired. 1364 1365 Token Cell T0 1366 T2 1367 1368 1369 Token Cell T1 1370 T2 1371 1372 1373 1374 Token Cell T2 1375 - 1376 1377 1378 Token Cell T3 1379 1380 1381 1382 Token Cell T4 1383 1384 1385 1387 Figure 15: FRR with LBF Using TCR 1389 The PLR was expecting to forward the packet normally and so is aware 1390 that the packet is latency sensitive and understands the semantics 1391 and hence importance of token 2. To maintain the expected path 1392 quality the PLR MUST use an FRR path while also ensuring the SLO is 1393 still being adhered to per the LBF token cell. The FRR path 1394 therefore needs to feature nodes able to support LBF token cells, and 1395 not incur a latency penalty that would physically prohibit being able 1396 to meet the SLO. This path can be selected by the SDN controller, or 1397 locally through the use of path attributes applied to the normal IP- 1398 FRR path selection process. 1400 On detecting a local repairable failure of the next hop or the link 1401 to the next hop the PLR pushes one or more tokens as is necessary to 1402 deliver the packet to the destination. In each case the next token 1403 pointer points to Token 2 the LBF token. When the packet arrives at 1404 the intermediate node chosen by the PLR for the next stage of the 1405 repair, the token (in this case Token 0) at the top of the token 1406 stack is popped and forwarding proceeds as dictated by token 1. The 1407 above operation can clearly be carried out as many times as necessary 1408 on the packet to provide a repair, by pushing as many tokens as are 1409 needed. 1411 Note that the scheme in this example uses an implicit disposition 1412 operation by intermediate nodes as described above in Section 10.1. 1414 11.6. Segment Routing with Latency-Based Forwarding 1416 The operation described in Section 11.5 in which repair tokens are 1417 pushed onto the packet is identical to the operation that happens 1418 when a packet is configured for segment routing (SR). Technically 1419 the packet depicted in Figure 15 IS a segment routed packet. 1421 It therefore follows that implementing segement routing and segment 1422 routing enhanced by features such as enhanced QoS is trivial in TCR. 1423 As we shall see in the next sections, TCR is capable of enhancing SR 1424 significantly beyond that. 1426 11.7. Enhanced Segment Routing with Latency-Based Forwarding 1428 This example illustrates how TCR can be used to add an enhanced QoS 1429 capability such as latency based forwarding to segment routing. We 1430 saw previously Section 11.6 how all segments can be made to execute a 1431 common policy but it might be desirable to execute a different policy 1432 in different segments. For example, some segments might underly 1433 their own latency objectives (just for that segment), without 1434 affecting the overall end-to-end objective. 1436 A packet that achieves this is depicted below in Figure 16. The 1437 packet in the example has three segments. Segments 1 and 3 apply LBF 1438 for the end-to-end latency objective, per T5. Segment 2 applies a 1439 "sub- SLO" just for that particular segment, per T4. Disposition of 1440 T1 and T2 is implicit (popping the token on reaching the segment 1441 destination), whereas disposition of T3 is explicit per T6. 1443 1444 Token Cell T1 1445 T5 1446 1447 1448 Token Cell T2 1449 T4 1450 1451 1452 Token Cell T3 1453 T5 1454 1455 1456 1458 - 1459 1460 1461 Token Cell T5 1462 - 1463 1464 1465 Token Cell T6 1466 1467 1468 1469 Token Cell T7 1470 1471 1472 1474 Figure 16: Enhanced Segment Routing Using TCR 1476 11.8. Enhanced Segment Routing with Differentiated iOAM 1478 The final example shows a variation of the previous example. Instead 1479 of applying a specific latency objectives for particular segments, it 1480 is possible to also invoke other functionality, such as collecting 1481 certain iOAM data only for particular segments, or collecting 1482 additional iOAM data for one of the segments, or even for collecting 1483 different sets of iOAM data along different segments. The particular 1484 example depicted (Figure 17) shows a packet with three segments. One 1485 set of parameters is collected for segments 1 and 3 using scratchpad 1486 T6, another set of parameters is collected for segment 2 using 1487 scratchpad T7. 1489 If instead, segment 2 should collect additional parameters beyond 1490 those collected for segments 1 and 2, this could be easily 1491 accommodated by simply setting "Next Token" of T4 to T5 instead of 1492 null. (This does not include a possible optimization for 1493 parallelization, but attests to the flexibility of the approach.) 1495 1496 Token Cell T1 1497 T4 1498 1499 1500 Token Cell T2 1501 T5 1502 1503 1504 Token Cell T3 1505 T4 1506 1507 1508 1510 - 1511 1512 1513 T6 1514 Token Cell T5 1515 - 1516 1517 1518 T7 1519 Token Cell T6 1520 - 1521 1522 1523 Token Cell T7 1524 - 1525 1526 1527 Token Cell T8 1528 1529 1530 1531 Token Cell T9 1532 1533 1534 1536 Figure 17: Enhanced Segment Routing Using TCR 1538 12. Items for further discussion 1540 This document does not constitute a finalized design and there are 1541 many design decisions that will require further discussion. The 1542 following is a partial list: 1544 o Preamble. The preamble needs further study. The goal is that it 1545 contains only the minimum of information as it needs to be popped 1546 and pushed when a token cell is popped or pushed. We need to 1547 understand if there is a need for a number of token cell's 1548 indicator, and/or a last child indicator. 1550 o Token Cell identification. As an alternative to referencing other 1551 token cells by offset, it is conceivable to introduce token cell 1552 identifiers and refer to token cells by their ID. 1554 o Manifest token cells. Rather to require processing of a full 1555 token cell, it is conceivable to express manifests as part of a 1556 token cell preamble, at least as long as the number of token cells 1557 to process in parallel are limited. This could save on stage in a 1558 token cell processing pipeline. However, it would result in a 1559 slightly longer or more complex preamble. 1561 o Manifest token cells (cont'd.) It should be noted that the 1562 current design of the manifest token cell type includes all 1563 pointers to subsequent token cells as part of the token cell blob. 1564 Alternative designs are conceivable. For example, the design 1565 might be altered to allow for the next token to be populated and 1566 only require any additional token cells to be referenced from the 1567 token cell blob. 1569 o Rendezvous token cells. As an optimization, it is conceivable to 1570 combine manifests and rendezvous points into a single token cell. 1572 o Security token cells. Further investigation is needed to 1573 determine the most effective structure, specifically compact yet 1574 efficient encodings for the token cell mask that is used to 1575 identify the portions of the packet being signed. 1577 o Disposition token cells. There are different ways that 1578 disposition token cells could be referred to for processing, 1579 including through a DT parameter (Disposition Token Cell) as part 1580 of forwarding token cells, through a separate field as part of the 1581 token cell structure, or indirectly by virtue of popping a 1582 forwarding token cell when the destination is reached and 1583 processing the token cell behind it. 1585 o Disposition token cells for processing by intermediate nodes. The 1586 example in Section 11.5 uses an implicit disposition operation of 1587 forwarding token cells by intermediate nodes, in which a 1588 forwarding token cell is simply popped and processing simply 1589 resumes with the subsequent token cell. It is conceivable to 1590 apply explicit disposition operations instead for the sake of more 1591 consistent semantics. Explicit semantics of "pop and resume 1592 processing with subsequent token cell" could be incorporated into 1593 the semantics of a forwarding token cell itself, or potentially 1594 indicated by a corresponding flag in the prefix. 1596 o Implementation profiles. Implementations may impose a reasonable 1597 limit on the number of token cells that can be serially processed 1598 at any one node. This will facilitate mapping to packet 1599 processing pipelines, which then support a predefined number of 1600 token cell processing stages per packet at line rate while 1601 maintaining constant packet processing delay at any given node. 1602 Determination of reasonable constraints is for further study. 1603 Analogous limitations may apply to the number of processing cycles 1604 that can be performed by callback functions, within which the 1605 processing of any given token cell needs to complete. 1607 o Implementation profiles (contd.). For further study are any 1608 mechanisms for the discovery of constraints as mentioned in the 1609 previous list item, as well as any capabilities for negotiation of 1610 corresponding profiles and the definition of node behavior when 1611 such limitations were to be breached (in all likelihood resulting 1612 in aborting the processing of the packet, dropping it with 1613 corresponding error code). 1615 o The design only requires forward token cell pointers and this 1616 prevents processing loops. We need to understand if this is too 1617 significant a restriction. If this restriction is removed some 1618 form of maximum tokens to be processed limit, similar in concept 1619 to TTL may be needed. 1621 13. Security Considerations 1623 This section concerns itself with the security of the TCR dataplane. 1625 The security of the control and management plane is a matter for the 1626 designers of those aspects of the solution. However, it is not 1627 anticipated that the securing of those components will be any more 1628 onerous than securing the control and management plane of other IETF 1629 designed dataplanes. 1631 Security of entry to the dataplane will depend on what entities have 1632 access to the dataplane. If TCR is used as a single domain sub-IP 1633 layer in the way that MPLS is used, then it will have the same 1634 security properties as MPLS in that it is extremely difficult for an 1635 unauthorized party to inject traffic into such a network because with 1636 TCR such traffic is easily recognized at network ingress and dropped. 1637 If the traffic is a TCR packet that is to be carried across the TCR 1638 network it will be encapsulated and so, in the absence of a TCR 1639 network error, will not be able to escape the encapsulation and cause 1640 harm. Only if a TCR network (or node) were to peer with another TCR 1641 network would there be a security concern with that third party 1642 having the ability to control the actions taken on the packet. Such 1643 a case is for further study. It should be noted that similar 1644 situations have been satisfactorily addressed in MPLS. 1646 We now need to consider the security of the contents of the packet. 1647 Clearly we could craft a token that signed the payload, and where the 1648 payload was a token, we have the option of including that signature 1649 in the token itself. However, securing the tokens themselves is more 1650 interesting because we need to authenticate selected components of 1651 the packet header, such as single tokens, groups of tokens or even 1652 components/portions of tokens. This is needed to allow the 1653 differentiation of metadata that must not be altered from scratchpad 1654 data that may be modified during packet transit. In addition, we 1655 need to allow intermediate nodes along a path to authenticate data 1656 that they modify, e.g. scratchpad data items that they create. 1658 The approach used in TCR allows the authentication of tokens and 1659 processing guidance they contain for additional security. 1660 Furthermore there is flexibility for intermediate hops to provide 1661 their own authentication, to secure scratchpad-type data added to 1662 packets along a path. This also allows for "authenticity chains" in 1663 which nodes verify the authenticity of data items they operate on 1664 before modifying and signing the update. 1666 It is clear that the above approach is different from earlier 1667 protocols where payloads are generally signed in their entirety and 1668 do not include support for differentiated signing, accommodating 1669 multiple signers of different packet aspects along a path. 1671 Most protocols secure their payload in its entirety, exposing only 1672 the packet header for processing (unless that is tunneled as well). 1673 TCR is a protocols that include additional packet components that may 1674 require more differentiated securing. Specifically, it includes 1675 guidance for how to process packets, including tokens, and metadata. 1676 In addition, TCR includes some packet components that can be modified 1677 or added by intermediate nodes in transit, specifically scratchpad 1678 data. This includes telemetry and iOAM data as well as data to 1679 indicate and verify certain properties of nodes that were traversed. 1680 While some data must not be modified, other data items might be added 1681 and/or be subject to modification. An example would be data that 1682 aggregates or analyzes telemetry data encountered in transit, for 1683 example an indicator of a minimum or maximum (queue length, 1684 utilization, latency) encountered along a path. TCR thus includes a 1685 mechanism that allows the operator to ensure the authenticity of 1686 packet data beyond the payload, but that allows them to do this in a 1687 way that exempts certain data items which are allowed to be modified 1688 along the way, with the option to allow the corresponding nodes to 1689 secure these modifications. This allows receiving applications to 1690 (for example) verify the authenticity of scratchpad data, and allow 1691 for the modification of data items where such modification is 1692 permitted without compromising the authenticity of the remaining 1693 portions of the packet. 1695 The design of the security token is described in Section 10.4 . This 1696 can only be used to sign itself and tokens or token contents after 1697 the security token. By including in the security token a mask 1698 structure it is possible to select what is to be signed. The 1699 efficiency of this method is described in Section 10.4. 1701 Matters related to inter-domain security will be considered in a 1702 future version of this text. 1704 14. IANA Considerations 1706 This document makes no IANA requests. 1708 15. References 1710 15.1. Normative References 1712 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1713 Requirement Levels", BCP 14, RFC 2119, 1714 DOI 10.17487/RFC2119, March 1997, 1715 . 1717 [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 1718 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, 1719 May 2017, . 1721 15.2. Informative References 1723 [DOI.10.1109_NOMS47738.2020.9110431] 1724 Clemm, A. and T. Eckert, "High-Precision Latency 1725 Forwarding over Packet-Programmable Networks", NOMS 2020 - 1726 2020 IEEE/IFIP Network Operations and 1727 Management Symposium, DOI 10.1109/noms47738.2020.9110431, 1728 April 2020. 1730 [I-D.ietf-ippm-ioam-data] 1731 Brockners, F., Bhandari, S., and T. Mizrahi, "Data Fields 1732 for In-situ OAM", draft-ietf-ippm-ioam-data-15 (work in 1733 progress), October 2021. 1735 [RFC3270] Le Faucheur, F., Wu, L., Davie, B., Davari, S., Vaananen, 1736 P., Krishnan, R., Cheval, P., and J. Heinanen, "Multi- 1737 Protocol Label Switching (MPLS) Support of Differentiated 1738 Services", RFC 3270, DOI 10.17487/RFC3270, May 2002, 1739 . 1741 [RFC3985] Bryant, S., Ed. and P. Pate, Ed., "Pseudo Wire Emulation 1742 Edge-to-Edge (PWE3) Architecture", RFC 3985, 1743 DOI 10.17487/RFC3985, March 2005, 1744 . 1746 [RFC8754] Filsfils, C., Ed., Dukes, D., Ed., Previdi, S., Leddy, J., 1747 Matsushima, S., and D. Voyer, "IPv6 Segment Routing Header 1748 (SRH)", RFC 8754, DOI 10.17487/RFC8754, March 2020, 1749 . 1751 Authors' Addresses 1753 Stewart Bryant 1754 University of Surrey 5GIC 1756 Email: sb@stewartbryant.com 1758 Alexander Clemm 1759 Futurewei Technologies, Inc. 1761 Email: ludwig@clemm.org