idnits 2.17.1 draft-izh-ccamp-flexe-fwk-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The abstract seems to contain references ([OIFFLEXE1], [OIFMLG3]), which it shouldn't. Please replace those with straight textual mentions of the documents in question. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (October 20, 2016) is 2737 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Missing Reference: '1-254' is mentioned on line 193, but not defined Summary: 1 error (**), 0 flaws (~~), 2 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Internet Engineering Task Force I. Hussain, Ed. 3 Internet-Draft R. Valiveti 4 Intended status: Informational K. Pithewan 5 Expires: April 23, 2017 Infinera Corp 6 Q. Wang, Ed. 7 ZTE 8 L. Andersson, Ed. 9 F. Zhang 10 M. Chen 11 J. Dong 12 Z. Du 13 Z. Haomian 14 X. Zhang 15 J. Huang 16 Q. Zhong 17 Huawei 18 October 20, 2016 20 GMPLS Routing and Signaling Framework for Flexible Ethernet (FlexE) 21 draft-izh-ccamp-flexe-fwk-00 23 Abstract 25 Traditionally, Ethernet MAC rates were constrained to match the rates 26 of the Ethernet PHY(s). OIF's implementation agreement [OIFMLG3] was 27 the first step in allowing MAC rates to be different than the PHY 28 rates. OIF has recently approved another implementation agreement 29 [OIFFLEXE1] which allows complete decoupling of the MAC data rates 30 and the Ethernet PHY(s) that support them. This includes support for 31 (a) MAC rates which are greater than the rate of a single PHY 32 (satisfied by bonding of multiple PHY(s)), (b) MAC rates which are 33 less than the rate of a PHY (sub-rate), (c) support of multiple FlexE 34 client signals carried over a single PHY, or over a collection of 35 bonded PHY(s). The FlexE SHIM functions which bond multipe Ethernet 36 PHY(s) to form a large "pipe" view the connectivity between two FlexE 37 aware devices as a collection of multiple point-to-point links (one 38 link per Ethernet PHY). These logical point-to-point links can 39 either be direct links (without an intervening transport network), or 40 realized via a Optical transport network. This draft catalogs the 41 usecases that capture the FlexE deployment scenarios -- including the 42 cases that include/exclude OTNs. 44 Status of This Memo 46 This Internet-Draft is submitted in full conformance with the 47 provisions of BCP 78 and BCP 79. 49 Internet-Drafts are working documents of the Internet Engineering 50 Task Force (IETF). Note that other groups may also distribute 51 working documents as Internet-Drafts. The list of current Internet- 52 Drafts is at http://datatracker.ietf.org/drafts/current/. 54 Internet-Drafts are draft documents valid for a maximum of six months 55 and may be updated, replaced, or obsoleted by other documents at any 56 time. It is inappropriate to use Internet-Drafts as reference 57 material or to cite them other than as "work in progress." 59 This Internet-Draft will expire on April 23, 2017. 61 Copyright Notice 63 Copyright (c) 2016 IETF Trust and the persons identified as the 64 document authors. All rights reserved. 66 This document is subject to BCP 78 and the IETF Trust's Legal 67 Provisions Relating to IETF Documents 68 (http://trustee.ietf.org/license-info) in effect on the date of 69 publication of this document. Please review these documents 70 carefully, as they describe your rights and restrictions with respect 71 to this document. Code Components extracted from this document must 72 include Simplified BSD License text as described in Section 4.e of 73 the Trust Legal Provisions and are provided without warranty as 74 described in the Simplified BSD License. 76 Table of Contents 78 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 79 1.1. Requirements Language . . . . . . . . . . . . . . . . . . 4 80 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4 81 3. Usecases . . . . . . . . . . . . . . . . . . . . . . . . . . 5 82 3.1. FlexE unware transport . . . . . . . . . . . . . . . . . 5 83 3.2. FlexE Aware . . . . . . . . . . . . . . . . . . . . . . . 7 84 3.2.1. FlexE Aware Case - No Resizing . . . . . . . . . . . 7 85 3.3. FlexE Termination - Transport . . . . . . . . . . . . . . 11 86 3.3.1. FlexE Client at Both endpoints . . . . . . . . . . . 11 87 3.3.2. Interworking of FlexE Client w/ Native Client at the 88 other endpoint . . . . . . . . . . . . . . . . . . . 12 89 3.3.3. Interworking of FlexE client w/ Client from OIF_MLG . 14 90 3.3.4. Back-to-Back FlexE . . . . . . . . . . . . . . . . . 15 91 3.3.4.1. FlexE Client BW Resizing . . . . . . . . . . . . 15 92 4. Requirements . . . . . . . . . . . . . . . . . . . . . . . . 16 93 5. Framework . . . . . . . . . . . . . . . . . . . . . . . . . . 17 94 6. Architecture . . . . . . . . . . . . . . . . . . . . . . . . 17 95 7. Solution . . . . . . . . . . . . . . . . . . . . . . . . . . 17 96 8. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 17 97 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 17 98 10. Security Considerations . . . . . . . . . . . . . . . . . . . 17 99 11. References . . . . . . . . . . . . . . . . . . . . . . . . . 17 100 11.1. Normative References . . . . . . . . . . . . . . . . . . 17 101 11.2. Informative References . . . . . . . . . . . . . . . . . 18 102 Appendix A. Additional Stuff . . . . . . . . . . . . . . . . . . 18 103 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 18 105 1. Introduction 107 Traditionally, Ethernet MAC rates were constrained to match the rates 108 of the Ethernet PHY(s). OIF's implementation agreement [OIFMLG3] was 109 the first step in allowing MAC rates to be different than the PHY 110 rates standardized by IEEE. OIF has recently approved another 111 implementation agreement [OIFFLEXE1] which allows complete decoupling 112 of the MAC data rates and the Ethernet PHY(s) that support them. 113 This includes support for (a) MAC rates which are greater than the 114 rate of a single PHY (satisfied by bonding of multiple PHY(s)), (b) 115 MAC rates which are less than the rate of a PHY (sub-rate), (c) 116 support of multiple FlexE client signals carried over a single PHY, 117 or over a collection of bonded PHY(s). The capabilities supported by 118 the OIF FlexE implementation agreement version 1.0 are: 120 a. Support a large rate Ethernet MAC over bonded Ethernet PHYs, e.g. 121 supporting a 200G MAC over 2 bonded 100GBASE-R PHY(s) 123 b. Support a sub-rate Ethernet MAC over a single Ethernet PHY, e.g. 124 supportnig a 50G MAC over a 100GBASE-R PHY 126 c. Support a collection of flexible Ethernet clients over a single 127 Ethernet PHY, e.g. supporting two MACs with the rates 25G, 50G 128 over a single 100GBASE-R PHY 130 d. Support a sub-rate Ethernet MAC over bonded PHYs, e.g. supporting 131 a 150G Ethernet client over 2 bonded 100GBASE-R PHY(s) 133 e. Support a collection of Ethernet MAC clients over bonded Ethernet 134 PHYs, e.g. supporting a 50G, and 150G MAC over 2 bonded Ethernet 135 PHY(s) 137 All networks which support the bonding of Ehernet interfaces (as per 138 [OIFFLEXE1]) include a basic building block -- which consists of two 139 FlexE SHIM functions (located at opposite ends of a link) and the 140 (logical) point to point links that carry the Ethernet PHY signals 141 between the two FlexE SHIM Functions. These logical point-to-point 142 PHY links can be realized in a variety of ways: 144 a. These are direct point-to-point links with no intervening 145 transport network. 147 b. The Ethernet PHY(s) are transparently transported via an Optical 148 Transport Network. Optical Transport Networks (defined by [G709] 149 and [G798]) have recently expanded the traditional bit (or 150 codeword) transparent transport of Ethernet client signals, and 151 included support for the usecases identified in the OIF FLexE 152 implementation agreement. 154 c. Realized by tunneling the Ethernet PHY(s) over some other type of 155 network (e.g. IP/MPLS). Thus, for example, the Ethernet PHY(s) 156 signals could be carried over a pseudowire (or a LSP)in the IP/ 157 MPLS network. Note that the OIF implementation agreement 158 [OIFFLEXE1] only includes support for 100G Ethernet PHY(s). As 159 a result of this encapsulation into a PW, the bandwidth of the PW 160 will be much larger than the bit rate of the Ethernet PHY (i.e. 161 100G), and such a pseudowire cannot be transported in networks 162 that only include 100G Ethernet links. This scenario is 163 realizable when (a) higher rate Ethernet PHY(s), e.g. 200G/40G 164 are supported) or (b) OIF extends the FlexE groups to include 165 lower rate Ethernet PHY(s), e.g. at the 25G/50G rate. Further 166 study is needed to ensure that these scenarios are realizable, 167 practical, and beneficial to operators. With this in mind, the 168 current draft doesn't include any coverage for this scenario. 170 Internet-draft examines the usescases that arise when the logical 171 links between FlexE capable devices are (a) point-to-point links 172 without any intervening network (b) realized via Optical transport 173 networks. This draft considers the variants in which fhe two peer 174 FlexE devices are both customer-edge devices, or customer-edge/ 175 provider edge devices. This list of usecases will help identify the 176 Control Plane (i.e. Routing and Signaling) extensions that may be 177 required). 179 1.1. Requirements Language 181 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 182 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 183 document are to be interpreted as described in RFC 2119 [RFC2119]. 185 2. Terminology 187 a. Ethernet PHY: an entity representing 100G-R Physical Coding 188 Sublayer (PCS), Physical Media Attachment (PMA), and Physical 189 Media Dependent (PMD) layers. 191 b. FlexE Group: A FlexE Group is composed of from 1 to n 100GBASE-R 192 Ethernet PHYs. Each PHY is identified by a number in the range 193 [1-254]. 195 c. FlexE Client: an Ethernet flow based on a MAC data rate that may 196 or may not correspond to any Ethernet PHY rate (e.g., 10, 40, m x 197 25 Gb/s). 199 d. FlexE Shim: the layer that maps or demaps the FlexE clients 200 carried over a FlexE group. 202 e. FlexE Calendar: The total capacity of a FlexE group is 203 represented as a collection of slots which have a granularty of 204 5G. The calendar for a FlexE group composed of n 100G PHYs is 205 represented as an array of 20n slots (each representing 5G of 206 bandwidth). This calendar is partitioned into sub-calendars, 207 with 20 slots per 100G PHY. Each FlexE client is mapped into one 208 or more calendar slots (based on the bandwidth of the FlexE 209 client). 211 3. Usecases 213 3.1. FlexE unware transport 215 The FlexE shim layer in a router maps the FlexE client(s) over the 216 FlexE group. The transport network is unware of the FlexE. Each of 217 the FlexE group PHY is carried independently across the transport 218 network over the same fiber route. The FlexE shim in the router 219 tolerates end-to-end skew across the network. In this usecase, the 220 router makes flexible use of the full capacity of the FlexE group, 221 and depends on legacy transport equipment to realize PCS-codeword- 222 transparent transport of 100GbE. It allows striping of PHYs in the 223 FlexE group over multiple line cards in the transport equipment. It 224 is worth mentioning that in this case, the FlexE SHIM layer is 225 terminated at the routers, and the coordination of operations related 226 to FlexE clients, e.g. creating new FlexE clients, deleting existing 227 FlexE clients, and resizing the bandwidth of existing FlexE clients 228 (if desired) happens between the two routers. Note that the 229 transport network is completely transparent to the FlexE signals, and 230 doesn't participate in any FlexE protocols. 232 ================================================================== 234 + FlexE Ethernet Client(s) + 235 +-----------------------------------------------------------+ 236 + + 237 + FlexE skew tolerance 238 +----------------------------------------+ 239 + for end-to-distance + 241 +-----------+ 2x100GE +---------+ +----------+ +------------+ 242 | | | | | | | | 243 | Router1 | | | | | | | 244 |FlexE Shim +---------+ A-end | | Z-end +-----+Router 2 | 245 | | | (FlexE | | (FlexE | |(FlexE Shim)| 246 | +---^-----+ unaware)| | unaware)+-----+ | 247 | | | | | | | | | 248 | | | | | | | | | 249 +-----------+ + +---------+ +----------+ +------------+ 250 FlexE Group 252 \----------Transport----------/ 253 network 254 +--------------+ +----------------+ 255 | FlexE Clients| | FlexE Client(s)| 256 +--------------+ +----------------+ 257 | FlexE Shim | | FlexE Shim | 258 +----+----+----+ +----+------+----+ 259 |PHY | | PHY | | PHY | | PHY | 260 +---+---+--+---+ +---+--+ +--+--+ 261 | | +-----+ +-----+ | | 262 | +----------+ PHY | | PHY |-------+ | 263 | +-----+ +-----+ | 264 | | ODU4+-----------+ ODU4| | 265 | +-----+ +-----+ | 266 | | 267 | +-----+ +-----+ | 268 +-----------------+ PHY | | PHY +-----------------+ 269 +-----+ +-----+ 270 | ODU4+-----------+ ODU4| 271 +-----+ +-----+ 273 ================================================================== 275 Figure 1: FlexE unaware transport 277 3.2. FlexE Aware 279 3.2.1. FlexE Aware Case - No Resizing 281 This scenario represents an optimization of the FlexE unaware 282 transport presented in Section 3.1, and illustrated in Figure 1. In 283 this application (see Figure 2), the devices at the edge of the 284 transport network do not terminate the FlexE shim layer, but are 285 aware of the (a) composition of the FlexE grpup (i.e. set of all 286 contained Ethernet PHYs) and (b) format of the FlexE overhead. They 287 "snoop" the FlexE overhead to determine the subset of the set of all 288 calendar slots that are available for use (i.e. these calendar slots 289 may be used, or unused). The transport network edge removes the 290 unavailable calendar slots at the ingress to the network, and adds 291 the same unavailable calendar slots back when exiting the network. 292 The result is that the FlexE Shim layers at both routers see exactly 293 the same input that they saw in the FlexE unware scenario -- with the 294 added benefit that the line (or DWDM) side bandwidth has been 295 optimized to be sufficient to carry only the available calendar slots 296 in all of the Ethernet PHY(s) in the FlexE group. This mode may be 297 used in cases where the bandwidth of the Ethernet PHY is greater than 298 the bit rate supported by a wavelength (and it is known that that all 299 calendar slots in the PHY are not "available"). 301 The transport network edge device could learn of the set of 302 unavailable calendar slots in a variety of ways; a few examples are 303 listed below: 305 a. The set of unavailable calendar slots could be configured against 306 each Ethernet PHY in the FlexE group. The FlexE demux function 307 in the transport network edge device (A) compares the information 308 about calendar slots which are expected to be unavailable (as per 309 user supplied configuration), with the corresponding information 310 encoded by the customer edge device in the FlexE overhead (as 311 specified in [OIFFLEXE1]). If there is a mismatch between the 312 unavailable calendar slots in any of the PHYs within a FlexE 313 group, the transport edge node software could raise an alarm to 314 report the inconsistency between the provisioning information at 315 the transport network edge, and the customer edge device. 317 b. The Transport network edge could be configured to act in a 318 "slave" mode. In this mode, the FlexE demux function at the 319 Transport network edge (A) receives the information about the 320 available/unavailable calendar slots by observing the FlexE 321 overhead (as specified in [OIFFLEXE1]) and uses this information 322 to select (a) the set of wavelengths (with appropriate 323 capacities) or (b) the bandwidth of the ODUflex (or fixed rate 324 ODUs) that could carry the FlexE PCS end-to-end. 326 c. The set of unavailable slots could be negotiated between FlexE 327 Shim entity in the customer device and the partial rate ODUflex 328 mapper located in the transport network element. Thus, for 329 example, the transport network element could declare the maximum 330 number of 5G slots that could be transported over a single 331 wavelength, and the customer network device can choose the number 332 of 5G slots that will be used between customer devices. This 333 process could be accomplished through control protocols such as 334 LMP,using the appropriate control channel for transporting the 335 messages. 337 In the basic FlexE aware mode, the transport network edge does not 338 expect the number of unavailable calendar slots to change 339 dynamically. 341 Note that the process of removing unavailable calendar slots from a 342 FlexE PHY is called "crunching" (see [OIFFLEXE1]). The following 343 additional notes apply to Figure 2: 345 a. The crunched FlexE PHYs are independently transported through the 346 transport network. The number of used (and unused) calendar 347 slots can be different across the FlexE group. In particular, if 348 all the calendar slots in a FlexE PHY are in use, the crunching 349 operation leaves the original signal intact. 351 b. In this illustration, the different FlexE PHY(s) are transported 352 using ODUflex containers in the transport network. These ODUflex 353 connections can be of different rates. 355 c. In the most general form, G.709 Section 17.12 allows for a FlexE 356 group consisting of m Ethernet PHY(s) to be crunched, combined, 357 and transported using n ODUFlex containers (where n can range 358 between 1 and m). In other words, the ITU G.709 recommendation 359 allows for (but not require the support for) the degenerate cases 360 in which (a) each Ethernet PHY within the group is transported 361 using its own ODUflex, and (b) all the PHY(s) are crunched, 362 combined and transported over a single ODUflex container. If all 363 the sub-calendar slots in a given PHY are available, it is 364 possible to transport the content of the PHY in one of two ways: 365 (a) as shown in Figure 2, or (b) using a FLexE unware (i.e. PCS- 366 codeword transparent transport) mode. The latter approach (of 367 using FlexE unaware transport) for a few select (fully-utilized) 368 PHYs is not attractive from the perspective of skew between the 369 PHYs that comprise the FlexE group. For simplicity, the 370 preferred mode of operation will be one in which the same mapping 371 procedure is used for member PHYs of a FlexE group. 373 d. When the crunched FlexE PHY(s) have a rate that is identical to 374 that of a standard Ethernet PHY, it is possible that the 375 transport network may utilize standard ODU containers such as 376 ODU2e, ODU4 etc. As currently defined by ITU G.709 377 Section 17.12, the crunched, sub-rate signal is always mapped to 378 an ODUflex, and the mapping to a fixed rate ODU signal is not 379 required. This option could be dropped if it results in any 380 significant simplification. 382 Note: The figure may need further editing to accurately depict the 383 signal hierarchy. 385 ================================================================ 386 FlexE Ethernet Client(s) 387 +-----------------------------------------------------+ 388 FlexE skew tolerance 389 +---------------------------------------------+ 390 for end+to+distance 392 +--------+ 2 x 100GE +---------+ +---------+ +------+ 393 | R1 | | | | +----+ R2 | 394 | (FlexE+-----------+ NE A | | NE Z | |(FlexE| 395 | Shim) | | (FlexE | | (FlexE +----+ Shim | 396 | +-----^-----+ aware) | | aware) | | | 397 | | | | | | | | | 398 +--------+ + +---------+ +---------+ +------+ 399 FlexE Group 400 \+--------+Transport+--------+/ 401 network 402 +-------------+ +-------------+ 403 |FlexE clients| |FlexE clients| 404 +-------------+ +-------------+ 405 | FlexE Shim | | FlexE Shim | 406 +-------------+ +-------------+ 407 | PHY | PHY | | PHY | PHY | 408 +-------------+ +-------------+ 409 | | | | 410 | | +-------------+ +------------+ | | 411 | | | FlexE-psg | | FlexE-psg | | | 412 | | +-------------+ +------------+ | | 413 | +--+ PHY|ODUflex +------- |ODUFlex|PHY +--+ | 414 | +-------------+ +------------+ | 415 | | 416 | +-------------+ +------------+ | 417 | | FlexE|psg | | FlexE|psg | | 418 | +-------------+ +------------+ | 419 +--------+ PHY|ODUflex +------- |ODUFlex|PHY +--------+ 420 +-------------+ +------------+ 422 + Legend: 423 | R1, R2 + Routers (supporting the FlexE clients) 424 | NE A, Z + Transport Network Edge nodes 425 + FlexE-psg: FlexE partial rate (sub) group signal 426 (per G.709:17.12) 428 =============================================================== 430 Figure 2: FlexE Aware Transport 432 3.3. FlexE Termination - Transport 434 These usecases build upon the basic router-transport equipment 435 connectivity illustrated in Figure 1. The FlexE shim layer at the 436 router maps to the set of FlexE clients over the FlexE group, as 437 usual. This section considers various usecases in which the 438 equipment located at the edge of the transport network instantiates 439 the FlexE Shim function which peers with the FlexE shim on the 440 customer device. In the router to network direction, the transport 441 edge node terminates the FlexE shim layer, and extracts one or more 442 FlexE client signals, and transports them through the network. That 443 is, these usecases are distinguished from the FlexE unaware cases in 444 that the FlexE group, and the FlexE shim layer end at the transport 445 network edge, and only the extracted FlexE client signals transit the 446 optical network. In the network to router direction, the transport 447 edge node maps a set of FlexE clients to the FlexE group (i.e. 448 performing the same functions as the router which connects to the 449 transport network).The various usecases differ in the combination of 450 service endpoints in the transport network. In the FlexE termination 451 scenarios, the distance between the FlexE Shims is limited the normal 452 Ethernet link distance. The FlexE shims in the router, and the 453 equipment need to support a small amount skew. 455 3.3.1. FlexE Client at Both endpoints 457 In this scenario, service consists of transporting a FlexE client 458 through the transport network, and possibly combining this FlexE 459 client with other FlexE clients into a FlexE group at the endpoints. 460 The FlexE client signal can be transported in two manners within the 461 OTN: (i) directly over one or more wavelengths (ii) mapped into an 462 ODUflex (of the appropriate rate) and then switched across the OTN. 463 Figure 3 illustrates the scenario involving the mapping of a FlexE 464 client to an ODUflex envelope; this figure only shows the signal 465 "stack" at the service endpoints, and doesn't illustrate the 466 switching of the ODUflex entity through the OTN. The ODUflex mapping 467 will be beneficial in scenarios where the rate of the FlexE client is 468 less than the capacity of a single wavelength deployed on the DWDM 469 side of the OTN network, and allows the network operators to packet 470 multiple FlexE client signals into the same wavelength -- thereby 471 improving the network efficiency. Although Figure 3 illustrates the 472 scenario in which one FlexE client is transported within the OTN, the 473 following points should be noted: 475 a. When the FlexE Shim termination function recovers multiple FlexE 476 client signals (at node A), the FlexE signals can be transported 477 independently. In other words, it is not a requirement that all 478 the FlexE client signals be co-routed. 480 b. Conversely, at the egress node, FlexE clients from different 481 endpoints can be combined via the FlexE shim, eventually exiting 482 the transport edge node over an Ethernet group. 484 ================================================================== 486 +--------+ 2 x 100GE +---------+ +----------+ +--------+ 487 | | | | | | | | 488 | Router1| | | | | | | 489 | FlexE +-----------+ A-end | | Z-end +------+Router2 | 490 | Shim | | (FlexE | | (FlexE | |FlexE | 491 | +-----^-----+ term) | term) +------+ Shim | 492 | | | | | | | | | 493 | | | | | | | | | 494 +--------+ + +---------+ +----------+ +--------+ 495 FlexE Group 496 \+--------+Transport+--------+/ 497 network 499 +-----------+ +--------------+ +-------------+ +-----------+ 500 | Client(s) | | Client | | Client | | Client(s) | 501 +-----------+ +--------+-----+ +------+------+ +-----------+ 502 | FlexE Shim| | Shim | | | | Shim | | FlexE Shim| 503 +-----------+ +--------+ ODU | | ODU +------+ +-----------+ 504 | PHY(s) | | PHY(s) | flex| | flex |PHY(s)| | PHY(s) | 505 +---+-------+ +---+----+--+--+ +---+--+---+--+ +---+-------+ 506 | | | | | | 507 +---------------+ +-----------+------+----------+ 509 ================================================================= 511 Figure 3: FlexE termination: FlexE clients at both endpoints 513 3.3.2. Interworking of FlexE Client w/ Native Client at the other 514 endpoint 516 The OIF implementation agreement [OIFFLEXE1] currently supports FlexE 517 client signals carried over one or more 100GBASE-R PHY(s). There is 518 a calendar of 5G timeslots associated with each PHY, and each FlexE 519 client can make use of a number of timeslots (possibly distributed 520 across the members of the FlexE group. This implies that the FlexE 521 client rates are multiples of 5Gbps. When the rates of the FlexE 522 client signals matches the MAC rates corresponding to existing 523 Ethernet PHYs, i.e. 10GBASE-R/40GBASE-R/100GBASE-R, there is a need 524 for the FlexE client signal to interwork with the native Ethernet 525 client received from a single (non-FlexE capable) Ethernet PHY. This 526 capability is expected to be extended to any future Ethernet PHY 527 rates that the IEEE may define in future (e.g. 25G, 50G, 200G etc.). 528 In these cases, although the bit rate of the FlexE client matches the 529 MAC rate of other endpoint, the 64B66B PCS codewords for the FlexE 530 client need to be transformed (via ordered set translation) to match 531 the specification for the specific Ethernt PHY. These details are 532 described in Section 7.2.2 of [OIFFLEXE1] and are not eloborated any 533 further in this document. 535 Figure 4 illustrates a scenario involving the interworking of a 10G 536 FlexE client with a 10GBASE-R native Ethernet signal. In this 537 example, the network wrapper is ODU2e. 539 ================================================================== 541 +--------+ 2 x 100GE +-------+ +-------+ +--------+ 542 | | | | | | | | 543 | Router1| | | | | | | 544 |(FlexE +-----------+ A-end | | Z-end | 10GE |Router 2| 545 | Shim) | |(FlexE | | +------+ | 546 | +-----^-----+ term) | | | | | 547 | | | | | | | | | 548 | | | | | | | | | 549 +--------+ + +-------+ +-------+ +--------+ 550 FlexE Group 551 \+---------Transport---------+/ 552 network 554 +-----------+ +---------------+ 555 | Client(s) | | Client | +------------+ +---------+ 556 +-----------+ +-------+-------+ | 10GE PCS | | 10GE PCS| 557 | FlexE Shim| | Shim | | +-------+----+ +---------+ 558 +-----------+ +-------+ ODU | | ODU2e | PHY| | PHY | 559 | PHY(s) | | PHY(s)| 2e | +---+---+--+-+ +-----+---+ 560 +---+-------+ +---+-------+---+ | | | 561 | | | | | | 562 | | | | | | 563 +---------------+ +-------------+ +------------+ 565 ================================================================= 567 Figure 4: FlexE client interop with Native Ethernet Client 569 3.3.3. Interworking of FlexE client w/ Client from OIF_MLG 571 As explained in the Introduction section ( Section 1 OIFMLG3 572 [OIFMLG3] introduced support for carrying 10GE and 40GE client 573 signals over a group of 100GBASE-R Ethernet PHY(s). While the most 574 recent implementation agreement doesn't call it out explicitly, it is 575 expected that the FlexE clients (as defined in [OIFFLEXE1]), and 576 10GBASE-R/40GBASE-R clients supported by OIFMLG3 [OIFMLG3]) will 577 interoperate. 579 Figure 5 illustrates a scenario involving the interworking of a 10G 580 FlexE client with a 10GBASE-R client supported by an OIFMLG3 581 interface. In this example, the network wrapper is ODU2e. 583 ================================================================== 585 +--------+ 2 x 100GE +---------+ +---------+ +---------+ 586 | | | | | | | | 587 | Router1| | | | | | | 588 | FlexE +-----------+ A-end | | Z-end +------+Router 2 | 589 | Shim | | (FlexE | | | |(MLG-3.0)| 590 | +-----^-----+ term) | | +------+ | 591 | | | | | | | | | 592 | | | | | | | | | 593 +--------+ + +---------+ +---------+ +---------+ 594 FlexE Group 596 \+--------+Transport+--------+/ 597 network 599 +-----------+ +-------------+ +--------------+ +----------+ 600 | Client(s) | | Client | | 10GE PCS | | 10GE Cl. | 601 +-----------+ +--------+----+ +------+-------+ +----------+ 602 | FlexE Shim| | Shim | | | | MLG3 | | MLG3 | 603 +-----------+ +--------+ ODU| | ODU +-------+ +----------+ 604 | PHY(s) | | PHY(s) | 2e | | 2e | PHY(s)| | PHY(s) | 605 +---+-------+ +---+----+--+-+ +---+--+---+---+ +---+------+ 606 | | | | | | 607 +---------------+ +------------+ +------------+ 609 ================================================================= 611 Figure 5: FlexE client interop with Ethernet Client supported by MLG3 613 3.3.4. Back-to-Back FlexE 615 This section covers a degenerate FlexE termination scenario in which 616 router1, router2, and router3 are interconnected through back-to-back 617 FlexE groups without an intermediate transport network (see 618 Figure 6). In this example, the FlexE SHIM at Router2 extracts one 619 or more FlexE client signals from the FlexE group connected to 620 Router1, and mutliplexes these extracted FlexE signals into the FlexE 621 group towards the appropriate router (e.g. Router3). Note that each 622 of the extracted FlexE client signals can be indepdenently routed 623 towards its respective FlexE group. 625 ================================================================== 627 +--------+ 2 x 100GE +---------+ 3 x 100GE +---------+ 628 | | | | | | 629 | Router1| | | | | 630 | FlexE +-----------+ Router2 +-----------+ Router3 | 631 | Shim | | FlexE +-----------+ FlexE | 632 | +-----^-----+ Shim +-----^-----+ Shim | 633 | | | | | | | | 634 | | | | | | | | 635 +--------+ + +---------+ + +---------+ 636 FlexE Group FlexE Group 638 ================================================================= 640 Figure 6: Back-to-Back FlexE 642 3.3.4.1. FlexE Client BW Resizing 644 In the scenario presented in Figure 6, it is possible to support the 645 FlexE client signal resizing on an end-to-end basis. Thus, for 646 example, the resizing of the end-to-end FlexE client circuit with a 647 scope of Router1-Router2-Router3 is accomplished by correctly 648 coordinating the resizing operations across these two segments: 649 Router1-Router2, Router2-Router3. The hop-by-hop FlexE client signal 650 resizing operations across each of these segments (or hops) are 651 accomplished by using the following FlexE overhead (as per 652 [OIFFLEXE1]): 654 a. Currently active FlexE calendar (containing a list of mapping 655 between the 5G tributary slots and the FlexE client signals 657 b. Future calendar to which the sender wants to transition to. 659 c. Calendar switch request bit (CR) 661 d. Calendar switch acknowldege bit (CA) 663 It is expected that the exact sequence of FlexE client resizing 664 operations will be different for the cases involving bandwidth 665 increase/decrease. 667 4. Requirements 669 This section summarizes solution requirements for the usecases 670 described in this document to help identify the Control Plane (i.e. 671 Routing and Signaling) extensions that may be required. 673 a. The solution SHALL support a FlexE group to address 674 abovementioned usecases including FlexE unaware (where FlexE mux 675 and demux can be separated by longer distances), FlexE aware 676 (where FlexE mux and demux can be separated by shorter 677 distances), and FlexE partially aware. 679 b. The solution SHALL support a flexible mechanism for configuring a 680 FlexE group -- such as a signaling protocol or a SDN controller/ 681 management system with network access to the FlexE mux/demux at 682 each end of the FlexE group. 684 c. The solution SHOULD support the ability to add/remove Ethernet 685 PHYs to/from a FlexE group. In the absence of this ability, it 686 is acceptable to permit changes to the group members only when 687 the group has been administratively locked (and hence not 688 providing any service). 690 d. The solution SHOULD allow decoupling of FlexE group's initial 691 configuration and bring up operation from an addition (or 692 removal) of FlexE clients to the FlexE group. For instance, it 693 SHOULD be possible to configure and bring up a FlexE group 694 without any FlexE client (e.g., with all calendar slots set to 695 unused or unavailable). 697 e. The solution SHALL allow adding or removing a FlexE client to a 698 FlexE group without affecting traffic on other clients. 700 f. The solution SHOULD allow resizing of FlexE client BW through 701 coordination of calendar updates within a single FlexE group. 702 There SHOULD be no expectation that FlexE client BW resizing be 703 hitless in all network scenarios. This capability can be 704 supported for the Back-to-Back FlexE scenario identified in 705 Section 3.3.4.1 707 g. For the FlexE unaware case, each of the 100GBASE-R PHYs in the 708 FlexE group SHALL be carried independently across transport 709 network using a PCS codeword transparent mapping. All PHYs of 710 the FlexE group SHALL be interconnected between the same two 711 FlexE shims. The Ethernet PHYs SHOULD be carried over the same 712 fiber route across the transport network (i.e., co-routed) 714 h. For the FlexE aware case, each of the 100GBASE-R PHYs in the 715 FlexE group SHALL be carried independently across transport 716 network. All PHYs of the FlexE group SHALL be interconnected 717 between the same two FlexE shims. The Ethernet PHYs SHOULD be 718 carried over the same fiber route across the transport network. 719 In the transport network, in mux direction, the OTN mapper SHALL 720 be able to discard unavailable slots (e.g., this can be based on 721 static configuration as the rate of a wavelength is not expected 722 to change in-service). In the transport network, in the demux 723 direction, the OTN mapper SHALL be able to restore unavailable 724 slots to match the original PHY rate. 726 i. For the FlexE termination case, the FlexE group SHALL be 727 terminated at the transport network edge. It SHOULD be possible 728 to carry (switch) each FlexE client extracted from the FlexE 729 group independently across transport network using OTN mapping 730 (e.g., ODUflex). 732 5. Framework 734 6. Architecture 736 7. Solution 738 8. Acknowledgements 740 9. IANA Considerations 742 This memo includes no request to IANA. 744 10. Security Considerations 746 None. 748 11. References 750 11.1. Normative References 752 [G709] ITU, "Optical Transport Network Interfaces 753 (http://www.itu.int/rec/T-REC-G.709-201606-P/en)", July 754 2016. 756 [G798] ITU, "Characteristics of optical transport network 757 hierarchy equipment functional blocks 758 (http://www.itu.int/rec/T-REC-G.798-201212-I/en)", 759 February 2014. 761 [OIFFLEXE1] 762 OIF, "FLex Ethernet Implementation Agreement Version 1.0 763 (OIF-FLEXE-01.0)", March 2016. 765 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 766 Requirement Levels", BCP 14, RFC 2119, 767 DOI 10.17487/RFC2119, March 1997, 768 . 770 11.2. Informative References 772 [OIFMLG3] OIF, "Multi-Lane Gearbox Implementation Agreement Version 773 3.0 (OIF-MLG-3.0)", April 2016. 775 Appendix A. Additional Stuff 777 This becomes an Appendix. 779 Authors' Addresses 781 Iftekhar Hussain (editor) 782 Infinera Corp 783 169 Java Drive 784 Sunnyvale, CA 94089 785 USA 787 Email: IHussain@infinera.com 789 Radha Valiveti 790 Infinera Corp 791 169 Java Drive 792 Sunnyvale, CA 94089 793 USA 795 Email: rvaliveti@infinera.com 796 Khuzema Pithewan 797 Infinera Corp 798 169 Java Drive 799 Sunnyvale, CA 94089 800 USA 802 Email: kpithewan@infinera.com 804 Qilei Wang (editor) 805 ZTE 806 Nanjing 807 CN 809 Email: wang.qilei@zte.com.cn 811 Loa Andersson (editor) 812 Huawei 813 Stockholm 814 Sweden 816 Email: loa@pi.nu 818 Fatai Zhang 819 Huawei 820 CN 822 Email: zhangfatai@huawei.com 824 Mach Chen 825 Huawei 826 CN 828 Email: mach.chen@huawei.com 830 Jie Dong 831 Huawei 832 CN 834 Email: jie.dong@huawei.com 835 Zongpeng Du 836 Huawei 837 CN 839 Email: duzongpeng@huawei.com 841 Zheng Haomian 842 Huawei 843 CN 845 Email: zhenghaomian@huawei.com 847 Xian Zhang 848 Huawei 849 CN 851 Email: zhang.xian@huawei.com 853 James Huang 854 Huawei 855 CN 857 Email: james.huang@huawei.com 859 Qiwen Zhong 860 Huawei 861 CN 863 Email: zhongqiwen@huawei.com