idnits 2.17.1 draft-izh-ccamp-flexe-fwk-02.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Using lowercase 'not' together with uppercase 'MUST', 'SHALL', 'SHOULD', or 'RECOMMENDED' is not an accepted usage according to RFC 2119. Please use uppercase 'NOT' together with RFC 2119 keywords (if that is what you mean). Found 'SHALL not' in this paragraph: Req-10 The system SHALL allow the addition (or removal) of one or more FlexE clients against the FlexE group which is being terminated. The addition (or removal) of FlexE client SHALL not affect the services for the other FlexE client signals. == Using lowercase 'not' together with uppercase 'MUST', 'SHALL', 'SHOULD', or 'RECOMMENDED' is not an accepted usage according to RFC 2119. Please use uppercase 'NOT' together with RFC 2119 keywords (if that is what you mean). Found 'SHOULD not' in this paragraph: Unavailable slot information: this information is used to indicate certain slots SHOULD not be considered when computing an end-to-end path. The unavailable slots can not be used to forward signal because of the transport constraints. -- The document date (March 13, 2017) is 2600 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Missing Reference: '1-254' is mentioned on line 221, but not defined == Missing Reference: 'FlexE-IA' is mentioned on line 1082, but not defined == Unused Reference: 'RFC3209' is defined on line 1250, but no explicit reference was found in the text == Unused Reference: 'RFC3471' is defined on line 1255, but no explicit reference was found in the text == Unused Reference: 'RFC3473' is defined on line 1260, but no explicit reference was found in the text == Unused Reference: 'RFC3630' is defined on line 1266, but no explicit reference was found in the text == Unused Reference: 'RFC4203' is defined on line 1276, but no explicit reference was found in the text == Unused Reference: 'RFC4204' is defined on line 1281, but no explicit reference was found in the text == Unused Reference: 'RFC3945' is defined on line 1290, but no explicit reference was found in the text Summary: 0 errors (**), 0 flaws (~~), 12 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Common Control and Measurment Plane I. Hussain, Ed. 3 Internet-Draft R. Valiveti 4 Intended status: Informational Infinera Corp 5 Expires: September 14, 2017 Q. Wang, Ed. 6 ZTE 7 L. Andersson, Ed. 8 M. Chen 9 H. Zheng 10 Huawei 11 March 13, 2017 13 GMPLS Routing and Signaling Framework for Flexible Ethernet (FlexE) 14 draft-izh-ccamp-flexe-fwk-02 16 Abstract 18 This document specifies GMPLS Control Plane Signalling and Routing 19 protocol extensions for Flexible Ethernet (FlexE). The FlexE data 20 plane were specified by Optical Internetworking Forum (OIF) in two 21 implementation agreements in 2016. 23 As different from earlier Ethernet data planes FlexE allows for 24 decoupling of the Ethernet Physical layer (PHY) and Media Access 25 Control layer (MAC) rates. 27 This document also specifies the use cases of FlexE technology, GMPLS 28 control plane requirements, framework and architecture. 30 Status of This Memo 32 This Internet-Draft is submitted in full conformance with the 33 provisions of BCP 78 and BCP 79. 35 Internet-Drafts are working documents of the Internet Engineering 36 Task Force (IETF). Note that other groups may also distribute 37 working documents as Internet-Drafts. The list of current Internet- 38 Drafts is at http://datatracker.ietf.org/drafts/current/. 40 Internet-Drafts are draft documents valid for a maximum of six months 41 and may be updated, replaced, or obsoleted by other documents at any 42 time. It is inappropriate to use Internet-Drafts as reference 43 material or to cite them other than as "work in progress." 45 This Internet-Draft will expire on September 14, 2017. 47 Copyright Notice 49 Copyright (c) 2017 IETF Trust and the persons identified as the 50 document authors. All rights reserved. 52 This document is subject to BCP 78 and the IETF Trust's Legal 53 Provisions Relating to IETF Documents 54 (http://trustee.ietf.org/license-info) in effect on the date of 55 publication of this document. Please review these documents 56 carefully, as they describe your rights and restrictions with respect 57 to this document. Code Components extracted from this document must 58 include Simplified BSD License text as described in Section 4.e of 59 the Trust Legal Provisions and are provided without warranty as 60 described in the Simplified BSD License. 62 Table of Contents 64 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 65 1.1. Requirements Language . . . . . . . . . . . . . . . . . . 5 66 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 5 67 3. Usecases . . . . . . . . . . . . . . . . . . . . . . . . . . 6 68 3.1. FlexE Unware transport . . . . . . . . . . . . . . . . . 6 69 3.2. FlexE Aware transport . . . . . . . . . . . . . . . . . . 8 70 3.3. FlexE Termination in Transport . . . . . . . . . . . . . 12 71 3.3.1. FlexE Client at Both endpoints . . . . . . . . . . . 12 72 3.3.2. Interworking of FlexE Client w/ Native Client at the 73 other endpoint . . . . . . . . . . . . . . . . . . . 14 74 3.3.3. Interworking of FlexE client w/ Client from OIF_MLG . 15 75 3.4. Back-to-Back FlexE . . . . . . . . . . . . . . . . . . . 16 76 3.5. FlexE Client BW Resizing . . . . . . . . . . . . . . . . 17 77 4. Requirements . . . . . . . . . . . . . . . . . . . . . . . . 19 78 5. Framework . . . . . . . . . . . . . . . . . . . . . . . . . . 21 79 5.1. FlexE Layer Model . . . . . . . . . . . . . . . . . . . . 21 80 5.1.1. Layer Model in FlexE Unaware Case . . . . . . . . . . 21 81 5.1.2. Layer Model in FlexE Terminating Case . . . . . . . . 22 82 5.1.3. Layer Model in FlexE Aware Case . . . . . . . . . . . 22 83 5.2. GMPLS Considerations . . . . . . . . . . . . . . . . . . 23 84 5.2.1. General Considerations . . . . . . . . . . . . . . . 23 85 5.2.2. Consideration of FlexE LSPs . . . . . . . . . . . . . 23 86 5.2.3. Control-Plane Modelling of FlexE Network Elements . . 24 87 5.2.4. FlexE Layer Resource Allocation Considerations . . . 24 88 5.2.5. Neighbour Discovery and Link Property Correlation . . 25 89 5.2.6. Routing and Topology Dissemination . . . . . . . . . 25 90 5.3. Control-Plane Protocol Requirements . . . . . . . . . . . 26 91 5.3.1. Support for Signalling of FlexE . . . . . . . . . . . 26 92 5.3.2. Support for Routing of FlexE . . . . . . . . . . . . 26 93 5.3.3. Support for Neighbour Discovery and Link Property and 94 Link Correlation . . . . . . . . . . . . . . . . . . 27 96 6. Architecture . . . . . . . . . . . . . . . . . . . . . . . . 27 97 7. Solution . . . . . . . . . . . . . . . . . . . . . . . . . . 27 98 8. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 27 99 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 27 100 10. Security Considerations . . . . . . . . . . . . . . . . . . . 27 101 11. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 28 102 12. References . . . . . . . . . . . . . . . . . . . . . . . . . 28 103 12.1. Normative References . . . . . . . . . . . . . . . . . . 28 104 12.2. Informative References . . . . . . . . . . . . . . . . . 29 105 Appendix A. Additional Stuff . . . . . . . . . . . . . . . . . . 29 106 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 30 108 1. Introduction 110 Traditionally, Ethernet MAC rates were constrained to match the rates 111 of the Ethernet PHY(s). OIF's implementation agreement [OIFMLG3] was 112 the first step in allowing MAC rates to be different than the PHY 113 rates standardized by IEEE. A recently approved implementation 114 agreement [OIFFLEXE1] allows for complete decoupling of the MAC data 115 rates and the Ethernet PHY(s). 117 This includes support for 119 a. MAC rates which are greater than the rate of a single PHY 120 (multiple PHYs) are bonded to achive this 122 b. MAC rates which are less than the rate of a PHY (sub-rate) 124 c. support of multiple FlexE CLients carried over a single PHY, or 125 over a collection of bonded PHY). 127 The capabilities supported by the FlexE implementation agreement 128 version 1.0 are: 130 a. Support a large rate Ethernet MAC over bonded Ethernet PHYs, e.g. 131 supporting a 200G MAC over 2 bonded 100GBASE-R PHY(s) 133 b. Support a sub-rate Ethernet MAC over a single Ethernet PHY, e.g. 134 supportnig a 50G MAC over a 100GBASE-R PHY 136 c. Support a collection of flexible Ethernet clients over a single 137 Ethernet PHY, e.g. supporting two MACs with the rates 25G, 50G 138 over a single 100GBASE-R PHY 140 d. Support a sub-rate Ethernet MAC over bonded PHYs, e.g. supporting 141 a 150G Ethernet client over 2 bonded 100GBASE-R PHY(s) 143 e. Support a collection of Ethernet MAC clients over bonded Ethernet 144 PHYs, e.g. supporting a 50G, and 150G MAC over 2 bonded Ethernet 145 PHY(s) 147 All networks which support the bonding of Ehernet interfaces (as per 148 [OIFFLEXE1]) include a basic building block -- which consists of two 149 FlexE Shim functions (located at opposite ends of a link) and the 150 (logical) point to point links that carry the Ethernet PHY signals 151 between the two FlexE Shim Functions. These logical point-to-point 152 PHY links can be realized in a variety of ways: 154 a. These are direct point-to-point links with no intervening 155 transport network. 157 b. The Ethernet PHY(s) are transparently transported via an Optical 158 Transport Network. Optical Transport Networks (defined by 159 [G.709] and [G798]) have recently expanded the traditional bit 160 (or codeword) transparent transport of Ethernet client signals, 161 and included support for the usecases identified in the OIF FlexE 162 implementation agreement. 164 c. Realized by tunneling the Ethernet PHY(s) over some other type of 165 network (e.g. IP/MPLS). Thus, for example, the Ethernet PHY(s) 166 signals could be carried over a pseudowire (or a LSP)in the IP/ 167 MPLS network. Note that the OIF implementation agreement 168 [OIFFLEXE1] only includes support for 100G Ethernet PHY(s). As 169 a result of this encapsulation into a PW, the bandwidth of the PW 170 will be much larger than the bit rate of the Ethernet PHY (i.e. 171 100G), and such a pseudowire cannot be transported in networks 172 that only include 100G Ethernet links. This scenario is 173 realizable when (a) higher rate Ethernet PHY(s), e.g. 200G/40G 174 are supported) or (b) OIF extends the FlexE groups to include 175 lower rate Ethernet PHY(s), e.g. at the 25G/50G rate. Further 176 study is needed to ensure that these scenarios are realizable, 177 practical, and beneficial to operators. With this in mind, the 178 current draft doesn't include any coverage for this scenario. 180 This Internet-draft examines the usescases that arise when the 181 logical links between FlexE capable devices are (a) point-to-point 182 links without any intervening network (b) realized via Optical 183 transport networks. This draft considers the variants in which fhe 184 two peer FlexE devices are both customer-edge devices, or customer- 185 edge/provider edge devices. This list of usecases will help identify 186 the Control Plane (i.e. Routing and Signaling) extensions that may 187 be required. 189 1.1. Requirements Language 191 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 192 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 193 document are to be interpreted as described in RFC 2119 [RFC2119]. 195 2. Terminology 197 a. CE (Customer Edge) - the group of functions that support the 198 termination/orignation of data received from or sent to the 199 network, 201 b. Crunching: (Editors note: text to be submitted>) 203 c. Ethernet PHY: an entity representing 100G-R Physical Coding 204 Sublayer (PCS), Physical Media Attachment (PMA), and Physical 205 Media Dependent (PMD) layers. 207 d. FlexE Calendar: The total capacity of a FlexE group is 208 represented as a collection of slots which have a granularty of 209 5G. The calendar for a FlexE group composed of n 100G PHYs is 210 represented as an array of 20n slots (each representing 5G of 211 bandwidth). This calendar is partitioned into sub-calendars, 212 with 20 slots per 100G PHY. Each FlexE client is mapped into one 213 or more calendar slots (based on the bandwidth of the FlexE 214 client). 216 e. FlexE Client: an Ethernet flow based on a MAC data rate that may 217 or may not correspond to any Ethernet PHY rate. 219 f. FlexE Group: A FlexE Group is composed of from 1 to n Ethernet 220 PHYs. In the first version of FlexE each PHY is identified by a 221 number in the range [1-254]. 223 g. FlexE Interface: A logic interface that is composed of from 1 to 224 n Ethernet interfaces. 226 h. FlexE Link: A logic link that connects two FexE interfaces 227 residing in two adjacent nodes. 229 i. FlexE Shim: the layer that maps or demaps the FlexE clients 230 carried over a FlexE group. 232 j. FlexE Sub-Interface: A channelized logic sub-interface that is 233 allocated specific slots from a FlexE interface, the number of 234 slots depends on the rate of the FlexE Client that will be 235 transmitted through this sub-interface. 237 k. FlexE Sub-Link: A logic link that connects two FlexE sub- 238 interfaces that residing in two adjacent nodes. 240 3. Usecases 242 3.1. FlexE Unware transport 244 The FlexE Shim layer in a router maps the FlexE client(s) over the 245 FlexE group. The transport network is unware of the FlexE. Each of 246 the FlexE group PHY is carried independently across the transport 247 network over the same fiber route. The FlexE Shim in the router 248 tolerates end-to-end skew across the network. In this usecase, the 249 router makes flexible use of the full capacity of the FlexE group, 250 and depends on legacy transport equipment to realize PCS-codeword- 251 transparent transport of 100GbE. It allows striping of PHYs in the 252 FlexE group over multiple line cards in the transport equipment. It 253 is worth mentioning that in this case, the FlexE Shim layer is 254 terminated at the routers, and the coordination of operations related 255 to FlexE clients, e.g. creating new FlexE clients, deleting existing 256 FlexE clients, and resizing the bandwidth of existing FlexE clients 257 (if desired) happens between the two routers. Note that the 258 transport network is completely transparent to the FlexE signals, and 259 doesn't participate in any FlexE protocols. 261 ================================================================== 263 + FlexE Ethernet Client(s) + 264 +-----------------------------------------------------------+ 265 + + 266 + FlexE skew tolerance + 267 +----------------------------------------+ 268 + for end-to-distance + 270 +-----------+ 2x100GE +---------+ +----------+ +------------+ 271 | | | | | | | | 272 | Router1 | | | | | | | 273 |FlexE Shim +---------+ A-end | | Z-end +-----+Router 2 | 274 | | | (FlexE | | (FlexE | |(FlexE Shim)| 275 | +---^-----+ unaware)| | unaware)+-----+ | 276 | | | | | | | | | 277 | | | | | | | | | 278 +-----------+ + +---------+ +----------+ +------------+ 279 FlexE Group 281 \----------Transport----------/ 282 network 283 +--------------+ +----------------+ 284 | FlexE Clients| | FlexE Client(s)| 285 +--------------+ +----------------+ 286 | FlexE Shim | | FlexE Shim | 287 +----+----+----+ +----+------+----+ 288 |PHY | | PHY | | PHY | | PHY | 289 +---+---+--+---+ +---+--+ +--+--+ 290 | | +-----+ +-----+ | | 291 | +----------+ PHY | | PHY |-------+ | 292 | +-----+ +-----+ | 293 | | ODU4+-----------+ ODU4| | 294 | +-----+ +-----+ | 295 | | 296 | +-----+ +-----+ | 297 +-----------------+ PHY | | PHY +-----------------+ 298 +-----+ +-----+ 299 | ODU4+-----------+ ODU4| 300 +-----+ +-----+ 302 ================================================================== 304 Figure 1: FlexE unaware transport 306 3.2. FlexE Aware transport 308 This scenario represents an optimization of the FlexE unaware 309 transport presented in Section 3.1, and illustrated in Figure 1. In 310 this application (see Figure 2), the devices at the edge of the 311 transport network do not terminate the FlexE shim layer, but are 312 aware of the (a) composition of the FlexE group (i.e. set of all 313 contained Ethernet PHYs) and (b) format of the FlexE overhead. At 314 the ingress to the transport network, the transport network edge 315 removes the unavailable calendar slots, and retains all available 316 calendar slots (whether they are allocated or not). At the egress 317 point of the transport network, the edge device adds the unavailable 318 calendar slots back. The result is that the FlexE Shim layers at 319 both routers see exactly the same input that they saw in the FlexE 320 unware scenario -- with the added benefit that the line (or DWDM) 321 side bandwidth has been optimized to be sufficient to carry only the 322 available calendar slots in all of the Ethernet PHY(s) in the FlexE 323 group. 325 The transport network edge device could learn of the set of 326 unavailable calendar slots in a variety of ways; a few examples are 327 listed below: 329 a. In this scenario, the transport network edge does not expect the 330 number of unavailable calendar slots to change dynamically. The 331 set of unavailable calendar slots is configured against each 332 Ethernet PHY in the FlexE group. The FlexE demux function in the 333 transport network edge device (A) compares the information about 334 calendar slots which are expected to be unavailable (as per user 335 supplied configuration), with the corresponding information 336 encoded by the customer edge device in the FlexE overhead (as 337 specified in [OIFFLEXE1]). If there is a mismatch between the 338 unavailable calendar slots in any of the PHYs within a FlexE 339 group, the transport edge node software could raise an alarm to 340 report the inconsistency between the provisioning information at 341 the transport network edge, and the customer edge device. 343 b. The Transport network edge is configured to act in a "slave" 344 mode. In this mode, the FlexE demux function at the Transport 345 network edge (A) receives the information about the available/ 346 unavailable calendar slots by observing the FlexE overhead (as 347 specified in [OIFFLEXE1]) and uses this information to calculate 348 the bandwidth of the ODUflex (or fixed rate ODUs) connection that 349 could carry the FlexE PCS end-to-end. This scenario allows for 350 the set of available/unavailable calendar slots to change 351 (slowly) with time -- but comes with the complexity of resizing 352 the ODUflex connection in response to changes in the number of 353 available calendar slots. 355 Note that the process of removing unavailable calendar slots from a 356 FlexE PHY is called "crunching" (see [OIFFLEXE1]). The following 357 additional notes apply to Figure 2: 359 a. As in the FlexE unaware case, all PHYs of the FlexE group MUST be 360 terminated between the same two FlexE shims. 362 b. The crunched FlexE PHYs are independently transported through the 363 transport network. The number of used (and unused) calendar 364 slots can be different across the FlexE group. In particular, if 365 all the calendar slots in a FlexE PHY are in use, the crunching 366 operation leaves the original signal intact. 368 c. In this illustration, the different FlexE PHY(s) are transported 369 using ODUflex containers in the transport network. These ODUflex 370 connections can be of different rates. 372 d. In the most general form, G.709 Section 17.12 [G.709] allows for 373 a FlexE group consisting of m Ethernet PHY(s) to be crunched, 374 combined, and transported using n ODUFlex containers (where n can 375 range between 1 and m). In other words, the ITU G.709 376 recommendation allows for (but not require the support for) the 377 degenerate cases in which (a) each Ethernet PHY within the group 378 is transported using its own ODUflex, and (b) all the PHY(s) are 379 crunched, combined and transported over a single ODUflex 380 container. If all the sub-calendar slots in a given PHY are 381 available, it is possible to transport the content of the PHY in 382 one of two ways: (a) as shown in Figure 2, or (b) using a FlexE 383 unware (i.e. PCS-codeword transparent transport) mode. The 384 latter approach (of using FlexE unaware transport) for a few 385 select (fully-utilized) PHYs is not attractive from the 386 perspective of skew between the PHYs that comprise the FlexE 387 group. For simplicity, the preferred mode of operation will be 388 one in which the same mapping procedure is used for member PHYs 389 of a FlexE group. 391 e. When the crunched FlexE PHY(s) have a rate that is identical to 392 that of a standard Ethernet PHY, it is possible that the 393 transport network may utilize standard ODU containers such as 394 ODU2e, ODU4 etc. As currently defined by ITU G.709 Section 17.12 395 [G.709], the crunched signal is always mapped to an ODUflex, and 396 the mapping to a fixed rate ODU signal is not required. This 397 option could be dropped if it results in any significant 398 simplification. 400 f. The bandwidth of the ODUFlex connections shall be computed based 401 on the total number of available 5G calendar slots which in the 402 subset of PHY(s) which are transported over this ODUflex entity 403 (see Section 3.2, G.709:Table 7-2 [G.709]). 405 g. As in the FlexE unaware case, the FlexE Shim layer is terminated 406 at the routers, and the coordination of operations related to 407 FlexE clients, e.g. creating new FlexE clients, deleting existing 408 FlexE clients, and resizing the bandwidth of existing FlexE 409 clients (if desired) happens between the two routers. Note that 410 the transport network is completely transparent to the FlexE 411 signals, and doesn't participate in any FlexE protocols. As long 412 as the set of available (and unavailable) calendar slots on the 413 PHY(s) does not change after the initial setup, the transport 414 network is not required to make any changes to the number/rates 415 of ODUflex connections which were created at service setup time. 417 h. In the FlexE aware case, the OTN pipes are sized to match the 418 currently configured set of available/unavailable calendar slots 419 across the FlexE group. If this set of available/unavailable 420 calendar slots on the PHY(s) is allowed to dynamically change, 421 the ODUflex connections would also require resizing to match the 422 new usage of available slots. However, the ODUflex hitless 423 resizing mechanism defined in G.7044 [G7044] has the following 424 restrictions: (a) ODUflex connection being resized must have 425 bandwidth of 100G or less (b) the ODUflex connection cannot 426 traverse OTUCn links which were introduced in the latest revision 427 of G.709. With the present limitations in the ODUflex resizing 428 mechanism, the dynamic adjustent of ODUflex bandwidth (for the 429 FlexE aware case) is possible only if (a) the transport network 430 edge maps each crunched PHY to its own ODUflex connection (b) the 431 Ethernet PHY rates are 100G or less (c) the ODUflex connection 432 does not traverse any OTUCn links along the end-to-end path. As 433 a result, this scenario is not considered in this document. 435 [[N1: The figure may need further editing to accurately depict the 436 signal hierarchy. --RV]] 438 ================================================================ 439 FlexE Ethernet Client(s) 440 +-----------------------------------------------------+ 441 FlexE skew tolerance 442 +---------------------------------------------+ 443 for end+to+distance 445 +--------+ 2 x 100GE +---------+ +---------+ +------+ 446 | R1 | | | | +----+ R2 | 447 | (FlexE+-----------+ NE A | | NE Z | |(FlexE| 448 | Shim) | | (FlexE | | (FlexE +----+ Shim | 449 | +-----^-----+ aware) | | aware) | | | 450 | | | | | | | | | 451 +--------+ + +---------+ +---------+ +------+ 452 FlexE Group 453 \+------+Transport+-------+/ 454 network 455 +-------------+ +-------------+ 456 |FlexE clients| |FlexE clients| 457 +-------------+ +-------------+ 458 | FlexE Shim | | FlexE Shim | 459 +-------------+ +-------------+ 460 | PHY | PHY | | PHY | PHY | 461 +-------------+ +-------------+ 462 | | | | 463 | | +-------------+ +------------+ | | 464 | | | FlexE-psg | | FlexE-psg | | | 465 | | +-------------+ +------------+ | | 466 | +--+ PHY|ODUflex +--------|ODUFlex|PHY +--+ | 467 | +-------------+ +------------+ | 468 | | 469 | +-------------+ +------------+ | 470 | | FlexE-psg | | FlexE-psg | | 471 | +-------------+ +------------+ | 472 +--------+ PHY|ODUflex +--------|ODUFlex|PHY +--------+ 473 +-------------+ +------------+ 475 + Legend: 476 | R1, R2 + Routers (supporting the FlexE clients) 477 | NE A, Z + Transport Network Edge nodes 478 + FlexE-psg: FlexE partial rate (sub) group signal 479 (per G.709:17.12) 481 =============================================================== 483 Figure 2: FlexE Aware Transport 485 3.3. FlexE Termination in Transport 487 These usecases build upon the basic router-transport equipment 488 connectivity illustrated in Figure 1. The FlexE shim layer at the 489 router maps to the set of FlexE clients over the FlexE group, as 490 usual. This section considers various usecases in which the 491 equipment located at the edge of the transport network instantiates 492 the FlexE Shim function which peers with the FlexE shim on the 493 customer device. In the router to network direction, the transport 494 edge node terminates the FlexE shim layer, and extracts one or more 495 FlexE client signals, and transports them through the network. That 496 is, these usecases are distinguished from the FlexE unaware cases in 497 that the FlexE group, and the FlexE shim layer end at the transport 498 network edge, and only the extracted FlexE client signals transit the 499 optical network. In the network to router direction, the transport 500 edge node maps a set of FlexE clients to the FlexE group (i.e. 501 performing the same functions as the router which connects to the 502 transport network).The various usecases differ in the combination of 503 service endpoints in the transport network. In the FlexE termination 504 scenarios, the distance between the FlexE Shims is limited the normal 505 Ethernet link distance. The FlexE shims in the router, and the 506 equipment need to support a small amount skew. 508 3.3.1. FlexE Client at Both endpoints 510 In this scenario, service consists of transporting a FlexE client 511 through the transport network, and possibly combining this FlexE 512 client with other FlexE clients into a FlexE group at the endpoints. 513 The FlexE client signal BMP mapped into an ODUflex (of the 514 appropriate rate) and then switched across the OTN. Figure 3 515 illustrates the scenario involving the mapping of a FlexE client to 516 an ODUflex envelope; this figure only shows the signal "stack" at the 517 service endpoints, and doesn't illustrate the switching of the 518 ODUflex entity through the OTN. The ODUflex signal then carried over 519 a sequence of OTUk links (with a maximum rate of 100G), and/or OTUCn 520 (with rates of n X 100G). Although Figure 3 illustrates the scenario 521 in which one FlexE client is transported within the OTN, the 522 following points should be noted: 524 a. When the FlexE Shim termination function recovers multiple FlexE 525 client signals (at node A), the FlexE signals can be transported 526 independently. In other words, it is not a requirement that all 527 the FlexE client signals be co-routed. 529 b. Conversely, at the egress node, FlexE clients from different 530 endpoints can be combined via the FlexE shim, eventually exiting 531 the transport edge node over an Ethernet group. 533 c. The description presented above(implicitly) assumes that the 534 FlexE Client signals have a constant bit rate which does not 535 change after the service setup. In the scenarios in which the 536 FlexE Client Signal rates are permitted to be dynamically 537 adjusted (i.e. resized), the resizing process would require 538 coordination across three resizing domains: (a) between Router1, 539 Node A (b) Resizing the ODUflex connection between the transport 540 edge nodes A, Z (c) between the Node Z, Router2. This usecase is 541 not considered in this document since G.709 [G.709] has dropped 542 support for the the hitless resizing of ODUflex connections with 543 bandwidths larger than 100G. In the absence of a hitless B100G 544 ODUflex resizing mechanism, this will have to be realized by 545 treating it like a request for new service with a new (increased 546 or decreased) rate. The FlexE client bandwidth resize 547 applicability for various use cases is summarized in Table 1. 549 ================================================================== 551 +--------+ 2 x 100GE +---------+ +----------+ +--------+ 552 | | | | | | | | 553 | Router1| | | | | | | 554 | FlexE +-----------+ A-end | | Z-end +------+Router2 | 555 | Shim | | (FlexE | | (FlexE | |FlexE | 556 | +-----^-----+ term) | term) +------+ Shim | 557 | | | | | | | | | 558 | | | | | | | | | 559 +--------+ + +---------+ +----------+ +--------+ 560 FlexE Group 561 \+--------+Transport+--------+/ 562 network 564 +-----------+ +--------------+ +-------------+ +-----------+ 565 | Client(s) | | Client | | Client | | Client(s) | 566 +-----------+ +--------+-----+ +------+------+ +-----------+ 567 | FlexE Shim| | Shim | | | | Shim | | FlexE Shim| 568 +-----------+ +--------+ ODU | | ODU +------+ +-----------+ 569 | PHY(s) | | PHY(s) | flex| | flex |PHY(s)| | PHY(s) | 570 +---+-------+ +---+----+--+--+ +---+--+---+--+ +---+-------+ 571 | | | | | | 572 +---------------+ +-----------+------+----------+ 574 ================================================================= 576 Figure 3: FlexE termination: FlexE clients at both endpoints 578 3.3.2. Interworking of FlexE Client w/ Native Client at the other 579 endpoint 581 The OIF implementation agreement [OIFFLEXE1] currently supports FlexE 582 client signals carried over one or more 100GBASE-R PHY(s). There is 583 a calendar of 5G timeslots associated with each PHY, and each FlexE 584 client can make use of a number of timeslots (possibly distributed 585 across the members of the FlexE group). This implies that the FlexE 586 client rates are multiples of 5Gbps. When the rates of the FlexE 587 client signals matches the MAC rates corresponding to existing 588 Ethernet PHYs, i.e. 10GBASE-R/40GBASE-R/100GBASE-R, there is a need 589 for the FlexE client signal to interwork with the native Ethernet 590 client received from a single (non-FlexE capable) Ethernet PHY. This 591 capability is expected to be extended to any future Ethernet PHY 592 rates that the IEEE may define in future (e.g. 25G, 50G, 200G etc.). 593 In these cases, although the bit rate of the FlexE client matches the 594 MAC rate of other endpoint, the 64B66B PCS codewords for the FlexE 595 client need to be transformed (via ordered set translation) to match 596 the specification for the specific Ethernt PHY. These details are 597 described in Section 7.2.2 of [OIFFLEXE1] and are not eloborated any 598 further in this document. 600 Figure 4 illustrates a scenario involving the interworking of a 10G 601 FlexE client with a 10GBASE-R native Ethernet signal. In this 602 example, the network wrapper is ODU2e. 604 ================================================================== 606 +--------+ 2 x 100GE +-------+ +-------+ +--------+ 607 | | | | | | | | 608 | Router1| | | | | | | 609 |(FlexE +-----------+ A-end | | Z-end | 10GE |Router 2| 610 | Shim) | |(FlexE | | +------+ | 611 | +-----^-----+ term) | | | | | 612 | | | | | | | | | 613 | | | | | | | | | 614 +--------+ + +-------+ +-------+ +--------+ 615 FlexE Group 616 \+---------Transport---------+/ 617 network 619 +-----------+ +---------------+ 620 | Client(s) | | Client | +------------+ +---------+ 621 +-----------+ +-------+-------+ | 10GE PCS | | 10GE PCS| 622 | FlexE Shim| | Shim | | +-------+----+ +---------+ 623 +-----------+ +-------+ ODU | | ODU2e | PHY| | PHY | 624 | PHY(s) | | PHY(s)| 2e | +---+---+--+-+ +-----+---+ 625 +---+-------+ +---+-------+---+ | | | 626 | | | | | | 627 | | | | | | 628 +---------------+ +-------------+ +------------+ 630 ================================================================= 632 Figure 4: FlexE client interop with Native Ethernet Client 634 3.3.3. Interworking of FlexE client w/ Client from OIF_MLG 636 As explained in the Introduction section ( Section 1 OIFMLG3 637 [OIFMLG3] introduced support for carrying 10GE and 40GE client 638 signals over a group of 100GBASE-R Ethernet PHY(s). While the most 639 recent implementation agreement doesn't call it out explicitly, it is 640 expected that the FlexE clients (as defined in [OIFFLEXE1]), and 641 10GBASE-R/40GBASE-R clients supported by OIFMLG3 [OIFMLG3]) will 642 interoperate. 644 Figure 5 illustrates a scenario involving the interworking of a 10G 645 FlexE client with a 10GBASE-R client supported by an OIFMLG3 646 interface. In this example, the network wrapper is ODU2e. 648 ================================================================== 650 +--------+ 2 x 100GE +---------+ +---------+ +---------+ 651 | | | | | | | | 652 | Router1| | | | | | | 653 | FlexE +-----------+ A-end | | Z-end +------+Router 2 | 654 | Shim | | (FlexE | | | |(MLG-3.0)| 655 | +-----^-----+ term) | | +------+ | 656 | | | | | | | | | 657 | | | | | | | | | 658 +--------+ + +---------+ +---------+ +---------+ 659 FlexE Group 661 \+--------+Transport+--------+/ 662 network 664 +-----------+ +-------------+ +--------------+ +----------+ 665 | Client(s) | | Client | | 10GE PCS | | 10GE Cl. | 666 +-----------+ +--------+----+ +------+-------+ +----------+ 667 | FlexE Shim| | Shim | | | | MLG3 | | MLG3 | 668 +-----------+ +--------+ ODU| | ODU +-------+ +----------+ 669 | PHY(s) | | PHY(s) | 2e | | 2e | PHY(s)| | PHY(s) | 670 +---+-------+ +---+----+--+-+ +---+--+---+---+ +---+------+ 671 | | | | | | 672 +---------------+ +------------+ +------------+ 674 ================================================================= 676 Figure 5: FlexE client interop with Ethernet Client supported by MLG3 678 3.4. Back-to-Back FlexE 680 This section covers a degenerate FlexE termination scenario in which 681 Router1, Router2, and Router3 are interconnected through back-to-back 682 FlexE groups without an intermediate transport network (see 683 Figure 6). Even in scenarios where there is a transport network 684 providing FlexE unaware/aware transport services for this pair of 685 FlexE groups, the FlexE layer network can be viewed as an overlay on 686 top of the underlying transport network. As such, all of the FlexE 687 Shim operations (e.g. adding/deleting FlexE clients, resizing 688 existing clients) proceed in the same manner -- regardless of whether 689 the routers are directly connected or not. 691 In this example, the FlexE Shim at Router2 extracts one or more FlexE 692 client signals from the FlexE group connected to Router1, and 693 mutliplexes these extracted FlexE signals into the FlexE group 694 towards the appropriate router (e.g. Router3). Note that each of 695 the extracted FlexE client signals can be independently routed 696 towards its respective FlexE group. 698 ================================================================== 700 +--------+ 2 x 100GE +---------+ 3 x 100GE +---------+ 701 | | | | | | 702 | Router1| | | | | 703 | FlexE +-----------+ Router2 +-----------+ Router3 | 704 | Shim | | FlexE +-----------+ FlexE | 705 | +-----^-----+ Shim +-----^-----+ Shim | 706 | | | | | | | | 707 | | | | | | | | 708 +--------+ + +---------+ + +---------+ 709 FlexE Group FlexE Group 711 ================================================================= 713 Figure 6: Back-to-Back FlexE 715 3.5. FlexE Client BW Resizing 717 The hop-by-hop (a hop is delimited by two FlexE Shim functions) 718 resizing of a FlexE client signal operates by maintaining two sets of 719 calendar slots for each client: the present and the future. Once the 720 configuration of both calendar slots for a specific client is 721 complete, the node signals to its peer to switch to from the present 722 set to the new set of calendar slots. Note that the switch to the 723 new set of calendar slots is unidirectional, and the process is 724 executed independently for both directions of transfer. This process 725 makes use of the following FlexE overhead (as per [OIFFLEXE1] 727 a. Currently active FlexE calendar (containing a list of mapping 728 between the 5G tributary slots and the FlexE client signals 730 b. Future calendar to which the sender wants to transition to. 732 c. Calendar switch request bit (CR) 734 d. Calendar switch acknowldege bit (CA) 736 FlexE client resizing operations are supported and can be achieved 737 via the configuration of Calendar A and Calendar B. It is worth 738 noting that there is no guarantee that such resizing will be hitless. 740 Table 1 provides a summary of client bandwidth resize applicability 741 in various use cases presented in this document. 743 +--------+---------+-------------+------------+---------------------+ 744 | FlexE | FlexE | Usecase | Transport | Resizing supported? | 745 | Shim e | Shim en | | Network | | 746 | ndpoin | dpoint | | Function | | 747 | t 1 | 2 | | | | 748 +--------+---------+-------------+------------+---------------------+ 749 | CE | CE | Section 3.1 | FlexE | Yes. Done at | 750 | (e.g. | | | unaware | endpoints. The OTN | 751 | router | | | transport | pipes are | 752 | ) | | | | configured for the | 753 | | | | | maximum number of | 754 | | | | | calendar slots | 755 | | | | | across each PHY in | 756 | | | | | the FlexE group. | 757 | | | | | Therefore, no | 758 | | | | | resizing is | 759 | | | | | required in the OTN | 760 | | | | | layer. | 761 | CE | CE | Section 3.2 | FlexE | Supported at the | 762 | (e.g. | | | aware | endpoints only if | 763 | router | | | transport | the set of availabl | 764 | ) | | | | e/unavailable | 765 | | | | | calendar slots is | 766 | | | | | constant. Not | 767 | | | | | supported otherwise | 768 | | | | | (see notes at the | 769 | | | | | end of Section | 770 | | | | | 3.2). | 771 | CE | Transpo | Section | FlexE Term | Not supported due | 772 | (e.g. | rt | 3.3.1 | ination in | to lack to lack of | 773 | router | Network | | Transport | a general (i.e. one | 774 | ) | Edge | | | that works | 775 | | | | | regardless of the | 776 | | | | | ODUflex bandwidth) | 777 | | | | | hitless ODUflex | 778 | | | | | resizing in G.709. | 779 | CE | CE | Section 3.4 | No | Yes. Done at | 780 | (e.g. | | | transport | endpoints by CE(s). | 781 | router | | | network | Thus, for example, | 782 | ) | | | | in Figure 6, the | 783 | | | | | resizing of the | 784 | | | | | end-to-end FlexE | 785 | | | | | client circuit with | 786 | | | | | a scope of Router1- | 787 | | | | | Router2-Router3 is | 788 | | | | | accomplished by | 789 | | | | | correctly | 790 | | | | | coordinating the | 791 | | | | | resizing operations | 792 | | | | | across these two | 793 | | | | | segments: | 794 | | | | | Router1-Router2, | 795 | | | | | Router2-Router3. It | 796 | | | | | is expected that | 797 | | | | | the exact sequence | 798 | | | | | of hop-by-hop | 799 | | | | | resize operations | 800 | | | | | is different | 801 | | | | | between bandwidth | 802 | | | | | increase/decrease | 803 | | | | | scenarios. | 804 +--------+---------+-------------+------------+---------------------+ 806 Table 1: FlexE Client Resizing 808 4. Requirements 810 This section summarizes the requirements for FlexE Group and FlexE 811 Client signaling and routing. The requirements are derived from the 812 usecases described in Section 3 of this document. Data plane 813 requirements (and/or solutions) (e.g. crunching of tributary slots, 814 adding unavailable tributary slots etc.) are not explicitly mentioned 815 in the following text. Given that the control plane sets up circuits 816 that transport client streams, there are no implications for the 817 control plane in matters of delay, jitter tolerance etc. The 818 requirements listed in this section will be used to identify the 819 Control Plane (i.e. Routing and Signaling) extensions that will be 820 required to support FlexE services in an OTN. 822 A Control Plane solution will be compliant to the specification in 823 Section 7 if it meets all the mandatory (MUST, SHALL) requirments, 824 the solution may also meet the optional (SHOULD, MAY) requirments. 826 Req-1 The solution SHALL support the creation of a FlexE group, 827 consisting of one or more (i.e., in the 1 to 254 range) 100GE 828 Ethernet PHY(s). 830 There are several alternatives that can meet this 831 requirement, e.g. routing and signaling protocols, or a 832 centrailized controller/management system with network access 833 to the FlexE mux/demux at each FlexE group termination point. 835 Req-2 The solution SHOULD be able to verify that the collection of 836 Ethernet PHY(s) included in a FlexE group have the same 837 characteristics (e.g. number of PHYs, rate of PHYs, etc.) at 838 the peer FlexE shims. 840 Req-3 The solution SHALL support the ability to delete a FlexE 841 group. 843 Req-4 It SHALL support the ability to administratively lock/unlock 844 a FlexE group. 846 Req-5 It SHALL be possible to add/remove PHY(s) to/from an 847 operational FlexE group while the group has been 848 administratively locked. 850 [Note: Since the addition/removal of Ethernet PHY(s) is done 851 only when the group has been locked, this dataplane operation 852 of the FlexE group ceases until it is placed in an unlocked 853 state.] 855 Req-6 The solution SHALL support the ability to advertise (and 856 discover) the information about FlexE capable nodes, and the 857 FlexE group instances they are supporting. 859 Req-7 It SHALL be possible to assign the transport network 860 treatment for a FlexE group to one the following choices: (a) 861 FlexE unaware transport (b) FlexE aware transport (c) FlexE 862 termination in Transport. 864 Req-8 For the FlexE unaware case, each of the Ethernet PHY(s) in 865 the FlexE group SHALL be mapped independently to the 866 appropriately sized ODU container (as per [G.709], and 867 switched across the transport network [OIFFLEXE1]. The 868 control plane SHALL be capable of co-routing the ODU signals 869 that are transporting the member PHY(s) between the two FlexE 870 Shim functions. 872 [Note: Insert applicable references to ITU, OIF spec for hard 873 skew tolerances] 875 Req-9 In the FlexE aware mode, the OTN SHALL crunch the PHY(s), and 876 map them to one or more ODUflex connections as per [G.709]. 878 When two or more ODUflex connections are used to transport 879 the collection of FlexE PHY(s) in a FlexE group, the system 880 SHALL support the ability to constrain the routes for these 881 ODUflex connections (e.g. co-route them) so that the end-to- 882 end skew is kept to a minimum (and within the range supported 883 by the FlexE Shims). 885 Req-10 The system SHALL allow the addition (or removal) of one or 886 more FlexE clients against the FlexE group which is being 887 terminated. The addition (or removal) of FlexE client SHALL 888 not affect the services for the other FlexE client signals. 890 Req-11 The system SHALL allow the FlexE client signals to flexibly 891 span the set of Ethernet PHY(s) which comprise the FlexE 892 group. In other words, it SHALL be possible to distribute 893 any FlexE client over an arbitrary combination of calendar 894 slots (whose total capacity matches the client bitrate) 895 chosen from a subset of the PHY(s). 897 Req-12 When the FlexE group is terminated on the Transport edge 898 node, this node SHOULD be capable of resizing one or more 899 FlexE client (using the "A/B" calendar signaling defined by 900 OIF) (see Section 3.5). It is acceptable that this resizing 901 is not hitless, and the client signal incurs a glitch during 902 the resizing operation. 904 There is no requirement for the OTN network to support the 905 hitless resizing of the ODUFlex connection which is 906 transporting the FlexE client signal. 908 Req-13 The solution SHALL support FlexE client resizing without 909 affecting any existing FlexE clients within the same FlexE 910 group. 912 5. Framework 914 This section discusses the environment where FlexE operates, this 915 should include both what FlexE runs over and what applications run on 916 top of FlexE. 918 5.1. FlexE Layer Model 920 Based on the cases addressed in Section 3, FlexE has different kinds 921 of mapping hierarchy accordingly. This section gives some 922 description of FlexE layer model in different cases. 924 5.1.1. Layer Model in FlexE Unaware Case 926 This case is depicted in Section 3.1. The FlexE Ethernet client 927 represents an end-to-end connection, which is from the Router 1 to 928 destination Router 2. The FlexE Ethernet client signal is first 929 mapped into the slots of FlexE at Router 1, then the FlexE signal is 930 carried by Ethernet PHYs towards the destination Router 2. When the 931 Ethernet PHYs arrive at Transport network edge node A-end, each PHY 932 will be mapped into a separate ODU4 connection and then forwarded 933 across the OTN network towards the ODU layer connection destination 934 Z-end. 936 Note: in this case, more than one FlexE clients can be carried by 937 FlexE layer. 939 Four different layers exist in this case, and the mapping hierarchy 940 can be seen in Figure 1. 942 5.1.2. Layer Model in FlexE Terminating Case 944 This case is depicted in Section 3.3. Take Section 3.3.1 for 945 example. The FlexE Ethernet signal is first mapped into the slots of 946 FlexE at Router 1, then the FlexE signal is carried by Ethernet PHYs 947 towards the Transport Network edge node A-end. When the FlexE signal 948 arrives at node A-end, node A-end first terminate Ethernet PHY signal 949 and FlexE signal, extracts the FlexE Ethernet client signal, then 950 maps the Ethernet client signal into ODU signal and forwards across 951 the OTN network towards destination node Z-end. Node Z-end first 952 terminate the ODU signal, extract the FlexE client signal from the 953 ODU signal, then map the Ethernet client signal into FlexE signal, 954 which will then be carried by Ethernet PHYs towards destination node 955 Router 2. 957 Two segments of FlexE connection exist in this case. one is from 958 Router 1 to node A-end, and the other is from node Z-end to Router 2. 959 The mapping hierarchy can be seen in Figure 3 961 5.1.3. Layer Model in FlexE Aware Case 963 This case is depicted in Section 3.2. The FlexE Ethernet client is 964 transferred from the R1 to destination R2, while the internal node NE 965 A and NE Z are capable of "crunching" and "combining" operation. The 966 FlexE Ethernet client signal is first mapped into the slots of FlexE 967 at R1, then the FlexE signal is carried by Ethernet PHYs towards the 968 destination R2. When the Ethernet PHY signal arrives at node NE A, 969 node NE A first discards unavailable slots, then map the remaining 970 FlexE slots onto ODU Connection. According to the description in 971 [G.709], these FlexE slots can be carried across the OTN network via 972 a couple of ODUflex signals which are carried in ODUCn/OTUCn/OTSiA 973 signals. 975 Two kinds of mapping hierarchy exist in this case, one is the FlexE 976 connection is carried by Ethernet PHYs, the other is FlexE connection 977 (e.g., FlexE-psg) is carried by ODUflex, which can be seen in 978 Figure 2. 980 5.2. GMPLS Considerations 982 The goal of this section is to provide an insight into the 983 application of GMPLS as a control mechanism in FlexE networks. 984 Specific control-plane requirements for the support of FlexE networks 985 are covered in Section 5.3. This section aims to describe the 986 modelling of controlling the FlexE shim layer specific attributes in 987 different network scenarios based on the capability of FlexE 988 described in OIF Flex Ethernet (FlexE) Implementation Agreement 989 [OIFFLEXE1]. 991 5.2.1. General Considerations 993 The GMPLS control of the FlexE layer deals with the establishment of 994 FlexE connections that are transferred in FlexE capable nodes. GMPLS 995 labels are used to locally represent the FlexE connections and its 996 associated slots assignment information for client. 998 5.2.2. Consideration of FlexE LSPs 1000 The FlexE LSP is a control-plane representation of a FlexE Connection 1001 and MUST be carried by Ethernet PHYs LSP or ODU LSP in the network. 1003 Figure 1 depicts a scenario that the FlexE LSP is carried over 1004 Ethernet PHYs LSP from Router 1 to Router 2. When there is a need to 1005 set up FlexE end-to-end connection to carry FlexE Ethernet client 1006 signal at R1, R1 will first check if there are enough resources for 1007 setting up FlexE LSP. If yes, R1 will first set up Ethernet PHYs LSP 1008 from R1 to R2, and then set up the FlexE LSP over the Ethernet PHYs 1009 LSP. This process actually includes three signalling procedures, the 1010 first one is to set up multiple ODU4 LSPs to carry Ethernet PHYs, the 1011 second one is to set up multiple Ethernet PHYs connection to carry 1012 FlexE LSP, and the third one is to set up FlexE connection to carry 1013 FlexE Ethernet client signal. The signalling of FlexE LSP SHOULD be 1014 able to reserve resource for Ethernet client. 1016 Figure 2 depicts the case that the FlexE LSP is carried over ODU LSP 1017 between NE A and NE Z. This case is different from that one in 1018 Figure 1, and is used to support cases such as the Ethernet PHY rate 1019 is be greater than the wavelength rate, the wavelength rate is not an 1020 integral multiple of the PHY rate. Both NE A and NE Z support the 1021 partial-rate ability ,which means when the FlexE LSP over Ethernet 1022 PHYs arrives at NE A, NE A should first discard the unavailable slots 1023 and then map the remaining FlexE slots into the ODU signal. 1025 5.2.3. Control-Plane Modelling of FlexE Network Elements 1027 FlexE is a new kinds of transport technology, which has many new 1028 constraints. These constraints are listed as follows: 1030 Unavailable slots: this is different from "unused" slot, in that 1031 it is known, due to transport network constraints, that not all of 1032 the calendar slots generated from the FlexE mux will reach the 1033 FlexE demux and therefore no FlexE client should be assigned to 1034 those slots. As defined in the Flex Ethernet Implementation 1035 Agreement, unavailable slots are always at the end of the sub- 1036 calendar configuration for the respective PHY. 1038 Unused slots: unused slots can be allocated to Ethernet client as 1039 available resource. 1041 Partial-rate capability: the partial-rate capability is usually 1042 supported by the OTN edge equipments. If an equipment supports 1043 partial-rate, it means this equipment has the capability of 1044 discarding unavailable slots and transfers the remaining slots 1045 across OTN transport network. 1047 Slot granularity: currently, only one kinds of 5G slot granularity 1048 is defined in OIF Flex Ethernet (FlexE) Implementation Agreement. 1050 5.2.4. FlexE Layer Resource Allocation Considerations 1052 FlexE LSP is used to provide resource service for its client, which 1053 is mainly reflected through the provision of the unused slot resource 1054 information towards the client layer. Besides the slot information, 1055 there are also some other attributes that need to be specified when 1056 allocating resource during connection setup process. 1058 FlexE group number: a bunch of Ethernet PHYs can be bounded 1059 together and used as a whole as one FlexE LSP. FlexE LSPs between 1060 the same source and destination equipment SHOULD NOT have the same 1061 FlexE group number. Source equipment and destination equipment 1062 SHOULD be aware of the existing of different FlexE groups and 1063 which Ethernet PHYs are in which FlexE group. 1065 PHY Number: it's a dynamic and logical number that is assigned 1066 through control plane or management plane, which is unique within 1067 the context of (source, destination), and has a one-to-one 1068 correlation with physical port. This information will also be 1069 carried in the FlexE overhead. Source equipment and destination 1070 equipment SHOULD negotiate a value for every Ethernet PHYs within 1071 one FlexE group. 1073 Slot Assignment information: the FlexE LSP transfers based on the 1074 slot positions, so the equipment SHOULD be able to tell which slot 1075 is assigned to which client. 1077 Partial-rate: during the process of resource allocation, where the 1078 partial-rate would happen should be indicated. 1080 Granularity: currently, only one kinds of 5G slot granularity is 1081 defined in OIF Flex Ethernet (FlexE) Implementation Agreement 1082 [FlexE-IA]. 1084 5.2.5. Neighbour Discovery and Link Property Correlation 1086 There are potential interworking problems between different FlexE 1087 capable equipment. Devices or equipments might not be able to 1088 support the interworking of every slot due to the constraints of 1089 transport network equipment or other constraints. In this case, two 1090 directly connected FlexE capable equipments SHOULD run the neighbour 1091 discovery process and correlate the link property to make sure which 1092 slots are unavailable, which slots can be used by the client. 1093 Neighbour discovery protocol can be communicated in in-band FlexE 1094 section management channel, and also can be communicated through out- 1095 of-band management channel. 1097 5.2.6. Routing and Topology Dissemination 1099 The topology and routing information is used by the path computation 1100 entity to compute an end-to-end path. Besides the basic 1101 interconnected information, there are also some FlexE specific 1102 attributes that should be taken into consideration. 1104 Partial-rate: partial-rate capability is a special feature which 1105 allows an equipment to discard unavailable slots and transfers the 1106 left slots across OTN transport network. Path computation entity 1107 is more likely to compute a feasible path if this capability is 1108 taken into consideration when computing path. 1110 Unavailable slot information: this information is used to indicate 1111 certain slots SHOULD not be considered when computing an end-to- 1112 end path. The unavailable slots can not be used to forward signal 1113 because of the transport constraints. 1115 Unused slot information: unused slot can be allocated to the path 1116 as available resource. 1118 5.3. Control-Plane Protocol Requirements 1120 The control of FlexE networks brings some new additional requirements 1121 to the GMPLS protocols. This section summarizes those requirements 1122 for signalling,routing and Link management protocol. 1124 5.3.1. Support for Signalling of FlexE 1126 Aim of the signaling is to set up an end-to-end LSP for FlexE signal. 1128 The signalling procedures shall be able to assign FlexE releated 1129 attributes for an LSP, which include FlexE group number for a FlexE 1130 LSP. This FlexE group number is unique and can be used to indicate a 1131 group of Ethernet PHYs bonded together. 1133 The signalling procedures shall be able to assign an unique PHY 1134 number for each bonded Ethernet PHY, and a correlation relationship 1135 SHOULD also be indicated between the assigned PHY number and real 1136 physical port number when signalling. 1138 The signalling procedures shall be able to configure the slots 1139 information allocated for a FlexE LSP. 1141 The Signalling procedures shall be able to indicate the palace where 1142 partial-rate mapping happens. 1144 The Signalling procedures shall be able to support the non-hitless 1145 resizing of FlexE client. 1147 5.3.2. Support for Routing of FlexE 1149 The routing protocol extensions are mainly based on the functionality 1150 that is described in [RFC4202] and these extensions are made to fit 1151 into FlexE network. 1153 The routing protocol SHALL distribute sufficient information to 1154 compute paths to enable the signalling procedure to establish LSPs as 1155 described in the previous sections. 1157 The routing protocol SHALL update its advertisements of available 1158 resources and capabilities to include the partial-rate support 1159 information and unused slot information on each Ethernet PHY port. 1161 5.3.3. Support for Neighbour Discovery and Link Property and Link 1162 Correlation 1164 The control plane MAY include support for neighbour discovery such 1165 that a FlexE network can be constructed in a "plug-and-play" manner. 1167 The control plane SHOULD allow the nodes at opposite ends of a link 1168 to correlate the properties that they will apply to the link. Such a 1169 correlation SHOULD include at least the identities of the nodes and 1170 the identities that they apply to the link. Other FlexE specific 1171 properties, such as the link characteristics of unavailable slot 1172 information, SHOULD also be correlated. Such neighbour discovery and 1173 link property correlation, if provided, MUST be able to operate in 1174 both in-band and out-of-band manner. 1176 6. Architecture 1178 This section discusses the different parts of FlexE signaling and 1179 routing and how these parts interoperte. 1181 FlexE control plane technology SHOULD be able to set up end-to-end 1182 connection in different cases, which may include the management of 1183 FlexE group, assignment of the resource to the FlexE client and so 1184 on. 1186 The FlexE routing mechanism is used to provide resource available 1187 information for set up FlexE connections, like Ethernet PHYs' 1188 information, partial-rate support information. Based on the resource 1189 available information advertised by routing protocol, an end-to-end 1190 FlexE connection is computed, and then the signalling protocol is 1191 used to set up an end-to-end connection. 1193 7. Solution 1195 8. Acknowledgements 1197 9. IANA Considerations 1199 This memo includes no request to IANA. 1201 Note to the RFC Editor: This section should be removed before 1202 publishing. 1204 10. Security Considerations 1206 None. 1208 11. Contributors 1210 Khuzema Pithewan, Infinera Corp, kpithewan@infinera.com 1212 Fatai Zhang, Huawei, zhangfatai@huawei.com 1214 Jie Dong, Huawei, jie.dong@huawei.com 1216 Zongpeng Du, Huawei, duzongpeng@huawei.com 1218 Xian Zhang, Huawei, zhang.xian@huawei.com 1220 James Huang, Huawei, james.huang@huawei.com 1222 Qiwen Zhong, Huawei, zhongqiwen@huawei.com 1224 12. References 1226 12.1. Normative References 1228 [G.709] ITU, "Optical Transport Network Interfaces 1229 (http://www.itu.int/rec/T-REC-G.709-201606-P/en)", July 1230 2016. 1232 [G7044] ITU, "Hitless adjustment of ODUflex(GFP) 1233 (https://www.itu.int/rec/T-REC-G.7044-201110-I/en)", 1234 Cctober 2011. 1236 [G798] ITU, "Characteristics of optical transport network 1237 hierarchy equipment functional blocks 1238 (http://www.itu.int/rec/T-REC-G.798-201212-I/en)", 1239 February 2014. 1241 [OIFFLEXE1] 1242 OIF, "FLex Ethernet Implementation Agreement Version 1.0 1243 (OIF-FLEXE-01.0)", March 2016. 1245 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1246 Requirement Levels", BCP 14, RFC 2119, 1247 DOI 10.17487/RFC2119, March 1997, 1248 . 1250 [RFC3209] Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V., 1251 and G. Swallow, "RSVP-TE: Extensions to RSVP for LSP 1252 Tunnels", RFC 3209, DOI 10.17487/RFC3209, December 2001, 1253 . 1255 [RFC3471] Berger, L., Ed., "Generalized Multi-Protocol Label 1256 Switching (GMPLS) Signaling Functional Description", 1257 RFC 3471, DOI 10.17487/RFC3471, January 2003, 1258 . 1260 [RFC3473] Berger, L., Ed., "Generalized Multi-Protocol Label 1261 Switching (GMPLS) Signaling Resource ReserVation Protocol- 1262 Traffic Engineering (RSVP-TE) Extensions", RFC 3473, 1263 DOI 10.17487/RFC3473, January 2003, 1264 . 1266 [RFC3630] Katz, D., Kompella, K., and D. Yeung, "Traffic Engineering 1267 (TE) Extensions to OSPF Version 2", RFC 3630, 1268 DOI 10.17487/RFC3630, September 2003, 1269 . 1271 [RFC4202] Kompella, K., Ed. and Y. Rekhter, Ed., "Routing Extensions 1272 in Support of Generalized Multi-Protocol Label Switching 1273 (GMPLS)", RFC 4202, DOI 10.17487/RFC4202, October 2005, 1274 . 1276 [RFC4203] Kompella, K., Ed. and Y. Rekhter, Ed., "OSPF Extensions in 1277 Support of Generalized Multi-Protocol Label Switching 1278 (GMPLS)", RFC 4203, DOI 10.17487/RFC4203, October 2005, 1279 . 1281 [RFC4204] Lang, J., Ed., "Link Management Protocol (LMP)", RFC 4204, 1282 DOI 10.17487/RFC4204, October 2005, 1283 . 1285 12.2. Informative References 1287 [OIFMLG3] OIF, "Multi-Lane Gearbox Implementation Agreement Version 1288 3.0 (OIF-MLG-3.0)", April 2016. 1290 [RFC3945] Mannie, E., Ed., "Generalized Multi-Protocol Label 1291 Switching (GMPLS) Architecture", RFC 3945, 1292 DOI 10.17487/RFC3945, October 2004, 1293 . 1295 Appendix A. Additional Stuff 1297 This becomes an Appendix. 1299 Authors' Addresses 1301 Iftekhar Hussain (editor) 1302 Infinera Corp 1303 169 Java Drive 1304 Sunnyvale, CA 94089 1305 USA 1307 Email: IHussain@infinera.com 1309 Radha Valiveti 1310 Infinera Corp 1311 169 Java Drive 1312 Sunnyvale, CA 94089 1313 USA 1315 Email: rvaliveti@infinera.com 1317 Qilei Wang (editor) 1318 ZTE 1319 Nanjing 1320 CN 1322 Email: wang.qilei@zte.com.cn 1324 Loa Andersson (editor) 1325 Huawei 1326 Stockholm 1327 Sweden 1329 Email: loa@pi.nu 1331 Mach Chen 1332 Huawei 1333 CN 1335 Email: mach.chen@huawei.com 1337 Haomian Zheng 1338 Huawei 1339 CN 1341 Email: zhenghaomian@huawei.com