idnits 2.17.1 draft-ietf-spring-segment-routing-msdc-03.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (March 3, 2017) is 2583 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- -- Looks like a reference, but probably isn't: '16000' on line 273 -- Looks like a reference, but probably isn't: '23999' on line 273 -- Looks like a reference, but probably isn't: '1000' on line 860 -- Looks like a reference, but probably isn't: '1999' on line 860 -- Looks like a reference, but probably isn't: '2000' on line 861 -- Looks like a reference, but probably isn't: '2999' on line 861 == Outdated reference: A later version (-27) exists of draft-ietf-idr-bgp-prefix-sid-04 == Outdated reference: A later version (-15) exists of draft-ietf-spring-segment-routing-11 ** Obsolete normative reference: RFC 3107 (Obsoleted by RFC 8277) == Outdated reference: A later version (-26) exists of draft-ietf-6man-segment-routing-header-05 == Outdated reference: A later version (-10) exists of draft-ietf-spring-segment-routing-central-epe-04 Summary: 1 error (**), 0 flaws (~~), 5 warnings (==), 7 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group C. Filsfils, Ed. 3 Internet-Draft S. Previdi, Ed. 4 Intended status: Informational Cisco Systems, Inc. 5 Expires: September 4, 2017 J. Mitchell 6 Unaffiliated 7 E. Aries 8 Juniper Networks 9 P. Lapukhov 10 Facebook 11 March 3, 2017 13 BGP-Prefix Segment in large-scale data centers 14 draft-ietf-spring-segment-routing-msdc-03 16 Abstract 18 This document describes the motivation and benefits for applying 19 segment routing in BGP-based large-scale data-centers. It describes 20 the design to deploy segment routing in those data-centers, for both 21 the MPLS and IPv6 dataplanes. 23 Requirements Language 25 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 26 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 27 document are to be interpreted as described in RFC 2119 [RFC2119]. 29 Status of This Memo 31 This Internet-Draft is submitted in full conformance with the 32 provisions of BCP 78 and BCP 79. 34 Internet-Drafts are working documents of the Internet Engineering 35 Task Force (IETF). Note that other groups may also distribute 36 working documents as Internet-Drafts. The list of current Internet- 37 Drafts is at http://datatracker.ietf.org/drafts/current/. 39 Internet-Drafts are draft documents valid for a maximum of six months 40 and may be updated, replaced, or obsoleted by other documents at any 41 time. It is inappropriate to use Internet-Drafts as reference 42 material or to cite them other than as "work in progress." 44 This Internet-Draft will expire on September 4, 2017. 46 Copyright Notice 48 Copyright (c) 2017 IETF Trust and the persons identified as the 49 document authors. All rights reserved. 51 This document is subject to BCP 78 and the IETF Trust's Legal 52 Provisions Relating to IETF Documents 53 (http://trustee.ietf.org/license-info) in effect on the date of 54 publication of this document. Please review these documents 55 carefully, as they describe your rights and restrictions with respect 56 to this document. Code Components extracted from this document must 57 include Simplified BSD License text as described in Section 4.e of 58 the Trust Legal Provisions and are provided without warranty as 59 described in the Simplified BSD License. 61 Table of Contents 63 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 64 2. Large Scale Data Center Network Design Summary . . . . . . . 3 65 2.1. Reference design . . . . . . . . . . . . . . . . . . . . 4 66 3. Some open problems in large data-center networks . . . . . . 5 67 4. Applying Segment Routing in the DC with MPLS dataplane . . . 6 68 4.1. BGP Prefix Segment (BGP-Prefix-SID) . . . . . . . . . . . 6 69 4.2. eBGP Labeled Unicast (RFC3107) . . . . . . . . . . . . . 7 70 4.2.1. Control Plane . . . . . . . . . . . . . . . . . . . . 7 71 4.2.2. Data Plane . . . . . . . . . . . . . . . . . . . . . 9 72 4.2.3. Network Design Variation . . . . . . . . . . . . . . 10 73 4.2.4. Global BGP Prefix Segment through the fabric . . . . 10 74 4.2.5. Incremental Deployments . . . . . . . . . . . . . . . 11 75 4.3. iBGP Labeled Unicast (RFC3107) . . . . . . . . . . . . . 12 76 5. Applying Segment Routing in the DC with IPv6 dataplane . . . 12 77 6. Communicating path information to the host . . . . . . . . . 13 78 7. Addressing the open problems . . . . . . . . . . . . . . . . 14 79 7.1. Per-packet and flowlet switching . . . . . . . . . . . . 14 80 7.2. Performance-aware routing . . . . . . . . . . . . . . . . 15 81 7.3. Deterministic network probing . . . . . . . . . . . . . . 16 82 8. Additional Benefits . . . . . . . . . . . . . . . . . . . . . 16 83 8.1. MPLS Dataplane with operational simplicity . . . . . . . 16 84 8.2. Minimizing the FIB table . . . . . . . . . . . . . . . . 17 85 8.3. Egress Peer Engineering . . . . . . . . . . . . . . . . . 17 86 8.4. Anycast . . . . . . . . . . . . . . . . . . . . . . . . . 18 87 9. Preferred SRGB Allocation . . . . . . . . . . . . . . . . . . 18 88 10. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 19 89 11. Manageability Considerations . . . . . . . . . . . . . . . . 19 90 12. Security Considerations . . . . . . . . . . . . . . . . . . . 19 91 13. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 20 92 14. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 20 93 15. References . . . . . . . . . . . . . . . . . . . . . . . . . 21 94 15.1. Normative References . . . . . . . . . . . . . . . . . . 21 95 15.2. Informative References . . . . . . . . . . . . . . . . . 21 96 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 22 98 1. Introduction 100 Segment Routing (SR), as described in 101 [I-D.ietf-spring-segment-routing] leverages the source routing 102 paradigm. A node steers a packet through an ordered list of 103 instructions, called segments. A segment can represent any 104 instruction, topological or service-based. A segment can have a 105 local semantic to an SR node or global within an SR domain. SR 106 allows to enforce a flow through any topological path and service 107 chain while maintaining per-flow state only at the ingress node to 108 the SR domain. Segment Routing can be applied to the MPLS and IPv6 109 data-planes. 111 The use-cases described in this document should be considered in the 112 context of the BGP-based large-scale data-center (DC) design 113 described in [RFC7938]. We extend it by applying SR both with IPv6 114 and MPLS dataplane. 116 2. Large Scale Data Center Network Design Summary 118 This section provides a brief summary of the informational document 119 [RFC7938] that outlines a practical network design suitable for data- 120 centers of various scales: 122 o Data-center networks have highly symmetric topologies with 123 multiple parallel paths between two server attachment points. The 124 well-known Clos topology is most popular among the operators. In 125 a Clos topology, the minimum number of parallel paths between two 126 elements is determined by the "width" of the "Tier-1" stage. See 127 Figure 1 below for an illustration of the concept. 129 o Large-scale data-centers commonly use a routing protocol, such as 130 BGP4 [RFC4271] in order to provide endpoint connectivity. 131 Recovery after a network failure is therefore driven either by 132 local knowledge of directly available backup paths or by 133 distributed signaling between the network devices. 135 o Within data-center networks, traffic is load-shared using the 136 Equal Cost Multipath (ECMP) mechanism. With ECMP, every network 137 device implements a pseudo-random decision, mapping packets to one 138 of the parallel paths by means of a hash function calculated over 139 certain parts of the packet, typically a combination of various 140 packet header fields. 142 The following is a schematic of a five-stage Clos topology, with four 143 devices in the "Tier-1" stage. Notice that number of paths between 144 Node1 and Node12 equals to four: the paths have to cross all of 145 Tier-1 devices. At the same time, the number of paths between Node1 146 and Node2 equals two, and the paths only cross Tier-2 devices. Other 147 topologies are possible, but for simplicity we'll only look into the 148 topologies that have a single path from Tier-1 to Tier-3. The rest 149 could be treated similarly, with a few modifications to the logic. 151 2.1. Reference design 153 Tier-1 154 +-----+ 155 |NODE | 156 +->| 5 |--+ 157 | +-----+ | 158 Tier-2 | | Tier-2 159 +-----+ | +-----+ | +-----+ 160 +------------>|NODE |--+->|NODE |--+--|NODE |-------------+ 161 | +-----| 3 |--+ | 6 | +--| 9 |-----+ | 162 | | +-----+ +-----+ +-----+ | | 163 | | | | 164 | | +-----+ +-----+ +-----+ | | 165 | +-----+---->|NODE |--+ |NODE | +--|NODE |-----+-----+ | 166 | | | +---| 4 |--+->| 7 |--+--| 10 |---+ | | | 167 | | | | +-----+ | +-----+ | +-----+ | | | | 168 | | | | | | | | | | 169 +-----+ +-----+ | +-----+ | +-----+ +-----+ 170 |NODE | |NODE | Tier-3 +->|NODE |--+ Tier-3 |NODE | |NODE | 171 | 1 | | 2 | | 8 | | 11 | | 12 | 172 +-----+ +-----+ +-----+ +-----+ +-----+ 173 | | | | | | | | 174 A O B O <- Servers -> Z O O O 176 Figure 1: 5-stage Clos topology 178 In the reference topology illustrated in Figure 1, we assume: 180 o Each node is its own AS (Node X has AS X) 182 * For simple and efficient route propagation filtering, Node5, 183 Node6, Node7 and Node8 use the same AS, Node3 and Node4 use the 184 same AS, Node9 and Node10 use the same AS. 186 * For efficient usage of the scarce 2-byte Private Use AS pool, 187 different Tier-3 nodes might use the same AS. 189 * Without loss of generality, we will simplify these details in 190 this document and assume that each node has its own AS. 192 o Each node peers with its neighbors with a BGP session. If not 193 specified, eBGP is assumed. In a specific use-case, iBGP will be 194 used but this will be called out explicitly in that case. 196 o Each node originates the IPv4 address of its loopback interface 197 into BGP and announces it to its neighbors. 199 * The loopback of Node X is 192.0.2.x/32. 201 In this document, we also refer to the Tier-1, Tier-2 and Tier-3 202 nodes respectively as Spine, Leaf and ToR (top of rack) nodes. When 203 a ToR node acts as a gateway to the "outside world", we call it a 204 border node. 206 3. Some open problems in large data-center networks 208 The data-center network design summarized above provides means for 209 moving traffic between hosts with reasonable efficiency. There are 210 few open performance and reliability problems that arise in such 211 design: 213 o ECMP routing is most commonly realized per-flow. This means that 214 large, long-lived "elephant" flows may affect performance of 215 smaller, short-lived "mouse" flows and reduce efficiency of per- 216 flow load-sharing. In other words, per-flow ECMP does not perform 217 efficiently when flow lifetime distribution is heavy-tailed. 218 Furthermore, due to hash-function inefficiencies it is possible to 219 have frequent flow collisions, where more flows get placed on one 220 path over the others. 222 o Shortest-path routing with ECMP implements an oblivious routing 223 model, which is not aware of the network imbalances. If the 224 network symmetry is broken, for example due to link failures, 225 utilization hotspots may appear. For example, if a link fails 226 between Tier-1 and Tier-2 devices (e.g. Node5 and Node9), Tier-3 227 devices Node1 and Node2 will not be aware of that, since there are 228 other paths available from perspective of Node3. They will 229 continue sending roughly equal traffic to Node3 and Node4 as if 230 the failure didn't exist which may cause a traffic hotspot. 232 o The absence of path visibility leaves transport protocols, such as 233 TCP, with a "blackbox" view of the network. Some TCP metrics, 234 such as SRTT, MSS, CWND and few others could be inferred and 235 cached based on past history, but those apply to destinations, 236 regardless of the path that has been chosen to get there. Thus, 237 for instance, TCP is not capable of remembering "bad" paths, such 238 as those that exhibited poor performance in the past. This means 239 that every new connection will be established obliviously (memory- 240 less) with regards to the paths chosen before, or chosen by other 241 nodes. 243 o Isolating faults in the network with multiple parallel paths and 244 ECMP-based routing is non-trivial due to lack of determinism. 245 Specifically, the connections from HostA to HostB may take a 246 different path every time a new connection is formed, thus making 247 consistent reproduction of a failure much more difficult. This 248 complexity scales linearly with the number of parallel paths in 249 the network, and stems from the random nature of path selection by 250 the network devices. 252 Further in this document, we are going to demonstrate how these 253 problems could be addressed within the framework of Segment Routing. 255 First, we will explain how to apply SR in the DC, for MPLS and IPv6 256 data-planes. 258 4. Applying Segment Routing in the DC with MPLS dataplane 260 4.1. BGP Prefix Segment (BGP-Prefix-SID) 262 A BGP Prefix Segment is a segment associated with a BGP prefix. A 263 BGP Prefix Segment is a network-wide instruction to forward the 264 packet along the ECMP-aware best path to the related prefix. 266 The BGP Prefix Segment is defined as the BGP-Prefix-SID Attribute in 267 [I-D.ietf-idr-bgp-prefix-sid] which contains an index. Throughout 268 this document the BGP Prefix Segment Attribute is referred as the 269 BGP-Prefix-SID and the encoded index as the label-index. 271 In this document, we make the network design decision to assume that 272 all the nodes are allocated the same SRGB (Segment Routing Global 273 Block), e.g. [16000, 23999] This is important to fulfill the 274 recommendation for operational simplification as explained in 275 [I-D.ietf-spring-segment-routing]. 277 Note well that the use of a common SRGB in all nodes is not a 278 requirement, one could use a different SRGB at every node. However, 279 this would make the operation of the DC fabric more complex as the 280 label allocated to the loopback of a remote node is then different at 281 every node. This also may increase the complexity of the centralized 282 controller. More on the SRGB allocation scheme is described in 283 Section 9. 285 For illustration purpose, when considering an MPLS data-plane, we 286 assume that the label-index allocated to prefix 192.0.2.x/32 is X. 287 As a result, a local label 1600x is allocated for prefix 192.0.2.x/32 288 by each node throughout the DC fabric. 290 When IPv6 data-plane is considered, we assume that Node X is 291 allocated IPv6 address (segment) 2001:DB8::X. 293 4.2. eBGP Labeled Unicast (RFC3107) 295 Referring to Figure 1 and [RFC7938], the following design 296 modifications are introduced: 298 o Each node peers with its neighbors via eBGP3107 session 300 o The forwarding plane at Tier-2 and Tier-1 is MPLS. 302 o The forwarding plane at Tier-3 is either IP2MPLS (if the host 303 sends IP traffic) or MPLS2MPLS (if the host sends MPLS- 304 encapsulated traffic). 306 Figure 2 zooms into a path from server A to server Z within the 307 topology of Figure 1. 309 +-----+ +-----+ +-----+ 310 +---------->|NODE | |NODE | |NODE | 311 | | 4 |--+->| 7 |--+--| 10 |---+ 312 | +-----+ +-----+ +-----+ | 313 | | 314 +-----+ +-----+ 315 |NODE | |NODE | 316 | 1 | | 11 | 317 +-----+ +-----+ 318 | | 319 A <- Servers -> Z 321 Figure 2: Path from A to Z via nodes 1, 4, 7, 10 and 11 323 Referring to Figure 1 and Figure 2 and assuming the IP address, AS 324 and label-index allocation previously described, the following 325 sections detail the control plane operation and the data plane states 326 for the prefix 192.0.2.11/32 (loopback of Node11) 328 4.2.1. Control Plane 330 Node11 originates 192.0.2.11/32 in BGP and allocates to it a BGP- 331 Prefix-SID with label-index: index11) [I-D.ietf-idr-bgp-prefix-sid]. 333 Node11 sends the following eBGP3107 update to Node10: 335 . IP Prefix: 192.0.2.11/32 336 . Label: Implicit-Null 337 . Next-hop: Node11's interface address on the link to Node10 338 . AS Path: {11} 339 . BGP-Prefix-SID: Label-Index 11 341 Node10 receives the above update. As it is SR capable, Node10 is 342 able to interpret the BGP-Prefix-SID and hence understands that it 343 should allocate the label from its own SRGB block, offset by the 344 Label-Index received in the BGP-Prefix-SID (16000+11 hence 16011) to 345 the NLRI instead of allocating a non-deterministic label out of a 346 dynamically allocated portion of the local label space. The 347 implicit-null label in the NLRI tells Node10 that it is the 348 penultimate hop and MUST pop the top label on the stack before 349 forwarding traffic for this prefix to Node11. 351 Then, Node10 sends the following eBGP3107 update to Node7: 353 . IP Prefix: 192.0.2.11/32 354 . Label: 16011 355 . Next-hop: Node10's interface address on the link to Node7 356 . AS Path: {10, 11} 357 . BGP-Prefix-SID: Label-Index 11 359 Node7 receives the above update. As it is SR capable, Node7 is able 360 to interpret the BGP-Prefix-SID and hence allocates the local 361 (incoming) label 16011 (16000 + 11) to the NLRI (instead of 362 allocating a "dynamic" local label from its label manager). Node7 363 uses the label in the received eBGP3107 NLRI as the outgoing label 364 (the index is only used to derive the local/incoming label). 366 Node7 sends the following eBGP3107 update to Node4: 368 . IP Prefix: 192.0.2.11/32 369 . Label: 16011 370 . Next-hop: Node7's interface address on the link to Node4 371 . AS Path: {7, 10, 11} 372 . BGP-Prefix-SID: Label-Index 11 374 Node4 receives the above update. As it is SR capable, Node4 is able 375 to interpret the BGP-Prefix-SID and hence allocates the local 376 (incoming) label 16011 to the NLRI (instead of allocating a "dynamic" 377 local label from its label manager). Node4 uses the label in the 378 received eBGP3107 NLRI as outgoing label (the index is only used to 379 derive the local/incoming label). 381 Node4 sends the following eBGP3107 update to Node1: 383 . IP Prefix: 192.0.2.11/32 384 . Label: 16011 385 . Next-hop: Node4's interface address on the link to Node1 386 . AS Path: {4, 7, 10, 11} 387 . BGP-Prefix-SID: Label-Index 11 389 Node1 receives the above update. As it is SR capable, Node1 is able 390 to interpret the BGP-Prefix-SID and hence allocates the local 391 (incoming) label 16011 to the NLRI (instead of allocating a "dynamic" 392 local label from its label manager). Node1 uses the label in the 393 received eBGP3107 NLRI as outgoing label (the index is only used to 394 derive the local/incoming label). 396 4.2.2. Data Plane 398 Referring to Figure 1, and assuming all nodes apply the same 399 advertisement rules described above and all nodes have the same SRGB 400 (16000-23999), here are the IP/MPLS forwarding tables for prefix 401 192.0.2.11/32 at Node1, Node4, Node7 and Node10. 403 ----------------------------------------------- 404 Incoming label | outgoing label | Outgoing 405 or IP destination | | Interface 406 ------------------+----------------+----------- 407 16011 | 16011 | ECMP{3, 4} 408 192.0.2.11/32 | 16011 | ECMP{3, 4} 409 ------------------+----------------+----------- 411 Figure 3: Node1 Forwarding Table 413 ----------------------------------------------- 414 Incoming label | outgoing label | Outgoing 415 or IP destination | | Interface 416 ------------------+----------------+----------- 417 16011 | 16011 | ECMP{7, 8} 418 192.0.2.11/32 | 16011 | ECMP{7, 8} 419 ------------------+----------------+----------- 421 Figure 4: Node4 Forwarding Table 423 ----------------------------------------------- 424 Incoming label | outgoing label | Outgoing 425 or IP destination | | Interface 426 ------------------+----------------+----------- 427 16011 | 16011 | 10 428 192.0.2.11/32 | 16011 | 10 429 ------------------+----------------+----------- 431 Figure 5: Node7 Forwarding Table 433 ----------------------------------------------- 434 Incoming label | outgoing label | Outgoing 435 or IP destination | | Interface 436 ------------------+----------------+----------- 437 16011 | POP | 11 438 192.0.2.11/32 | N/A | 11 439 ------------------+----------------+----------- 441 Node10 Forwarding Table 443 4.2.3. Network Design Variation 445 A network design choice could consist of switching all the traffic 446 through Tier-1 and Tier-2 as MPLS traffic. In this case, one could 447 filter away the IP entries at Node4, Node7 and Node10. This might be 448 beneficial in order to optimize the forwarding table size. 450 A network design choice could consist in allowing the hosts to send 451 MPLS-encapsulated traffic based on the Egress Peer Engineering (EPE) 452 use-case as defined in [I-D.ietf-spring-segment-routing-central-epe]. 453 For example, applications at HostA would send their Z-destined 454 traffic to Node1 with an MPLS label stack where the top label is 455 16011 and the next label is an EPE peer segment 456 ([I-D.ietf-spring-segment-routing-central-epe]) at Node11 directing 457 the traffic to Z. 459 4.2.4. Global BGP Prefix Segment through the fabric 461 When the previous design is deployed, the operator enjoys global BGP- 462 Prefix-SID and label allocation throughout the DC fabric. 464 A few examples follow: 466 o Normal forwarding to Node11: a packet with top label 16011 467 received by any node in the fabric will be forwarded along the 468 ECMP-aware BGP best-path towards Node11 and the label 16011 is 469 penultimate-popped at Node10. 471 o Traffic-engineered path to Node11: an application on a host behind 472 Node1 might want to restrict its traffic to paths via the Spine 473 node Node5. The application achieves this by sending its packets 474 with a label stack of {16005, 16011}. BGP Prefix SID 16005 directs 475 the packet up to Node5 along the path (Node1, Node3, Node5). BGP- 476 Prefix-SID 16011 then directs the packet down to Node11 along the 477 path (Node5, Node9, Node11). 479 4.2.5. Incremental Deployments 481 The design previously described can be deployed incrementally. Let 482 us assume that Node7 does not support the BGP-Prefix-SID and let us 483 show how the fabric connectivity is preserved. 485 From a signaling viewpoint, nothing would change: even though Node7 486 does not support the BGP-Prefix-SID, it does propagate the attribute 487 unmodified to its neighbors. 489 From a label allocation viewpoint, the only difference is that Node7 490 would allocate a dynamic (random) label to the prefix 192.0.2.11/32 491 (e.g. 123456) instead of the "hinted" label as instructed by the BGP- 492 Prefix-SID. The neighbors of Node7 adapt automatically as they 493 always use the label in the BGP3107 NLRI as outgoing label. 495 Node4 does understand the BGP-Prefix-SID and hence allocates the 496 indexed label in the SRGB (16011) for 192.0.2.11/32. 498 As a result, all the data-plane entries across the network would be 499 unchanged except the entries at Node7 and its neighbor Node4 as shown 500 in the figures below. 502 The key point is that the end-to-end Label Switched Path (LSP) is 503 preserved because the outgoing label is always derived from the 504 received label within the BGP3107 NLRI. The index in the BGP-Prefix- 505 SID is only used as a hint on how to allocate the local label (the 506 incoming label) but never for the outgoing label. 508 ------------------------------------------ 509 Incoming label | outgoing | Outgoing 510 or IP destination | label | Interface 511 -------------------+---------------------- 512 12345 | 16011 | 10 514 Figure 7: Node7 Forwarding Table 516 ------------------------------------------ 517 Incoming label | outgoing | Outgoing 518 or IP destination | label | Interface 519 -------------------+---------------------- 520 16011 | 12345 | 7 522 Figure 8: Node4 Forwarding Table 524 The BGP-Prefix-SID can thus be deployed incrementally one node at a 525 time. 527 When deployed together with a homogeneous SRGB (same SRGB across the 528 fabric), the operator incrementally enjoys the global prefix segment 529 benefits as the deployment progresses through the fabric. 531 4.3. iBGP Labeled Unicast (RFC3107) 533 The same exact design as eBGP3107 is used with the following 534 modifications: 536 All nodes use the same AS number. 538 iBGP3107 reflection with next-hop-self is used instead of 539 eBGP3107. 541 For simple and efficient route propagation filtering, Node5, 542 Node6, Node7 and Node8 use the same Cluster ID, Node3 and Node4 543 use the same Cluster ID, Node9 and Node10 use the same Cluster ID. 545 AIGP metric ([RFC7311]) is likely applied to the BGP-Prefix-SID as 546 part of a large-scale multi-domain design such as Seamless MPLS 547 [I-D.ietf-mpls-seamless-mpls]. 549 The control-plane behavior is mostly the same as described in the 550 previous section: the only difference is that the eBGP3107 path 551 propagation is simply replaced by an iBGP3107 path reflection with 552 next-hop changed to self. 554 The data-plane tables are exactly the same. 556 5. Applying Segment Routing in the DC with IPv6 dataplane 558 The design described in [RFC7938] is reused with one single 559 modification. We highlight it using the example of the reachability 560 to Node11 via spine node Node5. 562 Node5 originates 2001:DB8::5/128 with the attached BGP-Prefix-SID 563 advertising the support of the Segment Routing extension header (SRH, 565 [I-D.ietf-6man-segment-routing-header]) for IPv6 packets destined to 566 segment 2001:DB8::5 ([I-D.ietf-idr-bgp-prefix-sid]). 568 Tor11 originates 2001:DB8::11/128 with the attached BGP-Prefix-SID 569 advertising the support of the SRH for IPv6 packets destined to 570 segment 2001:DB8::11. 572 The control-plane and data-plane processing of all the other nodes in 573 the fabric is unchanged. Specifically, the routes to 2001:DB8::5 and 574 2001:DB8::11 are installed in the FIB along the eBGP best-path to 575 Node5 (spine node) and Node11 (ToR node) respectively. 577 An application on HostA which needs to send traffic to HostZ via only 578 Node5 (spine node) can do so by sending IPv6 packets with a SRH 579 extension header. The destination address and active segment is set 580 to 2001:DB8::5. The next and last segment is set to 2001:DB8::11. 582 The application must only use IPv6 addresses that have been 583 advertised as capable for SRv6 segment processing (e.g. for which the 584 BGP prefix segment capability has been advertised). How applications 585 learn this (e.g.: centralized controller and orchestration) is 586 outside the scope of this document. 588 6. Communicating path information to the host 590 There are two general methods for communicating path information to 591 the end-hosts: "proactive" and "reactive", aka "push" and "pull" 592 models. There are multiple ways to implement either of these 593 methods. Here, we note that one way could be using a centralized 594 controller: the controller either tells the hosts of the prefix-to- 595 path mappings beforehand and updates them as needed (network event 596 driven push), or responds to the hosts making request for a path to 597 specific destination (host event driven pull). It is also possible 598 to use a hybrid model, i.e., pushing some state from the controller 599 in response to particular network events, while the host pulls other 600 state on demand. 602 We note, that when disseminating network-related data to the end- 603 hosts a trade-off is made to balance the amount of information Vs. 604 the level of visibility in the network state. This applies both to 605 push and pull models. In the extreme case, the host would request 606 path information on every flow, and keep no local state at all. On 607 the other end of the spectrum, information for every prefix in the 608 network along with available paths could be pushed and continuously 609 updated on all hosts. 611 7. Addressing the open problems 613 This section demonstrates how the problems describe above could be 614 solved using the segment routing concept. It is worth noting that 615 segment routing signaling and data-plane are only parts of the 616 solution. Additional enhancements, e.g., such as the centralized 617 controller mentioned previously, and host networking stack support 618 are required to implement the proposed solutions. 620 7.1. Per-packet and flowlet switching 622 With the ability to choose paths on the host, one may go from per- 623 flow load-sharing in the network to per-packet or per-flowlet (see 624 [KANDULA04] for information on flowlets). The host may select 625 different segment routing instructions either per packet, or per 626 flowlet, and route them over different paths. This allows for 627 solving the "elephant flow" problem in the data-center and avoiding 628 link imbalances. 630 Note that traditional ECMP routing could be easily simulated with on- 631 host path selection, using method proposed in VL2 (see 632 [GREENBERG09]). The hosts would randomly pick a Tier-2 or Tier-1 633 device to "bounce" the packet off of, depending on whether the 634 destination is under the same Tier-2 nodes, or has to be reached 635 across Tier-1. The host would use a hash function that operates on 636 per-flow invariants, to simulate per-flow load-sharing in the 637 network. 639 Using Figure 1 as reference, let us illustrate this concept assuming 640 that HostA has an elephant flow to HostZ called Flow-f. 642 Normally, a flow is hashed on to a single path. Let's assume HostA 643 sends its packets associated with Flow-f with top label 16011 (the 644 label for the remote ToR, Node11, where HostZ is connected) and Node1 645 would hash all the packets of Flow-F via the same nhop (e.g. Node3). 646 Similarly, let's assume that leaf Node3 would hash all the packets of 647 Flow-F via the same next-hop (e.g.: spine node Node1). This normal 648 operation would restrict the elephant flow on a small subset of the 649 ECMP paths to HostZ and potentially create imbalance and congestion 650 in the fabric. 652 Leveraging the flowlet proposal, assuming HostA is made aware of 4 653 disjoint paths via intermediate segment 16005, 16006, 16007 and 16008 654 (the BGP prefix SID's of the 4 spine nodes) and also made aware of 655 the prefix segment of the remote ToR connected to the destination 656 (16011), then the application can break the elephant flow F into 657 flowlets F1, F2, F3, F4 and associate each flowlet with one of the 658 following 4 label stacks: {16005, 16011}, {16006, 16011}, {16007, 659 16011} and {16008, 16011}. This would spread the load of the elephant 660 flow through all the ECMP paths available in the fabric and re- 661 balance the load. 663 7.2. Performance-aware routing 665 Knowing the path associated with flows/packets, the end host may 666 deduce certain characteristics of the path on its own, and 667 additionally use the information supplied with path information 668 pushed from the controller or received via pull request. The host 669 may further share its path observations with the centralized agent, 670 so that the latter may keep up-to-date network health map to assist 671 other hosts with this information. 673 For example, an application A.1 at HostA may pin a TCP flow destined 674 to HostZ via Spine node Node5 using label stack {16005, 16011}. The 675 application A.1 may collect information on packet loss, deduced from 676 TCP retransmissions and other signals (e.g. RTT increases). A.1 may 677 additionally publish this information to a centralized agent, e.g. 678 after a flow completes, or periodically for longer lived flows. 679 Next, using both local and/or global performance data, application 680 A.1 as well as other applications sharing the same resources in the 681 DC fabric may pick up the best path for the new flow, or update an 682 existing path (e.g.: when informed of congestion on an existing 683 path). 685 One particularly interesting instance of performance-aware routing is 686 dynamic fault-avoidance. If some links or devices in the network 687 start discarding packets due to a fault, the end-hosts could detect 688 the path(s) being affected and steer their flows away from the 689 problem spot. Similar logic applies to failure cases where packets 690 get completely black-holed, e.g. when a link goes down. 692 For example, an application A.1 informed about 5 paths to Z {16005, 693 16011}, {16006, 16011}, {16007, 16011}, {16008, 16011} and {16011} 694 might use the last one by default (for simplicity). When performance 695 is degrading, A.1 might then start to pin TCP flows to each of the 4 696 other paths (each via a distinct spine) and monitor the performance. 697 It would then detect the faulty path and assign a negative preference 698 to the faulty path to avoid further flows using it. Gradually, over 699 time, it may re-assign flows on the faulty path to eventually detect 700 the resolution of the trouble and start reusing the path. 702 By leveraging Segment Routing, one avoids issues associated with 703 oblivious ECMP hashing. For example, if in the topology depicted on 704 Figure 1 a link between spine node Node5 and leaf node Node9 fails, 705 HostA may exclude the segment corresponding to Node5 from the prefix 706 matching the servers under Tier-2 devices Node9. In the push path 707 discovery model, the affected path mappings may be explicitly pushed 708 to all the servers for the duration of the failure. The new mapping 709 would instruct them to avoid the particular Tier-1 node until the 710 link has recovered. Alternatively, in pull path, the centralized 711 controller may start steering new flows immediately after it 712 discovers the issue. Until then, the existing flows may recover 713 using local detection of the path issues, as described in 714 Section 7.2. 716 7.3. Deterministic network probing 718 Active probing is a well-known technique for monitoring network 719 elements' health, constituting of sending continuous packet streams 720 simulating network traffic to the hosts in the data-center. Segment 721 routing makes possible to prescribe the exact paths that each probe 722 or series of probes would be taking toward their destination. This 723 allows for fast correlation and detection of failed paths, by 724 processing information from multiple actively probing agents. This 725 complements the data collected from the hosts routing stacks as 726 described in Section 7.2. 728 For example, imagine a probe agent sending packets to all machines in 729 the data-center. For every host, it may send packets over each of 730 the possible paths, knowing exactly which links and devices these 731 packets will be crossing. Correlating results for multiple 732 destinations with the topological data, it may automatically isolate 733 possible problem to a link or device in the network. 735 8. Additional Benefits 737 8.1. MPLS Dataplane with operational simplicity 739 As required by [RFC7938], no new signaling protocol is introduced. 740 The BGP-Prefix-SID is a lightweight extension to BGP Labeled Unicast 741 (RFC3107 [RFC3107]). It applies either to eBGP or iBGP based 742 designs. 744 Specifically, LDP and RSVP-TE are not used. These protocols would 745 drastically impact the operational complexity of the Data Center and 746 would not scale. This is in line with the requirements expressed in 747 [RFC7938]. 749 A key element of the operational simplicity is the deployment of the 750 design with a single and consistent SRGB across the DC fabric. 752 At every node in the fabric, the same label is associated to a given 753 BGP-Prefix-SID and hence a notion of global prefix segment arises. 755 When a controller programs HostA to send traffic to HostZ via the 756 normally available BGP ECMP paths, the controller uses label 16011 757 associated with the ToR node connected to the HostZ. The controller 758 does not need to pick the label based on the ToR that the source host 759 is connected to. 761 In a classic BGP Labeled Unicast design applied to the DC fabric 762 illustrated in Figure 1, the ToR Node1 connected to HostA would most 763 likely allocate a different label for 192.0.2.11/32 than the one 764 allocated by ToR Node2. As a consequence, the controller would need 765 to adapt the SR policy to each host, based on the ToR node that they 766 are connected to. This adds state maintenance and synchronization 767 problems. All of this unnecessary complexity is eliminated if a 768 single consistent SRGB is utilized across the fabric. 770 8.2. Minimizing the FIB table 772 The designer may decide to switch all the traffic at Tier-1 and Tier- 773 2's based on MPLS, hence drastically decreasing the IP table size at 774 these nodes. 776 This is easily accomplished by encapsulating the traffic either 777 directly at the host or at the source ToR node by pushing the BGP- 778 Prefix-SID of the destination ToR for intra-DC traffic or border node 779 for inter-DC or DC-to-outside-world traffic. 781 8.3. Egress Peer Engineering 783 It is straightforward to combine the design illustrated in this 784 document with the Egress Peer Engineering (EPE) use-case described in 785 [I-D.ietf-spring-segment-routing-central-epe]. 787 In such case, the operator is able to engineer its outbound traffic 788 on a per host-flow basis, without incurring any additional state at 789 intermediate points in the DC fabric. 791 For example, the controller only needs to inject a per-flow state on 792 the HostA to force it to send its traffic destined to a specific 793 Internet destination D via a selected border node (say Node12 in 794 Figure 1 instead of another border node, Node11) and a specific 795 egress peer of Node12 (say peer AS 9999 of local PeerNode segment 796 9999 at Node12 instead of any other peer which provides a path to the 797 destination D). Any packet matching this state at host A would be 798 encapsulated with SR segment list (label stack) {16012, 9999}. 16012 799 would steer the flow through the DC fabric, leveraging any ECMP, 800 along the best path to border node Node12. Once the flow gets to 801 border node Node12, the active segment is 9999 (because of PHP on the 802 upstream neighbor of Node12). This EPE PeerNode segment forces 803 border node Node12 to forward the packet to peer AS 9999, without any 804 IP lookup at the border node. There is no per-flow state for this 805 engineered flow in the DC fabric. A benefit of segment routing is 806 the per-flow state is only required at the source. 808 As well as allowing full traffic engineering control such a design 809 also offers FIB table minimization benefits as the Internet-scale FIB 810 at border node Node12 is not required if all FIB lookups are avoided 811 there by using EPE. 813 8.4. Anycast 815 The design presented in this document preserves the availability and 816 load-balancing properties of the base design presented in 817 [I-D.ietf-spring-segment-routing]. 819 For example, one could assign an anycast loopback 192.0.2.20/32 and 820 associate segment index 20 to it on the border Node11 and Node12 (in 821 addition to their node-specific loopbacks). Doing so, the EPE 822 controller could express a default "go-to-the-Internet via any border 823 node" policy as segment list {16020}. Indeed, from any host in the DC 824 fabric or from any ToR node, 16020 steers the packet towards the 825 border Node11 or Node12 leveraging ECMP where available along the 826 best paths to these nodes. 828 9. Preferred SRGB Allocation 830 In the MPLS case, we recommend to use same SRGBs at each node. 832 Different SRGBs in each node likely increase the complexity of the 833 solution both from an operational viewpoint and from a controller 834 viewpoint. 836 From an operation viewpoint, it is much simpler to have the same 837 global label at every node for the same destination (the MPLS 838 troubleshooting is then similar to the IPv6 troubleshooting where 839 this global property is a given). 841 From a controller viewpoint, this allows us to construct simple 842 policies applicable across the fabric. 844 Let us consider two applications A and B respectively connected to 845 ToR1 and ToR2. A has two flows FA1 and FA2 destined to Z. B has two 846 flows FB1 and FB2 destined to Z. The controller wants FA1 and FB1 to 847 be load-shared across the fabric while FA2 and FB2 must be 848 respectively steered via Node5 and Node8. 850 Assuming a consistent unique SRGB across the fabric as described in 851 the document, the controller can simply do it by instructing A and B 852 to use {16011} respectively for FA1 and FB1 and by instructing A and 853 B to use {16005 16011} and {16008 16011} respectively for FA2 and 854 FB2. 856 Let us assume a design where the SRGB is different at every node and 857 where the SRGB of each node is advertised using the Originator SRGB 858 TLV of the BGP-Prefix-SID as defined in 859 [I-D.ietf-idr-bgp-prefix-sid]: SRGB of Node K starts at value K*1000 860 and the SRGB length is 1000 (e.g. ToR1's SRGB is [1000, 1999], 861 ToR2's SRGB is [2000, 2999], ...). 863 In this case, not only the controller would need to collect and store 864 all of these different SRGB's (e.g., through the Originator SRGB TLV 865 of the BGP-Prefix-SID), furthermore it would need to adapt the policy 866 for each host. Indeed, the controller would instruct A to use {1011} 867 for FA1 while it would have to instruct B to use {2011} for FB1 868 (while with the same SRGB, both policies are the same {16011}). 870 Even worse, the controller would instruct A to use {1005, 5011} for 871 FA1 while it would instruct B to use {2011, 8011} for FB1 (while with 872 the same SRGB, the second segment is the same across both policies: 873 16011). When combining segments to create a policy, one need to 874 carefully update the label of each segment. This is obviously more 875 error-prone, more complex and more difficult to troubleshoot. 877 10. IANA Considerations 879 This document does not make any IANA request. 881 11. Manageability Considerations 883 The design and deployment guidelines described in this document rely 884 on Segment Routing extensions defined in BGP by 885 [I-D.ietf-idr-bgp-prefix-sid]. This document doesn't introduce any 886 change in the manageability of the BGP-Prefix-SID attribute other 887 than the ones defined in [I-D.ietf-idr-bgp-prefix-sid]. 889 In addition the SRGB allocation scheme recommended here is based on 890 the recommendation made in [I-D.ietf-spring-segment-routing] and no 891 other manageability aspects are introduced in this document. 893 12. Security Considerations 895 This document proposes to apply Segment Routing to a well known 896 scalability requirement expressed in [RFC7938] using the BGP-Prefix- 897 SID as defined in [I-D.ietf-idr-bgp-prefix-sid]. 899 The design does not introduce any additional security concerns from 900 what expressed in [RFC7938] and [I-D.ietf-idr-bgp-prefix-sid]. 902 13. Acknowledgements 904 The authors would like to thank Benjamin Black, Arjun Sreekantiah, 905 Keyur Patel, Acee Lindem and Anoop Ghanwani for their comments and 906 review of this document. 908 14. Contributors 910 Gaya Nagarajan 911 Facebook 912 US 914 Email: gaya@fb.com 916 Dmitry Afanasiev 917 Yandex 918 RU 920 Email: fl0w@yandex-team.ru 922 Tim Laberge 923 Cisco 924 US 926 Email: tlaberge@cisco.com 928 Edet Nkposong 929 Salesforce.com Inc. 930 US 932 Email: enkposong@salesforce.com 934 Mohan Nanduri 935 Microsoft 936 US 938 Email: mnanduri@microsoft.com 940 James Uttaro 941 ATT 942 US 944 Email: ju1738@att.com 945 Saikat Ray 946 Unaffiliated 947 US 949 Email: raysaikat@gmail.com 951 15. References 953 15.1. Normative References 955 [I-D.ietf-idr-bgp-prefix-sid] 956 Previdi, S., Filsfils, C., Lindem, A., Patel, K., 957 Sreekantiah, A., Ray, S., and H. Gredler, "Segment Routing 958 Prefix SID extensions for BGP", draft-ietf-idr-bgp-prefix- 959 sid-04 (work in progress), December 2016. 961 [I-D.ietf-spring-segment-routing] 962 Filsfils, C., Previdi, S., Decraene, B., Litkowski, S., 963 and R. Shakir, "Segment Routing Architecture", draft-ietf- 964 spring-segment-routing-11 (work in progress), February 965 2017. 967 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 968 Requirement Levels", BCP 14, RFC 2119, 969 DOI 10.17487/RFC2119, March 1997, 970 . 972 [RFC3107] Rekhter, Y. and E. Rosen, "Carrying Label Information in 973 BGP-4", RFC 3107, DOI 10.17487/RFC3107, May 2001, 974 . 976 [RFC4271] Rekhter, Y., Ed., Li, T., Ed., and S. Hares, Ed., "A 977 Border Gateway Protocol 4 (BGP-4)", RFC 4271, 978 DOI 10.17487/RFC4271, January 2006, 979 . 981 [RFC7311] Mohapatra, P., Fernando, R., Rosen, E., and J. Uttaro, 982 "The Accumulated IGP Metric Attribute for BGP", RFC 7311, 983 DOI 10.17487/RFC7311, August 2014, 984 . 986 15.2. Informative References 988 [GREENBERG09] 989 Greenberg, A., Hamilton, J., Jain, N., Kadula, S., Kim, 990 C., Lahiri, P., Maltz, D., Patel, P., and S. Sengupta, 991 "VL2: A Scalable and Flexible Data Center Network", 2009. 993 [I-D.ietf-6man-segment-routing-header] 994 Previdi, S., Filsfils, C., Field, B., Leung, I., Linkova, 995 J., Aries, E., Kosugi, T., Vyncke, E., and D. Lebrun, 996 "IPv6 Segment Routing Header (SRH)", draft-ietf-6man- 997 segment-routing-header-05 (work in progress), February 998 2017. 1000 [I-D.ietf-mpls-seamless-mpls] 1001 Leymann, N., Decraene, B., Filsfils, C., Konstantynowicz, 1002 M., and D. Steinberg, "Seamless MPLS Architecture", draft- 1003 ietf-mpls-seamless-mpls-07 (work in progress), June 2014. 1005 [I-D.ietf-spring-segment-routing-central-epe] 1006 Filsfils, C., Previdi, S., Aries, E., and D. Afanasiev, 1007 "Segment Routing Centralized BGP Egress Peer Engineering", 1008 draft-ietf-spring-segment-routing-central-epe-04 (work in 1009 progress), February 2017. 1011 [KANDULA04] 1012 Sinha, S., Kandula, S., and D. Katabi, "Harnessing TCP's 1013 Burstiness with Flowlet Switching", 2004. 1015 [RFC7938] Lapukhov, P., Premji, A., and J. Mitchell, Ed., "Use of 1016 BGP for Routing in Large-Scale Data Centers", RFC 7938, 1017 DOI 10.17487/RFC7938, August 2016, 1018 . 1020 Authors' Addresses 1022 Clarence Filsfils (editor) 1023 Cisco Systems, Inc. 1024 Brussels 1025 BE 1027 Email: cfilsfil@cisco.com 1029 Stefano Previdi (editor) 1030 Cisco Systems, Inc. 1031 Via Del Serafico, 200 1032 Rome 00142 1033 Italy 1035 Email: sprevidi@cisco.com 1036 Jon Mitchell 1037 Unaffiliated 1039 Email: jrmitche@puck.nether.net 1041 Ebben Aries 1042 Juniper Networks 1043 1133 Innovation Way 1044 Sunnyvale CA 94089 1045 US 1047 Email: exa@juniper.net 1049 Petr Lapukhov 1050 Facebook 1051 US 1053 Email: petr@fb.com