idnits 2.17.1 draft-ietf-spring-segment-routing-msdc-08.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (December 21, 2017) is 2316 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- -- Looks like a reference, but probably isn't: '16000' on line 270 -- Looks like a reference, but probably isn't: '23999' on line 270 -- Looks like a reference, but probably isn't: '1000' on line 883 -- Looks like a reference, but probably isn't: '1999' on line 883 -- Looks like a reference, but probably isn't: '2000' on line 884 -- Looks like a reference, but probably isn't: '2999' on line 884 == Unused Reference: 'RFC2119' is defined on line 1026, but no explicit reference was found in the text == Outdated reference: A later version (-27) exists of draft-ietf-idr-bgp-prefix-sid-07 == Outdated reference: A later version (-15) exists of draft-ietf-spring-segment-routing-14 == Outdated reference: A later version (-26) exists of draft-ietf-6man-segment-routing-header-07 Summary: 0 errors (**), 0 flaws (~~), 5 warnings (==), 7 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group C. Filsfils, Ed. 3 Internet-Draft S. Previdi 4 Intended status: Informational Cisco Systems, Inc. 5 Expires: June 24, 2018 J. Mitchell 6 Unaffiliated 7 E. Aries 8 Juniper Networks 9 P. Lapukhov 10 Facebook 11 December 21, 2017 13 BGP-Prefix Segment in large-scale data centers 14 draft-ietf-spring-segment-routing-msdc-08 16 Abstract 18 This document describes the motivation and benefits for applying 19 segment routing in BGP-based large-scale data-centers. It describes 20 the design to deploy segment routing in those data-centers, for both 21 the MPLS and IPv6 dataplanes. 23 Status of This Memo 25 This Internet-Draft is submitted in full conformance with the 26 provisions of BCP 78 and BCP 79. 28 Internet-Drafts are working documents of the Internet Engineering 29 Task Force (IETF). Note that other groups may also distribute 30 working documents as Internet-Drafts. The list of current Internet- 31 Drafts is at https://datatracker.ietf.org/drafts/current/. 33 Internet-Drafts are draft documents valid for a maximum of six months 34 and may be updated, replaced, or obsoleted by other documents at any 35 time. It is inappropriate to use Internet-Drafts as reference 36 material or to cite them other than as "work in progress." 38 This Internet-Draft will expire on June 24, 2018. 40 Copyright Notice 42 Copyright (c) 2017 IETF Trust and the persons identified as the 43 document authors. All rights reserved. 45 This document is subject to BCP 78 and the IETF Trust's Legal 46 Provisions Relating to IETF Documents 47 (https://trustee.ietf.org/license-info) in effect on the date of 48 publication of this document. Please review these documents 49 carefully, as they describe your rights and restrictions with respect 50 to this document. Code Components extracted from this document must 51 include Simplified BSD License text as described in Section 4.e of 52 the Trust Legal Provisions and are provided without warranty as 53 described in the Simplified BSD License. 55 Table of Contents 57 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 58 2. Large Scale Data Center Network Design Summary . . . . . . . 3 59 2.1. Reference design . . . . . . . . . . . . . . . . . . . . 4 60 3. Some open problems in large data-center networks . . . . . . 5 61 4. Applying Segment Routing in the DC with MPLS dataplane . . . 6 62 4.1. BGP Prefix Segment (BGP-Prefix-SID) . . . . . . . . . . . 6 63 4.2. eBGP Labeled Unicast (RFC8277) . . . . . . . . . . . . . 6 64 4.2.1. Control Plane . . . . . . . . . . . . . . . . . . . . 7 65 4.2.2. Data Plane . . . . . . . . . . . . . . . . . . . . . 9 66 4.2.3. Network Design Variation . . . . . . . . . . . . . . 10 67 4.2.4. Global BGP Prefix Segment through the fabric . . . . 10 68 4.2.5. Incremental Deployments . . . . . . . . . . . . . . . 11 69 4.3. iBGP Labeled Unicast (RFC8277) . . . . . . . . . . . . . 12 70 5. Applying Segment Routing in the DC with IPv6 dataplane . . . 14 71 6. Communicating path information to the host . . . . . . . . . 14 72 7. Addressing the open problems . . . . . . . . . . . . . . . . 15 73 7.1. Per-packet and flowlet switching . . . . . . . . . . . . 15 74 7.2. Performance-aware routing . . . . . . . . . . . . . . . . 16 75 7.3. Deterministic network probing . . . . . . . . . . . . . . 17 76 8. Additional Benefits . . . . . . . . . . . . . . . . . . . . . 17 77 8.1. MPLS Dataplane with operational simplicity . . . . . . . 18 78 8.2. Minimizing the FIB table . . . . . . . . . . . . . . . . 18 79 8.3. Egress Peer Engineering . . . . . . . . . . . . . . . . . 18 80 8.4. Anycast . . . . . . . . . . . . . . . . . . . . . . . . . 19 81 9. Preferred SRGB Allocation . . . . . . . . . . . . . . . . . . 19 82 10. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 20 83 11. Manageability Considerations . . . . . . . . . . . . . . . . 20 84 12. Security Considerations . . . . . . . . . . . . . . . . . . . 21 85 13. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 21 86 14. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 21 87 15. References . . . . . . . . . . . . . . . . . . . . . . . . . 22 88 15.1. Normative References . . . . . . . . . . . . . . . . . . 22 89 15.2. Informative References . . . . . . . . . . . . . . . . . 23 90 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 24 92 1. Introduction 94 Segment Routing (SR), as described in 95 [I-D.ietf-spring-segment-routing] leverages the source routing 96 paradigm. A node steers a packet through an ordered list of 97 instructions, called segments. A segment can represent any 98 instruction, topological or service-based. A segment can have a 99 local semantic to an SR node or global within an SR domain. SR 100 allows to enforce a flow through any topological path while 101 maintaining per-flow state only at the ingress node to the SR domain. 102 Segment Routing can be applied to the MPLS and IPv6 data-planes. 104 The use-cases described in this document should be considered in the 105 context of the BGP-based large-scale data-center (DC) design 106 described in [RFC7938]. This document extends it by applying SR both 107 with IPv6 and MPLS dataplane. 109 2. Large Scale Data Center Network Design Summary 111 This section provides a brief summary of the informational document 112 [RFC7938] that outlines a practical network design suitable for data- 113 centers of various scales: 115 o Data-center networks have highly symmetric topologies with 116 multiple parallel paths between two server attachment points. The 117 well-known Clos topology is most popular among the operators (as 118 described in [RFC7938]). In a Clos topology, the minimum number 119 of parallel paths between two elements is determined by the 120 "width" of the "Tier-1" stage. See Figure 1 below for an 121 illustration of the concept. 123 o Large-scale data-centers commonly use a routing protocol, such as 124 BGP-4 [RFC4271] in order to provide endpoint connectivity. 125 Recovery after a network failure is therefore driven either by 126 local knowledge of directly available backup paths or by 127 distributed signaling between the network devices. 129 o Within data-center networks, traffic is load-shared using the 130 Equal Cost Multipath (ECMP) mechanism. With ECMP, every network 131 device implements a pseudo-random decision, mapping packets to one 132 of the parallel paths by means of a hash function calculated over 133 certain parts of the packet, typically a combination of various 134 packet header fields. 136 The following is a schematic of a five-stage Clos topology, with four 137 devices in the "Tier-1" stage. Notice that number of paths between 138 Node1 and Node12 equals to four: the paths have to cross all of 139 Tier-1 devices. At the same time, the number of paths between Node1 140 and Node2 equals two, and the paths only cross Tier-2 devices. Other 141 topologies are possible, but for simplicity only the topologies that 142 have a single path from Tier-1 to Tier-3 are considered below. The 143 rest could be treated similarly, with a few modifications to the 144 logic. 146 2.1. Reference design 148 Tier-1 149 +-----+ 150 |NODE | 151 +->| 5 |--+ 152 | +-----+ | 153 Tier-2 | | Tier-2 154 +-----+ | +-----+ | +-----+ 155 +------------>|NODE |--+->|NODE |--+--|NODE |-------------+ 156 | +-----| 3 |--+ | 6 | +--| 9 |-----+ | 157 | | +-----+ +-----+ +-----+ | | 158 | | | | 159 | | +-----+ +-----+ +-----+ | | 160 | +-----+---->|NODE |--+ |NODE | +--|NODE |-----+-----+ | 161 | | | +---| 4 |--+->| 7 |--+--| 10 |---+ | | | 162 | | | | +-----+ | +-----+ | +-----+ | | | | 163 | | | | | | | | | | 164 +-----+ +-----+ | +-----+ | +-----+ +-----+ 165 |NODE | |NODE | Tier-3 +->|NODE |--+ Tier-3 |NODE | |NODE | 166 | 1 | | 2 | | 8 | | 11 | | 12 | 167 +-----+ +-----+ +-----+ +-----+ +-----+ 168 | | | | | | | | 169 A O B O <- Servers -> Z O O O 171 Figure 1: 5-stage Clos topology 173 In the reference topology illustrated in Figure 1, It is assumed: 175 o Each node is its own AS (Node X has AS X). 4-byte AS numbers are 176 recommended ([RFC6793]). 178 * For simple and efficient route propagation filtering, Node5, 179 Node6, Node7 and Node8 use the same AS, Node3 and Node4 use the 180 same AS, Node9 and Node10 use the same AS. 182 * In case of 2-byte autonomous system numbers are used and for 183 efficient usage of the scarce 2-byte Private Use AS pool, 184 different Tier-3 nodes might use the same AS. 186 * Without loss of generality, these details will be simplified in 187 this document and assume that each node has its own AS. 189 o Each node peers with its neighbors with a BGP session. If not 190 specified, eBGP is assumed. In a specific use-case, iBGP will be 191 used but this will be called out explicitly in that case. 193 o Each node originates the IPv4 address of its loopback interface 194 into BGP and announces it to its neighbors. 196 * The loopback of Node X is 192.0.2.x/32. 198 In this document, the Tier-1, Tier-2 and Tier-3 nodes are referred to 199 respectively as Spine, Leaf and ToR (top of rack) nodes. When a ToR 200 node acts as a gateway to the "outside world", it is referred to as a 201 border node. 203 3. Some open problems in large data-center networks 205 The data-center network design summarized above provides means for 206 moving traffic between hosts with reasonable efficiency. There are 207 few open performance and reliability problems that arise in such 208 design: 210 o ECMP routing is most commonly realized per-flow. This means that 211 large, long-lived "elephant" flows may affect performance of 212 smaller, short-lived "mouse" flows and reduce efficiency of per- 213 flow load-sharing. In other words, per-flow ECMP does not perform 214 efficiently when flow lifetime distribution is heavy-tailed. 215 Furthermore, due to hash-function inefficiencies it is possible to 216 have frequent flow collisions, where more flows get placed on one 217 path over the others. 219 o Shortest-path routing with ECMP implements an oblivious routing 220 model, which is not aware of the network imbalances. If the 221 network symmetry is broken, for example due to link failures, 222 utilization hotspots may appear. For example, if a link fails 223 between Tier-1 and Tier-2 devices (e.g. Node5 and Node9), Tier-3 224 devices Node1 and Node2 will not be aware of that, since there are 225 other paths available from perspective of Node3. They will 226 continue sending roughly equal traffic to Node3 and Node4 as if 227 the failure didn't exist which may cause a traffic hotspot. 229 o The absence of path visibility leaves transport protocols, such as 230 TCP, with a "blackbox" view of the network. Some TCP metrics, 231 such as SRTT, MSS, CWND and few others could be inferred and 232 cached based on past history, but those apply to destinations, 233 regardless of the path that has been chosen to get there. Thus, 234 for instance, TCP is not capable of remembering "bad" paths, such 235 as those that exhibited poor performance in the past. This means 236 that every new connection will be established obliviously (memory- 237 less) with regards to the paths chosen before, or chosen by other 238 nodes. 240 o Isolating faults in the network with multiple parallel paths and 241 ECMP-based routing is non-trivial due to lack of determinism. 242 Specifically, the connections from HostA to HostB may take a 243 different path every time a new connection is formed, thus making 244 consistent reproduction of a failure much more difficult. This 245 complexity scales linearly with the number of parallel paths in 246 the network, and stems from the random nature of path selection by 247 the network devices. 249 Further in this document (Section 7), it is demonstrated how these 250 problems could be addressed within the framework of Segment Routing. 252 First, it will be explained how to apply SR in the DC, for MPLS and 253 IPv6 data-planes. 255 4. Applying Segment Routing in the DC with MPLS dataplane 257 4.1. BGP Prefix Segment (BGP-Prefix-SID) 259 A BGP Prefix Segment is a segment associated with a BGP prefix. A 260 BGP Prefix Segment is a network-wide instruction to forward the 261 packet along the ECMP-aware best path to the related prefix. 263 The BGP Prefix Segment is defined as the BGP-Prefix-SID Attribute in 264 [I-D.ietf-idr-bgp-prefix-sid] which contains an index. Throughout 265 this document the BGP Prefix Segment Attribute is referred as the 266 BGP-Prefix-SID and the encoded index as the label-index. 268 In this document, the network design decision has been made to assume 269 that all the nodes are allocated the same SRGB (Segment Routing 270 Global Block), e.g. [16000, 23999]. This provides operational 271 simplification as explained in Section 9, but this is not a 272 requirement. 274 For illustration purpose, when considering an MPLS data-plane, it is 275 assumed that the label-index allocated to prefix 192.0.2.x/32 is X. 276 As a result, a local label (16000+x) is allocated for prefix 277 192.0.2.x/32 by each node throughout the DC fabric. 279 When IPv6 data-plane is considered, it is assumed that Node X is 280 allocated IPv6 address (segment) 2001:DB8::X. 282 4.2. eBGP Labeled Unicast (RFC8277) 284 Referring to Figure 1 and [RFC7938], the following design 285 modifications are introduced: 287 o Each node peers with its neighbors via a eBGP session with 288 extensions defined in [RFC8277] (named "eBGP8277" throughout this 289 document) and with the BGP-Prefix-SID attribute extension as 290 defined in [I-D.ietf-idr-bgp-prefix-sid]. 292 o The forwarding plane at Tier-2 and Tier-1 is MPLS. 294 o The forwarding plane at Tier-3 is either IP2MPLS (if the host 295 sends IP traffic) or MPLS2MPLS (if the host sends MPLS- 296 encapsulated traffic). 298 Figure 2 zooms into a path from server A to server Z within the 299 topology of Figure 1. 301 +-----+ +-----+ +-----+ 302 +---------->|NODE | |NODE | |NODE | 303 | | 4 |--+->| 7 |--+--| 10 |---+ 304 | +-----+ +-----+ +-----+ | 305 | | 306 +-----+ +-----+ 307 |NODE | |NODE | 308 | 1 | | 11 | 309 +-----+ +-----+ 310 | | 311 A <- Servers -> Z 313 Figure 2: Path from A to Z via nodes 1, 4, 7, 10 and 11 315 Referring to Figure 1 and Figure 2 and assuming the IP address with 316 the AS and label-index allocation previously described, the following 317 sections detail the control plane operation and the data plane states 318 for the prefix 192.0.2.11/32 (loopback of Node11) 320 4.2.1. Control Plane 322 Node11 originates 192.0.2.11/32 in BGP and allocates to it a BGP- 323 Prefix-SID with label-index: index11 [I-D.ietf-idr-bgp-prefix-sid]. 325 Node11 sends the following eBGP8277 update to Node10: 327 . IP Prefix: 192.0.2.11/32 328 . Label: Implicit-Null 329 . Next-hop: Node11's interface address on the link to Node10 330 . AS Path: {11} 331 . BGP-Prefix-SID: Label-Index 11 333 Node10 receives the above update. As it is SR capable, Node10 is 334 able to interpret the BGP-Prefix-SID and hence understands that it 335 should allocate the label from its own SRGB block, offset by the 336 Label-Index received in the BGP-Prefix-SID (16000+11 hence 16011) to 337 the NLRI instead of allocating a non-deterministic label out of a 338 dynamically allocated portion of the local label space. The 339 implicit-null label in the NLRI tells Node10 that it is the 340 penultimate hop and must pop the top label on the stack before 341 forwarding traffic for this prefix to Node11. 343 Then, Node10 sends the following eBGP8277 update to Node7: 345 . IP Prefix: 192.0.2.11/32 346 . Label: 16011 347 . Next-hop: Node10's interface address on the link to Node7 348 . AS Path: {10, 11} 349 . BGP-Prefix-SID: Label-Index 11 351 Node7 receives the above update. As it is SR capable, Node7 is able 352 to interpret the BGP-Prefix-SID and hence allocates the local 353 (incoming) label 16011 (16000 + 11) to the NLRI (instead of 354 allocating a "dynamic" local label from its label manager). Node7 355 uses the label in the received eBGP8277 NLRI as the outgoing label 356 (the index is only used to derive the local/incoming label). 358 Node7 sends the following eBGP8277 update to Node4: 360 . IP Prefix: 192.0.2.11/32 361 . Label: 16011 362 . Next-hop: Node7's interface address on the link to Node4 363 . AS Path: {7, 10, 11} 364 . BGP-Prefix-SID: Label-Index 11 366 Node4 receives the above update. As it is SR capable, Node4 is able 367 to interpret the BGP-Prefix-SID and hence allocates the local 368 (incoming) label 16011 to the NLRI (instead of allocating a "dynamic" 369 local label from its label manager). Node4 uses the label in the 370 received eBGP8277 NLRI as outgoing label (the index is only used to 371 derive the local/incoming label). 373 Node4 sends the following eBGP8277 update to Node1: 375 . IP Prefix: 192.0.2.11/32 376 . Label: 16011 377 . Next-hop: Node4's interface address on the link to Node1 378 . AS Path: {4, 7, 10, 11} 379 . BGP-Prefix-SID: Label-Index 11 381 Node1 receives the above update. As it is SR capable, Node1 is able 382 to interpret the BGP-Prefix-SID and hence allocates the local 383 (incoming) label 16011 to the NLRI (instead of allocating a "dynamic" 384 local label from its label manager). Node1 uses the label in the 385 received eBGP8277 NLRI as outgoing label (the index is only used to 386 derive the local/incoming label). 388 4.2.2. Data Plane 390 Referring to Figure 1, and assuming all nodes apply the same 391 advertisement rules described above and all nodes have the same SRGB 392 (16000-23999), here are the IP/MPLS forwarding tables for prefix 393 192.0.2.11/32 at Node1, Node4, Node7 and Node10. 395 ----------------------------------------------- 396 Incoming label | outgoing label | Outgoing 397 or IP destination | | Interface 398 ------------------+----------------+----------- 399 16011 | 16011 | ECMP{3, 4} 400 192.0.2.11/32 | 16011 | ECMP{3, 4} 401 ------------------+----------------+----------- 403 Figure 3: Node1 Forwarding Table 405 ----------------------------------------------- 406 Incoming label | outgoing label | Outgoing 407 or IP destination | | Interface 408 ------------------+----------------+----------- 409 16011 | 16011 | ECMP{7, 8} 410 192.0.2.11/32 | 16011 | ECMP{7, 8} 411 ------------------+----------------+----------- 413 Figure 4: Node4 Forwarding Table 415 ----------------------------------------------- 416 Incoming label | outgoing label | Outgoing 417 or IP destination | | Interface 418 ------------------+----------------+----------- 419 16011 | 16011 | 10 420 192.0.2.11/32 | 16011 | 10 421 ------------------+----------------+----------- 423 Figure 5: Node7 Forwarding Table 425 ----------------------------------------------- 426 Incoming label | outgoing label | Outgoing 427 or IP destination | | Interface 428 ------------------+----------------+----------- 429 16011 | POP | 11 430 192.0.2.11/32 | N/A | 11 431 ------------------+----------------+----------- 433 Node10 Forwarding Table 435 4.2.3. Network Design Variation 437 A network design choice could consist of switching all the traffic 438 through Tier-1 and Tier-2 as MPLS traffic. In this case, one could 439 filter away the IP entries at Node4, Node7 and Node10. This might be 440 beneficial in order to optimize the forwarding table size. 442 A network design choice could consist in allowing the hosts to send 443 MPLS-encapsulated traffic based on the Egress Peer Engineering (EPE) 444 use-case as defined in [I-D.ietf-spring-segment-routing-central-epe]. 445 For example, applications at HostA would send their Z-destined 446 traffic to Node1 with an MPLS label stack where the top label is 447 16011 and the next label is an EPE peer segment 448 ([I-D.ietf-spring-segment-routing-central-epe]) at Node11 directing 449 the traffic to Z. 451 4.2.4. Global BGP Prefix Segment through the fabric 453 When the previous design is deployed, the operator enjoys global BGP- 454 Prefix-SID and label allocation throughout the DC fabric. 456 A few examples follow: 458 o Normal forwarding to Node11: a packet with top label 16011 459 received by any node in the fabric will be forwarded along the 460 ECMP-aware BGP best-path towards Node11 and the label 16011 is 461 penultimate-popped at Node10 (or at Node 9). 463 o Traffic-engineered path to Node11: an application on a host behind 464 Node1 might want to restrict its traffic to paths via the Spine 465 node Node5. The application achieves this by sending its packets 466 with a label stack of {16005, 16011}. BGP Prefix SID 16005 directs 467 the packet up to Node5 along the path (Node1, Node3, Node5). BGP- 468 Prefix-SID 16011 then directs the packet down to Node11 along the 469 path (Node5, Node9, Node11). 471 4.2.5. Incremental Deployments 473 The design previously described can be deployed incrementally. Let 474 us assume that Node7 does not support the BGP-Prefix-SID and let us 475 show how the fabric connectivity is preserved. 477 From a signaling viewpoint, nothing would change: even though Node7 478 does not support the BGP-Prefix-SID, it does propagate the attribute 479 unmodified to its neighbors. 481 From a label allocation viewpoint, the only difference is that Node7 482 would allocate a dynamic (random) label to the prefix 192.0.2.11/32 483 (e.g. 123456) instead of the "hinted" label as instructed by the BGP- 484 Prefix-SID. The neighbors of Node7 adapt automatically as they 485 always use the label in the BGP8277 NLRI as outgoing label. 487 Node4 does understand the BGP-Prefix-SID and hence allocates the 488 indexed label in the SRGB (16011) for 192.0.2.11/32. 490 As a result, all the data-plane entries across the network would be 491 unchanged except the entries at Node7 and its neighbor Node4 as shown 492 in the figures below. 494 The key point is that the end-to-end Label Switched Path (LSP) is 495 preserved because the outgoing label is always derived from the 496 received label within the BGP8277 NLRI. The index in the BGP-Prefix- 497 SID is only used as a hint on how to allocate the local label (the 498 incoming label) but never for the outgoing label. 500 ------------------------------------------ 501 Incoming label | outgoing | Outgoing 502 or IP destination | label | Interface 503 -------------------+---------------------- 504 12345 | 16011 | 10 506 Figure 7: Node7 Forwarding Table 508 ------------------------------------------ 509 Incoming label | outgoing | Outgoing 510 or IP destination | label | Interface 511 -------------------+---------------------- 512 16011 | 12345 | 7 514 Figure 8: Node4 Forwarding Table 516 The BGP-Prefix-SID can thus be deployed incrementally one node at a 517 time. 519 When deployed together with a homogeneous SRGB (same SRGB across the 520 fabric), the operator incrementally enjoys the global prefix segment 521 benefits as the deployment progresses through the fabric. 523 4.3. iBGP Labeled Unicast (RFC8277) 525 The same exact design as eBGP8277 is used with the following 526 modifications: 528 All nodes use the same AS number. 530 Each node peers with its neighbors via an internal BGP session 531 (iBGP) with extensions defined in [RFC8277] (named "iBGP8277" 532 throughout this document). 534 Each node acts as a route-reflector for each of its neighbors and 535 with the next-hop-self option. Next-hop-self is a well known 536 operational feature which consists of rewriting the next-hop of a 537 BGP update prior to send it to the neighbor. Usually, it's a 538 common practice to apply next-hop-self behavior towards iBGP peers 539 for eBGP learned routes. In the case outlined in this section it 540 is proposed to use the next-hop-self mechanism also to iBGP 541 learned routes. 543 Cluster-1 544 +-----------+ 545 | Tier-1 | 546 | +-----+ | 547 | |NODE | | 548 | | 5 | | 549 Cluster-2 | +-----+ | Cluster-3 550 +---------+ | | +---------+ 551 | Tier-2 | | | | Tier-2 | 552 | +-----+ | | +-----+ | | +-----+ | 553 | |NODE | | | |NODE | | | |NODE | | 554 | | 3 | | | | 6 | | | | 9 | | 555 | +-----+ | | +-----+ | | +-----+ | 556 | | | | | | 557 | | | | | | 558 | +-----+ | | +-----+ | | +-----+ | 559 | |NODE | | | |NODE | | | |NODE | | 560 | | 4 | | | | 7 | | | | 10 | | 561 | +-----+ | | +-----+ | | +-----+ | 562 +---------+ | | +---------+ 563 | | 564 | +-----+ | 565 | |NODE | | 566 Tier-3 | | 8 | | Tier-3 567 +-----+ +-----+ | +-----+ | +-----+ +-----+ 568 |NODE | |NODE | +-----------+ |NODE | |NODE | 569 | 1 | | 2 | | 11 | | 12 | 570 +-----+ +-----+ +-----+ +-----+ 572 Figure 9: iBGP Sessions with Reflection and Next-Hop-Self 574 For simple and efficient route propagation filtering and as 575 illustrated in Figure 9: 577 Node5, Node6, Node7 and Node8 use the same Cluster ID (Cluster- 578 1) 580 Node3 and Node4 use the same Cluster ID (Cluster-2) 582 Node9 and Node10 use the same Cluster ID (Cluster-3) 584 The control-plane behavior is mostly the same as described in the 585 previous section: the only difference is that the eBGP8277 path 586 propagation is simply replaced by an iBGP8277 path reflection with 587 next-hop changed to self. 589 The data-plane tables are exactly the same. 591 5. Applying Segment Routing in the DC with IPv6 dataplane 593 The design described in [RFC7938] is reused with one single 594 modification. It is highlighted using the example of the 595 reachability to Node11 via spine node Node5. 597 Node5 originates 2001:DB8::5/128 with the attached BGP-Prefix-SID for 598 IPv6 packets destined to segment 2001:DB8::5 599 ([I-D.ietf-idr-bgp-prefix-sid]). 601 Node11 originates 2001:DB8::11/128 with the attached BGP-Prefix-SID 602 advertising the support of the SRH for IPv6 packets destined to 603 segment 2001:DB8::11. 605 The control-plane and data-plane processing of all the other nodes in 606 the fabric is unchanged. Specifically, the routes to 2001:DB8::5 and 607 2001:DB8::11 are installed in the FIB along the eBGP best-path to 608 Node5 (spine node) and Node11 (ToR node) respectively. 610 An application on HostA which needs to send traffic to HostZ via only 611 Node5 (spine node) can do so by sending IPv6 packets with a Segment 612 Routing header (SRH, [I-D.ietf-6man-segment-routing-header]). The 613 destination address and active segment is set to 2001:DB8::5. The 614 next and last segment is set to 2001:DB8::11. 616 The application must only use IPv6 addresses that have been 617 advertised as capable for SRv6 segment processing (e.g. for which the 618 BGP prefix segment capability has been advertised). How applications 619 learn this (e.g.: centralized controller and orchestration) is 620 outside the scope of this document. 622 6. Communicating path information to the host 624 There are two general methods for communicating path information to 625 the end-hosts: "proactive" and "reactive", aka "push" and "pull" 626 models. There are multiple ways to implement either of these 627 methods. Here, it is noted that one way could be using a centralized 628 controller: the controller either tells the hosts of the prefix-to- 629 path mappings beforehand and updates them as needed (network event 630 driven push), or responds to the hosts making request for a path to 631 specific destination (host event driven pull). It is also possible 632 to use a hybrid model, i.e., pushing some state from the controller 633 in response to particular network events, while the host pulls other 634 state on demand. 636 It is also noted, that when disseminating network-related data to the 637 end-hosts a trade-off is made to balance the amount of information 638 Vs. the level of visibility in the network state. This applies both 639 to push and pull models. In the extreme case, the host would request 640 path information on every flow, and keep no local state at all. On 641 the other end of the spectrum, information for every prefix in the 642 network along with available paths could be pushed and continuously 643 updated on all hosts. 645 7. Addressing the open problems 647 This section demonstrates how the problems described above (in 648 section 3) could be solved using the segment routing concept. It is 649 worth noting that segment routing signaling and data-plane are only 650 parts of the solution. Additional enhancements, e.g., such as the 651 centralized controller mentioned previously, and host networking 652 stack support are required to implement the proposed solutions. Also 653 the applicability of the solutions described below are not restricted 654 to the data-center alone, the same could be re-used in context of 655 other domains as well 657 7.1. Per-packet and flowlet switching 659 A flowlet is defined as a burst of packets from the same flow 660 followed by an idle interval. [KANDULA04] developed a scheme that 661 uses flowlets to split traffic across multiple parallel paths in 662 order to optimize traffic load sharing. 664 With the ability to choose paths on the host, one may go from per- 665 flow load-sharing in the network to per-packet or per-flowlet. The 666 host may select different segment routing instructions either per 667 packet, or per flowlet, and route them over different paths. This 668 allows for solving the "elephant flow" problem in the data-center and 669 avoiding link imbalances. 671 Note that traditional ECMP routing could be easily simulated with on- 672 host path selection, using method proposed in [GREENBERG09]. The 673 hosts would randomly pick a Tier-2 or Tier-1 device to "bounce" the 674 packet off of, depending on whether the destination is under the same 675 Tier-2 nodes, or has to be reached across Tier-1. The host would use 676 a hash function that operates on per-flow invariants, to simulate 677 per-flow load-sharing in the network. 679 Using Figure 1 as reference, let us illustrate this concept assuming 680 that HostA has an elephant flow to HostZ called Flow-f. 682 Normally, a flow is hashed on to a single path. Let's assume HostA 683 sends its packets associated with Flow-f with top label 16011 (the 684 label for the remote ToR, Node11, where HostZ is connected) and Node1 685 would hash all the packets of Flow-F via the same next-hop (e.g. 686 Node3). Similarly, let's assume that leaf Node3 would hash all the 687 packets of Flow-F via the same next-hop (e.g.: spine node Node5). 688 This normal operation would restrict the elephant flow on a small 689 subset of the ECMP paths to HostZ and potentially create imbalance 690 and congestion in the fabric. 692 Leveraging the flowlet proposal, assuming HostA is made aware of 4 693 disjoint paths via intermediate segment 16005, 16006, 16007 and 16008 694 (the BGP prefix SID's of the 4 spine nodes) and also made aware of 695 the prefix segment of the remote ToR connected to the destination 696 (16011), then the application can break the elephant flow F into 697 flowlets F1, F2, F3, F4 and associate each flowlet with one of the 698 following 4 label stacks: {16005, 16011}, {16006, 16011}, {16007, 699 16011} and {16008, 16011}. This would spread the load of the elephant 700 flow through all the ECMP paths available in the fabric and re- 701 balance the load. 703 7.2. Performance-aware routing 705 Knowing the path associated with flows/packets, the end host may 706 deduce certain characteristics of the path on its own, and 707 additionally use the information supplied with path information 708 pushed from the controller or received via pull request. The host 709 may further share its path observations with the centralized agent, 710 so that the latter may keep up-to-date network health map to assist 711 other hosts with this information. 713 For example, an application A.1 at HostA may pin a TCP flow destined 714 to HostZ via Spine node Node5 using label stack {16005, 16011}. The 715 application A.1 may collect information on packet loss, deduced from 716 TCP retransmissions and other signals (e.g. RTT increases). A.1 may 717 additionally publish this information to a centralized agent, e.g. 718 after a flow completes, or periodically for longer lived flows. 719 Next, using both local and/or global performance data, application 720 A.1 as well as other applications sharing the same resources in the 721 DC fabric may pick up the best path for the new flow, or update an 722 existing path (e.g.: when informed of congestion on an existing 723 path). 725 One particularly interesting instance of performance-aware routing is 726 dynamic fault-avoidance. If some links or devices in the network 727 start discarding packets due to a fault, the end-hosts could probe 728 and detect the path(s) that are affected and hence steer the affected 729 flows away from the problem spot. Similar logic applies to failure 730 cases where packets get completely black-holed, e.g., when a link 731 goes down and the failure is detected by the host while probing the 732 path. 734 For example, an application A.1 informed about 5 paths to Z {16005, 735 16011}, {16006, 16011}, {16007, 16011}, {16008, 16011} and {16011} 736 might use the last one by default (for simplicity). When performance 737 is degrading, A.1 might then start to pin TCP flows to each of the 4 738 other paths (each via a distinct spine) and monitor the performance. 739 It would then detect the faulty path and assign a negative preference 740 to the faulty path to avoid further flows using it. Gradually, over 741 time, it may re-assign flows on the faulty path to eventually detect 742 the resolution of the trouble and start reusing the path. 744 By leveraging Segment Routing, one avoids issues associated with 745 oblivious ECMP hashing. For example, if in the topology depicted on 746 Figure 1 a link between spine node Node5 and leaf node Node9 fails, 747 HostA may exclude the segment corresponding to Node5 from the prefix 748 matching the servers under Tier-2 devices Node9. In the push path 749 discovery model, the affected path mappings may be explicitly pushed 750 to all the servers for the duration of the failure. The new mapping 751 would instruct them to avoid the particular Tier-1 node until the 752 link has recovered. Alternatively, in pull path, the centralized 753 controller may start steering new flows immediately after it 754 discovers the issue. Until then, the existing flows may recover 755 using local detection of the path issues. 757 7.3. Deterministic network probing 759 Active probing is a well-known technique for monitoring network 760 elements' health, constituting of sending continuous packet streams 761 simulating network traffic to the hosts in the data-center. Segment 762 routing makes possible to prescribe the exact paths that each probe 763 or series of probes would be taking toward their destination. This 764 allows for fast correlation and detection of failed paths, by 765 processing information from multiple actively probing agents. This 766 complements the data collected from the hosts routing stacks as 767 described in Section 7.2. 769 For example, imagine a probe agent sending packets to all machines in 770 the data-center. For every host, it may send packets over each of 771 the possible paths, knowing exactly which links and devices these 772 packets will be crossing. Correlating results for multiple 773 destinations with the topological data, it may automatically isolate 774 possible problem to a link or device in the network. 776 8. Additional Benefits 777 8.1. MPLS Dataplane with operational simplicity 779 As required by [RFC7938], no new signaling protocol is introduced. 780 The BGP-Prefix-SID is a lightweight extension to BGP Labeled Unicast 781 [RFC8277]. It applies either to eBGP or iBGP based designs. 783 Specifically, LDP and RSVP-TE are not used. These protocols would 784 drastically impact the operational complexity of the Data Center and 785 would not scale. This is in line with the requirements expressed in 786 [RFC7938]. 788 Provided the same SRGB is configured on all nodes, all nodes use the 789 same MPLS label for a given IP prefix. This is simpler from an 790 operation standpoint, as discussed in Section 9 792 8.2. Minimizing the FIB table 794 The designer may decide to switch all the traffic at Tier-1 and Tier- 795 2's based on MPLS, hence drastically decreasing the IP table size at 796 these nodes. 798 This is easily accomplished by encapsulating the traffic either 799 directly at the host or the source ToR node by pushing the BGP- 800 Prefix-SID of the destination ToR for intra-DC traffic, or the BGP- 801 Prefix-SID for the the border node for inter-DC or DC-to-outside- 802 world traffic. 804 8.3. Egress Peer Engineering 806 It is straightforward to combine the design illustrated in this 807 document with the Egress Peer Engineering (EPE) use-case described in 808 [I-D.ietf-spring-segment-routing-central-epe]. 810 In such case, the operator is able to engineer its outbound traffic 811 on a per host-flow basis, without incurring any additional state at 812 intermediate points in the DC fabric. 814 For example, the controller only needs to inject a per-flow state on 815 the HostA to force it to send its traffic destined to a specific 816 Internet destination D via a selected border node (say Node12 in 817 Figure 1 instead of another border node, Node11) and a specific 818 egress peer of Node12 (say peer AS 9999 of local PeerNode segment 819 9999 at Node12 instead of any other peer which provides a path to the 820 destination D). Any packet matching this state at host A would be 821 encapsulated with SR segment list (label stack) {16012, 9999}. 16012 822 would steer the flow through the DC fabric, leveraging any ECMP, 823 along the best path to border node Node12. Once the flow gets to 824 border node Node12, the active segment is 9999 (because of PHP on the 825 upstream neighbor of Node12). This EPE PeerNode segment forces 826 border node Node12 to forward the packet to peer AS 9999, without any 827 IP lookup at the border node. There is no per-flow state for this 828 engineered flow in the DC fabric. A benefit of segment routing is 829 the per-flow state is only required at the source. 831 As well as allowing full traffic engineering control such a design 832 also offers FIB table minimization benefits as the Internet-scale FIB 833 at border node Node12 is not required if all FIB lookups are avoided 834 there by using EPE. 836 8.4. Anycast 838 The design presented in this document preserves the availability and 839 load-balancing properties of the base design presented in 840 [I-D.ietf-spring-segment-routing]. 842 For example, one could assign an anycast loopback 192.0.2.20/32 and 843 associate segment index 20 to it on the border Node11 and Node12 (in 844 addition to their node-specific loopbacks). Doing so, the EPE 845 controller could express a default "go-to-the-Internet via any border 846 node" policy as segment list {16020}. Indeed, from any host in the DC 847 fabric or from any ToR node, 16020 steers the packet towards the 848 border Node11 or Node12 leveraging ECMP where available along the 849 best paths to these nodes. 851 9. Preferred SRGB Allocation 853 In the MPLS case, it is recommend to use same SRGBs at each node. 855 Different SRGBs in each node likely increase the complexity of the 856 solution both from an operational viewpoint and from a controller 857 viewpoint. 859 From an operation viewpoint, it is much simpler to have the same 860 global label at every node for the same destination (the MPLS 861 troubleshooting is then similar to the IPv6 troubleshooting where 862 this global property is a given). 864 From a controller viewpoint, this allows us to construct simple 865 policies applicable across the fabric. 867 Let us consider two applications A and B respectively connected to 868 Node1 and Node2 (ToR nodes). A has two flows FA1 and FA2 destined to 869 Z. B has two flows FB1 and FB2 destined to Z. The controller wants 870 FA1 and FB1 to be load-shared across the fabric while FA2 and FB2 871 must be respectively steered via Node5 and Node8. 873 Assuming a consistent unique SRGB across the fabric as described in 874 the document, the controller can simply do it by instructing A and B 875 to use {16011} respectively for FA1 and FB1 and by instructing A and 876 B to use {16005 16011} and {16008 16011} respectively for FA2 and 877 FB2. 879 Let us assume a design where the SRGB is different at every node and 880 where the SRGB of each node is advertised using the Originator SRGB 881 TLV of the BGP-Prefix-SID as defined in 882 [I-D.ietf-idr-bgp-prefix-sid]: SRGB of Node K starts at value K*1000 883 and the SRGB length is 1000 (e.g. Node1's SRGB is [1000, 1999], 884 Node2's SRGB is [2000, 2999], ...). 886 In this case, not only the controller would need to collect and store 887 all of these different SRGB's (e.g., through the Originator SRGB TLV 888 of the BGP-Prefix-SID), furthermore it would need to adapt the policy 889 for each host. Indeed, the controller would instruct A to use {1011} 890 for FA1 while it would have to instruct B to use {2011} for FB1 891 (while with the same SRGB, both policies are the same {16011}). 893 Even worse, the controller would instruct A to use {1005, 5011} for 894 FA1 while it would instruct B to use {2011, 8011} for FB1 (while with 895 the same SRGB, the second segment is the same across both policies: 896 16011). When combining segments to create a policy, one need to 897 carefully update the label of each segment. This is obviously more 898 error-prone, more complex and more difficult to troubleshoot. 900 10. IANA Considerations 902 This document does not make any IANA request. 904 11. Manageability Considerations 906 The design and deployment guidelines described in this document are 907 based on the network design described in [RFC7938]. 909 The deployment model assumed in this document is based on a single 910 domain where the interconnected DCs are part of the same 911 administrative domain (which, of course, is split into different 912 autonomous systems). The operator has full control of the whole 913 domain and the usual operational and management mechanisms and 914 procedures are used in order to prevent any information related to 915 internal prefixes and topology to be leaked outside the domain. 917 As recommended in [I-D.ietf-spring-segment-routing], the same SRGB 918 should be allocated in all nodes in order to facilitate the design, 919 deployment and operations of the domain. 921 When EPE ([I-D.ietf-spring-segment-routing-central-epe]) is used (as 922 explained in Section 8.3, the same operational model is assumed. EPE 923 information is originated and propagated throughout the domain 924 towards an internal server and unless explicitly configured by the 925 operator, no EPE information is leaked outside the domain boundaries. 927 12. Security Considerations 929 This document proposes to apply Segment Routing to a well known 930 scalability requirement expressed in [RFC7938] using the BGP-Prefix- 931 SID as defined in [I-D.ietf-idr-bgp-prefix-sid]. 933 It has to be noted, as described in Section 11 that the design 934 illustrated in [RFC7938] and in this document, refer to a deployment 935 model where all nodes are under the same administration. In this 936 context, it is assumed that the operator doesn't want to leak outside 937 of the domain any information related to internal prefixes and 938 topology. The internal information includes prefix-sid and EPE 939 information. In order to prevent such leaking, the standard BGP 940 mechanisms (filters) are applied on the boundary of the domain. 942 Therefore, the solution proposed in this document does not introduce 943 any additional security concerns from what expressed in [RFC7938] and 944 [I-D.ietf-idr-bgp-prefix-sid]. It is assumed that the security and 945 confidentiality of the prefix and topology information is preserved 946 by outbound filters at each peering point of the domain as described 947 in Section 11. 949 13. Acknowledgements 951 The authors would like to thank Benjamin Black, Arjun Sreekantiah, 952 Keyur Patel, Acee Lindem and Anoop Ghanwani for their comments and 953 review of this document. 955 14. Contributors 957 Gaya Nagarajan 958 Facebook 959 US 961 Email: gaya@fb.com 963 Gaurav Dawra 964 Cisco Systems 965 US 967 Email: gdawra.ietf@gmail.com 968 Dmitry Afanasiev 969 Yandex 970 RU 972 Email: fl0w@yandex-team.ru 974 Tim Laberge 975 Cisco 976 US 978 Email: tlaberge@cisco.com 980 Edet Nkposong 981 Salesforce.com Inc. 982 US 984 Email: enkposong@salesforce.com 986 Mohan Nanduri 987 Microsoft 988 US 990 Email: mnanduri@microsoft.com 992 James Uttaro 993 ATT 994 US 996 Email: ju1738@att.com 998 Saikat Ray 999 Unaffiliated 1000 US 1002 Email: raysaikat@gmail.com 1004 15. References 1006 15.1. Normative References 1008 [I-D.ietf-idr-bgp-prefix-sid] 1009 Previdi, S., Filsfils, C., Lindem, A., Sreekantiah, A., 1010 and H. Gredler, "Segment Routing Prefix SID extensions for 1011 BGP", draft-ietf-idr-bgp-prefix-sid-07 (work in progress), 1012 October 2017. 1014 [I-D.ietf-spring-segment-routing] 1015 Filsfils, C., Previdi, S., Ginsberg, L., Decraene, B., 1016 Litkowski, S., and R. Shakir, "Segment Routing 1017 Architecture", draft-ietf-spring-segment-routing-14 (work 1018 in progress), December 2017. 1020 [I-D.ietf-spring-segment-routing-central-epe] 1021 Filsfils, C., Previdi, S., Dawra, G., Aries, E., and D. 1022 Afanasiev, "Segment Routing Centralized BGP Egress Peer 1023 Engineering", draft-ietf-spring-segment-routing-central- 1024 epe-10 (work in progress), December 2017. 1026 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1027 Requirement Levels", BCP 14, RFC 2119, 1028 DOI 10.17487/RFC2119, March 1997, 1029 . 1031 [RFC4271] Rekhter, Y., Ed., Li, T., Ed., and S. Hares, Ed., "A 1032 Border Gateway Protocol 4 (BGP-4)", RFC 4271, 1033 DOI 10.17487/RFC4271, January 2006, 1034 . 1036 [RFC7938] Lapukhov, P., Premji, A., and J. Mitchell, Ed., "Use of 1037 BGP for Routing in Large-Scale Data Centers", RFC 7938, 1038 DOI 10.17487/RFC7938, August 2016, 1039 . 1041 [RFC8277] Rosen, E., "Using BGP to Bind MPLS Labels to Address 1042 Prefixes", RFC 8277, DOI 10.17487/RFC8277, October 2017, 1043 . 1045 15.2. Informative References 1047 [GREENBERG09] 1048 Greenberg, A., Hamilton, J., Jain, N., Kadula, S., Kim, 1049 C., Lahiri, P., Maltz, D., Patel, P., and S. Sengupta, 1050 "VL2: A Scalable and Flexible Data Center Network", 2009. 1052 [I-D.ietf-6man-segment-routing-header] 1053 Previdi, S., Filsfils, C., Raza, K., Leddy, J., Field, B., 1054 daniel.voyer@bell.ca, d., daniel.bernier@bell.ca, d., 1055 Matsushima, S., Leung, I., Linkova, J., Aries, E., Kosugi, 1056 T., Vyncke, E., Lebrun, D., Steinberg, D., and R. Raszuk, 1057 "IPv6 Segment Routing Header (SRH)", draft-ietf-6man- 1058 segment-routing-header-07 (work in progress), July 2017. 1060 [KANDULA04] 1061 Sinha, S., Kandula, S., and D. Katabi, "Harnessing TCP's 1062 Burstiness with Flowlet Switching", 2004. 1064 [RFC6793] Vohra, Q. and E. Chen, "BGP Support for Four-Octet 1065 Autonomous System (AS) Number Space", RFC 6793, 1066 DOI 10.17487/RFC6793, December 2012, 1067 . 1069 Authors' Addresses 1071 Clarence Filsfils (editor) 1072 Cisco Systems, Inc. 1073 Brussels 1074 BE 1076 Email: cfilsfil@cisco.com 1078 Stefano Previdi 1079 Cisco Systems, Inc. 1080 Italy 1082 Email: stefano@previdi.net 1084 Jon Mitchell 1085 Unaffiliated 1087 Email: jrmitche@puck.nether.net 1089 Ebben Aries 1090 Juniper Networks 1091 1133 Innovation Way 1092 Sunnyvale CA 94089 1093 US 1095 Email: exa@juniper.net 1097 Petr Lapukhov 1098 Facebook 1099 US 1101 Email: petr@fb.com