idnits 2.17.1 draft-ietf-spring-segment-routing-msdc-06.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document doesn't use any RFC 2119 keywords, yet seems to have RFC 2119 boilerplate text. -- The document date (October 30, 2017) is 2364 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- -- Looks like a reference, but probably isn't: '16000' on line 276 -- Looks like a reference, but probably isn't: '23999' on line 276 -- Looks like a reference, but probably isn't: '1000' on line 892 -- Looks like a reference, but probably isn't: '1999' on line 892 -- Looks like a reference, but probably isn't: '2000' on line 893 -- Looks like a reference, but probably isn't: '2999' on line 893 == Unused Reference: 'RFC7311' is defined on line 1047, but no explicit reference was found in the text == Unused Reference: 'I-D.ietf-mpls-seamless-mpls' is defined on line 1072, but no explicit reference was found in the text == Outdated reference: A later version (-27) exists of draft-ietf-idr-bgp-prefix-sid-07 == Outdated reference: A later version (-15) exists of draft-ietf-spring-segment-routing-12 == Outdated reference: A later version (-10) exists of draft-ietf-spring-segment-routing-central-epe-06 ** Obsolete normative reference: RFC 3107 (Obsoleted by RFC 8277) == Outdated reference: A later version (-26) exists of draft-ietf-6man-segment-routing-header-07 Summary: 1 error (**), 0 flaws (~~), 8 warnings (==), 7 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group C. Filsfils, Ed. 3 Internet-Draft S. Previdi, Ed. 4 Intended status: Informational Cisco Systems, Inc. 5 Expires: May 3, 2018 J. Mitchell 6 Unaffiliated 7 E. Aries 8 Juniper Networks 9 P. Lapukhov 10 Facebook 11 October 30, 2017 13 BGP-Prefix Segment in large-scale data centers 14 draft-ietf-spring-segment-routing-msdc-06 16 Abstract 18 This document describes the motivation and benefits for applying 19 segment routing in BGP-based large-scale data-centers. It describes 20 the design to deploy segment routing in those data-centers, for both 21 the MPLS and IPv6 dataplanes. 23 Requirements Language 25 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 26 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 27 document are to be interpreted as described in RFC 2119 [RFC2119]. 29 Status of This Memo 31 This Internet-Draft is submitted in full conformance with the 32 provisions of BCP 78 and BCP 79. 34 Internet-Drafts are working documents of the Internet Engineering 35 Task Force (IETF). Note that other groups may also distribute 36 working documents as Internet-Drafts. The list of current Internet- 37 Drafts is at https://datatracker.ietf.org/drafts/current/. 39 Internet-Drafts are draft documents valid for a maximum of six months 40 and may be updated, replaced, or obsoleted by other documents at any 41 time. It is inappropriate to use Internet-Drafts as reference 42 material or to cite them other than as "work in progress." 44 This Internet-Draft will expire on May 3, 2018. 46 Copyright Notice 48 Copyright (c) 2017 IETF Trust and the persons identified as the 49 document authors. All rights reserved. 51 This document is subject to BCP 78 and the IETF Trust's Legal 52 Provisions Relating to IETF Documents 53 (https://trustee.ietf.org/license-info) in effect on the date of 54 publication of this document. Please review these documents 55 carefully, as they describe your rights and restrictions with respect 56 to this document. Code Components extracted from this document must 57 include Simplified BSD License text as described in Section 4.e of 58 the Trust Legal Provisions and are provided without warranty as 59 described in the Simplified BSD License. 61 Table of Contents 63 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 64 2. Large Scale Data Center Network Design Summary . . . . . . . 3 65 2.1. Reference design . . . . . . . . . . . . . . . . . . . . 4 66 3. Some open problems in large data-center networks . . . . . . 5 67 4. Applying Segment Routing in the DC with MPLS dataplane . . . 6 68 4.1. BGP Prefix Segment (BGP-Prefix-SID) . . . . . . . . . . . 6 69 4.2. eBGP Labeled Unicast (draft-ietf-mpls-rfc3107bis) . . . . 7 70 4.2.1. Control Plane . . . . . . . . . . . . . . . . . . . . 8 71 4.2.2. Data Plane . . . . . . . . . . . . . . . . . . . . . 9 72 4.2.3. Network Design Variation . . . . . . . . . . . . . . 10 73 4.2.4. Global BGP Prefix Segment through the fabric . . . . 10 74 4.2.5. Incremental Deployments . . . . . . . . . . . . . . . 11 75 4.3. iBGP Labeled Unicast (draft-ietf-mpls-rfc3107bis) . . . . 12 76 5. Applying Segment Routing in the DC with IPv6 dataplane . . . 14 77 6. Communicating path information to the host . . . . . . . . . 14 78 7. Addressing the open problems . . . . . . . . . . . . . . . . 15 79 7.1. Per-packet and flowlet switching . . . . . . . . . . . . 15 80 7.2. Performance-aware routing . . . . . . . . . . . . . . . . 16 81 7.3. Deterministic network probing . . . . . . . . . . . . . . 17 82 8. Additional Benefits . . . . . . . . . . . . . . . . . . . . . 17 83 8.1. MPLS Dataplane with operational simplicity . . . . . . . 18 84 8.2. Minimizing the FIB table . . . . . . . . . . . . . . . . 18 85 8.3. Egress Peer Engineering . . . . . . . . . . . . . . . . . 18 86 8.4. Anycast . . . . . . . . . . . . . . . . . . . . . . . . . 19 87 9. Preferred SRGB Allocation . . . . . . . . . . . . . . . . . . 19 88 10. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 20 89 11. Manageability Considerations . . . . . . . . . . . . . . . . 20 90 12. Security Considerations . . . . . . . . . . . . . . . . . . . 21 91 13. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 21 92 14. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 21 93 15. References . . . . . . . . . . . . . . . . . . . . . . . . . 22 94 15.1. Normative References . . . . . . . . . . . . . . . . . . 22 95 15.2. Informative References . . . . . . . . . . . . . . . . . 23 96 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 24 98 1. Introduction 100 Segment Routing (SR), as described in 101 [I-D.ietf-spring-segment-routing] leverages the source routing 102 paradigm. A node steers a packet through an ordered list of 103 instructions, called segments. A segment can represent any 104 instruction, topological or service-based. A segment can have a 105 local semantic to an SR node or global within an SR domain. SR 106 allows to enforce a flow through any topological path while 107 maintaining per-flow state only at the ingress node to the SR domain. 108 Segment Routing can be applied to the MPLS and IPv6 data-planes. 110 The use-cases described in this document should be considered in the 111 context of the BGP-based large-scale data-center (DC) design 112 described in [RFC7938]. This document extends it by applying SR both 113 with IPv6 and MPLS dataplane. 115 2. Large Scale Data Center Network Design Summary 117 This section provides a brief summary of the informational document 118 [RFC7938] that outlines a practical network design suitable for data- 119 centers of various scales: 121 o Data-center networks have highly symmetric topologies with 122 multiple parallel paths between two server attachment points. The 123 well-known Clos topology is most popular among the operators (as 124 described in [RFC7938]). In a Clos topology, the minimum number 125 of parallel paths between two elements is determined by the 126 "width" of the "Tier-1" stage. See Figure 1 below for an 127 illustration of the concept. 129 o Large-scale data-centers commonly use a routing protocol, such as 130 BGP-4 [RFC4271] in order to provide endpoint connectivity. 131 Recovery after a network failure is therefore driven either by 132 local knowledge of directly available backup paths or by 133 distributed signaling between the network devices. 135 o Within data-center networks, traffic is load-shared using the 136 Equal Cost Multipath (ECMP) mechanism. With ECMP, every network 137 device implements a pseudo-random decision, mapping packets to one 138 of the parallel paths by means of a hash function calculated over 139 certain parts of the packet, typically a combination of various 140 packet header fields. 142 The following is a schematic of a five-stage Clos topology, with four 143 devices in the "Tier-1" stage. Notice that number of paths between 144 Node1 and Node12 equals to four: the paths have to cross all of 145 Tier-1 devices. At the same time, the number of paths between Node1 146 and Node2 equals two, and the paths only cross Tier-2 devices. Other 147 topologies are possible, but for simplicity only the topologies that 148 have a single path from Tier-1 to Tier-3 are considered below. The 149 rest could be treated similarly, with a few modifications to the 150 logic. 152 2.1. Reference design 154 Tier-1 155 +-----+ 156 |NODE | 157 +->| 5 |--+ 158 | +-----+ | 159 Tier-2 | | Tier-2 160 +-----+ | +-----+ | +-----+ 161 +------------>|NODE |--+->|NODE |--+--|NODE |-------------+ 162 | +-----| 3 |--+ | 6 | +--| 9 |-----+ | 163 | | +-----+ +-----+ +-----+ | | 164 | | | | 165 | | +-----+ +-----+ +-----+ | | 166 | +-----+---->|NODE |--+ |NODE | +--|NODE |-----+-----+ | 167 | | | +---| 4 |--+->| 7 |--+--| 10 |---+ | | | 168 | | | | +-----+ | +-----+ | +-----+ | | | | 169 | | | | | | | | | | 170 +-----+ +-----+ | +-----+ | +-----+ +-----+ 171 |NODE | |NODE | Tier-3 +->|NODE |--+ Tier-3 |NODE | |NODE | 172 | 1 | | 2 | | 8 | | 11 | | 12 | 173 +-----+ +-----+ +-----+ +-----+ +-----+ 174 | | | | | | | | 175 A O B O <- Servers -> Z O O O 177 Figure 1: 5-stage Clos topology 179 In the reference topology illustrated in Figure 1, It is assumed: 181 o Each node is its own AS (Node X has AS X). 4-byte AS numbers are 182 recommended ([RFC6793]). 184 * For simple and efficient route propagation filtering, Node5, 185 Node6, Node7 and Node8 use the same AS, Node3 and Node4 use the 186 same AS, Node9 and Node10 use the same AS. 188 * In case of 2-byte autonomous system numbers are used and for 189 efficient usage of the scarce 2-byte Private Use AS pool, 190 different Tier-3 nodes might use the same AS. 192 * Without loss of generality, these details will be simplified in 193 this document and assume that each node has its own AS. 195 o Each node peers with its neighbors with a BGP session. If not 196 specified, eBGP is assumed. In a specific use-case, iBGP will be 197 used but this will be called out explicitly in that case. 199 o Each node originates the IPv4 address of its loopback interface 200 into BGP and announces it to its neighbors. 202 * The loopback of Node X is 192.0.2.x/32. 204 In this document, the Tier-1, Tier-2 and Tier-3 nodes are referred to 205 respectively as Spine, Leaf and ToR (top of rack) nodes. When a ToR 206 node acts as a gateway to the "outside world", it is referred to as a 207 border node. 209 3. Some open problems in large data-center networks 211 The data-center network design summarized above provides means for 212 moving traffic between hosts with reasonable efficiency. There are 213 few open performance and reliability problems that arise in such 214 design: 216 o ECMP routing is most commonly realized per-flow. This means that 217 large, long-lived "elephant" flows may affect performance of 218 smaller, short-lived "mouse" flows and reduce efficiency of per- 219 flow load-sharing. In other words, per-flow ECMP does not perform 220 efficiently when flow lifetime distribution is heavy-tailed. 221 Furthermore, due to hash-function inefficiencies it is possible to 222 have frequent flow collisions, where more flows get placed on one 223 path over the others. 225 o Shortest-path routing with ECMP implements an oblivious routing 226 model, which is not aware of the network imbalances. If the 227 network symmetry is broken, for example due to link failures, 228 utilization hotspots may appear. For example, if a link fails 229 between Tier-1 and Tier-2 devices (e.g. Node5 and Node9), Tier-3 230 devices Node1 and Node2 will not be aware of that, since there are 231 other paths available from perspective of Node3. They will 232 continue sending roughly equal traffic to Node3 and Node4 as if 233 the failure didn't exist which may cause a traffic hotspot. 235 o The absence of path visibility leaves transport protocols, such as 236 TCP, with a "blackbox" view of the network. Some TCP metrics, 237 such as SRTT, MSS, CWND and few others could be inferred and 238 cached based on past history, but those apply to destinations, 239 regardless of the path that has been chosen to get there. Thus, 240 for instance, TCP is not capable of remembering "bad" paths, such 241 as those that exhibited poor performance in the past. This means 242 that every new connection will be established obliviously (memory- 243 less) with regards to the paths chosen before, or chosen by other 244 nodes. 246 o Isolating faults in the network with multiple parallel paths and 247 ECMP-based routing is non-trivial due to lack of determinism. 248 Specifically, the connections from HostA to HostB may take a 249 different path every time a new connection is formed, thus making 250 consistent reproduction of a failure much more difficult. This 251 complexity scales linearly with the number of parallel paths in 252 the network, and stems from the random nature of path selection by 253 the network devices. 255 Further in this document (Section 7), it is demonstrated how these 256 problems could be addressed within the framework of Segment Routing. 258 First, it will be explained how to apply SR in the DC, for MPLS and 259 IPv6 data-planes. 261 4. Applying Segment Routing in the DC with MPLS dataplane 263 4.1. BGP Prefix Segment (BGP-Prefix-SID) 265 A BGP Prefix Segment is a segment associated with a BGP prefix. A 266 BGP Prefix Segment is a network-wide instruction to forward the 267 packet along the ECMP-aware best path to the related prefix. 269 The BGP Prefix Segment is defined as the BGP-Prefix-SID Attribute in 270 [I-D.ietf-idr-bgp-prefix-sid] which contains an index. Throughout 271 this document the BGP Prefix Segment Attribute is referred as the 272 BGP-Prefix-SID and the encoded index as the label-index. 274 In this document, the network design decision has been made to assume 275 that all the nodes are allocated the same SRGB (Segment Routing 276 Global Block), e.g. [16000, 23999]. This provides operational 277 simplification as explained in Section 9, but this is not a 278 requirement. 280 For illustration purpose, when considering an MPLS data-plane, it is 281 assumed that the label-index allocated to prefix 192.0.2.x/32 is X. 283 As a result, a local label (16000+x) is allocated for prefix 284 192.0.2.x/32 by each node throughout the DC fabric. 286 When IPv6 data-plane is considered, it is assumed that Node X is 287 allocated IPv6 address (segment) 2001:DB8::X. 289 4.2. eBGP Labeled Unicast (draft-ietf-mpls-rfc3107bis) 291 Referring to Figure 1 and [RFC7938], the following design 292 modifications are introduced: 294 o Each node peers with its neighbors via a eBGP session with 295 extensions defined in [I-D.ietf-mpls-rfc3107bis] (named "eBGP3107" 296 throughout this document) and with the BGP-Prefix-SID attribute 297 extension as defined in [I-D.ietf-idr-bgp-prefix-sid]. 299 o The forwarding plane at Tier-2 and Tier-1 is MPLS. 301 o The forwarding plane at Tier-3 is either IP2MPLS (if the host 302 sends IP traffic) or MPLS2MPLS (if the host sends MPLS- 303 encapsulated traffic). 305 Figure 2 zooms into a path from server A to server Z within the 306 topology of Figure 1. 308 +-----+ +-----+ +-----+ 309 +---------->|NODE | |NODE | |NODE | 310 | | 4 |--+->| 7 |--+--| 10 |---+ 311 | +-----+ +-----+ +-----+ | 312 | | 313 +-----+ +-----+ 314 |NODE | |NODE | 315 | 1 | | 11 | 316 +-----+ +-----+ 317 | | 318 A <- Servers -> Z 320 Figure 2: Path from A to Z via nodes 1, 4, 7, 10 and 11 322 Referring to Figure 1 and Figure 2 and assuming the IP address with 323 the AS and label-index allocation previously described, the following 324 sections detail the control plane operation and the data plane states 325 for the prefix 192.0.2.11/32 (loopback of Node11) 327 4.2.1. Control Plane 329 Node11 originates 192.0.2.11/32 in BGP and allocates to it a BGP- 330 Prefix-SID with label-index: index11 [I-D.ietf-idr-bgp-prefix-sid]. 332 Node11 sends the following eBGP3107 update to Node10: 334 . IP Prefix: 192.0.2.11/32 335 . Label: Implicit-Null 336 . Next-hop: Node11's interface address on the link to Node10 337 . AS Path: {11} 338 . BGP-Prefix-SID: Label-Index 11 340 Node10 receives the above update. As it is SR capable, Node10 is 341 able to interpret the BGP-Prefix-SID and hence understands that it 342 should allocate the label from its own SRGB block, offset by the 343 Label-Index received in the BGP-Prefix-SID (16000+11 hence 16011) to 344 the NLRI instead of allocating a non-deterministic label out of a 345 dynamically allocated portion of the local label space. The 346 implicit-null label in the NLRI tells Node10 that it is the 347 penultimate hop and must pop the top label on the stack before 348 forwarding traffic for this prefix to Node11. 350 Then, Node10 sends the following eBGP3107 update to Node7: 352 . IP Prefix: 192.0.2.11/32 353 . Label: 16011 354 . Next-hop: Node10's interface address on the link to Node7 355 . AS Path: {10, 11} 356 . BGP-Prefix-SID: Label-Index 11 358 Node7 receives the above update. As it is SR capable, Node7 is able 359 to interpret the BGP-Prefix-SID and hence allocates the local 360 (incoming) label 16011 (16000 + 11) to the NLRI (instead of 361 allocating a "dynamic" local label from its label manager). Node7 362 uses the label in the received eBGP3107 NLRI as the outgoing label 363 (the index is only used to derive the local/incoming label). 365 Node7 sends the following eBGP3107 update to Node4: 367 . IP Prefix: 192.0.2.11/32 368 . Label: 16011 369 . Next-hop: Node7's interface address on the link to Node4 370 . AS Path: {7, 10, 11} 371 . BGP-Prefix-SID: Label-Index 11 373 Node4 receives the above update. As it is SR capable, Node4 is able 374 to interpret the BGP-Prefix-SID and hence allocates the local 375 (incoming) label 16011 to the NLRI (instead of allocating a "dynamic" 376 local label from its label manager). Node4 uses the label in the 377 received eBGP3107 NLRI as outgoing label (the index is only used to 378 derive the local/incoming label). 380 Node4 sends the following eBGP3107 update to Node1: 382 . IP Prefix: 192.0.2.11/32 383 . Label: 16011 384 . Next-hop: Node4's interface address on the link to Node1 385 . AS Path: {4, 7, 10, 11} 386 . BGP-Prefix-SID: Label-Index 11 388 Node1 receives the above update. As it is SR capable, Node1 is able 389 to interpret the BGP-Prefix-SID and hence allocates the local 390 (incoming) label 16011 to the NLRI (instead of allocating a "dynamic" 391 local label from its label manager). Node1 uses the label in the 392 received eBGP3107 NLRI as outgoing label (the index is only used to 393 derive the local/incoming label). 395 4.2.2. Data Plane 397 Referring to Figure 1, and assuming all nodes apply the same 398 advertisement rules described above and all nodes have the same SRGB 399 (16000-23999), here are the IP/MPLS forwarding tables for prefix 400 192.0.2.11/32 at Node1, Node4, Node7 and Node10. 402 ----------------------------------------------- 403 Incoming label | outgoing label | Outgoing 404 or IP destination | | Interface 405 ------------------+----------------+----------- 406 16011 | 16011 | ECMP{3, 4} 407 192.0.2.11/32 | 16011 | ECMP{3, 4} 408 ------------------+----------------+----------- 410 Figure 3: Node1 Forwarding Table 412 ----------------------------------------------- 413 Incoming label | outgoing label | Outgoing 414 or IP destination | | Interface 415 ------------------+----------------+----------- 416 16011 | 16011 | ECMP{7, 8} 417 192.0.2.11/32 | 16011 | ECMP{7, 8} 418 ------------------+----------------+----------- 420 Figure 4: Node4 Forwarding Table 422 ----------------------------------------------- 423 Incoming label | outgoing label | Outgoing 424 or IP destination | | Interface 425 ------------------+----------------+----------- 426 16011 | 16011 | 10 427 192.0.2.11/32 | 16011 | 10 428 ------------------+----------------+----------- 430 Figure 5: Node7 Forwarding Table 432 ----------------------------------------------- 433 Incoming label | outgoing label | Outgoing 434 or IP destination | | Interface 435 ------------------+----------------+----------- 436 16011 | POP | 11 437 192.0.2.11/32 | N/A | 11 438 ------------------+----------------+----------- 440 Node10 Forwarding Table 442 4.2.3. Network Design Variation 444 A network design choice could consist of switching all the traffic 445 through Tier-1 and Tier-2 as MPLS traffic. In this case, one could 446 filter away the IP entries at Node4, Node7 and Node10. This might be 447 beneficial in order to optimize the forwarding table size. 449 A network design choice could consist in allowing the hosts to send 450 MPLS-encapsulated traffic based on the Egress Peer Engineering (EPE) 451 use-case as defined in [I-D.ietf-spring-segment-routing-central-epe]. 452 For example, applications at HostA would send their Z-destined 453 traffic to Node1 with an MPLS label stack where the top label is 454 16011 and the next label is an EPE peer segment 455 ([I-D.ietf-spring-segment-routing-central-epe]) at Node11 directing 456 the traffic to Z. 458 4.2.4. Global BGP Prefix Segment through the fabric 460 When the previous design is deployed, the operator enjoys global BGP- 461 Prefix-SID and label allocation throughout the DC fabric. 463 A few examples follow: 465 o Normal forwarding to Node11: a packet with top label 16011 466 received by any node in the fabric will be forwarded along the 467 ECMP-aware BGP best-path towards Node11 and the label 16011 is 468 penultimate-popped at Node10 (or at Node 9). 470 o Traffic-engineered path to Node11: an application on a host behind 471 Node1 might want to restrict its traffic to paths via the Spine 472 node Node5. The application achieves this by sending its packets 473 with a label stack of {16005, 16011}. BGP Prefix SID 16005 directs 474 the packet up to Node5 along the path (Node1, Node3, Node5). BGP- 475 Prefix-SID 16011 then directs the packet down to Node11 along the 476 path (Node5, Node9, Node11). 478 4.2.5. Incremental Deployments 480 The design previously described can be deployed incrementally. Let 481 us assume that Node7 does not support the BGP-Prefix-SID and let us 482 show how the fabric connectivity is preserved. 484 From a signaling viewpoint, nothing would change: even though Node7 485 does not support the BGP-Prefix-SID, it does propagate the attribute 486 unmodified to its neighbors. 488 From a label allocation viewpoint, the only difference is that Node7 489 would allocate a dynamic (random) label to the prefix 192.0.2.11/32 490 (e.g. 123456) instead of the "hinted" label as instructed by the BGP- 491 Prefix-SID. The neighbors of Node7 adapt automatically as they 492 always use the label in the BGP3107 NLRI as outgoing label. 494 Node4 does understand the BGP-Prefix-SID and hence allocates the 495 indexed label in the SRGB (16011) for 192.0.2.11/32. 497 As a result, all the data-plane entries across the network would be 498 unchanged except the entries at Node7 and its neighbor Node4 as shown 499 in the figures below. 501 The key point is that the end-to-end Label Switched Path (LSP) is 502 preserved because the outgoing label is always derived from the 503 received label within the BGP3107 NLRI. The index in the BGP-Prefix- 504 SID is only used as a hint on how to allocate the local label (the 505 incoming label) but never for the outgoing label. 507 ------------------------------------------ 508 Incoming label | outgoing | Outgoing 509 or IP destination | label | Interface 510 -------------------+---------------------- 511 12345 | 16011 | 10 513 Figure 7: Node7 Forwarding Table 515 ------------------------------------------ 516 Incoming label | outgoing | Outgoing 517 or IP destination | label | Interface 518 -------------------+---------------------- 519 16011 | 12345 | 7 521 Figure 8: Node4 Forwarding Table 523 The BGP-Prefix-SID can thus be deployed incrementally one node at a 524 time. 526 When deployed together with a homogeneous SRGB (same SRGB across the 527 fabric), the operator incrementally enjoys the global prefix segment 528 benefits as the deployment progresses through the fabric. 530 4.3. iBGP Labeled Unicast (draft-ietf-mpls-rfc3107bis) 532 The same exact design as eBGP3107 is used with the following 533 modifications: 535 All nodes use the same AS number. 537 Each node peers with its neighbors via an internal BGP session 538 (iBGP) with extensions defined in [I-D.ietf-mpls-rfc3107bis] 539 (named "iBGP3107" throughout this document) and with the BGP- 540 Prefix-SID attribute extension defined in this document. 542 Each node acts as a route-reflector for each of its neighbors and 543 with the next-hop-self option. Next-hop-self is a well known 544 operational feature which consists of rewriting the next-hop of a 545 BGP update prior to send it to the neighbor. Usually, it's a 546 common practice to apply next-hop-self behavior towards iBGP peers 547 for eBGP learned routes. In the case outlined in this section it 548 is proposed to use the next-hop-self mechanism also to iBGP 549 learned routes. 551 Cluster-1 552 +-----------+ 553 | Tier-1 | 554 | +-----+ | 555 | |NODE | | 556 | | 5 | | 557 Cluster-2 | +-----+ | Cluster-3 558 +---------+ | | +---------+ 559 | Tier-2 | | | | Tier-2 | 560 | +-----+ | | +-----+ | | +-----+ | 561 | |NODE | | | |NODE | | | |NODE | | 562 | | 3 | | | | 6 | | | | 9 | | 563 | +-----+ | | +-----+ | | +-----+ | 564 | | | | | | 565 | | | | | | 566 | +-----+ | | +-----+ | | +-----+ | 567 | |NODE | | | |NODE | | | |NODE | | 568 | | 4 | | | | 7 | | | | 10 | | 569 | +-----+ | | +-----+ | | +-----+ | 570 +---------+ | | +---------+ 571 | | 572 | +-----+ | 573 | |NODE | | 574 Tier-3 | | 8 | | Tier-3 575 +-----+ +-----+ | +-----+ | +-----+ +-----+ 576 |NODE | |NODE | +-----------+ |NODE | |NODE | 577 | 1 | | 2 | | 11 | | 12 | 578 +-----+ +-----+ +-----+ +-----+ 580 Figure 9: iBGP Sessions with Reflection and Next-Hop-Self 582 For simple and efficient route propagation filtering and as 583 illustrated in Figure 9: 585 Node5, Node6, Node7 and Node8 use the same Cluster ID (Cluster- 586 1) 588 Node3 and Node4 use the same Cluster ID (Cluster-2) 590 Node9 and Node10 use the same Cluster ID (Cluster-3) 592 The control-plane behavior is mostly the same as described in the 593 previous section: the only difference is that the eBGP3107 path 594 propagation is simply replaced by an iBGP3107 path reflection with 595 next-hop changed to self. 597 The data-plane tables are exactly the same. 599 5. Applying Segment Routing in the DC with IPv6 dataplane 601 The design described in [RFC7938] is reused with one single 602 modification. It is highlighted using the example of the 603 reachability to Node11 via spine node Node5. 605 Node5 originates 2001:DB8::5/128 with the attached BGP-Prefix-SID for 606 IPv6 packets destined to segment 2001:DB8::5 607 ([I-D.ietf-idr-bgp-prefix-sid]). 609 Node11 originates 2001:DB8::11/128 with the attached BGP-Prefix-SID 610 advertising the support of the SRH for IPv6 packets destined to 611 segment 2001:DB8::11. 613 The control-plane and data-plane processing of all the other nodes in 614 the fabric is unchanged. Specifically, the routes to 2001:DB8::5 and 615 2001:DB8::11 are installed in the FIB along the eBGP best-path to 616 Node5 (spine node) and Node11 (ToR node) respectively. 618 An application on HostA which needs to send traffic to HostZ via only 619 Node5 (spine node) can do so by sending IPv6 packets with a Segment 620 Routing header (SRH, [I-D.ietf-6man-segment-routing-header]). The 621 destination address and active segment is set to 2001:DB8::5. The 622 next and last segment is set to 2001:DB8::11. 624 The application must only use IPv6 addresses that have been 625 advertised as capable for SRv6 segment processing (e.g. for which the 626 BGP prefix segment capability has been advertised). How applications 627 learn this (e.g.: centralized controller and orchestration) is 628 outside the scope of this document. 630 6. Communicating path information to the host 632 There are two general methods for communicating path information to 633 the end-hosts: "proactive" and "reactive", aka "push" and "pull" 634 models. There are multiple ways to implement either of these 635 methods. Here, it is noted that one way could be using a centralized 636 controller: the controller either tells the hosts of the prefix-to- 637 path mappings beforehand and updates them as needed (network event 638 driven push), or responds to the hosts making request for a path to 639 specific destination (host event driven pull). It is also possible 640 to use a hybrid model, i.e., pushing some state from the controller 641 in response to particular network events, while the host pulls other 642 state on demand. 644 It is also noted, that when disseminating network-related data to the 645 end-hosts a trade-off is made to balance the amount of information 646 Vs. the level of visibility in the network state. This applies both 647 to push and pull models. In the extreme case, the host would request 648 path information on every flow, and keep no local state at all. On 649 the other end of the spectrum, information for every prefix in the 650 network along with available paths could be pushed and continuously 651 updated on all hosts. 653 7. Addressing the open problems 655 This section demonstrates how the problems described above (in 656 section 3) could be solved using the segment routing concept. It is 657 worth noting that segment routing signaling and data-plane are only 658 parts of the solution. Additional enhancements, e.g., such as the 659 centralized controller mentioned previously, and host networking 660 stack support are required to implement the proposed solutions. Also 661 the applicability of the solutions described below are not restricted 662 to the data-center alone, the same could be re-used in context of 663 other domains as well. 665 7.1. Per-packet and flowlet switching 667 A flowlet is defined as a burst of packets from the same flow 668 followed by an idle interval. [KANDULA04] developed a scheme that 669 uses flowlets to split traffic across multiple parallel paths in 670 order to optimize traffic load sharing. 672 With the ability to choose paths on the host, one may go from per- 673 flow load-sharing in the network to per-packet or per-flowlet. The 674 host may select different segment routing instructions either per 675 packet, or per flowlet, and route them over different paths. This 676 allows for solving the "elephant flow" problem in the data-center and 677 avoiding link imbalances. 679 Note that traditional ECMP routing could be easily simulated with on- 680 host path selection, using method proposed in [GREENBERG09]. The 681 hosts would randomly pick a Tier-2 or Tier-1 device to "bounce" the 682 packet off of, depending on whether the destination is under the same 683 Tier-2 nodes, or has to be reached across Tier-1. The host would use 684 a hash function that operates on per-flow invariants, to simulate 685 per-flow load-sharing in the network. 687 Using Figure 1 as reference, let us illustrate this concept assuming 688 that HostA has an elephant flow to HostZ called Flow-f. 690 Normally, a flow is hashed on to a single path. Let's assume HostA 691 sends its packets associated with Flow-f with top label 16011 (the 692 label for the remote ToR, Node11, where HostZ is connected) and Node1 693 would hash all the packets of Flow-F via the same next-hop (e.g. 694 Node3). Similarly, let's assume that leaf Node3 would hash all the 695 packets of Flow-F via the same next-hop (e.g.: spine node Node5). 696 This normal operation would restrict the elephant flow on a small 697 subset of the ECMP paths to HostZ and potentially create imbalance 698 and congestion in the fabric. 700 Leveraging the flowlet proposal, assuming HostA is made aware of 4 701 disjoint paths via intermediate segment 16005, 16006, 16007 and 16008 702 (the BGP prefix SID's of the 4 spine nodes) and also made aware of 703 the prefix segment of the remote ToR connected to the destination 704 (16011), then the application can break the elephant flow F into 705 flowlets F1, F2, F3, F4 and associate each flowlet with one of the 706 following 4 label stacks: {16005, 16011}, {16006, 16011}, {16007, 707 16011} and {16008, 16011}. This would spread the load of the elephant 708 flow through all the ECMP paths available in the fabric and re- 709 balance the load. 711 7.2. Performance-aware routing 713 Knowing the path associated with flows/packets, the end host may 714 deduce certain characteristics of the path on its own, and 715 additionally use the information supplied with path information 716 pushed from the controller or received via pull request. The host 717 may further share its path observations with the centralized agent, 718 so that the latter may keep up-to-date network health map to assist 719 other hosts with this information. 721 For example, an application A.1 at HostA may pin a TCP flow destined 722 to HostZ via Spine node Node5 using label stack {16005, 16011}. The 723 application A.1 may collect information on packet loss, deduced from 724 TCP retransmissions and other signals (e.g. RTT increases). A.1 may 725 additionally publish this information to a centralized agent, e.g. 726 after a flow completes, or periodically for longer lived flows. 727 Next, using both local and/or global performance data, application 728 A.1 as well as other applications sharing the same resources in the 729 DC fabric may pick up the best path for the new flow, or update an 730 existing path (e.g.: when informed of congestion on an existing 731 path). 733 One particularly interesting instance of performance-aware routing is 734 dynamic fault-avoidance. If some links or devices in the network 735 start discarding packets due to a fault, the end-hosts could probe 736 and detect the path(s) that are affected and hence steer the affected 737 flows away from the problem spot. Similar logic applies to failure 738 cases where packets get completely black-holed, e.g., when a link 739 goes down and the failure is detected by the host while probing the 740 path. 742 For example, an application A.1 informed about 5 paths to Z {16005, 743 16011}, {16006, 16011}, {16007, 16011}, {16008, 16011} and {16011} 744 might use the last one by default (for simplicity). When performance 745 is degrading, A.1 might then start to pin TCP flows to each of the 4 746 other paths (each via a distinct spine) and monitor the performance. 747 It would then detect the faulty path and assign a negative preference 748 to the faulty path to avoid further flows using it. Gradually, over 749 time, it may re-assign flows on the faulty path to eventually detect 750 the resolution of the trouble and start reusing the path. 752 By leveraging Segment Routing, one avoids issues associated with 753 oblivious ECMP hashing. For example, if in the topology depicted on 754 Figure 1 a link between spine node Node5 and leaf node Node9 fails, 755 HostA may exclude the segment corresponding to Node5 from the prefix 756 matching the servers under Tier-2 devices Node9. In the push path 757 discovery model, the affected path mappings may be explicitly pushed 758 to all the servers for the duration of the failure. The new mapping 759 would instruct them to avoid the particular Tier-1 node until the 760 link has recovered. Alternatively, in pull path, the centralized 761 controller may start steering new flows immediately after it 762 discovers the issue. Until then, the existing flows may recover 763 using local detection of the path issues. 765 7.3. Deterministic network probing 767 Active probing is a well-known technique for monitoring network 768 elements' health, constituting of sending continuous packet streams 769 simulating network traffic to the hosts in the data-center. Segment 770 routing makes possible to prescribe the exact paths that each probe 771 or series of probes would be taking toward their destination. This 772 allows for fast correlation and detection of failed paths, by 773 processing information from multiple actively probing agents. This 774 complements the data collected from the hosts routing stacks as 775 described in Section 7.2. 777 For example, imagine a probe agent sending packets to all machines in 778 the data-center. For every host, it may send packets over each of 779 the possible paths, knowing exactly which links and devices these 780 packets will be crossing. Correlating results for multiple 781 destinations with the topological data, it may automatically isolate 782 possible problem to a link or device in the network. 784 8. Additional Benefits 785 8.1. MPLS Dataplane with operational simplicity 787 As required by [RFC7938], no new signaling protocol is introduced. 788 The BGP-Prefix-SID is a lightweight extension to BGP Labeled Unicast 789 (RFC3107 [RFC3107]). It applies either to eBGP or iBGP based 790 designs. 792 Specifically, LDP and RSVP-TE are not used. These protocols would 793 drastically impact the operational complexity of the Data Center and 794 would not scale. This is in line with the requirements expressed in 795 [RFC7938]. 797 Provided the same SRGB is configured on all nodes, all nodes use the 798 same MPLS label for a given IP prefix. This is simpler from an 799 operation standpoint, as discussed in Section 9 801 8.2. Minimizing the FIB table 803 The designer may decide to switch all the traffic at Tier-1 and Tier- 804 2's based on MPLS, hence drastically decreasing the IP table size at 805 these nodes. 807 This is easily accomplished by encapsulating the traffic either 808 directly at the host or the source ToR node by pushing the BGP- 809 Prefix-SID of the destination ToR for intra-DC traffic, or the BGP- 810 Prefix-SID for the the border node for inter-DC or DC-to-outside- 811 world traffic. 813 8.3. Egress Peer Engineering 815 It is straightforward to combine the design illustrated in this 816 document with the Egress Peer Engineering (EPE) use-case described in 817 [I-D.ietf-spring-segment-routing-central-epe]. 819 In such case, the operator is able to engineer its outbound traffic 820 on a per host-flow basis, without incurring any additional state at 821 intermediate points in the DC fabric. 823 For example, the controller only needs to inject a per-flow state on 824 the HostA to force it to send its traffic destined to a specific 825 Internet destination D via a selected border node (say Node12 in 826 Figure 1 instead of another border node, Node11) and a specific 827 egress peer of Node12 (say peer AS 9999 of local PeerNode segment 828 9999 at Node12 instead of any other peer which provides a path to the 829 destination D). Any packet matching this state at host A would be 830 encapsulated with SR segment list (label stack) {16012, 9999}. 16012 831 would steer the flow through the DC fabric, leveraging any ECMP, 832 along the best path to border node Node12. Once the flow gets to 833 border node Node12, the active segment is 9999 (because of PHP on the 834 upstream neighbor of Node12). This EPE PeerNode segment forces 835 border node Node12 to forward the packet to peer AS 9999, without any 836 IP lookup at the border node. There is no per-flow state for this 837 engineered flow in the DC fabric. A benefit of segment routing is 838 the per-flow state is only required at the source. 840 As well as allowing full traffic engineering control such a design 841 also offers FIB table minimization benefits as the Internet-scale FIB 842 at border node Node12 is not required if all FIB lookups are avoided 843 there by using EPE. 845 8.4. Anycast 847 The design presented in this document preserves the availability and 848 load-balancing properties of the base design presented in 849 [I-D.ietf-spring-segment-routing]. 851 For example, one could assign an anycast loopback 192.0.2.20/32 and 852 associate segment index 20 to it on the border Node11 and Node12 (in 853 addition to their node-specific loopbacks). Doing so, the EPE 854 controller could express a default "go-to-the-Internet via any border 855 node" policy as segment list {16020}. Indeed, from any host in the DC 856 fabric or from any ToR node, 16020 steers the packet towards the 857 border Node11 or Node12 leveraging ECMP where available along the 858 best paths to these nodes. 860 9. Preferred SRGB Allocation 862 In the MPLS case, it is recommend to use same SRGBs at each node. 864 Different SRGBs in each node likely increase the complexity of the 865 solution both from an operational viewpoint and from a controller 866 viewpoint. 868 From an operation viewpoint, it is much simpler to have the same 869 global label at every node for the same destination (the MPLS 870 troubleshooting is then similar to the IPv6 troubleshooting where 871 this global property is a given). 873 From a controller viewpoint, this allows us to construct simple 874 policies applicable across the fabric. 876 Let us consider two applications A and B respectively connected to 877 Node1 and Node2 (ToR nodes). A has two flows FA1 and FA2 destined to 878 Z. B has two flows FB1 and FB2 destined to Z. The controller wants 879 FA1 and FB1 to be load-shared across the fabric while FA2 and FB2 880 must be respectively steered via Node5 and Node8. 882 Assuming a consistent unique SRGB across the fabric as described in 883 the document, the controller can simply do it by instructing A and B 884 to use {16011} respectively for FA1 and FB1 and by instructing A and 885 B to use {16005 16011} and {16008 16011} respectively for FA2 and 886 FB2. 888 Let us assume a design where the SRGB is different at every node and 889 where the SRGB of each node is advertised using the Originator SRGB 890 TLV of the BGP-Prefix-SID as defined in 891 [I-D.ietf-idr-bgp-prefix-sid]: SRGB of Node K starts at value K*1000 892 and the SRGB length is 1000 (e.g. Node1's SRGB is [1000, 1999], 893 Node2's SRGB is [2000, 2999], ...). 895 In this case, not only the controller would need to collect and store 896 all of these different SRGB's (e.g., through the Originator SRGB TLV 897 of the BGP-Prefix-SID), furthermore it would need to adapt the policy 898 for each host. Indeed, the controller would instruct A to use {1011} 899 for FA1 while it would have to instruct B to use {2011} for FB1 900 (while with the same SRGB, both policies are the same {16011}). 902 Even worse, the controller would instruct A to use {1005, 5011} for 903 FA1 while it would instruct B to use {2011, 8011} for FB1 (while with 904 the same SRGB, the second segment is the same across both policies: 905 16011). When combining segments to create a policy, one need to 906 carefully update the label of each segment. This is obviously more 907 error-prone, more complex and more difficult to troubleshoot. 909 10. IANA Considerations 911 This document does not make any IANA request. 913 11. Manageability Considerations 915 The design and deployment guidelines described in this document are 916 based on the network design described in [RFC7938]. 918 The deployment model assumed in this document is based on a single 919 domain where the interconnected DCs are part of the same 920 administrative domain (which, of course, is split into different 921 autonomous systems). The operator has full control of the whole 922 domain and the usual operational and management mechanisms and 923 procedures are used in order to prevent any information related to 924 internal prefixes and topology to be leaked outside the domain. 926 As recommended in [I-D.ietf-spring-segment-routing], the same SRGB 927 should be allocated in all nodes in order to facilitate the design, 928 deployment and operations of the domain. 930 When EPE ([I-D.ietf-spring-segment-routing-central-epe]) is used (as 931 explained in Section 8.3, the same operational model is assumed. EPE 932 information is originated and propagated throughout the domain 933 towards an internal server and unless explicitly configured by the 934 operator, no EPE information is leaked outside the domain boundaries. 936 12. Security Considerations 938 This document proposes to apply Segment Routing to a well known 939 scalability requirement expressed in [RFC7938] using the BGP-Prefix- 940 SID as defined in [I-D.ietf-idr-bgp-prefix-sid]. 942 It has to be noted, as described in Section 11 that the design 943 illustrated in [RFC7938] and in this document, refer to a deployment 944 model where all nodes are under the same administration. In this 945 context, it is assumed that the operator doesn't want to leak outside 946 of the domain any information related to internal prefixes and 947 topology. The internal information includes prefix-sid and EPE 948 information. In order to prevent such leaking, the standard BGP 949 mechanisms (filters) are applied on the boundary of the domain. 951 Therefore, the solution proposed in this document does not introduce 952 any additional security concerns from what expressed in [RFC7938] and 953 [I-D.ietf-idr-bgp-prefix-sid]. It is assumed that the security and 954 confidentiality of the prefix and topology information is preserved 955 by outbound filters at each peering point of the domain as described 956 in Section 11. 958 13. Acknowledgements 960 The authors would like to thank Benjamin Black, Arjun Sreekantiah, 961 Keyur Patel, Acee Lindem and Anoop Ghanwani for their comments and 962 review of this document. 964 14. Contributors 966 Gaya Nagarajan 967 Facebook 968 US 970 Email: gaya@fb.com 972 Dmitry Afanasiev 973 Yandex 974 RU 976 Email: fl0w@yandex-team.ru 977 Tim Laberge 978 Cisco 979 US 981 Email: tlaberge@cisco.com 983 Edet Nkposong 984 Salesforce.com Inc. 985 US 987 Email: enkposong@salesforce.com 989 Mohan Nanduri 990 Microsoft 991 US 993 Email: mnanduri@microsoft.com 995 James Uttaro 996 ATT 997 US 999 Email: ju1738@att.com 1001 Saikat Ray 1002 Unaffiliated 1003 US 1005 Email: raysaikat@gmail.com 1007 15. References 1009 15.1. Normative References 1011 [I-D.ietf-idr-bgp-prefix-sid] 1012 Previdi, S., Filsfils, C., Lindem, A., Sreekantiah, A., 1013 and H. Gredler, "Segment Routing Prefix SID extensions for 1014 BGP", draft-ietf-idr-bgp-prefix-sid-07 (work in progress), 1015 October 2017. 1017 [I-D.ietf-mpls-rfc3107bis] 1018 Rosen, E., "Using BGP to Bind MPLS Labels to Address 1019 Prefixes", draft-ietf-mpls-rfc3107bis-04 (work in 1020 progress), August 2017. 1022 [I-D.ietf-spring-segment-routing] 1023 Filsfils, C., Previdi, S., Decraene, B., Litkowski, S., 1024 and R. Shakir, "Segment Routing Architecture", draft-ietf- 1025 spring-segment-routing-12 (work in progress), June 2017. 1027 [I-D.ietf-spring-segment-routing-central-epe] 1028 Filsfils, C., Previdi, S., Aries, E., and D. Afanasiev, 1029 "Segment Routing Centralized BGP Egress Peer Engineering", 1030 draft-ietf-spring-segment-routing-central-epe-06 (work in 1031 progress), June 2017. 1033 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1034 Requirement Levels", BCP 14, RFC 2119, 1035 DOI 10.17487/RFC2119, March 1997, 1036 . 1038 [RFC3107] Rekhter, Y. and E. Rosen, "Carrying Label Information in 1039 BGP-4", RFC 3107, DOI 10.17487/RFC3107, May 2001, 1040 . 1042 [RFC4271] Rekhter, Y., Ed., Li, T., Ed., and S. Hares, Ed., "A 1043 Border Gateway Protocol 4 (BGP-4)", RFC 4271, 1044 DOI 10.17487/RFC4271, January 2006, 1045 . 1047 [RFC7311] Mohapatra, P., Fernando, R., Rosen, E., and J. Uttaro, 1048 "The Accumulated IGP Metric Attribute for BGP", RFC 7311, 1049 DOI 10.17487/RFC7311, August 2014, 1050 . 1052 [RFC7938] Lapukhov, P., Premji, A., and J. Mitchell, Ed., "Use of 1053 BGP for Routing in Large-Scale Data Centers", RFC 7938, 1054 DOI 10.17487/RFC7938, August 2016, 1055 . 1057 15.2. Informative References 1059 [GREENBERG09] 1060 Greenberg, A., Hamilton, J., Jain, N., Kadula, S., Kim, 1061 C., Lahiri, P., Maltz, D., Patel, P., and S. Sengupta, 1062 "VL2: A Scalable and Flexible Data Center Network", 2009. 1064 [I-D.ietf-6man-segment-routing-header] 1065 Previdi, S., Filsfils, C., Raza, K., Leddy, J., Field, B., 1066 daniel.voyer@bell.ca, d., daniel.bernier@bell.ca, d., 1067 Matsushima, S., Leung, I., Linkova, J., Aries, E., Kosugi, 1068 T., Vyncke, E., Lebrun, D., Steinberg, D., and R. Raszuk, 1069 "IPv6 Segment Routing Header (SRH)", draft-ietf-6man- 1070 segment-routing-header-07 (work in progress), July 2017. 1072 [I-D.ietf-mpls-seamless-mpls] 1073 Leymann, N., Decraene, B., Filsfils, C., Konstantynowicz, 1074 M., and D. Steinberg, "Seamless MPLS Architecture", draft- 1075 ietf-mpls-seamless-mpls-07 (work in progress), June 2014. 1077 [KANDULA04] 1078 Sinha, S., Kandula, S., and D. Katabi, "Harnessing TCP's 1079 Burstiness with Flowlet Switching", 2004. 1081 [RFC6793] Vohra, Q. and E. Chen, "BGP Support for Four-Octet 1082 Autonomous System (AS) Number Space", RFC 6793, 1083 DOI 10.17487/RFC6793, December 2012, 1084 . 1086 Authors' Addresses 1088 Clarence Filsfils (editor) 1089 Cisco Systems, Inc. 1090 Brussels 1091 BE 1093 Email: cfilsfil@cisco.com 1095 Stefano Previdi (editor) 1096 Cisco Systems, Inc. 1097 Italy 1099 Email: stefano@previdi.net 1101 Jon Mitchell 1102 Unaffiliated 1104 Email: jrmitche@puck.nether.net 1105 Ebben Aries 1106 Juniper Networks 1107 1133 Innovation Way 1108 Sunnyvale CA 94089 1109 US 1111 Email: exa@juniper.net 1113 Petr Lapukhov 1114 Facebook 1115 US 1117 Email: petr@fb.com