idnits 2.17.1 draft-filsfils-spring-segment-routing-msdc-02.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (July 6, 2015) is 3211 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- -- Looks like a reference, but probably isn't: '16000' on line 287 -- Looks like a reference, but probably isn't: '23999' on line 287 -- Looks like a reference, but probably isn't: '1000' on line 927 -- Looks like a reference, but probably isn't: '1999' on line 927 -- Looks like a reference, but probably isn't: '2000' on line 927 -- Looks like a reference, but probably isn't: '2999' on line 927 ** Obsolete normative reference: RFC 3107 (Obsoleted by RFC 8277) == Outdated reference: A later version (-05) exists of draft-filsfils-spring-segment-routing-central-epe-03 == Outdated reference: A later version (-11) exists of draft-ietf-rtgwg-bgp-routing-large-dc-03 == Outdated reference: A later version (-05) exists of draft-keyupate-idr-bgp-prefix-sid-02 == Outdated reference: A later version (-08) exists of draft-previdi-6man-segment-routing-header-06 Summary: 1 error (**), 0 flaws (~~), 5 warnings (==), 7 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group C. Filsfils, Ed. 3 Internet-Draft S. Previdi, Ed. 4 Intended status: Informational Cisco Systems, Inc. 5 Expires: January 7, 2016 J. Mitchell 6 Unaffiliated 7 E. Aries 8 P. Lapukhov 9 G. Nagarajan 10 Facebook 11 D. Afanasiev 12 Yandex 13 T. Laberge 14 E. Nkposong 15 M. Nanduri 16 Microsoft 17 J. Uttaro 18 ATT 19 S. Ray 20 Unaffiliated 21 July 6, 2015 23 BGP-Prefix Segment in large-scale data centers 24 draft-filsfils-spring-segment-routing-msdc-02 26 Abstract 28 This document describes the motivation and benefits for applying 29 segment routing in the data-center. It describes the design to 30 deploy segment routing in the data-center, for both the MPLS and IPv6 31 dataplanes. 33 Requirements Language 35 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 36 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 37 document are to be interpreted as described in RFC 2119 [RFC2119]. 39 Status of This Memo 41 This Internet-Draft is submitted in full conformance with the 42 provisions of BCP 78 and BCP 79. 44 Internet-Drafts are working documents of the Internet Engineering 45 Task Force (IETF). Note that other groups may also distribute 46 working documents as Internet-Drafts. The list of current Internet- 47 Drafts is at http://datatracker.ietf.org/drafts/current/. 49 Internet-Drafts are draft documents valid for a maximum of six months 50 and may be updated, replaced, or obsoleted by other documents at any 51 time. It is inappropriate to use Internet-Drafts as reference 52 material or to cite them other than as "work in progress." 54 This Internet-Draft will expire on January 7, 2016. 56 Copyright Notice 58 Copyright (c) 2015 IETF Trust and the persons identified as the 59 document authors. All rights reserved. 61 This document is subject to BCP 78 and the IETF Trust's Legal 62 Provisions Relating to IETF Documents 63 (http://trustee.ietf.org/license-info) in effect on the date of 64 publication of this document. Please review these documents 65 carefully, as they describe your rights and restrictions with respect 66 to this document. Code Components extracted from this document must 67 include Simplified BSD License text as described in Section 4.e of 68 the Trust Legal Provisions and are provided without warranty as 69 described in the Simplified BSD License. 71 Table of Contents 73 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 74 2. Large Scale Data Center Network Design Summary . . . . . . . 3 75 2.1. Reference design . . . . . . . . . . . . . . . . . . . . 4 76 3. Some open problems in large data-center networks . . . . . . 5 77 4. Applying Segment Routing in the DC with MPLS dataplane . . . 6 78 4.1. BGP Prefix Segment . . . . . . . . . . . . . . . . . . . 6 79 4.2. eBGP Labeled Unicast (RFC3107) . . . . . . . . . . . . . 7 80 4.2.1. Control Plane . . . . . . . . . . . . . . . . . . . . 8 81 4.2.2. Data Plane . . . . . . . . . . . . . . . . . . . . . 9 82 4.2.3. Network Design Variation . . . . . . . . . . . . . . 10 83 4.2.4. Global BGP Prefix Segment through the fabric . . . . 11 84 4.2.5. Incremental Deployments . . . . . . . . . . . . . . . 11 85 4.3. iBGP Labeled Unicast (RFC3107) . . . . . . . . . . . . . 12 86 4.4. eBGP Unlabelled (BGP4) . . . . . . . . . . . . . . . . . 13 87 5. Applying Segment Routing in the DC with IPv6 dataplane . . . 14 88 6. Communicating path information to the host . . . . . . . . . 14 89 7. Addressing the open problems . . . . . . . . . . . . . . . . 15 90 7.1. Per-packet and flowlet switching . . . . . . . . . . . . 15 91 7.2. Performance-aware routing . . . . . . . . . . . . . . . . 16 92 7.3. Non-oblivious routing . . . . . . . . . . . . . . . . . . 17 93 7.4. Deterministic network probing . . . . . . . . . . . . . . 17 94 8. Additional Benefits . . . . . . . . . . . . . . . . . . . . . 17 95 8.1. MPLS Dataplane with operational simplicity . . . . . . . 17 96 8.2. Minimizing the FIB table . . . . . . . . . . . . . . . . 18 97 8.3. Egress Peer Engineering . . . . . . . . . . . . . . . . . 18 98 8.4. Incremental Deployments . . . . . . . . . . . . . . . . . 19 99 8.5. Anycast . . . . . . . . . . . . . . . . . . . . . . . . . 19 100 9. Preferred SRGB Allocation . . . . . . . . . . . . . . . . . . 20 101 10. Alternative Options . . . . . . . . . . . . . . . . . . . . . 21 102 11. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 21 103 12. Manageability Considerations . . . . . . . . . . . . . . . . 21 104 13. Security Considerations . . . . . . . . . . . . . . . . . . . 21 105 14. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 21 106 15. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 21 107 16. References . . . . . . . . . . . . . . . . . . . . . . . . . 21 108 16.1. Normative References . . . . . . . . . . . . . . . . . . 21 109 16.2. Informative References . . . . . . . . . . . . . . . . . 22 110 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 23 112 1. Introduction 114 Segment Routing (SR), as described in 115 [I-D.filsfils-spring-segment-routing] leverages the source routing 116 paradigm. A node steers a packet through an ordered list of 117 instructions, called segments. A segment can represent any 118 instruction, topological or service-based. A segment can have a 119 local semantic to an SR node or global within an SR domain. SR 120 allows to enforce a flow through any topological path and service 121 chain while maintaining per-flow state only at the ingress node to 122 the SR domain. Segment Routing can be applied to the MPLS and IPv6 123 data-planes. 125 The use-cases described in this document should primarily be 126 considered in the context of the BGP-based large-scale data-center 127 (DC) design described in [I-D.ietf-rtgwg-bgp-routing-large-dc]. We 128 extend its applicability to large-scale design built on BGP (e.g. 129 metro aggregation) and consider variations where eBGP3107 and 130 iBGP3107 ([RFC3107]) are used instead of eBGP IPv4/IPv6 unlabeled. 132 2. Large Scale Data Center Network Design Summary 134 This section provides a brief summary of the informational document 135 [I-D.ietf-rtgwg-bgp-routing-large-dc] that outlines a practical 136 network design suitable for data-centers of various scales: 138 o Data-center networks have highly symmetric topologies with 139 multiple parallel paths between two server attachment points. The 140 well-known Clos topology is most popular among the operators. In 141 a Clos topology, the minimum number of parallel paths between two 142 elements is determined by the "width" of the middle stage. See 143 Figure 1 below for an illustration of the concept. 145 o Large-scale data-centers commonly use a routing protocol, such as 146 BGP4 [RFC4271] in order to provide endpoint connectivity. 147 Recovery after a network failure is therefore driven either by 148 local knowledge of directly available backup paths or by 149 distributed signaling between the network devices. 151 o Within data-center networks, traffic is load-shared using the 152 Equal Cost Multipath (ECMP) mechanism. With ECMP, every network 153 device implements a pseudo-random decision, mapping packets to one 154 of the parallel paths by means of a hash function calculated over 155 certain parts of the packet, typically a combination of various 156 packet header fields. 158 The following is a schematic of a five-stage Clos topology, with four 159 devices in the middle stage. Notice that number of paths between 160 Node1 and Node12 equals to four: the paths have to cross all of 161 Tier-1 devices. At the same time, the number of paths between Node1 162 and Node2 equals two, and the paths only cross Tier-2 devices. Other 163 topologies are possible, but for simplicity we'll only look into the 164 topologies that have a single path from Tier-1 to Tier-3. The rest 165 could be treated similarly, with a few modifications to the logic. 167 2.1. Reference design 169 Tier-1 170 +-----+ 171 |NODE | 172 +->| 5 |--+ 173 | +-----+ | 174 Tier-2 | | Tier-2 175 +-----+ | +-----+ | +-----+ 176 +------------>|NODE |--+->|NODE |--+--|NODE |-------------+ 177 | +-----| 3 |--+ | 6 | +--| 9 |-----+ | 178 | | +-----+ +-----+ +-----+ | | 179 | | | | 180 | | +-----+ +-----+ +-----+ | | 181 | +-----+---->|NODE |--+ |NODE | +--|NODE |-----+-----+ | 182 | | | +---| 4 |--+->| 7 |--+--| 10 |--+ | | | 183 | | | | +-----+ | +-----+ | +-----+ | | | | 184 | | | | | | | | | | 185 +-----+ +-----+ | +-----+ | +-----+ +-----+ 186 |NODE | |NODE | Tier-3 +->|NODE |--+ Tier-3 |NODE | |NODE | 187 | 1 | | 2 | | 8 | | 11 | | 12 | 188 +-----+ +-----+ +-----+ +-----+ +-----+ 189 | | | | | | | | 190 A O B O <- Servers -> Z O O O 192 Figure 1: 5-stage Clos topology 194 In the reference topology illustrated in Figure 1, we assume: 196 o Each node is its own AS (Node X has AS X) 198 * For simple and efficient route propagation filtering, Nodes 5, 199 6, 7 and 8 share the same AS, Nodes 3 and 4 share the same AS, 200 Nodes 9 and 10 share the same AS. 202 * For efficient usage of the scarce 2-byte Private Use AS pool, 203 different Tier-3 nodes might share the same AS. 205 * Without loss of generality, we will simplify these details in 206 this document and assume that each node has its own AS. 208 o Each node peers with its neighbors via BGP session 210 * If not specified, eBGP is assumed. In a specific use-case, 211 iBGP will be used but this will be called out explicitly in 212 that case. 214 o Each node originates the IPv4 address of it's loopback interface 215 into BGP and announces it to its neighbors. 217 * The loopback of Node X is 192.0.2.x/32. 219 In this document, we also refer to the Tier-1, Tier-2 and Tier-3 220 switches respectively as Spine, Leaf and ToR (top of rack) switches. 221 When a ToR switch acts as a gateway to the "outside world", we call 222 it a border switch. 224 3. Some open problems in large data-center networks 226 The data-center network design summarized above provides means for 227 moving traffic between hosts with reasonable efficiency. There are 228 few open performance and reliability problems that arise in such 229 design: 231 o ECMP routing is most commonly realized per-flow. This means that 232 large, long-lived "elephant" flows may affect performance of 233 smaller, short-lived "mouse" flows and reduce efficiency of per- 234 flow load-sharing. In other words, per-flow ECMP that does not 235 perform efficiently when flow life-time distribution is heavy- 236 tailed. Furthermore, due to hash-function inefficiencies it is 237 possible to have frequent flow collisions, where more flows get 238 placed on one path over the others 240 o Shortest-path routing with ECMP implements oblivious routing 241 model, which is not aware of the network imbalances. If the 242 network symmetry is broken, for example due to link failures, 243 utilization hotspots may appear. For example, if a link fails 244 between Tier-1 and Tier-2 devices (e.g. "Node5" and "Node9"), 245 Tier-3 devices "Node1" and "Node2" will not be aware of that, 246 since there are other paths available from perspective of "Node3". 247 They will continue sending roughly equal traffic to Node3 and 248 Node4 as if the failure didn't exist which may cause a traffic 249 hotspot. 251 o Absence of path visibility leaves transport protocols, such as 252 TCP, with a "blackbox" view of the network. Some TCP metrics, 253 such as SRTT, MSS, CWND and few others could be inferred and 254 cached based on past history, but those apply to destinations, 255 regardless of the path that has been chosen to get there. Thus, 256 for instance, TCP is not capable of remembering "bad" paths, such 257 as those that exhibited poor performance in the past. This means 258 that every new connection will be established obliviously (memory- 259 less) with regards to the paths chosen before, or chosen by other 260 nodes. 262 o Isolating faults in the network with multiple parallel paths and 263 ECMP-based routing is non-trivial due to lack of determinism. 264 Specifically, the connections from HostA to HostB may take a 265 different path every time a new connection is formed, thus making 266 consistent reproduction of a failure much more difficult. This 267 complexity scales linearly with the number of parallel paths in 268 the network, and stems from the random nature of path selection by 269 the network devices. 271 Further in this document, we are going to demonstrate how these 272 problems could be addressed within the framework of Segment Routing. 274 First, we will explain how to apply SR in the DC, for MPLS and IPv6 275 data-planes. 277 4. Applying Segment Routing in the DC with MPLS dataplane 279 4.1. BGP Prefix Segment 281 A BGP-Prefix Segment is a segment associated with a BGP prefix. A 282 BGP-Prefix Segment is a network-wide instruction to forward the 283 packet along the ECMP-aware best path to the related prefix 284 ([I-D.keyupate-idr-bgp-prefix-sid]). 286 In this document, we make the network design decision to assume that 287 all the nodes are allocated the same SRGB, e.g. [16000, 23999]. This 288 is important to fulfill the recommendation for operational 289 simplification as explained in [I-D.filsfils-spring-segment-routing] 290 and [I-D.filsfils-spring-segment-routing-use-cases]. 292 Note well that the use of a common SRGB in all nodes is not a 293 requirement, one could use a different SRGB at every node. However, 294 this would make the operation of the DC fabric more complex as the 295 label allocated to the loopback of a remote switch is then different 296 at every node. This also may increase the complexity of the 297 centralized controller. 299 For illustration purpose, when considering an MPLS data-plane, we 300 assume that the segment index allocated to prefix 192.0.2.x/32 is X. 301 As a result, a local label 1600x is allocated for prefix 192.0.2.x/32 302 by each node throughout the DC fabric. 304 When IPv6 data-plane is considered, we assume that Node X is 305 allocated IPv6 address (segment) 2001:DB8::X. 307 4.2. eBGP Labeled Unicast (RFC3107) 309 Referring to Figure 1 and [[I-D.ietf-rtgwg-bgp-routing-large-dc], the 310 following design modifications are introduced: 312 o Each node peers with its neighbors via eBGP3107 session 314 o The forwarding plane at Tier-2 and Tier-1 is MPLS. 316 o The forwarding plane at Tier-3 is either IP2MPLS (if the host 317 sends IP traffic) or MPLS2MPLS (if the host sends MPLS- 318 encapsulated traffic). 320 Figure 2 zooms on a path from server A to server Z within the 321 topology of Figure 1. 323 +-----+ +-----+ +-----+ 324 +---------->|NODE | |NODE | |NODE | 325 | | 4 |--+->| 7 |--+--| 10 |---+ 326 | +-----+ +-----+ +-----+ | 327 | | 328 +-----+ +-----+ 329 |NODE | |NODE | 330 | 1 | | 11 | 331 +-----+ +-----+ 332 | | 333 A <- Servers -> Z 335 Figure 2: Path from A to Z via nodes 1, 4, 7, 10 and 11 337 Referring to Figure 1 and Figure 2 and assuming the IP address, AS 338 and index allocation previously described, the following sections 339 detail the control plane operation and the data plane states for the 340 prefix 192.0.2.11/32 (loopback of Node11) 342 4.2.1. Control Plane 344 Node11 originates 192.0.2.11/32 in BGP and allocates to it the BGP- 345 Prefix Segment attribute (index11). 347 Node11 sends the following eBGP3107 update to Node10: 349 . NLRI: 192.0.2.11/32 350 . Label: Implicit-Null 351 . Next-hop: Node11's interface address on the link to Node10 352 . AS Path: {11} 353 . BGP-Prefix Attribute: Index 11 355 Node10 receives the above update. As it is SR capable, Node10 is 356 able to interpret the BGP-Prefix Attribute and hence understands that 357 it should allocate the label LOCAL-SRGB (16000) + "index" 11 (hence 358 16011) to the NLRI instead of allocating an nondeterministic label 359 out of a dynamically allocated portion of the local label space. The 360 implicit-null label in the NLRI tells Node10 that it is the 361 penultimate hop and MUST pop the top label on the stack before 362 forwarding traffic for this prefix to Node11. 364 Then, Node10 sends the following eBGP3107 update to Node7: 366 . NLRI: 192.0.2.11/32 367 . Label: 16011 368 . Next-hop: Node10's interface address on the link to Node7 369 . AS Path: {10, 11} 370 . BGP-Prefix Attribute: Index 11 372 Node7 receives the above update. As it is SR capable, Node7 is able 373 to interpret the BGP-Prefix Attribute and hence allocates the local 374 (incoming) label 16011 (16000 + 11) to the NLRI (instead of 375 allocating a "dynamic" local label from its label manager). Node7 376 uses the label in the received eBGP3107 NLRI as the outgoing label 377 (the index is only used to derive the local/incoming label). 379 Node7 sends the following eBGP3107 update to Node4: 381 . NLRI: 192.0.2.11/32 382 . Label: 16011 383 . Next-hop: Node7's interface address on the link to Node4 384 . AS Path: {7, 10, 11} 385 . BGP-Prefix Attribute: Index 11 387 Node4 receives the above update. As it is SR capable, Node4 is able 388 to interpret the BGP-Prefix Attribute and hence allocates the local 389 (incoming) label 16011 to the NLRI (instead of allocating a "dynamic" 390 local label from its label manager). Node4 uses the label in the 391 received eBGP3107 NLRI as outgoing label (the index is only used to 392 derive the local/incoming label). 394 Node4 sends the following eBGP3107 update to Node1: 396 . NLRI: 192.0.2.11/32 397 . Label: 16011 398 . Next-hop: Node4's interface address on the link to Node1 399 . AS Path: {4, 7, 10, 11} 400 . BGP-Prefix Attribute: Index 11 402 Node1 receives the above update. As it is SR capable, Node1 is able 403 to interpret the BGP-Prefix Attribute and hence allocates the local 404 (incoming) label 16011 to the NLRI (instead of allocating a "dynamic" 405 local label from its label manager). Node1 uses the label in the 406 received eBGP3107 NLRI as outgoing label (the index is only used to 407 derive the local/incoming label). 409 4.2.2. Data Plane 411 Referring to Figure 1Referring to Figure 1, and assuming all nodes 412 apply the same advertisement rules described above and all nodes have 413 the same SRGB (16000-23999), here are the IP/MPLS forwarding tables 414 for prefix 192.0.2.11/32 at Nodes 1, 4, 7 and 10. 416 ----------------------------------------------- 417 Incoming label | outgoing label | Outgoing 418 or IP destination | | Interface 419 ------------------+----------------+----------- 420 16011 | 16011 | ECMP{3, 4} 421 192.0.2.11/32 | 16011 | ECMP{3, 4} 422 ------------------+----------------+----------- 424 Figure 3: Node1 Forwarding Table 426 ----------------------------------------------- 427 Incoming label | outgoing label | Outgoing 428 or IP destination | | Interface 429 ------------------+----------------+----------- 430 16011 | 16011 | ECMP{7, 8} 431 192.0.2.11/32 | 16011 | ECMP{7, 8} 432 ------------------+----------------+----------- 434 Figure 4: Node4 Forwarding Table 436 ----------------------------------------------- 437 Incoming label | outgoing label | Outgoing 438 or IP destination | | Interface 439 ------------------+----------------+----------- 440 16011 | 16011 | 10 441 192.0.2.11/32 | 16011 | 10 442 ------------------+----------------+----------- 444 Figure 5: Node7 Forwarding Table 446 ----------------------------------------------- 447 Incoming label | outgoing label | Outgoing 448 or IP destination | | Interface 449 ------------------+----------------+----------- 450 16011 | POP | 11 451 192.0.2.11/32 | N/A | 11 452 ------------------+----------------+----------- 454 Node10 Forwarding Table 456 4.2.3. Network Design Variation 458 A network design choice could consist of switching all the traffic 459 through Tier-1 and Tier-2 as MPLS traffic. In this case, one could 460 filter away the IP entries at Nodes 4, 7 and 10. This might be 461 beneficial in order to optimize the forwarding table size. 463 A network design choice could consist in allowing the hosts to send 464 MPLS-encapsulated traffic (based on EPE use-case, 465 [I-D.filsfils-spring-segment-routing-central-epe]). For example, 466 applications at HostA would send their Z-destined traffic to Node1 467 with an MPLS label stack where the top label is 16011 and the next 468 label is an EPE peer segment at Node11 directing the traffic to Z. 470 4.2.4. Global BGP Prefix Segment through the fabric 472 When the previous design is deployed, the operator enjoys global BGP 473 prefix segment (label) allocation throughout the DC fabric. 475 A few examples follow: 477 o Normal forwarding to Node11: a packet with top label 16011 478 received by any switch in the fabric will be forwarded along the 479 ECMP-aware BGP best-path towards Node11 and the label 16011 is 480 penultimate-popped at Node10. 482 o Traffic-engineered path to Node11: an application on a host behind 483 Node1 might want to restrict its traffic to paths via the Spine 484 switch Node5. The application achieves this by sending its 485 packets with a label stack of {16005, 16011}. BGP Prefix segment 486 16005 directs the packet up to Node5 along the path (Node1, Node3, 487 Node5). BGP Prefix Segment 16011 then directs the packet down to 488 Node11 along the path (Node5, Node9, Node11). 490 4.2.5. Incremental Deployments 492 The design previously described can be deployed incrementally. Let 493 us assume that Node7 does not support the BGP-Prefix Segment 494 attribute and let us show how the fabric connectivity is preserved. 496 From a signaling viewpoint, nothing would change: if Node7 does not 497 understand the BGP-Prefix Segment attribute, it does propagate the 498 attribute unmodified to its neighbors. 500 From a label allocation viewpoint, the only difference is that Node7 501 would allocate a dynamic (random) label to the prefix 192.0.2.11/32 502 (e.g. 123456) instead of the "hinted" label as instructed by the BGP 503 Prefix Segment attribute. The neighbors of Node7 adapt automatically 504 as they always use the label in the BGP3107 NLRI as outgoing label. 506 Node4 does understand the BGP-Prefix Segment attribute and hence 507 allocates the indexed label in the SRGB (16011) for 192.0.2.11/32. 509 As a result, all the data-plane entries across the network would be 510 unchanged except the entries at Node7 and its neighbor Node4 as shown 511 in the figures below. 513 The key point is that the end-to-end LSP is preserved because the 514 outgoing label is always derived from the received label within the 515 BGP3107 NLRI. The index in the BGP Prefix SID is only used as a hint 516 on how to allocate the local label (the incoming label) but never for 517 the outgoing label. 519 ------------------------------------------ 520 Incoming label | outgoing | Outgoing 521 or IP destination | label | Interface 522 -------------------+---------------------- 523 12345 | 16011 | 10 525 Figure 7: Node7 Forwarding Table 527 ------------------------------------------ 528 Incoming label | outgoing | Outgoing 529 or IP destination | label | Interface 530 -------------------+---------------------- 531 16011 | 12345 | 7 533 Figure 8: Node4 Forwarding Table 535 The BGP-Prefix Segment functionality can thus be deployed 536 incrementally one node at a time. 538 When deployed together with a homogeneous SRGB (same SRGB across the 539 fabric), the operator incrementally enjoys the global prefix segment 540 benefits as the deployment progresses through the fabric. 542 4.3. iBGP Labeled Unicast (RFC3107) 544 The same exact design as eBGP3107 is used with the following 545 modifications: 547 All switches share the same AS 549 iBGP3107 reflection with nhop-self is used instead of eBGP3107 551 For simple and efficient route propagation filtering, Nodes 5, 6, 552 7 and 8 share the same Cluster ID, Nodes 3 and 4 share the same 553 Cluster ID, Nodes 9 and 10 share the same Cluster ID. 555 AIGP metric ([RFC7311]) is likely applied to the BGP prefix 556 segments as part of a large-scale multi-domain design such as 557 Seamless MPLS [I-D.ietf-mpls-seamless-mpls]. 559 The control-plane behavior is mostly the same as described in the 560 previous section: the only difference is that the eBGP3107 path 561 propagation is simply replaced by an iBGP3107 path reflection with 562 next-hop changed to self. 564 The data-plane tables are exactly the same. 566 4.4. eBGP Unlabelled (BGP4) 568 The same exact design as eBGP3107 is used with the following 569 modifications: 571 o The same exact design as eBGP3107 is used with the difference 572 consisting of using unlabeled eBGP IP sessions (instead of 573 eBGP3107). 575 o The control and data planes behavior is very similar to those 576 detailed for eBGP3107. The difference is explained here after 577 using the behavior of Node7 as an example. 579 o In eBGP3107 design, when Node7 receives the update for 580 192.0.2.11/32 from Node10: 582 * Node7 finds the outgoing label in the received BGP3107 NLRI. 584 * Node7 uses the index in the received BGP prefix SID as a hint 585 for how to allocate its local (incoming) label. 587 * Node7 installs, on top of the IP FIB entry to 192.0.2.11/32, an 588 MPLS FIB entry with incoming and outgoing labels defined 589 previously. 591 o In eBGP unlabeled design, when Node7 receives the update for 592 192.0.2.11/32 from 10: 594 * Node7 finds the outgoing label by summing B and I which are 595 respectively the neighbor SRGB TLV and the Index TLV found in 596 the BGP Prefix SID attribute (defined in 597 [I-D.keyupate-idr-bgp-prefix-sid]) of the received 598 192.0.2.11/32 route (this is the difference with eBGP3107). 600 * Node7 uses the index in the received BGP prefix SID as a hint 601 for allocating its local (incoming) label (same as eBGP3107). 603 * Node7 installs, on top of the IP FIB entry to 192.0.2.11/32, an 604 MPLS FIB entry with incoming and outgoing labels defined 605 previously (same as eBGP3107). 607 o Assuming the same IP address allocation, the same homogeneous 608 SRGB, the same index allocation, the data-planes through the DC 609 fabric would be exactly similar whether eBGP3107 + BGP-Prefix SID 610 is used or eBGP unlabeled + BGP-Prefix SID. 612 5. Applying Segment Routing in the DC with IPv6 dataplane 614 The design described in I-D.ietf-rtgwg-bgp-routing-large-dc 615 [I-D.ietf-rtgwg-bgp-routing-large-dc] is reused with one single 616 modification. We highlight it using the example of the reachability 617 to Node11 via spine switch Node5. 619 Spine5 originates 2001:DB8::5/128 with the attached BGP Prefix 620 Attribute adverting the support of the Segment Routing extension 621 header (SRH, [I-D.previdi-6man-segment-routing-header]) for IPv6 622 packets destined to segment 2001:DB8::5. 624 Tor11 originates 2001:DB8::11/128 with the attached BGP Prefix 625 Attribute adverting the support of the Segment Routing extension 626 header (SRH, [I-D.previdi-6man-segment-routing-header]) for IPv6 627 packets destined to segment 2001:DB8::11. 629 The control-plane and data-plane processing of all the other nodes in 630 the fabric is unchanged. Specifically, the routes to 2001:DB8::5 and 631 2001:DB8::11 are installed in the FIB along the eBGP best-path to 632 Node5 (spine node) and Node11 (ToR node) respectively. 634 An application on HostA which needs to send traffic to HostZ via only 635 Node5 (spine node) can do so by sending IPv6 packets with a SR 636 extension header. The destination address and active segment is set 637 to 2001:DB8::5. The next and last segment is set to 2001:DB8::11. 639 The application must only use IPv6 addresses that have been 640 advertised as capable for SRv6 segment processing (e.g. for which the 641 BGP prefix segment capability has been advertised). How applications 642 learn this (e.g.: centralized controller and orchestration) is 643 outside the scope of this document. 645 6. Communicating path information to the host 647 There are two general methods for communicating path information to 648 the end-hosts: "proactive" and "reactive", aka "push" and "pull" 649 models. There are multiple ways to implement either of these 650 methods. Here, we note that one way could be using a centralized 651 controller: the controller either tells the hosts of the prefix-to- 652 path mappings beforehand and updates them as needed (network event 653 driven push), or responds to the hosts making request for a path to 654 specific destination (host event driven pull). It is also possible 655 to use a hybrid model, i.e., pushing some state from the controller 656 in response to particular network events, while the host pulls other 657 state on demand. 659 We note, that when disseminating network-related data to the end- 660 hosts a trade-off is made to balance the amount of information vs the 661 level of visibility in the network state. This applies both to push 662 and pull models. In the extreme case, the host would request path 663 information on every flow, and keep no local state at all. On the 664 other end of the spectrum, information for every prefix in the 665 network along with available paths could be pushed and continuously 666 updated on all hosts. 668 7. Addressing the open problems 670 This section demonstrates how the problems describe above could be 671 solved using the segment routing concept. It is worth noting that 672 segment routing signaling and data-plane are only parts of the 673 solution. Additional enhancements, e.g. such as the centralized 674 controller mentioned previously, and host networking stack support 675 are required to implement the proposed solutions. 677 7.1. Per-packet and flowlet switching 679 With the ability to choose paths on the host, one may go from per- 680 flow load-sharing in the network to per-packet or per-flowlet (see 681 [KANDULA04] for information on flowlets). The host may select 682 different segment routing instructions either per packet, or per 683 flowlet, and route them over different paths. This allows for 684 solving the "elephant flow" problem in the data-center and avoiding 685 link imbalances. 687 Note that traditional ECMP routing could be easily simulated with on- 688 host path selection, using method proposed in VL2 (see 689 [GREENBERG09]). The hosts would randomly pick a Tier-2 or Tier-1 690 device to "bounce" the packet off of, depending on whether the 691 destination is under the same Tier-2 switches, or has to be reached 692 across Tier-1. The host would use a hash function that operates on 693 per-flow invariants, to simulate per-flow load-sharing in the 694 network. 696 Using Figure 1 as reference, let's illustrate this assuming that 697 HostA has an elephant flow to Z called Flow-f. 699 Normally, a flow is hashed on to a single path. Let's assume HostA 700 sends its packets associated with Flow-f with top label 16011 (the 701 label for the remote ToR, Node11, where HostZ is connected) and Node1 702 would hash all the packets of Flow-F via the same nhop (e.g. Node3). 703 Similarly, let's assume that leaf Node3 would hash all the packets of 704 Flow-F via the same next-hop (e.g.: spine switch Node1). This normal 705 operation would restrict the elephant flow on a small subset of the 706 ECMP paths to HostZ and potentially create imbalance and congestion 707 in the fabric. 709 Leveraging the flowlet proposal, assuming A is made aware of 4 710 disjoint paths via intermediate segment 16005, 16006, 16007 and 16008 711 (the BGP prefix SID's of the 4 spine switches) and also made aware of 712 the prefix segment of the remote ToR connected to the destination 713 (16011), then the application can break the elephant flow F into 714 flowlets F1, F2, F3, F4 and associate each flowlet with one of the 715 following 4 label stacks: {16005, 16011}, {16006, 16011}, {16007, 716 16011} and {16008, 16011}. This would spread the load of the elephant 717 flow through all the ECMP paths available in the fabric and rebalance 718 the load. 720 7.2. Performance-aware routing 722 Knowing the path associated with flows/packets, the end host may 723 deduce certain characteristics of the path on its own, and 724 additionally use the information supplied with path information 725 pushed from the controller or received via pull request. The host 726 may further share its path observations with the centralized agent, 727 so that the latter may keep up-to-date network health map to assist 728 other hosts with this information. 730 For example, an application A.1 at HostA may pin a TCP flow destined 731 to HostZ via Spine switch Node5 using label stack {16005, 16011}. The 732 application A.1 may collect information on packet loss, deduced from 733 TCP retransmissions and other signals (e.g. RTT increases). A.1 may 734 additionally publish this information to a centralized agent, e.g. 735 after a flow completes, or periodically for longer lived flows. 736 Next, using both local and/or global performance data, application 737 A.1 as well as other applications sharing the same resources in the 738 DC fabric may pick up the best path for the new flow, or update an 739 existing path (e.g.: when informed of congestion on an existing 740 path). 742 One particularly interesting instance of performance-aware routing is 743 dynamic fault-avoidance. If some links or devices in the network 744 start discarding packets due to a fault, the end-hosts could detect 745 the path(s) being affected and steer their flows away from the 746 problem spot. Similar logic applies to failure cases where packets 747 get completely black-holed, e.g. when a link goes down. 749 For example, an application A.1 informed about 5 paths to Z {16005, 750 16011}, {16006, 16011}, {16007, 16011}, {16008, 16011} and {16011} 751 might use the latter one by default (for simplicity). When 752 performance is degrading, A.1 might then start to pin TCP flows to 753 each of the 4 other paths (each via a distinct spine) and monitor the 754 performance. It would then detect the faulty path and assign a 755 negative preference to the faulty path to avoid further flows using 756 it. Gradually, over time, it may re-assign flows on the faulty path 757 to eventually detect the resolution of the trouble and start reusing 758 the path. 760 7.3. Non-oblivious routing 762 By leveraging Segment Routing, one avoids issues associated with 763 oblivious ECMP hashing. For example, if in the topology depicted on 764 Figure 1 a link between spine switch Node5 and leaf node Node9 fails, 765 HostA may exclude the segment corresponding to Node5 from the prefix 766 matching the servers under Tier-2 devices Node9. In the push path 767 discovery model, the affected path mappings may be explicitly pushed 768 to all the servers for the duration of the failure. The new mapping 769 would instruct them to avoid the particular Tier-1 switch until the 770 link has recovered. Alternatively, in pull path, the centralized 771 controller may start steering new flows immediately after it 772 discovers the issue. Until then, the existing flows may recover 773 using local detection of the path issues, as described in 774 Section 7.2. 776 7.4. Deterministic network probing 778 Active probing is a well-known technique for monitoring network 779 elements health, constituting of sending continuous packet streams 780 simulating network traffic to the hosts in the data-center. Segment 781 routing makes possible to prescribe the exact paths that each probe 782 or series of probes would be taking toward their destination. This 783 allows for fast correlation and detection of failed paths, by 784 processing information from multiple actively probing agents. This 785 complements the data collected from the hosts routing stacks as 786 described inSection 7.2. 788 For example, imagine a probe agent sending packets to all machines in 789 the data-center. For every host, it may send packets over each of 790 the possible paths, knowing exactly which links and devices these 791 packets will be crossing. Correlating results for multiple 792 destinations with the topological data, it may automatically isolate 793 possible problem to a link or device in the network. 795 8. Additional Benefits 797 8.1. MPLS Dataplane with operational simplicity 799 As required by [I-D.ietf-rtgwg-bgp-routing-large-dc], no new 800 signaling protocol is introduced. The Prefix Segment is a 801 lightweight extension to either BGP Labelled Unicast (RFC3107 803 [RFC3107]) or unlabeled BGP. It applies either to eBGP or iBGP based 804 designs 806 Specifically, LDP and RSVP-TE are not used. These protocols would 807 drastically impact the operational complexity of the Data Center and 808 would not scale. This is in line with the requirements expressed in 809 [I-D.ietf-rtgwg-bgp-routing-large-dc] 811 A key element of the operational simplicity is the deployment of the 812 design with a single and consistent SRGB across the DC fabric. 814 At every node in the fabric, the same label is associated to a given 815 BGP prefix segment and hence a notion of global prefix segment 816 arises. 818 When a controller programs HostA to send traffic to HostZ via the 819 normally available BGP ECMP paths, the controller uses label 16011 820 associated with the ToR switch connected to the HostZ. The 821 controller does not need to pick the label based on the ToR that the 822 source host is connected to. 824 In a classic BGP Labelled Unicast design applied to the DC fabric 825 illustrated in Figure 1, the ToR Node1 connected to HostA would most 826 likely allocate a different label for 192.0.2.11/32 than the one 827 allocated by ToR Node2. As a consequence, the controller would need 828 to adapt the SR policy to each host, based on the ToR switch that 829 they are connected to. This adds state maintenance and 830 synchronization problems. All of this unnecessary complexity is 831 eliminated if a single consistent SRGB is utilized across the fabric. 833 8.2. Minimizing the FIB table 835 The designer may decide to switch all the traffic at Tier-1 and Tier- 836 2's based on MPLS, hence drastically decreasing the IP table size at 837 these nodes. 839 This is easily accomplished by encapsulating the traffic either 840 directly at the host or at the source ToR switch by pushing the BGP- 841 Prefix Segment of the destination ToR for intra-DC traffic or border 842 switch for inter-DC or DC-to-outside-world traffic. 844 8.3. Egress Peer Engineering 846 It is straightforward to combine the design illustrated in this 847 document with the Egress Peer Engineering (EPE) use-case described in 848 [I-D.filsfils-spring-segment-routing-central-epe]. 850 In such case, the operator is able to engineer its outbound traffic 851 on a per host-flow basis, without incurring any additional state at 852 intermediate points in the DC fabric. 854 For example, the controller only needs to inject a per-flow state on 855 the HostA to force it to send its traffic destined to a specific 856 Internet destination D via a selected border switch (say Node12 in 857 Figure 1 instead of another border switch Node11) and a specific 858 egress peer of Node12 (say peer AS 9999 of local PeerNode segment 859 9999 at Node12 instead of any other peer which provides a path to the 860 destination D). Any packet matching this state at host A would be 861 encapsulated with SR segment list (label stack) {16012, 9999}. 16012 862 would steer the flow through the DC fabric, leveraging any ECMP, 863 along the best path to border switch Node12. Once the flow gets to 864 border switch Node12, the active segment is 9999 (thanks to PHP on 865 the upstream neighbor of Node12). This EPE PeerNode segment forces 866 border switch Node12 to forward the packet to peer AS 9999, without 867 any IP lookup at the border switch. There is no per-flow state for 868 this engineered flow in the DC fabric. A benefit of segment routing 869 is the per-flow state is only required at the source. 871 As well as allowing full traffic engineering control such a design 872 also offers FIB table minimization benefits as the Internet- scale 873 FIB at border switch Node12 is not required if all FIB lookups are 874 avoided there by using EPE. 876 8.4. Incremental Deployments 878 As explained in Section 4.2.5, this design can be deployed 879 incrementally. 881 8.5. Anycast 883 The design presented in this document preserves the availability and 884 load-balancing properties of the base design presented in 885 [I-D.filsfils-spring-segment-routing]. 887 For example, one could assign an anycast loopback 192.0.2.20/32 and 888 associate segment index 20 to it on the border switches 11 and 12 (in 889 addition to their node-specific loopbacks). Doing so, the EPE 890 controller could express a default "go-to-the- Internet via any 891 border switch" policy as segment list {16020}. Indeed, from any host 892 in the DC fabric or from any ToR switch, 16020 steers the packet 893 towards the border switches 11 or 12 leveraging ECMP where available 894 along the best paths to these switches. 896 9. Preferred SRGB Allocation 898 In the MPLS case, we do not recommend to use different SRGBs at each 899 node. 901 Different SRGBs in each node likely increase the complexity of the 902 solution both from an operation viewpoint and from a controller 903 viewpoint. 905 From an operation viewpoint, it is much simpler to have the same 906 global label at every node for the same destination (the MPLS 907 troubleshooting is then similar to the IPv6 troubleshooting where 908 this global property is a given). 910 From a controller viewpoint, this allows to construct simple policies 911 applicable across the fabric. 913 Let us consider two applications A and B respectively connected to 914 ToR1 and ToR2. A has two flows FA1 and FA2 destined to Z. B has two 915 flows FB1 and FB2 destined to Z. The controller wants FA1 and FB1 to 916 be load-shared across the fabric while FA2 and FB2 must be 917 respectively steered via Spine5 and spine 8. 919 Assuming a consistent unique SRGB across the fabric as described in 920 the document, the controller can simply do it by instructing A and B 921 to use {16011} respectively for FA1 and FB1 and by instructing A and 922 B to use {16005 16011} and {16008 16011} respectively for FA2 and 923 FB2. 925 Let us assume a design where the SRGB is different at every node: 926 SRGB of Node K starts at value K*1000 and the SRGB length is 1000 927 (e.g. ToR1's SRGB is [1000, 1999], ToR2's SRGB is [2000, 2999]...). 929 In this case, not only the controller would need to collect and store 930 all of these different SRGB's, furthermore it would need to adapt the 931 policy for each host. Indeed, the controller would instruct A to use 932 {1011} for FA1 while it would have to instruct B to use {2011} for 933 FB1 (while with the same SRGB, both policies are the same {16011}). 935 Even worse, the controller would instruct A to use {1005, 5011} for 936 FA1 while it would instruct B to use {2011, 8011} for FB1 (while with 937 the same SRGB, the second segment is the same across both policies: 938 16011). When combining segments to create a policy, one need to 939 carefully update the label of each segment. This is obviously more 940 error-prone, more complex and more difficult to troubleshoot. 942 10. Alternative Options 944 In order to support all the requirements and get consensus, the BGP 945 Prefix SID attribute has been extended to allow this design. 947 Specifically, the ORIGINATOR_SRGB TLV in the BGP Prefix SID signals 948 the SRGB of the switch that originated the BGP Prefix Segment. 950 This allows to determine the local label allocated by any switch for 951 any BGP Prefix Segment, despite the lack of a consistent unique SRGB 952 in the domain. 954 11. IANA Considerations 956 TBD 958 12. Manageability Considerations 960 TBD 962 13. Security Considerations 964 TBD 966 14. Contributors 968 Benjamin Black, Arjun Arjun Sreekantiah and Keyur Patel have 969 contributed to the content of this document. 971 15. Acknowledgements 973 The authors would like to thank Acee Lindem for his review. 975 16. References 977 16.1. Normative References 979 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 980 Requirement Levels", BCP 14, RFC 2119, March 1997. 982 [RFC3107] Rekhter, Y. and E. Rosen, "Carrying Label Information in 983 BGP-4", RFC 3107, May 2001. 985 [RFC4271] Rekhter, Y., Li, T., and S. Hares, "A Border Gateway 986 Protocol 4 (BGP-4)", RFC 4271, January 2006. 988 [RFC7311] Mohapatra, P., Fernando, R., Rosen, E., and J. Uttaro, 989 "The Accumulated IGP Metric Attribute for BGP", RFC 7311, 990 August 2014. 992 16.2. Informative References 994 [GREENBERG09] 995 Greenberg, A., Hamilton, J., Jain, N., Kadula, S., Kim, 996 C., Lahiri, P., Maltz, D., Patel, P., and S. Sengupta, 997 "VL2: A Scalable and Flexible Data Center Network", 2009. 999 [I-D.filsfils-spring-segment-routing] 1000 Filsfils, C., Previdi, S., Bashandy, A., Decraene, B., 1001 Litkowski, S., Horneffer, M., Milojevic, I., Shakir, R., 1002 Ytti, S., Henderickx, W., Tantsura, J., and E. Crabbe, 1003 "Segment Routing Architecture", draft-filsfils-spring- 1004 segment-routing-04 (work in progress), July 2014. 1006 [I-D.filsfils-spring-segment-routing-central-epe] 1007 Filsfils, C., Previdi, S., Patel, K., Aries, E., 1008 shaw@fb.com, s., Ginsburg, D., and D. Afanasiev, "Segment 1009 Routing Centralized Egress Peer Engineering", draft- 1010 filsfils-spring-segment-routing-central-epe-03 (work in 1011 progress), January 2015. 1013 [I-D.filsfils-spring-segment-routing-use-cases] 1014 Filsfils, C., Francois, P., Previdi, S., Decraene, B., 1015 Litkowski, S., Horneffer, M., Milojevic, I., Shakir, R., 1016 Ytti, S., Henderickx, W., Tantsura, J., Kini, S., and E. 1017 Crabbe, "Segment Routing Use Cases", draft-filsfils- 1018 spring-segment-routing-use-cases-01 (work in progress), 1019 October 2014. 1021 [I-D.ietf-mpls-seamless-mpls] 1022 Leymann, N., Decraene, B., Filsfils, C., Konstantynowicz, 1023 M., and D. Steinberg, "Seamless MPLS Architecture", draft- 1024 ietf-mpls-seamless-mpls-07 (work in progress), June 2014. 1026 [I-D.ietf-rtgwg-bgp-routing-large-dc] 1027 Lapukhov, P., Premji, A., and J. Mitchell, "Use of BGP for 1028 routing in large-scale data centers", draft-ietf-rtgwg- 1029 bgp-routing-large-dc-03 (work in progress), June 2015. 1031 [I-D.keyupate-idr-bgp-prefix-sid] 1032 Patel, K., Previdi, S., Filsfils, C., Sreekantiah, A., and 1033 S. Ray, "Segment Routing Prefix SID extensions for BGP", 1034 draft-keyupate-idr-bgp-prefix-sid-02 (work in progress), 1035 May 2015. 1037 [I-D.previdi-6man-segment-routing-header] 1038 Previdi, S., Filsfils, C., Field, B., and I. Leung, "IPv6 1039 Segment Routing Header (SRH)", draft-previdi-6man-segment- 1040 routing-header-06 (work in progress), May 2015. 1042 [KANDULA04] 1043 Sinha, S., Kandula, S., and D. Katabi, "Harnessing TCP's 1044 Burstiness with Flowlet Switching", 2004. 1046 Authors' Addresses 1048 Clarence Filsfils (editor) 1049 Cisco Systems, Inc. 1050 Brussels 1051 BE 1053 Email: cfilsfil@cisco.com 1055 Stefano Previdi (editor) 1056 Cisco Systems, Inc. 1057 Via Del Serafico, 200 1058 Rome 00142 1059 Italy 1061 Email: sprevidi@cisco.com 1063 Jon Mitchell 1064 Unaffiliated 1066 Email: jrmitche@puck.nether.net 1068 Ebben Aries 1069 Facebook 1070 US 1072 Email: exa@fb.com 1074 P. Lapukhov 1075 Facebook 1076 US 1078 Email: petr@fb.com 1079 G. Nagarajan 1080 Facebook 1081 US 1083 Email: gaya@fb.com 1085 Dmitry Afanasiev 1086 Yandex 1087 RU 1089 Email: fl0w@yandex-team.ru 1091 Tim Laberge 1092 Microsoft 1094 Email: tim.laberge@microsoft.com 1096 Edet Nkposong 1097 Microsoft 1099 Email: edetn@microsoft.com 1101 Mohan Nanduri 1102 Microsoft 1104 Email: mnanduri@microsoft.com 1106 James Uttaro 1107 ATT 1109 Email: ju1738@att.com 1111 Saikat Ray 1112 Unaffiliated 1114 Email: raysaikat@gmail.com