idnits 2.17.1 draft-ietf-spring-segment-routing-msdc-01.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (April 13, 2016) is 2927 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- -- Looks like a reference, but probably isn't: '16000' on line 273 -- Looks like a reference, but probably isn't: '23999' on line 273 -- Looks like a reference, but probably isn't: '1000' on line 863 -- Looks like a reference, but probably isn't: '1999' on line 863 -- Looks like a reference, but probably isn't: '2000' on line 863 -- Looks like a reference, but probably isn't: '2999' on line 863 ** Obsolete normative reference: RFC 3107 (Obsoleted by RFC 8277) == Outdated reference: A later version (-26) exists of draft-ietf-6man-segment-routing-header-01 == Outdated reference: A later version (-27) exists of draft-ietf-idr-bgp-prefix-sid-02 == Outdated reference: A later version (-11) exists of draft-ietf-rtgwg-bgp-routing-large-dc-09 == Outdated reference: A later version (-15) exists of draft-ietf-spring-segment-routing-07 == Outdated reference: A later version (-10) exists of draft-ietf-spring-segment-routing-central-epe-01 Summary: 1 error (**), 0 flaws (~~), 6 warnings (==), 7 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group C. Filsfils, Ed. 3 Internet-Draft S. Previdi, Ed. 4 Intended status: Informational Cisco Systems, Inc. 5 Expires: October 15, 2016 J. Mitchell 6 Unaffiliated 7 E. Aries 8 P. Lapukhov 9 Facebook 10 April 13, 2016 12 BGP-Prefix Segment in large-scale data centers 13 draft-ietf-spring-segment-routing-msdc-01 15 Abstract 17 This document describes the motivation and benefits for applying 18 segment routing in the data-center. It describes the design to 19 deploy segment routing in the data-center, for both the MPLS and IPv6 20 dataplanes. 22 Requirements Language 24 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 25 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 26 document are to be interpreted as described in RFC 2119 [RFC2119]. 28 Status of This Memo 30 This Internet-Draft is submitted in full conformance with the 31 provisions of BCP 78 and BCP 79. 33 Internet-Drafts are working documents of the Internet Engineering 34 Task Force (IETF). Note that other groups may also distribute 35 working documents as Internet-Drafts. The list of current Internet- 36 Drafts is at http://datatracker.ietf.org/drafts/current/. 38 Internet-Drafts are draft documents valid for a maximum of six months 39 and may be updated, replaced, or obsoleted by other documents at any 40 time. It is inappropriate to use Internet-Drafts as reference 41 material or to cite them other than as "work in progress." 43 This Internet-Draft will expire on October 15, 2016. 45 Copyright Notice 47 Copyright (c) 2016 IETF Trust and the persons identified as the 48 document authors. All rights reserved. 50 This document is subject to BCP 78 and the IETF Trust's Legal 51 Provisions Relating to IETF Documents 52 (http://trustee.ietf.org/license-info) in effect on the date of 53 publication of this document. Please review these documents 54 carefully, as they describe your rights and restrictions with respect 55 to this document. Code Components extracted from this document must 56 include Simplified BSD License text as described in Section 4.e of 57 the Trust Legal Provisions and are provided without warranty as 58 described in the Simplified BSD License. 60 Table of Contents 62 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 63 2. Large Scale Data Center Network Design Summary . . . . . . . 3 64 2.1. Reference design . . . . . . . . . . . . . . . . . . . . 4 65 3. Some open problems in large data-center networks . . . . . . 5 66 4. Applying Segment Routing in the DC with MPLS dataplane . . . 6 67 4.1. BGP Prefix Segment . . . . . . . . . . . . . . . . . . . 6 68 4.2. eBGP Labeled Unicast (RFC3107) . . . . . . . . . . . . . 7 69 4.2.1. Control Plane . . . . . . . . . . . . . . . . . . . . 7 70 4.2.2. Data Plane . . . . . . . . . . . . . . . . . . . . . 9 71 4.2.3. Network Design Variation . . . . . . . . . . . . . . 10 72 4.2.4. Global BGP Prefix Segment through the fabric . . . . 10 73 4.2.5. Incremental Deployments . . . . . . . . . . . . . . . 11 74 4.3. iBGP Labeled Unicast (RFC3107) . . . . . . . . . . . . . 12 75 5. Applying Segment Routing in the DC with IPv6 dataplane . . . 12 76 6. Communicating path information to the host . . . . . . . . . 13 77 7. Addressing the open problems . . . . . . . . . . . . . . . . 14 78 7.1. Per-packet and flowlet switching . . . . . . . . . . . . 14 79 7.2. Performance-aware routing . . . . . . . . . . . . . . . . 15 80 7.3. Non-oblivious routing . . . . . . . . . . . . . . . . . . 16 81 7.4. Deterministic network probing . . . . . . . . . . . . . . 16 82 8. Additional Benefits . . . . . . . . . . . . . . . . . . . . . 16 83 8.1. MPLS Dataplane with operational simplicity . . . . . . . 16 84 8.2. Minimizing the FIB table . . . . . . . . . . . . . . . . 17 85 8.3. Egress Peer Engineering . . . . . . . . . . . . . . . . . 17 86 8.4. Incremental Deployments . . . . . . . . . . . . . . . . . 18 87 8.5. Anycast . . . . . . . . . . . . . . . . . . . . . . . . . 18 88 9. Preferred SRGB Allocation . . . . . . . . . . . . . . . . . . 18 89 10. Alternative Options . . . . . . . . . . . . . . . . . . . . . 19 90 11. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 20 91 12. Manageability Considerations . . . . . . . . . . . . . . . . 20 92 13. Security Considerations . . . . . . . . . . . . . . . . . . . 20 93 14. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 20 94 15. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 20 95 16. References . . . . . . . . . . . . . . . . . . . . . . . . . 21 96 16.1. Normative References . . . . . . . . . . . . . . . . . . 21 97 16.2. Informative References . . . . . . . . . . . . . . . . . 21 98 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 22 100 1. Introduction 102 Segment Routing (SR), as described in 103 [I-D.ietf-spring-segment-routing] leverages the source routing 104 paradigm. A node steers a packet through an ordered list of 105 instructions, called segments. A segment can represent any 106 instruction, topological or service-based. A segment can have a 107 local semantic to an SR node or global within an SR domain. SR 108 allows to enforce a flow through any topological path and service 109 chain while maintaining per-flow state only at the ingress node to 110 the SR domain. Segment Routing can be applied to the MPLS and IPv6 111 data-planes. 113 The use-case use-cases described in this document should be 114 considered in the context of the BGP-based large-scale data-center 115 (DC) design described in[I-D.ietf-rtgwg-bgp-routing-large-dc]We 116 extend it by applying SR both with IPv6 and MPLS dataplane. 118 2. Large Scale Data Center Network Design Summary 120 This section provides a brief summary of the informational document 121 [I-D.ietf-rtgwg-bgp-routing-large-dc] that outlines a practical 122 network design suitable for data-centers of various scales: 124 o Data-center networks have highly symmetric topologies with 125 multiple parallel paths between two server attachment points. The 126 well-known Clos topology is most popular among the operators. In 127 a Clos topology, the minimum number of parallel paths between two 128 elements is determined by the "width" of the middle stage. See 129 Figure 1 below for an illustration of the concept. 131 o Large-scale data-centers commonly use a routing protocol, such as 132 BGP4 [RFC4271] in order to provide endpoint connectivity. 133 Recovery after a network failure is therefore driven either by 134 local knowledge of directly available backup paths or by 135 distributed signaling between the network devices. 137 o Within data-center networks, traffic is load-shared using the 138 Equal Cost Multipath (ECMP) mechanism. With ECMP, every network 139 device implements a pseudo-random decision, mapping packets to one 140 of the parallel paths by means of a hash function calculated over 141 certain parts of the packet, typically a combination of various 142 packet header fields. 144 The following is a schematic of a five-stage Clos topology, with four 145 devices in the middle stage. Notice that number of paths between 146 Node1 and Node12 equals to four: the paths have to cross all of 147 Tier-1 devices. At the same time, the number of paths between Node1 148 and Node2 equals two, and the paths only cross Tier-2 devices. Other 149 topologies are possible, but for simplicity we'll only look into the 150 topologies that have a single path from Tier-1 to Tier-3. The rest 151 could be treated similarly, with a few modifications to the logic. 153 2.1. Reference design 155 Tier-1 156 +-----+ 157 |NODE | 158 +->| 5 |--+ 159 | +-----+ | 160 Tier-2 | | Tier-2 161 +-----+ | +-----+ | +-----+ 162 +------------>|NODE |--+->|NODE |--+--|NODE |-------------+ 163 | +-----| 3 |--+ | 6 | +--| 9 |-----+ | 164 | | +-----+ +-----+ +-----+ | | 165 | | | | 166 | | +-----+ +-----+ +-----+ | | 167 | +-----+---->|NODE |--+ |NODE | +--|NODE |-----+-----+ | 168 | | | +---| 4 |--+->| 7 |--+--| 10 |--+ | | | 169 | | | | +-----+ | +-----+ | +-----+ | | | | 170 | | | | | | | | | | 171 +-----+ +-----+ | +-----+ | +-----+ +-----+ 172 |NODE | |NODE | Tier-3 +->|NODE |--+ Tier-3 |NODE | |NODE | 173 | 1 | | 2 | | 8 | | 11 | | 12 | 174 +-----+ +-----+ +-----+ +-----+ +-----+ 175 | | | | | | | | 176 A O B O <- Servers -> Z O O O 178 Figure 1: 5-stage Clos topology 180 In the reference topology illustrated in Figure 1, we assume: 182 o Each node is its own AS (Node X has AS X) 184 * For simple and efficient route propagation filtering, Nodes 5, 185 6, 7 and 8 share the same AS, Nodes 3 and 4 share the same AS, 186 Nodes 9 and 10 share the same AS. 188 * For efficient usage of the scarce 2-byte Private Use AS pool, 189 different Tier-3 nodes might share the same AS. 191 * Without loss of generality, we will simplify these details in 192 this document and assume that each node has its own AS. 194 o Each node peers with its neighbors via BGP session 196 * If not specified, eBGP is assumed. In a specific use-case, 197 iBGP will be used but this will be called out explicitly in 198 that case. 200 o Each node originates the IPv4 address of it's loopback interface 201 into BGP and announces it to its neighbors. 203 * The loopback of Node X is 192.0.2.x/32. 205 In this document, we also refer to the Tier-1, Tier-2 and Tier-3 206 switches respectively as Spine, Leaf and ToR (top of rack) switches. 207 When a ToR switch acts as a gateway to the "outside world", we call 208 it a border switch. 210 3. Some open problems in large data-center networks 212 The data-center network design summarized above provides means for 213 moving traffic between hosts with reasonable efficiency. There are 214 few open performance and reliability problems that arise in such 215 design: 217 o ECMP routing is most commonly realized per-flow. This means that 218 large, long-lived "elephant" flows may affect performance of 219 smaller, short-lived "mouse" flows and reduce efficiency of per- 220 flow load-sharing. In other words, per-flow ECMP that does not 221 perform efficiently when flow life-time distribution is heavy- 222 tailed. Furthermore, due to hash-function inefficiencies it is 223 possible to have frequent flow collisions, where more flows get 224 placed on one path over the others 226 o Shortest-path routing with ECMP implements oblivious routing 227 model, which is not aware of the network imbalances. If the 228 network symmetry is broken, for example due to link failures, 229 utilization hotspots may appear. For example, if a link fails 230 between Tier-1 and Tier-2 devices (e.g. "Node5" and "Node9"), 231 Tier-3 devices "Node1" and "Node2" will not be aware of that, 232 since there are other paths available from perspective of "Node3". 233 They will continue sending roughly equal traffic to Node3 and 234 Node4 as if the failure didn't exist which may cause a traffic 235 hotspot. 237 o Absence of path visibility leaves transport protocols, such as 238 TCP, with a "blackbox" view of the network. Some TCP metrics, 239 such as SRTT, MSS, CWND and few others could be inferred and 240 cached based on past history, but those apply to destinations, 241 regardless of the path that has been chosen to get there. Thus, 242 for instance, TCP is not capable of remembering "bad" paths, such 243 as those that exhibited poor performance in the past. This means 244 that every new connection will be established obliviously (memory- 245 less) with regards to the paths chosen before, or chosen by other 246 nodes. 248 o Isolating faults in the network with multiple parallel paths and 249 ECMP-based routing is non-trivial due to lack of determinism. 250 Specifically, the connections from HostA to HostB may take a 251 different path every time a new connection is formed, thus making 252 consistent reproduction of a failure much more difficult. This 253 complexity scales linearly with the number of parallel paths in 254 the network, and stems from the random nature of path selection by 255 the network devices. 257 Further in this document, we are going to demonstrate how these 258 problems could be addressed within the framework of Segment Routing. 260 First, we will explain how to apply SR in the DC, for MPLS and IPv6 261 data-planes. 263 4. Applying Segment Routing in the DC with MPLS dataplane 265 4.1. BGP Prefix Segment 267 A BGP-Prefix Segment is a segment associated with a BGP prefix. A 268 BGP-Prefix Segment is a network-wide instruction to forward the 269 packet along the ECMP-aware best path to the related prefix 270 ([I-D.ietf-idr-bgp-prefix-sid]). 272 In this document, we make the network design decision to assume that 273 all the nodes are allocated the same SRGB, e.g. [16000, 23999]. This 274 is important to fulfill the recommendation for operational 275 simplification as explained in [I-D.ietf-spring-segment-routing]. 277 Note well that the use of a common SRGB in all nodes is not a 278 requirement, one could use a different SRGB at every node. However, 279 this would make the operation of the DC fabric more complex as the 280 label allocated to the loopback of a remote switch is then different 281 at every node. This also may increase the complexity of the 282 centralized controller. 284 For illustration purpose, when considering an MPLS data-plane, we 285 assume that the segment index allocated to prefix 192.0.2.x/32 is X. 286 As a result, a local label 1600x is allocated for prefix 192.0.2.x/32 287 by each node throughout the DC fabric. 289 When IPv6 data-plane is considered, we assume that Node X is 290 allocated IPv6 address (segment) 2001:DB8::X. 292 4.2. eBGP Labeled Unicast (RFC3107) 294 Referring to Figure 1 and [[I-D.ietf-rtgwg-bgp-routing-large-dc], the 295 following design modifications are introduced: 297 o Each node peers with its neighbors via eBGP3107 session 299 o The forwarding plane at Tier-2 and Tier-1 is MPLS. 301 o The forwarding plane at Tier-3 is either IP2MPLS (if the host 302 sends IP traffic) or MPLS2MPLS (if the host sends MPLS- 303 encapsulated traffic). 305 Figure 2 zooms on a path from server A to server Z within the 306 topology of Figure 1. 308 +-----+ +-----+ +-----+ 309 +---------->|NODE | |NODE | |NODE | 310 | | 4 |--+->| 7 |--+--| 10 |---+ 311 | +-----+ +-----+ +-----+ | 312 | | 313 +-----+ +-----+ 314 |NODE | |NODE | 315 | 1 | | 11 | 316 +-----+ +-----+ 317 | | 318 A <- Servers -> Z 320 Figure 2: Path from A to Z via nodes 1, 4, 7, 10 and 11 322 Referring to Figure 1 and Figure 2 and assuming the IP address, AS 323 and index allocation previously described, the following sections 324 detail the control plane operation and the data plane states for the 325 prefix 192.0.2.11/32 (loopback of Node11) 327 4.2.1. Control Plane 329 Node11 originates 192.0.2.11/32 in BGP and allocates to it the BGP- 330 Prefix Segment attribute (index11). 332 Node11 sends the following eBGP3107 update to Node10: 334 . NLRI: 192.0.2.11/32 335 . Label: Implicit-Null 336 . Next-hop: Node11's interface address on the link to Node10 337 . AS Path: {11} 338 . BGP-Prefix Attribute: Index 11 340 Node10 receives the above update. As it is SR capable, Node10 is 341 able to interpret the BGP-Prefix Attribute and hence understands that 342 it should allocate the label LOCAL-SRGB (16000) + "index" 11 (hence 343 16011) to the NLRI instead of allocating an nondeterministic label 344 out of a dynamically allocated portion of the local label space. The 345 implicit-null label in the NLRI tells Node10 that it is the 346 penultimate hop and MUST pop the top label on the stack before 347 forwarding traffic for this prefix to Node11. 349 Then, Node10 sends the following eBGP3107 update to Node7: 351 . NLRI: 192.0.2.11/32 352 . Label: 16011 353 . Next-hop: Node10's interface address on the link to Node7 354 . AS Path: {10, 11} 355 . BGP-Prefix Attribute: Index 11 357 Node7 receives the above update. As it is SR capable, Node7 is able 358 to interpret the BGP-Prefix Attribute and hence allocates the local 359 (incoming) label 16011 (16000 + 11) to the NLRI (instead of 360 allocating a "dynamic" local label from its label manager). Node7 361 uses the label in the received eBGP3107 NLRI as the outgoing label 362 (the index is only used to derive the local/incoming label). 364 Node7 sends the following eBGP3107 update to Node4: 366 . NLRI: 192.0.2.11/32 367 . Label: 16011 368 . Next-hop: Node7's interface address on the link to Node4 369 . AS Path: {7, 10, 11} 370 . BGP-Prefix Attribute: Index 11 372 Node4 receives the above update. As it is SR capable, Node4 is able 373 to interpret the BGP-Prefix Attribute and hence allocates the local 374 (incoming) label 16011 to the NLRI (instead of allocating a "dynamic" 375 local label from its label manager). Node4 uses the label in the 376 received eBGP3107 NLRI as outgoing label (the index is only used to 377 derive the local/incoming label). 379 Node4 sends the following eBGP3107 update to Node1: 381 . NLRI: 192.0.2.11/32 382 . Label: 16011 383 . Next-hop: Node4's interface address on the link to Node1 384 . AS Path: {4, 7, 10, 11} 385 . BGP-Prefix Attribute: Index 11 387 Node1 receives the above update. As it is SR capable, Node1 is able 388 to interpret the BGP-Prefix Attribute and hence allocates the local 389 (incoming) label 16011 to the NLRI (instead of allocating a "dynamic" 390 local label from its label manager). Node1 uses the label in the 391 received eBGP3107 NLRI as outgoing label (the index is only used to 392 derive the local/incoming label). 394 4.2.2. Data Plane 396 Referring to Figure 1Referring to Figure 1, and assuming all nodes 397 apply the same advertisement rules described above and all nodes have 398 the same SRGB (16000-23999), here are the IP/MPLS forwarding tables 399 for prefix 192.0.2.11/32 at Nodes 1, 4, 7 and 10. 401 ----------------------------------------------- 402 Incoming label | outgoing label | Outgoing 403 or IP destination | | Interface 404 ------------------+----------------+----------- 405 16011 | 16011 | ECMP{3, 4} 406 192.0.2.11/32 | 16011 | ECMP{3, 4} 407 ------------------+----------------+----------- 409 Figure 3: Node1 Forwarding Table 411 ----------------------------------------------- 412 Incoming label | outgoing label | Outgoing 413 or IP destination | | Interface 414 ------------------+----------------+----------- 415 16011 | 16011 | ECMP{7, 8} 416 192.0.2.11/32 | 16011 | ECMP{7, 8} 417 ------------------+----------------+----------- 419 Figure 4: Node4 Forwarding Table 421 ----------------------------------------------- 422 Incoming label | outgoing label | Outgoing 423 or IP destination | | Interface 424 ------------------+----------------+----------- 425 16011 | 16011 | 10 426 192.0.2.11/32 | 16011 | 10 427 ------------------+----------------+----------- 429 Figure 5: Node7 Forwarding Table 431 ----------------------------------------------- 432 Incoming label | outgoing label | Outgoing 433 or IP destination | | Interface 434 ------------------+----------------+----------- 435 16011 | POP | 11 436 192.0.2.11/32 | N/A | 11 437 ------------------+----------------+----------- 439 Node10 Forwarding Table 441 4.2.3. Network Design Variation 443 A network design choice could consist of switching all the traffic 444 through Tier-1 and Tier-2 as MPLS traffic. In this case, one could 445 filter away the IP entries at Nodes 4, 7 and 10. This might be 446 beneficial in order to optimize the forwarding table size. 448 A network design choice could consist in allowing the hosts to send 449 MPLS-encapsulated traffic (based on EPE use-case, 450 [I-D.ietf-spring-segment-routing-central-epe]). For example, 451 applications at HostA would send their Z-destined traffic to Node1 452 with an MPLS label stack where the top label is 16011 and the next 453 label is an EPE peer segment at Node11 directing the traffic to Z. 455 4.2.4. Global BGP Prefix Segment through the fabric 457 When the previous design is deployed, the operator enjoys global BGP 458 prefix segment (label) allocation throughout the DC fabric. 460 A few examples follow: 462 o Normal forwarding to Node11: a packet with top label 16011 463 received by any switch in the fabric will be forwarded along the 464 ECMP-aware BGP best-path towards Node11 and the label 16011 is 465 penultimate-popped at Node10. 467 o Traffic-engineered path to Node11: an application on a host behind 468 Node1 might want to restrict its traffic to paths via the Spine 469 switch Node5. The application achieves this by sending its 470 packets with a label stack of {16005, 16011}. BGP Prefix segment 471 16005 directs the packet up to Node5 along the path (Node1, Node3, 472 Node5). BGP Prefix Segment 16011 then directs the packet down to 473 Node11 along the path (Node5, Node9, Node11). 475 4.2.5. Incremental Deployments 477 The design previously described can be deployed incrementally. Let 478 us assume that Node7 does not support the BGP-Prefix Segment 479 attribute and let us show how the fabric connectivity is preserved. 481 From a signaling viewpoint, nothing would change: if Node7 does not 482 understand the BGP-Prefix Segment attribute, it does propagate the 483 attribute unmodified to its neighbors. 485 From a label allocation viewpoint, the only difference is that Node7 486 would allocate a dynamic (random) label to the prefix 192.0.2.11/32 487 (e.g. 123456) instead of the "hinted" label as instructed by the BGP 488 Prefix Segment attribute. The neighbors of Node7 adapt automatically 489 as they always use the label in the BGP3107 NLRI as outgoing label. 491 Node4 does understand the BGP-Prefix Segment attribute and hence 492 allocates the indexed label in the SRGB (16011) for 192.0.2.11/32. 494 As a result, all the data-plane entries across the network would be 495 unchanged except the entries at Node7 and its neighbor Node4 as shown 496 in the figures below. 498 The key point is that the end-to-end LSP is preserved because the 499 outgoing label is always derived from the received label within the 500 BGP3107 NLRI. The index in the BGP Prefix SID is only used as a hint 501 on how to allocate the local label (the incoming label) but never for 502 the outgoing label. 504 ------------------------------------------ 505 Incoming label | outgoing | Outgoing 506 or IP destination | label | Interface 507 -------------------+---------------------- 508 12345 | 16011 | 10 510 Figure 7: Node7 Forwarding Table 512 ------------------------------------------ 513 Incoming label | outgoing | Outgoing 514 or IP destination | label | Interface 515 -------------------+---------------------- 516 16011 | 12345 | 7 518 Figure 8: Node4 Forwarding Table 520 The BGP-Prefix Segment functionality can thus be deployed 521 incrementally one node at a time. 523 When deployed together with a homogeneous SRGB (same SRGB across the 524 fabric), the operator incrementally enjoys the global prefix segment 525 benefits as the deployment progresses through the fabric. 527 4.3. iBGP Labeled Unicast (RFC3107) 529 The same exact design as eBGP3107 is used with the following 530 modifications: 532 All switches share the same AS 534 iBGP3107 reflection with nhop-self is used instead of eBGP3107 536 For simple and efficient route propagation filtering, Nodes 5, 6, 537 7 and 8 share the same Cluster ID, Nodes 3 and 4 share the same 538 Cluster ID, Nodes 9 and 10 share the same Cluster ID. 540 AIGP metric ([RFC7311]) is likely applied to the BGP prefix 541 segments as part of a large-scale multi-domain design such as 542 Seamless MPLS [I-D.ietf-mpls-seamless-mpls]. 544 The control-plane behavior is mostly the same as described in the 545 previous section: the only difference is that the eBGP3107 path 546 propagation is simply replaced by an iBGP3107 path reflection with 547 next-hop changed to self. 549 The data-plane tables are exactly the same. 551 5. Applying Segment Routing in the DC with IPv6 dataplane 553 The design described in I-D.ietf-rtgwg-bgp-routing-large-dc 554 [I-D.ietf-rtgwg-bgp-routing-large-dc] is reused with one single 555 modification. We highlight it using the example of the reachability 556 to Node11 via spine switch Node5. 558 Spine5 originates 2001:DB8::5/128 with the attached BGP Prefix 559 Attribute adverting the support of the Segment Routing extension 560 header (SRH, [I-D.ietf-6man-segment-routing-header]) for IPv6 packets 561 destined to segment 2001:DB8::5. 563 Tor11 originates 2001:DB8::11/128 with the attached BGP Prefix 564 Attribute adverting the support of the Segment Routing extension 565 header (SRH, [I-D.ietf-6man-segment-routing-header]) for IPv6 packets 566 destined to segment 2001:DB8::11. 568 The control-plane and data-plane processing of all the other nodes in 569 the fabric is unchanged. Specifically, the routes to 2001:DB8::5 and 570 2001:DB8::11 are installed in the FIB along the eBGP best-path to 571 Node5 (spine node) and Node11 (ToR node) respectively. 573 An application on HostA which needs to send traffic to HostZ via only 574 Node5 (spine node) can do so by sending IPv6 packets with a SR 575 extension header. The destination address and active segment is set 576 to 2001:DB8::5. The next and last segment is set to 2001:DB8::11. 578 The application must only use IPv6 addresses that have been 579 advertised as capable for SRv6 segment processing (e.g. for which the 580 BGP prefix segment capability has been advertised). How applications 581 learn this (e.g.: centralized controller and orchestration) is 582 outside the scope of this document. 584 6. Communicating path information to the host 586 There are two general methods for communicating path information to 587 the end-hosts: "proactive" and "reactive", aka "push" and "pull" 588 models. There are multiple ways to implement either of these 589 methods. Here, we note that one way could be using a centralized 590 controller: the controller either tells the hosts of the prefix-to- 591 path mappings beforehand and updates them as needed (network event 592 driven push), or responds to the hosts making request for a path to 593 specific destination (host event driven pull). It is also possible 594 to use a hybrid model, i.e., pushing some state from the controller 595 in response to particular network events, while the host pulls other 596 state on demand. 598 We note, that when disseminating network-related data to the end- 599 hosts a trade-off is made to balance the amount of information vs the 600 level of visibility in the network state. This applies both to push 601 and pull models. In the extreme case, the host would request path 602 information on every flow, and keep no local state at all. On the 603 other end of the spectrum, information for every prefix in the 604 network along with available paths could be pushed and continuously 605 updated on all hosts. 607 7. Addressing the open problems 609 This section demonstrates how the problems describe above could be 610 solved using the segment routing concept. It is worth noting that 611 segment routing signaling and data-plane are only parts of the 612 solution. Additional enhancements, e.g. such as the centralized 613 controller mentioned previously, and host networking stack support 614 are required to implement the proposed solutions. 616 7.1. Per-packet and flowlet switching 618 With the ability to choose paths on the host, one may go from per- 619 flow load-sharing in the network to per-packet or per-flowlet (see 620 [KANDULA04] for information on flowlets). The host may select 621 different segment routing instructions either per packet, or per 622 flowlet, and route them over different paths. This allows for 623 solving the "elephant flow" problem in the data-center and avoiding 624 link imbalances. 626 Note that traditional ECMP routing could be easily simulated with on- 627 host path selection, using method proposed in VL2 (see 628 [GREENBERG09]). The hosts would randomly pick a Tier-2 or Tier-1 629 device to "bounce" the packet off of, depending on whether the 630 destination is under the same Tier-2 switches, or has to be reached 631 across Tier-1. The host would use a hash function that operates on 632 per-flow invariants, to simulate per-flow load-sharing in the 633 network. 635 Using Figure 1 as reference, let's illustrate this assuming that 636 HostA has an elephant flow to Z called Flow-f. 638 Normally, a flow is hashed on to a single path. Let's assume HostA 639 sends its packets associated with Flow-f with top label 16011 (the 640 label for the remote ToR, Node11, where HostZ is connected) and Node1 641 would hash all the packets of Flow-F via the same nhop (e.g. Node3). 642 Similarly, let's assume that leaf Node3 would hash all the packets of 643 Flow-F via the same next-hop (e.g.: spine switch Node1). This normal 644 operation would restrict the elephant flow on a small subset of the 645 ECMP paths to HostZ and potentially create imbalance and congestion 646 in the fabric. 648 Leveraging the flowlet proposal, assuming A is made aware of 4 649 disjoint paths via intermediate segment 16005, 16006, 16007 and 16008 650 (the BGP prefix SID's of the 4 spine switches) and also made aware of 651 the prefix segment of the remote ToR connected to the destination 652 (16011), then the application can break the elephant flow F into 653 flowlets F1, F2, F3, F4 and associate each flowlet with one of the 654 following 4 label stacks: {16005, 16011}, {16006, 16011}, {16007, 655 16011} and {16008, 16011}. This would spread the load of the elephant 656 flow through all the ECMP paths available in the fabric and rebalance 657 the load. 659 7.2. Performance-aware routing 661 Knowing the path associated with flows/packets, the end host may 662 deduce certain characteristics of the path on its own, and 663 additionally use the information supplied with path information 664 pushed from the controller or received via pull request. The host 665 may further share its path observations with the centralized agent, 666 so that the latter may keep up-to-date network health map to assist 667 other hosts with this information. 669 For example, an application A.1 at HostA may pin a TCP flow destined 670 to HostZ via Spine switch Node5 using label stack {16005, 16011}. The 671 application A.1 may collect information on packet loss, deduced from 672 TCP retransmissions and other signals (e.g. RTT increases). A.1 may 673 additionally publish this information to a centralized agent, e.g. 674 after a flow completes, or periodically for longer lived flows. 675 Next, using both local and/or global performance data, application 676 A.1 as well as other applications sharing the same resources in the 677 DC fabric may pick up the best path for the new flow, or update an 678 existing path (e.g.: when informed of congestion on an existing 679 path). 681 One particularly interesting instance of performance-aware routing is 682 dynamic fault-avoidance. If some links or devices in the network 683 start discarding packets due to a fault, the end-hosts could detect 684 the path(s) being affected and steer their flows away from the 685 problem spot. Similar logic applies to failure cases where packets 686 get completely black-holed, e.g. when a link goes down. 688 For example, an application A.1 informed about 5 paths to Z {16005, 689 16011}, {16006, 16011}, {16007, 16011}, {16008, 16011} and {16011} 690 might use the latter one by default (for simplicity). When 691 performance is degrading, A.1 might then start to pin TCP flows to 692 each of the 4 other paths (each via a distinct spine) and monitor the 693 performance. It would then detect the faulty path and assign a 694 negative preference to the faulty path to avoid further flows using 695 it. Gradually, over time, it may re-assign flows on the faulty path 696 to eventually detect the resolution of the trouble and start reusing 697 the path. 699 7.3. Non-oblivious routing 701 By leveraging Segment Routing, one avoids issues associated with 702 oblivious ECMP hashing. For example, if in the topology depicted on 703 Figure 1 a link between spine switch Node5 and leaf node Node9 fails, 704 HostA may exclude the segment corresponding to Node5 from the prefix 705 matching the servers under Tier-2 devices Node9. In the push path 706 discovery model, the affected path mappings may be explicitly pushed 707 to all the servers for the duration of the failure. The new mapping 708 would instruct them to avoid the particular Tier-1 switch until the 709 link has recovered. Alternatively, in pull path, the centralized 710 controller may start steering new flows immediately after it 711 discovers the issue. Until then, the existing flows may recover 712 using local detection of the path issues, as described in 713 Section 7.2. 715 7.4. Deterministic network probing 717 Active probing is a well-known technique for monitoring network 718 elements health, constituting of sending continuous packet streams 719 simulating network traffic to the hosts in the data-center. Segment 720 routing makes possible to prescribe the exact paths that each probe 721 or series of probes would be taking toward their destination. This 722 allows for fast correlation and detection of failed paths, by 723 processing information from multiple actively probing agents. This 724 complements the data collected from the hosts routing stacks as 725 described inSection 7.2. 727 For example, imagine a probe agent sending packets to all machines in 728 the data-center. For every host, it may send packets over each of 729 the possible paths, knowing exactly which links and devices these 730 packets will be crossing. Correlating results for multiple 731 destinations with the topological data, it may automatically isolate 732 possible problem to a link or device in the network. 734 8. Additional Benefits 736 8.1. MPLS Dataplane with operational simplicity 738 As required by [I-D.ietf-rtgwg-bgp-routing-large-dc], no new 739 signaling protocol is introduced. The Prefix Segment is a 740 lightweight extension to BGP Labelled Unicast (RFC3107 [RFC3107]). 741 It applies either to eBGP or iBGP based designs. 743 Specifically, LDP and RSVP-TE are not used. These protocols would 744 drastically impact the operational complexity of the Data Center and 745 would not scale. This is in line with the requirements expressed in 746 [I-D.ietf-rtgwg-bgp-routing-large-dc] 747 A key element of the operational simplicity is the deployment of the 748 design with a single and consistent SRGB across the DC fabric. 750 At every node in the fabric, the same label is associated to a given 751 BGP prefix segment and hence a notion of global prefix segment 752 arises. 754 When a controller programs HostA to send traffic to HostZ via the 755 normally available BGP ECMP paths, the controller uses label 16011 756 associated with the ToR switch connected to the HostZ. The 757 controller does not need to pick the label based on the ToR that the 758 source host is connected to. 760 In a classic BGP Labelled Unicast design applied to the DC fabric 761 illustrated in Figure 1, the ToR Node1 connected to HostA would most 762 likely allocate a different label for 192.0.2.11/32 than the one 763 allocated by ToR Node2. As a consequence, the controller would need 764 to adapt the SR policy to each host, based on the ToR switch that 765 they are connected to. This adds state maintenance and 766 synchronization problems. All of this unnecessary complexity is 767 eliminated if a single consistent SRGB is utilized across the fabric. 769 8.2. Minimizing the FIB table 771 The designer may decide to switch all the traffic at Tier-1 and Tier- 772 2's based on MPLS, hence drastically decreasing the IP table size at 773 these nodes. 775 This is easily accomplished by encapsulating the traffic either 776 directly at the host or at the source ToR switch by pushing the BGP- 777 Prefix Segment of the destination ToR for intra-DC traffic or border 778 switch for inter-DC or DC-to-outside-world traffic. 780 8.3. Egress Peer Engineering 782 It is straightforward to combine the design illustrated in this 783 document with the Egress Peer Engineering (EPE) use-case described in 784 [I-D.ietf-spring-segment-routing-central-epe]. 786 In such case, the operator is able to engineer its outbound traffic 787 on a per host-flow basis, without incurring any additional state at 788 intermediate points in the DC fabric. 790 For example, the controller only needs to inject a per-flow state on 791 the HostA to force it to send its traffic destined to a specific 792 Internet destination D via a selected border switch (say Node12 in 793 Figure 1 instead of another border switch Node11) and a specific 794 egress peer of Node12 (say peer AS 9999 of local PeerNode segment 795 9999 at Node12 instead of any other peer which provides a path to the 796 destination D). Any packet matching this state at host A would be 797 encapsulated with SR segment list (label stack) {16012, 9999}. 16012 798 would steer the flow through the DC fabric, leveraging any ECMP, 799 along the best path to border switch Node12. Once the flow gets to 800 border switch Node12, the active segment is 9999 (thanks to PHP on 801 the upstream neighbor of Node12). This EPE PeerNode segment forces 802 border switch Node12 to forward the packet to peer AS 9999, without 803 any IP lookup at the border switch. There is no per-flow state for 804 this engineered flow in the DC fabric. A benefit of segment routing 805 is the per-flow state is only required at the source. 807 As well as allowing full traffic engineering control such a design 808 also offers FIB table minimization benefits as the Internet- scale 809 FIB at border switch Node12 is not required if all FIB lookups are 810 avoided there by using EPE. 812 8.4. Incremental Deployments 814 As explained in Section 4.2.5, this design can be deployed 815 incrementally. 817 8.5. Anycast 819 The design presented in this document preserves the availability and 820 load-balancing properties of the base design presented in 821 [I-D.ietf-spring-segment-routing]. 823 For example, one could assign an anycast loopback 192.0.2.20/32 and 824 associate segment index 20 to it on the border switches 11 and 12 (in 825 addition to their node-specific loopbacks). Doing so, the EPE 826 controller could express a default "go-to-the- Internet via any 827 border switch" policy as segment list {16020}. Indeed, from any host 828 in the DC fabric or from any ToR switch, 16020 steers the packet 829 towards the border switches 11 or 12 leveraging ECMP where available 830 along the best paths to these switches. 832 9. Preferred SRGB Allocation 834 In the MPLS case, we do not recommend to use different SRGBs at each 835 node. 837 Different SRGBs in each node likely increase the complexity of the 838 solution both from an operation viewpoint and from a controller 839 viewpoint. 841 From an operation viewpoint, it is much simpler to have the same 842 global label at every node for the same destination (the MPLS 843 troubleshooting is then similar to the IPv6 troubleshooting where 844 this global property is a given). 846 From a controller viewpoint, this allows to construct simple policies 847 applicable across the fabric. 849 Let us consider two applications A and B respectively connected to 850 ToR1 and ToR2. A has two flows FA1 and FA2 destined to Z. B has two 851 flows FB1 and FB2 destined to Z. The controller wants FA1 and FB1 to 852 be load-shared across the fabric while FA2 and FB2 must be 853 respectively steered via Spine5 and spine 8. 855 Assuming a consistent unique SRGB across the fabric as described in 856 the document, the controller can simply do it by instructing A and B 857 to use {16011} respectively for FA1 and FB1 and by instructing A and 858 B to use {16005 16011} and {16008 16011} respectively for FA2 and 859 FB2. 861 Let us assume a design where the SRGB is different at every node: 862 SRGB of Node K starts at value K*1000 and the SRGB length is 1000 863 (e.g. ToR1's SRGB is [1000, 1999], ToR2's SRGB is [2000, 2999]...). 865 In this case, not only the controller would need to collect and store 866 all of these different SRGB's, furthermore it would need to adapt the 867 policy for each host. Indeed, the controller would instruct A to use 868 {1011} for FA1 while it would have to instruct B to use {2011} for 869 FB1 (while with the same SRGB, both policies are the same {16011}). 871 Even worse, the controller would instruct A to use {1005, 5011} for 872 FA1 while it would instruct B to use {2011, 8011} for FB1 (while with 873 the same SRGB, the second segment is the same across both policies: 874 16011). When combining segments to create a policy, one need to 875 carefully update the label of each segment. This is obviously more 876 error-prone, more complex and more difficult to troubleshoot. 878 10. Alternative Options 880 In order to support all the requirements and get consensus, the BGP 881 Prefix SID attribute has been extended to allow this design. 883 Specifically, the ORIGINATOR_SRGB TLV in the BGP Prefix SID signals 884 the SRGB of the switch that originated the BGP Prefix Segment. 886 This allows to determine the local label allocated by any switch for 887 any BGP Prefix Segment, despite the lack of a consistent unique SRGB 888 in the domain. 890 11. IANA Considerations 892 TBD 894 12. Manageability Considerations 896 TBD 898 13. Security Considerations 900 TBD 902 14. Acknowledgements 904 The authors would like to thank Benjamin Black, Arjun Sreekantiah, 905 Keyur Patel and Acee Lindem for their comments and review of this 906 document. 908 15. Contributors 910 Gaya Nagarajan 911 Facebook 912 US 914 Email: gaya@fb.com 916 Dmitry Afanasiev 917 Yandex 918 RU 920 Email: fl0w@yandex-team.ru 922 Tim Laberge 923 Cisco 924 US 926 Email: tlaberge@cisco.com 928 Edet Nkposong 929 Microsoft 930 US 932 Email: edetn@microsoft.com 933 Mohan Nanduri 934 Microsoft 935 US 937 Email: mnanduri@microsoft.com 939 James Uttaro 940 ATT 941 US 943 Email: ju1738@att.com 945 Saikat Ray 946 Unaffiliated 947 US 949 Email: raysaikat@gmail.com 951 16. References 953 16.1. Normative References 955 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 956 Requirement Levels", BCP 14, RFC 2119, 957 DOI 10.17487/RFC2119, March 1997, 958 . 960 [RFC3107] Rekhter, Y. and E. Rosen, "Carrying Label Information in 961 BGP-4", RFC 3107, DOI 10.17487/RFC3107, May 2001, 962 . 964 [RFC4271] Rekhter, Y., Ed., Li, T., Ed., and S. Hares, Ed., "A 965 Border Gateway Protocol 4 (BGP-4)", RFC 4271, 966 DOI 10.17487/RFC4271, January 2006, 967 . 969 [RFC7311] Mohapatra, P., Fernando, R., Rosen, E., and J. Uttaro, 970 "The Accumulated IGP Metric Attribute for BGP", RFC 7311, 971 DOI 10.17487/RFC7311, August 2014, 972 . 974 16.2. Informative References 976 [GREENBERG09] 977 Greenberg, A., Hamilton, J., Jain, N., Kadula, S., Kim, 978 C., Lahiri, P., Maltz, D., Patel, P., and S. Sengupta, 979 "VL2: A Scalable and Flexible Data Center Network", 2009. 981 [I-D.ietf-6man-segment-routing-header] 982 Previdi, S., Filsfils, C., Field, B., Leung, I., Linkova, 983 J., Kosugi, T., Vyncke, E., and D. Lebrun, "IPv6 Segment 984 Routing Header (SRH)", draft-ietf-6man-segment-routing- 985 header-01 (work in progress), March 2016. 987 [I-D.ietf-idr-bgp-prefix-sid] 988 Previdi, S., Filsfils, C., Lindem, A., Patel, K., 989 Sreekantiah, A., Ray, S., and H. Gredler, "Segment Routing 990 Prefix SID extensions for BGP", draft-ietf-idr-bgp-prefix- 991 sid-02 (work in progress), December 2015. 993 [I-D.ietf-mpls-seamless-mpls] 994 Leymann, N., Decraene, B., Filsfils, C., Konstantynowicz, 995 M., and D. Steinberg, "Seamless MPLS Architecture", draft- 996 ietf-mpls-seamless-mpls-07 (work in progress), June 2014. 998 [I-D.ietf-rtgwg-bgp-routing-large-dc] 999 Lapukhov, P., Premji, A., and J. Mitchell, "Use of BGP for 1000 routing in large-scale data centers", draft-ietf-rtgwg- 1001 bgp-routing-large-dc-09 (work in progress), March 2016. 1003 [I-D.ietf-spring-segment-routing] 1004 Filsfils, C., Previdi, S., Decraene, B., Litkowski, S., 1005 and R. Shakir, "Segment Routing Architecture", draft-ietf- 1006 spring-segment-routing-07 (work in progress), December 1007 2015. 1009 [I-D.ietf-spring-segment-routing-central-epe] 1010 Filsfils, C., Previdi, S., Ginsburg, D., and D. Afanasiev, 1011 "Segment Routing Centralized BGP Peer Engineering", draft- 1012 ietf-spring-segment-routing-central-epe-01 (work in 1013 progress), March 2016. 1015 [KANDULA04] 1016 Sinha, S., Kandula, S., and D. Katabi, "Harnessing TCP's 1017 Burstiness with Flowlet Switching", 2004. 1019 Authors' Addresses 1021 Clarence Filsfils (editor) 1022 Cisco Systems, Inc. 1023 Brussels 1024 BE 1026 Email: cfilsfil@cisco.com 1027 Stefano Previdi (editor) 1028 Cisco Systems, Inc. 1029 Via Del Serafico, 200 1030 Rome 00142 1031 Italy 1033 Email: sprevidi@cisco.com 1035 Jon Mitchell 1036 Unaffiliated 1038 Email: jrmitche@puck.nether.net 1040 Ebben Aries 1041 Facebook 1042 US 1044 Email: exa@fb.com 1046 Petr Lapukhov 1047 Facebook 1048 US 1050 Email: petr@fb.com