idnits 2.17.1 draft-dunbar-sr-sdwan-over-hybrid-networks-02.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack a Security Considerations section. (A line matching the expected section header was found, but with an unexpected indentation: ' 6. Security Considerations' ) ** The document seems to lack an IANA Considerations section. (See Section 2.2 of https://www.ietf.org/id-info/checklist for how to handle the case when there are no actions for IANA.) (A line matching the expected section header was found, but with an unexpected indentation: ' 7. IANA Considerations' ) ** There are 5 instances of too long lines in the document, the longest one being 95 characters in excess of 72. ** The document seems to lack a both a reference to RFC 2119 and the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords. RFC 2119 keyword, line 445: '...appropriate SIDs SHOULD be chosen from...' Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (July 2, 2018) is 2126 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Missing Reference: 'RFC8085' is mentioned on line 446, but not defined == Missing Reference: 'RFC768' is mentioned on line 448, but not defined == Missing Reference: 'Dc1' is mentioned on line 674, but not defined == Unused Reference: 'RFC2735' is defined on line 764, but no explicit reference was found in the text == Unused Reference: 'RFC8192' is defined on line 767, but no explicit reference was found in the text == Unused Reference: 'ITU-T-X1036' is defined on line 770, but no explicit reference was found in the text == Unused Reference: 'RFC6071' is defined on line 774, but no explicit reference was found in the text == Unused Reference: 'RFC4364' is defined on line 777, but no explicit reference was found in the text == Unused Reference: 'RFC4664' is defined on line 780, but no explicit reference was found in the text == Unused Reference: 'RFC7510' is defined on line 794, but no explicit reference was found in the text == Outdated reference: A later version (-01) exists of draft-dukes-sr-for-sdwan-00 == Outdated reference: A later version (-26) exists of draft-ietf-6man-segment-routing-header-13 == Outdated reference: A later version (-22) exists of draft-ietf-spring-segment-routing-mpls-13 Summary: 4 errors (**), 0 flaws (~~), 14 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 Network Working Group L. Dunbar 2 Internet Draft Huawei 3 Intended status: Informational Mehmet Toy 4 Expires: January 2019 Verizon 5 July 2, 2018 7 Segment routing for SD-WAN paths over hybrid networks 8 draft-dunbar-sr-sdwan-over-hybrid-networks-02 10 Abstract 12 This document describes a method for end-to-end (E2E) SD-WAN paths 13 (most likely encrypted) to traverse specific list of network 14 segments, some of which are SR enabled and others may be IP networks 15 that do not support SR, to achieve the desired optimal E2E quality. 17 The method described in this draft uses the principle of segment 18 routing to enforce a SD-WAN path' head-end selected route traversing 19 through a list of specific nodes of multiple network segments 20 without requiring the nodes in each network segment to have the 21 intelligence (or maintaining states) of selecting next hop or next 22 domain. Those networks over which the SD-WAN path traverse have at 23 least one SR enabled network, and some network segments (especially 24 the last mile access portion) being existing IP networks (such as 25 existing IPv4, IPv6 or others). 27 Status of this Memo 29 This Internet-Draft is submitted in full conformance with the 30 provisions of BCP 78 and BCP 79. 32 This Internet-Draft is submitted in full conformance with the 33 provisions of BCP 78 and BCP 79. This document may not be modified, 34 and derivative works of it may not be created, except to publish it 35 as an RFC and to translate it into languages other than English. 37 Internet-Drafts are working documents of the Internet Engineering 38 Task Force (IETF), its areas, and its working groups. Note that 39 other groups may also distribute working documents as Internet- 40 Drafts. 42 Internet-Drafts are draft documents valid for a maximum of six 43 months and may be updated, replaced, or obsoleted by other documents 44 at any time. It is inappropriate to use Internet-Drafts as 45 reference material or to cite them other than as "work in progress." 47 The list of current Internet-Drafts can be accessed at 48 http://www.ietf.org/ietf/1id-abstracts.txt 50 The list of Internet-Draft Shadow Directories can be accessed at 51 http://www.ietf.org/shadow.html 53 This Internet-Draft will expire on January 2, 2009. 55 Copyright Notice 57 Copyright (c) 2018 IETF Trust and the persons identified as the 58 document authors. All rights reserved. 60 This document is subject to BCP 78 and the IETF Trust's Legal 61 Provisions Relating to IETF Documents 62 (http://trustee.ietf.org/license-info) in effect on the date of 63 publication of this document. Please review these documents 64 carefully, as they describe your rights and restrictions with 65 respect to this document. Code Components extracted from this 66 document must include Simplified BSD License text as described in 67 Section 4.e of the Trust Legal Provisions and are provided without 68 warranty as described in the Simplified BSD License. 70 Table of Contents 72 1. Introduction...................................................3 73 2. Definition of terms............................................4 74 3. Key Use Cases..................................................5 75 3.1. SD-WAN Path over LTE network and SR Domain................5 76 3.2. SD-WAN As Last Mile for Cloud DCs Access..................6 77 3.3. How & Why SR is useful for those use cases................7 79 4. Mechanism for SD-WAN path over one SR Domain and existing access 80 ..................................................................8 81 4.1. Controller Delivers SID Stack to SD-WAN Head-end..........9 82 4.2. Using GRE Key to Differentiate Flows.....................11 83 4.3. Using UDP Source Port Number to Differentiate Flows......12 84 4.4. GRE Header Extension.....................................15 85 5. SD-WAN path over multiple SP managed domains..................16 86 5.1. When Both SP domains support SR..........................17 87 5.2. When SP-2 does not support SR............................17 88 5.3. When SP-1 and SP-2 don't want to share network information18 89 5.4. TLV to pass Metadata through SRv6 Domain.................18 90 6. Security Considerations.......................................19 91 7. IANA Considerations...........................................20 92 8. References....................................................20 93 8.1. Normative References.....................................20 94 8.2. Informative References...................................21 95 9. Acknowledgments...............................................22 97 1. Introduction 99 This document describes a method to enforce a SD-WAN path's head-end 100 selected route traversing through a list of specific nodes of 101 multiple network segments without requiring the nodes in each 102 network segments to have the intelligence (or maintaining states) of 103 selecting next hop or next segments. Those networks over which the 104 SD-WAN path traverse have at least one SR enabled network, and some 105 network segments (especially the last mile access portion) being 106 existing IP networks (such as existing IPv4, IPv6 or others). 108 SD-WAN, as described by ONUG (Open Network User Group), is about 109 pooling WAN bandwidth from n service providers to get better WAN 110 bandwidth management, visibility & control. 112 Throughout this document, the term "Classic SD-WAN" refers to a pair 113 of CPEs in two locations aggregating N Service Providers' paths, 114 such as MPLS Paths and public internet paths. [SR-SD-WAN] describes 115 using explicit routes within the SRv6 or SR-MPLS enabled networks to 116 reach the desired quality for SD-WAN paths over the SRv6 or SR-MPLS 117 enabled networks respectively. 119 Another way of using SD-WAN is for network service providers to 120 extend its existing VPN to reach sites to which they do not have 121 presence yet, with detailed use cases described in Section 3 of this 122 document. Under this scenario, the SD-WAN path is laid over multiple 123 hybrid network segments. This document focuses on this type of SD- 124 WAN where a portion of SD-WAN path is over SR enabled networks and 125 the other portion of the SD-WAN path is over existing IP networks, 126 such as existing IPv4, LTE, etc. Under this scenario, the endpoints 127 of the SD-WAN path (e.g. the CPE devices, one or both) are not 128 directly attached to PEs of a SR domain. 130 The goal is to place a large portion of the SD-WAN path over a 131 provider VPN to reach desired transport quality or making the SD-WAN 132 path traversing specific ingress/egress PEs for the desired cost, 133 quality or other reasons. 135 2. Definition of terms 137 Cloud DC: Off-Premises Data Centers that usually host applications 138 and workload owned by different organizations or 139 tenants. 141 Controller: Used interchangeably with SD-WAN controller to manage 142 SD-WAN overlay path creation/deletion and monitoring the 143 path conditions between two or more sites. 145 DMVPN: Dynamic Multipoint Virtual Private Network. DMVPN is a 146 secure network that exchanges data between sites without 147 needing to pass traffic through an organization's 148 headquarter virtual private network (VPN) server or 149 router. 151 Heterogeneous Cloud: applications & workloads split among Cloud DCs 152 owned & managed by different Cloud Providers. 154 Hybrid Cloud: applications & workloads split between on-premises 155 Data centers and Cloud DCs. In this document Hybrid 156 Cloud also include heterogeneous cloud as well. 158 SD-WAN: Software Defined Wide Area Network, which can mean many 159 different things. In this document, "SD-WAN" refers to 160 the solutions specified by ONUG (Open Network User 161 Group), https://www.onug.net/software-defined-wide-area- 162 network-sd-wan/, which is about pooling WAN bandwidth 163 from n service providers to get better WAN bandwidth 164 management, visibility & control. 166 SP: Network Service Provider 168 SR: Segment Routing 170 SR Domain: A domain that supports Segment Routing 172 VPC: Virtual Private Cloud. A service offered by many Cloud 173 DC operators to allocate a logically isolated cloud 174 resources, including compute, networking and storage. 176 3. Key Use Cases 178 3.1. SD-WAN Path over LTE network and SR Domain 180 MEF Cloud Service Architecture [MEF-Cloud] describes a use case of 181 network operators needing to use SD-WAN over LTE for the last mile 182 access that they do not have physical infrastructure, as shown 183 below: 185 +********SD-WAN Overlay Path *********+ 186 * * 187 * +------------+ * 188 * |SD-WAN Ctrl | * 189 * +===+------------+====+ * 190 * // \\ * 191 * // <-----EVPN-VxLAN----> \\ * 192 * +-+--+ ++-+ ++-+ +--+-+ * 193 ***+ E1 |==|C1| |C4+==+ E2 |****+ 194 A --+ | | | | | | +----Z 195 A2--+----+ ++-+ ++-+ ++---+\---Z2 196 LTE || | | // 197 || | SR +-+---+ +----+ 198 +--------+ || | Network | C6 | |E3 | 199 | E4 | || | | |- --| | 200 | | +==+-----+ +-+---+ +----+ 201 +--------+-------+ C3 |-----+ 202 +---+-+ 203 -- Directly attached 204 == || Public Internet or LTE path 205 *** Overlay path 207 Figure 1: SD-WAN end points are attached to VPN via LTE 209 3.2. SD-WAN As Last Mile for Cloud DCs Access 211 Digital Transformation is propelling more and more enterprises to 212 move their workloads/Apps to cloud DCs that are geographically close 213 to their end users to improve end-to-end latency & overall user 214 experience, or to comply with local data protection regulations. 215 Conversely, workloads/Apps in those Cloud DCs can be easily shutdown 216 when their end users' geographic base changes. 218 Because of the ephemeral property of the selected Cloud DCs, an 219 enterprise or its network service provider may not have the direct 220 links to the Cloud DCs that are optimal for hosting the enterprise's 221 specific workloads/Apps. Under those circumstances, SD-WAN is a very 222 flexible choice to interconnect the enterprise on-premises data 223 centers & branch offices to its desired Cloud DCs. 225 However, SD-WAN paths over public internet can have unpredictable 226 performance, especially over long distances and cross state/country 227 boundaries. Therefore, it is highly desirable to place as much as 228 possible the portion of SD-WAN paths over service provider VPN (e.g. 229 enterprise's existing VPN) that have guaranteed SLA to minimize the 230 distance/segments over public internet. 232 Under this scenario, one or both of the SD-WAN end points may not 233 directly attached to the PEs of a SR Domain. 235 3.3. How & Why SR is useful for those use cases 237 Let us assume that the SD-WAN Controller is capable of computing 238 optimal paths between two end-points (e.g. E1<->E2 in the Figure 2), 239 either by communicating with the SR Domain controller/management- 240 system, or by other methods which is out of the scope of this 241 document. 243 The SR domain must have a set of PEs that have at least one port 244 facing the external networks (such as the public internet or LTE 245 termination). 247 Under this circumstance, SD-WAN end-points usually can reach 248 multiple PEs. 250 In the diagram below, E1 <-> E2 SD-WAN (most likely IPsec encrypted 251 tunnel) path can traverse C1 <-> C4, C1<->C6, C3<->C6, or C3<->C4 252 within the VPN. There are many flows (by different Apps) between E1 253 <-> E2. Some flows may need to traverse C1<->C4, others may need to 254 traverse C3<->C6 or other segments within the VPN, which are 255 determined by the SD-WAN controller based on the characteristics & 256 need of the Apps, such as cost, available bandwidth, latency, or 257 special functions only available at specific locations, etc. 259 Even with the same ingress/egress, some flows may need different 260 segments across the SR Domain. It is not practical, or even 261 possible, for PEs (e.g. C1, C2, C3 in this example) to determine 262 which Apps' flows should egress C4 or C6 where both C4&C6 can reach 263 E2. 265 Segment Routing can easily force the path to traverse the explicit 266 egress node (C4 or C6), or explicit segments through the SR Domain 267 based on the SLA requested by the SD-WAN head-end nodes. 269 +********SD-WAN Overlay Path *********+ 270 * * 271 * +------------+ * 272 * |SD-WAN Ctrl | * 273 * +===+------------+ * 274 * // \\ * 275 * // <-----EVPN-VxLAN----> \\ * 276 * +-+--+ ++-+ ++-+ +--+-+ * 277 ***+ E1 |==|C1| |C4+==+ E2 |****+ 278 A --+ | | | | | | +----Z 279 A2--+----+ ++-+ ++-+ ++---+\---Z2 280 LTE || | | // 281 || | SR +-+---+ +----+ 282 +--------+ || | Network | C6 | |E3 | 283 | E4 | || | | |- --| | 284 | | +==+-----+ +-+---+ +----+ 285 +-+--+-+-+-------+ C3 |-----+ 286 +---+-+ 288 -- Directly attached 289 == || Public Internet or LTE path 290 ** Overlay path 292 Figure 2: SDWAN end points not directly attached to PEs of SR Domain 294 4. Mechanism for SD-WAN path over one SR Domain and existing access 296 This section describes the mechanism to enforce a SD-WAN path' head- 297 end selected route traversing through a list of specific nodes of 298 multiple network segments without requiring the nodes in each 299 network segment to have the intelligence (or maintaining states) of 300 selecting next hop or next domain. 302 There may be two approaches here: 303 1) Controller installs the entire SID stack at E1. 304 2) Controller delivers to E1 a "Key" that the SR ingress PE can use 305 to map to the SID stack for the packets arriving at the SR Ingress 306 PE. Section 4.2 & 4.3 will describe how the "Key" is carried by the 307 packets. 309 The Approach 1) requires less processing at the SR Ingress PE nodes, 310 but only works if the remote CPEs are in the same Administrative 311 domain as the SR domain. SR domain usually is not willing to expose 312 its internal binding SIDs to devices in different administration 313 domains. This approach also requires more changes to SD-WAN end 314 nodes and need more header bytes added to the packets when rd traversing through 3 party internet. Some SD-WAN nodes might not be 315 capable of supporting encapsulating packets with the SID stack. 317 The Approach 2) above requires SR Ingress PE nodes to map the "Key" 318 to the SID Stack and prepend the SID stack to the packets (Same 319 processing for other traffic except the mapping is from the received 320 "Key" carried in the payload). 322 4.1. Controller Delivers SID Stack to SD-WAN Head-end 324 This approach is straightforward. 326 E1 -------------------------- > SD-WAN controller 327 request for a SD-WAN path E1<->E2 with a specific SLA 329 E1 <-------------------------- SD-WAN controller 330 Reply with the Ingress PE Node ID or address 331 & the Binding SID. 333 Here is the packet header for SD-WAN Source Node to prepend to the 334 payload: 336 0 1 2 3 337 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 339 IPv4 Header: 340 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 341 |Version| IHL |Type of Service| Total Length | 342 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 343 | Identification |Flags| Fragment Offset | 344 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 345 | Time to Live | Prot.=17(UDP) | Header Checksum | 346 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 347 | SD-WAN Source IPv4 Address | 348 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 349 | SR Ingress PE IPv4 Address | 350 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 352 UDP Header: 353 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 354 | Source Port = | Dest. Port = 4754/4755 | 355 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 356 | UDP Length | UDP Checksum | 357 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 359 GRE Header: 360 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 361 |C| |K|S| Reserved0 | Ver | Protocol Type | 362 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 363 | Checksum (optional) | Reserved1 (Optional) | 364 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 365 | Key (optional) | 366 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 367 | Sequence Number (optional) | 368 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 370 To traverse SRv6 domain, SRv6 Header is appended after the GRE 371 header [SRv6-SRH]: 373 To traverse MPLS-SR domain, a stack of MPLS labels is appended after 374 GRE Header [MPLS-SR]. 376 4.2. Using GRE Key to Differentiate Flows 378 This section describes a method of SD-WAN head-end node using GRE 379 Key to indicate the desired property for different flows between SD- 380 WAN end-points (E1<->E2 in the figure above): such as different 381 desired routes through the SR Domain, different egress PEs based on 382 cost, performance or other factors. It might be difficult or 383 impossible to DiffServ bits carried by the packets to describe those 384 flow properties. 386 The SR Domain ingress can map the GRE key to different SID through 387 the SR Domain. 389 We assume that the SD-WAN Controller can determine which ingress PE 390 can lead to the optimal path between E1<->E2. It is beyond the scope 391 of this document on how SD-WAN controller computes the paths and how 392 & what SD-WAN controller communicates with the SR Domain controller. 394 Here is the sequence of the flow: 396 E1 -------------------------- > SD-WAN controller 397 request for a SD-WAN path E1<->E2 with a specific SLA 399 E1 <-------------------------- SD-WAN controller 400 Reply with the Ingress PE Node ID or address 401 & the GRE Key. 403 Note: the GRE key from the SD-WAN controller is for the ingress PE 404 to correlate desired Path with the list of SIDs to prepend the 405 packet across the SR domain. 407 When SD-WAN Controller get the E1<->E2 path request, it will 408 communicate with the VPN Controller to get the optimal Ingress PE 409 Node ID (or IP address) and the GRE key to encapsulate the original 410 packets between E1 <-> E2 (assuming IPsec Tunnel mode is used). 412 Upon receiving the GRE encapsulated packets, the provider ingress 413 Edge C1/C3 decapsulates the outer GRE tunnel header, use the GRE key 414 to map to the pre-defined (by the network controller) Binding SIDs, 415 prepend the Binding SIDs to the packets, and forward its desired 416 paths across the provider VPN. 418 Depending on how the SD-WAN path destination can be reached by the 419 egress PE, the egress PE has different processing procedure: 421 - If the destination of the SD-WAN path is directly attached to the 422 egress VPN PE node, the egress VPN PE decapsulates SR header and 423 forward the packets to SD-WAN path destination node, such as the E2 424 in the figure above. 425 - If the destination of the SD-WAN path is IP reachable via IPv4 426 network from the egress VPN PE node, the egress VPN PE node 427 decapsulates SR header and forward the packets to SD-WAN path 428 destination node via its internet facing port to the SD-WAN path 429 destination (i.e. the E2 node in the figure above). 430 - If the SD-WAN path is traversing multiple domains owned by different 431 network operators, the egress PE processing is described in the next 432 session. 434 4.3. Using UDP Source Port Number to Differentiate Flows 436 [RFC8086] describes how to use GRE-in-UDP source port number as 437 entropy for better ECMP performance. When the remotely attached CPEs 438 is within very close proximity to the PEs, e.g. only one or two 439 hopes away like in LTE access, there is less issue if ECMP put all 440 flows with same traffic classifier into one path. Then, those UDP 441 numbers can also be used as a key to SR PE nodes to map to the 442 appropriate SID to the packets. 444 Same as RFC8086, UDP source port values used as a key for SR PEs to 445 map to appropriate SIDs SHOULD be chosen from the ephemeral port 446 range (49152-65535) [RFC8085]. 448 The GRE-in-UDP encapsulation format contains a UDP header [RFC768] 449 and a GRE header [RFC2890]. The format is shown as follow 450 (presented in bit order): 452 0 1 2 3 453 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 455 IPv4 Header: 456 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 457 |Version| IHL |Type of Service| Total Length | 458 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 459 | Identification |Flags| Fragment Offset | 460 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 461 | Time to Live | Prot.=17(UDP) | Header Checksum | 462 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 463 | SD-WAN Source IPv4 Address | 464 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 465 | SR Ingress PE IPv4 Address | 466 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 468 UDP Header: 469 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 470 | Source Port = SIDs key Value | Dest. Port = 4754/4755 | 471 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 472 | UDP Length | UDP Checksum | 473 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 475 GRE Header: 476 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 477 |C| |K|S| Reserved0 | Ver | Protocol Type | 478 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 479 | Checksum (optional) | Reserved1 (Optional) | 480 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 481 | Key (optional) | 482 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 483 | Sequence Number (optional) | 484 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 486 Figure 3: UDP + GRE Headers in IPv4 488 Here is the GRE Header for IPv6 network, i.e. the SD-WAN Source SD- 489 WAN Destination, and SR PEs are all in IPv6 domain: 491 0 1 2 3 492 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 494 IPv6 Header: 495 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 496 |Version| Traffic Class | Flow Label | 497 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 498 | Payload Length | NxtHdr=17(UDP)| Hop Limit | 499 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 500 | | 501 + + 502 | | 503 + SD-WAN Source IPv6 Address + 504 | | 505 + + 506 | | 507 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 508 | | 509 + + 510 | | 511 + SR Domain Ingress PE IPv6 Address + 512 | | 513 + + 514 | | 515 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 517 UDP Header: 518 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 519 | Source Port = SIDs key value | Dest. Port = 4754/4755 | 520 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 521 | UDP Length | UDP Checksum | 522 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 524 GRE Header: 525 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 526 |C| |K|S| Reserved0 | Ver | Protocol Type | 527 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 528 | Checksum (optional) | Reserved1 (Optional) | 529 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 530 | Key (optional) | 531 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 532 | Sequence Number (optional) | 533 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 535 Figure 4: GRE+UDP for IPv6 537 4.4. GRE Header Extension 539 A new protocol type can be added to the GRE header [RFC2890] to make 540 it easier for the SR PE to do the proper actions: 542 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 543 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 544 |C| Reserved0 | Ver | Protocol Type | 545 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 546 | Checksum (optional) | Reserved1 (Optional) | 547 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 549 The proposed GRE header will have the following format: 551 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 552 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 553 |C| |K|S| Reserved0 | Ver | Protocol Type | 554 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 555 | Checksum (optional) | Reserved1 (Optional) | 556 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 557 | Key (optional) | 558 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 559 | Sequence Number (Optional) | 560 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 562 New protocol type (value to be assigned by IANA): 563 UDP-Key: Using UDP source port value as a Key for SR Ingress PE to 564 map to the appropriate SIDs. 566 GRE-KEY: Using GRE Key value as a key for SR ingress PE to map to 567 the appropriate SIDs 569 5. SD-WAN path over multiple SP managed domains 570 The following figure shows a SD-WAN Path E1<->E2 over two SP domains 571 which are interconnected by public internet. 573 +********SD-WAN Overlay Path *************+ 574 * * 575 * +------------+ * 576 * |SD-WAN Ctrl | * 577 * +===+------------+====E2/E3/E4.. * 578 * // * 579 * // <-----EVPN-VxLAN----> * 580 * +-+--+ ++-+ ++-+ +--+-+ * 581 ***+ E1 |==|C1| |C7+--+ E7 | * 582 A --+ | | | | | | + * 583 A2--+----+ ++-+ ++-+ ++---+ * 584 LTE || | SP1 | // * 585 +---+ SR +-+---+ +----+ * 586 +--------+ |C3 | Network | C4 | |E3 | * 587 | E4 | | | | |- --| | * 588 | | +---+-----+---+---+-+ +----+ * 589 +--------+=======+ C2 | || * 590 +---+-+ // * 591 // // * 592 +-+-+--------+--+-+ * 593 |D1 | |D4 | * 594 | | | | * 595 ++--+ ++---+ * 596 | SP2 | * 597 | SR +--+--+ +-+--+ 598 +--------+ | Network | D2 | |E2 +----Z 599 | E6 | | | +====+ +---Z2 600 | | +--+-----+ +-+---+LTE +----+ 601 +-+--+-+-+-------+ D3 |-----+ 602 +---+-+ 604 -- Directly attached 605 == || Public Internet or LTE path 606 ** Overlay path 607 Figure 5: SD-WAN path over two different SP domains 609 Let's assume that the SP-1 domain's egress node for the SD-WAN path 610 E1<->E2 is C2, which can reach D1 or D4 of SP-2 via public IP 611 network (say IPv4 network). 613 Let's also assume that the optimal route for some flows over SD-WAN 614 path E1<->E2 are C1->C2->D1 and other flows are over C1->C2->D4 (out 615 of the scope of this document on how the path is calculated). 617 If SP-1 is SR enabled, the mechanism described in Section 4 is 618 applicable to the SD-WAN path source node E1 and the SP-1's ingress 619 PE (e.g. C1 or C3 in the figure). 620 However, the processing at egress node might be different depending 621 on how the SP-1's egress edges are connected to the SP-2's ingress 622 edge nodes. 624 5.1. When Both SP domains support SR 626 There may be three approaches here: 628 1) Controller installs the entire SID stack at E1, and the SID list 629 contains SID entries belong to both domains. 631 2) Controller delivers to E1 the SID stack that only for the first 632 domain, but delivers to C6 (egress node of first domain) the binding 633 SID of the second domain. 635 3) Controller delivers a "Key" to E1, which can be encoded as GRE 636 KEY or represented by the Source UDP port of the GRE encapsulation, 637 for Ingress PE of the first SR Domain to map to its own SID stack as 638 described in Section 4. The first SR Domain will reserve the "Key" 639 through its domain and pass the "Key" to the second SR domain. The 640 second SR Domain Ingress node will use the same method to map the 641 "Key" to its SID stack. 643 5.2. When SP-2 does not support SR 645 Under this circumstance (which can be caused by SP-2 not supporting 646 SR or not willing to share Binding SIDs to SP-1), if the packets 647 arriving at SP-1 egress node C6 do not have any metadata indicating 648 the types of encrypted payload, C6 does not really have much choice 649 other than simply forwarding the packets to E2 via public IP 650 network. This way, the packets may or may not traverse through the 651 SP-2 domain. If the distance between C6 and E2 is far, the quality 652 of service can be unpredictable. 654 5.3. When SP-1 and SP-2 don't want to share network information 656 If SP-1's ingress node C1 can include the GRE KEY it receives from 657 E1 in the data packets' SR header, the SP-1's egress node can map 658 the Key to the SP-2's Ingress node and encapsulate the data packet 659 in a new GRE header destined towards the SP-2's Ingress node. Then 660 the SP-2's Ingress node can follow the procedure described in the 661 Section 4 to forward the data packets across its domain. 663 If the first SR Domain does not support adding metadata to carry the 664 "key" through its domain, the controller can deliver the "key" to 665 SP-1's egress node the same time as it delivers the key to E1, 666 knowing the SD-WAN path will need to traverse two domains with the 667 second one does support SR but the two SPs don't want to exchange 668 network information. 670 5.4. TLV to pass Metadata through SRv6 Domain 672 If SP-1 is SRv6 based, the ingress node C1 can append a TLV to the 673 end of the SR Header [SRv6-SRH] to carry the KEY it receives from 674 E1[Dc1]. 676 The SP-1 egress node C6 can get the mapping between the KEYs and the 677 Node-IDs (or Addresses) of the next domain's ingress edge node (i.e. 678 D1 or D4 in the figure 3 above) from its network controller ahead of 679 time. 681 0 1 2 3 682 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 683 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 684 | Type | Length | RESERVED | 685 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 686 | Key ID (4 octets) from the GRE tunnel remote ingress node | 687 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 688 | Optional // 689 | Node ID or address for the ingress node Next domain // 690 | Variable length (0~32 octets) // 691 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 693 TYPE: (to be assigned by IANA) is to indicate the TLV is for 694 carrying the flow identifier of the packet encoded by the SD-WAN 695 source node. 697 Upon receiving the packet, the egress node (C6) can 699 - find the Node-ID (or the address) for the next domain's ingress 700 node, 701 - construct a GRE header with the Key received from the TLV above 702 and the destination address from the mapping given by the 703 controller, 704 - encapsulate the GRE header to the data packet (which has 705 decapsulated SR header), 706 - and forward the packet to the public internet. 708 6. Security Considerations 710 Remotely attached CPEs might brought the following security risks: 712 1) Potential DDoS attack to the PEs with ports facing internet. 713 I.e. the PE resource being attacked by unwanted traffic. 714 2) Potential risk of provider VPN network bandwidth being stolen 715 by the entities who spoofed the addresses of SD-WAN end nodes. 717 To mitigate security risk of 1) above, it is absolutely necessary 718 for PEs which accept remotely attached CPEs or simply have ports 719 facing internet to enable Anti-DDoS feature to prevent major DDoS 720 attack to those PEs. 722 To mitigate the security risk of 2) above, RFC7510 defines the use 723 of DTLS to authenticate and encrypt the RFC7510 encapsulation. 725 However, for the scenario of SD-WAN source node being remotely 726 attached to PEs, using the method recommended by RFC7510 means the 727 source node has to perform DTLS on top of the IPSec encryption 728 between SD-WAN end points E1<->E2. This can be too processing heavy 729 for the SD-WAN end nodes. In addition, if there are many SD-WAN 730 flows to traverse through the ingress PE (e.g. C1, C2, C4 in the 731 figure 1 above), heavy processing is required on the ingress PEs. 733 Since the payload between E2<->E2 is already encrypted, the 734 confidentiality of the payload is already ensured. The network 735 operators need to balance between how much they can tolerant some 736 percentage of bandwidth being stolen and how much extra cost they 737 are willing to pay for completely prevent any unpaid traffic 738 traversing through its VPN networks. For operators who opt for lower 739 cost ingress PEs and CPEs, but can tolerant some percentage of 740 bandwidth being used by unpaid subscribers, a simple approach can be 741 considered: 743 - Embed a key in the packets, which can be changed periodically, 744 like the digital signature used by a certificate authority or 745 certification authority (CA). 746 - The key can be encoded in the GRE Key field between SD-WAN end 747 node and Ingress PE. Since GRE has 24 bits, some fixed bits 748 can be used to represent the signature of paid subscribers. 750 7. IANA Considerations 752 This document requires new protocol type: 754 Protocol type to be added to GRE header: SR_Route 756 8. References 758 8.1. Normative References 759 [RFC2890] G. Dommety "Key and Sequence Number Extensions to GRE". 760 Sep. 2000. 762 8.2. Informative References 764 [RFC2735] B. Fox, et al "NHRP Support for Virtual Private 765 networks". Dec. 1999. 767 [RFC8192] S. Hares, et al "Interface to Network Security Functions 768 (I2NSF) Problem Statement and Use Cases", July 2017 770 [ITU-T-X1036] ITU-T Recommendation X.1036, "Framework for creation, 771 storage, distribution and enforcement of policies for 772 network security", Nov 2007. 774 [RFC6071] S. Frankel and S. Krishnan, "IP Security (IPsec) and 775 Internet Key Exchange (IKE) Document Roadmap", Feb 2011. 777 [RFC4364] E. Rosen and Y. Rekhter, "BGP/MPLS IP Virtual Private 778 Networks (VPNs)", Feb 2006 780 [RFC4664] L. Andersson and E. Rosen, "Framework for Layer 2 Virtual 781 Private Networks (L2VPNs)", Sept 2006. 783 [SR-SD-WAN] D. Dukes, et al, "SR for SDWAN: VPN with Underlay SLA", 784 draft-dukes-sr-for-sdwan-00, in progress, Oct 2017 786 [SRv6-SRH] S. Previdi, et al, "IPv6 Segment Routing Header (SRH)", 787 draft-ietf-6man-segment-routing-header-13, in progress, 788 April 2018. 790 [MPLS-SR] A. Bashandy, et al, "Segment Routing with MPLS data 791 plane", draft-ietf-spring-segment-routing-mpls-13, in 792 progress, April 2018. 794 [RFC7510] X. Xu, et al, "Encapsulating MPLS in UDP", April 2015. 796 [RFC8086] L. Yong, et al, "GRE-in-UDP Encapsulation", March 2017. 798 [MEF-Cloud] "Cloud Services Architecture Technical Specification", 799 Work in progress, April 2018 801 9. Acknowledgments 803 Many thanks to Dean Cheng and Jim Guichard for the discussion and 804 contributions. 806 Authors' Addresses 808 Linda Dunbar 809 Huawei 810 Email: Linda.Dunbar@huawei.com 812 Mehmet Toy 813 Verizon 814 One Verizon Way 815 Basking Ridge, NJ 07920 816 Email: mehmet.toy@verizon.com