idnits 2.17.1 draft-dunbar-sr-sdwan-over-hybrid-networks-01.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack a Security Considerations section. (A line matching the expected section header was found, but with an unexpected indentation: ' 6. Security Considerations' ) ** The document seems to lack an IANA Considerations section. (See Section 2.2 of https://www.ietf.org/id-info/checklist for how to handle the case when there are no actions for IANA.) (A line matching the expected section header was found, but with an unexpected indentation: ' 7. IANA Considerations' ) ** There are 5 instances of too long lines in the document, the longest one being 95 characters in excess of 72. ** The document seems to lack a both a reference to RFC 2119 and the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords. RFC 2119 keyword, line 447: '...appropriate SIDs SHOULD be chosen from...' Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (June 18, 2018) is 2140 days in the past. Is this intentional? -- Found something which looks like a code comment -- if you have code sections in the document, please surround them with '' and '' lines. Checking references for intended status: Informational ---------------------------------------------------------------------------- == Missing Reference: 'C2' is mentioned on line 598, but not defined == Missing Reference: 'SDWAN-C' is mentioned on line 577, but not defined == Missing Reference: 'SR-C' is mentioned on line 589, but not defined == Missing Reference: 'RFC8085' is mentioned on line 448, but not defined == Missing Reference: 'RFC768' is mentioned on line 450, but not defined == Missing Reference: 'D3' is mentioned on line 608, but not defined == Missing Reference: 'Dc1' is mentioned on line 678, but not defined == Unused Reference: 'RFC2735' is defined on line 768, but no explicit reference was found in the text == Unused Reference: 'RFC8192' is defined on line 771, but no explicit reference was found in the text == Unused Reference: 'ITU-T-X1036' is defined on line 774, but no explicit reference was found in the text == Unused Reference: 'RFC6071' is defined on line 778, but no explicit reference was found in the text == Unused Reference: 'RFC4364' is defined on line 781, but no explicit reference was found in the text == Unused Reference: 'RFC4664' is defined on line 784, but no explicit reference was found in the text == Unused Reference: 'RFC7510' is defined on line 798, but no explicit reference was found in the text == Outdated reference: A later version (-01) exists of draft-dukes-sr-for-sdwan-00 == Outdated reference: A later version (-26) exists of draft-ietf-6man-segment-routing-header-13 == Outdated reference: A later version (-22) exists of draft-ietf-spring-segment-routing-mpls-13 Summary: 4 errors (**), 0 flaws (~~), 18 warnings (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 Network Working Group L. Dunbar 2 Internet Draft Huawei 3 Intended status: Informational Mehmet Toy 4 Expires: January 2019 Verizon 5 June 18, 2018 7 Segment routing for SD-WAN paths over hybrid networks 8 draft-dunbar-sr-sdwan-over-hybrid-networks-01 10 Abstract 12 This document describes a method for end-to-end (E2E) SD-WAN paths 13 (most likely encrypted) to traverse specific list of network 14 segments, some of which are SR enabled and others may be legacy 15 networks that do not support SR, to achieve the desired optimal E2E 16 quality. 18 The method described in this draft uses the principle of segment 19 routing to enforce a SD-WAN path' head-end selected route traversing 20 through a list of specific nodes of multiple network segments 21 without requiring the nodes in each network segment to have the 22 intelligence (or maintaining states) of selecting next hop or next 23 domain. Those networks over which the SD-WAN path traverse have at 24 least one SR enabled network, and some network segments (especially 25 the last mile access portion) being legacy networks (such as legacy 26 IPv4, IPv6 or others). 28 Status of this Memo 30 This Internet-Draft is submitted in full conformance with the 31 provisions of BCP 78 and BCP 79. 33 This Internet-Draft is submitted in full conformance with the 34 provisions of BCP 78 and BCP 79. This document may not be modified, 35 and derivative works of it may not be created, except to publish it 36 as an RFC and to translate it into languages other than English. 38 Internet-Drafts are working documents of the Internet Engineering 39 Task Force (IETF), its areas, and its working groups. Note that 40 other groups may also distribute working documents as Internet- 41 Drafts. 43 Internet-Drafts are draft documents valid for a maximum of six 44 months and may be updated, replaced, or obsoleted by other documents 45 at any time. It is inappropriate to use Internet-Drafts as 46 reference material or to cite them other than as "work in progress." 48 The list of current Internet-Drafts can be accessed at 49 http://www.ietf.org/ietf/1id-abstracts.txt 51 The list of Internet-Draft Shadow Directories can be accessed at 52 http://www.ietf.org/shadow.html 54 This Internet-Draft will expire on December 18, 2018. 56 Copyright Notice 58 Copyright (c) 2018 IETF Trust and the persons identified as the 59 document authors. All rights reserved. 61 This document is subject to BCP 78 and the IETF Trust's Legal 62 Provisions Relating to IETF Documents 63 (http://trustee.ietf.org/license-info) in effect on the date of 64 publication of this document. Please review these documents 65 carefully, as they describe your rights and restrictions with 66 respect to this document. Code Components extracted from this 67 document must include Simplified BSD License text as described in 68 Section 4.e of the Trust Legal Provisions and are provided without 69 warranty as described in the Simplified BSD License. 71 Table of Contents 73 1. Introduction...................................................3 74 2. Definition of terms............................................4 75 3. Key Use Cases..................................................5 76 3.1. SD-WAN Path over LTE network and SR Domain................5 77 3.2. SD-WAN As Last Mile for Cloud DCs Access..................6 78 3.3. How & Why SR is useful for those use cases................7 79 4. Mechanism for SD-WAN path over one SR Domain and legacy access.8 80 4.1. Controller Delivers SID Stack to SD-WAN Head-end..........9 81 4.2. Using GRE Key to Differentiate Flows.....................11 82 4.3. Using UDP Source Port Number to Differentiate Flows......12 83 4.4. GRE Header Extension.....................................15 84 5. SD-WAN path over multiple SP managed domains..................16 85 5.1. When Both SP domains support SR..........................18 86 5.2. When SP-2 does not support SR............................18 87 5.3. When SP-1 and SP-2 don't want to share network information19 88 5.4. TLV to pass Metadata through SRv6 Domain.................19 89 6. Security Considerations.......................................20 90 7. IANA Considerations...........................................21 91 8. References....................................................21 92 8.1. Normative References.....................................21 93 8.2. Informative References...................................22 94 9. Acknowledgments...............................................23 96 1. Introduction 98 This document describes a method to enforce a SD-WAN path's head-end 99 selected route traversing through a list of specific nodes of 100 multiple network segments without requiring the nodes in each 101 network segments to have the intelligence (or maintaining states) of 102 selecting next hop or next segments. Those networks over which the 103 SD-WAN path traverse have at least one SR enabled network, and some 104 network segments (especially the last mile access portion) being 105 legacy networks (such as legacy IPv4, IPv6 or others). 107 SD-WAN, as described by ONUG (Open Network User Group), is about 108 pooling WAN bandwidth from n service providers to get better WAN 109 bandwidth management, visibility & control. 111 Throughout this document, the term "Classic SD-WAN" refers to a pair 112 of CPEs in two locations aggregating N Service Providers' paths, 113 such as MPLS Paths and public internet paths. [SR-SD-WAN] describes 114 using explicit routes within the SRv6 or SR-MPLS enabled networks to 115 reach the desired quality for SD-WAN paths over the SRv6 or SR-MPLS 116 enabled networks respectively. 118 Another way of using SD-WAN is for network service providers to 119 extend its existing VPN to reach sites to which they do not have 120 presence yet, with detailed use cases described in Section 3 of this 121 document. Under this scenario, the SD-WAN path is laid over multiple 122 hybrid network segments. This document focuses on this type of SD- 123 WAN where a portion of SD-WAN path is over SR enabled networks and 124 the other portion of the SD-WAN path is over legacy networks, such 125 as legacy IPv4, LTE, etc. Under this scenario, the endpoints of the 126 SD-WAN path (e.g. the CPE devices, one or both) are not directly 127 attached to PEs of a SR domain. 129 The goal is to place a large portion of the SD-WAN path over a 130 provider VPN to reach desired transport quality or making the SD-WAN 131 path traversing specific ingress/egress PEs for the desired cost, 132 quality or other reasons. 134 2. Definition of terms 136 Cloud DC: Off-Premises Data Centers that usually host applications 137 and workload owned by different organizations or 138 tenants. 140 Controller: Used interchangeably with SD-WAN controller to manage 141 SD-WAN overlay path creation/deletion and monitoring the 142 path conditions between two or more sites. 144 DMVPN: Dynamic Multipoint Virtual Private Network. DMVPN is a 145 secure network that exchanges data between sites without 146 needing to pass traffic through an organization's 147 headquarter virtual private network (VPN) server or 148 router. 150 Heterogeneous Cloud: applications & workloads split among Cloud DCs 151 owned & managed by different Cloud Providers. 153 Hybrid Cloud: applications & workloads split between on-premises 154 Data centers and Cloud DCs. In this document Hybrid 155 Cloud also include heterogeneous cloud as well. 157 SD-WAN: Software Defined Wide Area Network, which can mean many 158 different things. In this document, "SD-WAN" refers to 159 the solutions specified by ONUG (Open Network User 160 Group), https://www.onug.net/software-defined-wide-area- 161 network-sd-wan/, which is about pooling WAN bandwidth 162 from n service providers to get better WAN bandwidth 163 management, visibility & control. 165 SP: Network Service Provider 167 SR: Segment Routing 169 SR Domain: A domain that supports Segment Routing 171 VPC: Virtual Private Cloud. A service offered by many Cloud 172 DC operators to allocate a logically isolated cloud 173 resources, including compute, networking and storage. 175 3. Key Use Cases 177 3.1. SD-WAN Path over LTE network and SR Domain 179 MEF Cloud Service Architecture [MEF-Cloud] describes a use case of 180 network operators needing to use SD-WAN over LTE for the last mile 181 access that they do not have physical infrastructure, as shown 182 below: 184 SD-WAN Overlay 185 /-------------------------------------------/ 186 / [A]-----[E1]***********[E2]--------[Z] / 187 / * : / 188 / ******* : / 189 /-----------------------*--------:----------/ 190 LTE * : 191 Access * : 192 ***** ..***** 193 +------------+ /---*---*---------:----*--------/ 194 | SP | : / * * : * / 195 | Underlay | : / [C1]-[C3]------[C4]-[C6] / 196 +------------+ :/ \ / / / 197 : \ /--/----/ / 198 /: \ / / 199 / :...........[C2]..........................: 200 /-------------------------------/ 201 Figure 1: SD-WAN end points are attached to VPN via LTE 203 3.2. SD-WAN As Last Mile for Cloud DCs Access 205 Digital Transformation is propelling more and more enterprises to 206 move their workloads/Apps to cloud DCs that are geographically close 207 to their end users to improve end-to-end latency & overall user 208 experience, or to comply with local data protection regulations. 209 Conversely, workloads/Apps in those Cloud DCs can be easily shutdown 210 when their end users' geographic base changes. 212 Because of the ephemeral property of the selected Cloud DCs, an 213 enterprise or its network service provider may not have the direct 214 links to the Cloud DCs that are optimal for hosting the enterprise's 215 specific workloads/Apps. Under those circumstances, SD-WAN is a very 216 flexible choice to interconnect the enterprise on-premises data 217 centers & branch offices to its desired Cloud DCs. 219 However, SD-WAN paths over public internet can have unpredictable 220 performance, especially over long distances and cross state/country 221 boundaries. Therefore, it is highly desirable to place as much as 222 possible the portion of SD-WAN paths over service provider VPN (e.g. 223 enterprise's existing VPN) that have guaranteed SLA to minimize the 224 distance/segments over public internet. 226 Under this scenario, one or both of the SD-WAN end points may not 227 directly attached to the PEs of a SR Domain. 229 3.3. How & Why SR is useful for those use cases 231 Let us assume that the SD-WAN Controller is capable of computing 232 optimal paths between two end-points (e.g. E1<->E2 in the Figure 2), 233 either by communicating with the SR Domain controller/management- 234 system, or by other methods which is out of the scope of this 235 document. 237 The SR domain must have a set of PEs that have at least one port 238 facing the external networks (such as the public internet or LTE 239 termination). 241 Under this circumstance, SD-WAN end-points usually can reach 242 multiple PEs. 244 In the diagram below, E1 <-> E2 SD-WAN (most likely IPsec encrypted 245 tunnel) path can traverse C1 <-> C4, C1<->C6, C3<->C6, or C3<->C4 246 within the VPN. There are many flows (by different Apps) between E1 247 <-> E2. Some flows may need to traverse C1<->C4, others may need to 248 traverse C3<->C6 or other segments within the VPN, which are 249 determined by the SD-WAN controller based on the characteristics & 250 need of the Apps, such as cost, available bandwidth, latency, or 251 special functions only available at specific locations, etc. 253 Even with the same ingress/egress, some flows may need different 254 segments across the SR Domain. It is not practical, or even 255 possible, for PEs (e.g. C1, C2, C3 in this example) to determine 256 which Apps' flows should egress C4 or C6 where both C4&C6 can reach 257 E2. 259 Segment Routing can easily force the path to traverse the explicit 260 egress node (C4 or C6), or explicit segments through the SR Domain 261 based on the SLA requested by the SD-WAN head-end nodes. 263 +------------+ /-----------/ 264 | SDWAN | / / 265 | Control | / [SDWAN-C]... 266 +---+--------+ / / : 267 | /-----------/ : 268 | SD-WAN Overlay : 269 | /-------------------------------------------/ : 270 | / / : 271 | / [A]-----[E1]***********[E2]--------[Z] / : 272 | / * : / : 273 | / ******* : / : 274 | /-----------------------*--------:----------/ : 275 | * : : 276 | * : : 277 | /----------/ * : : 278 +---+--------+ / / * : : 279 | SP | / [SR-C] / * : : 280 | Control | / : / * : : 281 +------------+ /------:---/ ***** ..***** : 282 : * * : * : 283 : * * : * : 284 +------------+ : /---*---*---------:----*--------/ : 285 | SP | : / * * : * / : 286 | Underlay | : / [C1]-[C3]------[C4]-[C6] / : 287 +------------+ :/ \ / / / : 288 : \ /--/----/ / : 289 /: \ / / : 290 / :...........[C2]..........................: 291 / / 292 /-------------------------------/ 294 Figure 2: SDWAN end points not directly attached to PEs of SR Domain 296 4. Mechanism for SD-WAN path over one SR Domain and legacy access 298 This section describes the mechanism to enforce a SD-WAN path' head- 299 end selected route traversing through a list of specific nodes of 300 multiple network segments without requiring the nodes in each 301 network segment to have the intelligence (or maintaining states) of 302 selecting next hop or next domain. 304 There may be two approaches here: 305 1) Controller installs the entire SID stack at E1. 306 2) Controller delivers to E1 a "Key" that the SR ingress PE can use 307 to map to the SID stack for the packets arriving at the SR Ingress 308 PE. Section 4.2 & 4.3 will describe how the "Key" is carried by the 309 packets. 311 The Approach 1) requires less processing at the SR Ingress PE nodes, 312 but only works if the remote CPEs are in the same Administrative 313 domain as the SR domain. SR domain usually is not willing to expose 314 its internal binding SIDs to devices in different administration 315 domains. This approach also requires more changes to SD-WAN end 316 nodes and need more header bytes added to the packets when rd traversing through 3 party internet. Some SD-WAN nodes might not be 317 capable of supporting encapsulating packets with the SID stack. 319 The Approach 2) above requires SR Ingress PE nodes to map the "Key" 320 to the SID Stack and prepend the SID stack to the packets (Same 321 processing for other traffic except the mapping is from the received 322 "Key" carried in the payload). 324 4.1. Controller Delivers SID Stack to SD-WAN Head-end 326 This approach is straightforward. 328 E1 -------------------------- > SD-WAN controller 329 request for a SD-WAN path E1<->E2 with a specific SLA 331 E1 <-------------------------- SD-WAN controller 332 Reply with the Ingress PE Node ID or address 333 & the Binding SID. 335 Here is the packet header for SD-WAN Source Node to prepend to the 336 payload: 338 0 1 2 3 339 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 341 IPv4 Header: 342 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 343 |Version| IHL |Type of Service| Total Length | 344 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 345 | Identification |Flags| Fragment Offset | 346 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 347 | Time to Live | Prot.=17(UDP) | Header Checksum | 348 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 349 | SD-WAN Source IPv4 Address | 350 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 351 | SR Ingress PE IPv4 Address | 352 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 354 UDP Header: 355 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 356 | Source Port = | Dest. Port = 4754/4755 | 357 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 358 | UDP Length | UDP Checksum | 359 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 361 GRE Header: 362 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 363 |C| |K|S| Reserved0 | Ver | Protocol Type | 364 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 365 | Checksum (optional) | Reserved1 (Optional) | 366 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 367 | Key (optional) | 368 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 369 | Sequence Number (optional) | 370 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 372 To traverse SRv6 domain, SRv6 Header is appended after the GRE 373 header [SRv6-SRH]: 375 To traverse MPLS-SR domain, a stack of MPLS labels is appended after 376 GRE Header [MPLS-SR]. 378 4.2. Using GRE Key to Differentiate Flows 380 This section describes a method of SD-WAN head-end node using GRE 381 Key to indicate the desired property for different flows between SD- 382 WAN end-points (E1<->E2 in the figure above): such as different 383 desired routes through the SR Domain, different egress PEs based on 384 cost, performance or other factors. It might be difficult or 385 impossible to DiffServ bits carried by the packets to describe those 386 flow properties. 388 The SR Domain ingress can map the GRE key to different SID through 389 the SR Domain. 391 We assume that the SD-WAN Controller can determine which ingress PE 392 can lead to the optimal path between E1<->E2. It is beyond the scope 393 of this document on how SD-WAN controller computes the paths and how 394 & what SD-WAN controller communicates with the SR Domain controller. 396 Here is the sequence of the flow: 398 E1 -------------------------- > SD-WAN controller 399 request for a SD-WAN path E1<->E2 with a specific SLA 401 E1 <-------------------------- SD-WAN controller 402 Reply with the Ingress PE Node ID or address 403 & the GRE Key. 405 Note: the GRE key from the SD-WAN controller is for the ingress PE 406 to correlate desired Path with the list of SIDs to prepend the 407 packet across the SR domain. 409 When SD-WAN Controller get the E1<->E2 path request, it will 410 communicate with the VPN Controller to get the optimal Ingress PE 411 Node ID (or IP address) and the GRE key to encapsulate the original 412 packets between E1 <-> E2 (assuming IPsec Tunnel mode is used). 414 Upon receiving the GRE encapsulated packets, the provider ingress 415 Edge C1/C3 decapsulates the outer GRE tunnel header, use the GRE key 416 to map to the pre-defined (by the network controller) Binding SIDs, 417 prepend the Binding SIDs to the packets, and forward its desired 418 paths across the provider VPN. 420 Depending on how the SD-WAN path destination can be reached by the 421 egress PE, the egress PE has different processing procedure: 423 - If the destination of the SD-WAN path is directly attached to the 424 egress VPN PE node, the egress VPN PE decapsulates SR header and 425 forward the packets to SD-WAN path destination node, such as the E2 426 in the figure above. 427 - If the destination of the SD-WAN path is IP reachable via IPv4 428 network from the egress VPN PE node, the egress VPN PE node 429 decapsulates SR header and forward the packets to SD-WAN path 430 destination node via its internet facing port to the SD-WAN path 431 destination (i.e. the E2 node in the figure above). 432 - If the SD-WAN path is traversing multiple domains owned by different 433 network operators, the egress PE processing is described in the next 434 session. 436 4.3. Using UDP Source Port Number to Differentiate Flows 438 [RFC8086] describes how to use GRE-in-UDP source port number as 439 entropy for better ECMP performance. When the remotely attached CPEs 440 is within very close proximity to the PEs, e.g. only one or two 441 hopes away like in LTE access, there is less issue if ECMP put all 442 flows with same traffic classifier into one path. Then, those UDP 443 numbers can also be used as a key to SR PE nodes to map to the 444 appropriate SID to the packets. 446 Same as RFC8086, UDP source port values used as a key for SR PEs to 447 map to appropriate SIDs SHOULD be chosen from the ephemeral port 448 range (49152-65535) [RFC8085]. 450 The GRE-in-UDP encapsulation format contains a UDP header [RFC768] 451 and a GRE header [RFC2890]. The format is shown as follow 452 (presented in bit order): 454 0 1 2 3 455 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 457 IPv4 Header: 458 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 459 |Version| IHL |Type of Service| Total Length | 460 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 461 | Identification |Flags| Fragment Offset | 462 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 463 | Time to Live | Prot.=17(UDP) | Header Checksum | 464 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 465 | SD-WAN Source IPv4 Address | 466 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 467 | SR Ingress PE IPv4 Address | 468 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 470 UDP Header: 471 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 472 | Source Port = SIDs key Value | Dest. Port = 4754/4755 | 473 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 474 | UDP Length | UDP Checksum | 475 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 477 GRE Header: 478 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 479 |C| |K|S| Reserved0 | Ver | Protocol Type | 480 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 481 | Checksum (optional) | Reserved1 (Optional) | 482 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 483 | Key (optional) | 484 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 485 | Sequence Number (optional) | 486 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 488 Figure 3: UDP + GRE Headers in IPv4 490 Here is the GRE Header for IPv6 network, i.e. the SD-WAN Source SD- 491 WAN Destination, and SR PEs are all in IPv6 domain: 493 0 1 2 3 494 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 496 IPv6 Header: 497 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 498 |Version| Traffic Class | Flow Label | 499 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 500 | Payload Length | NxtHdr=17(UDP)| Hop Limit | 501 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 502 | | 503 + + 504 | | 505 + SD-WAN Source IPv6 Address + 506 | | 507 + + 508 | | 509 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 510 | | 511 + + 512 | | 513 + SR Domain Ingress PE IPv6 Address + 514 | | 515 + + 516 | | 517 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 519 UDP Header: 520 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 521 | Source Port = SIDs key value | Dest. Port = 4754/4755 | 522 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 523 | UDP Length | UDP Checksum | 524 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 526 GRE Header: 527 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 528 |C| |K|S| Reserved0 | Ver | Protocol Type | 529 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 530 | Checksum (optional) | Reserved1 (Optional) | 531 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 532 | Key (optional) | 533 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 534 | Sequence Number (optional) | 535 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 537 Figure 4: GRE+UDP for IPv6 539 4.4. GRE Header Extension 541 A new protocol type can be added to the GRE header [RFC2890] to make 542 it easier for the SR PE to do the proper actions: 544 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 545 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 546 |C| Reserved0 | Ver | Protocol Type | 547 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 548 | Checksum (optional) | Reserved1 (Optional) | 549 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 551 The proposed GRE header will have the following format: 553 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 554 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 555 |C| |K|S| Reserved0 | Ver | Protocol Type | 556 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 557 | Checksum (optional) | Reserved1 (Optional) | 558 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 559 | Key (optional) | 560 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 561 | Sequence Number (Optional) | 562 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 564 New protocol type (value to be assigned by IANA): 565 UDP-Key: Using UDP source port value as a Key for SR Ingress PE to 566 map to the appropriate SIDs. 568 GRE-KEY: Using GRE Key value as a key for SR ingress PE to map to 569 the appropriate SIDs 571 5. SD-WAN path over multiple SP managed domains 572 The following figure shows a SD-WAN Path E1<->E2 over two SP domains 573 which are interconnected by public internet. 575 +------------+ 576 | SDWAN | /-----------/ 577 | Control | / [SDWAN-C]... 578 +---+--------+ / / : 579 | /-----------/ : 580 | SD-WAN Overlay : 581 | /-------------------------------------------/ : 582 | / / : 583 | / [A]-----[E1]***********[E2]--------[Z] / : 584 +---+ / * : / : 585 | | / ******* : / : 586 | | /-----------------------*--------:----------/ : 587 | | /----------/ * .. .. .. .. .. :. : 588 | +-+--------+ / / * : : 589 | | SP-1 | / [SR-C] / * : : 590 | |Control | / : / * : : 591 | +----------+ /------:---/ ***** : : 592 | +----------+ : /---*---*-----------------------/ : : 593 | | SP-1 | : / * * / : : 594 | |Underlay | : / [C1]-[C3]------[C4]-[C6] / : : 595 | +----------+ :/ \ /* / / : : 596 | : \ /--/-*--/ / : : 597 | +----------+ /: \ / * / : : 598 +-+ SP-2 | / :...........[C2] * / : : 599 |Control | /-------------**-----*----------/ : : 600 +----------+ * * * .. .. .. .. ..: : 601 * * : : 602 +------------+ /---*---*---------:-------------/ : 603 | SP-2 | / * * : / : 604 | Underlay | / [D1]-[D4]------[D2]-[D5] / : 605 +------------+ / \ / / / : 606 / \ /--/----/ / : 607 / \ / / : 608 / [D3]..........................: 609 / / 610 /-------------------------------/ 611 Figure 5: SD-WAN path over two different SP domains 613 Let's assume that the SP-1 domain's egress node for the SD-WAN path 614 E1<->E2 is C2, which can reach D1 or D4 of SP-2 via public IP 615 network (say IPv4 network). 617 Let's also assume that the optimal route for some flows over SD-WAN 618 path E1<->E2 are C1->C2->D1 and other flows are over C1->C2->D4 (out 619 of the scope of this document on how the path is calculated). 621 If SP-1 is SR enabled, the mechanism described in Section 4 is 622 applicable to the SD-WAN path source node E1 and the SP-1's ingress 623 PE (e.g. C1 or C3 in the figure). 624 However, the processing at egress node might be different depending 625 on how the SP-1's egress edges are connected to the SP-2's ingress 626 edge nodes. 628 5.1. When Both SP domains support SR 630 There may be three approaches here: 632 1) Controller installs the entire SID stack at E1, and the SID list 633 contains SID entries belong to both domains. 635 2) Controller delivers to E1 the SID stack that only for the first 636 domain, but delivers to C6 (egress node of first domain) the binding 637 SID of the second domain. 639 3) Controller delivers a "Key" to E1, which can be encoded as GRE 640 KEY or represented by the Source UDP port of the GRE encapsulation, 641 for Ingress PE of the first SR Domain to map to its own SID stack as 642 described in Section 4. The first SR Domain will reserve the "Key" 643 through its domain and pass the "Key" to the second SR domain. The 644 second SR Domain Ingress node will use the same method to map the 645 "Key" to its SID stack. 647 5.2. When SP-2 does not support SR 649 Under this circumstance (which can be caused by SP-2 not supporting 650 SR or not willing to share Binding SIDs to SP-1), if the packets 651 arriving at SP-1 egress node C6 do not have any metadata indicating 652 the types of encrypted payload, C6 does not really have much choice 653 other than simply forwarding the packets to E2 via public IP 654 network. This way, the packets may or may not traverse through the 655 SP-2 domain. If the distance between C6 and E2 is far, the quality 656 of service can be unpredictable. 658 5.3. When SP-1 and SP-2 don't want to share network information 660 If SP-1's ingress node C1 can include the GRE KEY it receives from 661 E1 in the data packets' SR header, the SP-1's egress node can map 662 the Key to the SP-2's Ingress node and encapsulate the data packet 663 in a new GRE header destined towards the SP-2's Ingress node. Then 664 the SP-2's Ingress node can follow the procedure described in the 665 Section 4 to forward the data packets across its domain. 667 If the first SR Domain does not support adding metadata to carry the 668 "key" through its domain, the controller can deliver the "key" to 669 SP-1's egress node the same time as it delivers the key to E1, 670 knowing the SD-WAN path will need to traverse two domains with the 671 second one does support SR but the two SPs don't want to exchange 672 network information. 674 5.4. TLV to pass Metadata through SRv6 Domain 676 If SP-1 is SRv6 based, the ingress node C1 can append a TLV to the 677 end of the SR Header [SRv6-SRH] to carry the KEY it receives from 678 E1[Dc1]. 680 The SP-1 egress node C6 can get the mapping between the KEYs and the 681 Node-IDs (or Addresses) of the next domain's ingress edge node (i.e. 682 D1 or D4 in the figure 3 above) from its network controller ahead of 683 time. 685 0 1 2 3 686 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 687 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 688 | Type | Length | RESERVED | 689 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 690 | Key ID (4 octets) from the GRE tunnel remote ingress node | 691 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 692 | Optional // 693 | Node ID or address for the ingress node Next domain // 694 | Variable length (0~32 octets) // 695 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 697 TYPE: (to be assigned by IANA) is to indicate the TLV is for 698 carrying the flow identifier of the packet encoded by the SD-WAN 699 source node. 701 Upon receiving the packet, the egress node (C6) can 703 - find the Node-ID (or the address) for the next domain's ingress 704 node, 705 - construct a GRE header with the Key received from the TLV above 706 and the destination address from the mapping given by the 707 controller, 708 - encapsulate the GRE header to the data packet (which has 709 decapsulated SR header), 710 - and forward the packet to the public internet. 712 6. Security Considerations 714 Remotely attached CPEs might brought the following security risks: 716 1) Potential DDoS attack to the PEs with ports facing internet. 717 I.e. the PE resourced being attacked by unwanted traffic. 718 2) Potential risk of provider VPN network bandwidth being stolen 719 by the entities who spoofed the addresses of SD-WAN end nodes. 721 To mitigate security risk of 1) above, it is absolutely necessary 722 for PEs which accept remotely attached CPEs or simply have ports 723 facing internet to enable Anti-DDoS feature to prevent major DDoS 724 attack to those PEs. 726 To mitigate the security risk of 2) above, RFC7510 defines the use 727 of DTLS to authenticate and encrypt the RFC7510 encapsulation. 729 However, for the scenario of SD-WAN source node being remotely 730 attached to PEs, using the method recommended by RFC7510 means the 731 source node has to perform DTLS on top of the IPSec encryption 732 between SD-WAN end points E1<->E2. This can be too processing heavy 733 for the SD-WAN end nodes. In addition, if there are many SD-WAN 734 flows to traverse through the ingress PE (e.g. C1, C2, C4 in the 735 figure 1 above), heavy processing is required on the ingress PEs. 737 Since the payload between E2<->E2 is already encrypted, the 738 confidentiality of the payload is already ensured. The network 739 operators need to balance between how much they can tolerant some 740 percentage of bandwidth being stolen and how much extra cost they 741 are willing to pay for completely prevent any unpaid traffic 742 traversing through its VPN networks. For operators who opt for lower 743 cost ingress PEs and CPEs, but can tolerant some percentage of 744 bandwidth being used by unpaid subscribers, a simple approach can be 745 considered: 747 - Embed a key in the packets, which can be changed periodically, 748 like the digital signature used by a certificate authority or 749 certification authority (CA). 750 - The key can be encoded in the GRE Key field between SD-WAN end 751 node and Ingress PE. Since GRE has 24 bits, some fixed bits 752 can be used to represent the signature of paid subscribers. 754 7. IANA Considerations 756 This document requires new protocol type: 758 Protocol type to be added to GRE header: SR_Route 760 8. References 762 8.1. Normative References 763 [RFC2890] G. Dommety "Key and Sequence Number Extensions to GRE". 764 Sep. 2000. 766 8.2. Informative References 768 [RFC2735] B. Fox, et al "NHRP Support for Virtual Private 769 networks". Dec. 1999. 771 [RFC8192] S. Hares, et al "Interface to Network Security Functions 772 (I2NSF) Problem Statement and Use Cases", July 2017 774 [ITU-T-X1036] ITU-T Recommendation X.1036, "Framework for creation, 775 storage, distribution and enforcement of policies for 776 network security", Nov 2007. 778 [RFC6071] S. Frankel and S. Krishnan, "IP Security (IPsec) and 779 Internet Key Exchange (IKE) Document Roadmap", Feb 2011. 781 [RFC4364] E. Rosen and Y. Rekhter, "BGP/MPLS IP Virtual Private 782 Networks (VPNs)", Feb 2006 784 [RFC4664] L. Andersson and E. Rosen, "Framework for Layer 2 Virtual 785 Private Networks (L2VPNs)", Sept 2006. 787 [SR-SD-WAN] D. Dukes, et al, "SR for SDWAN: VPN with Underlay SLA", 788 draft-dukes-sr-for-sdwan-00, in progress, Oct 2017 790 [SRv6-SRH] S. Previdi, et al, "IPv6 Segment Routing Header (SRH)", 791 draft-ietf-6man-segment-routing-header-13, in progress, 792 April 2018. 794 [MPLS-SR] A. Bashandy, et al, "Segment Routing with MPLS data 795 plane", draft-ietf-spring-segment-routing-mpls-13, in 796 progress, April 2018. 798 [RFC7510] X. Xu, et al, "Encapsulating MPLS in UDP", April 2015. 800 [RFC8086] L. Yong, et al, "GRE-in-UDP Encapsulation", March 2017. 802 [MEF-Cloud] "Cloud Services Architecture Technical Specification", 803 Work in progress, April 2018 805 9. Acknowledgments 807 Many thanks to Dean Cheng and Jim Guichard for the discussion and 808 contributions. 810 Authors' Addresses 812 Linda Dunbar 813 Huawei 814 Email: Linda.Dunbar@huawei.com 816 Mehmet Toy 817 Verizon 818 One Verizon Way 819 Basking Ridge, NJ 07920 820 Email: mehmet.toy@verizon.com