idnits 2.17.1 draft-dunbar-sr-sdwan-over-hybrid-networks-05.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack a Security Considerations section. (A line matching the expected section header was found, but with an unexpected indentation: ' 6. Security Considerations' ) ** The document seems to lack an IANA Considerations section. (See Section 2.2 of https://www.ietf.org/id-info/checklist for how to handle the case when there are no actions for IANA.) (A line matching the expected section header was found, but with an unexpected indentation: ' 7. IANA Considerations' ) ** There is 1 instance of too long lines in the document, the longest one being 95 characters in excess of 72. ** The document seems to lack a both a reference to RFC 2119 and the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords. RFC 2119 keyword, line 465: '...appropriate SIDs SHOULD be chosen from...' Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 15 has weird spacing: '...f which can b...' -- The document date (July 8, 2019) is 1744 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Missing Reference: 'RFC8085' is mentioned on line 466, but not defined == Missing Reference: 'RFC768' is mentioned on line 468, but not defined == Unused Reference: 'RFC2735' is defined on line 784, but no explicit reference was found in the text == Unused Reference: 'RFC8192' is defined on line 787, but no explicit reference was found in the text == Unused Reference: 'ITU-T-X1036' is defined on line 790, but no explicit reference was found in the text == Unused Reference: 'RFC6071' is defined on line 794, but no explicit reference was found in the text == Unused Reference: 'RFC4364' is defined on line 797, but no explicit reference was found in the text == Unused Reference: 'RFC4664' is defined on line 800, but no explicit reference was found in the text == Unused Reference: 'RFC7510' is defined on line 814, but no explicit reference was found in the text == Unused Reference: 'SDWAN-Net2Cloud' is defined on line 822, but no explicit reference was found in the text == Outdated reference: A later version (-01) exists of draft-dukes-sr-for-sdwan-00 == Outdated reference: A later version (-26) exists of draft-ietf-6man-segment-routing-header-13 == Outdated reference: A later version (-22) exists of draft-ietf-spring-segment-routing-mpls-13 == Outdated reference: A later version (-08) exists of draft-dunbar-bess-bgp-sdwan-usage-01 == Outdated reference: A later version (-39) exists of draft-ietf-rtgwg-net2cloud-problem-statement-04 Summary: 4 errors (**), 0 flaws (~~), 17 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 Network Working Group L. Dunbar 2 Internet Draft Futurewei 3 Intended status: Informational Mehmet Toy 4 Expires: January 8, 2020 Verizon 5 July 8, 2019 7 Segment routing for SD-WAN paths over hybrid networks 8 draft-dunbar-sr-sdwan-over-hybrid-networks-05 10 Abstract 12 This document describes a method for end-to-end (E2E) SD-WAN paths 13 to traverse specific list of underlay network segments, some of 14 which can be private networks which include SR enabled segments, 15 some of which can be the public IP networks that do not support SR, 16 to achieve the desired optimal E2E quality. 18 The method described in this draft uses the principle of segment 19 routing to enforce a SD-WAN path' head-end selected route traversing 20 through a list of specific nodes of multiple network segments 21 without requiring the nodes in each network segment to have the 22 intelligence (or maintaining states) of selecting next hop or next 23 domain. 25 Status of this Memo 27 This Internet-Draft is submitted in full conformance with the 28 provisions of BCP 78 and BCP 79. 30 This Internet-Draft is submitted in full conformance with the 31 provisions of BCP 78 and BCP 79. This document may not be modified, 32 and derivative works of it may not be created, except to publish it 33 as an RFC and to translate it into languages other than English. 35 Internet-Drafts are working documents of the Internet Engineering 36 Task Force (IETF), its areas, and its working groups. Note that 37 other groups may also distribute working documents as Internet- 38 Drafts. 40 Internet-Drafts are draft documents valid for a maximum of six 41 months and may be updated, replaced, or obsoleted by other documents 42 at any time. It is inappropriate to use Internet-Drafts as 43 reference material or to cite them other than as "work in progress." 45 The list of current Internet-Drafts can be accessed at 46 http://www.ietf.org/ietf/1id-abstracts.txt 48 The list of Internet-Draft Shadow Directories can be accessed at 49 http://www.ietf.org/shadow.html 51 This Internet-Draft will expire on January 8, 2009. 53 Copyright Notice 55 Copyright (c) 2019 IETF Trust and the persons identified as the 56 document authors. All rights reserved. 58 This document is subject to BCP 78 and the IETF Trust's Legal 59 Provisions Relating to IETF Documents 60 (http://trustee.ietf.org/license-info) in effect on the date of 61 publication of this document. Please review these documents 62 carefully, as they describe your rights and restrictions with 63 respect to this document. Code Components extracted from this 64 document must include Simplified BSD License text as described in 65 Section 4.e of the Trust Legal Provisions and are provided without 66 warranty as described in the Simplified BSD License. 68 Table of Contents 70 1. Introduction...................................................3 71 2. Definition of terms............................................4 72 3. Key Use Cases..................................................5 73 3.1. SD-WAN Path over LTE network and SR Domain................5 74 3.2. SD-WAN as Last Mile for Cloud DCs Access..................6 75 3.3. How & Why SR is useful for those use cases................7 76 4. Mechanism for SD-WAN path over one SR Domain and existing access 77 ..................................................................8 78 4.1. Controller Delivers SID Stack to SD-WAN Head-end..........9 79 4.2. Using GRE Key or VXLAN ID to Differentiate Flows.........11 80 4.3. Using UDP Source Port Number to Differentiate Flows......12 81 4.4. GRE Header Extension.....................................15 82 5. SD-WAN path over multiple SP managed domains..................16 83 5.1. When Both SP domains support SR..........................18 84 5.2. When SP-2 does not support SR............................18 85 5.3. When SP-1 and SP-2 don't want to share network information19 86 5.4. TLV to pass Metadata through SRv6 Domain.................19 87 6. Security Considerations.......................................20 88 7. IANA Considerations...........................................21 89 8. References....................................................21 90 8.1. Normative References.....................................21 91 8.2. Informative References...................................21 92 9. Acknowledgments...............................................23 94 1. Introduction 96 This document describes a method to enforce a SD-WAN path's head-end 97 selected route traversing through a list of specific nodes of 98 multiple network segments without requiring the nodes in each 99 network segments to have the intelligence (or maintaining states) of 100 selecting next hop or next segments. Those networks over which the 101 SD-WAN path traverse have at least one SR enabled network, and some 102 network segments (especially the last mile access portion) being 103 existing IP networks (such as existing IPv4, IPv6 or others). 105 SD-WAN, as described by ONUG (Open Network User Group) and [SDWAN- 106 Net2Cloud], is about pooling WAN bandwidth from multiple service 107 providers to get better WAN bandwidth management, visibility & 108 control. 110 Throughout this document, the term "Classic SD-WAN" refers to a pair 111 of CPEs in two locations aggregating N Service Providers' paths, 112 such as MPLS Paths and public internet paths. [SR-SD-WAN] describes 113 using explicit routes within the SRv6 or SR-MPLS enabled networks to 114 reach the desired quality for SD-WAN paths over the SRv6 or SR-MPLS 115 enabled networks respectively. 117 [BGP-SDWAN-Usage] describes three distinct SD-WAN scenarios from an 118 edge node's perspective: 120 1) All traffic is encrypted over WAN ports, so that network 121 service providers can extend its existing VPN to reach sites only 122 reachable via third party untrusted networks (such as public 123 internet); 125 2) Edge node has some ports connected to VPNs over which traffic 126 can go natively without encryption and other ports to the public 127 Internet over which traffic are encrypted; or 129 3) VPN PEs adding Internet facing WAN ports to offload low 130 priority traffic when the VPN backbone paths/links are congested. 132 This document focuses on the scenario 1) above, where a SD-WAN path 133 is spanning over SR enabled networks and over public IP networks, 134 such as existing IPv4, LTE, etc. Under this scenario, the endpoints 135 of the SD-WAN path (e.g. the CPE devices, one or both) are not 136 directly attached to PEs of a SR domain. 138 The goal is to place as large portion as possible of the SD-WAN path 139 over a provider VPN to achieve more optimal transport quality, or 140 steering the SD-WAN path traversing specific ingress/egress PEs to 141 reach optimal cost, quality, regulatory or other reasons. 143 2. Definition of terms 145 Cloud DC: Off-Premises Data Centers that usually host applications 146 and workload owned by different organizations or 147 tenants. 149 Controller: Used interchangeably with SD-WAN controller to manage 150 SD-WAN overlay path creation/deletion and monitoring the 151 path conditions between two or more sites. 153 DMVPN: Dynamic Multipoint Virtual Private Network. DMVPN is a 154 secure network that exchanges data between sites without 155 needing to pass traffic through an organization's 156 headquarter virtual private network (VPN) server or 157 router. 159 Heterogeneous Cloud: applications & workloads split among Cloud DCs 160 owned & managed by different Cloud Providers. 162 Hybrid Cloud: applications & workloads split between on-premises 163 Data centers and Cloud DCs. In this document Hybrid 164 Cloud also include heterogeneous cloud as well. 166 SD-WAN: Software Defined Wide Area Network, "SDWAN" refers to 167 the solutions of pooling WAN bandwidth from multiple 168 underlay networks to get better WAN bandwidth 169 management, visibility & control. When the underlay 170 networks are private, traffic can traverse without 171 additional encryption; when the underlay networks are 172 public, such as the Internet, some traffic needs to be 173 encrypted when traversing through (depending on user 174 provided policies). 176 SP: Network Service Provider 178 SR: Segment Routing 180 SR Domain: A domain that supports Segment Routing 182 VPC: Virtual Private Cloud. A service offered by many Cloud 183 DC operators to allocate a logically isolated cloud 184 resources, including computing, networking and storage. 186 3. Key Use Cases 188 3.1. SD-WAN Path over LTE network and SR Domain 190 MEF Cloud Service Architecture [MEF-Cloud] describes a use case of 191 network operators needing to use SD-WAN over LTE for the last mile 192 access that they do not have physical infrastructure, as shown 193 below: 195 +********SD-WAN Overlay Path *********+ 196 * * 197 * +------------+ * 198 * |SD-WAN Ctrl | * 199 * +===+------------+====+ * 200 * // \\ * 201 * // <---Overlay Path ---> \\ * 202 * +-+--+ +--+ +--+ +--+-+ * 203 ***+ E1 |==|C1+--------+C4+==+ E2 |****+ 204 A --+ | | | | | | +----Z 205 A2--+--||+ ++-+ ++-+ ++---+\---Z2 206 LTE || | | // 207 || | SR +-+---+ +----+ 208 || | Network | C6 | |E3 | 209 || | | |----| | 210 +===+-+---+ +-+---+ +----+ 211 + C3 |-------+ 212 +---+-+ 213 | 214 +---+-+ 215 | E4 | 216 +-----+ 217 -- Directly attached 218 == || Public Internet or LTE path 219 *** Overlay path 221 Figure 1: SD-WAN end points are attached to VPN via LTE 223 3.2. SD-WAN as Last Mile for Cloud DCs Access 225 Digital Transformation is propelling more and more enterprises to 226 move their workloads/Apps to cloud DCs that are geographically close 227 to their end users to improve end-to-end latency & overall user 228 experience, or to comply with local data protection regulations. 229 Conversely, workloads/Apps in those Cloud DCs can be easily shutdown 230 when their end users' geographic base changes. 232 Because of the ephemeral property of the selected Cloud DCs, an 233 enterprise or its network service provider may not have the direct 234 links to the Cloud DCs that are optimal for hosting the enterprise's 235 specific workloads/Apps. Under those circumstances, SD-WAN is a very 236 flexible choice to interconnect the enterprise on-premises data 237 centers & branch offices to its desired Cloud DCs. 239 However, SD-WAN paths over public internet can have unpredictable 240 performance, especially over long distances and cross state/country 241 boundaries. Therefore, it is highly desirable to place as much as 242 possible the portion of SD-WAN paths over service provider VPN (e.g. 243 enterprise's existing VPN) that have guaranteed SLA and to minimize 244 the distance/segments over public internet. 246 Under this scenario, one or both of the SD-WAN end points may not 247 directly attached to the PEs of a SR Domain. 249 3.3. How & Why SR is useful for those use cases 251 Let us assume that the SD-WAN Controller is capable of computing 252 optimal paths between two end-points (e.g. E1<->E2 in the Figure 2), 253 either by communicating with the SR Domain controller/management- 254 system, or by other methods which is out of the scope of this 255 document. 257 When a SR domain has multiple PEs with ports facing the external 258 networks (such as the public internet or LTE termination), SD-WAN 259 paths can traverse the SR domain via different ingress/egress PEs 260 resulting in different E2E performance. 262 In the diagram below, E1 <-> E2 SD-WAN (most likely IPsec encrypted 263 tunnel) path can traverse C1 <-> C4, C1<->C6, C3<->C6, or C3<->C4 264 within the VPN. There are many flows (by different Apps) between E1 265 <-> E2. Some flows may need to traverse C1<->C4, others may need to 266 traverse C3<->C6 or other segments within the VPN, which are 267 determined by the SD-WAN controller based on the characteristics & 268 need of the Apps, such as cost, available bandwidth, latency, or 269 special functions only available at specific locations, etc. 271 Even with the same ingress/egress, some flows may need different 272 segments across the SR Domain. It is not practical, or even 273 possible, for PEs (e.g. C1, C2, C3 in this example) to determine 274 which Apps' flows should egress C4 or C6 where both C4&C6 can reach 275 E2. 277 Segment Routing can be used to steer packets (or path) to traverse 278 the explicit egress node (C4 or C6), or explicit segments through 279 the SR Domain based on the SLA requested by the SD-WAN head-end 280 nodes. 282 +********SD-WAN Overlay Path *********+ 283 * * 284 * +------------+ * 285 * |SD-WAN Ctrl | * 286 * +===+------------+====+ * 287 * // \\ * 288 * // <-- Overlay path----> \\ * 289 * +-+--+ ++-+ ++-+ +--+-+ * 290 ***+ E1 |==|C1|--------|C4+==+ E2 |****+ 291 A --+ | | | | | | +----Z 292 A2--+--+-+ ++-+ ++-+ ++---+\---Z2 293 LTE || | | // 294 || | SR +-+---+ +----+ 295 || | Network | C6 | |E3 | 296 || | | |- --| | 297 || +-+---+ +-+---+ +----+ 298 +==+ C3 |-------+ 299 +---+-+ 300 | 301 +---+-+ 302 | E4 | 303 +-----+ 305 -- Directly attached 306 == || Public Internet or LTE path 307 ** Overlay path 309 Figure 2: SDWAN end points not directly attached to PEs of SR Domain 311 4. Mechanism for SD-WAN path over one SR Domain and existing access 313 This section describes the mechanism to enforce a SD-WAN path' head- 314 end selected route traversing through a list of specific nodes of 315 multiple network segments without requiring the nodes in each 316 network segment to have the intelligence (or maintaining states) of 317 selecting next hop or next domain. 319 There may be two approaches here: 321 1) Controller installs the entire SID stack at E1. 322 2) Controller delivers to E1 a "Key" that the SR ingress PE can use 323 to map to the SID stack for the packets arriving at the SR Ingress 324 PE. Section 4.2 & 4.3 will describe how the "Key" is carried by the 325 packets. 327 The Approach 1) requires less processing at the SR Ingress PE nodes, 328 but only works if the remote CPEs are in the same administrative 329 domain as the SR domain. SR domain usually is not willing to expose 330 its internal binding SIDs to devices in different administration 331 domains. This approach also requires more changes to SD-WAN end 332 nodes and need more header bytes added to the packets when rd traversing through 3 party internet. Some SD-WAN nodes might not be 333 capable of supporting encapsulating packets with the SID stack. 335 The Approach 2) above requires SR Ingress PE nodes to map the "Key" 336 to the SID Stack and prepend the SID stack to the packets (Same 337 processing for other traffic except the mapping is from the received 338 "Key" carried in the payload). 340 4.1. Controller Delivers SID Stack to SD-WAN Head-end 342 This approach is straightforward. 344 E1 -------------------------- > SD-WAN controller 345 request for a SD-WAN path E1<->E2 with a specific SLA 347 E1 <-------------------------- SD-WAN controller 348 Reply with the Ingress PE Node ID or address 349 & the Binding SID. 351 Here is the packet header for SD-WAN Source Node to prepend to the 352 payload: 354 0 1 2 3 355 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 357 IPv4 Header: 358 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 359 |Version| IHL |Type of Service| Total Length | 360 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 361 | Identification |Flags| Fragment Offset | 362 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 363 | Time to Live | Prot.=17(UDP) | Header Checksum | 364 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 365 | SD-WAN Source IPv4 Address | 366 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 367 | SR Ingress PE IPv4 Address | 368 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 370 UDP Header: 371 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 372 | Source Port = | Dest. Port = 4754/4755 | 373 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 374 | UDP Length | UDP Checksum | 375 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 377 GRE Header: 378 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 379 |C| |K|S| Reserved0 | Ver | Protocol Type | 380 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 381 | Checksum (optional) | Reserved1 (Optional) | 382 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 383 | Key (For SR Ingress to map to its SID) | 384 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 385 | Sequence Number (optional) | 386 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 388 To traverse SRv6 domain, SRv6 Header is appended after the GRE 389 header [SRv6-SRH]: 391 To traverse MPLS-SR domain, a stack of MPLS labels is appended after 392 GRE Header [MPLS-SR]. 394 4.2. Using GRE Key or VXLAN ID to Differentiate Flows 396 This section describes a method of SD-WAN head-end node using GRE 397 Key or VXLAN Network ID (VNI) to indicate the desired property for 398 specific flows between SD-WAN end-points (E1<->E2 in the figure 399 above): such as different desired routes through the SR Domain, 400 different egress PEs based on cost, performance or other factors. 401 It might be difficult or impossible for DiffServ bits carried by the 402 packets to describe those flow properties because there can be more 403 than what DiffServ bits can represent. 405 The SR Domain ingress can map the GRE key to different SID through 406 the SR Domain. 408 We assume that the SD-WAN Controller can determine which ingress PE 409 can lead to the optimal path between E1<->E2. It is beyond the scope 410 of this document on how SD-WAN controller computes the paths and how 411 & what SD-WAN controller communicates with the SR Domain controller. 413 Here is the sequence of the flow: 415 E1 -------------------------- > SD-WAN controller 416 request for a SD-WAN path E1<->E2 with a specific SLA 418 E1 <-------------------------- SD-WAN controller 419 Reply with the Ingress PE Node ID or address 420 & (the GRE Key or VXLAN ID). 422 Note: the GRE key (or VXLAN ID) from the SD-WAN controller is for 423 the ingress PEs to correlate desired Path with the list of SIDs to 424 prepend the packet across the SR domain. 426 When SD-WAN Controller get the E1<->E2 path request, it will 427 communicate with the VPN Controller to get the optimal Ingress PE 428 Node ID (or IP address) and the GRE key (or VXLAN ID) to encapsulate 429 the original packets between E1 <-> E2 (assuming IPsec Tunnel mode 430 is used). 432 Upon receiving the GRE encapsulated packets, the provider ingress 433 Edge C1/C3 uses the GRE key (or VXLAN ID) to map to the pre-defined 434 (by the network controller) Binding SIDs, prepend the Binding SIDs 435 to the packets, and forward its desired paths across the provider 436 VPN. 438 Depending on how the SD-WAN path destination can be reached by the 439 egress PE, the egress PE has different processing procedure: 441 - If the destination of the SD-WAN path is directly attached to 442 the egress VPN PE node, the egress VPN PE decapsulates SR 443 header and forward the packets to SD-WAN path destination node, 444 such as the E2 in the figure above. 445 - If the destination of the SD-WAN path is IP reachable via IPv4 446 network from the egress VPN PE node, the egress VPN PE node 447 decapsulates SR header and forward the packets to SD-WAN path 448 destination node via its internet facing port to the SD-WAN 449 path destination (i.e. the E2 node in the figure above). 450 - If the SD-WAN path is traversing multiple domains owned by 451 different network operators, the egress PE processing is 452 described in the next session. 454 4.3. Using UDP Source Port Number to Differentiate Flows 456 [RFC8086] describes how to use GRE-in-UDP source port number as 457 entropy for better ECMP performance. When the remotely attached CPEs 458 is within very close proximity to the PEs, e.g. only one or two 459 hopes away like in LTE access, there is less issue if ECMP put all 460 flows with same traffic classifier into one path. Then, those UDP 461 numbers can also be used as a key to SR PE nodes to map to the 462 appropriate SID to the packets. 464 Same as RFC8086, UDP source port values used as a key for SR PEs to 465 map to appropriate SIDs SHOULD be chosen from the ephemeral port 466 range (49152-65535) [RFC8085]. 468 The GRE-in-UDP encapsulation format contains a UDP header [RFC768] 469 and a GRE header [RFC2890]. The format is shown as follow 470 (presented in bit order): 472 0 1 2 3 473 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 475 IPv4 Header: 476 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 477 |Version| IHL |Type of Service| Total Length | 478 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 479 | Identification |Flags| Fragment Offset | 480 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 481 | Time to Live | Prot.=17(UDP) | Header Checksum | 482 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 483 | SD-WAN Source IPv4 Address | 484 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 485 | SR Ingress PE IPv4 Address | 486 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 488 UDP Header: 489 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 490 | Source Port = SIDs key Value | Dest. Port = 4754/4755 | 491 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 492 | UDP Length | UDP Checksum | 493 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 495 GRE Header: 496 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 497 |C| |K|S| Reserved0 | Ver | Protocol Type | 498 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 499 | Checksum (optional) | Reserved1 (Optional) | 500 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 501 | Key (optional) | 502 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 503 | Sequence Number (optional) | 504 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 506 Figure 3: UDP + GRE Headers in IPv4 508 Here is the GRE Header for IPv6 network, i.e. the SD-WAN Source SD- 509 WAN Destination, and SR PEs are all in IPv6 domain: 511 0 1 2 3 512 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 514 IPv6 Header: 515 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 516 |Version| Traffic Class | Flow Label | 517 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 518 | Payload Length | NxtHdr=17(UDP)| Hop Limit | 519 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 520 | | 521 + + 522 | | 523 + SD-WAN Source IPv6 Address + 524 | | 525 + + 526 | | 527 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 528 | | 529 + + 530 | | 531 + SR Domain Ingress PE IPv6 Address + 532 | | 533 + + 534 | | 535 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 537 UDP Header: 538 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 539 | Source Port = SIDs key value | Dest. Port = 4754/4755 | 540 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 541 | UDP Length | UDP Checksum | 542 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 543 GRE Header: 544 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 545 |C| |K|S| Reserved0 | Ver | Protocol Type | 546 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 547 | Checksum (optional) | Reserved1 (Optional) | 548 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 549 | Key (optional) | 550 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 551 | Sequence Number (optional) | 552 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 554 Figure 4: GRE+UDP for IPv6 556 4.4. GRE Header Extension 558 A new protocol type can be added to the GRE header [RFC2890] to make 559 it easier for the SR PE to do the proper actions: 561 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 562 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 563 |C| Reserved0 | Ver | Protocol Type | 564 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 565 | Checksum (optional) | Reserved1 (Optional) | 566 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 568 The proposed GRE header will have the following format: 570 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 571 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 572 |C| |K|S| Reserved0 | Ver | Protocol Type | 573 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 574 | Checksum (optional) | Reserved1 (Optional) | 575 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 576 | Key (optional) | 577 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 578 | Sequence Number (Optional) | 579 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 581 New protocol type (value to be assigned by IANA): 583 UDP-Key: Using UDP source port value as a Key for SR Ingress PE to 584 map to the appropriate SIDs. 586 GRE-KEY: Using GRE Key value as a key for SR ingress PE to map to 587 the appropriate SIDs 589 5. SD-WAN path over multiple SP managed domains 590 The following figure shows a SD-WAN Path E1<->E2 over two SP domains 591 which are interconnected by public internet. 593 +********SD-WAN Overlay Path *************+ 594 * * 595 * +------------+ * 596 * |SD-WAN Ctrl | * 597 * +===+------------+====E2/E3/E4.. * 598 * // * 599 * // * 600 * +-+--+ ++-+ ++-+ +--+-+ * 601 ***+ E1 |==|C1+--------+C7+--+ E7 | * 602 A --+ | | | | | | + * 603 A2--+----+ ++-+ ++-+ ++---+ * 604 LTE || | SP1 | // * 605 +---+ SR +-+---+ +----+ * 606 +--------+ |C3 | Network | C4 | |E3 | * 607 | E4 | | | | |- --| | * 608 | | +---+-----+---+---+-+ +----+ * 609 +--------+=======+ C2 | || * 610 +---+-+ // * 611 // // * 612 +-+-+--------+--+-+ * 613 |D1 | |D4 | * 614 | | | | * 615 ++--+ ++---+ * 616 | SP2 | * 617 | SR +--+--+ +-+--+ 618 +--------+ | Network | D2 | |E2 +----Z 619 | E6 | | | +====+ +---Z2 620 | | +--+-----+ +-+---+LTE +----+ 621 +-+--+-+-+-------+ D3 |-----+ 622 +---+-+ 624 -- Directly attached 625 == || Public Internet or LTE path 626 ** Overlay path 627 Figure 5: SD-WAN path over two different SP domains 629 Let's assume that the SP-1 domain's egress node for the SD-WAN path 630 E1<->E2 is C2, which can reach D1 or D4 of SP-2 via public IP 631 network (say IPv4 network). 633 Let's also assume that the optimal route for some flows over SD-WAN 634 path E1<->E2 are C1->C2->D1 and other flows are over C1->C2->D4 (out 635 of the scope of this document on how the path is calculated). 637 If SP-1 is SR enabled, the mechanism described in Section 4 is 638 applicable to the SD-WAN path source node E1 and the SP-1's ingress 639 PE (e.g. C1 or C3 in the figure). 640 However, the processing at egress node might be different depending 641 on how the SP-1's egress edges are connected to the SP-2's ingress 642 edge nodes. 644 5.1. When Both SP domains support SR 646 There may be three approaches here: 648 1) Controller installs the entire SID stack at E1, and the SID list 649 contains SID entries belong to both domains. 651 2) Controller delivers to E1 the SID stack that only for the first 652 domain, but delivers to C6 (egress node of first domain) the binding 653 SID of the second domain. 655 3) Controller delivers a "Key" to E1, which can be encoded as GRE 656 KEY or represented by the Source UDP port of the GRE encapsulation, 657 for Ingress PE of the first SR Domain to map to its own SID stack as 658 described in Section 4. The first SR Domain will reserve the "Key" 659 through its domain and pass the "Key" to the second SR domain. The 660 second SR Domain Ingress node will use the same method to map the 661 "Key" to its SID stack. 663 5.2. When SP-2 does not support SR 665 Under this circumstance (which can be caused by SP-2 not supporting 666 SR or not willing to share Binding SIDs to SP-1), if the packets 667 arriving at SP-1 egress node C6 do not have any metadata indicating 668 the types of encrypted payload, C6 does not really have much choice 669 other than simply forwarding the packets to E2 via public IP 670 network. This way, the packets may or may not traverse through the 671 SP-2 domain. If the distance between C6 and E2 is far, the quality 672 of service can be unpredictable. 674 5.3. When SP-1 and SP-2 don't want to share network information 676 If SP-1's ingress node C1 can include the GRE KEY it receives from 677 E1 in the data packets' SR header, the SP-1's egress node can map 678 the Key to the SP-2's Ingress node and encapsulate the data packet 679 in a new GRE header destined towards the SP-2's Ingress node. Then 680 the SP-2's Ingress node can follow the procedure described in the 681 Section 4 to forward the data packets across its domain. 683 If the first SR Domain does not support adding metadata to carry the 684 "key" through its domain, the controller can deliver the "key" to 685 SP-1's egress node the same time as it delivers the key to E1, 686 knowing the SD-WAN path will need to traverse two domains with the 687 second one does support SR but the two SPs don't want to exchange 688 network information. 690 5.4. TLV to pass Metadata through SRv6 Domain 692 If SP-1 is SRv6 based, the ingress node C1 can append a TLV to the 693 end of the SR Header [SRv6-SRH] to carry the KEY it receives from 694 E1. 696 The SP-1 egress node C6 can get the mapping between the KEYs and the 697 Node-IDs (or Addresses) of the next domain's ingress edge node (i.e. 698 D1 or D4 in the figure 3 above) from its network controller ahead of 699 time. 701 0 1 2 3 702 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 703 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 704 | Type | Length | RESERVED | 705 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 706 | Key ID (4 octets) from the GRE tunnel remote ingress node | 707 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 708 | Optional // 709 | Node ID or address for the ingress node Next domain // 710 | Variable length (0~32 octets) // 711 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 713 TYPE: (to be assigned by IANA) is to indicate the TLV is for 714 carrying the flow identifier of the packet encoded by the SD-WAN 715 source node. 717 Upon receiving the packet, the egress node (C6) can 719 - find the Node-ID (or the address) for the next domain's ingress 720 node, 721 - construct a GRE header with the Key received from the TLV above 722 and the destination address from the mapping given by the 723 controller, 724 - encapsulate the GRE header to the data packet (which has 725 decapsulated SR header), 726 - and forward the packet to the public internet. 728 6. Security Considerations 730 Remotely attached CPEs might brought the following security risks: 732 1) Potential DDoS attack to the PEs with ports facing internet. 733 I.e. the PE resource being attacked by unwanted traffic. 734 2) Potential risk of provider VPN network bandwidth being stolen 735 by the entities who spoofed the addresses of SD-WAN end nodes. 737 To mitigate security risk of 1) above, it is absolutely necessary 738 for PEs which accept remotely attached CPEs or simply have ports 739 facing internet to enable Anti-DDoS feature to prevent major DDoS 740 attack to those PEs. 742 To mitigate the security risk of 2) above, RFC7510 defines the use 743 of DTLS to authenticate and encrypt the RFC7510 encapsulation. 745 However, for the scenario of SD-WAN source node being remotely 746 attached to PEs, using the method recommended by RFC7510 means the 747 source node has to perform DTLS on top of the IPSec encryption 748 between SD-WAN end points E1<->E2. This can be too processing heavy 749 for the SD-WAN end nodes. In addition, if there are many SD-WAN 750 flows to traverse through the ingress PE (e.g. C1, C2, C4 in the 751 figure 1 above), heavy processing is required on the ingress PEs. 753 Since the payload between E2<->E2 is already encrypted, the 754 confidentiality of the payload is already ensured. The network 755 operators need to balance between how much they can tolerant some 756 percentage of bandwidth being stolen and how much extra cost they 757 are willing to pay for completely prevent any unpaid traffic 758 traversing through its VPN networks. For operators who opt for lower 759 cost ingress PEs and CPEs, but can tolerant some percentage of 760 bandwidth being used by unpaid subscribers, a simple approach can be 761 considered: 763 - Embed a key in the packets, which can be changed periodically, 764 like the digital signature used by a certificate authority or 765 certification authority (CA). 766 - The key can be encoded in the GRE Key field between SD-WAN end 767 node and Ingress PE. Since GRE has 24 bits, some fixed bits 768 can be used to represent the signature of paid subscribers. 770 7. IANA Considerations 772 This document requires new protocol type: 774 Protocol type to be added to GRE header: SR_Route 776 8. References 778 8.1. Normative References 779 [RFC2890] G. Dommety "Key and Sequence Number Extensions to GRE". 780 Sep. 2000. 782 8.2. Informative References 784 [RFC2735] B. Fox, et al "NHRP Support for Virtual Private 785 networks". Dec. 1999. 787 [RFC8192] S. Hares, et al "Interface to Network Security Functions 788 (I2NSF) Problem Statement and Use Cases", July 2017 790 [ITU-T-X1036] ITU-T Recommendation X.1036, "Framework for creation, 791 storage, distribution and enforcement of policies for 792 network security", Nov 2007. 794 [RFC6071] S. Frankel and S. Krishnan, "IP Security (IPsec) and 795 Internet Key Exchange (IKE) Document Roadmap", Feb 2011. 797 [RFC4364] E. Rosen and Y. Rekhter, "BGP/MPLS IP Virtual Private 798 Networks (VPNs)", Feb 2006 800 [RFC4664] L. Andersson and E. Rosen, "Framework for Layer 2 Virtual 801 Private Networks (L2VPNs)", Sept 2006. 803 [SR-SD-WAN] D. Dukes, et al, "SR for SDWAN: VPN with Underlay SLA", 804 draft-dukes-sr-for-sdwan-00, in progress, Oct 2017 806 [SRv6-SRH] S. Previdi, et al, "IPv6 Segment Routing Header (SRH)", 807 draft-ietf-6man-segment-routing-header-13, in progress, 808 April 2018. 810 [MPLS-SR] A. Bashandy, et al, "Segment Routing with MPLS data 811 plane", draft-ietf-spring-segment-routing-mpls-13, in 812 progress, April 2018. 814 [RFC7510] X. Xu, et al, "Encapsulating MPLS in UDP", April 2015. 816 [RFC8086] L. Yong, et al, "GRE-in-UDP Encapsulation", March 2017. 818 [BGP-SDWAN-Usage] L. Dunbar, et al, "BGP Usage for SDWAN Overlay 819 Networks, draft-dunbar-bess-bgp-sdwan-usage-01, in 820 progress, July 2019. 822 [SDWAN-Net2Cloud] L. Dunbar, et al, "Dynamic Networks to Hybrid 823 Cloud DCs Problem Statement", draft-ietf-rtgwg-net2cloud- 824 problem-statement-04, in progress, July 2019. 826 [MEF-Cloud] "Cloud Services Architecture Technical Specification", 827 Work in progress, April 2018 829 9. Acknowledgments 831 Many thanks to Dean Cheng and Jim Guichard for the discussion and 832 contributions. 834 Authors' Addresses 836 Linda Dunbar 837 Futurewei 838 Email: Linda.Dunbar@futurewei.com 840 Mehmet Toy 841 Verizon 842 One Verizon Way 843 Basking Ridge, NJ 07920 844 Email: mehmet.toy@verizon.com