idnits 2.17.1 draft-dunbar-sr-sdwan-over-hybrid-networks-06.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack a Security Considerations section. (A line matching the expected section header was found, but with an unexpected indentation: ' 6. Security Considerations' ) ** The document seems to lack an IANA Considerations section. (See Section 2.2 of https://www.ietf.org/id-info/checklist for how to handle the case when there are no actions for IANA.) (A line matching the expected section header was found, but with an unexpected indentation: ' 7. IANA Considerations' ) ** There is 1 instance of too long lines in the document, the longest one being 89 characters in excess of 72. ** The document seems to lack a both a reference to RFC 2119 and the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords. RFC 2119 keyword, line 461: '...appropriate SIDs SHOULD be chosen from...' Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 15 has weird spacing: '... which can b...' -- The document date (November 4, 2019) is 1635 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Missing Reference: 'RFC8085' is mentioned on line 462, but not defined == Missing Reference: 'RFC768' is mentioned on line 464, but not defined == Unused Reference: 'RFC2735' is defined on line 780, but no explicit reference was found in the text == Unused Reference: 'RFC8192' is defined on line 783, but no explicit reference was found in the text == Unused Reference: 'ITU-T-X1036' is defined on line 786, but no explicit reference was found in the text == Unused Reference: 'RFC6071' is defined on line 790, but no explicit reference was found in the text == Unused Reference: 'RFC4364' is defined on line 793, but no explicit reference was found in the text == Unused Reference: 'RFC4664' is defined on line 796, but no explicit reference was found in the text == Unused Reference: 'RFC7510' is defined on line 810, but no explicit reference was found in the text == Unused Reference: 'SDWAN-Net2Cloud' is defined on line 818, but no explicit reference was found in the text == Outdated reference: A later version (-01) exists of draft-dukes-sr-for-sdwan-00 == Outdated reference: A later version (-26) exists of draft-ietf-6man-segment-routing-header-13 == Outdated reference: A later version (-22) exists of draft-ietf-spring-segment-routing-mpls-13 == Outdated reference: A later version (-08) exists of draft-dunbar-bess-bgp-sdwan-usage-01 == Outdated reference: A later version (-39) exists of draft-ietf-rtgwg-net2cloud-problem-statement-04 Summary: 4 errors (**), 0 flaws (~~), 17 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 Network Working Group L. Dunbar 2 Internet Draft Futurewei 3 Intended status: Informational Mehmet Toy 4 Expires: May 4, 2020 Verizon 5 November 4, 2019 7 Segment routing for SDWAN paths over hybrid networks 8 draft-dunbar-sr-sdwan-over-hybrid-networks-06 10 Abstract 12 This document describes a method for end-to-end (E2E) SDWAN paths to 13 traverse specific list of underlay network segments, some of which 14 can be private networks which include SR enabled segments, some of 15 which can be the public IP networks that do not support SR, to 16 achieve the desired optimal E2E quality. 18 The method described in this draft uses the principle of segment 19 routing to enforce a SDWAN path's head-end selected route traversing 20 through a list of specific nodes of multiple network segments 21 without requiring the nodes in each network segment to have the 22 intelligence (or maintaining states) of selecting next hop or next 23 domain. 25 Status of this Memo 27 This Internet-Draft is submitted in full conformance with the 28 provisions of BCP 78 and BCP 79. 30 This Internet-Draft is submitted in full conformance with the 31 provisions of BCP 78 and BCP 79. This document may not be modified, 32 and derivative works of it may not be created, except to publish it 33 as an RFC and to translate it into languages other than English. 35 Internet-Drafts are working documents of the Internet Engineering 36 Task Force (IETF), its areas, and its working groups. Note that 37 other groups may also distribute working documents as Internet- 38 Drafts. 40 Internet-Drafts are draft documents valid for a maximum of six 41 months and may be updated, replaced, or obsoleted by other documents 42 at any time. It is inappropriate to use Internet-Drafts as 43 reference material or to cite them other than as "work in progress." 45 The list of current Internet-Drafts can be accessed at 46 http://www.ietf.org/ietf/1id-abstracts.txt 48 The list of Internet-Draft Shadow Directories can be accessed at 49 http://www.ietf.org/shadow.html 51 This Internet-Draft will expire on May 4, 2009. 53 Copyright Notice 55 Copyright (c) 2019 IETF Trust and the persons identified as the 56 document authors. All rights reserved. 58 This document is subject to BCP 78 and the IETF Trust's Legal 59 Provisions Relating to IETF Documents 60 (http://trustee.ietf.org/license-info) in effect on the date of 61 publication of this document. Please review these documents 62 carefully, as they describe your rights and restrictions with 63 respect to this document. Code Components extracted from this 64 document must include Simplified BSD License text as described in 65 Section 4.e of the Trust Legal Provisions and are provided without 66 warranty as described in the Simplified BSD License. 68 Table of Contents 70 1. Introduction...................................................3 71 2. Definition of terms............................................4 72 3. Key Use Cases..................................................5 73 3.1. SDWAN Path over LTE network and SR Domain.................5 74 3.2. SDWAN as Last Mile for Cloud DCs Access...................6 75 3.3. How & Why SR is useful for those use cases................7 76 4. Mechanism for SDWAN path over one SR Domain and existing access8 77 4.1. Controller Delivers SID Stack to SDWAN Head-end...........9 78 4.2. Using GRE Key or VXLAN ID to Differentiate Flows.........11 79 4.3. Using UDP Source Port Number to Differentiate Flows......12 80 4.4. GRE Header Extension.....................................15 82 5. SDWAN path over multiple SP managed domains...................16 83 5.1. When Both SP domains support SR..........................17 84 5.2. When SP-2 does not support SR............................17 85 5.3. When SP-1 and SP-2 don't want to share network information18 86 5.4. TLV to pass Metadata through SRv6 Domain.................18 87 6. Security Considerations.......................................19 88 7. IANA Considerations...........................................20 89 8. References....................................................20 90 8.1. Normative References.....................................20 91 8.2. Informative References...................................20 92 9. Acknowledgments...............................................22 94 1. Introduction 96 This document describes a method to enforce a SDWAN path's head-end 97 selected route traversing through a list of specific nodes of 98 multiple network segments without requiring the nodes in each 99 network segments to have the intelligence (or maintaining states) of 100 selecting next hop or next segments. Those networks over which the 101 SDWAN path traverse have at least one SR enabled network, and some 102 network segments (especially the last mile access portion) being 103 existing IP networks (such as existing IPv4, IPv6 or others). 105 Throughout this document, the term "Classic SDWAN" refers to a pair 106 of CPEs in two locations aggregating N Service Providers' paths, 107 such as MPLS Paths and public internet paths. Classic SDWAN is like 108 what ONUG (Open Network User Group) has advocated, i.e. about 109 pooling WAN bandwidth from multiple service providers to get better 110 WAN bandwidth management, visibility & control. 112 [SR-SDWAN] describes using explicit routes within the SRv6 or SR- 113 MPLS enabled networks to reach the desired quality for SDWAN paths 114 over the SRv6 or SR-MPLS enabled networks respectively. 116 [BGP-SDWAN-Usage] describes three distinct SDWAN scenarios from an 117 edge node's perspective: 119 1) Edge nodes interconnected by IPsec tunnels. All traffic among 120 edge nodes are encrypted, i.e. over various IPsec tunnels. 122 2) Edge node has some ports connected to VPNs over which traffic 123 can go natively without encryption and other ports to the public 124 Internet over which traffic are encrypted; or 126 3) VPN PEs adding Internet facing WAN ports to offload low 127 priority traffic when the VPN backbone paths/links are congested. 129 This document discusses using SR to steer traffic over combination 130 of trusted network segments and untrusted segments, where traffic 131 are forwarded natively over the trusted network segments and 132 encrypted over the untrusted segments. 134 The goal is to place as large portion as possible of the SDWAN path 135 over a provider VPN to achieve more optimal transport quality or 136 steering the SDWAN path traversing specific ingress/egress PEs to 137 reach optimal cost, quality, regulatory or other reasons. 139 2. Definition of terms 141 Cloud DC: Off-Premises Data Centers that usually host applications 142 and workload owned by different organizations or 143 tenants. 145 Controller: Used interchangeably with SDWAN controller to manage 146 SDWAN overlay path creation/deletion and monitoring the 147 path conditions between two or more sites. 149 DMVPN: Dynamic Multipoint Virtual Private Network. DMVPN is a 150 secure network that exchanges data between sites without 151 needing to pass traffic through an organization's 152 headquarter virtual private network (VPN) server or 153 router. 155 Heterogeneous Cloud: applications & workloads split among Cloud DCs 156 owned & managed by different Cloud Providers. 158 Hybrid Cloud: applications & workloads split between on-premises 159 Data centers and Cloud DCs. In this document Hybrid 160 Cloud also include heterogeneous cloud as well. 162 SDWAN: Software Defined Wide Area Network, "SDWAN" refers to 163 the solutions of pooling WAN bandwidth from multiple 164 underlay networks to get better WAN bandwidth 165 management, visibility & control. When the underlay 166 networks are private, traffic can traverse without 167 additional encryption; when the underlay networks are 168 public, such as the Internet, some traffic needs to be 169 encrypted when traversing through (depending on user 170 provided policies). 172 SP: Network Service Provider 174 SR: Segment Routing 176 SR Domain: A domain that supports Segment Routing 178 VPC: Virtual Private Cloud. A service offered by many Cloud 179 DC operators to allocate a logically isolated cloud 180 resources, including computing, networking and storage. 182 3. Key Use Cases 184 3.1. SDWAN Path over LTE network and SR Domain 186 MEF Cloud Service Architecture [MEF-Cloud] describes a use case of 187 network operators needing to use SDWAN over LTE for the last mile 188 access that they do not have physical infrastructure, as shown 189 below: 191 +********SDWAN Overlay Path *********+ 192 * * 193 * +------------+ * 194 * |SDWAN Ctrl | * 195 * +===+------------+====+ * 196 * // \\ * 197 * // <---Overlay Path ---> \\ * 198 * +-+--+ +--+ +--+ +--+-+ * 199 ***+ E1 |==|C1+--------+C4+==+ E2 |****+ 200 A --+ | | | | | | +----Z 201 A2--+--||+ ++-+ ++-+ ++---+\---Z2 202 LTE || | | // 203 || | SR +-+---+ +----+ 204 || | Network | C6 | |E3 | 205 || | | |----| | 206 +===+-+---+ +-+---+ +----+ 207 + C3 |-------+ 208 +---+-+ 209 | 210 +---+-+ 211 | E4 | 212 +-----+ 213 -- Directly attached 214 == || Public Internet or LTE path 215 *** Overlay path 217 Figure 1: SDWAN end points are attached to VPN via LTE 219 3.2. SDWAN as Last Mile for Cloud DCs Access 221 Digital Transformation is propelling more and more enterprises to 222 move their workloads/Apps to cloud DCs that are geographically close 223 to their end users to improve end-to-end latency & overall user 224 experience, or to comply with local data protection regulations. 225 Conversely, workloads/Apps in those Cloud DCs can be easily shutdown 226 when their end users' geographic base changes. 228 Because of the ephemeral property of the selected Cloud DCs, an 229 enterprise or its network service provider may not have the direct 230 links to the Cloud DCs that are optimal for hosting the enterprise's 231 specific workloads/Apps. Under those circumstances, SDWAN is a very 232 flexible choice to interconnect the enterprise on-premises data 233 centers & branch offices to its desired Cloud DCs. 235 However, SDWAN paths over public internet can have unpredictable 236 performance, especially over long distances and cross state/country 237 boundaries. Therefore, it is highly desirable to place as much as 238 possible the portion of SDWAN paths over service provider VPN (e.g. 239 enterprise's existing VPN) that have guaranteed SLA and to minimize 240 the distance/segments over public internet. 242 Under this scenario, one or both of the SDWAN end points may not 243 directly attached to the PEs of a SR Domain. 245 3.3. How & Why SR is useful for those use cases 247 Let us assume that the SDWAN Controller is capable of computing 248 optimal paths between two end-points (e.g. E1<->E2 in the Figure 2), 249 either by communicating with the SR Domain controller/management- 250 system, or by other methods which is out of the scope of this 251 document. 253 When a SR domain has multiple PEs with ports facing the external 254 networks (such as the public internet or LTE termination), SDWAN 255 paths can traverse the SR domain via different ingress/egress PEs 256 resulting in different E2E performance. 258 In the diagram below, E1 <-> E2 SDWAN (most likely IPsec encrypted 259 tunnel) path can traverse C1 <-> C4, C1<->C6, C3<->C6, or C3<->C4 260 within the VPN. There are many flows (by different Apps) between E1 261 <-> E2. Some flows may need to traverse C1<->C4, others may need to 262 traverse C3<->C6 or other segments within the VPN, which are 263 determined by the SDWAN controller based on the characteristics & 264 need of the Apps, such as cost, available bandwidth, latency, or 265 special functions only available at specific locations, etc. 267 Even with the same ingress/egress, some flows may need different 268 segments across the SR Domain. It is not practical, or even 269 possible, for PEs (e.g. C1, C2, C3 in this example) to determine 270 which Apps' flows should egress C4 or C6 where both C4&C6 can reach 271 E2. 273 Segment Routing can be used to steer packets (or path) to traverse 274 the explicit egress node (C4 or C6), or explicit segments through 275 the SR Domain based on the SLA requested by the SDWAN head-end 276 nodes. 278 +********SDWAN Overlay Path *********+ 279 * * 280 * +------------+ * 281 * |SDWAN Ctrl | * 282 * +===+------------+====+ * 283 * // \\ * 284 * // <-- Overlay path----> \\ * 285 * +-+--+ ++-+ ++-+ +--+-+ * 286 ***+ E1 |==|C1|--------|C4+==+ E2 |****+ 287 A --+ | | | | | | +----Z 288 A2--+--+-+ ++-+ ++-+ ++---+\---Z2 289 LTE || | | // 290 || | SR +-+---+ +----+ 291 || | Network | C6 | |E3 | 292 || | | |- --| | 293 || +-+---+ +-+---+ +----+ 294 +==+ C3 |-------+ 295 +---+-+ 296 | 297 +---+-+ 298 | E4 | 299 +-----+ 301 -- Directly attached 302 == || Public Internet or LTE path 303 ** Overlay path 305 Figure 2: SDWAN end points not directly attached to PEs of SR Domain 307 4. Mechanism for SDWAN path over one SR Domain and existing access 309 This section describes the mechanism to enforce a SDWAN path' head- 310 end selected route traversing through a list of specific nodes of 311 multiple network segments without requiring the nodes in each 312 network segment to have the intelligence (or maintaining states) of 313 selecting next hop or next domain. 315 There may be two approaches here: 316 1) Controller installs the entire SID stack at E1. 318 2) Controller delivers to E1 a "Key" that the SR ingress PE can use 319 to map to the SID stack for the packets arriving at the SR Ingress 320 PE. Section 4.2 & 4.3 will describe how the "Key" is carried by the 321 packets. 323 The Approach 1) requires less processing at the SR Ingress PE nodes, 324 but only works if the remote CPEs are in the same administrative 325 domain as the SR domain. SR domain usually is not willing to expose 326 its internal binding SIDs to devices in different administration 327 domains. This approach also requires more changes to SDWAN end nodes 328 and need more header bytes added to the packets when traversing rd through 3 party internet. Some SDWAN nodes might not be capable of 329 supporting encapsulating packets with the SID stack. 331 The Approach 2) above requires SR Ingress PE nodes to map the "Key" 332 to the SID Stack and prepend the SID stack to the packets (Same 333 processing for other traffic except the mapping is from the received 334 "Key" carried in the payload). 336 4.1. Controller Delivers SID Stack to SDWAN Head-end 338 This approach is straightforward. 340 E1 -------------------------- > SDWAN controller 341 request for a SDWAN path E1<->E2 with a specific SLA 343 E1 <-------------------------- SDWAN controller 344 Reply with the Ingress PE Node ID or address 345 & the Binding SID. 347 Here is the packet header for SDWAN Source Node to prepend to the 348 payload: 350 0 1 2 3 351 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 353 IPv4 Header: 354 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 355 |Version| IHL |Type of Service| Total Length | 356 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 357 | Identification |Flags| Fragment Offset | 358 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 359 | Time to Live | Prot.=17(UDP) | Header Checksum | 360 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 361 | SDWAN Source IPv4 Address | 362 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 363 | SR Ingress PE IPv4 Address | 364 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 366 UDP Header: 367 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 368 | Source Port = | Dest. Port = 4754/4755 | 369 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 370 | UDP Length | UDP Checksum | 371 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 373 GRE Header: 374 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 375 |C| |K|S| Reserved0 | Ver | Protocol Type | 376 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 377 | Checksum (optional) | Reserved1 (Optional) | 378 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 379 | Key (For SR Ingress to map to its SID) | 380 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 381 | Sequence Number (optional) | 382 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 384 To traverse SRv6 domain, SRv6 Header is appended after the GRE 385 header [SRv6-SRH]: 387 To traverse MPLS-SR domain, a stack of MPLS labels is appended after 388 GRE Header [MPLS-SR]. 390 4.2. Using GRE Key or VXLAN ID to Differentiate Flows 392 This section describes a method of SDWAN head-end node using GRE Key 393 or VXLAN Network ID (VNI) to indicate the desired property for 394 specific flows between SDWAN end-points (E1<->E2 in the figure 395 above): such as different desired routes through the SR Domain, 396 different egress PEs based on cost, performance or other factors. 397 It might be difficult or impossible for DiffServ bits carried by the 398 packets to describe those flow properties because there can be more 399 than what DiffServ bits can represent. 401 The SR Domain ingress can map the GRE key to different SID through 402 the SR Domain. 404 We assume that the SDWAN Controller can determine which ingress PE 405 can lead to the optimal path between E1<->E2. It is beyond the scope 406 of this document on how SDWAN controller computes the paths and how 407 & what SDWAN controller communicates with the SR Domain controller. 409 Here is the sequence of the flow: 411 E1 -------------------------- > SDWAN controller 412 request for a SDWAN path E1<->E2 with a specific SLA 414 E1 <-------------------------- SDWAN controller 415 Reply with the Ingress PE Node ID or address 416 & (the GRE Key or VXLAN ID). 418 Note: the GRE key (or VXLAN ID) from the SDWAN controller is for the 419 ingress PEs to correlate desired Path with the list of SIDs to 420 prepend the packet across the SR domain. 422 When SDWAN Controller get the E1<->E2 path request, it will 423 communicate with the VPN Controller to get the optimal Ingress PE 424 Node ID (or IP address) and the GRE key (or VXLAN ID) to encapsulate 425 the original packets between E1 <-> E2 (assuming IPsec Tunnel mode 426 is used). 428 Upon receiving the GRE encapsulated packets, the provider ingress 429 Edge C1/C3 uses the GRE key (or VXLAN ID) to map to the pre-defined 430 (by the network controller) Binding SIDs, prepend the Binding SIDs 431 to the packets, and forward its desired paths across the provider 432 VPN. 434 Depending on how the SDWAN path destination can be reached by the 435 egress PE, the egress PE has different processing procedure: 437 - If the destination of the SDWAN path is directly attached to 438 the egress VPN PE node, the egress VPN PE decapsulates SR 439 header and forward the packets to SDWAN path destination node, 440 such as the E2 in the figure above. 441 - If the destination of the SDWAN path is IP reachable via IPv4 442 network from the egress VPN PE node, the egress VPN PE node 443 decapsulates SR header and forward the packets to SDWAN path 444 destination node via its internet facing port to the SDWAN path 445 destination (i.e. the E2 node in the figure above). 446 - If the SDWAN path is traversing multiple domains owned by 447 different network operators, the egress PE processing is 448 described in the next session. 450 4.3. Using UDP Source Port Number to Differentiate Flows 452 [RFC8086] describes how to use GRE-in-UDP source port number as 453 entropy for better ECMP performance. When the remotely attached CPEs 454 is within very close proximity to the PEs, e.g. only one or two 455 hopes away like in LTE access, there is less issue if ECMP put all 456 flows with same traffic classifier into one path. Then, those UDP 457 numbers can also be used as a key to SR PE nodes to map to the 458 appropriate SID to the packets. 460 Same as RFC8086, UDP source port values used as a key for SR PEs to 461 map to appropriate SIDs SHOULD be chosen from the ephemeral port 462 range (49152-65535) [RFC8085]. 464 The GRE-in-UDP encapsulation format contains a UDP header [RFC768] 465 and a GRE header [RFC2890]. The format is shown as follow 466 (presented in bit order): 468 0 1 2 3 469 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 471 IPv4 Header: 472 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 473 |Version| IHL |Type of Service| Total Length | 474 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 475 | Identification |Flags| Fragment Offset | 476 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 477 | Time to Live | Prot.=17(UDP) | Header Checksum | 478 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 479 | SDWAN Source IPv4 Address | 480 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 481 | SR Ingress PE IPv4 Address | 482 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 484 UDP Header: 485 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 486 | Source Port = SIDs key Value | Dest. Port = 4754/4755 | 487 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 488 | UDP Length | UDP Checksum | 489 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 491 GRE Header: 492 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 493 |C| |K|S| Reserved0 | Ver | Protocol Type | 494 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 495 | Checksum (optional) | Reserved1 (Optional) | 496 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 497 | Key (optional) | 498 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 499 | Sequence Number (optional) | 500 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 502 Figure 3: UDP + GRE Headers in IPv4 504 Here is the GRE Header for IPv6 network, i.e. the SDWAN Source SDWAN 505 Destination, and SR PEs are all in IPv6 domain: 507 0 1 2 3 508 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 510 IPv6 Header: 511 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 512 |Version| Traffic Class | Flow Label | 513 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 514 | Payload Length | NxtHdr=17(UDP)| Hop Limit | 515 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 516 | | 517 + + 518 | | 519 + SDWAN Source IPv6 Address + 520 | | 521 + + 522 | | 523 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 524 | | 525 + + 526 | | 527 + SR Domain Ingress PE IPv6 Address + 528 | | 529 + + 530 | | 531 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 533 UDP Header: 534 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 535 | Source Port = SIDs key value | Dest. Port = 4754/4755 | 536 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 537 | UDP Length | UDP Checksum | 538 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 540 GRE Header: 541 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 542 |C| |K|S| Reserved0 | Ver | Protocol Type | 543 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 544 | Checksum (optional) | Reserved1 (Optional) | 545 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 546 | Key (optional) | 547 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 548 | Sequence Number (optional) | 549 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 551 Figure 4: GRE+UDP for IPv6 553 4.4. GRE Header Extension 555 A new protocol type can be added to the GRE header [RFC2890] to make 556 it easier for the SR PE to do the proper actions: 558 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 559 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 560 |C| Reserved0 | Ver | Protocol Type | 561 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 562 | Checksum (optional) | Reserved1 (Optional) | 563 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 565 The proposed GRE header will have the following format: 567 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 568 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 569 |C| |K|S| Reserved0 | Ver | Protocol Type | 570 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 571 | Checksum (optional) | Reserved1 (Optional) | 572 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 573 | Key (optional) | 574 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 575 | Sequence Number (Optional) | 576 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 578 New protocol type (value to be assigned by IANA): 579 UDP-Key: Using UDP source port value as a Key for SR Ingress PE to 580 map to the appropriate SIDs. 582 GRE-KEY: Using GRE Key value as a key for SR ingress PE to map to 583 the appropriate SIDs 585 5. SDWAN path over multiple SP managed domains 586 The following figure shows a SDWAN Path E1<->E2 over two SP domains 587 which are interconnected by public internet. 589 +********SDWAN Overlay Path *************+ 590 * * 591 * +------------+ * 592 * |SDWAN Ctrl | * 593 * +===+------------+====E2/E3/E4.. * 594 * // * 595 * // * 596 * +-+--+ ++-+ ++-+ +--+-+ * 597 ***+ E1 |==|C1+--------+C7+--+ E7 | * 598 A --+ | | | | | | + * 599 A2--+----+ ++-+ ++-+ ++---+ * 600 LTE || | SP1 | // * 601 +---+ SR +-+---+ +----+ * 602 +--------+ |C3 | Network | C4 | |E3 | * 603 | E4 | | | | |- --| | * 604 | | +---+-----+---+---+-+ +----+ * 605 +--------+=======+ C2 | || * 606 +---+-+ // * 607 // // * 608 +-+-+--------+--+-+ * 609 |D1 | |D4 | * 610 | | | | * 611 ++--+ ++---+ * 612 | SP2 | * 613 | SR +--+--+ +-+--+ 614 +--------+ | Network | D2 | |E2 +----Z 615 | E6 | | | +====+ +---Z2 616 | | +--+-----+ +-+---+LTE +----+ 617 +-+--+-+-+-------+ D3 |-----+ 618 +---+-+ 620 -- Directly attached 621 == || Public Internet or LTE path 622 ** Overlay path 623 Figure 5: SDWAN path over two different SP domains 625 Let's assume that the SP-1 domain's egress node for the SDWAN path 626 E1<->E2 is C2, which can reach D1 or D4 of SP-2 via public IP 627 network (say IPv4 network). 629 Let's also assume that the optimal route for some flows over SDWAN 630 path E1<->E2 are C1->C2->D1 and other flows are over C1->C2->D4 (out 631 of the scope of this document on how the path is calculated). 633 If SP-1 is SR enabled, the mechanism described in Section 4 is 634 applicable to the SDWAN path source node E1 and the SP-1's ingress 635 PE (e.g. C1 or C3 in the figure). 636 However, the processing at egress node might be different depending 637 on how the SP-1's egress edges are connected to the SP-2's ingress 638 edge nodes. 640 5.1. When Both SP domains support SR 642 There may be three approaches here: 644 1) Controller installs the entire SID stack at E1, and the SID list 645 contains SID entries belong to both domains. 647 2) Controller delivers to E1 the SID stack that only for the first 648 domain, but delivers to C6 (egress node of first domain) the binding 649 SID of the second domain. 651 3) Controller delivers a "Key" to E1, which can be encoded as GRE 652 KEY or represented by the Source UDP port of the GRE encapsulation, 653 for Ingress PE of the first SR Domain to map to its own SID stack as 654 described in Section 4. The first SR Domain will reserve the "Key" 655 through its domain and pass the "Key" to the second SR domain. The 656 second SR Domain Ingress node will use the same method to map the 657 "Key" to its SID stack. 659 5.2. When SP-2 does not support SR 661 Under this circumstance (which can be caused by SP-2 not supporting 662 SR or not willing to share Binding SIDs to SP-1), if the packets 663 arriving at SP-1 egress node C6 do not have any metadata indicating 664 the types of encrypted payload, C6 does not really have much choice 665 other than simply forwarding the packets to E2 via public IP 666 network. This way, the packets may or may not traverse through the 667 SP-2 domain. If the distance between C6 and E2 is far, the quality 668 of service can be unpredictable. 670 5.3. When SP-1 and SP-2 don't want to share network information 672 If SP-1's ingress node C1 can include the GRE KEY it receives from 673 E1 in the data packets' SR header, the SP-1's egress node can map 674 the Key to the SP-2's Ingress node and encapsulate the data packet 675 in a new GRE header destined towards the SP-2's Ingress node. Then 676 the SP-2's Ingress node can follow the procedure described in the 677 Section 4 to forward the data packets across its domain. 679 If the first SR Domain does not support adding metadata to carry the 680 "key" through its domain, the controller can deliver the "key" to 681 SP-1's egress node the same time as it delivers the key to E1, 682 knowing the SDWAN path will need to traverse two domains with the 683 second one does support SR but the two SPs don't want to exchange 684 network information. 686 5.4. TLV to pass Metadata through SRv6 Domain 688 If SP-1 is SRv6 based, the ingress node C1 can append a TLV to the 689 end of the SR Header [SRv6-SRH] to carry the KEY it receives from 690 E1. 692 The SP-1 egress node C6 can get the mapping between the KEYs and the 693 Node-IDs (or Addresses) of the next domain's ingress edge node (i.e. 694 D1 or D4 in the figure 3 above) from its network controller ahead of 695 time. 697 0 1 2 3 698 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 699 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 700 | Type | Length | RESERVED | 701 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 702 | Key ID (4 octets) from the GRE tunnel remote ingress node | 703 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 704 | Optional // 705 | Node ID or address for the ingress node Next domain // 706 | Variable length (0~32 octets) // 707 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 709 TYPE: (to be assigned by IANA) is to indicate the TLV is for 710 carrying the flow identifier of the packet encoded by the SDWAN 711 source node. 713 Upon receiving the packet, the egress node (C6) can 715 - find the Node-ID (or the address) for the next domain's ingress 716 node, 717 - construct a GRE header with the Key received from the TLV above 718 and the destination address from the mapping given by the 719 controller, 720 - encapsulate the GRE header to the data packet (which has 721 decapsulated SR header), 722 - and forward the packet to the public internet. 724 6. Security Considerations 726 Remotely attached CPEs might brought the following security risks: 728 1) Potential DDoS attack to the PEs with ports facing internet. 729 I.e. the PE resource being attacked by unwanted traffic. 730 2) Potential risk of provider VPN network bandwidth being stolen 731 by the entities who spoofed the addresses of SDWAN end nodes. 733 To mitigate security risk of 1) above, it is absolutely necessary 734 for PEs which accept remotely attached CPEs or simply have ports 735 facing internet to enable Anti-DDoS feature to prevent major DDoS 736 attack to those PEs. 738 To mitigate the security risk of 2) above, RFC7510 defines the use 739 of DTLS to authenticate and encrypt the RFC7510 encapsulation. 741 However, for the scenario of SDWAN source node being remotely 742 attached to PEs, using the method recommended by RFC7510 means the 743 source node has to perform DTLS on top of the IPSec encryption 744 between SDWAN end points E1<->E2. This can be too processing heavy 745 for the SDWAN end nodes. In addition, if there are many SDWAN flows 746 to traverse through the ingress PE (e.g. C1, C2, C4 in the figure 1 747 above), heavy processing is required on the ingress PEs. 749 Since the payload between E2<->E2 is already encrypted, the 750 confidentiality of the payload is already ensured. The network 751 operators need to balance between how much they can tolerant some 752 percentage of bandwidth being stolen and how much extra cost they 753 are willing to pay for completely prevent any unpaid traffic 754 traversing through its VPN networks. For operators who opt for lower 755 cost ingress PEs and CPEs, but can tolerant some percentage of 756 bandwidth being used by unpaid subscribers, a simple approach can be 757 considered: 759 - Embed a key in the packets, which can be changed periodically, 760 like the digital signature used by a certificate authority or 761 certification authority (CA). 762 - The key can be encoded in the GRE Key field between SDWAN end 763 node and Ingress PE. Since GRE has 24 bits, some fixed bits 764 can be used to represent the signature of paid subscribers. 766 7. IANA Considerations 768 This document requires new protocol type: 770 Protocol type to be added to GRE header: SR_Route 772 8. References 774 8.1. Normative References 775 [RFC2890] G. Dommety "Key and Sequence Number Extensions to GRE". 776 Sep. 2000. 778 8.2. Informative References 780 [RFC2735] B. Fox, et al "NHRP Support for Virtual Private 781 networks". Dec. 1999. 783 [RFC8192] S. Hares, et al "Interface to Network Security Functions 784 (I2NSF) Problem Statement and Use Cases", July 2017 786 [ITU-T-X1036] ITU-T Recommendation X.1036, "Framework for creation, 787 storage, distribution and enforcement of policies for 788 network security", Nov 2007. 790 [RFC6071] S. Frankel and S. Krishnan, "IP Security (IPsec) and 791 Internet Key Exchange (IKE) Document Roadmap", Feb 2011. 793 [RFC4364] E. Rosen and Y. Rekhter, "BGP/MPLS IP Virtual Private 794 Networks (VPNs)", Feb 2006 796 [RFC4664] L. Andersson and E. Rosen, "Framework for Layer 2 Virtual 797 Private Networks (L2VPNs)", Sept 2006. 799 [SR-SDWAN] D. Dukes, et al, "SR for SDWAN: VPN with Underlay SLA", 800 draft-dukes-sr-for-sdwan-00, in progress, Oct 2017 802 [SRv6-SRH] S. Previdi, et al, "IPv6 Segment Routing Header (SRH)", 803 draft-ietf-6man-segment-routing-header-13, in progress, 804 April 2018. 806 [MPLS-SR] A. Bashandy, et al, "Segment Routing with MPLS data 807 plane", draft-ietf-spring-segment-routing-mpls-13, in 808 progress, April 2018. 810 [RFC7510] X. Xu, et al, "Encapsulating MPLS in UDP", April 2015. 812 [RFC8086] L. Yong, et al, "GRE-in-UDP Encapsulation", March 2017. 814 [BGP-SDWAN-Usage] L. Dunbar, et al, "BGP Usage for SDWAN Overlay 815 Networks, draft-dunbar-bess-bgp-sdwan-usage-01, in 816 progress, July 2019. 818 [SDWAN-Net2Cloud] L. Dunbar, et al, "Dynamic Networks to Hybrid 819 Cloud DCs Problem Statement", draft-ietf-rtgwg-net2cloud- 820 problem-statement-04, in progress, July 2019. 822 [MEF-Cloud] "Cloud Services Architecture Technical Specification", 823 Work in progress, April 2018 825 9. Acknowledgments 827 Many thanks to Dean Cheng and Jim Guichard for the discussion and 828 contributions. 830 Authors' Addresses 832 Linda Dunbar 833 Futurewei 834 Email: Linda.Dunbar@futurewei.com 836 Mehmet Toy 837 Verizon 838 One Verizon Way 839 Basking Ridge, NJ 07920 840 Email: mehmet.toy@verizon.com