idnits 2.17.1 draft-bottorff-ipsecme-mtdcuc-ipsec-lb-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (July 5, 2021) is 1019 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Unused Reference: 'RFC768' is defined on line 342, but no explicit reference was found in the text == Unused Reference: 'RFC1122' is defined on line 346, but no explicit reference was found in the text == Unused Reference: 'RFC1191' is defined on line 351, but no explicit reference was found in the text == Unused Reference: 'RFC2003' is defined on line 355, but no explicit reference was found in the text == Unused Reference: 'RFC3948' is defined on line 359, but no explicit reference was found in the text == Unused Reference: 'RFC4821' is defined on line 371, but no explicit reference was found in the text == Unused Reference: 'RFC7296' is defined on line 375, but no explicit reference was found in the text == Unused Reference: 'RFC6438' is defined on line 414, but no explicit reference was found in the text == Unused Reference: 'RFC8200' is defined on line 419, but no explicit reference was found in the text == Unused Reference: 'RFC8201' is defined on line 423, but no explicit reference was found in the text ** Obsolete normative reference: RFC 7321 (Obsoleted by RFC 8221) Summary: 1 error (**), 0 flaws (~~), 11 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 Network Working Group P. Bottorff 2 Internet-Draft Hewlett Packard Enterprise Co. 3 Intended status: Informational July 5, 2021 4 Expires: December 29, 2021 6 Multi-tenant Data Center Use Case for IPsec Load Balancing 7 draft-bottorff-ipsecme-mtdcuc-ipsec-lb-00 9 Abstract 11 IPsec is of increasing importance within data centers to secure 12 tunnels used to carry multi-tenant traffic encapsulated using the 13 Network Virtualization over L3 (NVO3) protocols. Encrypting NVO3 14 tunnels provides defense against bad actors within the physical 15 underlay network from monitoring or injecting overlay traffic from 16 outside the NVO3 infrastructure. When securing data center tunnels 17 using IPsec it becomes crucial to retain entropy within the outer 18 IPsec packet headers to facilitate load balancing over the highly 19 meshed networks used in these environments. While entropy is 20 necessary to support load distribution algorithms it is also 21 important that the entropy codes used retain integrity of flows to 22 prevent performance deterioration resulting from packet re-ordering. 23 Here, we describe a use case for load balancing IPsec traffic within 24 multi-tenant data centers. 26 Status of this Memo 28 This Internet-Draft is submitted in full conformance with the 29 provisions of BCP 78 and BCP 79. Internet-Drafts are working 30 documents of the Internet Engineering Task Force (IETF). Note that 31 other groups may also distribute working documents as Internet- 32 Drafts. The list of current Internet-Drafts is at 33 https://datatracker.ietf.org/drafts/current/. 35 Internet-Drafts are draft documents valid for a maximum of six months 36 and may be updated, replaced, or obsoleted by other documents at any 37 time. It is inappropriate to use Internet-Drafts as reference 38 material or to cite them other than as "work in progress." 40 This Internet-Draft will expire on December 22, 2021. 42 Copyright Notice 44 Copyright (c) 2021 IETF Trust and the persons identified as the 45 document authors. All rights reserved. 47 This document is subject to BCP 78 and the IETF Trust's Legal 48 Provisions Relating to IETF Documents 49 (http://trustee.ietf.org/license-info) in effect on the date of 50 publication of this document. Please review these documents 51 carefully, as they describe your rights and restrictions with respect 52 to this document. Code Components extracted from this document must 53 include Simplified BSD License text as described in Section 4.e of 54 the Trust Legal Provisions and are provided without warranty as 55 described in the Simplified BSD License. 57 Table of Contents 59 1. Introduction...................................................2 60 2. Generic Data Center Network Architecture.......................4 61 3. Network Virtualization Over L3 (NVO3) Architecture.............5 62 4. Load Balancing Secure NVO3 tunnels.............................6 63 5. Security Considerations........................................7 64 6. IANA Considerations............................................7 65 7. Conclusions....................................................7 66 8. Normative References...........................................8 67 9. Informative References.........................................9 68 10. Acknowledgments..............................................10 70 1. Introduction 72 Load balancing is essential within data centers to achieve high 73 utilization of the meshed networks common in these environments. 74 Typically, the load on the network is scattered over the mesh network 75 using a hash of the outer packet headers (i.e. a 5-tuple hash). When 76 a tunneling protocol is used over a data center mesh network packets 77 are addressed from tunnel end point to tunnel end point (e.g. 78 servers) which does not provide the entropy required to spread the 79 traffic over the data center mesh network. To provide load balancing 80 support, tunnel protocols used in the data center need to provide 81 entropy codes within their outer packet headers to support load 82 balancing. 84 While spreading traffic over a data center mesh network mis-ordering 85 of packet flows needs to be avoided to prevent slowing operations 86 caused by packet order recovery. To retain flow alignment within 87 tunneling protocols entropy codes need to be based on the flow of the 88 encapsulated packets. 90 Multi-tenant data center using Network Virtualization over L3 (NVO3) 91 [RFC7365, RFC8014, RFC8394] create virtual networks interconnecting 92 virtual servers within an overlay on top of the physical underlay 93 network. In these NVO3 networks virtual network packets are 94 multiplexed into tunnels which extend over the physical underlay 95 network. The encapsulations used to in the NVO3 tunnels (i.e. VxLAN, 96 GENEVE, GUE) have an outer UDP header followed by an outer NVO3 97 header. The NVO3 header carries a key called the Virtual Network 98 Identifier (VNI) which identifies the virtual network within the 99 virtual overlay network. Each of the virtual overlay networks is a 100 separate subnetwork which can have its own IP and L2 virtual address 101 space. 103 Since a single NVO3 tunnel is used between communicating servers, any 104 server to server connection has the same IP source, destination, and 105 UDP destination port. To retain entropy for load balancing the NVO3 106 protocols use the UDP source port to hold a hash of the encapsulated 107 inner packet. This outer UDP source port provides the entropy 108 necessary for spreading traffic over the mesh based on the 109 encapsulated flow. Other fields such as the IPv6 flow label could 110 also be used, however these are not universally supported in data 111 center switching infrastructures, while the use of UDP source port is 112 broadly available in data center switches, routers, and middle boxes. 114 The NVO3 protocols isolate tenant virtual networks based on the VNI 115 identifiers carried in the tunnel headers. Since any bad actor 116 connected to the data center underlay network could spoof an 117 encapsulation transporting a virtual network and any device in the 118 middle of the communication can monitor the tenant networks, NVO3 119 networks must operate in a secure perimeter. With the rise of more 120 aggressive bad actors it is desirable to provide secure connections 121 for NOV3 tunnels to eliminate the threat of a server or switch within 122 the data center underlay monitoring or interfering with the operation 123 of virtual networks throughout the data center. 125 Encryption of [RFC4301, RFC4303, RFC7321] the NVO3 tunnels can 126 provide protection against devices outside the virtual overlay from 127 monitoring, spoofing or interfering with the virtual networks. This 128 can be done using IPsec to encrypt the tunnels carrying virtual 129 networks between servers. Since the tunnels can be encrypted using 130 smart network interfaces this method can be very efficient, retaining 131 the high performance required within data centers. 133 If we apply IPsec directly to the NVO3 tunnels the IP source and 134 destination as well as the protocol type and SPI will be the same for 135 each server to server communication, therefore we will lose the 136 entropy needed to support the data center mesh network. Internet 137 Draft "Encapsulating IPsec ESP in UDP for Load-balancing" [IPSEC- 138 LB], which proposes using the source port of IPsec transport mode ESP 139 in UDP encapsulated packets for entropy, provides the solution needed 140 to support the use of IPsec to secure the tunnels used for NVO3 141 traffic in data centers. By using transport mode ESP in UDP 142 encapsulation of NVO3 tunnels entropy can be provided by using the 143 UDP source port just as was done originally for the NVO3 UDP 144 encapsulations. 146 2. Generic Data Center Network Architecture 148 Figure 1 depicts a typical Clos mesh network [RFC7938] used in a data 149 center. Each server is typically redundantly connected to a set of 150 switches located at the Top of the Rack (ToR) switch (also called a 151 leaf switch). Each of these switches is, in turn, connected to a set 152 of switches for the row of racks commonly called the End of Row (EoR) 153 switch (also called a spine switch). Typically these redundant links 154 are managed either by L2 link aggregation protocols [IEEE-AX, IEEE-Q] 155 or by L3 equal cost multi-path protocols [RFC7938]. The number of 156 links interconnecting these layers varies depending on the bandwidth 157 and resiliency requirements. The figure shows a two level hierarchy, 158 however it is common for data centers to have a three level hierarchy 159 where a similar Clos mesh is used to interconnect rows of servers. 161 +--------+ +--------+ 162 | EoR | | EoR | 163 | switch | | switch | 164 +--------+ +--------+ 165 / / | \ / | \ \ 166 / / +---\----/-----\-----------------+ 167 / / \/ | \ \ | 168 / / / \ | \ \ | 169 / +-------/--------/------\-+ \ \ | 170 / | / / \ \ \ | 171 +--------+ +--------+ +--------+ +--------+ 172 | ToR | | ToR | | ToR | | ToR | 173 | switch | | switch | | switch | | switch | 174 +--------+ +--------+ +--------+ +--------+ 175 / \ / \ / \ / \ 176 / \ / \ / \ / \ 177 / \/ \ / \/ \ 178 / / \ \ / / \ \ 179 / / \ \ / / \ \ 180 / / \ \ / / \ \ 181 '-----------' '-----------' '-----------' '-----------' 182 : Server : : Server : : Server : : Server : 183 : : : : : : : : 184 '-----------' '-----------' '-----------' '-----------' 186 Figure 1: Typical Data Center Mesh Network 187 To distribute packets over the network paths typically a hash 188 function is used to reduce the fields within the outer packet headers 189 into a group of flows. The group is then allocated to a network link 190 or path. In the simplest and most common implementation the 191 distributions are done based on a hash. In more sophisticated 192 implementation additional load data and timing information may be 193 used to move flow groups based on load estimates. 195 3. Network Virtualization Over L3 (NVO3) Architecture 197 In an NVO3 multi-tenant data center the physical interconnect 198 depicted in figure 1 is used as the underlay physical IP network 199 where IP addresses are assigned to servers. 201 +--------+ +--------+ 202 | Tenant +--+ +----| Tenant | 203 | System | | (') | System | 204 +--------+ | ................. ( ) +--------+ 205 | +---+ +---+ (_) 206 +--|NVE|---+ +---|NVE|-----+ 207 +---+ | | +---+ 208 / . +-----+ . 209 / . +--| NVA |--+ . 210 / . | +-----+ \ . 211 | . | \ . 212 | . | Overlay +--+--++--------+ 213 +--------+ | . | Network | NVE || Tenant | 214 | Tenant +--+ . | | || System | 215 | System | . \ +---+ +--+--++--------+ 216 +--------+ .....|NVE|......... 217 +---+ 218 | 219 | 220 ===================== 221 | | 222 +--------+ +--------+ 223 | Tenant | | Tenant | 224 | System | | System | 225 +--------+ +--------+ 227 Figure 2: NVO3 Architecture Reference Diagram 229 The NVO3 protocols are used on top of the physical underlay network 230 to create virtual networks which overlay the physical underlay 231 network. The virtual networks are carried over the underlay 232 encapsulated in an NVO3 encapsulation protocol such as GENEVE 233 [RFC8926], VxLAN [RFC7348], or GUE. These encapsulations indicate the 234 virtual network by encoding a Virtual Network Identifier (VNI) within 235 the encapsulation header. 237 Figure 2 is a copy of the NVO3 architecture [RFC8014] reference 238 diagram. In figure 2 the Network Virtual Edge (NVE) entities provide 239 the tunnel terminations for the encapsulation protocols. The NVEs can 240 be located within the server's hypervisor, within smart NICs on the 241 servers, or within switches of the physical network [RFC8394]. 243 The tenant systems (TS) are virtualized servers. These may be virtual 244 machines, containers, or physical servers that are connected over the 245 virtual networks and multiplexed into NOV3 tunnel by the NVEs. 247 The network virtualization authority (NVA) manages the creation and 248 configuration of virtual networks by configuring the NVEs. 250 4. Load Balancing Secure NVO3 tunnels 252 The NVO3 protocols isolate tenant virtual networks based on the 253 Virtual Network Identifiers carried in the tunnel header. Since any 254 bad actor inside the data center could spoof an encapsulation 255 transporting a virtual network and any device in the middle of the 256 communication can monitor the tenant networks, these networks are 257 only secure when operating within a secure perimeter. With the rise 258 of more aggressive bad actors it is desirable to provide secure 259 connections for NVO3 tunnels to eliminate the threat of a server or 260 switch within the data center underlay monitoring or interfering with 261 the operation of overlay virtual networks. 263 Encryption of [RFC4301, RFC4303, RFC7321] the NVO3 tunnels provides 264 protection against devices outside the virtual overlay from 265 monitoring, spoofing or interfering with the virtual networks. This 266 can be done using IPsec to encrypt the tunnels carrying virtual 267 networks between servers. Since the tunnels can be encrypted using 268 smart network interfaces this method can be very efficient, retaining 269 the high performance required within data centers. However since the 270 resulting tunnel headers don't provide enough entropy to support load 271 balancing over the data center mesh networks the resulting network 272 bandwidth could be greatly reduced. 274 To solve the load balancing problem produced by securing the NVO3 275 tunnels using IPsec the method proposed in Internet Draft 276 "Encapsulating IPsec ESP in UDP for Load-balancing" [IPSEC-LB] can be 277 used. Applying I-D.draft-xu-ipsecme-esp-in-udp-lb the NVO3 tunnels 278 are encapsulated as IPsec transparent mode ESP in UDP packets. Given 279 the UDP header on the outside of the IPsec tunnel the source port can 280 be used for entropy. By copying the source port from the original 281 NVO3 encapsulation into the IPsec ESP in UDP header it is possible to 282 retain the flow identity of the encrypted virtual network. Since the 283 NVO3 encapsulation source port contains an entropy code based on the 284 encapsulated overlay packet the resulting packet will provide the 285 entropy necessary to support load balancing in the data center mesh 286 network. 288 5. Security Considerations 290 Here we use IPsec to secure NVO3 tunnels between data center NVEs to 291 prevent attacks from servers or switches located within the data 292 center physical underlay, however outside of the overlay networks or 293 the NVE tunnel terminations and NVA managers. To fully secure a 294 multi-tenant network additional security methods need to be used to 295 prevent attackers from infiltrating the overlay infrastructure 296 including the Tenant Systems, NVEs and NVA. 298 6. IANA Considerations 300 The Internet Draft "Encapsulating IPsec ESP in UDP for Load- 301 balancing" [IPSEC-LB] proposes getting a new UDP destination port 302 assignment for use with load balanced IPsec. The use of a new port 303 would prevent existing implementations of IKE from operating with a 304 load balanced transparent mode ESP in UDP stream. It does not appear 305 this is necessary. Instead, the existing ESP in UDP port 4500 could 306 be used, provided both ends of the connection are configured to 307 exchanging ESP in UDP with an entropy code in the UDP source port. If 308 the existing ESP in UDP port 4500 is used, then there are no IANA 309 considerations since no new code points are necessary. 311 7. Conclusions 313 IPsec may be used to secure the underlay of a NVO3 multi-tenant data 314 center by encrypting the NVO3 tunnels. To make IPsec a viable 315 solution the IPsec tunnels need to provide load balancing. 317 By applying the proposal in Internet Draft "Encapsulating IPsec ESP 318 in UDP for Load-balancing" [IPSEC-LB], entropy can be added to the 319 IPsec packet header using the UDP source port of the ESP in UDP IPsec 320 packets. In particular, the source port of the original NVO3 tunnel 321 header can be copied to the new IPsec ESP in UDP source port 322 providing the necessary entropy while retaining the flow identity of 323 the encapsulated overlay packet. 325 8. Normative References 327 [IPSEC-LB] Xu, X., Hegde, S., Pismenny, B., Zhang, D., and Xia, L., 328 "Encapsulating IPsec ESP in UDP for Load-balancing", 329 December 2020, . 332 [IEEE-AX] "IEEE Standard for Local and Metropolitan Area Networks-- 333 Link Aggregation," in IEEE Std 802.1AX-2020 (Revision of 334 IEEE Std 802.1AX-2014), vol., no., pp.1-333, 29 May 2020, 335 doi: 10.1109/IEEESTD.2020.9105034. 337 [IEEE-Q] "IEEE Standard for Local and Metropolitan Area Network-- 338 Bridges and Bridged Networks," in IEEE Std 802.1Q-2018 339 (Revision of IEEE Std 802.1Q-2014), vol., no., pp.1-1993, 6 340 July 2018, doi: 10.1109/IEEESTD.2018.8403927. 342 [RFC768] Postel, J., "User Datagram Protocol", STD 6, RFC 768, DOI 343 10.17487/RFC0768, August 1980, . 346 [RFC1122] Braden, R., Ed., "Requirements for Internet Hosts - 347 Communication Layers", STD 3, RFC 112, DOI 348 10.17487/RFC1122, October 1989, . 351 [RFC1191] Mogul, J. and S. Deering, "Path MTU discovery", RFC 1191, 352 DOI 10.17487/RFC1191, November 1990, . 355 [RFC2003] Perkins, C., "IP Encapsulation within IP", RFC 2003, DOI 356 10.17487/RFC2003, October 1996, . 359 [RFC3948] Huttunen, A., Swander, B., Volpe, V., DiBurro, L., and M. 360 Stenberg, "UDP Encapsulation of IPsec ESP Packets", RFC 361 3948, DOI 10.17487/RFC3948, January 2005, . 364 [RFC4301] Kent,S., Seo, K., "Security Architecture for the Internet 365 Protocol", RFC 4301, December 2005, 366 368 [RFC4303] Kent, S., "IP Encapsulating Security Payload", RFC 4303, 369 December 2005, 371 [RFC4821] Mathis, M. and J. Heffner, "Packetization Layer Path MTU 372 Discovery", RFC 4821, DOI 10.17487/RFC4821, March 2007, 373 . 375 [RFC7296] Kaufman, C., Hoffman, P., Nir, Y., Eronen, P., and T. 376 Kivinen, "Internet Key Exchange Protocol Version 2 377 (IKEv2)", STD 79, RFC 7296, DOI 10.17487/RFC7296, October 378 2014, . 380 [RFC7321] McGrew, D., Hoffman, P., "Cryptographic Algorithm 381 Implementation Requirements and Usage Guidance for 382 Encapsulating Security Payload (ESP) and Authentication 383 Header (AH)", RFC 7321, August 2014, 384 https://datatracker.ietf.org/doc/rfc7321/ 386 [RFC7348] Mahalingam, M., et. al., "Virtual eXtensible Local Area 387 Network (VXLAN): A Framework for Overlaying Virtual Layer 2 388 Networks over Layer 3 Networks", RFC 7348, August 2014, 389 391 [RFC7365] Lasserre, M., et al, "Framework for Data Center (DC) 392 Network Virtualization", October 2014, 393 395 [RFC7938] Lapukhov, P., Premji, A., Mitchell, J., Ed., "Use of BGP 396 for Routing in Large-Scale Data Centers", August 2016, 397 . 399 [RFC8014] Black, D., et al, "An Architecture for Data-Center Network 400 Virtualization over Layer 3 (NVO3)", December 2016, 401 403 [RFC8394] Li, Y., Eastlake, D., Kreeger, L. Narten, T., Black, D., 404 "Split Network Virtualization Edge (Split-NVE) Control- 405 Plane Requirements", RFC 8394, May 2018, 406 408 [RFC8926] Gross, J., Ganga, I., Sridhar, T., "Geneve: Generic Network 409 Virtualization Encapsulation", November 2020, 410 412 9. Informative References 414 [RFC6438] Carpenter, B. and S. Amante, "Using the IPv6 Flow Label for 415 Equal Cost Multipath Routing and Link Aggregation in 416 Tunnels", RFC 6438, DOI 10.17487/RFC6438, November 2011, 417 . 419 [RFC8200] Deering, S., Hiden, R., "Internet Protocol, Version 6 420 (IPv6) Specification", STD 76, RFC 8200, July 2017, 421 423 [RFC8201] McCann, J., Deering, S., Mogul, J., Hiden, R., Ed., "Path 424 MTU Discovery for IP version 6", STD 87, RFC 8201, July 425 2017, 427 10. Acknowledgments 429 This document was prepared using 2-Word-v2.0.template.dot. 431 Authors' Addresses 433 Paul Bottorff 434 Aruba a Hewlett Packard Enterprise Co 435 8000 Foothill Blvd. 436 Roseville, CA 95747 438 Email: paul.bottorff@hpe.com