idnits 2.17.1
draft-bottorff-ipsecme-mtdcuc-ipsec-lb-00.txt:
Checking boilerplate required by RFC 5378 and the IETF Trust (see
https://trustee.ietf.org/license-info):
----------------------------------------------------------------------------
No issues found here.
Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt:
----------------------------------------------------------------------------
No issues found here.
Checking nits according to https://www.ietf.org/id-info/checklist :
----------------------------------------------------------------------------
No issues found here.
Miscellaneous warnings:
----------------------------------------------------------------------------
== The copyright year in the IETF Trust and authors Copyright Line does not
match the current year
-- The document date (July 5, 2021) is 1019 days in the past. Is this
intentional?
Checking references for intended status: Informational
----------------------------------------------------------------------------
== Unused Reference: 'RFC768' is defined on line 342, but no explicit
reference was found in the text
== Unused Reference: 'RFC1122' is defined on line 346, but no explicit
reference was found in the text
== Unused Reference: 'RFC1191' is defined on line 351, but no explicit
reference was found in the text
== Unused Reference: 'RFC2003' is defined on line 355, but no explicit
reference was found in the text
== Unused Reference: 'RFC3948' is defined on line 359, but no explicit
reference was found in the text
== Unused Reference: 'RFC4821' is defined on line 371, but no explicit
reference was found in the text
== Unused Reference: 'RFC7296' is defined on line 375, but no explicit
reference was found in the text
== Unused Reference: 'RFC6438' is defined on line 414, but no explicit
reference was found in the text
== Unused Reference: 'RFC8200' is defined on line 419, but no explicit
reference was found in the text
== Unused Reference: 'RFC8201' is defined on line 423, but no explicit
reference was found in the text
** Obsolete normative reference: RFC 7321 (Obsoleted by RFC 8221)
Summary: 1 error (**), 0 flaws (~~), 11 warnings (==), 1 comment (--).
Run idnits with the --verbose option for more detailed information about
the items above.
--------------------------------------------------------------------------------
1 Network Working Group P. Bottorff
2 Internet-Draft Hewlett Packard Enterprise Co.
3 Intended status: Informational July 5, 2021
4 Expires: December 29, 2021
6 Multi-tenant Data Center Use Case for IPsec Load Balancing
7 draft-bottorff-ipsecme-mtdcuc-ipsec-lb-00
9 Abstract
11 IPsec is of increasing importance within data centers to secure
12 tunnels used to carry multi-tenant traffic encapsulated using the
13 Network Virtualization over L3 (NVO3) protocols. Encrypting NVO3
14 tunnels provides defense against bad actors within the physical
15 underlay network from monitoring or injecting overlay traffic from
16 outside the NVO3 infrastructure. When securing data center tunnels
17 using IPsec it becomes crucial to retain entropy within the outer
18 IPsec packet headers to facilitate load balancing over the highly
19 meshed networks used in these environments. While entropy is
20 necessary to support load distribution algorithms it is also
21 important that the entropy codes used retain integrity of flows to
22 prevent performance deterioration resulting from packet re-ordering.
23 Here, we describe a use case for load balancing IPsec traffic within
24 multi-tenant data centers.
26 Status of this Memo
28 This Internet-Draft is submitted in full conformance with the
29 provisions of BCP 78 and BCP 79. Internet-Drafts are working
30 documents of the Internet Engineering Task Force (IETF). Note that
31 other groups may also distribute working documents as Internet-
32 Drafts. The list of current Internet-Drafts is at
33 https://datatracker.ietf.org/drafts/current/.
35 Internet-Drafts are draft documents valid for a maximum of six months
36 and may be updated, replaced, or obsoleted by other documents at any
37 time. It is inappropriate to use Internet-Drafts as reference
38 material or to cite them other than as "work in progress."
40 This Internet-Draft will expire on December 22, 2021.
42 Copyright Notice
44 Copyright (c) 2021 IETF Trust and the persons identified as the
45 document authors. All rights reserved.
47 This document is subject to BCP 78 and the IETF Trust's Legal
48 Provisions Relating to IETF Documents
49 (http://trustee.ietf.org/license-info) in effect on the date of
50 publication of this document. Please review these documents
51 carefully, as they describe your rights and restrictions with respect
52 to this document. Code Components extracted from this document must
53 include Simplified BSD License text as described in Section 4.e of
54 the Trust Legal Provisions and are provided without warranty as
55 described in the Simplified BSD License.
57 Table of Contents
59 1. Introduction...................................................2
60 2. Generic Data Center Network Architecture.......................4
61 3. Network Virtualization Over L3 (NVO3) Architecture.............5
62 4. Load Balancing Secure NVO3 tunnels.............................6
63 5. Security Considerations........................................7
64 6. IANA Considerations............................................7
65 7. Conclusions....................................................7
66 8. Normative References...........................................8
67 9. Informative References.........................................9
68 10. Acknowledgments..............................................10
70 1. Introduction
72 Load balancing is essential within data centers to achieve high
73 utilization of the meshed networks common in these environments.
74 Typically, the load on the network is scattered over the mesh network
75 using a hash of the outer packet headers (i.e. a 5-tuple hash). When
76 a tunneling protocol is used over a data center mesh network packets
77 are addressed from tunnel end point to tunnel end point (e.g.
78 servers) which does not provide the entropy required to spread the
79 traffic over the data center mesh network. To provide load balancing
80 support, tunnel protocols used in the data center need to provide
81 entropy codes within their outer packet headers to support load
82 balancing.
84 While spreading traffic over a data center mesh network mis-ordering
85 of packet flows needs to be avoided to prevent slowing operations
86 caused by packet order recovery. To retain flow alignment within
87 tunneling protocols entropy codes need to be based on the flow of the
88 encapsulated packets.
90 Multi-tenant data center using Network Virtualization over L3 (NVO3)
91 [RFC7365, RFC8014, RFC8394] create virtual networks interconnecting
92 virtual servers within an overlay on top of the physical underlay
93 network. In these NVO3 networks virtual network packets are
94 multiplexed into tunnels which extend over the physical underlay
95 network. The encapsulations used to in the NVO3 tunnels (i.e. VxLAN,
96 GENEVE, GUE) have an outer UDP header followed by an outer NVO3
97 header. The NVO3 header carries a key called the Virtual Network
98 Identifier (VNI) which identifies the virtual network within the
99 virtual overlay network. Each of the virtual overlay networks is a
100 separate subnetwork which can have its own IP and L2 virtual address
101 space.
103 Since a single NVO3 tunnel is used between communicating servers, any
104 server to server connection has the same IP source, destination, and
105 UDP destination port. To retain entropy for load balancing the NVO3
106 protocols use the UDP source port to hold a hash of the encapsulated
107 inner packet. This outer UDP source port provides the entropy
108 necessary for spreading traffic over the mesh based on the
109 encapsulated flow. Other fields such as the IPv6 flow label could
110 also be used, however these are not universally supported in data
111 center switching infrastructures, while the use of UDP source port is
112 broadly available in data center switches, routers, and middle boxes.
114 The NVO3 protocols isolate tenant virtual networks based on the VNI
115 identifiers carried in the tunnel headers. Since any bad actor
116 connected to the data center underlay network could spoof an
117 encapsulation transporting a virtual network and any device in the
118 middle of the communication can monitor the tenant networks, NVO3
119 networks must operate in a secure perimeter. With the rise of more
120 aggressive bad actors it is desirable to provide secure connections
121 for NOV3 tunnels to eliminate the threat of a server or switch within
122 the data center underlay monitoring or interfering with the operation
123 of virtual networks throughout the data center.
125 Encryption of [RFC4301, RFC4303, RFC7321] the NVO3 tunnels can
126 provide protection against devices outside the virtual overlay from
127 monitoring, spoofing or interfering with the virtual networks. This
128 can be done using IPsec to encrypt the tunnels carrying virtual
129 networks between servers. Since the tunnels can be encrypted using
130 smart network interfaces this method can be very efficient, retaining
131 the high performance required within data centers.
133 If we apply IPsec directly to the NVO3 tunnels the IP source and
134 destination as well as the protocol type and SPI will be the same for
135 each server to server communication, therefore we will lose the
136 entropy needed to support the data center mesh network. Internet
137 Draft "Encapsulating IPsec ESP in UDP for Load-balancing" [IPSEC-
138 LB], which proposes using the source port of IPsec transport mode ESP
139 in UDP encapsulated packets for entropy, provides the solution needed
140 to support the use of IPsec to secure the tunnels used for NVO3
141 traffic in data centers. By using transport mode ESP in UDP
142 encapsulation of NVO3 tunnels entropy can be provided by using the
143 UDP source port just as was done originally for the NVO3 UDP
144 encapsulations.
146 2. Generic Data Center Network Architecture
148 Figure 1 depicts a typical Clos mesh network [RFC7938] used in a data
149 center. Each server is typically redundantly connected to a set of
150 switches located at the Top of the Rack (ToR) switch (also called a
151 leaf switch). Each of these switches is, in turn, connected to a set
152 of switches for the row of racks commonly called the End of Row (EoR)
153 switch (also called a spine switch). Typically these redundant links
154 are managed either by L2 link aggregation protocols [IEEE-AX, IEEE-Q]
155 or by L3 equal cost multi-path protocols [RFC7938]. The number of
156 links interconnecting these layers varies depending on the bandwidth
157 and resiliency requirements. The figure shows a two level hierarchy,
158 however it is common for data centers to have a three level hierarchy
159 where a similar Clos mesh is used to interconnect rows of servers.
161 +--------+ +--------+
162 | EoR | | EoR |
163 | switch | | switch |
164 +--------+ +--------+
165 / / | \ / | \ \
166 / / +---\----/-----\-----------------+
167 / / \/ | \ \ |
168 / / / \ | \ \ |
169 / +-------/--------/------\-+ \ \ |
170 / | / / \ \ \ |
171 +--------+ +--------+ +--------+ +--------+
172 | ToR | | ToR | | ToR | | ToR |
173 | switch | | switch | | switch | | switch |
174 +--------+ +--------+ +--------+ +--------+
175 / \ / \ / \ / \
176 / \ / \ / \ / \
177 / \/ \ / \/ \
178 / / \ \ / / \ \
179 / / \ \ / / \ \
180 / / \ \ / / \ \
181 '-----------' '-----------' '-----------' '-----------'
182 : Server : : Server : : Server : : Server :
183 : : : : : : : :
184 '-----------' '-----------' '-----------' '-----------'
186 Figure 1: Typical Data Center Mesh Network
187 To distribute packets over the network paths typically a hash
188 function is used to reduce the fields within the outer packet headers
189 into a group of flows. The group is then allocated to a network link
190 or path. In the simplest and most common implementation the
191 distributions are done based on a hash. In more sophisticated
192 implementation additional load data and timing information may be
193 used to move flow groups based on load estimates.
195 3. Network Virtualization Over L3 (NVO3) Architecture
197 In an NVO3 multi-tenant data center the physical interconnect
198 depicted in figure 1 is used as the underlay physical IP network
199 where IP addresses are assigned to servers.
201 +--------+ +--------+
202 | Tenant +--+ +----| Tenant |
203 | System | | (') | System |
204 +--------+ | ................. ( ) +--------+
205 | +---+ +---+ (_)
206 +--|NVE|---+ +---|NVE|-----+
207 +---+ | | +---+
208 / . +-----+ .
209 / . +--| NVA |--+ .
210 / . | +-----+ \ .
211 | . | \ .
212 | . | Overlay +--+--++--------+
213 +--------+ | . | Network | NVE || Tenant |
214 | Tenant +--+ . | | || System |
215 | System | . \ +---+ +--+--++--------+
216 +--------+ .....|NVE|.........
217 +---+
218 |
219 |
220 =====================
221 | |
222 +--------+ +--------+
223 | Tenant | | Tenant |
224 | System | | System |
225 +--------+ +--------+
227 Figure 2: NVO3 Architecture Reference Diagram
229 The NVO3 protocols are used on top of the physical underlay network
230 to create virtual networks which overlay the physical underlay
231 network. The virtual networks are carried over the underlay
232 encapsulated in an NVO3 encapsulation protocol such as GENEVE
233 [RFC8926], VxLAN [RFC7348], or GUE. These encapsulations indicate the
234 virtual network by encoding a Virtual Network Identifier (VNI) within
235 the encapsulation header.
237 Figure 2 is a copy of the NVO3 architecture [RFC8014] reference
238 diagram. In figure 2 the Network Virtual Edge (NVE) entities provide
239 the tunnel terminations for the encapsulation protocols. The NVEs can
240 be located within the server's hypervisor, within smart NICs on the
241 servers, or within switches of the physical network [RFC8394].
243 The tenant systems (TS) are virtualized servers. These may be virtual
244 machines, containers, or physical servers that are connected over the
245 virtual networks and multiplexed into NOV3 tunnel by the NVEs.
247 The network virtualization authority (NVA) manages the creation and
248 configuration of virtual networks by configuring the NVEs.
250 4. Load Balancing Secure NVO3 tunnels
252 The NVO3 protocols isolate tenant virtual networks based on the
253 Virtual Network Identifiers carried in the tunnel header. Since any
254 bad actor inside the data center could spoof an encapsulation
255 transporting a virtual network and any device in the middle of the
256 communication can monitor the tenant networks, these networks are
257 only secure when operating within a secure perimeter. With the rise
258 of more aggressive bad actors it is desirable to provide secure
259 connections for NVO3 tunnels to eliminate the threat of a server or
260 switch within the data center underlay monitoring or interfering with
261 the operation of overlay virtual networks.
263 Encryption of [RFC4301, RFC4303, RFC7321] the NVO3 tunnels provides
264 protection against devices outside the virtual overlay from
265 monitoring, spoofing or interfering with the virtual networks. This
266 can be done using IPsec to encrypt the tunnels carrying virtual
267 networks between servers. Since the tunnels can be encrypted using
268 smart network interfaces this method can be very efficient, retaining
269 the high performance required within data centers. However since the
270 resulting tunnel headers don't provide enough entropy to support load
271 balancing over the data center mesh networks the resulting network
272 bandwidth could be greatly reduced.
274 To solve the load balancing problem produced by securing the NVO3
275 tunnels using IPsec the method proposed in Internet Draft
276 "Encapsulating IPsec ESP in UDP for Load-balancing" [IPSEC-LB] can be
277 used. Applying I-D.draft-xu-ipsecme-esp-in-udp-lb the NVO3 tunnels
278 are encapsulated as IPsec transparent mode ESP in UDP packets. Given
279 the UDP header on the outside of the IPsec tunnel the source port can
280 be used for entropy. By copying the source port from the original
281 NVO3 encapsulation into the IPsec ESP in UDP header it is possible to
282 retain the flow identity of the encrypted virtual network. Since the
283 NVO3 encapsulation source port contains an entropy code based on the
284 encapsulated overlay packet the resulting packet will provide the
285 entropy necessary to support load balancing in the data center mesh
286 network.
288 5. Security Considerations
290 Here we use IPsec to secure NVO3 tunnels between data center NVEs to
291 prevent attacks from servers or switches located within the data
292 center physical underlay, however outside of the overlay networks or
293 the NVE tunnel terminations and NVA managers. To fully secure a
294 multi-tenant network additional security methods need to be used to
295 prevent attackers from infiltrating the overlay infrastructure
296 including the Tenant Systems, NVEs and NVA.
298 6. IANA Considerations
300 The Internet Draft "Encapsulating IPsec ESP in UDP for Load-
301 balancing" [IPSEC-LB] proposes getting a new UDP destination port
302 assignment for use with load balanced IPsec. The use of a new port
303 would prevent existing implementations of IKE from operating with a
304 load balanced transparent mode ESP in UDP stream. It does not appear
305 this is necessary. Instead, the existing ESP in UDP port 4500 could
306 be used, provided both ends of the connection are configured to
307 exchanging ESP in UDP with an entropy code in the UDP source port. If
308 the existing ESP in UDP port 4500 is used, then there are no IANA
309 considerations since no new code points are necessary.
311 7. Conclusions
313 IPsec may be used to secure the underlay of a NVO3 multi-tenant data
314 center by encrypting the NVO3 tunnels. To make IPsec a viable
315 solution the IPsec tunnels need to provide load balancing.
317 By applying the proposal in Internet Draft "Encapsulating IPsec ESP
318 in UDP for Load-balancing" [IPSEC-LB], entropy can be added to the
319 IPsec packet header using the UDP source port of the ESP in UDP IPsec
320 packets. In particular, the source port of the original NVO3 tunnel
321 header can be copied to the new IPsec ESP in UDP source port
322 providing the necessary entropy while retaining the flow identity of
323 the encapsulated overlay packet.
325 8. Normative References
327 [IPSEC-LB] Xu, X., Hegde, S., Pismenny, B., Zhang, D., and Xia, L.,
328 "Encapsulating IPsec ESP in UDP for Load-balancing",
329 December 2020, .
332 [IEEE-AX] "IEEE Standard for Local and Metropolitan Area Networks--
333 Link Aggregation," in IEEE Std 802.1AX-2020 (Revision of
334 IEEE Std 802.1AX-2014), vol., no., pp.1-333, 29 May 2020,
335 doi: 10.1109/IEEESTD.2020.9105034.
337 [IEEE-Q] "IEEE Standard for Local and Metropolitan Area Network--
338 Bridges and Bridged Networks," in IEEE Std 802.1Q-2018
339 (Revision of IEEE Std 802.1Q-2014), vol., no., pp.1-1993, 6
340 July 2018, doi: 10.1109/IEEESTD.2018.8403927.
342 [RFC768] Postel, J., "User Datagram Protocol", STD 6, RFC 768, DOI
343 10.17487/RFC0768, August 1980, .
346 [RFC1122] Braden, R., Ed., "Requirements for Internet Hosts -
347 Communication Layers", STD 3, RFC 112, DOI
348 10.17487/RFC1122, October 1989, .
351 [RFC1191] Mogul, J. and S. Deering, "Path MTU discovery", RFC 1191,
352 DOI 10.17487/RFC1191, November 1990, .
355 [RFC2003] Perkins, C., "IP Encapsulation within IP", RFC 2003, DOI
356 10.17487/RFC2003, October 1996, .
359 [RFC3948] Huttunen, A., Swander, B., Volpe, V., DiBurro, L., and M.
360 Stenberg, "UDP Encapsulation of IPsec ESP Packets", RFC
361 3948, DOI 10.17487/RFC3948, January 2005, .
364 [RFC4301] Kent,S., Seo, K., "Security Architecture for the Internet
365 Protocol", RFC 4301, December 2005,
366
368 [RFC4303] Kent, S., "IP Encapsulating Security Payload", RFC 4303,
369 December 2005,
371 [RFC4821] Mathis, M. and J. Heffner, "Packetization Layer Path MTU
372 Discovery", RFC 4821, DOI 10.17487/RFC4821, March 2007,
373 .
375 [RFC7296] Kaufman, C., Hoffman, P., Nir, Y., Eronen, P., and T.
376 Kivinen, "Internet Key Exchange Protocol Version 2
377 (IKEv2)", STD 79, RFC 7296, DOI 10.17487/RFC7296, October
378 2014, .
380 [RFC7321] McGrew, D., Hoffman, P., "Cryptographic Algorithm
381 Implementation Requirements and Usage Guidance for
382 Encapsulating Security Payload (ESP) and Authentication
383 Header (AH)", RFC 7321, August 2014,
384 https://datatracker.ietf.org/doc/rfc7321/
386 [RFC7348] Mahalingam, M., et. al., "Virtual eXtensible Local Area
387 Network (VXLAN): A Framework for Overlaying Virtual Layer 2
388 Networks over Layer 3 Networks", RFC 7348, August 2014,
389
391 [RFC7365] Lasserre, M., et al, "Framework for Data Center (DC)
392 Network Virtualization", October 2014,
393
395 [RFC7938] Lapukhov, P., Premji, A., Mitchell, J., Ed., "Use of BGP
396 for Routing in Large-Scale Data Centers", August 2016,
397 .
399 [RFC8014] Black, D., et al, "An Architecture for Data-Center Network
400 Virtualization over Layer 3 (NVO3)", December 2016,
401
403 [RFC8394] Li, Y., Eastlake, D., Kreeger, L. Narten, T., Black, D.,
404 "Split Network Virtualization Edge (Split-NVE) Control-
405 Plane Requirements", RFC 8394, May 2018,
406
408 [RFC8926] Gross, J., Ganga, I., Sridhar, T., "Geneve: Generic Network
409 Virtualization Encapsulation", November 2020,
410
412 9. Informative References
414 [RFC6438] Carpenter, B. and S. Amante, "Using the IPv6 Flow Label for
415 Equal Cost Multipath Routing and Link Aggregation in
416 Tunnels", RFC 6438, DOI 10.17487/RFC6438, November 2011,
417 .
419 [RFC8200] Deering, S., Hiden, R., "Internet Protocol, Version 6
420 (IPv6) Specification", STD 76, RFC 8200, July 2017,
421
423 [RFC8201] McCann, J., Deering, S., Mogul, J., Hiden, R., Ed., "Path
424 MTU Discovery for IP version 6", STD 87, RFC 8201, July
425 2017,
427 10. Acknowledgments
429 This document was prepared using 2-Word-v2.0.template.dot.
431 Authors' Addresses
433 Paul Bottorff
434 Aruba a Hewlett Packard Enterprise Co
435 8000 Foothill Blvd.
436 Roseville, CA 95747
438 Email: paul.bottorff@hpe.com