IPv6 Operations (v6ops) Working Group E. Vasilenko Internet Draft X. Xiao Intended status: Informational Huawei Technologies Expires: March 2022 D. Khaustov Rostelecom September 17, 2021 IPv6 Oversized Packets Analysis draft-vasilenko-v6ops-ipv6-oversized-analysis-01 Abstract The IETF has some initiatives relying on IPv6 Extension Headers added in transit: SRv6, iOAM. Additionally, some recent developments are overlays (SRv6, VxLAN, NVO3, L2TPv3, and LISP). It could create oversized packets that need to be dealt with. This document analyzes available standards for the resolution of oversized packet drops. Status of this Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on February 2021. Copyright Notice Copyright (c) 2021 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents Vasilenko Expires February 25, 2022 [Page 1] Internet-Draft Oversized Analysis August 2021 carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Table of Contents 1. Terminology and pre-requisite..................................2 2. Problem statement..............................................3 3. Solutions......................................................5 3.1. Provision links with big enough MTU.......................6 3.2. Frugal usage of Extension Headers.........................7 3.3. Fragmentation and reassembly at the tunnel ends...........9 3.4. PMTUD by original packet source..........................12 3.5. Packetization Layer MTU Discovery........................14 4. Conclusion....................................................15 5. Security Considerations.......................................15 6. IANA Considerations...........................................15 7. References....................................................16 7.1. Normative References.....................................16 7.2. Informative References...................................18 8. Acknowledgments...............................................19 1. Terminology and pre-requisite We do assume good knowledge or frequent references to [PMTUD] and [IPv6 Tunneling]. Terminology is inherited from [PMTUD]. Link MTU - the maximum transmission unit, i.e., maximum packet size in octets that can be conveyed over a link. Path MTU (PMTU) - the minimum link MTU of all links in a path between a source node and a destination node. Path MTU Discovery (PMTUD) - the process by which a node learns the PMTU of a path. EMTU_S - Effective MTU for sending; used by upper-layer protocols to limit the size of IP packets they queue for sending. EMTU_R - Effective MTU for receiving; the largest packet that can be reassembled at the receiver. Vasilenko Expires February 25, 2022 [Page 2] Internet-Draft Oversized Analysis August 2021 Packetization Layer - the layer of the network stack that segments data into packets. PLPMTUD - Packetization Layer Path MTU Discovery, the method of detecting path MTU at packetization layer, which is an extension of classical PMTU Discovery. PTB (Packet Too Big) message - an ICMPv6 message reporting that an IPv6 packet is too large to forward through some link. MSS - the TCP Maximum Segment Size, the maximum payload size available to the TCP layer. This is typically the Path MTU minus the size of the IP and TCP headers. 2. Problem statement IPv6 is strict regarding fragmentation - it must NOT be done in transit (section 4.5 of [IPv6]). IPv6 sees rapid developments in recent years. A lot of additional functionality has been added primarily by adding options to Extension Headers and/or using overlay encapsulation. All of the above expand the packet size. This could lead to oversized packets that would be dropped on some links. Massive parallelism in traffic delivery is the additional challenge developed in the last 10 years: ECMP on one hop could reach 16 (or even more), which creates the end-to-end possibility for 64k paths on just 5 hops (example from big production network). Different paths could have a different set of Extension Headers and different PMTU as a result. PMTU is effectively becoming dynamic: we could never know how many additional headers would be added at a particular time to the particular packet on the particular path. The old classical PMTUD problems are still with us: filtered ICMPv6 messages, drops related to Extension Headers before next hop MTU has been evaluated (no Packet Too Big message sent). Standards have two important numbers that we would need for our discussion: o [IPv6] chapter 5 requires that every link should have the MTU of 1280 octets or greater (2^10+2^8 - it probably explains the choice of this size) Vasilenko Expires February 25, 2022 [Page 3] Internet-Draft Oversized Analysis August 2021 o [IPv6] requests minimum EMTU_R (reassembly buffer) in 1500 octets. An upper-layer protocol or application that depends on IPv6 fragmentation to send packets larger than the MTU of a path should not send packets larger than 1500 octets unless it has the assurance that the destination is capable of reassembling packets of that larger size The reassembly buffer is much above 1500B for the majority of desktop and server OSes. Windows 10 has "Reassemblylimit" almost 64MB (you could look by "netsh interface ipv6 show global"). Different flavors of Linux have "ipfrag_high_thresh" between 256KB and 4MB (you could look by "more /proc/sys/net/ipv4/ipfrag_high_thresh"). iOS has "maxReceiveIPv6BufferSize" 64Kb. The reassembly is not so good for embedded OSes. From the four primary OSes for IoT (Contiki, FreeRTOS, Mbed OS, MicroPython), only Mbed OS has the capability for 5 fragments by default, and it is possible to activate reassembly on Contiki. In all cases, the buffer is just a few packets of 1280B or 1500B. IoT devices may not be capable to reassemble the packet that the server in the cloud would send to it. Hence, ICMP PTB is still very important for some OSes. There is only one solution by [IPv6] architecture for the PMTU problem - decrease packet size on the original source. It is workable up to the minimum limit for IPv6 packets (1280B). The typical transit link had MTU not much bigger than 1500B for a long time, only the space for a few additional MPLS labels was reserved. 220B left could be considered as guaranteed for additional functionality in Extension and Encapsulation headers. It could be enough for the next decade if we would make some precautions - see discussion below. [Huston-2016] and [Huston-2021] did an investigation on a different topic (fragmentation), but he has good statistics related to MTU drops up to 1500B that did show a 5% drop for MTU as small as 1455B. Additionally, [Huston-2016] has found the big drop spike (69% from all drops!) at 1480B, 20B less is presumable for IPv6 encapsulation into IPv4. [Huston-2021] has shown twice bigger fragmentation drop for bigger packets with the peak at 1408 octets, especially for Asia. As you can see - 1500B is not always available now, the reason is not well understood. Hence, we do not have 220B for additional headers in all situations. We could be reasonably optimistic that such a type of tunneling would disappear in the long term. Later, we would stick to an optimistic assumption that 220B is available in most situations. It is still possible to have the more pessimistic estimation (200B? 175B?) looking to Huston's data. Vasilenko Expires February 25, 2022 [Page 4] Internet-Draft Oversized Analysis August 2021 The hungriest protocol known is SRv6 that could add 40B of IPv6 underlay tunnel header (called "outer IP header" in [SRH]), 16B of SRH header itself, and additionally up to 10 IPv6 addresses in the SID stack (potentially even more). It is already 216B - very close to 220B optimistic limit. It makes the introduction of any additional functionality quite challenging without rigorous expansion of all links to bigger MTU. Initial SRv6 implementations that trespassed safe limit in 220B are the reason for recent activities in MTU problem research. We see many recent efforts to improve Path MTU Discovery (which would be mentioned in the document) - let us find the rationale behind it. 3. Solutions Let's consider first the reassembly buffer problem as the simpler one. Minimal buffer for packet reassembly (1500B) is potentially possible to increase in new standard updates, but then would be the problem with the transition, because this limitation would be already programmed into billions of IoT hosts - it would need big time to be sure that we do not have old implementation anymore. There is no good solution for the problem of bloating headers above 220B for hosts. We need to keep headers below the 220B limit for embedded OSes. Fortunately, we are far from this problem yet - very limited additional functionality is implemented directly on the hosts (like [PMTU by HbH] or APN6). This problem should be looked at again in some number of years, it may be that in the future we would have to increase default EMTU_R on all hosts to give the possibility for new functionality. Let's now return to our primary problem of not enough PMTU. There is a low probability that the Internet community would agree to decrease the minimal IPv6 packet size (1280B). Hence, the oversized problem could not be resolved in that direction. It is possible to partially alleviate the MTU problem in some network zones where all transit nodes have big enough MTU. Transit nodes should delete extension headers before packets would leave "high MTU network zone". The leakage of a big header to a host could overflow EMTU_R buffer. The majority of RFCs recommend carriers delete additional headers before forwarding traffic to the client - this practice should be strictly followed. Vasilenko Expires February 25, 2022 [Page 5] Internet-Draft Oversized Analysis August 2021 The SPRING working group is actively developing a compressed version of SRv6 that should leave space for other functionality, even on current transit routers that sometimes do not support much above 1500B. All solutions for packet drop avoidance as a result of oversized packets could be classified into 4 classes. They are examined one by one. 3.1. Provision links with big enough MTU MTU supported by the host's links is typically 1500 Bytes. Backbone link's MTU could be up to 9000 Bytes on modern hardware. PMTUD is not needed in an ideal world. Reality is not that good: o Some old devices still support just a few additional MPLS labels above 1500B on Ethernet. It was historically a problem to cross 1536B because the IEEE specification for 802.3 assumes that a bigger number in the Length field means the Type of the payload. o We could have middleboxes that would not support MTU much bigger than 1500B MTU for a long time. o Ethernet is very mature now in relation to big MTU support, but that could be a challenge for other link-layer technologies (for example WiFi, satellite links, radio links, etc.). o Packet Links could be rented from a third party - no possibility to change the MTU. o Big MTU may negatively influence buffer size - see below. o The majority of vendors set the default MTU to 1500B (with variations on what is counted inside MTU). It is time-consuming to change the MTU on the production network. o Some hosts (especially for storage traffic in Data Centers) could use 2500B or 9000B MTU that challenges the possibility of having always bigger MTU in the backbone. Cost-optimized equipment architecture (especially used for switches, but applicable for many routers as well) may not split packets in the buffer memory. So small packet would occupy a bigger buffer space reserved for the packet with maximum MTU. This limitation effectively decreases the potential number of packets that could be buffered. Most of the host packets are still limited to 1500B size. MTU 9000B would just lead to wasting buffer memory with an efficiency of 1/6. The average packet size is twice smaller, hence in the worst-case buffer efficiency could be up to 1/12. Buffer memory is about 30% of the router cost. It is not acceptable to Vasilenko Expires February 25, 2022 [Page 6] Internet-Draft Oversized Analysis August 2021 increase buffer memory cost 12 times. Hence, in many cases, it does not make sense to increase MTU to the maximum supported by the switch or router. One should always check with the vendor the impact of using a big MTU on buffering for the particular product. MTU should be increased to the number that is bigger than the maximum MTU expected from hosts + the size of all possible network overhead + underlay IPv6 header (if present). 9000B MTU makes sense in DC, cross-DC environment, or for platforms that fragment packets for smaller sells in the memory. [MTU issues in Tunneling] section 3.3 discusses the opposite solution: decrease MTU on links to hosts to be sure that a host would always generate small enough MTU for the backbone. This solution was possible for small tunnel overhead, but now we are talking about the situation when 220B margin is not enough. [L3VPN] and [EVPN] do attach an additional label and could create oversized packets. Still, the MPLS header cannot point to the original MPLS router that has an attached service label. Additionally, a VPN IP packet could use private address space or no IP address at all (for EVPN). It blocks the possibility to properly organize the PMTUD process. Hence, [L3VPN] and [EVPN] have been developed under the assumption that all MTUs on the path would be expanded for at least 8 bytes that are needed for services over the MPLS data plane. We have recent [Generic Delivery Functions] that may permit fragmentation for MPLS services, but it is a personal draft yet. [Pseudowire Fragmentation] is the rear case when fragmentation is available over MPLS for one type of service. [VxLAN] section 4.3 also uses the approach: "it is RECOMMENDED that the MTUs across the physical network infrastructure be set to a value that accommodates the larger frame size due to the encapsulation". Packet drop statistics and big activity in IETF prove that the PMTUD problem persists. "Raise MTU on transit" is the best solution, if it is available. 3.2. Frugal usage of Extension Headers Some new functionality (especially source routing with a big SID stack) could decrease headers size without a big loss of functionality (for example, use loose node appointment in SID Vasilenko Expires February 25, 2022 [Page 7] Internet-Draft Oversized Analysis August 2021 stack). Some functionality (like iFIT or iOAM) could be completely omitted in the situation that would lead to packet drop. It is effectively "the tradeoff of functionality to PMTU control". The important point here is that the transit node attaching an additional header should be aware of all MTUs along the assumed packet path to predict how big MTU is still acceptable. [PMTUD] is readily available for tunneling interfaces - tunnel source should be aware of PMTU of the tunnel (by PTB feedback messages). But we have cases when it is not enough: o SDN controller (or management system in general) could assist in provisioning of extension headers (including SFC, iOAM, BIER) and encapsulation headers (SRv6, VxLAN) - should be the way to report MTUs to Controller. o ICMPv6 PTB would be directed to the transit control plane only in the case of problems inside the tunnel. PTB messages from outside of the tunnel would be directed to the source node. It is difficult to snoop PTB on transit nodes. Hence, we see many initiatives to collect and manage MTU by many popular protocols for routing and traffic engineering: [PMTU by ISIS], [PMTU by BGP-LS], [PMTU by PCEP], [PMTU by SR-Policy]. Moreover, these protocol extensions would become even more useful in the future when it would not be possible to squeeze all extension headers into 220B anymore. Frugal attachment of new headers on transit nodes would increase the need for awareness of PMTU - it should stimulate MTU collection by all other popular protocols (OSPF, normal BGP on peering borders). This approach has a fundamental problem: full knowledge about all MTUs in the domain could not help to estimate the real path for a packet, because of massive ECMP used by many networks (at least by all Carriers). Non-routing protocols do not have a proper engine to estimate traffic paths and predict PMTU as well. And even more, if L2 ECMP is used or some links are rented from another carrier it will again be impossible to predict the exact path and the PMTU. The second problem of this approach could be classified as "chicken and egg". We already have a much better solution for MTU drop - increase MTU (see the previous section). We are looking for other solutions only because upgrading equipment (to better MTU) is not possible for some reason. But new protocols introduction would also demand equipment upgrade and thus making frugal headers less Vasilenko Expires February 25, 2022 [Page 8] Internet-Draft Oversized Analysis August 2021 valuable. However, an upgrade for the control plane should be cheaper than an upgrade for the data plane, if the vendor would support such an approach. Hence, the solution discussed in this section has only limited applicability. 3.3. Fragmentation and reassembly at the tunnel ends The tunnel source behaves like a host with respect to the tunnel header. It is possible to properly adjust PMTU for the tunnel by [PMTUD], so it is potentially possible to fragment all packets bigger than PMTU. [IP Encapsulation] is the earliest standard for IP-in-IP encapsulation. Section 5.1 discusses that it is possible to fragment IP packets before tunnel encapsulation, so there is no need to reassemble packets on the other tunnel end - reassembly could happen on the destination host. It does not have additional cost implications on tunnel ends. This approach did work for IPv4 in the case of the "don't fragment" bit cleared. It fully contradicts IPv6 architecture that does not permit to fragment packets on transit - no standard has risked proposing such a solution for IPv6. Some standards do propose IPv6 fragmentation (primarily for packets 1280B and below), but fragmentation is recommended after encapsulation. It would lead to packet reassembly on the other tunnel end to hide (from destination host) the fact of transit fragmentation. It does minimize IPv6 architecture disruption. Many standards discussed below ([MPLS Encapsulation], [L2TPv3], [VxLAN], [NVO3]) forgot to mention that packets 1280 and below should be fragmented. This inaccuracy did not create any problem in production networks because we typically have 220B for all headers - it is big enough for many tunnels nested into each other. The situation could change in the next years because of Extension Headers expansion by different functions. It could create pressure to return to many mature standards and clarify the situation: what to do when the 1280B packet could not go through the tunnel. The Fragmentation has a few issues that make it not popular: o Fragmentation could double buffer requirements (we assume split only in 2 fragments). We could ignore small additional buffer requirements for packets that may be lost and need to wait some time before reassembly, the Internet is not productive anyway Vasilenko Expires February 25, 2022 [Page 9] Internet-Draft Oversized Analysis August 2021 after a few percentages of packet drops. The buffer memory is about 30% of the router cost. A 30% cost increase would not be accepted by the majority of owners. Albeit, some middleboxes already have enough buffer memory that could be reused for packet reassembly. o In general, IPv6 architecture does not approve fragmentation in transit in all standards (except the recent draft [IP Tunnels] - see below). [PMTUD] section 5.1: "packetization layers are encouraged to avoid sending messages that will require fragmentation". We would discuss in this section some situations when tunnel fragmentation is inevitable. o [Fragile Fragmentation] has a good collection of all problems related to fragmentation (additionally to the above: breaks ECMP, stateful processing, policy routing, and has many security attack vectors). [Fragile Fragmentation] strongly recommends avoiding fragmentation, but not deprecate yet. The primary RFC for tunneling is [IPv6 Tunneling] - it is the oldest standard that was later reused by many other standards (including the latest SRH). It permits fragmentation only for the case when the original packet is already minimal (1280B or less) - see section 7.1. It mandates dropping the packet and signaling ICMPv6 PTB to the source (request to decrease the PMTU size at the source) for all other cases. [MPLS Encapsulation] Section 5.1 has the name: "Preventing Fragmentation and Reassembly". It does stress again: "IPv6 intermediate nodes do not perform fragmentation in any event". [L2TPv3] section 4.1.4 has a similar comment: "Note that IPv6 does not support "in-flight" fragmentation of data packets". [VxLAN] section 4.3 is strict: "VTEPs MUST NOT fragment VXLAN packets." [NVO3] section 4.4.4 is strict too: "It is strongly RECOMMENDED that Path MTU Discovery ([PMTUD]) be used to prevent or minimize fragmentation." [IPv6 GRE] section 3.3 does recommend fragmentation only for packets that are less than 1280B. The most recent draft for all types of tunnels is [IP Tunnels]. It is already referenced by many IETF documents. It is complicated to cover all use cases (any IP over any IP in any situation), but the Vasilenko Expires February 25, 2022 [Page 10] Internet-Draft Oversized Analysis August 2021 net result is: much bigger part of the traffic proposed to be fragmented into the tunnel. Section 3.3: "The path between ingress and egress interfaces has a path MTU, but the endpoints can exchange messages as large as can be reassembled at the destination (egress interface), i.e., the EMTU_R of the egress interface". A short explanation of proposed functionality: original host would try to transmit biggest flows (by volume) on maximum PMTU, that tunnel source would not try to correct by PTB messages up to 1500B. Hence, the tunnel source would not have any option except to fragment. The principal problem here is the absence of PTB messages for the packet size between PMTU and statically appointed EMTU_R. Let's see how it has been formulated in more detail. [IP Tunnels] introduces a new variable "Tunnel MTU" that should not change as a result of PMTUD. The procedure to change "Tunnel MTU" is out of the draft discussion - it is pushed to specifications of particular tunnels in the last paragraph of section 4.2.2. Moreover, it is even assumed that PLPMTUD could be used on the router for "Tunnel MTU" discovery because this parameter is considered as an above network layer (like transport layer on the host). Separate section 4.2.3 is dedicated to the explanation that the newly introduced "Tunnel MTU" cannot be adjusted dynamically. There is a recommendation for the default "Tunnel MTU": typical host EMTU_R (1500B) minus tunnel outer headers overhead. The good question could be: if it is so difficult to manage "Tunnel MTU" dynamically, then why this variable was introduced? The MTU of the tunnel is renamed into MAP (maximum atomic packet), MAP should be corrected by PMTUD feedback from inside the tunnel. Section 4.2.2 states that everything up to "Tunnel MTU" should be accepted to the tunnel, one long packet (with inner and outer headers) should be created. Then it should be split into fragments below MAP size. Initially, "tunnel MTU" and MAP could be manually synchronized by the administrator (with the difference in tunnel overhead). But any additional overhead on the tunnel path (nested tunnel, smallest Extension Header) would result in PMTUD that decreases MAP, but would not change "Tunnel MTU". It would turn on fragmentation for all bulk traffic. This situation is quite probable now (see [Huston- 2020] on MTUs available on the Internet) and it would be even more probable in the future when many additional extension headers would be used. Hence, the requirement in section 5.3.1 "do NOT try to deprecate fragmentation" is indeed important. Section 3.6 has the same approach as all other standards to the question of when fragmentation should happen: "this document assumes that only outer fragmentation is viable because it is the only approach that works for both IPv4 datagrams with DF=1 and IPv6". a considerable increase in fragmentation is proposed for the reasons Vasilenko Expires February 25, 2022 [Page 11] Internet-Draft Oversized Analysis August 2021 of academic purity: the router part of the router should behave as a router, the host part of the router should behave as a host without any deviations. Additional fragmentations would create all of the problems discussed in [Fragile Fragmentation] and substantially increase the cost of tunnel endpoints. There is a high probability that draft [IP Tunnels] would be rejected by the market for cost reasons. Additionally, we should point that statistics for fragmented packet drop on the Internet is still high enough (7%) - see [Huston-2021]. Fragmentation is the least probable solution for oversized packet drops. 3.4. PMTUD by original packet source [PMTUD] is mandatory in IPv6 architecture, because IPv6 does not have fragmentation in transit. We could see recommendations in many RFCs not to block ICMPv6 PTB completely (it could be rate-limited - see [ICMPv6] section 2.4). [DPLPMTUD] section 1.1 has a very good collection of reasons why PTB message may not be delivered to the source - it is used as justification to augment PMTUD by [DPLPMTUD]. We should not see this problem for all non-tunneling protocols in the majority of environments. ICMPv6 PTB should be delivered to packet source, packet source would dynamically decrease PMTU to adapt to new realities. PMTU could change dynamically because some transit nodes could introduce additional extension header ad-hoc or ECMP could switch flow to a different path. [IPv6 Tunneling] mandates to relay ICMPv6 PTB by tunnel ends for ICMPv6 messages received from the inside tunnel. [IPv6 Tunneling] does not use "relay" terminology, but section 8 explains in detail how to reconstruct and retransmit ICMP messages to the original packet source (delete all tunnel-related information). [MTU issues in Tunneling] section 3.2 discusses the same approach. [L2TPv3] section 4.1.4 refers to the [IPv6 Tunneling]. We could assume it as the request for PTB messages relay too. [SRH] section 5.4 confirms full adherence to ICMPv6 PTB relay approach: "For IP packets encapsulated in an outer IPv6 header, ICMP error handling is as defined in [IPv6 Tunneling]". [VxLAN] section 4.3 proposes to use PMTUD: "Path MTU discovery MAY be used to address this requirement as well". [NVO3] section 4.4.4 assumes PMTUD too: "It is strongly RECOMMENDED that Path MTU Discovery ([PMTUD]) be used to prevent or minimize Vasilenko Expires February 25, 2022 [Page 12] Internet-Draft Oversized Analysis August 2021 fragmentation". [IPSec] section 8.2.1 requests that PMTU should be maintained for the tunnel and signaled to the original packet source as soon as any new packet would arrive. [IPv6 GRE] section 3.3 clearly instructs developers to drop the oversized packets and send PTB for packets bigger than tunnel MTU. The method of PMTU detection is fully IPv6 compliant: "the GMTU is equal to the PMTU associated with the path between the GRE ingress and the GRE egress, minus the GRE overhead". [MPLS Encapsulation] section 5.1 specifies the same approach: tunnel head-end should use [PMTUD] to understand tunnel MTU, then "the packet will have to be discarded, but the tunnel head should send the IP source of the discarded packet the proper ICMP error message". [VxLAN], [NVO3], [IPSec], [IPv6 GRE], and [MPLS Encapsulation] do not request for tunnel endpoint to relay PTB messages. PMTUD should be used to set proper MTU for the tunnel, then subsequent packets could trigger PTB messages to the packet source. It would create an additional round trip delay compared to the original [IPv6 Tunneling] relay approach for the first PTB message. This small deficiency could be partially explained by the desire of many standards to be universal for IPv6 as well as IPv4. As a reminder, IPv4 may not have enough information in the ICMP message to properly reconstruct a relay message (64bits of source packet by RFC 792). [IP Tunnels] is the only draft that contradicts [IPv6 Tunneling] (and every other protocol based on top) - it does clearly prohibit relay PTB messages. It states in section 3.3: "When such messages (PTB) arrive at the ingress interface ("ingress interface" is the tunnel interface in this draft), they may affect the properties of that interface (e.g., its MTU), but they should never directly cause new ICMPs in the outer network". This idea is generalized in section 5.1 as "ICMP messages MUST NOT be generated by the tunnel (as a link)". The motivation assumed in the draft is to fully mimic host behavior on the router virtual (tunnel) interface, because the host would not retranslate PTB messages. We see that "Flow Label" is gaining popularity. [IPv6 Tunneling] and [ICMPv6] do not have strong recommendations for "Flow Label" - it was not an important topic at that time. The only small improvement that makes sense to do for [IPv6 Tunneling] is to recommend coping "Flow Label" from source packet to tunnel packet and from source packet to ICMPv6 PTB message. It would permit to properly load balance PTB messages to the same path as original traffic - see the problem [ICMPv6 PTB in ECMP] about hash-based load balancing between Vasilenko Expires February 25, 2022 [Page 13] Internet-Draft Oversized Analysis August 2021 many hosts. Copy "Flow Label" to PTB message would not contradict neither IPv6 architecture nor any RFC - it is not mandatory to develop a special standard update for it. [MTU issues in Tunneling] section 3.2 has a concern that in the case of Lawful Intercept additional encapsulation could produce PTB messages that would show the fact to the monitored host. It is not a very realistic concern, because PMTU could change for many other reasons (especially with the proliferation of new protocols). If it is still a concern, then it makes sense to use another solution for this case: bigger MTU (better) or even fragmentation. [MTU issues in Tunneling] section 3.2 raises the question about the applicability of "MSS Clamping". The transit node could snoop the transport layer and change MSS exchanged between nodes. This "hack" is not recommended because it breaks the layered model of IETF or OSI. [PMTUD] is the only mechanism that is universal for all cases and fully compliant with IPv6 architecture. Vendors just need to use it, despite some challenges to relay PTB messages on tunnel ends. Moreover, it makes sense to standardize the relay of PTB messages on tunnel ends - it would improve PMTUD time on original traffic sources for round trip time. [IPv6] RFC: "It is strongly recommended that IPv6 nodes implement Path MTU Discovery [PMTUD]". 3.5. Packetization Layer MTU Discovery [PLPMTUD] and [DPLPMTUD] have been greatly developed in recent years. Packetization Layer (UDP/TCP) (1) has much more visibility (could see the size of transport layer buffers); (2) could operate under the absence of ICMPv6 PTB (too much filtering); (3) could be very granular (per-flow). It does have its use cases. Albeit, PLPMTUD/DPLPMTUD have their restrictions as they: (1) are not universal for all transport protocols; (2) need more resources from the host; (3) are challenging to share PMTU information between applications; (4) need much more round trip times to find suitable PMTU; (5) do not work well on congested paths (difficult to understand the reason for packet loss). Hence, PLPMTUD is not a replacement for PMTUD - both are needed. As a reminder from [PLPMTUD]: "Packetization Layer Path MTU Discovery (PLPMTUD) is most efficient when used in conjunction with the ICMP- based Path MTU Discovery". Vasilenko Expires February 25, 2022 [Page 14] Internet-Draft Oversized Analysis August 2021 PLPMTUD could play as a replacement for PMTUD in the worst-case scenario (ICMP is filtered). It would lead to the original host PMTU decrease too. PLPMTUD could be considered as a redundancy mechanism for PMTUD. Strictly speaking, [PMTU by HbH] is a network layer mechanism, not a packetization layer. It is mentioned in this section because its usage is very similar to PLPMTUD, [PMTU by HbH] could be considered to some degree as the extension to PLPMTUD. It is not expected to principally change the conclusions of this document. 4. Conclusion It is better not to have a problem with oversized packets in the first place. One should upgrade all links to a bigger MTU, if possible. The host could have MTU as big as transit node. It would be never possible to deprecate PMTUD. It is important to follow the recommendations of [PMTUD] and [IPv6 Tunneling] for ICMPv6 PTB message delivery to the original traffic source. Tunnel sources should perform the relay function to make sure that the original traffic source would get the PTB message faster. The temporary 220B limit for all headers pushes us to the frugal implementation of new extension headers. This limit would be alleviated after all backbone links would be upgraded to a much bigger MTU than 1500B. Additional protocols to collect MTU information could help in the transition period to attach additional headers frugally. It is true for all new protocols: SRv6, SFC, BIER, iOAM, APN6. [PLPMTUD] and [DPLPMTUD] are not the replacement for [PMTUD], but could help in some scenarios. Fragmentation is not at all a solution for oversized packet drops. 5. Security Considerations [PMTUD], [PLPMTUD], [DPLPMTUD], and [Fragile Fragmentation] have some attack vectors discussed. This document does not introduce additional security vulnerabilities. 6. IANA Considerations This document has no request to IANA. Vasilenko Expires February 25, 2022 [Page 15] Internet-Draft Oversized Analysis August 2021 7. References 7.1. Normative References [IPv6] S. Deering, R. Hinden, "Internet Protocol, Version 6 (IPv6) Specification", RFC 8200, DOI 10.17487/RFC8200, July 2017, . [ICMPv6] A. Conta, S. Deering, M. Gupta, "Internet Control Message Protocol (ICMPv6) for the Internet Protocol Version 6 (IPv6) Specification", RFC 4443, DOI 10.17487/RFC4443, March 2006, . [PMTUD] J. McCann, S. Deering, J. Mogul, R. Hinden, "Path MTU Discovery for IP version 6", RFC 8201, DOI 10.17487/RFC8201, July 2017, . [IPv6 Tunneling] A. Conta, S. Deering, "Generic Packet Tunneling in IPv6 Specification", RFC 2473, DOI 10.17487/RFC2473, December 1998, . [ICMPv6 PTB in ECMP] M. Byerly, M. Hite, J. Jaeggli, "Close Encounters of the ICMP Type 2 Kind", RFC 7690, DOI 10.17487/RFC7690, January 2016, . [MTU issues in Tunneling] P. Savola, "MTU and Fragmentation Issues with In-the-Network Tunneling", RFC 4459, DOI 10.17487/RFC4459, April 2006, . [IP Tunnels] J. Touch, M. Townsley, "IP Tunnels in the Internet Architecture", draft-ietf-intarea-tunnels-10 (work in progress), September 2019. [IP Encapsulation] C. Perkins, "IP Encapsulation within IP", RFC 2003, DOI 10.17487/RFC2003, October 1996, . [IPSec] S. Kent, K. Seo, "Security Architecture for the Internet Protocol", RFC 4301, DOI 10.17487/RFC4301, December 2005, . Vasilenko Expires February 25, 2022 [Page 16] Internet-Draft Oversized Analysis August 2021 [IPv6 GRE] C. Pignataro, R. Bonica, S. Krishnan, "IPv6 Support for Generic Routing Encapsulation (GRE)", RFC 7676, DOI 10.17487/RFC7676, October 2015, . [MPLS Encapsulation] T. Worster, Y. Rekhter, E. Rosen, "Encapsulating MPLS in IP or Generic Routing Encapsulation (GRE)", RFC 4023, DOI 10.17487/RFC4023, March 2005, . [L2TPv3] J. Lau, M. Townsley, I. Goyret, "Layer Two Tunneling Protocol - Version 3 (L2TPv3)", RFC 3931, DOI 10.17487/RFC3931, March 2005, . [VxLAN] M. Mahalingam, D. Dutt, K. Duda, P. Agarwal, L. Kreeger, T. Sridhar, M. Bursell, C. Wright, "Virtual eXtensible Local Area Network (VXLAN): A Framework for Overlaying Virtualized Layer 2 Networks over Layer 3 Networks", RFC 7348, DOI 10.17487/RFC7348, August 2014, . [NVO3] J. Gross, I. Ganga, T. Sridhar, "Geneve: Generic Network Virtualization Encapsulation", RFC 8926, DOI 10.17487/RFC8926, November 2020, . [L3VPN] E. Rosen, Y. Rekhter, "BGP/MPLS IP Virtual Private Networks (VPNs)", RFC 4364, DOI 10.17487/RFC4364, February 2006, . [EVPN] A. Sajassi, R. Aggarwal, N. Bitar, A. Isaac, J. Uttaro, J. Drake, W. Henderickx, "BGP MPLS-Based Ethernet VPN", RFC 7432, DOI 10.17487/RFC7432, February 2015, . [Huston-2016] Huston, G., "Fragmenting IPv6", Blog Post, 2016, . [Huston-2021] Huston, G., "IPv6 Fragmention Loss", Article, 2021, . [Fragile Fragmentation] R. Bonica, F. Baker, G. Huston, R. Hinden, O. Troan, F. Gont, "IP Fragmentation Considered Fragile", RFC 8900, DOI 10.17487/RFC8900, September 2020, . Vasilenko Expires February 25, 2022 [Page 17] Internet-Draft Oversized Analysis August 2021 7.2. Informative References [PLPMTUD] M. Mathis, J. Heffner, "Packetization Layer Path MTU Discovery", RFC 4821, DOI 10.17487/RFC4821, March 2007, . [DPLPMTUD] G. Fairhurst, T. Jones, M. Tuexen, I. Ruengeler, T. Voelker, "Packetization Layer Path MTU Discovery for Datagram Transports", RFC 8899, DOI 10.17487/RFC8899, March 2020, . [SRH] C. Filsfils, D. Dukes, S. Previdi, J. Leddy, S. Matsushima, D. Voyer, "IPv6 Segment Routing Header (SRH)", RFC 8754, DOI 10.17487/RFC8754, March 2020, . [PMTU by HbH] R. Hinden, G. Fairhurst, "IPv6 Minimum Path MTU Hop- by-Hop Option", draft-hinden-6man-mtu-option-02 (work in progress), July 2019. [PMTU by ISIS] Z. Hu, Y. Zhu, Z. Li, L. Dai, "IS-IS Extensions for Path MTU", draft-hu-lsr-isis-path-mtu-00 (work in progress), June 2018. [PMTU by PCEP] S. Peng, C. Li, L. Han, "Support for Path MTU (PMTU) in the Path Computation Element (PCE)communication Protocol (PCEP)", draft-li-pce-pcep-pmtu-03 (work in progress), October 2020. [PMTU by BGP-LS] Y. Zhu, Z. Hu, G. Yan, J. Yao, "BGP-LS Extensions for Advertising Path MTU", draft-zhu-idr-bgp-ls-path-mtu- 05 (work in progress), November 2020. [PMTU by SR-Policy] C. Li, Y. Zhu, A. Sawaf, Z. Li, "Segment Routing Path MTU in BGP", draft-li-idr-sr-policy-path-mtu-03 (work in progress), November 2019. [Generic Delivery Functions] Z. Zhang, R. Bonica, K. Kompella, "Generic Delivery Functions", draft-zzhang-intarea- generic-delivery-functions-01 (work in progress), April 2021. [Pseudowire Fragmentation] A. Malis, M. Townsley, "Pseudowire Emulation Edge-to-Edge (PWE3) Fragmentation and Reassembly", RFC 4623, DOI 10.17487/RFC4623, August 2006, . Vasilenko Expires February 25, 2022 [Page 18] Internet-Draft Oversized Analysis August 2021 8. Acknowledgments Thanks to v6ops working group for problem discussion Authors' Addresses Eduard Vasilenko Huawei Technologies 17/4 Krylatskaya st, Moscow, Russia 121614 Email: Vasilenko.Eduard@huawei.com Xiao Xipeng Huawei Technologies 205 Hansaallee, 40549 Dusseldorf, Germany Email: Xipengxiao@huawei.com Dmitriy Khaustov Rostelecom 13/2 Nikoloyamskaya st, Moscow, Russia 109240 Email: Dmitriy.Khaustov@rt.ru Vasilenko Expires February 25, 2022 [Page 19]