Network Working Group N. Banks Internet-Draft Microsoft Corporation Intended status: Experimental December 23, 2020 Expires: June 26, 2021 QUIC Performance draft-banks-quic-performance-00 Abstract The QUIC performance protocol provides a simple, general-purpose protocol for testing the performance characteristics of a QUIC implementation. With this protocol a generic server can support any number of client-driven performance tests and configurations. Standardizing the performance protocol allows for easy comparisons across different QUIC implementations. Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on June 26, 2021. Copyright Notice Copyright (c) 2020 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of Banks Expires June 26, 2021 [Page 1] Internet-Draft QUIC-PERF December 2020 the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 1.1. Terms and Definitions . . . . . . . . . . . . . . . . . . 3 2. Specification . . . . . . . . . . . . . . . . . . . . . . . . 3 2.1. Protocol Negotiation . . . . . . . . . . . . . . . . . . 3 2.2. Configuration . . . . . . . . . . . . . . . . . . . . . . 3 2.3. Streams . . . . . . . . . . . . . . . . . . . . . . . . . 3 2.3.1. Encoding Server Response Size . . . . . . . . . . . . 4 2.3.2. Bidirectional vs Unidirectional Streams . . . . . . . 4 3. Example Performance Scenarios . . . . . . . . . . . . . . . . 4 3.1. Single Connection Bulk Throughput . . . . . . . . . . . . 4 3.2. Requests Per Second . . . . . . . . . . . . . . . . . . . 5 3.3. Handshakes Per Second . . . . . . . . . . . . . . . . . . 6 3.4. Throughput Fairness Index . . . . . . . . . . . . . . . . 6 3.5. Maximum Number of Idle Connections . . . . . . . . . . . 7 4. Things to Note . . . . . . . . . . . . . . . . . . . . . . . 7 4.1. What Data Should be Sent? . . . . . . . . . . . . . . . . 7 4.2. Ramp up Congestion Control or Not? . . . . . . . . . . . 7 4.3. Disabling Encryption . . . . . . . . . . . . . . . . . . 7 5. Security Considerations . . . . . . . . . . . . . . . . . . . 8 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 8 7. Normative References . . . . . . . . . . . . . . . . . . . . 8 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 8 1. Introduction The various QUIC implementations are still quite young and not exhaustively tested for many different performance heavy scenarios. Some have done their own testing, but many are just starting this process. Additionally, most only test the performance between their own client and server. The QUIC performance protocol aims to standardize the performance testing mechanisms. This will hopefully achieve the following: o Remove the need to redesign a performance test for each QUIC implementation. o Provide standard test cases that can produce performance metrics that can be easily compared across different configurations and implementations. o Allow for easy cross-implementation performance testing. Banks Expires June 26, 2021 [Page 2] Internet-Draft QUIC-PERF December 2020 1.1. Terms and Definitions The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here. 2. Specification The sections below describe the mechanisms used by a client to connect to a QUIC perf server and execute various performance scenarios. 2.1. Protocol Negotiation The ALPN used by the QUIC performance protocol is "perf". It can be used on any UDP port, but UDP port 443 is used by default, if no other port is specified. No SNI is required to connect, but may be optionally provided if the client wishes. 2.2. Configuration TODO - Possible options: use the first stream to exchange configurations data OR use a custom transport parameter. 2.3. Streams The performance protocol is primarily centered around sending and receiving data. Streams are the primary vehicle for this. All performance tests are client-driven: o The client opens a stream. o The client encodes the size of the requested server response. o The client sends any data it wishes to. o The client cleanly closes the stream with a FIN. When a server receives a stream does the following: o The server accepts the new stream. o The server processes the encoded response size. o The server drains the rest of the client data. Banks Expires June 26, 2021 [Page 3] Internet-Draft QUIC-PERF December 2020 o The server then sends any response payload that was requested. *Note* - Should the server wait for FIN before replying? 2.3.1. Encoding Server Response Size Every stream opened by the client uses the first 8 bytes of the stream data to encode a 64-bit unsigned integer in network byte order to indicate the length of data the client wishes the server to respond with. An encoded value of zero is perfectly legal, and a value of MAX_UINT64 (0xFFFFFFFFFFFFFFFF) is practically used to indicate an unlimited server response. The client may then cancel the transfer at its convenience with a STOP_SENDING frame. On the server side, any stream that is closed before all 8 bytes are received should just be ignored, and gracefully closed on its end (if applicable). 2.3.2. Bidirectional vs Unidirectional Streams When a client uses a bidirectional stream to request a response payload from the server, the server sends the requested data on the same stream. If no data is requested by the client, the server merely closes its side of the stream. When a client uses a unidirectional stream to request a response payload from the server, the server opens a new unidirectional stream to send the requested data. If no data is requested by the client, the server need take no action. 3. Example Performance Scenarios All stream payload based tests below can be achieved either with bidirectional or unidirectional streams. Generally, the goal of all these performance tests is to measure the maximum load that can be achieved with the given QUIC implementation and hardware configuration. To that end, the network is not expected to be the bottleneck in any of these tests. To achieve that, the appropriate network hardware must be used so as to not limit throughput. 3.1. Single Connection Bulk Throughput Bulk data throughput on a single QUIC connection is probably the most common metric when first discussing the performance of a QUIC implementation. It uses only a single QUIC connection. It may be either an upload or download. It can be of any desired length. Banks Expires June 26, 2021 [Page 4] Internet-Draft QUIC-PERF December 2020 For an upload test, the client need only open a single stream, encodes a zero server response size, sends the upload payload and then closes (FIN) the stream. For a download test, the client again opens a single stream, encodes the server's response size (N bytes) and then closes the stream. The total throughput rate is measured by the client, and is calculated by dividing the total bytes sent or received by difference in time from when the client created its initial stream to the time the client received the server's FIN. 3.2. Requests Per Second Another very common performance metric is calculating the maximum requests per second that a QUIC server can handle. Unlike the bulk throughput test above, this test generally requires many parallel connections (possibly from multiple client machines) in order to saturate the server properly. There are several variables that tend to directly affect the results of this test: o The number of parallel connections. o The size of the client's request. o The size of the server's response. All of the above variables may be changed to measure the maximum RPS in the given scenario. The test starts with the client connecting all parallel connections and waiting for them to be connected. It's recommended to wait an additional couple of seconds for things to settle down. The client then starts sending "requests" on each connection. Specifically, the client should keep at least one request pending (preferrably at least two) on each connection at all times. When a request completes (receive server's FIN) the client should immediately queue another request. The client continues to do this for a configured period of time. From my testing, ten seconds seems to be a good amount of time to reach the steady state. Finally, the client measures the maximum requests per second rate as the total number of requests completed divided by the total execution time of the requests phase of the connection (not including the handshake and wait period). Banks Expires June 26, 2021 [Page 5] Internet-Draft QUIC-PERF December 2020 3.3. Handshakes Per Second Another metric that may reveal the connection setup efficiency is handshakes per second. It lets multiple clients (possibly from multiple machines) setup QUIC connections (then close them by CONNECTION_CLOSE) with a single server. Variables that may potentially affect the results are: o The number of client machines. o The number of connections a client can initialize in a second. o The size of ClientHello (long list of supported ciphers, versions, etc.). All the variables may be changed to measure the maximum handshakes per second in a given scenario. The test starts with the multiple clients initializing connections and waiting for them to be connected with the single server on the other machine. It's recommended to wait an additional couple of seconds for connections to settle down. The clients will initialize as many connections as possible to saturate the server. Once the client receive the handshake from the server, it terminates the connection by sending a CONNECTION_CLOSE to the server. The total handshakes per second are calculated by dividing the time period by the total number of connections that have successfully established during that time. 3.4. Throughput Fairness Index Connection fairness is able to help us reveal how the throughput is allocated among each connection. A way of doing it is to establish multiple hundreds or thousands of concurrent connections and request the same data block from a single server. Variables that have potential impact on the results are: o the size of the data being requested. o the number of the concurrent connections. The test starts with establishing several hundreds or thousands of concurrent connections and downloading the same data block from the server simultaneously. Banks Expires June 26, 2021 [Page 6] Internet-Draft QUIC-PERF December 2020 The index of fairness is calculated using the complete time of each connection and the size of the data block in [Jain's manner] (https://www.cse.wustl.edu/~jain/atmf/ftp/af_fair.pdf). Be noted that the relationship between fairness and whether the link is saturated is uncertain before any test. Thus it is recommended that both cases are covered in the test. TODO: is it necessary that we also provide tests on latency fairness in the multi-connection case? 3.5. Maximum Number of Idle Connections TODO 4. Things to Note There are a few important things to note when doing performance testing. 4.1. What Data Should be Sent? Since the goal here is to measure the efficiency of the QUIC implementation and not any application protocol, the performance application layer should be as light-weight as possible. To this end, the client and server application layer may use a single preallocated and initialized buffer that it queues to send when any payload needs to be sent out. 4.2. Ramp up Congestion Control or Not? When running CPU limited, and not network limited, performance tests ideally we don't care too much about the congestion control state. That being said, assuming the tests run for enough time, generally congestion control should ramp up very quickly and not be a measureable factor in the measurements that result. 4.3. Disabling Encryption A common topic when talking about QUIC performance is the effect that its encryption has. The draft-banks-quic-disable-encryption draft specifies a way for encryption to be mutually negotiated to be disabled so that an A:B test can be made to measure the "cost of encryption" in QUIC. Banks Expires June 26, 2021 [Page 7] Internet-Draft QUIC-PERF December 2020 5. Security Considerations Since the performance protocol allows for a client to trivially request the server to do a significant amount of work, it's generally advisable not to deploy a server running this protocol on the open internet. One possible mitigation for unauthenticated clients generating an unacceptable amount of work on the server would be to use client certificates to authenticate the client first. 6. IANA Considerations None 7. Normative References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, . [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, . Author's Address Nick Banks Microsoft Corporation Email: nibanks@microsoft.com Banks Expires June 26, 2021 [Page 8]