idnits 2.17.1 draft-white-httpbis-spdy-analysis-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack separate sections for Informative/Normative References. All references will be assumed normative when checking for downward references. ** The abstract seems to contain references ([SPDY-ANALYSIS]), which it shouldn't. Please replace those with straight textual mentions of the documents in question. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (July 6, 2012) is 4305 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Outdated reference: A later version (-04) exists of draft-agl-tls-nextprotoneg-03 == Outdated reference: A later version (-08) exists of draft-ietf-tcpm-initcwnd-03 ** Obsolete normative reference: RFC 793 (Obsoleted by RFC 9293) ** Obsolete normative reference: RFC 2001 (Obsoleted by RFC 2581) ** Obsolete normative reference: RFC 2414 (Obsoleted by RFC 3390) ** Obsolete normative reference: RFC 2616 (Obsoleted by RFC 7230, RFC 7231, RFC 7232, RFC 7233, RFC 7234, RFC 7235) Summary: 6 errors (**), 0 flaws (~~), 3 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group G. White 3 Internet-Draft J-F. Mule 4 Intended status: Informational D. Rice 5 Expires: January 7, 2013 CableLabs 6 July 6, 2012 8 Analysis of SPDY and TCP Initcwnd 9 draft-white-httpbis-spdy-analysis-00 11 Abstract 13 Making the Internet faster is the goal of many product and service 14 companies. Many strategies exist, from data caching to packet 15 switching, performance improvements in Internet browser clients and 16 server software, without forgetting the transport network scaling 17 from the fiber core to the access edges. 19 This report investigates two proposed protocol enhancements aimed at 20 making the web browsing user experience better by optimizing the 21 transport of web objects and thereby reducing the page load time. 22 The two enhancements are SPDY, a replacement of the HyperText 23 Transfer Protocol (HTTP) requiring client and server changes, and a 24 TCP tune-up, an increase in the TCP initial congestion window 25 accomplished by a setting change on the web servers. Both show 26 promise, but there are some caveats, particularly with SPDY. SPDY is 27 a candidate for standardization as HTTP/2.0. 29 In addition to a discussion of the two enhancements, this report 30 provides the results of laboratory testing on SPDY version 2 and the 31 proposed increase in the TCP initial congestion window in a variety 32 of simulated conditions, comparing the page load time when using the 33 proposed enhancement to the page load time for the default case of 34 HTTPS and current initial congestion window settings. 36 The proposed enhancements generate mixed results: web page load times 37 were reduced in some scenarios but increased significantly in others. 38 The performance improvement (or degradation) varied depending on the 39 number of servers, configuration of the initial TCP congestion 40 window, and especially any network packet loss. The following 41 results were obtained across all scenarios comparing SPDY and 42 congestion window enhancements to standard HTTPS. 44 o Average reduction in page load time was 29% 46 o Best improvement was over 78% reduction in page load time 47 o Worst cases showed a negative impact, resulting in a 3.3x increase 48 in page load time 50 These results lead us to the following conclusions: 52 o The SPDY protocol is currently a moving target, and thus it would 53 be challenging to realize a return on investment for general- 54 purpose usage in web servers. 56 o Protocol improvements, standardization in the IETF and wider 57 adoption by the client/server software may warrant a second look 58 at SPDY. 60 o Some applications in controlled environments may gain by 61 leveraging SPDY. SPDY might be a valuable tool where the a single 62 entity provides the servers, the client software, and the web 63 content. 65 o If SPDY were adopted very widely it may have some secondary 66 benefits for network operators through improved infrastructure 67 scalability due to a significant reduction in concurrent TCP 68 sessions, as well as a reduction in Packets Per Second. 70 o The proposed increase in the TCP initial congestion window is 71 straightforward, requires no client modifications, and on its own 72 provides consistent (albeit modest) performance gains. 74 This report is available in a somewhat more detailed form in 75 [SPDY-ANALYSIS]. 77 Status of this Memo 79 This Internet-Draft is submitted in full conformance with the 80 provisions of BCP 78 and BCP 79. 82 Internet-Drafts are working documents of the Internet Engineering 83 Task Force (IETF). Note that other groups may also distribute 84 working documents as Internet-Drafts. The list of current Internet- 85 Drafts is at http://datatracker.ietf.org/drafts/current/. 87 Internet-Drafts are draft documents valid for a maximum of six months 88 and may be updated, replaced, or obsoleted by other documents at any 89 time. It is inappropriate to use Internet-Drafts as reference 90 material or to cite them other than as "work in progress." 92 This Internet-Draft will expire on January 7, 2013. 94 Copyright Notice 95 Copyright (c) 2012 IETF Trust and the persons identified as the 96 document authors. All rights reserved. 98 This document is subject to BCP 78 and the IETF Trust's Legal 99 Provisions Relating to IETF Documents 100 (http://trustee.ietf.org/license-info) in effect on the date of 101 publication of this document. Please review these documents 102 carefully, as they describe your rights and restrictions with respect 103 to this document. Code Components extracted from this document must 104 include Simplified BSD License text as described in Section 4.e of 105 the Trust Legal Provisions and are provided without warranty as 106 described in the Simplified BSD License. 108 Table of Contents 110 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 5 111 2. SPDY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 112 2.1. The SPDY Protocol . . . . . . . . . . . . . . . . . . . . 7 113 2.2. SPDY Server Implementations . . . . . . . . . . . . . . . 9 114 2.3. SPDY Client Implementations . . . . . . . . . . . . . . . 9 115 2.4. SPDY in the Wild . . . . . . . . . . . . . . . . . . . . . 9 116 2.5. SPDY Protocol Development and IETF Standardization . . . . 9 117 3. TCP Initial Congestion Window . . . . . . . . . . . . . . . . 10 118 3.1. Historical Settings for initcwnd . . . . . . . . . . . . . 10 119 4. The Current Web Environment . . . . . . . . . . . . . . . . . 11 120 4.1. Content Environment . . . . . . . . . . . . . . . . . . . 11 121 4.2. Network Environment . . . . . . . . . . . . . . . . . . . 13 122 4.2.1. Round-Trip Time (RTT) . . . . . . . . . . . . . . . . 13 123 4.2.2. Access Network Data Rate . . . . . . . . . . . . . . . 15 124 4.2.3. Packet Loss . . . . . . . . . . . . . . . . . . . . . 15 125 5. Laboratory Testing and Results . . . . . . . . . . . . . . . . 16 126 5.1. Goals . . . . . . . . . . . . . . . . . . . . . . . . . . 16 127 5.2. Laboratory Environment . . . . . . . . . . . . . . . . . . 16 128 5.2.1. Web Server Configuration . . . . . . . . . . . . . . . 16 129 5.2.2. Client Configurations . . . . . . . . . . . . . . . . 17 130 5.3. Test Conditions . . . . . . . . . . . . . . . . . . . . . 17 131 5.3.1. Protocol Options . . . . . . . . . . . . . . . . . . . 17 132 5.3.2. Web Sites . . . . . . . . . . . . . . . . . . . . . . 18 133 5.4. Test Results . . . . . . . . . . . . . . . . . . . . . . . 20 134 5.4.1. CASE 1: SPDY vs. HTTPS . . . . . . . . . . . . . . . 21 135 5.4.2. CASE 2: initcwnd=10 vs. initcwnd=3 . . . . . . . . . 24 136 5.4.3. CASE 3: SPDY+initcwnd=10 vs. HTTPS+initcwnd=3 . . . . 27 137 5.4.4. Results Summary . . . . . . . . . . . . . . . . . . . 29 138 6. Recommendations and Conclusions . . . . . . . . . . . . . . . 30 139 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 31 140 8. Security Considerations . . . . . . . . . . . . . . . . . . . 31 141 9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 31 142 10. References . . . . . . . . . . . . . . . . . . . . . . . . . . 32 143 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 33 145 1. Introduction 147 The current method of transporting web objects from a server to a 148 client utilizes Hypertext Transfer Protocol version 1.1 (HTTP/1.1), 149 running atop the Transport Control Protocol (TCP) [RFC0793]. 150 HTTP/1.1 was published as [RFC2616] in 1999, and has since become the 151 most widely used application-layer protocol on the Internet. TCP 152 pre-dates HTTP/1.1 by about 18 years, and even though TCP has evolved 153 over time, it is fundamentally unchanged, and runs atop IPv4 and 154 IPv6. 156 Since the advent of the World Wide Web 15+ years ago, access network 157 bandwidths as well as server and client CPU horsepower have increased 158 by a factor of approximately 1.5x year-over-year, yet the apparent 159 speed of the web (from the web browsing user's perspective) has grown 160 much more slowly. In part, this can be explained by a concomitant 161 increase in web page complexity, as measured by any number of 162 attributes including total page size, number of linked resources, 163 amount of server-side code, and lines of JavaScript. However, there 164 is a view that the underlying protocols used to transport web 165 resources are becoming increasingly out-of-date with the network and 166 computing resources that are in use today, and that significant 167 improvements in performance (generally page load time) can be 168 achieved by revisiting these fundamental technologies. 170 Bolstering this view is the fact that while network bandwidths have 171 been improving rapidly, network latencies have not, and there is 172 little hope of achieving significant reductions in network latency, 173 as it is dominated by propagation delay. The result is that a 174 fundamental network parameter, the Bandwidth-Delay Product (BDP), is 175 much larger today than in the early days of the Internet. It is 176 becoming clear that HTTP/1.1 and TCP are not particularly optimized 177 for networks with high bandwidth-delay product. 179 Furthermore, in order to maximize performance using the existing 180 tools of HTTP/1.1 and TCP, web architects (browser, server and site 181 developers) have employed work-arounds that have some negative 182 implications. The most significant implications result from the use 183 of multiple simultaneous TCP connections. Modern browsers will open 184 up to six simultaneous TCP connections to each server from which they 185 need to retrieve resources, and optimized websites will compound this 186 by spreading content over multiple servers, a practice known as 187 "domain sharding". The result is that a single browser may have 20 188 or more simultaneous TCP connections to the hosts of a particular 189 site while it is downloading a single web page. The parallel nature 190 of this approach is intended to reduce the page download time 191 (compared to the alternative of a single, non-pipelined TCP 192 connection), and in many cases it succeeds. However, a web browser 193 with 20 or more simultaneous TCP sessions can drown out other 194 applications that use a small number of TCP sessions (e.g., one). 195 This is due to the fact that the TCP congestion avoidance algorithm 196 provides approximately fair sharing of the bandwidth on the 197 bottleneck link on a per-TCP-session basis. Additionally, some 198 browsers do not reuse TCP connections, so each web resource is 199 retrieved using a separate TCP connection. The result is that the 200 vast majority of sessions never reach the congestion avoidance phase, 201 and network resources can therefore either be underutilized or 202 excessively congested. The network overhead for establishing and 203 tearing down all of the TCP connections can also be a concern. 205 A number of efforts have sprung up to develop ways to improve page 206 download time and consequently improve the web browsing user 207 experience. This paper discusses two proposed enhancements, both 208 originally proposed by Google as part of their "Let's Make the Web 209 Faster" project [Faster]. The first is a replacement for HTTP 1.1, 210 called SPDY [SPDY], which aims to make more efficient use of the 211 network in common web browsing scenarios. The second is an 212 incremental enhancement to the TCP protocol in the form of an 213 increase in the server's initial congestion window (initcwnd). 215 In addition to a discussion of the two proposed enhancements, this 216 paper presents the results of laboratory testing conducted by 217 CableLabs on both technologies independently, and on the combination 218 of the two technologies. 220 2. SPDY 222 SPDY [SPDY] is an experimental protocol developed by Mike Belshe and 223 Roberto Peon of Google in late 2009 to replace HTTP/1.1 communication 224 for transporting web content between a client and a server. It is 225 deployed in production on many Google servers but requires a 226 compatible browser such as Google Chrome. The stated goals of SPDY 227 are: 229 o 50% reduction in page load time 231 o Minimize deployment complexity 233 o Avoid changes to content by web authors 235 o Solve this collaboratively via open-source software 237 At the time of this study, the current implemented version of SPDY is 238 version 2 (SPDY/2). Version 3 is currently being developed. 240 2.1. The SPDY Protocol 242 SPDY/2 retains the HTTP/1.1 metadata, but replaces the transport 243 aspects of HTTP with a more streamlined approach. SPDY has three 244 basic features that are used to accelerate the loading of a page: 246 o Multiplexed streams - The fundamental enhancement of SPDY is that 247 multiple resources can be retrieved via a single TCP connection. 248 The expectation (and current implementation in Chrome) is for the 249 client to open a single TCP connection to each server, and to 250 request all resources of the server over that single connection. 251 The use of a single TCP connection in this way allows the 252 congestion avoidance algorithm in TCP to more effectively manage 253 data flow across the network. Also, the client can include 254 multiple requests in a single message, thereby reducing overhead 255 and the number of round-trip times necessary to begin file 256 transfer for requests beyond the first. While this is similar to 257 HTTP pipelining [HTTP_Pipelining], the difference introduced by 258 SPDY is that the server can transfer all of the resources in 259 parallel (multiplexed) without the "head-of-line blocking" problem 260 that can occur with HTTP pipelining. 262 o Request prioritization - While SPDY may serve to decrease total 263 page load time, one side-effect of stream multiplexing is that a 264 critical page resource might be delivered more slowly due to the 265 concurrent delivery of a less critical resource. To prevent this, 266 SPDY provides a way for the client to indicate relative priorities 267 for the requested resources. For example, if a JavaScript library 268 is required in order to generate URLs for additional page 269 resources, the client can request that the library be delivered 270 with higher priority so that it isn't holding up those subsequent 271 requests. 273 o HTTP header compression - SPDY mandates that HTTP headers be 274 compressed to reduce the number of bytes transferred. HTTP Header 275 compression has been an Internet standard but it is not widely 276 used. Google SPDY makes it mandatory. This might be a fairly 277 small gain on the response headers, but it can provide a 278 significant benefit on the requests (which in many cases are 279 entirely headers). For example, each request from a particular 280 client to a particular server includes an identical user-agent 281 string (which can be in excess of 100 bytes), and in some cases 282 the same cookie is sent in each request. Both Chromium and 283 Firefox developers have reported approximately 90% header 284 compression using the proposed zlib compression. This could allow 285 a lot more requests to be packed into each request packet. 287 SPDY also has two advanced features that can be used in certain cases 288 to further accelerate the web user experience: 290 o Server Push - SPDY allows the server to create a stream to the 291 client, and via this stream send web resources that were not 292 explicitly requested by the client. The expectation is that the 293 client will cache the pushed resource, and then upon needing it, 294 will retrieve it from local cache. This function could 295 potentially be used by the server to push objects (of which the 296 client isn't yet aware) that are necessary for rendering the 297 current page. Additionally, this function could be used to push 298 resources for related pages that the user may be likely to 299 request. The heuristics used by the server to decide when/if to 300 push objects and what those objects are is left to the server 301 implementer. This feature is somewhat controversial, but the 302 authors defend it by pointing out that it is better than the 303 practice of "in-lining" resources into the html page, since it 304 allows the pushed resources to be cached for multiple future 305 usages. 307 o Server Hint - SPDY defines a new header that the server can use to 308 suggest additional resources that the client should request. This 309 is a less forceful approach than the Server Push, and allows the 310 client to be involved in the decision whether or not a particular 311 resource is delivered. The client might, for example, examine its 312 own cache and only request resources that are not resident in 313 local cache. On the other hand, Server Hint theoretically 314 requires one additional round trip that would be eliminated by 315 Server Push. 317 While it isn't fundamentally required for the protocol to function, 318 in practice SPDY/2 runs only over an encrypted (TLS) connection. The 319 rationale for this is three-fold: 321 o TLS involves a client-server handshake to negotiate cipher-suite. 322 The authors of SPDY extend this handshake to negotiate whether 323 SPDY can be used. This Next Protocol Negotiation (NPN) 324 [I-D.agl-tls-nextprotoneg] allows the use of the https:// URI, 325 rather than a new spdy:// URI, and as a result a single html page 326 can work for clients that support SPDY and clients that don't. 328 o TLS passes through firewalls and bypasses intermediaries. Many 329 HTTP requests today are processed (and modified) by transparent 330 proxies without the knowledge of the end-user. It would create a 331 tremendous barrier to adoption of SPDY if it were necessary for 332 intermediaries to be upgraded to handle the new protocol. 334 o Encryption is good. The authors of SPDY state a philosophical 335 belief that all HTTP traffic should be encrypted. They cite the 336 relative ease by which traffic (particularly Wi-Fi traffic) can be 337 snooped. 339 2.2. SPDY Server Implementations 341 There are a number of SPDY server implementations that are in various 342 stages of development as open source projects. The Chromium project 343 maintains a list of the implementations available [SPDY]. At the 344 time this study was initiated, many of these implementations 345 supported a subset of SPDY functionality and/or supported SPDY 346 version 1. At the time this study began, only two implementations 347 appeared to support SPDY/2 in a reliable way. The first is the 348 Chromium "FLIP Server" (FLIP was an early internal code-name for SPDY 349 within Google). This server is built off of the immense Chromium 350 source tree. The second is a Javascript implementation called "node- 351 spdy" which is built on the node.js server framework. There is also 352 an Apache module for SPDY, but at the time this study was initiated 353 it was incomplete. 355 2.3. SPDY Client Implementations 357 On the client side, both the Chrome browser and the Mozilla Firefox 358 browser support SPDY/2. Chrome also includes a built-in diagnostics 359 function that allows the user to examine the active SPDY sessions 360 . Finally, the Silk browser in the 361 Amazon Kindle Fire tablet computer purportedly utilizes SPDY/2 for 362 its connection to the Amazon EC2 cloud when performing web 363 acceleration. 365 2.4. SPDY in the Wild 367 In terms of live web content, the most significant, publicly 368 available sites that serve content via SPDY are run by Google. Many 369 of the Google properties have SPDY/2 enabled, including Gmail, Google 370 Docs, Picasa, Google+, and Google Encrypted Search. All of these 371 sites utilize only the basic SPDY features; there are no known live 372 instances of Server Push or Server Hint. In addition, the Twitter 373 and Wordpress websites utilize SPDY, and the web acceleration 374 companies Cotendo (acquired by Akamai in Dec. 2011) and Strangeloop 375 indicate that they have deployed SPDY in some capacity. 377 2.5. SPDY Protocol Development and IETF Standardization 379 Google has actively solicited input on enhancements to the SPDY/2 380 protocol. Up until very recently, development and discussion of 381 SPDY/3 has taken place on an open, Google-managed forum. However, 382 the SPDY/3 draft was submitted to the IETF HTTPbis working group on 383 February 23, 2012, for comments [I-D.mbelshe-httpbis-spdy]. 385 3. TCP Initial Congestion Window 387 The TCP initial congestion window (initcwnd) is used at the start of 388 a TCP connection. In the context of an HTTP session, the server's 389 initcwnd setting controls how many data packets will be sent in the 390 first burst of data from the server. It is a standard protocol 391 parameter that can be changed on Linux servers via a simple command 392 line. 394 Absent packet loss or receiver window limits, the TCP slow start 395 operation looks like: 397 Table 1. TCP Slow Start Operation 399 +--------------+---------------------------+-------------------------+ 400 | Round-Trip # | Client | Server | 401 +--------------+---------------------------+-------------------------+ 402 | 1 | TCP SYN | TCP SYN/ACK | 403 +--------------+---------------------------+-------------------------+ 404 | 2 | TCP ACK and HTTP GET | initcwnd data packets | 405 +--------------+---------------------------+-------------------------+ 406 | 3 | TCP ACKs | 2*initcwnd data packets | 407 +--------------+---------------------------+-------------------------+ 408 | 4 | TCP ACKs | 4*initcwnd data packets | 409 +--------------+---------------------------+-------------------------+ 410 | | | and so on.... | 411 +--------------+---------------------------+-------------------------+ 413 3.1. Historical Settings for initcwnd 415 A larger value for initcwnd will clearly result in fewer Round-Trip 416 Times (RTTs) to deliver a file. However, the downside to an 417 excessively large initcwnd is that there is an increased risk of 418 overflowing a router buffer on an intermediate hop, resulting in an 419 increase in latency from packet loss and retransmissions. A value of 420 initcwnd that is greater than the Bandwidth-Delay Product (BDP) of 421 the network path between the server and client has an increased 422 likelihood of causing packet loss that may lead to poor performance. 423 As network bandwidths have increased over time, the BDP has 424 increased, and as a result the defined value for initcwnd has been 425 adjusted. The early specifications for TCP ([RFC1122], [RFC2001]) 426 required that a TCP implementation set initcwnd to 1 packet. 427 Starting in 2002 ([RFC2414], [RFC3390]), the initcwnd value was 428 raised to 4000 bytes (effectively 3 packets in most networks). 430 Google has submitted an IETF draft [I-D.ietf-tcpm-initcwnd] proposing 431 that initcwnd now be increased to at least 10 packets, based on a 432 series of tests performed using production Google servers 434 [INITCWND-PAPER]. In Google's tests, a value of 10-16 packets 435 resulted in the minimum average latency for delivery of web 436 resources. 438 In November 2011, CDN Planet [CDNPLANET] performed an assessment of 439 CDNs to see what value is currently used for initcwnd. Many CDNs 440 have already increased their initcwnd beyond 3 packets, some as high 441 as 10-16 packets. 443 This increase from a value of 3 to a value of 10 will result in the 444 elimination of up to 2 RTTs for each file transfer. The maximum gain 445 will be seen for files that take 9 packets to deliver (i.e., files 446 around 11 kB in size). Files of that size would take 4 RTTs to 447 deliver using the default initcwnd, but can be delivered in 2 RTTs 448 with the proposed increase, a 50% improvement. Files smaller than 449 about 3 kB will not experience any acceleration. 451 4. The Current Web Environment 453 4.1. Content Environment 455 The construction of web page content plays a very important role in 456 determining the page load time. In order to validate the two 457 proposed enhancements, we will create a test bed that consists of web 458 servers, web clients and web content. We performed a survey of some 459 popular (and resource heavy) websites in order to understand the 460 content environment. The sites used in the survey were selected as 461 examples of sites that would stand to benefit from an acceleration 462 technology such as SPDY. Many of these sites are composed of a 463 fairly large number of resources, the content changes frequently 464 (reducing local cache acceleration), and they would presumably be 465 interested in page load performance. 467 The results shown in Table 2 indicate, for each site's home page, the 468 number of servers from which content was drawn, the number of 469 resources requested, the total transferred size of all page 470 resources, and the page load time. The reported page load time is 471 the total time to download all of the resources for the page. It is 472 possible, and even likely, that one or more of these pages are 473 largely complete (perhaps only missing some non-critical resources) 474 in significantly less time than is reported here. 476 Additionally, for each webpage we evaluated the percentage of total 477 resources that were requested from the top N servers for the page, 478 for values of N between 1 and 5. For example, Engadget receives 34% 479 of the web objects from the most used server, 82% of the objects from 480 3 most used servers and 87% of them come from the 5 servers with the 481 highest count. These servers may include the host servers for the 482 website, CDN servers for images and media content, advertising 483 servers for ad content, and analytics collection, among other types 484 of resources. 486 Table 2. Website Survey 488 +-------------+-------+-----+------+-------+------------------------+ 489 | | | |total | | % of GETs from | 490 | | | |page | total | top N servers | 491 | | | |size | time +----+----+----+----+----+ 492 |Page |servers|GETs |(KB) | (s) | N=1| N=2| N=3| N=4| N=5| 493 +-------------+-------+-----+------+-------+----+----+----+----+----+ 494 |Engadget | 26 | 278 | 1500 | 23.0 | 34%| 67%| 82%| 85%| 87%| 495 +-------------+-------+-----+------+-------+----+----+----+----+----+ 496 |NY Times | 27 | 148 | 1500 | 13.6 | 33%| 49%| 59%| 67%| 73%| 497 +-------------+-------+-----+------+-------+----+----+----+----+----+ 498 |Gizmodo | 25 | 116 | 3900 | 13.2 | 34%| 60%| 69%| 76%| 78%| 499 +-------------+-------+-----+------+-------+----+----+----+----+----+ 500 |CNN | 22 | 158 | 1180 | 12.6 | 59%| 70%| 75%| 80%| 83%| 501 +-------------+-------+-----+------+-------+----+----+----+----+----+ 502 |comcast.net | 12 | 55 | 606 | 12.0 | 51%| 67%| 75%| 80%| 84%| 503 +-------------+-------+-----+------+-------+----+----+----+----+----+ 504 |ESPN | 26 | 144 | 5400 | 11.6 | 40%| 51%| 60%| 67%| 70%| 505 +-------------+-------+-----+------+-------+----+----+----+----+----+ 506 |Amazon | 15 | 139 | 1100 | 11.0 | 34%| 58%| 81%| 86%| 90%| 507 +-------------+-------+-----+------+-------+----+----+----+----+----+ 508 |RoadRunner | 30 | 128 | 85 | 9.50 | 27%| 48%| 61%| 65%| 68%| 509 +-------------+-------+-----+------+-------+----+----+----+----+----+ 510 |Wired | 32 | 130 | 1200 | 8.90 | 48%| 56%| 61%| 65%| 68%| 511 +-------------+-------+-----+------+-------+----+----+----+----+----+ 512 |eBay | 14 | 53 | 520 | 8.60 | 23%| 40%| 55%| 68%| 75%| 513 +-------------+-------+-----+------+-------+----+----+----+----+----+ 514 |LinkedIn | 9 | 75 | 457 | 7.17 | 48%| 80%| 88%| 92%| 95%| 515 +-------------+-------+-----+------+-------+----+----+----+----+----+ 516 |Hulu | 12 | 193 | 2000 | 4.96 | 60%| 79%| 87%| 95%| 96%| 517 +-------------+-------+-----+------+-------+----+----+----+----+----+ 518 |Yahoo | 6 | 51 | 423 | 4.60 | 88%| 92%| 94%| 96%| 98%| 519 +-------------+-------+-----+------+-------+----+----+----+----+----+ 520 |YouTube | 6 | 37 | 2400 | 4.16 | 41%| 68%| 81%| 92%| 97%| 521 +-------------+-------+-----+------+-------+----+----+----+----+----+ 522 |Google | 3 | 20 | 73 | 2.45 | 45%| 90%|100%|100%|100%| 523 | shopping | | | | | | | | | | 524 +-------------+-------+-----+------+-------+----+----+----+----+----+ 525 |Average | 18 | 115 | 1490 | 9.82 | 44%| 65%| 75%| 81%| 84%| 526 +-------------+-------+-----+------+-------+----+----+----+----+----+ 528 4.2. Network Environment 530 The network environment between the servers and the client plays an 531 equally large role in determining page load time. The most important 532 parameters that drive performance are the round-trip time, the 533 bottleneck link bandwidth, and the packet loss rate. 535 4.2.1. Round-Trip Time (RTT) 537 In the absence of congestion, typical Internet round-trip times can 538 range from as low as 10 milliseconds when accessing content served in 539 close proximity, to as high as 500 milliseconds or more when 540 accessing servers across continents. For wireline broadband 541 customers in the U.S., the most popular websites can generally be 542 reached with RTTs ranging from 15 ms to 150 ms. For the websites 543 included in the survey in Section 4.1, a sample TCP RTT to each of 544 the top-five servers (in terms of number of resources requested) from 545 CableLabs headquarters in Louisville, Colorado, is shown (in 546 milliseconds) in Table 3 along with a summary of the minimum, mean, 547 and max RTT of those top-five servers. The RTT was calculated by 548 doing a TCP "ACK RTT" analysis of a packet capture of each page load 549 using the Wireshark tool. The ISP connection used during this 550 testing consisted of two load-balanced 100 Mbps duplex links with a 551 mean RTT of less than 2 ms. 553 Table 3. Round-Trip Times to Website Servers in Milliseconds 555 +-----------+------+------+------+------+------+-----+-------+-----+ 556 |Page |server|server|server|server|server| | | | 557 | | 1 | 2 | 3 | 4 | 5 | min |average| max | 558 +-----------+------+------+------+------+------+-----+-------+-----+ 559 |Engadget | 63* | 62* | 64* | 62* | 38 | 38 | 57.8 | 64 | 560 +-----------+------+------+------+------+------+-----+-------+-----+ 561 |NY Times | 64* | 64* | 62* | 64* | 64* | 62 | 63.6 | 64 | 562 +-----------+------+------+------+------+------+-----+-------+-----+ 563 |Gizmodo | 37* | 39* | 134 | 25 | 22 | 22 | 51.4 | 134 | 564 +-----------+------+------+------+------+------+-----+-------+-----+ 565 |CNN | 46* | 53 | 69* | 32 | 45 | 32 | 49.0 | 69 | 566 +-----------+------+------+------+------+------+-----+-------+-----+ 567 |Comcast.net| 65* | 69* | 94 | 28 | 71 | 28 | 65.4 | 94 | 568 +-----------+------+------+------+------+------+-----+-------+-----+ 569 |ESPN | 64* | 33 | 65* | 64* | 64* | 33 | 58.0 | 65 | 570 +-----------+------+------+------+------+------+-----+-------+-----+ 571 |Amazon | 35 | 72* | 16* | 98 | 33 | 16 | 50.8 | 98 | 572 +-----------+------+------+------+------+------+-----+-------+-----+ 573 |RoadRunner | 56 | 58 | 29* | 52* | 61* | 29 | 51.2 | 61 | 574 +-----------+------+------+------+------+------+-----+-------+-----+ 575 |Wired | 70* | 114 | 66* | 24* | 85 | 24 | 71.8 | 114 | 576 +-----------+------+------+------+------+------+-----+-------+-----+ 577 |eBay | 62* | 75 | 64* | 62* | 62 | 62 | 65.0 | 75 | 578 +-----------+------+------+------+------+------+-----+-------+-----+ 579 |LinkedIn | 50* | 72* | 78 | 54 | 63* | 50 | 63.4 | 78 | 580 +-----------+------+------+------+------+------+-----+-------+-----+ 581 |Hulu | 65* | 67* | 66* | 64* | 22 | 22 | 56.8 | 67 | 582 +-----------+------+------+------+------+------+-----+-------+-----+ 583 |Yahoo | 28 | 27 | 64 | 29 | 61 | 27 | 41.8 | 64 | 584 +-----------+------+------+------+------+------+-----+-------+-----+ 585 |YouTube | 23 | 25 | 23 | 22 | 22 | 22 | 23.0 | 25 | 586 +-----------+------+------+------+------+------+-----+-------+-----+ 587 |Google | 34 | 42 | 43 | 22 | - | 22 | 35.3 | 43 | 588 | shopping | | | | | | | | | 589 +-----------+------+------+------+------+------+-----+-------+-----+ 591 * denotes a host run by a CDN provider. 593 When network links are congested, a phenomenon referred to as 594 "bufferbloat" can result in an increase in RTT on the order of 595 hundreds of milliseconds. Bufferbloat arises due to the fact that 596 many network elements support much more buffering than is needed to 597 maintain full link utilization. These oversized buffers will be kept 598 full by the TCP (particularly when there are multiple simultaneous 599 sessions), thereby causing an increase in latency. In recent updates 600 to the DOCSIS 3.0 cable modem specification, a new feature has been 601 added to mitigate this effect by proper sizing of the cable modem's 602 upstream buffer. At this time, 14 cable modem models from 10 603 manufacturers have been certified as supporting this feature. For 604 more detail on bufferbloat and the DOCSIS Buffer Control feature, 605 see: [BUFFERCONTROL]. 607 4.2.2. Access Network Data Rate 609 Residential access network data rates have steadily risen over time, 610 with a 10 Mbps downstream rate and 1 Mbps upstream rate being fairly 611 common in North America, and 20 Mbps x 2 Mbps rates becoming 612 increasingly available. 614 As a result, for purposes of our investigation of the two proposed 615 technologies, we will utilize the 10x1 and 20x2 data rate 616 configurations. 618 Some access networks, in particular cable modem access networks, are 619 able to provide higher burst data rates that go well beyond the 620 sustained traffic rate for the connection. In this report we will 621 not attempt to study the impact that these "Powerboost" type features 622 have on the two proposed technologies. In addition, many operators 623 offer higher speed tiers than those mentioned above, with speeds in 624 Europe up to 360 Mbps on the very high end, but they currently have 625 fairly low take rates. Finally, some access networks may have lower 626 speed limitations placed on the services such as public and outdoor 627 Wi-Fi networks. 629 4.2.3. Packet Loss 631 It is important to note that packet loss due to router (or switch) 632 buffer overflow is expected behavior in networks, particularly when 633 TCP is utilized. In fact, packet loss due to buffer overflow is 634 fundamental to the congestion avoidance algorithm in TCP; it is the 635 only signal that the TCP has (absent the Explicit Congestion 636 Notification field in IP packets) that it has saturated the link and 637 needs to reduce its transmission rate. As a result, it is not a 638 concern in this testing. The concern here is random packet loss due 639 to noise, interference or network faults that will effectively send 640 TCP an erroneous signal to reduce its transmission rate. 642 In wired networks, packet loss due to noise is fairly uncommon, for 643 example with packet loss rates of approximately 10^-5 being typical 644 for DOCSIS cable modem networks. In wireless networks, packet loss 645 can vary dramatically as a result of interference and fluctuations in 646 carrier-to-noise ratio. 648 5. Laboratory Testing and Results 650 5.1. Goals 652 The goals of the testing are to investigate the page load performance 653 impacts of the two proposed technologies. In particular, we examine 654 Google SPDY with an eye toward evaluating SPDY's ability to achieve 655 the stated performance goal and to understand what aspects of SPDY 656 provide acceleration and in which scenarios. Testing is limited to 657 the SPDY "basic" features; no testing of server-push or server-hint 658 is included in this study, primarily due to the fact that the 659 performance enhancement gained by using them would be heavily 660 dependent on how they are configured and utilized by the website 661 administrator. Some benchmarks of server-push and server-hint have 662 been published by external sources; see [GOOGLE-BENCHMARKS] . The 663 results have not shown a statistically significant gain. 665 As stated previously, the TCP initcwnd increase is tested both 666 independently (using HTTPS) and in conjunction with SPDY. 668 5.2. Laboratory Environment 670 The entire test network was virtualized on a dedicated VMWare ESX 671 Virtual Server. The test network consisted of 7 virtual hosts, each 672 having two network interfaces. A single network interface on each 673 host was connected to a virtual LAN within the ESX environment. The 674 second network interface on each host was connected to the corporate 675 network. During the experiments, the CPU load on the virtualization 676 server was checked to ensure that it was not impacting the results. 678 5.2.1. Web Server Configuration 680 Four of the hosts were configured identically as web servers, with 681 the following software: 683 o OS: Ubuntu 11.04 685 o Web Server: "Express-SPDY" written in JavaScript for node.js 687 The "Express-SPDY" server is the melding of a compact and efficient 688 open-source HTTPS server (known as "Express") with an open-source 689 SPDY/2 implementation called "node-spdy". The implementation is 690 written in JavaScript using the node.js server environment. This 691 implementation was chosen over the Chromium package published by 692 Google due to the fact that it was lightweight and could be deployed 693 relatively easily. Unfortunately, the version of Express-SPDY 694 available at the time of this testing was found to not be stable, and 695 required some work to debug and fix prior to beginning any testing. 697 Since our testing commenced, the author of node-spdy rewrote a large 698 part of the package. 700 5.2.2. Client Configurations 702 The remaining three hosts were configured as client machines. The 703 primary client ran Windows 7, and the other two clients (used for 704 spot testing) ran Windows XP and Ubuntu 11.04. All clients used the 705 Chrome 16 browser, with the SPDY Benchmarking plugin, and were 706 installed with Wireshark to capture network traffic. The Dummynet 707 [DUMMYNET] network simulator was used to simulate the various network 708 conditions. 710 For each test condition, Dummynet was configured with the desired 711 upstream and downstream rates, with the desired RTT split equally 712 between the two links. In cases where packet loss was being tested, 713 the same packet loss rate was configured on both links. Both links 714 were left with the default buffering configuration of 50 packets. 716 Note that two of the hosts (Win7 and Ubuntu) support the TCP Window 717 Scaling Option by default, while the third (WinXP) does not. 719 5.3. Test Conditions 721 5.3.1. Protocol Options 723 As previously stated, the testing is intended to compare the SPDY 724 protocol to the HTTPS protocol, and to investigate the use of the 725 increased TCP Initial Congestion Window (initcwnd) (both with HTTPS 726 and in conjunction with SPDY). As a result, we measured the time it 727 took to load a web page using four different protocol options as 728 shown in Table 4. One option defines the baseline scenario or "Base 729 Case": with HTTPS for transport and the default initcwnd setting of 730 3. The results for the three cases involving a proposed enhancement 731 are then compared to the "Base Case". 733 Table 4. Matrix of Protocol Options 735 +-------------+---------+ 736 | HTTPS | SPDY | 737 +------------------+-------------+---------+ 738 | initcwnd = 3 | Base Case | CASE 1 | 739 +------------------+-------------+---------+ 740 | initcwnd = 10 | CASE 2 | CASE 3 | 741 +------------------+-------------+---------+ 743 5.3.2. Web Sites 745 Nine different "websites" were used to evaluate the impact that the 746 content and server configuration would have on the test results. The 747 nine websites were formed by the 3x3 matrix of three different server 748 configurations and three different web pages as follows. 750 5.3.2.1. Server Configurations 752 The server configuration parameter determined how all of the web 753 resources for a page were served. 755 o Server Case 1, all resources were served by a single server. 757 o Server Case 2, the resources were divided equally across two 758 servers. 760 o Server Case 3, the resources were divided equally across four 761 servers. 763 5.3.2.1.1. Web Pages 765 Three web pages were used in testing. These pages were chosen to 766 span a range of possible page configurations. In particular, Page B 767 was modeled based on the data collected in the website survey 768 (Section 4.1), while Pages A and C represent more extreme situations. 769 Page A is the smallest of the three pages, and is populated by a 770 number of small image files. Page C is the largest, and is populated 771 by a small number of very large image files. It could be argued that 772 a page like Page C would be fairly rare on the Internet compared to 773 pages like Page A or Page B. 775 Page A consisted of 102 total resources, with a total page size of 776 369 KB. It was composed of the following resources: 778 o 1 HTML file: 12.8 KB 780 o 1 JavaScript Library (jquery.js): 94.5 KB 782 o 100 JPEG images: min/mean/max/stddev size = 1.3/2.6/5.5/1.2 KB 784 Page B consisted of 101 total resources, with a total page size of 785 1.4 MB. It was composed of the following resources: 787 o 1 HTML file: 8.2 KB 789 o 100 JPEG/PNG/GIF images: 791 * min/mean/max/stddev size = 46B/14KB/103KB/24KB 793 * Approximately log-uniform size distribution 795 Page C consisted of 11 total resources, with a total page size of 3.0 796 MB. It was composed of the following resources: 798 o 1 HTML file: 0.8 KB 800 o 10 JPEG images: min/mean/max/stddev size = 298/302/310/4 KB 802 5.3.2.2. Channel Conditions 804 Eight different channel conditions were used in the testing. These 805 eight cases were formed from the 2x2x2 matrix of the following 806 parameters: 808 Configured Data Rate: 810 10 Mbps downstream, 1 Mbps upstream 812 20 Mbps downstream, 2 Mbps upstream 814 Round Trip Time: 816 20 ms 818 100 ms 820 Packet Loss Rate: 822 0% 824 1% 826 The two Configured Data Rates were chosen to match two commonly used 827 residential high-speed data service configurations. The Round Trip 828 Times correspond to a fairly short RTT that models a user accessing a 829 site that is located fairly close to them (or is accelerated by a 830 CDN), as well as a longer RTT which models a U.S. coast-to-coast page 831 load or a case where there is increased upstream latency due to 832 bufferbloat. Performance in higher RTT cases, such as an overseas 833 connection, is not considered in this study. The two packet loss 834 rates correspond to an idealistic case that might be approached by an 835 end-to-end wired network connection and a case with packet loss that 836 might be more typical of a network connection that includes a Wi-Fi 837 link. 839 5.3.2.3. Test Execution 841 The test execution will be performed in the following fashion: 843 o Run all 288 test conditions on Win 7 845 o Run spot tests on WinXP and Ubuntu 847 o For each test condition, perform 20 page loads (clearing cache and 848 resetting connections before each load) and calculate median load 849 time. 851 5.4. Test Results 853 The result of the spot tests on WinXP and Ubuntu were largely 854 consistent with the results for the Win7 testing. However, there was 855 a subset of the test cases with WinXP where SPDY showed degraded 856 performance compared to the other OSes. This occurred with the cases 857 that used a single server, 100 ms RTT, and 0% packet loss. This is 858 most likely a result of the lack of TCP Window Scaling in the WinXP 859 client, which limits the data rate of each TCP connection to 5 Mbps 860 in these conditions. Since SPDY is using a single TCP connection 861 (compared to six for HTTP), SPDY will see a performance hit due to 862 its inability to use the entire 10 Mbps or 20 Mbps pipe. 864 The raw data for all test cases (including the WinXP and Ubuntu test 865 cases) are provided in [SPDY-ANALYSIS]. The remainder of this report 866 will focus on the Win7 test results. 868 As described in Section 5.3.1, the 2x2 matrix of protocol options 869 will be presented as three different "cases", where each case 870 represents the gain achieved by using one of the three proposed 871 protocol enhancement options compared to the base case. This section 872 is broken into three subsections that correspond to the three cases. 874 For each case, the results comprise a five-dimensional matrix of test 875 conditions. In order to present the results in a compact way, each 876 subsection below will first examine the impact that website 877 configuration has on the achieved gain, then second will examine the 878 impact of the channel conditions. 880 The results provide a comparison between the median web page download 881 time achieved using the proposed protocol enhancement option and that 882 which is achieved in the base case. 884 5.4.1. CASE 1: SPDY vs. HTTPS 886 Case 1 compares the median web page download time achieved using 887 SPDY/2 to the median time achieved using HTTPS. The comparison is 888 made for each website configuration and channel condition 889 combination. The comparison is reported as a "gain", which is 890 calculated as the ratio of median page load time using HTTPS to the 891 median page load time using SPDY. As a result, gain values greater 892 than 1 indicate that SPDY/2 provides an advantage over HTTPS. As an 893 additional point of reference, a gain of 2 corresponds to a 50% 894 reduction in page load time, the goal to which SPDY aspires. 896 As stated above, the impact that the website configuration has on the 897 achieved gain will be examined first. Table 5 shows the 3x3 matrix 898 of website configurations. For each website, the maximum, minimum 899 and mean gain values (across all channel conditions) are provided, in 900 the manner shown below. 902 +-------------+ 903 | max | 904 | mean | 905 | min | 906 +-------------+ 908 Additionally, row-wise, column-wise, and overall statistics are 909 calculated and shown around the periphery of the 3x3 matrix. 911 Table 5. Website Impact on SPDY Gain 913 | 1 server | 2 servers | 4 servers | average | 914 ---------+-------------+-------------+-------------+-------------+ 915 | 4.1 | 1.8 | 1.8 | 4.1 | 916 Page A | 2.4 | 1.3 | 1.3 | 1.7 | 917 | 1.6 | 1.1 | 1.1 | 1.1 | 918 ---------+-------------+-------------+-------------+-------------+ 919 | 3.3 | 1.4 | 1.4 | 3.3 | 920 Page B | 1.7 | 1.0 | 1.0 | 1.3 | 921 | 0.8 | 0.6 | 0.7 | 0.6 | 922 ---------+-------------+-------------+-------------+-------------+ 923 | 1.0 | 1.0 | 1.1 | 1.1 | 924 Page C | 0.5 | 0.7 | 0.9 | 0.7 | 925 | 0.3 | 0.4 | 0.5 | 0.3 | 926 ---------+-------------+-------------+-------------+-------------+ 927 | 4.1 | 1.8 | 1.8 | 4.1 | 928 average | 1.6 | 1.0 | 1.1 | 1.2 | 929 | 0.3 | 0.4 | 0.5 | 0.3 | 930 ---------+-------------+-------------+-------------+-------------+ 931 These results show that SPDY worked well for Page A (many smaller 932 images), showing gains across all test conditions and an average gain 933 of 1.7. For Page B (many images more typical size), the results were 934 a bit more hit-or-miss, with SPDY not always resulting in improved 935 performance. Nonetheless, on average a gain of 1.3 was achieved. 936 Page C (small number of large images), on the other hand, resulted in 937 almost universally worse performance with SPDY than with HTTPS. In 938 the worst case, SPDY resulted in a 3.3x increase in page load time 939 (gain of 0.3). On average SPDY took 1.4x longer (gain of 0.7) to 940 download Page C than traditional HTTPS. 942 In general, the SPDY impact (positive or negative) diminished as more 943 servers were utilized. This is the result of the increased 944 parallelism and decreased number of objects requested per SPDY 945 session, which if taken to the extreme would result in a pattern of 946 requests that is similar to the HTTPS case. 948 The results shown in Table 6 examine the impact that channel 949 conditions have on SPDY performance (relative to HTTPS). Since the 950 channel conditions were formed from a 2x2x2 matrix of data-rate, RTT, 951 and packet-loss rate (PLR), the results are depicted as two 2x2 952 matrices of data-rate and RTT, one corresponding to the 0% PLR test 953 cases, and the other corresponding to the 1% PLR test cases. For 954 each channel condition, the results for all nine websites are 955 summarized via the maximum, mean, and minimum gain achieved. Similar 956 to the row and column statistics provided in the website impact 957 analysis in Table 5, summary statistics for all three dimensions are 958 provided around the periphery of the 2x2x2 cube (i.e., in the right- 959 most column, the last row, and the third table). 961 Table 6a. Channel Impact on SPDY Gain (0% PLR) 963 --------+-------------+-------------+-------------+ 964 0% PLR | 20ms | 100ms | average | 965 --------+-------------+-------------+-------------+ 966 | 2.1 | 4.1 | 4.1 | 967 10/1 | 1.4 | 1.7 | 1.6 | 968 | 0.7 | 0.8 | 0.7 | 969 --------+-------------+-------------+-------------+ 970 | 1.6 | 3.4 | 3.4 | 971 20/2 | 1.2 | 1.6 | 1.4 | 972 | 0.5 | 0.7 | 0.5 | 973 --------+-------------+-------------+-------------+ 974 | 2.1 | 4.1 | 4.1 | 975 average | 1.3 | 1.6 | 1.5 | 976 | 0.5 | 0.7 | 0.5 | 977 --------+-------------+-------------+-------------+ 978 Table 6b. Channel Impact on SPDY Gain (1% PLR) 980 --------+-------------+-------------+-------------+ 981 1% PLR | 20ms | 100ms | average | 982 --------------------------------------------------- 983 | 2.3 | 2.2 | 2.3 | 984 10/1 | 1.1 | 0.9 | 1.0 | 985 | 0.4 | 0.3 | 0.3 | 986 --------+-------------+-------------+-------------+ 987 | 2.0 | 1.7 | 2.0 | 988 20/2 | 1.0 | 0.8 | 0.9 | 989 | 0.3 | 0.3 | 0.3 | 990 --------+-------------+-------------+-------------+ 991 | 2.3 | 2.2 | 2.3 | 992 average | 1.0 | 0.8 | 0.9 | 993 | 0.3 | 0.3 | 0.3 | 994 --------+-------------+-------------+-------------+ 996 Table 6c. Channel Impact on SPDY Gain (all PLR) 998 --------+-------------+-------------+-------------+ 999 all PLR | 20ms | 100ms | average | 1000 --------+-------------+-------------+-------------+ 1001 | 2.3 | 4.1 | 4.1 | 1002 10/1 | 1.3 | 1.3 | 1.3 | 1003 | 0.4 | 0.3 | 0.3 | 1004 --------+-------------+-------------+-------------+ 1005 | 2.0 | 3.4 | 3.4 | 1006 20/2 | 1.1 | 1.2 | 1.1 | 1007 | 0.3 | 0.3 | 0.3 | 1008 --------+-------------+-------------+-------------+ 1009 | 2.3 | 4.1 | 4.1 | 1010 average | 1.2 | 1.2 | 1.2 | 1011 | 0.3 | 0.3 | 0.3 | 1012 --------+-------------+-------------+-------------+ 1014 The most interesting observation here is the comparison between the 1015 two PLR tests. SPDY provided significant gains when the PLR was 0% 1016 (average gain 1.5), but showed worse average performance than HTTPS 1017 when packet loss was 1% (average gain 0.9). This effect was more 1018 pronounced in the large RTT cases as compared to the small RTT. This 1019 result points to a weakness in the way that SPDY utilizes TCP. With 1020 a typical HTTPS download of a web page, the browser will open a large 1021 number of simultaneous TCP connections. In this case, a random 1022 packet loss will cause the one affected connection to temporarily 1023 reduce its congestion window (and hence the effective data rate), but 1024 the other connections will be unaffected, and in fact may be able to 1025 opportunistically make use of the bandwidth made available by the 1026 affected connection. The result for HTTPS is that random packet loss 1027 has only a minor impact on page download time. In the case of SPDY, 1028 the number of parallel TCP connections is dramatically reduced (by as 1029 much as a factor of six), so that random packet loss has a much 1030 bigger impact on the overall throughput. 1032 In the absence of packet loss, SPDY provides better gains as the RTT 1033 increases. This is the result of reducing the number of round trips 1034 needed to fetch all of the page resources. Nonetheless, there were 1035 test cases with 0 packet loss where SPDY performed worse than HTTPS. 1036 This only occurred with Page C; SPDY always provided a benefit for 1037 Pages A and B when there was no packet loss. The root cause of the 1038 degradation is unknown, but it may be the result of the buffering 1039 configuration of the network simulator Dummynet. As stated 1040 previously, testing was performed with the default configuration of 1041 50 packet buffers. This configuration is not reflective of real 1042 networks (which in some cases may have more than double that amount 1043 of buffering), and could bias the results somewhat against 1044 applications (such as SPDY) that use a small number of TCP sessions. 1046 Additionally, SPDY generally shows more gain in the lower Data Rate 1047 cases. 1049 The overall average gain of 1.2 seen in our experiments aligns well 1050 with the performance gains that Google is seeing in their live 1051 deployments. In results presented in a December 8, 2011, Google 1052 TechTalk [TECHTALK], they report a 15.4% improvement in page load 1053 time (equivalent to a gain of 1.18). 1055 5.4.2. CASE 2: initcwnd=10 vs. initcwnd=3 1057 The initcwnd increase provides very modest gain across the majority 1058 of test cases. In a few cases there was a marginal degradation of 1059 performance compared to the default initcwnd case. Across all test 1060 conditions an average gain of 1.1 (or 9% reduction in page load time) 1061 was seen, as indicated in Table 7 and Table 8. 1063 Table 7. Website Impact on Initcwnd Gain 1065 +-------------+-------------+-------------+-------------+ 1066 | 1 server | 2 servers | 4 servers | average | 1067 ---------+-------------+-------------+-------------+-------------+ 1068 | 1.1 | 1.1 | 1.1 | 1.1 | 1069 Page A | 1.0 | 1.0 | 1.0 | 1.0 | 1070 | 0.9 | 1.0 | 0.9 | 0.9 | 1071 ---------+-------------+-------------+-------------+-------------+ 1072 | 1.1 | 1.3 | 1.2 | 1.3 | 1073 Page B | 1.0 | 1.1 | 1.0 | 1.0 | 1074 | 1.0 | 1.0 | 0.9 | 0.9 | 1075 ---------+-------------+-------------+-------------+-------------+ 1076 | 1.2 | 1.3 | 1.5 | 1.5 | 1077 Page C | 1.1 | 1.1 | 1.2 | 1.1 | 1078 | 1.0 | 1.0 | 1.0 | 1.0 | 1079 ---------+-------------+-------------+-------------+-------------+ 1080 | 1.2 | 1.3 | 1.5 | 1.5 | 1081 average | 1.1 | 1.1 | 1.1 | 1.1 | 1082 | 0.9 | 1.0 | 0.9 | 0.9 | 1083 ---------+-------------+-------------+-------------+-------------+ 1085 Overall the Page C test cases showed a slightly higher gain over the 1086 other two pages. Since Page C involves large files, it isn't 1087 surprising that the effect is slightly more pronounced than in Page 1088 A. In Page A, the majority of resources can be delivered in 3 packets 1089 or less anyway, so the increase in initcwnd doesn't reduce the number 1090 of round-trips. 1092 Table 8a. Channel Impact on Initcwnd Gain (0% PLR) 1094 ---------+-------------+-------------+-------------+ 1095 0% PLR | 20ms | 100ms | average | 1096 ---------+-------------+-------------+-------------+ 1097 | 1.1 | 1.2 | 1.2 | 1098 10/1 | 1.0 | 1.1 | 1.0 | 1099 | 1.0 | 1.0 | 1.0 | 1100 ---------+-------------+-------------+-------------+ 1101 | 1.1 | 1.1 | 1.1 | 1102 20/2 | 1.0 | 1.0 | 1.0 | 1103 | 1.0 | 0.9 | 0.9 | 1104 ---------+-------------+-------------+-------------+ 1105 | 1.1 | 1.2 | 1.2 | 1106 average | 1.0 | 1.0 | 1.0 | 1107 | 1.0 | 0.9 | 0.9 | 1108 ---------+-------------+-------------+-------------+ 1109 Table 8b. Channel Impact on Initcwnd Gain (1% PLR) 1111 ---------+-------------+-------------+-------------+ 1112 1% PLR | 20ms | 100ms | average | 1113 ---------+-------------+-------------+-------------+ 1114 | 1.5 | 1.2 | 1.5 | 1115 10/1 | 1.1 | 1.1 | 1.1 | 1116 | 0.9 | 1.0 | 0.9 | 1117 ---------+-------------+-------------+-------------+ 1118 | 1.4 | 1.3 | 1.4 | 1119 20/2 | 1.1 | 1.1 | 1.1 | 1120 | 0.9 | 0.9 | 0.9 | 1121 ---------+-------------+-------------+-------------+ 1122 | 1.5 | 1.3 | 1.5 | 1123 average | 1.1 | 1.1 | 1.1 | 1124 | 0.9 | 0.9 | 0.9 | 1125 ---------+-------------+-------------+-------------+ 1127 Table 8c. Channel Impact on Initcwnd Gain (all PLR) 1129 ---------+-------------+-------------+-------------+ 1130 all PLR | 20ms | 100ms | average | 1131 ---------+-------------+-------------+-------------+ 1132 | 1.5 | 1.2 | 1.5 | 1133 10/1 | 1.1 | 1.1 | 1.1 | 1134 | 0.9 | 1.0 | 0.9 | 1135 ---------+-------------+-------------+-------------+ 1136 | 1.4 | 1.3 | 1.4 | 1137 20/2 | 1.1 | 1.1 | 1.1 | 1138 | 0.9 | 0.9 | 0.9 | 1139 ---------+-------------+-------------+-------------+ 1140 | 1.5 | 1.3 | 1.5 | 1141 average | 1.1 | 1.1 | 1.1 | 1142 | 0.9 | 0.9 | 0.9 | 1143 ---------+-------------+-------------+-------------+ 1145 When viewing the results broken down by channel condition, we don't 1146 see a significant difference in gain from one channel condition to 1147 the next. Notably, even in the high RTT cases we don't see much 1148 change in the gain. This is likely due to the fact that the majority 1149 of resources had sizes that either fell below the 3 packet default 1150 initcwnd, or were much larger (e.g., over 210 packets each for the 1151 images in Page C) so that the initcwnd change might save a single RTT 1152 for a transfer that took 9 RTTs using the default initcwnd. 1154 5.4.3. CASE 3: SPDY+initcwnd=10 vs. HTTPS+initcwnd=3 1156 The combination of SPDY and the initcwnd increase resulted in a 1157 benefit that was more than the product of the two individual gains. 1158 This is a result of the fact that, due to SPDY, all of the TCP 1159 sessions involved multiple round-trips, so the acceleration of 1160 initial data rate provided by the initcwnd increase resulted in 1161 tangible benefits. In fact, one case (Page A, single server) saw an 1162 astounding gain of 4.7. By examining the raw data in 1163 [SPDY-ANALYSIS], it can be seen that this combination reduced the 1164 median page load time of 8.7 seconds down to 1.9 seconds. 1165 Unfortunately, this result is not typical, and Page C again 1166 experienced worse performance when the new protocol options are used. 1167 The overall average gain seen was 1.4 across all test cases as shown 1168 in Table 9 and Table 10. 1170 Table 9. Website Impact on SPDY+Initcwnd Gain 1172 --------+-------------+-------------+-------------+-------------+ 1173 | 1 server | 2 servers | 4 servers | average | 1174 --------+-------------+-------------+-------------+-------------+ 1175 | 4.7 | 2.3 | 1.5 | 4.7 | 1176 Page A | 3.2 | 1.5 | 1.4 | 2.0 | 1177 | 2.2 | 1.2 | 1.2 | 1.2 | 1178 --------+-------------+-------------+-------------+-------------+ 1179 | 3.3 | 1.6 | 1.5 | 3.3 | 1180 Page B | 1.9 | 1.1 | 1.1 | 1.4 | 1181 | 0.9 | 0.6 | 0.7 | 0.6 | 1182 --------+-------------+-------------+-------------+-------------+ 1183 | 1.0 | 1.0 | 1.1 | 1.1 | 1184 Page C | 0.6 | 0.7 | 0.9 | 0.7 | 1185 | 0.3 | 0.4 | 0.6 | 0.3 | 1186 --------+-------------+-------------+-------------+-------------+ 1187 | 4.7 | 2.3 | 1.5 | 4.7 | 1188 average | 1.9 | 1.1 | 1.1 | 1.4 | 1189 | 0.3 | 0.4 | 0.6 | 0.3 | 1190 --------+-------------+-------------+-------------+-------------+ 1191 Table 10a. Channel Impact on SPDY+Initcwnd Gain (0% PLR) 1193 ---------+-------------+-------------+-------------+ 1194 0% PLR | 20ms | 100ms | average | 1195 ---------+-------------+-------------+-------------+ 1196 | 3.2 | 4.7 | 4.7 | 1197 10/1 | 1.7 | 1.8 | 1.8 | 1198 | 1.0 | 0.9 | 0.9 | 1199 ---------+-------------+-------------+-------------+ 1200 | 3.1 | 3.9 | 3.9 | 1201 20/2 | 1.6 | 1.7 | 1.6 | 1202 | 0.7 | 0.7 | 0.7 | 1203 ---------+-------------+-------------+-------------+ 1204 | 3.2 | 4.7 | 4.7 | 1205 average | 1.7 | 1.7 | 1.7 | 1206 | 0.7 | 0.7 | 0.7 | 1207 ---------+-------------+-------------+-------------+ 1209 Table 10b. Channel Impact on SPDY+Initcwnd Gain (1% PLR) 1211 ---------+-------------+-------------+-------------+ 1212 1% PLR | 20ms | 100ms | average | 1213 ---------+-------------+-------------+-------------+ 1214 | 3.1 | 2.4 | 3.1 | 1215 10/1 | 1.2 | 0.9 | 1.1 | 1216 | 0.4 | 0.3 | 0.3 | 1217 ---------+-------------+-------------+-------------+ 1218 | 2.8 | 2.2 | 2.8 | 1219 20/2 | 1.2 | 1.0 | 1.1 | 1220 | 0.3 | 0.3 | 0.3 | 1221 ---------+-------------+-------------+-------------+ 1222 | 3.1 | 2.4 | 3.1 | 1223 average | 1.2 | 1.0 | 1.1 | 1224 | 0.3 | 0.3 | 0.3 | 1225 ---------+-------------+-------------+-------------+ 1226 Table 10c. Channel Impact on SPDY+Initcwnd Gain (all PLR) 1228 ---------+-------------+-------------+-------------+ 1229 all PLR | 20ms | 100ms | average | 1230 ---------+-------------+-------------+-------------+ 1231 | 3.2 | 4.7 | 4.7 | 1232 10/1 | 1.5 | 1.4 | 1.4 | 1233 | 0.4 | 0.3 | 0.3 | 1234 ---------+-------------+-------------+-------------+ 1235 | 3.1 | 3.9 | 3.9 | 1236 20/2 | 1.4 | 1.3 | 1.3 | 1237 | 0.3 | 0.3 | 0.3 | 1238 ---------+-------------+-------------+-------------+ 1239 | 3.2 | 4.7 | 4.7 | 1240 average | 1.4 | 1.3 | 1.4 | 1241 | 0.3 | 0.3 | 0.3 | 1242 ---------+-------------+-------------+-------------+ 1244 In the packet loss test cases, there was a fairly significant 1245 performance loss with SPDY, as reported in earlier test cases. This 1246 is particularly true in the high RTT cases. 1248 5.4.4. Results Summary 1250 The results are summarized further in Table 11. A caveat on the 1251 average results presented here is worth noting: the 72 test cases 1252 were selected to test the protocol changes in a fairly wide range of 1253 conditions in order to understand the impact that individual factors 1254 have on the performance gains. As a result, the 72 test cases likely 1255 do not comprise a statistically accurate sampling of real-world 1256 sites, and so the average results presented here may not accurately 1257 reflect the average performance gain that would be seen in the real 1258 world. However, as noted previously, the average results do appear 1259 to align well with the average gains seen by Google via their live 1260 deployments with users utilizing the Chrome browser. In addition, it 1261 is interesting to examine the average gain achieved in a particular 1262 subset of the test conditions. For that we select the subset 1263 consisting of pages A and B operating on a wireline network (PLR=0%). 1264 For that subset, we see that the combination of SPDY/2 and the 1265 initcwnd increase achieves an average gain of 2.1 (52% reduction in 1266 page load time). 1268 Table 11. Summary of Results 1270 +--------------+--------------+--------------+ 1271 | Case 1 | Case 2 | Case 3 | 1272 | SPDY | initcwnd |SPDY+initcwnd | 1273 +----------------------+--------------+--------------+--------------+ 1274 |Best Gain | 4.1 | 1.5 | 4.7 | 1275 |(all test conditions) |(A,1,10,100,0)|(C,4,10,20,1) |(A,1,10,100,0)| 1276 +----------------------+--------------+--------------+--------------+ 1277 |Average Gain | | | | 1278 |(all test conditions) | 1.2 | 1.1 | 1.4 | 1279 +----------------------+--------------+--------------+--------------+ 1280 |Worst Gain | 0.3 | 0.9 | 0.3 | 1281 |(all test conditions) | (C,1,x,x,1) |(A,1,20,100,0)|(C,1,20,100,1)| 1282 +----------------------+--------------+--------------+--------------+ 1283 |# test cases achieving| 40 of 72 | 58 of 72 | 44 of 72 | 1284 | gain >= 1 | (56%) | (81%) | (61%) | 1285 +----------------------+--------------+--------------+--------------+ 1286 |# test cases achieving| 8 of 72 | 0 of 72 | 14 of 72 | 1287 | gain >= 2 | (11%) | (0%) | (19%) | 1288 +----------------------+--------------+--------------+--------------+ 1289 |Average Gain | | | | 1290 |(Pages A and B | 1.8 | 1.0 | 2.1 | 1291 | with PLR=0%) | | | | 1292 +----------------------+--------------+--------------+--------------+ 1294 6. Recommendations and Conclusions 1296 The results presented here indicate that SPDY/2 in conjunction with 1297 the TCP Initial Congestion Window increase has the potential to 1298 improve or impair web page download performance depending on a number 1299 of factors. 1301 Deployment of SPDY/2 for general-purpose web servers should be 1302 considered in light of the following concerns: 1304 o While the Chrome and Firefox browsers are important (approximately 1305 29% and 22% market share respectively as of June 2012), client 1306 support isn't ubiquitous. 1308 o Some web page downloads were significantly impaired by SPDY. 1310 o Work is underway on a revision to the protocol, and with 1311 standardization in the IETF a possibility in the near future, 1312 waiting may be worthwhile to see what develops in this space. 1314 On the other hand, for a controlled environment where a single entity 1315 provides the servers, the client software, and the web content, SPDY 1316 might be a valuable tool. An example application might be a remote 1317 user interface (RUI) that is delivered via HTTP to a CPE device. 1319 In addition to potential uses for SPDY by network operators, another 1320 area of interest is the impact that SPDY would have on network 1321 traffic if it were to be widely adopted. In general the story here 1322 is good, by reducing the number of simultaneous TCP sessions and 1323 extending the duration of many of the sessions, other applications 1324 could see improved performance as a result of TCP congestion 1325 avoidance being invoked far more often. Secondly, the expectation 1326 would be to see a slight reduction in data usage, due to the greater 1327 efficiency that SPDY provides (fewer TCP control packets), as well as 1328 a slight increase in average packet size, due to the multiplexing of 1329 HTTP responses. Both of these factors will serve to reduce the 1330 packets per second rate of the network, which improves scalability of 1331 CMTSs, routers, DPI boxes, etc. Finally, equipment implementing 1332 Carrier Grade NAT (which will be a key element of the IPv6 transition 1333 in some networks) could see improved scalability as the number of 1334 simultaneous TCP connections is reduced. 1336 In regards to the increase in the TCP Initial Congestion Window, 1337 while we see only marginal gains resulting from this enhancement, the 1338 change can be made simply by changing a server operating system 1339 parameter. It requires no client modifications. As a result, we see 1340 no reason not to set the server initcwnd at the proposed value of 10. 1342 7. IANA Considerations 1344 This document makes no request of IANA. 1346 Note to RFC Editor: this section may be removed on publication as an 1347 RFC. 1349 8. Security Considerations 1351 See [I-D.mbelshe-httpbis-spdy] and [I-D.ietf-tcpm-initcwnd] for 1352 Security Considerations. 1354 9. Acknowledgements 1356 The authors would like to thank Marc Weaver, Darshak Thakore, Eric 1357 Winkelman, Robin Sam Ku, Scott Maize, and Rob Moon for their 1358 assistance in developing and configuring the test bed, and collecting 1359 the data presented in this report. The authors would also like to 1360 gratefully express their appreciation to Chris Donley, Joan Strosin, 1361 Ken Barringer and Christie Poland for manuscript preparation 1362 assistance, and to Mike Belshe and Roberto Peon for their review and 1363 very insightful comments. 1365 10. References 1367 [BUFFERCONTROL] 1368 CableLabs, "Cable Modem Buffer Control", . 1372 [CDNPLANET] 1373 "www.cdnplanet.com", . 1375 [DUMMYNET] 1376 "Dummynet", . 1378 [Faster] Google, "Let's Make the Web Faster", 1379 . 1381 [GOOGLE-BENCHMARKS] 1382 Lloyd , M., "SPDY and Server Push: Analysis and 1383 Experiments", June 2010, 1384 . 1386 [HTTP_Pipelining] 1387 Wikipedia, "HTTP Pipelining", 1388 . 1390 [I-D.agl-tls-nextprotoneg] 1391 Langley, A., "Transport Layer Security (TLS) Next Protocol 1392 Negotiation Extension", draft-agl-tls-nextprotoneg-03 1393 (work in progress), April 2012. 1395 [I-D.ietf-tcpm-initcwnd] 1396 Chu, J., Dukkipati, N., Cheng, Y., and M. Mathis, 1397 "Increasing TCP's Initial Window", 1398 draft-ietf-tcpm-initcwnd-03 (work in progress), 1399 February 2012. 1401 [I-D.mbelshe-httpbis-spdy] 1402 Belshe, M. and R. Peon, "SPDY Protocol", 1403 draft-mbelshe-httpbis-spdy-00 (work in progress), 1404 February 2012. 1406 [INITCWND-PAPER] 1407 Dukkipati , N., Refice, T., Cheng, Y., Chu, J., Sutin, N., 1408 Agarwal, A., Herbert, T., and A. Jain, "An Argument for 1409 Increasing TCP's Initial Congestion Window", . 1412 [RFC0793] Postel, J., "Transmission Control Protocol", STD 7, 1413 RFC 793, September 1981. 1415 [RFC1122] Braden, R., "Requirements for Internet Hosts - 1416 Communication Layers", STD 3, RFC 1122, October 1989. 1418 [RFC2001] Stevens, W., "TCP Slow Start, Congestion Avoidance, Fast 1419 Retransmit, and Fast Recovery Algorithms", RFC 2001, 1420 January 1997. 1422 [RFC2414] Allman, M., Floyd, S., and C. Partridge, "Increasing TCP's 1423 Initial Window", RFC 2414, September 1998. 1425 [RFC2616] Fielding, R., Gettys, J., Mogul, J., Frystyk, H., 1426 Masinter, L., Leach, P., and T. Berners-Lee, "Hypertext 1427 Transfer Protocol -- HTTP/1.1", RFC 2616, June 1999. 1429 [RFC3390] Allman, M., Floyd, S., and C. Partridge, "Increasing TCP's 1430 Initial Window", RFC 3390, October 2002. 1432 [SPDY] "SPDY - The Chromium Projects", 1433 . 1435 [SPDY-ANALYSIS] 1436 White, G., "Analysis of Google Spdy and TCP initcwnd", 1437 May 2012, . 1440 [TECHTALK] 1441 "http://www.cnx-software.com/2012/01/27/ 1442 spdy-aims-to-make-the-web-faster-and-replace-http/", 1443 . 1445 Authors' Addresses 1447 Greg White 1448 CableLabs 1449 858 Coal Creek Circle 1450 Louisville, CO 80027-9750 1451 USA 1453 Email: g.white@cablelabs.com 1455 Jean-Francois Mule 1456 CableLabs 1457 180 Montgomery St 1458 Suite 2480 1459 San Francisco, CA 94104-4203 1460 USA 1462 Email: jf.mule@cablelabs.com 1464 Dan Rice 1465 CableLabs 1466 858 Coal Creek Circle 1467 Louisville, CO 80027-9750 1468 USA 1470 Email: d.rice@cablelabs.com