Re: [TLS] Preliminary data on Firefox TLS 1.3 Middlebox experiment

Eric Rescorla <ekr@rtfm.com> Tue, 05 December 2017 21:54 UTC

MIME-Version: 1.0
In-Reply-To: <CABcZeBNWjw2F9FuMM2muj263PpKnt+Md8DhskOwb2T7OrJvYdA@mail.gmail.com>
References: <CABcZeBNWjw2F9FuMM2muj263PpKnt+Md8DhskOwb2T7OrJvYdA@mail.gmail.com>
From: Eric Rescorla <ekr@rtfm.com>
Date: Tue, 05 Dec 2017 13:53:52 -0800
Message-ID: <CABcZeBPhRP9TNNUjvLZA6vOhz5y40VLF28bdqQYn1iePOewdLg@mail.gmail.com>
To: "tls@ietf.org" <tls@ietf.org>
Content-Type: multipart/alternative; boundary="001a11457576551a95055f9ee1a9"
Archived-At: <https://mailarchive.ietf.org/arch/msg/tls/or-8GSOSIpEbEQk7tkL7XPyrDHY>
Subject: Re: [TLS] Preliminary data on Firefox TLS 1.3 Middlebox experiment
Precedence: list

On Tue, Dec 5, 2017 at 1:35 PM, Eric Rescorla <ekr@rtfm.com> wrote:

> Hi folks,
>
> I now have some preliminary numbers to share with the group based on
> our Firefox experiments. The executive summary is that our data
> confirms Google's results. More detail below.
>
>
> EXPERIMENTAL DESIGN
> This is a forced experiment in which each client tries all the
> variants. The experiment is deployed via a system add-on (a remotely
> deployable, centrally managed piece of JavaScript code), and then
> takes measurements by trying to do an XHR to a given URL
> (https://mail.google.com/robots.txt) with a specific set of flags. We
> do the following three measurements:
>
> - TLS 1.2
> - TLS 1.3 draft-18
> - TLS 1.3 draft-18 with (approximately) PR#1092 ("7e02")
>
> We take five trials for each measurement, randomly shuffling the
> measurement order and then repeating the shuffled pattern five
> times. Each trial is done with a different connection and we declare
> "success" when any of the five trials succeeds.
>
>
> RESULTS
> This experiment was run on a 2% sample of the Firefox Beta population
> who have locale set to en-US, which we selected because of very
> high GMail blocking rates in some locales, which is a potential
> confounding factor. The experimen started 11/27 and has been running
> through today.
>
> This gave us an initial population of 161578, of whom 160809 (99.5%
> completed the experiment and reported results). This produced the
> following results:
>
>                      Success      Failure      Fail Rate
> --------------------------------------------------------
>
> TLS 1.2               158260         2549          .0158
> TLS 1.3-18            158194         4743          .0291
>

Oops. This first number should be 156066. This is what happens when you cut
and paste from your notebook.

-Ekr


TLS 1.3-Experiment    158194         2615          .0163
>
> For the statistics minded, the difference between -18 and 1.2 is
> significant at p < .001 and the 95% confidence interval of the failure
> rate difference is .0122-.0143 (using R's prop.test). There is no
> significant difference between 1.2 and 1.3-experiment (p = .36).
>
> We've got a -22 experiment in flight now, but it will only be on
> Nightly, so this is probably the strongest data we will have for
> a while.
>
> -Ekr
>
>
> ADDITIONAL DETAILS
> The relevant NSS version: https://dxr.mozilla.org/
> mozilla-beta/source/security/nss/lib/ssl
> Experimental code: https://github.com/mozilla/one-off-system-add-ons/tree/
> master/addons/tls13-middlebox-ghack
> iPython Notebook with analysis: https://gist.github.com/ekr/
> 598208b5399faf303453b10cb11647bf
>
>
>
>
>
>
>

[TLS] Preliminary data on Firefox TLS 1.3 Middleb… Eric Rescorla
Re: [TLS] Preliminary data on Firefox TLS 1.3 Mid… Eric Rescorla