[v6ops] Apple and IPv6 - Happy Eyeballs

[David Schinazi <dschinazi at apple.com>]

Hi everyone,

Today Apple released the first public seeds of iOS 9 and OS X El Capitan.
These seeds (and the third developer seeds released yesterday) include an improved version of Happy Eyeballs.

Based on our testing, this makes our Happy Eyeballs implementation go from roughly 50/50 IPv4/IPv6 in iOS 8 and Yosemite
to ~99% IPv6 in iOS 9 and El Capitan betas.

While our previous implementation from four years ago was designed to select the connection with lowest latency
no matter what, we agree that the Internet has changed since then and reports indicate that biasing towards IPv6 is now
beneficial for our customers: IPv6 is now mainstream instead of being an exception, there are less broken IPv6 tunnels,
IPv4 carrier-grade NATs are increasing in numbers, and throughput may even be better on average over IPv6.

The updated implementation performs the following:
- Query the DNS resolver for A and AAAA.
   If the DNS records are not in the cache, the requests are sent back to back on the wire, AAAA first.
- If the first reply we get is AAAA, we send out the v6 SYN immediately
- If the first reply we get is A and we're expecting a AAAA, we start a 25ms timer
   - If the timer fires, we send out the v4 SYN
   - If we get the AAAA during that 25ms window, we move on to address selection
- When we have a list of IP addresses (either from the DNS cache or by receiving them close together with v4 before v6),
   we perform our own address selection algorithm to sort them. This algorithm uses historical RTT data to prefer addresses
   that have lower latency - but has a 25ms leeway: if the historical RTT of two compared address are within 25ms of each
   other, we use RFC3484 to pick the best one.
- Once the list is sorted, we send out the SYN for the first address and start timers based on average and variance of the
   historical TCP RTT. Roughly speaking, we start the second address around the same time we send out a SYN retransmission
   for the first address.
- The first address to reply with a SYN-ACK wins the race, we then cancel the other TCP connection attempts.

If this behavior proves successful during the beta period, you should expect more IPv6 traffic from Apple products in the future.
Note however that this only describes the current beta and all these details are subject to change.

Please test this out if you have the means to, we'd love to see test results and receive feedback!

I would like to personally thank Jason Fesler and Paul Saab for their help investigating these issues and testing this.

Thanks,
David Schinazi
CoreOS Networking Engineer