Re: [dnsext] Chairs statement on "client-option" debate

Wilmer van der Gaast <wilmer@google.com> Sun, 25 July 2010 23:30 UTC

Return-Path: <owner-namedroppers@ops.ietf.org>
X-Original-To: ietfarch-dnsext-archive@core3.amsl.com
Delivered-To: ietfarch-dnsext-archive@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 0BC493A693B; Sun, 25 Jul 2010 16:30:23 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -101.273
X-Spam-Level:
X-Spam-Status: No, score=-101.273 tagged_above=-999 required=5 tests=[BAYES_50=0.001, FH_RELAY_NODNS=1.451, FM_FORGED_GMAIL=0.622, HELO_MISMATCH_COM=0.553, RCVD_IN_DNSWL_MED=-4, RDNS_NONE=0.1, USER_IN_WHITELIST=-100]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8mu19B6HLQsr; Sun, 25 Jul 2010 16:29:56 -0700 (PDT)
Received: from psg.com (psg.com [IPv6:2001:418:1::62]) by core3.amsl.com (Postfix) with ESMTP id DB9CF3A688F; Sun, 25 Jul 2010 16:29:54 -0700 (PDT)
Received: from majordom by psg.com with local (Exim 4.72 (FreeBSD)) (envelope-from <owner-namedroppers@ops.ietf.org>) id 1OdAZ0-00087c-0R for namedroppers-data0@psg.com; Sun, 25 Jul 2010 23:24:58 +0000
Received: from [216.239.44.51] (helo=smtp-out.google.com) by psg.com with esmtps (TLSv1:AES256-SHA:256) (Exim 4.72 (FreeBSD)) (envelope-from <wilmer@google.com>) id 1OdAYs-00085o-UZ for namedroppers@ops.ietf.org; Sun, 25 Jul 2010 23:24:52 +0000
Received: from kpbe20.cbf.corp.google.com (kpbe20.cbf.corp.google.com [172.25.105.84]) by smtp-out.google.com with ESMTP id o6PNOnex004086 for <namedroppers@ops.ietf.org>; Sun, 25 Jul 2010 16:24:49 -0700
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=google.com; s=beta; t=1280100290; bh=F8V/mQiy8fw9bopbKcYX3ClMWuY=; h=MIME-Version:In-Reply-To:References:Date:Message-ID:Subject:From: To:Cc:Content-Type; b=L4CFUZNh8hk5Ah2b11rN3ulPCDDXPga32bH7T6UHJUHIcVW47MR9/0CF5zi/pKbMu 7/pwBwPA3/qpqMhq5cgXw==
DomainKey-Signature: a=rsa-sha1; s=beta; d=google.com; c=nofws; q=dns; h=mime-version:in-reply-to:references:date:message-id:subject:from:to: cc:content-type:x-system-of-record; b=kv2Djf6RXCNS8pPcq0t5mr7fzcyD9xvnBTJG2begmVEPK51mKFdOzD5pz9fb8Rj2v aBJhjtw9nBsSVPSZxv5MQ==
Received: from bwz15 (bwz15.prod.google.com [10.188.26.15]) by kpbe20.cbf.corp.google.com with ESMTP id o6PNOlWR027368 for <namedroppers@ops.ietf.org>; Sun, 25 Jul 2010 16:24:48 -0700
Received: by bwz15 with SMTP id 15so2550683bwz.22 for <namedroppers@ops.ietf.org>; Sun, 25 Jul 2010 16:24:47 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.84.17 with SMTP id h17mr4860769bkl.101.1280100286632; Sun, 25 Jul 2010 16:24:46 -0700 (PDT)
Received: by 10.204.46.77 with HTTP; Sun, 25 Jul 2010 16:24:46 -0700 (PDT)
In-Reply-To: <4C054F87.5010608@ogud.com>
References: <4C054F87.5010608@ogud.com>
Date: Mon, 26 Jul 2010 01:24:46 +0200
Message-ID: <AANLkTinD5wwj+=vL9Y7ZXLMFuG7mcYy2LhnxYsuT2obp@mail.gmail.com>
Subject: Re: [dnsext] Chairs statement on "client-option" debate
From: Wilmer van der Gaast <wilmer@google.com>
To: Olafur Gudmundsson <ogud@ogud.com>
Cc: namedroppers <namedroppers@ops.ietf.org>
Content-Type: text/plain; charset="ISO-8859-1"
X-System-Of-Record: true
Sender: owner-namedroppers@ops.ietf.org
Precedence: bulk
List-ID: <namedroppers.ops.ietf.org>
List-Unsubscribe: To unsubscribe send a message to namedroppers-request@ops.ietf.org with
List-Unsubscribe: the word 'unsubscribe' in a single line as the message text body.
List-Archive: <http://ops.ietf.org/lists/namedroppers/>

As authors of edns-client-subnet, we never sent a reply addressing The
Chairs' comments.

The main issues seem to be related to 1 - clarifying the problem
statement, 2 - explaining why DNS is the best solution space, and 3 -
have a discussion about the costs, making sure that the right people
are paying for them.

We'll provide a clarified problem statement in edns-client-subnet-02
(really called -00 thanks to the counter reset). We believe 2 and 3
are discussions out of the scope of the draft itself, so we'll explain
our thoughts on those in this e-mail.

In short, we understand that edns-client-subnet is not a perfect
solution. We believe, however, that a perfect solution would require
making changes to how the user agent works. We can't do that, we only
get to change the nameservers and recursive resolvers. Changing all
user agents will take years (note how there are still many people
using MSIE6 or even Netscape). Having all interested parties (and only
them) roll out edns-client-subnet can be done in less than a year.

Currently users of open resolvers (which are used for various reasons
including performance, filtering of undesired (family-unfriendly,
etc.) content, and avoiding their ISP's ad-serving nameservers) have
problems with accessing some CDN content (going from severe latency
penalties to content simply being unreachable), we need to fix that.

This said... there were several alternative solutions proposed in
various threads that also don't need work on the client. Most of these
are known with the CDN industry, some are actually used. However, DNS
tricks are still the most popular. Here's some of the reasons why we
believe this is the case:

* If something breaks, one of the easiest and more reliable ways to
send users somewhere else is to... change DNS records, and have short
TTLs. No matter how much redundancy you build in your infrastructure,
eventually you'll have to do maintenances on it, or it'll break. DNS
is a simple, painless, and easy way to control where your users end
up. If you have a highly distributed infrastructure, this can happen
often, and from here.. to returning results that are geolocated, the
step is small.

Two ways were proposed to do this on the HTTP level instead:

* Framesets. These were deprecated several years ago for good reasons
outside the scope of this proposal, so no viable solution to this
problem.

* The other proposed solution was to use HTTP 30x redirects. There are
a few problems with this:
  - The main weakness is the latency penalty. A user has to resolve
www.foo.com first, connect to that server, send a query, get a 30x
response. Then disconnect, resolve the new hostname, etc etc. A
well-written app can limit this to just the first query, but this
requires putting this logic into the web application, which can be
considered a layer violation.
    The latency cost of establishing a TCP connection with servers
just for a redirect is sensibly more costly than a UDP query (which is
also only done on the first request, and often answered from cache),
and can definitively affect the overall latency and user experience.

  - This will either mean forwarding user agents to URLs with IP
addresses in them, or to URLs like http://lhr00.mail.foo.com/. This by
itself gives several problems:
    * These URLs look ugly, are not easy to remember, not practical to
bookmark/pass from person to person (many CDN nodes are accessible
only to customers of ISPs hosting the node), needlessly expose
internals to the user (yesterday my request went to London, and today
it goes to Amsterdam??).
    * With bookmarks in mind, content providers will be required to
keep old datacenters/servers in DNS even though they're
decommissioned. Also, taking down a datacenter temporarily gets more
complicated.
    * This is likely to break caching. The full URL is (part of) the
cache key of HTTP content. If the hostname changes, the client will
have to reload the full file. Webapps these days are often powered by
hundreds of kilobytes of JavaScript, being able to cache those is
essential.
    * This also causes problems with SSL certificates. Wildcards and
other ways are available to work around this problem, but both are
still not supported by all user agents.
    * And in a related note, in a world where phishing is a real
threat to many, it must be easy for users to recognise reliable URLs.
Expecting that someone unfamiliar with how URLs and DNS work that
"https://lhr00.mail.foo.com/" is good while
"https://lhr000.com/mail.foo.com/" is bad, is unrealistic.
    * Having said that, this technique is already used for things like
video streaming, where the URL is not ever going to be visible to the
user and throughput is more important than latency.

* Most of all, both solutions only solve the problem for HTTP.
Although HTTP is definitely our main focus here, other protocols can
benefit from this:
  - SIP came up as an example, although it's not a great one, as the
resulting RTP stream is much more latency-sensitive than SIP itself,
so the right RTP server can be selected by the SIP proxy already.
  - VPNs
  - Various e-mail protocols

Using BGP/anycast instead doesn't have any of these issues. However,
it has its own imperfections. Although anycast is fairly good at
directing packets to the nearest endpoint, the nearest endpoint is not
always the right endpoint.

CDNs are about more than just latency. They're also about capacity.
When a node is full, the excess traffic should go elsewhere. Pulling
the IP announcement wouldn't get redirect the excess but all traffic.
There may be ways to refine this, but they're fragile and hard to
control.

Another problem with using anycast is the fact that redirecting
traffic (i.e. when taking down a destination for maintenance) moves
new *and* existing TCP connections, resetting all the existing
connections, often with ugly user-visible consequences. These
redirections can't be avoided since they're not just within the power
of the content provider.

Using DNS has its own imperfections, but the CDNs of today have mostly
dealt with them already. Think of problems like not knowing how many
application queries will be generated by once returning a certain IP
address in a DNS response.

The last problem unsolved so far is the wrong assumption that users
are always close to their resolvers. edns-client-subnet is the key to
making them work completely. It comes with its own problems and
challenges, but they affecte only the CDNs and the operators of
recursive resolvers who opt in to using the option, as explained
below.


3) COST OF IMPLEMENTATION

In our mind, edns-client-subnet would be turned on and used only by
those third party resolvers and authoritative name servers that need a
solution for problem (2). We don't see this option being enabled by
anyone else, or worse, by default.

In any case, if we had to break down the cost of this option, we'd have:

a - implementation cost, authoritative nameserver, and recursive resolver.

and, on servers that enable the option:

b - additional load caused by more queries being sent
c - additional resources to handle larger caches, as replies would now
be tied to networks
d - configuration and management overhead

So, on the CDN/authoritative resolver side: authoritative servers that
don't use DNS tricks, can keep living happily without implementing
this option. Authoritative servers used by CDNs are likely to care
about the latency of their users, and are likely to implement support
for this option, paying their share on the cost. (They can finally
provide good latency to users of third party resolvers.)

On the recursive resolver side: if you're running a recursive resolver
for a small network or for a set of users that are close to the
resolver, you can keep living happily without enabling this option.

If you are running recursive resolvers for a large ISP or distributed
network, you're probably *already* paying the price: you probably have
lot of recursive resolvers, close to your users, to avoid paying the
price of the extra latency. In this case, you can keep doing your
business as usual, or ... you can *decide* if you *want* to, to enable
edns-client-subnet. Knowing explicitly which servers are returning
responses that are tied to a particular IP address also helps with
debugging, implementing smarter algorithms for shared caches, and
planning for capacity (how many servers we need, where, ...). So, if
you're willing to pay the cost, it's probably because you get a
benefit.

Once you have a software that supports edns-client-subnet, what you'll
have to do is configure how many bits of the network of the user to
forward, and for which networks your server will forward
edns-client-subnet.

If you are running a third party resolver, well: you can keep doing
business as usual, so, ... live with the fact that some of your users
will have bad latency when connecting to some CDNs. Or enable
edns-client-subnet, with more or less the same
advantages/disadvantages as in the previous case.

Again, this should not be enabled by default, we don't see this being
generally useful.

As for other authoritative nameservers, that don't need or want
support for edns-client-subnet, they might see additional EDNS0
options from time to time, but as long as they ignore them, they're
fine. Broken firewalls and filters will affect edns-client-subnet as
any other EDNS0 option, so nothing special to say here.

We hope this answers the questions on the table. We'll keep working on
-02 and hope to have it ready soon after IETF78.


Regards,

Wilmer v/d Gaast, Carlo Contavalli,
Sean Leach, Darryl Rodden.