On Feb 15, 2011, at 5:51 PM, Adam Barth wrote:
> On Tue, Feb 15, 2011 at 5:46 PM, Adam Barth <ietf@adambarth.com> wrote:
>> On Tue, Feb 15, 2011 at 3:55 PM, Roy T. Fielding <fielding@gbiv.com> wrote:
>>> On Feb 15, 2011, at 2:05 PM, Adam Barth wrote:
>>>> On Tue, Feb 15, 2011 at 1:22 PM, Julian Reschke <julian.reschke@gmx.de> wrote:
>>>>> On 15.02.2011 22:11, Adam Barth wrote:
>>>>>> ...
>>>>>> You really think we should recommend that servers use invalid UTF-8
>>>>>> sequences as cookie-values?  That sounds like bad advice...
>>>>>> ...
>>> 
>>> I am not recommending that they use invalid UTF-8 sequences.
>>> The ABNF only tells the implementer what octets can be used
>>> and what needs to be anticipated while parsing.  If I were
>>> designing a new Cookie protocol, I would exclude the high bits,
>>> but this is how the current Cookie protocol actually works in
>>> practice.
>> 
>> I feel like I'm just repeating myself.  The text you're complaining
>> about does not define how to parse a Set-Cookie header.
> 
> To be crystal clear, this is the only text in the document that refers
> to this figure:
> 
>          Servers SHOULD NOT send Set-Cookie headers
>          that fail to conform to the following grammar:

Which, as I said, is an incorrect requirement if this spec is
supposed to reflect how Cookie and Set-Cookie are used on
the Internet.  That should be more obvious in context:

Abstract:

   This document defines the HTTP Cookie and Set-Cookie header fields.
   These header fields can be used by HTTP servers to store state
   (called cookies) at HTTP user agents, letting the servers maintain a
   stateful session over the mostly stateless HTTP protocol.  Although
   cookies have many historical infelicities that degrade their security
   and privacy, the Cookie and Set-Cookie header fields are widely used
   on the Internet.

ToC

   4.  Server Requirements  . . . . . . . . . . . . . . . . . . . . . 12
     4.1.  Set-Cookie . . . . . . . . . . . . . . . . . . . . . . . . 12
       4.1.1.  Syntax . . . . . . . . . . . . . . . . . . . . . . . . 12
       4.1.2.  Semantics (Non-Normative)  . . . . . . . . . . . . . . 13
     4.2.  Cookie . . . . . . . . . . . . . . . . . . . . . . . . . . 16
       4.2.1.  Syntax . . . . . . . . . . . . . . . . . . . . . . . . 16
       4.2.2.  Semantics  . . . . . . . . . . . . . . . . . . . . . . 16
   5.  User Agent Requirements  . . . . . . . . . . . . . . . . . . . 17

4.1.1.  Syntax

   Informally, the Set-Cookie response header contains the header name
   "Set-Cookie" followed by a ":" and a cookie.  Each cookie begins with
   a name-value pair, followed by zero or more attribute-value pairs.
   Servers SHOULD NOT send Set-Cookie headers that fail to conform to
   the following grammar:


set-cookie-header = "Set-Cookie:" SP set-cookie-string
set-cookie-string = cookie-pair *( ";" SP cookie-av )
cookie-pair       = cookie-name "=" cookie-value
cookie-name       = token
cookie-value      = token / *base64-character
base64-character  = ALPHA / DIGIT / "+" / "/" / "="
token             = <token, defined in [RFC2616], Section 2.2>


[BTW, the grammar loses indentation because the comment on
max-age-av is too long]

Your argument appears to be that the cookie-value production
doesn't need to reflect actual usage because the specification
does not include a separate section on how servers should parse
received Cookie values.  In other words, the spec is incomplete.

My argument is that server developers (like me) already have
implementations that parse the Cookie value in a similar way
that user agents parse the Set-Cookie value, largely because
those values are set by third-party modules that the server
has no real control over.  BTW, server implementations like
libapreq <http://httpd.apache.org/apreq/> will only allow
embedded whitespace in value if it is a quoted-string.

Therefore, I would like you to change the ABNF so that it
reflects the reality of (Set-)Cookie usage on the Internet,
for the same reason that you have insisted the algorithm
for user agent parsing reflects reality.  Changing the ABNF
to include base64 does not do that -- it is just another
fantasy production that differs from all prior specs of
the cookie algorithm.  Changing it to

 cookie-value      = %x21-2B / %x2D-3A / %x3C-7E / %x80-FF

or just the minimum

 cookie-value      = %x21-2B / %x2D-3A / %x3C-7E

returns the definition to the original Netscape spec (at
least in the first case), reflects how they are implemented
on the Internet, and eliminates this artificial distinction
between the server and user agent requirements.

>>> I don't have any objection to a sentence that says servers
>>> SHOULD NOT send %x80-FF even though the grammar allows it.
>>> But I don't see any need for it either given the apparent
>>> interoperability shown by user agents.
>> 
>> That's essentially what the current document says.  The only
>> difference is that it also recommends that servers not generate
>> characters such as %x24, whose role in cookies has been muddied by RFC
>> 2965.

I don't see any restrictions on '$' (it is a valid token char).

The current document says

 cookie-value      = token / *base64-character

which unnecessarily excludes DQUOTE, "(" , ")" , "<" , ">" ,
"@", ":" , "\" , "[" , "]" , "?" , "{" , and "}".  It excludes
Google's very common __utmz cookie because that cookie's format
uses both pipe "|" (only allowed in token) and equals "="
(only allowed in *base64-character).

I do not see any reason for this working group to invent a
new grammar that has no implementation experience and differs
from every single prior specification of (Set-)Cookies.

The IESG should not approve such an error for publication,
since it will just require us to start this process over.

Cheers,

Roy T. Fielding                            <http://roy.gbiv.com/>
Principal Scientist, Adobe          <http://adobe.com/enterprise>