In reference to
https://svn.tools.ietf.org/svn/wg/appsawg/draft-ietf-appsawg-mime-default-charset/latest/draft-ietf-appsawg-mime-default-charset.html

First of all, thank you for finally taking on the much-needed update
to RFC 2046 rules.

Unfortunately, the draft doesn't address the problem the right way in
my opinion. Quotes from the draft.

> Each subtype of the "text" media type which uses the "charset" parameter can define its own default value for the "charset" parameter, including absence of any default.

Additionally, media types should be able to define circumstances where
in-band indicators override the charset parameter even if the charset
parameter is present.

In particular, media types should be allowed to override the charset
parameter if the first two or three bytes of the payload look like an
UTF-16 or UTF-8 BOM.

See:
https://www.w3.org/Bugs/Public/show_bug.cgi?id=15359
https://bugzilla.mozilla.org/show_bug.cgi?id=716579
https://bugzilla.mozilla.org/show_bug.cgi?id=687859

> In order to improve interoperability with deployed agents, "text/*" media type definitions SHOULD either a) specify that the "charset" parameter is not used for the defined subtype, because the charset information is transported inside the payload (as in "text/xml")

This seems wrong. If the charset parameter is present, it has an
effect for text/xml.

> or b) require explicit unconditional inclusion of the "charset" parameter eliminating the need for a default value.

This seems naïve. Formats need to specify what happens when a charset
parameter is missing, since no matter how much the format says it's
"required", the party sending data can omit the charset parameter.

> In accordance with option (a), above, "text/*" media types that can transport charset information inside the corresponding payloads, specifically including "text/html" and "text/xml", SHOULD NOT specify the use of a "charset" parameter, nor any default value, in order to avoid conflicting interpretations should the charset parameter value and the value specified in the payload disagree.

For backwards compatibility, pretty much every existing text/* type
will have to violate this "SHOULD NOT".

> New subtypes of the "text" media type, thus, SHOULD NOT define a default "charset" value. If there is a strong reason to do so despite this advice, they SHOULD use the "UTF-8" [RFC3629] charset as the default.

Seems reasonable.

> Specifications of how to specify the "charset" parameter, and what default value, if any, is used, are subtype-specific, NOT protocol-specific.

Seems reasonable.

> Protocols that use MIME, therefore, MUST NOT override default charset values for "text/*" media types to be different for their specific protocol. The protocol definitions MUST leave that to the subtype definitions.

Seems reasonable.

> The default charset parameter value for text/plain is unchanged from [RFC2046] and remains as "US-ASCII".

This is incompatible with reality. Web browsers, for instance, assume
a configuration-dependent default (which correlates with browser
localization) and may also (depending on configuration which, again,
correlates with localization by default) perform a heuristic analysis
on the payload.

I suggest specifying the following instead of sections 3 and 4 of the draft:

3. New rules for determining the character encoding for text/* media types

Each text/* media type MUST specify an algorithm for establishing the
character encoding of the entity body from the entity body (or
preferably the first N bytes thereof, preferably with N = 1024), the
charset parameter and Other Information. Other Information MAY include
configuration, an encoding label supplied by the referrer, the
previous encoding of an entity body retrieved from the same location
or the encoding of the referrer. New text/* media types MUST not use
Other Information in the algorithms they specify. New text/* media
types SHOULD use the following algorithm:

The character encoding is UTF-8. Terminate these steps.

4. Determining the character encoding for text/plain

If the first 2 octets of the entity body are 0xFE followed by 0xFF,
the character encoding is big-endian UTF-16. Terminate these steps.

If the first 2 octets of the entity body are 0xFF followed by 0xFE,
the character encoding is little-endian UTF-16. Terminate these steps.

If the first 3 octets of the entity body are 0xEF followed by 0xBB
followed by 0xBF, the character encoding is UTF-8. Terminate these
steps.

If the value of the charset parameter is a ASCII case-insensitive[1]
match for a label[2] of a supported encoding, the character encoding
is the encoding whose label was matched. Terminate these steps.

If the entity is being navigate to in a browsing context[3] and the
previous document had the same origin[4] as the text/plain entity, the
character encoding is the encoding of the referring document.
Terminate these steps. (Disclaimer: I'm not 100% sure that this step
is in the right order relative to the others.)

Optional: If a heuristic detector recognizes the octets of the entity
body as being encoding according to an encoding, the character
encoding is that encoding. Terminate these steps. This step SHOULD NOT
be implemented for locales where it has not been implemented
traditionally.

If the entity is being loaded into a nested browsing context that has
the same origin as the parent browsing context, the encoding is the
encoding of the document loaded in the parent browsing context.
Terminate these steps.

If the entity is being loaded into a browsing context and is being
fetched from a location from which an entity has been loaded before
and the previous character encoding has been cached, the character
encoding is the cached encoding. Terminate these steps.

If the entity is being loaded via a non-browsing context mechanism
(such as XMLHttpRequest) that defines a fallback encoding, use that
encoding. Terminate these steps.

Otherwise, the character encoding is a configuration-dependent
encoding. The default configuration SHOULD depend on the locale of the
user agent according to the table given in step 8 in [5]. Terminate
these steps.

[1] http://www.whatwg.org/specs/web-apps/current-work/#ascii-case-insensitive
[2] http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html#concept-encoding-label
[3] http://www.whatwg.org/specs/web-apps/current-work/#browsing-context
[4] http://tools.ietf.org/html/rfc6454
[5] http://www.whatwg.org/specs/web-apps/current-work/#determining-the-character-encoding

-- 
Henri Sivonen
hsivonen@iki.fi
http://hsivonen.iki.fi/