APPSAWG                                                     M. Kucherawy
Internet-Draft                                                G. Shapiro
Intended status: Informational                              May 17, 2013
Expires: November 18, 2013


             Advice for Safe Handling of Malformed Messages
                  draft-ietf-appsawg-malformed-mail-04

Abstract

   Although Internet mail formats have been precisely defined since the
   1970s, authoring and handling software often show only mild
   conformance to the specifications.  The distributed and non-
   interactive nature of email has often prompted adjustments to
   receiving software, to handle these variations, rather than trying to
   gain better conformance by senders, since the receiving operator is
   primarily driven by complaining recipient users and has no authority
   over the sending side of the system.  Processing with such
   flexibility comes at some cost, since mail software is faced with
   decisions about whether or not to permit non-conforming messages to
   continue toward their destinations unaltered, adjust them to conform
   (possibly at the cost of losing some of the original message), or
   outright rejecting them.

   A core requirement for interoperability is that both sides of an
   exchange work from the same details and semantics.  By having
   receivers be flexible, beyond the specifications, there can be -- and
   often has been -- a good chance that a message will not be fully
   interoperable.  Worse, a well-established pattern of tolerance for
   variations can sometimes be used as an attack vector.

   This document includes a collection of the best advice available
   regarding a variety of common malformed mail situations, to be used
   as implementation guidance.  It must be emphasized, however, that the
   intent of this document is not to standardize malformations or
   otherwise encourage their proliferation.  The messages are manifestly
   malformed, and the code and culture that generates them needs to be
   fixed.  Therefore, these messages should be rejected outright if at
   all possible.  Nevertheless, many malformed messages from otherwise
   legitimate senders are in circulation and will be for some time, and,
   unfortunately, commercial reality shows that we cannot always simply
   reject or discard them.  Accordingly, this document presents
   alternatives for dealing with them in ways that seem to do the least
   additional harm until the infrastructure is tightened up to match the
   standards.

Status of This Memo



Kucherawy & Shapiro     Expires November 18, 2013               [Page 1]

Internet-Draft             Safe Mail Handling                   May 2013


   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at http://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on November 18, 2013.

Copyright Notice

   Copyright (c) 2013 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (http://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.






















Kucherawy & Shapiro     Expires November 18, 2013               [Page 2]

Internet-Draft             Safe Mail Handling                   May 2013


Table of Contents

   1.  Introduction . . . . . . . . . . . . . . . . . . . . . . . . .  4
     1.1.  The Purpose Of This Work . . . . . . . . . . . . . . . . .  4
     1.2.  Not The Purpose Of This Work . . . . . . . . . . . . . . .  4
     1.3.  General Considerations . . . . . . . . . . . . . . . . . .  5
   2.  Document Conventions . . . . . . . . . . . . . . . . . . . . .  5
     2.1.  Examples . . . . . . . . . . . . . . . . . . . . . . . . .  5
   3.  Background . . . . . . . . . . . . . . . . . . . . . . . . . .  6
   4.  Internal Representations . . . . . . . . . . . . . . . . . . .  6
   5.  Invariant Content  . . . . . . . . . . . . . . . . . . . . . .  6
   6.  Mail Submission Agents . . . . . . . . . . . . . . . . . . . .  7
   7.  Line Terminaton  . . . . . . . . . . . . . . . . . . . . . . .  7
   8.  Header Anomalies . . . . . . . . . . . . . . . . . . . . . . .  8
     8.1.  Converting Obsolete and Invalid Syntaxes . . . . . . . . .  8
       8.1.1.  Host-Address Syntax  . . . . . . . . . . . . . . . . .  8
       8.1.2.  Excessive Angle Brackets . . . . . . . . . . . . . . .  8
       8.1.3.  Unbalanced Angle Brackets  . . . . . . . . . . . . . .  9
       8.1.4.  Unbalanced Parentheses . . . . . . . . . . . . . . . .  9
       8.1.5.  Commas in Address Lists  . . . . . . . . . . . . . . .  9
       8.1.6.  Unbalanced Quotes  . . . . . . . . . . . . . . . . . .  9
       8.1.7.  Naked Local-Parts  . . . . . . . . . . . . . . . . . . 10
     8.2.  Non-Header Lines . . . . . . . . . . . . . . . . . . . . . 10
     8.3.  Unusual Spacing  . . . . . . . . . . . . . . . . . . . . . 12
     8.4.  Header Malformations . . . . . . . . . . . . . . . . . . . 12
     8.5.  Header Field Counts  . . . . . . . . . . . . . . . . . . . 13
     8.6.  Missing Header Fields  . . . . . . . . . . . . . . . . . . 14
     8.7.  Missing or Incorrect Charset Information . . . . . . . . . 15
     8.8.  Eight-Bit Data . . . . . . . . . . . . . . . . . . . . . . 16
   9.  MIME Anomalies . . . . . . . . . . . . . . . . . . . . . . . . 16
     9.1.  Missing MIME-Version Field . . . . . . . . . . . . . . . . 17
     9.2.  Faulty Encodings . . . . . . . . . . . . . . . . . . . . . 17
   10. Body Anomalies . . . . . . . . . . . . . . . . . . . . . . . . 17
     10.1. Oversized Lines  . . . . . . . . . . . . . . . . . . . . . 17
   11. Security Considerations  . . . . . . . . . . . . . . . . . . . 18
   12. IANA Considerations  . . . . . . . . . . . . . . . . . . . . . 18
   13. References . . . . . . . . . . . . . . . . . . . . . . . . . . 18
     13.1. Normative References . . . . . . . . . . . . . . . . . . . 18
     13.2. Informative References . . . . . . . . . . . . . . . . . . 18
   Appendix A.  Acknowledgements  . . . . . . . . . . . . . . . . . . 19











Kucherawy & Shapiro     Expires November 18, 2013               [Page 3]

Internet-Draft             Safe Mail Handling                   May 2013


1.  Introduction

1.1.  The Purpose Of This Work

   The history of email standards, going back to [RFC733] and beyond,
   contains a fairly rigid evolution of specifications.  But
   implementations within that culture have also long had an
   undercurrent known formally as the robustness principle, but also
   known informally as Postel's Law: "Be conservative in what you do, be
   liberal in what you accept from others."

   Jon Postel's directive is often misinterpreted to mean that any
   deviance from a specification is acceptable.  Rather, it was intended
   only to account for legitimate variations in interpretation within
   specifications, as well as basic transit errors, like bit errors.
   Taken to its unintended extreme, excessive tolerance would imply that
   there are no limits to the liberties that a sender might take, while
   presuming a burden on a receiver to guess "correctly" at the meaning
   of any such variation.  These matters are further compounded by
   flawed receiver software -- the end users' mail readers -- which are
   also sometimes flawed, leaving senders to craft messages (sometimes
   bending the rules) to overcome those flaws.

   In general, this served the email ecosystem well by allowing a few
   errors in implementations without obstructing participation in the
   game.  The proverbial bar was set low.  However, as we have evolved
   into the current era, some of these lenient stances have begun to
   expose opportunities that can be exploited by malefactors.  Various
   email-based applications rely on strong application of these
   standards for simple security checks, while the very basic building
   blocks of that infrastructure, intending to be robust, fail utterly
   to assert those standards.

   This document presents some areas in which the more lenient stances
   can provide vectors for attack, and then presents the collected
   wisdom of numerous applications in and around the email ecosystem for
   dealing with them to mitigate their impact.

1.2.  Not The Purpose Of This Work

   It is important to understand that this work is not an effort to
   endorse or standardize certain common malformations.  The code and
   culture that introduces such messages into the mail stream needs to
   be repaired, as the security penalty now being paid for this lax
   processing arguably outweighs the reduction in support costs to end
   users who are not expected to understand the standards.  However, the
   reality is that this will not be fixed quickly.




Kucherawy & Shapiro     Expires November 18, 2013               [Page 4]

Internet-Draft             Safe Mail Handling                   May 2013


   Given this, it is beneficial to provide implementers with guidance
   about the safest or most effective way to handle malformed messages
   when they arrive, taking into consideration the tradeoffs of the
   choices available especially with respect to how various actors in
   the email ecosystem respond to such messages in terms of handling,
   parsing, or rendering to end users.

1.3.  General Considerations

   Many deviations from message format standards are considered by some
   receivers to be strong indications that the message is undesirable,
   i.e., is spam or contains malware.  Such receivers quickly decide
   that the best handling choice is simply to reject or discard the
   message.  This means malformations caused by innocent
   misunderstandings or ignorance of proper syntax can cause messages
   with no ill intent also to fail to be delivered.

   Senders that want to ensure message delivery are best advised to
   adhere strictly to the relevant standards (including, but not limited
   to, [MAIL], [MIME], and [DKIM]), as well as observe other industry
   best practices such as may be published from time to time either by
   the IETF or independently.

   Receivers that haven't the luxury of strict enforcement of the
   standards on inbound messages are usually best served by observing
   the following guidelines for handling of malformed messages:

   1.  Whenever possible, mitigation of syntactic malformations should
       be guided by an assessment of the most likely semantic intent.
       For example, it is reasonable to conclude that multiple sets of
       angle brackets around an address are simply superflous and can be
       dropped.

   2.  When the intent is unclear, or when it is clear but also
       impractical to change the content to reflect that intent,
       mitigation should be limited to cases where not taking any
       corrective action would clearly lead to a worse outcome.

   3.  Security issues, when present, need to be addressed and may force
       mitigation strategies that are otherwise suboptimal.

2.  Document Conventions

2.1.  Examples

   Examples of message content include a number within braces at the end
   of each line.  These are line numbers for use in subsequent
   discussion, and are not actually part of the message content



Kucherawy & Shapiro     Expires November 18, 2013               [Page 5]

Internet-Draft             Safe Mail Handling                   May 2013


   presented in the example.

   Blank lines are not numbered in the examples.

3.  Background

   The reader would benefit from reading [EMAIL-ARCH] for some general
   background about the overall email architecture.  Of particular
   interest is the Internet Message Format, detailed in [MAIL].
   Throughout this document, the use of the term "message" should be
   assumed to mean a block of text conforming to the Internet Message
   Format.

4.  Internal Representations

   Any agent handling a message could have one or two (or more) distinct
   representations of a message it is handling.  One is an internal
   representation, such as a block of storage used for the header and a
   block for the body.  These may be sorted, encoded, decoded, etc., as
   per the needs of that particular module.  The other is the
   representation that is output to the next agent in the handling
   chain.  This might be identical to the version that is input to the
   module, or it might have some changes such as added or reordered
   header fields, body modifications to remove malicious content, etc.

   In most cases, advice is provided only for internal representations.
   However, it is sometimes necessary to make changes between the input
   and output versions as well.  For the most part, such advice is
   avoided here; the goal is to ensure consistency among a set of
   transport and analysis agents, not necessarily the presentation of a
   "cleaned" message to the end user.

5.  Invariant Content

   An especially interesting handling sequence occurs within the
   destination Administrative Management Domain (ADMD; see
   [EMAIL-ARCH]).  From ingress to the ADMD, through the boundary agent,
   until delivery to the end user, it is beneficial to ensure that each
   agent sees an identical form for the message.  Absent this, it can be
   impossible for different agents in the chain to make consistent
   assertions about the content.

   For example, suppose a handling agent records that a message had some
   specific set of properties at ingress to the ADMD, then permitted it
   to continue inbound.  Some other agent alters the content for some
   reason.  The user, on viewing the delivered content, reports the
   message as abuse.  However, report processing often takes place at,
   or close to, the original point of ingress and is likely to be based



Kucherawy & Shapiro     Expires November 18, 2013               [Page 6]

Internet-Draft             Safe Mail Handling                   May 2013


   on the set of properties recorded there, rather at the end user's
   system.  This could render the complaint inactionable.  Similarly, a
   message that originally had properties a filtering agent would use to
   reject might not be rejected, and thus reach an end user, if an
   intermediate agent changed the message in a manner that alters one of
   those properties.

   Therefore, agents within an integrated message processing environment
   will simplify operational concerns by ensuring that each agent
   receives the same content -- except for the usual handling agent
   trace information additions -- and that this is what reaches the end
   user, unmodified.  Various policies, such as special handling for
   detected message abuse, will make exceptions appropriate.

   Note that some changes to a message alter syntax without changing
   semantics.  (Indeed, analyzing the semantics of malformations is the
   impetus for this work.)  For example, Section 8.4 describes a
   situation in which additional whitespace in a header is removed.
   This is a change in syntax without a change in semantics, though some
   systems (e.g., DKIM) are sensitive to such changes.  Care must be
   taken when developing message handling systems to be aware of the
   downstream impact of making either kind of change.

6.  Mail Submission Agents

   Within the email context, the single most influential component that
   can reduce the presence of malformed items in the email system is the
   Mail Handling Service (MHS; see [EMAIL-ARCH]).  This is the component
   that is essentially the interface between end users that create
   content and the mail stream.

   MHSes need to become more strict about enforcement of all relevant
   email standards, especially [MAIL] and the [MIME] family of
   documents.

   More strict conformance by relaying MTAs will also be helpful.
   although preventing the dissemination of malformed messages is
   desirable, the rejection of such mail already in transit also has a
   support cost, namely the creation of a [DSN] that many end users
   might not understand.

7.  Line Terminaton

   For interoperable Internet Mail messages, the only valid line
   separation sequence in messaging is ASCII 0x0D ("carriage return", or
   CR) followed by ASCII 0x0A ("line feed", or LF), commonly referred to
   as CRLF.  Common UNIX user tools, however, typically only use LF for
   internal line termination.  This means that a protocol engine, which



Kucherawy & Shapiro     Expires November 18, 2013               [Page 7]

Internet-Draft             Safe Mail Handling                   May 2013


   converts between UNIX and Internet Mail formats, has to convert
   between these two end-of-line representations before transmitting a
   message or after receiving it.

   Non-compliant implementations can cause messages to be transmitted
   with a mix of line terminations, such as LF everywhere except CRLF
   only at the end of the message.  According to [SMTP] and [MAIL], this
   means the entire message actually exists on a single line.

   Within modern Internet Mail it is highly unlikely that an isolated CR
   or LF is valid in common ASCII text.  Furthermore [MIME] presents
   mechanisms for encoding content that actually does need to contain
   such an unusual character sequence.

   Thus, it will typically be safe and helpful to treat a naked CR or LF
   as equivalent to a CRLF when parsing a message.

8.  Header Anomalies

   This section covers common syntactical and semantic anomalies found
   in a message header, and presents preferred mitigations.

8.1.  Converting Obsolete and Invalid Syntaxes

   A message using an obsolete header syntax might confound an agent
   that is attempting to be robust in its handling of syntax variations.
   A bad actor could exploit such a weakness in order to get abuse or
   malicious content through a filter.  This section presents some
   examples of such variations.  Messages including them ought be
   rejected; where this is not possible, recommended internal
   interpretations are provided.

8.1.1.  Host-Address Syntax

   The following obsolete syntax attempts to specify source routing:

       To: <@example.net:fran@example.com>

   This means "send to fran@example.com via the mail service at
   example.net".  It can safely be interpreted as:

       To: <fran@example.com>

8.1.2.  Excessive Angle Brackets

   The following over-use of angle brackets, e.g.:

       To: <<<user2@example.org>>>



Kucherawy & Shapiro     Expires November 18, 2013               [Page 8]

Internet-Draft             Safe Mail Handling                   May 2013


   can safely be interpreted as:

       To: <user2@example.org>

8.1.3.  Unbalanced Angle Brackets

   The following use of unbalanced angle brackets:

       To: <another@example.net
       To: second@example.org>

   can usually be treated as:

       To: <another@example.net>
       To: second@example.org

8.1.4.  Unbalanced Parentheses

   The following use of unbalanced parentheses:

       To: (Testing <fran@example.com>
       To: Testing) <sam@example.com>

   should be interpreted as:

       To: (Testing) <fran@example.com>
       To: "Testing)" <sam@example.com>

8.1.5.  Commas in Address Lists

   This use of an errant comma:

       To: <third@example.net, fourth@example.net>

   can reasonably be interpreted as ending an address, so the above
   should really be interpreted as:

       To: third@example.net, fourth@example.net

8.1.6.  Unbalanced Quotes

   The following use of unbalanced quotation marks:

       To: "Joe <joe@example.com>

   leaves software with no obvious "good" interpretation.  If it is
   essential to extract an address from the above, one possible
   interpretation is:



Kucherawy & Shapiro     Expires November 18, 2013               [Page 9]

Internet-Draft             Safe Mail Handling                   May 2013


       To: "Joe <joe@example.com>"@example.net

   where "example.net" is the domain name or host name of the handling
   agent making the interpretation.  Another possible interpretation is
   simply:

       To: "Joe" <joe@example.com&gt

8.1.7.  Naked Local-Parts

   [MAIL] defines a local-part as the user portion of an email address,
   and the display-name as the "user-friendly" label that accompanies
   the address specification.

   Some broken submission agents might introduce messages with only a
   local-part or only a display-name and no properly formed address.
   For example:

       To: Joe

   A submission agent ought to reject this or, at a minimum, append "@"
   followed by its own host name or some other valid name likely to
   enable a reply to be delivered to the correct mailbox.  Where this is
   not done, an agent receiving such a message will probably be
   successful by synthesizing a valid header field for evaluation using
   the techniques described in Section 8.6.

8.2.  Non-Header Lines

   Some messages contain a line of text in the header that is not a
   valid message header field of any kind.  For example:

       From: user@example.com {1}
       To: userpal@example.net {2}
       Subject: This is your reminder {3}
       about the football game tonight {4}
       Date: Wed, 20 Oct 2010 20:53:35 -0400 {5}

       Don't forget to meet us for the tailgate party! {7}

   The cause of this is typically a bug in a message generator of some
   kind.  Line {4} was intended to be a continuation of line {3}; it
   should have been indented by whitespace as set out in Section 2.2.3
   of [MAIL].

   This anomaly has varying impacts on processing software, depending on
   the implementation:




Kucherawy & Shapiro     Expires November 18, 2013              [Page 10]

Internet-Draft             Safe Mail Handling                   May 2013


   1.  some agents choose to separate the header of the message from the
       body only at the first empty line (i.e. a CRLF immediately
       followed by another CRLF);

   2.  some agents assume this anomaly should be interpreted to mean the
       body starts at line {4}, as the end of the header is assumed by
       encountering something that is not a valid header field or folded
       portion thereof;

   3.  some agents assume this should be interpreted as an intended
       header folding as described above and thus simply append a single
       space character (ASCII 0x20) and the content of line {4} to that
       of line {3};

   4.  some agents reject this outright as line {4} is neither a valid
       header field nor a folded continuation of a header field prior to
       an empty line.

   This can be exploited if it is known that one message handling agent
   will take one action while the next agent in the handling chain will
   take another.  Consider, for example, a message filter that searches
   message headers for properties indicative of abusive of malicious
   content that is attached to a Mail Transfer Agent (MTA) implementing
   option 2 above.  An attacker could craft a message that includes this
   malformation at a position above the property of interest, knowing
   the MTA will not consider that content part of the header, and thus
   the MTA will not feed it to the filter, thus avoiding detection.
   Meanwhile, the Mail User Agent (MUA) which presents the content to an
   end user, implements option 1 or 3, which has some undesirable
   effect.

   It should be noted that a few implementations choose option 4 above
   since any reputable message generation program will get header
   folding right, and thus anything so blatant as this malformation is
   likely an error caused by a malefactor.

   The preferred implementation if option 4 above is not employed is to
   apply the following heuristic when this malformation is detected:

   1.  Search forward for an empty line.  If one is found, then apply
       option 3 above to the anomalous line, and continue.

   2.  Search forward for another line that appears to be a new header
       field, i.e., a name followed by a colon.  If one is found, then
       apply option 3 above to the anomalous line, and continue.






Kucherawy & Shapiro     Expires November 18, 2013              [Page 11]

Internet-Draft             Safe Mail Handling                   May 2013


8.3.  Unusual Spacing

   The following message is valid per [MAIL]:

       From: user@example.com {1}
       To: userpal@example.net {2}
       Subject: This is your reminder {3}
        {4}
        about the football game tonight {5}
       Date: Wed, 20 Oct 2010 20:53:35 -0400 {6}

       Don't forget to meet us for the tailgate party! {8}

   Line {4} contains a single whitespace.  The intended result is that
   lines {3}, {4}, and {5} comprise a single continued header field.
   However, some agents are aggressive at stripping trailing whitespace,
   which will cause line {4} to be treated as an empty line, and thus
   the separator line between header and body.  This can affect header-
   specific processing algorithms as described in the previous section.

   This example was legal in earlier versions of the Internet Mail
   format standard.

   The best handling of this example is for a message parsing engine to
   behave as if line {4} was not present in the message and for a
   message creation engine to emit the message with line {4} removed.

8.4.  Header Malformations

   Among the many possible malformations, a common one is insertion of
   whitespace at unusual locations, such as:

       From: user@example.com {1}
       To: userpal@example.net {2}
       Subject: This is your reminder {3}
       MIME-Version : 1.0 {4}
       Content-Type: text/plain {5}
       Date: Wed, 20 Oct 2010 20:53:35 -0400 {6}

       Don't forget to meet us for the tailgate party! {8}

   Note the addition of whitespace in line {4} after the header field
   name but before the colon that separates the name from the value.

   The acceptance grammar of [MAIL] permits that extra whitespace, so it
   cannot be considered invalid.  However, a consensus of
   implementations prefers to remove that whitespace.  There is no
   perceived change to the semantics of the header field being altered



Kucherawy & Shapiro     Expires November 18, 2013              [Page 12]

Internet-Draft             Safe Mail Handling                   May 2013


   as the whitespace is itself semantically meaningless.  Therefore, it
   is best to remove all whitespace after the field name but before the
   colon and to emit the field in this modified form.

8.5.  Header Field Counts

   Section 3.6 of [MAIL] prescribes specific header field counts for a
   valid message.  Few agents actually enforce these in the sense that a
   message whose header contents exceed one or more limits set there are
   generally allowed to pass; they typically add any required fields
   that are missing, however.

   Also, few agents that use messages as input, including Mail User
   Agents (MUAs) that actually display messages to users, verify that
   the input is valid before proceeding.  Some popular open source
   filtering programs and some popular Mailing List Management (MLM)
   packages select either the first or last instance of a particular
   field name, such as From, to decide who sent a message.  Absent
   strict enforcement of [MAIL], an attacker can craft a message with
   multiple fields if that attacker knows the filter will make a
   decision based on one but the user will be shown the other.

   This situation is exacerbated when message validity is assessed, such
   as through enhanced authentication methods.  Such methods might cover
   one instance of a constrained field but not another, taking the wrong
   one as "good" or "safe".  An MUA, for example could show the first of
   two From fields to an end user as "good" or "safe" while an
   authentication method actually only verified the second.

   In attempting to counter this exposure, one of the following can be
   enacted:

   1.  reject outright or refuse to process further any input message
       that does not conform to Section 3.6 of [MAIL];

   2.  remove or, in the case of an MUA, refuse to render any instances
       of a header field whose presence exceeds a limit prescribed in
       Section 3.6 of [MAIL] when generating its output;

   3.  where a field has a limited instance count, combine additional
       instances into a single compound instance;

   4.  alter the name of any header field whose presence exceeds a limit
       prescribed in Section 3.6 of [MAIL] when generating its output so
       that later agents can produce a consistent result.  Any
       alteration likely to cause the field to be ignored by downstream
       agents is acceptable.  A common approach is to prefix the field
       names with a string such as "BAD-".



Kucherawy & Shapiro     Expires November 18, 2013              [Page 13]

Internet-Draft             Safe Mail Handling                   May 2013


   Selecting a mitigation action from the above list, or some other
   action, must consider the needs of the operator making the decision,
   and the nature of its user base.

8.6.  Missing Header Fields

   Similar to the previous section, there are messages seen in the wild
   that lack certain required header fields.  In particular, [MAIL]
   requires that a From and Date field be present in all messages.

   When presented with a message lacking these fields, the MTA might
   perform one of the following:

   1.  Make no changes

   2.  Add an instance of the missing field(s) using synthesized content
       based on data provided in other parts of the protocol

   Option 2 is recommended for handling this case.  Handling agents
   should add these for internal hanlding if they are missing, but
   should not add them to the external representation.  The reason for
   this advice is that there are some filter modules that would consider
   the absence of such fields to be a condition warranting special
   treatment (e.g., rejection), and thus the effectiveness of such
   modules would be stymied by an upstream filter adding them in a way
   visible to other components.

   The synthesized fields should contain a best guess as to what should
   have been there; for From, the SMTP MAIL command's address can be
   used (if not null) or a placeholder address followed by an address
   literal (e.g., unknown@[192.0.2.1]); for Date, a date extracted from
   a Received field is a reasonable choice.

   One other important case to consider is a missing Message-Id field.
   An MTA that encounters a message missing this field should synthesize
   a valid one using techniques described above and add it to the
   external rpresentation, since many deployed tools use the content of
   that field as a common unique message reference, so its absence
   inhibits correlation of message processing.  One possible synthesis
   would be based on based on an encoding of the current date/time and
   an internal MTA ID (e.g., queue ID) followed by @ and the fully
   qualified hostname of the machine synthesizing the header value.  For
   example:

       tm = gmtime(&now);
       (void) snprintf(buf, sizeof(buf), "%04d%02d%02d%02d%02d.%s@%s",
                       tm->tm_year + 1900, tm->tm_mon + 1, tm->tm_mday,
                       tm->tm_hour, tm->tm_min, queueID, fqhn);



Kucherawy & Shapiro     Expires November 18, 2013              [Page 14]

Internet-Draft             Safe Mail Handling                   May 2013


8.7.  Missing or Incorrect Charset Information

   MIME provides the means to include textual material employing
   charsets other than US-ASCII.  Such material is required to have an
   identifiable charset.  Charset identification is done using a
   "charset" parameter in the Content-Type header field, a character set
   label within the MIME entity itself, or the character set may be
   implicitly specified by the Content-Type (see [CHARSET]).

   It is unfortunately fairly common for required character set
   information to be missing or incorrect in textual MIME entities.  As
   such, processing agents should perform basic sanity checks, e.g.:

   o  US-ASCII is 7bit only, so 8bit material is necessarily not US-
      ASCII.

   o  UTF-8 has a very specific syntactic structure that other 8bit
      charsets are unlikely to follow.

   o  Null bytes (ASCII 0x00) are not allowed in either 7bit or 8bit
      data.

   o  Not all 7bit material is US-ASCII.  The presence of the various
      escape sequences used for character switching may be used as an
      indication of the various ISO-2022 charsets.

   When a character set error is detected, processing agents should:

   a.  apply heuristics to determine the most likely character set and,
       if successful, proceed using that information; or

   b.  refuse to process the malformed MIME entity.

   A null byte inside a textual MIME entity can cause typical string
   processing functions to mis-identify the end of a string, which can
   be exploited to hide malicious content from analysis processes.
   Accordingly, null bytes require additional special handling.

   A few null bytes in isolation is likely to be the result of poor
   message construction practices.  Such nulls should be silently
   dropped.

   Large numbers of null bytes are usually the result of binary material
   that is improperly encoded, improperly labeled, or both.  Such
   material is likely to be damaged beyond the hope of recovery, so the
   best course of action is to refuse to process it.

   Finally, the presence of null bytes may be used as indication of



Kucherawy & Shapiro     Expires November 18, 2013              [Page 15]

Internet-Draft             Safe Mail Handling                   May 2013


   possible malicious intent.

8.8.  Eight-Bit Data

   Standards-compliant email messages do not contain any non-ASCII data
   without indicating that such content is present by means of published
   SMTP extensions.  Absent that, MIME encodings are typically used to
   convert non-ASCII data to ASCII in a way that can be reversed by
   other handling agents or end users.

   The best way to handle non-compliant 8bit material depends on its
   location.

   Non-compliant 8bit in MIME entity content should simply be processed
   as if the necessary SMTP extensions had been used to transfer the
   message.  Note that improperly labeled 8bit material in textual MIME
   entities may require treatment as described in Section 8.7.

   Non-compliant 8bit in message or MIME entity header fields can be
   handled as follows:

   o  Occurrences in unstructured text fields, comments, and phrases,
      can be converted into encoded-words (see [MIME3] if a likely
      character set can be determined.  Alternatively, 8bit characters
      can be removed or replaced with some other character.

   o  Occurrences in header fields whose syntax is unknown may be
      handled by dropping the field entirely or by removing/replacing
      the 8bit character as described above.

   o  Occurrences in addresses are especially problematic.  Agents
      supporting [EAI] may, if the 8bit conforms to 8bit syntax, elect
      to treat the messages as an EAI message and process it
      accordingly.  Otherwise, it is in most cases best to exclude the
      address from any sort of processing -- which may mean dropping it
      entirely -- since any attempt to fix interpret it definitively is
      unlikely to be successful.

9.  MIME Anomalies

   [MIME], et seq, include a mechanism of message extensions for
   providing text in character sets other than ASCII, non-text
   attachments to messages, multi-part message bodies, and similar
   facilities.

   Some anomalies with MIME-compliant generation are also common.  This
   section discusses some of those and presents preferred mitigations.




Kucherawy & Shapiro     Expires November 18, 2013              [Page 16]

Internet-Draft             Safe Mail Handling                   May 2013


9.1.  Missing MIME-Version Field

   Any message that uses [MIME] constructs is required to have a MIME-
   Version header field.  Without it, the Content-Type and associated
   fields have no semantic meaning.

   It is often observed that a message has complete MIME structure, yet
   lacks this header field.  It is prudent to disregard this absence and
   conduct analysis of the message as if it were present, especially by
   agents attempting to identify malicious material.

   Further, the absence of MIME-Version might be an indication of
   malicious intent, and extra scrutiny of the message may be warranted.
   Such omissions are not expected from compliant message generators.

9.2.  Faulty Encodings

   There have been a few different specifications of base64 in the past.
   The implemenation defined in [MIME] instructs decoders to discard
   characters that are not part of the base64 alphabet.  Other
   implementations consider an encoded body containing such characters
   to be completely invalid.  Very early specifications of base64 (see
   [PEM], for example) allowed email-style comments within base64-
   encoded data.

   The attack vector here involves constructing a base64 body whose
   meaning varies given different possible decodings.  If a security
   analysis module wishes to be thorough, it should consider scanning
   the possible outputs of the known decoding dialects in an attempt to
   anticipate how the MUA will interpret the data.

10.  Body Anomalies

10.1.  Oversized Lines

   A message containing a line of content that exceeds 998 characters
   plus the line terminator (1000 total) violates Section 2.1.1 of
   [MAIL].  Some handling agents may not look at content in a single
   line past the first 998 bytes, providing bad actors an opportunity to
   hide malicious content.

   There is no specified way to handle such messages, other than to
   observe that they are non-compliant and reject them, or rewrite the
   oversized line such that the message is compliant.

   Handling agents MUST take one of the following actions:





Kucherawy & Shapiro     Expires November 18, 2013              [Page 17]

Internet-Draft             Safe Mail Handling                   May 2013


   1.  Break such lines into multiple lines at a position that does not
       change the semantics of the text being thus altered.  For
       example, breaking an oversized line such that a [URI] then spans
       two lines could inhibit the proper identification of that URI.

   2.  Rewrite the MIME part (or the entire message if not MIME) that
       contains the excessively long line using a content encoding that
       breaks the line in the transmission but would still result in the
       line being intact on decoding for presentation to the user.  Both
       of the encodings declared in [MIME] can accomplish this.

11.  Security Considerations

   The discussions of the anomalies above and their prescribed solutions
   are themselves security considerations.  The practises enumerated in
   this document are generally perceived as attempts to resolve security
   considerations that already exist rather than introducing new ones.
   However, some of the attacks described here may not have appeared in
   previous email specifications.

12.  IANA Considerations

   This document contains no actions for IANA.

   [RFC Editor: Please remove this section prior to publication.]

13.  References

13.1.  Normative References

   [MAIL]        Resnick, P., "Internet Message Format", RFC 5322,
                 October 2008.

13.2.  Informative References

   [CHARSET]     Melnikov, A. and J. Reschke, "Update to MIME regarding
                 "charset" Parameter Handling in Textual Media Types",
                 RFC 6657, July 2012.

   [DKIM]        Allman, E., Callas, J., Delany, M., Libbey, M., Fenton,
                 J., and M. Thomas, "DomainKeys Identified Mail (DKIM)
                 Signatures", RFC 4871, May 2007.

   [DSN]         Moore, K. and G. Vaudreuil, "An Extensible Message
                 Format for Delivery Status Notifications", RFC 3464,
                 January 2003.

   [EAI]         Yang, A., Steele, S., and N. Freed, "Internationalized



Kucherawy & Shapiro     Expires November 18, 2013              [Page 18]

Internet-Draft             Safe Mail Handling                   May 2013


                 Email Headers", RFC 6532, February 2012.

   [EMAIL-ARCH]  Crocker, D., "Internet Mail Architecture", RFC 5598,
                 July 2009.

   [MIME]        Freed, N. and N. Borenstein, "Multipurpose Internet
                 Mail Extensions (MIME) Part One: Format of Internet
                 Message Bodies", RFC 2045, November 1996.

   [MIME3]       Moore, K., "MIME (Multipurpose Internet Mail
                 Extensions) Part Three: Message Header Extensions for
                 Non-ASCII Text", RFC 2047, November 1996.

   [PEM]         Linn, J., "Privacy Enhancement for Internet Electronic
                 Mail: Part I -- Message Encipherment and Authentication
                 Procedures", RFC 1113, August 1989.

   [RFC733]      Crocker, D., Vittal, J., Pogran, K., and D. Henderson,
                 Jr., "Standard for the Format of Internet Text
                 Messages", RFC 733, November 1977.

   [SMTP]        Klensin, J., "Simple Mail Transfer Protocol", RFC 5321,
                 October 2008.

   [URI]         Berners-Lee, T., Fielding, R., and L. Masinter,
                 "Uniform Resource Identifier (URI): Generic Syntax",
                 RFC 3986, January 2005.

Appendix A.  Acknowledgements

   The author wishes to acknowledge the following for their review and
   constructive criticism of this proposal: Dave Crocker, Ned Freed,
   Tony Hansen, Franck Martin, and Timo Serainen

Authors' Addresses

   Murray S. Kucherawy

   EMail: superuser@gmail.com


   Gregory N. Shapiro

   EMail: gshapiro@sendmail.com







Kucherawy & Shapiro     Expires November 18, 2013              [Page 19]