<?xml version="1.0" encoding="us-ascii"?>
<!DOCTYPE rfc SYSTEM "rfc2629.dtd">
<?rfc toc="yes"?>
<?rfc rfcedstyle="yes"?>
<?rfc subcompact="no"?>
<?rfc symrefs="yes"?>
<?rfc comments="yes" ?>
<?rfc inline="yes" ?>

<rfc ipr="trust200902" category="std" docName='draft-melnikov-smtp-metadata-01'>
  <front>
    <title abbrev="Metadata Transfer SMTP Extension">
      Simple Mail Transfer Protocol extension for relaying Metadata
    </title>
    <author initials="A." surname="Melnikov" fullname="Alexey Melnikov">
      <organization>Isode Ltd</organization>
      <address>
        <postal>
          <street>5 Castle Business Village</street>
          <street>36 Station Road</street>
          <city>Hampton</city>
          <region>Middlesex</region>
          <code>TW12 2BX</code>
          <country>UK</country>
        </postal>
        <email>Alexey.Melnikov@isode.com</email>
      </address>
    </author>
      
    <date year="2015"/>
    
    <keyword>SMTP</keyword>
    <keyword>BDAT</keyword>

    <abstract>
      <t>
   This memo defines an extension to the SMTP (Simple Mail Transfer Protocol)
   service whereby message metadata (such as Trace header fields, IMAP flags, Keying material, etc)
   can be transferred in separate containers similar to BDAT (RFC 3030, SMTP CHUNKING) command.
   This allows clean separation of transaction related state from the message itself.
      </t>
    </abstract>
    
  </front>
  <middle>
    <section title="Introduction">

      <t>
        This memo defines an extension to the SMTP (Simple Mail Transfer Protocol)
        service whereby message metadata (such as Trace header fields, IMAP flags, Keying material, etc)
        can be transferred in separate containers similar to BDAT (RFC 3030, SMTP CHUNKING) command.
        This allows clean separation of transaction related state from the message itself.
      </t>

    </section>
    
    <section title="Conventions Used in This Document">
      
      <t>The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
	    "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in
	    this document are to be interpreted as described in
	    <xref target="RFC2119"/> when they appear in ALL CAPS.
      These words also appear in this document in lower case as plain
      English words, absent their normative meanings.</t>
      
      <t>The formal syntax use the <xref target="RFC5234">Augmented
	  Backus-Naur Form (ABNF)</xref> notation including the core rules
	defined in Appendix B of <xref target="RFC5234">RFC 5234</xref>.
      </t>

<t>
In examples, "C:" and "S:" indicate lines sent by the client and
server respectively.  Line breaks that do not start with a new "C:" or
"S:" exist for editorial reasons and are not a part of the protocol.
</t>

    </section>

    
    <section title="Definition of the Metadata SMTP Extension">

	<t>The Metadata SMTP service extension is defined as follows:
	
	    <list style="numbers">
		
   <t>The textual name of this extension is "Metadata Transfer".</t>

   <t>The EHLO keyword value associated with this extension is "METADATA".
   Any server that advertises support for the "METADATA" extension MUST also support SMTP CHUNKING (RFC 3030).</t>

   <t>The EHLO keyword has no parameters
<!--////Add optional parameters for conveying list of metadata containers accepted? E.g.
   an OPTIONAL parameter which conveys names of metadata container types supported. (See the &lt;metadata-XXXX-ehlo&gt; ABNF non terminal
    in <xref target="abnf"/> for details of its syntax.) Absence of the parameter means that the server
    is unwilling to disclose any specific container types.
-->
    </t>

   <t>
     <cref>Should BMTD be allowed before the DATA command? There is no reason why not.</cref>
     A new SMTP verb, BMTD, is defined. The BMTD verb takes one argument,
     which indicates the length, in octets, of the binary metadata
     container that follows immediately after the command.
     See <xref target="bmtd-command"/> for the description of the BMTD command
     and <xref target="abnf"/> for its syntax.
   </t>

   <t>This extension doesn't add any new parameters to
      MAIL FROM or RCPT TO commands.</t>
        
   <t>The Metadata extension is valid for the submission service
      <xref target="RFC6409"/> and LMTP <xref target="RFC2033"/>.</t>

	    </list>
	</t>

  <section title="BMTD command" anchor="bmtd-command">

    <t>
      After all MAIL and RCPT responses are collected and processed, the
      message metadata is sent using a series of BMTD commands.
      The BMTD command takes one required argument, the exact length of the metadata segment ("container")
      in octets.  The metadata is sent immediately after the trailing &lt;CR&gt;
      &lt;LF&gt; of the BMTD command line.  Once the receiver-SMTP receives the
      specified number of octets, it will return a 250 reply code.
    </t>

    <t>
      BMTD commands MUST be sent before any BDAT <xref target="RFC3030"/> or
      BURL <xref target="RFC4468"/> commands. If a server encounters BMTD command
      after BDAT/BURL, it MUST respond with 503 "Bad sequence of commands" reply code.
      The state resulting from this error is indeterminate.  A RSET command
      MUST be sent to clear the transaction before continuing.
    </t>

    <t>
      Each BMTD container starts with 2 octet container type,
      followed by container type specific data.
      This means that the metadata segment length can never be the value 1
      (it can either be 0 or be equal or greater than 2).
    </t>

    <t>
      A 250 response MUST be sent to each successful BMTD data block ("chunk") within
      a mail transaction.  If a failure occurs after a BMTD command is
      received, the receiver-SMTP MUST accept and discard the associated
      metadata and message data before sending the appropriate 5XX or 4XX code.  If a
      5XX or 4XX code is received by the sender-SMTP in response to a BMTD
      chunk, the transaction should be considered failed and the sender-
      SMTP MUST NOT send any additional BMTD segments.  If the receiver-
      SMTP has declared support for command pipelining <xref target="RFC2920"/>, the receiver
      SMTP MUST be prepared to accept and discard additional BDAT/BURL/BMTD chunks
      already in the pipeline after the failed BMTD.
    </t>

    <t>
      Note: An error on the receiver-SMTP such as disk full or imminent
      shutdown can only be reported after the BMTD segment has been
      received.  It is therefore important to choose a reasonable chunk
      size given the expected end-to-end bandwidth.
    </t>

    <t>
      Note:  Because the receiver-SMTP does not acknowledge the BMTD
      command before the message data is sent, it is important to send
      the BMTD only to systems that have declared their capability to
      accept BMTD commands.  Illegally sending a BMTD command and
      associated message data to a non-METADATA capable system will
      result in the receiver-SMTP parsing the associated message data as
      if it were a potentially very long, ESMTP command line containing
      binary data.
    </t>

    <t>
      More than one BMTD command can occur in a transaction.
      (However some BMTD container types only allow for a single BMTD command
      with that particular container type.)
      Any BMTD command MUST be followed by one or more of BMTD/BDAT/BURL commands.
    </t>

      </section>

      <section title="Initial List of Metadata Container types">

        <t>
        Type 0: Trace header fields: Received, Return-Path, Authentication-Results (RFC 7001), etc
        encoded as if they are a part of a message header.
        Containers of this type can appear multiple types in a transaction.
        MUST be supported by all compliant servers.
        </t>

        <t>
        Type 1: IMAP Keywords <xref target="RFC3501"/> associated with the message (e.g. $MDNSent, $Forwarded, \Answered).
        This is a space separated list of IMAP keywords/flags.
        Container of this type MUST NOT appear more than once in a transaction.
        If the final LMTP delivers the message to an IMAP capable mailstore, it MUST
        attempt setting the listed IMAP keywords/flags on the message.
        Flags/keywords not supported by the mailstore (or disallowed when a message is injected
        via LMTP) MUST be ignored.
        </t>

        <t>Keying material, a la Dark Mail. TBD if there is interest.</t>
        
      </section>

      <section title="Requirements on a Metadata Container type definition">

        <t>
          Each container type definition MUST specify if it can appear more than once.
        </t>
        
        <t>
          Unless specified by an extension mutually agreed by SMTP sender and SMTP recipient,
          no container type can be defined as required (i.e. appearing at least once
          in a SMTP transaction) or define how it can be relayed to a non compliant MTA.
        </t>

        <t>
          Each container type definition MUST describe how it is going to be handled by the final MTA/LMTP server.
        </t>

      </section>


    </section>

    <section title="Handling of messages received via SMTP">
      
	<t>
	This section describes how a conforming SMTP server should handle any
	messages received via SMTP.
	</t>

	<section title="Relay of messages to other conforming SMTP/LMTP servers" anchor="relay-conform">
	    
	    <t>
	    The following rules govern the behavior of a conforming MTA (in the
	    role of an SMTP/LMTP client), when
	    relaying a message which was received via the SMTP protocol, to an
	    SMTP/LMTP server that supports the METADATA extension:

<list style="numbers">
    
		<t>
    Instead of prepending trace fields to the message itself as specified in RFC 5321,
    a relaying MTA SHOULD <cref>Cross check with RFC 5321 regarding insertion of Received header fields</cref>
    insert a single BMTD container of type 0 (Trace fields) containing its own trace header fields such as
    Received <xref target="RFC5321"/>, Authentication-Results <xref target="RFC7001"/>, etc.
    
    <!--////Multiple chunks can be inserted if there are multiple independent agents doing processing?-->
      
    <!--////Talk about firewalls stripping some type 0 containers?-->   
    </t>


    <t>
    All other BMTD commands are relayed to conforming SMTP/LMTP server in the order received.
    Intermediary servers SHOULD NOT coalesce or reorder metadata containers of type 0 or any other type that they understand.
    Intermediary servers MUST NOT coalesce, reorder or drop metadata containers of any types that they don't recognize.
    </t>
    
</list>

	    </t>

	</section>

  <section title="Relay of messages to non-conforming SMTP/LMTP servers" anchor="relay-non-conform">

	   <t>
	   The following rules govern the behavior of a conforming MTA (in the
	   role of an SMTP/LMTP client), when relaying a message which was received via the
	   SMTP protocol, to an SMTP/LMTP server that does not support the METADATA extension:

<list style="numbers">

    <t>Data from each metadata container of type 0 (Trace fields) MUST be extracted
    and prepended to the header of the message in the order of BMTD commands.</t>
  
    <t>
    All other BMTD chunks are discarded. <cref>OPEN ISSUE. They can also be converted to some magic header fields for logging and debugging?</cref>
    </t>

</list>

    </t>

  </section>



<!--////
	<section title="Mailing lists and Aliases">

<t>
  Several types of mechanisms exist to redirect or forward messages to
  alternative or multiple addresses <xref target="RFC5598"/>.
  Examples for this are aliases and mailing lists <xref target="RFC5321"/>.
</t>

<t>
  If a message is subject to such processing,
  the Mediator node (Section 2.1 of <xref target="RFC5598"/>),
  SHOULD pass BMTD chunks (or perform downconversion when relaying to
  SMTP servers which don't support the METADATA extension)
  for all expanded and/or translated addresses.
</t>
	    
	</section>
-->


	
	<section title="Gatewaying a message into a foreign environment">
	    
	  <t>
	The following rules govern the behavior of a conforming MTA, when
	gatewaying a message that was received via the SMTP protocol, into a
	foreign (non-SMTP) environment:

<list style="numbers">

	<t>
	If the destination environment is unable to provide an equivalent
	of the BMTD command, the conforming MTA SHOULD behave
	as if it is relaying to a non-conformant SMTP server (<xref target="relay-non-conform"/>).
	</t>
    
	<t>
	If the destination environment is capable of providing an equivalent
	of the BMTD command, the conforming MTA SHOULD behave
	as if it is relaying to a conformant SMTP server (<xref target="relay-conform"/>),
  converting any BMTD command to the equivalent in the destination environment.
  </t>

</list>

	  </t>
	
	</section>

    </section>

    <section title="Use of METADATA with LMTP">

      <t>An LMTP server can advertise support for the METADATA extension:

	      <list style="numbers">

        <t>
        Data from containers of type 0 (Trace fields) is extracted (in the order of the
        corresponding BMTD commands) and prepended to the header of the message.
        </t>

        <t>Handling of other container type is specific to the container type.</t>
          
        <t>
        Unsupported BMTD container types are discarded.
        <!--////Keep in sync with anchor="relay-non-conform"-->
        <cref>OPEN ISSUE. They can also be converted to some magic header fields for logging and debugging?</cref>
        </t>

        </list>
        
      </t>

    </section>

    <section title="Syntax" anchor="abnf">

      <figure>
        <artwork type="ABNF">
<![CDATA[
   metadata-ehlo  = "METADATA"
             ; Complies with the <ehlo-line> ABNF production from RFC 5321.

   bmtd-cmd       = "BMTD" SP chunk-size CR LF
   chunk-size     = 1*DIGIT
   
   bmtd-container = container-type container-specific-data
   
   container-type = <2 octets, extensible>
   
   container-specific-data = <remaining container data>
  
   DIGIT = <Defined in RFC 5234>
]]>
        </artwork>
      </figure>

<!--////RFC 5321 doesn't seem to have ABNF for the response line?!
      ; Syntax of the X.7.TBD3 response
      ["X.7.TBD3" SP] priority-value SP human-readable-text
-->

    </section>
    
    <section title="Example">

      <t>The original submission (from MUA to MSA) might look like shown below.
      Note that the example is also making use of the STARTTLS <xref target="RFC3207"/>,
      and
	    DSN <xref target="RFC3461"/> SMTP extensions, even though there is no requirement
	    that these other extensions are to be supported when the METADATA SMTP extension
	    is implemented.</t>
      
<figure><artwork>
       <![CDATA[
     S: 220 example.com SMTP server here
     C: EHLO mua.example.com
     S: 250-example.com
     S: 250-STARTTLS
     S: 250-AUTH SCRAM-SHA-1 DIGEST-MD5
     S: 250-DSN
     S: 250-CHUNKING
     S: 250-ENHANCEDSTATUSCODES
     S: 250 METADATA
     C: AUTH SCRAM-SHA-1
     [...authentication exchange...]
     S: 235 2.7.0 Authentication successful
     C: MAIL FROM:<eljefe@example.com> ENVID=QQ314159
     S: 250 2.1.0 <eljefe@example.com> sender ok
     C: RCPT TO:<topbanana@example.net>
     S: 250 2.1.5 <topbanana@example.net> recipient ok
     C: RCPT TO:<Dana@Ivory.example.net> NOTIFY=SUCCESS,FAILURE
         ORCPT=rfc822;Dana@Ivory.example.net
     S: 250 2.1.5 <Dana@Ivory.example.net> recipient ok
     C: BMTD 40
     C: <2 octets == type> ...
     S: 250 2.1.0 message metadata accepted
     C: BMTD 12
     C: <2 octets == type 1>$Forwarded
     S: 250 2.1.0 message metadata accepted
     C: BDAT 86 LAST
     C: To: Susan@random.com
     C: From: Sam@random.com
     C: Subject: This is a bodyless test message
     S: 250 2.1.0 message accepted
     C: QUIT
     S: 221 2.0.0 goodbye
]]></artwork></figure>

      <t>[[Need to fix byte counts/BMTD commands in the example]]</t>

      <t>[[Add another example with PIPELINING]]</t>
      
    </section>

    <section title="Deployment Considerations">

      <section title="Multiple MX records">

        <t>
        If multiple DNS MX records are used to specify multiple servers for a
        domain in section 5 of <xref target="RFC5321"/>, it is strongly advised that
        all of them support the METADATA extension <!--and handle unrecognized container types in
        exactly the same way-->. If one or more servers behave differently in this respect,
        then it is strongly suggested that none of the servers
        support the METADATA extension. Otherwise,
        unexpected differences in message rejections
        can happen during temporary or permanent
        failures, which users might perceive as serious reliability issues.
        </t>
        
      </section>

    </section>

    <section title="Open Issues/To Do">

      <t>Document interaction with the SIZE extension. (Proposal: count each BMTD chunk size against the SIZE limit)</t>

      <t>Decide what should be allowed behaviour for handling of container types
      unrecognized by intermediate server and final delivery agents.</t>
      
    </section>

    <section title="IANA Considerations" anchor="iana-regs">
      
      <t>This specification requests IANA to add the METADATA SMTP extension
      to the "SMTP Service Extensions" registry (in http://www.iana.org/assignments/mail-parameters).
      This extension is suitable for the Submit port.
      </t>      

<!--
      <t>This specification requests IANA to add the following Enumerated Status Codes
      to the "Simple Mail Transfer Protocol (SMTP) Enhanced Status Codes" registry
      established by <xref target="RFC5248"/> (in
      http://www.iana.org/assignments/smtp-enhanced-status-codes/smtp-enhanced-status-codes.xml):

        <list style="numbers">

          <t><list style='hanging'>
   <t hangText="Code:">X.7.TBD1</t>
   <t hangText="Sample Text:">Priority Level is too low</t>
   <t hangText="Associated basic status code:">450, 550 (other 4XX or 5XX codes are allowed)</t>
   <t hangText="Description:">The specified priority level is below the lowest
                       priority acceptable for the receiving SMTP server.
                       This condition might be temporary, for example
                       the server is operating in a mode
                       where only higher priority messages are accepted
                       for transfer and delivery,
                       while lower priority messages are rejected.</t>
   <t hangText="Reference:">RFC XXXX</t>
   <t hangText="Submitter:">A. Melnikov</t>
   <t hangText="Change controller:">IESG</t>
          </list></t>

          <t><list style='hanging'>
   <t hangText="Code:">X.3.TBD2</t>
   <t hangText="Sample Text:">Message is too big for the specified priority</t>
   <t hangText="Associated basic status code:">552 (other 4XX or 5XX codes are allowed)</t>
   <t hangText="Description:">The message is too big for the specified priority.
                       This condition might be temporary, for example
                       the server is operating in a mode
                       where only higher priority messages below certain size are accepted
                       for transfer and delivery.</t>
   <t hangText="Reference:">RFC XXXX</t>
   <t hangText="Submitter:">A. Melnikov</t>
   <t hangText="Change controller:">IESG</t>
          </list></t>

          <t><list style='hanging'>
   <t hangText="Code:">X.3.TBD3</t>
   <t hangText="Sample Text:">Requested priority was changed</t>
   <t hangText="Associated basic status code:">250 or 251</t>
   <t hangText="Description:">The message was accepted for relay/delivery, but the requested
                       priority (possibly the implied default) was not honoured.
                       The human readable text after the status code contains the new priority,
                       followed by SP (space) and explanatory human readable text.</t>
   <t hangText="Reference:">RFC XXXX</t>
   <t hangText="Submitter:">A. Melnikov</t>
   <t hangText="Change controller:">IESG</t>
          </list></t>
          
	  </list>
      </t>
-->
    
  </section>

    
  <section title="Security Considerations" anchor="seccons">

    <t>TBD</t>

  </section>

  </middle>
  <back>
    <references title="Normative References">
      <?rfc include="reference.RFC.1870"?> <!-- SMTP SIZE -->
      <?rfc include="reference.RFC.2033"?> <!-- LMTP -->
      <?rfc include="reference.RFC.2034"?> <!-- SMTP extension for returning Enhanced Status Codes -->
      <?rfc include="reference.RFC.2119"?> <!-- Keywords -->
      <?rfc include="reference.RFC.2920"?> <!-- SMTP PIPELINING -->
      <?rfc include="reference.RFC.3030"?> <!-- SMTP CHUNKING -->
      <?rfc include="reference.RFC.3461"?> <!-- SMTP DSN -->
      <?rfc include="reference.RFC.3501"?> <!-- IMAP -->
      <?rfc include="reference.RFC.5321"?> <!-- SMTP -->
      <?rfc include="reference.RFC.5322"?> <!-- Email -->
      <?rfc include="reference.RFC.5226"?> <!-- IANA Policies -->
      <?rfc include="reference.RFC.5234"?> <!-- ABNF -->
      <?rfc include="reference.RFC.5248"?> <!-- Enhanced Status Codes Registry -->
      <?rfc include="reference.RFC.6409"?> <!-- Submission Service -->
      <?rfc include="reference.RFC.7001"?> <!-- Authentication-Results -->      
    </references>

    <references title="Informative References">

      <?rfc include="reference.RFC.5598"?>
      <?rfc include="reference.RFC.1845"?> <!-- SMTP CHECKPOINT -->
      <?rfc include="reference.RFC.3207"?> <!-- SMTP STARTTLS -->
      <?rfc include="reference.RFC.4468"?> <!-- SMTP BURL -->
      <?rfc include="reference.RFC.5228"?> <!-- Sieve -->

    </references>
    
    <section title="Background on Design Choices" anchor="faq">

      <t>This Section provides some background on design choices made during development of
      the METADATA SMTP extension.</t>

      <t>Use of a new command like BDAT makes it very easy to send chunks of binary data.
      Byte counted blobs are easy to parse and generate.</t>      

    </section>

    <section title="Acknowledgements">

      <t>The idea suggested in this document is not new. John Klensin and Paul Smith have suggested
      use of an SMTP extension for separating metadata from the rest of email messages.
      Thank you to Chris Newman for providing comments and suggesting how to make the extension
      easier to implement.
      This document was also inspired by the Dark Mail project.</t>

      <t>This document cuts &amp; pastes lots of text from RFC 3030.</t>
      
    </section>
  </back>
</rfc>
