Simple Mail Transfer Protocol extension for relaying Metadata
Isode Ltd5 Castle Business Village36 Station RoadHamptonMiddlesexTW12 2BXUKAlexey.Melnikov@isode.comSMTPBDAT
This memo defines an extension to the SMTP (Simple Mail Transfer Protocol)
service whereby message metadata (such as Trace header fields, IMAP flags, Keying material, etc)
can be transferred in separate containers similar to BDAT (RFC 3030, SMTP CHUNKING) command.
This allows clean separation of transaction related state from the message itself.
This memo defines an extension to the SMTP (Simple Mail Transfer Protocol)
service whereby message metadata (such as Trace header fields, IMAP flags, Keying material, etc)
can be transferred in separate containers similar to BDAT (RFC 3030, SMTP CHUNKING) command.
This allows clean separation of transaction related state from the message itself.
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in
this document are to be interpreted as described in
when they appear in ALL CAPS.
These words also appear in this document in lower case as plain
English words, absent their normative meanings.The formal syntax use the Augmented
Backus-Naur Form (ABNF) notation including the core rules
defined in Appendix B of RFC 5234.
In examples, "C:" and "S:" indicate lines sent by the client and
server respectively. Line breaks that do not start with a new "C:" or
"S:" exist for editorial reasons and are not a part of the protocol.
The Metadata SMTP service extension is defined as follows:
The textual name of this extension is "Metadata Transfer".The EHLO keyword value associated with this extension is "METADATA".
Any server that advertises support for the "METADATA" extension MUST also support SMTP CHUNKING (RFC 3030).The EHLO keyword has no parameters
Should BMTD be allowed before the DATA command? There is no reason why not.
A new SMTP verb, BMTD, is defined. The BMTD verb takes one argument,
which indicates the length, in octets, of the binary metadata
container that follows immediately after the command.
See for the description of the BMTD command
and for its syntax.
This extension doesn't add any new parameters to
MAIL FROM or RCPT TO commands.The Metadata extension is valid for the submission service
and LMTP .
After all MAIL and RCPT responses are collected and processed, the
message metadata is sent using a series of BMTD commands.
The BMTD command takes one required argument, the exact length of the metadata segment ("container")
in octets. The metadata is sent immediately after the trailing <CR>
<LF> of the BMTD command line. Once the receiver-SMTP receives the
specified number of octets, it will return a 250 reply code.
BMTD commands MUST be sent before any BDAT or
BURL commands. If a server encounters BMTD command
after BDAT/BURL, it MUST respond with 503 "Bad sequence of commands" reply code.
The state resulting from this error is indeterminate. A RSET command
MUST be sent to clear the transaction before continuing.
Each BMTD container starts with 2 octet container type,
followed by container type specific data.
This means that the metadata segment length can never be the value 1
(it can either be 0 or be equal or greater than 2).
A 250 response MUST be sent to each successful BMTD data block ("chunk") within
a mail transaction. If a failure occurs after a BMTD command is
received, the receiver-SMTP MUST accept and discard the associated
metadata and message data before sending the appropriate 5XX or 4XX code. If a
5XX or 4XX code is received by the sender-SMTP in response to a BMTD
chunk, the transaction should be considered failed and the sender-
SMTP MUST NOT send any additional BMTD segments. If the receiver-
SMTP has declared support for command pipelining , the receiver
SMTP MUST be prepared to accept and discard additional BDAT/BURL/BMTD chunks
already in the pipeline after the failed BMTD.
Note: An error on the receiver-SMTP such as disk full or imminent
shutdown can only be reported after the BMTD segment has been
received. It is therefore important to choose a reasonable chunk
size given the expected end-to-end bandwidth.
Note: Because the receiver-SMTP does not acknowledge the BMTD
command before the message data is sent, it is important to send
the BMTD only to systems that have declared their capability to
accept BMTD commands. Illegally sending a BMTD command and
associated message data to a non-METADATA capable system will
result in the receiver-SMTP parsing the associated message data as
if it were a potentially very long, ESMTP command line containing
binary data.
More than one BMTD command can occur in a transaction.
(However some BMTD container types only allow for a single BMTD command
with that particular container type.)
Any BMTD command MUST be followed by one or more of BMTD/BDAT/BURL commands.
Type 0: Trace header fields: Received, Return-Path, Authentication-Results (RFC 7001), etc
encoded as if they are a part of a message header.
Containers of this type can appear multiple types in a transaction.
MUST be supported by all compliant servers.
Type 1: IMAP Keywords associated with the message (e.g. $MDNSent, $Forwarded, \Answered).
This is a space separated list of IMAP keywords/flags.
Container of this type MUST NOT appear more than once in a transaction.
If the final LMTP delivers the message to an IMAP capable mailstore, it MUST
attempt setting the listed IMAP keywords/flags on the message.
Flags/keywords not supported by the mailstore (or disallowed when a message is injected
via LMTP) MUST be ignored.
Keying material, a la Dark Mail. TBD if there is interest.
Each container type definition MUST specify if it can appear more than once.
Unless specified by an extension mutually agreed by SMTP sender and SMTP recipient,
no container type can be defined as required (i.e. appearing at least once
in a SMTP transaction) or define how it can be relayed to a non compliant MTA.
Each container type definition MUST describe how it is going to be handled by the final MTA/LMTP server.
This section describes how a conforming SMTP server should handle any
messages received via SMTP.
The following rules govern the behavior of a conforming MTA (in the
role of an SMTP/LMTP client), when
relaying a message which was received via the SMTP protocol, to an
SMTP/LMTP server that supports the METADATA extension:
Instead of prepending trace fields to the message itself as specified in RFC 5321,
a relaying MTA SHOULD Cross check with RFC 5321 regarding insertion of Received header fields
insert a single BMTD container of type 0 (Trace fields) containing its own trace header fields such as
Received , Authentication-Results , etc.
All other BMTD commands are relayed to conforming SMTP/LMTP server in the order received.
Intermediary servers SHOULD NOT coalesce or reorder metadata containers of type 0 or any other type that they understand.
Intermediary servers MUST NOT coalesce, reorder or drop metadata containers of any types that they don't recognize.
The following rules govern the behavior of a conforming MTA (in the
role of an SMTP/LMTP client), when relaying a message which was received via the
SMTP protocol, to an SMTP/LMTP server that does not support the METADATA extension:
Data from each metadata container of type 0 (Trace fields) MUST be extracted
and prepended to the header of the message in the order of BMTD commands.
All other BMTD chunks are discarded. OPEN ISSUE. They can also be converted to some magic header fields for logging and debugging?
The following rules govern the behavior of a conforming MTA, when
gatewaying a message that was received via the SMTP protocol, into a
foreign (non-SMTP) environment:
If the destination environment is unable to provide an equivalent
of the BMTD command, the conforming MTA SHOULD behave
as if it is relaying to a non-conformant SMTP server ().
If the destination environment is capable of providing an equivalent
of the BMTD command, the conforming MTA SHOULD behave
as if it is relaying to a conformant SMTP server (),
converting any BMTD command to the equivalent in the destination environment.
An LMTP server can advertise support for the METADATA extension:
Data from containers of type 0 (Trace fields) is extracted (in the order of the
corresponding BMTD commands) and prepended to the header of the message.
Handling of other container type is specific to the container type.
Unsupported BMTD container types are discarded.
OPEN ISSUE. They can also be converted to some magic header fields for logging and debugging?The original submission (from MUA to MSA) might look like shown below.
Note that the example is also making use of the STARTTLS ,
and
DSN SMTP extensions, even though there is no requirement
that these other extensions are to be supported when the METADATA SMTP extension
is implemented.[[Need to fix byte counts/BMTD commands in the example]][[Add another example with PIPELINING]]
If multiple DNS MX records are used to specify multiple servers for a
domain in section 5 of , it is strongly advised that
all of them support the METADATA extension . If one or more servers behave differently in this respect,
then it is strongly suggested that none of the servers
support the METADATA extension. Otherwise,
unexpected differences in message rejections
can happen during temporary or permanent
failures, which users might perceive as serious reliability issues.
Document interaction with the SIZE extension. (Proposal: count each BMTD chunk size against the SIZE limit)Decide what should be allowed behaviour for handling of container types
unrecognized by intermediate server and final delivery agents.This specification requests IANA to add the METADATA SMTP extension
to the "SMTP Service Extensions" registry (in http://www.iana.org/assignments/mail-parameters).
This extension is suitable for the Submit port.
TBDThis Section provides some background on design choices made during development of
the METADATA SMTP extension.Use of a new command like BDAT makes it very easy to send chunks of binary data.
Byte counted blobs are easy to parse and generate.The idea suggested in this document is not new. John Klensin and Paul Smith have suggested
use of an SMTP extension for separating metadata from the rest of email messages.
Thank you to Chris Newman for providing comments and suggesting how to make the extension
easier to implement.
This document was also inspired by the Dark Mail project.This document cuts & pastes lots of text from RFC 3030.