idnits 2.17.1 draft-melnikov-smtp-metadata-01.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** There is 1 instance of too long lines in the document, the longest one being 3 characters in excess of 72. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (June 17, 2015) is 3229 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Unused Reference: 'RFC1870' is defined on line 381, but no explicit reference was found in the text == Unused Reference: 'RFC2034' is defined on line 388, but no explicit reference was found in the text == Unused Reference: 'RFC5322' is defined on line 411, but no explicit reference was found in the text == Unused Reference: 'RFC5226' is defined on line 414, but no explicit reference was found in the text == Unused Reference: 'RFC5248' is defined on line 421, but no explicit reference was found in the text == Unused Reference: 'RFC5598' is defined on line 432, but no explicit reference was found in the text == Unused Reference: 'RFC1845' is defined on line 435, but no explicit reference was found in the text == Unused Reference: 'RFC5228' is defined on line 444, but no explicit reference was found in the text ** Downref: Normative reference to an Informational RFC: RFC 2033 ** Obsolete normative reference: RFC 3501 (Obsoleted by RFC 9051) ** Obsolete normative reference: RFC 5226 (Obsoleted by RFC 8126) ** Obsolete normative reference: RFC 7001 (Obsoleted by RFC 7601) Summary: 5 errors (**), 0 flaws (~~), 9 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group A. Melnikov 3 Internet-Draft Isode Ltd 4 Intended status: Standards Track June 17, 2015 5 Expires: December 19, 2015 7 Simple Mail Transfer Protocol extension for relaying Metadata 8 draft-melnikov-smtp-metadata-01 10 Abstract 12 This memo defines an extension to the SMTP (Simple Mail Transfer 13 Protocol) service whereby message metadata (such as Trace header 14 fields, IMAP flags, Keying material, etc) can be transferred in 15 separate containers similar to BDAT (RFC 3030, SMTP CHUNKING) 16 command. This allows clean separation of transaction related state 17 from the message itself. 19 Status of This Memo 21 This Internet-Draft is submitted in full conformance with the 22 provisions of BCP 78 and BCP 79. 24 Internet-Drafts are working documents of the Internet Engineering 25 Task Force (IETF). Note that other groups may also distribute 26 working documents as Internet-Drafts. The list of current Internet- 27 Drafts is at http://datatracker.ietf.org/drafts/current/. 29 Internet-Drafts are draft documents valid for a maximum of six months 30 and may be updated, replaced, or obsoleted by other documents at any 31 time. It is inappropriate to use Internet-Drafts as reference 32 material or to cite them other than as "work in progress." 34 This Internet-Draft will expire on December 19, 2015. 36 Copyright Notice 38 Copyright (c) 2015 IETF Trust and the persons identified as the 39 document authors. All rights reserved. 41 This document is subject to BCP 78 and the IETF Trust's Legal 42 Provisions Relating to IETF Documents 43 (http://trustee.ietf.org/license-info) in effect on the date of 44 publication of this document. Please review these documents 45 carefully, as they describe your rights and restrictions with respect 46 to this document. Code Components extracted from this document must 47 include Simplified BSD License text as described in Section 4.e of 48 the Trust Legal Provisions and are provided without warranty as 49 described in the Simplified BSD License. 51 Table of Contents 53 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 54 2. Conventions Used in This Document . . . . . . . . . . . . . . 2 55 3. Definition of the Metadata SMTP Extension . . . . . . . . . . 3 56 3.1. BMTD command . . . . . . . . . . . . . . . . . . . . . . 3 57 3.2. Initial List of Metadata Container types . . . . . . . . 4 58 3.3. Requirements on a Metadata Container type definition . . 5 59 4. Handling of messages received via SMTP . . . . . . . . . . . 5 60 4.1. Relay of messages to other conforming SMTP/LMTP servers . 5 61 4.2. Relay of messages to non-conforming SMTP/LMTP servers . . 6 62 4.3. Gatewaying a message into a foreign environment . . . . . 6 63 5. Use of METADATA with LMTP . . . . . . . . . . . . . . . . . . 6 64 6. Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 65 7. Example . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 66 8. Deployment Considerations . . . . . . . . . . . . . . . . . . 8 67 8.1. Multiple MX records . . . . . . . . . . . . . . . . . . . 8 68 9. Open Issues/To Do . . . . . . . . . . . . . . . . . . . . . . 9 69 10. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 9 70 11. Security Considerations . . . . . . . . . . . . . . . . . . . 9 71 12. References . . . . . . . . . . . . . . . . . . . . . . . . . 9 72 12.1. Normative References . . . . . . . . . . . . . . . . . . 9 73 12.2. Informative References . . . . . . . . . . . . . . . . . 10 74 Appendix A. Background on Design Choices . . . . . . . . . . . . 11 75 Appendix B. Acknowledgements . . . . . . . . . . . . . . . . . . 11 76 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 11 78 1. Introduction 80 This memo defines an extension to the SMTP (Simple Mail Transfer 81 Protocol) service whereby message metadata (such as Trace header 82 fields, IMAP flags, Keying material, etc) can be transferred in 83 separate containers similar to BDAT (RFC 3030, SMTP CHUNKING) 84 command. This allows clean separation of transaction related state 85 from the message itself. 87 2. Conventions Used in This Document 89 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 90 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 91 document are to be interpreted as described in [RFC2119] when they 92 appear in ALL CAPS. These words also appear in this document in 93 lower case as plain English words, absent their normative meanings. 95 The formal syntax use the Augmented Backus-Naur Form (ABNF) [RFC5234] 96 notation including the core rules defined in Appendix B of RFC 5234 97 [RFC5234]. 99 In examples, "C:" and "S:" indicate lines sent by the client and 100 server respectively. Line breaks that do not start with a new "C:" 101 or "S:" exist for editorial reasons and are not a part of the 102 protocol. 104 3. Definition of the Metadata SMTP Extension 106 The Metadata SMTP service extension is defined as follows: 108 1. The textual name of this extension is "Metadata Transfer". 110 2. The EHLO keyword value associated with this extension is 111 "METADATA". Any server that advertises support for the 112 "METADATA" extension MUST also support SMTP CHUNKING (RFC 3030). 114 3. The EHLO keyword has no parameters 116 4. [[CREF1: Should BMTD be allowed before the DATA command? There 117 is no reason why not.]] A new SMTP verb, BMTD, is defined. The 118 BMTD verb takes one argument, which indicates the length, in 119 octets, of the binary metadata container that follows immediately 120 after the command. See Section 3.1 for the description of the 121 BMTD command and Section 6 for its syntax. 123 5. This extension doesn't add any new parameters to MAIL FROM or 124 RCPT TO commands. 126 6. The Metadata extension is valid for the submission service 127 [RFC6409] and LMTP [RFC2033]. 129 3.1. BMTD command 131 After all MAIL and RCPT responses are collected and processed, the 132 message metadata is sent using a series of BMTD commands. The BMTD 133 command takes one required argument, the exact length of the metadata 134 segment ("container") in octets. The metadata is sent immediately 135 after the trailing of the BMTD command line. Once the 136 receiver-SMTP receives the specified number of octets, it will return 137 a 250 reply code. 139 BMTD commands MUST be sent before any BDAT [RFC3030] or BURL 140 [RFC4468] commands. If a server encounters BMTD command after BDAT/ 141 BURL, it MUST respond with 503 "Bad sequence of commands" reply code. 143 The state resulting from this error is indeterminate. A RSET command 144 MUST be sent to clear the transaction before continuing. 146 Each BMTD container starts with 2 octet container type, followed by 147 container type specific data. This means that the metadata segment 148 length can never be the value 1 (it can either be 0 or be equal or 149 greater than 2). 151 A 250 response MUST be sent to each successful BMTD data block 152 ("chunk") within a mail transaction. If a failure occurs after a 153 BMTD command is received, the receiver-SMTP MUST accept and discard 154 the associated metadata and message data before sending the 155 appropriate 5XX or 4XX code. If a 5XX or 4XX code is received by the 156 sender-SMTP in response to a BMTD chunk, the transaction should be 157 considered failed and the sender- SMTP MUST NOT send any additional 158 BMTD segments. If the receiver- SMTP has declared support for 159 command pipelining [RFC2920], the receiver SMTP MUST be prepared to 160 accept and discard additional BDAT/BURL/BMTD chunks already in the 161 pipeline after the failed BMTD. 163 Note: An error on the receiver-SMTP such as disk full or imminent 164 shutdown can only be reported after the BMTD segment has been 165 received. It is therefore important to choose a reasonable chunk 166 size given the expected end-to-end bandwidth. 168 Note: Because the receiver-SMTP does not acknowledge the BMTD command 169 before the message data is sent, it is important to send the BMTD 170 only to systems that have declared their capability to accept BMTD 171 commands. Illegally sending a BMTD command and associated message 172 data to a non-METADATA capable system will result in the receiver- 173 SMTP parsing the associated message data as if it were a potentially 174 very long, ESMTP command line containing binary data. 176 More than one BMTD command can occur in a transaction. (However some 177 BMTD container types only allow for a single BMTD command with that 178 particular container type.) Any BMTD command MUST be followed by one 179 or more of BMTD/BDAT/BURL commands. 181 3.2. Initial List of Metadata Container types 183 Type 0: Trace header fields: Received, Return-Path, Authentication- 184 Results (RFC 7001), etc encoded as if they are a part of a message 185 header. Containers of this type can appear multiple types in a 186 transaction. MUST be supported by all compliant servers. 188 Type 1: IMAP Keywords [RFC3501] associated with the message (e.g. 189 $MDNSent, $Forwarded, \Answered). This is a space separated list of 190 IMAP keywords/flags. Container of this type MUST NOT appear more 191 than once in a transaction. If the final LMTP delivers the message 192 to an IMAP capable mailstore, it MUST attempt setting the listed IMAP 193 keywords/flags on the message. Flags/keywords not supported by the 194 mailstore (or disallowed when a message is injected via LMTP) MUST be 195 ignored. 197 Keying material, a la Dark Mail. TBD if there is interest. 199 3.3. Requirements on a Metadata Container type definition 201 Each container type definition MUST specify if it can appear more 202 than once. 204 Unless specified by an extension mutually agreed by SMTP sender and 205 SMTP recipient, no container type can be defined as required (i.e. 206 appearing at least once in a SMTP transaction) or define how it can 207 be relayed to a non compliant MTA. 209 Each container type definition MUST describe how it is going to be 210 handled by the final MTA/LMTP server. 212 4. Handling of messages received via SMTP 214 This section describes how a conforming SMTP server should handle any 215 messages received via SMTP. 217 4.1. Relay of messages to other conforming SMTP/LMTP servers 219 The following rules govern the behavior of a conforming MTA (in the 220 role of an SMTP/LMTP client), when relaying a message which was 221 received via the SMTP protocol, to an SMTP/LMTP server that supports 222 the METADATA extension: 224 1. Instead of prepending trace fields to the message itself as 225 specified in RFC 5321, a relaying MTA SHOULD [[CREF2: Cross check 226 with RFC 5321 regarding insertion of Received header fields]] 227 insert a single BMTD container of type 0 (Trace fields) 228 containing its own trace header fields such as Received 229 [RFC5321], Authentication-Results [RFC7001], etc. 231 2. All other BMTD commands are relayed to conforming SMTP/LMTP 232 server in the order received. Intermediary servers SHOULD NOT 233 coalesce or reorder metadata containers of type 0 or any other 234 type that they understand. Intermediary servers MUST NOT 235 coalesce, reorder or drop metadata containers of any types that 236 they don't recognize. 238 4.2. Relay of messages to non-conforming SMTP/LMTP servers 240 The following rules govern the behavior of a conforming MTA (in the 241 role of an SMTP/LMTP client), when relaying a message which was 242 received via the SMTP protocol, to an SMTP/LMTP server that does not 243 support the METADATA extension: 245 1. Data from each metadata container of type 0 (Trace fields) MUST 246 be extracted and prepended to the header of the message in the 247 order of BMTD commands. 249 2. All other BMTD chunks are discarded. [[CREF3: OPEN ISSUE. They 250 can also be converted to some magic header fields for logging and 251 debugging?]] 253 4.3. Gatewaying a message into a foreign environment 255 The following rules govern the behavior of a conforming MTA, when 256 gatewaying a message that was received via the SMTP protocol, into a 257 foreign (non-SMTP) environment: 259 1. If the destination environment is unable to provide an equivalent 260 of the BMTD command, the conforming MTA SHOULD behave as if it is 261 relaying to a non-conformant SMTP server (Section 4.2). 263 2. If the destination environment is capable of providing an 264 equivalent of the BMTD command, the conforming MTA SHOULD behave 265 as if it is relaying to a conformant SMTP server (Section 4.1), 266 converting any BMTD command to the equivalent in the destination 267 environment. 269 5. Use of METADATA with LMTP 271 An LMTP server can advertise support for the METADATA extension: 273 1. Data from containers of type 0 (Trace fields) is extracted (in 274 the order of the corresponding BMTD commands) and prepended to 275 the header of the message. 277 2. Handling of other container type is specific to the container 278 type. 280 3. Unsupported BMTD container types are discarded. [[CREF4: OPEN 281 ISSUE. They can also be converted to some magic header fields 282 for logging and debugging?]] 284 6. Syntax 286 metadata-ehlo = "METADATA" 287 ; Complies with the ABNF production from RFC 5321. 289 bmtd-cmd = "BMTD" SP chunk-size CR LF 290 chunk-size = 1*DIGIT 292 bmtd-container = container-type container-specific-data 294 container-type = <2 octets, extensible> 296 container-specific-data = 298 DIGIT = 300 7. Example 302 The original submission (from MUA to MSA) might look like shown 303 below. Note that the example is also making use of the STARTTLS 304 [RFC3207], and DSN [RFC3461] SMTP extensions, even though there is no 305 requirement that these other extensions are to be supported when the 306 METADATA SMTP extension is implemented. 308 S: 220 example.com SMTP server here 309 C: EHLO mua.example.com 310 S: 250-example.com 311 S: 250-STARTTLS 312 S: 250-AUTH SCRAM-SHA-1 DIGEST-MD5 313 S: 250-DSN 314 S: 250-CHUNKING 315 S: 250-ENHANCEDSTATUSCODES 316 S: 250 METADATA 317 C: AUTH SCRAM-SHA-1 318 [...authentication exchange...] 319 S: 235 2.7.0 Authentication successful 320 C: MAIL FROM: ENVID=QQ314159 321 S: 250 2.1.0 sender ok 322 C: RCPT TO: 323 S: 250 2.1.5 recipient ok 324 C: RCPT TO: NOTIFY=SUCCESS,FAILURE 325 ORCPT=rfc822;Dana@Ivory.example.net 326 S: 250 2.1.5 recipient ok 327 C: BMTD 40 328 C: <2 octets == type> ... 329 S: 250 2.1.0 message metadata accepted 330 C: BMTD 12 331 C: <2 octets == type 1>$Forwarded 332 S: 250 2.1.0 message metadata accepted 333 C: BDAT 86 LAST 334 C: To: Susan@random.com 335 C: From: Sam@random.com 336 C: Subject: This is a bodyless test message 337 S: 250 2.1.0 message accepted 338 C: QUIT 339 S: 221 2.0.0 goodbye 341 [[Need to fix byte counts/BMTD commands in the example]] 343 [[Add another example with PIPELINING]] 345 8. Deployment Considerations 347 8.1. Multiple MX records 349 If multiple DNS MX records are used to specify multiple servers for a 350 domain in section 5 of [RFC5321], it is strongly advised that all of 351 them support the METADATA extension . If one or more servers behave 352 differently in this respect, then it is strongly suggested that none 353 of the servers support the METADATA extension. Otherwise, unexpected 354 differences in message rejections can happen during temporary or 355 permanent failures, which users might perceive as serious reliability 356 issues. 358 9. Open Issues/To Do 360 Document interaction with the SIZE extension. (Proposal: count each 361 BMTD chunk size against the SIZE limit) 363 Decide what should be allowed behaviour for handling of container 364 types unrecognized by intermediate server and final delivery agents. 366 10. IANA Considerations 368 This specification requests IANA to add the METADATA SMTP extension 369 to the "SMTP Service Extensions" registry (in 370 http://www.iana.org/assignments/mail-parameters). This extension is 371 suitable for the Submit port. 373 11. Security Considerations 375 TBD 377 12. References 379 12.1. Normative References 381 [RFC1870] Klensin, J., Freed, N., and K. Moore, "SMTP Service 382 Extension for Message Size Declaration", STD 10, RFC 1870, 383 November 1995. 385 [RFC2033] Myers, J., "Local Mail Transfer Protocol", RFC 2033, 386 October 1996. 388 [RFC2034] Freed, N., "SMTP Service Extension for Returning Enhanced 389 Error Codes", RFC 2034, October 1996. 391 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 392 Requirement Levels", BCP 14, RFC 2119, March 1997. 394 [RFC2920] Freed, N., "SMTP Service Extension for Command 395 Pipelining", STD 60, RFC 2920, September 2000. 397 [RFC3030] Vaudreuil, G., "SMTP Service Extensions for Transmission 398 of Large and Binary MIME Messages", RFC 3030, December 399 2000. 401 [RFC3461] Moore, K., "Simple Mail Transfer Protocol (SMTP) Service 402 Extension for Delivery Status Notifications (DSNs)", RFC 403 3461, January 2003. 405 [RFC3501] Crispin, M., "INTERNET MESSAGE ACCESS PROTOCOL - VERSION 406 4rev1", RFC 3501, March 2003. 408 [RFC5321] Klensin, J., "Simple Mail Transfer Protocol", RFC 5321, 409 October 2008. 411 [RFC5322] Resnick, P., Ed., "Internet Message Format", RFC 5322, 412 October 2008. 414 [RFC5226] Narten, T. and H. Alvestrand, "Guidelines for Writing an 415 IANA Considerations Section in RFCs", BCP 26, RFC 5226, 416 May 2008. 418 [RFC5234] Crocker, D. and P. Overell, "Augmented BNF for Syntax 419 Specifications: ABNF", STD 68, RFC 5234, January 2008. 421 [RFC5248] Hansen, T. and J. Klensin, "A Registry for SMTP Enhanced 422 Mail System Status Codes", BCP 138, RFC 5248, June 2008. 424 [RFC6409] Gellens, R. and J. Klensin, "Message Submission for Mail", 425 STD 72, RFC 6409, November 2011. 427 [RFC7001] Kucherawy, M., "Message Header Field for Indicating 428 Message Authentication Status", RFC 7001, September 2013. 430 12.2. Informative References 432 [RFC5598] Crocker, D., "Internet Mail Architecture", RFC 5598, July 433 2009. 435 [RFC1845] Crocker, D. and N. Freed, "SMTP Service Extension for 436 Checkpoint/Restart", RFC 1845, September 1995. 438 [RFC3207] Hoffman, P., "SMTP Service Extension for Secure SMTP over 439 Transport Layer Security", RFC 3207, February 2002. 441 [RFC4468] Newman, C., "Message Submission BURL Extension", RFC 4468, 442 May 2006. 444 [RFC5228] Guenther, P. and T. Showalter, "Sieve: An Email Filtering 445 Language", RFC 5228, January 2008. 447 Appendix A. Background on Design Choices 449 This Section provides some background on design choices made during 450 development of the METADATA SMTP extension. 452 Use of a new command like BDAT makes it very easy to send chunks of 453 binary data. Byte counted blobs are easy to parse and generate. 455 Appendix B. Acknowledgements 457 The idea suggested in this document is not new. John Klensin and 458 Paul Smith have suggested use of an SMTP extension for separating 459 metadata from the rest of email messages. Thank you to Chris Newman 460 for providing comments and suggesting how to make the extension 461 easier to implement. This document was also inspired by the Dark 462 Mail project. 464 This document cuts & pastes lots of text from RFC 3030. 466 Author's Address 468 Alexey Melnikov 469 Isode Ltd 470 5 Castle Business Village 471 36 Station Road 472 Hampton, Middlesex TW12 2BX 473 UK 475 EMail: Alexey.Melnikov@isode.com