idnits 2.17.1 draft-ietf-extra-imap-objectid-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** There are 6 instances of too long lines in the document, the longest one being 1 character in excess of 72. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (April 20, 2018) is 2197 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Missing Reference: 'UIDVALIDITY 1524195797' is mentioned on line 180, but not defined == Outdated reference: A later version (-16) exists of draft-ietf-jmap-mail-04 ** Obsolete normative reference: RFC 3501 (Obsoleted by RFC 9051) ** Obsolete normative reference: RFC 4234 (Obsoleted by RFC 5234) Summary: 3 errors (**), 0 flaws (~~), 3 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 EXTRA B. Gondwana, Ed. 3 Internet-Draft FastMail 4 Intended status: Standards Track April 20, 2018 5 Expires: October 22, 2018 7 IMAP Extension for object identifiers 8 draft-ietf-extra-imap-objectid-00 10 Abstract 12 This document adds new properties to IMAP mailboxes and messages to 13 allow clients to more efficiently re-use cached data for resources 14 which have changed location on the server. 16 Status of This Memo 18 This Internet-Draft is submitted in full conformance with the 19 provisions of BCP 78 and BCP 79. 21 Internet-Drafts are working documents of the Internet Engineering 22 Task Force (IETF). Note that other groups may also distribute 23 working documents as Internet-Drafts. The list of current Internet- 24 Drafts is at https://datatracker.ietf.org/drafts/current/. 26 Internet-Drafts are draft documents valid for a maximum of six months 27 and may be updated, replaced, or obsoleted by other documents at any 28 time. It is inappropriate to use Internet-Drafts as reference 29 material or to cite them other than as "work in progress." 31 This Internet-Draft will expire on October 22, 2018. 33 Copyright Notice 35 Copyright (c) 2018 IETF Trust and the persons identified as the 36 document authors. All rights reserved. 38 This document is subject to BCP 78 and the IETF Trust's Legal 39 Provisions Relating to IETF Documents 40 (https://trustee.ietf.org/license-info) in effect on the date of 41 publication of this document. Please review these documents 42 carefully, as they describe your rights and restrictions with respect 43 to this document. Code Components extracted from this document must 44 include Simplified BSD License text as described in Section 4.e of 45 the Trust Legal Provisions and are provided without warranty as 46 described in the Simplified BSD License. 48 Table of Contents 50 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 51 2. Conventions Used In This Document . . . . . . . . . . . . . . 3 52 3. CAPABILITY Identification . . . . . . . . . . . . . . . . . . 3 53 4. MAILBOXID object identifier . . . . . . . . . . . . . . . . . 3 54 4.1. New response code for CREATE . . . . . . . . . . . . . . 4 55 4.2. New OK Untagged Response for SELECT and EXAMINE . . . . . 4 56 4.3. New attribute for STATUS . . . . . . . . . . . . . . . . 4 57 5. EMAILID object identifier and THREADID correlator . . . . . . 5 58 5.1. EMAILID identifier for identical messages . . . . . . . . 6 59 5.2. THREADID identifer for related messages . . . . . . . . . 6 60 5.3. New Message Data Items in FETCH and UID FETCH Commands . 6 61 6. New Filters on SEARCH command . . . . . . . . . . . . . . . . 9 62 7. Formal syntax . . . . . . . . . . . . . . . . . . . . . . . . 9 63 8. Implementation considerations . . . . . . . . . . . . . . . . 10 64 8.1. Assigning object identifiers . . . . . . . . . . . . . . 10 65 8.2. Interaction with special cases . . . . . . . . . . . . . 11 66 8.3. Ideas for implementing object identifiers . . . . . . . . 11 67 9. Future considerations . . . . . . . . . . . . . . . . . . . . 12 68 10. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 12 69 11. Security Considerations . . . . . . . . . . . . . . . . . . . 12 70 12. Changes . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 71 12.1. 01 . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 72 13. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 13 73 14. References . . . . . . . . . . . . . . . . . . . . . . . . . 13 74 14.1. Normative References . . . . . . . . . . . . . . . . . . 13 75 14.2. Informative References . . . . . . . . . . . . . . . . . 14 76 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 14 78 1. Introduction 80 IMAP stores are often used by many clients. Each client may cache 81 data from the server so that they don't need to re-download 82 information. [RFC3501] defines that a mailbox can be uniquely 83 referenced by its name and UIDVALIDITY, and a message within that 84 mailbox can be uniquely referenced by its mailbox (name + 85 UIDVALIDITY) and UID. The triple of mailbox name, UIDVALIDITY and 86 UID is guaranteed to be immutable. 88 [RFC4315] defines a COPYUID response which allows a client which 89 copies messages to know the mapping between the UIDs in the source 90 and destination mailboxes, and hence update its local cache. 92 If a mailbox is successfully renamed, the client knows that the same 93 messages exist in the destination mailbox name as previously existed 94 in the source mailbox name. 96 The result is that the client which copies (or [RFC6851] moves) 97 messages or renames a mailbox can update its local cache, but any 98 other client connected to the same store can not know with certainty 99 that the messages are identical, and so will re-download everything. 101 This extension adds new properties to a message (EMAILID) and mailbox 102 (MAILBOXID) which allow a client to quickly identify messages or 103 mailboxes which have been renamed by another client. 105 This extension also adds an optional thread identifier (THREADID) to 106 messages, which can be used by the server to indicate messages which 107 it has identified to be related. 109 2. Conventions Used In This Document 111 In examples, "C:" indicates lines sent by a client that is connected 112 to a server. "S:" indicates lines sent by the server to the client. 114 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 115 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 116 document are to be interpreted as described in [RFC2119] when they 117 appear in ALL CAPS. These words may also appear in this document in 118 lower case as plain English words, absent their normative meanings. 120 3. CAPABILITY Identification 122 IMAP servers that support this extension MUST include "OBJECTID" in 123 the response list to the CAPABILITY command. 125 4. MAILBOXID object identifier 127 The MAILBOXID is a server-allocated unique identifer for each 128 mailbox. 130 The server SHOULD return the same MAILBOXID for a server with the 131 same mailbox name and UIDVALIDITY. This is almost MUST, but weakened 132 to allow for the possibility of loss of OBJECTID data during disaster 133 recovery while still keeping the name and UIDVALIDITY the same. 135 The server MUST NOT report the same MAILBOXID for two mailboxes at 136 the same time. 138 The server MUST NOT reuse the same MAILBOXID for a mailbox which does 139 not obey all the invarients that [RFC3501] defines for a mailbox 140 which does not change name or UIDVALIDITY. 142 The server MAY choose to create a MAILBOXID value in a way that does 143 not survive RENAME (e.g. a digest of name + uidvalidity could be 144 used), however the client then loses the major benefit of having an 145 identifier. 147 4.1. New response code for CREATE 149 This document extends the CREATE command to have the response code 150 MAILBOXID on successful mailbox creation. 152 Syntax: "MAILBOXID" SP "(" ")" 154 Response code in tagged OK for successful CREATE command. 156 Example: 158 C: 3 create foo 159 S: 3 OK [MAILBOXID (F2212ea87-6097-4256-9d51-716686338625)] Completed 160 C: 4 create bar 161 S: 4 OK [MAILBOXID (F6352ae03-b7f5-463c-896f-d47cf8b48ee3)] Completed 162 C: 5 create foo 163 S: 5 NO Mailbox already exists 165 4.2. New OK Untagged Response for SELECT and EXAMINE 167 This document adds a new untagged response code to any successful 168 SELECT or EXAMINE command. A server advertising OBJECTID capability 169 MUST return this untagged response to all successful SELECT and 170 EXAMINE commands. 172 Syntax: "OK" SP "[" "MAILBOXID" SP "(" ")" "]" text 174 Untagged OK response to SELECT or EXAMINE. 176 Example: 178 C: 27 select "foo" 179 [...] 180 S: * OK [UIDVALIDITY 1524195797] Ok 181 S: * OK [MAILBOXID (F2212ea87-6097-4256-9d51-716686338625)] Ok 182 [...] 183 S: 27 OK [READ-WRITE] Completed 185 4.3. New attribute for STATUS 187 This document adds the MAILBOXID attribute to the STATUS command 188 using the extended syntax defined in [RFC4466]. 190 Syntax: "MAILBOXID" 191 The attribute in the STATUS command. 193 Syntax: "MAILBOXID" SP "(" ")" 195 The response item in the STATUS response contains the objectid 196 assigned by the server for this mailbox. 198 Example: 200 C: 6 status foo (mailboxid) 201 S: * STATUS foo (MAILBOXID (F2212ea87-6097-4256-9d51-716686338625)) 202 S: 6 OK Completed 203 C: 7 status bar (mailboxid) 204 S: * STATUS bar (MAILBOXID (F6352ae03-b7f5-463c-896f-d47cf8b48ee3)) 205 S: 7 OK Completed 206 C: 8 rename foo renamed 207 S: * OK rename foo renamed 208 S: 8 OK Completed 209 C: 9 status renamed (mailboxid) 210 S: * STATUS renamed (MAILBOXID (F2212ea87-6097-4256-9d51-716686338625)) 211 S: 9 OK Completed 212 C: 10 status bar (mailboxid) 213 S: * STATUS bar (MAILBOXID (F6352ae03-b7f5-463c-896f-d47cf8b48ee3)) 214 S: 10 OK Completed 216 When the LIST-STATUS IMAP capability [RFC5819] is also available, the 217 STATUS command can be combined with the LIST command to further 218 improve efficiency. This way, the MAILBOXIDs of many mailboxes can 219 be queried with just one LIST command. 221 Example: 223 C: 11 list "" "*" return (status (mailboxid)) 224 S: * LIST (\HasNoChildren) "." INBOX 225 S: * STATUS INBOX (MAILBOXID (Ff8e3ead4-9389-4aff-adb1-d8d89efd8cbf)) 226 S: * LIST (\HasNoChildren) "." bar 227 S: * STATUS bar (MAILBOXID (F6352ae03-b7f5-463c-896f-d47cf8b48ee3)) 228 S: * LIST (\HasNoChildren) "." renamed 229 S: * STATUS renamed (MAILBOXID (F2212ea87-6097-4256-9d51-716686338625)) 230 S: 11 OK Completed (0.001 secs 3 calls) 232 5. EMAILID object identifier and THREADID correlator 234 This document adds two additional data items to individual email 235 messages within a mailbox, EMAILID and THREADID. 237 5.1. EMAILID identifier for identical messages 239 The EMAILID data item is an objectid which uniquely identifies the 240 content of a single message. Anything which must remain immutable on 241 a {name, uidvalidity, uid} triple must also be the same between 242 messages with the same EMAILID. 244 The server SHOULD return the same EMAILID for the same UID triple. 245 As with MAILBOXID above, this is almost a MUST, but allows for the 246 possibility of loss of OBJECTID data in disaster recovery without 247 having to change UIDVALIDITY. 249 The server SHOULD return the same EMAILID for the exact same message 250 content in different folders after a COPY or [RFC6851] MOVE command. 252 The server MAY us content detection to assign the same EMAILID upon 253 APPEND of an identical message. 255 5.2. THREADID identifer for related messages 257 The THREADID data item is an objectid which uniquely identifies as 258 set of messages which the server believes are related in some way. 260 THREADID calculation is generally based on some combination of 261 References, In-Reply-To and Subject, but the exact logic is left up 262 to the server implementation. 264 The server MUST return the same THREADID for all messages with the 265 same EMAILID. 267 The server SHOULD return the same THREADID for related messages even 268 if they are in different mailboxes. 270 THREADID is optional, if the server is unable to calculate 271 relationships between messages, it MUST return NIL to in all FETCH 272 responses for the THREADID data item, and it MUST NOT match any 273 messages for any value of THREADID. 275 The server MAY use the same value for both EMAILID and THREADID, for 276 example the THREADID could be the EMAILID of the first message seen 277 in each thread. 279 5.3. New Message Data Items in FETCH and UID FETCH Commands 281 This document specifies two new FETCH items for messages: 283 Syntax: EMAILID 284 The EMAILID message data item causes the server to return EMAILID 285 FETCH response data items. 287 Syntax: THREADID 289 The THREADID message data item causes the server to return THREADID 290 FETCH response data items. 292 And the following responses: 294 Syntax: EMAILID ( ) 296 The EMAILID response data item contains the server-assigned objectid 297 for each message. 299 Syntax: THREADID ( ) 301 The THREADID response data item contains the server-assigned objectid 302 for the set of related messages to which this message belongs. 304 Syntax: THREADID NIL 306 The NIL value to the THREADID response data item is returned when 307 the server mailbox does not support THREADID calculation. 309 Example: 311 C: 5 append inbox "20-Mar-2018 03:07:37 +1100" {733} 312 [...] 313 Subject: Message A 314 Message-ID: 315 [...] 316 S: 5 OK [APPENDUID 1521475658 1] Completed 318 C: 11 append inbox "20-Mar-2018 03:07:37 +1100" {793} 319 [...] 320 Subject: Re: Message A 321 Message-ID: 322 References: 323 [...] 324 S: 11 OK [APPENDUID 1521475658 2] Completed 326 C: 17 append inbox "20-Mar-2018 03:07:37 +1100" {736} 327 [...] 328 Subject: Message C 329 Message-ID: 330 [...] 331 S: 17 OK [APPENDUID 1521475658 3] Completed 333 C: 22 fetch 1:* (emailid threadid) 334 S: * 1 FETCH (EMAILID (Md8976d99ac3275bb4e) THREADID (T4964b478a75b7ea9)) 335 S: * 2 FETCH (EMAILID (Mdd3c288836c4c7a762) THREADID (T4964b478a75b7ea9)) 336 S: * 3 FETCH (EMAILID (Mf2e25fdc09b49ea703) THREADID (T6311863d02dd95b5)) 337 S: 22 OK Completed (0.000 sec) 339 C: 23 move 2 foo 340 S: * OK [COPYUID 1521475659 2 1] Completed 341 S: * 2 EXPUNGE 342 S: 23 OK Completed 344 C: 24 fetch 1:* (emailid threadid) 345 S: * 1 FETCH (EMAILID (Md8976d99ac3275bb4e) THREADID (T4964b478a75b7ea9)) 346 S: * 2 FETCH (EMAILID (Mf2e25fdc09b49ea703) THREADID (T6311863d02dd95b5)) 347 S: 24 OK Completed (0.000 sec) 348 C: 25 select "foo" 350 C: 25 select "foo" 351 [...] 352 S: 25 OK [READ-WRITE] Completed 353 C: 26 fetch 1:* (emailid threadid) 354 S: * 1 FETCH (EMAILID (Mdd3c288836c4c7a762) THREADID (T4964b478a75b7ea9)) 355 S: 26 OK Completed (0.000 sec) 357 Example: (no THREADID support) 358 C: 26 fetch 1:* (emailid threadid) 359 S: * 1 FETCH (EMAILID (M00000001) THREADID NIL) 360 S: * 2 FETCH (EMAILID (M00000002) THREADID NIL) 361 S: 26 OK Completed (0.000 sec) 363 6. New Filters on SEARCH command 365 This extension defines the EMAILID and THREADID filters for the 366 SEARCH command. 368 EMAILID 370 Messages whose EMAILID is exactly the specified objectid. 372 THREADID 374 Messages whose THREADID is exactly the specified objectid. 376 Example: (as if run before the MOVE above when the mailbox had 3 377 messages) 379 C: 27 search emailid Md8976d99ac3275bb4e918af4 380 S: * SEARCH 1 381 S: 27 OK Completed (1 msgs in 0.000 secs) 382 C: 28 search threadid T4964b478a75b7ea9 383 S: * SEARCH 1 2 384 S: 28 OK Completed (2 msgs in 0.000 secs) 386 7. Formal syntax 388 The following syntax specification uses the Augmented Backus-Naur 389 Form (ABNF) [RFC4234] notation. Elements not defined here can be 390 found in the formal syntax of the ABNF [RFC4234], IMAP [RFC3501], and 391 IMAP ABNF extensions [RFC4466] specifications. 393 Except as noted otherwise, all alphabetic characters are case- 394 insensitive. The use of upper- or lowercase characters to define 395 token strings is for editorial clarity only. Implementations MUST 396 accept these strings in a case-insensitive fashion. 398 capability =/ "OBJECTID" 400 fetch-att =/ "EMAILID" / "THREADID" 402 fetch-emailid-resp = "EMAILID" SP "(" objectid ")" ;; follows tagged- 403 ext production from [RFC4466] 404 fetch-threadid-resp = "THREADID" SP "(" objectid ")" ;; follows 405 tagged-ext production from [RFC4466] 407 objectid = 1*255(ALPHA / DIGIT / "_" / "-") ;; characters in object 408 identifiers are case ;; significant 410 resp-text-code =/ "MAILBOXID" SP "(" objectid ")" ;; incorporated 411 before the expansion rule of ;; atom [SP 1*] ;; that appears in [RFC3501] 414 search-key =/ "EMAILID" SP objectid / "THREADID" SP objectid 416 status-att =/ "MAILBOXID" 418 status-att-value =/ "MAILBOXID" SP "(" objectid ")" ;; follows 419 tagged-ext production from [RFC4466] 421 8. Implementation considerations 423 8.1. Assigning object identifiers 425 All objectid values are allocated by the server. 427 In the interests of reducing the possibilities of encoding mistakes, 428 objectids are restricted to a safe subset of possible byte values, 429 and in order to allow clients to allocate storage, they are 430 restricted in length. 432 An objectid is a string of 1 to 255 characters from the following set 433 of 64 codepoints. a-z, A-Z, 0-9, '_', '-'. These characters are safe 434 to use in almost any context (e.g. filesystems, URIs, IMAP atoms). 436 For maximum safety, servers SHOULD also follow defensive allocation 437 strategies to avoid creating risks where glob completion or data type 438 detection may be present (e.g. on filesystems or in spreadsheets). 439 In particular it is wise to avoid: 441 o ids starting with - 443 o ids starting with digits 445 o ids which contain only digits 447 o ids which differ only by ASCII case (A vs a) 449 o the specific sequence of 3 characters NIL 450 A good solution to these issues is to prefix every ID with a single 451 alphabetical character. 453 8.2. Interaction with special cases 455 The case of RENAME INBOX may need special handling for unique ids. 457 It is advisable (though not required) to have MAILBOXID be globally 458 unique, but they it is only required to be unique within a single 459 server. 461 8.3. Ideas for implementing object identifiers 463 This section contains non-normative suggestions. 465 Ideas for implementing MAILBOXID: 467 o Digest of (MailboxName/UIDVALIDITY) - not kept when renaming, but 468 is guaranteed unique and doesn't require storage. 470 o [RFC4122] UUID 472 o Server assigned sequence number (guaranteed not to be reused) 474 Ideas for implementing EMAILID: 476 o Digest of (MailboxName/UIDVALIDITY/UID) - is not kept when copying 477 messages, but is guaranteed unique and doesn't require storage. 479 o Digest of message content (RFC822 bytes) - expensive unless cached 481 o [RFC4122] UUID 483 o Server assigned sequence number (guaranteed not to be reused) 485 Ideas for implementing THREADID: 487 o Derive from EMAILID of first seen message in the thread. 489 o [RFC4122] UUID 491 o Server assigned sequence number (guaranteed not to be reused) 493 There is a need to index and look up reference/in-reply-to data at 494 message creation to efficiently find matching messages for threading. 495 Threading may be either across folders, or within each folder only. 496 The server has significant leeway here. 498 9. Future considerations 500 This extension is intentionally defined to be compatible with the 501 data model in [I-D.ietf-jmap-mail]. 503 A future extension could be proposed to give a way to SELECT a 504 mailbox by MAILBOXID rather than name. 506 An extension to allow fetching message content directly via EMAILID 507 and message listings by THREADID could be proposed. 509 10. IANA Considerations 511 The IANA is requested to add "OBJECTID" to the "IMAP Capabilities" 512 registry located at . 515 11. Security Considerations 517 It is strongly advised that servers generate OBJECTIDs which are safe 518 to use as filesystem names, and unlikely to be auto-detected as 519 numbers. See implementation considerations. 521 The use of a digest for ID generation may be used as proof that a 522 particular sequence of bytes was seen by the server. 524 12. Changes 526 To be removed by the editor before publication 528 12.1. 01 530 o renamed UNIQUEID (capability) to OBJECTID (and renamed the draft 531 as well) 533 o renamed UNIQUEID (status item) to MAILBOXID 535 o renamed MSGID to EMAILID 537 o renamed THRID to THREADID 539 o restricted objectid to 64 safe characters 541 o added security considerations and advice about choosing objectid 543 o wrapped all responses in () for RFC4466 compatibility 545 o signifiant rewrite of all sections 547 13. Acknowledgments 549 The EXTRA working group at IETF. 551 The gmail team's X-GM-THRID and X-GM-MSGID implementation. 553 14. References 555 14.1. Normative References 557 [I-D.ietf-jmap-mail] 558 Jenkins, N., "JMAP for Mail", draft-ietf-jmap-mail-04 559 (work in progress), March 2018. 561 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 562 Requirement Levels", BCP 14, RFC 2119, 563 DOI 10.17487/RFC2119, March 1997, 564 . 566 [RFC3501] Crispin, M., "INTERNET MESSAGE ACCESS PROTOCOL - VERSION 567 4rev1", RFC 3501, DOI 10.17487/RFC3501, March 2003, 568 . 570 [RFC4234] Crocker, D., Ed. and P. Overell, "Augmented BNF for Syntax 571 Specifications: ABNF", RFC 4234, DOI 10.17487/RFC4234, 572 October 2005, . 574 [RFC4315] Crispin, M., "Internet Message Access Protocol (IMAP) - 575 UIDPLUS extension", RFC 4315, DOI 10.17487/RFC4315, 576 December 2005, . 578 [RFC4466] Melnikov, A. and C. Daboo, "Collected Extensions to IMAP4 579 ABNF", RFC 4466, DOI 10.17487/RFC4466, April 2006, 580 . 582 [RFC5819] Melnikov, A. and T. Sirainen, "IMAP4 Extension for 583 Returning STATUS Information in Extended LIST", RFC 5819, 584 DOI 10.17487/RFC5819, March 2010, 585 . 587 [RFC6851] Gulbrandsen, A. and N. Freed, Ed., "Internet Message 588 Access Protocol (IMAP) - MOVE Extension", RFC 6851, 589 DOI 10.17487/RFC6851, January 2013, 590 . 592 14.2. Informative References 594 [RFC4122] Leach, P., Mealling, M., and R. Salz, "A Universally 595 Unique IDentifier (UUID) URN Namespace", RFC 4122, 596 DOI 10.17487/RFC4122, July 2005, 597 . 599 Author's Address 601 Bron Gondwana (editor) 602 FastMail 603 Level 2, 114 William St 604 Melbourne VIC 3000 605 Australia 607 Email: brong@fastmailteam.com 608 URI: https://www.fastmail.com