idnits 2.17.1 draft-melnikov-imap-disc-04.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** It looks like you're using RFC 3978 boilerplate. You should update this to the boilerplate described in the IETF Trust License Policy document (see https://trustee.ietf.org/license-info), which is required now. -- Found old boilerplate from RFC 3667, Section 5.1 on line 13. -- Found old boilerplate from RFC 3978, Section 5.5 on line 1334. -- Found old boilerplate from RFC 3979, Section 5, paragraph 1 on line 1307. -- Found old boilerplate from RFC 3979, Section 5, paragraph 2 on line 1314. -- Found old boilerplate from RFC 3979, Section 5, paragraph 3 on line 1320. ** The document seems to lack an RFC 3978 Section 5.1 IPR Disclosure Acknowledgement -- however, there's a paragraph with a matching beginning. Boilerplate error? ** This document has an original RFC 3978 Section 5.4 Copyright Line, instead of the newer IETF Trust Copyright according to RFC 4748. ** This document has an original RFC 3978 Section 5.5 Disclaimer, instead of the newer disclaimer which includes the IETF Trust according to RFC 4748. ** The document uses RFC 3667 boilerplate or RFC 3978-like boilerplate instead of verbatim RFC 3978 boilerplate. After 6 May 2005, submission of drafts without verbatim RFC 3978 boilerplate is not accepted. The following non-3978 patterns matched text found in the document. That text should be removed or replaced: By submitting this Internet-Draft, I certify that any applicable patent or other IPR claims of which I am aware have been disclosed, or will be disclosed, and any of which I become aware will be disclosed, in accordance with RFC 3668. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- ** The document seems to lack a 1id_guidelines paragraph about Internet-Drafts being working documents. ** The document seems to lack a 1id_guidelines paragraph about 6 months document validity. ** The document is more than 15 pages and seems to lack a Table of Contents. == No 'Intended status' indicated for this document; assuming Proposed Standard == The page length should not exceed 58 lines per page, but there was 1 longer page, the longest (page 1) being 1379 lines Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack an Introduction section. ** The document seems to lack an IANA Considerations section. (See Section 2.2 of https://www.ietf.org/id-info/checklist for how to handle the case when there are no actions for IANA.) ** There are 293 instances of too long lines in the document, the longest one being 17 characters in excess of 72. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the RFC 3978 Section 5.4 Copyright Line does not match the current year -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (August 2004) is 7187 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Missing Reference: 'HEADER' is mentioned on line 778, but not defined -- Looks like a reference, but probably isn't: '3' on line 787 -- Looks like a reference, but probably isn't: '4' on line 787 -- Looks like a reference, but probably isn't: '5' on line 787 -- Looks like a reference, but probably isn't: '6' on line 787 -- Looks like a reference, but probably isn't: '7' on line 787 -- Looks like a reference, but probably isn't: '8' on line 787 -- Looks like a reference, but probably isn't: '9' on line 788 -- Looks like a reference, but probably isn't: '10' on line 788 -- Looks like a reference, but probably isn't: '11' on line 788 -- Looks like a reference, but probably isn't: '13' on line 788 -- Looks like a reference, but probably isn't: '14' on line 788 -- Looks like a reference, but probably isn't: '15' on line 788 -- Looks like a reference, but probably isn't: '16' on line 788 -- Looks like a reference, but probably isn't: '21' on line 789 == Missing Reference: 'UNSEEN 12' is mentioned on line 1205, but not defined == Missing Reference: 'UIDVALIDITY 3857529045' is mentioned on line 1206, but not defined == Missing Reference: 'HIGHESTMODSEQ 20010715194045007' is mentioned on line 1209, but not defined == Missing Reference: 'IMAP' is mentioned on line 1230, but not defined ** Obsolete normative reference: RFC 3501 (ref. 'IMAP4') (Obsoleted by RFC 9051) ** Obsolete normative reference: RFC 2359 (ref. 'UIDPLUS') (Obsoleted by RFC 4315) -- No information found for draft-melnikov-imap-condstore-XX - is the name correct? -- Possible downref: Normative reference to a draft: ref. 'CONDSTORE' -- No information found for draft-ietf-imapext-acl-XX - is the name correct? Summary: 13 errors (**), 0 flaws (~~), 8 warnings (==), 24 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 IMAPEXT Working Group A. Melnikov 2 Internet Draft: IMAP4 Disconnected Access Editor 3 Document: draft-melnikov-imap-disc-04.txt August 2004 4 Expires: February 2005 6 Synchronization operations for disconnected IMAP4 clients 8 Status of this Memo 10 By submitting this Internet-Draft, I certify that any applicable 11 patent or other IPR claims of which I am aware have been disclosed, or 12 will be disclosed, and any of which I become aware will be disclosed, 13 in accordance with RFC 3668. 15 Internet Drafts are working documents of the Internet Engineering 16 Task Force (IETF), its Areas, and its Working Groups. Note that 17 other groups may also distribute working documents as Internet 18 Drafts. Internet Drafts are draft documents valid for a maximum of 19 six months. Internet Drafts may be updated, replaced, or obsoleted 20 by other documents at any time. It is not appropriate to use 21 Internet Drafts as reference material or to cite them other than as 22 ``work in progress''. 24 The list of current Internet-Drafts can be accessed at 25 http://www.ietf.org/ietf/1id-abstracts.txt 27 The list of Internet-Draft Shadow Directories can be accessed at 28 http://www.ietf.org/shadow.html. 30 This is a draft document based on the expired draft written by 31 the IETF IMAP Working Group. A revised version of this draft document 32 will be submitted to the RFC editor as an Informational (or BCP) RFC for the 33 Internet Community. Discussion and suggestions for improvement are 34 requested, and should be sent to imap@CAC.Washington.EDU. 36 This memo is for informational use and does not constitute a 37 standard. Distribution of this memo is unlimited. 39 Abstract 41 This document attempts to address some of the issues involved in building 42 a disconnected IMAP4 client. In particular, it deals with the issues 43 of what might be called the "driver" portion of the synchronization 44 tool: the portion of the code responsible for issuing the correct set 45 of IMAP4 commands to synchronize the disconnected client in the way 46 that is most likely to make the human who uses the disconnected 47 client happy. 49 This note describes different strategies that can be used by disconnected 50 clients as well as shows how to use IMAP protocol in order to minimize the 51 time of synchronization process. 53 1. Conventions Used in this Document 55 In examples, "C:" and "S:" indicate lines sent by the client and 56 server respectively. 58 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 59 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 60 document are to be interpreted as described in RFC 2119 [KEYWORDS]. 62 Let's call an IMAP command idempotent, if the result of executing the 63 command twice sequentially is the same as the result of executing the 64 command just once. 66 Several recommendations presented in this document are generally 67 applicable to all types of IMAP clients. However this document tries 68 to concentrate on disconnected mail clients. It also suggests some IMAP 69 extensions that should be implemented by IMAP servers in order to make 70 life of disconnected clients easier. 72 Readers of this document are also strongly advised to read RFC 2683 73 [RFC 2683]. 75 Editorial comments/questions or missing paragraphs are marked in the 76 text with << and >>. 78 2. Design Principles 80 All mailbox state or content information stored on the disconnected 81 client should be viewed strictly as a cache of the state of the 82 server. The "master" state remains on the server, just as it would 83 with an interactive IMAP4 client. The one exception to this rule is 84 that information about the state of the disconnected client's cache 85 (the state includes flag changes while offline and scheduled message uploads) 86 remains on the disconnected client: that is, the IMAP4 server is not 87 responsible for remembering the state of the disconnected IMAP4 client. 89 We assume that a disconnected client is a client that, for whatever 90 reason, wants to minimize the length of time that it is "on the 91 phone" to the IMAP4 server. Often this will be because the client is 92 using a dialup connection, possibly with very low bandwidth, but 93 sometimes it might just be that the human is in a hurry to catch an 94 airplane, or some other event beyond our control. Whatever the 95 reason, we assume that we must make efficient use of the network 96 connection, both in the usual sense (not generating spurious traffic) 97 and in the sense that we would prefer not to have the connection 98 sitting idle while the client and/or the server is performing 99 strictly local computation or I/O. Another, perhaps simpler way of 100 stating this is that we assume that network connections are 101 "expensive". 103 Practical experience with disconnected mail systems has shown that 104 there is no single synchronization strategy that is appropriate 105 for all cases. Different humans have different preferences, 106 and the same human's preference will vary depending both on 107 external circumstance (how much of a hurry the human is in today) 108 and on the value that the human places on the messages being 109 transferred. The point here is that there is no way that 110 the synchronization program can guess exactly what the human 111 wants to do, so the human will have to provide some guidance. 113 Taken together, the preceding two principles lead to the conclusion 114 that the synchronization program must make its decisions based on 115 some kind of guidance provided by the human by selecting the appropriate 116 options in UI or through some sort of configuration file, but almost 117 certainly should not pause for I/O with the human during the middle 118 of the synchronization process. The human will almost certainly have 119 several different configurations for the synchronization program, for 120 different circumstances. 122 Since a disconnected client has no way of knowing what changes might 123 have occurred to the mailbox while it was disconnected, message 124 numbers are not useful to a disconnected client. All disconnected 125 client operations should be performed using UIDs, so that the client 126 can be sure that it and the server are talking about the same 127 messages during the synchronization process. 129 3. Overall picture of synchronization 131 The basic strategy for synchronization is outlined below. 132 Note that the real strategy may vary from one application to another 133 or may depend on a synchronization mode. 135 a) Process any "actions" that were pending on the client that 136 were not associated with any mailbox (in particular sending 137 messages composed offline with SMTP. This is not part of IMAP 138 synchronization, but it is mentioned here for completeness); 140 b) Fetch the current list of "interesting" mailboxes (The disconnected 141 client should allow the user to skip this step completely); 143 c) "Client-to-server synchronization" - for each IMAP "action" that 144 were pending on the client: 146 1) If the action implies opening a new mailbox (any operation 147 that operates on messages) - open the mailbox. Check its UID 148 validity value (see section 4.1 for more details) returned in 149 the UIDVALIDITY response code. If the UIDVALIDITY value returned 150 by the server differs, the client MUST empty the local cache of 151 the mailbox and remove any pending "actions" which refer to UIDs 152 in that mailbox (and consider them failed). Note, this doesn't 153 affect actions performed on client generated fake UIDs (see 154 section 5). 156 2) Perform the action. If the action is to delete a mailbox (DELETE), 157 make sure that the mailbox is closed first. 159 d) "Server-to-client synchronization" - for each mailbox that requires 160 synchronization, do the following: 162 1) Check the mailbox UIDVALIDITY (see section 4.1 for more details). 163 with SELECT/EXAMINE/STATUS. 164 If UIDVALIDITY value returned by the server differs, 165 the client MUST 167 * empty the local cache of that mailbox; 168 * remove any pending "actions" which refer to UIDs in 169 that mailbox and consider them failed; 170 * skip step 2-II; 172 2) Fetch the current "descriptors"; 174 I) Discover new messages. 176 II) Discover changes to old messages. 178 3) Fetch the bodies of any "interesting" messages that the client 179 doesn't already have. 181 d) Close all open mailboxes not required for further operations 182 (if staying online) or disconnect all open connections (if going 183 offline). 185 Terms used: 187 "Actions" are queued requests that were made by the human to the 188 client's MUA software while the client was disconnected. 190 Let define "descriptors" as a set of IMAP4 FETCH data items. 191 Conceptually, a message's descriptor is that set of 192 information that allows the synchronization program to decide what 193 protocol actions are necessary to bring the local cache to the 194 desired state for this message; since this decision is really up 195 to the human, this information probably includes a at least a few 196 header fields intended for human consumption. Exactly what will 197 constitute a descriptor depends on the client implementation. At 198 a minimum, the descriptor contains the message's UID and FLAGS. 199 Other likely candidates are the RFC822.SIZE, RFC822.HEADER and 200 BODYSTRUCTURE data items. 202 Comments: 204 1). The list of actions should be ordered. E.g., if the human deletes 205 message A1 in mailbox A, then expunges mailbox A, then deletes 206 message A2 in mailbox A, the human will expect that message A1 is 207 gone and that message A2 is still present but is now deleted. 209 By processing all the actions before proceeding with 210 synchronization, we avoid having to compensate for the local MUA's 211 changes to the server's state. That is, once we have processed 212 all the pending actions, the steps that the client must take to 213 synchronize itself will be the same no matter where the changes to 214 the server's state originated. 216 2). Steps a) and b) can be performed in parallel. Alternatively step a) 217 can be performed after d). 219 3). On step b) the set of "interesting" mailboxes pretty much has to be 220 determined by the human. What mailboxes belong to this set may 221 vary between different IMAP4 sessions with the same server, 222 client, and human. An interesting mailbox can be a mailbox 223 returned by LSUB command. Special mailbox "INBOX" SHOULD always 224 be considered "interesting". 226 4). On step d-2-II) the client also finds out about 227 changes to the flags of messages that the client already has in 228 its local cache, as well as finding out about messages in the 229 local cache that no longer exist on the server (i.e., messages that 230 have been expunged). 232 5). "Interesting" messages are those messages that the synchronization 233 program thinks the human wants to have cached locally, based on 234 the configuration and the data retrieved in step (b). 236 The rest of this discussion will focus primarily on the synchronization 237 issues for a single mailbox. 239 4. Mailbox synchronization steps and strategies 241 4.1. Checking UID Validity 243 The "UID validity" of a mailbox is a number returned in an 244 UIDVALIDITY response code in an OK untagged response at mailbox 245 selection time. The UID validity value changes between sessions when 246 UIDs fail to persist between sessions. 248 Whenever the client selects a mailbox, the client must compare the 249 returned UID validity value with the value stored in the local cache. 250 If the UID validity values differ, the UIDs in the client's cache are 251 no longer valid. The client MUST then empty the local cache of 252 that mailbox and remove any pending "actions" which refer to UIDs in 253 that mailbox. The client MAY also issue a warning to the human. 254 The client MUST NOT cancel any scheduled uploads (i.e. APPENDs) for 255 the mailbox. 257 Note that UIDVALIDITY is not only returned on a mailbox selection. 258 COPYUID and APPENDUID response codes defined in [UIDPLUS] extension 259 (see also 4.2.2) and UIDVALIDITY STATUS response data item also contain 260 a UIDVALIDITY value for some other mailbox. The client SHOULD behave as 261 described in the previous paragraph (but it should act on the other mailbox's 262 cache), no matter how it obtained the UIDVALIDITY value. 264 4.2. Synchronizing local changes with the server 266 4.2.1. Uploading messages to the mailbox 268 There are two most typical examples of operations that will result in message 269 uploads: 271 1) Saving a draft message 272 2) Message copy between remote mailboxes on two different IMAP servers 273 or a local mailbox and a remote mailbox. 275 Message upload is performed with APPEND command. A message scheduled to be 276 uploaded has no UID associated with it, as all UIDs are assigned by the 277 server. The APPEND command will effectively associate a UID with the uploaded 278 message that can be stored in the local cache for a future reference. 279 However [IMAP4] doesn't describe a simple mechanism to discover the message UID 280 by just performing the APPEND command. In order to discover UID the client can 281 do one of the following: 283 1) Remove the uploaded message from cache. After that use the mechanism described 284 in 4.3 to fetch the information about the uploaded message as if it was uploaded 285 by some other client. 287 2) Try to fetch header information as described in 4.2.2 in order to find a message 288 that corresponds to the uploaded message. One strategy of doing that is described 289 in 4.2.2. 291 Case 1) describes a non particularly smart client. 293 C: A003 APPEND Drafts (\Seen $MDNSent) {310} 294 S: + Ready for literal data 295 C: Date: Mon, 7 Feb 1994 21:52:25 -0800 (PST) 296 C: From: Fred Foobar 297 C: Subject: afternoon meeting 298 C: To: mooch@owatagu.siam.edu 299 C: Message-Id: 300 C: MIME-Version: 1.0 301 C: Content-Type: TEXT/PLAIN; CHARSET=US-ASCII 302 C: 303 C: Hello Joe, do you think we can meet at 3:30 tomorrow? 304 C: 305 S: A003 OK APPEND Completed 307 Fortunately there is a simpler way to discover the message UID in the presence 308 of [UIDPLUS] extension: 310 C: A003 APPEND Drafts (\Seen $MDNSent) {310} 311 S: + Ready for literal data 312 C: Date: Mon, 7 Feb 1994 21:52:25 -0800 (PST) 313 C: From: Fred Foobar 314 C: Subject: afternoon meeting 315 C: To: mooch@owatagu.siam.edu 316 C: Message-Id: 317 C: MIME-Version: 1.0 318 C: Content-Type: TEXT/PLAIN; CHARSET=US-ASCII 319 C: 320 C: Hello Joe, do you think we can meet at 3:30 tomorrow? 321 C: 322 S: A003 OK APPEND [APPENDUID 1022843275 77712] completed 324 The UID of the appended message is the second parameter of APPENDUID response code. 326 4.2.2. Optimizing "move" and "copy" operations 328 Practical experience with IMAP, and other mailbox access 329 protocols that support multiple mailboxes suggests that moving a 330 message from one mailbox to another is an extremely common operation. 332 4.2.2.1. Moving a message between two mailboxes on the same server 334 In IMAP4 a "move" operation between two mailboxes on the same server 335 is really a combination of a COPY operation and a STORE +FLAGS (\Deleted) 336 operation. This makes good protocol sense for IMAP, but it leaves 337 a simple-minded disconnected client in the silly position of deleting 338 and possibly expunging its cached copy of a message, then fetching 339 an identical copy via the network. 341 However, the presence of UIDPLUS extension support in the server can help: 342 A001 UID COPY 567,414 "Interesting Messages" 343 A001 OK [COPYUID 1022843275 414,567 5:6] Completed 344 This tells the client that the message with UID 414 in the current mailbox 345 was successfully copied to the mailbox "Interesting Messages" and was given 346 the UID 5, and that the message with UID 567 was given the UID 6. 348 In the absence of UIDPLUS extension support in the server the following 349 trick can be used. By including the Message-ID: header and the INTERNALDATE 350 data item as part of the descriptor, the client can check the descriptor of a 351 "new" message against messages that are already in its cache, and 352 avoid fetching the extra copy. Of course, it's possible that the 353 cost of checking to see if the message is already in the local cache 354 may exceed the cost of just fetching it, so this technique should not 355 be used blindly. If the MUA implements a "move" command, it make 356 special provisions to use this technique when it knows that a 357 copy/delete sequence is the result of a "move" command. 359 Since it's theoretically possible for this algorithm to find the 360 wrong message (given sufficiently malignant Message-ID headers), 361 implementors should provide a way to disable this optimization, both 362 permanently and on a message-by-message basis. 364 << Example >> 366 4.2.2.2. Moving a message from a remote mailbox to a local 368 Moving a message from a remote mailbox to a local is done with FETCH 369 (that includes FLAGS and INTERNALDATE) followed by 370 UID STORE +FLAGS.SILENT (\Deleted): 372 C: A003 UID FETCH 123 (RFC822 INTERNALDATE FLAGS) 373 S: * 27 FETCH (UID 123 INTERNALDATE "31-May-2002 05:26:59 -0600" 374 FLAGS (\Seen $MDNSent) RFC822 375 S: ...message body... 376 S: ) 377 S: A003 OK UID FETCH completed 378 C: A004 UID STORE +FLAGS.SILENT (\Deleted) 379 S: A004 STORE completed 381 Note, that there is no reason to fetch the message during synchronization 382 if it already in the client's cache. Also, the client SHOULD preserve 383 delivery date in the local cache. 385 4.2.2.3. Moving a message from a local mailbox to a remote 387 Moving a message from a local mailbox to a remote is done with APPEND: 389 C: A003 APPEND Drafts (\Seen $MDNSent) "31-May-2002 05:26:59 -0600" {310} 390 S: + Ready for literal data 391 C: Date: Mon, 7 Feb 1994 21:52:25 -0800 (PST) 392 C: From: Fred Foobar 393 C: Subject: afternoon meeting 394 C: To: mooch@owatagu.siam.edu 395 C: Message-Id: 396 C: MIME-Version: 1.0 397 C: Content-Type: TEXT/PLAIN; CHARSET=US-ASCII 398 C: 399 C: Hello Joe, do you think we can meet at 3:30 tomorrow? 400 C: 401 S: A003 OK APPEND [APPENDUID 1022843275 77712] completed 403 The client SHOULD specify delivery date from the local cache in the APPEND. 405 4.2.2.4. Moving a message between two mailbox on two different servers 407 Moving a message between two mailbox on two different servers is a 408 combination of the operations described in 4.2.2.2 followed by the 409 operations described in 4.2.2.3. 411 4.2.2.5. Uploading multiple messages to a remote mailbox with MULTIAPPEND 413 When there is a need to upload multiple messages to a remote mailbox 414 (e.g. as per 4.2.2.3), the presence of certain IMAP extensions may 415 significantly improve performance. One of them is [MULTIAPPEND]. 417 For some mail stores opening a mailbox for appending might be expensive. 418 [MULTIAPPEND] tells the server to open mailbox once (instead of opening 419 and closing it "n" times per "n" messages to be uploaded) and keep it 420 open while a group of messages is being uploaded to the server. 422 Also, if the server supports both [MULTIAPPEND] and [LITERAL+] extensions, 423 the entire upload is accomplished in a single command/response round trip. 425 Note: Client implementors should be aware, that [MULTIAPPEND] performs 426 append of multiple messages atomically. This means, for example, 427 if there is not enough space to save "n"-th message (or the message 428 has invalid structure and is rejected by the server) after successful 429 upload of "n-1" messages, the whole upload operation fails and no 430 message will be saved in the mailbox. Although, this behavior might 431 be desirable in certain situations, it might not be what you want. 432 See also section 5.1 for discussions about error recovery. 434 Note: MULTIAPPEND can be used together with UIDPLUS extension in a way 435 similar to what was described in section 4.2.1. [MULTIAPPEND] 436 extends syntax of APPENDUID response code to allow for multiple 437 message UIDs in the second parameter. 439 Example: 440 An example below demonstrates the use of MULTIAPPEND together with 441 UIDPLUS (synchronization points where the client waits for confirmations 442 from the server are marked with "<--->"): 444 C: A003 APPEND Jan-2002 (\Seen $MDNSent) "31-May-2002 05:26:59 -0600" {310} 445 <---> 446 S: + Ready for literal data 447 C: Date: Mon, 7 Feb 1994 21:52:25 -0800 (PST) 448 C: From: Fred Foobar 449 C: Subject: afternoon meeting 450 C: To: mooch@owatagu.siam.edu 451 C: Message-Id: 452 C: MIME-Version: 1.0 453 C: Content-Type: TEXT/PLAIN; CHARSET=US-ASCII 454 C: 455 C: Hello Joe, do you think we can meet at 3:30 tomorrow? 456 C: (\Seen) " 1-Jun-2002 22:43:04 -0800" {286} 457 <---> 458 S: + Ready for literal data 459 C: Date: Mon, 7 Feb 1994 22:43:04 -0800 (PST) 460 C: From: Joe Mooch 461 C: Subject: Re: afternoon meeting 462 C: To: foobar@blurdybloop.com 463 C: Message-Id: 464 C: MIME-Version: 1.0 465 C: Content-Type: TEXT/PLAIN; CHARSET=US-ASCII 466 C: 467 C: 3:30 is fine with me. 468 C: 469 S: A003 OK APPEND [APPENDUID 1022843275 77712,77713] completed 471 The upload takes 3 round trips. 473 Example: 474 The example above was modified for the case when the server supports 475 MULTIAPPEND, LITERAL+ and UIDPLUS. The upload takes only 1 round trip. 477 C: A003 APPEND Jan-2002 (\Seen $MDNSent) "31-May-2002 05:26:59 -0600" {310+} 478 C: Date: Mon, 7 Feb 1994 21:52:25 -0800 (PST) 479 C: From: Fred Foobar 480 C: Subject: afternoon meeting 481 C: To: mooch@owatagu.siam.edu 482 C: Message-Id: 483 C: MIME-Version: 1.0 484 C: Content-Type: TEXT/PLAIN; CHARSET=US-ASCII 485 C: 486 C: Hello Joe, do you think we can meet at 3:30 tomorrow? 487 C: (\Seen) " 1-Jun-2002 22:43:04 -0800" {286+} 488 C: Date: Mon, 7 Feb 1994 22:43:04 -0800 (PST) 489 C: From: Joe Mooch 490 C: Subject: Re: afternoon meeting 491 C: To: foobar@blurdybloop.com 492 C: Message-Id: 493 C: MIME-Version: 1.0 494 C: Content-Type: TEXT/PLAIN; CHARSET=US-ASCII 495 C: 496 C: 3:30 is fine with me. 497 C: 498 S: A003 OK APPEND [APPENDUID 1022843275 77712,77713] completed 500 4.2.3. Replaying local flag changes 502 The disconnected client uses STORE command to synchronize local flag state 503 with the server. The disconnected client SHOULD use +FLAGS.SILENT or -FLAGS.SILENT 504 in order to set or unset flags modified by the user while offline. FLAGS 505 form must not be used, as there is a risk that this will overwrite flags 506 on the server that has been changed by some other client. 508 Example: 509 For the message with UID 15, the disconnected client stores the following 510 flags \Seen and $Highest. The flags were modified on the server by some other 511 client: \Seen, \Answered and $Highest. 512 While offline the user requested to remove $Highest flags and to add \Deleted. 513 The flag synchronization sequence for the message should look like: 515 C: A001 UID STORE 15 +FLAGS.SILENT (\Deleted) 516 S: A001 STORE completed 517 C: A002 UID STORE 15 -FLAGS.SILENT ($Highest) 518 S: A002 STORE completed 520 If the disconnected client is able to store an additional binary state 521 information (or a piece of information that can take a value from a predefined 522 set of values) in the local cache of an IMAP mailbox or in a local mailbox 523 (e.g. message priority), and if the server supports storing of arbitrary 524 keywords, the client MUST use keywords to store this state on the server. 526 Example: 527 Imagine a speculative mail client that can mark a message as one of work-related 528 ($Work), personal ($Personal) or spam ($Spam). In order to mark a message as 529 personal the client issues: 531 C: A001 UID STORE 15 +FLAGS.SILENT ($Personal) 532 S: A001 STORE completed 533 C: A002 UID STORE 15 -FLAGS.SILENT ($Work $Spam) 534 S: A002 STORE completed 536 In order to mark the message as neither work, nor personal, not spam, the client 537 issues: 539 C: A003 UID STORE 15 -FLAGS.SILENT ($Personal $Work $Spam) 540 S: A003 STORE completed 542 4.2.4. Processing mailbox compression (EXPUNGE) requests 544 A naive disconnected client implementation that supports compressing a mailbox 545 while offline may decide to issue an EXPUNGE command to the server in order 546 to expunge messages marked \Deleted. The problem with this command during 547 synchronization is that it permanently erases all messages with \Deleted flag set, 548 i.e. even those messages that were marked as \Deleted on the server while the user 549 was offline. Doing so might lead the user to an unpleasant surprise. 551 Fortunately [UIDPLUS] extension can help in this case as well. The extension 552 introduces UID EXPUNGE command, that, unlike EXPUNGE, takes a UID set parameter, 553 that lists UIDs of all messages that can be expunged. When processing this command 554 server erases only messages with \Deleted flag listed in the UID list. Thus, 555 a message not listed in the UID set will not be expunged even if it has \Deleted 556 flag set. 558 Example: While offline 3 messages with UIDs 7, 27 and 65 were marked \Deleted 559 when the user requested to compress the open mailbox. Another client marked 560 a message \Deleted on the server (UID 34). During synchronization the 561 disconnected client issues: 563 C: A001 UID EXPUNGE 7,27,65 564 S: * ... EXPUNGE 565 S: * ... EXPUNGE 566 S: * ... EXPUNGE 567 S: A001 UID EXPUNGE completed 569 If another client issues UID SEARCH DELETED command (to find all messages with 570 \Deleted flag) before and after the UID EXPUNGE it will get: 572 Before: 573 C: B001 UID SEARCH DELETED 574 S: * SEARCH 65 34 27 7 575 S: B001 UID SEARCH completed 577 After: 578 C: B002 UID SEARCH DELETED 579 S: * SEARCH 34 580 S: B002 UID SEARCH completed 582 In the absence of [UIDPLUS] extension the following sequence of command can be 583 used as an approximation. Note, that when this sequence is performed, there is a 584 possibility that another client marks additional messages as deleted and these 585 messages will be expunged as well. 587 1). Find all messages marked \Deleted on the server: 589 C: A001 UID SEARCH DELETED 590 S: * SEARCH 65 34 27 7 591 S: A001 UID SEARCH completed 593 2). Find all messages that must not be erased (for the previous example 594 the list will consist of the message with UID 34) 596 3). Temporary remove \Deleted flag on all messages found in step 2) 598 C: A002 UID STORE 34 -FLAGS.SILENT (\Deleted) 599 S: A002 UID STORE completed 601 4). Expunge the mailbox 603 C: A003 EXPUNGE 604 S: * 20 EXPUNGE 605 S: * 7 EXPUNGE 606 S: * 1 EXPUNGE 607 S: A003 EXPUNGE completed 609 Here message with UID 7 has message number 1; with UID 27 - message 610 number 7 and with UID 65 - message number 20. 612 5). Restore \Deleted flag on all messages found when performing step 2) 614 C: A004 UID STORE 34 +FLAGS.SILENT (\Deleted) 615 S: A004 UID STORE completed 617 4.2.5. Closing a mailbox 619 When the disconnected client has to close a mailbox, it should not use 620 CLOSE command, because CLOSE does a silent EXPUNGE (section 4.2.4 explains 621 why EXPUNGE should not be used by a disconnected client). It is safe to use 622 CLOSE only if the mailbox was opened with EXAMINE. 624 If the mailbox was opened with SELECT, the client can use one of the 625 following commands to implicitly close the mailbox and prevent the silent 626 expunge: 628 1). UNSELECT - This is a command described in [UNSELECT] that works as 629 CLOSE, but doesn't cause the silent EXPUNGE. This command is 630 supported by the server if it reports UNSELECT in its CAPABILITY list. 631 2). SELECT - SELECT causes implicit CLOSE without EXPUNGE. 632 3). If the client intends to issue LOGOUT after closing the mailbox, it may 633 just issue LOGOUT, because LOGOUT causes implicit CLOSE without EXPUNGE 634 as well. 635 4). SELECT - if the client knows a mailbox that doesn't 636 exist or can't be selected, it MAY SELECT it. 638 If the client opened the mailbox with SELECT and just wants to avoid 639 implicit EXPUNGE without closing the mailbox, it may also use the following: 641 5). EXAMINE - reselect the same mailbox in read-only mode. 643 4.3. Details of "Normal" synchronization of a single mailbox 645 The most common form of synchronization is where the human trusts the 646 integrity of the client's copy of the state of a particular mailbox, 647 and simply wants to bring the client's cache up to date so that it 648 accurately reflects the mailbox's current state on the server. 650 4.3.1. Discovering new messages and changes to old messages 652 Let represent the highest UID that the client knows about 653 in this mailbox. Since UIDs are allocated in strictly ascending 654 order, this is simply the UID of the last message in the mailbox that 655 the client knows about. Let represent 's UID 656 plus one. Let represent a list consisting of all the 657 FETCH data item items that the implementation considers to be part of 658 the descriptor; at a minimum this is just the FLAGS data item, but 659 it usually also includes BODYSTRUCTURE and RFC822.SIZE. At this step 660 SHOULD NOT include RFC822. 662 With no further information, the client can issue the following 663 two commands: 664 tag1 UID FETCH :* 665 tag2 UID FETCH 1: FLAGS 667 The first command will request some information about "new" messages 668 (i.e. messages received by the server since the last synchronization). 669 It will also allow the client to build a message number to UID map 670 (only for new messages). The second command allows the client to 671 1) update cached flags for old messages; 672 2) find out which old messages got expunged; 673 3) build a mapping between message numbers and UIDs (for old messages). 675 The order here is significant. We want the server to start returning 676 the list of new message descriptors as fast as it can, so that the 677 client can start issuing more FETCH commands, so we start out by 678 asking for the descriptors of all the messages we know the client 679 cannot possibly have cached yet. The second command fetches the 680 information we need to determine what changes may have occurred to 681 messages that the client already has cached. Note, that the latter 682 command should only be issued if the UIDNEXT value cached by the client 683 differs from the one returned by the server. Once the client has 684 issued these two commands, there's nothing more the client can do 685 with this mailbox until the responses to the first command start 686 arriving. A clever synchronization program might use this time to 687 fetch its local cache state from disk, or start the process of 688 synchronizing another mailbox. 690 Example of the first FETCH: 691 C: A011 UID fetch 131:* (FLAGS BODYSTRUCTURE INTERNALDATE RFC822.SIZE) 692 S: ... 694 The second FETCH command will result in nil or more untagged fetch 695 responses. Each response will have a corresponding UID FETCH data item. 696 All messages that didn't have a matching untagged FETCH response 697 MUST be removed from the local cache. 699 For example, if the had a value 15000 and the local cache 700 contained 3 messages with the UIDs 12, 777 and 14999 respectively, than 701 after receiving the following responses from the server: 703 S: * 1 FETCH (UID 12 FLAGS (\Seen)) 704 S: * 2 FETCH (UID 777 FLAGS (\Answered \Deleted)) 706 the client must remove the message with UID 14999 from its local cache. 708 Note: If the client is not interested in flag changes (i.e. the client 709 only wants to know which old messages are still on the server), the second 710 FETCH command can be substituted with: 711 tag2 UID SEARCH UID 1: 713 This command will generate less traffic. However an implementor should be 714 aware that in order to build the mapping table from message numbers to UIDs 715 the output of the SEARCH command MUST be sorted first, because there is 716 no requirement for a server to return UIDs in SEARCH response in the ascending 717 order. 719 4.3.2. Searching for "interesting" messages. 721 This step is either performed entirely on the client (from the information received 722 in step 4.3.1), after performing additional searches on the server or both. 723 The decision on what is an "interesting" message is up to the client software 724 and the human. One easy criterion that should probably be implemented in any 725 client is whether the message is "too big" for automatic retrieval, where "too big" 726 is a parameter defined in the client's configuration. 728 Another commonly used criteria is the age of a message. For example, the client 729 may choose to download only messages received in the last week (in this case, 730 would be today's date minus 7 days): 732 tag3 UID SEARCH UID SINCE 734 Keep in mind that a date search disregards time and timezone. 735 The client can avoid doing this search if it specified INTERNALDATE in 736 on step 4.3.1. If the client did, it can perform the search itself. 738 At this step the client also decides what kind of information about a particular 739 message to fetch from the server. In particular, even for a message that is considered 740 to be "too big" the client MAY choose to fetch some part(s) of it. For example, 741 if the message is a multipart/mixed containing a text part and a MPEG attachment, 742 there is no reason for the client not to fetch the text part. The decision of which 743 part should or should not be fetched can be based on the information received in 744 BODYSTRUCTURE FETCH response data item (i.e. if BODYSTRUCTURE was included in 745 on step 4.3.1). 747 4.3.3. Populating cache with "interesting" messages. 749 Once the client found out which messages are "interesting", the client 750 can start issuing appropriate FETCH commands for "interesting" messages or 751 bodyparts thereof. 753 It is important to note that fetching a message into the disconnected 754 client's local cache does NOT imply that the human has (or even will) 755 read the message. Thus, the synchronization program for a 756 disconnected client should always be careful to use the .PEEK 757 variants of the FETCH data items that implicitly set the \Seen flag. 759 Once the last descriptor has arrived and the last FETCH command has 760 been issued, the client simply needs to process the incoming fetch 761 items, using them to update the local message cache. 763 In order to avoid deadlock problems, the client must give processing 764 of received messages priority over issuing new FETCH commands during 765 this synchronization process. This may necessitate temporary local 766 queuing of FETCH requests that cannot be issued without causing a 767 deadlock. In order to achieve the best use of the "expensive" network 768 connection, the client will almost certainly need to pay careful 769 attention to any flow-control information that it can obtain from the 770 underlying transport connection (usually a TCP connection). 772 Example: After fetching a message BODYSTRUCTURE the client discovers 773 a complex MIME message. Than it decides to fetch MIME headers 774 of the nested MIME messages and some body parts. 776 C: A011 UID fetch 11 (BODYSTRUCTURE) 777 S: ... 778 C: A012 UID fetch 11 (BODY[HEADER] BODY[1.MIME] BODY[1.1.MIME] 779 BODY[1.2.MIME] BODY[2.MIME] BODY[3.MIME] BODY[4.MIME] BODY[5.MIME] 780 BODY[6.MIME] BODY[7.MIME] BODY[8.MIME] BODY[9.MIME] BODY[10.MIME] 781 BODY[11.MIME] BODY[12.MIME] BODY[13.MIME] BODY[14.MIME] BODY[15.MIME] 782 BODY[16.MIME] BODY[17.MIME] BODY[18.MIME] BODY[19.MIME] BODY[20.MIME] 783 BODY[21.MIME]) 784 S: ... 785 C: A013 UID fetch 11 (BODY[1.1] BODY[1.2]) 786 S: ... 787 C: A014 UID fetch 11 (BODY[3] BODY[4] BODY[5] BODY[6] BODY[7] BODY[8] 788 BODY[9] BODY[10] BODY[11] BODY[13] BODY[14] BODY[15] BODY[16] 789 BODY[21]) 790 S: ... 792 4.3.4. User initiated synchronization 794 After the client finished the main synchronization that was described in 795 4.3.1-4.3.3 the user may optionally request additional synchronization steps 796 while the client is still online. This is not any different from the process 797 described in 4.3.2 and 4.3.3. 799 Typical examples are: 800 1) fetch all messages selected in UI. 801 2) fetch all messages marked as \Flagged on the server. 803 4.4. Special case: descriptor-only synchronization 805 For some mailboxes, fetching the descriptors might be the entire 806 synchronization step. Practical experience with IMAP has shown that 807 a certain class of mailboxes (e.g., "archival" mailboxes) are used 808 primarily for long-term storage of important messages that the human 809 wants to have instantly available on demand but does not want 810 cluttering up the disconnected client's cache at any other time. 811 Messages in this kind of mailbox would be fetched exclusively by 812 explicit actions queued by the local MUA. Thus, the only 813 synchronization that is necessary for a mailbox of this kind is 814 fetching the descriptor information that the human will use to 815 identify messages that should be explicitly fetched. 817 Special mailboxes that receive traffic from a high volume, low 818 priority mailing list might also be in this category, at least when 819 the human is in a hurry. 821 4.5. Special case: fast new-only synchronization 823 In some cases the human might be in such a hurry that s/he doesn't 824 care about changes to old messages, just about new messages. In this 825 case, the client can skip the UID FETCH command that obtains the 826 flags and UIDs for old messages (1:). 828 4.6. Special case: blind FETCH 830 In some cases the human may know (for whatever reason) that s/he 831 always wants to fetch any new messages in a particular mailbox, 832 unconditionally. In this case, the client can just fetch the 833 messages themselves, rather than just the descriptors, by using a 834 command like: 835 tag1 UID FETCH :* (FLAGS RFC822.PEEK) 837 Note, that this example ignores the fact that the messages can 838 be arbitrary long. The disconnected client MUST always check 839 for message size before downloading, unless explicitly told otherwise. 840 A good behaving client should use instead something like the following: 842 1) Issue "tag1 UID FETCH :* (FLAGS RFC822.SIZE)" 843 2) From the message sizes returned in step 1 construct UID set 844 845 3) Issue "tag2 UID FETCH (RFC822.PEEK)" 847 or 849 1) Issue "tag1 UID FETCH :* (FLAGS)" 850 2) Construct UID set from the responses of 1) 851 3) Issue "tag2 SEARCH UID SMALLER " 852 Construct UID set from the result of 853 the SEARCH command. 854 4) Issue "tag3 UID FETCH (RFC822.PEEK)" 856 5. Implementation considerations 858 Below are listed some common implementation pitfalls that should be 859 considered when implementing a disconnected client. 861 1) Implementing fake UIDs on the client. 863 A message scheduled to be uploaded has no UID, as UIDs are selected by 864 the server. The client may implement fake UIDs internally in order to 865 reference not yet uploaded messages in further operations. For example, 866 message was scheduled to be uploaded and later marked deleted or copied 867 to another mailbox). However, the client MUST NOT under any circumstances 868 sent these fake UIDs to the server. Also, client implementors should 869 be reminded that according to [IMAP4] an UID is a 32bit unsigned integer 870 excluding 0. So, both 4294967295 and 2147483648 are valid UIDs and 0 and -1 871 are both invalid. Existing disconnected mail clients are known to send 872 negative numbers (e.g. "-1") as message UIDs to servers during synchronization. 874 Example 1: The user starts composing a new message, edits it, saves it, 875 continues to edit and saves it again. 877 A disconnected client may record in its replay log (log of operations 878 to be replayed on the server during synchronization) the sequence of 879 operations as shown below. For the purpose of this example we assume 880 that all draft messages are stored in the mailbox called Drafts on an 881 IMAP server. We will also use the following conventions: 882 UID of the intermediate version of the draft when it was saved 883 for the first time. This is a fake UID generated on the client. 884 UID of the final version of the draft. This is another fake UID 885 generated on the client. 887 1). APPEND Drafts (\Seen $MDNSent \Drafts) {} 888 ...first version of the message follows... 889 2). APPEND Drafts (\Seen $MDNSent \Drafts) {} 890 ...final version of the message follows... 891 3). STORE +FLAGS (\Deleted) 893 Step 1 corresponds to the first attempt to save the draft message, 894 step 2 corresponds to the second attempt to save the draft message 895 and the step 3 deletes the first version of the draft message saved 896 in step 1. 898 A naive disconnected client may send the command in step 3 without 899 replacing the fake client generated with the value returned 900 by the server in step 1. 902 2) Section 5.1 talks about common implementation errors related to error 903 recovery during playback. 905 3) Don't assume that the disconnected client is the only client used by 906 the user. 908 <> 909 Example 2: Some clients may use the \Deleted flag as an indicator that 910 the message should not appear in the user's view. Usage of the \Deleted 911 flag for this purpose is not safe, as other clients (e.g. online 912 clients) might EXPUNGE the mailbox at any time. 914 4) Beware of "data" dependencies between synchronization operations. 916 It might be very tempting for a client writer to perform some optimizations 917 on playback log. Such optimizations might include removing redundant 918 operations (see also optimization #2 in section 5.3), operation reordering, etc. 920 It is not always safe to reorder or remove redundant operations during 921 synchronization, because some operations may have dependencies. So if 922 in doubt, don't do that. The following example demonstrates this: 924 Example 3: The user copies a message out of a mailbox and then deletes 925 the mailbox. 927 C: A001 SELECT Old-Mail 928 S: ... 929 C: A002 UID COPY 111 ToDo 930 S: A002 OK [COPYUID 1022843345 111 94] Copy completed 931 ... 932 C: A015 CLOSE 933 S: A015 OK Completed 934 C: A016 DELETE Old-Mail 935 S: A016 OK Mailbox deletion completed successfully 937 If the client performs DELETE (tag A016) first and COPY (tag A002) second, 938 than the COPY fails. 940 5.1. Error recovery during playback 942 Error recovery during synchronization is one of the trickiest parts 943 to get right. Below, we will discuss certain error conditions 944 and suggest possible choices to handle them: 946 1). Lost connection to the server. 948 The client MUST remember the current position in playback (replay) log and 949 replay it starting from the interrupted operation (the last command 950 issued by the client, but not acknowledged by the server) next time it is 951 successfully connected to the same server. If the connection was lost while 952 executing a non-idempotent IMAP command (see definition in Section 1), when 953 reconnected the client MUST make sure that the interrupted command was 954 indeed not executed. If it wasn't executed, the client must restart playback 955 from the interrupted command, otherwise from the following command. 957 When reconnected, care must be taken in order to properly reapply logical 958 operations that are represented by multiple IMAP commands, e.g. UID EXPUNGE 959 emulation when UID EXPUNGE is not supported by the server (see section 4.2.4). 961 Once the client detects that the connection to the server was lost, 962 it MUST stop replaying log. There are existing disconnected clients 963 that, to the great annoyance of a user, will pop up an error dialog 964 for each playback operation that has failed. 966 2). Copying/appending messages to a mailbox that doesn't exist. 968 The user should be advised about the situation and be given 969 one of the following choices: 970 a). Try to recreate a mailbox; 971 b). Copy/upload messages to another mailbox; 972 c). Skip copy/upload. 973 d). Abort replay. 975 3). Copying messages from, rename or get/change ACLs [ACL] on 976 a mailbox that doesn't exist: 978 a). Skip operation 979 b). Abort replay 981 4). Deleting mailbox or deleting/expunging a message that no longer exist. 983 This is actually is not an error and should be ignored by the client. 985 5). Message to perform operation on doesn't exist. 987 a). Skip operation 988 b). Abort replay 990 In the case of changing flags on an expunged message the client should 991 silently ignore the error. 993 Note 1: Several synchronization operations map to multiple IMAP commands 994 (for example "move" described in 4.2.2). The client must guaranty 995 atomicity of each such multistep operation. For example, 996 when performing "move" between two mailboxes on the same server, 997 if the server is unable to copy messages, the client MUST NOT attempt to 998 set the \Deleted flag on the messages being copied, all the more expunging 999 them. However the client MAY consider that move operation succeeded 1000 even if the server was unable to set \Deleted flag on copied messages. 1002 Note 2: Many synchronization operations have "data" dependencies. 1003 A failed operation must cause all dependent operations to fail as 1004 well. The client should check that and MUST NOT try to perform 1005 all dependent operations blindly (unless the user corrected the original 1006 problem). For example, a message may be scheduled to be appended to 1007 a mailbox on the server and later on the appended message may be copied 1008 to another mailbox. If the APPEND operation fails, the client must not 1009 attempt to COPY the failed message later on. (See also Section 5, example 3). 1011 5.2. Quality of implementation issues. 1013 Below listed some quality of implementation issues for disconnected clients. 1014 They will help to write a disconnected client that works correctly, performs 1015 synchronization as quickly as possible (and thus can make the user 1016 happier as well as save her some money) and minimizes the server load: 1018 1) Don't loose information. 1020 No matter how smart your client is other areas, if it looses information - 1021 users will get very upset. 1023 2) Don't do work unless explicitly asked. Be flexible. Ask all questions 1024 BEFORE starting synchronization, if possible. 1026 3) Minimize traffic 1028 The client MUST NOT issue a command if the client already received 1029 the required information from the server. 1031 The client MUST make use of UIDPLUS extension if it is supported 1032 by the server. 1034 See also optimization #1 in Section 5.3. 1036 4) Minimize number of round trips. 1038 Round trips kill performance, especially on links with high latency. 1039 Sections 4.2.2.5 and 5.2 give some advices how to minimize number of 1040 round trips. 1042 See also optimization #1 in Section 5.3. 1044 5.3. Optimizations 1046 Some useful optimizations are described in this section. A disconnected 1047 client that supports recommendations listed below will give the user 1048 a more pleasant experience. 1050 1) Initial OK or PREATH responses may contain CAPABILITY response code 1051 as described in section 7.1 of [IMAP4]. This response code gives 1052 the same information as returned by the CAPABILITY command(*). 1053 A disconnected client that pays attention to this response code can 1054 avoid sending CAPABILITY command and will save a round trip. 1056 (*) - Note, that some servers report in the CAPABILITY response code 1057 extensions that are only relevant in unauthenticated state or 1058 in all states. Such servers usually send another CAPABILITY 1059 response code upon successful authentication using LOGIN or 1060 AUTHENTICATE command (that negotiates no security layer, see 1061 section 6.2.2 of [IMAP4]). The CAPABILITY response code 1062 sent upon successful LOGIN/AUTHENTICATE might be different 1063 from the CAPABILITY response code in the initial OK response, 1064 as extensions only relevant for unauthenticated state will not 1065 be advertised and some additional extensions available only 1066 in authenticated and/or selected state will be. 1068 Example 1: 1070 S: * OK [CAPABILITY IMAP4REV1 LOGIN-REFERRALS STARTTLS AUTH=DIGEST-MD5 AUTH=SRP] 1071 imap.example.com ready 1072 C: 2 authenticate DIGEST-MD5 1073 ... 1074 S: 2 OK [CAPABILITY IMAP4REV1 IDLE NAMESPACE MAILBOX-REFERRALS SCAN SORT 1075 THREAD=REFERENCES THREAD=ORDEREDSUBJECT MULTIAPPEND] User authenticated 1076 (no layer) 1078 2) An advanced disconnected client may choose to optimize its replay log. 1079 For example, there might be some operations which are redundant (the list 1080 is not complete): 1081 a) an EXPUNGE followed by another EXPUNGE or CLOSE; 1082 b) changing flags (other than the \Deleted flag) on a message that 1083 gets immediately expunged; 1084 c) opening and closing the same mailbox. 1086 When optimizing, be careful about data dependencies between commands. 1087 For example, if the client is wishing to optimize (see case b) above) 1089 tag1 UID STORE +FLAGS (\Deleted) 1090 ... 1091 tag2 UID STORE +FLAGS (\Flagged) 1092 ... 1093 tag3 UID COPY "Backup" 1094 ... 1095 tag4 UID EXPUNGE 1097 it can't remove the second UID STORE command, because the message is being 1098 copied before it gets expunged. 1100 In general, it might be a good idea to keep mailboxes open during 1101 synchronization (see case c) above), if possible. This can be more easily 1102 achieved in conjunction with optimization #3 described below. 1104 3) Perform some synchronization steps in parallel, if possible. 1106 Several synchronization steps don't depend on each other and thus can 1107 be performed in parallel. Because the server machine is usually more 1108 powerful than the client machine and can perform some operations in 1109 parallel, this may speed up the total time of synchronization. 1111 In order to achieve such parallelization the client will have to open 1112 more than one connection to the same server. Client writers should not 1113 forget about non-trivial cost associated with establishing a TCP connection 1114 and performing an authentication. The disconnected client MUST NOT use 1115 a connection per each mailbox. In most cases it is sufficient to have 1116 two connections. 1118 Any mailbox synchronization MUST start with checking of the UIDVALIDITY 1119 as described in section 4.1 of this document. The client MAY use STATUS 1120 command to check UID Validity of a non selected mailbox. This is preferable 1121 to opening many connections to the same server to perform synchronization 1122 of multiple mailboxes simultaneously. As described in section 5.3.10 of 1123 [IMAP4], this SHOULD NOT be used on the selected mailbox. 1125 6. IMAP extensions that may help 1127 The following extensions can save traffic and/or number of round trips: 1129 1) The use of [UIDPLUS] is discussed in sections 4.1, 4.2.1, 4.2.2.1 and 4.2.4. 1131 2) The use of MULTIAPPEND and LITERAL+ extensions for uploading messages 1132 is discussed in 4.2.2.5. 1134 3) Use CONDSTORE extension (see section 6.1) for quick flag resynchronization. 1136 6.1. CONDSTORE extension 1138 An advance disconnected mail client should use [CONDSTORE] extension 1139 when it is supported by the server. The client must cache the value from 1140 HIGHESTMODSEQ OK response code received on mailbox opening and update 1141 it whenever the server sends MODSEQ FETCH data items. 1143 If the client receives NOMODSEQ OK untagged response instead of 1144 HIGHESTMODSEQ, it MUST remove the last known HIGHESTMODSEQ value from its 1145 cache and follow more general instructions in section 3. 1147 When the client opens the mailbox for synchronization it first compares 1148 UIDVALIDITY as described in step d)1) in section 3. If the cached 1149 UIDVALIDITY value matches the one returned by the server, the client 1150 MUST compare the cached value of HIGHESTMODSEQ with the one returned 1151 by the server. If the cached HIGHESTMODSEQ value also matches the 1152 one returned by the server, then the client MUST NOT fetch flags for 1153 cached messages, as they hasn't changed. If the value on the server 1154 is higher than the cached one, the client MAY use 1155 "SEARCH MODSEQ " to find all messages with flags 1156 changed since the last time the client was online and had the mailbox 1157 opened. Alternatively the client MAY use 1158 "FETCH 1:* (FLAGS) (CHANGEDSINCE )". The latter operation 1159 combines searching for changed messages and fetching new information. 1161 In all cases the client still need to fetch information about new 1162 messages (if requested by the user) and discover which messages got 1163 expunged. 1165 Step d) ("Server-to-client synchronization") in section 4 in the presence 1166 of CONDSTORE extension is amended as follows: 1168 d) "Server-to-client synchronization" - for each mailbox that requires 1169 synchronization, do the following: 1171 1a) Check the mailbox UIDVALIDITY (see section 4.1 for more details). 1172 with SELECT/EXAMINE/STATUS. 1173 If UIDVALIDITY value returned by the server differs, 1174 the client MUST 1176 * empty the local cache of that mailbox; 1177 * "forget" the cached HIGHESTMODSEQ value for the mailbox; 1178 * remove any pending "actions" which refer to UIDs in 1179 that mailbox. Note, this doesn't affect actions performed on 1180 client generated fake UIDs (see section 5); 1181 * skip steps 1b and 2-II; 1183 1b) Check the mailbox HIGHESTMODSEQ. If the cached value is the same 1184 as the one returned by the server, skip fetching message flags 1185 on step 2-II, i.e. the client only has to find out which messages 1186 got expunged. 1188 2) Fetch the current "descriptors"; 1190 I) Discover new messages. 1192 II) Discover changes to old messages using 1193 "FETCH 1:* (FLAGS) (CHANGEDSINCE )" or 1194 "SEARCH MODSEQ ". 1196 3) Fetch the bodies of any "interesting" messages that the client 1197 doesn't already have. 1199 Example (the UIDVALIDITY value is the same, but the HIGHESTMODSEQ value 1200 has changed on the server while the client was offline): 1202 C: A142 SELECT INBOX 1203 S: * 172 EXISTS 1204 S: * 1 RECENT 1205 S: * OK [UNSEEN 12] Message 12 is first unseen 1206 S: * OK [UIDVALIDITY 3857529045] UIDs valid 1207 S: * FLAGS (\Answered \Flagged \Deleted \Seen \Draft) 1208 S: * OK [PERMANENTFLAGS (\Deleted \Seen \*)] Limited 1209 S: * OK [HIGHESTMODSEQ 20010715194045007] 1210 S: A142 OK [READ-WRITE] SELECT completed 1212 after that either: 1213 C: A143 UID FETCH 1:* (FLAGS) (CHANGEDSINCE 20010715194032001) 1214 S: * 2 FETCH (UID 6 MODSEQ (20010715205008000) FLAGS (\Deleted)) 1215 S: * 5 FETCH (UID 9 MODSEQ (20010715195517000) FLAGS ($NoJunk 1216 $AutoJunk $MDNSent)) 1217 ... 1218 S: A143 OK FETCH completed 1220 or: 1222 C: A143 SEARCH MODSEQ 20010715194032001 1223 S: * SEARCH 2 5 6 7 11 12 18 19 20 23 (MODSEQ 20010917162500) 1224 S: A143 OK Search complete 1226 7. Security Considerations 1228 It is believed that this document does not raise any new security concerns 1229 that are not alread present in the base [IMAP] protocol, and these issues 1230 are discussed in [IMAP]. Additional security considerations may be found 1231 in different extensions mentioned in this document, in particular in 1232 [UIDPLUS], [LITERAL+], [CONDSTORE], [MULTIAPPEND] and [UNSELECT]. 1234 Implementors are also reminded about importance of thorough testing. 1236 8. References 1238 8.1. Normative References 1240 [KEYWORDS] Bradner, "Key words for use in RFCs to Indicate 1241 Requirement Levels", RFC 2119, Harvard University, March 1997. 1243 [IMAP4] Crispin, M., "Internet Message Access Protocol - Version 1244 4rev1", RFC 3501, University of Washington, March 2003. 1246 [UIDPLUS] Myers, J., "IMAP4 UIDPLUS extension", RFC 2359, June 1988. 1248 [LITERAL+] Myers, J. "IMAP4 non-synchronizing literals", RFC 2088, 1249 January 1997. 1251 [CONDSTORE] Melnikov, A., Hole, S., "IMAP Extension for Conditional 1252 STORE operation", Work in progress, draft-melnikov-imap-condstore-XX.txt, 1253 Isode Limited, ACI WorldWide/MessagingDirect. 1255 [MULTIAPPEND] Crispin, M., "INTERNET MESSAGE ACCESS PROTOCOL - 1256 MULTIAPPEND EXTENSION", RFC 3502, University of Washington, 1257 March 2003. 1259 [UNSELECT] Melnikov, A., "Internet Message Access Protocol (IMAP) 1260 UNSELECT command", RFC 3691, Isode Limited, February 2004. 1262 8.2. Informative References 1264 [RFC 2683] Leiba, B., "IMAP4 Implementation Recommendations", RFC 2683, 1265 September 1999. 1267 [ACL] Myers, J., "IMAP4 ACL Extension", RFC 2086, January 1997. 1268 and 1269 Melnikov, A., "IMAP4 ACL Extension", draft-ietf-imapext-acl-XX.txt, 1270 Work in Progress. 1272 9. Acknowledgment 1274 This document is based on the draft-ietf-imap-disc-01.txt written 1275 by Rob Austein in November 1994. 1277 The editor appreciate comments posted by Mark Crispin to the IMAP mailing 1278 list and the comments and corrections and ideas received from Cyrus Daboo, 1279 John G. Myers and Chris Newman. 1281 The editor would also like to thank the developers of Netscape Messenger 1282 and Mozilla mail clients for providing examples of disconnected mail clients 1283 that served as a base for many recommendations in this document. 1285 10. Editor's Address 1287 Alexey Melnikov 1288 mailto: alexey.melnikov@isode.com 1290 Isode Limited 1291 5 Castle Business Village, 1292 36 Station Road, 1293 Hampton, Middlesex, 1294 United Kingdom, TW12 2BX 1296 Phone: +44 77 53759732 1298 11. Intellectual Property 1300 The IETF takes no position regarding the validity or scope of any 1301 Intellectual Property Rights or other rights that might be claimed to 1302 pertain to the implementation or use of the technology described in 1303 this document or the extent to which any license under such rights 1304 might or might not be available; nor does it represent that it has 1305 made any independent effort to identify any such rights. Information 1306 on the procedures with respect to rights in RFC documents can be 1307 found in BCP 78 and BCP 79. 1309 Copies of IPR disclosures made to the IETF Secretariat and any 1310 assurances of licenses to be made available, or the result of an 1311 attempt made to obtain a general license or permission for the use of 1312 such proprietary rights by implementers or users of this 1313 specification can be obtained from the IETF on-line IPR repository at 1314 http://www.ietf.org/ipr. 1316 The IETF invites any interested party to bring to its attention any 1317 copyrights, patents or patent applications, or other proprietary 1318 rights that may cover technology that may be required to implement 1319 this standard. Please address the information to the IETF at 1320 ietf-ipr@ietf.org. 1322 12. Full Copyright Statement 1324 Copyright (C) The Internet Society (2004). This document is subject 1325 to the rights, licenses and restrictions contained in BCP 78, and 1326 except as set forth therein, the authors retain all their rights. 1328 This document and the information contained herein are provided on an 1329 "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS 1330 OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET 1331 ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, 1332 INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE 1333 INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED 1334 WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. 1336 Acknowledgement 1338 Funding for the RFC Editor function is currently provided by 1339 the Internet Society.