idnits 2.17.1 draft-melnikov-imap-disc-00.txt: ** The Abstract section seems to be numbered Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** Looks like you're using RFC 2026 boilerplate. This must be updated to follow RFC 3978/3979, as updated by RFC 4748. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- ** Missing expiration date. The document expiration date should appear on the first and last page. ** The document seems to lack a 1id_guidelines paragraph about Internet-Drafts being working documents. ** The document seems to lack a 1id_guidelines paragraph about 6 months document validity. ** The document is more than 15 pages and seems to lack a Table of Contents. == No 'Intended status' indicated for this document; assuming Proposed Standard == The page length should not exceed 58 lines per page, but there was 1 longer page, the longest (page 1) being 932 lines Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack an Introduction section. ** The document seems to lack an IANA Considerations section. (See Section 2.2 of https://www.ietf.org/id-info/checklist for how to handle the case when there are no actions for IANA.) ** The document seems to lack separate sections for Informative/Normative References. All references will be assumed normative when checking for downward references. ** There are 180 instances of too long lines in the document, the longest one being 58 characters in excess of 72. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the RFC 3978 Section 5.4 Copyright Line does not match the current year -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (June 2002) is 7985 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Missing Reference: 'HEADER' is mentioned on line 662, but not defined -- Looks like a reference, but probably isn't: '3' on line 671 -- Looks like a reference, but probably isn't: '4' on line 671 -- Looks like a reference, but probably isn't: '5' on line 671 -- Looks like a reference, but probably isn't: '6' on line 671 -- Looks like a reference, but probably isn't: '7' on line 671 -- Looks like a reference, but probably isn't: '8' on line 671 -- Looks like a reference, but probably isn't: '9' on line 672 -- Looks like a reference, but probably isn't: '10' on line 672 -- Looks like a reference, but probably isn't: '11' on line 672 -- Looks like a reference, but probably isn't: '13' on line 672 -- Looks like a reference, but probably isn't: '14' on line 672 -- Looks like a reference, but probably isn't: '15' on line 672 -- Looks like a reference, but probably isn't: '16' on line 672 -- Looks like a reference, but probably isn't: '21' on line 673 ** Obsolete normative reference: RFC 2060 (ref. 'IMAP4') (Obsoleted by RFC 3501) ** Obsolete normative reference: RFC 2359 (ref. 'UIDPLUS') (Obsoleted by RFC 4315) Summary: 12 errors (**), 0 flaws (~~), 4 warnings (==), 16 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 IMAPEXT Working Group A. Melnikov 2 Internet Draft: IMAP4 Disconnected Access Editor Editor 3 Document: draft-melnikov-imap-disc-00.txt June 2002 5 Synchronization operations for disconnected IMAP4 clients 7 Status of this Memo 9 This document is an Internet Draft and is in full conformance with 10 all provisions of Section 10 of RFC 2026. 12 Internet Drafts are working documents of the Internet Engineering 13 Task Force (IETF), its Areas, and its Working Groups. Note that 14 other groups may also distribute working documents as Internet 15 Drafts. Internet Drafts are draft documents valid for a maximum of 16 six months. Internet Drafts may be updated, replaced, or obsoleted 17 by other documents at any time. It is not appropriate to use 18 Internet Drafts as reference material or to cite them other than as 19 ``work in progress''. 21 The list of current Internet-Drafts can be accessed at 22 http://www.ietf.org/ietf/1id-abstracts.txt 24 The list of Internet-Draft Shadow Directories can be accessed at 25 http://www.ietf.org/shadow.html. 27 This is a draft document based on the expired draft written by 28 the IETF IMAP Working Group. A revised version of this draft document 29 will be submitted to the RFC editor as an Informational RFC for the 30 Internet Community. Discussion and suggestions for improvement are 31 requested, and should be sent to imap@CAC.Washington.EDU. 33 This memo is for informational use and does not constitute a 34 standard. Distribution of this memo is unlimited. 36 1. Abstract 38 This document attempts to address some of the issues involved in building 39 a disconnected IMAP4 client. In particular, it deals with the issues 40 of what might be called the "driver" portion of the synchronization 41 tool: the portion of the code responsible for issuing the correct set 42 of IMAP4 commands to synchronize the disconnected client in the way 43 that is most likely to make the human who uses the disconnected 44 client happy. 46 This note describes different strategies that can be used by disconnected 47 clients as well as shows how to use IMAP protocol in order to minimize the 48 time of synchronization process. 50 2. Conventions Used in this Document 52 In examples, "C:" and "S:" indicate lines sent by the client and 53 server respectively. 55 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 56 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 57 document are to be interpreted as described in RFC 2119 [KEYWORDS]. 59 Editorial comments/questions or missing paragraphs are marked in the 60 text with << and >>. 62 3. Design Principles 64 All mailbox state or content information stored on the disconnected 65 client should be viewed strictly as a cache of the state of the 66 server. The "master" state remains on the server, just as it would 67 with an interactive IMAP4 client. The one exception to this rule is 68 that information about the state of the disconnected client's cache 69 (the state includes flag changes while offline and scheduled message uploads) 70 remains on the disconnected client: that is, the IMAP4 server is not 71 responsible for remembering the state of the disconnected IMAP4 client. 73 We assume that a disconnected client is a client that, for whatever 74 reason, wants to minimize the length of time that it is "on the 75 phone" to the IMAP4 server. Often this will be because the client is 76 using a dialup connection, possibly with very low bandwidth, but 77 sometimes it might just be that the human is in a hurry to catch an 78 airplane, or some other event beyond our control. Whatever the 79 reason, we assume that we must make efficient use of the network 80 connection, both in the usual sense (not generating spurious traffic) 81 and in the sense that we would prefer not to have the connection 82 sitting idle while the client and/or the server is performing 83 strictly local computation or I/O. Another, perhaps simpler way of 84 stating this is that we assume that network connections are 85 "expensive". 87 Practical experience with disconnected mail systems has shown that 88 there is no single synchronization strategy that is appropriate 89 for all cases. Different humans have different preferences, 90 and the same human's preference will vary depending both on 91 external circumstance (how much of a hurry the human is in today) 92 and on the value that the human places on the messages being 93 transfered. The point here is that there is no way that 94 the synchronization program can guess exactly what the human 95 wants to do, so the human will have to provide some guidance. 97 Taken together, the preceeding two principles lead to the conclusion 98 that the synchronization program must make its decisions based on 99 some kind of guidance provided by the human by selecting the appropriate 100 options in UI or throught some sort of configuration file, but almost 101 certainly should not pause for I/O with the human during the middle 102 of the synchronization process. The human will almost certainly have 103 several different configurations for the synchronization program, for 104 different circumstances. 106 Since a disconnected client has no way of knowing what changes might 107 have occured to the mailbox while it was disconnected, message 108 numbers are not useful to a disconnected client. All disconnected 109 client operations should be performed using UIDs, so that the client 110 can be sure that it and the server are talking about the same 111 messages during the synchronization process. 113 4. Overall picture of synchronization 115 The basic strategy for synchronization is outlined below. 116 Note that the real strategy may vary from one application to another 117 or may depend on a synchronization mode. 119 a) Process any "actions" that were pending on the client that 120 were not associated with any mailbox (in particular sending 121 messages composed offline with SMTP. This is not part of IMAP 122 synchronization, but it is mentioned here for completeness); 124 b) Fetch the current list of "interesting" mailboxes; 126 c) "Client-to-server synchronization" - for each IMAP "action" that 127 were pending on the client: 129 1) If the action implies opening a new mailbox (any operation 130 that operates on messages) - open the mailbox. Check its UID 131 validity value (see section 5.1 for more details) returned in 132 the UIDVALIDITY response code. If the UIDVALIDITY value returned 133 by the server differs, the client MUST empty the local cache of 134 the mailbox and remove any pending "actions" which refer to UIDs 135 in that mailbox. 137 2) Perform the action. If the action is to delete a mailbox (DELETE), 138 make sure that the mailbox is closed first. 140 d) "Server-to-client synchronization" - for each mailbox that requires 141 synchronization, do the following: 143 1) Check its UIDVALIDITY (see section 5.1 for more details). 144 with SELECT/EXAMINE/STATUS. 145 If UIDVALIDITY value returned by the server differs, 146 the client MUST empty the local cache of that mailbox 147 and remove any pending "actions" which refer to UIDs in 148 that mailbox. 150 2) Fetch the current "descriptors"; 152 << Be more detailed? >> 154 3) Fetch the bodies of any "interesting" messages that the client 155 doesn't already have. 157 d) Close all open mailboxes not required for further operations 158 (if staying online) or disconnect all open connections (if going offline). 160 Terms used: 162 "Actions" are queued requests that were made by the human to the 163 client's MUA software while the client was disconnected. 165 Let define "descriptors" as a set of IMAP4 FETCH data items. 166 Conceptually, a message's descriptor is that set of 167 information that allows the synchronization program to decide what 168 protocol actions are necessary to bring the local cache to the 169 desired state for this message; since this decision is really up 170 to the human, this information probably includes a at least a few 171 header fields intended for human consumption. Exactly what will 172 constitute a descriptor depends on the client implementation. At 173 a minimum, the descriptor contains the message's UID and FLAGS. 174 Other likely candidates are the RFC822.SIZE, RFC822.HEADER and 175 BODYSTRUCTURE data items. 177 Comments: 179 1). The list of actions should be ordered. Eg, if the human deletes 180 message A1 in mailbox A, then expunges mailbox A, then deletes 181 message A2 in mailbox A, the human will expect that message A1 is 182 gone and that message A2 is still present but is now deleted. 184 By processing all the actions before proceeding with 185 synchronization, we avoid having to compensate for the local MUA's 186 changes to the server's state. That is, once we have processed 187 all the pending actions, the steps that the client must take to 188 synchronize itself will be the same no matter where the changes to 189 the server's state originated. 191 2). Steps a) and b) can be performed in parallel. Alternatively step a) 192 can be performed after d). 194 3). On step b) the set of "interesting" mailboxes pretty much has to be 195 determined by the human. What mailboxes belong to this set may 196 vary between different IMAP4 sessions with the same server, 197 client, and human. An interesting mailbox can be a mailbox 198 returned by LSUB command. Special mailbox "INBOX" SHOULD always 199 be considered "interesting". 201 4). On step d2) the client also finds out about 202 changes to the flags of messages that the client already has in 203 its local cache, as well as finding out about messages in the 204 local cache that no longer exist on the server (i.e. , messages that 205 have been expunged). 207 6). "Interesting" messages are those messages that the synchronization 208 program thinks the human wants to have cached locally, based on 209 the configuration and the data retrieved in step (b). 211 The rest of this discussion will focus primarily on the synchronization 212 issues for a single mailbox. 214 5. Mailbox synchronization steps and strategies 216 5.1. Checking UID Validity 218 The "UID validity" of a mailbox is a number returned in an 219 UIDVALIDITY response code in an OK untagged response at mailbox 220 selection time. The UID validity value changes between sessions when 221 UIDs fail to persist between sessions. 223 Whenever the client selects a mailbox, the client must compare the 224 returned UID validity value with the value stored in the local cache. 225 If the UID validity values differ, the UIDs in the client's cache are 226 no longer valid. The client MUST then empty the local cache of 227 that mailbox and remove any pending "actions" which refer to UIDs in 228 that mailbox. The client MAY also issue a warning to the human. 229 The client MUST NOT cancel any scheduled uploads (i.e. APPENDs) for 230 the mailbox. 232 Note that UIDVALIDITY is not only returned on a mailbox selection. 233 COPYUID and APPENDUID response codes defined in [UIDPLUS] extension 234 (see also 5.2.2) and UIDVALIDITY STATUS response data item also contain 235 a UIDVALIDITY value for some other mailbox. The client SHOULD behave as 236 described in the previous paragraph (but it should act on the other mailbox's 237 cache), no matter how it obtained the UIDVALIDITY value. 239 5.2. Synchronizing local changes with the server 241 5.2.1. Uploading messages to the mailbox 243 There are two most typical examples of operations that will result in message 244 uploads: 246 1) Saving a draft message 247 2) Message copy between remote mailboxes on two different IMAP servers 248 or a local mailbox and a remote mailbox. 250 Note, that alternatively this step can be performed at the end 251 of the automatic mailbox synchronization. 253 Message upload is performed with APPEND command. A message scheduled to be 254 uploaded has no UID associated with it, as all UIDs are assigned by the 255 server. The APPEND command will effectively associate a UID with the uploaded 256 message that can be stored in the local cache for a future reference. 257 However [IMAP4] doesn't describe a simple mechanism to discover the message UID 258 by just performing the APPEND command. In order to discover UID the client can 259 do one of the following: 261 1) Remove the uploaded message from cache. After that use the mechanism described 262 in 5.3 to fetch the information about the uploaded message as if it was uploaded 263 by some other client. 265 2) Try to fetch header information as described in 5.2.2 in order to find a message 266 that corresponds to the uploaded message. One strategy of doing that is described 267 in 5.2.2. 269 Case 1) discribes a non particularly smart client. 271 C: A003 APPEND Drafts (\Seen $MDNSent) {310} 272 S: + Ready for literal data 273 C: Date: Mon, 7 Feb 1994 21:52:25 -0800 (PST) 274 C: From: Fred Foobar 275 C: Subject: afternoon meeting 276 C: To: mooch@owatagu.siam.edu 277 C: Message-Id: 278 C: MIME-Version: 1.0 279 C: Content-Type: TEXT/PLAIN; CHARSET=US-ASCII 280 C: 281 C: Hello Joe, do you think we can meet at 3:30 tomorrow? 282 C: 283 S: A003 OK APPEND Completed 285 Fortunately there is a simpler way to discover the message UID in the presence 286 of [UIDPLUS] extension: 288 C: A003 APPEND Drafts (\Seen $MDNSent) {310} 289 S: + Ready for literal data 290 C: Date: Mon, 7 Feb 1994 21:52:25 -0800 (PST) 291 C: From: Fred Foobar 292 C: Subject: afternoon meeting 293 C: To: mooch@owatagu.siam.edu 294 C: Message-Id: 295 C: MIME-Version: 1.0 296 C: Content-Type: TEXT/PLAIN; CHARSET=US-ASCII 297 C: 298 C: Hello Joe, do you think we can meet at 3:30 tomorrow? 299 C: 300 S: A003 OK APPEND [APPENDUID 1022843275 77712] completed 302 The UID of the appended message is the second parameter of APPENDUID response code. 304 5.2.2. Optimizing "move" operations 306 Practical experience with IMAP, and other mailbox access 307 protocols that support multiple mailboxes suggests that moving a 308 message from one mailbox to another is an extremely common operation. 310 5.2.2.1. Moving a message between two mailboxes on the same server 312 In IMAP4 a "move" operation between two mailboxes on the same server 313 is really a combination of a COPY operation and a STORE +FLAGS (\Deleted) 314 operation. This makes good protocol sense for IMAP, but it leaves 315 a simple-minded disconnected client in the silly position of deleting 316 and possibly expunging its cached copy of a message, then fetching 317 an identical copy via the network. 319 However the presence of UIDPLUS extension support in the server can help: 320 A001 UID COPY 567,414 "Interesting Messages" 321 A001 OK [COPYUID 1022843275 414,567 5:6] Completed 322 This tells the client that the message with UID 414 in the current mailbox 323 was successfully copied to the mailbox "Interesting Messages" and was given 324 the UID 5, and that the message with UID 567 was given the UID 6. 326 In the absence of UIDPLUS extension support in the server the following 327 trick can be used. By including the Message-ID: header and the INTERNALDATE 328 data item as part of the descriptor, the client can check the descriptor of a 329 "new" message against messages that are already in its cache, and 330 avoid fetching the extra copy. Of course, it's possible that the 331 cost of checking to see if the message is already in the local cache 332 may exceed the cost of just fetching it, so this technique should not 333 be used blindly. If the MUA implements a "move" command, it make 334 special provisions to use this technique when it knows that a 335 copy/delete sequence is the result of a "move" command. 337 Since it's theoretically possible for this algorithm to find the 338 wrong message (given sufficiently malignant Message-ID headers), 339 implementors should provide a way to disable this optimization, both 340 permanently and on a message-by-message basis. 342 << Example >> 344 5.2.2.2. Moving a message from a remote mailbox to a local 346 Moving a message from a remote mailbox to a local is done with FETCH 347 (that includes FLAGS and INTERNALDATE) followed by 348 UID STORE +FLAGS.SILENT (\Deleted): 350 C: A003 UID FETCH 123 (RFC822 INTERNALDATE FLAGS) 351 S: * 27 FETCH (UID 123 INTERNALDATE "31-May-2002 05:26:59 -0600" 352 FLAGS (\Seen $MDNSent) RFC822 353 S: ...message body... 354 S: ) 355 S: A003 OK UID FETCH completed 356 C: A004 UID STORE +FLAGS.SILENT (\Deleted) 357 S: A004 STORE completed 359 Note, that there is no reason to fetch the message during synchronization 360 if it already in the client's cache. Also, the client SHOULD preserve 361 delivery date in the local cache. 363 5.2.2.3. Moving a message from a local mailbox to a remote 365 Moving a message from a local mailbox to a remote is done with APPEND: 367 C: A003 APPEND Drafts (\Seen $MDNSent) "31-May-2002 05:26:59 -0600" {310} 368 S: + Ready for literal data 369 C: Date: Mon, 7 Feb 1994 21:52:25 -0800 (PST) 370 C: From: Fred Foobar 371 C: Subject: afternoon meeting 372 C: To: mooch@owatagu.siam.edu 373 C: Message-Id: 374 C: MIME-Version: 1.0 375 C: Content-Type: TEXT/PLAIN; CHARSET=US-ASCII 376 C: 377 C: Hello Joe, do you think we can meet at 3:30 tomorrow? 378 C: 379 S: A003 OK APPEND [APPENDUID 1022843275 77712] completed 381 The client SHOULD specify delivery date from the local cache in the APPEND. 383 5.2.2.4. Moving a message between two mailbox on two different servers 385 Moving a message between two mailbox on two different servers is a combination 386 of the 5.2.2.2 followed by 5.2.2.3. 388 5.2.3. Replaying local flag changes 390 The disconnected client uses STORE command to synchronize local flag state 391 with the server. The disconnected client SHOULD use +FLAGS.SILENT or -FLAGS.SILENT 392 in order to set or unset flags modified by the user while offline. FLAGS 393 form must not be used, as there is a risk that this will overwrite flags 394 on the server that has been changed by some other client. 396 Example: 397 For the message with UID 15, the disconnected client stores the following 398 flags \Seen and $Highest. The flags were modified on the server by some other 399 client: \Seen, \Answered and $Highest. 400 While offline the user requested to remove $Highest flags and to add \Deleted. 401 The flag synchronization sequence for the message should look like: 403 C: A001 UID STORE 15 +FLAGS.SILENT (\Deleted) 404 S: A001 STORE completed 405 C: A002 UID STORE 15 -FLAGS.SILENT ($Highest) 406 S: A002 STORE completed 408 If the disconnected client is able to store an additional binary state 409 information (or a piece of information that can take a value from a predefined 410 set of values) in the local cache of an IMAP mailbox or in a local mailbox 411 (e.g. message priority), and if the server supports storing of arbitrary 412 keywords, the client MUST use keywords to store this state on the server. 414 Example: 415 Imagine a speculative mail client that can mark a message as one of work-related 416 ($Work), personal ($Personal) or spam ($Spam). In order to mark a message as 417 personal the client issues: 419 C: A001 UID STORE 15 +FLAGS.SILENT ($Personal) 420 S: A001 STORE completed 421 C: A002 UID STORE 15 -FLAGS.SILENT ($Work $Spam) 422 S: A002 STORE completed 424 In order to mark the message as neither work, nor personal, not spam, the client 425 issues: 427 C: A003 UID STORE 15 -FLAGS.SILENT ($Personal $Work $Spam) 428 S: A003 STORE completed 430 5.2.4. Processing mailbox compression (expunge) requests 432 A naive disconnected client implementation that supports compressing a mailbox 433 while offline may decide to issue an EXPUNGE command to the server in order 434 to expunge messages marked \Deleted. The problem with this command during 435 synchronization is that it permanently erases all messages with \Deleted flag set, 436 i.e. even those messages that were marked as \Deleted on the server while the user 437 was offline. Doing so will lead the user to an unpleasant surprise. 439 Fortunately [UIDPLUS] extension can help in this case as well. The extension 440 introduces UID EXPUNGE command, that, unlike EXPUNGE, takes a UID set parameter, 441 that lists UIDs of all messages that can be expunged. When processing this command 442 server erases only messages with \Deleted flag listed in the UID list. Thus, 443 a message not listed in the UID set will not be expunged even if it has \Deleted 444 flag set. 446 Example: While offline 3 messages with UIDs 7, 27 and 65 were marked \Deleted 447 when the user requested to compress the open mailbox. Another client marked 448 a message \Deleted on the server (UID 34). During synchronization the 449 disconnected client issues: 451 C: A001 UID EXPUNGE 7,27,65 452 S: * ... EXPUNGE 453 S: * ... EXPUNGE 454 S: * ... EXPUNGE 455 S: A001 UID EXPUNGE completed 457 If another client issues UID SEARCH DELETED command (to find all messages with 458 \Deleted flag) before and after the UID EXPUNGE it will get: 460 Before: 461 C: B001 UID SEARCH DELETED 462 S: * SEARCH 65 34 27 7 463 S: B001 UID SEARCH completed 465 After: 466 C: B002 UID SEARCH DELETED 467 S: * SEARCH 34 468 S: B002 UID SEARCH completed 470 In the absence of [UIDPLUS] extension the following sequence of command can be 471 used as an approximation. Note, that when this sequence is performed, there is a 472 possibility that another client marks additional messages as deleted and these 473 messages will be expunged as well. 475 1). Find all messages marked \Deleted on the server: 477 C: A001 UID SEARCH DELETED 478 S: * SEARCH 65 34 27 7 479 S: A001 UID SEARCH completed 481 2). Find all messages that must not be erased (for the previous example 482 the list will consist of the message with UID 34) 484 3). Temporary remove \Deleted flag on all messages found on the step 2) 486 C: A002 UID STORE 34 -FLAGS.SILENT (\Deleted) 487 S: A002 UID STORE completed 489 4). Expunge the mailbox 491 C: A003 EXPUNGE 492 S: * 20 EXPUNGE 493 S: * 7 EXPUNGE 494 S: * 1 EXPUNGE 495 S: A003 EXPUNGE completed 497 Here message with UID 7 has message number 1; with UID 27 - message 498 number 7 and with UID 65 - message number 20. 500 5). Restore \Deleted flag on all messages found when performing step 2) 502 C: A004 UID STORE 34 +FLAGS.SILENT (\Deleted) 503 S: A004 UID STORE completed 505 5.2.5. Closing a mailbox 507 When the disconnected client has to close a mailbox, it SHOULD NOT use 508 CLOSE command, because CLOSE does a silent EXPUNGE (section 5.2.4 explains 509 why EXPUNGE must not be used by a disconnected client). It is safe to use 510 CLOSE only if the mailbox was opened with EXAMINE. 512 If the mailbox was opened with SELECT, the client can use one of the 513 following commands to implicitely close the mailbox and prevent the silent 514 expunge: 516 1). UNSELECT - This is an undocumented command that works as CLOSE, but 517 doesn't cause the silent EXPUNGE. This command is supported by the server 518 if it reports UNSELECT in its CAPABILITY list. 519 2). EXAMINE - reselect the same mailbox in read-only mode. 520 3). SELECT - SELECT causes implicit CLOSE without EXPUNGE. 521 4). If the client intends to issue LOGOUT after closing the mailbox, it may 522 just issue LOGOUT, because LOGOUT causes implicit CLOSE without EXPUNGE 523 as well. 524 5). SELECT - if the client knows a mailbox that doesn't 525 exist or can't be selected, it MAY SELECT it. 527 5.3. Details of "Normal" synchronization of a single mailbox 529 The most common form of synchronization is where the human trusts the 530 integrity of the client's copy of the state of a particular mailbox, 531 and simply wants to bring the client's cache up to date so that it 532 accurately reflects the mailbox's current state on the server. 534 5.3.1. Discovering new messages and changes to old messages 536 Let represent the highest UID that the client knows about 537 in this mailbox. Since UIDs are allocated in strictly ascending 538 order, this is simply the UID of the last message in the mailbox that 539 the client knows about. Let represent 's UID 540 plus one. Let represent a list consisting of all the 541 FETCH data item items that the implementation considers to be part of 542 the descriptor; at a minimum this is just the FLAGS data item, but 543 it usually also includes BODYSTRUCTURE and RFC822.SIZE. At this step 544 SHOULD NOT include RFC822. 546 With no further information, the client can issue issue the following 547 two commands: 548 tag1 UID FETCH :* 549 tag2 UID FETCH 1: FLAGS 551 The first command will request some information about "new" messages 552 (i.e. messages received by the server since the last synchronization). 553 It will also allow the client to build a message number to UID map 554 (only for new messages). The second command allows the client to 555 1) update cached flags for old messages; 556 2) find out which old messages got expunged; 557 3) build a mapping between message numbers and UIDs (for old messages). 559 The order here is significant. We want the server to start returning 560 the list of new message descriptors as fast as it can, so that the 561 client can start issuing more FETCH commands, so we start out by 562 asking for the descriptors of all the messages we know the client 563 cannot possibly have cached yet. The second command fetches the 564 information we need to determine what changes may have occurred to 565 messages that the client already has cached. Note, that the latter 566 command should only be issued if the UIDNEXT value cached by the client 567 differs from the one returned by the server. Once the client has 568 issued these two commands, there's nothing more the client can do 569 with this mailbox until the responses to the first command start 570 arriving. A clever synchronization program might use this time to 571 fetch its local cache state from disk, or start the process of 572 synchronizing another mailbox. 574 Example of the first FETCH: 575 C: A011 UID fetch 131:* (FLAGS BODYSTRUCTURE INTERNALDATE RFC822.SIZE) 576 S: ... 578 The second FETCH command will result in nil or more untagged fetch 579 responses. Each response will have a corresponding UID FETCH data item. 580 All messages that didn't have a matching untagged FETCH response 581 MUST be removed from the local cache. 583 For example, if the had a value 15000 and the local cache 584 contained 3 messages with the UIDs 12, 777 and 14999 respectively, than 585 after receiving the following responses from the server: 587 S: * 1 FETCH (UID 12 FLAGS (\Seen)) 588 S: * 2 FETCH (UID 777 FLAGS (\Answered \Deleted)) 590 the client must remove the message with UID 14999 from its local cache. 592 Note: If the client is not interested in flag changes (i.e. the client 593 only wants to know which old messages are still on the server), the second 594 FETCH command can be substituted with: 595 tag2 UID SEARCH UID 1: 597 This command will generate less traffic. However an implementor should be 598 aware that in order to build the mapping table from message numbers to UIDs 599 the output of the SEARCH command MUST be sorted first, because there is 600 no requirement for a server to return UIDs in SEARCH response in the ascending 601 order. 603 5.3.2. Searching for "interesting" messages. 605 This step is either performed entirely on the client (from the information received 606 in step 5.3.1), after performing additional searches on the server or both. 607 The decision on what is an "interesting" message is up to the client software 608 and the human. One easy criterion that should probably be implemented in any 609 client is whether the message is "too big" for automatic retrieval, where "too big" 610 is a parameter defined in the client's configuration. 612 Another commonly used criteria is the age of a message. For example, the client 613 may choose to download only messages received in the last week (in this case, 614 would be today's date minus 7 days): 616 tag3 UID SEARCH UID SINCE 618 Keep in mind that a date search disregards time and timezone. 619 The client can avoid doing this search if it specified INTERNALDATE in 620 on step 5.3.1. If the client did, it can perform the search itself. 622 At this step the client also decides what kind of information about a particular 623 message to fetch from the server. In particular, even for a message that is considered 624 to be "too big" the client MAY choose to fetch some part(s) of it. For example, 625 if the message is a multipart/mixed containing a text part and a MPEG attachment, 626 there is no reason for the client not to fetch the text part. The decision of which 627 part should or should not be fetched can be based on the information received in 628 BODYSTRUCTURE FETCH response data item (i.e. if BODYSTRUCTURE was included in 629 on step 5.3.1). 631 5.3.3. Populating cache with "interesting" messages. 633 Once the client found out which messages are "interesting", the client 634 can start issuing appropriate FETCH commands for "interesting" messages or 635 bodyparts thereof. 637 It is important to note that fetching a message into the disconnected 638 client's local cache does NOT imply that the human has (or even will) 639 read the message. Thus, the synchronization program for a 640 disconnected client should always be careful to use the .PEEK 641 variants of the FETCH data items that implicitly set the \Seen flag. 643 Once the last descriptor has arrived and the last FETCH command has 644 been issued, the client simply needs to process the incoming fetch 645 items, using them to update the local message cache. 647 In order to avoid deadlock problems, the client must give processing 648 of received messages priority over issuing new FETCH commands during 649 this synchronization process. This may necessitate temporary local 650 queuing of FETCH requests that cannot be issued without causing a 651 deadlock. In order to achive the best use of the "expensive" network 652 connection, the client will almost certainly need to pay careful 653 attention to any flow-control information that it can obtain from the 654 underlying transport connection (usually a TCP connection). 656 Example: After fetching a message BODYSTRUCTURE the client discovers 657 a complex MIME message. Than it decides to fetch MIME headers 658 of the nested MIME messages and some body parts. 660 C: A011 UID fetch 11 (BODYSTRUCTURE) 661 S: ... 662 C: A012 UID fetch 11 (BODY[HEADER] BODY[1.MIME] BODY[1.1.MIME] 663 BODY[1.2.MIME] BODY[2.MIME] BODY[3.MIME] BODY[4.MIME] BODY[5.MIME] 664 BODY[6.MIME] BODY[7.MIME] BODY[8.MIME] BODY[9.MIME] BODY[10.MIME] 665 BODY[11.MIME] BODY[12.MIME] BODY[13.MIME] BODY[14.MIME] BODY[15.MIME] 666 BODY[16.MIME] BODY[17.MIME] BODY[18.MIME] BODY[19.MIME] BODY[20.MIME] 667 BODY[21.MIME]) 668 S: ... 669 C: A013 UID fetch 11 (BODY[1.1] BODY[1.2]) 670 S: ... 671 C: A014 UID fetch 11 (BODY[3] BODY[4] BODY[5] BODY[6] BODY[7] BODY[8] 672 BODY[9] BODY[10] BODY[11] BODY[13] BODY[14] BODY[15] BODY[16] 673 BODY[21]) 674 S: ... 676 5.3.4. User initiated synchronization 678 After the client finished the main synchronization that was described in 679 5.3.1-5.3.3 the user may optionally request additional synchronization steps 680 while the client is still online. This is not any different from the process 681 described in 5.3.3. 683 << Example: Fetch all flagged messages. >> 685 << Example: Fetch all messages selected in UI. >> 687 5.4. Special case: descriptor-only synchronization 689 For some mailboxes, fetching the descriptors might be the entire 690 synchronization step. Practical experience with IMAP has shown that 691 a certain class of mailboxes (eg, "archival" mailboxes) are used 692 primarily for long-term storage of important messages that the human 693 wants to have instantly available on demand but does not want 694 cluttering up the disconnected client's cache at any other time. 695 Messages in this kind of mailbox would be fetched exclusively by 696 explicit actions queued by the local MUA. Thus, the only 697 synchronization that is necessary for a mailbox of this kind is 698 fetching the descriptor information that the human will use to 699 identify messages that should be explicitly fetched. 701 Special mailboxes that receive traffic from a high volume, low 702 priority mailing list might also be in this catagory, at least when 703 the human is in a hurry. 705 5.5. Special case: fast new-only synchronization 707 In some cases the human might be in such a hurry that s/he doesn't 708 care about changes to old messages, just about new messages. In this 709 case, the client can skip the UID FETCH command that obtains the 710 flags and UIDs for old messages (1:). 712 5.6. Special case: blind FETCH 714 In some cases the human may know (for whatever reason) that s/he 715 always wants to fetch any new messages in a particular mailbox, 716 unconditionally. In this case, the client can just fetch the 717 messages themselves, rather than just the descriptors, by using a 718 command like: 719 tag1 UID FETCH :* (FLAGS RFC822.PEEK) 721 Note, that this example ignores the fact that the messages can 722 be arbitrary long. The disconnected client MUST always check 723 for message size before downloading, unless explicitely told otherwise. 724 A good behaving client should use instead something like the following: 726 1) Issue "tag1 UID FETCH :* (FLAGS RFC822.SIZE)" 727 2) From the message sizes returned in step 1 construct UID set 728 729 3) Issue "tag2 UID FETCH (RFC822.PEEK)" 731 or 733 1) Issue "tag1 UID FETCH :* (FLAGS)" 734 2) Construct UID set from the responses of 1) 735 3) Issue "tag2 SEARCH UID SMALLER " 736 Construct UID set from the result of 737 the SEARCH command. 738 4) Issue "tag3 UID FETCH (RFC822.PEEK)" 740 6. Implementation considerations 742 Below are listed some golden rules that should be considered when implementing 743 a good disconnected IMAP client. They will help to write a disconnected client 744 that works correctly, performs synchronization as quickly as possible 745 (and thus can save the user money) as well as minimizes the load of the server: 747 1) Don't reorder operations during synchronization. 749 It is not always safe to reorder operations during synchronization, 750 because some operations may have dependencies. So if in doubt, don't do that. 751 The following example demonstrate this: 753 Example 1: The user copies a message out of a mailbox and then deletes 754 the mailbox. 756 C: A001 SELECT Old-Mail 757 S: ... 758 C: A002 UID COPY 111 ToDo 759 S: A002 OK [COPYUID 1022843345 111 94] Copy completed 760 ... 761 C: A015 CLOSE 762 S: A005 OK Completed 763 C: A016 DELETE Old-Mail 764 S: A016 OK Mailbox deletion completed successfully 766 If the client performs DELETE (tag A016) first and COPY (tag A002) second, 767 than the COPY fails. 769 << Describe one case when it is safe to reorder: the disconnected client 770 doesn't allow to perform DELETE and RENAME while offline and EXPUNGE 771 is never used (UID EXPUNGE is used instead or its emulation as described 772 in 5.2.3 >> 774 2) Minimize traffic 776 The client MUST NOT issue a command if the client already received 777 the required information from the server. 779 The client MUST make use of UIDPLUS extension if it is supported 780 by the server. 782 3) Minimize number of round trips. 784 Round trips kill performance, especially on links with high latency. 785 This section gives some advices how to minimize number of round trips. 787 4) Perform some synchronization steps in parallel if possible. 789 Several synchronization steps don't depend on each other and thus can 790 be performed in parallel. Because the server machine is usually more 791 powerful than the client machine and can perform some operations in 792 parallel, this may speed up the total time of synchronization. 794 In order to achieve such parallelization the client will have to open 795 more than one connection to the same server. Client writers should be 796 aware of the non-trivial cost associated with establishing TCP connection 797 and performing authentication. The disconnected client MUST NOT use 798 a connection per each mailbox. In most cases it is sufficient to have 799 two connections. 801 Any mailbox synchronization MUST start with checking of the UIDVALIDITY 802 as described in section 5.1 of this document. The client MAY use STATUS 803 command to check UID Validity of a non selected mailbox. This is preferable 804 to opening many connections to the same server to perform synchronization 805 of multiple mailboxes simultaneously. As described in section 6.3.10 of 806 [IMAP4], this MUST NOT be used on the selected mailbox. 808 Below listed some quality of implementation issues for disconnected clients: 810 1) Don't loose information. 812 << To be added later >> 814 6.1. Optimizations 816 << To be added later >> 818 6.2. Error recovery during playback 820 << To be added later >> 822 7. IMAP extensions that may help 824 The following extensions can save traffic and/or number of round trips: 826 1) The use of [UIDPLUS] was discussed in 5.1, 5.2.1, 5.2.2.1 and 5.2.4. 828 << Describe how MULTIAPPEND and LITERAL+ can be used >> 830 << Describe how CONDSTORE can be used >> 832 8. Security Considerations 834 Security considerations are not discussed in this memo. 836 9. References 838 [KEYWORDS] Bradner, "Key words for use in RFCs to Indicate 839 Requirement Levels", RFC 2119, Harvard University, March 1997. 841 [IMAP4] Crispin, M., "Internet Message Access Protocol - Version 842 4rev1", RFC 2060, University of Washington, December 1996. 844 [UIDPLUS] Myers, J., "IMAP4 UIDPLUS extension", RFC 2359, June 1988. 846 [LITERAL+] Myers, J. "IMAP4 non-synchronizing literals", RFC 2088, 847 January 1997. 849 << Add missing references: MULTIAPPEND, CONDSTORE, ACL? >> 851 10. Aknowledgement 853 This document is a revision of the draft-ietf-imap-disc-01.txt written 854 by Rob Austein in November 1994. 856 The editor also appreciate comments posted by Mark Crispin to the IMAP mailing 857 list and the comments and corrections received from Cyrus Daboo. 859 The editor would also like to thank the developers of Netscape Messenger 860 and Mozilla mail clients for providing examples of disconnected mail clients 861 that served as a base for many recommendations in this document. 863 11. Editor's Address 865 Alexey Melnikov 866 mailto: mel@messagingdirect.com 868 ACI WorldWide/MessagingDirect 869 22 The Quadrant, Richmond, 870 Surrey, United Kingdom, TW9 1BP 872 12. Full Copyright Statement 874 Copyright (C) The Internet Society 2002. All Rights Reserved. 876 This document and translations of it may be copied and furnished to 877 others, and derivative works that comment on or otherwise explain it 878 or assist in its implementation may be prepared, copied, published 879 and distributed, in whole or in part, without restriction of any 880 kind, provided that the above copyright notice and this paragraph 881 are included on all such copies and derivative works. However, this 882 document itself may not be modified in any way, such as by removing 883 the copyright notice or references to the Internet Society or other 884 Internet organizations, except as needed for the purpose of 885 developing Internet standards in which case the procedures for 886 copyrights defined in the Internet Standards process must be 887 followed, or as required to translate it into languages other than 888 English. 890 The limited permissions granted above are perpetual and will not be 891 revoked by the Internet Society or its successors or assigns. 893 This document and the information contained herein is provided on an 894 "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING 895 TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING 896 BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION 897 HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF 898 MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.