May 2006 Lemonade Internet Draft: VFOLDER S. H. Maes Document: draft-ietf-lemonade-vfolder-01 R. Cromwell A. Srivastava A. Gulbrandsen Eds. Expires: November 2006 May 2006 Persistent Virtual Folder extension to the IMAP Protocol Status of this Memo By submitting this Internet-Draft, each author represents that any applicable patent or other IPR claims of which he or she is aware have been or will be disclosed, and any of which he or she becomes aware will be disclosed, in accordance with Section 6 of BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. This Internet-Draft will expire on November 30, 2006. Abstract Persistent Extensions to the IMAP Protocol (LPSEARCH) defines extension parameters to the [RFC3501] CREATE command to allow virtual mailboxes to be created which are views of other mailboxes narrowed by search criteria. Conventions used in this document In examples, "C:" and "S:" indicate lines sent by the client and server respectively. Maes [Page 1] May 2006 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119]. An implementation is not compliant if it fails to satisfy one or more of the MUST or REQUIRED level requirements for the protocol(s) it implements. An implementation that satisfies all the MUST or REQUIRED level and all the SHOULD level requirements for a protocol is said to be "unconditionally compliant" to that protocol; one that satisfies all the MUST level requirements but not all the SHOULD level requirements is said to be "conditionally compliant." When describing the general syntax, some definitions are omitted as they are defined in [RFC3501]. Table of Contents Status of this Memo...............................................1 Abstract..........................................................1 Conventions used in this document.................................1 Table of Contents.................................................2 1. Introduction................................................2 2. VFOLDER semantics...........................................3 3. VFOLDER Capability..........................................4 4. CREATE Command Extension....................................4 5. Response Codes..............................................4 A.1. BADBACKING................................................4 A.2. BADSEARCH.................................................5 6. Formal Syntax...............................................5 7. Examples....................................................5 8. Message and Mailbox changes.................................6 9. List Extension..............................................7 10. ACL.........................................................7 A. Avoiding Duplicates (Informative)...........................8 Security Considerations...........................................8 Normative References..............................................8 Future Work.......................................................9 Version History...................................................9 Acknowledgments..................................................10 Authors Addresses................................................10 Intellectual Property Statement..................................11 Disclaimer of Validity...........................................11 Copyright Statement..............................................11 1. Introduction Maes Expires – November 2006 [Page 2] May 2006 The VFOLDER extension is present in any IMAP4 implementation which returns "VFOLDER" as one of the supported capabilities in the CAPABILITY command. A virtual folder (vfolder) is an IMAP4 folder with attached search criteria. A new CREATE parameter allows clients to specify a backing mailbox and search criteria, such that messages in the backing mailbox which satisfy the search criteria are shown in the vfolder. Once created, operations applied to the virtual mailbox, such as APPEND and STORE, may be applied to the backing mailbox. For all intents and purposes, the virtual folder looks and behaves like a real IMAP4 folder, except that it does not contain any messages itself. The intent of this extension is to provide efficient access to potentially large or high velocity mailboxes, for all clients, particularly resource restricted mobile clients. VFOLDER also defines two new response codes and if [LISTEXT] is supported, it defines 1 new selection option and 1 new return option. 2. VFOLDER semantics When a change is made to the backing mailbox, such as the deposit of a new message, or the mutation of a dynamic message attribute, the change must pass the search criteria of the virtual folder before being visible in it. Changes made to dynamic attributes of messages in a vfolder are propagated to the backing mailbox (e.g. STORE) APPEND/COPY to a vfolder SHOULD be supported as well, but may not on some implementations. From the client's perspective, a vfolder should appear to function as a regular mailbox. This includes the ability for a new virtual folder to be created by using another virtual folder as a backing mailbox. A VFOLDER search MUST NOT reference session dependent keys such as MSN sequence sets, NEW, OLD, and RECENT. A VFOLDER implementation MAY permit search on dynamic message attributes (e.g. flags) but clients should not assume support without checking for BADSEARCH response codes. If a server does support search on dynamic attributes, the possibility arises that a message may be included in a vfolder, excluded, and then included again. When this happens, the server MUST generate a new UID each time, and this implies that the order of messages in a vfolder need not match the underlying order of the Maes Expires – November 2006 [Page 3] May 2006 backing mailbox. VFOLDER aware clients may wish to try and detect this case and prevent duplicate downloads. The VFOLDER extension makes a message appear in multiple "mailboxes" at a time; one actual mailbox and zero or more views. Messages can also disappear and reappear in views. This complicates the semantics of the \Recent pseudoflag considerably. To simplify implementation, the server MAY omit computing any \Recent pseudoflag for view mailboxes. In that case, a message is only \Recent when viewed in the underlying mailbox. If it does compute \Recent, it should present the view exactly as an ordinary mailbox. 3. VFOLDER Capability A server which supports LVFOLDER returns "VFOLDER" as one of the responses of the CAPABILITY command. VFOLDER adheres to [CREATEPARAM] and [ABNFEXTEND] syntax so a server MAY also wish to report additional capabilities for extended CREATE. 4. CREATE Command Extension Arguments: mailbox name Optional "VFOLDER" backing mailbox name & search criteria Responses: optional NO responses BADSEARCH, BADBACKING Result: OK created lpsearch completed NO can't create mailbox with that name BAD command unknown or arguments invalid All of the semantics of CREATE as defined in 6.3.3 of [RFC3501] must hold. Additionally, if the backing mailbox name doesn't exist, the creation MUST fail with a NO result and BADBACKING response code. If the search criteria are invalid because the search references search keys which cannot be used (e.g. session dependent keys like NEW, OLD, RECENT) or because the server deems a persistent search on those keys too expensive or not implemented (e.g. mailbox flags), BADSEARCH must be reported with a NO response, or if the SEARCH contains an error in one of its argument values, a NO with a BADSEARCH response is returned. The response SHOULD provide enough explanation to allow a user to correct the search. 5. Response Codes A.1. BADBACKING The mailbox name used for the backing mailbox doesn't exist. Maes Expires – November 2006 [Page 4] May 2006 A.2. BADSEARCH The search criteria violates the pre-conditions mentioned in section 2, or some of the arguments of the search are invalid. 6. Formal Syntax The following syntax specification uses the Augmented Backus-Naur Form (ABNF) notation. Elements not defined here can be found in the formal syntax of the [ABNF], [RFC3501], and [ABNFEXTEND]. The create ABNF grammar in [RFC3501] is hereby modified to the grammar defined in [ABNFEXTEND]. An additional CREATE param "VFOLDER" is introduced whose value is a list containing the backing store mailbox and the search parameters. create_param =/ "VFOLDER" SP "(" backing-mailbox psearch ")" ;; conforms to generic "create-param" syntax as defined in [ABNFEXTEND] backing-mailbox = mailbox psearch = search-program ; defined in [ABNFEXTEND] ; RECENT, NEW, and OLD should not be used. option-extension =/ "VFOLDER" ; option-extension is in [LISTEXT] vfolder-extended-item = "VFOLDER" SP "(" mailbox SP nstring ")" 7. Examples C: a1 CREATE lemonade (VFOLDER(INBOX HEADER "Sender" "lemonade- bounces")) S: a1 OK CREATE VFOLDER Completed Create a persistent mailbox which shows only messages sent to lemonade mailing list. C: a2 CREATE mobile (VFOLDER(INBOX FROM "boss@mycompany.com")) S: a2 OK CREATE VFOLDER Completed Create a mailbox to be synchronized (not in scope of this document) with a mobile device. Maes Expires – November 2006 [Page 5] May 2006 C: a2 CREATE mobile (VFOLDER (INBOX FROM "boss@mycompany.com" WITHIN 3)) S: a2 OK CREATE LPSEARCH Completed Create a mailbox that contains all messages from boss@mycompany.com that were sent within the last 3 days according to the time of the server, utilizing the [WITHIN] draft extension. C: a3 CREATE foo (VFOLDER (IMBOX FROM "boss@mycompany.com")) S: a3 NO [BADBACKING] CREATE failed. IMBOX is not a valid mailbox. Attempt to create a mailbox with a non-existent backing mailbox (fail) C: a3 CREATE foo (VFOLDER (INBOX RECENT)) S: a3 NO [BADSEARCH] CREATE failed. SEARCH refers to session dependent properties. Attempt to create a mailbox with a search based on session dependent keys. C: a3 CREATE foo (VFOLDER (INBOX UNSEEN)) S: a3 NO [BADSEARCH] CREATE failed. SEARCH refers to message flags. VFOLDER with dynamic attributes not implemented by this server. 8. Message and Mailbox changes The DELETE Command (RFC 3501 section 6.3.4) offers a special problem if a mailbox is deleted while there are vfolders onto that mailbox. Servers MUST NOT show messages in deleted mailboxes to clients. If a DELETE command deletes a mailbox, existing vfolders which reference the mailbox should be deleted and not appear in future LIST commands. If a client has a vfolder currently SELECTed, the server MUST not show any messages. The RENAME Command (RFC 3501 section 6.3.5) has a similar problem: If a mailbox is renamed, what happens to views onto that mailbox? The server MAY treat it in the same way as a DELETE command by removing all vfolders attached to the old mailbox name, OR it MAY track the new name and modify all dependent vfolders to use the new folder name. The APPEND Command (RFC 3501 section 6.3.11) and the COPY Command Maes Expires – November 2006 [Page 6] May 2006 (RFC 3501 section 6.4.7) MAY be used to append/copy messages to vfolder, depending on the implementation. A NO response should be generated if the server lacks APPEND/COPY support on VFOLDER. The LIST Command (RFC 3501 section 6.3.8) MUST tag vfolders with the new \Vfolder mailbox flag. (LIST is also described below.) The EXPUNGE Command (RFC 3501 section 6.4.3) causes the underlying mailbox to be expunged when a view is expunged. Servers SHOULD expunge only the messages visible in the view, or MAY expunge the entire mailbox. The former is more desirable, if possible. The IDLE command should treat vfolders the same as any normal mailbox. When new messages arrive, or messages are expunged, an untagged response MUST be sent to the client just as it would if the backing mailbox was selected. 9. List Extension If the server also supports [LISTEXT], a client can find existing vfolders, and can read the search expression for an existing vfolder. The selection option vfolder instructs the server to return LIST responses only for vfolders. The return option vfolders instructs the server to include the view's search and underlying mailbox in a LIST response. Some servers MAY elect to hide vfolders by default so that non- vfolder aware clients cannot see them. In such cases, if a client uses the vfolder selection option, the server MUST return responses with vfolders. 10. ACL SETACL can be used to set access control lists on vfolders, just like on mailboxes. The i right (COPY/APPEND) MAY or MAY NOT be granted on a view. LISTRIGHTS acts as for mailboxes. MYRIGHTS computes access as for mailboxes. However, it may or may not consider the underlying mailbox ACL, depending on how a server implements VFOLDER. Maes Expires – November 2006 [Page 7] May 2006 If it considers the underlying mailbox ACL, the ACL on a mailbox controls all access to the messages stored there. From a security perspective, this may be considered an advantage. If it works independently of the underlying mailbox ACL, vfolders can be used to selectively grant access to a few messages in a mailbox. This can also be viewed as a security advantage, since it allows more finegrained access control. A. Avoiding Duplicates (Informative) With the introduction of VFOLDER, two problems arise. First, a message may appear in one or more vfolders as well as the backing mailbox. Clients may wish to prevent the duplicate retrieval of such messages. Secondly, it is possible for a message to appear, disappear, and reappear in a VFOLDER, which causes the message to be assigned a new UID by the server. This may result in unnecessary duplication of message on a client, may confuse users, and may interfere with notification mechanisms. Clients wishing to check if a message is a duplicate at this point may have to fetch message headers of new message and compare that against the local client cache of messages. This may require some reasonable persistence of deleted messages in the cache. See [IMAP- DISC] section 4.2.2.1 for a discussion of a possible technique. Future versions of VFOLDER may introduce a new server supported mechanism for efficiently determining the original mailbox and UID of a message Security Considerations The VFOLDER extension does not raise any security considerations which are not present in the base protocol. Considerations are the same as for IMAP [RFC 3501]. Normative References [ABNF] D. Crocker, et al. "Augmented BNF for Syntax Specifications: ABNF", RFC 4234, October 2005. http://www.ietf.org/rfc/rfc4234 [ABNFEXTEND] Melnikov, A., and C. Daboo, "Collected extensions to IMAP4 ABNF", work in progress, draft-melnikov-imap-ext-abnf-XX.txt. Maes Expires – November 2006 [Page 8] May 2006 [ACL] Melnikov, A., "IMAP4 Access Control List (ACL) Extension", RFC 4314, Isode Ltd., December 2005. [CREATEPARAM] Melnikov, A., "IMAP CREATE/RENAME parameters", draft- melnikov-imap-createparams-01.txt, September 2005. [IMAP-DISC] Melnikov, A. " Synchronization operations for disconnected IMAP4 clients", draft-melnikov-imap-disc-06.txt, October 2004. [LISTEXT] Leiba, B. and A. Melnikov, "IMAP4 LIST Command Extensions", work in progress, draft-ietf-imapext-list- extensions-xx.txt. [RFC3501] Crispin, M. "IMAP4, Internet Message Access Protocol Version 4 rev1", RFC 3501, March 2003. http://www.ietf.org/rfc/rfc3501 [WITHIN] Maes, S.H., Cromwell, R., "WITHIN Search extension to the IMAP Protocol", draft-maes-lemonade-search-within-02.txt, May 2006 Future Work [1] Determine what, if any, of VIEW and VFOLDER's former restrictions on search parameters must be restored. [2] Better solution for the duplicate download problem [3] Tighten semantics and defined behavior, promoting some MAY or SHOULD to MUST. Version History Release 04 Overhauled since last IETF Plenary. Merged Arnt Gulbrandsen's VIEW draft with VFOLDER 03 draft. Release 04 represents a superset, with some of both VFOLDER and VIEW's restrictions removed. Release 03 Separate SEARCH extension to separate draft Release 02 Update to address comments from Alexey Melnikov, and a new restricted model using immutable message properties Release 01 Maes Expires – November 2006 [Page 9] May 2006 Update to address comments from Alexey Melnikov to follow appropriately the generic syntax provided in draft-melnikov-imap-ext- abnf-05.txt. Release 00 Initial release Acknowledgments We want to give a special thanks to A. Melnikov for his review and suggestions, and to thank Arnt Gulbrandsen for his excellent VIEW proposal, which has now been merged with VFOLDER. The authors want to thank all who have contributed key insight and extensively reviewed and discussed the concepts of LPSEARCH and its early introduction P-IMAP [P-IMAP]. In particular, this includes the authors of the P-IMAP draft: Rafiul Ahad – Oracle Corporation, Eugene Chiu – Oracle Corporation, Ray Cromwell – Oracle Corporation, Jia-der Day – Oracle Corporation, Vi Ha – Oracle Corporation, Wook-Hyun Jeong – Samsung Electronics Co. LTF, Chang Kuang – Oracle Corporation, Rodrigo Lima – Oracle Corporation, Stephane H. Maes – Oracle Corporation, Gustaf Rosell - Sony Ericsson, Jean Sini – Symbol Technologies, Sung-Mu Son – LG Electronics, Fan Xiaohui - CHINA MOBILE COMMUNICATIONS CORPORATION (CMCC), Zhao Lijun - CHINA MOBILE COMMUNICATIONS CORPORATION (CMCC). Authors Addresses Stephane H. Maes Oracle Corporation 500 Oracle Parkway M/S 4op634 Redwood Shores, CA 94065 USA Phone: +1-650-607-6296 Email: stephane.maes@oracle.com Ray Cromwell Oracle Corporation 500 Oracle Parkway Redwood Shores, CA 94065 USA Arnt Gulbrandsen Oryx Mail Systems GmbH Schweppermannstr. 8 D-81671 Muenchen Germany Maes Expires – November 2006 [Page 10] May 2006 Anil Srivastava Sun Microsystems 4150 Network Circle SCA15/201 Santa Clara, CA 94065 anil.srivastava@sun.com Intellectual Property Statement The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79. Copies of IPR disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr. The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at ietf- ipr@ietf.org. Disclaimer of Validity This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Copyright Statement Copyright (C) The Internet Society (2006). This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights. Acknowledgement Maes Expires – November 2006 [Page 11] May 2006 Funding for the RFC Editor function is currently provided by the Internet Society. Maes Expires – November 2006 [Page 12]