< draft-ietf-lemonade-compress-00.txt   draft-ietf-lemonade-compress-01.txt >
<COMPRESSION> February 2006 Network Working Group Arnt Gulbrandsen
Request for Comments: DRAFT Oryx Mail Systems GmbH
Lemonade draft-ietf-lemonade-compress-01.txt June 2006
Internet Draft: LZIP S. H. Maes
Document: draft-ietf-lemonade-compress-00 R. Cromwell
(Editors)
Expires: August 2006 February 2006 The IMAP COMPRESS=DEFLATE Extension
COMPRESSION
Status of this Memo Status of this Memo
By submitting this Internet-Draft, each author represents that any By submitting this Internet-Draft, each author represents that any
applicable patent or other IPR claims of which he or she is aware applicable patent or other IPR claims of which he or she is aware
have been or will be disclosed, and any of which he or she becomes have been or will be disclosed, and any of which he or she becomes
aware will be disclosed, in accordance with Section 6 of BCP 79. aware will be disclosed, in accordance with Section 6 of BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet-
Drafts.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are working documents of the Internet Engineering
and may be updated, replaced, or obsoleted by other documents at any Task Force (IETF), its areas, and its working groups. Note that
time. It is inappropriate to use Internet-Drafts as reference other groups may also distribute working documents as Internet-
material or to cite them other than as "work in progress." Drafts.
The list of current Internet-Drafts can be accessed at Internet-Drafts are draft documents valid for a maximum of six
http://www.ietf.org/ietf/1id-abstracts.txt. months and may be updated, replaced, or obsoleted by other documents
at any time. It is inappropriate to use Internet-Drafts as
reference material or to cite them other than as "work in progress".
The list of Internet-Draft Shadow Directories can be accessed at The list of current Internet-Drafts can be accessed at
http://www.ietf.org/shadow.html. http://www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-
Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html.
Copyright Notice Copyright Notice
Copyright (C) The Internet Society (2006). Copyright (C) The Internet Society 2006.
Abstract Abstract
Lemonade investigates adding mobile optimizations for the next The COMPRESS=DEFLATE extension allows an IMAP connection to be
version of the Lemonade Profile. LZIP addresses this task and compressed using the DEFLATE algorithm, such that effective
provides an extension to allow compression of the exchanged text and compression is available even when TLS is used.
binary literals, typically message body parts.
Conventions used in this document Conventions Used in This Document
<COMPRESSION> February 2006
In examples, "C:" and "S:" indicate lines sent by the client and The key words "REQUIRED", "MUST", "MUST NOT", "SHOULD", "SHOULD
server respectively. NOT", and "MAY" in this document are to be interpreted as described
in "Key words for use in RFCs to Indicate Requirement Levels"
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", [KEYWORDS]. Formal syntax is defined by [ABNF] as modified by
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this [IMAP].
document are to be interpreted as described in [RFC2119].
An implementation is not compliant if it fails to satisfy one or more In the example, "C:" and "S:" indicate lines sent by the client and
of the MUST or REQUIRED level requirements for the protocol(s) it server respectively.
implements. An implementation that satisfies all the MUST or REQUIRED
level and all the SHOULD level requirements for a protocol is said to
be "unconditionally compliant" to that protocol; one that satisfies
all the MUST level requirements but not all the SHOULD level
requirements is said to be "conditionally compliant." When
describing the general syntax, some definitions are omitted as they
are defined in [RFC3501].
Table of Contents Introduction and Overview
Status of this Memo ........................................ 1 An IMAP server that supports this extension announces
Copyright Notice............................................ 1 "COMPRESS=DEFLATE" as one of its capabilities.
Abstract.................................................... 1
Conventions used in this document........................... 1
Table of Contents........................................... 2
1. Introduction............................................. 2
2. The CAPABILITY Command................................... 3
3. LZIP Commands............................................ 3
4. LZIP Response............................................ 3
5. Formal Syntax............................................ 4
Security Considerations..................................... 4
References.................................................. 4
Future Work................................................. 5
Version History............................................. 5
Acknowledgments............................................. 5
Authors Addresses........................................... 6
Intellectual Property Statement............................. 6
Disclaimer of Validity...................................... 7
Copyright Statement ........................................ 7
1. The goal of COMPRESS=DEFLATE is to reduce the bandwidth usage of
Introduction IMAP. On regular IMAP connections, the PPP or MNP compression used
with many low-bandwidth links compresses IMAP well. However, when
TLS is used, PPP/MNP compression is ineffective. TLS too may provide
compression, but few or no implementations do so in practice.
LZIP provides an extension to allow compression of text and binary In order to increase interoperation, it is desirable to have as few
literals. different compression algorithms as possible, so this document
specifies only one. The DEFLATE algorithm is standard, widely
available, unencumbered by patents and fairly efficient. Hopefully
it will not be necessary to define additional algorithms.
While it could be argued that transport could provide generic The extension adds one new command (COMPRESS) and no new responses.
compression of the data (e.g. TLS with NULL Cipher), application
level compression presents the advantage to be better tunable to the
<COMPRESSION> February 2006
type of data being requested, for example, to avoid compression of The COMPRESS Command
already compressed data.
Compression performances depend on the actual types of e-mail that Arguments: Name of compression mechanism: "DEFLATE".
are received. They change between text bodies and different types of Direction: "UP", "DOWN" or "BOTH".
attachments. In general, LZIP presents a worthwhile gain over
uncompressed or network compressed only approached at very little
extra cost for the implementer.
Bandwidth optimization are important features required in particular Responses: None
to support mobile email use cases [MEMAIL][OMA-ME-RD]
2. Result: OK The server will compress its responses (if the direction
The CAPABILITY Command is DOWN or BOTH) and expects the client to compress its
commands (if the direction is UP or BOTH).
NO The connection already is compressed, or the server
doesn't support the requested mechanism, or the direction
specified is unknown.
BAD Command unknown or invalid argument.
Servers which support LZIP MUST return ‘LZIP’ in the response list to The COMPRESS command instructs the server to use the named
a capability command. compression mechanism ("DEFLATE" is the only one defined) for future
commands and/or responses. If the direction specified is "UP", only
commands are compressed. If the direction specified is "DOWN", only
For DEFLATE (as for many other compression mechanisms), the
compressor can trade speed against quality. When decompressing
there isn't much of a tradeoff. Consequently, the client and server
are both free to pick the best reasonable rate of compression for
the data they send.
Example: A LEMONADE server that implements LZIP. The client MUST NOT send additional commands until it has seen the
C: a001 CAPABILITY result of COMPRESS.
S: * CAPABILITY IMAP4rev1 AUTH=LOGIN IDLE LZIP
S: a001 OK CAPABILITY completed
3. If both SASL/TLS and COMPRESS are in use, the data should be
LZIP Commands compressed before it is encrypted (and decrypted before it is
decompressed), independent of the order in which the client issues
COMPRESS, AUTHENTICATE and STARTTLS.
The LZIP command is an extension of [RFC3516] IMAP BINARY, which Example
introduces three new commands “LZIP”, “LZIP.PEEK”, “LZIP.SIZE” that
parallel the syntax and semantics of “BINARY”, “BINARY.PEEK”, and
“BINARY.SIZE” in [RFC3516]. In general, LZIP inherits all of the
requirements and semantics of [RFC3516]’s “BINARY” and “BINARY.PEEK”,
except that the content transfer encoding being requested is
understood to be the result of what would be returned from BINARY
decoding, followed by the application of the DEFLATE algorithm.
Example: Zipping a body part fetch This example shows a simple login sequence. The client uses TLS for
C: A1 FETCH 123 LZIP.PEEK[1.2] privacy and [DEFLATE] for compression.
S: * LZIP[1.2]~{1234}
S: ….binary decoded and deflated data….
S: A1 OK FETCH completed
As mentioned in RFC3516, LZIP.SIZE is a potentially expensive S: * OK [CAPABILITY IMAP4REV1 STARTTLS COMPRESS=DEFLATE]
operation, as in LZIP, so clients should be aware that making C: a starttls
successive requests for the same part may be expensive. S: a OK
C: b compress deflate
S: b OK
C: c login arnt tnra
S: c OK
4. Compression Efficiency
LZIP Response
As the result of processing an LZIP command, two new responses, LZIP IMAP poses some unusual problems for a compression layer.
and LZIP.SIZE which parallel that responses of [RFC3516] are
<COMPRESSION> February 2006
introduced. They are identical in syntax and semantics of the BINARY Upstream is fairly simple. Most IMAP clients send the same few
responses in [RFC3516] in everyway, except that the resulting binary commands again and again, so any compression algorith which can
literal is understood to be in DEFLATE format. exploit quotes works efficiently. The APPEND command is an
exception; clients which send many APPEND commands may want to take
special care.
5. Downstream has the unusual property that 3-4 kinds of data are sent,
Formal Syntax confusing all dictionary-based compression algorithms.
The following syntax specification uses the Augmented Backus-Naur The first type is IMAP responses. These are highly compressible;
Form (ABNF) notation. Elements not defined here can be found in zlib using its least CPU-intensive setting compresses typical
the formal syntax of the [ABNF], [RFC3501], and [ABNFEXTEND]. responses to 25-40% of their original size.
The create ABNF grammar in [RFC3501] is hereby modified to the The second is email headers. These are equally compressible, and
grammar defined in [ABNFEXTEND] benefit from using the same dictionary as the IMAP responses.
fetch-att =/ "LZIP" [".PEEK"] section-binary [partial] The third is email body text. Text is usually fairly short and
/ "LZIP.SIZE" section-binary includes much ASCII, so the same compression dictionary will do a
good job here, too. When multiple messages in the same thread are
read at the same time, quoted lines etc. can often be compressed
almost to zero.
msg-att-static =/ "LZIP" section-binary SP (nstring / literal8) Finally, attachments (non-text email bodies) are transmitted, either
/ "LZIP.SIZE" section-binary SP number in [BINARY] form or encoded with base-64.
Security Considerations When attachments are retrieved in [BINARY] form, DEFLATE may be able
to compress them, but the format of the attachment is usually not
IMAP-like, so the dictionary built while compressing IMAP does not
help. The compressor has to adapt from IMAP to the attachment's
format, and then back.
LZIP does not introduce additional security consideration with When attachments are retrieved in base-64 form, the same problems
respect to IMAPv4Rev1. apply, but the base-64 encoding adds another problem. 8-bit
compression algorithms such as deflate work well on 8-bit file
formats, however base-64 turns a file into something resembling a
6-bit bytes in an 8-bit format.
References A few file formats aren't compressible using deflate, e.g. .gz, .zip
and .jpg files.
[LEMONADEPROFILE] Maes, S.H. and Melnikov A., "Lemonade Profile", According to the author's measurements, the compression level used
draft-ietf-lemonade-profile-XX.txt, (work in progress). makes little difference. zlib's level 1 compresses IMAP almost as
well as level 9, and for the receiver, level 1 seems to require
(just a tiny bit) pmore CPU than level 9. Independent verification
is strongly desired.
[MEMAIL] Maes, S.H., “Lemonade and Mobile e-mail", draft-maes- Implementation Notes
lemonade-mobile-email-xx.txt, (work in progress).
[OMA-ME-RD] Open Mobile Alliance Mobile Email Requirement Document, When using the zlib library (see [DEFLATE]), the functions
(Work in progress). http://www.openmobilealliance.org/ deflateInit(), deflate(), inflateInit() and inflate() suffice to
implement this extension.
[P-IMAP] Maes, S.H., Lima R., Kuang, C., Cromwell, R., Ha, V. and Note that when using TLS, compression may actually decrease the CPU
Chiu, E., Day, J., Ahad R., Jeong W-H., Rosell G., Sini, J., Sohn usage, depending on which algorithms are used in TLS. This is
S-M., Xiaohui F. and Lijun Z., "Push Extensions to the IMAP because fewer bytes need to be encrypted, and encryption is
Protocol (P-IMAP)", draft-maes-lemonade-p-imap-xx.txt, (work in generally more expensive than compression.
progress).
[RFC1951] Deutsch, P. “DEFLATE Compressed Data Format Specification A client can improve downstream compression by implementing [BINARY]
version 1.3”, RFC1951, May 1996. and using FETCH BINARY instead of FETCH BODY.
http://www.ietf.org/rfc/rfc1951
<COMPRESSION> February 2006
[RFC2119] Brader, S. "Keywords for use in RFCs to Indicate A server can improve downstream compression if it hints to the
Requirement Levels", RFC 2119, March 1997. compressor that the data type is about to change strongly, e.g. by
http://www.ietf.org/rfc/rfc2119 sending a Z_FULL_FLUSH at the start and end of large non-text
literals (before and after '*CHAR8' in the definition of literal in
RFC 3501, page 86).
[RFC3501] Crispin, M. "IMAP4, Internet Message Access Protocol A server can improve the CPU efficiency both of the server and the
Version 4 rev1", RFC 3501, March 2003. client if it adjusts the compression level (e.g. using the
http://www.ietf.org/rfc/rfc3501 deflateParams() function in zlib) at these points. A very simple
strategy is to change the level 0 to at the start of a literal
provided the first two bytes are either 0x1F 0x8B (as in deflate-
compressed files) or 0xFF 0xD8 (JPEG), and to keep it at 1-5 the
rest of the time.
[RFC3516] Nerenberg, L. “IMAP4 Binary Content Extension”, RFC3516, Formal Syntax
April 2003.
http://www.ietf.org/rfc/rfc3516
Future Work The following syntax specification uses the Augmented Backus-Naur
Form (ABNF) notation as specified in [ABNF]. Non-terminals
referenced but not defined below are as defined by [ABNF] (SP, CRLF)
or [IMAP] (all others).
Should a new “compressed literal” be considered paralleling the Except as noted otherwise, all alphabetic characters are case-
binary literal8 syntax? For example, %~{nz-number}? Potential insensitive. The use of upper or lower case characters to define
applications could be its usage in APPEND/CATENATE. token strings is for editorial clarity only. Implementations MUST
accept these strings in a case-insensitive fashion.
Version History command-any =/ compress
Release 00 of draft-maes-lemonadel-lzip compress = "COMPRESS" SP algorithm SP ( "UP" / "DOWN" /
Initial release published in June 2005 "BOTH" )
Release 01 of draft-maes-lemonadel-lzip
Shortened list of editors. Authors pushed to acknowledgements
Section 2: Addition of exact compression algorithm
references
Section 4:
Addition of exact compression algorithm references
Considerations on command compression added
Correction and updates of examples
References:
Additional references on compression algorithms and IMAP4
Binary.
Release 02 of draft-maes-lemonadel-lzip
Reworked to model IMAP BINARY
Release 00 of IETF draft
Re-cast LZIP to focus on compression of text and binary
literals.
Acknowledgments algorithm = "DEFLATE"
The authors want to thank all who have contributed key insight and Security considerations
extensively reviewed and discussed the concepts of LPSEARCH and its
early introduction P-IMAP [P-IMAP]. In particular, this includes the
authors of the P-IMAP draft: Rafiul Ahad – Oracle Corporation, Eugene
Chiu – Oracle Corporation, Ray Cromwell – Oracle Corporation, Jia-der
Day – Oracle Corporation, Vi Ha – Oracle Corporation, Wook-Hyun Jeong
– Samsung Electronics Co. LTF, Chang Kuang – Oracle Corporation,
Rodrigo Lima – Oracle Corporation, Stephane H. Maes – Oracle
<COMPRESSION> February 2006
Corporation, Gustaf Rosell - Sony Ericsson, Jean Sini – Symbol (As for [TLSCOMP] RFC 3749.)
Technologies, Sung-Mu Son – LG Electronics, Fan Xiaohui - CHINA
MOBILE COMMUNICATIONS CORPORATION (CMCC), Zhao Lijun - CHINA MOBILE
COMMUNICATIONS CORPORATION (CMCC). We also want to give a special
thanks to A. Melnikov for his review and suggestions.
Authors Addresses IANA Considerations
Stephane H. Maes The IANA is requested to add COMPRESS=DEFLATE to the list of IMAP
Oracle Corporation extensions.
500 Oracle Parkway
M/S 4op634
Redwood Shores, CA 94065
USA
Phone: +1-650-607-6296
Email: stephane.maes@oracle.com
Ray Cromwell Credits
Oracle Corporation
500 Oracle Parkway
Redwood Shores, CA 94065
USA
Anil Srivastava Quite a few people on the LEMONADE mailing list have offered
Sun Microsystems comments, including Dave Cridland, Ned Freed and Tony Hansen. And
4150 Network Circle SCA15/201 various people in the rooms at meetings. Send me mail, I'll add you.
Santa Clara, CA 94065
anil.srivastava@sun.com
Intellectual Property Statement Open Issues
The IETF takes no position regarding the validity or scope of any Both ends can already disable compression at any point by calling
Intellectual Property Rights or other rights that might be claimed to deflateParams(). The only missing feature is for the client to
pertain to the implementation or use of the technology described in request that the server stop compressing - are there use-cases for
this document or the extent to which any license under such rights that? It requires adding more server-side state, so I'm wary.
might or might not be available; nor does it represent that it has
made any independent effort to identify any such rights. Information
on the procedures with respect to rights in RFC documents can be
found in BCP 7878 and BCP 79.
Copies of IPR disclosures made to the IETF Secretariat and any What text and numbers are needed wrt. compression levels? A bit of
assurances of licenses to be made available, or the result of an solid information is not amiss.
attempt made to obtain a general license or permission for the use of
such proprietary rights by implementers or users of this
specification can be obtained from the IETF on-line IPR repository at
http://www.ietf.org/ipr.
<COMPRESSION> February 2006 Normative References
The IETF invites any interested party to bring to its attention any [ABNF] Crocker, Overell, "Augmented BNF for Syntax
copyrights, patents or patent applications, or other proprietary Specifications: ABNF", RFC 2234, Internet Mail
rights that may cover technology that may be required to implement Consortium, Demon Internet Ltd, November 1997.
this standard. Please address the information to the IETF at ietf-
ipr@ietf.org.
Disclaimer of Validity [IMAP] Crispin, "Internet Message Access Protocol - Version
This document and the information contained herein are provided on an 4rev1", RFC 3501, University of Washington, June 2003.
"AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET [KEYWORDS] Bradner, "Key words for use in RFCs to Indicate
ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, Requirement Levels", RFC 2119, Harvard University, March
INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE 1997.
INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. [DEFLATE] Deutsch, "DEFLATE Compressed Data Format Specification
version 1.3", RFC 1951, Aladdin Enterprises, May 1996.
[STARTTLS] Newman, C. "Using TLS with IMAP, POP3 and ACAP", RFC
2595, June 1999.
Informative References
[TLSCOMP] Hollenbeck, "Transport Layer Security Protocol
Compression Methods", RFC 3749, VeriSign, May 2004.
Author's Address
Arnt Gulbrandsen
Oryx Mail Systems GmbH
Schweppermannstr. 8
D-81671 Muenchen
Germany
Fax: +49 89 4502 9758
Email: arnt@oryx.com
Intellectual Property Statement
The IETF takes no position regarding the validity or scope of any
Intellectual Property Rights or other rights that might be claimed to
pertain to the implementation or use of the technology described in this
document or the extent to which any license under such rights might or
might not be available; nor does it represent that it has made any
independent effort to identify any such rights. Information on the
procedures with respect to rights in RFC documents can be found in BCP 78
and BCP 79.
Copies of IPR disclosures made to the IETF Secretariat and any assurances
of licenses to be made available, or the result of an attempt made to
obtain a general license or permission for the use of such proprietary
rights by implementers or users of this specification can be obtained from
the IETF on-line IPR repository at http://www.ietf.org/ipr.
The IETF invites any interested party to bring to its attention any
copyrights, patents or patent applications, or other proprietary rights
that may cover technology that may be required to implement this standard.
Please address the information to the IETF at ietf-ipr@ietf.org.
Copyright Statement Copyright Statement
Copyright (C) The Internet Society (2006). This document is subject Copyright (C) The Internet Society (2006).
to the rights, licenses and restrictions contained in BCP 78, and
except as set forth therein, the authors retain all their rights.
Acknowledgement This document is subject to the rights, licenses and restrictions
contained in BCP 78, and except as set forth therein, the authors retain
all their rights.
Funding for the RFC Editor function is currently provided by the Disclaimer of Validity
Internet Society.
This document and the information contained herein are provided on an "AS
IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS
SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET ENGINEERING
TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT
LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT
INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR
FITNESS FOR A PARTICULAR PURPOSE.
Acknowledgment
Funding for the RFC Editor function is currently provided by the Internet
Society.
 End of changes. 67 change blocks. 
246 lines changed or deleted 230 lines changed or added

This html diff was produced by rfcdiff 1.48. The latest version is available from http://tools.ietf.org/tools/rfcdiff/