wdiff draft-ietf-lemonade-compress-00.txt draft-ietf-lemonade-compress-01.txt

<COMPRESSION> February 2006

Lemonade
Internet Draft: LZIP S. H. Maes
Document: draft-ietf-lemonade-compress-00 R. Cromwell
(Editors)

Expires: August 2006 FebruaryNetwork Working Group Arnt Gulbrandsen
Request for Comments: DRAFT Oryx Mail Systems GmbH
draft-ietf-lemonade-compress-01.txt June 2006

COMPRESSION

The IMAP COMPRESS=DEFLATE Extension

Status of this Memo

By submitting this Internet-Draft, each author represents that any
applicable patent or other IPR claims of which he or she is aware
have been or will be disclosed, and any of which he or she becomes
aware will be disclosed, in accordance with Section 6 of BCP 79.

Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet-
Drafts.

Internet-Drafts are draft documents valid for a maximum of six
months and may be updated, replaced, or obsoleted by other documents
at any time. It is inappropriate to use Internet-Drafts as
reference material or to cite them other than as "work in progress." progress".

The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-Draft Internet-
Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html.

Abstract

Lemonade investigates adding mobile optimizations for the next
version of the Lemonade Profile. LZIP addresses this task and
provides an

The COMPRESS=DEFLATE extension allows an IMAP connection to allow compression of be
compressed using the exchanged text and
binary literals, typically message body parts. DEFLATE algorithm, such that effective
compression is available even when TLS is used.

Conventions used Used in this document
<COMPRESSION> February 2006

In examples, "C:" and "S:" indicate lines sent by the client and
server respectively. This Document

The key words "REQUIRED", "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD
NOT", "RECOMMENDED", "MAY", and "OPTIONAL" "MAY" in this document are to be interpreted as described
in [RFC2119].

An implementation is not compliant if it fails to satisfy one or more
of the MUST or REQUIRED level requirements "Key words for use in RFCs to Indicate Requirement Levels"

[KEYWORDS]. Formal syntax is defined by [ABNF] as modified by
[IMAP].

In the protocol(s) it
implements. An implementation that satisfies all the MUST or REQUIRED
level example, "C:" and all "S:" indicate lines sent by the SHOULD level requirements for a protocol is said to
be "unconditionally compliant" to client and
server respectively.

Introduction and Overview

An IMAP server that protocol; supports this extension announces
"COMPRESS=DEFLATE" as one that satisfies
all the MUST level requirements but not all the SHOULD level
requirements of its capabilities.

The goal of COMPRESS=DEFLATE is said to be "conditionally compliant." When
describing reduce the general syntax, some definitions are omitted as they
are defined in [RFC3501].

Table of Contents

Status bandwidth usage of this Memo ........................................ 1
Copyright Notice............................................ 1
Abstract.................................................... 1
Conventions
IMAP. On regular IMAP connections, the PPP or MNP compression used
with many low-bandwidth links compresses IMAP well. However, when
TLS is used, PPP/MNP compression is ineffective. TLS too may provide
compression, but few or no implementations do so in this document........................... 1
Table of Contents........................................... 2
1. Introduction............................................. 2
2. The CAPABILITY Command................................... 3
3. LZIP Commands............................................ 3
4. LZIP Response............................................ 3
5. Formal Syntax............................................ 4
Security Considerations..................................... 4
References.................................................. 4
Future Work................................................. 5
Version History............................................. 5
Acknowledgments............................................. 5
Authors Addresses........................................... 6
Intellectual Property Statement............................. 6
Disclaimer of Validity...................................... 7
Copyright Statement ........................................ 7

1.
Introduction

LZIP provides an extension practice.

In order to allow increase interoperation, it is desirable to have as few
different compression of text algorithms as possible, so this document
specifies only one. The DEFLATE algorithm is standard, widely
available, unencumbered by patents and binary
literals.

While fairly efficient. Hopefully
it could will not be argued that transport could provide generic
compression necessary to define additional algorithms.

The extension adds one new command (COMPRESS) and no new responses.

The COMPRESS Command

Arguments: Name of the data (e.g. TLS with NULL Cipher), application
level compression presents mechanism: "DEFLATE".
Direction: "UP", "DOWN" or "BOTH".

Responses: None

Result: OK The server will compress its responses (if the advantage to be better tunable direction
is DOWN or BOTH) and expects the client to compress its
commands (if the
<COMPRESSION> February 2006

type of data being requested, for example, direction is UP or BOTH).
NO The connection already is compressed, or the server
doesn't support the requested mechanism, or the direction
specified is unknown.
BAD Command unknown or invalid argument.

The COMPRESS command instructs the server to avoid use the named
compression of
already compressed data.

Compression performances depend on mechanism ("DEFLATE" is the actual types of e-mail that only one defined) for future
commands and/or responses. If the direction specified is "UP", only
commands are received. They change between text bodies and different types of
attachments. In general, LZIP presents a worthwhile gain over
uncompressed or network compressed compressed. If the direction specified is "DOWN", only approached at very little
extra cost
For DEFLATE (as for many other compression mechanisms), the
compressor can trade speed against quality. When decompressing
there isn't much of a tradeoff. Consequently, the implementer.

Bandwidth optimization client and server
are important features required in particular both free to support mobile email use cases [MEMAIL][OMA-ME-RD]

2. pick the best reasonable rate of compression for
the data they send.

The CAPABILITY Command

Servers which support LZIP client MUST return ‘LZIP’ NOT send additional commands until it has seen the
result of COMPRESS.

If both SASL/TLS and COMPRESS are in use, the response list to data should be
compressed before it is encrypted (and decrypted before it is
decompressed), independent of the order in which the client issues
COMPRESS, AUTHENTICATE and STARTTLS.

Example

This example shows a capability command.

Example: A LEMONADE server that implements LZIP.
C: a001 CAPABILITY simple login sequence. The client uses TLS for
privacy and [DEFLATE] for compression.

S: * CAPABILITY IMAP4rev1 AUTH=LOGIN IDLE LZIP OK [CAPABILITY IMAP4REV1 STARTTLS COMPRESS=DEFLATE]
C: a starttls
S: a001 a OK CAPABILITY completed

3.
LZIP Commands
C: b compress deflate
S: b OK
C: c login arnt tnra
S: c OK

Compression Efficiency

IMAP poses some unusual problems for a compression layer.

Upstream is fairly simple. Most IMAP clients send the same few
commands again and again, so any compression algorith which can
exploit quotes works efficiently. The LZIP APPEND command is an extension of [RFC3516] IMAP BINARY,
exception; clients which
introduces three new send many APPEND commands “LZIP”, “LZIP.PEEK”, “LZIP.SIZE” that
parallel may want to take
special care.

Downstream has the syntax and semantics unusual property that 3-4 kinds of “BINARY”, “BINARY.PEEK”, and
“BINARY.SIZE” in [RFC3516]. In general, LZIP inherits data are sent,
confusing all of the
requirements and semantics of [RFC3516]’s “BINARY” and “BINARY.PEEK”,
except that the content transfer encoding being requested dictionary-based compression algorithms.

The first type is
understood IMAP responses. These are highly compressible;
zlib using its least CPU-intensive setting compresses typical
responses to be the result 25-40% of what would be returned their original size.

The second is email headers. These are equally compressible, and
benefit from BINARY
decoding, followed by using the application of same dictionary as the DEFLATE algorithm.

Example: Zipping a IMAP responses.

The third is email body part fetch
C: A1 FETCH 123 LZIP.PEEK[1.2]
S: * LZIP[1.2]~{1234}
S: ….binary decoded and deflated data….
S: A1 OK FETCH completed

As mentioned in RFC3516, LZIP.SIZE text. Text is usually fairly short and
includes much ASCII, so the same compression dictionary will do a potentially expensive
operation, as
good job here, too. When multiple messages in LZIP, so clients should be aware that making
successive requests for the same part thread are
read at the same time, quoted lines etc. can often be compressed
almost to zero.

Finally, attachments (non-text email bodies) are transmitted, either
in [BINARY] form or encoded with base-64.

When attachments are retrieved in [BINARY] form, DEFLATE may be expensive.

4.
LZIP Response

As able
to compress them, but the result format of processing the attachment is usually not
IMAP-like, so the dictionary built while compressing IMAP does not
help. The compressor has to adapt from IMAP to the attachment's
format, and then back.

When attachments are retrieved in base-64 form, the same problems
apply, but the base-64 encoding adds another problem. 8-bit
compression algorithms such as deflate work well on 8-bit file
formats, however base-64 turns a file into something resembling a
6-bit bytes in an LZIP command, two new responses, LZIP 8-bit format.

A few file formats aren't compressible using deflate, e.g. .gz, .zip
and .jpg files.

According to the author's measurements, the compression level used
makes little difference. zlib's level 1 compresses IMAP almost as
well as level 9, and LZIP.SIZE which parallel for the receiver, level 1 seems to require
(just a tiny bit) pmore CPU than level 9. Independent verification
is strongly desired.

Implementation Notes

When using the zlib library (see [DEFLATE]), the functions
deflateInit(), deflate(), inflateInit() and inflate() suffice to
implement this extension.

Note that responses of [RFC3516] are
<COMPRESSION> February 2006

introduced. They when using TLS, compression may actually decrease the CPU
usage, depending on which algorithms are identical used in syntax TLS. This is
because fewer bytes need to be encrypted, and encryption is
generally more expensive than compression.

A client can improve downstream compression by implementing [BINARY]
and semantics using FETCH BINARY instead of FETCH BODY.

A server can improve downstream compression if it hints to the BINARY
responses in [RFC3516] in everyway, except
compressor that the resulting binary data type is about to change strongly, e.g. by
sending a Z_FULL_FLUSH at the start and end of large non-text
literals (before and after '*CHAR8' in the definition of literal in
RFC 3501, page 86).

A server can improve the CPU efficiency both of the server and the
client if it adjusts the compression level (e.g. using the
deflateParams() function in zlib) at these points. A very simple
strategy is understood to be change the level 0 to at the start of a literal
provided the first two bytes are either 0x1F 0x8B (as in DEFLATE format.

5. deflate-
compressed files) or 0xFF 0xD8 (JPEG), and to keep it at 1-5 the
rest of the time.

Formal Syntax

The following syntax specification uses the Augmented Backus-Naur
Form (ABNF) notation. Elements notation as specified in [ABNF]. Non-terminals
referenced but not defined here can be found in
the formal syntax of the [ABNF], [RFC3501], and [ABNFEXTEND]. below are as defined by [ABNF] (SP, CRLF)
or [IMAP] (all others).

Except as noted otherwise, all alphabetic characters are case-
insensitive. The create ABNF grammar in [RFC3501] use of upper or lower case characters to define
token strings is hereby modified to the
grammar defined for editorial clarity only. Implementations MUST
accept these strings in [ABNFEXTEND]

fetch-att =/ "LZIP" [".PEEK"] section-binary [partial]
/ "LZIP.SIZE" section-binary

msg-att-static a case-insensitive fashion.

command-any =/ "LZIP" section-binary compress

compress = "COMPRESS" SP algorithm SP (nstring ( "UP" / literal8) "DOWN" / "LZIP.SIZE" section-binary SP number
"BOTH" )

algorithm = "DEFLATE"

Security considerations

(As for [TLSCOMP] RFC 3749.)

IANA Considerations

LZIP does not introduce additional security consideration with
respect

The IANA is requested to IMAPv4Rev1.

References

[LEMONADEPROFILE] Maes, S.H. and Melnikov A., "Lemonade Profile",
draft-ietf-lemonade-profile-XX.txt, (work in progress).

[MEMAIL] Maes, S.H., “Lemonade and Mobile e-mail", draft-maes-
lemonade-mobile-email-xx.txt, (work in progress).

[OMA-ME-RD] Open Mobile Alliance Mobile Email Requirement Document,
(Work in progress). http://www.openmobilealliance.org/

[P-IMAP] Maes, S.H., Lima R., Kuang, C., Cromwell, R., Ha, V. and
Chiu, E., Day, J., Ahad R., Jeong W-H., Rosell G., Sini, J., Sohn
S-M., Xiaohui F. and Lijun Z., "Push Extensions add COMPRESS=DEFLATE to the list of IMAP
Protocol (P-IMAP)", draft-maes-lemonade-p-imap-xx.txt, (work
extensions.

Credits

Quite a few people on the LEMONADE mailing list have offered
comments, including Dave Cridland, Ned Freed and Tony Hansen. And
various people in
progress).

[RFC1951] Deutsch, P. “DEFLATE Compressed Data Format Specification
version 1.3”, RFC1951, May 1996.
http://www.ietf.org/rfc/rfc1951
<COMPRESSION> February 2006

[RFC2119] Brader, S. "Keywords the rooms at meetings. Send me mail, I'll add you.

Open Issues

Both ends can already disable compression at any point by calling
deflateParams(). The only missing feature is for use in RFCs the client to Indicate
Requirement Levels",
request that the server stop compressing - are there use-cases for
that? It requires adding more server-side state, so I'm wary.

What text and numbers are needed wrt. compression levels? A bit of
solid information is not amiss.

Normative References

[ABNF] Crocker, Overell, "Augmented BNF for Syntax
Specifications: ABNF", RFC 2119, March 2234, Internet Mail
Consortium, Demon Internet Ltd, November 1997.
http://www.ietf.org/rfc/rfc2119

[RFC3501]

[IMAP] Crispin, M. "IMAP4, Internet "Internet Message Access Protocol - Version 4 rev1",
4rev1", RFC 3501, March 2003.
http://www.ietf.org/rfc/rfc3501

[RFC3516] Nerenberg, L. “IMAP4 Binary Content Extension”, RFC3516,
April University of Washington, June 2003.
http://www.ietf.org/rfc/rfc3516

Future Work

Should a new “compressed literal” be considered paralleling the
binary literal8 syntax? For example, %~{nz-number}? Potential
applications could be its usage in APPEND/CATENATE.

Version History

Release 00 of draft-maes-lemonadel-lzip
Initial release published

[KEYWORDS] Bradner, "Key words for use in June 2005
Release 01 of draft-maes-lemonadel-lzip
Shortened list of editors. Authors pushed to acknowledgements
Section 2: Addition of exact compression algorithm
references
Section 4:
Addition of exact compression algorithm references
Considerations on command compression added
Correction and updates of examples
References:
Additional references on compression algorithms and IMAP4
Binary.
Release 02 of draft-maes-lemonadel-lzip
Reworked to model IMAP BINARY
Release 00 of IETF draft
Re-cast LZIP to focus on compression of text and binary
literals.

Acknowledgments

The authors want to thank all who have contributed key insight and
extensively reviewed and discussed the concepts of LPSEARCH and its
early introduction P-IMAP [P-IMAP]. In particular, this includes the
authors of the P-IMAP draft: Rafiul Ahad – Oracle Corporation, Eugene
Chiu – Oracle Corporation, Ray Cromwell – Oracle Corporation, Jia-der
Day – Oracle Corporation, Vi Ha – Oracle Corporation, Wook-Hyun Jeong
– Samsung Electronics Co. LTF, Chang Kuang – Oracle Corporation,
Rodrigo Lima – Oracle Corporation, Stephane H. Maes – Oracle
<COMPRESSION> February 2006

Corporation, Gustaf Rosell - Sony Ericsson, Jean Sini – Symbol
Technologies, Sung-Mu Son – LG Electronics, Fan Xiaohui - CHINA
MOBILE COMMUNICATIONS CORPORATION (CMCC), Zhao Lijun - CHINA MOBILE
COMMUNICATIONS CORPORATION (CMCC). We also want to give a special
thanks RFCs to A. Melnikov for his review Indicate
Requirement Levels", RFC 2119, Harvard University, March
1997.

[DEFLATE] Deutsch, "DEFLATE Compressed Data Format Specification
version 1.3", RFC 1951, Aladdin Enterprises, May 1996.

[STARTTLS] Newman, C. "Using TLS with IMAP, POP3 and suggestions.

Authors Addresses

Stephane H. Maes
Oracle Corporation
500 Oracle Parkway
M/S 4op634
Redwood Shores, CA 94065
USA
Phone: +1-650-607-6296 ACAP", RFC
2595, June 1999.

Informative References

[TLSCOMP] Hollenbeck, "Transport Layer Security Protocol
Compression Methods", RFC 3749, VeriSign, May 2004.

Author's Address

Arnt Gulbrandsen
Oryx Mail Systems GmbH
Schweppermannstr. 8
D-81671 Muenchen
Germany

Fax: +49 89 4502 9758

Email: stephane.maes@oracle.com

Ray Cromwell
Oracle Corporation
500 Oracle Parkway
Redwood Shores, CA 94065
USA

Anil Srivastava
Sun Microsystems
4150 Network Circle SCA15/201
Santa Clara, CA 94065
anil.srivastava@sun.com arnt@oryx.com

Intellectual Property Statement

The IETF takes no position regarding the validity or scope of any
Intellectual Property Rights or other rights that might be claimed to
pertain to the implementation or use of the technology described in this
document or the extent to which any license under such rights might or
might not be available; nor does it represent that it has made any
independent effort to identify any such rights. Information on the
procedures with respect to rights in RFC documents can be found in BCP 7878 78
and BCP 79.

Copies of IPR disclosures made to the IETF Secretariat and any assurances
of licenses to be made available, or the result of an attempt made to
obtain a general license or permission for the use of such proprietary
rights by implementers or users of this specification can be obtained from
the IETF on-line IPR repository at http://www.ietf.org/ipr.

<COMPRESSION> February 2006

The IETF invites any interested party to bring to its attention any
copyrights, patents or patent applications, or other proprietary rights
that may cover technology that may be required to implement this standard.
Please address the information to the IETF at ietf-
ipr@ietf.org. ietf-ipr@ietf.org.

This document is subject to the rights, licenses and restrictions
contained in BCP 78, and except as set forth therein, the authors retain
all their rights.

Disclaimer of Validity

This document and the information contained herein are provided on an "AS
IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS
SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET ENGINEERING
TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT
LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT
INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR
FITNESS FOR A PARTICULAR PURPOSE.

Copyright (C) The Internet Society (2006). This document is subject
to the rights, licenses and restrictions contained in BCP 78, and
except as set forth therein, the authors retain all their rights.

Acknowledgement

Acknowledgment

Funding for the RFC Editor function is currently provided by the Internet
Society.