[hybi] [permessage-deflate] Compressing fragmented data

Takeshi Yoshino <tyoshino@google.com> Tue, 14 January 2014 04:05 UTC

MIME-Version: 1.0
From: Takeshi Yoshino <tyoshino@google.com>
Date: Tue, 14 Jan 2014 13:05:07 +0900
Message-ID: <CAH9hSJbeY5VOY_iuwrdBq-KYcoVkW_8KArPp70hP4tdZj6eQfg@mail.gmail.com>
To: Arman Djusupov <arman@noemax.com>
Content-Type: multipart/alternative; boundary="e89a8f3bad456f293504efe64deb"
Cc: "hybi@ietf.org" <hybi@ietf.org>
Subject: [hybi] [permessage-deflate] Compressing fragmented data
Precedence: list

Hi Arman,

Let me change the subject.

On Thu, Jan 9, 2014 at 11:46 PM, Arman Djusupov <arman@noemax.com> wrote:

> Hello Takeshi,
>
>
>
> In the current draft, the description of the compression process states :
>
>
>
>    1.  Compress all the octets of the payload of the message using
>
>        DEFLATE.
>
>
>
> This doesn’t take into account outbound messages of arbitrary size that
> NEED to be fragmented, for which it is more favorable to use
> fragmentation and to compress & send fragments until the end of the
> message is reached.
>

OK. I'll add some text to explain how this works for fragmented messages.


>
>
> In such cases the implementation buffers the input from its source and
> then flushes it out into a compressed fragment, repeating this processuntil the end of the source data is
> reached. At that point it flushes any remaining bytes buffered into a
> final frame, or flushes a final frame of 0 length.
>
>
>
>
>
> When the implementation produces compressed fragments, it periodically
> produces frames with 0x00 0x00 0xFF 0xFF at the end of the frame due to
> flushing DEFLATE. But because of the following requirements:
>
>
>
>    2. If the resulting data does not end with an empty DEFLATE block
>
>        with no compression (the "BTYPE" bits is set to 00), append an
>
>        empty DEFLATE block with no compression to the tail end.
>
>
>
>    3.  Remove 4 octets (that are 0x00 0x00 0xff 0xff) from the tail end.
>
>        After this step, the last octet of the compressed data contains
>
>        (possibly part of) the DEFLATE header bits with the "BTYPE" bits
>
>        set to 00.
>
>
>
> the implementation must ensure that the message ends empty blocks without a
> trailing 0x0000FFFF, so the implementation must keep track on what type
> of block was at the end of the frame that was sent last. If the last frame
> sent ends with 0x0000FFFF then the implementation cannot remove those 4
> bytes from the wire, but it must artificially produce an empty DEFLATE
> block to send as the final frame. This is only due to the requirement to
> remove 0x0000FFFF.
>

Yes. It's just two byte long. DEFLATE block header (3bit) + Fixed Huffman
code for end of block symbol (7bit) + padding (6bit).


>
>

>
> Wouldn’t it be easier to make the removal 0x00 0x00 0xFF 0xFF optional,
> while at the same time requiring that the receiving side appends a 0x00
> 0x00 0xFF 0xFF to the final frame in case when its missing?
>
>
>

I think it complicates decoders. Code which doesn't know DEFLATE
decompressor's state cannot check if 0x00 0x00 0xFF 0xFF at the end is
really an uncompressed block body or not. It could be something else, e.g.
body of compressed block.


>  It is much simpler to just check the final bytes of the message instead
> of having to remember the state of the previous frame.
>

[hybi] [permessage-deflate] Compressing fragmente… Takeshi Yoshino
Re: [hybi] [permessage-deflate] Compressing fragm… Arman Djusupov
Re: [hybi] [permessage-deflate] Compressing fragm… Takeshi Yoshino