This will summarize all my input and conclusion on the frame size header
issue. This corrects numerous errors and incoherence in my prior posts on
this issue.  Please read, it is worth it.

> On 23.08.2010 01:55, John Tamplin wrote:
>> ...
>> Feel free to add new ideas if I missed one, and vote for (the checkbox)
>> any idea you would be happy with, vote against (the X) any idea you
>> ...
>
> The IETF doesn't vote
> (<http://www.ietf.org/tao.html#getting.things.done>).
>
> But call it an "informal poll", then this could be useful in finding out
> where we are.

Gentlepeople, let me see if I can help move us towards concensus. I have
only been trying to help on this frame header issue for about 2 days, so
please give me reasonable chance/patience, before judging me.

Technical points follow the "Justication" section.

Justification
=============

Some have publicly expressed negatively about the delay in reaching
concensus, two people told me privately that I was adding too much verbage
and delaying the process.  I remind them:

http://www.ietf.org/tao.html#getting.things.done

"'rough consensus', meaning that a very large majority of those who care
must agree"

"strongly held objections must be debated until most people are satisfied
that these objections are wrong"

"The lack of formal voting has caused some very long delays for some
proposals, but most IETF participants who have witnessed rough consensus
after acrimonious debates feel that the delays often result in better
protocols."

We do not yet have 'rough concensus' evidenced by the distribution of the
informal poll:

https://www.google.com/moderator/#15/e=2809f&t=2809f.40

It is my understanding that this is caused by a lack of complete
understanding of the issues, not by a valid difference of opinion. In
fact, most disagreement in life derives from incomplete communication.
When we think someone else is 'wrong', 99% of the time it is merely the
opposing sides do not yet understand the logic of each other. This is why
I continue trying to explain my logic, because I think others do not yet
understand. Some on the list may have dismissed my points as being
'irrelevant' to their opinion, but I think this is because they do not yet
fully understand my points. So with that belief in mind, I will now try to
explain my logic again more carefully. Because unless someone can point
out a flaw in my logic, I think I have considered ALL the concerns
expressed by the list and proposed the ONLY option which meets all of them
with the best quantitative fit. Unfortunately, this logic is convoluted,
so I hope I can explain it non-ambiguiously. Based on the informal poll,
it seems I either failed on my prior attempts, or someone failed to refute
my logic. The above paragraph is long-winded, and it is to make sure
people don't misconstrue my objectives and method of communication here. I
am not arguing for my original choice, Option #2, I am now arguing for
this list's not yet clearly articulated choice. If anyone can put a hole
in my logic below, then it will help us reach mutual understanding, so
that the list's choice will converge.


Technical Points
================

1) Originally I thought "Option #2 - 7/16/63-bit" was fully optimized:

http://www.ietf.org/mail-archive/web/hybi/current/msg03412.html

because it provided the least possible header overhead in all reasonably
expected scenarios of frame size, known to me at the time, and it removed
the discontinuity in header overhead at 127+ frame size, as compared to
the slightly[1] less optimized "Option #1 - 7/63-bit".


2) However, feedback on the list was that a (negotiated at connection
handshake) maximum frame size was necessary for at least 2 orthogonal
reasons. Some want to reject larger frames as a DoS and/or general
security filter (afaics a _STRICT_ maximum is better here too):

http://www.ietf.org/mail-archive/web/hybi/current/msg03458.html

I will make multiple orthogonal points below, invalidating one of them may
or may not invalidate all.

Others want to eliminate as much possible CPU overhead in the
pre-processing the frame header (even for single-core implementations),
and orthogonally is that UNAVOIDABLY MUST precede passing the processing
to another core (for multi-core):

http://www.ietf.org/mail-archive/web/hybi/current/msg03453.html

I explained (with some prior errors corrected herein) that an unreasonable
outcome of not minimizing all (not just any conditionals) the CPU cost of
pre-processing the frame header, is that the network bandwidth could be
wasted. This is always true and the waste is proportion to the ratio of
the CPU time (including multi-core signaling+shunting overhead time) to
process the header, divided by the time to send the entire frame over the
network (could this be eliminated with NIC buffering incoming frames in
parallel with DMA?). Thus the waste increases as frame size decreases.

And I explained that to eliminate the relatively high (as compared to
other header pre-processing) CPU cost of conditional branches (aka
comparisons), a _STRICT_ maximum frame size is required:

http://www.ietf.org/mail-archive/web/hybi/current/msg03510.html
http://www.ietf.org/mail-archive/web/hybi/current/msg03517.html
http://www.ietf.org/mail-archive/web/hybi/current/msg03511.html

John Tamplin questioned whether the conditionals CPU cost was significant,
but I think he was arguing the point of whether the speed of CPU is
increasing faster than the network bandwidth, and I now realize and make
the point above that the waste is any CPU cost, regardless of relative
rate of improvement in network versus CPU speed.

John Tamplin made the point that for intermediaries, they will ignore the
WebSocket frames and just process the TCP packets. I am responding now, to
say that isn't true always, for example refragmentation.

Also, there is a related orthogonal point which is convoluted to explain.
The ability to use all the multi-cores (if not due to a bottleneck at the
network bandwidth plus the aforementioned waste) can be limited by the
average (and thus perhaps the maximum) frame size. The limit can be if avg
is too small causing signalling+shunting load to cause performance to
decrease if more cores are employed. Tangentially, the limit can be if avg
is too high, requiring intra-frame (splitting) parallelization, which
depends on the intra-core signalling cost due to complexity of the
state-machine for the frame data processing (extension protocols, any
application on top, etc).

Thus I explained that due to its internal economics, an implementation
might be optimized with its maximum size larger than 126 if there were no
costs of doing so, but perhaps not if the cost of processing conditionals
combined with the fact that 98% of frames are less than 126, justified
setting a _STRICT_ maximum size of 126, if that eliminated the
conditionals. Remember above I pointed out that the _STRICT_ maximum size
was a benefit when coupled with frame size that uses conditionals, because
it allows the implementation to control its costs and maximize its
economics. But the problem is that if we choose an option which can only
do 0 - 126 size with conditionals eliminated by the _STRICT_ maximum size,
then the (implementation self-optimization) optimization is a disaster,
because it means a large frame has to be fragmented by the sender to 126
byte frames! However, the receiving implementation probably did not have
any benefit of limiting the maximum frame size to 126, except for the
minimization of the conditionals cost.

So ladies and gentlemen, that is why I proposed "Option 15/63-bit" (I
formerly called it "Option 1/15-bit"). Do you understand now?

The 15-bit (32767 bytes) size when conditionals are eliminated by
optimization of the _STRICT_ maximum size, is more than sufficient to
prevent the above disaster scenario.  Also it has negligible (2%? [1])
costs relative to the prior best option, "Option #2 - 7/16/63-bit":

http://www.ietf.org/mail-archive/web/hybi/current/msg03482.html
(see near bottom of the post, extra byte for 0-126, save byte 126-32766)

So clearly "Option 15/63-bit" is superior to "Option #2 - 7/16/63-bit".


3) After I proposed "Option 15/63-bit", a new option was proposed "Option
8/16/32/64-bit". I can't yet decide between these two options, so I voted
for both of them in the informal poll (and voted against all other
options, except "Option #2 - 7/16/63-bit"). The "Option 8/16/32/64-bit"
gives more reversed bits, but it adds a CPU cost of loading the LenLen,
loading a value of 1 and left shifting by LenLen. The "Option
8/16/32/64-bit" does not have any explicit conditionals that can be
eliminated with a _STRICT_ maximum size, but a _STRICT_ maximum size is
still required, else there will be an UNNECESSARY conditional to test if
frame size is greater than maximum size (for those implementations that
gain from such). Thus I am reasonably confident that "Option
8/16/32/64-bit" is superior to "Option 15/63-bit", but I am waiting to
hear any other feedback on the list about this comparison.

Let us contemplate how the _STRICT_ maximum size impacts the 63-bit
sendFile() optimization?  An implementation will choose to minimize its
costs, so we have to consider the selfish economics of what ever we
choose. Perhaps with "Option 8/16/32/64-bit" we can tolerate the one
conditional and make maximum sizes a hint, not _STRICT_?


[1] I write that only 'slightly', because John Tamplin had provided a
distribution of frame sizes from some real world applications, which
claims that only 2% of frame sizes will be greater than 126. I write
'claims', because statistics of the present, do not always predict the
future (aka Murphy's Law). One thing I just realized now, is I forgot to
check/ask John the mean and std deviation of frame size? To better
quantify the impact of adding 1 byte to the header as the other options
do.