[hybi] Bidirectional protocols over port 80 - making it work everywhere

[Jamie Lokier <jamie at shareable.org>]

Jamie Lokier wrote:
> You can't expect Upgrade to work on port 80 on a substantial portion
> of the internet (possibly growing, due to 3G mobile phones), and a
> good number of places don't allow access to general internet services
> via any port other than 80.
> 
> BWTP or anything else "emulated" over HTTP is a way to sort that out.
> 
> Of course then it is just another protocol over Comet (etc.), with
> sucky performance.
> 
> So something of great value would be a way to Upgrade to the proper
> protocol over the real internet in a way works despite transparent
> proxies on port 80, to the extent that it's possible, and work
> properly when it is not possible to upgrade.

By the way, I've been working on a solution to this exact problem.
There might actually be one which doesn't involve trying to guess
which proxies there are in the way.

I don't have a full solution yet, so think about how it might work and
help me out.

It's similar to Greg's idea of BWTP proxy involvement.

We use HTTP proxy involvement.

We start by "emulating" a bidirectional protocol over HTTP,
Comet-style, but include a specific header which means "if you
recognise this header, you may 'lift' the embedded bidirectional
protocol out of it's Comet-style framing and handle it directly in
it's native framing".

Proxies which don't recognise the header will process the HTTP
normally, and it will just work with Comet-style behaviour.

Proxies which do recognise the header will intercept the embedded
protocol and convert it to something which works better, over a single
socket upstream (eventually).

If the Comet-style protocol consisted of a "long POST", with an
infinitely long request and infinitely long response, there would be
no problem.  The HTTP part would just be an initial header.

However, you can't do that: many HTTP implementations don't permit
simultaneous request and response data (even though it's allowed and
some do permit it - I've tried).  So, in general you have two
connections with Comet.  One's the long-GET/POST for server-initiated
messages, and one is for client-initiated messages, roughly speaking.
(You may have tricks like alternating between them for keepalive).

The tricky bit will be finding a way for a participating proxy to lift
the bidirectional protocol out of those two connections, into a single
connection.

Fun, but it might just work over the whole internet.

Especially if we follow the guidelines on good Comet implementation
for the HTTP framing.

I'm going to try implementing that and see where it leads.

-- Jamie