Justin Erenkrantz wrote:
> On Mon, Feb 1, 2010 at 7:56 PM, Maciej Stachowiak <mjs@apple.com> wrote:
> > If either of these concerns makes half-close-only impractical, then I think Jamie's suggestion is the best: allow any of half-close, explicit close frame, or both.
> 
> Given how hard it is to get useful (or consistent) TCP error
> information from most OS stacks, I think having an explicit close
> frame in the common case is going to be well worth it in the long run.

I agree.  I didn't think of it until a really long hard think.

And then I realised that an explicit close message(*) is essential if
you want to know that the application cleanly closed the socket, versus:

    - Application crash looks the same as shutdown() to the receiver.

    - OS killing the application due to, e.g., memory pressure looks
      the same as shutdown() to the receiver.

    - Intermediate TCP relay (there are many types; see my other post)
      killing the connection looks the same as shutdown() to the receiver.

    - Virtual machine containing the application is hard-rebooted, looks
      the same as shutdown() to the receiver with some VM types.

    - HTTP proxy forwarding over CONNECT, killing the connection looks
      the same as shutdown() to the receiver.

In a nutshell, there are *two* distinct uses for signalling clean shutdown:

    1. To avoid the TCP reset hazard.

    2. To signal that it was a clean shutdown, so there is no protocol
       uncertainty about what has been processed / what can be retried
       on another connection.(**)(***)

shutdown() and lingering close does deal with 1, but it isn't reliable
for 2 because of all the other things which can look like shutdown().

So I agree with Justin and others, that an explicit "clean close"
message is more useful and we'll be glad of it in the long run,
compared with just shutdown().

Oh yes, and there is another reason not to rely on shutdown():

    - Not all TCP relays will forward a half-close.  I know
      for a fact that some I've written (long ago ;-) didn't have a
      shutdown() call anywhere.

    - I would not be surprised to find some HTTP proxy over CONNECT
      doesn't forward a half-close reliably, or at all.

Here are some more technical issues we should consider:

    a. Because the sender must have a timeout after sending the close
       message(***), for clean shutdown from *both* sides, it's
       desirable that the receiver finishes the connection quickly,
       rather than writing "all queued messages", which can be any
       amount of data and take any amount of time.  The receiver
       should detect the close message as soon as possible, and finish
       sending its current message only, or even abort the current
       message (if we used byte 0xfe for that purpose, say).
       Detecting the close message is made quicker by following it
       with FIN (EOF), because processing code will see the EOF before
       it parses incoming queued data.  (OOB would expedite it further
       but I don't see it being popular :-)

    b. It is possible for *both* sides to decide to close around the
       same time, independently.  E.g. if they both implement an idle
       timeout or are driven by resource limits.  I'd appreciate if
       someone double checks this works out ok.

    c. This "lingering close" (of any kind) takes time - up to the
       timeout, which might be 30 seconds, 2 minutes or whatever makes
       sense.  (E.g. see Apache).  What should applications do when
       they want to terminate and are in this state?  E.g. a web
       browser sends a close message, and then the user quits the
       browser.  Does it abruptly close(), or is it important to leave
       a background process handling the lingering close?

    d. Sending a close message followed by shutdown() to half-close.
       Is that *actually* reliable over HTTP proxies when using
       CONNECT?  I wonder if some proxies will close() the next hop,
       thereby breaking orderly close if shutdown() is used.  Perhaps
       Anne's idea to send the close message _without_ using
       shutdown() is a good idea after all.  Does anyone have data?

Footnotes

(*)    Let's not call them frames, to avoid confusion with TCP
       frames/packets :-)

(**)   ACK messages alone cannot do this, because without a clean
       shutdown message, you can't tell if an ACK was lost.

(***)  It also signals "don't bother sending more requests because I
       won't be sending any more responses on this connection".
       Unless you get into the dubious area of reviving half-dead
       connections.

(****) It's related to keepalive.  You can't use a keepalive to avoid
       needing the timeout; that'd just be an equivalent timeout.

And a couple of notes about TCP stacks:

     - Good TCP stacks (e.g. Linux (MSG_MORE/TCP_CORK), FreeBSD
       (TCP_NOPUSH)) can be made to combine the close message and FIN
       in the same packet.  So the close message is essentially free
       on them anyway.

     - Many TCP stacks actually do implement "lingering close" in the
       OS.  See SO_LINGER.  But it is often not correctly implemented
       or not useful, which is why Apache et al must do it themselves.

-- Jamie