A proposal for a scientific approach to this question [was Re: I'm struggling with 2219 language again]

Marc Petit-Huguenin <petithug@acm.org> Sat, 05 January 2013 19:13 UTC

Return-Path: <petithug@acm.org>
X-Original-To: ietf@ietfa.amsl.com
Delivered-To: ietf@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 69A2E21F8554 for <ietf@ietfa.amsl.com>; Sat, 5 Jan 2013 11:13:56 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -101.308
X-Spam-Level:
X-Spam-Status: No, score=-101.308 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, MISSING_HEADERS=1.292, NO_RELAYS=-0.001, USER_IN_WHITELIST=-100]
Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id BT4TAUW3Cn-l for <ietf@ietfa.amsl.com>; Sat, 5 Jan 2013 11:13:55 -0800 (PST)
Received: from implementers.org (implementers.org [IPv6:2604:3400:dc1:41:216:3eff:fe5b:8240]) by ietfa.amsl.com (Postfix) with ESMTP id 279F221F8546 for <ietf@ietf.org>; Sat, 5 Jan 2013 11:13:55 -0800 (PST)
Received: from [IPv6:2601:9:4b80:32:5de6:df9d:4864:4f45] (unknown [IPv6:2601:9:4b80:32:5de6:df9d:4864:4f45]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (Client CN "Marc Petit-Huguenin", Issuer "implementers.org" (verified OK)) by implementers.org (Postfix) with ESMTPS id 5A0262049D for <ietf@ietf.org>; Sat, 5 Jan 2013 19:13:53 +0000 (UTC)
Message-ID: <50E87B7B.5070605@acm.org>
Date: Sat, 05 Jan 2013 11:14:03 -0800
From: Marc Petit-Huguenin <petithug@acm.org>
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0.11) Gecko/20121122 Icedove/10.0.11
MIME-Version: 1.0
CC: ietf@ietf.org
Subject: A proposal for a scientific approach to this question [was Re: I'm struggling with 2219 language again]
References: <7ED55FF1-3E1A-4DF7-918E-07790517B848@softarmor.com>
In-Reply-To: <7ED55FF1-3E1A-4DF7-918E-07790517B848@softarmor.com>
X-Enigmail-Version: 1.4.1
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
X-BeenThere: ietf@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: IETF-Discussion <ietf.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/ietf>, <mailto:ietf-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/ietf>
List-Post: <mailto:ietf@ietf.org>
List-Help: <mailto:ietf-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ietf>, <mailto:ietf-request@ietf.org?subject=subscribe>
X-List-Received-Date: Sat, 05 Jan 2013 19:13:56 -0000

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

I read the responses so far, and what can be said today is that there is 2
philosophies, with supporters in both camps.  The goal of the IETF is to make
the Internet work better, and I do believe that RFC 2119 is one of the
fundamental tool to reach this goal, but having two ways to use it does not
help this goal.

One way to find out would be to measure which philosophy results in the best
implementations.  Let's say that we can associate with each Standard Track RFC
one of these two philosophy.  If we had statistics on implementations then it
would be a simple matter of counting which one produce the less
interoperability problems, security issues and congestion problems (is there
other criteria?).  But as far as I know, there is no such data available -
maybe we should start collecting these, but that does not help for our current
problem.

Another way to look at it would be to run the following experiment:

1. Someone design a new protocol, something simple but not obvious, and write
in a formal language and keep it secret.

2. The same protocol is rewritten in RFC language but in two different
variants according to the two philosophies.  These also are kept secret.

3. The two variants are distributed randomly to a set of volunteer
implementers, who all implement the spec they received the best they can and
submit the result back, keeping their implementation secret.

4.  A test harness is written from the formal description, and all
implementations are run against each other, collecting stats related to the
criteria listed above (some criterion may be tricky to automatically assess,
we'll see).

5. Results are published, together with the protocol in formal form, the
specs, the results and the recommendation for one or the other philosophy.


That could be an interesting research project, and could even find some
funding from interested parties.


On 01/03/2013 09:15 PM, Dean Willis wrote:
> 
> I've always held to the idea that RFC 2119 language is for defining levels
> of compliance to requirements, and is best used very sparingly (as
> recommended in RFC 2119 itself). To me, RFC 2119 language doesn't make
> behavior normative -- rather, it describes the implications of doing
> something different than the defined behavior, from "will break the
> protocol if you change it" to "we have reason to think that there might be
> a reason we don't want to describe here that might influence you not to do
> this" to "here are some reasons that would cause you to do something
> different" and on to "doing something different might offend the
> sensibilities of the protocol author, but probably won't hurt anything
> else."
> 
> But I'm ghost-editing a document right now whose gen-art review suggested 
> replacing the vast majority of "is" "does" and "are" prose with MUST. The 
> comments seem to indicate that protocol-defining text not using RFC 2119 
> language (specifically MUST) is "not normative".
> 
> This makes me cringe. But my co-editor likes it a lot. And I see smart
> people like Ole also echoing the though that RFC 2119 language is what
> makes text normative.
> 
> For example, the protocol under discussion uses TLS or DTLS for a plethora
> of security reasons. So, every time the draft discusses sending a response
> to a request, we would say "the node MUST send a response, and this
> response MUST be constructed by (insert some concatenation procedure here)
> and MUST be transmitted using TLS or DTLS.
> 
> Or, a more specific example:
> 
> For the text:
> 
> In order to originate a message to a given Node-ID or Resource-ID, a node 
> constructs an appropriate destination list.
> 
> 
> The Gen-ART comment here is: -First sentence: "a node constructs" -> "a
> node MUST construct"
> 
> 
> We'll literally end up with hundreds of RFC 2119 invocations (mostly MUST)
> in a protocol specification.
> 
> Is this a good or bad thing? My co-editor and I disagree -- he likes 
> formalization of the description language, and I like the English prose.
> But it raises process questions for the IETF as a whole:
> 
> Are we deliberately evolving our language to use RFC 2119 terms as the 
> principle verbs  of a formal specification language?
> 
> Either way, I'd like to see some consensus. Because my head is throbbing
> and I want to know if it MUST hurt, SHOULD hurst, or just hurts. But I
> MUST proceed in accordance with consensus, because to do otherwise would
> undermine the clarity of our entire specification family.
> 

- -- 
Marc Petit-Huguenin
Email: marc@petit-huguenin.org
Blog: http://blog.marc.petit-huguenin.org
Profile: http://www.linkedin.com/in/petithug
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iQIcBAEBCAAGBQJQ6Ht3AAoJECnERZXWan7ETHwP/1MwWKyX4ZoTqS2AZr5VdCwx
jGO/0+tbHppplfippPlJRR6cV5rfrrtkKp9j3Xbr477Jeuaaadjv3y0CfkGF+DUb
fDhcB/GQLiN1oC6s3cjiib46Rnd18Ela6xUAZleiLjKKoo0TJKfQ8oAt3tYonokK
onb95NAsF0FsbiqBzoUi23aEf/SFoKOg3a67DAt5XmntnNh5K6jVOmT4GFYtF3LB
vW6d1x0hI0INUSXzjypD+MaqzCgHvdxAjqx44qlrFjFYz7GLcRAR3Z5ynOCCfeBi
fM4xxGhytTYolrXdOQK1BgtUIewdCHMqPFZjclB2DiEITkUzxRrGKI5MizQEModB
8vwmJQJ0E98veMvBP5ce9eKPZP0gHMxEWMu2zgGulb2mLVxyfSfBIy2dIuqpHsUM
dWuvLzx1HJouZ4sFfCGMarpLvoYwYu1grL+oaJiPXd1TI26BCyI703gdQE1dBRhK
XrfIEgmP1VG1QhhBA6otLQlWfB0IGG2c80y0KOZ4x9rQ8Wn+Cxz4vQg6CvHPys+z
ClvFH4r/v3CpavSpugcD7sJAssfp9wbe9Jjw1pCPiXgs0fN+yQtQRIVDCSCQHNJ4
iWFtwExIDQyh/lKmJNlf8P3qBSmhzqvAG1B9bAZY9JroJoqY84kkgWKHRodAQ/HM
DqC0PVCig1bKYwZpT1Ov
=oOlZ
-----END PGP SIGNATURE-----