[core] Preliminary Summary from CoAP#4 plugtest in London

Carsten Bormann <cabo@tzi.org> Sun, 09 March 2014 18:17 UTC

Content-Type: text/plain; charset="windows-1252"
Mime-Version: 1.0 (Mac OS X Mail 7.2 \(1874\))
From: Carsten Bormann <cabo@tzi.org>
Date: Sun, 09 Mar 2014 18:16:45 +0000
Content-Transfer-Encoding: quoted-printable
Message-Id: <07326969-B456-4E89-9417-5C07BDE8B3CA@tzi.org>
To: "core@ietf.org WG" <core@ietf.org>
Archived-At: http://mailarchive.ietf.org/arch/msg/core/p539wP2H0PNmekHFnlB2bJfXXZQ
Subject: [core] Preliminary Summary from CoAP#4 plugtest in London
Precedence: list

Here is an impression from the CoAP#4 plugtest that is ending today:

For the CoAP test suite, overall interoperability was 98.5 %, with
1346 interoperable out of 1367 tests completed.

The core-coap tests were interoperable at 99.3 % this time.

More spectacularly, the tests for Observe achieved 99.4 %.  This
result really reinforces shipping this specification to the IESG now.
I'm even more impressed that 48 hours after the Friday decision on
Observation cancellation, seven pairings (unfortunately, not everyone
had the local access to source code needed to do this) completed
testing the outcome, with 100 % interoperability.

Link-format fell back a little at 96.5 % (with problems mostly in
complex queries), and Block is at 95.1 %.  A remaining hotspot is
Block1 POST, which for over half of the pairings both sides had
implemented, but only achieved 70 % interoperability within this
group.  On the other hand, while only seven pairings had both
implemented Block1/Block2 POST, these achieved 100 % interoperability.

Thanks go to ETSI for running and to OMA for supporting the event, as
well as to all the participants for bringing lots of implementations
to the event.  After the IETF (and, for some of us, STRINT), this was
quite a marathon.

For those who just can’t stop testing (or have missed the event):

I have deployed the updates for CoAP#4 to http://coap.me, so you can
use coap://coap.me as a test CoAP server and use the web interface of
coap.me as a test CoAP client against your server.

The test descriptions for CoAP and DTLS can be found at
https://github.com/cabo/td-coap4 — these are not the official ETSI
document (which will be available soon) but the “source code” for
that.

Also don't miss testing with http://vs0.inf.ethz.ch, which complements
coap.me (e.g., it has an Observe implementation in the server, which
coap.me hasn't got around to).

If you want to arrange some timeslot for attended remote testing,
please don’t hesitate to ask me.

Grüße, Carsten

[core] Preliminary Summary from CoAP#4 plugtest i… Carsten Bormann