[core] Preliminary Summary from CoAP#3 plug test in Las Vegas

Carsten Bormann <cabo@tzi.org> Fri, 22 November 2013 20:13 UTC

Return-Path: <cabo@tzi.org>
X-Original-To: core@ietfa.amsl.com
Delivered-To: core@ietfa.amsl.com
Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 307191AE19E for <core@ietfa.amsl.com>; Fri, 22 Nov 2013 12:13:24 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: 1.149
X-Spam-Level: *
X-Spam-Status: No, score=1.149 tagged_above=-999 required=5 tests=[BAYES_50=0.8, HELO_EQ_DE=0.35, SPF_HELO_PASS=-0.001] autolearn=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Dn1ktE2mBs4D for <core@ietfa.amsl.com>; Fri, 22 Nov 2013 12:13:22 -0800 (PST)
Received: from informatik.uni-bremen.de (mailhost.informatik.uni-bremen.de [IPv6:2001:638:708:30c9::12]) by ietfa.amsl.com (Postfix) with ESMTP id 1F44A1AE1E8 for <core@ietf.org>; Fri, 22 Nov 2013 12:13:20 -0800 (PST)
X-Virus-Scanned: amavisd-new at informatik.uni-bremen.de
Received: from smtp-fb3.informatik.uni-bremen.de (smtp-fb3.informatik.uni-bremen.de [134.102.224.120]) by informatik.uni-bremen.de (8.14.5/8.14.5) with ESMTP id rAMKD6F4027208; Fri, 22 Nov 2013 21:13:06 +0100 (CET)
Received: from tzi-v4.plugtests.net (wsip-24-234-25-210.lv.lv.cox.net [24.234.25.210]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by smtp-fb3.informatik.uni-bremen.de (Postfix) with ESMTPSA id 5453B4CB; Fri, 22 Nov 2013 21:13:04 +0100 (CET)
Content-Type: text/plain; charset="iso-8859-1"
Mime-Version: 1.0 (Mac OS X Mail 7.0 \(1822\))
From: Carsten Bormann <cabo@tzi.org>
Date: Fri, 22 Nov 2013 12:13:01 -0800
Content-Transfer-Encoding: quoted-printable
Message-Id: <D3302F9F-4858-4A9F-9F9C-3BF0DBB4BA2A@tzi.org>
To: core <core@ietf.org>
X-Mailer: Apple Mail (2.1822)
Cc: IOT_PLUGTESTS@LIST.ETSI.ORG
Subject: [core] Preliminary Summary from CoAP#3 plug test in Las Vegas
X-BeenThere: core@ietf.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: "Constrained RESTful Environments \(CoRE\) Working Group list" <core.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/core>, <mailto:core-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/core/>
List-Post: <mailto:core@ietf.org>
List-Help: <mailto:core-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/core>, <mailto:core-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 22 Nov 2013 20:13:24 -0000

While a couple of final tests are still ongoing, here is an impression from the CoAP#3 plugtest that is ending today:

Overall interoperability was 94.1 % at the time of writing, with 809 interoperable out of 860 tests completed.
Note that this figure includes a number of completely new implementations as well as a number of newer specifications, including OMA LWM2M.

The core-coap tests were interoperable at 95.9 %, link-format was even better at 98.4 %.
Remaining hotspots (~ 80 %) are ETags/Validation and separate responses in a lossy context.
Block and Observe are at 84.9 and 84.8 percent, respectively, reflecting that some of the implementations are rather new and some recent changes may not be fully implemented.

OMA LWM2M was at a stunning 97.4 % given that the 1.0 spec draft is not even a month old, but that result reflects a smaller sample size and an early test spec with limited coverage.
We found a number of issues where clarifications are desirable in the LWM2M spec draft.

Finally, we only completed very few DTLS tests, at 66.7 % interoperability.
The problems observed were generally with the retransmission mechanism.
We will certainly expand this segment in the next events.

Thanks go to ETSI for running and to OMA for hosting the event, as well as to all the participants for bringing lots of implementations to the event.

If you missed this one: The next CoAP/DTLS/LWM2M interop is not far away.
Discussions on the details are ongoing, but we might even have one in London colocated with IETF89.

Grüße, Carsten