Subject: [Anima-signaling] Discuss: Using CBOR for anima signaling

From: Toerless Eckert <[email-masked]>
Date: Thu, 21 May 2015 00:56:41 -0700

I would like to suggest considering CBOR as the encoding/"presentation"?
layer for new protocol/signaling we introduce in ANIMA:

Traditionally, protocols in eg: routing area in IETF did always
reinvent their own binary encodings, often with their own TLV formats,
requiring re-implementing coding/decoding layers for each protocols
as well as troubleshooting tool chains (eg: wireshark modules). And
a lot of time is lost re-discussing details of those protocol layers.

Application folks in the IETF did evolve from HTTP->XML->JSON with a
common "encoding" layer. Lately it became clar that when reusing
these application layer mechanisms in the IoT space, it is necessary
to support more compact encoding with otherwise the same benefits as
JSON, but without the elements of JSON that where determined to be too
difficult or not working. This is what ultimately became CBOR (RFC7049).

IAB folks (Joe Hildebrand) and others will also start to propose the
idea of using CBOR for future work also in the routing area in Prague
(routing area meeting). 

So, would like to see this considered by anima signaling, eg:
If it's choosing not to use CBOR, why not.

Another aspect is binary vs. string "TLV" encoding.
CBOR does allow both binary-numeric and string-name TLVs.

I personally would prefer string-name TLVs because binary TLV means
you need a parallel TLV-schemata distribution mechanism for diagnostics
and policy control operations.

Cheers
   Toerless

---

From: Michael Richardson <[email-masked]>
Date: Thu, 21 May 2015 11:51:22 -0400

> layer for new protocol/signaling we introduce in ANIMA:

+1.

But, let's be clear: we are writing JSON, with encoding in CBOR, not
presenting objects described by CBOR.

--
Michael Richardson <[email-masked]>, Sandelman Software Works
 -= IPv6 IoT consulting =-

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 22 May 2015 08:13:11 +1200

> presenting objects described by CBOR.

Well, I have no answer because I need to study CBOR first.
I remain very concerned about software bloat, however. I am
thinking about low end devices; unless we accept that ASAs
can only run in significantly powerful devices.

     Brian

---

From: Jéferson Campos Nobre <[email-masked]>
Date: Thu, 21 May 2015 18:47:06 -0300

I am not familiar with CBOR either, but after a quick google search I
found that constrained devices are mentioned in CBOR docs.
Besides that, there a new WG (under review) called "CBOR Object
Signing and Encryption"  (cose), what also should be interessant in
terms of the security features expected in anima.
In any case, I need to study CBOR too.
Best.

Jéferson Campos Nobre
PhD Student
Computer Networks Group -  Institute of Informatics
Federal University of Rio Grande do Sul
http://www.inf.ufrgs.br/~jcnobre

On Thu, May 21, 2015 at 5:13 PM, Brian E Carpenter
<[email-masked]> wrote:
> _______________________________________________
> Anima-signaling mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/anima-signaling

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 22 May 2015 16:31:53 +1200

Well, I have a team-leader challenge for Michael R and Toerless,
in two parts. I'm not trying to be clever, just trying to get
a clear grasp of what you are proposing:

1. Please describe in a few paragraphs the advantages and disadvantages
of using a human-readable text-based format for signaling between
autonomic service agents (which are expected to operate entirely
without human intervention).

2. Then, take it for granted for the moment that the protocol
messages and options defined in GDNP draft 04 are correct. Give us
a rough draft of how they would look in JSON. No polishing required;
this is just to put some meat on the bones.

ASAP, because this would be a fairly major change to the draft
and we only have about 6 weeks.

After reminding myself about CBOR, yes, I agree, but the first
question is whether to use a text-based format at all.

Regards
   Brian

---

From: "Liubing (Leo)" <[email-masked]>
Date: Fri, 22 May 2015 09:03:42 +0000

Hi Toerless,

Thanks for sharing the knowledge and your thoughts. I believe it will make the design considerations more comprehensive.
I'm also not familiar with the CBOR encoding, so I have a couple of basic questions, please see inline.

> -----Original Message-----
> From: Anima-signaling [mailto:[email-masked]] On Behalf
> Of Toerless Eckert
> Sent: Thursday, May 21, 2015 3:57 PM
> To: [email-masked]; [email-masked]
> Subject: [Anima-signaling] Discuss: Using CBOR for anima signaling
> 
> Traditionally, protocols in eg: routing area in IETF did always reinvent their
> own binary encodings, often with their own TLV formats, requiring
> re-implementing coding/decoding layers for each protocols as well as
> troubleshooting tool chains (eg: wireshark modules). And a lot of time is lost
> re-discussing details of those protocol layers.
> 
> common "encoding" layer. Lately it became clar that when reusing these
> application layer mechanisms in the IoT space, it is necessary to support
> more compact encoding with otherwise the same benefits as JSON, but
> without the elements of JSON that where determined to be too difficult or
> not working. This is what ultimately became CBOR (RFC7049).
> 
> IAB folks (Joe Hildebrand) and others will also start to propose the idea of
> using CBOR for future work also in the routing area in Prague (routing area
> meeting).
[Bing] For app folks, the HTTP->XML->JSON->CBOR approaching road might be clear. 
But in the routing area, I don't understand why CBOR is suddenly (maybe it's only the my lack of knowledge) on the table?
If there is some reference stuff, I'd be glad to have a learn.

> If it's choosing not to use CBOR, why not.
[Bing] Is the main motivation to make the signaling compatible to the very small devices?

> I personally would prefer string-name TLVs because binary TLV means you
> need a parallel TLV-schemata distribution mechanism for diagnostics and
> policy control operations.
[Bing] The "string TLV encoding" is not "text-based encoding" which we discussed in the DNS-SD thread?

Best regards,
Bing

---

From: Michael Richardson <[email-masked]>
Date: Fri, 22 May 2015 09:45:23 -0400

> without human intervention).

But, that's not what I'm suggesting.
ASN.1 is human readable, (as are SNMP MIBs) but, are transmitted in BER or
DER format.
In contrast, RFC760 is human readable too, but we standardize IPv4 headers.

So, let's go back....
    XML, when simple, is UTF-8 and human readable, but when complex, becomes
         almost unreadable, and very expensive to parse.
    JSON is straight Type-Value pairs, with lengths from encapsulation.
    CBOR is the IoT friendly way to encode JSON.

CBOR is to JSON, what BER is to ASN.1, except that in this case, CBOR
can be mechanically translated into JSON, which means that it can be
trivially used in the current HTML5/Javascript web based processing.

My point is that we aren't going to put CBOR encoding into our documents,
(which might make think that the session ID is always at byte offset 5)
we are going to put JSON objects, and we might give CBOR encodings as
examples.

    > this is just to put some meat on the bones.

I only see -03 in the repo...

{ "message_type" : "G1",
  "waiting" : "1042000",  # JS also seems to prefer miliseconds
  "device_id": "00D8-C76C",
  "locator_ipv6": "fe80::224:21ff:fea7:5e27",
}

of course, it would be CBOR encoded.

(why is there is constant "4" in the packet, btw?)

---

From: Michael Richardson <[email-masked]>
Date: Fri, 22 May 2015 09:57:03 -0400

> But in the routing area, I don't understand why CBOR is suddenly (maybe
    > it's only the my lack of knowledge) on the table?

Because IoT devices already are going to have CBOR encoders/decoders,
and they are the most limited in terms of code space.

    > [Bing] Is the main motivation to make the signaling compatible to the

Yes.

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 23 May 2015 08:10:55 +1200

Hi Michael,

>          almost unreadable, and very expensive to parse.

Yes, that's exactly what traumatised me a few years ago and is the
source of my concern.

> we are going to put JSON objects, and we might give CBOR encodings as

OK, I think I understand how that works. I don't understand yet why it's
better than the traditional approach to a TLV protocol in this case,
since we are not talking about an HTML/JS environment.

I also need to understand how the API would look - would it also
be based on JSON?

> I only see -03 in the repo...

The unpublished -04 is in the Github repo, but for the present discussion,
-03 is fine.

>   "locator_ipv6": "fe80::224:21ff:fea7:5e27",

Yes. As a way of expressing the construct that's perfectly fine.
I see how we could do the whole thing.

> of course, it would be CBOR encoded.

Sure.

> (why is there is constant "4" in the packet, btw?)

That's the TLV length field, defined in the DNCP format that
we have borrowed. (The options are subsequent TLVs, each with
their own length field.) We wouldn't need it in JSON.

Maybe you should be having this same conversation in homenet
about DNCP, which is in WG Last Call with its old-fashioned
TLV format. If we switch to JSON, we will by definition be
independent from DNCP.

Thanks, this was helpful!

    Brian

---

From: Michael Richardson <[email-masked]>
Date: Fri, 22 May 2015 20:04:38 -0400

> since we are not talking about an HTML/JS environment.

The things we are doing would feed into management stations, and you'd be
surprised at how many things people do in JS with things like node.js.
Also how many languages now compile *into* JS...

    > I also need to understand how the API would look - would it also

Max and I went through the API, and I don't know that it matters what's
underneath.   I think talking APIs is too soon.

    > I see how we could do the whole thing.

Good.

    > their own length field.) We wouldn't need it in JSON.

okay, got.

    > TLV format. If we switch to JSON, we will by definition be

True.
I think the TLV format is well specified, and probably I don't have a problem
with it in the end; but I am concerned that we will need a lot more variety
in types going forward.  A lot of vendor experimentation, and custom values.

--
]               Never tell me the odds!                 | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works        | network architect  [
]     [email-masked]  http://www.sandelman.ca/        |   ruby on rails    [

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 23 May 2015 17:11:20 +1200

> underneath.   I think talking APIs is too soon.

It's too soon for anything definitive, but I need a model in my mind
of how "things that an ASA wants to do" map into "things that the
protocol can do".

> in types going forward.  A lot of vendor experimentation, and custom values.

Yes, that's a powerful argument. As the requirements say: "A basic requirement
for the protocol is therefore the ability to represent, discover, synchronize
and negotiate almost any kind of network parameter." I can see how JSON
enables that.

    Brian

---

From: Toerless Eckert <[email-masked]>
Date: Sat, 23 May 2015 07:21:47 -0700

> can only run in significantly powerful devices.

If you just run a single protocol on your system, i am sure
a hand coded binary parser for just this protocol is most lightweight.
As soon as you run two or more, i'd think a common parser/creator
code is more efficient. Thats what CBOR would give you.

Cheers
    Toerless

---

From: Toerless Eckert <[email-masked]>
Date: Sat, 23 May 2015 07:40:17 -0700

> without human intervention).

a) Choice of string TLV as opposed to numeric ones is a secondary
option. The first one should be discuss whether to use a common
data description with CBOR, eg: with binary TLV.

b) At programming level of your autonomic service agent, at a policy
language level and at diagnostics level, you need the textual 
representation. If you use binary encoding, your toolchain for all
of these need to include schemadata to allow text<->binary encoding.
If you want to provide extensibility especially in policy and diagnostics,
you need to figure out how to propagate those schematas arounds. I
haven't seen good solutions for those in embedded enviroments.

> this is just to put some meat on the bones.

Yes, that would definitely be a lot easier for someone
more familiar with JSON/CBOR than myself. Lets me see if we
can find someone ;-)

> and we only have about 6 weeks.

IMHO, the choice of encoding layer would NOT need to be finalized by prague.

IMHO, the higher layer aspects (message exchanges etc.) are initially more
important to decide on. If we agree that we should investigate
the use of a generic encoding layer like CBOR, then we should just
not spend unnecessary cycles on fine-tuning the current manually
defined TLV format because the more time we spend on this the more
its a foregone conclusion to use it.

> question is whether to use a text-based format at all.

1. Figure out the higher layer interactions are as you said right.
2. Figure out if/how we'd use a generic encoding.
3. Assuming 2. = CBOR,
   Decide if/how to decide between binary or string encoding within CBOR.

Cheers
    Toerless

---

From: Toerless Eckert <[email-masked]>
Date: Sun, 24 May 2015 05:58:14 -0700

On Sun, May 24, 2015 at 08:48:38AM +1200, Brian E Carpenter wrote:
> I think that's a risky comparison.

Sure, I was too quick on the keyboard with the comparison ;-)
Pick your data-type safe language of choice.Obviously JavaScript
would be a logical equivalent given how it also caused JSON.

> C is, as far as I'm concerned,
> the worst thing that ever happened in computer science: the language
> of buffer overflows, weak typing, and sloppy practice generally.
> C programmers produce code much more quickly than assembler programmers,
> but it's equally error-prone. We had many better high-level languages
> to choose from in the 1970s, and it's sad that the industry chose C.

C was successfull because it is like assembler - without all the
boring bits.

> Yes, low level protocol designers have usually thought in assembly
> language terms (even after they started coding in C). Assuming for a
> moment that there is no efficiency, penalty, I see a trade-off here:
> with our present traditional binary TLV model, we have well-understood
> packet formats on the wire, but with a CBOR/JSON model we have a legible
> description in the source code, but the bits on the wire are scrambled
> eggs.

Not sure about the scrambled part. If you try tracing whats 
happening on the wire, the TLV model is a hexdump plus protocol
documentation and your brain hard at work (had to do this recently
and i think it was multiple people that overlooked some bad bit
in the encoding causing interop issues). With CBOR its a pre-existing
CBOR parser independent of your protocol. Yes, if you had to brain
disassemble a CBOR hexdump there would be a few more bits to understand
than with TLV, but you'd really never shuold have to do this.

> Question: how do tools like Wireshark deal with CBOR/JSON protocols?

JSON would be the target diagnostic format. If you get CBOR
on the wire, this could be translated to JSON. With string encoded
types in CBOR, this would work directly. With numeric types
you would first need to install a schema with the numeric to string
mappings of the individual types possible in the structure of your
protocol (like a MIB file). 

This is how something like wireshark should/could work.
Haven't checked the actual product against CBOR, but shouldn't be
too difficult to write a CBOR parser in wirescript once. 

> I don't think we're arguing that ASAs cannot use DNS-SD when it meets
> their need for discovery. We're arguing that Anima signaling *also* needs
> a basic discovery mechanism that doesn't have any dependencies and that
> does mesh naturally with the rest of the signaling mechanism.

a) define "no dependencies"
b) Not sure i see "meshing"  as a intrinsically good property.
   "modular composition of components and concepts" is what
   maintains sanity. But then again i've been burned by a protocol
   that was deployed for 10 years without a well defined 
   state machinery.

> That's completely compatible with moving to a JSON/CBOR model, afaik.

Sure.

Cheers
    Toerless

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 25 May 2015 08:04:30 +1200

> a) define "no dependencies"

GDNP can operate (possibly in an insecure mode) when there is nothing else
but a basic TCP/IP stack and nothing has been configured by humans.

>    that was deployed for 10 years without a well defined 

What I mean is that if discovery is performed based on the same
data model as the subsequent operations (i.e. based on objectives,
whether they are TLVs or JSON objects), we are pretty much guaranteed
consistent behaviour. An entity that responds to discovery can be
assumed to respond to negotiation or syncronization too.

When I started drafting the API, it also made think about the implied
state machine inside the GDNP engine. It wouldn't be a luxury to work on
that explicitly.

Thanks, this is a useful conversation.

   Brian

---

From: Michael Richardson <[email-masked]>
Date: Sun, 24 May 2015 16:49:54 -0400

> Question: how do tools like Wireshark deal with CBOR/JSON protocols?

Let me answer that with my [email-masked] hat on.
1) every time we have new binary TLV protocol, we have a high risk of a
   CVE in tcpdump. (all your comments about C are well taken, but it
   seems everyone who tried writing network protocols in Pascal or Modula 2
   failed. Tcpdump is indeed written in C. It sucks.)

2) we do not yet have a CBOR or CoAP decoder in tcpdump. I imagine that we
   will get one soon.  Wireshark has a CoAP decoder. Wireshark has a plugin
   model that permits things to be written in python (I think) as well as C,
   so things show up much soon, and wireshark is very windows friendly.
   XML, HTTP and JSON decoders are implemented with the "print-ascii"
   decoder :-)
   Tcpdump does not assemble TCP flows, Wireshark does, and also has a nice
   way to show traffic in the two directions, which makes the print-ascii
   function almost everything you need for some things that run over TCP,
   or seem to run over TCP.
   (SMTP, FTP, SIP, HTTP, XMPP, many SQL connections, etc...)

3) the lack of a single unified IETF standard TLV format has resulted in
   hundreds and hundreds CVEs.

</HAT>

But, as I said before, what motivates me to look for an "open" format is
the ability of a vendor to easily use "x-cisco-foobar" as a type, no
registration required.  Most attempts to predefine vendor extensions
in TLV formats have failed. Radius and DHCP succeeded in part because they
provided from day one with explicit vendor extension formats. Radius more
so than DHCP here too.

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 25 May 2015 16:58:23 +1200

Michael,
On 25/05/2015 08:49, Michael Richardson wrote:

Thanks for the ongoing education.

Is there some information about the runtime footprint of JSON and CBOR?
I know I'm a dinosaur, but I really want a feel for what sort of impact
this approach would have on implementations.

<snip>

> provided from day one with explicit vendor extension formats. Radius more

And indeed we have tried to do that in GDNP, because it is clearly essential.
We took the approach of using a PEN to guarantee uniqueness, but
something like "ext-<FQDN>-foobar" works too, of course.

   Brian

---

From: "Liubing (Leo)" <[email-masked]>
Date: Mon, 25 May 2015 10:45:20 +0000

Hi Toerless, Michael, and all,

Thanks for your patient explanation and education.
Please let me to combine the following two topics together for discussion:

1. The data model, which is to represent the "objectives" in GDNP.
Toerless, Brian and I had some off-list discussion of using DNS-SD data model, so that we might re-use some existing infrastructure. 
But we all believed the DNS-SD should not be the only data model. In an ideal way, the GDNP could support arbitrary data models, including potential newly defined GDNP data models.
I'd like to hear you others' thoughts.
2. The encoding model, which is to encode "Messages" and "Options" in GDNP.
We are now discussing using CBOR to replace current TLV encoding.

My questions/comments are:

a) If I understand correctly, the data model and encoding model should be de-coupled? In other words, this CBOR discussion doesn't bother the DNS-SD discussion, right?

b) Say if we use DNS-SD&CBOR, the GDNP packet will like this:
|TCP/UDP Header | CBOR Encoded GDNP messages | CBOR Encoded GDNP Option|
If the GDNP Option is an " Objective Option ", then should it be:
|TCP/UDP Header | CBOR Encoded GDNP messages | CBOR Encoded DNS-SD Data| ?
Can CBOR simply encapsulate the DNS-SD data, or it needs to translate the DNS-SD data into JSON?

c) In theory, I think JASON/CBOR-based encoding is more flexible than the traditional TLV. But CBOR is quite new and hasn't been solidly proved by practice, is it safe for Anima to rely on it?

Best regards,
Bing

> important to decide on. If we agree that we should investigate the use of a
> generic encoding layer like CBOR, then we should just not spend unnecessary
> cycles on fine-tuning the current manually defined TLV format because the
> more time we spend on this the more its a foregone conclusion to use it.
>

---

From: Michael Richardson <[email-masked]>
Date: Mon, 25 May 2015 10:24:06 -0400

> this approach would have on implementations.

I'm sorry, I don't have any info; I'm just at this point, a "fanboy"
for CBOR.  JSON parsers vary in size.  I've some C libraries to good success
in a recent project, but while it was a router appliance, it wasn't
constrained.

    >> in TLV formats have failed. Radius and DHCP succeeded in part because
    >> they provided from day one with explicit vendor extension
    >>formats. Radius more so than DHCP here too.

---

From: Toerless Eckert <[email-masked]>
Date: Mon, 25 May 2015 07:33:24 -0700

http://cbor.io/impls.html

-- 
---
Toerless Eckert, [email-masked]

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 26 May 2015 07:56:42 +1200

OK...
"A C implementation for highly constrained nodes, which achieves a full CBOR decoder in 880 bytes of ARM code (and now also
includes an encoder), has recently become available."
I like this. That still leaves the same question open for JSON processing.

Regards
   Brian

---

From: Toerless Eckert <[email-masked]>
Date: Mon, 25 May 2015 13:02:32 -0700

I did not suggest to use JSON on the wire.

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 26 May 2015 08:12:38 +1200

> I did not suggest to use JSON on the wire.

Now I'm confused. If it isn't JSON, what am I representing in
a CBOR-encoded message?

   Brian

---

From: Toerless Eckert <[email-masked]>
Date: Mon, 25 May 2015 14:00:44 -0700

> Now I'm confused. If it isn't JSON, what am I representing in

CBOR is a compact wire-represtantion of a structured object model.
JSON is a human-readible representation of a superset of the same
object model. Lets just care about the common subject of the
CBOR/JSON object model.

When you write an ASA in JS, and you are programming structured
data, you are effectively writing JSON. So the interpreter or
compiler is parsing your JSON and converting it into run-time
representation. And when you send it on the wire, you'd call some
CBOR.encode to convert the run-time representation to CBOR,
and CBOR.decode back into runtime. So... except for diagnostics,
You would only really have JSON encode in interpreter/compiler.

If you use some other language that also has the ability to
natively represent the CBOR/JSON data model as objects, then
you would not program JSON, but the syntax of that language. 
Perl and Python for example can represent easily JSON data structures
as well, but with their own slightly different syntax, 
so you'd never find JSON encode/decode except for diagnostics.

If you have some assmbler like language like C/C++ without
real structured data (with string/numeric indices), this all
becomes a problem of the poor JSON SDK - and the poor programmer ;-)
But we've already established that C is a bad idea for the future
anyhow.

Cheers
    Toerless

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Mon, 25 May 2015 20:53:45 +0000

Sorry, finally started paying attention to this thread.  I'll top post some general remarks, then see if there's anything else in the thread that needs attention.

A profile of CBOR (removing all of the bits you don't need, like floats, chunked encoding, and most of the tags) is a good fit for places where you want fast processing of structured data with generic code.  

At the app layer, with web APIs, we have found that JSON-shaped data is good at modeling lots of different data, good at being bound into various programming languages, does NOT require schema to syntax-parse, and is adequate at extensibility.  This combination is a good fit for lots of problems.  When combined with a fast binary encoding such as CBOR, without JSON's warts (search RFC 7159 for the word "interoperable" to find some of them) and the ability to handle binary data gracefully, a different set of problems are a good fit.

As to whether Wireshark can parse CBOR, here is a proof point:

https://github.com/iptube/SPUDshark

If I was any kind of lua programmer, there would be a generic CBOR implementation for lua that this plugin would call.  However, you can see that I got the basics done in a relatively small amount of inexpert code.  I would welcome pull requests to tighten it up.  Probably, I would prefer to just display the diagnostic format specified in 7049.

As to whether a profile of CBOR is a good fit for the routing layer, I don't think I can say without gathering data.  I'm working on getting a FPGA implementation of CBOR done so we can look at performance versus a less-structured format.  There are a couple of known security edge cases that would need to be called out and defended against (such as a string of 0x818181... that might blow out a careless implementation's stack).  My intuition is that by the time you created a custom TLV that did everything you wanted and was small enough to be interesting, you would have created a less-optimal CBOR that everyone would have to implement from scratch.

> Yes, if you had to brain disassemble a CBOR hexdump there would 
> be a few more bits to understand than with TLV, but you'd really 
> never shuold have to do this.

Not that many more bits.  On days when I've been hacking CBOR, I can parse it by sight.  Agree that you should never have to do it, though, because it's not as much fun as solving the real problem at hand.

> JSON would be the target diagnostic format.

Well, the augmented diagnostic notation in Section 6 of RFC 7049 (http://tools.ietf.org/html/rfc7049#section-6) is likely exactly what you want.

> Give us a rough draft of how they would look in JSON.

If anyone who knows GDNP wants to sit down with me and bang out some JSON and CBOR representations, I'd be happy to help.

> Is there some information about the runtime footprint of JSON and CBOR?

Carsten's CBOR parser in C is about 880 bytes of ARM code when compiled with -Os.  More is needed for the DOM-like code, if you want that.  JSON code is actually a bit larger, usually, because of the need to deal with string escaping, etc.  However, here is a really small version that is not known to be widely-used or portable: 

https://github.com/quartzjer/js0n/blob/master/src/js0n.c

> Now I'm confused. If it isn't JSON, what am I representing 
> in a CBOR-encoded message?

CBOR is a different syntax than JSON, that is effectively a superset (as long as you didn't want integers > 64bits, which don't interop in JSON anyway).  What you're encoding in CBOR is a data model that you decide on at protocol design time, with a defined extensibility model.

-- 
Joe Hildebrand

---

From: Michael Richardson <[email-masked]>
Date: Mon, 25 May 2015 17:17:45 -0400

> Now I'm confused. If it isn't JSON, what am I representing in

yes, but as you decode the CBOR, you don't *have* to turn it into
ASCII/UTF-8-formatted JSON.  You turn it into an object, such a dictionary.
(Of course, you can turn it into UTF8)

From the abstract point of view, we would write in JSON.

(Someone needs to release Joe's postings.
And [email-masked] is someone's typo.)

---

From: Toerless Eckert <[email-masked]>
Date: Mon, 25 May 2015 14:20:17 -0700

> (Of course, you can turn it into UTF8)

Right.

> From the abstract point of view, we would write in JSON.

abstract -- IETF draft/spec ?!

> (Someone needs to release Joe's postings.

I did try to release it. Not sure it worked. I've asked him to re-send.

> And [email-masked] is someone's typo.)

Yeah, that was me too ;-(. But its a bit late for this thread now on anima...
I guess we'll post a summary later.

Cheers
    toerless

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Mon, 25 May 2015 21:22:58 +0000

>From the abstract point of view, we would write in JSON.

CBOR diagnostic format for examples, and CDDL for specification:

https://tools.ietf.org/html/draft-greevenbosch-appsawg-cbor-cddl-05

-- 
Joe Hildebrand

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Mon, 25 May 2015 22:14:24 +0000

>Can CBOR simply encapsulate the DNS-SD data, or it needs to translate the DNS-SD data into JSON?

There's no need to have multiple CBOR messages.  You can either have a CBOR array that contains multiple bits, or mix the bits together into a map with different keys, depending on how separate everything needs to be kept.

Once you have a parser that can discern structure, try to avoid creating the need for other parsers.

>c) In theory, I think JASON/CBOR-based encoding is more flexible than the traditional TLV. But CBOR is quite new and hasn't been solidly proved by practice, is it safe for Anima to rely on it?

It's more widely-implemented and proven that whatever custom TLV format we would come up with.

-- 
Joe Hildebrand

---

From: "Liubing (Leo)" <[email-masked]>
Date: Tue, 26 May 2015 02:37:15 +0000

Hi Joe,

Thanks for your explanation.

> There's no need to have multiple CBOR messages.  You can either have a
> CBOR array that contains multiple bits, or mix the bits together into a map
> with different keys, depending on how separate everything needs to be kept.
[Bing] Understood. 
The other aspect of my question was that when there is DNS-SD data carried in the GDNP message, the DNS-SD is a nested structure under the GDNP message structure. Then, the DNS-SD data should also be in JASON objects as well as the GDNP message structure, rather than simply encapsulating it in a binary/string form to be a value of the GDNP object. Did I understand it correctly?

> Once you have a parser that can discern structure, try to avoid creating the
[Bing] By reading the other threads, my understanding is that there is a common parser for CBOR encoding -> JASON objects; and there should be another customized parser for the JASON objects as defined in the specific protocol. I guess you were referring to the latter? 

Best regards,
Bing

> >c) In theory, I think JASON/CBOR-based encoding is more flexible than the
> traditional TLV. But CBOR is quite new and hasn't been solidly proved by
> practice, is it safe for Anima to rely on it?
>

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Tue, 26 May 2015 17:22:56 +0000

>The other aspect of my question was that when there is DNS-SD data carried in the GDNP message, the DNS-SD is a nested structure under the GDNP message structure. Then, the DNS-SD data should also b…

There are several options.  You could keep the same encoding as the original DNS-SD, and just send them as a CBOR byte array.  You could also create a new CBOR structure that sent the DNS-SD data in a parsed form. That would likely require either defining a CBOR structure for each RR type, or defining some more generic rule for how to do the translation.  I'm not enough of a DNS expert to know if the latter is possible.  I thought I remembered there being a repository of unpacking rules for the different RR types, but I can't seem to find it at the moment.

>[Bing] By reading the other threads, my understanding is that there is a common parser for CBOR encoding -> JASON objects; 

Usually not.  Think of it this way: when you parse XML, you get a set of objects that you can manipulate using the XML DOM.  That DOM happens to be standardized (which leads to it being a pretty terrible API to work with), but there are also XML libraries that provide an easier API to get at the data you parsed.

When you parse JSON, you get something DOM-like.  There's a set of APIs that allow you to access the type of the parsed thing, or access the 12th item of a parsed array.  This is a different API than the XML DOM.  In JavaScript, it's the same API that you use to access any variable, but in other languages without native support for variant types and maps (such as C), you need to type a little more to access the API.

When you parse CBOR, you get something DOM-like, different, but probably similar to a JSON API.

So instead, I would say, there is a common parser for CBOR which creates native data structures in your programming environment, with an API to access those data structures.

>and there should be another customized parser for the JASON objects as defined in the specific protocol. I guess you were referring to the latter? 

No.  You can parse JSON or CBOR without need for customizing the parser or needing to know the schema beforehand.  This is one of the reasons people tend to like this style of format.

What I'm saying is I don't recommend creating a protocol that looks like this (in JSON, for clarity):

{
  "foo": "bar",
  "struct": "{\"boo\": \"baz\"}"
}

Notice that the internal structure of "struct" is hidden from the parser.  To process it, you would have to get the value of "struct", know somehow ahead of time that it contained JSON, instantiate a new JSON parser, and re-parse.  Instead, you would write:

{
  "foo": "bar",
  "struct": {"boo": "baz"}
}

One pass of the parser would then allow the receiver to retrieve struct.boo.

-- 
Joe Hildebrand

---

From: "Toerless Eckert (eckert)" <[email-masked]>
Date: Tue, 26 May 2015 11:42:27 -0700

Here is an ad-hoc example of how a JSON structure for a
service announcmeent/definition could look like. 
A query for the service could simply leave out the instance sub-object
but might instead have a "param" section when eg: a query is
trying to find a service instance with specific params (prio < val etc. pp).

This whole service object could be part of an overall GDNP JSON
message object whose other element would be message type etc. pp -
whatever we need.

I haven't tried to find other message element JSON definitions,
so i hope Joe will set me straight if this is totally out of line
with best practices ;-)

Cheers
  Toerless

 "service" : { 
     "name" : {
         "dns-sd" : {
             "name" : "_ntp._udp"
         }
         "gdnp" : {
           "name" : "NTP"
         }
     }
     "instances" : [
         // Each instance could be expressed in one or more
         // ways. The following shows a compact representation
         // good enough for us ("gdnp"), and then a representation of
         // the DNS-SD RRs.
         {  "gdnp" : {
            "name"  : "MyStratum1Server.local",
            "l3addrs" : [ "10.10.1.1", "5000::1234" ],
            "l4port"  : 123,
            "ttl"     : 120,
            "params"  : { // we could easy leave out default param/values
                 "priority" : 0,
                 "weight"   : 0,
                 "password" : "foobar",
                 "stratum"  : 1,
            }
            // A DNS-SD service announcemenet is a set of RRs.
            // Represent each RR as a JSON object. Optimize
            // by using a JSON array for multiple RRs for the same DN.
            "dns-rr" : [
                { "dn"    : "_ntp._udp",
                  "type"  : "PTR",
                  "class" : "IN",
                  "ttl"   : 86400,
                  "val"   : "MyStratum1Server._ntp._udp"
                },
                // The most basic service instance parameters are
                // in a SRV RR. Service specific parameters usually in
                // TXT.
                { "dn"    " "MyStratum1Server._ntp._udp",
                  "type"  : "SRV",
                  "class" : "IN",
                  "ttl"   : 86400,
                  "val"   : [ {
                          "type" : "SRV",
                          "val"  : {
                              "priority" : 0,
                              "weight"   : 0,
                              "port"     : 123,
                              "target"   : "MyStratum1Server.local"
                              }
                           }, {
                           "type" : "TXT",
                           "val"  : "passwd=foobar;stratum=1"
                           }
                  ]
                },
                // To actually connect to a service you need
                // network addresses of the instance.
                { "dn"    : "MyStratum1Server.local",
                  "class" : "IN",
                  "ttl"   : 120,
                  "val"   : [ {
                          "type : "A",
                          "val" : "10.10.1.1"
                      }, {
                          "type : "AAAA",
                          "val" : "5000::1234"  // IPv6
                      }
                  ]
                }
            ]
         } }
     ]
 }

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Tue, 26 May 2015 19:55:07 +0000

In case anyone cares, a cleaned-up and linted version of Toerless' JSON is attached, along with a naïve conversion to CBOR.  Here are the sizes, in bytes:

544 gdnp.cbor
     712 gdnp.json

Other than syntax issues, I didn't see any glaring errors in your approach.

-- 
Joe Hildebrand

---

From: "Toerless Eckert (eckert)" <[email-masked]>
Date: Tue, 26 May 2015 13:11:59 -0700

Thanks Joe. 

Brian Carpenter would most certainly like to know whats the
encoding size if we used numeric instead of string "type" fields.

Haven't tried to count it exactly, but probably 200 bytes
less in CPOR if we replaced all types with numeric 8-bit registered
values. Except the exampe "password" and "stratum" types because
those would be service-specific and therefore difficult to register
upfront and string encoding would IMHO be instrumental to extensibility.

Cheers
    Toerless

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 27 May 2015 12:33:02 +1200

> There are several options.  You could keep the same encoding as the original DNS-SD, and just send them as a CBOR byte array.  You could also create a new CBOR structure that sent the DNS-SD data i…

In any case, for the moment I think we should keep two discussions separate
a) should we use JSON/CBOR for our signaling protocol?
b) should we have our own discovery mechanism as well as using DNS-SD
   in cases where it's useful?

In any case, the question an ASA would be asking DNS-SD is "do you
have a locator for a service that supports the following autonomic
objective?" If we switch to expressing objectives in JSON, DNS-SD
would need to deal with that instead of a type code.

   Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 27 May 2015 13:06:23 +1200

> Thanks Joe. 

Ditto.

> encoding size if we used numeric instead of string "type" fields.

The IP address calls out to be numeric and some objectives will
naturally have numeric parameters. The point is, though, that we
will get to the practical limit for using UDP much more quickly
than with a binary protocol.

BTW, GDNP won't discover a service like NTP in the general case.
It will discover an ASA that supports a particular objective.
We might have an objective called NTP, of course. I think this
is only terminology.

> upfront and string encoding would IMHO be instrumental to extensibility.

Let me make a quick guess at the payload size in GDNP as currently
drafted:

A Discovery message would be 16 bytes.

The Response message would be 40 bytes if it contained one
IPv6 address, and longer of course if it contained multiple locators.

This isn't quite as dramatic a difference as might appear, because
in both cases the payload will have a security wrapper.

Regards
   Brian

---

From: "Toerless Eckert (eckert)" <[email-masked]>
Date: Tue, 26 May 2015 18:28:44 -0700

> than with a binary protocol.

Sure. I was just too lazy to look up the right binary types in CBOR
for these.

> We might have an objective called NTP, of course. I think this

If we avoid constraining GDNP to entities called ASAs, it would
be a lot easier to explain how it's reuseable outside beyond a fully
autonomic network.

> in both cases the payload will have a security wrapper.

Sure. Right now we compare apple and oranges because i was suggesting
what i think would be good to ahve in service announcement/discoveries
AND encoding into JSON/CBOR. SHould simply try to give a JSON encoding
of your current proposed data elements and compare that to your binary
message format. 

Cheers
    Toerless

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 27 May 2015 13:44:09 +1200

> of your current proposed data elements and compare that to your binary

Agreed. I'm going to study up on JSON, so give me a little time :-).

   Brian

---

From: "Liubing (Leo)" <[email-masked]>
Date: Wed, 27 May 2015 02:26:21 +0000

Hi Joe,

Many thanks for your patient and detailed answer. It helps a lot!
Replies are inline.

> From: Joe Hildebrand (jhildebr) [mailto:[email-masked]]
> Sent: Wednesday, May 27, 2015 1:23 AM
> To: Liubing (Leo); Toerless Eckert (eckert); Michael Richardson; Brian E
> Carpenter
> Cc: [email-masked]
> Subject: Re: [Anima-signaling] Data modle & Encoding model (DNS-SD,
> CBOR)-//Discuss: Using CBOR for anima signaling
> 
> >The other aspect of my question was that when there is DNS-SD data
> carried in the GDNP message, the DNS-SD is a nested structure under the
> GDNP message structure. Then, the DNS-SD data should also be in JASON
> objects as well as the GDNP message structure, rather than simply
> encapsulating it in a binary/string form to be a value of the GDNP object. Did
> I understand it correctly?
> 
> There are several options.  You could keep the same encoding as the
> original DNS-SD, and just send them as a CBOR byte array.  You could also
> create a new CBOR structure that sent the DNS-SD data in a parsed form.
> That would likely require either defining a CBOR structure for each RR type,
> or defining some more generic rule for how to do the translation.  I'm not
> enough of a DNS expert to know if the latter is possible.  I thought I
> remembered there being a repository of unpacking rules for the different RR
> types, but I can't seem to find it at the moment.

[Bing] Got it. Then I guess using CBOR byte array might be an easier way for GDNP, since we could avoid the unnecessary work to define CBOR structures for RR types.

> >[Bing] By reading the other threads, my understanding is that there is
> >a common parser for CBOR encoding -> JASON objects;
> 
> Usually not.  Think of it this way: when you parse XML, you get a set of
> objects that you can manipulate using the XML DOM.  That DOM happens
> to be standardized (which leads to it being a pretty terrible API to work with),
> but there are also XML libraries that provide an easier API to get at the data
> you parsed.
> 
> When you parse JSON, you get something DOM-like.  There's a set of APIs
> that allow you to access the type of the parsed thing, or access the 12th
> item of a parsed array.  This is a different API than the XML DOM.  In
> JavaScript, it's the same API that you use to access any variable, but in other
> languages without native support for variant types and maps (such as C), you
> need to type a little more to access the API.
> 
> So instead, I would say, there is a common parser for CBOR which creates
> native data structures in your programming environment, with an API to
> access those data structures.

[Bing] Got it. So there's no need to parse CBOR via JSON, CBOR is directly parsed into the structures that the current program could manipulate.

> >and there should be another customized parser for the JASON objects as
> defined in the specific protocol. I guess you were referring to the latter?
> 
> No.  You can parse JSON or CBOR without need for customizing the parser
> or needing to know the schema beforehand.  This is one of the reasons
> people tend to like this style of format.

[Bing] So the JSON/CBOR parser doesn't need to know the mapping of the parsed objects and the protocol fields, it only parses the objects to data structures that the application can access/deal with. And it's the application's job to handle the mapping and other semantic processing.
In this sense, the CBOR parser should be fairly lightweight.

> Notice that the internal structure of "struct" is hidden from the parser.  To
> process it, you would have to get the value of "struct", know somehow
> ahead of time that it contained JSON, instantiate a new JSON parser, and
> re-parse.  Instead, you would write:
> 
> {
>   "foo": "bar",
>   "struct": {"boo": "baz"}
> }
> 
[Bing] Understood. 
But the former case doesn't apply to the "DNS-SD as CBOR byte array" case, right? The CBOR array doesn't need another JSON parser, it only need the application to parse the array into RR records as I understand.

Best regards,
Bing

---

From: "Liubing (Leo)" <[email-masked]>
Date: Wed, 27 May 2015 03:39:13 +0000

Hi Brian,

Please see replies inline.

> From: Brian E Carpenter [mailto:[email-masked]]
> Sent: Wednesday, May 27, 2015 8:33 AM
> To: Joe Hildebrand (jhildebr); Liubing (Leo); Toerless Eckert (eckert); Michael
> Richardson

[Bing] Agreed. My intention to make them into one thread was to make sure the two things are de-coupled.

> In any case, the question an ASA would be asking DNS-SD is "do you have a
> locator for a service that supports the following autonomic objective?" 

[Bing] Ah, I think we have two different approaches in our mind. Let me try to explain it:
1) DNS-SD in parallel with GDNP. 
ASAs could call the GDNP module as well as calling the DNS-SD module. I guess that's what you said "b) should we have our own discovery mechanism as well as using DNS-SD in cases where it's useful?"?
2) DNS-SD embedded in GDNP
We always use the GDNP discovery mechanism, and just use the "DNS-SD data set" as a kind of GNDP-Objective representation.
In this case, an ASA would ask GDNP "who provides the DNS-SD service (say, _bootstrapRegistrar._udp.) ?"

I'm not sure which is better so far. First of all, did I correctly parse your thought?

> If we switch to expressing objectives in JSON, DNS-SD would need to deal with
> that instead of a type code.

[Bing] My understanding is that we are expressing the whole "GDNP messages" in JSON rather than only the "GDNP objectives"?
If I understand correctly, and you did mean "DNS-SD in parallel with GDNP", then I think the DNS-SD has no relationship with the JSON objects of the GDNP messages.

Best regards,
Bing

> >> [Bing] By reading the other threads, my understanding is that there
> >> is a common parser for CBOR encoding -> JASON objects;
> >
> > No.  You can parse JSON or CBOR without need for customizing the
> parser or needing to know the schema beforehand.  This is one of the
> reasons people tend to like this style of format.
> >

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 27 May 2015 16:18:23 +1200

Hello Bing,

> [Bing] Agreed. My intention to make them into one thread was to make sure the two things are de-coupled.

Good

> I'm not sure which is better so far. First of all, did I correctly parse your thought?

Yes. I'm not sure I see any advantage in 2) though, it seems to complicate
the data representation for no real reason.

> [Bing] My understanding is that we are expressing the whole "GDNP messages" in JSON rather than only the "GDNP objectives"?

Correct. I think the advantage is the flexibility/extensibility for
defining new objectives, but the method would apply to all our existing
message types and TLVs.

> If I understand correctly, and you did mean "DNS-SD in parallel with GDNP", then I think the DNS-SD has no relationship with the JSON objects of the GDNP messages.

That was my thinking; in a case where DNS-SD seems more natural, just use
it! We aren't designing GDNP as a printing solution, but if it's used
to set up printer servers, why not use the existing way to discover them?
For something designed for GDNP from the start, use GDNP discovery.

Regards
    Brian

---

From: "Liubing (Leo)" <[email-masked]>
Date: Wed, 27 May 2015 09:57:20 +0000

> objective?"
> > ASAs could call the GDNP module as well as calling the DNS-SD module. I
> guess that's what you said "b) should we have our own discovery mechanism
> as well as using DNS-SD in cases where it's useful?"?
> > We always use the GDNP discovery mechanism, and just use the "DNS-SD
> data set" as a kind of GNDP-Objective representation.

[Bing] I think the advantages might be:
1) We could have an unified signaling protocol for autonomic networking. This could simplify the interaction with ASAs. ASAs only need to deal with the signaling protocol.
2) The DNS-SD could utilize the signaling protocol's off-link flooding capability. DNS-SD normally uses mDNS which is mainly for on-link discovery. AFAIK it lacks a comprehensive design for off-link discovery. One case provided by Toerless in the private discussion was: an NOC located in the non-autonomic domain announces DNS-SD services through mDNS; at the edge, the autonomic node floods the DNS-SD service announcement through signaling protocol into the autonomic domain; then the ASAs learn the info of NOC.
3) The DNS-SD data set could be a starter of the GDNP-Objectives at the first stage.

However, I don't see these as very solid advantages, but some food for discussion. Do you think these reasons make sense?

> >> If we switch to expressing objectives in JSON, DNS-SD would need to
> >> deal with that instead of a type code.
> >
> > [Bing] My understanding is that we are expressing the whole "GDNP
> messages" in JSON rather than only the "GDNP objectives"?
> 
> Correct. I think the advantage is the flexibility/extensibility for defining new
> objectives, but the method would apply to all our existing message types and
> TLVs.
> 
> > If I understand correctly, and you did mean "DNS-SD in parallel with GDNP",
> then I think the DNS-SD has no relationship with the JSON objects of the
> GDNP messages.
> 
> it! We aren't designing GDNP as a printing solution, but if it's used to set up
> printer servers, why not use the existing way to discover them?

[Bing] I believe in some cases it is a reasonable approach.
But in this case, the DNS-SD is a dedicated plane which doesn't bother the signaling design at all. So, do you mean what we need to do is just simply saying ASAs could use mDNS/DNS-SD as well as the signaling protocol?

Best regards,
Bing

> >> to be standardized (which leads to it being a pretty terrible API to
> >> work with), but there are also XML libraries that provide an easier
> >> API to get at the data you parsed.
> >>>
> >> that allow you to access the type of the parsed thing, or access the
> >> 12th item of a parsed array.  This is a different API than the XML
> >> DOM.  In JavaScript, it's the same API that you use to access any
> >> variable, but in other languages without native support for variant
> >> types and maps (such as C), you need to type a little more to access the
> API.
> >>>
> >> native data structures in your programming environment, with an API
> >> to access those data structures.
> >>>
> >>> What I'm saying is I don't recommend creating a protocol that looks
> >>> like
> >> this (in JSON, for clarity):
> >>>
> >>> {
> >>>   "foo": "bar",
> >> ahead of time that it contained JSON, instantiate a new JSON parser,
> >> and re-parse.  Instead, you would write:
> >>>
> >>> {
> >>>   "foo": "bar",
> >>>   "struct": {"boo": "baz"}
> >>> }
> >>>

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Wed, 27 May 2015 16:10:23 +0000

>[Bing] Got it. Then I guess using CBOR byte array might be an easier way for GDNP, since we could avoid the unnecessary work to define CBOR structures for RR types.

Maybe.  That would mean further application-specific parsing of those binary fields, however.  It's not clear to me what the right design is without knowing more about the programming environments of the target developers, etc.

>[Bing] Got it. So there's no need to parse CBOR via JSON, CBOR is directly parsed into the structures that the current program could manipulate.

Correct.

>In this sense, the CBOR parser should be fairly lightweight.

Yes.  The parser can be very small, and completely generic.

>But the former case doesn't apply to the "DNS-SD as CBOR byte array" case, right? The CBOR array doesn't need another JSON parser, it only need the application to parse the array into RR records as…

Sorry, the JSON above was just an example that was easy (for me) to see.  Imagine if instead of JSON here, I did a DNS SRV record (in CBOR diagnostic format):

{ "answer": h'c00c0021000100000375...' }

vs:

{
  "answer": {
    "type": "SRV",
    "class": "IN",
    ...
  }
}

So yes, in the first approach, you would need a DNS RR parser.

-- 
Joe Hildebrand

---

From: Michael Richardson <[email-masked]>
Date: Wed, 27 May 2015 12:40:50 -0400

Joe Hildebrand (jhildebr) <[email-masked]> wrote:
    > In case anyone cares, a cleaned-up and linted version of Toerless' JSON
    > is attached, along with a naïve conversion to CBOR.  Here are the
    > sizes, in bytes:

    > 712 gdnp.json

At some point, people thought that CBOR would become compressed, but in
the end that did not happen.  I wonder what happens if you run gzip on
those two files?

---

From: "Toerless Eckert (eckert)" <[email-masked]>
Date: Wed, 27 May 2015 11:07:32 -0700

Michael:

I think Joe's comparison still has strings for the Type fields in
CBOR, not numeric encoding. As i said, i'd expect another 200 byte
or so when going numeric, And i'd expect maybe only 50 bytes or so
better if one would try a custom handmade TLV format for the structures
i proposed.

GZIP would give you likely better than CBOR compression of JSON
because GZIP compresses strings, which CBOR does not. But you'd end up
with a lot more memory and CPU consumption. I wouldn't expect GZIP
to create more than 10% savings though on a binary only CBOR
dataset.

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Wed, 27 May 2015 18:23:41 +0000

>the end that did not happen.  I wonder what happens if you run gzip on

With gzip -9 on the existing files:

312 gdnp.cbor.gz
     332 gdnp.json.gz

which gives another data point that Claude Shannon was pretty sharp.

I don't know enough about your use case, but can't imagine wanting to take the code size, power and latency hit for gzip for small devices.

Just by changing the keys to integers, I got the attached file, at 342 bytes, assuming the following strings had be standardized in a registry:

 0: name
 1: dns-sd
-1: gdnp
 2: class
-2: ttl
 3: type
-3: val
 4: dn
-4: port
 5: weight
-5: priority
 6: target
-6: dns-rr
 7: password
-7: stratum
 8: params
-8: l3addrs
 9: l4port
-9: instances
10: service

Crappy python code here:

https://gist.github.com/8c5fd4ed2ffd98d1e5b5

Note that the integers alternate positive/negative.  Numbers that are smaller by absolute value use fewer bytes in CBOR.

To recap the sizes:

544 gdnp.cbor
     312 gdnp.cbor.gz
     712 gdnp.json
     332 gdnp.json.gz
342 gdnp_int.cbor
     235 gdnp_int.cbor.gz

The important takeaway is that slightly-idiomatic CBOR is less than half the size of the equivalent JSON for this dataset.

-- 
Joe Hildebrand

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 28 May 2015 09:18:51 +1200

> I don't know enough about your use case, but can't imagine wanting to take the code size, power and latency hit for gzip for small devices.

I agree. This is control-plane messaging, so I don't see that saving every
drop of bandwidth is essential; what is relevant, perhaps, is being pushed
over the UDP/TCP boundary, but as I said yesterday, we'll already have a
security wrapper that will increase the message size anyway.

    Brian

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Wed, 27 May 2015 21:20:54 +0000

>security wrapper that will increase the message size anyway.

Oh, I forgot to respond to that.  If you want a more generic security wrapper, and you end up wanting to use CBOR, then you probably want the COSE work to go forward.  It would be nice if one or more of you indicated that on the ietf@ietf discussion to show that it's not just apparea folks that want the work to progress.

-- 
Joe Hildebrand

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 28 May 2015 09:29:24 +1200

On the DNS-SD topic:

> 1) We could have an unified signaling protocol for autonomic networking. This could simplify the interaction with ASAs. ASAs only need to deal with the signaling protocol.

OK, but that could also be achieved by simply forgetting about DNS-SD completely ;-)

> 2) The DNS-SD could utilize the signaling protocol's off-link flooding capability. DNS-SD normally uses mDNS which is mainly for on-link discovery. AFAIK it lacks a comprehensive design for off-link discovery. 

I assume the the DNSSD WG is trying to fix that? To be honest I have
not looked at their current work in detail. (That's why I hoped for a
DNS-SD expert in this design team, but it didn't work out.)

> One case provided by Toerless in the private discussion was: an NOC located in the non-autonomic domain announces DNS-SD services through mDNS; at the edge, the autonomic node floods the DNS-SD service announcement through signaling protocol into the autonomic domain; then the ASAs learn the info of NOC.

Sure, but that doesn't require DNS-SD to rely on GDNP...

> 3) The DNS-SD data set could be a starter of the GDNP-Objectives at the first stage.

Well, I am not clear that DNS-SD will work without human intervention
from a cold start; I do believe that GDNP discovery will do so.

> However, I don't see these as very solid advantages, but some food for discussion. Do you think these reasons make sense?

Well, we need to understand how DNSSD plans to solve the cold-start
multi-link situation.

> But in this case, the DNS-SD is a dedicated plane which doesn't bother the signaling design at all. So, do you mean what we need to do is just simply saying ASAs could use mDNS/DNS-SD as well as th…

That is my current idea - but if the DNS-SD guys want to come play with
us, we should listen.

    Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 28 May 2015 09:33:28 +1200

> Oh, I forgot to respond to that.  If you want a more generic security wrapper, and you end up wanting to use CBOR, then you probably want the COSE work to go forward.  It would be nice if one or mo…

I absolutely and utterly don't understand that discussion.

The plan here is that autonomic signalling MUST always run in a secure
environment, but that it doesn't need to be built in: we would run
within a secure autonomic control plane, or a TLS-protected channel.
At the moment we see that as pretty much orthogonal to the protocol
and its message format.

   Brian

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Wed, 27 May 2015 21:36:42 +0000

>I absolutely and utterly don't understand that discussion.

It's mostly the usual sort of pushback from Phil, and a bunch of fallout from that.  Nothing really interesting.  More use cases that might use the work is always a good thing for any working group, however.

>At the moment we see that as pretty much orthogonal to the protocol

If you're doing TLS, you don't need COSE.  If you're doing UDP, you can use DTLS, not COSE.  If you merely need object signing, or more interesting security properties than a protected channel, JOSE or COSE may be interesting if you pick JSON or CBOR (respectively), in order to reduce code complexity.

-- 
Joe Hildebrand

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 28 May 2015 12:52:19 +1200

> If you're doing TLS, you don't need COSE.  If you're doing UDP, you can use DTLS, not COSE.  If you merely need object signing, or more interesting security properties than a protected channel, JOS…

Thanks Joe, that's very clearly explained. At the moment my personal opinion
is that we don't need object-level security, because only trusted ASAs
will be sending the objects.

    Brian

---

From: "Liubing (Leo)" <[email-masked]>
Date: Thu, 28 May 2015 01:40:53 +0000

> Of Brian E Carpenter
> > If you're doing TLS, you don't need COSE.  If you're doing UDP, you can
> use DTLS, not COSE.  If you merely need object signing, or more interesting
> security properties than a protected channel, JOSE or COSE may be
> interesting if you pick JSON or CBOR (respectively), in order to reduce code
> complexity.
> 
> Thanks Joe, that's very clearly explained. At the moment my personal
> opinion is that we don't need object-level security, because only trusted ASAs

+1.

Best regards,
Bing

---

From: "Toerless Eckert (eckert)" <[email-masked]>
Date: Wed, 27 May 2015 18:47:57 -0700

> On the DNS-SD topic:

Let me maybe give a quick overview of where my interest in DNS-SD comes
from because as Leo mentioned, we just had that discussion in PM so far.

Before we see a lot of really cool new interesting autonomic agents pop up
in networks, the autonomic infra with its security, zero-touch connectivity
(eg: ACP) could well serve to help support automation of NOC to network
setup/operations . There are a lot of NOC services that every router needs:
Radius/Tacacs/Diameter servers, syslog servers, TFTP/http config/system
file download servers, NTP servers, DNS, DHCP, Lawfull Intercept
server (kidding.. i hope ;-), yada yada. 

These servers could easily be autoconfigured, therefore eliminating a lot
of day 1 device setup problems as well as server redundancy setup issues.
The solution to do this would have something like two agents:

One ASA sits in the NOC and needs to talk a protocol with all these NOC
servers that these servers easily can speak. This is where DNS-SD with
mDNS comes in. Its already supported by some existing servers, and where
not, its easy to setup by using bonjour/avahi and creating DNS-SD
config. And Its the best/only service definition framework the IETF
has real experience with AFAIK.

So, this NOC-services agent would take those services announcements,
and convert them into GDNP service announcements (which is where eg:
my suggested JSON/CBOR encoding comes in). Then the second agent on every
AN node is one that learns all those service announcements and "autoconfigures'
the network devices use of those services.

In Ciscos autonomic implementation we do this for an initial subset
of services, but we use hop-by-hop mDNS propagation instead of GDNP
and through that experience it's clear for me that DNS-SD is a good
starting point to define/announce all those services (including magic
parameters of each like multiple ports and passwords and the like),
but its really difficult to multi-hop propagate with mDNS messages.

So, given how GDNP wants/needs to do service (oops: "Objective" ;-)
discovery anyhow, i thought that this could really be a a good initial
killer use-case for GDNP because i can see how networks might want to
start deploying GDNP with these two agents.

SO, the next step would be to use ASA to automate configuration of
other existing services. Lets say for example: We did come up with
service-special protocols like BSR to auto-negotiate IP Multicast
"agents" called rendesvous points. Instead of doing those per-service
protocols, i could equally use the same GDNP + DNS-SD setup. Whats
the benefit now in DNS-SD ? Probably not the implementation anymore
as it's in-network, but it avoids me having to reinvent something new.
And if i want to invent something new than DNS-SD can at least serve
as a bar to measure up against!

Cheers
    Toerless

---

From: Michael Richardson <[email-masked]>
Date: Wed, 27 May 2015 22:31:08 -0400

Joe Hildebrand (jhildebr) <[email-masked]> wrote:
    > It's mostly the usual sort of pushback from Phil, and a bunch of
    > fallout from that.  Nothing really interesting.  More use cases that
    > might use the work is always a good thing for any working group,
    > however.

    > use DTLS, not COSE.  If you merely need object signing, or more
    > interesting security properties than a protected channel, JOSE or COSE
    > may be interesting if you pick JSON or CBOR (respectively), in order to
    > reduce code complexity.

We need signed objects that can transverse across a multitude of borders.
Sadly, we need 802.1AR objects, and I don't think we can greenfield a new
PKI, so COSE won't help us.

---

From: Sheng Jiang <[email-masked]>
Date: Thu, 28 May 2015 04:00:46 +0000

Hi, Brian and signaling design team members,

While the discussion regarding to Data model & Encoding model keep going within our anima signaling mail list, I am getting worried that we are lacking of experts and expert knowledge, particularly for DNS-SD. It worth for the leader, Brian to summary the current status and purpose of this discussion, and report to ANIMA main mail list, and also solicit more experts to join the discussion. That email may also copy to other relevant WG, such as dnssd or jose WGs, too.

Many thanks and best regards,

Sheng

>Sent: Thursday, May 28, 2015 8:52 AM
>To: Joe Hildebrand (jhildebr); Michael Richardson
>Cc: Toerless Eckert (eckert); [email-masked]; Liubing (Leo)
>>>> Oh, I forgot to respond to that.  If you want a more generic security
>wrapper, and you end up wanting to use CBOR, then you probably want the
>COSE work to go forward.  It would be nice if one or more of you indicated
>that on the ietf@ietf discussion to show that it's not just apparea folks that
>want the work to progress.
>>>
>> It's mostly the usual sort of pushback from Phil, and a bunch of fallout from
>that.  Nothing really interesting.  More use cases that might use the work is
>always a good thing for any working group, however.
>>
>> If you're doing TLS, you don't need COSE.  If you're doing UDP, you can use
>DTLS, not COSE.  If you merely need object signing, or more interesting
>security properties than a protected channel, JOSE or COSE may be interesting
>if you pick JSON or CBOR (respectively), in order to reduce code complexity.
>

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 28 May 2015 16:16:24 +1200

Michael,

> PKI, so COSE won't help us.

We need trust between ASAs across borders, but why does Anima require
signed objects? (I am sure there are other contexts where signed
objects are needed.)

    Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 28 May 2015 16:19:24 +1200

Certainly we need to do that. But it may take a few more days
before it's all clear enough in my mind to draft a summary.

Regards
   Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 28 May 2015 16:25:26 +1200

Thanks Toerless. That does seem like an interesting way to bridge
the old and the new worlds, which is necessary. Let me just focus
on one point for now:

> my suggested JSON/CBOR encoding comes in).

I get that. But that doesn't mean that the conversion needs to be
a simple wrapping of the DNS-derived representation. Surely we'd be
better to actually map it into a representation that is designed
specifically for autonomic operations? In other words we would
inherit the information but not the syntax.

Regards
   Brian

---

From: "Toerless Eckert (eckert)" <[email-masked]>
Date: Wed, 27 May 2015 21:59:20 -0700

> inherit the information but not the syntax.

That's why i included in my JSON example two options. One is
the more verbatim representation of DNS-SD record structure,
and the other one which i think covers the essential data needed. 
I don't know if or why the verbatim JSON representation of DNS-SD
records was necessary to achieve our goals, that was more an
exercise in JSON. But would like to hear some DNS experts on the topic.

Cheers
    toerless

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 28 May 2015 17:11:57 +1200

> exercise in JSON. But would like to hear some DNS experts on the topic.

OK, then we are on the same page. So I will try to formulate a summary
but I need to do a little reading first.

   Brian

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Thu, 28 May 2015 15:52:42 +0000

I remain unencumbered with actual knowledge of the problem you're trying to solve, so this sounds a lot like something we tried to configure applications:

https://tools.ietf.org/html/draft-daboo-aggregated-service-discovery-03

The idea was you do one HTTPS request to a .well-known URL, and get all of your app config data at once as JSON, without having to grovel over a thousand different DNS queries.

We couldn't explain this quickly enough to the normal hate-anything-new crowd, and none of us cared enough to waste the time to put it into small words, so it died.

Have you guys considered just passing around a HTTPS URL over your UDP channel?  That would be a lot easier to secure, since it would rely on existing code.  You could also be as verbose as you want, at which point JSON would probably be the right choice instead of CBOR.

-- 
Joe Hildebrand

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 29 May 2015 08:27:47 +1200

Joe,

We aren't trying to compete with traditional top-down configuration
protocols. That's NETCONF, RESTCONF and YANG territory. Also, discovery
is not the main issue; it's how autonomic service agents will negotiate
parameters with each other, after discovery.

All the same, I'm less convinced than I was a couple of weeks ago
that low-level optimization is vital compared to coding convenience.

Regards
   Brian

---

From: Michael Richardson <[email-masked]>
Date: Sat, 30 May 2015 19:37:22 -0400

> which gives another data point that Claude Shannon was pretty sharp.

Thanks.

    > I don't know enough about your use case, but can't imagine wanting to
    > take the code size, power and latency hit for gzip for small devices.

It certainly is a consideration.
If running over (D)TLS, what about the possibility of turning on TLS
level compression?  I also suspect that a much stupider compression (such as
plain-old huffman encoding would an almost as good job here.

    > Just by changing the keys to integers, I got the attached file, at 342
    > bytes, assuming the following strings had be standardized in a
    > registry:

So one might conclude that gzip is basically acting as an adhoc assigner of
integer keys to strings...  The problem with the explicit registry is that it
essentially removes the major advantage of JSON/CBOR over some sane binary
TLV: which is that the string format provides for easy and non-centralized
expansion of our lexicon.

    > The important takeaway is that slightly-idiomatic CBOR is less than
    > half the size of the equivalent JSON for this dataset.

My conclusion is that doing CBOR (uncompressed) with strings is only slightly
better than TLV due to the lack of a need for a registry.   If we have to
go to a registry, then we might as well use GNDP-style TLV.

My experience with IKE1/2, Radius, etc. is that after turning the TLVs into
an internal datastructure/dictionary, that one has to verify that the right
data is there anyway.  Same as with JSON...

As such, I propose that we temporarily shelve this discussion, and return to
it later on.

---

From: Michael Richardson <[email-masked]>
Date: Sat, 30 May 2015 19:42:37 -0400

> signed objects? (I am sure there are other contexts where signed

The following statements need to be signed:
- I am (vendor's BAR's reseller BAZ's) device XYZ.
- Network FOO owns device XYZ.
- Network FOO is intended to provide service ABC.

Some of these things can and will occur in the form of 802.1AR objects, and
once we have the ASN.1 code, we might wind up going to PKIX format signed
objects for everything.
Some of these things may occur via PSKs shared with MASA or other deployment
aids, but that just means that they are non-transitive signed objects.

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Sun, 31 May 2015 18:16:06 +0000

>plain-old huffman encoding would an almost as good job here.

DEFLATE and LZS are currently defined, but you probably don't want to use them because of the effort that would be required to avoid CRIME-style attacks.

>expansion of our lexicon.

Almost.  You're missing one bit, which is that in CBOR, the keys can be any type, and don't have to be homogenous, which means you can do:

{
  0: "standardized key",
  "extension.example.com": "non-standard extension"
}

In this approach, any integer key would be controlled by IANA.

>go to a registry, then we might as well use GNDP-style TLV.

If the TLV uses strings as keys, it would likely be of roughly the same size as CBOR with string keys.  You might do better encoding if you know exactly what kind of data you're dealing with, at the expense of being able to encode generic data with adequate type info.

In particular, if you want to do length-counted strings, and you know that they're always going to be 255 bytes or less, you could save a byte over CBOR; you could also do NULL-terminated strings and save a byte at the expense of opening up potential attacks on sloppy implementations.

Yes.

>As such, I propose that we temporarily shelve this discussion, and return to it later on.

Cool.  I'm going to unsubscribe from the list after a day or two.  Please let me know if you want input later.

-- 
Joe Hildebrand