idnits 2.17.1 draft-ietf-cbor-file-magic-03.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- == There are 2 instances of lines with non-ascii characters in the document. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (4 August 2021) is 989 days in the past. Is this intentional? Checking references for intended status: Best Current Practice ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Outdated reference: A later version (-09) exists of draft-ietf-cose-cbor-encoded-cert-02 == Outdated reference: A later version (-25) exists of draft-ietf-rats-eat-10 == Outdated reference: A later version (-24) exists of draft-ietf-sacm-coswid-18 -- Obsolete informational reference (is this intentional?): RFC 8152 (Obsoleted by RFC 9052, RFC 9053) Summary: 0 errors (**), 0 flaws (~~), 5 warnings (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 CBOR Working Group M. Richardson 3 Internet-Draft Sandelman Software Works 4 Intended status: Best Current Practice 4 August 2021 5 Expires: 5 February 2022 7 On storing CBOR encoded items on stable storage 8 draft-ietf-cbor-file-magic-03 10 Abstract 12 This document proposes an on-disk format for CBOR objects that is 13 friendly to common on-disk recognition systems like the Unix file(1) 14 command. 16 This document is being discussed at: https://github.com/cbor-wg/cbor- 17 magic-number 19 Status of This Memo 21 This Internet-Draft is submitted in full conformance with the 22 provisions of BCP 78 and BCP 79. 24 Internet-Drafts are working documents of the Internet Engineering 25 Task Force (IETF). Note that other groups may also distribute 26 working documents as Internet-Drafts. The list of current Internet- 27 Drafts is at https://datatracker.ietf.org/drafts/current/. 29 Internet-Drafts are draft documents valid for a maximum of six months 30 and may be updated, replaced, or obsoleted by other documents at any 31 time. It is inappropriate to use Internet-Drafts as reference 32 material or to cite them other than as "work in progress." 34 This Internet-Draft will expire on 5 February 2022. 36 Copyright Notice 38 Copyright (c) 2021 IETF Trust and the persons identified as the 39 document authors. All rights reserved. 41 This document is subject to BCP 78 and the IETF Trust's Legal 42 Provisions Relating to IETF Documents (https://trustee.ietf.org/ 43 license-info) in effect on the date of publication of this document. 44 Please review these documents carefully, as they describe your rights 45 and restrictions with respect to this document. Code Components 46 extracted from this document must include Simplified BSD License text 47 as described in Section 4.e of the Trust Legal Provisions and are 48 provided without warranty as described in the Simplified BSD License. 50 Table of Contents 52 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 53 1.1. Terminology . . . . . . . . . . . . . . . . . . . . . . . 3 54 2. Requirements for a Magic Number . . . . . . . . . . . . . . . 4 55 3. Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . 4 56 3.1. The CBOR Protocol Specific Tag . . . . . . . . . . . . . 4 57 3.2. CBOR Tag Wrapped . . . . . . . . . . . . . . . . . . . . 4 58 3.3. CBOR Tag Sequence . . . . . . . . . . . . . . . . . . . . 5 59 4. Advice to Protocol Developers . . . . . . . . . . . . . . . . 5 60 4.1. Is the on-wire format new? . . . . . . . . . . . . . . . 7 61 4.2. Can many items be trivially concatenated? . . . . . . . . 7 62 4.3. Are there tags at the start? . . . . . . . . . . . . . . 7 63 5. Security Considerations . . . . . . . . . . . . . . . . . . . 7 64 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 7 65 6.1. CBOR Sequence Tag . . . . . . . . . . . . . . . . . . . . 8 66 6.2. CBOR Tags for CoAP Content-Format Numbers . . . . . . . . 8 67 7. References . . . . . . . . . . . . . . . . . . . . . . . . . 8 68 7.1. Normative References . . . . . . . . . . . . . . . . . . 8 69 7.2. Informative References . . . . . . . . . . . . . . . . . 9 70 Appendix A. CBOR Tags for CoAP Content Formats . . . . . . . . . 10 71 A.1. Content-Format Tag Examples . . . . . . . . . . . . . . . 10 72 Appendix B. Example from Openswan . . . . . . . . . . . . . . . 11 73 Appendix C. Changelog . . . . . . . . . . . . . . . . . . . . . 12 74 Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . 12 75 Contributors . . . . . . . . . . . . . . . . . . . . . . . . . . 12 76 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 12 78 1. Introduction 80 Since very early in computing, operating systems have sought ways to 81 mark which files could be processed by which programs. 83 For instance, the Unix file(1) command, which has existed since 1973 84 [file], has been able to identify many file formats for decades. 85 Many systems (Linux, MacOS, Windows) will select the correct 86 application based upon the file contents, if the system can not 87 determine it by other means: for instance, the classic MacOS 88 maintained a resource fork that includes media type ("MIME type") 89 information and therefore ideally never needs to know anything about 90 the file. Other systems do this by file extensions. 92 While having a media type associated with the file is a better 93 solution in general, when files become disconnected from their type 94 information, such as when attempting to do forensics on a damaged 95 system, then being able to identify a file type can become very 96 important. 98 It is noted that in the media type registration, that a magic number 99 is asked for, if available, as is a file extension. 101 A challenge for the file(1) program is often that it can be confused 102 by the encoding vs the content. For instance, an Android "apk" used 103 to transfer and store an application may be identified as a ZIP file. 104 Both OpenOffice or MSOffice files are ZIP files of XML files. 105 (Unless OpenOffice files are flat (fodp) files, in which case they 106 may appear to be generic XML files.) 108 As CBOR becomes a more and more common encoding for a wide variety of 109 artifacts, identifying them as just "CBOR" is probably not 110 sufficient. This document provides a way to encode a magic number 111 into the beginning of a CBOR format file. Two options are presented: 112 typically a CBOR Protocol author will specify one. 114 A CBOR Protocol is a specification which uses CBOR as its encoding. 115 Examples of CBOR Protocols currently under development include CoSWID 116 [I-D.ietf-sacm-coswid], and EAT [I-D.ietf-rats-eat]. COSE itself 117 [RFC8152] is considered infrastructure, however the encoding of 118 public keys in CBOR as described in [I-D.ietf-cose-cbor-encoded-cert] 119 would be an identified CBOR Protocol. 121 A major inspiration for this document is observing the mess in ASN.1 122 based systems where most files are PEM encoded, identified by the 123 extension "pem", confusing public keys, private keys, certificate 124 requests and S/MIME content. 126 These proposals are invasive to how CBOR protocols are written to 127 disk, but in both cases, the proposed envelope does not require that 128 the tag be transferred on the wire. 130 In addition to the on-disk identification aspects, there are some 131 protocols which may benefit from having such a magic number on the 132 wire if they presently using a different (legacy) encoding scheme. 133 The presence of the identifiable magic sequence signals that CBOR is 134 being used or a legacy scheme. 136 1.1. Terminology 138 The term "diagnostic notation" refers to the human-readable notation 139 for CBOR data items defined in Section 8 of [RFC8949] and Appendix G 140 of [RFC8610]. 142 The term CDDL (Concise Data Definition Language) refers to the 143 language defined in [RFC8610]. 145 2. Requirements for a Magic Number 147 A magic number is ideally a unique fingerprint, present in the first 148 4 or 8 bytes of the file, which does not change when the contents 149 change, and does not depend upon the length of the file. 151 Less ideal solutions have a pattern that needs to be matched, but in 152 which some bytes need to be ignored. While the Unix file(1) command 153 can be told to ignore bytes, this can lead to ambiguities. 155 3. Protocol 157 There are two variations of this practice. Both use CBOR Tags in a 158 way that results in a deterministic first 8 to 12 bytes. 160 3.1. The CBOR Protocol Specific Tag 162 CBOR Protocol designers should obtain a tag for each major type of 163 object that they might store on disk. As there are more than 4 164 billion available 4-byte tags, there should be little issue in 165 allocating a few to each available CBOR Protocol. 167 The policy is First Come First Served, so all that is required is an 168 email to IANA, having filled in the small template provided in 169 Section 9.2 of [RFC8949]. 171 This tag should be allocated by the author of the CBOR Protocol, and 172 to be in the four-byte range, it should be at least 0x01000000 173 (decimal 16777216) in value. 175 The use of a sequence of four US-ASCII codes which are mnemonic to 176 the protocol is encouraged, but not required. 178 For CBOR byte strings that happen to contain a representation that is 179 described by a CoAP Content-Format Number (Section 12.3 of [RFC7252], 180 Subregistry Content-Formats of [IANA.core-parameters]), a tag number 181 has been allocated in Section 6.2 (see Appendix A for details and 182 examples). 184 3.2. CBOR Tag Wrapped 186 This proposal starts with the Self-described CBOR tag, 55799, as 187 described in Section 3.4.6 of [RFC8949]. 189 A second CBOR Tag is then allocated to describe the specific Protocol 190 involved, as described above. 192 This proposal wraps the CBOR value as tags usually do. Applications 193 that need to send the CBOR value across a constrained link may wish 194 to remove the two tags if the use is implicitly understood. This is 195 a decision of the CBOR Protocol specification. 197 3.3. CBOR Tag Sequence 199 This proposal makes use of CBOR Sequences as described in [RFC8742]. 201 This proposal consists of two tags and a constant string for a total 202 of 12 bytes. 204 1. The file shall start with the Self-described CBOR Sequence tag, 205 55800. 207 2. The file shall continue with a CBOR tag, from the First Come 208 First Served space, which uniquely identifies the CBOR Protocol. 209 The use of a four-byte tag is encouraged. 211 3. The three byte CBOR byte string containing 0x42_4F_52. 213 The first part identifies the file as being a CBOR Sequence, and does 214 so with all the desirable properties explained in Section 3.4.6 of 215 [RFC8949]. Specifically, it does not seem to conflict with any known 216 file types, and it is not valid Unicode in any Unicode encoding. 218 The second part identifies which CBOR Protocol is used, as described 219 above. 221 The third part is represented as a constant byte sequence 222 0x43_42_4f_52, the ASCII characters "CBOR", which is the CBOR encoded 223 data item for the three byte sequence 0x42_4f_52 ('BOR' in diagnostic 224 notation). This is the data item that is being tagged. 226 The actual CBOR Protocol value then follows as the next data item(s) 227 in the CBOR sequence, without a need for any further specific tag. 228 The use of a CBOR Sequence allows the application to trivially remove 229 the first item with the two tags. 231 This means that should a file be reviewed by a human (directly in an 232 editor, or in a hexdump display), it will include the ASCII 233 characters "CBOR" prominently. This value is also included simply 234 because the two tags need to tag something. 236 4. Advice to Protocol Developers 238 This document introduces a choice between a CBOR Sequence and a 239 wrapped CBOR Tag. Which should a protocol designer use? 240 In this discussion, one assumes that there is an object stored in a 241 file, perhaps specified by a system operator in a configuration file. 243 For example: a private key used in COSE operations, a public key/ 244 certificate in C509 or CBOR format, a recorded sensor reading stored 245 for later transmission, or a COVID vaccination certificate that needs 246 to be displayed in QRcode form. 248 Both the CBOR Tag Sequence and the wrapped tag can be trivially 249 removed by an application before sending the CBOR content out on the 250 wire. 252 The CBOR Tag Sequence is a little bit easier to remove as in most 253 cases, CBOR parsers will return it as a unit, and then return the 254 actual CBOR item, which could be anything at all, and could include 255 CBOR tags that _do_ need to be sent on wire. 257 On the other hand, having the CBOR Tag Sequence in the file requires 258 that all programs that expect to examine that file are able to skip 259 what appears to be an empty CBOR item. Programs which might not 260 expect the CBOR Tag Sequence, but which would operate without a 261 problem would include any program that expects to process CBOR 262 Sequences from the file. 264 As an example of where there was a problem with previous security 265 systems, "PEM" format certificate files grew to be able to contain 266 multiple certificates by simple concatenation. The PKCS1 format 267 could also contain a private key object followed by a one or more 268 certificate objects: but only when in PEM format. But, when in 269 binary DER format, concatenation of certificates was not compatible 270 with most programs. 272 The use of CBOR Tag Wrapped format is easier to retrofit to an 273 existing format with existing and unchangeable on-disk format. This 274 new sequence of tags are expected to be trivially ignored by an 275 existing program when reading CBOR from disk, even if the program 276 only supports decoding a single data item (and not a CBOR sequence). 277 But, a naive program might also then transmit the additional tags 278 across the network. Removing the CBOR Tag Wrapped format requires 279 knowledge of the two tags involved. Other tags present might be 280 needed. For a representation matching a specific media-type that is 281 carried in a CBOR byte string, the byte string head will already have 282 to be removed for use as such a representation, so it should be easy 283 to remove the enclosing tag heads as well. This is of particular 284 interest with the pre-defined tags provided by Appendix A for media- 285 types with CoAP Content-Format numbers. 287 Here are some considerations: 289 4.1. Is the on-wire format new? 291 If the on-wire format is new, then it could be specified with the 292 CBOR Tag Wrapped format if the extra eight bytes are not a problem. 293 The disk format is then identical to the on-wire format. 295 If the eight bytes are a problem (and they usually are if CBOR is 296 being considered), then the CBOR Tag Sequence format should be 297 adopted for on-disk storage. 299 4.2. Can many items be trivially concatenated? 301 If the programs that read the contents of the file already expect to 302 process all of the items in the file (not just the first), then the 303 CBOR Tag Sequence format may be easily retrofitted. 305 The program involved may throw errors or warnings on the CBOR Tag 306 Sequence if they have not yet been updated, but this may not be a 307 problem. If it is, then consideration should be given to CBOR Tag 308 Wrapped. 310 If only one item is ever expected in the file, the the use of CBOR 311 Tag Sequence may present an implementation hurdle to programs that 312 previously just read a single value and used it. 314 4.3. Are there tags at the start? 316 If the Protocol expects to use other tags values at the top-level, 317 then it may be easier to explain if the CBOR Tag Sequence format is 318 used. 320 5. Security Considerations 322 This document provides a way to identify CBOR Protocol objects. 323 Clearly identifying CBOR contents on disk may have a variety of 324 impacts. 326 The most obvious is that it may allow malware to identify interesting 327 objects on disk, and then corrupt them. 329 6. IANA Considerations 331 Section 6.1 documents the allocation that was done for a CBOR tag to 332 be used in a CBOR sequence to identify the sequence (an example for 333 using this tag is found in Appendix B). Section 6.2 allocates a CBOR 334 tag for each actual or potential CoAP Content-Format number (examples 335 are in Appendix A). 337 6.1. CBOR Sequence Tag 339 IANA has allocated tag 55800 as the CBOR Sequence tag. This tag is 340 from the First Come/First Served area. 342 The value has been picked to have properties similar to the 55799 tag 343 (Section 3.4.6 of [RFC8949]). 345 The hexadecimal representation is: 0xd9_d9_f8. 347 This is not valid UTF-8: the first 0xd9 puts the value into the 348 three-byte value of UTF-8, but the 0xd9 as the second value is not a 349 valid second byte for UTF-8. 351 This is not valid UTF-16: the byte sequence 0xd9d9 (in either endian 352 order) puts this value into the UTF-16 high-half zone, which would 353 signal that this a 32-bit Unicode value. However, the following 354 16-bit big-endian value 0xf8.. is not a valid second sequence 355 according to [RFC2781]. On a little-endian system, it would be 356 necessary to examine the fourth byte to determine if it is valid. 357 That next byte is determined by the subsequent encoding, and 358 Section 3.4.6 of [RFC8949] has already determined that no valid CBOR 359 encodings result in a valid UTF-16. 361 Data Item: 362 byte string 364 Semantics: 365 indicates that the file contains CBOR Sequences 367 6.2. CBOR Tags for CoAP Content-Format Numbers 369 IANA is requested to allocate the tag numbers 1668546560 (0x63740000) 370 to 1668612095 (0x6374FFFF) as follows: 372 Data Item: 373 byte string 375 Semantics: 376 for each tag number NNNN, the representation of content-format 377 (RFC7252) NNNN-1668546560 379 Reference: 380 RFCthis 382 7. References 384 7.1. Normative References 386 [RFC8742] Bormann, C., "Concise Binary Object Representation (CBOR) 387 Sequences", RFC 8742, DOI 10.17487/RFC8742, February 2020, 388 . 390 [RFC8949] Bormann, C. and P. Hoffman, "Concise Binary Object 391 Representation (CBOR)", STD 94, RFC 8949, 392 DOI 10.17487/RFC8949, December 2020, 393 . 395 7.2. Informative References 397 [file] Wikipedia, "file (command)", 20 January 2021, 398 . 400 [I-D.ietf-cose-cbor-encoded-cert] 401 Mattsson, J. P., Selander, G., Raza, S., Höglund, J., and 402 M. Furuhed, "CBOR Encoded X.509 Certificates (C509 403 Certificates)", Work in Progress, Internet-Draft, draft- 404 ietf-cose-cbor-encoded-cert-02, 12 July 2021, 405 . 408 [I-D.ietf-rats-eat] 409 Mandyam, G., Lundblade, L., Ballesteros, M., and J. 410 O'Donoghue, "The Entity Attestation Token (EAT)", Work in 411 Progress, Internet-Draft, draft-ietf-rats-eat-10, 7 June 412 2021, . 415 [I-D.ietf-sacm-coswid] 416 Birkholz, H., Fitzgerald-McKay, J., Schmidt, C., and D. 417 Waltermire, "Concise Software Identification Tags", Work 418 in Progress, Internet-Draft, draft-ietf-sacm-coswid-18, 12 419 July 2021, . 422 [IANA.core-parameters] 423 IANA, "Constrained RESTful Environments (CoRE) 424 Parameters", 425 . 427 [RFC2781] Hoffman, P. and F. Yergeau, "UTF-16, an encoding of ISO 428 10646", RFC 2781, DOI 10.17487/RFC2781, February 2000, 429 . 431 [RFC7252] Shelby, Z., Hartke, K., and C. Bormann, "The Constrained 432 Application Protocol (CoAP)", RFC 7252, 433 DOI 10.17487/RFC7252, June 2014, 434 . 436 [RFC8152] Schaad, J., "CBOR Object Signing and Encryption (COSE)", 437 RFC 8152, DOI 10.17487/RFC8152, July 2017, 438 . 440 [RFC8610] Birkholz, H., Vigano, C., and C. Bormann, "Concise Data 441 Definition Language (CDDL): A Notational Convention to 442 Express Concise Binary Object Representation (CBOR) and 443 JSON Data Structures", RFC 8610, DOI 10.17487/RFC8610, 444 June 2019, . 446 Appendix A. CBOR Tags for CoAP Content Formats 448 Often, there is a need to identify a media type (or content type, 449 i.e., media type optionally used with parameters) that describes a 450 byte string in a CBOR data item. 452 Section 5.10.3 of [RFC7252] defines the concept of a Content-Format, 453 which is a short 16-bit unsigned integer that identifies a specific 454 content type (media type plus optionally parameters), optionally 455 together with a content encoding. 457 This specification allocates CBOR tag numbers 1668546560 (0x63740000) 458 to 1668612095 (0x6374FFFF) for the tagging of representations of 459 specific content formats. The tag content tagged with tag number 460 NNNN (in above range) is a byte string that is to be interpreted as a 461 representation of the content format NNNN-1668546560. 463 A.1. Content-Format Tag Examples 465 Subregistry Content-Formats of [IANA.core-parameters] defines content 466 formats that can be used as examples: 468 * Content-Format 432 stands for media type application/td+json (no 469 parameters). The corresponding tag number is 1668546992 (i.e., 470 1668546560+432). 472 So the following CDDL snippet can be used to identify application/ 473 td+json representations: 475 td-json = #6.1668546992(bstr) 476 Note that a byte string is used as the type of the tag content, 477 because a media type representation in general can be any byte 478 string. 480 * Content-Format 11050 stands for media type application/json in 481 deflate encoding. 483 The corresponding tag number is 1668557610 (i.e., 484 1668546560+11050). 486 So the following CDDL snippet can be used to identify application/ 487 json representations compressed in deflate encoding: 489 json-deflate = #6.1668557610(bstr) 491 The byte string is appropriate here as the type for the tag 492 content, because the compressed form is an instance of a general 493 byte string. 495 Appendix B. Example from Openswan 497 The Openswan IPsec project has a daemon ("pluto"), and two control 498 programs ("addconn", and "whack"). They communicate via a Unix- 499 domain socket, over which a C-structure containing pointers to 500 strings is serialized using a bespoke mechanism. This is normally 501 not a problem as the structure is compiled by the same compiler; but 502 when there are upgrades it is possible for the daemon and the control 503 programs to get out of sync by the bespoke serialization. As a 504 result, there are extra compensations to deal with shutting the 505 daemon down. During testing it is sometimes the case that upgrades 506 are backed out. 508 In addition, when doing unit testing, the easiest way to load policy 509 is to use the normal policy reading process, but that is not normally 510 loaded in the daemon. Instead the IPC that is normally sent across 511 the wire is compiled/serialized and placed in a file. The above 512 magic number is included in the file, and also on the IPC in order to 513 distinguish the "shutdown" command CBOR operation. 515 In order to reduce the problems due to serialization, the 516 serialization is being changed to CBOR. Additionally, this change 517 allows the IPC to be described by CDDL, and for any language that 518 encode to CBOR can be used. 520 IANA has allocated the tag 1330664270, or 0x4f_50_53_4e for this 521 purpose. As a result, each file and each IPC is prefixed with: 523 In diagnostic notation: 525 55800(1330664270(h'424F52')) 527 Or in hex: 529 00000000 d9 d9 f8 da 4f 50 53 4e 43 42 4f 52 |....OPSNCBOR| 531 Appendix C. Changelog 533 Acknowledgements 535 The CBOR WG brainstormed this protocol on January 20, 2021. 537 Contributors 539 Carsten Bormann 540 Universität Bremen TZI 541 Germany 543 Email: cabo@tzi.org 545 Josef 'Jeff' Sipek 547 Email: jeffpc@josefsipek.net 549 Author's Address 551 Michael Richardson 552 Sandelman Software Works 554 Email: mcr+ietf@sandelman.ca