idnits 2.17.1 draft-ietf-cbor-file-magic-05.txt: -(5): Line appears to be too long, but this could be caused by non-ascii characters in UTF-8 encoding Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- == There are 3 instances of lines with non-ascii characters in the document. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (13 September 2021) is 955 days in the past. Is this intentional? Checking references for intended status: Best Current Practice ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Outdated reference: A later version (-09) exists of draft-ietf-cose-cbor-encoded-cert-02 == Outdated reference: A later version (-25) exists of draft-ietf-rats-eat-10 == Outdated reference: A later version (-24) exists of draft-ietf-sacm-coswid-18 -- Obsolete informational reference (is this intentional?): RFC 8152 (Obsoleted by RFC 9052, RFC 9053) Summary: 0 errors (**), 0 flaws (~~), 5 warnings (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 CBOR Working Group M. Richardson 3 Internet-Draft Sandelman Software Works 4 Intended status: Best Current Practice C. Bormann 5 Expires: 17 March 2022 Universität Bremen TZI 6 13 September 2021 8 On storing CBOR encoded items on stable storage 9 draft-ietf-cbor-file-magic-05 11 Abstract 13 This document proposes an on-disk format for CBOR objects that is 14 friendly to common on-disk recognition systems like the Unix file(1) 15 command. 17 This document is being discussed at: https://github.com/cbor-wg/cbor- 18 magic-number 20 Status of This Memo 22 This Internet-Draft is submitted in full conformance with the 23 provisions of BCP 78 and BCP 79. 25 Internet-Drafts are working documents of the Internet Engineering 26 Task Force (IETF). Note that other groups may also distribute 27 working documents as Internet-Drafts. The list of current Internet- 28 Drafts is at https://datatracker.ietf.org/drafts/current/. 30 Internet-Drafts are draft documents valid for a maximum of six months 31 and may be updated, replaced, or obsoleted by other documents at any 32 time. It is inappropriate to use Internet-Drafts as reference 33 material or to cite them other than as "work in progress." 35 This Internet-Draft will expire on 17 March 2022. 37 Copyright Notice 39 Copyright (c) 2021 IETF Trust and the persons identified as the 40 document authors. All rights reserved. 42 This document is subject to BCP 78 and the IETF Trust's Legal 43 Provisions Relating to IETF Documents (https://trustee.ietf.org/ 44 license-info) in effect on the date of publication of this document. 45 Please review these documents carefully, as they describe your rights 46 and restrictions with respect to this document. Code Components 47 extracted from this document must include Simplified BSD License text 48 as described in Section 4.e of the Trust Legal Provisions and are 49 provided without warranty as described in the Simplified BSD License. 51 Table of Contents 53 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 54 1.1. Terminology . . . . . . . . . . . . . . . . . . . . . . . 4 55 2. Requirements for a Magic Number . . . . . . . . . . . . . . . 4 56 3. Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . 4 57 3.1. The CBOR Protocol Specific Tag . . . . . . . . . . . . . 4 58 3.2. CBOR Tag Wrapped . . . . . . . . . . . . . . . . . . . . 5 59 3.3. CBOR Tag Sequence . . . . . . . . . . . . . . . . . . . . 5 60 4. Advice to Protocol Developers . . . . . . . . . . . . . . . . 6 61 4.1. Is the on-wire format new? . . . . . . . . . . . . . . . 7 62 4.2. Can many items be trivially concatenated? . . . . . . . . 7 63 4.3. Are there tags at the start? . . . . . . . . . . . . . . 8 64 5. Security Considerations . . . . . . . . . . . . . . . . . . . 8 65 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 8 66 6.1. CBOR Sequence Tag . . . . . . . . . . . . . . . . . . . . 8 67 6.2. CBOR Tags for CoAP Content-Format Numbers . . . . . . . . 9 68 7. References . . . . . . . . . . . . . . . . . . . . . . . . . 9 69 7.1. Normative References . . . . . . . . . . . . . . . . . . 9 70 7.2. Informative References . . . . . . . . . . . . . . . . . 9 71 Appendix A. CBOR Tags for CoAP Content Formats . . . . . . . . . 10 72 A.1. Content-Format Tag Examples . . . . . . . . . . . . . . . 11 73 Appendix B. Example from Openswan . . . . . . . . . . . . . . . 11 74 Appendix C. Changelog . . . . . . . . . . . . . . . . . . . . . 12 75 Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . 12 76 Contributors . . . . . . . . . . . . . . . . . . . . . . . . . . 12 77 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 12 79 1. Introduction 81 Since very early in computing, operating systems have sought ways to 82 mark which files could be processed by which programs. 84 For instance, the Unix file(1) command, which has existed since 1973 85 [file], has been able to identify many file formats for decades based 86 upon the contents of the file. 88 Many systems (Linux, macOS, Windows) will select the correct 89 application based upon the file contents, if the system can not 90 determine it by other means. For instance, starting in Mac OS, a 91 resource fork was maintained that includes media type ("MIME type") 92 information and therefore ideally never needs to know anything about 93 the file. 95 But, many other systems do this by file extensions. Many common web 96 servers derive the MIME-type information from file extensions. 98 While having a media type associated with the file is a better 99 solution in general, when files become disconnected from their type 100 information, such as when attempting to do forensics on a damaged 101 system, then being able to identify a file type can become very 102 important. 104 It is noted that in the media type registration, that a magic number 105 is asked for, if available, as is a file extension. 107 A challenge for the file(1) program is often that it can be confused 108 by the encoding vs the content. For instance, an Android "apk" used 109 to transfer and store an application may be identified as a ZIP file. 110 Additionally, both OpenOffice and MSOffice files are ZIP files of XML 111 files. 113 As CBOR becomes a more and more common encoding for a wide variety of 114 artifacts, identifying them as just "CBOR" is probably not 115 sufficient. This document provides a way to encode a magic number 116 into the beginning of a CBOR format file. Two possible methods are 117 presented: a CBOR Protocol author will specify one. (A CBOR Protocol 118 is a specification which uses CBOR as its encoding) 120 Examples of CBOR Protocols currently under development include CoSWID 121 [I-D.ietf-sacm-coswid], and EAT [I-D.ietf-rats-eat]. COSE itself 122 [RFC8152] is considered infrastructure, however the encoding of 123 public keys in CBOR as described in [I-D.ietf-cose-cbor-encoded-cert] 124 would be an identified CBOR Protocol as well. 126 A major inspiration for this document is observing the mess in ASN.1 127 based systems where most files are PEM encoded, identified by the 128 extension "pem", confusing public keys, private keys, certificate 129 requests, and S/MIME content. 131 While these proposals are invasive to how CBOR Protocols are stored 132 in files, but in both cases, the proposed envelope does not require 133 that the tag be transferred on the wire. 135 In addition to the on-disk identification aspects, there are some 136 protocols which may benefit from having such a magic number on the 137 wire if they are presently using a different (legacy) encoding 138 scheme. The presence of the identifiable magic sequence signals that 139 CBOR is being used vs a legacy scheme. 141 1.1. Terminology 143 The term "diagnostic notation" refers to the human-readable notation 144 for CBOR data items defined in Section 8 of [RFC8949] and Appendix G 145 of [RFC8610]. 147 The term CDDL (Concise Data Definition Language) refers to the 148 language defined in [RFC8610]. 150 2. Requirements for a Magic Number 152 A magic number is ideally a unique fingerprint, present in the first 153 4 or 8 bytes of the file, which does not change when the contents 154 change, and does not depend upon the length of the file. 156 Less ideal solutions have a pattern that needs to be matched, but in 157 which some bytes need to be ignored. While the Unix file(1) command 158 can be told to ignore bytes, this can lead to ambiguities. 160 3. Protocol 162 There are two methods presented. Which one is to be used is up to 163 the CBOR Protocol author to determine. Both use CBOR Tags in a way 164 that results in a deterministic first 8 to 12 bytes. 166 3.1. The CBOR Protocol Specific Tag 168 CBOR Protocol designers need to obtain a CBOR tag for each major type 169 of object that they might store on disk. As there are more than 4 170 billion available 4-byte tags, there should be little issue in 171 allocating a few to each available CBOR Protocol. 173 The IANA policy for 4-byte CBOR Tags is First Come First Served, so 174 all that is required is an email to IANA, having filled in the small 175 template provided in Section 9.2 of [RFC8949]. 177 This tag needs to be allocated by the author of the CBOR Protocol. 178 In order to be in the four-byte range, and so that there are no 179 leading zeros, the value needs to be in the range 0x01000000 (decimal 180 16777216) to 0xFFFFFFFF (decimal 4294967295). It is further 181 suggested to avoid values that have an embedded zero byte in the four 182 bytes of their binary representation (e.g., 0x12003456). 184 The use of a sequence of four US-ASCII codes which are mnemonic to 185 the protocol is encouraged, but not required. 187 For CBOR byte strings that happen to contain a representation that is 188 described by a CoAP Content-Format Number (Section 12.3 of [RFC7252], 189 Registry Content-Formats of [IANA.core-parameters]), a tag number has 190 already been allocated in Section 6.2 (see Appendix A for details and 191 examples). 193 3.2. CBOR Tag Wrapped 195 The CBOR Tag Wrapped method starts with the Self-described CBOR tag, 196 55799, as described in Section 3.4.6 of [RFC8949]. 198 A second CBOR Tag is then allocated to describe the specific Protocol 199 involved, as described above. 201 This proposal wraps the CBOR value as tags usually do. Applications 202 that need to send the CBOR value across a constrained link may wish 203 to remove the two tags if the use is implicitly understood. 205 Whether or not to remove the tag is a decision made by the CBOR 206 Protocol specification. 208 3.3. CBOR Tag Sequence 210 The CBOR Taq Sequence method makes use of CBOR Sequences as described 211 in [RFC8742]. 213 This method consists of two tags nested around a constant string for 214 a total of 12 bytes. 216 1. The file shall start with the Self-described CBOR Sequence tag, 217 55800. 219 2. The file shall continue with a CBOR tag, from the First Come 220 First Served space, which uniquely identifies the CBOR Protocol. 221 The use of a four-byte tag is encouraged. 223 3. The three byte CBOR byte string containing 0x42_4F_52. 225 The first part identifies the file as being a CBOR Sequence, and does 226 so with all the desirable properties explained in Section 3.4.6 of 227 [RFC8949]. Specifically, it does not seem to conflict with any known 228 file types, and it is not valid Unicode in any Unicode encoding. 230 The second part identifies which CBOR Protocol is used, as described 231 above. 233 The third part is represented as a constant byte sequence 234 0x43_42_4f_52, the ASCII characters "CBOR", which is the CBOR encoded 235 data item for the three byte sequence 0x42_4f_52 ('BOR' in diagnostic 236 notation). This is the data item that is being tagged. 238 The actual CBOR Protocol value then follows as the next data item(s) 239 in the CBOR sequence, without a need for any further specific tag. 240 The use of a CBOR Sequence allows the application to trivially remove 241 the first item with the two tags. 243 Should this file be reviewed by a human (directly in an editor, or in 244 a hexdump display), it will include the ASCII characters "CBOR" 245 prominently. This value is also included simply because the two tags 246 need to tag something. 248 4. Advice to Protocol Developers 250 This document introduces a choice between a CBOR Sequence and a 251 wrapped CBOR Tag. Which should a protocol designer use? 253 In this discussion, one assumes that there is an object stored in a 254 file, perhaps specified by a system operator in a configuration file. 256 For example: a private key used in COSE operations, a public key/ 257 certificate in C509 or CBOR format, a recorded sensor reading stored 258 for later transmission, or a COVID vaccination certificate that needs 259 to be displayed in QR code form. 261 Both the CBOR Tag Sequence and the wrapped tag can be trivially 262 removed by an application before sending the CBOR content out on the 263 wire. 265 The CBOR Tag Sequence is a little bit easier to remove as in most 266 cases, CBOR parsers will return it as a unit, and then return the 267 actual CBOR item, which could be anything at all, and could include 268 CBOR tags that _do_ need to be sent on wire. 270 On the other hand, having the CBOR Tag Sequence in the file requires 271 that all programs that expect to examine that file are able to skip 272 what appears to be an empty CBOR item. Programs which might not 273 expect the CBOR Tag Sequence, but which would operate without a 274 problem would include any program that expects to process CBOR 275 Sequences from the file. 277 As an example of where there was a problem with previous security 278 systems, "PEM" format certificate files grew to be able to contain 279 multiple certificates by simple concatenation. The PKCS1 format 280 could also contain a private key object followed by a one or more 281 certificate objects: but only when in PEM format. Annoyingly, when 282 in binary DER format (which like CBOR is self-delimiting), 283 concatenation of certificates was not compatible with most programs 284 as they did not expect to read more than one item in the file. 286 The use of CBOR Tag Wrapped format is easier to retrofit to an 287 existing format with existing and unchangeable on-disk format. This 288 new sequence of tags is expected to be trivially ignored by an 289 existing program when reading CBOR from disk, even if the program 290 only supports decoding a single data item (and not a CBOR sequence). 291 But, a naive program might also then transmit the additional tags 292 across the network. Removing the CBOR Tag Wrapped format requires 293 knowledge of the two tags involved. Other tags present might be 294 needed. 296 For a representation matching a specific media-type that is carried 297 in a CBOR byte string, the byte string head will already have to be 298 removed for use as such a representation, so it should be easy to 299 remove the enclosing tag heads as well. This is of particular 300 interest with the pre-defined tags provided by Appendix A for media- 301 types with CoAP Content-Format numbers. 303 Here are some considerations in the form of survey questions: 305 4.1. Is the on-wire format new? 307 If the on-wire format is new, then it could be specified with the 308 CBOR Tag Wrapped format if the extra eight bytes are not a problem. 309 The disk format is then identical to the on-wire format. 311 If the eight bytes are a problem (and they usually are if CBOR is 312 being considered), then the CBOR Tag Sequence format should be 313 adopted for on-disk storage. 315 4.2. Can many items be trivially concatenated? 317 If the programs that read the contents of the file already expect to 318 process all of the items in the file (not just the first), then the 319 CBOR Tag Sequence format may be easily retrofitted. 321 The program involved may throw errors or warnings on the CBOR Tag 322 Sequence if they have not yet been updated, but this may not be a 323 problem. If it is, then consideration should be given to CBOR Tag 324 Wrapped. 326 If only one item is ever expected in the file, the use of CBOR Tag 327 Sequence may present an implementation hurdle to programs that 328 previously just read a single value and used it. 330 4.3. Are there tags at the start? 332 If the Protocol expects to use other tags values at the top-level, 333 then it may be easier to explain if the CBOR Tag Sequence format is 334 used. 336 5. Security Considerations 338 This document provides a way to identify CBOR Protocol objects. 339 Clearly identifying CBOR contents on disk may have a variety of 340 impacts. 342 The most obvious is that it may allow malware to identify interesting 343 objects on disk, and then corrupt them. 345 6. IANA Considerations 347 Section 6.1 documents the allocation that was done for a CBOR tag to 348 be used in a CBOR sequence to identify the sequence (an example for 349 using this tag is found in Appendix B). Section 6.2 allocates a CBOR 350 tag for each actual or potential CoAP Content-Format number (examples 351 are in Appendix A). 353 6.1. CBOR Sequence Tag 355 IANA has allocated tag 55800 as the CBOR Sequence tag. This tag is 356 from the First Come/First Served area. 358 The value has been picked to have properties similar to the 55799 tag 359 (Section 3.4.6 of [RFC8949]). 361 The hexadecimal representation is: 0xd9_d9_f8. 363 This is not valid UTF-8: the first 0xd9 puts the value into the 364 three-byte value of UTF-8, but the 0xd9 as the second value is not a 365 valid second byte for UTF-8. 367 This is not valid UTF-16: the byte sequence 0xd9d9 (in either endian 368 order) puts this value into the UTF-16 high-half zone, which would 369 signal that this a 32-bit Unicode value. However, the following 370 16-bit big-endian value 0xf8.. is not a valid second sequence 371 according to [RFC2781]. On a little-endian system, it would be 372 necessary to examine the fourth byte to determine if it is valid. 373 That next byte is determined by the subsequent encoding, and 374 Section 3.4.6 of [RFC8949] has already determined that no valid CBOR 375 encodings result in a valid UTF-16. 377 Data Item: 379 byte string 381 Semantics: 382 indicates that the file contains CBOR Sequences 384 6.2. CBOR Tags for CoAP Content-Format Numbers 386 IANA is requested to allocate the tag numbers 1668546560 (0x63740000) 387 to 1668612095 (0x6374FFFF) as follows: 389 Data Item: 390 byte string 392 Semantics: 393 for each tag number NNNNNNNN, the representation of content-format 394 (RFC7252) NNNNNNNN-1668546560 396 Reference: 397 RFCthis 399 7. References 401 7.1. Normative References 403 [RFC8742] Bormann, C., "Concise Binary Object Representation (CBOR) 404 Sequences", RFC 8742, DOI 10.17487/RFC8742, February 2020, 405 . 407 [RFC8949] Bormann, C. and P. Hoffman, "Concise Binary Object 408 Representation (CBOR)", STD 94, RFC 8949, 409 DOI 10.17487/RFC8949, December 2020, 410 . 412 7.2. Informative References 414 [file] Wikipedia, "file (command)", 20 January 2021, 415 . 417 [I-D.ietf-cose-cbor-encoded-cert] 418 Mattsson, J. P., Selander, G., Raza, S., Höglund, J., and 419 M. Furuhed, "CBOR Encoded X.509 Certificates (C509 420 Certificates)", Work in Progress, Internet-Draft, draft- 421 ietf-cose-cbor-encoded-cert-02, 12 July 2021, 422 . 425 [I-D.ietf-rats-eat] 426 Mandyam, G., Lundblade, L., Ballesteros, M., and J. 427 O'Donoghue, "The Entity Attestation Token (EAT)", Work in 428 Progress, Internet-Draft, draft-ietf-rats-eat-10, 7 June 429 2021, . 432 [I-D.ietf-sacm-coswid] 433 Birkholz, H., Fitzgerald-McKay, J., Schmidt, C., and D. 434 Waltermire, "Concise Software Identification Tags", Work 435 in Progress, Internet-Draft, draft-ietf-sacm-coswid-18, 12 436 July 2021, . 439 [IANA.core-parameters] 440 IANA, "Constrained RESTful Environments (CoRE) 441 Parameters", 442 . 444 [RFC2781] Hoffman, P. and F. Yergeau, "UTF-16, an encoding of ISO 445 10646", RFC 2781, DOI 10.17487/RFC2781, February 2000, 446 . 448 [RFC7252] Shelby, Z., Hartke, K., and C. Bormann, "The Constrained 449 Application Protocol (CoAP)", RFC 7252, 450 DOI 10.17487/RFC7252, June 2014, 451 . 453 [RFC8152] Schaad, J., "CBOR Object Signing and Encryption (COSE)", 454 RFC 8152, DOI 10.17487/RFC8152, July 2017, 455 . 457 [RFC8610] Birkholz, H., Vigano, C., and C. Bormann, "Concise Data 458 Definition Language (CDDL): A Notational Convention to 459 Express Concise Binary Object Representation (CBOR) and 460 JSON Data Structures", RFC 8610, DOI 10.17487/RFC8610, 461 June 2019, . 463 Appendix A. CBOR Tags for CoAP Content Formats 465 Often, there is a need to identify a media type (or content type, 466 i.e., media type optionally used with parameters) that describes a 467 byte string in a CBOR data item. 469 Section 5.10.3 of [RFC7252] defines the concept of a Content-Format, 470 which is a short 16-bit unsigned integer that identifies a specific 471 content type (media type plus optionally parameters), optionally 472 together with a content encoding. 474 This specification allocates CBOR tag numbers 1668546560 (0x63740000) 475 to 1668612095 (0x6374FFFF) for the tagging of representations of 476 specific content formats. The tag content tagged with tag number 477 NNNNNNNN (in above range) is a byte string that is to be interpreted 478 as a representation of the content format NNNNNNNN-1668546560. 480 A.1. Content-Format Tag Examples 482 Registry Content-Formats of [IANA.core-parameters] defines content 483 formats that can be used as examples: 485 * Content-Format 432 stands for media type application/td+json (no 486 parameters). The corresponding tag number is 1668546992 (i.e., 487 1668546560+432). 489 So the following CDDL snippet can be used to identify application/ 490 td+json representations: 492 td-json = #6.1668546992(bstr) 494 Note that a byte string is used as the type of the tag content, 495 because a media type representation in general can be any byte 496 string. 498 * Content-Format 11050 stands for media type application/json in 499 deflate encoding. 501 The corresponding tag number is 1668557610 (i.e., 502 1668546560+11050). 504 So the following CDDL snippet can be used to identify application/ 505 json representations compressed in deflate encoding: 507 json-deflate = #6.1668557610(bstr) 509 The byte string is appropriate here as the type for the tag 510 content, because the compressed form is an instance of a general 511 byte string. 513 Appendix B. Example from Openswan 515 The Openswan IPsec project has a daemon ("pluto"), and two control 516 programs ("addconn", and "whack"). They communicate via a Unix- 517 domain socket, over which a C-structure containing pointers to 518 strings is serialized using a bespoke mechanism. This is normally 519 not a problem as the structure is compiled by the same compiler; but 520 when there are upgrades it is possible for the daemon and the control 521 programs to get out of sync by the bespoke serialization. As a 522 result, there are extra compensations to deal with shutting the 523 daemon down. During testing it is sometimes the case that upgrades 524 are backed out. 526 In addition, when doing unit testing, the easiest way to load policy 527 is to use the normal policy reading process, but that is not normally 528 loaded in the daemon. Instead the IPC that is normally sent across 529 the wire is compiled/serialized and placed in a file. The above 530 magic number is included in the file, and also on the IPC in order to 531 distinguish the "shutdown" command CBOR operation. 533 In order to reduce the problems due to serialization, the 534 serialization is being changed to CBOR. Additionally, this change 535 allows the IPC to be described by CDDL, and for any language that 536 encode to CBOR can be used. 538 IANA has allocated the tag 1330664270, or 0x4f_50_53_4e for this 539 purpose. As a result, each file and each IPC is prefixed with a CBOR 540 TAG Sequence. 542 In diagnostic notation: 544 55800(1330664270(h'424F52')) 546 Or in hex: 548 00000000 d9 d9 f8 da 4f 50 53 4e 43 42 4f 52 |....OPSNCBOR| 550 Appendix C. Changelog 552 Acknowledgements 554 The CBOR WG brainstormed this protocol on January 20, 2021 via a 555 number of productive email exchanges on the mailing list. 557 Contributors 559 Josef 'Jeff' Sipek 561 Email: jeffpc@josefsipek.net 563 Authors' Addresses 565 Michael Richardson 566 Sandelman Software Works 568 Email: mcr+ietf@sandelman.ca 569 Carsten Bormann 570 Universität Bremen TZI 571 Germany 573 Email: cabo@tzi.org