idnits 2.17.1 draft-hallambaker-mesh-dare-13.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack an Authors' Addresses Section. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (19 September 2021) is 948 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Missing Reference: 'EOF' is mentioned on line 759, but not defined ** Obsolete normative reference: RFC 7159 (Obsoleted by RFC 8259) Summary: 2 errors (**), 0 flaws (~~), 2 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group P. M. Hallam-Baker 3 Internet-Draft ThresholdSecrets.com 4 Intended status: Informational 19 September 2021 5 Expires: 23 March 2022 7 Mathematical Mesh 3.0 Part III : Data At Rest Encryption (DARE) 8 draft-hallambaker-mesh-dare-13 10 Abstract 12 This document describes the Data At Rest Encryption (DARE) Envelope 13 and Container syntax. 15 The DARE Envelope syntax is used to digitally sign, digest, 16 authenticate, or encrypt arbitrary content data. 18 The DARE Container syntax describes an append-only sequence of 19 entries, each containing a DARE Envelope. DARE Containers may 20 support cryptographic integrity verification of the entire data 21 container content by means of a Merkle tree. 23 [Note to Readers] 25 Discussion of this draft takes place on the MATHMESH mailing list 26 (mathmesh@ietf.org), which is archived at 27 https://mailarchive.ietf.org/arch/search/?email_list=mathmesh. 29 This document is also available online at 30 http://mathmesh.com/Documents/draft-hallambaker-mesh-dare.html. 32 Status of This Memo 34 This Internet-Draft is submitted in full conformance with the 35 provisions of BCP 78 and BCP 79. 37 Internet-Drafts are working documents of the Internet Engineering 38 Task Force (IETF). Note that other groups may also distribute 39 working documents as Internet-Drafts. The list of current Internet- 40 Drafts is at https://datatracker.ietf.org/drafts/current/. 42 Internet-Drafts are draft documents valid for a maximum of six months 43 and may be updated, replaced, or obsoleted by other documents at any 44 time. It is inappropriate to use Internet-Drafts as reference 45 material or to cite them other than as "work in progress." 47 This Internet-Draft will expire on 23 March 2022. 49 Copyright Notice 51 Copyright (c) 2021 IETF Trust and the persons identified as the 52 document authors. All rights reserved. 54 This document is subject to BCP 78 and the IETF Trust's Legal 55 Provisions Relating to IETF Documents (https://trustee.ietf.org/ 56 license-info) in effect on the date of publication of this document. 57 Please review these documents carefully, as they describe your rights 58 and restrictions with respect to this document. 60 Table of Contents 62 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 4 63 1.1. Encryption and Integrity . . . . . . . . . . . . . . . . 5 64 1.1.1. Key Exchange . . . . . . . . . . . . . . . . . . . . 5 65 1.1.2. Data Erasure . . . . . . . . . . . . . . . . . . . . 6 66 1.2. Signature . . . . . . . . . . . . . . . . . . . . . . . . 6 67 1.2.1. Signing Individual Plaintext Envelopes . . . . . . . 7 68 1.2.2. Signing Individual Encrypted Envelopes . . . . . . . 7 69 1.2.3. Signing sequences of envelopes . . . . . . . . . . . 7 70 1.3. Container . . . . . . . . . . . . . . . . . . . . . . . . 8 71 1.3.1. Container Format . . . . . . . . . . . . . . . . . . 8 72 1.3.2. Write . . . . . . . . . . . . . . . . . . . . . . . . 9 73 1.3.3. Encryption and Authentication . . . . . . . . . . . . 9 74 1.3.4. Integrity and Signature . . . . . . . . . . . . . . . 10 75 1.3.5. Redaction . . . . . . . . . . . . . . . . . . . . . . 10 76 1.3.6. Alternative approaches . . . . . . . . . . . . . . . 11 77 1.3.7. Efficiency . . . . . . . . . . . . . . . . . . . . . 11 78 2. Definitions . . . . . . . . . . . . . . . . . . . . . . . . . 12 79 2.1. Related Specifications . . . . . . . . . . . . . . . . . 12 80 2.2. Requirements Language . . . . . . . . . . . . . . . . . . 12 81 2.3. Defined terms . . . . . . . . . . . . . . . . . . . . . . 13 82 3. DARE Envelope Architecture . . . . . . . . . . . . . . . . . 14 83 3.1. Processing Considerations . . . . . . . . . . . . . . . . 15 84 3.2. Content Metadata and Annotations . . . . . . . . . . . . 15 85 3.3. Encoded Data Sequence . . . . . . . . . . . . . . . . . . 16 86 3.4. Encryption and Integrity . . . . . . . . . . . . . . . . 17 87 3.4.1. Key Exchange . . . . . . . . . . . . . . . . . . . . 17 88 3.4.2. Key Identifiers . . . . . . . . . . . . . . . . . . . 18 89 3.4.3. Salt Derivation . . . . . . . . . . . . . . . . . . . 19 90 3.4.4. Key Derivation . . . . . . . . . . . . . . . . . . . 19 91 3.5. Signature . . . . . . . . . . . . . . . . . . . . . . . . 20 92 3.6. Algorithms . . . . . . . . . . . . . . . . . . . . . . . 20 93 3.6.1. Field: kwd . . . . . . . . . . . . . . . . . . . . . 20 94 4. DARE Container Architecture . . . . . . . . . . . . . . . . . 20 95 4.1. Container Navigation . . . . . . . . . . . . . . . . . . 21 96 4.1.1. Tree . . . . . . . . . . . . . . . . . . . . . . . . 22 97 4.1.2. Position Index . . . . . . . . . . . . . . . . . . . 22 98 4.1.3. Metadata Index . . . . . . . . . . . . . . . . . . . 22 99 4.2. Integrity Mechanisms . . . . . . . . . . . . . . . . . . 22 100 4.2.1. Digest Chain calculation . . . . . . . . . . . . . . 22 101 4.2.2. Binary Merkle tree calculation . . . . . . . . . . . 23 102 4.2.3. Signature . . . . . . . . . . . . . . . . . . . . . . 23 103 5. DARE Schema . . . . . . . . . . . . . . . . . . . . . . . . . 24 104 5.1. Envelope Classes . . . . . . . . . . . . . . . . . . . . 24 105 5.1.1. Structure: DareEnvelopeSequence . . . . . . . . . . . 24 106 5.2. Header and Trailer Classes . . . . . . . . . . . . . . . 24 107 5.2.1. Structure: DareTrailer . . . . . . . . . . . . . . . 25 108 5.2.2. Structure: DareHeader . . . . . . . . . . . . . . . . 25 109 5.2.3. Structure: ContentMeta . . . . . . . . . . . . . . . 26 110 5.3. Cryptographic Data . . . . . . . . . . . . . . . . . . . 27 111 5.3.1. Structure: DareSignature . . . . . . . . . . . . . . 27 112 5.3.2. Structure: X509Certificate . . . . . . . . . . . . . 28 113 5.3.3. Structure: DareRecipient . . . . . . . . . . . . . . 28 114 5.3.4. Structure: DarePolicy . . . . . . . . . . . . . . . . 28 115 5.3.5. Structure: FileEntry . . . . . . . . . . . . . . . . 29 116 6. DARE Container Schema . . . . . . . . . . . . . . . . . . . . 29 117 6.1. Container Headers . . . . . . . . . . . . . . . . . . . . 29 118 6.1.1. Structure: SequenceInfo . . . . . . . . . . . . . . . 29 119 6.2. Index Structures . . . . . . . . . . . . . . . . . . . . 30 120 6.2.1. Structure: SequenceIndex . . . . . . . . . . . . . . 30 121 6.2.2. Structure: IndexPosition . . . . . . . . . . . . . . 31 122 6.2.3. Structure: KeyValue . . . . . . . . . . . . . . . . . 31 123 7. Dare Container Applications . . . . . . . . . . . . . . . . . 31 124 7.1. Catalog . . . . . . . . . . . . . . . . . . . . . . . . . 31 125 7.2. Spool . . . . . . . . . . . . . . . . . . . . . . . . . . 32 126 7.3. Archive . . . . . . . . . . . . . . . . . . . . . . . . . 32 127 8. Future Work . . . . . . . . . . . . . . . . . . . . . . . . . 33 128 8.1. Terminal integrity check . . . . . . . . . . . . . . . . 33 129 8.2. Terminal index record . . . . . . . . . . . . . . . . . . 33 130 8.3. Deferred indexing . . . . . . . . . . . . . . . . . . . . 33 131 9. Security Considerations . . . . . . . . . . . . . . . . . . . 34 132 9.1. Encryption/Signature nesting . . . . . . . . . . . . . . 34 133 9.2. Side channel . . . . . . . . . . . . . . . . . . . . . . 34 134 9.3. Salt reuse . . . . . . . . . . . . . . . . . . . . . . . 34 135 10. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 34 136 11. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 34 137 12. Appendix A: DARE Envelope Examples and Test Vectors . . . . . 34 138 13. Test Examples . . . . . . . . . . . . . . . . . . . . . . . . 34 139 13.1. Plaintext Message . . . . . . . . . . . . . . . . . . . 35 140 13.2. Plaintext Message with EDS . . . . . . . . . . . . . . . 35 141 13.3. Encrypted Message . . . . . . . . . . . . . . . . . . . 35 142 13.4. Signed Message . . . . . . . . . . . . . . . . . . . . . 37 143 13.5. Signed and Encrypted Message . . . . . . . . . . . . . . 38 144 14. Appendix B: DARE Container Examples and Test Vectors . . . . 38 145 14.1. Simple sequence . . . . . . . . . . . . . . . . . . . . 39 146 14.2. Payload and chain digests . . . . . . . . . . . . . . . 39 147 14.3. Merkle Tree . . . . . . . . . . . . . . . . . . . . . . 41 148 14.4. Signed sequence . . . . . . . . . . . . . . . . . . . . 44 149 14.5. Encrypted sequence . . . . . . . . . . . . . . . . . . . 45 150 15. Appendix C: Previous Frame Function . . . . . . . . . . . . . 49 151 16. Appendix D: Outstanding Issues . . . . . . . . . . . . . . . 49 152 17. Normative References . . . . . . . . . . . . . . . . . . . . 50 153 18. Informative References . . . . . . . . . . . . . . . . . . . 52 155 1. Introduction 157 This document describes the Data At Rest Encryption (DARE) Envelope 158 and Container Syntax. The DARE Envelope syntax is used to digitally 159 sign, digest, authenticate, or encrypt arbitrary message content. 160 The DARE Container syntax describes an append-only sequence of data 161 frames, each containing a DARE Envelope that supports efficient 162 incremental signature and encryption. 164 The DARE Envelope Syntax is based on a subset of the JSON Web 165 Signature [RFC7515] and JSON Web Encryption [RFC7516] standards and 166 shares many fields and semantics. The processing model and data 167 structures have been streamlined to remove alternative means of 168 specifying the same content and to enable multiple data sequences to 169 be signed and encrypted under a single master encryption key without 170 compromise to security. 172 A DARE Envelope consists of a _Header_, _Payload_ and an optional 173 _Trailer_. To enable single pass encoding and decoding, the Header 174 contains all the information required to perform cryptographic 175 processing of the Payload and authentication data (digest, MAC, 176 signature values) MAY be deferred to the Trailer section. 178 A DARE Container is an append-only log format consisting of a 179 sequence of frames. Cryptographic enhancements (signature, 180 encryption) may be applied to individual frames or to sets of frames. 181 Thus, a single key exchange may be used to provide a master key to 182 encrypt multiple frames and a single signature may be used to 183 authenticate all the frames in the container up to and including the 184 frame in which the signature is presented. 186 The DARE Envelope syntax may be used either as a standalone 187 cryptographic message syntax or as a means of presenting a single 188 DARE Container frame together with the complete cryptographic context 189 required to verify the contents and decrypt them. 191 1.1. Encryption and Integrity 193 A key innovation in the DARE Envelope Syntax is the separation of key 194 exchange and data encryption operations so that a Master Key (MK) 195 established in a single exchange to be applied to multiple data 196 sequences. This means that a single public key operation MAY be used 197 to encrypt and/or authenticate multiple parts of the same DARE 198 Envelope or multiple frames in a DARE Container. 200 To avoid reuse of the key and to avoid the need to communicate 201 separate IVs, each octet sequence is encrypted under a different 202 encryption key (and IV if required) derived from the Master Key by 203 means of a salt that is unique for each octet sequence that is 204 encrypted. The same approach is used to generate keys for 205 calculating a MAC over the octet sequence if required. This approach 206 allows encryption and integrity protections to be applied to the 207 envelope payload, to header or trailer fields or to application 208 defined Enhanced Data Sequences in the header or trailer. 210 1.1.1. Key Exchange 212 Traditional cryptographic containers describe the application of a 213 single key exchange to encryption of a single octet sequence. 214 Examples include PCKS#7/CMS [RFC2315], OpenPGP [RFC4880] and JSON Web 215 Encryption [RFC7516]. 217 To encrypt data using RSA, the encoder first generates a random 218 encryption key and initialization vector (IV). The encryption key is 219 encrypted under the public key of each recipient to create a per- 220 recipient decryption entry. The encryption key, plaintext and IV are 221 used to generate the ciphertext (figure 1). 223 (Artwork only available as svg: No external link available, see 224 draft-hallambaker-mesh-dare-13.html for artwork.) 226 Figure 1: Monolithic Key Exchange and Encrypt 228 This approach is adequate for the task of encrypting a single octet 229 stream. It is less than satisfactory when encrypting multiple octet 230 streams or very long streams for which a rekeying operation is 231 desirable. 233 In the DARE approach, key exchange and key derivation are separate 234 operations and keys MAY be derived for encryption or integrity 235 purposes or both. A single key exchange MAY be used to derive keys 236 to apply encryption and integrity enhancements to multiple data 237 sequences. 239 The DARE key exchange begins with the same key exchange used to 240 produce the CEK in JWE but instead of using the CEK to encipher data 241 directly, it is used as one of the inputs to a Key Derivation 242 Function (KDF) that is used to derive parameters for each block of 243 data to be encrypted. To avoid the need to introduce additional 244 terminology, the term 'CEK' is still used to describe the output of 245 the key agreement algorithm (including key unwrapping if required) 246 but it is more appropriately described as a Master Key (figure 2). 248 (Artwork only available as svg: No external link available, see 249 draft-hallambaker-mesh-dare-13.html for artwork.) 251 Figure 2: Exchange of Master Key 253 A Master Key may be used to encrypt any number of data items. Each 254 data item is encrypted under a different encryption key and IV (if 255 required). This data is derived from the Master Key using the HKDF 256 function [RFC5869] using a different salt for each data item and 257 separate info tags for each cryptographic function (figure 3). 259 (Artwork only available as svg: No external link available, see 260 draft-hallambaker-mesh-dare-13.html for artwork.) 262 Figure 3: Data item encryption under Master Key and per-item salt. 264 This approach to encryption offers considerably greater flexibility 265 allowing the same format for data item encryption to be applied at 266 the transport, message or field level. 268 1.1.2. Data Erasure 270 Each encrypted DARE Envelope specifies a unique Master Salt value of 271 at least 128 bits which is used to derive the salt values used to 272 derive cryptographic keys for the envelope payload and annotations. 274 Erasure of the Master Salt value MAY be used to effectively render 275 the envelope payload and annotations undecipherable without altering 276 the envelope payload data. The work factor for decryption will be 277 O(2^128) even if the decryption key is compromised. 279 1.2. Signature 281 As with encryption, DARE Envelope signatures MAY be applied to an 282 individual envelope or a sequence of envelope. 284 1.2.1. Signing Individual Plaintext Envelopes 286 When an individual plaintext envelope is signed, the digest value 287 used to create the signature is calculated over the binary value of 288 the payload data. That is, the value of the payload before the 289 encoding (Base-64, JSON-B) is applied. 291 1.2.2. Signing Individual Encrypted Envelopes 293 When an individual plaintext envelope is signed, the digest value 294 used to create the signature is calculated over the binary value of 295 the payload data. That is, the value of the payload after encryption 296 but before the encoding (Base-64, JSON-B) is applied. 298 Use of signing and encryption in combination presents the risk of 299 subtle attacks depending on the order in which signing and encryption 300 take place [Davis2001]. 302 Na?ve approaches in which an envelope is encrypted and then signed 303 present the possibility of a surreptitious forwarding attack. For 304 example, Alice signs an envelope and sends it to Mallet who then 305 strips off Alice's signature and sends the envelope to Bob. 307 Na?ve approaches in which an envelope is signed and then encrypted 308 present the possibility of an attacker claiming authorship of a 309 ciphertext. For example, Alice encrypts a ciphertext for Bob and 310 then signs it. Mallet then intercepts the envelope and sends it to 311 Bob. 313 While neither attack is a concern in all applications, both attacks 314 pose potential hazards for the unwary and require close inspection of 315 application protocol design to avoid exploitation. 317 To prevent these attacks, each signature on an envelope that is 318 signed and encrypted MUST include a witness value that is calculated 319 by applying a MAC function to the signature value as described in 320 section XXX. 322 1.2.3. Signing sequences of envelopes 324 To sign multiple envelopes with a single signature, we first 325 construct a Merkle tree of the envelope payload digest values and 326 then sign the root of the Merkle tree. 328 [This is not yet implemented but will be soon.] 330 1.3. Container 332 DARE Container is a message and file syntax that allows a sequence of 333 data frames to be represented with cryptographic integrity, 334 signature, and encryption enhancements to be constructed in an append 335 only format. 337 The format is designed to meet the requirements of a wide range of 338 use cases including: 340 * Recording transactions in persistent storage. 342 * Synchronizing transaction logs between hosts. 344 * File archive. 346 * Message spool. 348 * Signing and encrypting single data items. 350 * Incremental encryption and authentication of server logs. 352 1.3.1. Container Format 354 A Container consists of a sequence of variable length Frames. Each 355 frame consists of a forward length indicator, the framed data and a 356 reverse length indicator. The reverse length indicator is written 357 out backwards allowing the length and thus the frame to be read in 358 the reverse direction: 360 (Artwork only available as svg: No external link available, see 361 draft-hallambaker-mesh-dare-13.html for artwork.) 363 Figure 4: JBCD Bidirectional Frame 365 Each frame contains a single DARE Envelope consisting of a Header, 366 Payload and Trailer (if required). The first frame in a container 367 describes the container format options and defaults. These include 368 the range of encoding options for frame metadata supported and the 369 container profiles to which the container conforms. 371 All internal data formats support use of pointers of up to 64 bits 372 allowing containers of up to 18 exabytes to be written. 374 Five container types are currently specified: 376 Simple The container does not provide any index or content integrity 377 checks. 379 Tree Frame headers contain entries that specify the start position 380 of previous frames at the apex of the immediately enclosing binary 381 tree. This enables efficient random access to any frame in the 382 file. 384 Digest Each frame trailer contains a "PayloadDigest" field. 385 Modification of the payload will cause verification of the 386 "PayloadDigest" value to fail on that frame. 388 Chain Each frame trailer contains "PayloadDigest" and "ChainDigest" 389 fields allowing modifications to the payload data to be detected. 390 Modification of the payload will cause verification of the 391 "PayloadDigest" value to fail on that frame and verification of 392 the "ChainDigest" value to fail on all subsequent frames. 394 Merkle Tree Frame headers contain entries that specify the start 395 position of previous frames at the apex of the immediately 396 enclosing binary tree. Frame Trailers contain TreeDigestPartial 397 and TreeDigestFinal entries forming a Merkle digest tree. 399 1.3.2. Write 401 In normal circumstances, Containers are written as an append only 402 log. As with Envelopes, integrity information (payload digest, 403 signatures) is written to the entry trailer. Thus, large payloads 404 may be written without the need to buffer the payload data _provided 405 that_ the content length is known in advance. 407 Should exceptional circumstances require, Container entries MAY be 408 erased by overwriting the Payload and/or parts of the Header content 409 without compromising the ability to verify other entries in the 410 container. If the entry Payload is encrypted, it is sufficient to 411 erase the container salt value to render the container entry 412 effectively inaccessible (though recovery might still be possible if 413 the original salt value can be recovered from the storage media. 415 1.3.3. Encryption and Authentication 417 Frame payloads and associated attributes MAY be encrypted and/or 418 authenticated in the same manner as Envelopes. 420 _Incremental encryption_ is supported allowing encryption parameters 421 from a single public key exchange operation to be applied to encrypt 422 multiple frames. The public key exchange information is specified in 423 the first encrypted frame and subsequent frames encrypted under those 424 parameters specify the location at which the key exchange information 425 is to be found by means of the ExchangePosition field which MUST 426 specify a location that is earlier in the file. 428 To avoid cryptographic vulnerabilities resulting from key re-use, the 429 DARE key exchange requires that each encrypted sequence use an 430 encryption key and initialization vector derived from the master key 431 established in the public key exchange by means of a unique salt. 433 Each Envelope and by extension, each Container frame MUST specify a 434 unique salt value of at least 128 bits. Since the encryption key is 435 derived from the salt value by means of a Key Derivation Function, 436 erasure of the salt MAY be used as a means of rendering the payload 437 plaintext value inaccessible without changing the payload value. 439 1.3.4. Integrity and Signature 441 Signatures MAY be applied to a payload digest, the final digest in a 442 chain or tree. The chain and tree digest modes allow a single 443 signature to be used to authenticate all frame payloads in a 444 container. 446 The tree signature mode is particularly suited to applications such 447 as file archives as it allows files to be verified individually 448 without requiring the signer to sign each individually. Furthermore, 449 in applications such as code signing, it allows a single signature to 450 be used to verify both the integrity of the code and its membership 451 of the distribution. 453 As with DARE Envelope, the signature mechanism does not specify the 454 interpretation of the signature semantics. The presence of a 455 signature demonstrates that the holder of the private key applied it 456 to the specified digest value but not their motive for doing so. 457 Describing such semantics is beyond the scope of this document and is 458 deferred to future work. 460 1.3.5. Redaction 462 The chief disadvantage of using an append-only format is that 463 containers only increase in size. In many applications, much of the 464 data in the container becomes redundant or obsolete and a process 465 analogous to garbage collection is required. This process is called 466 _redaction_. 468 The simplest method of redaction is to create a new container and 469 sequentially copy each entry from the old container to the new, 470 discarding redundant frames and obsolete header information. 472 For example, partial index records may be consolidated into a single 473 index record placed in the last frame of the container. Unnecessary 474 signature and integrity data may be discarded and so on. 476 While redaction could in principle be effected by moving data in- 477 place in the existing container, supporting this approach in a robust 478 fashion is considerably more complex as it requires backward 479 references in subsequent frames to be overridden as each frame is 480 moved. 482 1.3.6. Alternative approaches 484 Many file proprietary formats are in use that support some or all of 485 these capabilities but only a handful have public, let alone open, 486 standards. DARE Container is designed to provide a superset of the 487 capabilities of existing message and file syntaxes, including: 489 * Cryptographic Message Syntax [RFC5652] defines a syntax used to 490 digitally sign, digest, authenticate, or encrypt arbitrary message 491 content. 493 * The.ZIP File Format specification [ZIPFILE] developed by Phil 494 Katz. 496 * The BitCoin Block chain [BLOCKCHAIN]. 498 * JSON Web Encryption and JSON Web Signature 500 Attempting to make use of these specifications in a layered fashion 501 would require at least three separate encoders and introduce 502 unnecessary complexity. Furthermore, there is considerable overlap 503 between the specifications providing multiple means of achieving the 504 same ends, all of which must be supported if decoders are to work 505 reliably. 507 1.3.7. Efficiency 509 Every data format represents a compromise between different concerns, 510 in particular: 512 Compactness The space required to record data in the encoding. 514 Memory Overhead The additional volatile storage (RAM) required to 515 maintain indexes etc. to support efficient retrieval operations. 517 Number of Operations The number of operations required to retrieve 518 data from or append data to an existing encoded sequence. 520 Number of Disk Seek Operations Optimizing the response time of 521 magnetic storage media to random access read requests has 522 traditionally been one of the central concerns of database design. 523 The DARE Container format is designed to the assumption that this 524 will cease to be a concern as solid state media replaces magnetic. 526 While the cost of storage of all types has declined rapidly over the 527 past decades, so has the amount of data to be stored. DARE Container 528 represents a pragmatic balance of these considerations for current 529 technology. In particular, since payload volumes are likely to be 530 very large, memory and operational efficiency are considered higher 531 priorities than compactness. 533 2. Definitions 535 2.1. Related Specifications 537 The DARE Envelope and Container formats are based on the following 538 existing standards and specifications. 540 Object serialization The JSON-B [draft-hallambaker-jsonbcd] encoding 541 is used for object serialization. This encoding is an extension 542 of the JavaScript Object Notation (JSON) [RFC7159]. 544 Message syntax The cryptographic processing model is based on JSON 545 Web Signature (JWS) [RFC7515], JSON Web Encryption (JWE) [RFC7516] 546 and JSON Web Key (JWK) [RFC7517]. 548 Cryptographic primitives. The HMAC-based Extract-and-Expand Key 549 Derivation Function [RFC5869] and Advanced Encryption Standard 550 (AES) Key Wrap with Padding Algorithm [RFC3394] are used. 552 The Uniform Data Fingerprint method of presenting data digests is 553 used for key identifiers and other purposes 554 [draft-hallambaker-mesh-udf]. 556 Cryptographic algorithms The cryptographic algorithms and 557 identifiers described in JSON Web Algorithms (JWA) [RFC7518] are 558 used together with additional algorithms as defined in the JSON 559 Object Signing and Encryption IANA registry [IANAJOSE]. 561 2.2. Requirements Language 563 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 564 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 565 document are to be interpreted as described in [RFC2119]. 567 2.3. Defined terms 569 The terms "Authentication Tag", "Content Encryption Key", "Key 570 Management Mode", "Key Encryption", "Direct Key Agreement", "Key 571 Agreement with Key Wrapping" and "Direct Encryption" are defined in 572 the JWE specification [RFC7516]. 574 The terms "Authentication", "Ciphertext", "Digital Signature", 575 "Encryption", "Initialization Vector (IV)", "Message Authentication 576 Code (MAC)", "Plaintext" and "Salt" are defined by the Internet 577 Security Glossary, Version 2 [RFC4949]. 579 Annotated Envelope A DARE Envelope that contains an "Annotations" 580 field with at least one entry. 582 Authentication Data A Message Authentication Code or authentication 583 tag. 585 Complete Envelope A DARE envelope that contains the key exchange 586 information necessary for the intended recipient(s) to decrypt it. 588 Detached Envelope A DARE envelope that does not contain the key 589 exchange information necessary for the intended recipient(s) to 590 decrypt it. 592 Encryption Context The master key, encryption algorithms and 593 associated parameters used to generate a set of one or more 594 enhanced data sequences. 596 Encoded data sequence (EDS) A sequence consisting of a salt, content 597 data and authentication data (if required by the encryption 598 context). 600 Enhancement Applying a cryptographic operation to a data sequence. 601 This includes encryption, authentication and both at the same 602 time. 604 Generator The party that generates a DARE envelope. 606 Group Encryption Key A key used to encrypt data to be read by a 607 group of users. This is typically achieved by means of some form 608 of proxy re-encryption or distributed key generation. 610 Group Encryption Key Identifier A key identifier for a group 611 encryption key. 613 Master Key (MK) The master secret from which keys are derived for 614 authenticating enhanced data sequences. 616 Recipient Any party that receives and processes at least some part 617 of a DARE envelope. 619 Related Envelope A set of DARE envelopes that share the same key 620 exchange information and hence the same Master Key. 622 Uniform Data Fingerprint (UDF) The means of presenting the result of 623 a cryptographic digest function over a data sequence and content 624 type identifier specified in the Uniform Data Fingerprint 625 specification [draft-hallambaker-mesh-udf] 627 3. DARE Envelope Architecture 629 A DARE Envelope is a sequence of three parts: 631 Header A JSON object containing information a reader requires to 632 begin processing the envelope. 634 Payload An array of octets. 636 Trailer A JSON object containing information calculated from the 637 envelope payload. 639 For example, the following sequence is a JSON encoded Envelope with 640 an empty header, a payload of zero length and an empty trailer: 642 [ {}, "", {} ] 644 DARE Envelopes MAY be encoded using JSON serialization or a binary 645 serialization for greater efficiency. 647 JSON [RFC7159] Offers compatibility with applications and libraries 648 that support JSON. Payload data is encoded using Base64 incurring 649 a 33% overhead. 651 JSON-B [draft-hallambaker-jsonbcd] A superset of JSON encoding that 652 permits binary data to be encoded as a sequence of length-data 653 segments. This avoids the Base64 overhead incurred by JSON 654 encoding. Since JSON-B is a superset of JSON encoding, an 655 application can use a single decoder for either format. 657 JSON-C [draft-hallambaker-jsonbcd] A superset of JSON-C which 658 provides additional efficiency by allowing field tags and other 659 repeated string data to be encoded by reference to a dictionary. 660 Since JSON-C is a superset of JSON and JSON-B encodings, an 661 application can use a single decoder for all three formats. 663 DARE Envelope processors MUST support JSON serialization and SHOULD 664 support JSON-B serialization. 666 3.1. Processing Considerations 668 The DARE Envelope Syntax supports single pass encoding and decoding 669 without buffering of data. All the information required to begin 670 processing a DARE envelope (key agreement information, digest 671 algorithms), is provided in the envelope header. All the information 672 that is derived from envelope processing (authentication codes, 673 digest values, signatures) is presented in the envelope trailer. 675 The choice of envelope encoding does not affect the semantics of 676 envelope processing. A DARE Envelope MAY be reserialized under the 677 same serialization or converted from any of the specified 678 serialization to any other serialization without changing the 679 semantics or integrity properties of the envelope. 681 3.2. Content Metadata and Annotations 683 A header MAY contain header fields describing the payload content. 684 These include: 686 ContentType Specifies the IANA Media Type [RFC6838]. 688 Annotations A list of Encoded Data Sequences that provide 689 application specific annotations to the envelope. 691 For example, consider the following mail message: 693 From: Alice@example.com 694 To: bob@example.com 695 Subject: TOP-SECRET Product Launch Today! 697 The CEO told me the product launch is today. Tell no-one! 699 Existing encryption approaches require that header fields such as the 700 subject line be encrypted with the body of the message or not 701 encrypted at all. Neither approach is satisfactory. In this 702 example, the subject line gives away important information that the 703 sender probably assumed would be encrypted. But if the subject line 704 is encrypted together with the message body, a mail client must 705 retrieve at least part of the message body to provide a 'folder' 706 view. 708 The plaintext form of the equivalent DARE Message encoding is: 710 [{ 711 "annotations":["iAEAiBdGcm9tOiBBbGljZUBleGFtcGxlLmNvbQ", 712 "iAEBiBNUbzogYm9iQGV4YW1wbGUuY29t", 713 "iAECiClTdWJqZWN0OiBUT1AtU0VDUkVUIFByb2R1Y3QgTGF1bmNoIFRvZG 714 F5IQ" 715 ], 716 "ContentMetaData":"ewogICJjdHkiOiAiYXBwbGljYXRpb24vZXhhbXBsZS 717 1tYWlsIn0"}, 718 "VGhlIENFTyB0b2xkIG1lIHRoZSBwcm9kdWN0IGxhdW5jaCBpcyB0b2RheS4gVG 719 VsbCBuby1vbmUh" 720 ] 722 This contains the same information as before but the mail message 723 headers are now presented as a list of Encoded Data Sequences. 725 3.3. Encoded Data Sequence 727 An encoded data sequence (EDS) is a sequence of octets that encodes a 728 data sequence according to cryptographic enhancements specified in 729 the context in which it is presented. An EDS MAY be encrypted and 730 MAY be authenticated by means of a MAC. The keys and other 731 cryptographic parameters used to apply these enhancements are derived 732 from the cryptographic context and a Salt prefix specified in the EDS 733 itself. 735 An EDS sequence contains exactly three binary fields encoded in 736 JSON-B serialization as follows: 738 Salt Prefix A sequence of octets used to derive the encryption key, 739 Initialization Vector and MAC key as required. 741 Body The plaintext or encrypted content. 743 Authentication Tag The authentication code value in the case that 744 the cryptographic context specifies use of authenticated 745 encryption or a MAC, otherwise is a zero-length field. 747 Requiring all three fields to be present, even in cases where they 748 are unnecessary simplifies processing at the cost of up to six 749 additional data bytes. 751 The encoding of the 'From' header of the previous example as a 752 plaintext EDS is as follows: 754 88 01 755 00 756 88 17 757 46 72 6f 6d 3a 20 41 6c 69 63 65 40 65 78 61 6d 758 70 6c 65 2e 63 6f 6d 759 [EOF] 761 3.4. Encryption and Integrity 763 Encryption and integrity protections MAY be applied to any DARE 764 Envelope Payload and Annotations. 766 The following is an encrypted version of the message shown earlier. 767 The payload and annotations have both increased in size as a result 768 of the block cipher padding. The header now includes Recipients and 769 Salt fields to enable the content to be decoded. 771 [{ 772 "enc":"A256CBC", 773 "kid":"EBQD-DR7C-4YOS-EEO2-BDAY-5YWL-HZQS", 774 "Salt":"VbqZd7LxspE1ns0vTRCtew", 775 "annotations":["iAEAiCBUkMuqTusi2Y7CVPpbXJsBbhG_CiUSz4RJ8CB6Jg 776 FExg", 777 "iAEBiCDojkQmvG6_d4V4-aCHOeKAu9QFkGdcqX1MuuO0P4wUaQ", 778 "iAECiDDmXF5TSZRWnGNE0fCBKpuNYOAuH8tab2-v6Y5JDjmzHsJyWtJaS6 779 46fySmWTob6BQ" 780 ], 781 "recipients":[{ 782 "kid":"MDW3-XKC7-T2YL-2WCJ-C5VK-46QT-CQP4", 783 "epk":{ 784 "PublicKeyECDH":{ 785 "crv":"Ed25519", 786 "Public":"prvRHMk13IigvR8yW5WSjJXVDPrdYo6oLGXl6iP5908"}}, 787 "wmk":"P_RC1ZLYDMVbbyqGJmfG_1riU-_yYt_d_5D331ZtdniVnBcOOJ 788 87cQ"} 789 ], 790 "ContentMetaData":"ewogICJjdHkiOiAiYXBwbGljYXRpb24vZXhhbXBsZS 791 1tYWlsIn0"}, 792 "X_thVIKHkw7ni2DydiaIN1pgSAAG4Zh4XsFPBtYQi_FwpMwHDhLxOg9so1ZwUx 793 rEajkoTf_C5GUiY1Pf87kQsQ" 794 ] 796 3.4.1. Key Exchange 798 The DARE key exchange is based on the JWE key exchange except that 799 encryption modes are intentionally limited and the output of the key 800 exchange is the DARE Master Key rather than the Content Encryption 801 Key. 803 A DARE Key Exchange MAY contain any number of Recipient entries, each 804 providing a means of decrypting the Master Key using a different 805 private key. 807 If the Key Exchange mechanism supports message recovery, Direct Key 808 Agreement is used, in all other cases, Key Wrapping is used. 810 This approach allows envelopes with one intended recipient to be 811 handled in the exact same fashion as envelopes with multiple 812 recipients. While this does require an additional key wrapping 813 operation, that could be avoided if an envelope has exactly one 814 intended recipient, this is offset by the reduction in code 815 complexity. 817 If the key exchange algorithm does not support message recovery (e.g. 818 Diffie Hellman and Elliptic Curve Diffie-Hellman), the HKDF Extract- 819 and-Expand Key Derivation Function is used to derive a master key 820 using the following info tag: 822 "dare-master" [64 61 72 65 2d 6d 61 73 74 65 72] Key derivation info 823 field used when deriving a master key from the output of a key 824 exchange. 826 The master key length is the maximum of the key size of the 827 encryption algorithm specified by the key exchange header, the key 828 size of the MAC algorithm specified by the key exchange header (if 829 used) and 256. 831 3.4.2. Key Identifiers 833 The JWE/JWS specifications define a kid field for use as a key 834 identifier but not how the identifier itself is constructed. All 835 DARE key identifiers are either UDF key fingerprints 836 [draft-hallambaker-mesh-udf] or Mesh/Recrypt Group Key Identifiers. 838 A UDF fingerprint is formed as the digest of an IANA content type and 839 the digested data. A UDF key fingerprint is formed with the content 840 type "application/pkix-keyinfo" and the digested data is the ASN.1 841 DER encoded PKIX certificate "keyInfo" sequence for the corresponding 842 public key. 844 A Group Key Identifier has the form @. Where 845 is a UDF key fingerprint and is the DNS 846 address of a service that provides the encryption service to support 847 decryption by group members. 849 3.4.3. Salt Derivation 851 A Master Salt is a sequence of 16 or more octets that is specified in 852 the Salt field of the header. 854 The Master Salt is used to derive salt values for the envelope 855 payload and associated encoded data sequences as follows. 857 Payload Salt = Master Salt 859 EDS Salt = Concatenate (Payload Salt Prefix, Master Salt) 861 Encoders SHOULD NOT generate salt values that exceed 1024 octets. 863 The salt value is opaque to the DARE encoding but MAY be used to 864 encode application specific semantics including: 866 * Frame number to allow reassembly of a data sequence split over a 867 sequence of envelopes which may be delivered out of order. 869 * Transmit the Master Key in the manner of a Kerberos ticket. 871 * Identify the Master Key under which the Enhanced Data Sequence was 872 generated. 874 * Enable access to the plaintext to be eliminated by erasure of the 875 encryption key. 877 For data erasure to be effective, the salt MUST be constructed so 878 that the difficulty of recovering the key is sufficiently high that 879 it is infeasible. For most purposes, a salt with 128 bits of 880 appropriately random data is sufficient. 882 3.4.4. Key Derivation 884 Encryption and/or authentication keys are derived from the Master Key 885 using a Extract-and-Expand Key Derivation Function as follows: 887 0. The Master Key and salt value are used to extract the PRK 888 (pseudorandom key) 890 1. The PRK is used to derive the algorithm keys using the 891 application specific information input for that key type. 893 The application specific information inputs are: 895 "dare-encrypt" [64 61 72 65 2d 65 6e 63 72 79 70 74] To generate an 896 encryption or encryption with authentication key. 898 "dare-iv" [64 61 72 65 2d 65 6e 63 72 79 70 74] To generate an 899 initialization vector. 901 "dare-mac" [dare-mac] To generate a Message Authentication Code key. 903 3.5. Signature 905 While encryption and integrity enhancements can be applied to any 906 part of a DARE Envelope, signatures are only applied to payload 907 digest values calculated over one or more envelope payloads. 909 The payload digest value for an envelope is calculated over the 910 binary payload data. That is, after any encryption enhancement has 911 been applied but before the envelope encoding is applied. This 912 allows envelopes to be converted from one encoding to another without 913 affecting signature verification. 915 Single Payload The signed value is the payload digest of the 916 envelope payload. 918 Multiple Payload. The signed value is the root of a Merkle Tree in 919 which the payload digest of the envelope is one of the leaves. 921 Verification of a multiple payload signature naturally requires the 922 additional digest values required to construct the Merkle Tree. 923 These are provided in the Trailer in a format that permits multiple 924 signers to reference the same tree data. 926 3.6. Algorithms 928 3.6.1. Field: kwd 930 The key wrapping and derivation algorithms. 932 Since the means of public key exchange is determined by the key 933 identifier of the recipient key, it is only necessary to specify the 934 algorithms used for key wrapping and derivation. 936 The default (and so far only) algorithm is kwd-aes-sha2-256-256. 938 Advanced Encryption Standard (AES) Key Wrap with Padding Algorithm 939 [RFC3394] is used to wrap the Master Exchange Key. AES 256 is used. 941 HMAC-based Extract-and-Expand Key Derivation Function [RFC5869] is 942 used for key derivation. SHA-2-256 is used for the hash function. 944 4. DARE Container Architecture 945 4.1. Container Navigation 947 Three means of locating frames in a container are supported: 949 Sequential Access frames sequentially starting from the start or the 950 end of the container. 952 Binary search Access any container frame by frame number in 953 O(log_2(n)) time by means of a binary tree constructed while the 954 container is written. 956 Index Access and container frame by frame number or by key by means 957 of an index record. 959 All DARE Containers support sequential access. Only tree and Merkle 960 tree containers support binary search access. An index frame MAY be 961 written appended to any container and provides O(1) access to any 962 frame listed in the index. 964 Two modes of compilation are considered: 966 Monolithic Frames are added to the container in a single operation, 967 e.g. file archives, 969 Incremental Additional frames are written to the container at 970 various intervals after it was originally created, e.g. server 971 logs, message spools. 973 In the monolithic mode, navigation requirements are best met by 974 writing an index frame to the end of the container when it is 975 complete. It is not necessary to construct a binary search tree 976 unless a Merkle tree integrity check is required. 978 In the incremental mode, Binary search provides an efficient means of 979 locating frames by frame number but not by key. Writing a complete 980 index to the container every _m_ write operations provides _O(m)_ 981 search access but requires O(n^2) storage. 983 Use of partial indexes provides a better compromise between speed and 984 efficiency. A partial index is written out every _m_ frames where 985 _m_ is a power of two. A complete index is written every time a 986 binary tree apex record is written. This approach provides for 987 O(log_2(n)) search with incremental compilation with approximately 988 double the overhead of the monolithic case. 990 4.1.1. Tree 992 As previously described, the JBCD frame structure allows incremental 993 navigation to the immediately preceding frame. The "TreePosition" 994 parameter specifies the start position of _any_ previous frame in the 995 container, thus allowing rapid navigation to that point. 997 The TreePosition parameter MAY be used to enable any frame in the 998 container to be retrieved in log_2(n) time by means of a binary 999 search. The TreePosition parameter specifies the immediately 1000 preceding apex of a binary tree formed from the container entries. 1002 For example, the TreePosition of frame 6 in a container gives the 1003 location of frame 5, the TreePosition of frame 5 gives the location 1004 of frame 3, the TreePosition of frame 3 gives the location of frame 1005 1, and the TreePosition of frame 1 gives the location of frame 0: 1007 (Artwork only available as svg: No external link available, see 1008 draft-hallambaker-mesh-dare-13.html for artwork.) 1010 Figure 5: Binary search tree. 1012 An algorithm for efficiently calculating the immediately preceding 1013 apex is provided in Appendix C. 1015 4.1.2. Position Index 1017 Contains a table of frame number, position pairs pointing to prior 1018 locations in the file. 1020 4.1.3. Metadata Index 1022 Contains a list of IndexMeta entries. Each entry contains a metadata 1023 description and a list of frame indexes (not positions) of frames 1024 that match the description. 1026 4.2. Integrity Mechanisms 1028 Frame sequences in a DARE container MAY be protected against a frame 1029 insertion attack by means of a digest chain, a binary Merkle tree or 1030 both. 1032 4.2.1. Digest Chain calculation 1034 A digest chain is simple to implement but can only be verified if the 1035 full chain of values is known. Appending a frame to the chain has 1036 _O(1)_ complexity but verification has _O(n)_ complexity: 1038 (Artwork only available as svg: No external link available, see 1039 draft-hallambaker-mesh-dare-13.html for artwork.) 1041 Figure 6: Hash chain integrity check 1043 The value of the chain digest for the first frame (frame 0) is 1044 _H(H(null)+H(Payload_0)), where null is a zero length octet sequence 1045 and payloadn is the sequence of payload data bytes for frame n_ 1047 The value of the chain digest for frame _n is H(H(Payload_(n-1) 1048 +H(Payloadn)), where A+B stands for concatenation of the byte 1049 sequences A and B._ 1051 4.2.2. Binary Merkle tree calculation 1053 The tree index mechanism describe earlier may be used to implement a 1054 binary Merkle tree. The value TreeDigest specifies the apex value of 1055 the tree for that node. 1057 Appending a frame to the chain has O(log_2 (n)) complexity provided 1058 that the container format supports at least the binary tree index. 1059 Verifying a chain has O(log_2 (n)) complexity, provided that the set 1060 of necessary digest inputs is known. 1062 To calculate the value of the tree digest for a node, we first 1063 calculate the values of all the sub trees that have their apex at 1064 that node and then calculate the digest of that value and the 1065 immediately preceding local apex. 1067 4.2.3. Signature 1069 Payload data MAY be signed using a JWS [RFC7515] as applied in the 1070 Envelope. 1072 Signatures are specified by the "Signatures" parameter in the content 1073 header. The data that the signature is calculated over is defined by 1074 the typ parameter of the Signature as follows. 1076 "Payload" The value of the "PayloadDigest" parameter 1078 "Chain" The value of the "ChainDigest" parameter 1080 "Tree" The value of the "TreeDigestFinal" parameter 1082 If the "typ" parameter is absent, the value Payload is implied. 1084 A frame MAY contain multiple signatures created with the same signing 1085 key and different typ values. 1087 The use of signatures over chain and tree digest values permit 1088 multiple frames to be validated using a single signature verification 1089 operation. 1091 5. DARE Schema 1093 A DARE Envelope consists of a Header, an Enhanced Data Sequence (EDS) 1094 and an optional trailer. This section describes the JSON data fields 1095 used to construct headers, trailers and complete envelopes. 1097 Wherever possible, fields from JWE, JWS and JWK have been used. In 1098 these cases, the fields have the exact same semantics. Note however 1099 that the classes in which these fields are presented have different 1100 structure and nesting. 1102 5.1. Envelope Classes 1104 A DARE envelope contains a single DAREMessageSequence in either the 1105 JSON or Compact serialization as directed by the protocol in which it 1106 is applied. 1108 5.1.1. Structure: DareEnvelopeSequence 1110 A DARE envelope containing Header, EDS and Trailer in JSON object 1111 encoding. Since a DAREMessage is almost invariably presented in JSON 1112 sequence or compact encoding, use of the DAREMessage subclass is 1113 preferred. 1115 Although a DARE envelope is functionally an object, it is serialized 1116 as an ordered sequence. This ensures that the envelope header field 1117 will always precede the body in a serialization, this allowing 1118 processing of the header information to be performed before the 1119 entire body has been received. 1121 Header: DareHeader (Optional) The envelope header. May specify the 1122 key exchange data, pre-signature or signature data, cloaked 1123 headers and/or encrypted data sequences. 1125 Body: Binary (Optional) The envelope body 1127 Trailer: DareTrailer (Optional) The envelope trailer. If present, 1128 this contains the signature. 1130 5.2. Header and Trailer Classes 1132 A DARE sequence MUST contain a (possibly empty) DAREHeader and MAY 1133 contain a DARETrailer. 1135 5.2.1. Structure: DareTrailer 1137 A DARE envelope Trailer 1139 Signatures: DareSignature [0..Many] A list of signatures. A 1140 envelope trailer MUST NOT contain a signatures field if the header 1141 contains a signatures field. 1143 SignedData: Binary (Optional) Contains a DAREHeader object 1145 PayloadDigest: Binary (Optional) If present, contains the digest of 1146 the Payload. 1148 ChainDigest: Binary (Optional) If present, contains the digest of 1149 the PayloadDigest values of this frame and the frame immediately 1150 preceding. 1152 TreeDigest: Binary (Optional) If present, contains the Binary Merkle 1153 Tree digest value. 1155 5.2.2. Structure: DareHeader 1157 Inherits: DareTrailer 1159 A DARE Envelope Header. Since any field that is present in a trailer 1160 MAY be placed in a header instead, the envelope header inherits from 1161 the trailer. 1163 EnvelopeId: String (Optional) Unique identifier 1165 EncryptionAlgorithm: String (Optional) The encryption algorithm as 1166 specified in JWE 1168 DigestAlgorithm: String (Optional) Digest Algorithm. If specified, 1169 tells decoder that the digest algorithm is used to construct a 1170 signature over the envelope payload. 1172 KeyIdentifier: String (Optional) Base seed identifier. 1174 Salt: Binary (Optional) Salt value used to derrive cryptographic 1175 parameters for the content data. 1177 Malt: Binary (Optional) Hash of the Salt value used to derrive 1178 cryptographic parameters for the content data. This field SHOULD 1179 NOT be present if the Salt field is present. It is used to allow 1180 the salt value to be erased (thus rendering the payload content 1181 irrecoverable) without affecting the ability to calculate the 1182 payload digest value. 1184 Cloaked: Binary (Optional) If present in a header or trailer, 1185 specifies an encrypted data block containing additional header 1186 fields whose values override those specified in the envelope and 1187 context headers. 1189 When specified in a header, a cloaked field MAY be used to conceal 1190 metadata (content type, compression) and/or to specify an 1191 additional layer of key exchange. That applies to both the 1192 envelope body and to headers specified within the cloaked header. 1194 Processing of cloaked data is described in... 1196 EDSS: Binary [0..Many] If present, the Annotations field contains a 1197 sequence of Encrypted Data Segments encrypted under the envelope 1198 base seed. The interpretation of these fields is application 1199 specific. 1201 Signers: DareSignature [0..Many] A list of 'presignature' 1203 Recipients: DareRecipient [0..Many] A list of recipient key exchange 1204 information blocks. 1206 Policy: DarePolicy (Optional) A DARE security policy governing 1207 future additions to the container. 1209 ContentMetaData: Binary (Optional) If present contains a JSON 1210 encoded ContentInfo structure which specifies plaintext content 1211 metadata and forms one of the inputs to the envelope digest value. 1213 SequenceInfo: SequenceInfo (Optional) Information that describes 1214 container information 1216 SequenceIndex: SequenceIndex (Optional) An index of records in the 1217 current container up to but not including this one. 1219 Received: DateTime (Optional) Date on which the envelope was 1220 received. 1222 5.2.3. Structure: ContentMeta 1224 UniqueId: String (Optional) Unique object identifier 1226 Labels: String [0..Many] List of labels that are applied to the 1227 payload of the frame. 1229 KeyValues: KeyValue [0..Many] List of key/value pairs describing the 1230 payload of the frame. 1232 MessageType: String (Optional) The mesh message type 1234 ContentType: String (Optional) The content type field as specified 1235 in JWE 1237 Paths: String [0..Many] List of filename paths for the payload of 1238 the frame. 1240 Filename: String (Optional) The original filename under which the 1241 data was stored. 1243 Event: String (Optional) Operation on the header 1245 Created: DateTime (Optional) Initial creation date. 1247 Modified: DateTime (Optional) Date of last modification. 1249 Expire: DateTime (Optional) Date at which the associated transaction 1250 will expire 1252 First: Integer (Optional) Frame number of the first object instance 1253 value. 1255 Previous: Integer (Optional) Frame number of the immediately prior 1256 object instance value 1258 FileEntry: FileEntry (Optional) Information describing the file 1259 entry on disk. 1261 5.3. Cryptographic Data 1263 DARE envelope uses the same fields as JWE and JWS but with different 1264 structure. In particular, DARE envelopes MAY have multiple 1265 recipients and multiple signers. 1267 5.3.1. Structure: DareSignature 1269 The signature value 1271 Dig: String (Optional) Digest algorithm hint. Specifying the digest 1272 algorithm to be applied to the envelope body allows the body to be 1273 processed in streaming mode. 1275 Alg: String (Optional) Key exchange algorithm 1277 KeyIdentifier: String (Optional) Key identifier of the signature 1278 key. 1280 Certificate: X509Certificate (Optional) PKIX certificate of signer. 1282 Path: X509Certificate (Optional) PKIX certificates that establish a 1283 trust path for the signer. 1285 Manifest: Binary (Optional) The data description that was signed. 1287 SignatureValue: Binary (Optional) The signature value as an Enhanced 1288 Data Sequence under the envelope base seed. 1290 WitnessValue: Binary (Optional) The signature witness value used on 1291 an encrypted envelope to demonstrate that the signature was 1292 authorized by a party with actual knowledge of the encryption key 1293 used to encrypt the envelope. 1295 5.3.2. Structure: X509Certificate 1297 X5u: String (Optional) URL identifying an X.509 public key 1298 certificate 1300 X5: Binary (Optional) An X.509 public key certificate 1302 5.3.3. Structure: DareRecipient 1304 Recipient information 1306 KeyIdentifier: String (Optional) Key identifier for the encryption 1307 key. 1309 The Key identifier MUST be either a UDF fingerprint of a key or a 1310 Group Key Identifier 1312 KeyWrapDerivation: String (Optional) The key wrapping and derivation 1313 algorithms. 1315 WrappedBaseSeed: Binary (Optional) The wrapped base seed. The base 1316 seed is encrypted under the result of the key exchange. 1318 RecipientKeyData: String (Optional) The per-recipient key exchange 1319 data. 1321 5.3.4. Structure: DarePolicy 1323 EncryptionAlgorithm: String (Optional) The encryption algorithm to 1324 be used to compute the payload. 1326 DigestAlgorithm: String (Optional) The digest algorithm to be used 1327 to compute the payload digest. 1329 Encryption: String (Optional) The encryption policy specifier, 1330 determines how often a key exchange is required. 'Single': All 1331 entries are encrypted under the key exchange specified in the 1332 entry specifying this policy. 'Isolated': All entries are 1333 encrypted under a separate key exchange. 'All': All entries are 1334 encrypted. 'None': No entries are encrypted. 1336 Default value is 'None' if EncryptKeys is null, and 'All' 1337 otherwise. 1339 Signature: String (Optional) The signature policy 'None': No entries 1340 are signed. 'Last': The last entry in the container is signed. 1341 'Isolated': All entries are independently signed. 'Any': Entries 1342 may be signed. 1344 Default value is 'None' if SignKeys is null, and 'Any' otherwise. 1346 Sealed: Boolean (Optional) If true the policy is immutable and 1347 cannot be changed by a subsequent policy override. 1349 5.3.5. Structure: FileEntry 1351 Path: String (Optional) The file path in canonical form. 1353 CreationTime: DateTime (Optional) The creation time of the file on 1354 disk in UTC 1356 LastAccessTime: DateTime (Optional) The last access time of the file 1357 on disk in UTC 1359 LastWriteTime: DateTime (Optional) The last write time of the file 1360 on disk in UTC 1362 Attributes: Integer (Optional) The file attribues as a bitmapped 1363 integer. 1365 6. DARE Container Schema 1367 TBS stuff 1369 6.1. Container Headers 1371 TBS stuff 1373 6.1.1. Structure: SequenceInfo 1375 Information that describes container information 1376 DataEncoding: String (Optional) Specifies the data encoding for the 1377 header section of for the following frames. This value is ONLY 1378 valid in Frame 0 which MUST have a header encoded in JSON. 1380 ContainerType: String (Optional) Specifies the container type for 1381 the following records. This value is ONLY valid in Frame 0 which 1382 MUST have a header encoded in JSON. 1384 Index: Integer (Optional) The record index within the file. This 1385 MUST be unique and satisfy any additional requirements determined 1386 by the ContainerType. 1388 IsMeta: Boolean (Optional) If true, the current frame is a meta 1389 frame and does not contain a payload. 1391 Note: Meta frames MAY be present in any container. Applications 1392 MUST accept containers that contain meta frames at any position in 1393 the file. Applications MUST NOT interpret a meta frame as a data 1394 frame with an enpty payload. 1396 Default: Boolean (Optional) If set true in a persistent container, 1397 specifies that this record contains the default object for the 1398 container. 1400 TreePosition: Integer (Optional) Position of the frame containing 1401 the apex of the preceding sub-tree. 1403 IndexPosition: Integer (Optional) Specifies the position in the file 1404 at which the last index entry is to be found 1406 ExchangePosition: Integer (Optional) Specifies the position in the 1407 file at which the key exchange data is to be found 1409 6.2. Index Structures 1411 TBS stuff 1413 6.2.1. Structure: SequenceIndex 1415 A container index 1417 Full: Boolean (Optional) If true, the index is complete and contains 1418 position entries for all the frames in the file. If absent or 1419 false, the index is incremental and only contains position entries 1420 for records added since the last frame containing a 1421 ContainerIndex. 1423 Positions: IndexPosition [0..Many] List of container position 1424 entries 1426 6.2.2. Structure: IndexPosition 1428 Specifies the position in a file at which a specified record index is 1429 found 1431 Index: Integer (Optional) The record index within the file. 1433 Position: Integer (Optional) The record position within the file 1434 relative to the index base. 1436 UniqueId: String (Optional) Unique object identifier 1438 6.2.3. Structure: KeyValue 1440 Specifies a key/value entry 1442 Key: String (Optional) The key 1444 Value: String (Optional) The value corresponding to the key 1446 7. Dare Container Applications 1448 DARE Containers are used to implement two forms of persistence store 1449 to support Mesh operations: 1451 Catalogs A set of related items which MAY be added, modified or 1452 deleted at any time. 1454 Spools A list of related items whose status MAY be changed at any 1455 time but which are immutable once added. 1457 Since DARE Containers are an append only log format, entries can only 1458 be modified or deleted by adding items to the log to change the 1459 status of previous entries. It is always possible to undo any 1460 operation on a catalog or spool unless the underlying container is 1461 purged or the individual entries modified. 1463 7.1. Catalog 1465 Catalogs contain a set of entries, each of which is distinguished by 1466 a unique identifier. 1468 Three operations are supported: 1470 Add Addition of the entry to the catalog 1471 Update Modification of the data associated with the entry excluding 1472 the identifier 1474 Delete Removal of the entry from the catalog 1476 The set of valid state transitions is defined by the Finite State 1477 machine: 1479 (Add-Update*-Delete)* 1481 Catalogs are used to represent sets of persistent objects associated 1482 with a Mesh Service Account. The user's set of contacts for example. 1483 Each contact entry may be modified many times over time but refers to 1484 the same subject for its entire lifetime. 1486 7.2. Spool 1488 Spools contain lists of entries, each of which is distinguished by a 1489 unique identifier. 1491 Four operations are supported: 1493 Post Addition of the entry to the spool 1495 Processed Marks the entry as having been processed. 1497 Unprocessed Returns the entry to the unread state. 1499 Delete Mark the entry as deleted allowing recovery of associated 1500 storage in a subsequent purge operation. 1502 The set of valid state transitions is defined by the Finite State 1503 machine: 1505 Post-(Processed| Unprocessed| Delete *) 1507 Spools are used to represent time sequence ordered entries such as 1508 lists of messages being sent or received, task queues and transaction 1509 logs. 1511 7.3. Archive 1513 A DARE Archive is a DARE Container whose entries contain files. This 1514 affords the same functionality as a traditional ZIP or tar archive 1515 but with the added cryptographic capabilities provided by the DARE 1516 format. 1518 8. Future Work 1520 The current specification describes an approach in which containers 1521 are written according to a strict append-only policy. Greater 1522 flexibility may be achieved by loosening this requirement allowing 1523 record(s) at the end of the container to be overwritten. 1525 8.1. Terminal integrity check 1527 A major concern when operating a critical service is the possibility 1528 of a hardware or power failure occurring during a write operation 1529 causing the file update to be incomplete. While most modern 1530 operating systems have effective mechanisms in place to prevent 1531 corruption of the file system itself in such circumstances, this does 1532 not provide sufficient protection at the application level. 1534 Appending a null record containing a container-specific magic number 1535 provides an effective means of detecting this circumstance that can 1536 be quickly verified. 1538 If a container specifies a terminal integrity check value in the 1539 header of frame zero, the container is considered to be in an 1540 incomplete write state if the final frame is not a null record 1541 specifying the magic number. 1543 When appending new records to such containers, the old terminal 1544 integrity check record is overwritten by the data being added and a 1545 new integrity check record appended to the end. 1547 8.2. Terminal index record 1549 A writer can maintain a complete (or partial) index of the container 1550 in its final record without additional space overhead by overwriting 1551 the prior index on each update. 1553 8.3. Deferred indexing 1555 The task of updating terminal indexes may be deferred to a time when 1556 the machine is not busy. This improves responsiveness and may avoid 1557 the need to re-index containers receiving a sequence of updates. 1559 This approach may be supported by appending new entries to the end of 1560 the container in the usual fashion and maintaining a record of 1561 containers to be updated as a separate task. 1563 When updating the index on a container that has been updated in this 1564 fashion, the writer must ensure that no data is lost even if the 1565 process is interrupted. The use of guard records and other 1566 precautions against loss of state is advised. 1568 9. Security Considerations 1570 This section describes security considerations arising from the use 1571 of DARE in general applications. 1573 Additional security considerations for use of DARE in Mesh services 1574 and applications are described in the Mesh Security Considerations 1575 guide [draft-hallambaker-mesh-security]. 1577 9.1. Encryption/Signature nesting 1579 9.2. Side channel 1581 9.3. Salt reuse 1583 10. IANA Considerations 1585 11. Acknowledgements 1587 A list of people who have contributed to the design of the Mesh is 1588 presented in [draft-hallambaker-mesh-architecture]. 1590 The name Data At Rest Encryption was proposed by Melhi Abdulhayo?lu. 1592 12. Appendix A: DARE Envelope Examples and Test Vectors 1594 13. Test Examples 1596 In the following examples, Alice's encryption private key parameters 1597 are: 1599 { 1600 "PrivateKeyECDH":{ 1601 "crv":"Ed25519", 1602 "Private":"4rPu23meICbPMLvmjAaZm6N-CZ3wqnjA5gCeNvs0Svc"}} 1604 Alice's signature private key parameters are: 1606 { 1607 "PrivateKeyECDH":{ 1608 "crv":"Ed25519", 1609 "Private":"znX4E3eT_qsOrNKtzVo5yQXjKLS_Xe302S6_zE8MU5E"}} 1611 The body of the test message is the UTF8 representation of the 1612 following string: 1614 "This is a test long enough to require multiple blocks" 1616 The EDS sequences, are the UTF8 representation of the following 1617 strings: 1619 "Subject: Message metadata should be encrypted" 1620 "2018-02-01" 1622 13.1. Plaintext Message 1624 A plaintext message without associated EDS sequences is an empty 1625 header followed by the message body: 1627 { 1628 "DareEnvelope":[{}, 1629 "VGhpcyBpcyBhIHRlc3QgbG9uZyBlbm91Z2ggdG8gcmVxdWlyZSBtdWx0aXBs 1630 ZSBibG9ja3M" 1631 ]} 1633 13.2. Plaintext Message with EDS 1635 If a plaintext message contains EDS sequences, these are also in 1636 plaintext: 1638 { 1639 "DareEnvelope":[{ 1640 "annotations":["iAEAiC1TdWJqZWN0OiBNZXNzYWdlIG1ldGFkYXRhIHNo 1641 b3VsZCBiZSBlbmNyeXB0ZWQ", 1642 "iAEBiAoyMDE4LTAyLTAx" 1643 ]}, 1644 "VGhpcyBpcyBhIHRlc3QgbG9uZyBlbm91Z2ggdG8gcmVxdWlyZSBtdWx0aXBs 1645 ZSBibG9ja3M" 1646 ]} 1648 13.3. Encrypted Message 1650 The creator generates a base seed: 1652 40 0E B3 EC 04 73 1E A6 45 64 B0 6A ED F9 AC B1 1653 9A B6 C7 72 E3 DF 77 10 5A 71 21 54 9B 52 3D 71 1655 For each recipient of the message: 1657 The creator generates an ephemeral key: 1659 { 1660 "PrivateKeyECDH":{ 1661 "crv":"Ed25519", 1662 "Private":"78dh0AIeOv4nem043ll2nlDp52k16N0PXpjErrGyWBw"}} 1664 The key agreement value is calculated: 1666 B4 0C 18 C0 87 12 91 25 BA A0 03 5D 52 BE AB 99 1667 4E 7B 02 9F A8 02 C1 6C E0 60 BC C8 81 24 1E 0C 1669 The key agreement value is used as the input to a HKDF key derivation 1670 function with the info parameter master to create the key used to 1671 wrap the base seed: 1673 BA 5D 91 1B E4 F6 8E 5B B5 70 3A FA B7 09 82 07 1674 60 0E A5 34 B0 91 A5 4B 56 08 38 B7 8E 75 54 67 1676 The wrapped base seed is: 1678 3F F4 42 D5 92 D8 0C C5 5B 6F 2A 86 26 67 C6 FF 1679 5A E2 53 EF F2 62 DF DD FF 90 F7 DF 56 6D 76 78 1680 95 9C 17 0E 38 9F 3B 71 1682 This information is used to calculate the Recipient information shown 1683 in the example below. 1685 To encrypt a message, we first generate a unique salt value: 1687 D2 F1 FB 60 3D 06 1F F3 42 47 91 13 8B 77 E2 EE 1689 The base seed and salt value are used to generate the payload 1690 encryption key: 1692 69 11 25 F3 65 4A 4C 49 B2 6B CA 13 48 C3 45 BE 1693 B5 38 6E A7 0F 7E A8 E5 B3 8C F5 0F 33 3A A8 20 1695 Since AES is a block cipher, we also require an initializarion 1696 vector: 1698 6A 00 1A 6B 26 18 8D 26 C8 2A 54 1F DE 18 B0 59 1700 The output sequence is the encrypted bytes: 1702 D2 5E 42 E8 5B 19 7E 49 D8 BC E8 63 47 AF E3 A1 1703 9E 8C F9 98 03 A2 35 1F B2 87 79 CF 2D E6 1E 65 1704 38 5B 1F 76 A2 69 0F CA 88 8E A1 B3 EC 8D EF 28 1705 8C 16 6C 9D 58 AF 62 4E 86 2B 5E CC 99 4A 50 9C 1707 Since the message is not signed, there is no need for a trailer. The 1708 completed message is: 1710 { 1711 "DareEnvelope":[{ 1712 "enc":"A256CBC", 1713 "kid":"EBQO-ZHLV-PTHT-7EH5-LDGK-LCU3-HAPV", 1714 "Salt":"0vH7YD0GH_NCR5ETi3fi7g", 1715 "recipients":[{ 1716 "kid":"MDW3-XKC7-T2YL-2WCJ-C5VK-46QT-CQP4", 1717 "epk":{ 1718 "PublicKeyECDH":{ 1719 "crv":"Ed25519", 1720 "Public":"2MmwZTfeBCRImFB2kwFRgNoay9IjJHcOuX7jrl9pM 1721 yo"}}, 1722 "wmk":"4PXvuukffEuZyUJXanJke7GREmPbAr9Y2Wz89DwN-CK7gHVx 1723 4II6xQ"} 1724 ]}, 1725 "0l5C6FsZfknYvOhjR6_joZ6M-ZgDojUfsod5zy3mHmU4Wx92omkPyoiOobPs 1726 je8ojBZsnVivYk6GK17MmUpQnA" 1727 ]} 1729 13.4. Signed Message 1731 Signed messages specify the digest algorithm to be used in the header 1732 and the signature value in the trailer. Note that the digest 1733 algorithm is not optional since it serves as notice that a decoder 1734 should digest the payload value to enable signature verification. 1736 { 1737 "DareEnvelope":[{ 1738 "dig":"S512"}, 1739 "VGhpcyBpcyBhIHRlc3QgbG9uZyBlbm91Z2ggdG8gcmVxdWlyZSBtdWx0aXBs 1740 ZSBibG9ja3M", 1741 { 1742 "signatures":[{ 1743 "alg":"S512", 1744 "kid":"MD4C-AWRW-JSSJ-CP2D-JSNY-HUPI-SOH3", 1745 "signature":"Y76p0SE2zmoyKRjgL4T-iq7U2gMWQR1WJUHkFfO98U 1746 W-YaDdmw7dAeEE0AxUPBGu0XHowSZ0p4HF76GYbqZjAg"} 1747 ], 1748 "PayloadDigest":"raim8SV5adPbWWn8FMM4mrRAQCO9A2jZ0NZAnFXWlG 1749 0xF6sWGJbnKSdtIJMmMU_hjarlIPEoY3vy9UdVlH5KAg"} 1750 ]} 1752 13.5. Signed and Encrypted Message 1754 A signed and encrypted message is encrypted and then signed. The 1755 signer proves knowledge of the payload plaintext by providing the 1756 plaintext witness value. 1758 { 1759 "DareEnvelope":[{ 1760 "enc":"A256CBC", 1761 "dig":"S512", 1762 "kid":"EBQB-DDWU-V4YU-AUPL-H5JQ-W7K4-SDXD", 1763 "Salt":"VBS4crdNcEZobOlnIrC-zA", 1764 "recipients":[{ 1765 "kid":"MDW3-XKC7-T2YL-2WCJ-C5VK-46QT-CQP4", 1766 "epk":{ 1767 "PublicKeyECDH":{ 1768 "crv":"Ed25519", 1769 "Public":"IjE5rpUzw4QQq0lR-DyGVlWFLG0HpDTo6oG73MgUM 1770 mk"}}, 1771 "wmk":"IfL6epBv1t-wSbLBU9sVW84AaNQ43_2VEkkatrq7jTBqx7ui 1772 psv2gw"} 1773 ]}, 1774 "bx8sBdxb3DTgZw4K7Bwf6idZC-0sGlHWrytgYuT0ERoTqUMmgb-0LNbz6Vd1 1775 dp5-_-EkhpNZf2ERYXDXixXQ9g", 1776 { 1777 "signatures":[{ 1778 "alg":"S512", 1779 "kid":"MD4C-AWRW-JSSJ-CP2D-JSNY-HUPI-SOH3", 1780 "signature":"4lcQgA8cVIWlfA8mvCzSmw1wQm9pGyXgi9C3V-oH_p 1781 dr_eNVT2sq_FVCQiGYtdpxT3MW17wSl1kXo9fNsVB7CQ", 1782 "witness":"W_Uz9WvEdi_xTZf2q1cDoUDeC3S7JnHi36pRUiezCAo"} 1783 ], 1784 "PayloadDigest":"O-Phb87EkcBt6sM6qSCAjOjGvU_zD6DlkqCWEGK3En 1785 zzd-HGvDO5sTyhke4sKtHjyDKkj2iDNqroOcluIUqE_A"} 1786 ]} 1788 14. Appendix B: DARE Container Examples and Test Vectors 1790 The data payloads in all the following examples are identical, only 1791 the authentication and/or encryption is different. 1793 * Frame 1..n consists of 300 bytes being the byte sequence 00, 01, 1794 02, etc. repeating after 256 bytes. 1796 For conciseness, the raw data format is omitted for examples after 1797 the first, except where the data payload has been transformed, (i.e. 1798 encrypted). 1800 14.1. Simple sequence 1802 The following example shows a simple sequence with first frame and a 1803 single data frame: 1805 f4 91 1806 f0 8b 1807 f0 00 1808 f0 00 1809 91 f4 1810 f5 01 73 1811 f0 42 1812 f1 01 2c 1813 73 01 f5 1815 Since there is no integrity check, there is no need for trailer 1816 entries. The header values are: 1818 Frame 0 1820 { 1821 "DareHeader":{ 1822 "policy":{}, 1823 "ContentMetaData":"e30", 1824 "SequenceInfo":{ 1825 "DataEncoding":"JSON", 1826 "ContainerType":"List", 1827 "Index":0}}} 1829 [Empty trailer] 1831 Frame 1 1833 { 1834 "DareHeader":{ 1835 "ContentMetaData":"e30", 1836 "SequenceInfo":{ 1837 "Index":1}}} 1839 [Empty trailer] 1841 14.2. Payload and chain digests 1843 The following example shows a chain sequence with a first frame and 1844 three data frames. The headers of these frames is the same as before 1845 but the frames now have trailers specifying the PayloadDigest and 1846 ChainDigest values: 1848 Frame 0 1850 { 1851 "DareHeader":{ 1852 "dig":"S512", 1853 "policy":{}, 1854 "ContentMetaData":"e30", 1855 "SequenceInfo":{ 1856 "DataEncoding":"JSON", 1857 "ContainerType":"Chain", 1858 "Index":0}}} 1860 [Empty trailer] 1862 Frame 1 1864 { 1865 "DareHeader":{ 1866 "dig":"S512", 1867 "ContentMetaData":"e30", 1868 "SequenceInfo":{ 1869 "Index":1}}} 1871 { 1872 "DareTrailer":{ 1873 "PayloadDigest":"8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZ 1874 DlZeaWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw", 1875 "ChainDigest":"T7S1FcrgY3AaWD4L-t5W1K-3XYkPTcOdGEGyjglTD6yMYV 1876 RVz9tn_KQc6GdA-P4VSRigBygV65OEd2Vv3YDhww"}} 1878 Frame 2 1880 { 1881 "DareHeader":{ 1882 "dig":"S512", 1883 "ContentMetaData":"e30", 1884 "SequenceInfo":{ 1885 "Index":2}}} 1887 { 1888 "DareTrailer":{ 1889 "PayloadDigest":"8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZ 1890 DlZeaWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw", 1891 "ChainDigest":"T7S1FcrgY3AaWD4L-t5W1K-3XYkPTcOdGEGyjglTD6yMYV 1892 RVz9tn_KQc6GdA-P4VSRigBygV65OEd2Vv3YDhww"}} 1894 Frame 3 1895 { 1896 "DareHeader":{ 1897 "dig":"S512", 1898 "ContentMetaData":"e30", 1899 "SequenceInfo":{ 1900 "Index":3}}} 1902 { 1903 "DareTrailer":{ 1904 "PayloadDigest":"8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZ 1905 DlZeaWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw", 1906 "ChainDigest":"T7S1FcrgY3AaWD4L-t5W1K-3XYkPTcOdGEGyjglTD6yMYV 1907 RVz9tn_KQc6GdA-P4VSRigBygV65OEd2Vv3YDhww"}} 1909 14.3. Merkle Tree 1911 The following example shows a chain sequence with a first frame and 1912 six data frames. The trailers now contain the TreePosition and 1913 TreeDigest values: 1915 Frame 0 1917 { 1918 "DareHeader":{ 1919 "dig":"S512", 1920 "policy":{}, 1921 "ContentMetaData":"e30", 1922 "SequenceInfo":{ 1923 "DataEncoding":"JSON", 1924 "ContainerType":"Merkle", 1925 "Index":0}}} 1927 [Empty trailer] 1929 Frame 1 1930 { 1931 "DareHeader":{ 1932 "dig":"S512", 1933 "ContentMetaData":"e30", 1934 "SequenceInfo":{ 1935 "Index":1, 1936 "TreePosition":0}}} 1938 { 1939 "DareTrailer":{ 1940 "PayloadDigest":"8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZ 1941 DlZeaWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw", 1942 "TreeDigest":"T7S1FcrgY3AaWD4L-t5W1K-3XYkPTcOdGEGyjglTD6yMYVR 1943 Vz9tn_KQc6GdA-P4VSRigBygV65OEd2Vv3YDhww"}} 1945 Frame 2 1947 { 1948 "DareHeader":{ 1949 "dig":"S512", 1950 "ContentMetaData":"e30", 1951 "SequenceInfo":{ 1952 "Index":2, 1953 "TreePosition":392}}} 1955 { 1956 "DareTrailer":{ 1957 "PayloadDigest":"8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZ 1958 DlZeaWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw", 1959 "TreeDigest":"7fHmkEIsPkN6sDYAOLvpIJn5Dg3PxDDAaq-ll2kh8722kok 1960 kFnZQcYtjuVC71aHNXI18q-lPnfRkmwryG-bhqQ"}} 1962 Frame 3 1964 { 1965 "DareHeader":{ 1966 "dig":"S512", 1967 "ContentMetaData":"e30", 1968 "SequenceInfo":{ 1969 "Index":3, 1970 "TreePosition":392}}} 1972 { 1973 "DareTrailer":{ 1974 "PayloadDigest":"8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZ 1975 DlZeaWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw", 1976 "TreeDigest":"T7S1FcrgY3AaWD4L-t5W1K-3XYkPTcOdGEGyjglTD6yMYVR 1977 Vz9tn_KQc6GdA-P4VSRigBygV65OEd2Vv3YDhww"}} 1979 Frame 4 1981 { 1982 "DareHeader":{ 1983 "dig":"S512", 1984 "ContentMetaData":"e30", 1985 "SequenceInfo":{ 1986 "Index":4, 1987 "TreePosition":1676}}} 1989 { 1990 "DareTrailer":{ 1991 "PayloadDigest":"8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZ 1992 DlZeaWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw", 1993 "TreeDigest":"vJ6ngNATvZcXSMALi5IUqzl1GBxBnTNVcC87VL_BhMRCbAv 1994 KSj8gs0VFgxxLkZ2myrtaDIwhHoswiTiBMLNWug"}} 1996 Frame 5 1998 { 1999 "DareHeader":{ 2000 "dig":"S512", 2001 "ContentMetaData":"e30", 2002 "SequenceInfo":{ 2003 "Index":5, 2004 "TreePosition":1676}}} 2006 { 2007 "DareTrailer":{ 2008 "PayloadDigest":"8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZ 2009 DlZeaWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw", 2010 "TreeDigest":"T7S1FcrgY3AaWD4L-t5W1K-3XYkPTcOdGEGyjglTD6yMYVR 2011 Vz9tn_KQc6GdA-P4VSRigBygV65OEd2Vv3YDhww"}} 2013 Frame 6 2014 { 2015 "DareHeader":{ 2016 "dig":"S512", 2017 "ContentMetaData":"e30", 2018 "SequenceInfo":{ 2019 "Index":6, 2020 "TreePosition":2963}}} 2022 { 2023 "DareTrailer":{ 2024 "PayloadDigest":"8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZ 2025 DlZeaWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw", 2026 "TreeDigest":"WgHlz3EHczVPqgtpc39Arv7CFIsCbFVsk8wg0j2qLlEfur9 2027 SZ0mdr65Ka-HF0Qx8gg_DAoiJwUrwADDXyxVJOg"}} 2029 14.4. Signed sequence 2031 The following example shows a tree sequence with a signature in the 2032 final record. The signing key parameters are: 2034 { 2035 "PrivateKeyECDH":{ 2036 "crv":"Ed25519", 2037 "Private":"znX4E3eT_qsOrNKtzVo5yQXjKLS_Xe302S6_zE8MU5E"}} 2039 The sequence headers and trailers are: 2041 Frame 0 2043 { 2044 "DareHeader":{ 2045 "dig":"S512", 2046 "policy":{}, 2047 "ContentMetaData":"e30", 2048 "SequenceInfo":{ 2049 "DataEncoding":"JSON", 2050 "ContainerType":"Merkle", 2051 "Index":0}}} 2053 [Empty trailer] 2055 Frame 1 2056 { 2057 "DareHeader":{ 2058 "dig":"S512", 2059 "ContentMetaData":"e30", 2060 "SequenceInfo":{ 2061 "Index":1, 2062 "TreePosition":0}}} 2064 { 2065 "DareTrailer":{ 2066 "PayloadDigest":"8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZ 2067 DlZeaWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw", 2068 "TreeDigest":"T7S1FcrgY3AaWD4L-t5W1K-3XYkPTcOdGEGyjglTD6yMYVR 2069 Vz9tn_KQc6GdA-P4VSRigBygV65OEd2Vv3YDhww"}} 2071 Frame 2 2073 { 2074 "DareHeader":{ 2075 "dig":"S512", 2076 "ContentMetaData":"e30", 2077 "SequenceInfo":{ 2078 "Index":2, 2079 "TreePosition":392}}} 2081 { 2082 "DareTrailer":{ 2083 "PayloadDigest":"8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZ 2084 DlZeaWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw", 2085 "TreeDigest":"7fHmkEIsPkN6sDYAOLvpIJn5Dg3PxDDAaq-ll2kh8722kok 2086 kFnZQcYtjuVC71aHNXI18q-lPnfRkmwryG-bhqQ"}} 2088 14.5. Encrypted sequence 2090 The following example shows a sequence in which all the frame 2091 payloads are encrypted under the same base seed established in a key 2092 agreement specified in the first frame. 2094 Frame 0 2095 { 2096 "DareHeader":{ 2097 "enc":"A256CBC", 2098 "kid":"EBQG-JZST-KORS-PJP3-W6CF-YNO5-H3RM", 2099 "Salt":"JeDK58O63CKsJLGdCxIvFA", 2100 "recipients":[{ 2101 "kid":"MDW3-XKC7-T2YL-2WCJ-C5VK-46QT-CQP4", 2102 "epk":{ 2103 "PublicKeyECDH":{ 2104 "crv":"Ed25519", 2105 "Public":"Tcur0dyEfGVT9_Xu9pcGLrtS1KVUffgS7Y1j2h564tM"}}, 2106 "wmk":"mc1T1J_3BT8ItVdG7kNCf3M1PBPU6wxhfh_FHkm4Gu9EUJi-88 2107 Ivqw"} 2108 ], 2109 "policy":{ 2110 "enc":"none", 2111 "dig":"none", 2112 "EncryptKeys":[{ 2113 "PublicKeyECDH":{ 2114 "crv":"Ed25519", 2115 "Public":"iNEOsydUCtD9SGDoqcUCwtiKWbicIIsbDMcJbebUGT0"}} 2116 ], 2117 "Sealed":true}, 2118 "ContentMetaData":"e30", 2119 "SequenceInfo":{ 2120 "DataEncoding":"JSON", 2121 "ContainerType":"List", 2122 "Index":0}}} 2124 [Empty trailer] 2126 Frame 1 2127 { 2128 "DareHeader":{ 2129 "enc":"A256CBC", 2130 "kid":"EBQF-Q6TK-QUCT-3XKN-25LW-A5XK-RW4M", 2131 "Salt":"PJGUkGc004GKrjSHYbzgNg", 2132 "recipients":[{ 2133 "kid":"MDW3-XKC7-T2YL-2WCJ-C5VK-46QT-CQP4", 2134 "epk":{ 2135 "PublicKeyECDH":{ 2136 "crv":"Ed25519", 2137 "Public":"BfRjTIiWJH-OlldLEG5FK0O9acKG19xQIldOcV97S4I"}}, 2138 "wmk":"--1mN4A2BCOjO4P0iExai0FbxHMMqqbwEDdw2BxtQKjPoNYn2Y 2139 y-_Q"} 2140 ], 2141 "ContentMetaData":"e30", 2142 "SequenceInfo":{ 2143 "Index":1}}} 2145 [Empty trailer] 2147 Frame 2 2149 { 2150 "DareHeader":{ 2151 "enc":"A256CBC", 2152 "kid":"EBQA-ILT2-NAYH-M2DH-DYFF-NBBI-OYL4", 2153 "Salt":"prtpxhuoYw5zBVGF13UoHQ", 2154 "recipients":[{ 2155 "kid":"MDW3-XKC7-T2YL-2WCJ-C5VK-46QT-CQP4", 2156 "epk":{ 2157 "PublicKeyECDH":{ 2158 "crv":"Ed25519", 2159 "Public":"89ZZZzHV2qZJsZED5WgQbccSz2H3ebbZdLkl5GP3f4Y"}}, 2160 "wmk":"ltaB2QTJT5RFZA_sLe64IzdhQ3spSBeXUgz9jFd3JcMFbp7HmG 2161 FmLw"} 2162 ], 2163 "ContentMetaData":"e30", 2164 "SequenceInfo":{ 2165 "Index":2}}} 2167 [Empty trailer] 2169 Here are the sequence bytes. Note that the content is now encrypted 2170 and has expanded by 25 bytes. These are the salt (16 bytes), the AES 2171 padding (4 bytes) and the JSON-B framing (5 bytes). 2173 f5 02 ef 2174 f1 02 d8 2175 f0 10 2176 f0 00 2177 ef 02 f5 2178 f5 02 f9 2179 f1 01 c3 2180 f1 01 30 2181 f9 02 f5 2182 f5 02 f9 2183 f1 01 c3 2184 f1 01 30 2185 f9 02 f5 2187 The following example shows a sequence in which all the frame 2188 payloads are encrypted under separate key agreements specified in the 2189 payload frames. 2191 Frame 0 2193 { 2194 "DareHeader":{ 2195 "policy":{ 2196 "enc":"none", 2197 "dig":"none", 2198 "Sealed":true}, 2199 "ContentMetaData":"e30", 2200 "SequenceInfo":{ 2201 "DataEncoding":"JSON", 2202 "ContainerType":"List", 2203 "Index":0}}} 2205 [Empty trailer] 2207 Frame 1 2209 { 2210 "DareHeader":{ 2211 "ContentMetaData":"e30", 2212 "SequenceInfo":{ 2213 "Index":1}}} 2215 [Empty trailer] 2217 Frame 2 2218 { 2219 "DareHeader":{ 2220 "ContentMetaData":"e30", 2221 "SequenceInfo":{ 2222 "Index":2}}} 2224 [Empty trailer] 2226 15. Appendix C: Previous Frame Function 2228 public long PreviousFrame (long Frame) { 2229 long x2 = Frame + 1; 2230 long d = 1; 2232 while (x2 > 0) { 2233 if ((x2 & 1) == 1) { 2234 return x2 == 1 ? (d / 2) - 1 : Frame - d; 2235 } 2236 d = d * 2; 2237 x2 = x2 / 2; 2238 } 2239 return 0; 2240 } 2242 16. Appendix D: Outstanding Issues 2244 The following issues need to be addressed. 2246 +================+===============================================+ 2247 | Issue | Description | 2248 +================+===============================================+ 2249 | Indexing | No examples are given of indexing a container | 2250 +----------------+-----------------------------------------------+ 2251 | Archive | Should include a file archive example | 2252 +----------------+-----------------------------------------------+ 2253 | File Path | Mention the file path security issue in the | 2254 | | security considerations | 2255 +----------------+-----------------------------------------------+ 2256 | Security | Write Security considerations | 2257 | Considerations | | 2258 +----------------+-----------------------------------------------+ 2259 | AES-GCM | Switch to using AES GCM in the examples | 2260 +----------------+-----------------------------------------------+ 2261 | Witness | Complete handling of witness values. | 2262 +----------------+-----------------------------------------------+ 2263 | Schema | Complete the schema documentation | 2264 +----------------+-----------------------------------------------+ 2266 Table 1 2268 17. Normative References 2270 [draft-hallambaker-jsonbcd] 2271 Hallam-Baker, P., "Binary Encodings for JavaScript Object 2272 Notation: JSON-B, JSON-C, JSON-D", Work in Progress, 2273 Internet-Draft, draft-hallambaker-jsonbcd-21, 5 August 2274 2021, . 2277 [draft-hallambaker-mesh-architecture] 2278 Hallam-Baker, P., "Mathematical Mesh 3.0 Part I: 2279 Architecture Guide", Work in Progress, Internet-Draft, 2280 draft-hallambaker-mesh-architecture-17, 5 August 2021, 2281 . 2284 [draft-hallambaker-mesh-security] 2285 Hallam-Baker, P., "Mathematical Mesh 3.0 Part VII: 2286 Security Considerations", Work in Progress, Internet- 2287 Draft, draft-hallambaker-mesh-security-07, 5 August 2021, 2288 . 2291 [draft-hallambaker-mesh-udf] 2292 Hallam-Baker, P., "Mathematical Mesh 3.0 Part II: Uniform 2293 Data Fingerprint.", Work in Progress, Internet-Draft, 2294 draft-hallambaker-mesh-udf-13, 5 August 2021, 2295 . 2298 [IANAJOSE] "[Reference Not Found!]". 2300 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 2301 Requirement Levels", BCP 14, RFC 2119, 2302 DOI 10.17487/RFC2119, March 1997, 2303 . 2305 [RFC2315] Kaliski, B., "PKCS #7: Cryptographic Message Syntax 2306 Version 1.5", RFC 2315, DOI 10.17487/RFC2315, March 1998, 2307 . 2309 [RFC3394] Schaad, J. and R. Housley, "Advanced Encryption Standard 2310 (AES) Key Wrap Algorithm", RFC 3394, DOI 10.17487/RFC3394, 2311 September 2002, . 2313 [RFC4880] Callas, J., Donnerhacke, L., Finney, H., Shaw, D., and R. 2314 Thayer, "OpenPGP Message Format", RFC 4880, 2315 DOI 10.17487/RFC4880, November 2007, 2316 . 2318 [RFC4949] Shirey, R., "Internet Security Glossary, Version 2", 2319 FYI 36, RFC 4949, DOI 10.17487/RFC4949, August 2007, 2320 . 2322 [RFC5869] Krawczyk, H. and P. Eronen, "HMAC-based Extract-and-Expand 2323 Key Derivation Function (HKDF)", RFC 5869, 2324 DOI 10.17487/RFC5869, May 2010, 2325 . 2327 [RFC6838] Freed, N., Klensin, J., and T. Hansen, "Media Type 2328 Specifications and Registration Procedures", BCP 13, 2329 RFC 6838, DOI 10.17487/RFC6838, January 2013, 2330 . 2332 [RFC7159] Bray, T., "The JavaScript Object Notation (JSON) Data 2333 Interchange Format", RFC 7159, DOI 10.17487/RFC7159, March 2334 2014, . 2336 [RFC7515] Jones, M., Bradley, J., and N. Sakimura, "JSON Web 2337 Signature (JWS)", RFC 7515, DOI 10.17487/RFC7515, May 2338 2015, . 2340 [RFC7516] Jones, M. and J. Hildebrand, "JSON Web Encryption (JWE)", 2341 RFC 7516, DOI 10.17487/RFC7516, May 2015, 2342 . 2344 [RFC7517] Jones, M., "JSON Web Key (JWK)", RFC 7517, 2345 DOI 10.17487/RFC7517, May 2015, 2346 . 2348 [RFC7518] Jones, M., "JSON Web Algorithms (JWA)", RFC 7518, 2349 DOI 10.17487/RFC7518, May 2015, 2350 . 2352 18. Informative References 2354 [BLOCKCHAIN] 2355 Chain.com, "Blockchain Specification". 2357 [Davis2001] 2358 Davis, D., "Defective Sign & Encrypt in S/MIME, PKCS#7, 2359 MOSS, PEM, PGP, and XML", May 2001. 2361 [RFC5652] Housley, R., "Cryptographic Message Syntax (CMS)", STD 70, 2362 RFC 5652, DOI 10.17487/RFC5652, September 2009, 2363 . 2365 [ZIPFILE] PKWARE Inc, "APPNOTE.TXT - .ZIP File Format 2366 Specification", October 2014.