idnits 2.17.1 draft-hallambaker-mesh-dare-11.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack an Authors' Addresses Section. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (13 January 2021) is 1170 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Missing Reference: 'EOF' is mentioned on line 759, but not defined ** Obsolete normative reference: RFC 7159 (Obsoleted by RFC 8259) Summary: 2 errors (**), 0 flaws (~~), 2 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group P. M. Hallam-Baker 3 Internet-Draft ThresholdSecrets.com 4 Intended status: Informational 13 January 2021 5 Expires: 17 July 2021 7 Mathematical Mesh 3.0 Part III : Data At Rest Encryption (DARE) 8 draft-hallambaker-mesh-dare-11 10 Abstract 12 This document describes the Data At Rest Encryption (DARE) Envelope 13 and Container syntax. 15 The DARE Envelope syntax is used to digitally sign, digest, 16 authenticate, or encrypt arbitrary content data. 18 The DARE Container syntax describes an append-only sequence of 19 entries, each containing a DARE Envelope. DARE Containers may 20 support cryptographic integrity verification of the entire data 21 container content by means of a Merkle tree. 23 [Note to Readers] 25 Discussion of this draft takes place on the MATHMESH mailing list 26 (mathmesh@ietf.org), which is archived at 27 https://mailarchive.ietf.org/arch/search/?email_list=mathmesh. 29 This document is also available online at 30 http://mathmesh.com/Documents/draft-hallambaker-mesh-dare.html. 32 Status of This Memo 34 This Internet-Draft is submitted in full conformance with the 35 provisions of BCP 78 and BCP 79. 37 Internet-Drafts are working documents of the Internet Engineering 38 Task Force (IETF). Note that other groups may also distribute 39 working documents as Internet-Drafts. The list of current Internet- 40 Drafts is at https://datatracker.ietf.org/drafts/current/. 42 Internet-Drafts are draft documents valid for a maximum of six months 43 and may be updated, replaced, or obsoleted by other documents at any 44 time. It is inappropriate to use Internet-Drafts as reference 45 material or to cite them other than as "work in progress." 47 This Internet-Draft will expire on 17 July 2021. 49 Copyright Notice 51 Copyright (c) 2021 IETF Trust and the persons identified as the 52 document authors. All rights reserved. 54 This document is subject to BCP 78 and the IETF Trust's Legal 55 Provisions Relating to IETF Documents (https://trustee.ietf.org/ 56 license-info) in effect on the date of publication of this document. 57 Please review these documents carefully, as they describe your rights 58 and restrictions with respect to this document. 60 Table of Contents 62 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 4 63 1.1. Encryption and Integrity . . . . . . . . . . . . . . . . 5 64 1.1.1. Key Exchange . . . . . . . . . . . . . . . . . . . . 5 65 1.1.2. Data Erasure . . . . . . . . . . . . . . . . . . . . 6 66 1.2. Signature . . . . . . . . . . . . . . . . . . . . . . . . 6 67 1.2.1. Signing Individual Plaintext Envelopes . . . . . . . 7 68 1.2.2. Signing Individual Encrypted Envelopes . . . . . . . 7 69 1.2.3. Signing sequences of envelopes . . . . . . . . . . . 7 70 1.3. Container . . . . . . . . . . . . . . . . . . . . . . . . 8 71 1.3.1. Container Format . . . . . . . . . . . . . . . . . . 8 72 1.3.2. Write . . . . . . . . . . . . . . . . . . . . . . . . 9 73 1.3.3. Encryption and Authentication . . . . . . . . . . . . 9 74 1.3.4. Integrity and Signature . . . . . . . . . . . . . . . 10 75 1.3.5. Redaction . . . . . . . . . . . . . . . . . . . . . . 10 76 1.3.6. Alternative approaches . . . . . . . . . . . . . . . 11 77 1.3.7. Efficiency . . . . . . . . . . . . . . . . . . . . . 11 78 2. Definitions . . . . . . . . . . . . . . . . . . . . . . . . . 12 79 2.1. Related Specifications . . . . . . . . . . . . . . . . . 12 80 2.2. Requirements Language . . . . . . . . . . . . . . . . . . 12 81 2.3. Defined terms . . . . . . . . . . . . . . . . . . . . . . 13 82 3. DARE Envelope Architecture . . . . . . . . . . . . . . . . . 14 83 3.1. Processing Considerations . . . . . . . . . . . . . . . . 15 84 3.2. Content Metadata and Annotations . . . . . . . . . . . . 15 85 3.3. Encoded Data Sequence . . . . . . . . . . . . . . . . . . 16 86 3.4. Encryption and Integrity . . . . . . . . . . . . . . . . 17 87 3.4.1. Key Exchange . . . . . . . . . . . . . . . . . . . . 17 88 3.4.2. Key Identifiers . . . . . . . . . . . . . . . . . . . 18 89 3.4.3. Salt Derivation . . . . . . . . . . . . . . . . . . . 19 90 3.4.4. Key Derivation . . . . . . . . . . . . . . . . . . . 19 91 3.5. Signature . . . . . . . . . . . . . . . . . . . . . . . . 20 92 3.6. Algorithms . . . . . . . . . . . . . . . . . . . . . . . 20 93 3.6.1. Field: kwd . . . . . . . . . . . . . . . . . . . . . 20 94 4. DARE Container Architecture . . . . . . . . . . . . . . . . . 20 95 4.1. Container Navigation . . . . . . . . . . . . . . . . . . 21 96 4.1.1. Tree . . . . . . . . . . . . . . . . . . . . . . . . 22 97 4.1.2. Position Index . . . . . . . . . . . . . . . . . . . 22 98 4.1.3. Metadata Index . . . . . . . . . . . . . . . . . . . 22 99 4.2. Integrity Mechanisms . . . . . . . . . . . . . . . . . . 22 100 4.2.1. Digest Chain calculation . . . . . . . . . . . . . . 22 101 4.2.2. Binary Merkle tree calculation . . . . . . . . . . . 23 102 4.2.3. Signature . . . . . . . . . . . . . . . . . . . . . . 23 103 5. DARE Schema . . . . . . . . . . . . . . . . . . . . . . . . . 24 104 5.1. Envelope Classes . . . . . . . . . . . . . . . . . . . . 24 105 5.1.1. Structure: DareEnvelopeSequence . . . . . . . . . . . 24 106 5.2. Header and Trailer Classes . . . . . . . . . . . . . . . 25 107 5.2.1. Structure: DareTrailer . . . . . . . . . . . . . . . 25 108 5.2.2. Structure: DareHeader . . . . . . . . . . . . . . . . 25 109 5.2.3. Structure: ContentMeta . . . . . . . . . . . . . . . 26 110 5.3. Cryptographic Data . . . . . . . . . . . . . . . . . . . 27 111 5.3.1. Structure: DareSignature . . . . . . . . . . . . . . 27 112 5.3.2. Structure: X509Certificate . . . . . . . . . . . . . 28 113 5.3.3. Structure: DareRecipient . . . . . . . . . . . . . . 28 114 5.3.4. Structure: DarePolicy . . . . . . . . . . . . . . . . 29 115 5.3.5. Structure: FileEntry . . . . . . . . . . . . . . . . 29 116 6. DARE Container Schema . . . . . . . . . . . . . . . . . . . . 29 117 6.1. Container Headers . . . . . . . . . . . . . . . . . . . . 30 118 6.1.1. Structure: SequenceInfo . . . . . . . . . . . . . . . 30 119 6.2. Index Structures . . . . . . . . . . . . . . . . . . . . 30 120 6.2.1. Structure: SequenceIndex . . . . . . . . . . . . . . 31 121 6.2.2. Structure: IndexPosition . . . . . . . . . . . . . . 31 122 6.2.3. Structure: KeyValue . . . . . . . . . . . . . . . . . 31 123 7. Dare Container Applications . . . . . . . . . . . . . . . . . 31 124 7.1. Catalog . . . . . . . . . . . . . . . . . . . . . . . . . 32 125 7.2. Spool . . . . . . . . . . . . . . . . . . . . . . . . . . 32 126 7.3. Archive . . . . . . . . . . . . . . . . . . . . . . . . . 33 127 8. Future Work . . . . . . . . . . . . . . . . . . . . . . . . . 33 128 8.1. Terminal integrity check . . . . . . . . . . . . . . . . 33 129 8.2. Terminal index record . . . . . . . . . . . . . . . . . . 33 130 8.3. Deferred indexing . . . . . . . . . . . . . . . . . . . . 34 131 9. Security Considerations . . . . . . . . . . . . . . . . . . . 34 132 9.1. Encryption/Signature nesting . . . . . . . . . . . . . . 34 133 9.2. Side channel . . . . . . . . . . . . . . . . . . . . . . 34 134 9.3. Salt reuse . . . . . . . . . . . . . . . . . . . . . . . 34 135 10. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 34 136 11. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 34 137 12. Appendix A: DARE Envelope Examples and Test Vectors . . . . . 34 138 13. Test Examples . . . . . . . . . . . . . . . . . . . . . . . . 34 139 13.1. Plaintext Message . . . . . . . . . . . . . . . . . . . 35 140 13.2. Plaintext Message with EDS . . . . . . . . . . . . . . . 35 141 13.3. Encrypted Message . . . . . . . . . . . . . . . . . . . 36 142 13.4. Signed Message . . . . . . . . . . . . . . . . . . . . . 37 143 13.5. Signed and Encrypted Message . . . . . . . . . . . . . . 38 144 14. Appendix B: DARE Container Examples and Test Vectors . . . . 39 145 14.1. Simple sequence . . . . . . . . . . . . . . . . . . . . 39 146 14.2. Payload and chain digests . . . . . . . . . . . . . . . 40 147 14.3. Merkle Tree . . . . . . . . . . . . . . . . . . . . . . 42 148 14.4. Signed sequence . . . . . . . . . . . . . . . . . . . . 45 149 14.5. Encrypted sequence . . . . . . . . . . . . . . . . . . . 46 150 15. Appendix C: Previous Frame Function . . . . . . . . . . . . . 50 151 16. Appendix D: Outstanding Issues . . . . . . . . . . . . . . . 50 152 17. Normative References . . . . . . . . . . . . . . . . . . . . 51 153 18. Informative References . . . . . . . . . . . . . . . . . . . 53 155 1. Introduction 157 This document describes the Data At Rest Encryption (DARE) Envelope 158 and Container Syntax. The DARE Envelope syntax is used to digitally 159 sign, digest, authenticate, or encrypt arbitrary message content. 160 The DARE Container syntax describes an append-only sequence of data 161 frames, each containing a DARE Envelope that supports efficient 162 incremental signature and encryption. 164 The DARE Envelope Syntax is based on a subset of the JSON Web 165 Signature [RFC7515] and JSON Web Encryption [RFC7516] standards and 166 shares many fields and semantics. The processing model and data 167 structures have been streamlined to remove alternative means of 168 specifying the same content and to enable multiple data sequences to 169 be signed and encrypted under a single master encryption key without 170 compromise to security. 172 A DARE Envelope consists of a _Header_, _Payload_ and an optional 173 _Trailer_. To enable single pass encoding and decoding, the Header 174 contains all the information required to perform cryptographic 175 processing of the Payload and authentication data (digest, MAC, 176 signature values) MAY be deferred to the Trailer section. 178 A DARE Container is an append-only log format consisting of a 179 sequence of frames. Cryptographic enhancements (signature, 180 encryption) may be applied to individual frames or to sets of frames. 181 Thus, a single key exchange may be used to provide a master key to 182 encrypt multiple frames and a single signature may be used to 183 authenticate all the frames in the container up to and including the 184 frame in which the signature is presented. 186 The DARE Envelope syntax may be used either as a standalone 187 cryptographic message syntax or as a means of presenting a single 188 DARE Container frame together with the complete cryptographic context 189 required to verify the contents and decrypt them. 191 1.1. Encryption and Integrity 193 A key innovation in the DARE Envelope Syntax is the separation of key 194 exchange and data encryption operations so that a Master Key (MK) 195 established in a single exchange to be applied to multiple data 196 sequences. This means that a single public key operation MAY be used 197 to encrypt and/or authenticate multiple parts of the same DARE 198 Envelope or multiple frames in a DARE Container. 200 To avoid reuse of the key and to avoid the need to communicate 201 separate IVs, each octet sequence is encrypted under a different 202 encryption key (and IV if required) derived from the Master Key by 203 means of a salt that is unique for each octet sequence that is 204 encrypted. The same approach is used to generate keys for 205 calculating a MAC over the octet sequence if required. This approach 206 allows encryption and integrity protections to be applied to the 207 envelope payload, to header or trailer fields or to application 208 defined Enhanced Data Sequences in the header or trailer. 210 1.1.1. Key Exchange 212 Traditional cryptographic containers describe the application of a 213 single key exchange to encryption of a single octet sequence. 214 Examples include PCKS#7/CMS [RFC2315], OpenPGP [RFC4880] and JSON Web 215 Encryption [RFC7516]. 217 To encrypt data using RSA, the encoder first generates a random 218 encryption key and initialization vector (IV). The encryption key is 219 encrypted under the public key of each recipient to create a per- 220 recipient decryption entry. The encryption key, plaintext and IV are 221 used to generate the ciphertext (figure 1). 223 (Artwork only available as svg: No external link available, see 224 draft-hallambaker-mesh-dare-11.html for artwork.) 226 Figure 1: Monolithic Key Exchange and Encrypt 228 This approach is adequate for the task of encrypting a single octet 229 stream. It is less than satisfactory when encrypting multiple octet 230 streams or very long streams for which a rekeying operation is 231 desirable. 233 In the DARE approach, key exchange and key derivation are separate 234 operations and keys MAY be derived for encryption or integrity 235 purposes or both. A single key exchange MAY be used to derive keys 236 to apply encryption and integrity enhancements to multiple data 237 sequences. 239 The DARE key exchange begins with the same key exchange used to 240 produce the CEK in JWE but instead of using the CEK to encipher data 241 directly, it is used as one of the inputs to a Key Derivation 242 Function (KDF) that is used to derive parameters for each block of 243 data to be encrypted. To avoid the need to introduce additional 244 terminology, the term 'CEK' is still used to describe the output of 245 the key agreement algorithm (including key unwrapping if required) 246 but it is more appropriately described as a Master Key (figure 2). 248 (Artwork only available as svg: No external link available, see 249 draft-hallambaker-mesh-dare-11.html for artwork.) 251 Figure 2: Exchange of Master Key 253 A Master Key may be used to encrypt any number of data items. Each 254 data item is encrypted under a different encryption key and IV (if 255 required). This data is derived from the Master Key using the HKDF 256 function [RFC5869] using a different salt for each data item and 257 separate info tags for each cryptographic function (figure 3). 259 (Artwork only available as svg: No external link available, see 260 draft-hallambaker-mesh-dare-11.html for artwork.) 262 Figure 3: Data item encryption under Master Key and per-item salt. 264 This approach to encryption offers considerably greater flexibility 265 allowing the same format for data item encryption to be applied at 266 the transport, message or field level. 268 1.1.2. Data Erasure 270 Each encrypted DARE Envelope specifies a unique Master Salt value of 271 at least 128 bits which is used to derive the salt values used to 272 derive cryptographic keys for the envelope payload and annotations. 274 Erasure of the Master Salt value MAY be used to effectively render 275 the envelope payload and annotations undecipherable without altering 276 the envelope payload data. The work factor for decryption will be 277 O(2^128) even if the decryption key is compromised. 279 1.2. Signature 281 As with encryption, DARE Envelope signatures MAY be applied to an 282 individual envelope or a sequence of envelope. 284 1.2.1. Signing Individual Plaintext Envelopes 286 When an individual plaintext envelope is signed, the digest value 287 used to create the signature is calculated over the binary value of 288 the payload data. That is, the value of the payload before the 289 encoding (Base-64, JSON-B) is applied. 291 1.2.2. Signing Individual Encrypted Envelopes 293 When an individual plaintext envelope is signed, the digest value 294 used to create the signature is calculated over the binary value of 295 the payload data. That is, the value of the payload after encryption 296 but before the encoding (Base-64, JSON-B) is applied. 298 Use of signing and encryption in combination presents the risk of 299 subtle attacks depending on the order in which signing and encryption 300 take place [Davis2001]. 302 Na?ve approaches in which an envelope is encrypted and then signed 303 present the possibility of a surreptitious forwarding attack. For 304 example, Alice signs an envelope and sends it to Mallet who then 305 strips off Alice's signature and sends the envelope to Bob. 307 Na?ve approaches in which an envelope is signed and then encrypted 308 present the possibility of an attacker claiming authorship of a 309 ciphertext. For example, Alice encrypts a ciphertext for Bob and 310 then signs it. Mallet then intercepts the envelope and sends it to 311 Bob. 313 While neither attack is a concern in all applications, both attacks 314 pose potential hazards for the unwary and require close inspection of 315 application protocol design to avoid exploitation. 317 To prevent these attacks, each signature on an envelope that is 318 signed and encrypted MUST include a witness value that is calculated 319 by applying a MAC function to the signature value as described in 320 section XXX. 322 1.2.3. Signing sequences of envelopes 324 To sign multiple envelopes with a single signature, we first 325 construct a Merkle tree of the envelope payload digest values and 326 then sign the root of the Merkle tree. 328 [This is not yet implemented but will be soon.] 330 1.3. Container 332 DARE Container is a message and file syntax that allows a sequence of 333 data frames to be represented with cryptographic integrity, 334 signature, and encryption enhancements to be constructed in an append 335 only format. 337 The format is designed to meet the requirements of a wide range of 338 use cases including: 340 * Recording transactions in persistent storage. 342 * Synchronizing transaction logs between hosts. 344 * File archive. 346 * Message spool. 348 * Signing and encrypting single data items. 350 * Incremental encryption and authentication of server logs. 352 1.3.1. Container Format 354 A Container consists of a sequence of variable length Frames. Each 355 frame consists of a forward length indicator, the framed data and a 356 reverse length indicator. The reverse length indicator is written 357 out backwards allowing the length and thus the frame to be read in 358 the reverse direction: 360 (Artwork only available as svg: No external link available, see 361 draft-hallambaker-mesh-dare-11.html for artwork.) 363 Figure 4: JBCD Bidirectional Frame 365 Each frame contains a single DARE Envelope consisting of a Header, 366 Payload and Trailer (if required). The first frame in a container 367 describes the container format options and defaults. These include 368 the range of encoding options for frame metadata supported and the 369 container profiles to which the container conforms. 371 All internal data formats support use of pointers of up to 64 bits 372 allowing containers of up to 18 exabytes to be written. 374 Five container types are currently specified: 376 Simple The container does not provide any index or content integrity 377 checks. 379 Tree Frame headers contain entries that specify the start position 380 of previous frames at the apex of the immediately enclosing binary 381 tree. This enables efficient random access to any frame in the 382 file. 384 Digest Each frame trailer contains a "PayloadDigest" field. 385 Modification of the payload will cause verification of the 386 "PayloadDigest" value to fail on that frame. 388 Chain Each frame trailer contains "PayloadDigest" and "ChainDigest" 389 fields allowing modifications to the payload data to be detected. 390 Modification of the payload will cause verification of the 391 "PayloadDigest" value to fail on that frame and verification of 392 the "ChainDigest" value to fail on all subsequent frames. 394 Merkle Tree Frame headers contain entries that specify the start 395 position of previous frames at the apex of the immediately 396 enclosing binary tree. Frame Trailers contain TreeDigestPartial 397 and TreeDigestFinal entries forming a Merkle digest tree. 399 1.3.2. Write 401 In normal circumstances, Containers are written as an append only 402 log. As with Envelopes, integrity information (payload digest, 403 signatures) is written to the entry trailer. Thus, large payloads 404 may be written without the need to buffer the payload data _provided 405 that_ the content length is known in advance. 407 Should exceptional circumstances require, Container entries MAY be 408 erased by overwriting the Payload and/or parts of the Header content 409 without compromising the ability to verify other entries in the 410 container. If the entry Payload is encrypted, it is sufficient to 411 erase the container salt value to render the container entry 412 effectively inaccessible (though recovery might still be possible if 413 the original salt value can be recovered from the storage media. 415 1.3.3. Encryption and Authentication 417 Frame payloads and associated attributes MAY be encrypted and/or 418 authenticated in the same manner as Envelopes. 420 _Incremental encryption_ is supported allowing encryption parameters 421 from a single public key exchange operation to be applied to encrypt 422 multiple frames. The public key exchange information is specified in 423 the first encrypted frame and subsequent frames encrypted under those 424 parameters specify the location at which the key exchange information 425 is to be found by means of the ExchangePosition field which MUST 426 specify a location that is earlier in the file. 428 To avoid cryptographic vulnerabilities resulting from key re-use, the 429 DARE key exchange requires that each encrypted sequence use an 430 encryption key and initialization vector derived from the master key 431 established in the public key exchange by means of a unique salt. 433 Each Envelope and by extension, each Container frame MUST specify a 434 unique salt value of at least 128 bits. Since the encryption key is 435 derived from the salt value by means of a Key Derivation Function, 436 erasure of the salt MAY be used as a means of rendering the payload 437 plaintext value inaccessible without changing the payload value. 439 1.3.4. Integrity and Signature 441 Signatures MAY be applied to a payload digest, the final digest in a 442 chain or tree. The chain and tree digest modes allow a single 443 signature to be used to authenticate all frame payloads in a 444 container. 446 The tree signature mode is particularly suited to applications such 447 as file archives as it allows files to be verified individually 448 without requiring the signer to sign each individually. Furthermore, 449 in applications such as code signing, it allows a single signature to 450 be used to verify both the integrity of the code and its membership 451 of the distribution. 453 As with DARE Envelope, the signature mechanism does not specify the 454 interpretation of the signature semantics. The presence of a 455 signature demonstrates that the holder of the private key applied it 456 to the specified digest value but not their motive for doing so. 457 Describing such semantics is beyond the scope of this document and is 458 deferred to future work. 460 1.3.5. Redaction 462 The chief disadvantage of using an append-only format is that 463 containers only increase in size. In many applications, much of the 464 data in the container becomes redundant or obsolete and a process 465 analogous to garbage collection is required. This process is called 466 _redaction_. 468 The simplest method of redaction is to create a new container and 469 sequentially copy each entry from the old container to the new, 470 discarding redundant frames and obsolete header information. 472 For example, partial index records may be consolidated into a single 473 index record placed in the last frame of the container. Unnecessary 474 signature and integrity data may be discarded and so on. 476 While redaction could in principle be effected by moving data in- 477 place in the existing container, supporting this approach in a robust 478 fashion is considerably more complex as it requires backward 479 references in subsequent frames to be overridden as each frame is 480 moved. 482 1.3.6. Alternative approaches 484 Many file proprietary formats are in use that support some or all of 485 these capabilities but only a handful have public, let alone open, 486 standards. DARE Container is designed to provide a superset of the 487 capabilities of existing message and file syntaxes, including: 489 * Cryptographic Message Syntax [RFC5652] defines a syntax used to 490 digitally sign, digest, authenticate, or encrypt arbitrary message 491 content. 493 * The.ZIP File Format specification [ZIPFILE] developed by Phil 494 Katz. 496 * The BitCoin Block chain [BLOCKCHAIN]. 498 * JSON Web Encryption and JSON Web Signature 500 Attempting to make use of these specifications in a layered fashion 501 would require at least three separate encoders and introduce 502 unnecessary complexity. Furthermore, there is considerable overlap 503 between the specifications providing multiple means of achieving the 504 same ends, all of which must be supported if decoders are to work 505 reliably. 507 1.3.7. Efficiency 509 Every data format represents a compromise between different concerns, 510 in particular: 512 Compactness The space required to record data in the encoding. 514 Memory Overhead The additional volatile storage (RAM) required to 515 maintain indexes etc. to support efficient retrieval operations. 517 Number of Operations The number of operations required to retrieve 518 data from or append data to an existing encoded sequence. 520 Number of Disk Seek Operations Optimizing the response time of 521 magnetic storage media to random access read requests has 522 traditionally been one of the central concerns of database design. 523 The DARE Container format is designed to the assumption that this 524 will cease to be a concern as solid state media replaces magnetic. 526 While the cost of storage of all types has declined rapidly over the 527 past decades, so has the amount of data to be stored. DARE Container 528 represents a pragmatic balance of these considerations for current 529 technology. In particular, since payload volumes are likely to be 530 very large, memory and operational efficiency are considered higher 531 priorities than compactness. 533 2. Definitions 535 2.1. Related Specifications 537 The DARE Envelope and Container formats are based on the following 538 existing standards and specifications. 540 Object serialization The JSON-B [draft-hallambaker-jsonbcd] encoding 541 is used for object serialization. This encoding is an extension 542 of the JavaScript Object Notation (JSON) [RFC7159]. 544 Message syntax The cryptographic processing model is based on JSON 545 Web Signature (JWS) [RFC7515], JSON Web Encryption (JWE) [RFC7516] 546 and JSON Web Key (JWK) [RFC7517]. 548 Cryptographic primitives. The HMAC-based Extract-and-Expand Key 549 Derivation Function [RFC5869] and Advanced Encryption Standard 550 (AES) Key Wrap with Padding Algorithm [RFC3394] are used. 552 The Uniform Data Fingerprint method of presenting data digests is 553 used for key identifiers and other purposes 554 [draft-hallambaker-mesh-udf]. 556 Cryptographic algorithms The cryptographic algorithms and 557 identifiers described in JSON Web Algorithms (JWA) [RFC7518] are 558 used together with additional algorithms as defined in the JSON 559 Object Signing and Encryption IANA registry [IANAJOSE]. 561 2.2. Requirements Language 563 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 564 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 565 document are to be interpreted as described in [RFC2119]. 567 2.3. Defined terms 569 The terms "Authentication Tag", "Content Encryption Key", "Key 570 Management Mode", "Key Encryption", "Direct Key Agreement", "Key 571 Agreement with Key Wrapping" and "Direct Encryption" are defined in 572 the JWE specification [RFC7516]. 574 The terms "Authentication", "Ciphertext", "Digital Signature", 575 "Encryption", "Initialization Vector (IV)", "Message Authentication 576 Code (MAC)", "Plaintext" and "Salt" are defined by the Internet 577 Security Glossary, Version 2 [RFC4949]. 579 Annotated Envelope A DARE Envelope that contains an "Annotations" 580 field with at least one entry. 582 Authentication Data A Message Authentication Code or authentication 583 tag. 585 Complete Envelope A DARE envelope that contains the key exchange 586 information necessary for the intended recipient(s) to decrypt it. 588 Detached Envelope A DARE envelope that does not contain the key 589 exchange information necessary for the intended recipient(s) to 590 decrypt it. 592 Encryption Context The master key, encryption algorithms and 593 associated parameters used to generate a set of one or more 594 enhanced data sequences. 596 Encoded data sequence (EDS) A sequence consisting of a salt, content 597 data and authentication data (if required by the encryption 598 context). 600 Enhancement Applying a cryptographic operation to a data sequence. 601 This includes encryption, authentication and both at the same 602 time. 604 Generator The party that generates a DARE envelope. 606 Group Encryption Key A key used to encrypt data to be read by a 607 group of users. This is typically achieved by means of some form 608 of proxy re-encryption or distributed key generation. 610 Group Encryption Key Identifier A key identifier for a group 611 encryption key. 613 Master Key (MK) The master secret from which keys are derived for 614 authenticating enhanced data sequences. 616 Recipient Any party that receives and processes at least some part 617 of a DARE envelope. 619 Related Envelope A set of DARE envelopes that share the same key 620 exchange information and hence the same Master Key. 622 Uniform Data Fingerprint (UDF) The means of presenting the result of 623 a cryptographic digest function over a data sequence and content 624 type identifier specified in the Uniform Data Fingerprint 625 specification [draft-hallambaker-mesh-udf] 627 3. DARE Envelope Architecture 629 A DARE Envelope is a sequence of three parts: 631 Header A JSON object containing information a reader requires to 632 begin processing the envelope. 634 Payload An array of octets. 636 Trailer A JSON object containing information calculated from the 637 envelope payload. 639 For example, the following sequence is a JSON encoded Envelope with 640 an empty header, a payload of zero length and an empty trailer: 642 [ {}, "", {} ] 644 DARE Envelopes MAY be encoded using JSON serialization or a binary 645 serialization for greater efficiency. 647 JSON [RFC7159] Offers compatibility with applications and libraries 648 that support JSON. Payload data is encoded using Base64 incurring 649 a 33% overhead. 651 JSON-B [draft-hallambaker-jsonbcd] A superset of JSON encoding that 652 permits binary data to be encoded as a sequence of length-data 653 segments. This avoids the Base64 overhead incurred by JSON 654 encoding. Since JSON-B is a superset of JSON encoding, an 655 application can use a single decoder for either format. 657 JSON-C [draft-hallambaker-jsonbcd] A superset of JSON-C which 658 provides additional efficiency by allowing field tags and other 659 repeated string data to be encoded by reference to a dictionary. 660 Since JSON-C is a superset of JSON and JSON-B encodings, an 661 application can use a single decoder for all three formats. 663 DARE Envelope processors MUST support JSON serialization and SHOULD 664 support JSON-B serialization. 666 3.1. Processing Considerations 668 The DARE Envelope Syntax supports single pass encoding and decoding 669 without buffering of data. All the information required to begin 670 processing a DARE envelope (key agreement information, digest 671 algorithms), is provided in the envelope header. All the information 672 that is derived from envelope processing (authentication codes, 673 digest values, signatures) is presented in the envelope trailer. 675 The choice of envelope encoding does not affect the semantics of 676 envelope processing. A DARE Envelope MAY be reserialized under the 677 same serialization or converted from any of the specified 678 serialization to any other serialization without changing the 679 semantics or integrity properties of the envelope. 681 3.2. Content Metadata and Annotations 683 A header MAY contain header fields describing the payload content. 684 These include: 686 ContentType Specifies the IANA Media Type [RFC6838]. 688 Annotations A list of Encoded Data Sequences that provide 689 application specific annotations to the envelope. 691 For example, consider the following mail message: 693 From: Alice@example.com 694 To: bob@example.com 695 Subject: TOP-SECRET Product Launch Today! 697 The CEO told me the product launch is today. Tell no-one! 699 Existing encryption approaches require that header fields such as the 700 subject line be encrypted with the body of the message or not 701 encrypted at all. Neither approach is satisfactory. In this 702 example, the subject line gives away important information that the 703 sender probably assumed would be encrypted. But if the subject line 704 is encrypted together with the message body, a mail client must 705 retrieve at least part of the message body to provide a 'folder' 706 view. 708 The plaintext form of the equivalent DARE Message encoding is: 710 [{ 711 "annotations":["iAEAiBdGcm9tOiBBbGljZUBleGFtcGxlLmNvbQ", 712 "iAEBiBNUbzogYm9iQGV4YW1wbGUuY29t", 713 "iAECiClTdWJqZWN0OiBUT1AtU0VDUkVUIFByb2R1Y3QgTGF1bmNoIFRvZG 714 F5IQ" 715 ], 716 "ContentMetaData":"ewogICJjdHkiOiAiYXBwbGljYXRpb24vZXhhbXBsZS 717 1tYWlsIn0"}, 718 "VGhlIENFTyB0b2xkIG1lIHRoZSBwcm9kdWN0IGxhdW5jaCBpcyB0b2RheS4gVG 719 VsbCBuby1vbmUh" 720 ] 722 This contains the same information as before but the mail message 723 headers are now presented as a list of Encoded Data Sequences. 725 3.3. Encoded Data Sequence 727 An encoded data sequence (EDS) is a sequence of octets that encodes a 728 data sequence according to cryptographic enhancements specified in 729 the context in which it is presented. An EDS MAY be encrypted and 730 MAY be authenticated by means of a MAC. The keys and other 731 cryptographic parameters used to apply these enhancements are derived 732 from the cryptographic context and a Salt prefix specified in the EDS 733 itself. 735 An EDS sequence contains exactly three binary fields encoded in 736 JSON-B serialization as follows: 738 Salt Prefix A sequence of octets used to derive the encryption key, 739 Initialization Vector and MAC key as required. 741 Body The plaintext or encrypted content. 743 Authentication Tag The authentication code value in the case that 744 the cryptographic context specifies use of authenticated 745 encryption or a MAC, otherwise is a zero-length field. 747 Requiring all three fields to be present, even in cases where they 748 are unnecessary simplifies processing at the cost of up to six 749 additional data bytes. 751 The encoding of the 'From' header of the previous example as a 752 plaintext EDS is as follows: 754 88 01 755 00 756 88 17 757 46 72 6f 6d 3a 20 41 6c 69 63 65 40 65 78 61 6d 758 70 6c 65 2e 63 6f 6d 759 [EOF] 761 3.4. Encryption and Integrity 763 Encryption and integrity protections MAY be applied to any DARE 764 Envelope Payload and Annotations. 766 The following is an encrypted version of the message shown earlier. 767 The payload and annotations have both increased in size as a result 768 of the block cipher padding. The header now includes Recipients and 769 Salt fields to enable the content to be decoded. 771 [{ 772 "enc":"A256CBC", 773 "kid":"EBQH-3NAT-FW7Z-IRDK-QC3M-SC2B-UI77", 774 "Salt":"-7YjIinh89vumJlW5P2GMg", 775 "annotations":["iAEAiCC8xyPT3IaxCUiUUDdUKfJ4FojwKinCRX9DIQ5UXd 776 14vQ", 777 "iAEBiCCwXnWWrKxt6ijdvmVm8QkKBWwWu3zcjOsQ2HiYNI5qFg", 778 "iAECiDCPK9HdJbLx0DRO4mWcUc36Q5vPpfjK0kchlTnJOAaAJ6BBM1RJw_ 779 hzAaekH0FeD64" 780 ], 781 "recipients":[{ 782 "kid":"MCZG-SJCC-WPAU-MNRF-6TCG-KUUQ-ICAQ", 783 "epk":{ 784 "PublicKeyECDH":{ 785 "crv":"Ed25519", 786 "Public":"9vIwyosnE76w1gtCYqY0nKu5MrIf3jPeiPFE_6T3-ks"}}, 787 "wmk":"QZwtHQUzXS6qlipGvjQhz_0GjHNFV27sAUP_nPf5vxTBJbtxUb 788 El-Q"} 789 ], 790 "ContentMetaData":"ewogICJjdHkiOiAiYXBwbGljYXRpb24vZXhhbXBsZS 791 1tYWlsIn0"}, 792 "DvWZ9em6q2rIrWAAUcaS-W7DN1dbEVeyO9L5iqC-1okIbwm2V4pP8prKH1Acy7 793 IJrWKK8d8Smxm5ep9XiIrjuw" 794 ] 796 3.4.1. Key Exchange 798 The DARE key exchange is based on the JWE key exchange except that 799 encryption modes are intentionally limited and the output of the key 800 exchange is the DARE Master Key rather than the Content Encryption 801 Key. 803 A DARE Key Exchange MAY contain any number of Recipient entries, each 804 providing a means of decrypting the Master Key using a different 805 private key. 807 If the Key Exchange mechanism supports message recovery, Direct Key 808 Agreement is used, in all other cases, Key Wrapping is used. 810 This approach allows envelopes with one intended recipient to be 811 handled in the exact same fashion as envelopes with multiple 812 recipients. While this does require an additional key wrapping 813 operation, that could be avoided if an envelope has exactly one 814 intended recipient, this is offset by the reduction in code 815 complexity. 817 If the key exchange algorithm does not support message recovery (e.g. 818 Diffie Hellman and Elliptic Curve Diffie-Hellman), the HKDF Extract- 819 and-Expand Key Derivation Function is used to derive a master key 820 using the following info tag: 822 "dare-master" [64 61 72 65 2d 6d 61 73 74 65 72] Key derivation info 823 field used when deriving a master key from the output of a key 824 exchange. 826 The master key length is the maximum of the key size of the 827 encryption algorithm specified by the key exchange header, the key 828 size of the MAC algorithm specified by the key exchange header (if 829 used) and 256. 831 3.4.2. Key Identifiers 833 The JWE/JWS specifications define a kid field for use as a key 834 identifier but not how the identifier itself is constructed. All 835 DARE key identifiers are either UDF key fingerprints 836 [draft-hallambaker-mesh-udf] or Mesh/Recrypt Group Key Identifiers. 838 A UDF fingerprint is formed as the digest of an IANA content type and 839 the digested data. A UDF key fingerprint is formed with the content 840 type "application/pkix-keyinfo" and the digested data is the ASN.1 841 DER encoded PKIX certificate "keyInfo" sequence for the corresponding 842 public key. 844 A Group Key Identifier has the form @. Where 845 is a UDF key fingerprint and is the DNS 846 address of a service that provides the encryption service to support 847 decryption by group members. 849 3.4.3. Salt Derivation 851 A Master Salt is a sequence of 16 or more octets that is specified in 852 the Salt field of the header. 854 The Master Salt is used to derive salt values for the envelope 855 payload and associated encoded data sequences as follows. 857 Payload Salt = Master Salt 859 EDS Salt = Concatenate (Payload Salt Prefix, Master Salt) 861 Encoders SHOULD NOT generate salt values that exceed 1024 octets. 863 The salt value is opaque to the DARE encoding but MAY be used to 864 encode application specific semantics including: 866 * Frame number to allow reassembly of a data sequence split over a 867 sequence of envelopes which may be delivered out of order. 869 * Transmit the Master Key in the manner of a Kerberos ticket. 871 * Identify the Master Key under which the Enhanced Data Sequence was 872 generated. 874 * Enable access to the plaintext to be eliminated by erasure of the 875 encryption key. 877 For data erasure to be effective, the salt MUST be constructed so 878 that the difficulty of recovering the key is sufficiently high that 879 it is infeasible. For most purposes, a salt with 128 bits of 880 appropriately random data is sufficient. 882 3.4.4. Key Derivation 884 Encryption and/or authentication keys are derived from the Master Key 885 using a Extract-and-Expand Key Derivation Function as follows: 887 0. The Master Key and salt value are used to extract the PRK 888 (pseudorandom key) 890 1. The PRK is used to derive the algorithm keys using the 891 application specific information input for that key type. 893 The application specific information inputs are: 895 "dare-encrypt" [64 61 72 65 2d 65 6e 63 72 79 70 74] To generate an 896 encryption or encryption with authentication key. 898 "dare-iv" [64 61 72 65 2d 65 6e 63 72 79 70 74] To generate an 899 initialization vector. 901 "dare-mac" [dare-mac] To generate a Message Authentication Code key. 903 3.5. Signature 905 While encryption and integrity enhancements can be applied to any 906 part of a DARE Envelope, signatures are only applied to payload 907 digest values calculated over one or more envelope payloads. 909 The payload digest value for an envelope is calculated over the 910 binary payload data. That is, after any encryption enhancement has 911 been applied but before the envelope encoding is applied. This 912 allows envelopes to be converted from one encoding to another without 913 affecting signature verification. 915 Single Payload The signed value is the payload digest of the 916 envelope payload. 918 Multiple Payload. The signed value is the root of a Merkle Tree in 919 which the payload digest of the envelope is one of the leaves. 921 Verification of a multiple payload signature naturally requires the 922 additional digest values required to construct the Merkle Tree. 923 These are provided in the Trailer in a format that permits multiple 924 signers to reference the same tree data. 926 3.6. Algorithms 928 3.6.1. Field: kwd 930 The key wrapping and derivation algorithms. 932 Since the means of public key exchange is determined by the key 933 identifier of the recipient key, it is only necessary to specify the 934 algorithms used for key wrapping and derivation. 936 The default (and so far only) algorithm is kwd-aes-sha2-256-256. 938 Advanced Encryption Standard (AES) Key Wrap with Padding Algorithm 939 [RFC3394] is used to wrap the Master Exchange Key. AES 256 is used. 941 HMAC-based Extract-and-Expand Key Derivation Function [RFC5869] is 942 used for key derivation. SHA-2-256 is used for the hash function. 944 4. DARE Container Architecture 945 4.1. Container Navigation 947 Three means of locating frames in a container are supported: 949 Sequential Access frames sequentially starting from the start or the 950 end of the container. 952 Binary search Access any container frame by frame number in 953 O(log_(2)(n)) time by means of a binary tree constructed while the 954 container is written. 956 Index Access and container frame by frame number or by key by means 957 of an index record. 959 All DARE Containers support sequential access. Only tree and Merkle 960 tree containers support binary search access. An index frame MAY be 961 written appended to any container and provides O(1) access to any 962 frame listed in the index. 964 Two modes of compilation are considered: 966 Monolithic Frames are added to the container in a single operation, 967 e.g. file archives, 969 Incremental Additional frames are written to the container at 970 various intervals after it was originally created, e.g. server 971 logs, message spools. 973 In the monolithic mode, navigation requirements are best met by 974 writing an index frame to the end of the container when it is 975 complete. It is not necessary to construct a binary search tree 976 unless a Merkle tree integrity check is required. 978 In the incremental mode, Binary search provides an efficient means of 979 locating frames by frame number but not by key. Writing a complete 980 index to the container every _m_ write operations provides _O(m)_ 981 search access but requires O(n^(2)) storage. 983 Use of partial indexes provides a better compromise between speed and 984 efficiency. A partial index is written out every _m_ frames where 985 _m_ is a power of two. A complete index is written every time a 986 binary tree apex record is written. This approach provides for 987 O(log_(2)(n)) search with incremental compilation with approximately 988 double the overhead of the monolithic case. 990 4.1.1. Tree 992 As previously described, the JBCD frame structure allows incremental 993 navigation to the immediately preceding frame. The "TreePosition" 994 parameter specifies the start position of _any_ previous frame in the 995 container, thus allowing rapid navigation to that point. 997 The TreePosition parameter MAY be used to enable any frame in the 998 container to be retrieved in log_(2)(n) time by means of a binary 999 search. The TreePosition parameter specifies the immediately 1000 preceding apex of a binary tree formed from the container entries. 1002 For example, the TreePosition of frame 6 in a container gives the 1003 location of frame 5, the TreePosition of frame 5 gives the location 1004 of frame 3, the TreePosition of frame 3 gives the location of frame 1005 1, and the TreePosition of frame 1 gives the location of frame 0: 1007 (Artwork only available as svg: No external link available, see 1008 draft-hallambaker-mesh-dare-11.html for artwork.) 1010 Figure 5: Binary search tree. 1012 An algorithm for efficiently calculating the immediately preceding 1013 apex is provided in Appendix C. 1015 4.1.2. Position Index 1017 Contains a table of frame number, position pairs pointing to prior 1018 locations in the file. 1020 4.1.3. Metadata Index 1022 Contains a list of IndexMeta entries. Each entry contains a metadata 1023 description and a list of frame indexes (not positions) of frames 1024 that match the description. 1026 4.2. Integrity Mechanisms 1028 Frame sequences in a DARE container MAY be protected against a frame 1029 insertion attack by means of a digest chain, a binary Merkle tree or 1030 both. 1032 4.2.1. Digest Chain calculation 1034 A digest chain is simple to implement but can only be verified if the 1035 full chain of values is known. Appending a frame to the chain has 1036 _O(1)_ complexity but verification has _O(n)_ complexity: 1038 (Artwork only available as svg: No external link available, see 1039 draft-hallambaker-mesh-dare-11.html for artwork.) 1041 Figure 6: Hash chain integrity check 1043 The value of the chain digest for the first frame (frame 0) is 1044 _H(H(null)+H(Payload_(0))), where null is a zero length octet 1045 sequence and payloadn is the sequence of payload data bytes for frame 1046 n_ 1048 The value of the chain digest for frame _n is H(H(Payload_(n-1) 1049 +H(Payloadn)), where A+B stands for concatenation of the byte 1050 sequences A and B._ 1052 4.2.2. Binary Merkle tree calculation 1054 The tree index mechanism describe earlier may be used to implement a 1055 binary Merkle tree. The value TreeDigest specifies the apex value of 1056 the tree for that node. 1058 Appending a frame to the chain has O(log_(2) (n)) complexity provided 1059 that the container format supports at least the binary tree index. 1060 Verifying a chain has O(log_(2) (n)) complexity, provided that the 1061 set of necessary digest inputs is known. 1063 To calculate the value of the tree digest for a node, we first 1064 calculate the values of all the sub trees that have their apex at 1065 that node and then calculate the digest of that value and the 1066 immediately preceding local apex. 1068 4.2.3. Signature 1070 Payload data MAY be signed using a JWS [RFC7515] as applied in the 1071 Envelope. 1073 Signatures are specified by the "Signatures" parameter in the content 1074 header. The data that the signature is calculated over is defined by 1075 the typ parameter of the Signature as follows. 1077 "Payload" The value of the "PayloadDigest" parameter 1079 "Chain" The value of the "ChainDigest" parameter 1081 "Tree" The value of the "TreeDigestFinal" parameter 1083 If the "typ" parameter is absent, the value Payload is implied. 1085 A frame MAY contain multiple signatures created with the same signing 1086 key and different typ values. 1088 The use of signatures over chain and tree digest values permit 1089 multiple frames to be validated using a single signature verification 1090 operation. 1092 5. DARE Schema 1094 A DARE Envelope consists of a Header, an Enhanced Data Sequence (EDS) 1095 and an optional trailer. This section describes the JSON data fields 1096 used to construct headers, trailers and complete envelopes. 1098 Wherever possible, fields from JWE, JWS and JWK have been used. In 1099 these cases, the fields have the exact same semantics. Note however 1100 that the classes in which these fields are presented have different 1101 structure and nesting. 1103 5.1. Envelope Classes 1105 A DARE envelope contains a single DAREMessageSequence in either the 1106 JSON or Compact serialization as directed by the protocol in which it 1107 is applied. 1109 5.1.1. Structure: DareEnvelopeSequence 1111 A DARE envelope containing Header, EDS and Trailer in JSON object 1112 encoding. Since a DAREMessage is almost invariably presented in JSON 1113 sequence or compact encoding, use of the DAREMessage subclass is 1114 preferred. 1116 Although a DARE envelope is functionally an object, it is serialized 1117 as an ordered sequence. This ensures that the envelope header field 1118 will always precede the body in a serialization, this allowing 1119 processing of the header information to be performed before the 1120 entire body has been received. 1122 Header: DareHeader (Optional) The envelope header. May specify the 1123 key exchange data, pre-signature or signature data, cloaked 1124 headers and/or encrypted data sequences. 1126 Body: Binary (Optional) The envelope body 1128 Trailer: DareTrailer (Optional) The envelope trailer. If present, 1129 this contains the signature. 1131 5.2. Header and Trailer Classes 1133 A DARE sequence MUST contain a (possibly empty) DAREHeader and MAY 1134 contain a DARETrailer. 1136 5.2.1. Structure: DareTrailer 1138 A DARE envelope Trailer 1140 Signatures: DareSignature [0..Many] A list of signatures. A 1141 envelope trailer MUST NOT contain a signatures field if the header 1142 contains a signatures field. 1144 SignedData: Binary (Optional) Contains a DAREHeader object 1146 PayloadDigest: Binary (Optional) If present, contains the digest of 1147 the Payload. 1149 ChainDigest: Binary (Optional) If present, contains the digest of 1150 the PayloadDigest values of this frame and the frame immediately 1151 preceding. 1153 TreeDigest: Binary (Optional) If present, contains the Binary Merkle 1154 Tree digest value. 1156 5.2.2. Structure: DareHeader 1158 Inherits: DareTrailer 1160 A DARE Envelope Header. Since any field that is present in a trailer 1161 MAY be placed in a header instead, the envelope header inherits from 1162 the trailer. 1164 EnvelopeId: String (Optional) Unique identifier 1166 EncryptionAlgorithm: String (Optional) The encryption algorithm as 1167 specified in JWE 1169 DigestAlgorithm: String (Optional) Digest Algorithm. If specified, 1170 tells decoder that the digest algorithm is used to construct a 1171 signature over the envelope payload. 1173 KeyIdentifier: String (Optional) Base seed identifier. 1175 Salt: Binary (Optional) Salt value used to derrive cryptographic 1176 parameters for the content data. 1178 Malt: Binary (Optional) Hash of the Salt value used to derrive 1179 cryptographic parameters for the content data. This field SHOULD 1180 NOT be present if the Salt field is present. It is used to allow 1181 the salt value to be erased (thus rendering the payload content 1182 irrecoverable) without affecting the ability to calculate the 1183 payload digest value. 1185 Cloaked: Binary (Optional) If present in a header or trailer, 1186 specifies an encrypted data block containing additional header 1187 fields whose values override those specified in the envelope and 1188 context headers. 1190 When specified in a header, a cloaked field MAY be used to conceal 1191 metadata (content type, compression) and/or to specify an 1192 additional layer of key exchange. That applies to both the 1193 envelope body and to headers specified within the cloaked header. 1195 Processing of cloaked data is described in... 1197 EDSS: Binary [0..Many] If present, the Annotations field contains a 1198 sequence of Encrypted Data Segments encrypted under the envelope 1199 base seed. The interpretation of these fields is application 1200 specific. 1202 Signers: DareSignature [0..Many] A list of 'presignature' 1204 Recipients: DareRecipient [0..Many] A list of recipient key exchange 1205 information blocks. 1207 Policy: DarePolicy (Optional) A DARE security policy governing 1208 future additions to the container. 1210 ContentMetaData: Binary (Optional) If present contains a JSON 1211 encoded ContentInfo structure which specifies plaintext content 1212 metadata and forms one of the inputs to the envelope digest value. 1214 SequenceInfo: SequenceInfo (Optional) Information that describes 1215 container information 1217 SequenceIndex: SequenceIndex (Optional) An index of records in the 1218 current container up to but not including this one. 1220 Received: DateTime (Optional) Date on which the envelope was 1221 received. 1223 5.2.3. Structure: ContentMeta 1225 UniqueId: String (Optional) Unique object identifier 1226 Labels: String [0..Many] List of labels that are applied to the 1227 payload of the frame. 1229 KeyValues: KeyValue [0..Many] List of key/value pairs describing the 1230 payload of the frame. 1232 MessageType: String (Optional) The mesh message type 1234 ContentType: String (Optional) The content type field as specified 1235 in JWE 1237 Paths: String [0..Many] List of filename paths for the payload of 1238 the frame. 1240 Filename: String (Optional) The original filename under which the 1241 data was stored. 1243 Event: String (Optional) Operation on the header 1245 Created: DateTime (Optional) Initial creation date. 1247 Modified: DateTime (Optional) Date of last modification. 1249 Expire: DateTime (Optional) Date at which the associated transaction 1250 will expire 1252 First: Integer (Optional) Frame number of the first object instance 1253 value. 1255 Previous: Integer (Optional) Frame number of the immediately prior 1256 object instance value 1258 FileEntry: FileEntry (Optional) Information describing the file 1259 entry on disk. 1261 5.3. Cryptographic Data 1263 DARE envelope uses the same fields as JWE and JWS but with different 1264 structure. In particular, DARE envelopes MAY have multiple 1265 recipients and multiple signers. 1267 5.3.1. Structure: DareSignature 1269 The signature value 1271 Dig: String (Optional) Digest algorithm hint. Specifying the digest 1272 algorithm to be applied to the envelope body allows the body to be 1273 processed in streaming mode. 1275 Alg: String (Optional) Key exchange algorithm 1277 KeyIdentifier: String (Optional) Key identifier of the signature 1278 key. 1280 Certificate: X509Certificate (Optional) PKIX certificate of signer. 1282 Path: X509Certificate (Optional) PKIX certificates that establish a 1283 trust path for the signer. 1285 Manifest: Binary (Optional) The data description that was signed. 1287 SignatureValue: Binary (Optional) The signature value as an Enhanced 1288 Data Sequence under the envelope base seed. 1290 WitnessValue: Binary (Optional) The signature witness value used on 1291 an encrypted envelope to demonstrate that the signature was 1292 authorized by a party with actual knowledge of the encryption key 1293 used to encrypt the envelope. 1295 5.3.2. Structure: X509Certificate 1297 X5u: String (Optional) URL identifying an X.509 public key 1298 certificate 1300 X5: Binary (Optional) An X.509 public key certificate 1302 5.3.3. Structure: DareRecipient 1304 Recipient information 1306 KeyIdentifier: String (Optional) Key identifier for the encryption 1307 key. 1309 The Key identifier MUST be either a UDF fingerprint of a key or a 1310 Group Key Identifier 1312 KeyWrapDerivation: String (Optional) The key wrapping and derivation 1313 algorithms. 1315 WrappedBaseSeed: Binary (Optional) The wrapped base seed. The base 1316 seed is encrypted under the result of the key exchange. 1318 RecipientKeyData: String (Optional) The per-recipient key exchange 1319 data. 1321 5.3.4. Structure: DarePolicy 1323 EncryptionAlgorithm: String (Optional) The encryption algorithm to 1324 be used to compute the payload. 1326 DigestAlgorithm: String (Optional) The digest algorithm to be used 1327 to compute the payload digest. 1329 Encryption: String (Optional) The encryption policy specifier, 1330 determines how often a key exchange is required. 'Single': All 1331 entries are encrypted under the key exchange specified in the 1332 entry specifying this policy. 'Isolated': All entries are 1333 encrypted under a separate key exchange. 'All': All entries are 1334 encrypted. 'None': No entries are encrypted. 1336 Default value is 'None' if EncryptKeys is null, and 'All' 1337 otherwise. 1339 Signature: String (Optional) The signature policy 'None': No entries 1340 are signed. 'Last': The last entry in the container is signed. 1341 'Isolated': All entries are independently signed. 'Any': Entries 1342 may be signed. 1344 Default value is 'None' if SignKeys is null, and 'Any' otherwise. 1346 Sealed: Boolean (Optional) If true the policy is immutable and 1347 cannot be changed by a subsequent policy override. 1349 5.3.5. Structure: FileEntry 1351 Path: String (Optional) The file path in canonical form. 1353 CreationTime: DateTime (Optional) The creation time of the file on 1354 disk in UTC 1356 LastAccessTime: DateTime (Optional) The last access time of the file 1357 on disk in UTC 1359 LastWriteTime: DateTime (Optional) The last write time of the file 1360 on disk in UTC 1362 Attributes: Integer (Optional) The file attribues as a bitmapped 1363 integer. 1365 6. DARE Container Schema 1367 TBS stuff 1369 6.1. Container Headers 1371 TBS stuff 1373 6.1.1. Structure: SequenceInfo 1375 Information that describes container information 1377 DataEncoding: String (Optional) Specifies the data encoding for the 1378 header section of for the following frames. This value is ONLY 1379 valid in Frame 0 which MUST have a header encoded in JSON. 1381 ContainerType: String (Optional) Specifies the container type for 1382 the following records. This value is ONLY valid in Frame 0 which 1383 MUST have a header encoded in JSON. 1385 Index: Integer (Optional) The record index within the file. This 1386 MUST be unique and satisfy any additional requirements determined 1387 by the ContainerType. 1389 IsMeta: Boolean (Optional) If true, the current frame is a meta 1390 frame and does not contain a payload. 1392 Note: Meta frames MAY be present in any container. Applications 1393 MUST accept containers that contain meta frames at any position in 1394 the file. Applications MUST NOT interpret a meta frame as a data 1395 frame with an enpty payload. 1397 Default: Boolean (Optional) If set true in a persistent container, 1398 specifies that this record contains the default object for the 1399 container. 1401 TreePosition: Integer (Optional) Position of the frame containing 1402 the apex of the preceding sub-tree. 1404 IndexPosition: Integer (Optional) Specifies the position in the file 1405 at which the last index entry is to be found 1407 ExchangePosition: Integer (Optional) Specifies the position in the 1408 file at which the key exchange data is to be found 1410 6.2. Index Structures 1412 TBS stuff 1414 6.2.1. Structure: SequenceIndex 1416 A container index 1418 Full: Boolean (Optional) If true, the index is complete and contains 1419 position entries for all the frames in the file. If absent or 1420 false, the index is incremental and only contains position entries 1421 for records added since the last frame containing a 1422 ContainerIndex. 1424 Positions: IndexPosition [0..Many] List of container position 1425 entries 1427 6.2.2. Structure: IndexPosition 1429 Specifies the position in a file at which a specified record index is 1430 found 1432 Index: Integer (Optional) The record index within the file. 1434 Position: Integer (Optional) The record position within the file 1435 relative to the index base. 1437 UniqueId: String (Optional) Unique object identifier 1439 6.2.3. Structure: KeyValue 1441 Specifies a key/value entry 1443 Key: String (Optional) The key 1445 Value: String (Optional) The value corresponding to the key 1447 7. Dare Container Applications 1449 DARE Containers are used to implement two forms of persistence store 1450 to support Mesh operations: 1452 Catalogs A set of related items which MAY be added, modified or 1453 deleted at any time. 1455 Spools A list of related items whose status MAY be changed at any 1456 time but which are immutable once added. 1458 Since DARE Containers are an append only log format, entries can only 1459 be modified or deleted by adding items to the log to change the 1460 status of previous entries. It is always possible to undo any 1461 operation on a catalog or spool unless the underlying container is 1462 purged or the individual entries modified. 1464 7.1. Catalog 1466 Catalogs contain a set of entries, each of which is distinguished by 1467 a unique identifier. 1469 Three operations are supported: 1471 Add Addition of the entry to the catalog 1473 Update Modification of the data associated with the entry excluding 1474 the identifier 1476 Delete Removal of the entry from the catalog 1478 The set of valid state transitions is defined by the Finite State 1479 machine: 1481 (Add-Update*-Delete)* 1483 Catalogs are used to represent sets of persistent objects associated 1484 with a Mesh Service Account. The user's set of contacts for example. 1485 Each contact entry may be modified many times over time but refers to 1486 the same subject for its entire lifetime. 1488 7.2. Spool 1490 Spools contain lists of entries, each of which is distinguished by a 1491 unique identifier. 1493 Four operations are supported: 1495 Post Addition of the entry to the spool 1497 Processed Marks the entry as having been processed. 1499 Unprocessed Returns the entry to the unread state. 1501 Delete Mark the entry as deleted allowing recovery of associated 1502 storage in a subsequent purge operation. 1504 The set of valid state transitions is defined by the Finite State 1505 machine: 1507 Post-(Processed| Unprocessed| Delete *) 1509 Spools are used to represent time sequence ordered entries such as 1510 lists of messages being sent or received, task queues and transaction 1511 logs. 1513 7.3. Archive 1515 A DARE Archive is a DARE Container whose entries contain files. This 1516 affords the same functionality as a traditional ZIP or tar archive 1517 but with the added cryptographic capabilities provided by the DARE 1518 format. 1520 8. Future Work 1522 The current specification describes an approach in which containers 1523 are written according to a strict append-only policy. Greater 1524 flexibility may be achieved by loosening this requirement allowing 1525 record(s) at the end of the container to be overwritten. 1527 8.1. Terminal integrity check 1529 A major concern when operating a critical service is the possibility 1530 of a hardware or power failure occurring during a write operation 1531 causing the file update to be incomplete. While most modern 1532 operating systems have effective mechanisms in place to prevent 1533 corruption of the file system itself in such circumstances, this does 1534 not provide sufficient protection at the application level. 1536 Appending a null record containing a container-specific magic number 1537 provides an effective means of detecting this circumstance that can 1538 be quickly verified. 1540 If a container specifies a terminal integrity check value in the 1541 header of frame zero, the container is considered to be in an 1542 incomplete write state if the final frame is not a null record 1543 specifying the magic number. 1545 When appending new records to such containers, the old terminal 1546 integrity check record is overwritten by the data being added and a 1547 new integrity check record appended to the end. 1549 8.2. Terminal index record 1551 A writer can maintain a complete (or partial) index of the container 1552 in its final record without additional space overhead by overwriting 1553 the prior index on each update. 1555 8.3. Deferred indexing 1557 The task of updating terminal indexes may be deferred to a time when 1558 the machine is not busy. This improves responsiveness and may avoid 1559 the need to re-index containers receiving a sequence of updates. 1561 This approach may be supported by appending new entries to the end of 1562 the container in the usual fashion and maintaining a record of 1563 containers to be updated as a separate task. 1565 When updating the index on a container that has been updated in this 1566 fashion, the writer must ensure that no data is lost even if the 1567 process is interrupted. The use of guard records and other 1568 precautions against loss of state is advised. 1570 9. Security Considerations 1572 This section describes security considerations arising from the use 1573 of DARE in general applications. 1575 Additional security considerations for use of DARE in Mesh services 1576 and applications are described in the Mesh Security Considerations 1577 guide [draft-hallambaker-mesh-security]. 1579 9.1. Encryption/Signature nesting 1581 9.2. Side channel 1583 9.3. Salt reuse 1585 10. IANA Considerations 1587 11. Acknowledgements 1589 A list of people who have contributed to the design of the Mesh is 1590 presented in [draft-hallambaker-mesh-architecture]. 1592 The name Data At Rest Encryption was proposed by Melhi Abdulhayo?lu. 1594 12. Appendix A: DARE Envelope Examples and Test Vectors 1596 13. Test Examples 1598 In the following examples, Alice's encryption private key parameters 1599 are: 1601 { 1602 "PrivateKeyECDH":{ 1603 "crv":"Ed25519", 1604 "Private":"6o3XJvRF5WrG4-DY3hBG_qPb3j5JiAvWPfts3VhdQeg"}} 1606 Alice's signature private key parameters are: 1608 { 1609 "PrivateKeyECDH":{ 1610 "crv":"Ed25519", 1611 "Private":"3JhYW8Q2YHJ2_cPjLYOLbzprvecQi4xsfuat0eOKLxw"}} 1613 The body of the test message is the UTF8 representation of the 1614 following string: 1616 "This is a test long enough to require multiple blocks" 1618 The EDS sequences, are the UTF8 representation of the following 1619 strings: 1621 "Subject: Message metadata should be encrypted" 1622 "2018-02-01" 1624 13.1. Plaintext Message 1626 A plaintext message without associated EDS sequences is an empty 1627 header followed by the message body: 1629 { 1630 "DareEnvelope":[{}, 1631 "VGhpcyBpcyBhIHRlc3QgbG9uZyBlbm91Z2ggdG8gcmVxdWlyZSBtdWx0aXBs 1632 ZSBibG9ja3M" 1633 ]} 1635 13.2. Plaintext Message with EDS 1637 If a plaintext message contains EDS sequences, these are also in 1638 plaintext: 1640 { 1641 "DareEnvelope":[{ 1642 "annotations":["iAEAiC1TdWJqZWN0OiBNZXNzYWdlIG1ldGFkYXRhIHNo 1643 b3VsZCBiZSBlbmNyeXB0ZWQ", 1644 "iAEBiAoyMDE4LTAyLTAx" 1645 ]}, 1646 "VGhpcyBpcyBhIHRlc3QgbG9uZyBlbm91Z2ggdG8gcmVxdWlyZSBtdWx0aXBs 1647 ZSBibG9ja3M" 1648 ]} 1650 13.3. Encrypted Message 1652 The creator generates a base seed: 1654 60 14 70 EB 98 22 EB 47 61 24 46 E2 C8 F0 86 31 1655 95 DD 37 38 C7 08 08 8C 1F 16 05 EE B7 CC F3 8B 1657 For each recipient of the message: 1659 The creator generates an ephemeral key: 1661 { 1662 "PrivateKeyECDH":{ 1663 "crv":"Ed25519", 1664 "Private":"c3B3RFkR8tSm71a0nzZmHbWHgPPtrjGdu6ZB21NjBQ4"}} 1666 The key agreement value is calculated: 1668 D5 CB 99 BE FC F0 BA DF FA C5 9A 29 6E AE 8F DB 1669 76 4E B0 95 76 F1 03 36 E7 1E 78 DA FC 8D EF 72 1671 The key agreement value is used as the input to a HKDF key derivation 1672 function with the info parameter master to create the key used to 1673 wrap the base seed: 1675 3C 79 76 F7 22 61 62 CB 2B C8 98 31 16 75 71 39 1676 CC E3 9E 6A 25 6C F9 21 A9 6C 58 F2 DB 97 23 27 1678 The wrapped base seed is: 1680 41 9C 2D 1D 05 33 5D 2E AA 96 2A 46 BE 34 21 CF 1681 FD 06 8C 73 45 57 6E EC 01 43 FF 9C F7 F9 BF 14 1682 C1 25 BB 71 51 B1 25 F9 1684 This information is used to calculate the Recipient information shown 1685 in the example below. 1687 To encrypt a message, we first generate a unique salt value: 1689 6D 64 72 61 7D 48 99 D4 04 F2 36 7F DA 78 E1 8D 1691 The base seed and salt value are used to generate the payload 1692 encryption key: 1694 84 7F 2C EB 72 4A CB B2 0F 66 21 81 30 76 43 06 1695 4F 63 80 B4 F2 68 68 EB CA 7B 96 B7 FC 04 DF DC 1697 Since AES is a block cipher, we also require an initializarion 1698 vector: 1700 A2 87 C4 F3 FA 8A 89 5B 4F CA 40 6E BD E2 E5 76 1702 The output sequence is the encrypted bytes: 1704 5A 34 06 42 1C 6A 5C 4B BF D7 44 43 4B F8 A2 B7 1705 3E 56 65 27 AD AC 98 B1 07 59 2F F8 43 D5 A0 53 1706 13 A1 BB D3 90 76 FB F5 1C 30 58 FB 0B 06 67 54 1707 7A 66 FB 1D 43 50 3E C5 21 15 BA 79 AB 96 C1 1F 1709 Since the message is not signed, there is no need for a trailer. The 1710 completed message is: 1712 { 1713 "DareEnvelope":[{ 1714 "enc":"A256CBC", 1715 "kid":"EBQG-J7ZP-4NPO-BS3J-YFE4-3YPM-TICI", 1716 "Salt":"bWRyYX1ImdQE8jZ_2njhjQ", 1717 "recipients":[{ 1718 "kid":"MCZG-SJCC-WPAU-MNRF-6TCG-KUUQ-ICAQ", 1719 "epk":{ 1720 "PublicKeyECDH":{ 1721 "crv":"Ed25519", 1722 "Public":"5DmkW-Ieh_XdDKRYKi-_4gPeH3QwdNZ4BYajMsxdk 1723 e0"}}, 1724 "wmk":"8-F2jDSBxmuCE2iYqP-BVcZiZUKcE8CrL3Vxis_487W-xjUx 1725 3b8F-A"} 1726 ]}, 1727 "WjQGQhxqXEu_10RDS_iitz5WZSetrJixB1kv-EPVoFMTobvTkHb79RwwWPsL 1728 BmdUemb7HUNQPsUhFbp5q5bBHw" 1729 ]} 1731 13.4. Signed Message 1733 Signed messages specify the digest algorithm to be used in the header 1734 and the signature value in the trailer. Note that the digest 1735 algorithm is not optional since it serves as notice that a decoder 1736 should digest the payload value to enable signature verification. 1738 { 1739 "DareEnvelope":[{ 1740 "dig":"S512"}, 1741 "VGhpcyBpcyBhIHRlc3QgbG9uZyBlbm91Z2ggdG8gcmVxdWlyZSBtdWx0aXBs 1742 ZSBibG9ja3M", 1743 { 1744 "signatures":[{ 1745 "alg":"S512", 1746 "kid":"MCQU-ONOR-SMME-AJM3-4RNU-2EDR-KWX4", 1747 "signature":"NthXHWAJ3b4r9ulhG5VRzcH3d-_RJ7PlT7F9wzD-dr 1748 XFLFVTUfrvuXteEuBaE4-9c-DLWGc_pIc9eGLjXBB9Dw"} 1749 ], 1750 "PayloadDigest":"raim8SV5adPbWWn8FMM4mrRAQCO9A2jZ0NZAnFXWlG 1751 0xF6sWGJbnKSdtIJMmMU_hjarlIPEoY3vy9UdVlH5KAg"} 1752 ]} 1754 13.5. Signed and Encrypted Message 1756 A signed and encrypted message is encrypted and then signed. The 1757 signer proves knowledge of the payload plaintext by providing the 1758 plaintext witness value. 1760 { 1761 "DareEnvelope":[{ 1762 "enc":"A256CBC", 1763 "dig":"S512", 1764 "kid":"EBQE-FQSV-HURW-44LI-W7E3-4HMX-AR37", 1765 "Salt":"zel4BwvOFCzTs6BrNL0yzQ", 1766 "recipients":[{ 1767 "kid":"MCZG-SJCC-WPAU-MNRF-6TCG-KUUQ-ICAQ", 1768 "epk":{ 1769 "PublicKeyECDH":{ 1770 "crv":"Ed25519", 1771 "Public":"bYxb8Q4_gnY3CKnppk-82V33DHseB_caD8vu7s8wT 1772 40"}}, 1773 "wmk":"hiLLca5zdVe8BFkT8n5dPxNOlDa5l4bhCXnjFRvSil9J19g7 1774 tnwMIQ"} 1775 ]}, 1776 "KlaXVXwrVm1t5xny6xD8U1qAvOo6Iaw1uPV_7-XvmplQCU5sGT9kyZ6Xay3H 1777 AKP3gJ8CsxriOvxZjMwnwn4OZg", 1778 { 1779 "signatures":[{ 1780 "alg":"S512", 1781 "kid":"MCQU-ONOR-SMME-AJM3-4RNU-2EDR-KWX4", 1782 "signature":"9pXy6nHEzFjYSohDG9lR1h2ml4bvHGz6jGbMzCm_dn 1783 8ANcNZR_E6K7JM-sTexlooTgmdMKflGylCvqNJFiyNAA", 1784 "witness":"eqhzn-rVajjxuYGTjpOVNjJOOkeGiYa5E9iUnSr3wSg"} 1785 ], 1786 "PayloadDigest":"HNflV2DRiLjMWSWfJ5VZW7oeXA5YxmDuq0MN9F5rCG 1787 Kwh58BoxTE7CNyH85Cv4EH6-fKxtlmJayUmS4oxTv0Pw"} 1788 ]} 1790 14. Appendix B: DARE Container Examples and Test Vectors 1792 The data payloads in all the following examples are identical, only 1793 the authentication and/or encryption is different. 1795 * Frame 1..n consists of 300 bytes being the byte sequence 00, 01, 1796 02, etc. repeating after 256 bytes. 1798 For conciseness, the raw data format is omitted for examples after 1799 the first, except where the data payload has been transformed, (i.e. 1800 encrypted). 1802 14.1. Simple sequence 1804 The following example shows a simple sequence with first frame and a 1805 single data frame: 1807 f4 91 1808 f0 8b 1809 f0 00 1810 f0 00 1811 91 f4 1812 f5 01 73 1813 f0 42 1814 f1 01 2c 1815 73 01 f5 1817 Since there is no integrity check, there is no need for trailer 1818 entries. The header values are: 1820 Frame 0 1822 { 1823 "DareHeader":{ 1824 "policy":{}, 1825 "ContentMetaData":"e30", 1826 "SequenceInfo":{ 1827 "DataEncoding":"JSON", 1828 "ContainerType":"List", 1829 "Index":0}}} 1831 [Empty trailer] 1833 Frame 1 1835 { 1836 "DareHeader":{ 1837 "ContentMetaData":"e30", 1838 "SequenceInfo":{ 1839 "Index":1}}} 1841 [Empty trailer] 1843 14.2. Payload and chain digests 1845 The following example shows a chain sequence with a first frame and 1846 three data frames. The headers of these frames is the same as before 1847 but the frames now have trailers specifying the PayloadDigest and 1848 ChainDigest values: 1850 Frame 0 1851 { 1852 "DareHeader":{ 1853 "dig":"S512", 1854 "policy":{}, 1855 "ContentMetaData":"e30", 1856 "SequenceInfo":{ 1857 "DataEncoding":"JSON", 1858 "ContainerType":"Chain", 1859 "Index":0}}} 1861 [Empty trailer] 1863 Frame 1 1865 { 1866 "DareHeader":{ 1867 "dig":"S512", 1868 "ContentMetaData":"e30", 1869 "SequenceInfo":{ 1870 "Index":1}}} 1872 { 1873 "DareTrailer":{ 1874 "PayloadDigest":"8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZ 1875 DlZeaWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw", 1876 "ChainDigest":"T7S1FcrgY3AaWD4L-t5W1K-3XYkPTcOdGEGyjglTD6yMYV 1877 RVz9tn_KQc6GdA-P4VSRigBygV65OEd2Vv3YDhww"}} 1879 Frame 2 1881 { 1882 "DareHeader":{ 1883 "dig":"S512", 1884 "ContentMetaData":"e30", 1885 "SequenceInfo":{ 1886 "Index":2}}} 1888 { 1889 "DareTrailer":{ 1890 "PayloadDigest":"8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZ 1891 DlZeaWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw", 1892 "ChainDigest":"T7S1FcrgY3AaWD4L-t5W1K-3XYkPTcOdGEGyjglTD6yMYV 1893 RVz9tn_KQc6GdA-P4VSRigBygV65OEd2Vv3YDhww"}} 1895 Frame 3 1896 { 1897 "DareHeader":{ 1898 "dig":"S512", 1899 "ContentMetaData":"e30", 1900 "SequenceInfo":{ 1901 "Index":3}}} 1903 { 1904 "DareTrailer":{ 1905 "PayloadDigest":"8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZ 1906 DlZeaWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw", 1907 "ChainDigest":"T7S1FcrgY3AaWD4L-t5W1K-3XYkPTcOdGEGyjglTD6yMYV 1908 RVz9tn_KQc6GdA-P4VSRigBygV65OEd2Vv3YDhww"}} 1910 14.3. Merkle Tree 1912 The following example shows a chain sequence with a first frame and 1913 six data frames. The trailers now contain the TreePosition and 1914 TreeDigest values: 1916 Frame 0 1918 { 1919 "DareHeader":{ 1920 "dig":"S512", 1921 "policy":{}, 1922 "ContentMetaData":"e30", 1923 "SequenceInfo":{ 1924 "DataEncoding":"JSON", 1925 "ContainerType":"Merkle", 1926 "Index":0}}} 1928 [Empty trailer] 1930 Frame 1 1931 { 1932 "DareHeader":{ 1933 "dig":"S512", 1934 "ContentMetaData":"e30", 1935 "SequenceInfo":{ 1936 "Index":1, 1937 "TreePosition":0}}} 1939 { 1940 "DareTrailer":{ 1941 "PayloadDigest":"8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZ 1942 DlZeaWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw", 1943 "TreeDigest":"T7S1FcrgY3AaWD4L-t5W1K-3XYkPTcOdGEGyjglTD6yMYVR 1944 Vz9tn_KQc6GdA-P4VSRigBygV65OEd2Vv3YDhww"}} 1946 Frame 2 1948 { 1949 "DareHeader":{ 1950 "dig":"S512", 1951 "ContentMetaData":"e30", 1952 "SequenceInfo":{ 1953 "Index":2, 1954 "TreePosition":392}}} 1956 { 1957 "DareTrailer":{ 1958 "PayloadDigest":"8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZ 1959 DlZeaWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw", 1960 "TreeDigest":"7fHmkEIsPkN6sDYAOLvpIJn5Dg3PxDDAaq-ll2kh8722kok 1961 kFnZQcYtjuVC71aHNXI18q-lPnfRkmwryG-bhqQ"}} 1963 Frame 3 1965 { 1966 "DareHeader":{ 1967 "dig":"S512", 1968 "ContentMetaData":"e30", 1969 "SequenceInfo":{ 1970 "Index":3, 1971 "TreePosition":392}}} 1973 { 1974 "DareTrailer":{ 1975 "PayloadDigest":"8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZ 1976 DlZeaWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw", 1977 "TreeDigest":"T7S1FcrgY3AaWD4L-t5W1K-3XYkPTcOdGEGyjglTD6yMYVR 1978 Vz9tn_KQc6GdA-P4VSRigBygV65OEd2Vv3YDhww"}} 1980 Frame 4 1982 { 1983 "DareHeader":{ 1984 "dig":"S512", 1985 "ContentMetaData":"e30", 1986 "SequenceInfo":{ 1987 "Index":4, 1988 "TreePosition":1676}}} 1990 { 1991 "DareTrailer":{ 1992 "PayloadDigest":"8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZ 1993 DlZeaWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw", 1994 "TreeDigest":"vJ6ngNATvZcXSMALi5IUqzl1GBxBnTNVcC87VL_BhMRCbAv 1995 KSj8gs0VFgxxLkZ2myrtaDIwhHoswiTiBMLNWug"}} 1997 Frame 5 1999 { 2000 "DareHeader":{ 2001 "dig":"S512", 2002 "ContentMetaData":"e30", 2003 "SequenceInfo":{ 2004 "Index":5, 2005 "TreePosition":1676}}} 2007 { 2008 "DareTrailer":{ 2009 "PayloadDigest":"8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZ 2010 DlZeaWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw", 2011 "TreeDigest":"T7S1FcrgY3AaWD4L-t5W1K-3XYkPTcOdGEGyjglTD6yMYVR 2012 Vz9tn_KQc6GdA-P4VSRigBygV65OEd2Vv3YDhww"}} 2014 Frame 6 2015 { 2016 "DareHeader":{ 2017 "dig":"S512", 2018 "ContentMetaData":"e30", 2019 "SequenceInfo":{ 2020 "Index":6, 2021 "TreePosition":2963}}} 2023 { 2024 "DareTrailer":{ 2025 "PayloadDigest":"8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZ 2026 DlZeaWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw", 2027 "TreeDigest":"WgHlz3EHczVPqgtpc39Arv7CFIsCbFVsk8wg0j2qLlEfur9 2028 SZ0mdr65Ka-HF0Qx8gg_DAoiJwUrwADDXyxVJOg"}} 2030 14.4. Signed sequence 2032 The following example shows a tree sequence with a signature in the 2033 final record. The signing key parameters are: 2035 { 2036 "PrivateKeyECDH":{ 2037 "crv":"Ed25519", 2038 "Private":"3JhYW8Q2YHJ2_cPjLYOLbzprvecQi4xsfuat0eOKLxw"}} 2040 The sequence headers and trailers are: 2042 Frame 0 2044 { 2045 "DareHeader":{ 2046 "dig":"S512", 2047 "policy":{}, 2048 "ContentMetaData":"e30", 2049 "SequenceInfo":{ 2050 "DataEncoding":"JSON", 2051 "ContainerType":"Merkle", 2052 "Index":0}}} 2054 [Empty trailer] 2056 Frame 1 2057 { 2058 "DareHeader":{ 2059 "dig":"S512", 2060 "ContentMetaData":"e30", 2061 "SequenceInfo":{ 2062 "Index":1, 2063 "TreePosition":0}}} 2065 { 2066 "DareTrailer":{ 2067 "PayloadDigest":"8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZ 2068 DlZeaWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw", 2069 "TreeDigest":"T7S1FcrgY3AaWD4L-t5W1K-3XYkPTcOdGEGyjglTD6yMYVR 2070 Vz9tn_KQc6GdA-P4VSRigBygV65OEd2Vv3YDhww"}} 2072 Frame 2 2074 { 2075 "DareHeader":{ 2076 "dig":"S512", 2077 "ContentMetaData":"e30", 2078 "SequenceInfo":{ 2079 "Index":2, 2080 "TreePosition":392}}} 2082 { 2083 "DareTrailer":{ 2084 "PayloadDigest":"8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZ 2085 DlZeaWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw", 2086 "TreeDigest":"7fHmkEIsPkN6sDYAOLvpIJn5Dg3PxDDAaq-ll2kh8722kok 2087 kFnZQcYtjuVC71aHNXI18q-lPnfRkmwryG-bhqQ"}} 2089 14.5. Encrypted sequence 2091 The following example shows a sequence in which all the frame 2092 payloads are encrypted under the same base seed established in a key 2093 agreement specified in the first frame. 2095 Frame 0 2096 { 2097 "DareHeader":{ 2098 "enc":"A256CBC", 2099 "kid":"EBQM-SNWC-5R6N-4JVS-JUHM-364P-FO7Y", 2100 "Salt":"eK7VIQt1h6emMMXXJDagGA", 2101 "recipients":[{ 2102 "kid":"MCZG-SJCC-WPAU-MNRF-6TCG-KUUQ-ICAQ", 2103 "epk":{ 2104 "PublicKeyECDH":{ 2105 "crv":"Ed25519", 2106 "Public":"pHHafE97kxgz89p-IwkFMlbkWUHzXJ-0aTMqDJIS0Y0"}}, 2107 "wmk":"ptMrMLScJdZ7AmxcUGxJP6jQBCaunNYNbTMUQNqaQI6c4Rs7uE 2108 fCVg"} 2109 ], 2110 "policy":{ 2111 "enc":"none", 2112 "dig":"none", 2113 "EncryptKeys":[{ 2114 "PublicKeyECDH":{ 2115 "crv":"Ed25519", 2116 "Public":"-NDENZ8r4M2h3G_W7iW6IOBJlHCjMLxWnwsMnIOBcxo"}} 2117 ], 2118 "Sealed":true}, 2119 "ContentMetaData":"e30", 2120 "SequenceInfo":{ 2121 "DataEncoding":"JSON", 2122 "ContainerType":"List", 2123 "Index":0}}} 2125 [Empty trailer] 2127 Frame 1 2128 { 2129 "DareHeader":{ 2130 "enc":"A256CBC", 2131 "kid":"EBQI-WIWG-AI3H-DDR3-X6CC-CP5Q-DJIR", 2132 "Salt":"QHNWV-Q5Amrce1_r-YA4BA", 2133 "recipients":[{ 2134 "kid":"MCZG-SJCC-WPAU-MNRF-6TCG-KUUQ-ICAQ", 2135 "epk":{ 2136 "PublicKeyECDH":{ 2137 "crv":"Ed25519", 2138 "Public":"GLlHst-jwA9J6Dj0x0ktFPh4_feoq9epvvdzKvd6Ils"}}, 2139 "wmk":"2M4v64eG0D-9k50EnHSFFbPbQ5GKgtu_dRET-dHFRjAGfmj6JC 2140 EeKQ"} 2141 ], 2142 "ContentMetaData":"e30", 2143 "SequenceInfo":{ 2144 "Index":1}}} 2146 [Empty trailer] 2148 Frame 2 2150 { 2151 "DareHeader":{ 2152 "enc":"A256CBC", 2153 "kid":"EBQK-6MSJ-J4DD-ZYWE-B3SJ-IXIN-55FV", 2154 "Salt":"W-428mJAs68fIoZkLXAlEg", 2155 "recipients":[{ 2156 "kid":"MCZG-SJCC-WPAU-MNRF-6TCG-KUUQ-ICAQ", 2157 "epk":{ 2158 "PublicKeyECDH":{ 2159 "crv":"Ed25519", 2160 "Public":"5lQ_frHzXaQUyMtUTAWC8E2kWDiiNCO6XeqPBrTXYn8"}}, 2161 "wmk":"hKwGLOdl56Kf2o_v2Pgr4KRfJjKtEYXutxm3Oo50kzltAAs668 2162 60Dw"} 2163 ], 2164 "ContentMetaData":"e30", 2165 "SequenceInfo":{ 2166 "Index":2}}} 2168 [Empty trailer] 2170 Here are the sequence bytes. Note that the content is now encrypted 2171 and has expanded by 25 bytes. These are the salt (16 bytes), the AES 2172 padding (4 bytes) and the JSON-B framing (5 bytes). 2174 f5 02 ef 2175 f1 02 d8 2176 f0 10 2177 f0 00 2178 ef 02 f5 2179 f5 02 f9 2180 f1 01 c3 2181 f1 01 30 2182 f9 02 f5 2183 f5 02 f9 2184 f1 01 c3 2185 f1 01 30 2186 f9 02 f5 2188 The following example shows a sequence in which all the frame 2189 payloads are encrypted under separate key agreements specified in the 2190 payload frames. 2192 Frame 0 2194 { 2195 "DareHeader":{ 2196 "policy":{ 2197 "enc":"none", 2198 "dig":"none", 2199 "Sealed":true}, 2200 "ContentMetaData":"e30", 2201 "SequenceInfo":{ 2202 "DataEncoding":"JSON", 2203 "ContainerType":"List", 2204 "Index":0}}} 2206 [Empty trailer] 2208 Frame 1 2210 { 2211 "DareHeader":{ 2212 "ContentMetaData":"e30", 2213 "SequenceInfo":{ 2214 "Index":1}}} 2216 [Empty trailer] 2218 Frame 2 2219 { 2220 "DareHeader":{ 2221 "ContentMetaData":"e30", 2222 "SequenceInfo":{ 2223 "Index":2}}} 2225 [Empty trailer] 2227 15. Appendix C: Previous Frame Function 2229 public long PreviousFrame (long Frame) { 2230 long x2 = Frame + 1; 2231 long d = 1; 2233 while (x2 > 0) { 2234 if ((x2 & 1) == 1) { 2235 return x2 == 1 ? (d / 2) - 1 : Frame - d; 2236 } 2237 d = d * 2; 2238 x2 = x2 / 2; 2239 } 2240 return 0; 2241 } 2243 16. Appendix D: Outstanding Issues 2245 The following issues need to be addressed. 2247 +================+===============================================+ 2248 | Issue | Description | 2249 +================+===============================================+ 2250 | Indexing | No examples are given of indexing a container | 2251 +----------------+-----------------------------------------------+ 2252 | Archive | Should include a file archive example | 2253 +----------------+-----------------------------------------------+ 2254 | File Path | Mention the file path security issue in the | 2255 | | security considerations | 2256 +----------------+-----------------------------------------------+ 2257 | Security | Write Security considerations | 2258 | Considerations | | 2259 +----------------+-----------------------------------------------+ 2260 | AES-GCM | Switch to using AES GCM in the examples | 2261 +----------------+-----------------------------------------------+ 2262 | Witness | Complete handling of witness values. | 2263 +----------------+-----------------------------------------------+ 2264 | Schema | Complete the schema documentation | 2265 +----------------+-----------------------------------------------+ 2267 Table 1 2269 17. Normative References 2271 [draft-hallambaker-jsonbcd] 2272 Hallam-Baker, P., "Binary Encodings for JavaScript Object 2273 Notation: JSON-B, JSON-C, JSON-D", Work in Progress, 2274 Internet-Draft, draft-hallambaker-jsonbcd-19, 2 November 2275 2020, . 2278 [draft-hallambaker-mesh-architecture] 2279 Hallam-Baker, P., "Mathematical Mesh 3.0 Part I: 2280 Architecture Guide", Work in Progress, Internet-Draft, 2281 draft-hallambaker-mesh-architecture-15, 2 November 2020, 2282 . 2285 [draft-hallambaker-mesh-security] 2286 Hallam-Baker, P., "Mathematical Mesh 3.0 Part VII: 2287 Security Considerations", Work in Progress, Internet- 2288 Draft, draft-hallambaker-mesh-security-06, 2 November 2289 2020, . 2292 [draft-hallambaker-mesh-udf] 2293 Hallam-Baker, P., "Mathematical Mesh 3.0 Part II: Uniform 2294 Data Fingerprint.", Work in Progress, Internet-Draft, 2295 draft-hallambaker-mesh-udf-11, 2 November 2020, 2296 . 2299 [IANAJOSE] "[Reference Not Found!]". 2301 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 2302 Requirement Levels", BCP 14, RFC 2119, 2303 DOI 10.17487/RFC2119, March 1997, 2304 . 2306 [RFC2315] Kaliski, B., "PKCS #7: Cryptographic Message Syntax 2307 Version 1.5", RFC 2315, DOI 10.17487/RFC2315, March 1998, 2308 . 2310 [RFC3394] Schaad, J. and R. Housley, "Advanced Encryption Standard 2311 (AES) Key Wrap Algorithm", RFC 3394, DOI 10.17487/RFC3394, 2312 September 2002, . 2314 [RFC4880] Callas, J., Donnerhacke, L., Finney, H., Shaw, D., and R. 2315 Thayer, "OpenPGP Message Format", RFC 4880, 2316 DOI 10.17487/RFC4880, November 2007, 2317 . 2319 [RFC4949] Shirey, R., "Internet Security Glossary, Version 2", 2320 FYI 36, RFC 4949, DOI 10.17487/RFC4949, August 2007, 2321 . 2323 [RFC5869] Krawczyk, H. and P. Eronen, "HMAC-based Extract-and-Expand 2324 Key Derivation Function (HKDF)", RFC 5869, 2325 DOI 10.17487/RFC5869, May 2010, 2326 . 2328 [RFC6838] Freed, N., Klensin, J., and T. Hansen, "Media Type 2329 Specifications and Registration Procedures", BCP 13, 2330 RFC 6838, DOI 10.17487/RFC6838, January 2013, 2331 . 2333 [RFC7159] Bray, T., "The JavaScript Object Notation (JSON) Data 2334 Interchange Format", RFC 7159, DOI 10.17487/RFC7159, March 2335 2014, . 2337 [RFC7515] Jones, M., Bradley, J., and N. Sakimura, "JSON Web 2338 Signature (JWS)", RFC 7515, DOI 10.17487/RFC7515, May 2339 2015, . 2341 [RFC7516] Jones, M. and J. Hildebrand, "JSON Web Encryption (JWE)", 2342 RFC 7516, DOI 10.17487/RFC7516, May 2015, 2343 . 2345 [RFC7517] Jones, M., "JSON Web Key (JWK)", RFC 7517, 2346 DOI 10.17487/RFC7517, May 2015, 2347 . 2349 [RFC7518] Jones, M., "JSON Web Algorithms (JWA)", RFC 7518, 2350 DOI 10.17487/RFC7518, May 2015, 2351 . 2353 18. Informative References 2355 [BLOCKCHAIN] 2356 Chain.com, "Blockchain Specification". 2358 [Davis2001] 2359 Davis, D., "Defective Sign & Encrypt in S/MIME, PKCS#7, 2360 MOSS, PEM, PGP, and XML", May 2001. 2362 [RFC5652] Housley, R., "Cryptographic Message Syntax (CMS)", STD 70, 2363 RFC 5652, DOI 10.17487/RFC5652, September 2009, 2364 . 2366 [ZIPFILE] PKWARE Inc, "APPNOTE.TXT - .ZIP File Format 2367 Specification", October 2014.