idnits 2.17.1 draft-hallambaker-dare-container-02.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The abstract seems to contain references ([1]), which it shouldn't. Please replace those with straight textual mentions of the documents in question. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (August 27, 2018) is 2040 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- -- Looks like a reference, but probably isn't: '1' on line 1151 -- Looks like a reference, but probably isn't: '2' on line 1154 -- Looks like a reference, but probably isn't: '3' on line 1157 -- Looks like a reference, but probably isn't: '4' on line 1160 ** Obsolete normative reference: RFC 7159 (Obsoleted by RFC 8259) Summary: 2 errors (**), 0 flaws (~~), 1 warning (==), 5 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group P. Hallam-Baker 3 Internet-Draft Comodo Group Inc. 4 Intended status: Informational August 27, 2018 5 Expires: February 28, 2019 7 Data At Rest Encryption Part 2: DARE Container 8 draft-hallambaker-dare-container-02 10 Abstract 12 This document describes DARE Container, a message and file syntax 13 that allows an append-only sequence of data frames to be represented 14 with cryptographic integrity, signature and encryption enhancements. 15 The format supports data integrity checks using digest chains and 16 Merkle trees. The simplest supports efficient write operations and 17 efficient read operations in either the forward or reverse direction. 18 Support for efficient random-access reads may be provided through the 19 use of binary trees or index records appended to the end of the file. 21 This document is also available online at 22 http://mathmesh.com/Documents/draft-hallambaker-dare-container.html 23 [1] . 25 Status of This Memo 27 This Internet-Draft is submitted in full conformance with the 28 provisions of BCP 78 and BCP 79. 30 Internet-Drafts are working documents of the Internet Engineering 31 Task Force (IETF). Note that other groups may also distribute 32 working documents as Internet-Drafts. The list of current Internet- 33 Drafts is at https://datatracker.ietf.org/drafts/current/. 35 Internet-Drafts are draft documents valid for a maximum of six months 36 and may be updated, replaced, or obsoleted by other documents at any 37 time. It is inappropriate to use Internet-Drafts as reference 38 material or to cite them other than as "work in progress." 40 This Internet-Draft will expire on February 28, 2019. 42 Copyright Notice 44 Copyright (c) 2018 IETF Trust and the persons identified as the 45 document authors. All rights reserved. 47 This document is subject to BCP 78 and the IETF Trust's Legal 48 Provisions Relating to IETF Documents 49 (https://trustee.ietf.org/license-info) in effect on the date of 50 publication of this document. Please review these documents 51 carefully, as they describe your rights and restrictions with respect 52 to this document. Code Components extracted from this document must 53 include Simplified BSD License text as described in Section 4.e of 54 the Trust Legal Provisions and are provided without warranty as 55 described in the Simplified BSD License. 57 Table of Contents 59 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 60 1.1. Container Format . . . . . . . . . . . . . . . . . . . . 3 61 1.2. Write . . . . . . . . . . . . . . . . . . . . . . . . . . 4 62 1.3. Read Access . . . . . . . . . . . . . . . . . . . . . . . 5 63 1.4. Encryption and Authentication . . . . . . . . . . . . . . 5 64 1.5. Integrity and Signature . . . . . . . . . . . . . . . . . 5 65 1.6. Redaction . . . . . . . . . . . . . . . . . . . . . . . . 6 66 1.7. Alternative approaches . . . . . . . . . . . . . . . . . 6 67 1.8. Efficiency . . . . . . . . . . . . . . . . . . . . . . . 7 68 2. Definitions . . . . . . . . . . . . . . . . . . . . . . . . . 7 69 2.1. Related Specifications . . . . . . . . . . . . . . . . . 7 70 2.2. Requirements Language . . . . . . . . . . . . . . . . . . 8 71 3. Container Navigation . . . . . . . . . . . . . . . . . . . . 8 72 3.1. Tree . . . . . . . . . . . . . . . . . . . . . . . . . . 9 73 3.2. Position Index . . . . . . . . . . . . . . . . . . . . . 9 74 3.3. Metadata Index . . . . . . . . . . . . . . . . . . . . . 10 75 4. Integrity Mechanisms . . . . . . . . . . . . . . . . . . . . 10 76 4.1. Digest Chain calculation . . . . . . . . . . . . . . . . 10 77 4.2. Binary Merkle tree calculation . . . . . . . . . . . . . 10 78 5. Reference . . . . . . . . . . . . . . . . . . . . . . . . . . 11 79 5.1. Container Headers . . . . . . . . . . . . . . . . . . . . 11 80 5.1.1. Structure: ContainerHeaderFirst . . . . . . . . . . . 11 81 5.1.2. Structure: ContainerHeader . . . . . . . . . . . . . 11 82 5.2. Content Metadata Structure . . . . . . . . . . . . . . . 12 83 5.2.1. Structure: ContentMeta . . . . . . . . . . . . . . . 12 84 5.3. Index Structures . . . . . . . . . . . . . . . . . . . . 13 85 5.3.1. Structure: ContainerIndex . . . . . . . . . . . . . . 13 86 5.3.2. Structure: IndexPosition . . . . . . . . . . . . . . 13 87 5.3.3. Structure: KeyValue . . . . . . . . . . . . . . . . . 13 88 5.3.4. Structure: IndexMeta . . . . . . . . . . . . . . . . 13 89 5.4. Signature . . . . . . . . . . . . . . . . . . . . . . . . 14 90 6. Security Considerations . . . . . . . . . . . . . . . . . . . 14 91 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 14 92 8. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 14 93 9. Appendix A: Examples and Test Vectors . . . . . . . . . . . . 14 94 9.1. Simple container . . . . . . . . . . . . . . . . . . . . 15 95 9.2. Payload and chain digests . . . . . . . . . . . . . . . . 16 96 9.3. Merkle Tree . . . . . . . . . . . . . . . . . . . . . . . 17 97 9.4. Signed container . . . . . . . . . . . . . . . . . . . . 19 98 9.5. Encrypted container . . . . . . . . . . . . . . . . . . . 21 99 10. Appendix B . . . . . . . . . . . . . . . . . . . . . . . . . 25 100 11. References . . . . . . . . . . . . . . . . . . . . . . . . . 25 101 11.1. Normative References . . . . . . . . . . . . . . . . . . 25 102 11.2. Informative References . . . . . . . . . . . . . . . . . 26 103 11.3. URIs . . . . . . . . . . . . . . . . . . . . . . . . . . 26 104 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 26 106 1. Introduction 108 DARE Container is a message and file syntax that allows a sequence of 109 data frames to be represented with cryptographic integrity, 110 signature, and encryption enhancements to be constructed in an append 111 only format. DARE Container was developed in response to needs that 112 arose out of the design of the Mathematical Mesh 113 [draft-hallambaker-mesh-architecture] . It is built on the binary 114 encodings of JSON data objects, JSON-B and JSON-C 115 [draft-hallambaker-jsonbcd] and the DARE Message format 116 [draft-hallambaker-dare-message] . 118 The format is designed to meet the requirements of a wide range of 119 use cases including: 121 o Recording transactions in persistent storage. 123 o Synchronizing transaction logs between hosts. 125 o File archive. 127 o Message spool. 129 o Signing and encrypting single data items. 131 o Incremental encryption and authentication of server logs. 133 1.1. Container Format 135 A DARE Container consists of a sequence of variable length frames. 136 Each frame consists of a forward length indicator, the framed data 137 and a reverse length indicator. The reverse length indicator is 138 written out backwards allowing the length and thus the frame to be 139 read in the reverse direction: 141 [[This figure is not viewable in this format. The figure is 142 available at http://mathmesh.com/Documents/draft-hallambaker-dare- 143 container.html [2].]] 145 JBCD Bidirectional Frame 147 Each frame contains a single DARE Message consisting of a Header, 148 Payload and Trailer (if required). The first frame in a container 149 describes the container format options and defaults. These include 150 the range of encoding options for frame metadata supported and the 151 container profiles to which the container conforms. 153 All internal data formats are 64 bit clean allowing for containers of 154 up to 18 exabytes to be written. 156 Five container types are currently specified but this may be reduced 157 if the Digest and Tree types are withdrawn. These are: 159 Simple The container does not provide any index or content integrity 160 checks. 162 Tree Frame headers contain entries that specify the start position 163 of previous frames at the apex of the immediately enclosing binary 164 tree. This enables efficient random access to any frame in the 165 file. 167 Digest Each frame trailer contains a PayloadDigest field. 168 Modification of the payload will cause verification of the 169 PayloadDigest value to fail on that frame. 171 Chain Each frame trailer contains PayloadDigest and ChainDigest 172 fields allowing modifications to the payload data to be detected. 173 Modification of the payload will cause verification of the 174 PayloadDigest value to fail on that frame and verification of the 175 ChainDigest value to fail on all subsequent frames. 177 Merkle Tree Frame headers contain entries that specify the start 178 position of previous frames at the apex of the immediately 179 enclosing binary tree. Frame Trailers contain 180 TreeDigestPartial and TreeDigestFinal entries forming a Merkle 181 digest tree. 183 1.2. Write 185 In normal circumstances, DARE Containers are written as an append 186 only log. As with DARE Messages, integrity information (payload 187 digest, signatures) is written to the message trailer. Thus, large 188 payloads may be written without the need to buffer the payload data 189 provided that the content length is known in advance. 191 1.3. Read Access 193 The use of reverse length indicators allows DARE containers to 194 support efficient sequential access in either the forward or reverse 195 directions. 197 Random access to any part of a file MAY be supported by means of a 198 binary tree index and/or an index record providing direct access to 199 any part of the file. 201 1.4. Encryption and Authentication 203 Frame payloads and associated attributes MAY be encrypted and/or 204 authenticated using the approach described in 205 [draft-hallambaker-dare-message] . 207 Incremental encryption is supported allowing encryption parameters 208 from a single public key exchange operation to be applied to encrypt 209 multiple frames. The public key exchange information is specified in 210 the first encrypted frame and subsequent frames encrypted under those 211 parameters specify the location at which the key exchange information 212 is to be found by means of the ExchangePosition field. 214 The only restriction on the use of incremental encryption is that the 215 frame containing the key exchange information MUST precede the frames 216 that reference the exchange parameters. 218 To avoid cryptographic vulnerabilities resulting from key re-use, the 219 DARE key exchange requires that each encrypted sequence use an 220 encryption key and initialization vector derived from the master key 221 established in the public key exchange by means of a unique salt. 223 Each DARE Message and by extension, each DARE Container frame MUST 224 specify a unique salt value of at least 128 bits. Since the 225 encryption key is derived from the salt value by means of a Key 226 Derivation Function, erasure of the salt MAY be used as a means of 227 rendering the payload plaintext value inaccessible without changing 228 the payload value. 230 1.5. Integrity and Signature 232 Signatures MAY be applied to a payload digest, the final digest in a 233 chain or tree. The chain and tree digest modes allow a single 234 signature to be used to authenticate all frame payloads in a 235 container. 237 The tree signature mode is particularly suited to applications such 238 as file archives as it allows files to be verified individually 239 without requiring the signer to sign each individually. Furthermore, 240 in applications such as code signing, it allows a single signature to 241 be used to verify both the integrity of the code and its membership 242 of the distribution. 244 As with DARE Message, the signature mechanism does not specify the 245 interpretation of the signature semantics. The presence of a 246 signature demonstrates that the holder of the private key applied it 247 to the specified digest value but not their motive for doing so. 248 Describing such semantics is beyond the scope of this document and is 249 deferred to future work. 251 1.6. Redaction 253 The chief disadvantage of using an append-only format is that 254 containers only increase in size. In many applications, much of the 255 data in the container becomes redundant or obsolete and a process 256 analogous to garbage collection is required. This process is called 257 redaction. 259 The simplest method of redaction is to create a new container and 260 sequentially copy each entry from the old container to the new, 261 discarding redundant frames and obsolete header information. 263 For example, partial index records may be consolidated into a single 264 index record placed in the last frame of the container. Unnecessary 265 signature and integrity data may be discarded and so on. 267 It is also possible but not necessarily advisable to perform such a 268 redaction in-place provided that the redaction process does not 269 increase the length of any individual frame and that appropriate 270 provision is made for file locking to prevent conflicts and to 271 provide for safe resumption should an interruption occur during the 272 process. 274 1.7. Alternative approaches 276 Many file proprietary formats are in use that support some or all of 277 these capabilities but only a handful have public, let alone open, 278 standards. DARE Container is designed to provide a superset of the 279 capabilities of existing message and file syntaxes, including: 281 o Cryptographic Message Syntax [RFC5652] defines a syntax used to 282 digitally sign, digest, authenticate, or encrypt arbitrary message 283 content. 285 o The.ZIP File Format specification [ZIPFILE] developed by Phil 286 Katz. 288 o The BitCoin Block chain [BLOCKCHAIN] . 290 o JSON Web Encryption and JSON Web Signature 292 Attempting to make use of these specifications in a layered fashion 293 would require at least three separate encoders and introduce 294 unnecessary complexity. Furthermore, there is considerable overlap 295 between the specifications providing multiple means of achieving the 296 same ends, all of which must be supported if decoders are to work 297 reliably. 299 1.8. Efficiency 301 Every data format represents a compromise between different concerns, 302 in particular: 304 Compactness The space required to record data in the encoding. 306 Memory Overhead The additional volatile storage (RAM) required to 307 maintain indexes etc. to support efficient retrieval operations. 309 Number of Operations The number of operations required to retrieve 310 data from or append data to an existing encoded sequence. 312 Number of Disk Seek Operations Optimizing the response time of 313 magnetic storage media to random access read requests has 314 traditionally been one of the central concerns of database design. 315 The DARE Container format is designed to the assumption that this 316 will cease to be a concern as solid state media replaces magnetic. 318 While the cost of storage of all types has declined rapidly over the 319 past decades, so has the amount of data to be stored. DARE Container 320 represents a pragmatic balance of these considerations for current 321 technology. In particular, since payload volumes are likely to be 322 very large, memory and operational efficiency are considered higher 323 priorities than compactness. 325 2. Definitions 327 2.1. Related Specifications 329 DARE Container makes use of the following related standards and 330 specifications. 332 Encoding Content frame headers are encoded using JavaScript Object 333 Notation (JSON) [RFC7159] , JSON-B or JSON-C 334 [draft-hallambaker-jsonbcd] . 336 Cryptography The encryption and signature schemes used are based on 337 JSON Web Signature [RFC7515] and JSON Web Encryption [RFC7516] as 338 applied in DARE Message [draft-hallambaker-dare-message] . 340 2.2. Requirements Language 342 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 343 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 344 document are to be interpreted as described in RFC 2119 [RFC2119] . 346 3. Container Navigation 348 Three means of locating frames in a container are supported: 350 Sequential Access frames sequentially starting from the start or the 351 end of the container. 353 Binary search Access any container frame by frame number in 354 O(log2(n)) time by means of a binary tree constructed while the 355 container is written. 357 Index Access and container frame by frame number or by key by means 358 of an index record. 360 All DARE Containers support sequential access. Only tree and Merkle 361 tree containers support binary search access. An index frame MAY be 362 written appended to any container and provides O(1) access to any 363 frame listed in the index. 365 Two modes of compilation are considered: 367 Monolithic Frames are added to the container in a single operation, 368 e.g. file archives, 370 Incremental Additional frames are written to the container at 371 various intervals after it was originally created, e.g. server 372 logs, message spools. 374 In the monolithic mode, navigation requirements are best met by 375 writing an index frame to the end of the container when it is 376 complete. It is not necessary to construct a binary search tree 377 unless a Merkle tree integrity check is required. 379 In the incremental mode, Binary search provides an efficient means of 380 locating frames by frame number but not by key. Writing a complete 381 index to the container every m write operations provides O(m) search 382 access but requires O(n2) storage. 384 Use of partial indexes provides a better compromise between speed and 385 efficiency. A partial index is written out every m frames where m is 386 a power of two. A complete index is written every time a binary tree 387 apex record is written. This approach provides for O(log2(n)) search 388 with incremental compilation with approximately double the overhead 389 of the monolithic case. 391 3.1. Tree 393 Binary search is supported by means of the TreePosition parameter 394 specified in the FrameHeader. This parameter specifies the value of 395 the immediately preceding apex. 397 Calculation of the immediately preceding apex is most easily 398 described by representing the array index in binary with base of 1 399 (rather than 0). An array index that is a power of 2 (2, 4, 8, 16, 400 etc.) will be the apex of a complete tree. Every other array index 401 has the value of the sum of a set of powers of 2 and the immediately 402 preceding apex will be the value of the next smallest power of 2 in 403 the sum. 405 For example, to find the immediately preceding apex for frame 5, we 406 add 1 to get 6. 6 = 4 + 2, so we ignore the 2 and the preceding frame 407 is 4. 409 The values of Tree Position are shown for the first 8 frames in 410 figure xx below: 412 [[This figure is not viewable in this format. The figure is 413 available at http://mathmesh.com/Documents/draft-hallambaker-dare- 414 container.html [3].]] 416 Merkle Tree Integrity check 418 An algorithm for efficiently calculating the immediately preceding 419 apex is provided in Appendix C. 421 3.2. Position Index 423 Contains a table of frame number, position pairs pointing to prior 424 locations in the file. 426 3.3. Metadata Index 428 Contains a list of IndexMeta entries. Each entry contains a metadata 429 description and a list of frame indexes (not positions) of frames 430 that match the description. 432 4. Integrity Mechanisms 434 Frame sequences in a DARE container MAY be protected against a frame 435 insertion attack by means of a digest chain, a binary Merkle tree or 436 both. 438 4.1. Digest Chain calculation 440 A digest chain is simple to implement but can only be verified if the 441 full chain of values is known. Appending a frame to the chain has 442 O(1) complexity but verification has O(n) complexity: 444 [[This figure is not viewable in this format. The figure is 445 available at http://mathmesh.com/Documents/draft-hallambaker-dare- 446 container.html [4].]] 448 Hash chain integrity check 450 The value of the chain digest for the first frame (frame 0) is 451 H(H(null)+H(Payload0)), where null is a zero length octet sequence 452 and payloadn is the sequence of payload data bytes for frame n 454 The value of the chain digest for frame n is H(H(Payloadn-1 455 +H(Payloadn)), where A+B stands for concatenation of the byte 456 sequences A and B. 458 4.2. Binary Merkle tree calculation 460 The tree index mechanism describe earlier may be used to implement a 461 binary Merkle tree. The value TreeDigest specifies the apex value of 462 the tree for that node. 464 Appending a frame to the chain has O(log2n) complexity provided that 465 the container format supports at least the binary tree index. 466 Verifying a chain has O(log2 n) complexity, provided that the set of 467 necessary digest inputs is known. 469 To calculate the value of the tree digest for a node, we first 470 calculate the values of all the sub trees that have their apex at 471 that node and then calculate the digest of that value and the 472 immediately preceding local apex. 474 5. Reference 476 TBS stuff 478 5.1. Container Headers 480 TBS stuff 482 5.1.1. Structure: ContainerHeaderFirst 484 Inherits: ContainerHeader 486 Inherits: ContainerHeader 488 DataEncoding: String (Optional) Specifies the data encoding for the 489 header section of for the following frames. This value is ONLY 490 valid in Frame 0 which MUST have a header encoded in JSON. 492 5.1.2. Structure: ContainerHeader 494 Inherits: DAREHeader 496 Describes a container header. A container header MAY contain any 497 DARE Message header. 499 Index: Integer (Optional) The record index within the file. This 500 MUST be unique and satisfy any additional requirements determined 501 by the ContainerType. 503 ContainerType: String (Optional) Specifies the container type for 504 the following records. 506 IsMeta: Boolean (Optional) If true, the current frame is a meta 507 frame and does not contain a payload. 509 Note: Meta frames MAY be present in any container. Applications 510 MUST accept containers that contain meta frames at any position in 511 the file. Applications MUST NOT interpret a meta frame as a data 512 frame with an enpty payload. 514 UniqueID: String (Optional) Unique object identifier 516 ContentMeta: ContentMeta (Optional) Content meta data. 518 TreePosition: Integer (Optional) Position of the frame containing 519 the apex of the preceding sub-tree. 521 IndexPosition: Integer (Optional) Specifies the position in the file 522 at which the last index entry is to be found 524 ExchangePosition: Integer (Optional) Specifies the position in the 525 file at which the key exchange data is to be found 527 ContainerIndex: ContainerIndex (Optional) An index of records in the 528 current container up to but not including this one. 530 PayloadDigest: Binary (Optional) If present, contains the digest of 531 the Payload. 533 ChainDigest: Binary (Optional) If present, contains the digest of 534 the PayloadDigest values of this frame and the frame immediately 535 preceding. 537 TreeDigest: Binary (Optional) If present, contains the Binary Merkle 538 Tree digest value. 540 Event: String (Optional) Unique object identifier 542 Labels: String [0..Many] List of labels that are applied to the 543 payload of the frame. 545 KeyValues: KeyValue [0..Many] List of key/value pairs describing the 546 payload of the frame. 548 First: Integer (Optional) Frame number of the first object instance 549 value. 551 Previous: Integer (Optional) Frame number of the immediately prior 552 object instance value 554 5.2. Content Metadata Structure 556 TBS stuff 558 5.2.1. Structure: ContentMeta 560 Information describing the object instance 562 ContentType: String (Optional) The content type field as specified 563 in JWE 565 Paths: String [0..Many] List of filename paths for the payload of 566 the frame. 568 UniqueID: String (Optional) Unique object identifier 569 Created: DateTime (Optional) Initial creation date. 571 Modified: DateTime (Optional) Date of last modification. 573 5.3. Index Structures 575 TBS stuff 577 5.3.1. Structure: ContainerIndex 579 A container index 581 Full: Boolean (Optional) If true, the index is complete and contains 582 position entries for all the frames in the file. If absent or 583 false, the index is incremental and only contains position entries 584 for records added since the last frame containing a 585 ContainerIndex. 587 Positions: IndexPosition [0..Many] List of container position 588 entries 590 Metas: IndexMeta [0..Many] List of container position entries 592 5.3.2. Structure: IndexPosition 594 Specifies the position in a file at which a specified record index is 595 found 597 Index: Integer (Optional) The record index within the file. 599 Position: Integer (Optional) The record position within the file 600 relative to the index base. 602 5.3.3. Structure: KeyValue 604 Specifies a key/value entry 606 Key: String (Optional) The key 608 Value: String (Optional) The value corresponding to the key 610 5.3.4. Structure: IndexMeta 612 Specifies the list of index entries at which a record with the 613 specified metadata occurrs. 615 Index: Integer [0..Many] List of record indicies within the file 616 where frames matching the specified criteria are found. 618 ContentType: String (Optional) Content type parameter 620 Paths: String [0..Many] List of filename paths for the current 621 frame. 623 Labels: String [0..Many] List of labels that are applied to the 624 current frame. 626 5.4. Signature 628 Payload data MAY be signed using a JWS [RFC7515] as applied in the 629 DARE Message format [draft-hallambaker-dare-message] . 631 Signatures are specified by the Signatures parameter in the content 632 header. The data that the signature is calculated over is defined by 633 the typ parameter of the Signature as follows. 635 Payload The value of the PayloadDigest parameter 637 Chain The value of the ChainDigest parameter 639 Tree The value of the TreeDigestFinal parameter 641 If the typ parameter is absent, the value Payload is implied. 643 A frame MAY contain multiple signatures created with the same signing 644 key and different typ values. 646 The use of signatures over chain and tree digest values permit 647 multiple frames to be validated using a single signature verification 648 operation. 650 6. Security Considerations 652 7. IANA Considerations 654 8. Acknowledgements 656 9. Appendix A: Examples and Test Vectors 658 The data payloads in all the following examples are identical, only 659 the authentication and/or encryption is different. 661 o Frame 1..n consists of 300 bytes being the byte sequence 00, 01, 662 02, etc. repeating after 256 bytes. 664 For conciseness, the raw data format is omitted for examples after 665 the first, except where the data payload has been transformed, (i.e. 666 encrypted). 668 9.1. Simple container 670 the following example shows a simple container with first frame and a 671 single data frame: 673 f4 5d 674 f0 59 675 f0 00 676 5d f4 677 f5 01 40 678 f0 0f 679 f1 01 2c 680 40 01 f5 682 Figure 1 684 Since there is no integrity check, there is no need for trailer 685 entries. The header values are: 687 Frame 0 689 { 690 "Index":0, 691 "ContainerType":"List", 692 "ContentMeta":{}, 693 "DataEncoding":"JSON"} 695 [Empty trailer] 697 Figure 2 699 Frame 1 701 { 702 "Index":1} 704 [Empty trailer] 706 Figure 3 708 9.2. Payload and chain digests 710 The following example shows a chain container with a first frame and 711 three data frames. The headers of these frames is the same as before 712 but the frames now have trailers specifying the PayloadDigest and 713 ChainDigest values: 715 Frame 0 717 { 718 "Index":0, 719 "ContainerType":"Chain", 720 "ContentMeta":{}, 721 "DataEncoding":"JSON"} 723 [Empty trailer] 725 Figure 4 727 Frame 1 729 { 730 "Index":1} 732 { 733 "PayloadDigest":"8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZDlZe 734 aWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw", 735 "ChainDigest":"T7S1FcrgY3AaWD4L-t5W1K-3XYkPTcOdGEGyjglTD6yMYVRVz9 736 tn_KQc6GdA-P4VSRigBygV65OEd2Vv3YDhww"} 738 Figure 5 740 Frame 2 742 { 743 "Index":2} 745 { 746 "PayloadDigest":"8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZDlZe 747 aWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw", 748 "ChainDigest":"T7S1FcrgY3AaWD4L-t5W1K-3XYkPTcOdGEGyjglTD6yMYVRVz9 749 tn_KQc6GdA-P4VSRigBygV65OEd2Vv3YDhww"} 751 Figure 6 753 Frame 3 754 { 755 "Index":3} 757 { 758 "PayloadDigest":"8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZDlZe 759 aWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw", 760 "ChainDigest":"T7S1FcrgY3AaWD4L-t5W1K-3XYkPTcOdGEGyjglTD6yMYVRVz9 761 tn_KQc6GdA-P4VSRigBygV65OEd2Vv3YDhww"} 763 Figure 7 765 9.3. Merkle Tree 767 The following example shows a chain container with a first frame and 768 six data frames. The trailers now contain the TreePosition and 769 TreeDigest values: 771 Frame 0 773 { 774 "Index":0, 775 "ContainerType":"Merkle", 776 "ContentMeta":{}, 777 "DataEncoding":"JSON"} 779 [Empty trailer] 781 Figure 8 783 Frame 1 785 { 786 "Index":1, 787 "TreePosition":0} 789 { 790 "PayloadDigest":"8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZDlZe 791 aWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw", 792 "TreeDigest":"T7S1FcrgY3AaWD4L-t5W1K-3XYkPTcOdGEGyjglTD6yMYVRVz9t 793 n_KQc6GdA-P4VSRigBygV65OEd2Vv3YDhww"} 795 Figure 9 797 Frame 2 798 { 799 "Index":2, 800 "TreePosition":319} 802 { 803 "PayloadDigest":"8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZDlZe 804 aWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw", 805 "TreeDigest":"7fHmkEIsPkN6sDYAOLvpIJn5Dg3PxDDAaq-ll2kh8722kokkFnZ 806 QcYtjuVC71aHNXI18q-lPnfRkmwryG-bhqQ"} 808 Figure 10 810 Frame 3 812 { 813 "Index":3, 814 "TreePosition":319} 816 { 817 "PayloadDigest":"8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZDlZe 818 aWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw", 819 "TreeDigest":"T7S1FcrgY3AaWD4L-t5W1K-3XYkPTcOdGEGyjglTD6yMYVRVz9t 820 n_KQc6GdA-P4VSRigBygV65OEd2Vv3YDhww"} 822 Figure 11 824 Frame 4 826 { 827 "Index":4, 828 "TreePosition":1451} 830 { 831 "PayloadDigest":"8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZDlZe 832 aWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw", 833 "TreeDigest":"vJ6ngNATvZcXSMALi5IUqzl1GBxBnTNVcC87VL_BhMRCbAvKSj8 834 gs0VFgxxLkZ2myrtaDIwhHoswiTiBMLNWug"} 836 Figure 12 838 Frame 5 839 { 840 "Index":5, 841 "TreePosition":1451} 843 { 844 "PayloadDigest":"8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZDlZe 845 aWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw", 846 "TreeDigest":"T7S1FcrgY3AaWD4L-t5W1K-3XYkPTcOdGEGyjglTD6yMYVRVz9t 847 n_KQc6GdA-P4VSRigBygV65OEd2Vv3YDhww"} 849 Figure 13 851 Frame 6 853 { 854 "Index":6, 855 "TreePosition":2586} 857 { 858 "PayloadDigest":"8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZDlZe 859 aWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw", 860 "TreeDigest":"WgHlz3EHczVPqgtpc39Arv7CFIsCbFVsk8wg0j2qLlEfur9SZ0m 861 dr65Ka-HF0Qx8gg_DAoiJwUrwADDXyxVJOg"} 863 Figure 14 865 9.4. Signed container 867 The following example shows a tree container with a signature in the 868 final record. The signing key parameters are: 870 { 871 "PrivateKeyRSA":{ 872 "kid":"MBWNO-2J43U-ESWKW-XQWL6-6YGEW-UOPWU-A", 873 "n":"1NzbmakMalVH1mRv7TEDEhwXDNojn5wZbq1tv1gp5PgZwzX-klYXuFhj0- 874 MpO0zcwptsUaYJhwdvvgW_1udUpISQYluXOB3UMj2e_0yl64MvnqTL47SZQuAN3QQ 875 9cuCw_-_Eyj_jerspauqa6RpNzGcabZrtRl7J7DPVZ3SNlw-H_Wxd4HkrFVW_Yqup 876 htNL1JciQJYm2DSu9dbetqPZ80x6IBargY850mBYOzvNNE5S-dRJQHoJY4SG-ESYF 877 BuAHtBlOMgbI0XNiq96EegA-vPW9XRF-SHdX5mkafefDGK4rT_RoE4gRwhDM3jbZ8 878 1-F2ZA_GpNVEvB-25_vF96lQ", 879 "e":"AQAB", 880 "d":"k_v_h7Jo-TvUt44X6jSax-pTdBHrljk1zSYxGEe4yIBbmMVe-Gl2ECkTLe 881 nNbnafO4RGJ_Vgxkk7PEZO-p7Uz5OBtX-rf83tCgihEyg8aaFIZ-h1_xY9Pqr5uGA 882 MQGNJaoVMsLb99QNNZhE4JTquP56mVvDQaI3Zn6bhhA0ZqpxS_x6iRUV5KnHCRd47 883 DKGHcAsQR_caxGec7M_XNPqpl0tcoGQz46-I0SVVcqtjb_YysEVez4eJhb_ZTU4C7 884 pz4wXDj6B0ppFJVZEMaIhKo8FCnoQdXEJl4vSiBzUFrSlPc6gjQk2CBhxc_kb782w 885 VwW5wtgOVxhV1k2piQ2NrlOQ", 886 "p":"6osXfrJLiPfK0ky17vqzRs-M5mOxZU2LEFGyHTXxx6EYTWixEsx2Sdf6kx 887 UD5K_QdYnqqd3yobbGdDpMwwEuwgogWVCz90nc9QdCUy4MCpz8lpOQd1i_tXMmtOg 888 GBu9mapMTEOwc7HlLlSDRezv_TzTAH4izv-CUEZ_M5EcwyFM", 889 "q":"6FYD6NV4rKU1ACtGQyIvwGrkrS_F6phB-whx0nSVFkbkwppJiPnC6XqjZ3 890 OYPCZylxaTHFnxCs0nntrraeNEcWfNPrpTN5XIZjbOiIaKA2iCQkWlDoi8oduEtTK 891 oIcuy32oBz6MpUeCWzl0ZQ4EeTF3RyCc_jpf8oRvZ0e3ItHc", 892 "dp":"hfjbe9BmWx-HqCaPSanEW-9UQYmym_X2OGUiA5N7vxci5ZymgOFvs_B9v 893 iQj7C4NOgaEl3EjFgJsS5m9nSoAxm-4WKxDkD6NyxzRYugLkshnc69otvNn1kKnWn 894 CqeK2o57mJC4KDZwRGCzIK1oTH6jtsfta8Lh8fFQ4doEuV7uc", 895 "dq":"r6R_ViE0Foja1aLhflU09mmZMViBbkXm86nBqtHZ97pmrLvJRdVTxgCh0 896 c6w0yBZ1uEJHBDeykSoZE6qVCWtE3Le1kI0MTx6ANQENXBInCUA_Kr8Ck3TFSYIYJ 897 fIRaxiMMZKUjfOQAji2WXGeKL_TcpLkt4hDWLXaNDOTgdOiSc", 898 "qi":"DfHtLB1Ox1Kgp3E4jqy5Qxeb7-v7_uv8n_5-E1OQ3NLSRV2m_auojkR19 899 nY3gokHKNSXM41qKlJLU00lROjOO2KUq57s8GZkheVfbJLNCJ6KAw_aRT2IgyJm2b 900 e2v5OCHSkm88tgJWbtKj-OPKTFV5gOMVdeCzGX286ErjDHGCM"}} 902 Figure 15 904 The container headers and trailers are: 906 Frame 0 908 { 909 "Index":0, 910 "ContainerType":"Merkle", 911 "ContentMeta":{}, 912 "DataEncoding":"JSON"} 914 [Empty trailer] 916 Figure 16 918 Frame 1 920 { 921 "Index":1, 922 "TreePosition":0} 924 { 925 "PayloadDigest":"8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZDlZe 926 aWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw", 927 "TreeDigest":"T7S1FcrgY3AaWD4L-t5W1K-3XYkPTcOdGEGyjglTD6yMYVRVz9t 928 n_KQc6GdA-P4VSRigBygV65OEd2Vv3YDhww"} 930 Figure 17 932 Frame 2 934 { 935 "dig":"S512", 936 "Index":2, 937 "TreePosition":319} 939 { 940 "PayloadDigest":"8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZDlZe 941 aWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw", 942 "TreeDigest":"7fHmkEIsPkN6sDYAOLvpIJn5Dg3PxDDAaq-ll2kh8722kokkFnZ 943 QcYtjuVC71aHNXI18q-lPnfRkmwryG-bhqQ"} 945 Figure 18 947 9.5. Encrypted container 949 The following example shows a container in which all the frame 950 payloads are encrypted under the same master secret established in a 951 key agreement specified in the first frame. 953 Frame 0 954 { 955 "enc":"A256CBC", 956 "Salt":"Th7V_b2--KdV5rETQZbHsw", 957 "recipients":[{ 958 "kid":"MDAD3-E4BYE-MK6CH-QA2HD-TKRS2-KIX5Y-A", 959 "epk":{ 960 "PublicKeyDH":{ 961 "kid":"MANS7-VSJ2X-75NSK-4SEOM-FRITG-6Q7EO-A", 962 "Domain":"YE6bnq1MlX5ojaJto6PLP_PEwA", 963 "Public":"MYw__h5Ii4honclOGFKY42KdvJed5T47CZI3QTRSBXUPiOD 964 0D0W49YIyf-m30K-BvMWVzNsh9qAQFpK53A5UJO_pQmfbmrZLHq3AjdYoRlc44YOn 965 usUokZN43xtGly4m7Rcf_42M3cofa80UWejL3mz3zQKTojxwXz3cykLA2tsTjHdKE 966 O0Q6jWef952YkPQNKJmR-42aX7Pfz5kZt1QtgfUV4QYk_b0NFfHSNk9CAs6Klav5K 967 qGjxL4mfIxvmrksyIR5solCo2BMbh_AOdyZNuEa4nURvS9YtWrXZRjQtMZPKB8MXG 968 _mSAxUco3U1em2hmdkgIkTveF7-da6otILw"}}, 969 "wmk":"0HfYk15gAvxngunLoHymHSunrb0KlGjGuTERlpSxB5JLpFJj6KJZXA"} 970 ], 971 "Index":0, 972 "ContainerType":"List", 973 "ContentMeta":{}, 974 "DataEncoding":"JSON"} 976 [Empty trailer] 978 Figure 19 980 Frame 1 982 { 983 "enc":"A256CBC", 984 "Salt":"BmY8ZYeD8Kg1snIrBIAUyA", 985 "Index":1} 987 [Empty trailer] 989 Figure 20 991 Frame 2 993 { 994 "enc":"A256CBC", 995 "Salt":"pA-kigxfhhsGC5lM9xrAZQ", 996 "Index":2} 998 [Empty trailer] 1000 Figure 21 1002 Here are the container bytes. Note that the content is now encrypted 1003 and has expanded by 25 bytes. These are the salt (16 bytes), the AES 1004 padding (4 bytes) and the JSON-B framing (5 bytes). 1006 f5 03 3c 1007 f1 03 27 1008 f0 10 1009 3c 03 f5 1010 f5 01 7c 1011 f0 47 1012 f1 01 30 1013 7c 01 f5 1014 f5 01 7c 1015 f0 47 1016 f1 01 30 1017 7c 01 f5 1019 Figure 22 1021 The following example shows a container in which all the frame 1022 payloads are encrypted under separate key agreements specified in the 1023 payload frames. 1025 Frame 0 1027 { 1028 "Index":0, 1029 "ContainerType":"List", 1030 "ContentMeta":{}, 1031 "DataEncoding":"JSON"} 1033 [Empty trailer] 1035 Figure 23 1037 Frame 1 1038 { 1039 "enc":"A256CBC", 1040 "Salt":"LUAkC3V-ztXJfF7OS-RNwQ", 1041 "recipients":[{ 1042 "kid":"MDAD3-E4BYE-MK6CH-QA2HD-TKRS2-KIX5Y-A", 1043 "epk":{ 1044 "PublicKeyDH":{ 1045 "kid":"MCYIZ-PH7YZ-XAY6Q-2EMQR-GMRXG-66YPK-A", 1046 "Domain":"YE6bnq1MlX5ojaJto6PLP_PEwA", 1047 "Public":"4cDrUAlsZpyGZGhgVbLlunEyl7O9a8ku-6IGVZaSqyOmsb5 1048 rCt_pjnN49fjy7hqsS364S4eUDRj7e3vfIBEav-zeynaiJB6lY6eJBMmlaEuomyyy 1049 01Srq5eaX6i0A8CRWuuRK8IJAHQ31Q89Bah5XayMxc3bqPxciOL1gqU75w65fKmY9 1050 4mRdAIw2Yk-PTcQVztVDk2ooGJuvLENuIk29g-leFbMmCAXND9hhDErVupBND3Ljh 1051 BXylTvfIx-mrw0SMPO_IsVLbhrXQ6lTMygucttT3GPf25lgWBaEdiHVuPqmTBtagB 1052 Vgq9-gHySYtuCNrK7RfB5rtz6CUmENEwx1QA"}}, 1053 "wmk":"25yqAeqIG1S6BrCSQxU_M8WKB4t8J_lHWI-68DNYORTUoCpaitosig"} 1054 ], 1055 "Index":1} 1057 [Empty trailer] 1059 Figure 24 1061 Frame 2 1063 { 1064 "enc":"A256CBC", 1065 "Salt":"EVrEz8ai_v9s9hZ0ZXMB1Q", 1066 "recipients":[{ 1067 "kid":"MDAD3-E4BYE-MK6CH-QA2HD-TKRS2-KIX5Y-A", 1068 "epk":{ 1069 "PublicKeyDH":{ 1070 "kid":"MB43P-C7BVX-SROQ6-Z5LW5-5DLRO-PS675-A", 1071 "Domain":"YE6bnq1MlX5ojaJto6PLP_PEwA", 1072 "Public":"mJRyK45fAf2ksxW7M8W961hCKI1g7dQqn9rA2K6vdSh1o4v 1073 gLqjHDzLYqr9yQKteGUfl0PLIuHAOFq3p_oJDhaioJKX6Akv-y-dcfDdi2WgOOr_k 1074 R9NPtl-nmsbJ8fx6v1QdXLzdoGZjsj_t6wyIo5w287p7DylFdYkhriqVv7tgBELZI 1075 53Pur4F5QbK-qJYeSwY3fAksQYrviMZbFVwic8v5QQNoAGcoGguBOOz-aljFuo1g9 1076 0o7pNT-Soj3DAPfmXS1hmPR-hpqyzELAat-q_O22P4d_NtUOmaLwZXSZVYUBLE8Fj 1077 1okt4_8zGfCsGpsHl9rwqlyoVIZ4VadldWw"}}, 1078 "wmk":"P6UjESsvq5bATPHc5vPfEcs5u1pCkLVBCv5_aYdHdwQG3rpPihHj2g"} 1079 ], 1080 "Index":2} 1082 [Empty trailer] 1084 Figure 25 1086 10. Appendix B 1088 public long PreviousFrame (long Frame) { 1089 long x2 = Frame + 1; 1090 long d = 1; 1092 while (x2 > 0) { 1093 if ((x2 & 1) == 1) { 1094 return x2 == 1 ? (d / 2) - 1 : Frame - d; 1095 } 1096 d = d * 2; 1097 x2 = x2 / 2; 1098 } 1099 return 0; 1100 } 1102 Figure 26 1104 11. References 1106 11.1. Normative References 1108 [draft-hallambaker-dare-message] 1109 Hallam-Baker, P., "Data At Rest Encryption Part 1: DARE 1110 Message", draft-hallambaker-dare-message-01 (work in 1111 progress), July 2018. 1113 [draft-hallambaker-jsonbcd] 1114 Hallam-Baker, P., "Binary Encodings for JavaScript Object 1115 Notation: JSON-B, JSON-C, JSON-D", draft-hallambaker- 1116 jsonbcd-13 (work in progress), July 2018. 1118 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1119 Requirement Levels", BCP 14, RFC 2119, 1120 DOI 10.17487/RFC2119, March 1997. 1122 [RFC7159] Bray, T., "The JavaScript Object Notation (JSON) Data 1123 Interchange Format", RFC 7159, DOI 10.17487/RFC7159, March 1124 2014. 1126 [RFC7515] Jones, M., Bradley, J., and N. Sakimura, "JSON Web 1127 Signature (JWS)", RFC 7515, DOI 10.17487/RFC7515, May 1128 2015. 1130 [RFC7516] Jones, M. and J. Hildebrand, "JSON Web Encryption (JWE)", 1131 RFC 7516, DOI 10.17487/RFC7516, May 2015. 1133 11.2. Informative References 1135 [BLOCKCHAIN] 1136 Chain.com, "Blockchain Specification". 1138 [draft-hallambaker-mesh-architecture] 1139 Hallam-Baker, P., "Mathematical Mesh: Architecture", 1140 draft-hallambaker-mesh-architecture-04 (work in progress), 1141 September 2017. 1143 [RFC5652] Housley, R., "Cryptographic Message Syntax (CMS)", STD 70, 1144 RFC 5652, DOI 10.17487/RFC5652, September 2009. 1146 [ZIPFILE] PKWARE Inc, "APPNOTE.TXT - .ZIP File Format 1147 Specification", October 2014. 1149 11.3. URIs 1151 [1] http://mathmesh.com/Documents/draft-hallambaker-dare- 1152 container.html 1154 [2] http://mathmesh.com/Documents/draft-hallambaker-dare- 1155 container.html 1157 [3] http://mathmesh.com/Documents/draft-hallambaker-dare- 1158 container.html 1160 [4] http://mathmesh.com/Documents/draft-hallambaker-dare- 1161 container.html 1163 Author's Address 1165 Phillip Hallam-Baker 1166 Comodo Group Inc. 1168 Email: philliph@comodo.com