idnits 2.17.1 draft-belchior-gateway-recovery-03.txt: -(4): Line appears to be too long, but this could be caused by non-ascii characters in UTF-8 encoding Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- == There are 3 instances of lines with non-ascii characters in the document. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack an IANA Considerations section. (See Section 2.2 of https://www.ietf.org/id-info/checklist for how to handle the case when there are no actions for IANA.) ** There are 105 instances of too long lines in the document, the longest one being 115 characters in excess of 72. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document seems to lack the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords. (The document does seem to have the reference to RFC 2119 which the ID-Checklist requires). -- The document date (3 October 2021) is 935 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- -- Looks like a reference, but probably isn't: '1' on line 800 -- Looks like a reference, but probably isn't: '2' on line 803 -- Looks like a reference, but probably isn't: '3' on line 807 -- Looks like a reference, but probably isn't: '4' on line 810 -- Looks like a reference, but probably isn't: '5' on line 814 -- Looks like a reference, but probably isn't: '6' on line 817 -- Looks like a reference, but probably isn't: '7' on line 820 -- Looks like a reference, but probably isn't: '8' on line 823 -- Looks like a reference, but probably isn't: '9' on line 829 -- Looks like a reference, but probably isn't: '10' on line 832 -- Looks like a reference, but probably isn't: '11' on line 835 -- Looks like a reference, but probably isn't: '12' on line 838 -- Looks like a reference, but probably isn't: '13' on line 841 -- Looks like a reference, but probably isn't: '14' on line 845 -- Looks like a reference, but probably isn't: '15' on line 848 -- Looks like a reference, but probably isn't: '16' on line 851 -- Looks like a reference, but probably isn't: '17' on line 854 -- Looks like a reference, but probably isn't: '18' on line 857 == Missing Reference: 'CD16' is mentioned on line 897, but not defined == Unused Reference: 'RFC2119' is defined on line 908, but no explicit reference was found in the text == Unused Reference: 'BHG87' is defined on line 923, but no explicit reference was found in the text == Unused Reference: 'Clar88' is defined on line 933, but no explicit reference was found in the text == Unused Reference: 'HS2019' is defined on line 944, but no explicit reference was found in the text == Unused Reference: 'SRC84' is defined on line 954, but no explicit reference was found in the text == Outdated reference: A later version (-03) exists of draft-hargreaves-odap-00 Summary: 2 errors (**), 0 flaws (~~), 10 warnings (==), 19 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Internet Engineering Task Force R. Belchior 3 Internet-Draft M. Correia 4 Intended status: Informational INESC-ID, Instituto Superior Técnico 5 Expires: 6 April 2022 T. Hardjono 6 MIT 7 3 October 2021 9 DLT Gateway Crash Recovery Mechanism 10 draft-belchior-gateway-recovery-03 12 Abstract 14 This memo describes the crash recovery mechanism for the Open Digital 15 Asset Protocol (ODAP), called ODAP-2PC. The goal is to assure 16 gateways running ODAP to be able to recover from crashes, and thus 17 preserve the consistency of an asset across ledgers (i.e., double 18 spend does not occur). This draft includes the description of the 19 messaging and logging flow necessary for the correct functioning of 20 ODAP-2PC. 22 Status of This Memo 24 This Internet-Draft is submitted in full conformance with the 25 provisions of BCP 78 and BCP 79. 27 Internet-Drafts are working documents of the Internet Engineering 28 Task Force (IETF). Note that other groups may also distribute 29 working documents as Internet-Drafts. The list of current Internet- 30 Drafts is at https://datatracker.ietf.org/drafts/current/. 32 Internet-Drafts are draft documents valid for a maximum of six months 33 and may be updated, replaced, or obsoleted by other documents at any 34 time. It is inappropriate to use Internet-Drafts as reference 35 material or to cite them other than as "work in progress." 37 This Internet-Draft will expire on 6 April 2022. 39 Copyright Notice 41 Copyright (c) 2021 IETF Trust and the persons identified as the 42 document authors. All rights reserved. 44 This document is subject to BCP 78 and the IETF Trust's Legal 45 Provisions Relating to IETF Documents (https://trustee.ietf.org/ 46 license-info) in effect on the date of publication of this document. 47 Please review these documents carefully, as they describe your rights 48 and restrictions with respect to this document. Code Components 49 extracted from this document must include Simplified BSD License text 50 as described in Section 4.e of the Trust Legal Provisions and are 51 provided without warranty as described in the Simplified BSD License. 53 Table of Contents 55 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 56 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3 57 3. Logging Model . . . . . . . . . . . . . . . . . . . . . . . . 4 58 3.1. Example . . . . . . . . . . . . . . . . . . . . . . . . . 5 59 3.2. Log Storage Modes . . . . . . . . . . . . . . . . . . . . 7 60 3.3. Log Storage API: . . . . . . . . . . . . . . . . . . . . 8 61 3.3.1. Response Codes . . . . . . . . . . . . . . . . . . . 10 62 4. Format of log entries . . . . . . . . . . . . . . . . . . . . 10 63 5. ODAP-2PC . . . . . . . . . . . . . . . . . . . . . . . . . . 13 64 5.1. Crash Recovery Model . . . . . . . . . . . . . . . . . . 13 65 5.2. Recovery Procedure . . . . . . . . . . . . . . . . . . . 14 66 5.2.1. Transfer Initiation Flow . . . . . . . . . . . . . . 14 67 5.2.2. Lock-Evidence Flow . . . . . . . . . . . . . . . . . 14 68 5.2.3. Commitment Establishment Flow . . . . . . . . . . . . 15 69 5.3. ODAP-2PC Messages . . . . . . . . . . . . . . . . . . . . 15 70 5.3.1. RECOVER . . . . . . . . . . . . . . . . . . . . . . . 15 71 5.3.2. RECOVER-UDPDATE . . . . . . . . . . . . . . . . . . . 16 72 5.3.3. RECOVER-UPDATE ACK . . . . . . . . . . . . . . . . . 16 73 5.3.4. RECOVER-SUCCESS . . . . . . . . . . . . . . . . . . . 16 74 5.3.5. ROLLBACK . . . . . . . . . . . . . . . . . . . . . . 17 75 5.4. Examples . . . . . . . . . . . . . . . . . . . . . . . . 17 76 5.4.1. Crashing before issuing a command to the counterparty 77 gateway . . . . . . . . . . . . . . . . . . . . . . . 17 78 5.4.2. Crashing after issuing a command to the counterparty 79 gateway . . . . . . . . . . . . . . . . . . . . . . . 19 80 6. Security Considerations . . . . . . . . . . . . . . . . . . . 20 81 7. References . . . . . . . . . . . . . . . . . . . . . . . . . 21 82 7.1. Normative References . . . . . . . . . . . . . . . . . . 21 83 7.2. Informative References . . . . . . . . . . . . . . . . . 21 84 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 22 86 1. Introduction 88 Gateway systems that perform virtual asset transfers among DLTs must 89 possess a degree of resiliency and fault tolerance in the face of 90 possible crashes. Accounting for the possibility of crashes is 91 particularly important to guarantee asset consistency across DLTs. 93 ODAP-2PC [HERMES] uses 2PC, an atomic commitment protocol (ACP). 2PC 94 considers two roles: a coordinator who manages the protocol's 95 execution and participants who manage the resources that must be kept 96 consistent. The source gateway plays the ACP role of Coordinator, 97 and the recipient gateway plays the Participant role in relay mode. 98 Gateways exchange messages corresponding to the protocol execution, 99 generating log entries for each one. 101 Log entries are organized into logs. Logs enable either the same or 102 other backup gateways to resume any phase of ODAP. This log can also 103 serve as an accountability tool in case of disputes. Another key 104 component is an atomic commit protocol (ACP) that guarantees that the 105 source and target DLTs are modified consistently (atomicity) and 106 permanently (durability), e.g., that assets that are taken from the 107 source DLT are persisted into the recipient DLT. 109 Log entries are then the basis satisfying one of the key deployment 110 requirements of gateways for asset transfers: a high degree of 111 availability. In this document, we consider two common strategies to 112 increase availability: (1) to support the recovery of the gateways 113 (self-healing model) and (2) to employ backup gateways with the 114 ability to resume a stalled transfer (primary-backup model) [HERMES]. 116 This memo proposes: (i) the logging model of ODAP-2PC; (ii) the log 117 storage types; (iii) the log storage API; (iv) the log entry format; 118 (v) the recovery and rollback procedures. 120 2. Terminology 122 There following are some terminology used in the current document: 124 * Gateway: The nodes of a DLT system that are functionally capable 125 of handling an asset transfer with another DLT. Gateway nodes 126 implement the gateway-to-gateway asset transfer protocol. 128 * Primary Gateway: The node of a DLT system that has been selected 129 or elected to act as a gateway in an asset transfer. 131 * Backup Gateway: The node of a DLT system that has been selected or 132 elected to act as a backup gateway to a primary gateway. 134 * Message Flow Parameters: The parameters and payload employed in a 135 message flow between a sending gateway and receiving gateway. 137 * Source Gateway (or G1): The gateway that initiates the transfer 138 protocol. Acts as a coordinator of the ACP and mediates the 139 message flow. 141 * Recipient Gateway (or G2): The gateway that is the target of an 142 asset transfer. It follows instructions from the source gateway. 144 * Source DLT: The DLT of the source gateway. 146 * Recipient DLT: The DLT of the recipient gateway. 148 * Log: Set of log entries such that those are ordered by the time of 149 its creation. 151 * Public (or Shared) Log: log where several nodes can read and write 152 from it. 154 * Private Log: log where only one node can read and write from it. 156 * Log data: The log information is retained by a gateway connected 157 to an exchanged message within an asset transfer protocol. 159 * Log entry: The log information generated and persisted by a 160 gateway regarding one specific message flow step. 162 * Log format: The format of log-data generated by a gateway. 164 * Atomic commit protocol (ACP): A protocol that guarantees that 165 assets that are taken from a DLT are persisted into the other DLT. 166 Examples are two and three-phase commit protocols (2PC, 3PC, 167 respectively) and non-blocking atomic commit protocols. 169 * Fault: A fault is an event that alters the expected behavior of a 170 system. 172 * Crash-fault tolerant models: models allowing a system to keep 173 operating correctly despite having a set of faulty components. 175 * Digital asset: a form of digital medium record used as a digital 176 representation of a tangible or intangible asset. 178 3. Logging Model 180 We consider the log file to be a stack of log entries. Each time a 181 log entry is added, it goes to the top of the stack (the highest 182 index). For each protocol step a gateway performs, a log entry is 183 created immediately before executing and immediately after executing 184 a given operation. 186 To manipulate the log, we define a set of log primitives that 187 translate log entry requests from a process into log entries, 188 realized by the log storage API (for the context of ODAP, 189 Section 3.5): 191 * writeLogEntry(e,L) (WRITE) - appends a log entry e in the log L 192 (held by the corresponding Log Storage Support). 194 * getLogEntry(i,L) (READ) - retrieves a log entry with index i from 195 log L. 197 From these primitives, other functions can be built: 199 * getLogLength (L) (READ) - obtains the number of log entries from 200 log L. 202 * getLogDiff(l1,l2) (READ) - obtains the difference between two 203 logs. 205 * getLastEntry(L): obtains the last log entry from log L. 207 * getLog(L): retrieves the whole log L. 209 * updateLog(l1,l2): updates l1 based on l2 (uses getLogDiff and 210 writeLogEntry). 212 Example 3.1 shows a simplified version log referring to the transfer 213 initiation flow ODAP phase. Each log entry (simplified, see the 214 definition in Section 3) is composed of metadata (phase, sequence 215 number) and one attribute from the payload (operation). Operations 216 map behavior to state (see Section 3). 218 The following table illustrates the log storage API. The Function 219 describes the primitive supported by the log storage API. The 220 Parameters column specifies the parameters given to the endpoint as 221 query parameters. Endpoint specifies the endpoint mapping a specific 222 log primitive. The column Returns specifies what the contents of 223 "response_data" mean. The column Response Example illustrates this 224 last field. 226 3.1. Example 227 ,--. ,--. ,-------. 228 |G1| |G2| |Log API| 229 `--' `--' `-------' 230 | [1]: writeLogEntry <1,1,init-validate> | 231 | ---------------------------------------------------------------> 232 | | | 233 | initiate ODAP's phase 1| | 234 | -----------------------> | 235 | | | 236 | | [2]: writeLogEntry <1,2,exec-validate>| 237 | | --------------------------------------> 238 | | | 239 | |----. | 240 | | | execute validate from p1 | 241 | |<---' | 242 | | | 243 | | [3]: writeLogEntry <1,3,done-validate>| 244 | | --------------------------------------> 245 | | | 246 | | [4]: writeLogEntry <1,4,ack-validate> | 247 | | --------------------------------------> 248 | | | 249 | validation complete | | 250 | <----------------------- | 251 ,--. ,--. ,-------. 252 |G1| |G2| |Log API| 253 `--' `--' `-------' 255 Figure 1 257 Example 2.1 shows the sequence of logging operations over part of the 258 first phase of ODAP (simplified): 260 * 1. At step 1, G1 writes an init-validate operation, meaning it 261 will require G2 to initiate the validate function: This step 262 generates a log entry (p1, 1, init-validate). 264 * 2. At step 2, G2 writes an exec-validate operation, meaning it 265 will try to execute the validate function: This step generates a 266 log entry (p1, 2, exec-validate). 268 * 3. At step 3, G2 writes a done-validate operation, meaning it 269 successfully executed the validate function: This step generates a 270 log entry (p1, 3, done-validate). 272 * 4. At step 4, G2 writes an ack-validate operation, meaning it 273 will send an acknowledgment to G1 regarding the done-validate: 274 This step generates a log entry (p1, 4, ack-validate). 276 3.2. Log Storage Modes 278 Gateways store state mapped by logs. Gateways have private logs 279 recording enterprise-sensitive data that can be used, for instance, 280 for analytics (enterprise log). Entries can include end-to-end 281 cross-jurisdiction transaction latency and throughput. 283 Apart from the enterprise log, a state log can be public or private, 284 centralized or decentralized. This log is meant to be shared with 285 everyone with an Internet connection (public) or only within the 286 gateway consortium (private). Logs can be stored locally or in a 287 cloud service, per gateway (centralized), or in a decentralized 288 infrastructure (i.e., decentralized ledger, decentralized database). 289 We call the latter option decentralized log storage. The type of the 290 state log depends on the trust assumptions among gateways and the 291 access mode [ODAP]. 293 In greater detail: 295 * 1. Public decentralized log: log entries are stored on a 296 decentralized public log (e.g., Ethereum blockchain, IPFS). Each 297 gateway writes non-encrypted log entries to a decentralized log 298 storage. Although this is the best option for providing 299 accountability of gateways, availability, and integrity of the 300 logs, leading to shorter dispute resolution, this can lead to 301 privacy issues. The integrity of the log can be asserted by 302 hashing the entries and comparing it to each stored hash on the 303 decentralized log storage. A solution to the privacy problems 304 could be given by gateways publishing a hash of the log entry plus 305 metadata to the decentralized log storage instead of the log 306 entries. Although this is a first step towards resolving privacy 307 issues, a tradeoff with data availability is done. In particular, 308 this choice leads to lower availability guarantees since a gateway 309 needs to wait for the counter-party gateway to deliver the logs in 310 case logs need to be shared. In this case, the decentralized log 311 storage acts as a notarizing service. This mode is recommended 312 when gateways operate in the Relay Mode: Client-initiated Gateway 313 to Gateway. This mode can also be used by the Direct Mode: Client 314 to Multiple Gateway access mode because gateways may need to share 315 state between themselves. Note: the difference between the 316 mentioned modes is that in Direct Mode: Client to Multiple 317 Gateway, a single client/organization controls all the gateways, 318 whereas, in the Relay Mode, gateways are controlled by different 319 organizations. 321 * 2. Public centralized log: log entries are published in a 322 bulletin that more organizations control. That bulletin can be 323 updated or removed at any time. Accountability is only guaranteed 324 provided that there are multiple copies of such bulletin by 325 conflicting parties. Availability and integrity can be obtained 326 via redundancy. 328 * 3. Private centralized log. Each gateway stores logs locally or 329 in a cloud in the private log storage mode but does not share them 330 by default with other gateways. If needed, logs are requested 331 from the counter-party gateway. Saving logs locally is faster 332 than saving them on the respective ledger since issuing a 333 transaction is several orders of magnitude slower than writing on 334 a disk or accessing a cloud service. Nonetheless, this model 335 delivers weaker integrity and availability guarantees. 337 Each log storage mode provides a different process to recover the 338 state from crashes. In the private log, a gateway requires the most 339 recent log from the counter-party gateway. This mode is the one 340 where the most trust is needed. The gateway publishes hashes of log 341 entries and metadata on a decentralized log storage in the 342 centralized public log. Gateways who need the logs request them from 343 other gateways and perform integrity checks of the received logs. In 344 the public decentralized mode, the gateways publish the plain log 345 entries on decentralized log storage. This is the most trustless and 346 decentralized mode of operation. 348 By default, if there are gateways from different institutions 349 involved in an asset transfer, the storage mode should be a 350 decentralized log storage. The decentralized log storage can provide 351 a common source of truth to solve disputes and maintain a shared 352 state, alleviating trust assumptions between gateways. 354 3.3. Log Storage API: 356 The log storage API allows developers to be abstracted from the log 357 storage support, providing a standardized way to interact with logs 358 (e.g., relational vs. non-relational, local vs. on-chain). It also 359 handles access control if needed. 361 +---------------------------------------+----------------------------------+------------------------------------------------------------------------+ 362 | Function | Parameters | Endpoint | 363 +---------------------------------------+----------------------------------+------------------------------------------------------------------------+ 364 | Append log entry | logId - log entry to be appended | POST / writeLogEntry/:logId Host: example.org Accept: application/json | 365 +---------------------------------------+----------------------------------+------------------------------------------------------------------------+ 366 | Obtains a log entry | id - log entry id | GET getLogEntry/:id Host: example.org | 367 +---------------------------------------+----------------------------------+------------------------------------------------------------------------+ 368 | Obtains the length of the log | None | GET getLogLength Host: example.org | 369 +---------------------------------------+----------------------------------+------------------------------------------------------------------------+ 370 | Obtains the difference | log - log to be compared | POST /getLogDiff/:log Host: example.org | 371 | between a given log and a current log | | | 372 +---------------------------------------+----------------------------------+------------------------------------------------------------------------+ 373 | Obtains the last log entry | None | GET getLastEntry Host: example.org | 374 +---------------------------------------+----------------------------------+------------------------------------------------------------------------+ 375 | Obtains the whole log | None | GET getLog Host: example.org | 376 +---------------------------------------+----------------------------------+------------------------------------------------------------------------+ 378 Figure 2 380 The following table maps the respecetive return values and response 381 examples: 383 +---------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+ 384 | Returns | Response Example | 385 +=================================+=======================================================================================================================================================+ 386 | The entry index of the last log | HTTP/1.1 200 OK Cache-Control: private Date: Mon, 02 Mar 2020 05:07:35 GMT Content-Type: application/json { "success": true, "response_data":"2" } | 387 | (string) | | 388 +---------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+ 389 | A log entry | HTTP/1.1 200 OK Cache-Control: private Date: Mon, 02 Mar 2020 05:07:35 GMT Content-Type: application/json { "success": true, "response_data": {...} } | 390 +---------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+ 391 | The length of the log | HTTP/1.1 200 OK Cache-Control: private Date: Mon, 02 Mar 2020 05:07:35 GMT Content-Type: application/json { "success": true, "response_data":"2" } | 392 | (string) | | 393 +---------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+ 394 | The difference between two logs | HTTP/1.1 200 OK Cache-Control: private Date: Mon, 02 Mar 2020 05:07:35 GMT Content-Type: application/json { "success": true, "response_data": {...} } | 395 +---------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+ 396 | A log entry | HTTP/1.1 200 OK Cache-Control: private Date: Mon, 02 Mar 2020 05:07:35 GMT Content-Type: application/json { "success": true, "response_data": {...} } | 397 +---------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+ 398 | The log | HTTP/1.1 200 OK Cache-Control: private Date: Mon, 02 Mar 2020 05:07:35 GMT Content-Type: application/json { "success": true, "response_data": {...} } | 399 +---------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+ 401 Figure 3 403 3.3.1. Response Codes 405 The log storage API MUST respond with return codes indicating the 406 failure (error 5XX) or success of the operation (200). The 407 application may carry out a further operation in the future to 408 determine the ultimate status of the operation. 410 The log storage API response is in JSON format and contains two 411 fields: 1) success: true if the operation was successful, and 2) 412 response_data: contains the payload of the response generated by the 413 log storage API. 415 4. Format of log entries 417 A gateway stores the log entries in its log, and they capture 418 gateways operations. Entries account for the current status of one 419 of the three ODAP flows: Transfer Initiation flow, Lock-Evidence 420 flow, and Commitment Establishment flow. 422 The recommended format for log entries is JSON, with protocol- 423 specific mandatory fields supporting a free format field for 424 plaintext or encrypted payloads directed at the DLT gateway or an 425 underlying DLT. Although the recommended format is JSON, other 426 formats can be used (e.g., XML). 428 The mandatory fields of a log entry, that are generated by ODAP, are: 430 * Version: ODAP protocol Version (major, minor). 432 * Session ID: unique identifier (UUIDv2) representing a session. 434 * Sequence Number: monotonically increasing counter that uniquely 435 represents a message from a session. 437 * ODAP Phase: current ODAP phase. 439 * Resource URL: Location of Resource to be accessed. 441 * Developer URN: Assertion of developer / application identity. 443 * Action/Response: GET/POST and arguments (or Response Code). 445 * Credential Profile: Specify type of auth (e.g. SAML, OAuth, 446 X.509). 448 * Credential Block: Credential token, certificate, string. 450 * Payload Profile: Asset Profile provenance and capabilities. 452 * Application Profile: Vendor or Application specific profile. 454 * Payload: Payload for POST, responses, and native DLT txns. The 455 payload is specific to the current ODAP phase. 457 * Payload Hash: hash of the current message payload. 459 In addition to the attributes that belong to ODAP s schema, each log 460 entry REQUIRES the following attributes: 462 * timestamp REQUIRED: timestamp referring to when the log entry was 463 generated (UNIX format). 465 * source_gateway_pubkey REQUIRED: the public key of the gateway 466 initiating a transfer. 468 * source_gateway_dlt_system REQUIRED: the ID of the source DLT. 470 * recipient_gateway_pubkey REQUIRED: the public key of the gateway 471 involved in a transfer. 473 * recipient_gateway_dlt_system REQUIRED: the ID of the recipient 474 gatewayinvolved in a transfer. 476 * logging_profile REQUIRED: contains the profile regarding the 477 logging procedure. Default is local store. 479 * Message_signature REQUIRED: Gateway EDCSA signature over the log 480 entry. 482 * Last_entry_hash REQUIRED: Hash of previous log entry. 484 * Access_control_profile REQUIRED: the profile regarding the 485 confidentiality of the log entries being stored. Default is only 486 the gateway that created the logs can access them. 488 * Operation: the high level operation being executed by the gateway 489 on that step. There are five types of operations: Operation init- 490 states the intention of a node to execute a particular operation; 491 Operation exec- expresses that the node is executing the 492 operation; Operation done- states when a node successfully 493 executed a step of the protocol; Operation ack- refers to when a 494 node acknowledges a message received from another (e.g., command 495 executed); Operation fail- occurs when an agent fails to execute a 496 specific step. 498 Optional field entries are: 500 * source_gateway_uid OPTIONAL: the uid of the source gateway 501 involved in a transfer. 503 * recipient_gateway_uid : the uid of the recipient gateway involved 504 in a transfer. 506 * recovery message: the type of recovery message, if gateway is 507 involved in a recovery procedure. 509 * recovery payload: the payload associated with the recovery 510 message. 512 Example of a log entry created by G1, corresponding to locking an 513 asset (phase 2.3 of the ODAP protocol) : 515 { 516 "sessionId": "4eb424c8-aead-4e9e-a321-a160ac3909ac", 517 "seqNumber": 6, 518 "phaseId": "lock", 519 "sourceGatewayId": "5.47.165.186", 520 "sourceDltId": "Hyperledger-Fabric-JusticeChain", 521 "targetGatewayId": "192.47.113.116", 522 "targetDltId": "Ethereum", 523 "timestamp": "1606157330", 524 "payload": { 525 "messageType": "2pc-log", 526 "message": "LOCK_ASSET", 527 "votes": "none" 528 }, 529 "payloadHash": "80BCF1C7421E98B097264D1C6F1A514576D6C9F4EF04955FA3AEF1C0664B34E3", 530 "logEntryHash": "[...]" 531 } 533 Figure 4 535 Example of a log entry created by G2, acknowledging G1 locking an 536 asset (phase 2.4 of the ODAP protocol) : 538 { 539 "sessionId": "4eb424c8-aead-4e9e-a321-a160ac3909ac", 540 "seqNumber": 7, 541 "phaseId": "lock", 542 "sourceGatewayId": "5.47.165.186", 543 "sourceDltId": "Hyperledger-Fabric-JusticeChain", 544 "targetGatewayId": "192.47.113.116", 545 "targetDltId": "Ethereum", 546 "timestamp": "1606157333", 547 "payload": { 548 "messageType": "2pc-log", 549 "message": "LOCK_ASSET_ACK", 550 "votes": "none" 551 } 553 Figure 5 555 5. ODAP-2PC 557 This section defines general considerations about crash recovery. 558 ODAP-2PC is the application of the gateway crash recovery mechanism 559 to asset transfers across all ODAP phases. 561 5.1. Crash Recovery Model 563 Gateways can fail by crashing (i.e., becoming silent). In order to 564 be able to recover from these crashes, gateways store log entries in 565 a persistent data storage. Thus, gateways can recover by obtaining 566 the latest successful operation and continuing from there. We 567 consider two recovery models: 569 * 1. Self-healing mode: assumes that after a crash, a gateway 570 eventually recovers. The gateway does not lose its long-term keys 571 (public-private key pair) and can reestablish all TLS connections. 573 * 2. Primary-backup mode assumes that a gateway may never recover 574 after a crash but that this failure can be detected by timeout 575 [AD76]. If the timeout is exceeded, a backup gateway detects that 576 failure unequivocally and takes the role of the primary gateway. 577 The failure is detected using heartbeat messages and a 578 conservative period. 580 In both modes, after a gateway recovers, the gateways follow a 581 general recovery procedure (in Section 6.2 explained in detail for 582 each phase): 584 * 1. Crash communication: using the self-healing or primary-backup 585 modes, a node recovers. After that, it sends a message RECOVER to 586 the counterparty gateways. 588 * 2. State update: The gateway syncs its state with the latest 589 state, either by requesting it from the decentralized log storage 590 or other gateways (depending on the log storage mode). If a 591 decentralized log storage is available, the crashed gateway 592 attempts to update its local log, using getLogDiff from the shared 593 log. If there is no shared log, the crashed gateway needs to 594 synchronize itself with the counterparty gateway by querying the 595 counterparty gateway with a recovery message RECOVER containing 596 the latest log before the crash. The counterparty gateway sends 597 back a RECOVER-UPDATE message with its log. The recovered gateway 598 can now reconstruct the updated log via getLogDiff, and derive the 599 current state of the asset transfer. The gateways now share the 600 same state and can proceed with its operation. 602 * 3. Recovery communication: The gateway and informs other gateways 603 of the recovery with a recovery confirmation message is sent 604 (RECOVERY-CONFIRM), and the respective acknowledgment is sent by 605 the counterparty gateway (RECOVERY-ACK). 607 Finally, the gateway resumes the normal execution of ODAP. 609 5.2. Recovery Procedure 611 The previous section explained the general procedure that gateways 612 follow upon crashing. In more detail, for each ODAP phase, we define 613 the recovery procedure called ODAP-2PC: 615 5.2.1. Transfer Initiation Flow 617 This phase of ODAP follows the Crash Recovery Model from Section 6.1. 619 5.2.2. Lock-Evidence Flow 621 This phase of ODAP follows the Crash Recovery Model from Section 6.1. 622 Note that, in this phase, distributed ledgers were changed by 623 gateways. The crash gateways' recovery should take place in less 624 than the timeout specified for the asset transfer. Otherwise, the 625 rollback protocol present in the next section is applied. 627 5.2.3. Commitment Establishment Flow 629 This phase of ODAP follows the Crash Recovery Model from Section 6.1 630 and extra steps because in the third phase, distributed gateways 631 changed ledgers. As transactions cannot be undone on blockchains, 632 reverting a transaction includes issuing new transactions (with the 633 contrary effect of the ones to be reverted). We use a rollback list 634 [HERMES] to keep track of which transaction may be rolled back. The 635 crash recovery protocol for the Commitment Establishment Flow is as 636 follows (steps according to Figure 4 [HERMES]): 638 * 1. Rollback lists for all the gateways involved are initialized. 640 * 2. On step 2.3, add a pre-lock transaction to the source gateway 641 rollback list. 643 * 3. On step 3.2, if the request is denied, abort the transaction 644 and apply rollbacks on the source gateway. 646 * 4. On step 3.3, add a lock transaction to the source gateway 647 rollback list. 649 * 5. On step 3.4, if the commit fails, abort the transaction and 650 apply rollbacks on the source gateway. 652 * 6. On step 3.5, add a create asset transaction to the rollback 653 list of the recipient gateway. 655 * 7. On step 3.8, if the commit is successful, ODAP terminates. 657 * 8: Otherwise, if the last commit is unsuccessful, then abort the 658 transaction and apply rollbacks to both gateways. 660 5.3. ODAP-2PC Messages 662 ODAP-2PC messages are used to recover from crashes at the several 663 ODAP phases. These messages inform gateways of the current state of 664 a recovery procedure. ODAP-2PC messages follow the log format from 665 Section 4. 667 5.3.1. RECOVER 669 A recover message is sent from the crashed gateway to the 670 counterparty gateway, sending its most recent state. This message 671 type is encoded on the recovery message field of an ODAP log. 673 The parameters of the recovery message payload consists of the 674 following: 676 * ODAP phase: latest ODAP phase registered. 678 * Sequence number: latest sequence number registered. 680 * Last_entry_hash REQUIRED: Hash of previous log entry. 682 5.3.2. RECOVER-UDPDATE 684 The recover update message is sent by the counterparty gateway after 685 receiving a recover message from a recovered gateway. The recovered 686 gateway informs of its current state (via the current state of the 687 log). The counterparty gateway now calculates the difference between 688 the log entry corresponding to the received sequence number from the 689 recovered gateway and the latest sequence number (corresponding to 690 the latest log entry). This state is sent to the recovered gateway. 692 The parameters of the recover update payload consists of the 693 following: 695 * recovered logs: the list of log messages that the recovered 696 gateway needs to update. 698 5.3.3. RECOVER-UPDATE ACK 700 The recover-update ack message (response to RECOVER-UPDATE) states if 701 the recovered gateway's logs has been successfully updated. If 702 inconsistencies are detected, the recovered gateway answers with 703 initiates a dispute (RECOVER-DISPUTE message). 705 The parameters of this message consists of the following: 707 * success: true/false. 709 * entries changed: list of hashes of log entries that were appeded 710 to the recovered gateway log. 712 5.3.4. RECOVER-SUCCESS 714 The recover-ack message is sent by the counterparty gateway to the 715 recovered gateway acknowledging that the state is synchronized. 717 The parameters of this message consists of the following: 719 * success: true/false. 721 5.3.5. ROLLBACK 723 A rollback message is sent by a gateway that initiated a rollback as 724 defined by ODAP-2PC. 726 The parameters of this message consists of the following: 728 * success: true/false. 730 * actions performed: actions performed to rollback a state (e.g., 731 UNLOCK; BURN). 733 * proofs: TBD. 735 5.4. Examples 737 There are several situations when a crash may occur. 739 5.4.1. Crashing before issuing a command to the counterparty gateway 741 The following figure represents the source gateway (G1) crashing 742 before it issued an init command to the recipient gateway (G2). 744 ,--. ,--. ,-------. 745 |G1| |G2| |Log API| 746 `--' `--' `-------' 747 | [1]: writeLogEntry <1, 1, init-validate> | 748 | -------------------------------------------------> 749 | | | 750 |----. | | 751 | | [2] Crash | | 752 |<---' ... | | 753 | [3]recover | | 754 | | | 755 | | | 756 | [4] <1, 2, RECOVER> | | 757 | -----------------------------> | 758 | | | 759 | | [5] getLogEntry(i)| 760 | | ------------------> 761 | | | 762 | | [6] logEntries | 763 | | <- - - - - - - - - 764 | | | 765 | [7] <1,3,RECOVER-UPDATE> | | 766 | <----------------------------- | 767 | | | 768 |----. | | 769 | | [8] process log | | 770 |<---' | | 771 | | | 772 | [9] <1,4,writeLogEntry> | 773 | -------------------------------------------------> 774 | | | 775 | [10] <1,5,RECOVER-UPDATE-ACK>| | 776 | -----------------------------> | 777 | | | 778 | [11] <1,6,RECOVER-SUCESS> | | 779 | <----------------------------- | 780 | | | 781 | [12]: <1,7,init-validateNext> | 782 | -------------------------------------------------> 783 ,--. ,--. ,-------. 784 |G1| |G2| |Log API| 785 `--' `--' `-------' 787 Figure 6 789 5.4.2. Crashing after issuing a command to the counterparty gateway 791 The second scenario requires further synchronization (figure below). 792 At the retrieval of the latest log entry, G1 notices its log is 793 outdated. It updates it upon necessary validation and then 794 communicates its recovery to G2. The process then continues as 795 defined. 797 ,--. ,--. ,-------. 798 |G1| |G2| |Log API| 799 `--' `--' `-------' 800 | [1]: writeLogEntry <1,1,init-validate> | 801 | -----------------------------------------------------------------> 802 | | | 803 | [2]: <1,1,init-validate> | | 804 | -----------------------------> | 805 | | | 806 |----. | | 807 | | [3] Crash | | 808 |<---' | | 809 | | | 810 | | [4]: writeLogEntry | 811 | | ----------------------------------> 812 | | | 813 | |----. | 814 | | | [5]: execute init | 815 | |<---' | 816 | | | 817 | | [6]: writeLogEntry | 818 | | ----------------------------------> 819 | | | 820 | | [7]: writeLogEntry | 821 | | ----------------------------------> 822 | | | 823 | [8] <1,2,init-validate-ack> | | 824 | discovers that G1 crashed | | 825 | via timeout | | 826 | <----------------------------- | 827 | | | 828 |----. | | 829 | | [9] Recover | | 830 |<---' | | 831 | | | 832 | [10] <1, 2, RECOVER> | | 833 | -----------------------------> | 834 | | | 835 | | [11] getLogEntry(i) | 836 | | ----------------------------------> 837 | | | 838 | | [12] logEntries | 839 | | <- - - - - - - - - - - - - - - - - 840 | | | 841 | [13] <1,3,RECOVER-UPDATE> | | 842 | <----------------------------- | 843 | | | 844 |----. | | 845 | | [14] process log | | 846 |<---' | | 847 | | | 848 | [15] <1,4,writeLogEntry> | 849 | -----------------------------------------------------------------> 850 | | | 851 | [16] <1,5,RECOVER-UPDATE-ACK>| | 852 | -----------------------------> | 853 | | | 854 | [17] <1,6,RECOVER-SUCESS> | | 855 | <----------------------------- | 856 | | | 857 | [18]: <1,7,init-validateNext> | 858 | -----------------------------------------------------------------> 859 ,--. ,--. ,-------. 860 |G1| |G2| |Log API| 861 `--' `--' `-------' 863 Figure 7 865 6. Security Considerations 867 We assume a trusted, authenticated, secure, reliable communication 868 channel between gateways (i.e., messages cannot be spoofed and/or 869 altered by an adversary) using TLS/HTTPS [TLS]. Clients support 870 acceptable credential schemes such as OAuth2.0. We assume the 871 storage service used provides the means necessary to assure the logs' 872 confidentiality and integrity, stored and in transit. The service 873 must provide an authentication and authorization scheme, e.g., based 874 on OAuth and OIDC [OIDC], and use secure channels based on TLS/HTTPS 875 [TLS]. The present protocol is crash fault-tolerant, meaning that it 876 handles gateways that crash for several reasons (e.g., power outage). 877 The present protocol does not support Byzantine faults, where 878 gateways can behave arbitrarily (including being malicious). This 879 implies that both gateways are considered trusted. We assume logs 880 are not tampered with or lost. Log entries need integrity, 881 availability, and confidentiality guarantees, as they are an 882 attractive point of attack [BVC19]. Every log entry contains a hash 883 of its payload for guaranteeing integrity. If extra guarantees are 884 needed (e.g., non-repudiation), a log entry might be signed by its 885 creator. Availability is guaranteed by the usage of the log storage 886 API that connects a gateway to a dependable storage (local, external, 887 or DLT-based). Each underlying storage provides different 888 guarantees. Access control can be enforced via the access control 889 profile that each log can have associated with, i.e., the profile can 890 be resolved, indicating who can access the log entry in which 891 condition. Access control profiles can be implemented with access 892 control lists for simple authorization. The authentication of the 893 entities accessing the logs is done at the Log Storage API level 894 (e.g., username+password authentication in local storage vs. 895 blockchain-based access control in a DLT). For extra guarantees, the 896 nodes running the log storage API (or the gateway nodes themselves) 897 can be protected by hardening technologies such as Intel SGX [CD16]. 899 7. References 901 7.1. Normative References 903 [ODAP] Hargreaves, M. and T. Hardjono, "Open Digital Asset 904 Protocol, October 2020, IETF, draft-hargreaves-odap-00.", 905 October 2020, 906 . 908 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 909 Requirement Levels", BCP 14, RFC 2119, 910 DOI 10.17487/RFC2119, March 1997, 911 . 913 [TLS] Rescorla, E., "The Transport Layer Security (TLS) Protocol 914 Version 1.3?, RFC 8446.", 2018, 915 . 917 7.2. Informative References 919 [AD76] Alsberg, P. and D. Day, "A principle for resilient sharing 920 of distributed resources. In Proc. of the 2nd Int. Conf. 921 on Software Engineering", 1976, <978-0-201-10715-9>. 923 [BHG87] Bernstein, P., Hadzilacos, V., and N. Goodman, 924 "Concurrency Control and Recovery in Database Systems, 925 Chapter 7. Addison Wesley Publishing Company", 1987, 926 . 928 [BVC19] Belchior, R., Vasconcelos, A., and M. Correia, "Towards 929 Secure, Decentralized, and Automatic Audits with 930 Blockchain. European Conference on Information Systems", 931 2019, . 933 [Clar88] Clark, D., "The Design Philosophy of the DARPA Internet 934 Protocols, ACM Computer Communication Review, Proc SIGCOMM 935 88, vol. 18, no. 4, pp. 106-114", August 1988. 937 [HERMES] Belchior, R., Vasconcelos, A., Correia, M., and T. 938 Hardjono, "HERMES: Fault-Tolerant Middleware for 939 Blockchain Interoperability", 2021, 940 . 944 [HS2019] Hardjono, T. and N. Smith, "Decentralized Trusted 945 Computing Base for Blockchain Infrastructure Security, 946 Frontiers Journal, Special Issue on Blockchain Technology, 947 Vol. 2, No. 24", December 2019, 948 . 950 [OIDC] Sakimura, N., Bradley, J., Jones, M., de Medeiros, B., and 951 C. Mortimore, "OpenID Connect Core 1.0", 2014, 952 . 954 [SRC84] Saltzer, J., Reed, D., and D. Clark, "End-to-End Arguments 955 in System Design, ACM Transactions on Computer Systems, 956 vol. 2, no. 4, pp. 277-288", November 1984. 958 Authors' Addresses 960 Rafael Belchior 961 INESC-ID, Instituto Superior Técnico 963 Email: rafael.belchior@tecnico.ulisboa.pt 965 Miguel Correia 966 INESC-ID, Instituto Superior Técnico 968 Email: miguel.p.correia@tecnico.ulisboa.pt 970 Thomas Hardjono 971 MIT 973 Email: hardjono@mit.edu