idnits 2.17.1 draft-aspb-rof-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack a Security Considerations section. ** There are 28 instances of lines with control characters in the document. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 196 has weird spacing: '... if any and M...' -- Couldn't find a document date in the document -- date freshness check skipped. Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Missing Reference: 'RFC2119' is mentioned on line 105, but not defined == Unused Reference: 'RFC7159' is defined on line 259, but no explicit reference was found in the text ** Obsolete normative reference: RFC 7159 (Obsoleted by RFC 8259) Summary: 3 errors (**), 0 flaws (~~), 4 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 INTERNET-DRAFT Pushpita Bhattacharjee 3 Intended Status: Proposed Standard Abel Sanchez 4 Expires: July 11 2019 MIT 5 January 7 2019 7 Reproduce Object framework(ROF) 8 draft-aspb-rof-00 10 Abstract 12 This document postulates a standard for computational models to help 13 with reproducibility. This standard proposes a lightweight,programing 14 language agnostic, JavaScript Object Notation(JSON) based Framework 15 to describe the environment and behavior of a computational model. 17 Status of this Memo 19 This Internet-Draft is submitted to IETF in full conformance with the 20 provisions of BCP 78 and BCP 79. 22 Internet-Drafts are working documents of the Internet Engineering 23 Task Force (IETF), its areas, and its working groups. Note that 24 other groups may also distribute working documents as 25 Internet-Drafts. 27 Internet-Drafts are draft documents valid for a maximum of six months 28 and may be updated, replaced, or obsoleted by other documents at any 29 time. It is inappropriate to use Internet-Drafts as reference 30 material or to cite them other than as "work in progress." 32 The list of current Internet-Drafts can be accessed at 33 http://www.ietf.org/1id-abstracts.html 35 The list of Internet-Draft Shadow Directories can be accessed at 36 http://www.ietf.org/shadow.html 38 INTERNET DRAFT Reproduce Object framework(ROF) 40 Copyright and License Notice 42 Copyright (c) 2018 IETF Trust and the persons identified as the 43 document authors. All rights reserved. 45 This document is subject to BCP 78 and the IETF Trust's Legal 46 Provisions Relating to IETF Documents 47 (http://trustee.ietf.org/license-info) in effect on the date of 48 publication of this document. Please review these documents 49 carefully, as they describe your rights and restrictions with respect 50 to this document. Code Components extracted from this document must 51 include Simplified BSD License text as described in Section 4.e of 52 the Trust Legal Provisions and are provided without warranty as 53 described in the Simplified BSD License. 55 Table of Contents 57 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 58 1.1 Terminology . . . . . . . . . . . . . . . . . . . . . . . . 3 59 2. Grammar . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 60 3 Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 61 4 Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 62 5 Parser . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 63 6 Generator . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 64 7 IANA Considerations . . . . . . . . . . . . . . . . . . . . . . 6 65 8 References . . . . . . . . . . . . . . . . . . . . . . . . . . 6 66 8.1 Normative References . . . . . . . . . . . . . . . . . . . 6 67 8.2 Informative References . . . . . . . . . . . . . . . . . . 7 68 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 7 70 INTERNET DRAFT Reproduce Object framework(ROF) 72 1 Introduction 74 Reproducibility in computational research is an unsolved complex 75 challenge faced by numerous researchers today. The nature of the 76 complexity stems from various stakeholders involved in the end to end 77 research paradigm and from the multiple components involved in this 78 process, such as the computational model, software platform, data 79 sets, publication methods etc. Scientific research like any other 80 field has gone through a major digital transformation in the past 81 couple of decades with the rise of the world wide web. Many types of 82 research now depend on developing complex computational models, using 83 various software platforms and analyzing big sets of data. But when 84 it comes to publishing the results of research we still use the 85 archetypical only text-based publication process. This process makes 86 it increasingly difficult to reproduce research results from a 87 published study. Although research has become increasingly 88 multidisciplinary and dependent on computer software models and large 89 sets of data, the traditional way of scientific publications has not 90 changed much over time. Various studies have found that the text- 91 based analysis with supporting material presented in a paper is not 92 enough to reproduce research fully. This has been one of the factors 93 leading to the reproducibility crisis. Reproduce Object framework 94 takes a digital automation approach in trying to propose a solution 95 for the reproducibility challenge. The Reproduce Object Framework is 96 a JavaScript Object Notation (JSON) based lightweight standard to 97 define a computational model configuration, inputs to the model, its 98 results and environment to reproduce the model. 100 1.1 Terminology 102 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 103 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 104 document are to be interpreted as described in RFC 2119 [RFC2119]. 106 2. Grammar 108 This standard follows the basic JSON syntax of immutable key string 109 and user-defined value allowed by JSON grammar, the key-value pair is 110 separated by a colon(:). Starting with a right curly bracket ({)and 111 ending with the left curly bracket (}).The left of the colon(:) entry 112 signifies the key identifier of the entry and the right side provides 113 the value for each of the identifiers. It contains 6 mandatory 114 elements: 116 1. code_repository- this key corresponds to the code location for 118 INTERNET DRAFT Reproduce Object framework(ROF) 120 the model in the publicly accessible cloud. 122 2. language- this key identifies the programing language used to 123 develop the model. 125 3.language_version- this key declares the version of the programming 126 language used to develop the model. 128 4.input_file_location- this points to the relative location of the 129 input file that the computational model requires for the analysis. 130 The path is decided from the root of the code base folder structure 131 and includes the name and format of the file. 133 5.output_file- this key points to the relative location of the 134 results file that the computational model renders as the output of 135 the analysis is saved. The path is decided from the root of the code 136 base folder structure and includes the name and format of the file. 138 6.main_file- this key expects the file location of the executable 139 file that will trigger the execution. The path is decided from the 140 root of the code base folder structure and includes the name and 141 format of the file. 143 7.read_me- this key points to the location of the file with the 144 description of the model. The path is decided from the root of the 145 code base folder and it includes the name and format of the file. 146 The order of the keys may depend upon the implementation by the 147 platform. 149 3 Values 151 The Reproduce Object must contain only one permissible value for 152 each key.This specification allows the following as the values for 153 each key: 155 1. code_repository- must contain a single location where all the 156 source code for the model is uploaded. It can be in any of the 157 cloud source code sharing platforms like GitHub, Google Drive, 158 Dropbox etc. The path should be the same as the cloud repository 159 convention and accessible freely without any special 160 authorization. 162 2.language- must have the official programming language name 163 declared by the official language distribution. It will contain a 165 INTERNET DRAFT Reproduce Object framework(ROF) 167 single language name.It can have values like python, R, node_js 168 etc. This language compiler should be publicly available for 169 download and use. 171 3.language_version- must contain a single specific version number 172 of the programming language used to develop the computational 173 model. This value will be formatted with 3 dot notation, denoting 174 the major, minor and patch version as officially declared by the 175 programming language. This version should be publicly available 176 for download and use. 178 4.input_file-This value must contain a single path 179 location relative to the root folder of the source code. If the 180 file exists in the root folder itself , the declaration must be 181 ./filename.extension.Location path must include the file name and 182 extension of the file. 184 5.output_file- Must contain the location of results file 185 generated by the computational model, location is defined relative 186 to the root of the source code folder. Location path must include 187 the file name and extension of the file. 189 6.main_file - Must contain a single file path location 190 relative to the root folder of the source code. This key refers to 191 the file which triggers the execution of the model. 193 7.read_me- Must contain a single file path location 194 relative to the root folder of the source code. This file contains 195 details regarding the author of the model, copyright restrictions 196 if any and Model description. 198 The implementation can limit the size of string for memory and 199 other consideration. 201 4 Example 203 A brief, single-entry Reproduce standard Object: 205 { 207 code_repository: git@github.com:pushpitab/ML-Demo.git 209 language: node_js 211 INTERNET DRAFT Reproduce Object framework(ROF) 213 language_version:10.9.0 215 input_file: ./input/the_input_file.json 217 output_file:./results/the_output_file.json 219 main_file:./src/main.js 221 read_me:./README.md 223 } 225 It describes a computational model with the following 226 details: This model's source code is shared in public cloud 227 repository GitHub.The code can be accessed freely without special 228 authorization.This contains GitHub notation of sharing and user 229 identifying syntax.The programing language used in the model is 230 node_js. The version of node_js used is declared as 10.9.0. The 231 path to the input file required for the model to run is under the 232 root folder and one level down in a folder name input.The name of 233 the file is the_input_file.json After executing the model, the 234 result is saved in the results folder, inside the root of the 235 source folder by the name the_output_file.json.The main executable 236 file is located in the root of the src folder and named as 237 main.js. The last value refers to the read_me document with 238 details about the description, author and copyright. Its located 239 in the root of the source folder. 241 5 Parser A Reproduce Standard Object parser transforms the object to 242 the required actionable elements. Any standard JSON parser can be 243 used to parse the Reproduce Object. 245 6 Generator 247 To create the Reproduce Object, generating method MUST follow the 248 JSON grammar and the Reproduce Object grammar. 250 7 IANA Considerations 252 This memo includes no request to IANA. 254 8 References 256 8.1 Normative References 257 INTERNET DRAFT Reproduce Object framework(ROF) 259 [RFC7159] Bray, T., Ed., "The JavaScript Object 260 Notation (JSON) Data Interchange Format", RFC 261 7159, DOI 10.17487/RFC7159, March 2014, 262 . 263 8.2 Informative References 265 Authors' Addresses 267 Abel Sanchez 268 Massachusetts Institute of Technology 270 EMail: doval@mit.edu 272 Pushpita bhattacharjee 274 Massachusetts Institute of Technology 276 EMail: pushpital@mit.edu