idnits 2.17.1 draft-huitema-rfc-eval-project-01.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack an Introduction section. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (August 7, 2019) is 1724 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- -- Obsolete informational reference (is this intentional?): RFC 8312 (Obsoleted by RFC 9438) Summary: 1 error (**), 0 flaws (~~), 1 warning (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group C. Huitema 3 Internet-Draft Private Octopus Inc. 4 Intended status: Informational August 7, 2019 5 Expires: February 8, 2020 7 RFC Evaluation Project - First Step 8 draft-huitema-rfc-eval-project-01 10 Abstract 12 This document presents a first attempt at evaluating the recently 13 published RFC. We analyze a set of randomly chosen RFC approved in 14 2018, looking for history and delays, and using Google Scholar as a 15 proxy for the RFC popularity. The results are interesting, and 16 inform further evaluation efforts. 18 Status of This Memo 20 This Internet-Draft is submitted in full conformance with the 21 provisions of BCP 78 and BCP 79. 23 Internet-Drafts are working documents of the Internet Engineering 24 Task Force (IETF). Note that other groups may also distribute 25 working documents as Internet-Drafts. The list of current Internet- 26 Drafts is at https://datatracker.ietf.org/drafts/current/. 28 Internet-Drafts are draft documents valid for a maximum of six months 29 and may be updated, replaced, or obsoleted by other documents at any 30 time. It is inappropriate to use Internet-Drafts as reference 31 material or to cite them other than as "work in progress." 33 This Internet-Draft will expire on February 8, 2020. 35 Copyright Notice 37 Copyright (c) 2019 IETF Trust and the persons identified as the 38 document authors. All rights reserved. 40 This document is subject to BCP 78 and the IETF Trust's Legal 41 Provisions Relating to IETF Documents 42 (https://trustee.ietf.org/license-info) in effect on the date of 43 publication of this document. Please review these documents 44 carefully, as they describe your rights and restrictions with respect 45 to this document. Code Components extracted from this document must 46 include Simplified BSD License text as described in Section 4.e of 47 the Trust Legal Provisions and are provided without warranty as 48 described in the Simplified BSD License. 50 Table of Contents 52 1. RFC Evaluation project . . . . . . . . . . . . . . . . . . . 2 53 2. Methodology . . . . . . . . . . . . . . . . . . . . . . . . . 3 54 3. Analysis of 20 selected RFC . . . . . . . . . . . . . . . . . 6 55 3.1. 8411 . . . . . . . . . . . . . . . . . . . . . . . . . . 6 56 3.2. 8456 . . . . . . . . . . . . . . . . . . . . . . . . . . 7 57 3.3. 8446 . . . . . . . . . . . . . . . . . . . . . . . . . . 7 58 3.4. 8355 . . . . . . . . . . . . . . . . . . . . . . . . . . 8 59 3.5. 8441 . . . . . . . . . . . . . . . . . . . . . . . . . . 9 60 3.6. 8324 . . . . . . . . . . . . . . . . . . . . . . . . . . 10 61 3.7. 8377 . . . . . . . . . . . . . . . . . . . . . . . . . . 10 62 3.8. 8498 . . . . . . . . . . . . . . . . . . . . . . . . . . 11 63 3.9. 8479 . . . . . . . . . . . . . . . . . . . . . . . . . . 11 64 3.10. 8453 . . . . . . . . . . . . . . . . . . . . . . . . . . 12 65 3.11. 8429 . . . . . . . . . . . . . . . . . . . . . . . . . . 12 66 3.12. 8312 . . . . . . . . . . . . . . . . . . . . . . . . . . 13 67 3.13. 8492 . . . . . . . . . . . . . . . . . . . . . . . . . . 14 68 3.14. 8378 . . . . . . . . . . . . . . . . . . . . . . . . . . 15 69 3.15. 8361 . . . . . . . . . . . . . . . . . . . . . . . . . . 15 70 3.16. 8472 . . . . . . . . . . . . . . . . . . . . . . . . . . 16 71 3.17. 8466 . . . . . . . . . . . . . . . . . . . . . . . . . . 16 72 3.18. 8362 . . . . . . . . . . . . . . . . . . . . . . . . . . 17 73 3.19. 8468 . . . . . . . . . . . . . . . . . . . . . . . . . . 17 74 4. Observations . . . . . . . . . . . . . . . . . . . . . . . . 19 75 4.1. Publication delays . . . . . . . . . . . . . . . . . . . 19 76 4.2. Preparation and Publication delays . . . . . . . . . . . 24 77 4.3. Copy editing . . . . . . . . . . . . . . . . . . . . . . 27 78 4.4. Independent Series . . . . . . . . . . . . . . . . . . . 30 79 4.5. Citation Counts . . . . . . . . . . . . . . . . . . . . . 30 80 5. Next steps . . . . . . . . . . . . . . . . . . . . . . . . . 36 81 6. Security considerations . . . . . . . . . . . . . . . . . . . 37 82 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 37 83 8. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 37 84 9. Informative References . . . . . . . . . . . . . . . . . . . 37 85 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 40 87 1. RFC Evaluation project 89 As stated on the organization's web site, "The IETF is a large open 90 international community of network designers, operators, vendors, and 91 researchers concerned with the evolution of the Internet architecture 92 and the smooth operation of the Internet." In this memo, we attempt 93 to evaluate the RFC production process. 95 The IETF data tracker provides information about RFC and drafts, from 96 which we can infer statistics about the production system. We can 97 measure how long it takes to drive a proposition from initial draft 98 to final publication, and how these delays can be split between 99 Working Group discussions, IETF reviews, IESG assessment, RFC Editor 100 delays and final reviews by the authors. 102 Just measuring production delays may be misleading. If the IETF 103 simply rubber-stamped draft proposals and published them, the delays 104 would be short but the quality and impact might suffer. We hope that 105 most of the RFC that are published are useful, but we need a way to 106 measure that usefulness. We try to do that by measuring the number 107 of references of the published RFCs in Google Scholar, and also by 108 checking whether the protocols and technologies defined in the RFCs 109 were implemented and used on the Internet. 111 2. Methodology 113 In this exploration, we want to evaluate not just the mechanics of 114 the RFC production, but also the quality and impact of the results. 115 This evaluation of quality and impact is subjective. We start with 116 two ideas: 118 1- Use Google Scholar to assess the citation counts of published 119 documents 121 2- Ask the RFC authors whether the specifications resulted in the 122 deployment of products or services 124 When accessing Google Scholar, we search for quoted strings of the 125 form "RFC xxxx". This is an arbitrary choice, we could for example 126 have chosen to search for "RFCxxx" or a combination of the two forms. 127 We retained the simpler alternative, because we don't believe that 128 picking one or the other would introduce a significant bias. 130 Basic production mechanisms could be evaluated by processing data 131 from the IETF tracker, but subjective data requires manual assessment 132 of results, which can be time consuming. Google Scholar also 133 requires manual access because the site does not offer an open API. 134 Since our resources are limited, we will only perform this analysis 135 for a small sample of RFC, selected at random from the list of RFC 136 approved in 2018. Specifically, we will pick 20 RFC at random 137 between: 139 o RFC 8307, published in January 2018, and 141 o RFC 8511, published December 2018. 143 In order to avoid injecting personal bias in the random selecton, we 144 use a random selection process similar to the Nomination Committee 145 selection process defined in [RFC3797]. The process is seeded with 146 the text string "vanitas vanitatum et omnia vanitas", and the results 147 are: 149 Picking 20 numbers between 8307 and 8511, 150 using MD5(vanitas vanitatum et omnia vanitas) 151 Rank 1: 8411 -- md5=daba041224a879199b698748808f917d 152 Rank 2: 8456 -- md5=f5570484d91ada6a672edbdca61d808c 153 Rank 3: 8446 -- md5=8340e918bb8faf69d197f79c9a58d7b8 154 Rank 4: 8355 -- md5=19474df74efd9917cf3fe8acce2ac374 155 Rank 5: 8441 -- md5=5acce2b730f3c24a4a91a5fc1921d1cd 156 Rank 6: 8324 -- md5=411c11a1cf4c292f83458865599c6921 157 Rank 7: 8377 -- md5=ac16a89192c0f0727febd35aacbc1f24 158 Rank 8: 8498 -- md5=bba44f2ba1ab240a1265a82ab71f7e02 159 Rank 9: 8479 -- md5=1653606b0af95d529a473a8f85ffaea4 160 Rank 10: 8453 -- md5=0cbfe105667c5a83b027dcfa85062f98 161 Rank 11: 8429 -- md5=fa51d7738562d990926a0d199fb060b8 162 Rank 12: 8312 -- md5=96d061523b1a57343356ae7a1e498ca5 163 Rank 13: 8492 -- md5=1b72b746eb05f79af40ed2bd3faccbe8 164 Rank 14: 8378 -- md5=645833b936d36cdcc797256518d7c483 165 Rank 15: 8361 -- md5=2064622c868e410beb0d9c18d0cb522c 166 Rank 16: 8472 -- md5=ca8a823072a21df011d0ea8b96a6aa47 167 Rank 17: 8471 -- md5=01b293a7dd0793e6f3297f2a973cd7e3 168 Rank 18: 8466 -- md5=8e411babe271557fe83bcdececc1643f 169 Rank 19: 8362 -- md5=8a1ba3efd82856a12b2b35fc5237e1b7 170 Rank 20: 8468 -- md5=57ae50ee0e1e0708d356d96d116dbfe1 172 When evaluating delays and impact, we will compare the year 2018 to 173 2008 and 1998, 10 and 20 years ago. To drive this comparison, we 174 pick 20 RFC at random among those published in 2008, and another 20 175 among those published in 1998. We use the same nomcom-like 176 methodology. 178 For 2008, we picking random RFC numbers between RFC 5134 (January 179 2008) and RFC 5405 (December 2008), using the sentence "sed fugit 180 interea fugit irreparabile tempus" as a seed. We actually list here 181 21 numbers, because the random draw place RFC 5315 in 20th position, 182 but that RFC was never issued. We replace it by RFC 5301, which came 183 in 21st position. 185 Picking 21 numbers between 5134 and 5405, 186 using MD5(sed fugit interea fugit irreparabile tempus) 187 Rank 1: 5227 -- md5=61e5b3cd97fda4e75450e93df0744b6d 188 Rank 2: 5174 -- md5=eed49542394e9d392bcd756ee0f5beed 189 Rank 3: 5172 -- md5=72055ca953d6a8ee0d60a9fca0b8d738 190 Rank 4: 5354 -- md5=7a00ef15897479d1d15255159f6d8674 191 Rank 5: 5195 -- md5=54813be7bb56f48a05af8c894799b51f 192 Rank 6: 5236 -- md5=153263bbd8c0349501b75a33f3d66f6c 193 Rank 7: 5348 -- md5=b2d19aa9c1250ef2ddf169045f6e99d7 194 Rank 8: 5281 -- md5=1c3e61643d46d1da4ba16f0fd7a0aff5 195 Rank 9: 5186 -- md5=5e87001b830183b1a9427d479a7b0e42 196 Rank 10: 5326 -- md5=024347839f83d8082549c08bdfa1b43e 197 Rank 11: 5277 -- md5=049a83016ab08552841c59400480cd9d 198 Rank 12: 5373 -- md5=a1ce374aaebdacca2e7d6eeff039d970 199 Rank 13: 5404 -- md5=fb0d6b582a27ce34175e39de33598556 200 Rank 14: 5329 -- md5=df043ef1f9d42ba12a03a84434d26ead 201 Rank 15: 5283 -- md5=c40d3f966bc7800d6508d3d82df2371d 202 Rank 16: 5358 -- md5=6fea5bdb26b19e68befd409a09cb335d 203 Rank 17: 5142 -- md5=a8844b73287781762e6548fc6f533508 204 Rank 18: 5271 -- md5=c19eb02984265ecfe4ca076f2c160cfa 205 Rank 19: 5349 -- md5=33d756f81bf6e40ff344cf6ccaf29f13 206 Rank 20: 5315 -- md5=d4c30875f88328d72c9f78def2d1dde5 207 Rank 21: 5301 -- md5=3356419e5560901f0d31309b39d14a80 209 For 1998 we picking random RFC numbers between RFC 2257 (January 210 1998) and RFC 2479 (December 1998), using the sentence "pulvis et 211 umbra sumus" as a seed. 213 Picking 20 numbers between 2257 and 2479, 214 using MD5(pulvis et umbra sumus) 215 Rank 1: 2431 -- md5=6eba444bb3349339fcde7e2be726f0f0 216 Rank 2: 2381 -- md5=0ce53f69f1a49a49309054af1e9a1d42 217 Rank 3: 2387 -- md5=53726e77ffeed903d244155d28e30f10 218 Rank 4: 2348 -- md5=69fcab2d2085555bac9eb0e0f4346523 219 Rank 5: 2391 -- md5=93358a26a7fce9fa9d61a31cdfdb87b7 220 Rank 6: 2267 -- md5=652dcab91bd9d5c58f98bdab21b23d80 221 Rank 7: 2312 -- md5=2505b0bed4af0a00ee663891c6f294d8 222 Rank 8: 2448 -- md5=6ede4b0dfa935f6ca656a5104b8dc5d0 223 Rank 9: 2374 -- md5=5312bfe5a9ca45563eb0bf35510d8daa 224 Rank 10: 2398 -- md5=7748a6deec3860898678f992b0b22792 225 Rank 11: 2283 -- md5=d73ff67466eb42d971a0e134b9284f83 226 Rank 12: 2382 -- md5=de1f667ac3e4c64aa529872e08823dff 227 Rank 13: 2289 -- md5=37773c2569dc25fdd0ab400ea401d5c7 228 Rank 14: 2282 -- md5=6b3df671a0a0becf9e42d203b59acd08 229 Rank 15: 2404 -- md5=e1b6819e5355924f456eb79f93beb8fd 230 Rank 16: 2449 -- md5=4f057df7c226efea773e7013c9c62081 231 Rank 17: 2317 -- md5=40eee3b536abe4afdb8834a0650c0a04 232 Rank 18: 2394 -- md5=044f09c53fc9fd1c50fe6bd0c39318e1 233 Rank 19: 2297 -- md5=78ee7e128436c969c80900fef80c075c 234 Rank 20: 2323 -- md5=ea6935bbda5f6d97756d3df5c3e2fdfb 236 3. Analysis of 20 selected RFC 238 We review each of the RFC listed in (#methodology) for the year 2018, 239 trying both to answer the known questions and to gather insight for 240 further analyzes. In many cases, the analysis of the data is 241 complemented by direct feedback from the RFC authors. 243 3.1. 8411 245 IANA Registration for the Cryptographic Algorithm Object Identifier 246 Range [RFC8411]: 248 Informational, 5 pages 249 4 drafts (personal),first May 8, 2017. Published August 2018. 250 Last call announced 2017/10/09 251 IESG evaluation starts 2017/12/28 252 Approved 2018/02/26, draft 03 253 Auth 48 2018/04/20 254 Auth 48 complete 2018/07/17 255 Published 2018/08/06 256 IANA action: create table 258 The draft underwent minor copy edit before publication. 260 The long delay in Auth48 is probaby due to clustering with [RFC8410], 261 which entered AUTH 48 on 06/05. The MISSREF tracker code was cleared 262 then. 264 2 references on Google Scholar. 266 3.2. 8456 268 Benchmarking Methodology for Software-Defined Networking (SDN) 269 Controller Performance [RFC8456]: 271 Informational, 64 pages 272 2 personal drafts, 9 WG drafts, first in March 2015 273 Last call announced 2018-01-19 274 IESG evaluation starts 2018-02-27 275 IESG approved 2018-05-25 276 Auth 48 2018-08-31 277 Auth 48 complete 2018-10-16 278 Published 2018-10-30 280 The draft underwent very extensive copy editing, covering use of 281 articles, turn of phrases, choice of vocabulary. The changes are 282 enough to cause pagination differences. The "diff" tool marks pretty 283 much every page as changed. Some diagrams see change in protocol 284 elements like message names. 286 According to the author, the experience of producing this draft 287 mirrors a typical one in the Benchmarking Methodologies Working Group 288 (BMWG).There were multiple authors in multiple time zones, which 289 slowed down the AUTH process somewhat, although the Auth48 delay of 290 46 is only a bit longer than the average draft. 292 The RFC was part of cluster with [RFC8455]. 294 Google Scholar shows 3 references. BMWG publishes informational RFCs 295 centered around benchmarking, and the methodologies in RFC 8456 have 296 been implemented in benchmarking products. 298 3.3. 8446 300 The Transport Layer Security (TLS) Protocol Version 1.3 [RFC8446], as 301 the title indicates, defines the new version of the TLS protocol. 302 From the datatracker, we extract the following: 304 Proposed standard 305 160 pages 306 29 WG drafts first April 17, 2014. 307 Last call announced 2018/02/15 308 IESG evaluation starts 2018/03/02 309 Approved 2018/03/21, draft 28 310 Auth 48 2018/06/14 311 Auth 48 complete 2018/08/10 312 Published 2018/08/10 314 The RFC was a major effort in the IETF. Working group members 315 developed and tested several implementations. Researchers analyzed 316 the specifications and performed formal verifications. Deployment 317 tests outlined issues that caused extra work when the specification 318 was almost ready. These complexity largely explains the time spent 319 in the working group. 321 Comparing the final draft to the published version, we find 322 relatively light copy editing. It includes explaining acronyms on 323 first use, clarifying some definitions standardizing punctiation and 324 capitalization, and spelling out some numbers in text. This 325 generally fall in the category of "style", although some of the 326 clarifications go into message definitions. However, that simple 327 analysis does not explain why the Auth48 phase took almost two 328 months. 330 This document's Auth48 process was part of the "Github experiment", 331 which tried to use github pull requests to track the AUTH48 changes 332 and review comments. The RPC staff had to learn using Github for 333 that process, and this required more work than the usual RFC. Author 334 and AD thoroughly reviewed each proposed edit, accepting some and 335 rejecting some. The concern there was that any change in a complex 336 specification might affect a protocol that was extensively reviewed 337 in the working group, but of course these reviews added time to the 338 Aouth48 delays. 340 The RFC has 123 references in Google Scholar. There are 21 341 implementations listed in the Wiki of the TLS 1.3 project. It has 342 been deployed on major browsers, and is already used in a large 343 fraction of TLS connections. 345 3.4. 8355 347 Resiliency Use Cases in Source Packet Routing in Networking (SPRING) 348 Networks [RFC8355] is an informational RFC. It originated from a use 349 case informational draft that was mostly used for the BOF creating 350 the WG, and then to drive initial work/evolutions from the WG. 352 Informational, 13 pages. 353 2 personal drafts (personal), first January 31, 2014. 13 WG drafts. 354 Last call announced 2017-04-20 355 IESG evaluation starts 2017-05-04, draft 09 356 Approved 2017-12-19, draft 12 357 Auth 48 2018-03-12 358 Auth 48 complete 2018-03-27 359 Published 2018-03-28 361 Minor set of copy edit, mostly for style. 363 9 references on Google Scholar. No implementation of the RFC itself, 364 but the technology behind it such as Segment Routing (architecture 365 RFC 8402, TI-LFA draft-ietf-rtgwg-segment-routing-ti-lfa) is widely 366 implemented and deployment is ongoing. 368 3.5. 8441 370 Bootstrapping WebSockets with HTTP/2 [RFC8441] 372 Proposed standard, 8 pages. Updates RFC 6455. 373 3 personal drafts (personal), first 10/15/2017. 8 WG drafts. 374 Last call announced 2018-05-07, draft 05 375 IESG evaluation starts 2018-05-29, draft 06 376 Approved 2018-06-07, draft 07 377 Auth 48 2018-08-13 378 Auth 48 complete 2018-09-15 379 Published 2018-09-21 380 IANA Action: table entries 382 This RFC defines the support of WebSockets in HTTP/2, which is 383 different from the mechanism defined for HTTP/1.1 in [RFC6455]. The 384 process was relatively straightforward, involving the usual type of 385 discussions, some on details and some on important points. 387 Comparing final draft and published RFC shows a minor set of copy 388 edit, mostly for style. However, the author recalls a painful 389 process. The RFC includes many charts and graphs that were very 390 difficult to format correctly in the author's production process that 391 involve conversions from markdown to XML, and then from XML to text. 392 The author had to get substantial help from the RFC editor. 394 No references on Google Scholar. (RFC 6455 had over 1000 results) 396 There are several implementations, including Firefox and Chrome, 397 making RFC 8441 a very successful standard. 399 3.6. 8324 401 DNS Privacy, Authorization, Special Uses, Encoding, Characters, 402 Matching, and Root Structure: Time for Another Look? [RFC8324]. 403 This is an opinion piece on DNS development, published on the 404 Independent Stream. 406 Informational, 29 pages. Independent stream. 407 5 personal drafts (personal), first June 2, 2017. 408 ISE review started 2017-07-10, draft 03 409 IETF conflict review and IESG review started 2017-10-29 410 Approved 2017-12-18, draft 04 411 Auth 48 2018-01-29, draft 05 412 Auth 48 complete 2018-02-26 413 Published 2018-02-27 415 This RFC took only 9 months from first draft to publication, which is 416 the shortest in the 2018 sample set. In part, this is because the 417 text was privately circulated and reviewed before the first draft was 418 published. The nature of the document is another reason for the 419 short delay. It is an opinion piece, and does not require the same 420 type of consensus building and reviews than a protocol specification. 422 Comparing the final draft and the published version shows only minor 423 copy edit, mostly for style. According to the author, because this 424 is because he knows how to write in RFC Style with the result that 425 his documents often need a minimum of editing. He also makes sure 426 that the document on which the Production Center starts working 427 already has changes discussed and approved during Last Call and IESG 428 review incorporated rather than expecting the Production Center to 429 operate off of notes about changed to be made. 431 2 references on Google Scholar. 433 3.7. 8377 435 Transparent Interconnection of Lots of Links (TRILL): Multi-Topology 436 [RFC8377] 438 Proposed standard, 20 pages. Updates RFC 6325, 7177. 439 3 personal drafts (personal), first September 3, 2013. 7 WG drafts. 440 Last call announced 2018-02-19, draft 05 441 IESG evaluation starts 2018-03-02, draft 06 442 Approved 2018-03-12, draft 05 443 Auth 48 2018-04-20, draft 06 444 Auth 48 complete 2018-07-31 445 Published 2018-07-31 446 IANA Table, table entries 447 Minor set of copy edit, mostly for style, also clarity. 449 1 reference on Google Scholar. 451 3.8. 8498 453 A P-Served-User Header Field Parameter for an Originating Call 454 Diversion (CDIV) Session Case in the Session Initiation Protocol 455 (SIP) [RFC8498]. 457 Informational, 15 pages. 458 5 personal drafts (personal), first March 21, 2016. 9 WG drafts. 459 Last call announced 2018-10-12, draft 05 460 IESG evaluation starts 2018-11-28, draft 07 461 Approved 2018-12-10, draft 08 462 Auth 48 2019-01-28 463 Auth 48 complete 2019-02-13 464 Published 2019-02-15 465 IANA Action, table rows added. 467 Copy edit for style, but also clarification of ambiguous sentences. 469 No references on Google Scholar. 471 3.9. 8479 473 Storing Validation Parameters in PKCS#8 [RFC8479] 475 Informational, 8 pages. Independent stream. 476 5 personal drafts (personal), first August 8, 2017. 477 ISE review started 2018-03-29, draft 02 478 IETF conflict review and IESG review started 2018-03-29 479 Approved 2018-08-20, draft 03 480 Auth 48 2018-09-20, draft 04 481 Auth 48 complete 2018-09-25 482 Published 2018-09-26 484 The goal of the draft was to document what the gnutls implementation 485 was using for storing provably generated RSA keys. This is a short 486 RFC that was published relatively quickly, although discussion 487 between the author, the Independent Series Editor and the IESG lasted 488 several months. In the initial conflict review, Tthe IESG asked the 489 ISE to not publish this document before IETF working groups had an 490 opportunity to pick up the work. The author met that requirement by 491 a presentation to the SECDISPATCH WG in IETF 102. Since no WG was 492 interested in pickup the work, the document progressed on the 493 Independent Stream. 495 Very minor set of copy edit, moving some references from normative to 496 informative. 498 No reference on Google Scholar. 500 The author is not aware of other implementations than gnutls relying 501 on this RFC. 503 3.10. 8453 505 Framework for Abstraction and Control of TE Networks (ACTN) [RFC8453] 507 Informational, 42 pages. 508 3 personal drafts, first June 15, 2015. 16 WG drafts. 509 Out of WG 2018-01-26, draft 11 510 Expert review requested, 2018-02-13 511 Last call announced 2018-04-16, draft 13 512 IESG evaluation starts 2018-05-16, draft 14 513 Approved 2018-06-01, draft 15 514 Auth 48 2018-08-13 515 Auth 48 complete 2018-08-20 516 Published 2018-08-20 517 IANA Action, table rows added. 519 Minor copy editing. 521 8 references on Google Scholar. 523 3.11. 8429 525 Deprecate Triple-DES (3DES) and RC4 in Kerberos [RFC8429] 527 BCP, 10 pages. 528 6 WG drafts, first 5/1/2017. 529 Last call announced 7/16/2017, draft 03 530 IESG evaluation starts 8/18/2017, draft 04 531 Approved 5/25/2018, draft 05 532 Auth 48 7/24/2018 533 Auth 48 complete 10/31/2018 534 Published 10/31/2018 535 IANA Action, table rows added. 537 This RFC recommends to deprecate two encryption algorithms that are 538 now considered obsolete and possibly broken. The document was sent 539 back to the WG after the first last call, edited, and then there was 540 a second last call. The delay from first draft to working group last 541 call was relatively short, but the number may be misleading. The 542 initial draft was a replacement of a similar draft in the KITTEN 543 working group, which stagnated for some time before the CURDLE 544 working group took up the work. The deprecation of RC4 was somewhat 545 contentious, but the WG had already debated this prior to the 546 production of this draft, and the draft was not delayed by this 547 debate. 549 Most of the 280 days between IETF LC and IESG approval was because 550 the IESG had to talk about whether this document should obsolete or 551 move to historic RFC 4757, and no one was really actively pushing 552 that discussion for a while. 554 The 99 days in AUTH48 are mostly because one of the authors was a 555 sitting AD, and those duties ended up taking precedence over 556 reviewing this document. 558 Minor copy editing, for style. 560 1 reference on Google Scholar. 562 The implementation of the draft would be the actual removal of 563 support for 3DES and RC4 in major implementations. This is 564 happening, but very slowly. 566 3.12. 8312 568 CUBIC for Fast Long-Distance Networks [RFC8312] 570 Informational, 18 pages. 571 2 personal drafts, first 9/1/2014. 8 WG drafts 572 Last call announced 9/18/2017, draft 06 573 IESG evaluation starts 2017-11-14 574 Approved 2017-10-04, draft 07 575 Auth 48 2018-01-08 576 Auth 48 complete 2018-02-07 577 Published 2018-02-07 578 IANA Action, table rows added. 580 Minor copy editing, for style. 582 9 references on Google Scholar. 584 The TCP congestion control algorithm Cubic was defined first in 2005, 585 was implemented in Linux soon after, and was implemented in major 586 OSes after that. After some debates from 2015 to 2015, the TCPM 587 working group adopted the draft, with a goal of documenting Cubic in 588 the RFc series. According to the authors, this was not a high 589 priority effort, as Cubic was already implemented in multiple OSes 590 and documented in research papers. At some point, only one of the 591 authors was actively working on the draft. Ths may explain why 592 another two years was spent progressing the draft after adoption by 593 the WG. 595 The RFC publication may or may not have triggered further 596 implementations. On the other hand, several OSes picked up bug fixes 597 from the draft and the RFC. 599 3.13. 8492 601 Secure Password Ciphersuites for Transport Layer Security (TLS) 602 [RFC8492] 604 Informational, 40 pages. (Independent Stream) 605 10 personal drafts, first 9/2/2012. 8 WG drafts 606 ISE review started 2017-05-10, draft 01 607 IETF conflict review and IESG review started 2017-09-04 608 Approved 2017-10-29, draft 04 609 Auth 48 10/19/2018, draft 05 610 Auth 48 complete 2/19/2019 611 Published 2/21/2019 612 IANA Action, table rows added. 614 This RFC has a complex history. The first individual draft was 615 submitted to the TLS working group on September 7, 2012. It 616 progressed there, and was adopted by the WG after 3 revisions. There 617 were then 8 revisions in the TLS WG, until the WG decided to not 618 progress it. The draft was parked in 2013 by the WG chairs after 619 failing to get consensus in WG last call. The AD finally pulled the 620 plug in 2016, and the draft was then resubmitted to the ISE. 622 At that point, the author was busy and was treating this RFC with a 623 low priority because, in his words, it would not be a "real RFC". 624 There were problems with the draft that only came up late. In 625 particular, it had to wait for a change in registry policy that only 626 came about with the publication of TLS 1.3, which caused the draft to 627 only be published after RFC 8446, and also required adding references 628 to TLS 1.3. The author also got a very late comment while in AUTH48 629 that caused some rewrite. Finally, there was some IANA issue with 630 the extension registry where a similar extension was added by someone 631 else. The draft was changed to just use it. 633 Changes in AUTH48 include added reference to TLS 1.3, copy-editing 634 for style, some added requirements, added paragraphs, and changes in 635 algorithms specification. 637 2 references on Google Scholar. Only the author implemented the 638 specification. 640 3.14. 8378 642 Signal-Free Locator/ID Separation Protocol (LISP) Multicast [RFC8378] 643 is an experimental RFC, defining how to implement Multicast in the 644 LISP architecture. 646 Experimental, 21 pages. 647 5 personal drafts, first 2/28/2014. 10 WG drafts 648 Last call announced 2018-02-13, draft 07 649 IESG evaluation starts 2018-02-28, draft 08 650 Approved 2018-03-12, draft 09 651 Auth 48 2018-04-23 652 Auth 48 complete 2018-05-02 653 Published 2018-05-02 655 Preparing the RFC took more than 4 years. According to the authors, 656 they were not aggressive pushing it and just let the working group 657 process decide to pace it. They also did implementations during that 658 time. 660 Minor copy editing, for style. 662 1 reference on Google Scholar. The RFC was implemented by 663 lispers.net and cisco, and was used in doing IPv6 multicast over IPv4 664 unicast/multicast at the Olympics in PyeungChang. The plan is to 665 work on a proposedstandard once the experiment concludes. 667 3.15. 8361 669 Transparent Interconnection of Lots of Links (TRILL): Centralized 670 Replication for Active-Active Broadcast, Unknown Unicast, and 671 Multicast (BUM) Traffic [RFC8361] 673 Proposed Standard, 17 pages. 674 3 personal drafts, first 11/12/2013. 14 WG drafts 675 Last call announced 2017-11-28, draft 10 676 IESG evaluation starts 2017-12-18, draft 11 677 Approved 2018-01-29, draft 13 678 Auth 48 2018-09-17 679 Auth 48 complete 4/9/2018 680 Published 2018-10-08 682 According to the authors, the long delays in producing this RFC was 683 due to a slow uptake of the technology in the industry. 685 Minor copy editing, for style. 687 1 reference on Google Scholar. 689 There was at least 1 partial implementation. 691 3.16. 8472 693 Transport Layer Security (TLS) Extension for Token Binding Protocol 694 Negotiation [RFC8472] 696 Proposed Standard, 8 pages. 697 1 personal drafts, first 5/29/2015. 15 WG drafts 698 Last call announced 2017-11-13, draft 10 699 IESG evaluation starts 2018-03-19 700 Approved 2018-07-20, draft 14 701 Auth 48 2018-09-17 702 Auth 48 complete 2018-09-25 703 Published 2018-10-08 705 This is a pretty simple document, but it took over 3 years from 706 individual draft to RFC. According to the authors,the biggest 707 setbacks occurred at the start: it took a while to find a home for 708 this draft. It was presented in the TLS WG (because it's a TLS 709 extension) and UTA WG (because it has to do with applications using 710 TLS). Then the ADs determined that a new WG was needed, so the 711 authors had to work through the WG creation process, including 712 running a BOF. 714 Minor copy editing, for style, with the addition of a reference to 715 TLS 1.3. 717 5 references on Google Scholar. 719 Perhaps partially due to the delays, some of the implementers lost 720 interest in supporting this RFC. 722 3.17. 8466 724 A YANG Data Model for Layer 2 Virtual Private Network (L2VPN) Service 725 Delivery [RFC8466] 727 Proposed Standard, 158 pages. 728 5 personal drafts, first 9/1/2016. 11 WG drafts 729 Last call announced 2018-02-21, draft 07 730 IESG evaluation starts 2018-03-14, draft 08 731 Approved 2018-06-25, draft 10 732 Auth 48 2018-09-17 733 Auth 48 complete 2018-10-09 734 Published 2018-10-12 735 Copy editing for style and clarity, with also corrections to the yang 736 model. 738 2 references on Google Scholar. 740 3.18. 8362 742 OSPFv3 Link State Advertisement (LSA) Extensibility [RFC8362] is a 743 major extension to the OSPF protocol. It makes OSPFv3 fully 744 extensible. 746 Proposed Standard, 33 pages. 747 4 personal drafts, first February 17, 2013. 24 WG drafts 748 Last call announced 2017-12-19, draft 19 749 IESG evaluation starts 2018-01-18, draft 20 750 Approved 2018-01-29, draft 23 751 Auth 48 2018-03-19 752 Auth 48 complete 2018-03-30 753 Published 2018-04-03 755 The specification was first submitted as a personal draft in the IPv6 756 WG, then moved to the OSPF WG. The long delay of producing this RFC 757 is due to the complexity of the problem, and the need to wait for 758 implementations. It is a very important change to OSPF that makes 759 OSPFv3 fully extensible. Since it was a non-backward compatible 760 change, the developers started out with some very complex migration 761 scenarios but ended up with either legacy or extended OSPFv3 LSAs 762 within an OSPFv3 routing domain. The initial attempts to have a 763 hybrid mode of operation with both legacy and extended LSAs also 764 delayed implementation due to the complexity. 766 Copy editing for style and clarity. 768 7 references on Google Scholar. It either was or will be implemented 769 by all the router vendors. 771 3.19. 8468 773 IPv4, IPv6, and IPv4-IPv6 Coexistence: Updates for the IP Performance 774 Metrics (IPPM) Framework [rfc8468]. 776 Informational, 15 pages. 777 3 personal drafts, first August 6, 2015. 7 WG drafts 778 Last call announced 2018-04-11, draft 04 779 IESG evaluation starts 2018-05-24, draft 05 780 Approved 2018-07-10, draft 06 781 Auth 48 2018-09-13 782 Auth 48 complete 2018-11-05 783 Published 2018-11-14 785 RFC8468 was somehow special in that there was not a technical reason/ 786 interest that triggered it, but rather a formal requirement. While 787 writing RFC7312 the IP Performance Metrics working group (IPPM) 788 realized that RFC 2330, the IP Performance Metrics Framework 789 supported IPv4 only and explicitly excluded support for IPv6. 790 Nevertheless, people used the metrics that were defined on top of RFC 791 2330 (and, therefore, IPv4 only) for IPv6, too. Although the IPPM WG 792 agreed that the work was needed, the interest of IPPM attendees in 793 progressing (and reading/reviewing) the IPv6 draft was limited. 794 Resolving the IPv6 technical part was straight-forward, but 795 subsequently some people asked for a broader scope (topics like 796 header compression, 6lo, etc.) and it took some time to figure out 797 and later on convince people that these topics are out of scope. The 798 group also had to resolve contentious topics, for example how to 799 measure the processing of IPv6 extension headers, which is sometimes 800 non-standard. 802 The Auth48 delay for this draft was longer than average. According 803 to the authors, the main reasons include: 805 o Work-load and travel caused by busy-work-periods of all co-authors 807 o Time zone difference between co-authors and editor (at least US, 808 Europe, India, not considering travel) 810 o Editor proposing and committing some unacceptable modifications 811 that needed to be reverted 813 o Lengthy discussions on a new document title (required high effort 814 and took a long time, in particular reaching consensus between co- 815 authors and editor was time-consuming and involved the AD) 817 o Editor correctly identifying some nits (obsoleted personal 818 websites of co-authors) and co-authors attempting to fix them. 820 The differences between the final draft and the publish RFC show copy 821 editing for style and clarity, but do not account for the back and 822 forth between authors and editors mentioned by the authors. 824 2 references on Google Scholar. In contrast, RFC 2330 has more than 825 3000 references, including over 70 that are more recent than RFC 826 8468. The authors believe that many of these references include IPv6 827 work that should formally reference RFC 8468 in addition to RFC 2330, 828 but do not. 830 4. Observations 832 We examine the 20 RFC in the sample, measuring various 833 characteristics such as delay and citation counts, in an attempt to 834 identify patterns in the IETF processes. 836 4.1. Publication delays 838 We look at the distribution of delays between the submission of the 839 first draft and the publication of the RFC. We break out the total 840 delay in three components: 842 o The working group delay, from the first draft to the start of the 843 IETF last call; 845 o The IETF delay, which lasts from the beginning of the IETF last 846 call to the approval by the IESG, including the reviews by various 847 directorates; 849 o The RFC production, from approval by the IESG to publication, 850 including the Auth48 reviews. 852 For submissions to the independent stream, we don't have a working 853 group. We consider instead the progression of the individual draft 854 until the adoption by the ISE as the equivalent of the "working 855 group" period, and the delay from adoption by the ISE until 856 submission to the RFC Editor as the equivalent of the IETF delay. 857 The following table shows the delays for the 20 RFC in the sample: 859 +------+---------+-------+---------+------+------+------+ 860 | RFC | Status | Pages | Overall | WG | IETF | Edit | 861 +------+---------+-------+---------+------+------+------+ 862 | 8411 | Info | 5 | 455 | 154 | 140 | 161 | 863 | | | | | | | | 864 | 8456 | Info | 64 | 1107 | 823 | 126 | 158 | 865 | | | | | | | | 866 | 8446 | PS | 160 | 1576 | 1400 | 34 | 142 | 867 | | | | | | | | 868 | 8355 | Info | 13 | 1517 | 1175 | 243 | 99 | 869 | | | | | | | | 870 | 8441 | PS | 8 | 341 | 204 | 31 | 106 | 871 | | | | | | | | 872 | 8324 | ISE | 29 | 270 | 38 | 161 | 71 | 873 | | | | | | | | 874 | 8377 | PS | 8 | 1792 | 1630 | 21 | 141 | 875 | | | | | | | | 876 | 8498 | Info | 15 | 1061 | 935 | 59 | 67 | 877 | | | | | | | | 878 | 8479 | ISE | 8 | 414 | 233 | 144 | 37 | 879 | | | | | | | | 880 | 8453 | Info | 42 | 1162 | 1036 | 46 | 80 | 881 | | | | | | | | 882 | 8429 | BCP | 10 | 548 | 76 | 313 | 159 | 883 | | | | | | | | 884 | 8312 | Info | 18 | 1255 | 1113 | 16 | 126 | 885 | | | | | | | | 886 | 8492 | ISE | 40 | 2358 | 1706 | 172 | 480 | 887 | | | | | | | | 888 | 8378 | Exp | 21 | 1524 | 1446 | 27 | 51 | 889 | | | | | | | | 890 | 8361 | PS | 17 | 1612 | 1477 | 62 | 73 | 891 | | | | | | | | 892 | 8472 | PS | 8 | 1228 | 899 | 249 | 80 | 893 | | | | | | | | 894 | 8466 | PS | 158 | 771 | 538 | 124 | 109 | 895 | | | | | | | | 896 | 8362 | PS | 33 | 1871 | 1766 | 41 | 64 | 897 | | | | | | | | 898 | 8468 | Info | 15 | 1196 | 979 | 90 | 127 | 899 | | | | | | | | 900 | | average | 35 | 1161 | 928 | 110 | 123 | 901 +------+---------+-------+---------+------+------+------+ 903 The average delay from first draft to publication is about 3 years, 904 but this varies widely. Excluding the independent stream 905 submissions, the average delay from start to finish is 3 years and 3 906 months, of which on average 2 years and 8 months are spent getting 907 consensus in the working group. 909 The longest delay is found for [RFC8492], 6.5 years from start to 910 finish. This is however a very special case, a draft that was 911 prepared for the TLS working group and failed to reach consensus. 912 After that, it was resubmitted to the ISE, and incurred atypical 913 production delays. 915 On average, we see that 80% of the delay is incurred in WG 916 processing, 10% in IETF review, and 10% for edition and publication. 917 We can compare these delays to those observed 10 years ago and 20 918 years ago: 920 +------------+--------+-------+-------+ 921 | RFC (2008) | Status | Pages | Delay | 922 +------------+--------+-------+-------+ 923 | 5326 | Exp | 54 | 1584 | 924 | | | | | 925 | 5348 | PS | 58 | 823 | 926 | | | | | 927 | 5281 | Info | 51 | 1308 | 928 | | | | | 929 | 5354 | Exp | 23 | 2315 | 930 | | | | | 931 | 5227 | PS | 21 | 2434 | 932 | | | | | 933 | 5329 | PS | 12 | 1980 | 934 | | | | | 935 | 5277 | PS | 35 | 912 | 936 | | | | | 937 | 5236 | ISE | 26 | 1947 | 938 | | | | | 939 | 5358 | BCP | 7 | 884 | 940 | | | | | 941 | 5271 | Info | 22 | 1066 | 942 | | | | | 943 | 5195 | PS | 10 | 974 | 944 | | | | | 945 | 5283 | PS | 12 | 1096 | 946 | | | | | 947 | 5186 | Info | 6 | 2253 | 948 | | | | | 949 | 5142 | PS | 13 | 1005 | 950 | | | | | 951 | 5373 | PS | 24 | 1249 | 952 | | | | | 953 | 5404 | PS | 27 | 214 | 954 | | | | | 955 | 5172 | PS | 7 | 305 | 956 | | | | | 957 | 5349 | Info | 10 | 1096 | 958 | | | | | 959 | 5301 | PS | 6 | 396 | 960 | | | | | 961 | 5174 | Info | 8 | 427 | 962 +------------+--------+-------+-------+ 964 +------------+--------+-------+---------+ 965 | RFC (1998) | Status | Pages | Delay | 966 +------------+--------+-------+---------+ 967 | 2289 | PS | 25 | 396 | 968 | | | | | 969 | 2267 | Info | 10 | unknown | 970 | | | | | 971 | 2317 | BCP | 10 | 485 | 972 | | | | | 973 | 2404 | PS | 7 | 488 | 974 | | | | | 975 | 2374 | PS | 12 | 289 | 976 | | | | | 977 | 2449 | PS | 19 | 273 | 978 | | | | | 979 | 2283 | PS | 9 | 153 | 980 | | | | | 981 | 2394 | Info | 6 | 365 | 982 | | | | | 983 | 2348 | DS | 5 | 699 | 984 | | | | | 985 | 2382 | Info | 30 | 396 | 986 | | | | | 987 | 2297 | ISE | 109 | 28 | 988 | | | | | 989 | 2381 | PS | 43 | 699 | 990 | | | | | 991 | 2312 | Info | 20 | 365 | 992 | | | | | 993 | 2387 | PS | 10 | 122 | 994 | | | | | 995 | 2398 | Info | 15 | 396 | 996 | | | | | 997 | 2391 | PS | 10 | 122 | 998 | | | | | 999 | 2431 | PS | 10 | 457 | 1000 | | | | | 1001 | 2282 | Info | 14 | 215 | 1002 | | | | | 1003 | 2323 | ISE | 5 | unknown | 1004 | | | | | 1005 | 2448 | ISE | 7 | 92 | 1006 +------------+--------+-------+---------+ 1008 We can compare the median delay, and the delays observed by the 1009 fastest and slowest quartiles in the three years: 1011 +------+-------------+--------+-------------+ 1012 | Year | Fastest 25% | Median | Slowest 25% | 1013 +------+-------------+--------+-------------+ 1014 | 2018 | 604 | 1179 | 1522 | 1015 | | | | | 1016 | 2008 | 869 | 1081 | 1675 | 1017 | | | | | 1018 | 1998 | 169 | 365 | 442 | 1019 +------+-------------+--------+-------------+ 1021 The IETF takes three to four times more times to produce an RFC in 1022 2018 than it did in 1998, but about the sametime as it did in 2008. 1024 The increased delay does not mean increased work per RFC. The number 1025 of RFC published per year remained between 200 and 300 all those 1026 years, and the number of participants is not greater now than in 1027 1998. If we estimated the "level of attention" by dividing the 1028 number of participants by the number of RFC produced, we would see a 1029 number that remains stable. People are probably not working much 1030 more on each RFC now than they were 20 years ago, but the same amount 1031 of work is stretched over a much longer period. 1033 4.2. Preparation and Publication delays 1035 The preparation and publication delays include three components: 1037 o the delay from submission to the RFC Editor to beginning of 1038 Auth48, during which the document is prepared; 1040 o the AUTH48 delay, during which authors review and eventually 1041 approve the changes proposed by the editors; 1043 o the publication delay, from final agreement by authors and editors 1044 to actual publication. 1046 The breakdown of the publication delays for each RFC is shown in the 1047 following table. 1049 +-------+---------+-------+--------+---------+--------+-------------+ 1050 | RFC | Status | Pages | RFC | Auth 48 | RFC | Edit(total) | 1051 | | | | edit | | Pub | | 1052 +-------+---------+-------+--------+---------+--------+-------------+ 1053 | 8411 | Info | 5 | 53 | 88 | 20 | 161 | 1054 | | | | | | | | 1055 | 8456 | Info | 64 | 98 | 46 | 14 | 158 | 1056 | | | | | | | | 1057 | 8446 | PS | 160 | 85 | 57 | 0 | 142 | 1058 | | | | | | | | 1059 | 8355 | Info | 13 | 83 | 15 | 1 | 99 | 1060 | | | | | | | | 1061 | 8441 | PS | 8 | 67 | 33 | 6 | 106 | 1062 | | | | | | | | 1063 | 8324 | ISE | 29 | 42 | 28 | 1 | 71 | 1064 | | | | | | | | 1065 | 8377 | PS | 8 | 39 | 102 | 0 | 141 | 1066 | | | | | | | | 1067 | 8498 | Info | 15 | 49 | 16 | 2 | 67 | 1068 | | | | | | | | 1069 | 8479 | ISE | 8 | 31 | 5 | 1 | 37 | 1070 | | | | | | | | 1071 | 8453 | Info | 42 | 73 | 7 | 0 | 80 | 1072 | | | | | | | | 1073 | 8429 | BCP | 10 | 60 | 99 | 0 | 159 | 1074 | | | | | | | | 1075 | 8312 | Info | 18 | 96 | 30 | 0 | 126 | 1076 | | | | | | | | 1077 | 8492 | ISE | 40 | 355 | 123 | 2 | 480 | 1078 | | | | | | | | 1079 | 8378 | Exp | 21 | 42 | 9 | 0 | 51 | 1080 | | | | | | | | 1081 | 8361 | PS | 17 | 39 | 31 | 3 | 73 | 1082 | | | | | | | | 1083 | 8472 | PS | 8 | 59 | 8 | 13 | 80 | 1084 | | | | | | | | 1085 | 8466 | PS | 158 | 84 | 22 | 3 | 109 | 1086 | | | | | | | | 1087 | 8362 | PS | 33 | 49 | 11 | 4 | 64 | 1088 | | | | | | | | 1089 | 8468 | Info | 15 | 65 | 53 | 9 | 127 | 1090 | | | | | | | | 1091 | | Average | | 77.3 | 41.2 | 4.2 | 122.7 | 1092 | | | | | | | | 1093 | -8492 | Average | | 62 | 37 | 4 | 103 | 1094 +-------+---------+-------+--------+---------+--------+-------------+ 1095 On average, the total delay appears to be a bit more than four month, 1096 but the average is skewed by the extreme values encountered for 1097 [RFC8492]. If we exclude that RFC from the computations, the average 1098 delay drops to a just a bit more than 3 months: about 2 months for 1099 the preparation, a bit more than one month for the Auth48 phase, and 1100 4 days for the publishing. 1102 Of course, these delays vary from RFC to RFC. To try explain the 1103 causes of the delay, we compute the correlation factor between the 1104 observed delays and 4 plausible explanation factors: 1106 o The number of pages in the document, 1108 o The amount of copy edit, as discussed in (#copy-editing), 1110 o Whether or not an IANA action was required. 1112 We find the following values: 1114 +-------------+----------+---------+-------------+ 1115 | Correlation | RFC edit | Auth 48 | Edit(total) | 1116 +-------------+----------+---------+-------------+ 1117 | Nb pages | 0.50 | -0.04 | 0.21 | 1118 | | | | | 1119 | Copy-Edit | 0.42 | 0.24 | 0.45 | 1120 | | | | | 1121 | IANA | -0.13 | 0.26 | 0.15 | 1122 +-------------+----------+---------+-------------+ 1124 None of these indicate strong correlations. The greater number of 1125 pages will tend to increase the preparation delay, but it does not 1126 appear to impact the Auth48 delay at all. The amount of copy editing 1127 also tend to increase the the preparation delay somewhat and the 1128 Auth48 delay a little. The presence or absence of IANA action has 1129 very ittle correlation with the delays. 1131 We also observe that the Auth48 delay varies much more than the 1132 preparation delay, with a standard deviation of 20 days for Auth48 1133 versus 10 days for the preparation delay. In theory, Auth48 is just 1134 a final verification: the authors receive the document prepared by 1135 the RFC production center, and just have to give their approval, or 1136 maybe request a last minute correction. The name indicates that this 1137 is expected to last just two days, but in average it lasts more than 1138 a month. 1140 We tested a variety of hypotheses that might explain the duration of 1141 AUTH48 by computing the correlation coefficients between various 1142 properties of the RFC and the production delays. The results are 1143 listed in the following table: 1145 +-------------+----------------+---------+-----------------+ 1146 | Correlation | RFC production | Auth 48 | RFC Edit(total) | 1147 +-------------+----------------+---------+-----------------+ 1148 | Nb drafts | 0.19 | -0.30 | -0.17 | 1149 | | | | | 1150 | Nb Authors | 0.40 | -0.04 | 0.16 | 1151 | | | | | 1152 | WG delay | 0.03 | -0.16 | -0.15 | 1153 +-------------+----------------+---------+-----------------+ 1155 The results show that there is no simple answer. The number of 1156 pages, the required amount of copy editing and to a very small extent 1157 the number of drafts can help predict the production delay, but there 1158 is no obvious predictor for the Auth 48 delay. In particular, there 1159 is no numerical evidence that the number of authors influences the 1160 Auth48 delay, or that authors who have spent a long time working on 1161 the document in the working group somehow spend even longer to answer 1162 questions during Auth48 - if anything, the numerical results point in 1163 the opposite direction. 1165 After asking the authors of the RFC in the sample why the AUTH48 1166 phase took a long time, and we got three explanations: 1168 1- Some RFC have multiple authors in multiple time zones. This slows 1169 down the coordination required for approving changes. 1171 2- Some authors found some of the proposed changes unnecessary or 1172 undesirable, and asked that they be reversed. This required long 1173 exchanges between authors and editors. 1175 3- Some authors were not giving high priority to AUTH48 responses. 1177 As mentioned above, we were not able to verify these hypotheses by 1178 looking at the data. 1180 4.3. Copy editing 1182 We can assess the amount of copy editing applied to each published 1183 RFC by comparing the text of the draft approved for publication and 1184 the text of the RFC. We do expect differences in the "boilerplate" 1185 and in the IANA section, but we will also see differences due to copy 1186 editing. Assessing the amount of copy editing is subjective, and we 1187 do it using a scale of 1 to 4: 1189 1- Minor editing 1190 2- Editing for style, such as capitalization, hyphens, that versus 1191 which, and expending all acronyms at least one. 1193 3- Editing for clarity in addition to style, such as rewriting 1194 ambiguous sentences and clarifying use of internal references. For 1195 Yang models, that may include model corrections suggested by the 1196 verifier. 1198 4- Extensive editing. 1200 The following table shows that with about half of the RFC required 1201 editing for style, and the other half at least some editing for 1202 clarity. 1204 +------+--------+-----------+ 1205 | RFC | Status | Copy Edit | 1206 +------+--------+-----------+ 1207 | 8411 | Info | 2 | 1208 | | | | 1209 | 8456 | Info | 4 | 1210 | | | | 1211 | 8446 | PS | 3 | 1212 | | | | 1213 | 8355 | Info | 2 | 1214 | | | | 1215 | 8441 | PS | 2 | 1216 | | | | 1217 | 8324 | ISE | 2 | 1218 | | | | 1219 | 8377 | PS | 3 | 1220 | | | | 1221 | 8498 | Info | 3 | 1222 | | | | 1223 | 8479 | ISE | 1 | 1224 | | | | 1225 | 8453 | Info | 2 | 1226 | | | | 1227 | 8429 | BCP | 2 | 1228 | | | | 1229 | 8312 | Info | 2 | 1230 | | | | 1231 | 8492 | ISE | 3 | 1232 | | | | 1233 | 8378 | Exp | 2 | 1234 | | | | 1235 | 8361 | PS | 2 | 1236 | | | | 1237 | 8472 | PS | 2 | 1238 | | | | 1239 | 8466 | PS | 3 | 1240 | | | | 1241 | 8362 | PS | 3 | 1242 | | | | 1243 | 8468 | Info | 3 | 1244 +------+--------+-----------+ 1246 This method of assessment does not take into account the number of 1247 changes proposed by the editors and eventually rejected by the 1248 authors, since these changes are not present in either the final 1249 draft or the published RFC. It might be possible to get an 1250 evaluation of these "phantom changes" from the RFC Production Center. 1252 4.4. Independent Series 1254 Out of 20 randomly selected RFC, 3 were published through the 1255 "independent series". One is an independent opinion, another a 1256 description of a non-IETF protocol format, and the third was 1257 [RFC8492], which is a special case. Apart from this special case, 1258 the publication delays were significantly shorter for the Independent 1259 Stream than for the IETF stream. This seems to indicate that the 1260 Independent Series is functioning as expected. 1262 The authors of these 3 RFC are regular IETF contributors. This 1263 observation motivated a secondary analysis of all the RFC published 1264 in the "independent" stream in 2018. There are 14 such RFC: 8507, 1265 8494, 8493, 8492, 8483, 8479, 8433, 8409, 8374, 8369, 8367, 8351, 1266 8328 and 8324. (RFC 8367 and 8369 were published on 1 April 2018.) 1267 We can ask whether the authors of these RFC are these outsiders, part 1268 of a "wider community" or are people who are also contributing to the 1269 IETF. The overwhelming response is, "insiders". Pretty much all the 1270 authors are or were involved in the IETF, many of them with a 1271 prominent track record. There are just 2 exceptions, a single RFC in 1272 which only 3 of the 5 authors are well associated with the IETF. 1274 4.5. Citation Counts 1276 Part of the exercise is to test whether citation counts provide a 1277 useful measure of the popularity of the IETF production. These 1278 citation counts vary widely: 1280 +------+--------+---------+ 1281 | RFC | Status | Scholar | 1282 +------+--------+---------+ 1283 | 8411 | Info | 2 | 1284 | | | | 1285 | 8456 | Info | 3 | 1286 | | | | 1287 | 8446 | PS | 123 | 1288 | | | | 1289 | 8355 | Info | 9 | 1290 | | | | 1291 | 8441 | PS | 0 | 1292 | | | | 1293 | 8324 | ISE | 2 | 1294 | | | | 1295 | 8377 | PS | 1 | 1296 | | | | 1297 | 8498 | Info | 0 | 1298 | | | | 1299 | 8479 | ISE | 0 | 1300 | | | | 1301 | 8453 | Info | 8 | 1302 | | | | 1303 | 8429 | BCP | 1 | 1304 | | | | 1305 | 8312 | Info | 9 | 1306 | | | | 1307 | 8492 | ISE | 2 | 1308 | | | | 1309 | 8378 | Exp | 1 | 1310 | | | | 1311 | 8361 | PS | 1 | 1312 | | | | 1313 | 8472 | PS | 5 | 1314 | | | | 1315 | 8466 | PS | 2 | 1316 | | | | 1317 | 8362 | PS | 7 | 1318 | | | | 1319 | 8468 | Info | 2 | 1320 +------+--------+---------+ 1322 The results indicate that [RFC8446] is by far the most popular of the 1323 20 RFC in our sample. This is not surprising, since TLS is a key 1324 Internet Protocol. Of the other publications, only 4 have 5 to 9 1325 citations, and the others have three or less. 1327 In order to get a baseline, we can look at the number of references 1328 for the RFC published in 2008 and 1998. However, we need totake time 1329 into account. Documents published a long time ago are expected to 1330 have accrued more references. We try to address this by looking at 1331 three counts for each document: the overall number of references over 1332 the document's lifetime, the number of references obtained in the 1333 year following publication, and the number of references observed 1334 since 2018: 1336 +------------+---------+----------+-------+ 1337 | RFC (2008) | Overall | 08 to 09 | >2018 | 1338 +------------+---------+----------+-------+ 1339 | 5326 | 234 | 27 | 26 | 1340 | | | | | 1341 | 5348 | 364 | 34 | 13 | 1342 | | | | | 1343 | 5281 | 182 | 27 | 12 | 1344 | | | | | 1345 | 5354 | 30 | 15 | 7 | 1346 | | | | | 1347 | 5227 | 81 | 9 | 7 | 1348 | | | | | 1349 | 5329 | 57 | 11 | 5 | 1350 | | | | | 1351 | 5277 | 85 | 10 | 5 | 1352 | | | | | 1353 | 5236 | 48 | 8 | 4 | 1354 | | | | | 1355 | 5358 | 45 | 6 | 3 | 1356 | | | | | 1357 | 5271 | 16 | 3 | 3 | 1358 | | | | | 1359 | 5195 | 21 | 12 | 1 | 1360 | | | | | 1361 | 5283 | 26 | 5 | 1 | 1362 | | | | | 1363 | 5186 | 13 | 4 | 1 | 1364 | | | | | 1365 | 5142 | 42 | 14 | 0 | 1366 | | | | | 1367 | 5373 | 16 | 4 | 0 | 1368 | | | | | 1369 | 5404 | 12 | 3 | 0 | 1370 | | | | | 1371 | 5172 | 11 | 2 | 0 | 1372 | | | | | 1373 | 5349 | 11 | 1 | 0 | 1374 | | | | | 1375 | 5301 | 11 | 3 | 0 | 1376 | | | | | 1377 | 5174 | 1 | 1 | 0 | 1378 +------------+---------+----------+-------+ 1379 +------------+---------+----------+-------+ 1380 | RFC (1998) | Overall | 98 to 99 | >2018 | 1381 +------------+---------+----------+-------+ 1382 | 2289 | 369 | 13 | 14 | 1383 | | | | | 1384 | 2267 | 607 | 13 | 7 | 1385 | | | | | 1386 | 2317 | 81 | 9 | 6 | 1387 | | | | | 1388 | 2404 | 446 | 20 | 5 | 1389 | | | | | 1390 | 2374 | 280 | 18 | 4 | 1391 | | | | | 1392 | 2449 | 54 | 6 | 3 | 1393 | | | | | 1394 | 2283 | 240 | 25 | 1 | 1395 | | | | | 1396 | 2394 | 69 | 4 | 1 | 1397 | | | | | 1398 | 2348 | 27 | 2 | 1 | 1399 | | | | | 1400 | 2382 | 89 | 30 | 0 | 1401 | | | | | 1402 | 2297 | 68 | 21 | 0 | 1403 | | | | | 1404 | 2381 | 86 | 20 | 0 | 1405 | | | | | 1406 | 2312 | 115 | 18 | 0 | 1407 | | | | | 1408 | 2387 | 292 | 8 | 0 | 1409 | | | | | 1410 | 2398 | 72 | 7 | 0 | 1411 | | | | | 1412 | 2391 | 110 | 5 | 0 | 1413 | | | | | 1414 | 2431 | 25 | 4 | 0 | 1415 | | | | | 1416 | 2282 | 12 | 3 | 0 | 1417 | | | | | 1418 | 2323 | 12 | 1 | 0 | 1419 | | | | | 1420 | 2448 | 5 | 1 | 0 | 1421 +------------+---------+----------+-------+ 1423 We can compare the median number of references, and the numbers of 1424 references for the least and most popular quartiles in the three 1425 years: 1427 +----------------------------+-----------+--------+------------+ 1428 | References | Lower 25% | Median | Higher 25% | 1429 +----------------------------+-----------+--------+------------+ 1430 | RFC (2018) | 1 | 2 | 6 | 1431 | | | | | 1432 | RFC (2008) | 12 | 26 | 57 | 1433 | | | | | 1434 | RFC (2008), until 2009 | 3 | 6 | 12 | 1435 | | | | | 1436 | RFC (2008), 2018 and after | 0 | 1 | 5 | 1437 | | | | | 1438 | RFC (1998) | 47 | 84 | 250 | 1439 | | | | | 1440 | RFC (1998), until 1999 | 4 | 9 | 19 | 1441 | | | | | 1442 | RFC (1998), 2018 and after | 0 | 0 | 3 | 1443 +----------------------------+-----------+--------+------------+ 1445 The total numbers shows new documents with fewer references than the 1446 older ones. This can be explained to some degree by the passage of 1447 time. If we restrict the analysis to the number of references 1448 accrued in the year of publishing and the year after that, we still 1449 see higher reference counts in 1998 than in 2008 or 2018. For 1450 example, the top quartile of RFC published in 1998 had at least 19 1451 references by the end of 1999, while the top quartile of RFC 1452 published in 2008 only had at least 12 references, which is twice the 1453 corresponding number for the RFC of 2018. 1455 We also see that the number of references to RFC fades over time. 1456 Only the most popular of the RFC produced in 1998 are still 1457 referenced in 2019. The overall popularity of the RFC series benefit 1458 from a history of publishing relevant documents, but over time the 1459 references to historic documents will decrease and the overall 1460 popularity will depend on more recent documents. 1462 We need however to be a bit cautious before asserting that 1463 publications with a low citation count have limited impact: 1465 o some documents may well accumulate more citations over time. For 1466 example, [RFC8377] updates [RFC6455]. There are more than 1000 1467 citations of [RFC6455] on Google Scholar. We might expect that 1468 the citation count of [RFC8377] will increase in the coming years. 1470 o citation counts largely come from academic publications, and thus 1471 reflect popularity within researchers more than popularity with 1472 network operators and vendors. 1474 We should be able to assess the popularity of specifications with 1475 vendors, operators and designers by asking questions about deployed 1476 services and products. 1478 5. Next steps 1480 This draft is not really complete. We have obtained feedback from 1481 many authors but not all. We should also get a review from the RFC 1482 Production Center. We may want to find a better way of looking at 1483 citations than simple queries on Google Scholar, and in any case we 1484 want to update the reference counts before publication, as they will 1485 keep changing over time. When an RFC describes a protocol, we may 1486 want to also compare the citation counts and the deployment status of 1487 the protocol. 1489 Even with those limitations, the exercise shows some promise, and 1490 also shows the interest of doing more studies. 1492 It is tempting to go from this exercise on published RFCs to a 1493 broader evaluation of the IETF productivity, but this would require 1494 more than just evaluating the RFCs. The IETF produces standard 1495 proposals and informative memos that get published in the RFC series, 1496 but that is not its only purpose. Two other purposes would be the 1497 organization of fruitful discussions between members of the technical 1498 community, and the filtering of ill-thought proposals so they are not 1499 endorsed in IETF publications. 1501 We don't have good ideas yet for evaluating the propagation of ideas, 1502 but we could perhaps evaluate the filtering: not enough filtering 1503 would cause bad ideas to be published; too much filtering would cause 1504 good ideas to be rejected. 1506 Not enough filtering should be visible by analyzing the published 1507 RFCs. Successful RFC will accrue many references and would drive 1508 many implementations. Unsuccessful RFCs would lack both. These 1509 criterias are hard to predict in advance, so we expect a fraction of 1510 the published RFCs to be unsuccessful. Too small a fraction would 1511 indicate a process that is too conservative, too large a fraction 1512 would indicate a process that's too lax. We can see that in the 1513 current analysis. 1515 On the other hand, a reasonable balance between success and lack of 1516 it does not guarantee that the process is very efficient. It could 1517 be that the culling of bad ideas also culls good ones, and we would 1518 not know by just looking at the publications. For that, we would 1519 need to look at publication attempts that were abandoned, for example 1520 drafts that expired without progressing or being replaced. The 1521 sampling methodology could also be used for that purpose. Pick maybe 1522 20 drafts at random, among those abandoned in a target year, and 1523 investigate why they were abandoned. Was it because better solutions 1524 emerged in the working group? Or maybe because the authors 1525 discovered a flaw in their proposal? Or was it because some 1526 factional struggle blocked a good idea? Was the idea pursued in a 1527 different venue? Hopefully, someone will try this kind of 1528 investigation. 1530 6. Security considerations 1532 This draft does not specify any protocol. 1534 We might want to analyze whether security issues were discovered 1535 after publication of specific standards. 1537 7. IANA Considerations 1539 This draft does not require any IANA action. 1541 Peliminary analysis does not indicate that IANA is causing any 1542 particular delay in the publication process. 1544 8. Acknowledgements 1546 Many thanks to the authors of the selected RFC who were willing to 1547 provide feedback on the process: Michael Ackermann, Zafar Ali, Sarah 1548 Banks, Bruno Decraene, Lars Eggert, Nalini Elkins, Joachim Fabini, 1549 Dino Farinacci, Clarence Filsfils, Sujay Gupta, Dan Harkins, Vinayak 1550 Hegde, Benjamin Kaduk, John Klensin, Acee Lindem, Nikos 1551 Mavrogiannopoulos, Patrick McManus, Victor Moreno, Al Morton, Andrei 1552 Popov, Eric Rescorla, Michiko Short, Bhuvaneswaran Vengainathan, Lao 1553 Weiguo, and Li Yizhou. Many thanks to Adrian Farrel for his useful 1554 advice. 1556 9. Informative References 1558 [RFC3797] Eastlake 3rd, D., "Publicly Verifiable Nominations 1559 Committee (NomCom) Random Selection", RFC 3797, 1560 DOI 10.17487/RFC3797, June 2004, 1561 . 1563 [RFC6455] Fette, I. and A. Melnikov, "The WebSocket Protocol", 1564 RFC 6455, DOI 10.17487/RFC6455, December 2011, 1565 . 1567 [RFC8312] Rhee, I., Xu, L., Ha, S., Zimmermann, A., Eggert, L., and 1568 R. Scheffenegger, "CUBIC for Fast Long-Distance Networks", 1569 RFC 8312, DOI 10.17487/RFC8312, February 2018, 1570 . 1572 [RFC8324] Klensin, J., "DNS Privacy, Authorization, Special Uses, 1573 Encoding, Characters, Matching, and Root Structure: Time 1574 for Another Look?", RFC 8324, DOI 10.17487/RFC8324, 1575 February 2018, . 1577 [RFC8355] Filsfils, C., Ed., Previdi, S., Ed., Decraene, B., and R. 1578 Shakir, "Resiliency Use Cases in Source Packet Routing in 1579 Networking (SPRING) Networks", RFC 8355, 1580 DOI 10.17487/RFC8355, March 2018, 1581 . 1583 [RFC8361] Hao, W., Li, Y., Durrani, M., Gupta, S., and A. Qu, 1584 "Transparent Interconnection of Lots of Links (TRILL): 1585 Centralized Replication for Active-Active Broadcast, 1586 Unknown Unicast, and Multicast (BUM) Traffic", RFC 8361, 1587 DOI 10.17487/RFC8361, April 2018, 1588 . 1590 [RFC8362] Lindem, A., Roy, A., Goethals, D., Reddy Vallem, V., and 1591 F. Baker, "OSPFv3 Link State Advertisement (LSA) 1592 Extensibility", RFC 8362, DOI 10.17487/RFC8362, April 1593 2018, . 1595 [RFC8377] Eastlake 3rd, D., Zhang, M., and A. Banerjee, "Transparent 1596 Interconnection of Lots of Links (TRILL): Multi-Topology", 1597 RFC 8377, DOI 10.17487/RFC8377, July 2018, 1598 . 1600 [RFC8378] Moreno, V. and D. Farinacci, "Signal-Free Locator/ID 1601 Separation Protocol (LISP) Multicast", RFC 8378, 1602 DOI 10.17487/RFC8378, May 2018, 1603 . 1605 [RFC8410] Josefsson, S. and J. Schaad, "Algorithm Identifiers for 1606 Ed25519, Ed448, X25519, and X448 for Use in the Internet 1607 X.509 Public Key Infrastructure", RFC 8410, 1608 DOI 10.17487/RFC8410, August 2018, 1609 . 1611 [RFC8411] Schaad, J. and R. Andrews, "IANA Registration for the 1612 Cryptographic Algorithm Object Identifier Range", 1613 RFC 8411, DOI 10.17487/RFC8411, August 2018, 1614 . 1616 [RFC8429] Kaduk, B. and M. Short, "Deprecate Triple-DES (3DES) and 1617 RC4 in Kerberos", BCP 218, RFC 8429, DOI 10.17487/RFC8429, 1618 October 2018, . 1620 [RFC8441] McManus, P., "Bootstrapping WebSockets with HTTP/2", 1621 RFC 8441, DOI 10.17487/RFC8441, September 2018, 1622 . 1624 [RFC8446] Rescorla, E., "The Transport Layer Security (TLS) Protocol 1625 Version 1.3", RFC 8446, DOI 10.17487/RFC8446, August 2018, 1626 . 1628 [RFC8453] Ceccarelli, D., Ed. and Y. Lee, Ed., "Framework for 1629 Abstraction and Control of TE Networks (ACTN)", RFC 8453, 1630 DOI 10.17487/RFC8453, August 2018, 1631 . 1633 [RFC8455] Bhuvaneswaran, V., Basil, A., Tassinari, M., Manral, V., 1634 and S. Banks, "Terminology for Benchmarking Software- 1635 Defined Networking (SDN) Controller Performance", 1636 RFC 8455, DOI 10.17487/RFC8455, October 2018, 1637 . 1639 [RFC8456] Bhuvaneswaran, V., Basil, A., Tassinari, M., Manral, V., 1640 and S. Banks, "Benchmarking Methodology for Software- 1641 Defined Networking (SDN) Controller Performance", 1642 RFC 8456, DOI 10.17487/RFC8456, October 2018, 1643 . 1645 [RFC8466] Wen, B., Fioccola, G., Ed., Xie, C., and L. Jalil, "A YANG 1646 Data Model for Layer 2 Virtual Private Network (L2VPN) 1647 Service Delivery", RFC 8466, DOI 10.17487/RFC8466, October 1648 2018, . 1650 [rfc8468] Morton, A., Fabini, J., Elkins, N., Ackermann, M., and V. 1651 Hegde, "IPv4, IPv6, and IPv4-IPv6 Coexistence: Updates for 1652 the IP Performance Metrics (IPPM) Framework", RFC 8468, 1653 DOI 10.17487/RFC8468, November 2018, 1654 . 1656 [RFC8472] Popov, A., Ed., Nystroem, M., and D. Balfanz, "Transport 1657 Layer Security (TLS) Extension for Token Binding Protocol 1658 Negotiation", RFC 8472, DOI 10.17487/RFC8472, October 1659 2018, . 1661 [RFC8479] Mavrogiannopoulos, N., "Storing Validation Parameters in 1662 PKCS#8", RFC 8479, DOI 10.17487/RFC8479, September 2018, 1663 . 1665 [RFC8492] Harkins, D., Ed., "Secure Password Ciphersuites for 1666 Transport Layer Security (TLS)", RFC 8492, 1667 DOI 10.17487/RFC8492, February 2019, 1668 . 1670 [RFC8498] Mohali, M., "A P-Served-User Header Field Parameter for an 1671 Originating Call Diversion (CDIV) Session Case in the 1672 Session Initiation Protocol (SIP)", RFC 8498, 1673 DOI 10.17487/RFC8498, February 2019, 1674 . 1676 Author's Address 1678 Christian Huitema 1679 Private Octopus Inc. 1680 427 Golfcourse Rd 1681 Friday Harbor WA 98250 1682 U.S.A 1684 Email: huitema@huitema.net