idnits 2.17.1 draft-huitema-rfc-eval-project-02.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack an Introduction section. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (October 7, 2019) is 1663 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- -- Obsolete informational reference (is this intentional?): RFC 2267 (Obsoleted by RFC 2827) -- Obsolete informational reference (is this intentional?): RFC 8312 (Obsoleted by RFC 9438) Summary: 1 error (**), 0 flaws (~~), 1 warning (==), 3 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group C. Huitema 3 Internet-Draft Private Octopus Inc. 4 Intended status: Informational October 7, 2019 5 Expires: April 9, 2020 7 Evaluation of a Sample of RFC Produced in 2018 8 draft-huitema-rfc-eval-project-02 10 Abstract 12 This document presents a first attempt at evaluating the recently 13 published RFC. We analyze a set of randomly chosen RFC approved in 14 2018, looking for history and delays. The average RFC was produced 15 in 3 years and 4 months, of which 2 years and 10 months were spent in 16 the working group, 3 to 4 months for IETF consensus and IESG review, 17 and 3 to 4 months in RFC production. The main variation in RFC 18 production delays comes from the AUTH48 phase. 20 We also measure the number of citations of the chosen RFC using 21 Semantic Scholar, and compare citation counts with what we know about 22 deployment. We show that citation counts indicate academic interest, 23 but correlate only loosely with deployment or usage of the 24 specifications. 26 The measurement of delays could be automated by processing dates and 27 events recorded in the data tracker. The measurement of published 28 RFC could be complemented by statistics on abandoned drafts, which 29 would measure the efficiency of the IETF triaging process. 31 Status of This Memo 33 This Internet-Draft is submitted in full conformance with the 34 provisions of BCP 78 and BCP 79. 36 Internet-Drafts are working documents of the Internet Engineering 37 Task Force (IETF). Note that other groups may also distribute 38 working documents as Internet-Drafts. The list of current Internet- 39 Drafts is at https://datatracker.ietf.org/drafts/current/. 41 Internet-Drafts are draft documents valid for a maximum of six months 42 and may be updated, replaced, or obsoleted by other documents at any 43 time. It is inappropriate to use Internet-Drafts as reference 44 material or to cite them other than as "work in progress." 46 This Internet-Draft will expire on April 9, 2020. 48 Copyright Notice 50 Copyright (c) 2019 IETF Trust and the persons identified as the 51 document authors. All rights reserved. 53 This document is subject to BCP 78 and the IETF Trust's Legal 54 Provisions Relating to IETF Documents 55 (https://trustee.ietf.org/license-info) in effect on the date of 56 publication of this document. Please review these documents 57 carefully, as they describe your rights and restrictions with respect 58 to this document. Code Components extracted from this document must 59 include Simplified BSD License text as described in Section 4.e of 60 the Trust Legal Provisions and are provided without warranty as 61 described in the Simplified BSD License. 63 Table of Contents 65 1. RFC Evaluation project . . . . . . . . . . . . . . . . . . . 3 66 2. Selecting a random sample of RFC . . . . . . . . . . . . . . 4 67 3. Analysis of 20 selected RFC . . . . . . . . . . . . . . . . . 6 68 3.1. 8411 . . . . . . . . . . . . . . . . . . . . . . . . . . 6 69 3.2. 8456 . . . . . . . . . . . . . . . . . . . . . . . . . . 7 70 3.3. 8446 . . . . . . . . . . . . . . . . . . . . . . . . . . 7 71 3.4. 8355 . . . . . . . . . . . . . . . . . . . . . . . . . . 8 72 3.5. 8441 . . . . . . . . . . . . . . . . . . . . . . . . . . 9 73 3.6. 8324 . . . . . . . . . . . . . . . . . . . . . . . . . . 10 74 3.7. 8377 . . . . . . . . . . . . . . . . . . . . . . . . . . 10 75 3.8. 8498 . . . . . . . . . . . . . . . . . . . . . . . . . . 11 76 3.9. 8479 . . . . . . . . . . . . . . . . . . . . . . . . . . 11 77 3.10. 8453 . . . . . . . . . . . . . . . . . . . . . . . . . . 12 78 3.11. 8429 . . . . . . . . . . . . . . . . . . . . . . . . . . 12 79 3.12. 8312 . . . . . . . . . . . . . . . . . . . . . . . . . . 13 80 3.13. 8492 . . . . . . . . . . . . . . . . . . . . . . . . . . 14 81 3.14. 8378 . . . . . . . . . . . . . . . . . . . . . . . . . . 15 82 3.15. 8361 . . . . . . . . . . . . . . . . . . . . . . . . . . 15 83 3.16. 8472 . . . . . . . . . . . . . . . . . . . . . . . . . . 16 84 3.17. 8466 . . . . . . . . . . . . . . . . . . . . . . . . . . 16 85 3.18. 8362 . . . . . . . . . . . . . . . . . . . . . . . . . . 17 86 3.19. 8468 . . . . . . . . . . . . . . . . . . . . . . . . . . 17 87 4. Analysis of process and delays . . . . . . . . . . . . . . . 18 88 4.1. Publication delays . . . . . . . . . . . . . . . . . . . 18 89 4.2. Preparation and Publication delays . . . . . . . . . . . 24 90 4.3. Copy editing . . . . . . . . . . . . . . . . . . . . . . 27 91 4.4. Independent Series . . . . . . . . . . . . . . . . . . . 30 92 5. Citation Counts . . . . . . . . . . . . . . . . . . . . . . . 30 93 5.1. Citation Numbers . . . . . . . . . . . . . . . . . . . . 31 94 5.2. Comparison to 1998 and 2008 . . . . . . . . . . . . . . . 32 95 5.3. Citations versus deployments . . . . . . . . . . . . . . 35 97 6. Observations and Next Steps . . . . . . . . . . . . . . . . . 37 98 7. Security considerations . . . . . . . . . . . . . . . . . . . 38 99 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 38 100 9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 38 101 10. Informative References . . . . . . . . . . . . . . . . . . . 39 102 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 42 104 1. RFC Evaluation project 106 As stated on the organization's web site, "The IETF is a large open 107 international community of network designers, operators, vendors, and 108 researchers concerned with the evolution of the Internet architecture 109 and the smooth operation of the Internet." In this memo, we attempt 110 to evaluate the RFC production process. 112 The IETF data tracker provides information about RFC and drafts, from 113 which we can infer statistics about the production system. We can 114 measure how long it takes to drive a proposition from initial draft 115 to final publication, and how these delays can be split between 116 Working Group discussions, IETF reviews, IESG assessment, RFC Editor 117 delays and final reviews by the authors. 119 Just measuring production delays may be misleading. If the IETF 120 simply rubber-stamped draft proposals and published them, the delays 121 would be short but the quality and impact might suffer. We hope that 122 most of the RFC that are published are useful, but we need a way to 123 measure that usefulness. We try to do that by measuring the number 124 of references of the published RFCs in Semantic Scholar, and also by 125 checking whether the protocols and technologies defined in the RFCs 126 were implemented and used on the Internet. 128 In order to limit the resource required for this study, we selected 129 at random 20 RFC published in 2018, as explained in (#sample- 130 selection). For comparison purposes, we also selected at random 20 131 RFC published in 1998 and 20 published in 2008. The information 132 gathered for every RFC in the sample is presented in (#sample-rfc- 133 analysis). In (#process-analysis) we analyze the production process 134 and the sources of delays, comparing the 2018 sample to the selected 135 samples for 1998 and 2018. In (#citation-numbers) we present 136 citation counts for the RFC in the samples, and analyze whether 137 citation counts could be used to evaluate the quality of RFC. 138 Finally, we present the conclusion of the study and propose next 139 steps in (#conclusion). 141 2. Selecting a random sample of RFC 143 Basic production mechanisms could be evaluated by processing data 144 from the IETF tracker, but subjective data requires manual assessment 145 of results, which can be time consuming. Since our resources are 146 limited, we will only perform this analysis for a small sample of 147 RFC, selected at random from the list of RFC approved in 2018. 148 Specifically, we will pick 20 RFC at random between: 150 o RFC 8307, published in January 2018, and 152 o RFC 8511, published December 2018. 154 In order to avoid injecting personal bias in the random selection, we 155 use a random selection process similar to the Nomination Committee 156 selection process defined in [RFC3797]. The process is seeded with 157 the text string "vanitas vanitatum et omnia vanitas", and the results 158 are: 160 Picking 20 numbers between 8307 and 8511, 161 using MD5(vanitas vanitatum et omnia vanitas) 162 Rank 1: 8411 -- md5=daba041224a879199b698748808f917d 163 Rank 2: 8456 -- md5=f5570484d91ada6a672edbdca61d808c 164 Rank 3: 8446 -- md5=8340e918bb8faf69d197f79c9a58d7b8 165 Rank 4: 8355 -- md5=19474df74efd9917cf3fe8acce2ac374 166 Rank 5: 8441 -- md5=5acce2b730f3c24a4a91a5fc1921d1cd 167 Rank 6: 8324 -- md5=411c11a1cf4c292f83458865599c6921 168 Rank 7: 8377 -- md5=ac16a89192c0f0727febd35aacbc1f24 169 Rank 8: 8498 -- md5=bba44f2ba1ab240a1265a82ab71f7e02 170 Rank 9: 8479 -- md5=1653606b0af95d529a473a8f85ffaea4 171 Rank 10: 8453 -- md5=0cbfe105667c5a83b027dcfa85062f98 172 Rank 11: 8429 -- md5=fa51d7738562d990926a0d199fb060b8 173 Rank 12: 8312 -- md5=96d061523b1a57343356ae7a1e498ca5 174 Rank 13: 8492 -- md5=1b72b746eb05f79af40ed2bd3faccbe8 175 Rank 14: 8378 -- md5=645833b936d36cdcc797256518d7c483 176 Rank 15: 8361 -- md5=2064622c868e410beb0d9c18d0cb522c 177 Rank 16: 8472 -- md5=ca8a823072a21df011d0ea8b96a6aa47 178 Rank 17: 8471 -- md5=01b293a7dd0793e6f3297f2a973cd7e3 179 Rank 18: 8466 -- md5=8e411babe271557fe83bcdececc1643f 180 Rank 19: 8362 -- md5=8a1ba3efd82856a12b2b35fc5237e1b7 181 Rank 20: 8468 -- md5=57ae50ee0e1e0708d356d96d116dbfe1 183 When evaluating delays and impact, we will compare the year 2018 to 184 2008 and 1998, 10 and 20 years ago. To drive this comparison, we 185 pick 20 RFC at random among those published in 2008, and another 20 186 among those published in 1998. We use the same nomcom-like 187 methodology. 189 For 2008, we picking random RFC numbers between RFC 5134 (January 190 2008) and RFC 5405 (December 2008), using the sentence "sed fugit 191 interea fugit irreparabile tempus" as a seed. We actually list here 192 21 numbers, because the random draw place RFC 5315 in 20th position, 193 but that RFC was never issued. We replace it by RFC 5301, which came 194 in 21st position. 196 Picking 21 numbers between 5134 and 5405, 197 using MD5(sed fugit interea fugit irreparabile tempus) 198 Rank 1: 5227 -- md5=61e5b3cd97fda4e75450e93df0744b6d 199 Rank 2: 5174 -- md5=eed49542394e9d392bcd756ee0f5beed 200 Rank 3: 5172 -- md5=72055ca953d6a8ee0d60a9fca0b8d738 201 Rank 4: 5354 -- md5=7a00ef15897479d1d15255159f6d8674 202 Rank 5: 5195 -- md5=54813be7bb56f48a05af8c894799b51f 203 Rank 6: 5236 -- md5=153263bbd8c0349501b75a33f3d66f6c 204 Rank 7: 5348 -- md5=b2d19aa9c1250ef2ddf169045f6e99d7 205 Rank 8: 5281 -- md5=1c3e61643d46d1da4ba16f0fd7a0aff5 206 Rank 9: 5186 -- md5=5e87001b830183b1a9427d479a7b0e42 207 Rank 10: 5326 -- md5=024347839f83d8082549c08bdfa1b43e 208 Rank 11: 5277 -- md5=049a83016ab08552841c59400480cd9d 209 Rank 12: 5373 -- md5=a1ce374aaebdacca2e7d6eeff039d970 210 Rank 13: 5404 -- md5=fb0d6b582a27ce34175e39de33598556 211 Rank 14: 5329 -- md5=df043ef1f9d42ba12a03a84434d26ead 212 Rank 15: 5283 -- md5=c40d3f966bc7800d6508d3d82df2371d 213 Rank 16: 5358 -- md5=6fea5bdb26b19e68befd409a09cb335d 214 Rank 17: 5142 -- md5=a8844b73287781762e6548fc6f533508 215 Rank 18: 5271 -- md5=c19eb02984265ecfe4ca076f2c160cfa 216 Rank 19: 5349 -- md5=33d756f81bf6e40ff344cf6ccaf29f13 217 Rank 20: 5315 -- md5=d4c30875f88328d72c9f78def2d1dde5 218 Rank 21: 5301 -- md5=3356419e5560901f0d31309b39d14a80 220 For 1998 we picking random RFC numbers between RFC 2257 (January 221 1998) and RFC 2479 (December 1998), using the sentence "pulvis et 222 umbra sumus" as a seed. 224 Picking 20 numbers between 2257 and 2479, 225 using MD5(pulvis et umbra sumus) 226 Rank 1: 2431 -- md5=6eba444bb3349339fcde7e2be726f0f0 227 Rank 2: 2381 -- md5=0ce53f69f1a49a49309054af1e9a1d42 228 Rank 3: 2387 -- md5=53726e77ffeed903d244155d28e30f10 229 Rank 4: 2348 -- md5=69fcab2d2085555bac9eb0e0f4346523 230 Rank 5: 2391 -- md5=93358a26a7fce9fa9d61a31cdfdb87b7 231 Rank 6: 2267 -- md5=652dcab91bd9d5c58f98bdab21b23d80 232 Rank 7: 2312 -- md5=2505b0bed4af0a00ee663891c6f294d8 233 Rank 8: 2448 -- md5=6ede4b0dfa935f6ca656a5104b8dc5d0 234 Rank 9: 2374 -- md5=5312bfe5a9ca45563eb0bf35510d8daa 235 Rank 10: 2398 -- md5=7748a6deec3860898678f992b0b22792 236 Rank 11: 2283 -- md5=d73ff67466eb42d971a0e134b9284f83 237 Rank 12: 2382 -- md5=de1f667ac3e4c64aa529872e08823dff 238 Rank 13: 2289 -- md5=37773c2569dc25fdd0ab400ea401d5c7 239 Rank 14: 2282 -- md5=6b3df671a0a0becf9e42d203b59acd08 240 Rank 15: 2404 -- md5=e1b6819e5355924f456eb79f93beb8fd 241 Rank 16: 2449 -- md5=4f057df7c226efea773e7013c9c62081 242 Rank 17: 2317 -- md5=40eee3b536abe4afdb8834a0650c0a04 243 Rank 18: 2394 -- md5=044f09c53fc9fd1c50fe6bd0c39318e1 244 Rank 19: 2297 -- md5=78ee7e128436c969c80900fef80c075c 245 Rank 20: 2323 -- md5=ea6935bbda5f6d97756d3df5c3e2fdfb 247 3. Analysis of 20 selected RFC 249 We review each of the RFC listed in (#methodology) for the year 2018, 250 trying both to answer the known questions and to gather insight for 251 further analyzes. In many cases, the analysis of the data is 252 complemented by direct feedback from the RFC authors. 254 3.1. 8411 256 IANA Registration for the Cryptographic Algorithm Object Identifier 257 Range [RFC8411]: 259 Informational, 5 pages 260 4 drafts (personal),first May 8, 2017. Published August 2018. 261 Last call announced 2017/10/09 262 IESG evaluation starts 2017/12/28 263 Approved 2018/02/26, draft 03 264 Auth 48 2018/04/20 265 Auth 48 complete 2018/07/17 266 Published 2018/08/06 267 IANA action: create table 269 The draft underwent minor copy edit before publication. 271 The long delay in Auth48 is probaby due to clustering with [RFC8410], 272 which entered AUTH 48 on 06/05. The MISSREF tracker code was cleared 273 then. 275 3.2. 8456 277 Benchmarking Methodology for Software-Defined Networking (SDN) 278 Controller Performance [RFC8456]: 280 Informational, 64 pages 281 2 personal drafts, 9 WG drafts, first in March 2015 282 Last call announced 2018-01-19 283 IESG evaluation starts 2018-02-27 284 IESG approved 2018-05-25 285 Auth 48 2018-08-31 286 Auth 48 complete 2018-10-16 287 Published 2018-10-30 289 The draft underwent very extensive copy editing, covering use of 290 articles, turn of phrases, choice of vocabulary. The changes are 291 enough to cause pagination differences. The "diff" tool marks pretty 292 much every page as changed. Some diagrams see change in protocol 293 elements like message names. 295 According to the author, the experience of producing this draft 296 mirrors a typical one in the Benchmarking Methodologies Working Group 297 (BMWG).There were multiple authors in multiple time zones, which 298 slowed down the AUTH process somewhat, although the Auth48 delay of 299 46 is only a bit longer than the average draft. 301 The RFC was part of cluster with [RFC8455]. 303 BMWG publishes informational RFCs centered around benchmarking, and 304 the methodologies in RFC 8456 have been implemented in benchmarking 305 products. 307 3.3. 8446 309 The Transport Layer Security (TLS) Protocol Version 1.3 [RFC8446], as 310 the title indicates, defines the new version of the TLS protocol. 311 From the datatracker, we extract the following: 313 Proposed standard 314 160 pages 315 29 WG drafts first April 17, 2014. 316 Last call announced 2018/02/15 317 IESG evaluation starts 2018/03/02 318 Approved 2018/03/21, draft 28 319 Auth 48 2018/06/14 320 Auth 48 complete 2018/08/10 321 Published 2018/08/10 323 The RFC was a major effort in the IETF. Working group members 324 developed and tested several implementations. Researchers analyzed 325 the specifications and performed formal verifications. Deployment 326 tests outlined issues that caused extra work when the specification 327 was almost ready. These complexity largely explains the time spent 328 in the working group. 330 Comparing the final draft to the published version, we find 331 relatively light copy editing. It includes explaining acronyms on 332 first use, clarifying some definitions standardizing punctiation and 333 capitalization, and spelling out some numbers in text. This 334 generally fall in the category of "style", although some of the 335 clarifications go into message definitions. However, that simple 336 analysis does not explain why the Auth48 phase took almost two 337 months. 339 This document's Auth48 process was part of the "Github experiment", 340 which tried to use github pull requests to track the AUTH48 changes 341 and review comments. The RPC staff had to learn using Github for 342 that process, and this required more work than the usual RFC. Author 343 and AD thoroughly reviewed each proposed edit, accepting some and 344 rejecting some. The concern there was that any change in a complex 345 specification might affect a protocol that was extensively reviewed 346 in the working group, but of course these reviews added time to the 347 Aouth48 delays. 349 There are 21 implementations listed in the Wiki of the TLS 1.3 350 project. It has been deployed on major browsers, and is already used 351 in a large fraction of TLS connections. 353 3.4. 8355 355 Resiliency Use Cases in Source Packet Routing in Networking (SPRING) 356 Networks [RFC8355] is an informational RFC. It originated from a use 357 case informational draft that was mostly used for the BOF creating 358 the WG, and then to drive initial work/evolutions from the WG. 360 Informational, 13 pages. 361 2 personal drafts (personal), first January 31, 2014. 13 WG drafts. 362 Last call announced 2017-04-20 363 IESG evaluation starts 2017-05-04, draft 09 364 Approved 2017-12-19, draft 12 365 Auth 48 2018-03-12 366 Auth 48 complete 2018-03-27 367 Published 2018-03-28 369 Minor set of copy edit, mostly for style. 371 No implementation of the RFC itself, but the technology behind it 372 such as Segment Routing (architecture RFC 8402, TI-LFA draft-ietf- 373 rtgwg-segment-routing-ti-lfa) is widely implemented and deployment is 374 ongoing. 376 According to participants in the discussion, the process of adoption 377 of the source packet routing standards was very contentious. The 378 establishment of consensus at both the working group level and the 379 IETF level was difficult and time consuming. 381 3.5. 8441 383 Bootstrapping WebSockets with HTTP/2 [RFC8441] 385 Proposed standard, 8 pages. Updates RFC 6455. 386 3 personal drafts (personal), first 10/15/2017. 8 WG drafts. 387 Last call announced 2018-05-07, draft 05 388 IESG evaluation starts 2018-05-29, draft 06 389 Approved 2018-06-07, draft 07 390 Auth 48 2018-08-13 391 Auth 48 complete 2018-09-15 392 Published 2018-09-21 393 IANA Action: table entries 395 This RFC defines the support of WebSockets in HTTP/2, which is 396 different from the mechanism defined for HTTP/1.1 in [RFC6455]. The 397 process was relatively straightforward, involving the usual type of 398 discussions, some on details and some on important points. 400 Comparing final draft and published RFC shows a minor set of copy 401 edit, mostly for style. However, the author recalls a painful 402 process. The RFC includes many charts and graphs that were very 403 difficult to format correctly in the author's production process that 404 involve conversions from markdown to XML, and then from XML to text. 405 The author had to get substantial help from the RFC editor. 407 There are several implementations, including Firefox and Chrome, 408 making RFC 8441 a very successful standard. 410 3.6. 8324 412 DNS Privacy, Authorization, Special Uses, Encoding, Characters, 413 Matching, and Root Structure: Time for Another Look? [RFC8324]. 414 This is an opinion piece on DNS development, published on the 415 Independent Stream. 417 Informational, 29 pages. Independent stream. 418 5 personal drafts (personal), first June 2, 2017. 419 ISE review started 2017-07-10, draft 03 420 IETF conflict review and IESG review started 2017-10-29 421 Approved 2017-12-18, draft 04 422 Auth 48 2018-01-29, draft 05 423 Auth 48 complete 2018-02-26 424 Published 2018-02-27 426 This RFC took only 9 months from first draft to publication, which is 427 the shortest in the 2018 sample set. In part, this is because the 428 text was privately circulated and reviewed before the first draft was 429 published. The nature of the document is another reason for the 430 short delay. It is an opinion piece, and does not require the same 431 type of consensus building and reviews than a protocol specification. 433 Comparing the final draft and the published version shows only minor 434 copy edit, mostly for style. According to the author, because this 435 is because he knows how to write in RFC Style with the result that 436 his documents often need a minimum of editing. He also makes sure 437 that the document on which the Production Center starts working 438 already has changes discussed and approved during Last Call and IESG 439 review incorporated rather than expecting the Production Center to 440 operate off of notes about changed to be made. 442 3.7. 8377 444 Transparent Interconnection of Lots of Links (TRILL): Multi-Topology 445 [RFC8377] 446 Proposed standard, 20 pages. Updates RFC 6325, 7177. 447 3 personal drafts (personal), first September 3, 2013. 7 WG drafts. 448 Last call announced 2018-02-19, draft 05 449 IESG evaluation starts 2018-03-02, draft 06 450 Approved 2018-03-12, draft 05 451 Auth 48 2018-04-20, draft 06 452 Auth 48 complete 2018-07-31 453 Published 2018-07-31 454 IANA Table, table entries 456 Minor set of copy edit, mostly for style, also clarity. 458 3.8. 8498 460 A P-Served-User Header Field Parameter for an Originating Call 461 Diversion (CDIV) Session Case in the Session Initiation Protocol 462 (SIP) [RFC8498]. 464 Informational, 15 pages. 465 5 personal drafts (personal), first March 21, 2016. 9 WG drafts. 466 Last call announced 2018-10-12, draft 05 467 IESG evaluation starts 2018-11-28, draft 07 468 Approved 2018-12-10, draft 08 469 Auth 48 2019-01-28 470 Auth 48 complete 2019-02-13 471 Published 2019-02-15 472 IANA Action, table rows added. 474 Copy edit for style, but also clarification of ambiguous sentences. 476 3.9. 8479 478 Storing Validation Parameters in PKCS#8 [RFC8479] 480 Informational, 8 pages. Independent stream. 481 5 personal drafts (personal), first August 8, 2017. 482 ISE review started 2018-03-29, draft 02 483 IETF conflict review and IESG review started 2018-03-29 484 Approved 2018-08-20, draft 03 485 Auth 48 2018-09-20, draft 04 486 Auth 48 complete 2018-09-25 487 Published 2018-09-26 489 The goal of the draft was to document what the gnutls implementation 490 was using for storing provably generated RSA keys. This is a short 491 RFC that was published relatively quickly, although discussion 492 between the author, the Independent Series Editor and the IESG lasted 493 several months. In the initial conflict review, Tthe IESG asked the 494 ISE to not publish this document before IETF working groups had an 495 opportunity to pick up the work. The author met that requirement by 496 a presentation to the SECDISPATCH WG in IETF 102. Since no WG was 497 interested in pickup the work, the document progressed on the 498 Independent Stream. 500 Very minor set of copy edit, moving some references from normative to 501 informative. 503 The author is not aware of other implementations than gnutls relying 504 on this RFC. 506 3.10. 8453 508 Framework for Abstraction and Control of TE Networks (ACTN) [RFC8453] 510 Informational, 42 pages. 511 3 personal drafts, first June 15, 2015. 16 WG drafts. 512 Out of WG 2018-01-26, draft 11 513 Expert review requested, 2018-02-13 514 Last call announced 2018-04-16, draft 13 515 IESG evaluation starts 2018-05-16, draft 14 516 Approved 2018-06-01, draft 15 517 Auth 48 2018-08-13 518 Auth 48 complete 2018-08-20 519 Published 2018-08-20 520 IANA Action, table rows added. 522 Minor copy editing. 524 3.11. 8429 526 Deprecate Triple-DES (3DES) and RC4 in Kerberos [RFC8429] 528 BCP, 10 pages. 529 6 WG drafts, first 5/1/2017. 530 Last call announced 7/16/2017, draft 03 531 IESG evaluation starts 8/18/2017, draft 04 532 Approved 5/25/2018, draft 05 533 Auth 48 7/24/2018 534 Auth 48 complete 10/31/2018 535 Published 10/31/2018 536 IANA Action, table rows added. 538 This RFC recommends to deprecate two encryption algorithms that are 539 now considered obsolete and possibly broken. The document was sent 540 back to the WG after the first last call, edited, and then there was 541 a second last call. The delay from first draft to working group last 542 call was relatively short, but the number may be misleading. The 543 initial draft was a replacement of a similar draft in the KITTEN 544 working group, which stagnated for some time before the CURDLE 545 working group took up the work. The deprecation of RC4 was somewhat 546 contentious, but the WG had already debated this prior to the 547 production of this draft, and the draft was not delayed by this 548 debate. 550 Most of the 280 days between IETF LC and IESG approval was because 551 the IESG had to talk about whether this document should obsolete or 552 move to historic RFC 4757, and no one was really actively pushing 553 that discussion for a while. 555 The 99 days in AUTH48 are mostly because one of the authors was a 556 sitting AD, and those duties ended up taking precedence over 557 reviewing this document. 559 Minor copy editing, for style. 561 The implementation of the draft would be the actual removal of 562 support for 3DES and RC4 in major implementations. This is 563 happening, but very slowly. 565 3.12. 8312 567 CUBIC for Fast Long-Distance Networks [RFC8312] 569 Informational, 18 pages. 570 2 personal drafts, first 9/1/2014. 8 WG drafts 571 Last call announced 9/18/2017, draft 06 572 IESG evaluation starts 2017-11-14 573 Approved 2017-10-04, draft 07 574 Auth 48 2018-01-08 575 Auth 48 complete 2018-02-07 576 Published 2018-02-07 577 IANA Action, table rows added. 579 Minor copy editing, for style. 581 The TCP congestion control algorithm Cubic was defined first in 2005, 582 was implemented in Linux soon after, and was implemented in major 583 OSes after that. After some debates from 2015 to 2015, the TCPM 584 working group adopted the draft, with a goal of documenting Cubic in 585 the RFc series. According to the authors, this was not a high 586 priority effort, as Cubic was already implemented in multiple OSes 587 and documented in research papers. At some point, only one of the 588 authors was actively working on the draft. Ths may explain why 589 another two years was spent progressing the draft after adoption by 590 the WG. 592 The RFC publication may or may not have triggered further 593 implementations. On the other hand, several OSes picked up bug fixes 594 from the draft and the RFC. 596 3.13. 8492 598 Secure Password Ciphersuites for Transport Layer Security (TLS) 599 [RFC8492] 601 Informational, 40 pages. (Independent Stream) 602 10 personal drafts, first 9/2/2012. 8 WG drafts 603 ISE review started 2017-05-10, draft 01 604 IETF conflict review and IESG review started 2017-09-04 605 Approved 2017-10-29, draft 04 606 Auth 48 10/19/2018, draft 05 607 Auth 48 complete 2/19/2019 608 Published 2/21/2019 609 IANA Action, table rows added. 611 This RFC has a complex history. The first individual draft was 612 submitted to the TLS working group on September 7, 2012. It 613 progressed there, and was adopted by the WG after 3 revisions. There 614 were then 8 revisions in the TLS WG, until the WG decided to not 615 progress it. The draft was parked in 2013 by the WG chairs after 616 failing to get consensus in WG last call. The AD finally pulled the 617 plug in 2016, and the draft was then resubmitted to the ISE. 619 At that point, the author was busy and was treating this RFC with a 620 low priority because, in his words, it would not be a "real RFC". 621 There were problems with the draft that only came up late. In 622 particular, it had to wait for a change in registry policy that only 623 came about with the publication of TLS 1.3, which caused the draft to 624 only be published after RFC 8446, and also required adding references 625 to TLS 1.3. The author also got a very late comment while in AUTH48 626 that caused some rewrite. Finally, there was some IANA issue with 627 the extension registry where a similar extension was added by someone 628 else. The draft was changed to just use it. 630 Changes in AUTH48 include added reference to TLS 1.3, copy-editing 631 for style, some added requirements, added paragraphs, and changes in 632 algorithms specification. 634 3.14. 8378 636 Signal-Free Locator/ID Separation Protocol (LISP) Multicast [RFC8378] 637 is an experimental RFC, defining how to implement Multicast in the 638 LISP architecture. 640 Experimental, 21 pages. 641 5 personal drafts, first 2/28/2014. 10 WG drafts 642 Last call announced 2018-02-13, draft 07 643 IESG evaluation starts 2018-02-28, draft 08 644 Approved 2018-03-12, draft 09 645 Auth 48 2018-04-23 646 Auth 48 complete 2018-05-02 647 Published 2018-05-02 649 Preparing the RFC took more than 4 years. According to the authors, 650 they were not aggressive pushing it and just let the working group 651 process decide to pace it. They also did implementations during that 652 time. 654 Minor copy editing, for style. 656 The RFC was implemented by lispers.net and cisco, and was used in 657 doing IPv6 multicast over IPv4 unicast/multicast at the Olympics in 658 PyeungChang. The plan is to work on a proposedstandard once the 659 experiment concludes. 661 3.15. 8361 663 Transparent Interconnection of Lots of Links (TRILL): Centralized 664 Replication for Active-Active Broadcast, Unknown Unicast, and 665 Multicast (BUM) Traffic [RFC8361] 667 Proposed Standard, 17 pages. 668 3 personal drafts, first 11/12/2013. 14 WG drafts 669 Last call announced 2017-11-28, draft 10 670 IESG evaluation starts 2017-12-18, draft 11 671 Approved 2018-01-29, draft 13 672 Auth 48 2018-09-17 673 Auth 48 complete 4/9/2018 674 Published 2018-10-08 676 According to the authors, the long delays in producing this RFC was 677 due to a slow uptake of the technology in the industry. 679 Minor copy editing, for style. 681 There was at least 1 partial implementation. 683 3.16. 8472 685 Transport Layer Security (TLS) Extension for Token Binding Protocol 686 Negotiation [RFC8472] 688 Proposed Standard, 8 pages. 689 1 personal drafts, first 5/29/2015. 15 WG drafts 690 Last call announced 2017-11-13, draft 10 691 IESG evaluation starts 2018-03-19 692 Approved 2018-07-20, draft 14 693 Auth 48 2018-09-17 694 Auth 48 complete 2018-09-25 695 Published 2018-10-08 697 This is a pretty simple document, but it took over 3 years from 698 individual draft to RFC. According to the authors,the biggest 699 setbacks occurred at the start: it took a while to find a home for 700 this draft. It was presented in the TLS WG (because it's a TLS 701 extension) and UTA WG (because it has to do with applications using 702 TLS). Then the ADs determined that a new WG was needed, so the 703 authors had to work through the WG creation process, including 704 running a BOF. 706 Minor copy editing, for style, with the addition of a reference to 707 TLS 1.3. 709 Perhaps partially due to the delays, some of the implementers lost 710 interest in supporting this RFC. 712 3.17. 8466 714 A YANG Data Model for Layer 2 Virtual Private Network (L2VPN) Service 715 Delivery [RFC8466] 717 Proposed Standard, 158 pages. 718 5 personal drafts, first 9/1/2016. 11 WG drafts 719 Last call announced 2018-02-21, draft 07 720 IESG evaluation starts 2018-03-14, draft 08 721 Approved 2018-06-25, draft 10 722 Auth 48 2018-09-17 723 Auth 48 complete 2018-10-09 724 Published 2018-10-12 726 Copy editing for style and clarity, with also corrections to the yang 727 model. 729 3.18. 8362 731 OSPFv3 Link State Advertisement (LSA) Extensibility [RFC8362] is a 732 major extension to the OSPF protocol. It makes OSPFv3 fully 733 extensible. 735 Proposed Standard, 33 pages. 736 4 personal drafts, first February 17, 2013. 24 WG drafts 737 Last call announced 2017-12-19, draft 19 738 IESG evaluation starts 2018-01-18, draft 20 739 Approved 2018-01-29, draft 23 740 Auth 48 2018-03-19 741 Auth 48 complete 2018-03-30 742 Published 2018-04-03 744 The specification was first submitted as a personal draft in the IPv6 745 WG, then moved to the OSPF WG. The long delay of producing this RFC 746 is due to the complexity of the problem, and the need to wait for 747 implementations. It is a very important change to OSPF that makes 748 OSPFv3 fully extensible. Since it was a non-backward compatible 749 change, the developers started out with some very complex migration 750 scenarios but ended up with either legacy or extended OSPFv3 LSAs 751 within an OSPFv3 routing domain. The initial attempts to have a 752 hybrid mode of operation with both legacy and extended LSAs also 753 delayed implementation due to the complexity. 755 Copy editing for style and clarity. 757 This specification either was or will be implemented by all the 758 router vendors. 760 3.19. 8468 762 IPv4, IPv6, and IPv4-IPv6 Coexistence: Updates for the IP Performance 763 Metrics (IPPM) Framework [rfc8468]. 765 Informational, 15 pages. 766 3 personal drafts, first August 6, 2015. 7 WG drafts 767 Last call announced 2018-04-11, draft 04 768 IESG evaluation starts 2018-05-24, draft 05 769 Approved 2018-07-10, draft 06 770 Auth 48 2018-09-13 771 Auth 48 complete 2018-11-05 772 Published 2018-11-14 774 RFC8468 was somehow special in that there was not a technical reason/ 775 interest that triggered it, but rather a formal requirement. While 776 writing RFC7312 the IP Performance Metrics working group (IPPM) 777 realized that RFC 2330, the IP Performance Metrics Framework 778 supported IPv4 only and explicitly excluded support for IPv6. 779 Nevertheless, people used the metrics that were defined on top of RFC 780 2330 (and, therefore, IPv4 only) for IPv6, too. Although the IPPM WG 781 agreed that the work was needed, the interest of IPPM attendees in 782 progressing (and reading/reviewing) the IPv6 draft was limited. 783 Resolving the IPv6 technical part was straight-forward, but 784 subsequently some people asked for a broader scope (topics like 785 header compression, 6lo, etc.) and it took some time to figure out 786 and later on convince people that these topics are out of scope. The 787 group also had to resolve contentious topics, for example how to 788 measure the processing of IPv6 extension headers, which is sometimes 789 non-standard. 791 The Auth48 delay for this draft was longer than average. According 792 to the authors, the main reasons include: 794 o Work-load and travel caused by busy-work-periods of all co-authors 796 o Time zone difference between co-authors and editor (at least US, 797 Europe, India, not considering travel) 799 o Editor proposing and committing some unacceptable modifications 800 that needed to be reverted 802 o Lengthy discussions on a new document title (required high effort 803 and took a long time, in particular reaching consensus between co- 804 authors and editor was time-consuming and involved the AD) 806 o Editor correctly identifying some nits (obsoleted personal 807 websites of co-authors) and co-authors attempting to fix them. 809 The differences between the final draft and the publish RFC show copy 810 editing for style and clarity, but do not account for the back and 811 forth between authors and editors mentioned by the authors. 813 4. Analysis of process and delays 815 We examine the 20 RFC in the sample, measuring various 816 characteristics such as delay and citation counts, in an attempt to 817 identify patterns in the IETF processes. 819 4.1. Publication delays 821 We look at the distribution of delays between the submission of the 822 first draft and the publication of the RFC. We break out the total 823 delay in three components: 825 o The working group delay, from the first draft to the start of the 826 IETF last call; 828 o The IETF delay, which lasts from the beginning of the IETF last 829 call to the approval by the IESG, including the reviews by various 830 directorates; 832 o The RFC production, from approval by the IESG to publication, 833 including the Auth48 reviews. 835 For submissions to the independent stream, we don't have a working 836 group. We consider instead the progression of the individual draft 837 until the adoption by the ISE as the equivalent of the "working 838 group" period, and the delay from adoption by the ISE until 839 submission to the RFC Editor as the equivalent of the IETF delay. 840 The following table shows the delays for the 20 RFC in the sample: 842 +------+------------------+-------+---------+------+------+------+ 843 | RFC | Status | Pages | Overall | WG | IETF | Edit | 844 +------+------------------+-------+---------+------+------+------+ 845 | 8411 | Info | 5 | 455 | 154 | 140 | 161 | 846 | | | | | | | | 847 | 8456 | Info | 64 | 1107 | 823 | 126 | 158 | 848 | | | | | | | | 849 | 8446 | PS | 160 | 1576 | 1400 | 34 | 142 | 850 | | | | | | | | 851 | 8355 | Info | 13 | 1517 | 1175 | 243 | 99 | 852 | | | | | | | | 853 | 8441 | PS | 8 | 341 | 204 | 31 | 106 | 854 | | | | | | | | 855 | 8324 | ISE | 29 | 270 | 38 | 161 | 71 | 856 | | | | | | | | 857 | 8377 | PS | 8 | 1792 | 1630 | 21 | 141 | 858 | | | | | | | | 859 | 8498 | Info | 15 | 1061 | 935 | 59 | 67 | 860 | | | | | | | | 861 | 8479 | ISE | 8 | 414 | 233 | 144 | 37 | 862 | | | | | | | | 863 | 8453 | Info | 42 | 1162 | 1036 | 46 | 80 | 864 | | | | | | | | 865 | 8429 | BCP | 10 | 548 | 76 | 313 | 159 | 866 | | | | | | | | 867 | 8312 | Info | 18 | 1255 | 1113 | 16 | 126 | 868 | | | | | | | | 869 | 8492 | ISE | 40 | 2358 | 1706 | 172 | 480 | 870 | | | | | | | | 871 | 8378 | Exp | 21 | 1524 | 1446 | 27 | 51 | 872 | | | | | | | | 873 | 8361 | PS | 17 | 1612 | 1477 | 62 | 73 | 874 | | | | | | | | 875 | 8472 | PS | 8 | 1228 | 899 | 249 | 80 | 876 | | | | | | | | 877 | 8466 | PS | 158 | 771 | 538 | 124 | 109 | 878 | | | | | | | | 879 | 8362 | PS | 33 | 1871 | 1766 | 41 | 64 | 880 | | | | | | | | 881 | 8468 | Info | 15 | 1196 | 979 | 90 | 127 | 882 | | | | | | | | 883 | | average | 35 | 1161 | 928 | 110 | 123 | 884 | | | | | | | | 885 | | average(not ISE) | 37 | 1188 | 978 | 101 | 109 | 886 +------+------------------+-------+---------+------+------+------+ 888 The average delay from first draft to publication is about 3 years, 889 but this varies widely. Excluding the independent stream 890 submissions, the average delay from start to finish is 3 years and 4 891 months, of which on average 2 years and 10 months are spent getting 892 consensus in the working group, and 3 to 4 months each for IETF 893 consensus and for RFC production. 895 The longest delay is found for [RFC8492], 6.5 years from start to 896 finish. This is however a very special case, a draft that was 897 prepared for the TLS working group and failed to reach consensus. 898 After that, it was resubmitted to the ISE, and incurred atypical 899 production delays. 901 On average, we see that 80% of the delay is incurred in WG 902 processing, 10% in IETF review, and 10% for edition and publication. 903 We can compare these delays to those observed 10 years ago and 20 904 years ago: 906 +------------+--------+-------+-------+ 907 | RFC (2008) | Status | Pages | Delay | 908 +------------+--------+-------+-------+ 909 | 5326 | Exp | 54 | 1584 | 910 | | | | | 911 | 5348 | PS | 58 | 823 | 912 | | | | | 913 | 5281 | Info | 51 | 1308 | 914 | | | | | 915 | 5354 | Exp | 23 | 2315 | 916 | | | | | 917 | 5227 | PS | 21 | 2434 | 918 | | | | | 919 | 5329 | PS | 12 | 1980 | 920 | | | | | 921 | 5277 | PS | 35 | 912 | 922 | | | | | 923 | 5236 | ISE | 26 | 1947 | 924 | | | | | 925 | 5358 | BCP | 7 | 884 | 926 | | | | | 927 | 5271 | Info | 22 | 1066 | 928 | | | | | 929 | 5195 | PS | 10 | 974 | 930 | | | | | 931 | 5283 | PS | 12 | 1096 | 932 | | | | | 933 | 5186 | Info | 6 | 2253 | 934 | | | | | 935 | 5142 | PS | 13 | 1005 | 936 | | | | | 937 | 5373 | PS | 24 | 1249 | 938 | | | | | 939 | 5404 | PS | 27 | 214 | 940 | | | | | 941 | 5172 | PS | 7 | 305 | 942 | | | | | 943 | 5349 | Info | 10 | 1096 | 944 | | | | | 945 | 5301 | PS | 6 | 396 | 946 | | | | | 947 | 5174 | Info | 8 | 427 | 948 +------------+--------+-------+-------+ 950 +------------+--------+-------+---------+ 951 | RFC (1998) | Status | Pages | Delay | 952 +------------+--------+-------+---------+ 953 | 2289 | PS | 25 | 396 | 954 | | | | | 955 | 2267 | Info | 10 | unknown | 956 | | | | | 957 | 2317 | BCP | 10 | 485 | 958 | | | | | 959 | 2404 | PS | 7 | 488 | 960 | | | | | 961 | 2374 | PS | 12 | 289 | 962 | | | | | 963 | 2449 | PS | 19 | 273 | 964 | | | | | 965 | 2283 | PS | 9 | 153 | 966 | | | | | 967 | 2394 | Info | 6 | 365 | 968 | | | | | 969 | 2348 | DS | 5 | 699 | 970 | | | | | 971 | 2382 | Info | 30 | 396 | 972 | | | | | 973 | 2297 | ISE | 109 | 28 | 974 | | | | | 975 | 2381 | PS | 43 | 699 | 976 | | | | | 977 | 2312 | Info | 20 | 365 | 978 | | | | | 979 | 2387 | PS | 10 | 122 | 980 | | | | | 981 | 2398 | Info | 15 | 396 | 982 | | | | | 983 | 2391 | PS | 10 | 122 | 984 | | | | | 985 | 2431 | PS | 10 | 457 | 986 | | | | | 987 | 2282 | Info | 14 | 215 | 988 | | | | | 989 | 2323 | ISE | 5 | unknown | 990 | | | | | 991 | 2448 | ISE | 7 | 92 | 992 +------------+--------+-------+---------+ 994 We can compare the median delay, and the delays observed by the 995 fastest and slowest quartiles in the three years: 997 +------+-------------+--------+-------------+ 998 | Year | Fastest 25% | Median | Slowest 25% | 999 +------+-------------+--------+-------------+ 1000 | 2018 | 604 | 1179 | 1522 | 1001 | | | | | 1002 | 2008 | 869 | 1081 | 1675 | 1003 | | | | | 1004 | 1998 | 169 | 365 | 442 | 1005 +------+-------------+--------+-------------+ 1007 The IETF takes three to four times more times to produce an RFC in 1008 2018 than it did in 1998, but about the same time as it did in 2008. 1010 The increased delay does not mean increased work per RFC. The number 1011 of RFC published per year remained between 200 and 300 all those 1012 years, and the number of participants is not greater now than in 1013 1998. If we estimated the "level of attention" by dividing the 1014 number of participants by the number of RFC produced, we would see a 1015 number that remains stable. People are probably not working much 1016 more on each RFC now than they were 20 years ago, but the same amount 1017 of work is stretched over a much longer period. 1019 4.2. Preparation and Publication delays 1021 The preparation and publication delays include three components: 1023 o the delay from submission to the RFC Editor to beginning of 1024 Auth48, during which the document is prepared; 1026 o the AUTH48 delay, during which authors review and eventually 1027 approve the changes proposed by the editors; 1029 o the publication delay, from final agreement by authors and editors 1030 to actual publication. 1032 The breakdown of the publication delays for each RFC is shown in the 1033 following table. 1035 +-------+---------+-------+--------+---------+--------+-------------+ 1036 | RFC | Status | Pages | RFC | Auth 48 | RFC | Edit(total) | 1037 | | | | edit | | Pub | | 1038 +-------+---------+-------+--------+---------+--------+-------------+ 1039 | 8411 | Info | 5 | 53 | 88 | 20 | 161 | 1040 | | | | | | | | 1041 | 8456 | Info | 64 | 98 | 46 | 14 | 158 | 1042 | | | | | | | | 1043 | 8446 | PS | 160 | 85 | 57 | 0 | 142 | 1044 | | | | | | | | 1045 | 8355 | Info | 13 | 83 | 15 | 1 | 99 | 1046 | | | | | | | | 1047 | 8441 | PS | 8 | 67 | 33 | 6 | 106 | 1048 | | | | | | | | 1049 | 8324 | ISE | 29 | 42 | 28 | 1 | 71 | 1050 | | | | | | | | 1051 | 8377 | PS | 8 | 39 | 102 | 0 | 141 | 1052 | | | | | | | | 1053 | 8498 | Info | 15 | 49 | 16 | 2 | 67 | 1054 | | | | | | | | 1055 | 8479 | ISE | 8 | 31 | 5 | 1 | 37 | 1056 | | | | | | | | 1057 | 8453 | Info | 42 | 73 | 7 | 0 | 80 | 1058 | | | | | | | | 1059 | 8429 | BCP | 10 | 60 | 99 | 0 | 159 | 1060 | | | | | | | | 1061 | 8312 | Info | 18 | 96 | 30 | 0 | 126 | 1062 | | | | | | | | 1063 | 8492 | ISE | 40 | 355 | 123 | 2 | 480 | 1064 | | | | | | | | 1065 | 8378 | Exp | 21 | 42 | 9 | 0 | 51 | 1066 | | | | | | | | 1067 | 8361 | PS | 17 | 39 | 31 | 3 | 73 | 1068 | | | | | | | | 1069 | 8472 | PS | 8 | 59 | 8 | 13 | 80 | 1070 | | | | | | | | 1071 | 8466 | PS | 158 | 84 | 22 | 3 | 109 | 1072 | | | | | | | | 1073 | 8362 | PS | 33 | 49 | 11 | 4 | 64 | 1074 | | | | | | | | 1075 | 8468 | Info | 15 | 65 | 53 | 9 | 127 | 1076 | | | | | | | | 1077 | | Average | | 77.3 | 41.2 | 4.2 | 122.7 | 1078 | | | | | | | | 1079 | -8492 | Average | | 62 | 37 | 4 | 103 | 1080 +-------+---------+-------+--------+---------+--------+-------------+ 1081 On average, the total delay appears to be a bit more than four month, 1082 but the average is skewed by the extreme values encountered for 1083 [RFC8492]. If we exclude that RFC from the computations, the average 1084 delay drops to a just a bit more than 3 months: about 2 months for 1085 the preparation, a bit more than one month for the Auth48 phase, and 1086 4 days for the publishing. 1088 Of course, these delays vary from RFC to RFC. To try explain the 1089 causes of the delay, we compute the correlation factor between the 1090 observed delays and 4 plausible explanation factors: 1092 o The number of pages in the document, 1094 o The amount of copy edit, as discussed in (#copy-editing), 1096 o Whether or not an IANA action was required. 1098 We find the following values: 1100 +-------------+----------+---------+-------------+ 1101 | Correlation | RFC edit | Auth 48 | Edit(total) | 1102 +-------------+----------+---------+-------------+ 1103 | Nb pages | 0.50 | -0.04 | 0.21 | 1104 | | | | | 1105 | Copy-Edit | 0.42 | 0.24 | 0.45 | 1106 | | | | | 1107 | IANA | -0.13 | 0.26 | 0.15 | 1108 +-------------+----------+---------+-------------+ 1110 None of these indicate strong correlations. The greater number of 1111 pages will tend to increase the preparation delay, but it does not 1112 appear to impact the Auth48 delay at all. The amount of copy editing 1113 also tend to increase the the preparation delay somewhat and the 1114 Auth48 delay a little. The presence or absence of IANA action has 1115 very ittle correlation with the delays. 1117 We also observe that the Auth48 delay varies much more than the 1118 preparation delay, with a standard deviation of 20 days for Auth48 1119 versus 10 days for the preparation delay. In theory, Auth48 is just 1120 a final verification: the authors receive the document prepared by 1121 the RFC production center, and just have to give their approval, or 1122 maybe request a last minute correction. The name indicates that this 1123 is expected to last just two days, but in average it lasts more than 1124 a month. 1126 We tested a variety of hypotheses that might explain the duration of 1127 AUTH48 by computing the correlation coefficients between various 1128 properties of the RFC and the production delays. The results are 1129 listed in the following table: 1131 +-------------+----------------+---------+-----------------+ 1132 | Correlation | RFC production | Auth 48 | RFC Edit(total) | 1133 +-------------+----------------+---------+-----------------+ 1134 | Nb drafts | 0.19 | -0.30 | -0.17 | 1135 | | | | | 1136 | Nb Authors | 0.40 | -0.04 | 0.16 | 1137 | | | | | 1138 | WG delay | 0.03 | -0.16 | -0.15 | 1139 +-------------+----------------+---------+-----------------+ 1141 The results show that there is no simple answer. The number of 1142 pages, the required amount of copy editing and to a very small extent 1143 the number of drafts can help predict the production delay, but there 1144 is no obvious predictor for the Auth 48 delay. In particular, there 1145 is no numerical evidence that the number of authors influences the 1146 Auth48 delay, or that authors who have spent a long time working on 1147 the document in the working group somehow spend even longer to answer 1148 questions during Auth48 - if anything, the numerical results point in 1149 the opposite direction. 1151 After asking the authors of the RFC in the sample why the AUTH48 1152 phase took a long time, and we got three explanations: 1154 1- Some RFC have multiple authors in multiple time zones. This slows 1155 down the coordination required for approving changes. 1157 2- Some authors found some of the proposed changes unnecessary or 1158 undesirable, and asked that they be reversed. This required long 1159 exchanges between authors and editors. 1161 3- Some authors were not giving high priority to AUTH48 responses. 1163 As mentioned above, we were not able to verify these hypotheses by 1164 looking at the data. 1166 4.3. Copy editing 1168 We can assess the amount of copy editing applied to each published 1169 RFC by comparing the text of the draft approved for publication and 1170 the text of the RFC. We do expect differences in the "boilerplate" 1171 and in the IANA section, but we will also see differences due to copy 1172 editing. Assessing the amount of copy editing is subjective, and we 1173 do it using a scale of 1 to 4: 1175 1- Minor editing 1176 2- Editing for style, such as capitalization, hyphens, that versus 1177 which, and expending all acronyms at least one. 1179 3- Editing for clarity in addition to style, such as rewriting 1180 ambiguous sentences and clarifying use of internal references. For 1181 Yang models, that may include model corrections suggested by the 1182 verifier. 1184 4- Extensive editing. 1186 The following table shows that with about half of the RFC required 1187 editing for style, and the other half at least some editing for 1188 clarity. 1190 +------+--------+-----------+ 1191 | RFC | Status | Copy Edit | 1192 +------+--------+-----------+ 1193 | 8411 | Info | 2 | 1194 | | | | 1195 | 8456 | Info | 4 | 1196 | | | | 1197 | 8446 | PS | 3 | 1198 | | | | 1199 | 8355 | Info | 2 | 1200 | | | | 1201 | 8441 | PS | 2 | 1202 | | | | 1203 | 8324 | ISE | 2 | 1204 | | | | 1205 | 8377 | PS | 3 | 1206 | | | | 1207 | 8498 | Info | 3 | 1208 | | | | 1209 | 8479 | ISE | 1 | 1210 | | | | 1211 | 8453 | Info | 2 | 1212 | | | | 1213 | 8429 | BCP | 2 | 1214 | | | | 1215 | 8312 | Info | 2 | 1216 | | | | 1217 | 8492 | ISE | 3 | 1218 | | | | 1219 | 8378 | Exp | 2 | 1220 | | | | 1221 | 8361 | PS | 2 | 1222 | | | | 1223 | 8472 | PS | 2 | 1224 | | | | 1225 | 8466 | PS | 3 | 1226 | | | | 1227 | 8362 | PS | 3 | 1228 | | | | 1229 | 8468 | Info | 3 | 1230 +------+--------+-----------+ 1232 This method of assessment does not take into account the number of 1233 changes proposed by the editors and eventually rejected by the 1234 authors, since these changes are not present in either the final 1235 draft or the published RFC. It might be possible to get an 1236 evaluation of these "phantom changes" from the RFC Production Center. 1238 4.4. Independent Series 1240 Out of 20 randomly selected RFC, 3 were published through the 1241 "independent series". One is an independent opinion, another a 1242 description of a non-IETF protocol format, and the third was 1243 [RFC8492], which is a special case. Apart from this special case, 1244 the publication delays were significantly shorter for the Independent 1245 Stream than for the IETF stream. This seems to indicate that the 1246 Independent Series is functioning as expected. 1248 The authors of these 3 RFC are regular IETF contributors. This 1249 observation motivated a secondary analysis of all the RFC published 1250 in the "independent" stream in 2018. There are 14 such RFC: 8507, 1251 8494, 8493, 8492, 8483, 8479, 8433, 8409, 8374, 8369, 8367, 8351, 1252 8328 and 8324. (RFC 8367 and 8369 were published on 1 April 2018.) 1253 We can ask whether the authors of these RFC are these outsiders, part 1254 of a "wider community" or are people who are also contributing to the 1255 IETF. The overwhelming response is, "insiders". Pretty much all the 1256 authors are or were involved in the IETF, many of them with a 1257 prominent track record. There are just 2 exceptions, a single RFC in 1258 which only 3 of the 5 authors are well associated with the IETF. 1260 5. Citation Counts 1262 In this exploration, we want to assess whether citation counts 1263 provide a meaningful assessment of the popularity of RFC. We obtain 1264 the citation counts through the Semantic Scholar API, using queries 1265 of the form: ~~~ http://api.semanticscholar.org/ v1/paper/10.17487/ 1266 rfc8446?include_unknown_references=true ~~~ In these queries, the RFC 1267 is uniquely identified by its DOI reference, which is composed of the 1268 RFC Series prefix 10.17487 and the rfc identifier. The queries 1269 return a series of properties, including a list of citations for the 1270 RFC. Based on that list of citations, we compute three numbers: 1272 o The total number of citations 1274 o The number of citations in the year of publication and the year 1275 after that 1277 o For the RFC published in 1998 or 2008 that we use for comparison, 1278 the number of citations in the years 2018 and 2019. 1280 All the numbers were retrieved on October 6, 2019. 1282 5.1. Citation Numbers 1284 As measured on October 6, 2019, the citation counts for the RFC in 1285 our sample set were: 1287 +-----------+--------+-------+-----------+ 1288 | RFC(2018) | Status | Total | 2018-2019 | 1289 +-----------+--------+-------+-----------+ 1290 | 8411 | Info | 1 | 0 | 1291 | | | | | 1292 | 8456 | Info | 1 | 1 | 1293 | | | | | 1294 | 8446 | PS | 418 | 204 | 1295 | | | | | 1296 | 8355 | Info | 3 | 3 | 1297 | | | | | 1298 | 8441 | PS | 1 | 1 | 1299 | | | | | 1300 | 8324 | ISE | 0 | 0 | 1301 | | | | | 1302 | 8377 | PS | 0 | 0 | 1303 | | | | | 1304 | 8498 | Info | 0 | 0 | 1305 | | | | | 1306 | 8479 | ISE | 0 | 0 | 1307 | | | | | 1308 | 8453 | Info | 3 | 3 | 1309 | | | | | 1310 | 8429 | BCP | 0 | 0 | 1311 | | | | | 1312 | 8312 | Info | 25 | 16 | 1313 | | | | | 1314 | 8492 | ISE | 4 | 4 | 1315 | | | | | 1316 | 8378 | Exp | 1 | 1 | 1317 | | | | | 1318 | 8361 | PS | 0 | 0 | 1319 | | | | | 1320 | 8472 | PS | 1 | 1 | 1321 | | | | | 1322 | 8466 | PS | 0 | 0 | 1323 | | | | | 1324 | 8362 | PS | 1 | 1 | 1325 | | | | | 1326 | 8468 | Info | 1 | 1 | 1327 +-----------+--------+-------+-----------+ 1329 The results indicate that [RFC8446] is by far the most cited of the 1330 20 RFC in our sample. This is not surprising, since TLS is a key 1331 Internet Protocol. The TLS 1.3 protocol was also the subject of 1332 extensive studies by researchers, and thus was mentioned in a number 1333 of published papers. Surprisingly, the Semantic Scholar mentions a 1334 number of citations that predate the publication date. These are 1335 probably citations of the various draft versions of the protocol. 1337 The next most cited RFC in the sample is [RFC8312] which describes 1338 the Cubic congestion control algorithm for TCP. That protocol was 1339 also the target of a large number of academic publications.The other 1340 RFC in the sample only have a small number of citations. 1342 5.2. Comparison to 1998 and 2008 1344 In order to get a baseline, we can look at the number of references 1345 for the RFC published in 2008 and 1998. However, we need totake time 1346 into account. Documents published a long time ago are expected to 1347 have accrued more references. We try to address this by looking at 1348 three counts for each document: the overall number of references over 1349 the document's lifetime, the number of references obtained in the 1350 year following publication, and the number of references observed 1351 since 2018: 1353 +-----------+--------+-------+-----------+-----------+ 1354 | RFC(2008) | Status | Total | 2008-2009 | 2018-2019 | 1355 +-----------+--------+-------+-----------+-----------+ 1356 | 5326 | Exp | 138 | 14 | 15 | 1357 | | | | | | 1358 | 5348 | PS | 14 | 3 | 0 | 1359 | | | | | | 1360 | 5281 | Info | 69 | 15 | 7 | 1361 | | | | | | 1362 | 5354 | Exp | 17 | 13 | 0 | 1363 | | | | | | 1364 | 5227 | PS | 19 | 1 | 2 | 1365 | | | | | | 1366 | 5329 | PS | 24 | 6 | 1 | 1367 | | | | | | 1368 | 5277 | PS | 32 | 3 | 2 | 1369 | | | | | | 1370 | 5236 | ISE | 25 | 5 | 4 | 1371 | | | | | | 1372 | 5358 | BCP | 21 | 2 | 0 | 1373 | | | | | | 1374 | 5271 | Info | 7 | 2 | 0 | 1375 | | | | | | 1376 | 5195 | PS | 7 | 4 | 2 | 1377 | | | | | | 1378 | 5283 | PS | 8 | 1 | 0 | 1379 | | | | | | 1380 | 5186 | Info | 14 | 4 | 2 | 1381 | | | | | | 1382 | 5142 | PS | 8 | 4 | 0 | 1383 | | | | | | 1384 | 5373 | PS | 5 | 2 | 0 | 1385 | | | | | | 1386 | 5404 | PS | 1 | 1 | 0 | 1387 | | | | | | 1388 | 5172 | PS | 2 | 0 | 0 | 1389 | | | | | | 1390 | 5349 | Info | 8 | 0 | 2 | 1391 | | | | | | 1392 | 5301 | PS | 5 | 1 | 0 | 1393 | | | | | | 1394 | 5174 | Info | 0 | 0 | 0 | 1395 +-----------+--------+-------+-----------+-----------+ 1396 +-----------+--------+-------+-----------+-----------+ 1397 | RFC(1998) | Status | Total | 1998-1999 | 2018-2019 | 1398 +-----------+--------+-------+-----------+-----------+ 1399 | 2289 | PS | 2 | 0 | 1 | 1400 | | | | | | 1401 | 2267 | Info | 982 | 5 | 61 | 1402 | | | | | | 1403 | 2317 | BCP | 9 | 1 | 2 | 1404 | | | | | | 1405 | 2404 | PS | 137 | 6 | 1 | 1406 | | | | | | 1407 | 2374 | PS | 42 | 4 | 0 | 1408 | | | | | | 1409 | 2449 | PS | 7 | 2 | 0 | 1410 | | | | | | 1411 | 2283 | PS | 17 | 3 | 2 | 1412 | | | | | | 1413 | 2394 | Info | 13 | 2 | 1 | 1414 | | | | | | 1415 | 2348 | DS | 5 | 0 | 0 | 1416 | | | | | | 1417 | 2382 | Info | 17 | 12 | 0 | 1418 | | | | | | 1419 | 2297 | ISE | 36 | 11 | 0 | 1420 | | | | | | 1421 | 2381 | PS | 39 | 12 | 0 | 1422 | | | | | | 1423 | 2312 | Info | 14 | 3 | 0 | 1424 | | | | | | 1425 | 2387 | PS | 4 | 1 | 0 | 1426 | | | | | | 1427 | 2398 | Info | 17 | 0 | 1 | 1428 | | | | | | 1429 | 2391 | PS | 31 | 3 | 0 | 1430 | | | | | | 1431 | 2431 | PS | 3 | 0 | 0 | 1432 | | | | | | 1433 | 2282 | Info | 8 | 0 | 0 | 1434 | | | | | | 1435 | 2323 | ISE | 1 | 0 | 0 | 1436 | | | | | | 1437 | 2448 | ISE | 0 | 0 | 0 | 1438 +-----------+--------+-------+-----------+-----------+ 1440 We can compare the median number of citations and the numbers of 1441 citations for the least and most popular quartiles in the three 1442 years: 1444 +----------------------------+-----------+--------+------------+ 1445 | References | Lower 25% | Median | Higher 25% | 1446 +----------------------------+-----------+--------+------------+ 1447 | RFC (2018) | 0 | 1 | 3 | 1448 | | | | | 1449 | RFC (2008) | 6.5 | 11 | 21.75 | 1450 | | | | | 1451 | RFC (2008), until 2009 | 1 | 2.5 | 4.5 | 1452 | | | | | 1453 | RFC (2008), 2018 and after | 0 | 0 | 2 | 1454 | | | | | 1455 | RFC (1998) | 4.75 | 13.5 | 32.25 | 1456 | | | | | 1457 | RFC (1998), until 1999 | 0 | 2 | 4.25 | 1458 | | | | | 1459 | RFC (1998), 2018 and after | 0 | 0 | 1 | 1460 +----------------------------+-----------+--------+------------+ 1462 The total numbers shows new documents with fewer citations than the 1463 older ones. This can be explained to some degree by the passage of 1464 time. If we restrict the analysis to the number of citations accrued 1465 in the year of publishing and the year after that, we still see about 1466 the same distribution for the three samples. 1468 We also see that the number of references to RFC fades over time. 1469 Only the most popular of the RFC produced in 1998 are still cited in 1470 2019. 1472 5.3. Citations versus deployments 1474 The following table shows side by side the number of citations as 1475 measured in (#citation-numbers) and the estimation of deployment as 1476 indicated in (#sample-rfc-analysis). 1478 +-----------+--------+-----------+------------+ 1479 | RFC(2018) | Status | Citations | Deployment | 1480 +-----------+--------+-----------+------------+ 1481 | 8411 | Info | 1 | medium | 1482 | | | | | 1483 | 8456 | Info | 1 | medium | 1484 | | | | | 1485 | 8446 | PS | 418 | high | 1486 | | | | | 1487 | 8355 | Info | 3 | medium | 1488 | | | | | 1489 | 8441 | PS | 1 | high | 1490 | | | | | 1491 | 8324 | ISE | 0 | N/A | 1492 | | | | | 1493 | 8377 | PS | 0 | unknown | 1494 | | | | | 1495 | 8498 | Info | 0 | unknown | 1496 | | | | | 1497 | 8479 | ISE | 0 | one | 1498 | | | | | 1499 | 8453 | Info | 3 | unknown | 1500 | | | | | 1501 | 8429 | BCP | 0 | some | 1502 | | | | | 1503 | 8312 | Info | 25 | high | 1504 | | | | | 1505 | 8492 | ISE | 4 | one | 1506 | | | | | 1507 | 8378 | Exp | 1 | some | 1508 | | | | | 1509 | 8361 | PS | 0 | one | 1510 | | | | | 1511 | 8472 | PS | 1 | medium | 1512 | | | | | 1513 | 8466 | PS | 0 | unknown | 1514 | | | | | 1515 | 8362 | PS | 1 | medium | 1516 | | | | | 1517 | 8468 | Info | 1 | some | 1518 +-----------+--------+-----------+------------+ 1520 From looking at these results, it is fairly obvious that citation 1521 counts cannot be used as proxies for the "value" of an RFC. In our 1522 sample, the two RFC that have high citation counts were both widely 1523 deployed, and can certainly be described as successful, but we also 1524 see many RFC that saw significant deployment without garnering a high 1525 level of citations. 1527 Citation counts are driven by academic interest, but are only loosely 1528 correlated with actual deployment. We saw that [RFC8446] was widely 1529 cited in part because the standardization process involved many 1530 researchers, and that the high citation count of [RFC8312] is largely 1531 due to the academic interest in evaluating congestion control 1532 protocols. If we look at previous years, the most cited RFC in the 1533 2008 sample is [RFC5326], an experimental RFC defining security 1534 extensions to an experimental delay tolerant transport protocol. 1535 This protocol does not carry a significant proportion of Internet 1536 traffic, but has been the object of a fair number of academic 1537 studies. 1539 The citation process tends to privilege the first expression of a 1540 concept. We see that with the most cited RFC in the 1998 set is 1541 [RFC2267], an informational RFC defining Network Ingress Filtering 1542 that was obsoleted in May 2000 by [RFC2827]. It is still cited 1543 frequently in 2018 and 2019, regardless of its formal status in the 1544 RFC series. We see the same effect at work with [RFC8441], which 1545 garners very few citations although it obsoletes [RFC6455] that has a 1546 large number of citations. The same goes for [RFC8468], which is 1547 sparsely cited while the [RFC2330] is widely cited. Just counting 1548 citations will not indicate whether developers still use an old 1549 specification or have adopted the revised RFC. 1551 6. Observations and Next Steps 1553 The goal of this study was to evaluate how the IETF produces 1554 standards, from working group processing to RFC production. As shown 1555 in (#process-analysis), the average RFC was produced in 3 years and 4 1556 months, which is similar to what was found in the 2008 sample, but 1557 more than three times larger than the delays for the 1998 sample. 1559 The Working group process appears to be the main source of delays. 1560 Efforts to diminish delays should probably focus there, instead of on 1561 the IETF and IESG reviews of the RFC production. For the RFC 1562 production phase, most of the variability originates in the AUTH48 1563 process, which is influenced by a variety of factors such as number 1564 of authors or level of engagement of these authors. 1566 The Independent Submission stream operates as expected. The authors 1567 of the independent RFC appear to be mostly IETF insiders. 1569 The analysis of citations in (#citation-numbers) shows that citation 1570 numbers are a very poor indication of the "value" of an RFC. 1571 Citation numbers measure the engagement of academic researchers with 1572 specific topics, but have little correlation with the level of 1573 adoption and deployment of a specific RFC. 1575 This document analyses a small sample of RFC "in depth". This 1576 allowed gathering of detailed feedback on the process and the 1577 deployments. On the other hand, much of the data on delays is 1578 available from the data tracker. It may be worth considering adding 1579 an automated reporting of delay metrics in the data tracker. 1581 This document only considers the RFC that were published in a given 1582 year. This approach can be criticized as introducing a form of 1583 "survivor bias". There are many drafts proposed to the IETF, and 1584 only a fraction of them end up being published as RFC. On one hand 1585 this is expected, because part of the process is to triage between 1586 ideas that can gather consensus and those that don't. On the other 1587 hand, we don't know whether that triage is too drastic and 1588 discouraged progress on good ideas. 1590 One way to evaluate the triage process would be to look at 1591 publication attempts that were abandoned, for example drafts that 1592 expired without progressing or being replaced. The sampling 1593 methodology could also be used for that purpose. Pick maybe 20 1594 drafts at random, among those abandoned in a target year, and 1595 investigate why they were abandoned. Was it because better solutions 1596 emerged in the working group? Or maybe because the authors 1597 discovered a flaw in their proposal? Or was it because some 1598 factional struggle blocked a good idea? Was the idea pursued in a 1599 different venue? Hopefully, someone will try this kind of 1600 investigation. 1602 7. Security considerations 1604 This draft does not specify any protocol. 1606 We might want to analyze whether security issues were discovered 1607 after publication of specific standards. 1609 8. IANA Considerations 1611 This draft does not require any IANA action. 1613 Peliminary analysis does not indicate that IANA is causing any 1614 particular delay in the RFC publication process. 1616 9. Acknowledgements 1618 Many thanks to the authors of the selected RFC who were willing to 1619 provide feedback on the process: Michael Ackermann, Zafar Ali, Sarah 1620 Banks, Bruno Decraene, Lars Eggert, Nalini Elkins, Joachim Fabini, 1621 Dino Farinacci, Clarence Filsfils, Sujay Gupta, Dan Harkins, Vinayak 1622 Hegde, Benjamin Kaduk, John Klensin, Acee Lindem, Nikos 1623 Mavrogiannopoulos, Patrick McManus, Victor Moreno, Al Morton, Andrei 1624 Popov, Eric Rescorla, Michiko Short, Bhuvaneswaran Vengainathan, Lao 1625 Weiguo, and Li Yizhou. Many thanks to Adrian Farrel for his useful 1626 advice, and to Stephen Farrel and Colin Perkins for their guidance on 1627 the use of citations. 1629 10. Informative References 1631 [RFC2267] Ferguson, P. and D. Senie, "Network Ingress Filtering: 1632 Defeating Denial of Service Attacks which employ IP Source 1633 Address Spoofing", RFC 2267, DOI 10.17487/RFC2267, January 1634 1998, . 1636 [RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis, 1637 "Framework for IP Performance Metrics", RFC 2330, 1638 DOI 10.17487/RFC2330, May 1998, 1639 . 1641 [RFC2827] Ferguson, P. and D. Senie, "Network Ingress Filtering: 1642 Defeating Denial of Service Attacks which employ IP Source 1643 Address Spoofing", BCP 38, RFC 2827, DOI 10.17487/RFC2827, 1644 May 2000, . 1646 [RFC3797] Eastlake 3rd, D., "Publicly Verifiable Nominations 1647 Committee (NomCom) Random Selection", RFC 3797, 1648 DOI 10.17487/RFC3797, June 2004, 1649 . 1651 [RFC5326] Ramadas, M., Burleigh, S., and S. Farrell, "Licklider 1652 Transmission Protocol - Specification", RFC 5326, 1653 DOI 10.17487/RFC5326, September 2008, 1654 . 1656 [RFC6455] Fette, I. and A. Melnikov, "The WebSocket Protocol", 1657 RFC 6455, DOI 10.17487/RFC6455, December 2011, 1658 . 1660 [RFC8312] Rhee, I., Xu, L., Ha, S., Zimmermann, A., Eggert, L., and 1661 R. Scheffenegger, "CUBIC for Fast Long-Distance Networks", 1662 RFC 8312, DOI 10.17487/RFC8312, February 2018, 1663 . 1665 [RFC8324] Klensin, J., "DNS Privacy, Authorization, Special Uses, 1666 Encoding, Characters, Matching, and Root Structure: Time 1667 for Another Look?", RFC 8324, DOI 10.17487/RFC8324, 1668 February 2018, . 1670 [RFC8355] Filsfils, C., Ed., Previdi, S., Ed., Decraene, B., and R. 1671 Shakir, "Resiliency Use Cases in Source Packet Routing in 1672 Networking (SPRING) Networks", RFC 8355, 1673 DOI 10.17487/RFC8355, March 2018, 1674 . 1676 [RFC8361] Hao, W., Li, Y., Durrani, M., Gupta, S., and A. Qu, 1677 "Transparent Interconnection of Lots of Links (TRILL): 1678 Centralized Replication for Active-Active Broadcast, 1679 Unknown Unicast, and Multicast (BUM) Traffic", RFC 8361, 1680 DOI 10.17487/RFC8361, April 2018, 1681 . 1683 [RFC8362] Lindem, A., Roy, A., Goethals, D., Reddy Vallem, V., and 1684 F. Baker, "OSPFv3 Link State Advertisement (LSA) 1685 Extensibility", RFC 8362, DOI 10.17487/RFC8362, April 1686 2018, . 1688 [RFC8377] Eastlake 3rd, D., Zhang, M., and A. Banerjee, "Transparent 1689 Interconnection of Lots of Links (TRILL): Multi-Topology", 1690 RFC 8377, DOI 10.17487/RFC8377, July 2018, 1691 . 1693 [RFC8378] Moreno, V. and D. Farinacci, "Signal-Free Locator/ID 1694 Separation Protocol (LISP) Multicast", RFC 8378, 1695 DOI 10.17487/RFC8378, May 2018, 1696 . 1698 [RFC8410] Josefsson, S. and J. Schaad, "Algorithm Identifiers for 1699 Ed25519, Ed448, X25519, and X448 for Use in the Internet 1700 X.509 Public Key Infrastructure", RFC 8410, 1701 DOI 10.17487/RFC8410, August 2018, 1702 . 1704 [RFC8411] Schaad, J. and R. Andrews, "IANA Registration for the 1705 Cryptographic Algorithm Object Identifier Range", 1706 RFC 8411, DOI 10.17487/RFC8411, August 2018, 1707 . 1709 [RFC8429] Kaduk, B. and M. Short, "Deprecate Triple-DES (3DES) and 1710 RC4 in Kerberos", BCP 218, RFC 8429, DOI 10.17487/RFC8429, 1711 October 2018, . 1713 [RFC8441] McManus, P., "Bootstrapping WebSockets with HTTP/2", 1714 RFC 8441, DOI 10.17487/RFC8441, September 2018, 1715 . 1717 [RFC8446] Rescorla, E., "The Transport Layer Security (TLS) Protocol 1718 Version 1.3", RFC 8446, DOI 10.17487/RFC8446, August 2018, 1719 . 1721 [RFC8453] Ceccarelli, D., Ed. and Y. Lee, Ed., "Framework for 1722 Abstraction and Control of TE Networks (ACTN)", RFC 8453, 1723 DOI 10.17487/RFC8453, August 2018, 1724 . 1726 [RFC8455] Bhuvaneswaran, V., Basil, A., Tassinari, M., Manral, V., 1727 and S. Banks, "Terminology for Benchmarking Software- 1728 Defined Networking (SDN) Controller Performance", 1729 RFC 8455, DOI 10.17487/RFC8455, October 2018, 1730 . 1732 [RFC8456] Bhuvaneswaran, V., Basil, A., Tassinari, M., Manral, V., 1733 and S. Banks, "Benchmarking Methodology for Software- 1734 Defined Networking (SDN) Controller Performance", 1735 RFC 8456, DOI 10.17487/RFC8456, October 2018, 1736 . 1738 [RFC8466] Wen, B., Fioccola, G., Ed., Xie, C., and L. Jalil, "A YANG 1739 Data Model for Layer 2 Virtual Private Network (L2VPN) 1740 Service Delivery", RFC 8466, DOI 10.17487/RFC8466, October 1741 2018, . 1743 [rfc8468] Morton, A., Fabini, J., Elkins, N., Ackermann, M., and V. 1744 Hegde, "IPv4, IPv6, and IPv4-IPv6 Coexistence: Updates for 1745 the IP Performance Metrics (IPPM) Framework", RFC 8468, 1746 DOI 10.17487/RFC8468, November 2018, 1747 . 1749 [RFC8468] Morton, A., Fabini, J., Elkins, N., Ackermann, M., and V. 1750 Hegde, "IPv4, IPv6, and IPv4-IPv6 Coexistence: Updates for 1751 the IP Performance Metrics (IPPM) Framework", RFC 8468, 1752 DOI 10.17487/RFC8468, November 2018, 1753 . 1755 [RFC8472] Popov, A., Ed., Nystroem, M., and D. Balfanz, "Transport 1756 Layer Security (TLS) Extension for Token Binding Protocol 1757 Negotiation", RFC 8472, DOI 10.17487/RFC8472, October 1758 2018, . 1760 [RFC8479] Mavrogiannopoulos, N., "Storing Validation Parameters in 1761 PKCS#8", RFC 8479, DOI 10.17487/RFC8479, September 2018, 1762 . 1764 [RFC8492] Harkins, D., Ed., "Secure Password Ciphersuites for 1765 Transport Layer Security (TLS)", RFC 8492, 1766 DOI 10.17487/RFC8492, February 2019, 1767 . 1769 [RFC8498] Mohali, M., "A P-Served-User Header Field Parameter for an 1770 Originating Call Diversion (CDIV) Session Case in the 1771 Session Initiation Protocol (SIP)", RFC 8498, 1772 DOI 10.17487/RFC8498, February 2019, 1773 . 1775 Author's Address 1777 Christian Huitema 1778 Private Octopus Inc. 1779 427 Golfcourse Rd 1780 Friday Harbor WA 98250 1781 U.S.A 1783 Email: huitema@huitema.net