idnits 2.17.1
draft-huitema-rfc-eval-project-05.txt:
Checking boilerplate required by RFC 5378 and the IETF Trust (see
https://trustee.ietf.org/license-info):
----------------------------------------------------------------------------
No issues found here.
Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt:
----------------------------------------------------------------------------
No issues found here.
Checking nits according to https://www.ietf.org/id-info/checklist :
----------------------------------------------------------------------------
No issues found here.
Miscellaneous warnings:
----------------------------------------------------------------------------
== The copyright year in the IETF Trust and authors Copyright Line does not
match the current year
-- The document date (April 5, 2020) is 1475 days in the past. Is this
intentional?
Checking references for intended status: Informational
----------------------------------------------------------------------------
-- Obsolete informational reference (is this intentional?): RFC 2267
(Obsoleted by RFC 2827)
-- Obsolete informational reference (is this intentional?): RFC 8312
(Obsoleted by RFC 9438)
Summary: 0 errors (**), 0 flaws (~~), 1 warning (==), 3 comments (--).
Run idnits with the --verbose option for more detailed information about
the items above.
--------------------------------------------------------------------------------
2 Network Working Group C. Huitema
3 Internet-Draft Private Octopus Inc.
4 Intended status: Informational April 5, 2020
5 Expires: October 7, 2020
7 Evaluation of a Sample of RFC Produced in 2018
8 draft-huitema-rfc-eval-project-05
10 Abstract
12 This document presents the author's effort to understand the delays
13 involved in publishing an idea in the IETF, from the first individual
14 draft to the publication of the RFC. We analyze a set of randomly
15 chosen RFC approved in 2018, looking for history and delays. We also
16 use two randomly chosen sets of RFC published in 2008 and 1998 for
17 comparing delays seen in 2018 to those observed 10 or 20 years ago.
18 The average RFC in the 2018 sample was produced in 3 years and 4
19 months, of which 2 years and 10 months were spent in the working
20 group, 3 to 4 months for IETF consensus and IESG review, and 3 to 4
21 months in RFC production. The main variation in RFC production
22 delays comes from the AUTH-48 phase.
24 We also measure the number of citations of the chosen RFC using
25 Semantic Scholar, and compare citation counts with what we know about
26 deployment. We show that citation counts indicate academic interest,
27 but correlate only loosely with deployment or usage of the
28 specifications. Counting web references could complement that.
30 The RFCs selected for this survey were chosen at random and represent
31 a small sample of all RFCs produced, and only approximately 10% of
32 the RFCs produced in each of 1998, 2008, and 2018. It is possible
33 that different samples would produce different results. Furthermore,
34 the conclusions drawn from the observations made in this document
35 represent the author's opinions and do not have consensus of the
36 IETF.
38 Status of This Memo
40 This Internet-Draft is submitted in full conformance with the
41 provisions of BCP 78 and BCP 79.
43 Internet-Drafts are working documents of the Internet Engineering
44 Task Force (IETF). Note that other groups may also distribute
45 working documents as Internet-Drafts. The list of current Internet-
46 Drafts is at https://datatracker.ietf.org/drafts/current/.
48 Internet-Drafts are draft documents valid for a maximum of six months
49 and may be updated, replaced, or obsoleted by other documents at any
50 time. It is inappropriate to use Internet-Drafts as reference
51 material or to cite them other than as "work in progress."
53 This Internet-Draft will expire on October 7, 2020.
55 Copyright Notice
57 Copyright (c) 2020 IETF Trust and the persons identified as the
58 document authors. All rights reserved.
60 This document is subject to BCP 78 and the IETF Trust's Legal
61 Provisions Relating to IETF Documents
62 (https://trustee.ietf.org/license-info) in effect on the date of
63 publication of this document. Please review these documents
64 carefully, as they describe your rights and restrictions with respect
65 to this document. Code Components extracted from this document must
66 include Simplified BSD License text as described in Section 4.e of
67 the Trust Legal Provisions and are provided without warranty as
68 described in the Simplified BSD License.
70 Table of Contents
72 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3
73 2. Methodology . . . . . . . . . . . . . . . . . . . . . . . . . 4
74 2.1. Defining the Important Milestones . . . . . . . . . . . . 5
75 2.2. Selecting a Random Sample of RFC . . . . . . . . . . . . 6
76 3. Analysis of 20 Selected RFC . . . . . . . . . . . . . . . . . 6
77 3.1. 8411 . . . . . . . . . . . . . . . . . . . . . . . . . . 6
78 3.2. 8456 . . . . . . . . . . . . . . . . . . . . . . . . . . 7
79 3.3. 8446 . . . . . . . . . . . . . . . . . . . . . . . . . . 8
80 3.4. 8355 . . . . . . . . . . . . . . . . . . . . . . . . . . 9
81 3.5. 8441 . . . . . . . . . . . . . . . . . . . . . . . . . . 9
82 3.6. 8324 . . . . . . . . . . . . . . . . . . . . . . . . . . 10
83 3.7. 8377 . . . . . . . . . . . . . . . . . . . . . . . . . . 11
84 3.8. 8498 . . . . . . . . . . . . . . . . . . . . . . . . . . 11
85 3.9. 8479 . . . . . . . . . . . . . . . . . . . . . . . . . . 12
86 3.10. 8453 . . . . . . . . . . . . . . . . . . . . . . . . . . 12
87 3.11. 8429 . . . . . . . . . . . . . . . . . . . . . . . . . . 13
88 3.12. 8312 . . . . . . . . . . . . . . . . . . . . . . . . . . 13
89 3.13. 8492 . . . . . . . . . . . . . . . . . . . . . . . . . . 14
90 3.14. 8378 . . . . . . . . . . . . . . . . . . . . . . . . . . 15
91 3.15. 8361 . . . . . . . . . . . . . . . . . . . . . . . . . . 16
92 3.16. 8472 . . . . . . . . . . . . . . . . . . . . . . . . . . 16
93 3.17. 8471 . . . . . . . . . . . . . . . . . . . . . . . . . . 17
94 3.18. 8466 . . . . . . . . . . . . . . . . . . . . . . . . . . 17
95 3.19. 8362 . . . . . . . . . . . . . . . . . . . . . . . . . . 18
96 3.20. 8468 . . . . . . . . . . . . . . . . . . . . . . . . . . 18
97 4. Analysis of Process and Delays . . . . . . . . . . . . . . . 19
98 4.1. First Draft to RFC Delays . . . . . . . . . . . . . . . . 20
99 4.2. Working Group Processing Time . . . . . . . . . . . . . . 25
100 4.3. Preparation and Publication Delays . . . . . . . . . . . 28
101 4.4. Copy Editing . . . . . . . . . . . . . . . . . . . . . . 31
102 4.5. Independent Series . . . . . . . . . . . . . . . . . . . 34
103 5. Citation Counts . . . . . . . . . . . . . . . . . . . . . . . 34
104 5.1. Citation Numbers . . . . . . . . . . . . . . . . . . . . 35
105 5.2. Comparison to 1998 and 2008 . . . . . . . . . . . . . . . 37
106 5.3. Citations Versus Deployments . . . . . . . . . . . . . . 40
107 5.4. Citations versus web references . . . . . . . . . . . . . 42
108 6. Observations and Next Steps . . . . . . . . . . . . . . . . . 44
109 7. Security Considerations . . . . . . . . . . . . . . . . . . . 46
110 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 46
111 9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 46
112 10. Informative References . . . . . . . . . . . . . . . . . . . 46
113 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 50
115 1. Introduction
117 As stated on the organization's web site, "The IETF is a large open
118 international community of network designers, operators, vendors, and
119 researchers concerned with the evolution of the Internet architecture
120 and the smooth operation of the Internet." The specifications
121 produced by the IETF are published in the RFC series, along with
122 independent submissions, research papers and IAB documents. In this
123 memo, the author attempts to understand the delays involved in
124 publishing an idea in the IETF, from the first individual draft to
125 the publication of the RFC. This is an individual effort, and the
126 author's conclusions presented here are personal. There was no
127 attempt to seek IETF consensus.
129 The IETF keeps records of documents and process actions in the IETF
130 data tracker [TRKR]. The IETF data tracker provides information
131 about RFC and drafts, from which we can infer statistics about the
132 production system. We can measure how long it takes to drive a
133 proposition from initial draft to final publication, and how these
134 delays can be split between Working Group discussions, IETF reviews,
135 IESG assessment, RFC Editor delays and final reviews by the authors -
136 or, for independent stream RFCs, draft production, reviews by the
137 Independent Stream Editor, conflict reviews, RFC Editor delays and
138 final reviews. Tracker data is available for all RFCs, not just IETF
139 stream RFCs.
141 Just measuring production delays may be misleading. If the IETF or
142 the editors of the other series simply rubber-stamped draft proposals
143 and published them, the delays would be short but the quality and
144 impact might suffer. We hope that most of the RFC that are published
145 are useful, but we need a way to measure that usefulness. We try to
146 do that by measuring the number of references of the published RFCs
147 in Semantic Scholar [SSCH], and also by asking the authors of each
148 RFC in the sample whether the protocols and technologies defined in
149 the RFCs were implemented and used on the Internet. The citations
150 measured by the Semantic Scholar include citations in other RFC and
151 in Internet drafts. We also measure the number of references on the
152 web, which provides some results but would be hard to automate.
154 In order to limit the resource required for this study, we selected
155 at random 20 RFC published in 2018, as explained in Section 2.2. The
156 statistical sampling picked both IETF stream and Independent stream
157 documents. For comparison purposes, we also selected at random 20
158 RFC published in 1998 and 20 published in 2008. Limiting the sample
159 to 20 out of 209 RFCs published in 2018 allows for in depth analysis
160 of each RFC, but readers should be reminded that the this is a small
161 sample. The sample is too small to apply general statistical
162 techniques and quantify specific ratios, and discussions of
163 correlation techniques would be inappropriate. Instead, the purpose
164 is to identify trends, spot issues and document future work.
166 The information gathered for every RFC in the sample is presented in
167 Section 3. In Section 4 we analyze the production process and the
168 sources of delays, comparing the 2018 sample to the selected samples
169 for 1998 and 2018. In Section 5.1 we present citation counts for the
170 RFC in the samples, and analyze whether citation counts could be used
171 to evaluate the quality of RFC.
173 The measurement of delays could be automated by processing dates and
174 events recorded in the data tracker. The measurement of published
175 RFC could be complemented by statistics on abandoned drafts, which
176 would measure the efficiency of the IETF triaging process. More
177 instrumentation would help understanding how large delays happen
178 during working group processes. These potential next steps are
179 developed in Section 6.
181 2. Methodology
183 The study reported here started with a simple idea: take a sample of
184 RFC, and perform an in-depth analysis of the path from the first
185 presentation of the idea to its publication, while also trying to
186 access the success of the resulting specification. This requires
187 defining the key milestones that we want to track, and drawing a
188 random sample using an unbiased process.
190 2.1. Defining the Important Milestones
192 The IETF data tracker records a list of events for each document
193 processed by IETF working groups. This has a high granularity, and
194 also a high variability. Most documents start life as an individual
195 draft, are adopted by a working group, undergo a working group last
196 call, are submitted to the IESG, undergo an IETF last call and an
197 IESG review, get eventually approved by the IESG, and are processed
198 for publication by the RFC Editor, but there are exceptions. Some
199 documents are first submitted to one working group and then moved to
200 another. Some documents are published through the Independent
201 Stream, and are submitted to the Independent Stream Editor instead of
202 the IESG.
204 In order to simplify tabulation, we break the delay from between the
205 submission of the first draft and the publication of the RFC in three
206 big components:
208 o The working group processing time, from the first draft to the
209 start of the IETF last call;
211 o The IETF processing time, which lasts from the beginning of the
212 IETF last call to the approval by the IESG, including the reviews
213 by various directorates;
215 o The RFC production, from approval by the IESG to publication,
216 including the AUTH-48 reviews.
218 For submissions to the independent stream, we don't have a working
219 group. We consider instead the progression of the individual draft
220 until the adoption by the ISE as the equivalent of the "working
221 group" period, and the delay from adoption by the ISE until
222 submission to the RFC Editor as the equivalent of the IETF delay.
224 We measure the staring point of the process using the date of
225 submission of the first draft listed on that RFC page in the IETF
226 data tracker. In most case, this first draft is an individual draft
227 that then resubmitted as a working group draft, or maybe resubmitted
228 with a new name as the draft was searching for a home in an IETF
229 working group, or before deciding for submission on the independent
230 stream.
232 The IETF Data Tracker entries for RFC and drafts do not list working
233 group events like Working Group Last Call. The only intermediate
234 event that we list between the first draft and the submission to the
235 IESG is the Working Group Adoption. For that, we use the date of
236 submission of the version 00 of the draft eventually published as
237 RFC. We use the same definition for drafts submitted to the
238 Independent Stream.
240 2.2. Selecting a Random Sample of RFC
242 Basic production mechanisms could be evaluated by processing data
243 from the IETF data tracker, but subjective data requires manual
244 assessment of results, which can be time consuming. Since our
245 resources are limited, we will only perform this analysis for a small
246 sample of RFC, selected at random from the list of RFC approved in
247 2018. Specifically, we will pick 20 RFC numbers at random between:
249 o RFC 8307, published in January 2018, and
251 o RFC 8511, published December 2018.
253 The list of 20 selected RFC is: RFC 8411, RFC 8456, RFC 8446, RFC
254 8355, RFC 8441, RFC 8324, RFC 8377, RFC 8498, RFC 8479, RFC 8453, RFC
255 8429, RFC 8312, RFC 8492 , RFC 8378, RFC 8361, RFC 8472, RFC 8471,
256 RFC 8466, RFC 8362, and RFC 8468.
258 When evaluating delays and impact, we will compare the year 2018 to
259 2008 and 1998, 10 and 20 years ago. To drive this comparison, we
260 pick 20 RFC at random among those published in 2008, and another 20
261 among those published in 1998.
263 The list of the 20 randomly selected RFC from 2008 is: RFC 5227, RFC
264 5174, RFC 5172, RFC 5354, RFC 5195, RFC 5236, RFC 5348, RFC 5281, RFC
265 5186, RFC 5326, RFC 5277, RFC 5373, RFC 5404, RFC 5329, RFC 5283, RFC
266 5358, RFC 5142, RFC 5271, RFC 5349, and RFC 5301.
268 The list of the 20 randomly selected RFC from 2008 is: RFC 2431, RFC
269 2381, RFC 2387, RFC 2348, RFC 2391, RFC 2267, RFC 2312, RFC 2448, RFC
270 2374, RFC 2398, RFC 2283, RFC 2382, RFC 2289, RFC 2282, RFC 2404, RFC
271 2449, RFC 2317, RFC 2394, RFC 2297, and RFC 2323.
273 3. Analysis of 20 Selected RFC
275 We review each of the RFC listed in Section 2.2 for the year 2018,
276 trying both to answer the known questions and to gather insight for
277 further analyzes. In many cases, the analysis of the data is
278 complemented by direct feedback from the RFC authors.
280 3.1. 8411
282 IANA Registration for the Cryptographic Algorithm Object Identifier
283 Range [RFC8411]:
285 Informational, 5 pages
286 4 drafts (personal), first 2017-05-08.
287 Last call announced 2017-10-09
288 IESG evaluation starts 2017-12-28
289 Approved 2018-02-26, draft 03
290 AUTH-48 2018-04-20
291 AUTH-48 complete 2018-07-17
292 Published 2018-08-06
293 IANA action: create table
295 This RFC was published from the individual draft, which was not
296 resubmitted as a working group draft.
298 The draft underwent minor copy edit before publication.
300 Some but not all of the long delay in AUTH-48 is due to clustering
301 with [RFC8410]. MISSREF was cleared on 2018-05-09 and the document
302 re-entered AUTH-48 at once. AUTH-48 lasted over two months after
303 that.
305 The time after AUTH-48 and before publication (3 weeks) partly
306 overlaps with travel for IETF-102 and is partly due to coordinating
307 the cluster.
309 3.2. 8456
311 Benchmarking Methodology for Software-Defined Networking (SDN)
312 Controller Performance [RFC8456]:
314 Informational, 64 pages
315 2 personal drafts, 9 WG drafts, first 2015-03-23
316 WG adoption on 2015-10-18
317 Last call announced 2018-01-19
318 IESG evaluation starts 2018-02-27
319 IESG approved 2018-05-25
320 AUTH-48 2018-08-31
321 AUTH-48 complete 2018-10-16
322 Published 2018-10-30
324 The draft underwent very extensive copy editing, covering use of
325 articles, turn of phrases, choice of vocabulary. The changes are
326 enough to cause pagination differences. The "diff" tool marks pretty
327 much every page as changed. Some diagrams see change in protocol
328 elements like message names.
330 According to the author, the experience of producing this draft
331 mirrors a typical one in the Benchmarking Methodologies Working Group
332 (BMWG). There were multiple authors in multiple time zones, which
333 slowed down the AUTH-48 process somewhat, although the AUTH-48 delay
334 of 46 days is only a bit longer than the average draft.
336 The RFC was part of cluster with [RFC8455].
338 BMWG publishes informational RFCs centered around benchmarking, and
339 the methodologies in RFC 8456 have been implemented in benchmarking
340 products.
342 3.3. 8446
344 The Transport Layer Security (TLS) Protocol Version 1.3 [RFC8446], as
345 the title indicates, defines the new version of the TLS protocol.
346 From the IETF data tracker, we extract the following:
348 Proposed standard
349 160 pages
350 29 WG drafts first 2014-04-17.
351 Last call announced 2018-02-15
352 IESG evaluation starts 2018-03-02
353 Approved 2018-03-21, draft 28
354 AUTH-48 2018-06-14
355 AUTH-48 complete 2018-08-10
356 Published 2018-08-10
358 This draft started as a WG effort.
360 The RFC was a major effort in the IETF. Working group members
361 developed and tested several implementations. Researchers analyzed
362 the specifications and performed formal verifications. Deployment
363 tests outlined issues that caused extra work when the specification
364 was almost ready. These complexity largely explains the time spent
365 in the working group.
367 Comparing the final draft to the published version, we find
368 relatively light copy editing. It includes explaining acronyms on
369 first use, clarifying some definitions standardizing punctiation and
370 capitalization, and spelling out some numbers in text. This
371 generally fall in the category of "style", although some of the
372 clarifications go into message definitions. However, that simple
373 analysis does not explain why the AUTH-48 phase took almost two
374 months.
376 This document's AUTH-48 process was part of the "Github experiment",
377 which tried to use github pull requests to track the AUTH-48 changes
378 and review comments. The RPC staff had to learn using Github for
379 that process, and this required more work than the usual RFC. Author
380 and AD thoroughly reviewed each proposed edit, accepting some and
381 rejecting some. The concern there was that any change in a complex
382 specification might affect a protocol that was extensively reviewed
383 in the working group, but of course these reviews added time to the
384 AUTH-48 delays.
386 There are 21 implementations listed in the Wiki of the TLS 1.3
387 project [TLS13IMP]. It has been deployed on major browsers, and is
388 already used in a large fraction of TLS connections.
390 3.4. 8355
392 Resiliency Use Cases in Source Packet Routing in Networking (SPRING)
393 Networks [RFC8355] is an informational RFC. It originated from a use
394 case informational draft that was mostly used for the BOF creating
395 the WG, and then to drive initial work/evolutions from the WG.
397 Informational, 13 pages.
398 2 personal drafts (personal), first 2014-01-31. 13 WG drafts.
399 WG adoption on 2014-05-13
400 Last call announced 2017-04-20
401 IESG evaluation starts 2017-05-04, draft 09
402 Approved 2017-12-19, draft 12
403 AUTH-48 2018-03-12
404 AUTH-48 complete 2018-03-27
405 Published 2018-03-28
407 Minor set of copy edits, mostly for style.
409 No implementation of the RFC itself, but the technology behind it
410 such as Segment Routing (architecture RFC 8402, TI-LFA draft-ietf-
411 rtgwg-segment-routing-ti-lfa) is widely implemented and deployment is
412 ongoing.
414 According to participants in the discussion, the process of adoption
415 of the source packet routing standards was very contentious. The
416 establishment of consensus at both the working group level and the
417 IETF level was difficult and time consuming.
419 3.5. 8441
421 Bootstrapping WebSockets with HTTP/2 [RFC8441]
422 Proposed standard, 8 pages. Updates RFC 6455.
423 3 personal drafts (personal), first 2017-10-15. 8 WG drafts.
424 WG adoption on 2017-12-19
425 Last call announced 2018-05-07, draft 05
426 IESG evaluation starts 2018-05-29, draft 06
427 Approved 2018-06-07, draft 07
428 AUTH-48 2018-08-13
429 AUTH-48 complete 2018-09-15
430 Published 2018-09-21
431 IANA Action: table entries
433 This RFC defines the support of WebSockets in HTTP/2, which is
434 different from the mechanism defined for HTTP/1.1 in [RFC6455]. The
435 process was relatively straightforward, involving the usual type of
436 discussions, some on details and some on important points.
438 Comparing final draft and published RFC shows a minor set of copy
439 edit, mostly for style. However, the author recalls a painful
440 process. The RFC includes many charts and graphs that were very
441 difficult to format correctly in the author's production process that
442 involve conversions from markdown to XML, and then from XML to text.
443 The author had to get substantial help from the RFC editor.
445 There are several implementations, including Firefox and Chrome,
446 making RFC 8441 a very successful specification.
448 3.6. 8324
450 DNS Privacy, Authorization, Special Uses, Encoding, Characters,
451 Matching, and Root Structure: Time for Another Look? [RFC8324].
452 This is an opinion piece on DNS development, published on the
453 Independent Stream.
455 Informational, 29 pages. Independent stream.
456 5 personal drafts (personal), first 2017-06-02.
457 ISE review started 2017-07-10, draft 03
458 IETF conflict review and IESG review started 2017-10-29
459 Approved 2017-12-18, draft 04
460 AUTH-48 2018-01-29, draft 05
461 AUTH-48 complete 2018-02-26
462 Published 2018-02-27
464 This RFC took only 9 months from first draft to publication, which is
465 the shortest in the 2018 sample set. In part, this is because the
466 text was privately circulated and reviewed by ISE designated experts
467 before the first draft was published. The nature of the document is
468 another reason for the short delay. It is an opinion piece, and does
469 not require the same type of consensus building and reviews than a
470 protocol specification.
472 Comparing the final draft and the published version shows only minor
473 copy edit, mostly for style. According to the author, because this
474 is because he knows how to write in RFC Style with the result that
475 his documents often need a minimum of editing. He also makes sure
476 that the document on which the Production Center starts working
477 already has changes discussed and approved during Last Call and IESG
478 review incorporated rather than expecting the Production Center to
479 operate off of notes about changed to be made.
481 3.7. 8377
483 Transparent Interconnection of Lots of Links (TRILL): Multi-Topology
484 [RFC8377]
486 Proposed standard, 20 pages. Updates RFC 6325, 7177.
487 3 personal drafts (personal), first 2013-09-03. 7 WG drafts.
488 WG adoption on 2015-09-01
489 Last call announced 2018-02-19, draft 05
490 IESG evaluation starts 2018-03-02, draft 06
491 Approved 2018-03-12, draft 05
492 AUTH-48 2018-04-20, draft 06
493 AUTH-48 complete 2018-07-31
494 Published 2018-07-31
495 IANA Table, table entries
497 Minor set of copy edits, mostly for style, also clarity.
499 3.8. 8498
501 A P-Served-User Header Field Parameter for an Originating Call
502 Diversion (CDIV) Session Case in the Session Initiation Protocol
503 (SIP) [RFC8498].
505 Informational, 15 pages.
506 5 personal drafts (personal), first 2016-03-21. 9 WG drafts.
507 WG adoption on 2017-05-15
508 Last call announced 2018-10-12, draft 05
509 IESG evaluation starts 2018-11-28, draft 07
510 Approved 2018-12-10, draft 08
511 AUTH-48 2019-01-28
512 AUTH-48 complete 2019-02-13
513 Published 2019-02-15
514 IANA Action, table rows added.
516 Copy edit for style, but also clarification of ambiguous sentences.
518 3.9. 8479
520 Storing Validation Parameters in PKCS#8 [RFC8479]
522 Informational, 8 pages. Independent stream.
523 5 personal drafts (personal), first 2017-08-08.
524 ISE review started 2018-12-10, draft 00
525 IETF conflict review and IESG review started 2018-03-29
526 Approved 2018-08-20, draft 03
527 AUTH-48 2018-09-20, draft 04
528 AUTH-48 complete 2018-09-25
529 Published 2018-09-26
531 The goal of the draft was to document what the gnutls implementation
532 was using for storing provably generated RSA keys. This is a short
533 RFC that was published relatively quickly, although discussion
534 between the author, the Independent Series Editor and the IESG lasted
535 several months. In the initial conflict review, The IESG asked the
536 ISE to not publish this document before IETF working groups had an
537 opportunity to pick up the work. The author met that requirement by
538 a presentation to the SECDISPATCH WG in IETF 102. Since no WG was
539 interested in pickup the work, the document progressed on the
540 Independent Stream.
542 Very minor set of copy edit, moving some references from normative to
543 informative.
545 The author is not aware of other implementations than gnutls relying
546 on this RFC.
548 3.10. 8453
550 Framework for Abstraction and Control of TE Networks (ACTN) [RFC8453]
552 Informational, 42 pages.
553 3 personal drafts, first 2015-06-15. 16 WG drafts.
554 WG adoption on 2016-07-15
555 Out of WG 2018-01-26, draft 11
556 Expert review requested, 2018-02-13
557 Last call announced 2018-04-16, draft 13
558 IESG evaluation starts 2018-05-16, draft 14
559 Approved 2018-06-01, draft 15
560 AUTH-48 2018-08-13
561 AUTH-48 complete 2018-08-20
562 Published 2018-08-20
563 IANA Action, table rows added.
565 Minor copy editing.
567 3.11. 8429
569 Deprecate Triple-DES (3DES) and RC4 in Kerberos [RFC8429]
571 BCP, 10 pages.
572 6 WG drafts, first 2017-05-01.
573 Last call announced 2017-07-16, draft 03
574 IESG evaluation starts 2017-08-18, draft 04
575 Approved 2018-05-25, draft 05
576 AUTH-48 2018-07-24
577 AUTH-48 complete 2018-10-31
578 Published 2018-10-31
579 IANA Action, table rows added.
581 This draft started as a working group effort.
583 This RFC recommends to deprecate two encryption algorithms that are
584 now considered obsolete and possibly broken. The document was sent
585 back to the WG after the first last call, edited, and then there was
586 a second last call. The delay from first draft to working group last
587 call was relatively short, but the number may be misleading. The
588 initial draft was a replacement of a similar draft in the KITTEN
589 working group, which stagnated for some time before the CURDLE
590 working group took up the work. The deprecation of RC4 was somewhat
591 contentious, but the WG had already debated this prior to the
592 production of this draft, and the draft was not delayed by this
593 debate.
595 Most of the 280 days between IETF LC and IESG approval was because
596 the IESG had to talk about whether this document should obsolete or
597 move to historic RFC 4757, and no one was really actively pushing
598 that discussion for a while.
600 The 99 days in AUTH-48 are mostly because one of the authors was a
601 sitting AD, and those duties ended up taking precedence over
602 reviewing this document.
604 Minor copy editing, for style.
606 The implementation of the draft would be the actual removal of
607 support for 3DES and RC4 in major implementations. This is
608 happening, but very slowly.
610 3.12. 8312
612 CUBIC for Fast Long-Distance Networks [RFC8312]
613 Informational, 18 pages.
614 2 personal drafts, first 2014-09-01. 8 WG drafts
615 WG adoption on 2015-06-08
616 Last call announced 2017-09-18, draft 06
617 IESG evaluation starts 2017-11-14
618 Approved 2017-10-04, draft 07
619 AUTH-48 2018-01-08
620 AUTH-48 complete 2018-02-07
621 Published 2018-02-07
622 IANA Action, table rows added.
624 Minor copy editing, for style.
626 The TCP congestion control algorithm Cubic was defined first in 2005,
627 was implemented in Linux soon after, and was implemented in major
628 OSes after that. After some debates from 2015 to 2015, the TCPM
629 working group adopted the draft, with a goal of documenting Cubic in
630 the RFc series. According to the authors, this was not a high
631 priority effort, as Cubic was already implemented in multiple OSes
632 and documented in research papers. At some point, only one of the
633 authors was actively working on the draft. Ths may explain why
634 another two years was spent progressing the draft after adoption by
635 the WG.
637 The RFC publication may or may not have triggered further
638 implementations. On the other hand, several OSes picked up bug fixes
639 from the draft and the RFC.
641 3.13. 8492
643 Secure Password Ciphersuites for Transport Layer Security (TLS)
644 [RFC8492]
646 Informational, 40 pages. (Independent Stream)
647 10 personal drafts, first 2012-09-07. 8 WG drafts
648 Targeted to ISE stream 2016-08-05
649 ISE review started 2017-05-10, draft 01
650 IETF conflict review and IESG review started 2017-09-04
651 Approved 2017-10-29, draft 04
652 AUTH-48 2018-10-19, draft 05
653 AUTH-48 complete 2019-02-19
654 Published 2019-02-21
655 IANA Action, table rows added.
657 This RFC has a complex history. The first individual draft was
658 submitted to the TLS working group on September 7, 2012. It
659 progressed there, and was adopted by the WG after 3 revisions. There
660 were then 8 revisions in the TLS WG, until the WG decided to not
661 progress it. The draft was parked in 2013 by the WG chairs after
662 failing to get consensus in WG last call. The AD finally pulled the
663 plug in 2016, and the draft was then resubmitted to the ISE.
665 At that point, the author was busy and was treating this RFC with a
666 low priority because, in his words, it would not be a "real RFC".
667 There were problems with the draft that only came up late. In
668 particular, it had to wait for a change in registry policy that only
669 came about with the publication of TLS 1.3, which caused the draft to
670 only be published after RFC 8446, and also required adding references
671 to TLS 1.3. The author also got a very late comment while in AUTH-48
672 that caused some rewrite. Finally, there was some IANA issue with
673 the extension registry where a similar extension was added by someone
674 else. The draft was changed to just use it.
676 Changes in AUTH-48 include added reference to TLS 1.3, copy-editing
677 for style, some added requirements, added paragraphs, and changes in
678 algorithms specification.
680 3.14. 8378
682 Signal-Free Locator/ID Separation Protocol (LISP) Multicast [RFC8378]
683 is an experimental RFC, defining how to implement Multicast in the
684 LISP architecture.
686 Experimental, 21 pages.
687 5 personal drafts, first 2014-02-28. 10 WG drafts
688 WG adoption on 2015-12-21
689 Last call announced 2018-02-13, draft 07
690 IESG evaluation starts 2018-02-28, draft 08
691 Approved 2018-03-12, draft 09
692 AUTH-48 2018-04-23
693 AUTH-48 complete 2018-05-02
694 Published 2018-05-02
696 Preparing the RFC took more than 4 years. According to the authors,
697 they were not aggressive pushing it and just let the working group
698 process decide to pace it. They also did implementations during that
699 time.
701 Minor copy editing, for style.
703 The RFC was implemented by lispers.net and cisco, and was used in
704 doing IPv6 multicast over IPv4 unicast/multicast at the Olympics in
705 PyeungChang. The plan is to work on a proposedstandard once the
706 experiment concludes.
708 3.15. 8361
710 Transparent Interconnection of Lots of Links (TRILL): Centralized
711 Replication for Active-Active Broadcast, Unknown Unicast, and
712 Multicast (BUM) Traffic [RFC8361]
714 Proposed Standard, 17 pages.
715 3 personal drafts, first 2013-11-12. 14 WG drafts
716 WG adoption on 2014-12-16
717 Last call announced 2017-11-28, draft 10
718 IESG evaluation starts 2017-12-18, draft 11
719 Approved 2018-01-29, draft 13
720 AUTH-48 2018-03-09
721 AUTH-48 complete 2018-04-09
722 Published 2018-04-12
724 According to the authors, the long delays in producing this RFC was
725 due to a slow uptake of the technology in the industry.
727 Minor copy editing, for style.
729 There was at least 1 partial implementation.
731 3.16. 8472
733 Transport Layer Security (TLS) Extension for Token Binding Protocol
734 Negotiation [RFC8472]
736 Proposed Standard, 8 pages.
737 1 personal drafts, 2015-05-29. 15 WG drafts
738 WG adoption on 2015-09-11
739 Last call announced 2017-11-13, draft 10
740 IESG evaluation starts 2018-03-19
741 Approved 2018-07-20, draft 14
742 AUTH-48 2018-09-17
743 AUTH-48 complete 2018-09-25
744 Published 2018-10-08
746 This is a pretty simple document, but it took over 3 years from
747 individual draft to RFC. According to the authors,the biggest
748 setbacks occurred at the start: it took a while to find a home for
749 this draft. It was presented in the TLS WG (because it's a TLS
750 extension) and UTA WG (because it has to do with applications using
751 TLS). Then the ADs determined that a new WG was needed, so the
752 authors had to work through the WG creation process, including
753 running a BOF.
755 Minor copy editing, for style, with the addition of a reference to
756 TLS 1.3.
758 Perhaps partially due to the delays, some of the implementers lost
759 interest in supporting this RFC.
761 3.17. 8471
763 The Token Binding Protocol Version 1.0 [RFC8471]
765 Proposed Standard, 18 pages.
766 1 personal drafts, 2014-10-13. 19 WG drafts
767 WG adoption on 2015-03-15
768 Last call announced 2017-11-13, draft 16
769 IESG evaluation starts 2018-03-19
770 Approved 2018-07-20, draft 19
771 AUTH-48 2018-09-17
772 AUTH-48 complete 2018-09-25
773 Published 2018-10-08
775 Presentation of a Token Binding Protocol for TLS. We can notice a
776 delay of 5 months before adoption of the draft by the WG. That
777 explains in part the overall delay of almost 4 years from first draft
778 to publication.
780 Minor copy editing, for style.
782 The web references indicates adoption in multiple development
783 projects.
785 3.18. 8466
787 A YANG Data Model for Layer 2 Virtual Private Network (L2VPN) Service
788 Delivery [RFC8466]
790 Proposed Standard, 158 pages.
791 5 personal drafts, first 2016-09-01. 11 WG drafts
792 WG adoption on 2017-02-26
793 Last call announced 2018-02-21, draft 07
794 IESG evaluation starts 2018-03-14, draft 08
795 Approved 2018-06-25, draft 10
796 AUTH-48 2018-09-17
797 AUTH-48 complete 2018-10-09
798 Published 2018-10-12
800 Copy editing for style and clarity, with also corrections to the yang
801 model.
803 3.19. 8362
805 OSPFv3 Link State Advertisement (LSA) Extensibility [RFC8362] is a
806 major extension to the OSPF protocol. It makes OSPFv3 fully
807 extensible.
809 Proposed Standard, 33 pages.
810 4 personal drafts, first 2013-02-17. 24 WG drafts
811 WG adoption on 2013-10-15
812 Last call announced 2017-12-19, draft 19
813 IESG evaluation starts 2018-01-18, draft 20
814 Approved 2018-01-29, draft 23
815 AUTH-48 2018-03-19
816 AUTH-48 complete 2018-03-30
817 Published 2018-04-03
819 The specification was first submitted as a personal draft in the IPv6
820 WG, then moved to the OSPF WG. The long delay of producing this RFC
821 is due to the complexity of the problem, and the need to wait for
822 implementations. It is a very important change to OSPF that makes
823 OSPFv3 fully extensible. Since it was a non-backward compatible
824 change, the developers started out with some very complex migration
825 scenarios but ended up with either legacy or extended OSPFv3 LSAs
826 within an OSPFv3 routing domain. The initial attempts to have a
827 hybrid mode of operation with both legacy and extended LSAs also
828 delayed implementation due to the complexity.
830 Copy editing for style and clarity.
832 This specification either was or will be implemented by all the
833 router vendors.
835 3.20. 8468
837 IPv4, IPv6, and IPv4-IPv6 Coexistence: Updates for the IP Performance
838 Metrics (IPPM) Framework [rfc8468].
840 Informational, 15 pages.
841 3 personal drafts, first 2015-08-06. 7 WG drafts
842 WG adoption on 2016-07-04
843 Last call announced 2018-04-11, draft 04
844 IESG evaluation starts 2018-05-24, draft 05
845 Approved 2018-07-10, draft 06
846 AUTH-48 2018-09-13
847 AUTH-48 complete 2018-11-05
848 Published 2018-11-14
849 RFC8468 was somehow special in that there was not a technical reason/
850 interest that triggered it, but rather a formal requirement. While
851 writing RFC7312 the IP Performance Metrics working group (IPPM)
852 realized that RFC 2330, the IP Performance Metrics Framework
853 supported IPv4 only and explicitly excluded support for IPv6.
854 Nevertheless, people used the metrics that were defined on top of RFC
855 2330 (and, therefore, IPv4 only) for IPv6, too. Although the IPPM WG
856 agreed that the work was needed, the interest of IPPM attendees in
857 progressing (and reading/reviewing) the IPv6 draft was limited.
858 Resolving the IPv6 technical part was straight-forward, but
859 subsequently some people asked for a broader scope (topics like
860 header compression, 6lo, etc.) and it took some time to figure out
861 and later on convince people that these topics are out of scope. The
862 group also had to resolve contentious topics, for example how to
863 measure the processing of IPv6 extension headers, which is sometimes
864 non-standard.
866 The AUTH-48 delay for this draft was longer than average. According
867 to the authors, the main reasons include:
869 o Work-load and travel caused by busy-work-periods of all co-authors
871 o Time zone difference between co-authors and editor (at least US,
872 Europe, India, not considering travel)
874 o Editor proposing and committing some unacceptable modifications
875 that needed to be reverted
877 o Lengthy discussions on a new document title (required high effort
878 and took a long time, in particular reaching consensus between co-
879 authors and editor was time-consuming and involved the AD)
881 o Editor correctly identifying some nits (obsoleted personal
882 websites of co-authors) and co-authors attempting to fix them.
884 The differences between the final draft and the publish RFC show copy
885 editing for style and clarity, but do not account for the back and
886 forth between authors and editors mentioned by the authors.
888 4. Analysis of Process and Delays
890 We examine the 20 RFC in the sample, measuring various
891 characteristics such as delay and citation counts, in an attempt to
892 identify patterns in the IETF processes.
894 4.1. First Draft to RFC Delays
896 We look at the distribution of delays between the submission of the
897 first draft and the publication of the RFC, using the three big
898 milestones defined in Section 2.1: processing time in the Working
899 Group, IETF processing time, and publication delay. The following
900 table shows the delays for the 20 RFC in the sample:
902 +------+------------------+-------+---------+------+------+------+
903 | RFC | Status | Pages | Overall | WG | IETF | Edit |
904 +------+------------------+-------+---------+------+------+------+
905 | 8411 | Info | 5 | 455 | 154 | 140 | 161 |
906 | | | | | | | |
907 | 8456 | Info | 64 | 1317 | 1033 | 126 | 158 |
908 | | | | | | | |
909 | 8446 | PS | 160 | 1576 | 1400 | 34 | 142 |
910 | | | | | | | |
911 | 8355 | Info | 13 | 1517 | 1175 | 243 | 99 |
912 | | | | | | | |
913 | 8441 | PS | 8 | 341 | 204 | 31 | 106 |
914 | | | | | | | |
915 | 8324 | ISE | 29 | 270 | 38 | 161 | 71 |
916 | | | | | | | |
917 | 8377 | PS | 8 | 1792 | 1630 | 21 | 141 |
918 | | | | | | | |
919 | 8498 | Info | 15 | 1061 | 935 | 59 | 67 |
920 | | | | | | | |
921 | 8479 | ISE | 8 | 414 | 233 | 144 | 37 |
922 | | | | | | | |
923 | 8453 | Info | 42 | 1162 | 1036 | 46 | 80 |
924 | | | | | | | |
925 | 8429 | BCP | 10 | 548 | 76 | 313 | 159 |
926 | | | | | | | |
927 | 8312 | Info | 18 | 1255 | 1113 | 16 | 126 |
928 | | | | | | | |
929 | 8492 | ISE | 40 | 2358 | 1706 | 172 | 480 |
930 | | | | | | | |
931 | 8378 | Exp | 21 | 1524 | 1446 | 27 | 51 |
932 | | | | | | | |
933 | 8361 | PS | 17 | 1612 | 1477 | 62 | 73 |
934 | | | | | | | |
935 | 8472 | PS | 8 | 1228 | 899 | 249 | 80 |
936 | | | | | | | |
937 | 8471 | PS | 18 | 1228 | 899 | 249 | 80 |
938 | | | | | | | |
939 | 8466 | PS | 158 | 771 | 538 | 124 | 109 |
940 | | | | | | | |
941 | 8362 | PS | 33 | 1871 | 1766 | 41 | 64 |
942 | | | | | | | |
943 | 8468 | Info | 15 | 1196 | 979 | 90 | 127 |
944 | | | | | | | |
945 | | average | 35 | 1186 | 948 | 117 | 121 |
946 | | | | | | | |
947 | | average(not ISE) | 36 | 1217 | 999 | 110 | 107 |
948 +------+------------------+-------+---------+------+------+------+
950 The average delay from first draft to publication is about 3 years
951 and 3 months, but this varies widely. Excluding the independent
952 stream submissions, the average delay from start to finish is 3 years
953 and 4 months, of which on average 2 years and 9 months are spent
954 getting consensus in the working group, and 3 to 4 months each for
955 IETF consensus and for RFC production.
957 The longest delay is found for [RFC8492], 6.5 years from start to
958 finish. This is however a very special case, a draft that was
959 prepared for the TLS working group and failed to reach consensus.
960 After that, it was resubmitted to the ISE, and incurred atypical
961 production delays.
963 On average, we see that 80% of the delay is incurred in WG
964 processing, 10% in IETF review, and 10% for edition and publication.
966 For IETF stream RFC, it appears that the delays for informational
967 documents are slightly shorter than those for protocol
968 specifications, maybe six months shorter on average. However, there
969 are lots of differences between individual documents. The delays
970 range from less than a year to more than 5 years for protocol
971 specifications, and from a year and 3 months to a bit more than 4
972 years for informational documents.
974 We can compare the delays in the 2018 samples to those observed 10
975 years ago and 20 years before:
977 +------------+--------+-------+-------+
978 | RFC (2008) | Status | Pages | Delay |
979 +------------+--------+-------+-------+
980 | 5326 | Exp | 54 | 1584 |
981 | | | | |
982 | 5348 | PS | 58 | 823 |
983 | | | | |
984 | 5281 | Info | 51 | 1308 |
985 | | | | |
986 | 5354 | Exp | 23 | 2315 |
987 | | | | |
988 | 5227 | PS | 21 | 2434 |
989 | | | | |
990 | 5329 | PS | 12 | 1980 |
991 | | | | |
992 | 5277 | PS | 35 | 912 |
993 | | | | |
994 | 5236 | ISE | 26 | 1947 |
995 | | | | |
996 | 5358 | BCP | 7 | 884 |
997 | | | | |
998 | 5271 | Info | 22 | 1066 |
999 | | | | |
1000 | 5195 | PS | 10 | 974 |
1001 | | | | |
1002 | 5283 | PS | 12 | 1096 |
1003 | | | | |
1004 | 5186 | Info | 6 | 2253 |
1005 | | | | |
1006 | 5142 | PS | 13 | 1005 |
1007 | | | | |
1008 | 5373 | PS | 24 | 1249 |
1009 | | | | |
1010 | 5404 | PS | 27 | 214 |
1011 | | | | |
1012 | 5172 | PS | 7 | 305 |
1013 | | | | |
1014 | 5349 | Info | 10 | 1096 |
1015 | | | | |
1016 | 5301 | PS | 6 | 396 |
1017 | | | | |
1018 | 5174 | Info | 8 | 427 |
1019 +------------+--------+-------+-------+
1021 +------------+--------+-------+---------+
1022 | RFC (1998) | Status | Pages | Delay |
1023 +------------+--------+-------+---------+
1024 | 2289 | PS | 25 | 396 |
1025 | | | | |
1026 | 2267 | Info | 10 | unknown |
1027 | | | | |
1028 | 2317 | BCP | 10 | 485 |
1029 | | | | |
1030 | 2404 | PS | 7 | 488 |
1031 | | | | |
1032 | 2374 | PS | 12 | 289 |
1033 | | | | |
1034 | 2449 | PS | 19 | 273 |
1035 | | | | |
1036 | 2283 | PS | 9 | 153 |
1037 | | | | |
1038 | 2394 | Info | 6 | 365 |
1039 | | | | |
1040 | 2348 | DS | 5 | 699 |
1041 | | | | |
1042 | 2382 | Info | 30 | 396 |
1043 | | | | |
1044 | 2297 | ISE | 109 | 28 |
1045 | | | | |
1046 | 2381 | PS | 43 | 699 |
1047 | | | | |
1048 | 2312 | Info | 20 | 365 |
1049 | | | | |
1050 | 2387 | PS | 10 | 122 |
1051 | | | | |
1052 | 2398 | Info | 15 | 396 |
1053 | | | | |
1054 | 2391 | PS | 10 | 122 |
1055 | | | | |
1056 | 2431 | PS | 10 | 457 |
1057 | | | | |
1058 | 2282 | Info | 14 | 215 |
1059 | | | | |
1060 | 2323 | ISE | 5 | unknown |
1061 | | | | |
1062 | 2448 | ISE | 7 | 92 |
1063 +------------+--------+-------+---------+
1065 We can compare the median delay, and the delays observed by the
1066 fastest and slowest quartiles in the three years:
1068 +------+-------------+--------+-------------+
1069 | Year | Fastest 25% | Median | Slowest 25% |
1070 +------+-------------+--------+-------------+
1071 | 2018 | 604 | 1179 | 1522 |
1072 | | | | |
1073 | 2008 | 869 | 1081 | 1675 |
1074 | | | | |
1075 | 1998 | 169 | 365 | 442 |
1076 +------+-------------+--------+-------------+
1078 The IETF takes three to four times more times to produce an RFC in
1079 2018 than it did in 1998, but about the same time as it did in 2008.
1080 We can get a rough estimate of how this translates in term of "level
1081 of attention" per RFC by comparing the number of participants in the
1082 IETF meetings of 2018, 2008 and 1998 [IETFCOUNT] to the number of RFC
1083 published these years [RFCYEAR].
1085 +------+------+---------+---------+------+----------+---------------+
1086 | Year | Nb | Spring | Summer | Fall | Average | Attendees/RFC |
1087 | | RFC | P. | P. | | P. | |
1088 +------+------+---------+---------+------+----------+---------------+
1089 | 2018 | 208 | 1235 | 1078 | 879 | 1064 | 5.1 |
1090 | | | | | | | |
1091 | 2008 | 290 | 1128 | 1181 | 962 | 1090 | 3.8 |
1092 | | | | | | | |
1093 | 1998 | 234 | 1775 | 2106 | 1705 | 1862 | 9.0 |
1094 +------+------+---------+---------+------+----------+---------------+
1096 The last column in the table provides the ratio of average number of
1097 participants by number of RFC produced. If the IETF was a
1098 centralized organization, if all participants and documents were
1099 equivalent, this ratio would be the number of participants dedicated
1100 to produce an RFC on a given year. This is of course a completely
1101 abstract figure because none of the hypotheses above is true, but it
1102 still gives a vague indication of the "level of attention" applied to
1103 documents. We see that this ratio has increased from 2008 to 2018,
1104 as the number of participants was about the same for these two years
1105 but the number of published RFC decreased. However, that ratio was
1106 much higher in 1998. The IETF had many more participants, and there
1107 were probably many more eyes available to review any given draft. If
1108 we applied the ratios of 1998, the IETF would be producing 119
1109 documents in 2018 instead of 208.
1111 4.2. Working Group Processing Time
1113 The largest part of the delays is spent in the working groups, before
1114 the draft is submitted to the IESG for IETF review. As mentioned in
1115 Section 2.1, the only intermediate milestone that we can extract from
1116 the IETF data tracker is the date at which the document was adopted
1117 by the working group, or targeted for independent submission. The
1118 breakdown of the delays for the documents in our sample is:
1120 +------+---------+------+----------------+----------------+
1121 | RFC | Status | WG | Until adoption | After adoption |
1122 +------+---------+------+----------------+----------------+
1123 | 8411 | Info | 154 | 0 | 154 |
1124 | | | | | |
1125 | 8456 | Info | 1033 | 209 | 824 |
1126 | | | | | |
1127 | 8446 | PS | 1400 | 0 | 1400 |
1128 | | | | | |
1129 | 8355 | Info | 1175 | 102 | 1073 |
1130 | | | | | |
1131 | 8441 | PS | 204 | 65 | 139 |
1132 | | | | | |
1133 | 8324 | ISE | 38 | 0 | 38 |
1134 | | | | | |
1135 | 8377 | PS | 1630 | 728 | 902 |
1136 | | | | | |
1137 | 8498 | Info | 935 | 420 | 515 |
1138 | | | | | |
1139 | 8479 | ISE | 233 | 0 | 233 |
1140 | | | | | |
1141 | 8453 | Info | 1036 | 396 | 640 |
1142 | | | | | |
1143 | 8429 | BCP | 76 | 0 | 76 |
1144 | | | | | |
1145 | 8312 | Info | 1113 | 280 | 833 |
1146 | | | | | |
1147 | 8492 | ISE | 1706 | 1428 | 278 |
1148 | | | | | |
1149 | 8378 | Exp | 1446 | 661 | 785 |
1150 | | | | | |
1151 | 8361 | PS | 1477 | 399 | 1078 |
1152 | | | | | |
1153 | 8472 | PS | 899 | 105 | 794 |
1154 | | | | | |
1155 | 8471 | PS | 1127 | 153 | 794 |
1156 | | | | | |
1157 | 8466 | PS | 538 | 178 | 360 |
1158 | | | | | |
1159 | 8362 | PS | 1766 | 240 | 1526 |
1160 | | | | | |
1161 | 8468 | Info | 979 | 333 | 646 |
1162 | | | | | |
1163 | | Average | 948 | 285 | 663 |
1164 +------+---------+------+----------------+----------------+
1166 The time before working group adoption average to a bit more than 9
1167 months, compared to 1 years and almost 10 months for processing time
1168 after adoption. We see that RFC 8492 stands out, with long delays
1169 spent attempting publication through a working group before
1170 submission to the independent editor. If we removed RFC 8492 from
1171 the list, the average time until adoption drops to just over 7
1172 months, and becomes just 25% of the total processing time in the WG.
1174 There are a few documents that started immediately as working group
1175 efforts, or were immediately targeted for publication in the
1176 independent series. Those documents tend to see short processing
1177 times, with the exception of RFC 8446 on which the TLS working group
1178 spent a long time working.
1180 4.3. Preparation and Publication Delays
1182 The preparation and publication delays include three components:
1184 o the delay from submission to the RFC Editor to beginning of AUTH-
1185 48, during which the document is prepared;
1187 o the AUTH-48 delay, during which authors review and eventually
1188 approve the changes proposed by the editors;
1190 o the publication delay, from final agreement by authors and editors
1191 to actual publication.
1193 The breakdown of the publication delays for each RFC is shown in the
1194 following table.
1196 +-------+---------+-------+--------+---------+--------+-------------+
1197 | RFC | Status | Pages | RFC | AUTH-48 | RFC | Edit(total) |
1198 | | | | edit | | Pub | |
1199 +-------+---------+-------+--------+---------+--------+-------------+
1200 | 8411 | Info | 5 | 53 | 88 | 20 | 161 |
1201 | | | | | | | |
1202 | 8456 | Info | 64 | 98 | 46 | 14 | 158 |
1203 | | | | | | | |
1204 | 8446 | PS | 160 | 85 | 57 | 0 | 142 |
1205 | | | | | | | |
1206 | 8355 | Info | 13 | 83 | 15 | 1 | 99 |
1207 | | | | | | | |
1208 | 8441 | PS | 8 | 67 | 33 | 6 | 106 |
1209 | | | | | | | |
1210 | 8324 | ISE | 29 | 42 | 28 | 1 | 71 |
1211 | | | | | | | |
1212 | 8377 | PS | 8 | 39 | 102 | 0 | 141 |
1213 | | | | | | | |
1214 | 8498 | Info | 15 | 49 | 16 | 2 | 67 |
1215 | | | | | | | |
1216 | 8479 | ISE | 8 | 31 | 5 | 1 | 37 |
1217 | | | | | | | |
1218 | 8453 | Info | 42 | 73 | 7 | 0 | 80 |
1219 | | | | | | | |
1220 | 8429 | BCP | 10 | 60 | 99 | 0 | 159 |
1221 | | | | | | | |
1222 | 8312 | Info | 18 | 96 | 30 | 0 | 126 |
1223 | | | | | | | |
1224 | 8492 | ISE | 40 | 355 | 123 | 2 | 480 |
1225 | | | | | | | |
1226 | 8378 | Exp | 21 | 42 | 9 | 0 | 51 |
1227 | | | | | | | |
1228 | 8361 | PS | 17 | 39 | 31 | 3 | 73 |
1229 | | | | | | | |
1230 | 8472 | PS | 8 | 59 | 8 | 13 | 80 |
1231 | | | | | | | |
1232 | 8471 | PS | 18 | 59 | 8 | 13 | 80 |
1233 | | | | | | | |
1234 | 8466 | PS | 158 | 84 | 22 | 3 | 109 |
1235 | | | | | | | |
1236 | 8362 | PS | 33 | 49 | 11 | 4 | 64 |
1237 | | | | | | | |
1238 | 8468 | Info | 15 | 65 | 53 | 9 | 127 |
1239 | | | | | | | |
1240 | | Average | | 76 | 40 | 5 | 121 |
1241 | | | | | | | |
1242 | -8492 | Average | | 62 | 35 | 5 | 102 |
1243 +-------+---------+-------+--------+---------+--------+-------------+
1244 On average, the total delay appears to be about four months, but the
1245 average is skewed by the extreme values encountered for [RFC8492].
1246 If we exclude that RFC from the computations, the average delay drops
1247 to a just a bit more than 3 months: about 2 months for the
1248 preparation, a bit more than one month for the AUTH-48 phase, and 5
1249 days for the publishing.
1251 Of course, these delays vary from RFC to RFC. To try explain the
1252 causes of the delay, we compute the correlation factor between the
1253 observed delays and several plausible explanation factors:
1255 o The number of pages in the document,
1257 o The amount of copy edit, as discussed in Section 4.4,
1259 o Whether or not an IANA action was required,
1261 o The number of authors,
1263 o The number of drafts revisions,
1265 o The working group delay.
1267 We find the following values:
1269 +-------------+----------+---------+-------------+
1270 | Correlation | RFC edit | AUTH-48 | Edit(total) |
1271 +-------------+----------+---------+-------------+
1272 | Nb pages | 0.50 | -0.04 | 0.21 |
1273 | | | | |
1274 | Copy-Edit | 0.42 | 0.24 | 0.45 |
1275 | | | | |
1276 | IANA | -0.14 | -0.21 | 0.12 |
1277 | | | | |
1278 | Nb Authors | 0.39 | -0.07 | 0.18 |
1279 | | | | |
1280 | Nb drafts | 0.18 | -0.33 | -0.19 |
1281 | | | | |
1282 | WG delay | 0.03 | -0.16 | -0.15 |
1283 +-------------+----------+---------+-------------+
1285 We see some plausible explanations for the production delay. It will
1286 be somewhat longer for longer documents, or for documents that
1287 require a lot of copy editing (see Section 4.4). Somewhat
1288 surprisingly, it also tend to increase with the number of authors.
1289 It does not appear significantly correlated with the presence or
1290 absence of IANA action.
1292 The analysis of RFC 8324 in Section 3.6 explains its short editing
1293 delays by the experience of the author. This makes sense: if a
1294 document needs less editing, the editing delays would be shorter.
1295 This is partially confirmed by the relation between the amount of
1296 copy editing and the publication delay.
1298 We see fewer plausible explanations for the AUTH48 delays. These
1299 delays vary much more than the preparation delay, with a standard
1300 deviation of 20 days for AUTH-48 versus 10 days for the preparation
1301 delay. In theory, AUTH-48 is just a final verification: the authors
1302 receive the document prepared by the RFC production center, and just
1303 have to give their approval, or maybe request a last minute
1304 correction. The name indicates that this is expected to last just
1305 two days, but in average it lasts more than a month.
1307 We often hypothesize that the number of authors influences the
1308 AUTH-48 delay, or that authors who have spent a long time working on
1309 the document in the working group somehow get demotivated and spend
1310 even longer to answer questions during AUTH-48. This may happen
1311 sometimes, but our statistics don't show that - if anything, the
1312 numerical results point in the opposite direction.
1314 After asking the authors of the RFC in the sample why the AUTH-48
1315 phase took a long time, and we got three explanations:
1317 1- Some RFC have multiple authors in multiple time zones. This slows
1318 down the coordination required for approving changes.
1320 2- Some authors found some of the proposed changes unnecessary or
1321 undesirable, and asked that they be reversed. This required long
1322 exchanges between authors and editors.
1324 3- Some authors were not giving high priority to AUTH-48 responses.
1326 As mentioned above, we were not able to verify these hypotheses by
1327 looking at the data.
1329 4.4. Copy Editing
1331 We can assess the amount of copy editing applied to each published
1332 RFC by comparing the text of the draft approved for publication and
1333 the text of the RFC. We do expect differences in the "boilerplate"
1334 and in the IANA section, but we will also see differences due to copy
1335 editing. Assessing the amount of copy editing is subjective, and we
1336 do it using a scale of 1 to 4:
1338 1- Minor editing
1339 2- Editing for style, such as capitalization, hyphens, that versus
1340 which, and expending all acronyms at least one.
1342 3- Editing for clarity in addition to style, such as rewriting
1343 ambiguous sentences and clarifying use of internal references. For
1344 Yang models, that may include model corrections suggested by the
1345 verifier.
1347 4- Extensive editing.
1349 The following table shows that with about half of the RFC required
1350 editing for style, and the other half at least some editing for
1351 clarity.
1353 +------+--------+-----------+
1354 | RFC | Status | Copy Edit |
1355 +------+--------+-----------+
1356 | 8411 | Info | 2 |
1357 | | | |
1358 | 8456 | Info | 4 |
1359 | | | |
1360 | 8446 | PS | 3 |
1361 | | | |
1362 | 8355 | Info | 2 |
1363 | | | |
1364 | 8441 | PS | 2 |
1365 | | | |
1366 | 8324 | ISE | 2 |
1367 | | | |
1368 | 8377 | PS | 3 |
1369 | | | |
1370 | 8498 | Info | 3 |
1371 | | | |
1372 | 8479 | ISE | 1 |
1373 | | | |
1374 | 8453 | Info | 2 |
1375 | | | |
1376 | 8429 | BCP | 2 |
1377 | | | |
1378 | 8312 | Info | 2 |
1379 | | | |
1380 | 8492 | ISE | 3 |
1381 | | | |
1382 | 8378 | Exp | 2 |
1383 | | | |
1384 | 8361 | PS | 2 |
1385 | | | |
1386 | 8472 | PS | 2 |
1387 | | | |
1388 | 8471 | PS | 2 |
1389 | | | |
1390 | 8466 | PS | 3 |
1391 | | | |
1392 | 8362 | PS | 3 |
1393 | | | |
1394 | 8468 | Info | 3 |
1395 +------+--------+-----------+
1397 This method of assessment does not take into account the number of
1398 changes proposed by the editors and eventually rejected by the
1399 authors, since these changes are not present in either the final
1400 draft or the published RFC. It might be possible to get an
1401 evaluation of these "phantom changes" from the RFC Production Center.
1403 4.5. Independent Series
1405 Out of 20 randomly selected RFC, 3 were published through the
1406 "independent series". One is an independent opinion, another a
1407 description of a non-IETF protocol format, and the third was
1408 [RFC8492], which is a special case. Apart from this special case,
1409 the publication delays were significantly shorter for the Independent
1410 Stream than for the IETF stream.
1412 The authors of these 3 RFC are regular IETF contributors. This
1413 observation motivated a secondary analysis of all the RFC published
1414 in the "independent" stream in 2018. There are 14 such RFC: 8507,
1415 8494, 8493, 8492, 8483, 8479, 8433, 8409, 8374, 8369, 8367, 8351,
1416 8328 and 8324. (RFC 8367 and 8369 were published on 1 April 2018.)
1417 The majority of the documents were published by regular IETF
1418 participants, but two of them were not. One describes "The BagIt
1419 File Packaging Format (V1.0)" [RFC8493], and the other the "Yeti DNS
1420 Testbed" [RFC8483]. They document a data format and a system
1421 developed outside the IETF, and illustrate the outreach function of
1422 the Independent Stream. In both cases, the authors include one
1423 experienced IETF participant, who presumably helped outsiders
1424 navigate the publication process.
1426 5. Citation Counts
1428 In this exploration, we want to assess whether citation counts
1429 provide a meaningful assessment of the popularity of RFC. We obtain
1430 the citation counts through the Semantic Scholar API, using queries
1431 of the form:
1433 http://api.semanticscholar.org/
1434 v1/paper/10.17487/rfc8446?include_unknown_references=true
1436 In these queries, the RFC is uniquely identified by its DOI
1437 reference, which is composed of the RFC Series prefix 10.17487 and
1438 the rfc identifier. The queries return a series of properties,
1439 including a list of citations for the RFC. Based on that list of
1440 citations, we compute three numbers:
1442 o The total number of citations
1444 o The number of citations in the year of publication and the year
1445 after that
1447 o For the RFC published in 1998 or 2008 that we use for comparison,
1448 the number of citations in the years 2018 and 2019.
1450 All the numbers were retrieved on October 6, 2019.
1452 5.1. Citation Numbers
1454 As measured on October 6, 2019, the citation counts for the RFC in
1455 our sample set were:
1457 +-----------+--------+-------+-----------+
1458 | RFC(2018) | Status | Total | 2018-2019 |
1459 +-----------+--------+-------+-----------+
1460 | 8411 | Info | 1 | 0 |
1461 | | | | |
1462 | 8456 | Info | 1 | 1 |
1463 | | | | |
1464 | 8446 | PS | 418 | 204 |
1465 | | | | |
1466 | 8355 | Info | 3 | 3 |
1467 | | | | |
1468 | 8441 | PS | 1 | 1 |
1469 | | | | |
1470 | 8324 | ISE | 0 | 0 |
1471 | | | | |
1472 | 8377 | PS | 0 | 0 |
1473 | | | | |
1474 | 8498 | Info | 0 | 0 |
1475 | | | | |
1476 | 8479 | ISE | 0 | 0 |
1477 | | | | |
1478 | 8453 | Info | 3 | 3 |
1479 | | | | |
1480 | 8429 | BCP | 0 | 0 |
1481 | | | | |
1482 | 8312 | Info | 25 | 16 |
1483 | | | | |
1484 | 8492 | ISE | 4 | 4 |
1485 | | | | |
1486 | 8378 | Exp | 1 | 1 |
1487 | | | | |
1488 | 8361 | PS | 0 | 0 |
1489 | | | | |
1490 | 8472 | PS | 1 | 1 |
1491 | | | | |
1492 | 8471 | PS | 1 | 1 |
1493 | | | | |
1494 | 8466 | PS | 0 | 0 |
1495 | | | | |
1496 | 8362 | PS | 1 | 1 |
1497 | | | | |
1498 | 8468 | Info | 1 | 1 |
1499 +-----------+--------+-------+-----------+
1501 The results indicate that [RFC8446] is by far the most cited of the
1502 20 RFC in our sample. This is not surprising, since TLS is a key
1503 Internet Protocol. The TLS 1.3 protocol was also the subject of
1504 extensive studies by researchers, and thus was mentioned in a number
1505 of published papers. Surprisingly, the Semantic Scholar mentions a
1506 number of citations that predate the publication date. These are
1507 probably citations of the various draft versions of the protocol.
1509 The next most cited RFC in the sample is [RFC8312] which describes
1510 the Cubic congestion control algorithm for TCP. That protocol was
1511 also the target of a large number of academic publications.The other
1512 RFC in the sample only have a small number of citations.
1514 There is probably a small bias when measuring citations at a fixed
1515 date. An RFC published in January 2018 would have more time to
1516 accrue citations than one published in December. That may be true to
1517 some extent, as the second most cited RFC in the set was published in
1518 January. However, the effect has to be limited. The most cited RFC
1519 was published in August, and the second most cited was published in
1520 2019. (That RFC got an RFC number in 2018, but publication was
1521 slowed by long AUTH-48 delays.)
1523 5.2. Comparison to 1998 and 2008
1525 In order to get a baseline, we can look at the number of references
1526 for the RFC published in 2008 and 1998. However, we need totake time
1527 into account. Documents published a long time ago are expected to
1528 have accrued more references. We try to address this by looking at
1529 three counts for each document: the overall number of references over
1530 the document's lifetime, the number of references obtained in the
1531 year following publication, and the number of references observed
1532 since 2018:
1534 +-----------+--------+-------+-----------+-----------+
1535 | RFC(2008) | Status | Total | 2008-2009 | 2018-2019 |
1536 +-----------+--------+-------+-----------+-----------+
1537 | 5326 | Exp | 138 | 14 | 15 |
1538 | | | | | |
1539 | 5348 | PS | 14 | 3 | 0 |
1540 | | | | | |
1541 | 5281 | Info | 69 | 15 | 7 |
1542 | | | | | |
1543 | 5354 | Exp | 17 | 13 | 0 |
1544 | | | | | |
1545 | 5227 | PS | 19 | 1 | 2 |
1546 | | | | | |
1547 | 5329 | PS | 24 | 6 | 1 |
1548 | | | | | |
1549 | 5277 | PS | 32 | 3 | 2 |
1550 | | | | | |
1551 | 5236 | ISE | 25 | 5 | 4 |
1552 | | | | | |
1553 | 5358 | BCP | 21 | 2 | 0 |
1554 | | | | | |
1555 | 5271 | Info | 7 | 2 | 0 |
1556 | | | | | |
1557 | 5195 | PS | 7 | 4 | 2 |
1558 | | | | | |
1559 | 5283 | PS | 8 | 1 | 0 |
1560 | | | | | |
1561 | 5186 | Info | 14 | 4 | 2 |
1562 | | | | | |
1563 | 5142 | PS | 8 | 4 | 0 |
1564 | | | | | |
1565 | 5373 | PS | 5 | 2 | 0 |
1566 | | | | | |
1567 | 5404 | PS | 1 | 1 | 0 |
1568 | | | | | |
1569 | 5172 | PS | 2 | 0 | 0 |
1570 | | | | | |
1571 | 5349 | Info | 8 | 0 | 2 |
1572 | | | | | |
1573 | 5301 | PS | 5 | 1 | 0 |
1574 | | | | | |
1575 | 5174 | Info | 0 | 0 | 0 |
1576 +-----------+--------+-------+-----------+-----------+
1577 +-----------+--------+-------+-----------+-----------+
1578 | RFC(1998) | Status | Total | 1998-1999 | 2018-2019 |
1579 +-----------+--------+-------+-----------+-----------+
1580 | 2289 | PS | 2 | 0 | 1 |
1581 | | | | | |
1582 | 2267 | Info | 982 | 5 | 61 |
1583 | | | | | |
1584 | 2317 | BCP | 9 | 1 | 2 |
1585 | | | | | |
1586 | 2404 | PS | 137 | 6 | 1 |
1587 | | | | | |
1588 | 2374 | PS | 42 | 4 | 0 |
1589 | | | | | |
1590 | 2449 | PS | 7 | 2 | 0 |
1591 | | | | | |
1592 | 2283 | PS | 17 | 3 | 2 |
1593 | | | | | |
1594 | 2394 | Info | 13 | 2 | 1 |
1595 | | | | | |
1596 | 2348 | DS | 5 | 0 | 0 |
1597 | | | | | |
1598 | 2382 | Info | 17 | 12 | 0 |
1599 | | | | | |
1600 | 2297 | ISE | 36 | 11 | 0 |
1601 | | | | | |
1602 | 2381 | PS | 39 | 12 | 0 |
1603 | | | | | |
1604 | 2312 | Info | 14 | 3 | 0 |
1605 | | | | | |
1606 | 2387 | PS | 4 | 1 | 0 |
1607 | | | | | |
1608 | 2398 | Info | 17 | 0 | 1 |
1609 | | | | | |
1610 | 2391 | PS | 31 | 3 | 0 |
1611 | | | | | |
1612 | 2431 | PS | 3 | 0 | 0 |
1613 | | | | | |
1614 | 2282 | Info | 8 | 0 | 0 |
1615 | | | | | |
1616 | 2323 | ISE | 1 | 0 | 0 |
1617 | | | | | |
1618 | 2448 | ISE | 0 | 0 | 0 |
1619 +-----------+--------+-------+-----------+-----------+
1621 We can compare the median number of citations and the numbers of
1622 citations for the least and most popular quartiles in the three
1623 years:
1625 +----------------------------+-----------+--------+------------+
1626 | References | Lower 25% | Median | Higher 25% |
1627 +----------------------------+-----------+--------+------------+
1628 | RFC (2018) | 0 | 1 | 3 |
1629 | | | | |
1630 | RFC (2008) | 6.5 | 11 | 21.75 |
1631 | | | | |
1632 | RFC (2008), until 2009 | 1 | 2.5 | 4.5 |
1633 | | | | |
1634 | RFC (2008), 2018 and after | 0 | 0 | 2 |
1635 | | | | |
1636 | RFC (1998) | 4.75 | 13.5 | 32.25 |
1637 | | | | |
1638 | RFC (1998), until 1999 | 0 | 2 | 4.25 |
1639 | | | | |
1640 | RFC (1998), 2018 and after | 0 | 0 | 1 |
1641 +----------------------------+-----------+--------+------------+
1643 The total numbers shows new documents with fewer citations than the
1644 older ones. This can be explained to some degree by the passage of
1645 time. If we restrict the analysis to the number of citations accrued
1646 in the year of publishing and the year after that, we still see about
1647 the same distribution for the three samples.
1649 We also see that the number of references to RFC fades over time.
1650 Only the most popular of the RFC produced in 1998 are still cited in
1651 2019.
1653 5.3. Citations Versus Deployments
1655 The following table shows side by side the number of citations as
1656 measured in Section 5.1 and the estimation of deployment as indicated
1657 in Section 3.
1659 +-----------+--------+-----------+------------+
1660 | RFC(2018) | Status | Citations | Deployment |
1661 +-----------+--------+-----------+------------+
1662 | 8411 | Info | 1 | medium |
1663 | | | | |
1664 | 8456 | Info | 1 | medium |
1665 | | | | |
1666 | 8446 | PS | 418 | high |
1667 | | | | |
1668 | 8355 | Info | 3 | medium |
1669 | | | | |
1670 | 8441 | PS | 1 | high |
1671 | | | | |
1672 | 8324 | ISE | 0 | N/A |
1673 | | | | |
1674 | 8377 | PS | 0 | unknown |
1675 | | | | |
1676 | 8498 | Info | 0 | unknown |
1677 | | | | |
1678 | 8479 | ISE | 0 | one |
1679 | | | | |
1680 | 8453 | Info | 3 | unknown |
1681 | | | | |
1682 | 8429 | BCP | 0 | some |
1683 | | | | |
1684 | 8312 | Info | 25 | high |
1685 | | | | |
1686 | 8492 | ISE | 4 | one |
1687 | | | | |
1688 | 8378 | Exp | 1 | some |
1689 | | | | |
1690 | 8361 | PS | 0 | one |
1691 | | | | |
1692 | 8472 | PS | 1 | medium |
1693 | | | | |
1694 | 8471 | PS | 1 | medium |
1695 | | | | |
1696 | 8466 | PS | 0 | unknown |
1697 | | | | |
1698 | 8362 | PS | 1 | medium |
1699 | | | | |
1700 | 8468 | Info | 1 | some |
1701 +-----------+--------+-----------+------------+
1703 From looking at these results, it is fairly obvious that citation
1704 counts cannot be used as proxies for the "value" of an RFC. In our
1705 sample, the two RFC that have high citation counts were both widely
1706 deployed, and can certainly be described as successful, but we also
1707 see many RFC that saw significant deployment without garnering a high
1708 level of citations.
1710 Citation counts are driven by academic interest, but are only loosely
1711 correlated with actual deployment. We saw that [RFC8446] was widely
1712 cited in part because the standardization process involved many
1713 researchers, and that the high citation count of [RFC8312] is largely
1714 due to the academic interest in evaluating congestion control
1715 protocols. If we look at previous years, the most cited RFC in the
1716 2008 sample is [RFC5326], an experimental RFC defining security
1717 extensions to an experimental delay tolerant transport protocol.
1718 This protocol does not carry a significant proportion of Internet
1719 traffic, but has been the object of a fair number of academic
1720 studies.
1722 The citation process tends to privilege the first expression of a
1723 concept. We see that with the most cited RFC in the 1998 set is
1724 [RFC2267], an informational RFC defining Network Ingress Filtering
1725 that was obsoleted in May 2000 by [RFC2827]. It is still cited
1726 frequently in 2018 and 2019, regardless of its formal status in the
1727 RFC series. We see the same effect at work with [RFC8441], which
1728 garners very few citations although it obsoletes [RFC6455] that has a
1729 large number of citations. The same goes for [RFC8468], which is
1730 sparsely cited while the [RFC2330] is widely cited. Just counting
1731 citations will not indicate whether developers still use an old
1732 specification or have adopted the revised RFC.
1734 5.4. Citations versus web references
1736 Web references might be another indiactor of the popularity of an
1737 RFC. In order to evaluate these references, we list here the number
1738 of results returned by searches on Google and Bing, looking for the
1739 search term "RFCnnnn" (e.g., RFC8411), and copying the number of
1740 results returned by the search engines. The table below presents the
1741 results of these searches, performed on April 4, 2020.
1743 +-----------+--------+-----------+--------+-------+
1744 | RFC(2018) | Status | Citations | Google | Bing |
1745 +-----------+--------+-----------+--------+-------+
1746 | 8411 | Info | 1 | 301 | 94 |
1747 | | | | | |
1748 | 8456 | Info | 1 | 266 | 8456 |
1749 | | | | | |
1750 | 8446 | PS | 418 | 25900 | 47800 |
1751 | | | | | |
1752 | 8355 | Info | 3 | 521 | 114 |
1753 | | | | | |
1754 | 8441 | PS | 1 | 2430 | 59500 |
1755 | | | | | |
1756 | 8324 | ISE | 0 | 393 | 138 |
1757 | | | | | |
1758 | 8377 | PS | 0 | 264 | 10900 |
1759 | | | | | |
1760 | 8498 | Info | 0 | 335 | 10100 |
1761 | | | | | |
1762 | 8479 | ISE | 0 | 564 | 11000 |
1763 | | | | | |
1764 | 8453 | Info | 3 | 817 | 11400 |
1765 | | | | | |
1766 | 8429 | BCP | 0 | 391 | 41600 |
1767 | | | | | |
1768 | 8312 | Info | 25 | 1620 | 2820 |
1769 | | | | | |
1770 | 8492 | ISE | 4 | 323 | 9400 |
1771 | | | | | |
1772 | 8378 | Exp | 1 | 418 | 11600 |
1773 | | | | | |
1774 | 8361 | PS | 0 | 499 | 92 |
1775 | | | | | |
1776 | 8472 | PS | 1 | 496 | 169 |
1777 | | | | | |
1778 | 8471 | PS | 1 | 1510 | 11600 |
1779 | | | | | |
1780 | 8466 | PS | 0 | 766 | 173 |
1781 | | | | | |
1782 | 8362 | PS | 1 | 67 | 147 |
1783 | | | | | |
1784 | 8468 | Info | 1 | 453 | 127 |
1785 +-----------+--------+-----------+--------+-------+
1787 The results counts from Bing are sometimes surprising. Why would RFC
1788 8441 gather 59,500 web references? Looking at the results in detail,
1789 we find a mix of data. Some of them are logs of development projects
1790 implementing Web Sockets, which is exactly what we are looking for,
1791 but others appear spurious. For example, a shop selling rugby
1792 jerseys is listed because its phone number ends with "8441". Other
1793 pages were listed because street numbers or product numbers matched
1794 the RFC number. The same type of collision may explain the large
1795 reference counts on Bing for RFC 8377, 8498, 8479, 8453, 8429, 8378,
1796 and 8471. The result counts on Bing do not appear to provide a good
1797 metric.
1799 On Google, all RFC garner at least a 250 references, largely because
1800 the whole RFC catalog is replicated on a large number of web servers.
1801 Deviations from that base line are largely correlated with the number
1802 of citations in the Semantic Scholar, with a couple of exception: RFC
1803 8441, and 8471 garner more references than the low citation counts
1804 would predict. Looking at the results, we find many references in
1805 development databases explaining how these protocols are implemented
1806 in various code bases and open source projects. This means that
1807 counting Google results would give some indication about an RFC's
1808 popularity, complementing the citation counts.
1810 There are some practical problems in using the counts of Google
1811 results. Google searches are personalized, the results depend on the
1812 source of the queries, and the counts may vary as well. The search
1813 result depend on the search algorithm, and there is no guarantee that
1814 counts will not change when the algorithm changes. On the other
1815 hand, the results do indicate that some of the RFC in our sample are
1816 beeing used by developers or in deployments.
1818 6. Observations and Next Steps
1820 The author's goal was to get a personal understanding of the "chain
1821 of production" of the RFCs, and in particular to look at the various
1822 causes of delays in the process. As shown in Section 4, the average
1823 RFC was produced in 3 years and 4 months, which is similar to what
1824 was found in the 2008 sample, but more than three times larger than
1825 the delays for the 1998 sample.
1827 The Working group process appears to be the main source of delays.
1828 Efforts to diminish delays should probably focus there, instead of on
1829 the IETF and IESG reviews of the RFC production. For the RFC
1830 production phase, most of the variability originates in the AUTH-48
1831 process, which is influenced by a variety of factors such as number
1832 of authors or level of engagement of these authors.
1834 Most of the delay is spent in the working group, but the IETF data
1835 tracker does not hold much information about what happens inside the
1836 working groups. For example, events like Working Group Last Calls
1837 are not recorded in the history of the drafts available in the
1838 datatracker. Such information would have been interesting. Of
1839 course, requiring that information would create an administrative
1840 burden, so there is clearly a trade-off between requiring more work
1841 from working group chairs and providing better data for process
1842 analysis.
1844 The Independent Submission stream operates as expected. The majority
1845 of the authors of the independent RFC appear to be in IETF insiders,
1846 but there is significant amount of engagement by outside parties.
1848 The analysis of citations in Section 5.1 shows that citation numbers
1849 are a very poor indication of the "value" of an RFC. Citation
1850 numbers measure the engagement of academic researchers with specific
1851 topics, but have little correlation with the level of adoption and
1852 deployment of a specific RFC. The result counts of Google searches
1853 do capture references outside academia, such as logs of development
1854 projects. This might be informative, but it is not clear that the
1855 counts would not change over time due to algorithm changes or
1856 personaliztion.
1858 This document analyses a small sample of RFC "in depth". This
1859 allowed gathering of detailed feedback on the process and the
1860 deployments. On the other hand, much of the data on delays is
1861 available from the IETF data tracker. It may be worth considering
1862 adding an automated reporting of delay metrics in the IETF data
1863 tracker.
1865 This document only considers the RFC that were published in a given
1866 year. This approach can be criticized as introducing a form of
1867 "survivor bias". There are many drafts proposed to the IETF, and
1868 only a fraction of them end up being published as RFC. On one hand
1869 this is expected, because part of the process is to triage between
1870 ideas that can gather consensus and those that don't. On the other
1871 hand, we don't know whether that triage is too drastic and
1872 discouraged progress on good ideas.
1874 One way to evaluate the triage process would be to look at
1875 publication attempts that were abandoned, for example drafts that
1876 expired without progressing or being replaced. The sampling
1877 methodology could also be used for that purpose. Pick maybe 20
1878 drafts at random, among those abandoned in a target year, and
1879 investigate why they were abandoned. Was it because better solutions
1880 emerged in the working group? Or maybe because the authors
1881 discovered a flaw in their proposal? Or was it because some
1882 factional struggle blocked a good idea? Was the idea pursued in a
1883 different venue? Hopefully, someone will try this kind of
1884 investigation.
1886 7. Security Considerations
1888 This draft does not specify any protocol.
1890 We might want to analyze whether security issues were discovered
1891 after publication of specific standards.
1893 8. IANA Considerations
1895 This draft does not require any IANA action.
1897 Peliminary analysis does not indicate that IANA is causing any
1898 particular delay in the RFC publication process.
1900 9. Acknowledgements
1902 Many thanks to the authors of the selected RFC who were willing to
1903 provide feedback on the process: Michael Ackermann, Zafar Ali, Sarah
1904 Banks, Bruno Decraene, Lars Eggert, Nalini Elkins, Joachim Fabini,
1905 Dino Farinacci, Clarence Filsfils, Sujay Gupta, Dan Harkins, Vinayak
1906 Hegde, Benjamin Kaduk, John Klensin, Acee Lindem, Nikos
1907 Mavrogiannopoulos, Patrick McManus, Victor Moreno, Al Morton, Andrei
1908 Popov, Eric Rescorla, Michiko Short, Bhuvaneswaran Vengainathan, Lao
1909 Weiguo, and Li Yizhou. Many thanks to Adrian Farrel for his useful
1910 advice, to Stephen Farrell and Colin Perkins for their guidance on
1911 the use of citations, and to Dave Crocker for a comprehensive review.
1913 10. Informative References
1915 [IETFCOUNT]
1916 IETF, "Past IETF Meetings", 2020,
1917 .
1919 [RFC2267] Ferguson, P. and D. Senie, "Network Ingress Filtering:
1920 Defeating Denial of Service Attacks which employ IP Source
1921 Address Spoofing", RFC 2267, DOI 10.17487/RFC2267, January
1922 1998, .
1924 [RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis,
1925 "Framework for IP Performance Metrics", RFC 2330,
1926 DOI 10.17487/RFC2330, May 1998,
1927 .
1929 [RFC2827] Ferguson, P. and D. Senie, "Network Ingress Filtering:
1930 Defeating Denial of Service Attacks which employ IP Source
1931 Address Spoofing", BCP 38, RFC 2827, DOI 10.17487/RFC2827,
1932 May 2000, .
1934 [RFC5326] Ramadas, M., Burleigh, S., and S. Farrell, "Licklider
1935 Transmission Protocol - Specification", RFC 5326,
1936 DOI 10.17487/RFC5326, September 2008,
1937 .
1939 [RFC6455] Fette, I. and A. Melnikov, "The WebSocket Protocol",
1940 RFC 6455, DOI 10.17487/RFC6455, December 2011,
1941 .
1943 [RFC8312] Rhee, I., Xu, L., Ha, S., Zimmermann, A., Eggert, L., and
1944 R. Scheffenegger, "CUBIC for Fast Long-Distance Networks",
1945 RFC 8312, DOI 10.17487/RFC8312, February 2018,
1946 .
1948 [RFC8324] Klensin, J., "DNS Privacy, Authorization, Special Uses,
1949 Encoding, Characters, Matching, and Root Structure: Time
1950 for Another Look?", RFC 8324, DOI 10.17487/RFC8324,
1951 February 2018, .
1953 [RFC8355] Filsfils, C., Ed., Previdi, S., Ed., Decraene, B., and R.
1954 Shakir, "Resiliency Use Cases in Source Packet Routing in
1955 Networking (SPRING) Networks", RFC 8355,
1956 DOI 10.17487/RFC8355, March 2018,
1957 .
1959 [RFC8361] Hao, W., Li, Y., Durrani, M., Gupta, S., and A. Qu,
1960 "Transparent Interconnection of Lots of Links (TRILL):
1961 Centralized Replication for Active-Active Broadcast,
1962 Unknown Unicast, and Multicast (BUM) Traffic", RFC 8361,
1963 DOI 10.17487/RFC8361, April 2018,
1964 .
1966 [RFC8362] Lindem, A., Roy, A., Goethals, D., Reddy Vallem, V., and
1967 F. Baker, "OSPFv3 Link State Advertisement (LSA)
1968 Extensibility", RFC 8362, DOI 10.17487/RFC8362, April
1969 2018, .
1971 [RFC8377] Eastlake 3rd, D., Zhang, M., and A. Banerjee, "Transparent
1972 Interconnection of Lots of Links (TRILL): Multi-Topology",
1973 RFC 8377, DOI 10.17487/RFC8377, July 2018,
1974 .
1976 [RFC8378] Moreno, V. and D. Farinacci, "Signal-Free Locator/ID
1977 Separation Protocol (LISP) Multicast", RFC 8378,
1978 DOI 10.17487/RFC8378, May 2018,
1979 .
1981 [RFC8410] Josefsson, S. and J. Schaad, "Algorithm Identifiers for
1982 Ed25519, Ed448, X25519, and X448 for Use in the Internet
1983 X.509 Public Key Infrastructure", RFC 8410,
1984 DOI 10.17487/RFC8410, August 2018,
1985 .
1987 [RFC8411] Schaad, J. and R. Andrews, "IANA Registration for the
1988 Cryptographic Algorithm Object Identifier Range",
1989 RFC 8411, DOI 10.17487/RFC8411, August 2018,
1990 .
1992 [RFC8429] Kaduk, B. and M. Short, "Deprecate Triple-DES (3DES) and
1993 RC4 in Kerberos", BCP 218, RFC 8429, DOI 10.17487/RFC8429,
1994 October 2018, .
1996 [RFC8441] McManus, P., "Bootstrapping WebSockets with HTTP/2",
1997 RFC 8441, DOI 10.17487/RFC8441, September 2018,
1998 .
2000 [RFC8446] Rescorla, E., "The Transport Layer Security (TLS) Protocol
2001 Version 1.3", RFC 8446, DOI 10.17487/RFC8446, August 2018,
2002 .
2004 [RFC8453] Ceccarelli, D., Ed. and Y. Lee, Ed., "Framework for
2005 Abstraction and Control of TE Networks (ACTN)", RFC 8453,
2006 DOI 10.17487/RFC8453, August 2018,
2007 .
2009 [RFC8455] Bhuvaneswaran, V., Basil, A., Tassinari, M., Manral, V.,
2010 and S. Banks, "Terminology for Benchmarking Software-
2011 Defined Networking (SDN) Controller Performance",
2012 RFC 8455, DOI 10.17487/RFC8455, October 2018,
2013 .
2015 [RFC8456] Bhuvaneswaran, V., Basil, A., Tassinari, M., Manral, V.,
2016 and S. Banks, "Benchmarking Methodology for Software-
2017 Defined Networking (SDN) Controller Performance",
2018 RFC 8456, DOI 10.17487/RFC8456, October 2018,
2019 .
2021 [RFC8466] Wen, B., Fioccola, G., Ed., Xie, C., and L. Jalil, "A YANG
2022 Data Model for Layer 2 Virtual Private Network (L2VPN)
2023 Service Delivery", RFC 8466, DOI 10.17487/RFC8466, October
2024 2018, .
2026 [rfc8468] Morton, A., Fabini, J., Elkins, N., Ackermann, M., and V.
2027 Hegde, "IPv4, IPv6, and IPv4-IPv6 Coexistence: Updates for
2028 the IP Performance Metrics (IPPM) Framework", RFC 8468,
2029 DOI 10.17487/RFC8468, November 2018,
2030 .
2032 [RFC8468] Morton, A., Fabini, J., Elkins, N., Ackermann, M., and V.
2033 Hegde, "IPv4, IPv6, and IPv4-IPv6 Coexistence: Updates for
2034 the IP Performance Metrics (IPPM) Framework", RFC 8468,
2035 DOI 10.17487/RFC8468, November 2018,
2036 .
2038 [RFC8471] Popov, A., Ed., Nystroem, M., Balfanz, D., and J. Hodges,
2039 "The Token Binding Protocol Version 1.0", RFC 8471,
2040 DOI 10.17487/RFC8471, October 2018,
2041 .
2043 [RFC8472] Popov, A., Ed., Nystroem, M., and D. Balfanz, "Transport
2044 Layer Security (TLS) Extension for Token Binding Protocol
2045 Negotiation", RFC 8472, DOI 10.17487/RFC8472, October
2046 2018, .
2048 [RFC8479] Mavrogiannopoulos, N., "Storing Validation Parameters in
2049 PKCS#8", RFC 8479, DOI 10.17487/RFC8479, September 2018,
2050 .
2052 [RFC8483] Song, L., Ed., Liu, D., Vixie, P., Kato, A., and S. Kerr,
2053 "Yeti DNS Testbed", RFC 8483, DOI 10.17487/RFC8483,
2054 October 2018, .
2056 [RFC8492] Harkins, D., Ed., "Secure Password Ciphersuites for
2057 Transport Layer Security (TLS)", RFC 8492,
2058 DOI 10.17487/RFC8492, February 2019,
2059 .
2061 [RFC8493] Kunze, J., Littman, J., Madden, E., Scancella, J., and C.
2062 Adams, "The BagIt File Packaging Format (V1.0)", RFC 8493,
2063 DOI 10.17487/RFC8493, October 2018,
2064 .
2066 [RFC8498] Mohali, M., "A P-Served-User Header Field Parameter for an
2067 Originating Call Diversion (CDIV) Session Case in the
2068 Session Initiation Protocol (SIP)", RFC 8498,
2069 DOI 10.17487/RFC8498, February 2019,
2070 .
2072 [RFCYEAR] RFC Editor, "Number of RFC Published per YEAR", 2020,
2073 .
2075 [SSCH] Allen Institute for AI, "Semantic Scholar", 2020,
2076 .
2078 [TLS13IMP]
2079 TLS WG, "TLS 1.3 Implementations", 2020,
2080 .
2083 [TRKR] IETF, "IETF Data Tracker", 2020,
2084 .
2086 Author's Address
2088 Christian Huitema
2089 Private Octopus Inc.
2090 427 Golfcourse Rd
2091 Friday Harbor WA 98250
2092 U.S.A
2094 Email: huitema@huitema.net