idnits 2.17.1 draft-kucherawy-spfbis-experiment-04.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack a both a reference to RFC 2119 and the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords. RFC 2119 keyword, line 341: '... SHOULD publish both types and M...' Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (April 4, 2012) is 4399 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- -- Obsolete informational reference (is this intentional?): RFC 4408 (ref. 'SPF') (Obsoleted by RFC 7208) Summary: 1 error (**), 0 flaws (~~), 1 warning (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 SPFBIS Working Group M. Kucherawy 3 Internet-Draft Cloudmark 4 Intended status: Informational April 4, 2012 5 Expires: October 6, 2012 7 Resolution of The SPF/Sender-ID Experiment 8 draft-kucherawy-spfbis-experiment-04 10 Abstract 12 In 2006 the IETF published a suite of protocol documents comprising 13 SPF and Sender-ID, two proposed email authentication protocols. 14 Because of interoperability concerns created by simultaneous use of 15 the two protocols by a receiver, and some concerns with Sender-ID and 16 compatibility with existing standards, the IESG required them to have 17 Experimental status and invited the community to observe their 18 deployments for a period of time, hoping convergence would be 19 possible later. 21 After six years, sufficient experience and evidence have been 22 collected that the experiment thus created can be considered 23 concluded, and a single protocol can be advanced. This memo presents 24 those findings. 26 Status of this Memo 28 This Internet-Draft is submitted in full conformance with the 29 provisions of BCP 78 and BCP 79. 31 Internet-Drafts are working documents of the Internet Engineering 32 Task Force (IETF). Note that other groups may also distribute 33 working documents as Internet-Drafts. The list of current Internet- 34 Drafts is at http://datatracker.ietf.org/drafts/current/. 36 Internet-Drafts are draft documents valid for a maximum of six months 37 and may be updated, replaced, or obsoleted by other documents at any 38 time. It is inappropriate to use Internet-Drafts as reference 39 material or to cite them other than as "work in progress." 41 This Internet-Draft will expire on October 6, 2012. 43 Copyright Notice 45 Copyright (c) 2012 IETF Trust and the persons identified as the 46 document authors. All rights reserved. 48 This document is subject to BCP 78 and the IETF Trust's Legal 49 Provisions Relating to IETF Documents 50 (http://trustee.ietf.org/license-info) in effect on the date of 51 publication of this document. Please review these documents 52 carefully, as they describe your rights and restrictions with respect 53 to this document. Code Components extracted from this document must 54 include Simplified BSD License text as described in Section 4.e of 55 the Trust Legal Provisions and are provided without warranty as 56 described in the Simplified BSD License. 58 Table of Contents 60 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 61 2. The Need For Consensus . . . . . . . . . . . . . . . . . . . . 3 62 3. Evidence of Deployment . . . . . . . . . . . . . . . . . . . . 4 63 4. Evidence of Differences . . . . . . . . . . . . . . . . . . . . 5 64 5. Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . 5 65 5.1. From the Evidence . . . . . . . . . . . . . . . . . . . . . 5 66 5.2. Recommendations to the IESG . . . . . . . . . . . . . . . . 6 67 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . . 7 68 7. Security Considerations . . . . . . . . . . . . . . . . . . . . 7 69 8. Informative References . . . . . . . . . . . . . . . . . . . . 7 70 Appendix A. Experiences Developing SPF . . . . . . . . . . . . . . 8 71 Appendix B. Acknowledgments . . . . . . . . . . . . . . . . . . . 9 72 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . . 9 74 1. Introduction 76 In April, 2006, the IETF published the [SPF] and Sender-ID email 77 authentication protocols, the latter consisting of three documents 78 ([SUBMITTER], [SENDER-ID], and [PRA]). Both of these enable one to 79 publish via the Domain Name System a policy declaring which mail 80 servers were authorized to send email on behalf of a specific domain 81 name. The two protocols made use of this policy statement and some 82 specific (but different) logic to evaluate whether or not the email 83 client sending or relaying a message was authorized to do so. 85 Because Sender-ID could use the same policy statement as SPF, the 86 IESG at the time was concerned that an implementation of Sender-ID 87 might erroneously apply that statement to a message and, depending on 88 selected recipient actions, could improperly interfere with message 89 delivery. As a result, the IESG required the publication of all of 90 these documents as Experimental, and requested that the community 91 observe deployment and operation of the protocols over a period of 92 two years from publication in order to determine a reasonable path 93 forward. (For further details about the IESG's concern, see the IESG 94 Note prepended to all of those documents.) 96 Accordingly, this memo resolves the experiment by presenting evidence 97 regarding both deployment and efficacy of the two protocols, and 98 further discusses the increasing need for consensus. At the end it 99 presents conclusions based on the data collected. 101 2. The Need For Consensus 103 These two protocols fall into a family of protocols that provide 104 domain-level email authentication services. For reference, another 105 prominent one is [DKIM]. Various efforts exist that use these as 106 building blocks to increased abuse filtering capabilities, and indeed 107 this sort of work has spawned another working group in the 108 Applications area, with still more of these incubating in 109 associations and trade groups outside of the IETF. 111 There is thus some palpable interest in having a path authorization 112 scheme, as well as a domain-level signing scheme, on the Standards 113 Track so that these newer technologies can develop with confidence. 114 This is, in part, why the community has decided to expend the effort 115 to bring this experiment to a conclusion and document the results, 116 and then advance a single path authorization technology. 118 3. Evidence of Deployment 120 Two large-scale DNS surveys were run that looked for the two 121 supported kinds of resource records (RR) that can contain SPF policy 122 statements. 124 One data source for this report requested SPF records from one 125 million domains. Approximately 287,267 domains included non-error 126 and non-empty replies. Of these, 287,250 included type 16 (DNS RR 127 TXT) replies, 4,613 included type 99 (DNS RR SPF) replies, and 4,596 128 included both types. 130 Another source requested SPF records from 251,651 domains for which 131 there was a history of previous observed SPF evaluations. Of these, 132 136,018 returned type 16 answers, 2,605 returned type 99 answers, 133 2,439 returned both types, and 115,465 returned neither. Of those 134 answers retrieved, 6,479 included records that start with the string 135 "spf2.0/pra" which are specific requests for Sender-ID processing by 136 receivers. 138 During this second survey, some domains were observed to provide 139 immediate answers for type 16 queries, but would time out waiting for 140 replies to type 99 queries. For example, it was observed that 3,953 141 distinct domains in the survey returned a result of some kind (a 142 record or an error) for the TXT query in time N, while the SPF query 143 ultimately failed but only after at least time 4N. 145 It is likely impossible to determine from a survey which MTAs have 146 SPF and/or Sender-ID checking enabled at message ingress since it 147 does not appear, for example, in the reply to the EHLO command from 148 extended [SMTP]. We therefore rely on evidence found via web 149 searches, and observed the following: 151 o A web site [SID-IMPL] dedicated to highlighting Sender-ID 152 implementations last updated in late 2007 listed 13 153 implementations, which we assume means they implement the PRA 154 checks. At least one of them is known no longer to be supported 155 by its vendor. 157 o The [OPENSPF] web site maintains a list of known implementations 158 of SPF. At the time of this memo's writing it listed six 159 libraries, 22 MTAs with built-in SPF implementations, and numerous 160 patches for MTAs and mail clients. 162 In a survey of numerous MTAs in current or recent use, only two 163 (Santronics WinServer and McAfee MxLogic) were found to contain 164 implementations of the SMTP SUBMITTER extension as part of the MTA 165 service, which could act as an enabler to Sender-ID. An unknown 166 number of clients implement it; although there is substantial 167 activity showing its use in logs, it is unclear whether these are 168 separate implementations by legitimate senders, or merely instances 169 of distributed automated malware seeking to improve their odds of 170 reaching the end user. 172 A survey was done of queries to type 16 and type 99 records by 173 observing nameserver logs. Only a few queries were ever received for 174 type 99 records, and those almost exclusively came from one large 175 email service provider that queried for both types. The vast 176 majority of other querying agents only ever requested type 16. 178 [pending: SPF query results from Hotmail] 180 [other data TBD] 182 4. Evidence of Differences 184 It is plain from inspection of the two protocols that they have much 185 in common: For a single message, both require the same number of DNS 186 queries, and both require the same code to parse the result. The PRA 187 algorithm applied by Sender-ID is, however, more expensive than 188 simply extracting the domain name from the omnipresent 189 RFC5321.MailFrom. Thus, SPF is cheaper to apply to a message. 191 One set of specific data collected by a working group contributor 192 shows that in more than 95.5% of cases, Sender-ID and SPF reach the 193 same conclusion about a message, meaning either both protocols return 194 a "pass" result or both return a "fail" result. The data set 195 yielding this response could not further characterize the cases in 196 which the answers differed. 198 [pending: MAIL FROM/PRA comparison report from Hotmail] 200 [other data TBD] 202 5. Conclusions 204 It is standard procedure within the IETF to document as standard 205 those protocols and practices that have come into sufficient common 206 use as to become part of the basic infrastructure. 208 5.1. From the Evidence 210 Given the six years that have passed since the publication of the 211 experimental RFCs, and the evidence reported in the earlier sections 212 of this document, the following conclusions are supported: 214 1. There has not been substantial adoption of the type 99 (SPF) DNS 215 resource record. In both large-scale surveys performed for this 216 work, less than 2% of responding domains published type 99 217 records, and almost no clients requested them. 219 2. Of the records retrieved, fewer than 5% requested processing of 220 messages using the PRA algorithm that was an integral part of 221 Sender-ID. 223 3. No data collected showed any substantial operational benefit 224 (e.g., cheaper processing, improved accuracy) to using Sender-ID 225 over SPF. 227 4. A survey of implementations shows significant support for both 228 protocols, though there were more implementations in support of 229 SPF than of Sender-ID. Further, the SPF implementations showed 230 better upkeep and current interest than the Sender-ID 231 implemenations. 233 5. A survey of implementations shows no significant use of the 234 SUBMITTER extension by servers, but some by clients. 236 5.2. Recommendations to the IESG 238 In light of the above, the working group recommends to the IESG the 239 following: 241 1. that the experiment comprising the series of RFCs defining the 242 SUBMITTER SMTP extension, the Sender-ID mechanism, the Purported 243 Responsible address algorithm, and SPF, be considered concluded; 245 2. that [SPF] be advanced to the standards track after appropriate 246 technical review with respect to the deployed base; 248 3. that this revision to SPF deprecate the use of the type 99 DNS 249 resource record by both clients and servers; 251 4. that [SUBMITTER], [SENDER-ID], and [PRA] have failed to gain any 252 substantial adoption and are thus, de facto, obsolete (and thus 253 could be marked "Obsoleted" and "Historic" by some future 254 action). 256 Appendix A is offered as a cautionary review of problems that 257 affected the process of developing SPF and Sender-ID in terms of use 258 of the DNS. 260 6. IANA Considerations 262 This memo presents no actions for IANA. [RFC Editor: Please remove 263 this section prior to publication.] 265 7. Security Considerations 267 This memo contains information for the community only, akin to an 268 implementation report, and does not introduce any new security 269 concerns. Its implications could, in fact, resolve some. 271 8. Informative References 273 [DKIM] Crocker, D., Ed., Hansen, T., Ed., and M. Kucherawy, Ed., 274 "DomainKeys Identified Mail (DKIM) Signatures", RFC 6376, 275 September 2011. 277 [OPENSPF] "Sender Policy Framework: Project Overview", 278 . 280 [PRA] Lyon, J., "Purported Responsible Address in E-Mail 281 Messages", RFC 4407, April 2006. 283 [SENDER-ID] 284 Lyon, J. and M. Wong, "Sender ID: Authenticating E-Mail", 285 RFC 4406, April 2006. 287 [SID-IMPL] 288 "Sender ID Framework Industry Support and Solutions", 289 October 2007, . 292 [SMTP] Klensin, J., "Simple Mail Transfer Protocol", RFC 5321, 293 October 2008. 295 [SPF] Wong, M. and W. Schlitt, "Sender Policy Framework (SPF) 296 for Authorizing Use of Domains in E-Mail, Version 1", 297 RFC 4408, April 2006. 299 [SUBMITTER] 300 Allman, E. and H. Katz, "SMTP Service Extension for 301 Indicating the Responsible Submitter of an E-Mail 302 Message", RFC 4405, April 2006. 304 Appendix A. Experiences Developing SPF 306 SPF was originally developed by a community of interested developers 307 outside the IETF. It was brought to the IETF for standardization 308 only after the specification was relatively mature and ready for the 309 rigors of the IETF publication process. 311 At the time, the prospect of getting a DNS resource record (RR) type 312 allocated for SPF was not seriously considered, partly because it was 313 perceived to have high barriers to entry. As a result, by the time 314 the working group was formed, there was already a substantial and 315 growing installed base that had SPF running using TXT RRs. 316 Eventually the application was made for the new RR type as a result 317 of pressure from the DNS experts in the community, who encouraged 318 doing so as the preferred path toward using the DNS for storing such 319 things as policy data. 321 Later, after type 99 was assigned (long after IESG approval of the 322 document, in fact), a plan was put into place to effect a gradual 323 transition to using type 99 instead of using type 16. This plan 324 failed to take effect for four primary reasons: 326 1. there was hesitation to make the transition because of concerns 327 that nameservers (and, in fact, DNS-aware firewalls) would drop 328 or reject requests for unknown RR types (see Section 3 for 329 evidence of this), which means successful rollout of a new RR 330 type is contingent upon widespread adoption of updated 331 nameservers and resolver functions; 333 2. many DNS provisioning tools (e.g., web interfaces to controlling 334 DNS zone data) were, and still are, typically lethargic about 335 adding support for new RR types; 337 3. the substantial deployed base was already using type 16, and it 338 was working just fine, leading to inertia; 340 4. [SPF] itself included a faulty transition plan: It said a server 341 SHOULD publish both types and MUST publish at least one, while a 342 client can query either or both, which means both can claim to be 343 fully compliant while failing utterly to interoperate. 345 It is likely that this will happen again if the bar to creating new 346 RR types even for experimental development purposes is not lowered, 347 and handling of unknown RR types becomes generally more graceful. 349 There are DNS experts within the community that will undoubtedly 350 point to DNS servers and firewalls that mistreat queries for unknown 351 RR types, and claim they are broken, as a way of answering this 352 concern. This is undoubtedly correct, but the reality is that they 353 are among us and likely will be for some time, and this needs to be 354 considered as new protocols and IETF procedures are developed. 356 Appendix B. Acknowledgments 358 The following provided operational data that contributed to the 359 evidence presented above: 361 Cisco: contributed data about observed Sender-ID and SPF records in 362 the DNS for a large number of domains 364 Hotmail: contributed data about the difference between 365 RFC5321.MailFrom and RFC5322.From domains across large mail 366 volumes, and a survey of DNS queries observed in response to 367 outgoing mail traffic 369 John Levine: conducted a survey of DNS server logs to evaluate SPF- 370 related query traffic 372 Santronics: contributed data about the use of the SUBMITTER 373 extension in aggregate SMTP client traffic 375 The Trusted Domain Project: contributed data about the difference 376 between Sender-ID and SPF results, and conducted one of the two 377 detailed TXT/SPF record surveys including collecting timing data 379 The author would also like to thank the following for their 380 contributions to the development of the text in this memo: Dave 381 Crocker, and Scott Kitterman 383 Author's Address 385 Murray S. Kucherawy 386 Cloudmark 387 128 King St., 2nd Floor 388 San Francisco, CA 94107 389 USA 391 Phone: +1 415 946 3800 392 Email: msk@cloudmark.com