idnits 2.17.1 draft-dulaunoy-dnsop-passive-dns-cof-03.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** There is 1 instance of too long lines in the document, the longest one being 7 characters in excess of 72. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (June 12, 2017) is 2500 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Missing Reference: 'WEIMERPDNS' is mentioned on line 432, but not defined == Missing Reference: 'REST' is mentioned on line 428, but not defined == Missing Reference: 'DNSDB' is mentioned on line 407, but not defined == Missing Reference: 'PDNSCERTAT' is mentioned on line 409, but not defined == Missing Reference: 'PDNSCIRCL' is mentioned on line 415, but not defined == Missing Reference: 'PDNSCOF' is mentioned on line 424, but not defined == Missing Reference: 'PDNSCLIENT' is mentioned on line 419, but not defined == Missing Reference: 'BAILIWICK' is mentioned on line 398, but not defined == Missing Reference: 'CACHEPOISONING' is mentioned on line 403, but not defined == Unused Reference: 'RFC5001' is defined on line 380, but no explicit reference was found in the text == Unused Reference: 'I-D.narten-iana-considerations-rfc2434bis' is defined on line 439, but no explicit reference was found in the text == Unused Reference: 'RFC3552' is defined on line 445, but no explicit reference was found in the text ** Obsolete normative reference: RFC 2234 (Obsoleted by RFC 4234) ** Obsolete normative reference: RFC 4627 (Obsoleted by RFC 7158, RFC 7159) Summary: 3 errors (**), 0 flaws (~~), 13 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Domain Name System Operations A. Dulaunoy 3 Internet-Draft CIRCL 4 Intended status: Informational A. Kaplan 5 Expires: December 14, 2017 CERT.at 6 P. Vixie 7 H. Stern 8 Farsight Security, Inc. 9 June 12, 2017 11 Passive DNS - Common Output Format 12 draft-dulaunoy-dnsop-passive-dns-cof-03 14 Abstract 16 This document describes a common output format of Passive DNS Servers 17 which clients can query. The output format description includes also 18 in addition a common semantic for each Passive DNS system. By having 19 multiple Passive DNS Systems adhere to the same output format for 20 queries, users of multiple Passive DNS servers will be able to 21 combine result sets easily. 23 Status of This Memo 25 This Internet-Draft is submitted in full conformance with the 26 provisions of BCP 78 and BCP 79. 28 Internet-Drafts are working documents of the Internet Engineering 29 Task Force (IETF). Note that other groups may also distribute 30 working documents as Internet-Drafts. The list of current Internet- 31 Drafts is at http://datatracker.ietf.org/drafts/current/. 33 Internet-Drafts are draft documents valid for a maximum of six months 34 and may be updated, replaced, or obsoleted by other documents at any 35 time. It is inappropriate to use Internet-Drafts as reference 36 material or to cite them other than as "work in progress." 38 This Internet-Draft will expire on December 14, 2017. 40 Copyright Notice 42 Copyright (c) 2017 IETF Trust and the persons identified as the 43 document authors. All rights reserved. 45 This document is subject to BCP 78 and the IETF Trust's Legal 46 Provisions Relating to IETF Documents 47 (http://trustee.ietf.org/license-info) in effect on the date of 48 publication of this document. Please review these documents 49 carefully, as they describe your rights and restrictions with respect 50 to this document. Code Components extracted from this document must 51 include Simplified BSD License text as described in Section 4.e of 52 the Trust Legal Provisions and are provided without warranty as 53 described in the Simplified BSD License. 55 Table of Contents 57 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 58 1.1. Requirements Language . . . . . . . . . . . . . . . . . . 3 59 2. Limitation . . . . . . . . . . . . . . . . . . . . . . . . . 3 60 3. Common Output Format . . . . . . . . . . . . . . . . . . . . 3 61 3.1. Overview . . . . . . . . . . . . . . . . . . . . . . . . 4 62 3.2. ABNF grammar . . . . . . . . . . . . . . . . . . . . . . 4 63 3.3. Mandatory Fields . . . . . . . . . . . . . . . . . . . . 4 64 3.3.1. rrname . . . . . . . . . . . . . . . . . . . . . . . 5 65 3.3.2. rrtype . . . . . . . . . . . . . . . . . . . . . . . 5 66 3.3.3. rdata . . . . . . . . . . . . . . . . . . . . . . . . 5 67 3.3.4. time_first . . . . . . . . . . . . . . . . . . . . . 5 68 3.3.5. time_last . . . . . . . . . . . . . . . . . . . . . . 6 69 3.4. Optional Fields . . . . . . . . . . . . . . . . . . . . . 6 70 3.4.1. count . . . . . . . . . . . . . . . . . . . . . . . . 6 71 3.4.2. bailiwick . . . . . . . . . . . . . . . . . . . . . . 6 72 3.5. Additional Fields . . . . . . . . . . . . . . . . . . . . 6 73 3.5.1. sensor_id . . . . . . . . . . . . . . . . . . . . . . 6 74 3.5.2. zone_time_first . . . . . . . . . . . . . . . . . . . 6 75 3.5.3. zone_time_last . . . . . . . . . . . . . . . . . . . 7 76 3.5.4. origin . . . . . . . . . . . . . . . . . . . . . . . 7 77 3.6. Additional Fields Registry . . . . . . . . . . . . . . . 7 78 4. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 7 79 5. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 7 80 6. Privacy Considerations . . . . . . . . . . . . . . . . . . . 7 81 7. Security Considerations . . . . . . . . . . . . . . . . . . . 8 82 8. References . . . . . . . . . . . . . . . . . . . . . . . . . 8 83 8.1. Normative References . . . . . . . . . . . . . . . . . . 8 84 8.2. References . . . . . . . . . . . . . . . . . . . . . . . 9 85 8.3. Informative References . . . . . . . . . . . . . . . . . 10 86 Appendix A. Examples . . . . . . . . . . . . . . . . . . . . . . 10 87 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 11 89 1. Introduction 91 Passive DNS is a technique described by Florian Weimer in 2005 in 92 Passive DNS replication, F Weimer - 17th Annual FIRST Conference on 93 Computer Security [WEIMERPDNS]. Since then multiple Passive DNS 94 implementations were created and evolved over time. Users of these 95 Passive DNS servers may query a server (often via WHOIS [RFC3912] or 96 HTTP REST [REST]), parse the results and process them in other 97 applications. 99 There are multiple implementations of Passive DNS software. Users of 100 passive DNS query each implementation and aggregate the results for 101 their search. This document describes the output format of four 102 Passive DNS Systems ([DNSDB], [PDNSCERTAT], [PDNSCIRCL] and 103 [PDNSCOF]) which are in use today and which already share a nearly 104 identical output format. As the format and the meaning of output 105 fields from each Passive DNS need to be consistent, we propose in 106 this document a solution to commonly name each field along with their 107 corresponding interpretation. The format follows a simple key-value 108 structure in JSON [RFC4627] format. The benefit of having a 109 consistent Passive DNS output format is that multiple client 110 implementations can query different servers without having to have a 111 separate parser for each individual server. passivedns-client 112 [PDNSCLIENT] currently implements multiple parsers due to a lack of 113 standardization. The document does not describe the protocol (e.g. 114 WHOIS [RFC3912], HTTP REST [REST]) nor the query format used to query 115 the Passive DNS. Neither does this document describe "pre-recursor" 116 Passive DNS Systems. Both of these are separate topics and deserve 117 their own RFC document. 119 1.1. Requirements Language 121 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 122 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 123 document are to be interpreted as described in RFC 2119 [RFC2119]. 125 2. Limitation 127 As a Passive DNS servers can include protection mechanisms for their 128 operation, results might be different due to those protection 129 measures. These mechanisms filter out DNS answers if they fail some 130 criteria. The bailiwick algorithm [BAILIWICK] protects the Passive 131 DNS Database from cache poisoning attacks [CACHEPOISONING]. Another 132 limitation that clients querying the database need to be aware of is 133 that each query simply gets a snapshot-answer of the time of 134 querying. Clients MUST NOT rely on consistent answers. Nor must 135 they assume that answers must be identical across multiple Passive 136 DNS Servers. 138 3. Common Output Format 139 3.1. Overview 141 The formatting of the answer follows the JSON [RFC4627] format. In 142 fact, it is a subset of the full JSON language. Notable differences 143 are the modified definition of whitespace ("ws"). The order of the 144 fields is not significant for the same resource type. 146 The intent of this output format is to be easily parsable by scripts. 147 Each JSON object is expressed on a single line to be processed by the 148 client line-by-line. Every implementation MUST support the JSON 149 output format. 151 Examples of JSON (Appendix A) output are in the appendix. 153 3.2. ABNF grammar 155 Formal grammar as defined in ABNF [RFC2234] 157 answer = entries 158 entries = * ( entry CR) 159 entry = "{" keyvallist "}" 160 keyvallist = [ member *( value-separator member ) ] 161 member = qm field qm name-separator value 162 name-separator = ws %x3A ws ; a ":" colon 163 value = value ; as defined in the JSON RFC 164 value-separator = ws %x2C ws ; , comma. As defined in JSON 165 field = "rrname" | "rrtype" | "rdata" | "time_first" | 166 "time_last" | "count" | "bailiwick" | "sensor_id" | 167 "zone_time_first" | "zone_time_last" | "origin" | futureField 168 futureField = string 169 CR = %x0D 170 qm = %x22 ; " a quotation mark 171 ws = *( 172 %x20 | ; Space 173 %x09 ; Horizontal tab 174 ) 176 Note that value is defined in JSON [RFC4627] and has the exact same 177 specification as there. The same goes for the definition of string. 179 3.3. Mandatory Fields 181 Implementation MUST support all the mandatory fields. 183 Uniqueness property: the tuple (rrname,rrtype,rdata) will always be 184 unique within one answer per server. While rrname and rrtype are 185 always individual JSON primitive types (strings, numbers, booleans or 186 null), rdata MAY return multiple resource records or a single record. 187 When multiple resource records are returned, rdata MUST be a JSON 188 array. In the case of a single resource record is returned, rdata 189 MUST be a JSON string. 191 3.3.1. rrname 193 This field returns the name of the queried resource. 195 3.3.2. rrtype 197 This field returns the resource record type as seen by the passive 198 DNS. The key is rrtype and the value is in the interpreted record 199 type represented as a JSON [RFC4627] string. If the value cannot be 200 interpreted the decimal value is returned following the principle of 201 transparency as described in RFC 3597 [RFC3597]. Then the decimal 202 value is represented as a JSON [RFC4627] number. The resource record 203 type can be any values as described by IANA in the DNS parameters 204 document in the section 'Resource Record (RR) TYPEs' 205 (http://www.iana.org/assignments/dns-parameters). Currently known 206 and supported textual descriptions of rrtypes are: A, AAAA, CNAME, 207 PTR, SOA, TXT, DNAME, NS, SRV, RP, NAPTR, HINFO, A6. A client MUST 208 be able to understand these textual rrtype values represented as a 209 JSON [RFC4627] string. In addition, a client MUST be able to handle 210 a decimal value (as mentioned above) as answer represented as a JSON 211 [RFC4627] number. 213 3.3.3. rdata 215 This field returns the resource records of the queried resource. 216 When multiple resource records are returned, rdata MUST be a JSON 217 array. In the case of a single resource record is returned, rdata 218 MUST be a JSON string. Each resource record is represented as a JSON 219 [RFC4627] string. Each resource record MUST be escaped as defined in 220 section 2.6 of RFC4627 [RFC4627]. Depending on the rrtype, this can 221 be an IPv4 or IPv6 address, a domain name (as in the case of CNAMEs), 222 an SPF record, etc. A client MUST be able to interpret any value 223 which is legal as the right hand side in a DNS master file RFC 1035 224 [RFC1035] and RFC 1034 [RFC1034]. If the rdata came from an unknown 225 DNS resource records, the server must follow the transparency 226 principle as described in RFC 3597 [RFC3597]. 228 3.3.4. time_first 230 This field returns the first time that the record / unique tuple 231 (rrname, rrtype, rdata) has been seen by the passive DNS. The date 232 is expressed in seconds (decimal) since 1st of January 1970 (Unix 233 timestamp). The time zone MUST be UTC. This field is represented as 234 a JSON [RFC4627] number. 236 3.3.5. time_last 238 This field returns the last time that the unique tuple (rrname, 239 rrtype, rdata) record has been seen by the passive DNS. The date is 240 expressed in seconds (decimal) since 1st of January 1970 (Unix 241 timestamp). The time zone MUST be UTC. This field is represented as 242 a JSON [RFC4627] number. 244 3.4. Optional Fields 246 Implementations SHOULD support one or more fields. 248 3.4.1. count 250 Specifies how many authoritative DNS answers were received at the 251 Passive DNS Server's collectors with exactly the given set of values 252 as answers (i.e. same data in the answer set - compare with the 253 uniqueness property in "Mandatory Fields"). The number of requests 254 is expressed as a decimal value. This field is represented as a JSON 255 [RFC4627] number. 257 3.4.2. bailiwick 259 The bailiwick is the best estimate of the apex of the zone where this 260 data is authoritative. 262 3.5. Additional Fields 264 Implementations MAY support the following fields: 266 3.5.1. sensor_id 268 This field returns the sensor information where the record was seen. 269 It is represented as a JSON [RFC4627] string. 271 If the data originate from sensors or probes which are part of a 272 publicly-known gathering or measurement system (e.g. RIPE Atlas), a 273 JSON [RFC4627] string SHOULD be prefixed. 275 3.5.2. zone_time_first 277 This field returns the first time that the unique tuple (rrname, 278 rrtype, rdata) record has been seen via master file import. The date 279 is expressed in seconds (decimal) since 1st of January 1970 (Unix 280 timestamp). The time zone MUST be UTC. This field is represented as 281 a JSON [RFC4627] number. 283 3.5.3. zone_time_last 285 This field returns the last time that the unique tuple (rrname, 286 rrtype, rdata) record has been seen via master file import. The date 287 is expressed in seconds (decimal) since 1st of January 1970 (Unix 288 timestamp). The time zone MUST be UTC. This field is represented as 289 a JSON [RFC4627] number. 291 3.5.4. origin 293 Specifies the resource origin of the Passive DNS response. This 294 field is represented as a Uniform Resource Identifier [RFC3986] 295 (URI). 297 3.6. Additional Fields Registry 299 In accordance with [RFC6648], designers of new passive DNS 300 applications that would need additional fields can request and 301 register new field name at https://github.com/adulau/pdns-qof/wiki/ 302 Additional-Fields. 304 4. Acknowledgements 306 Thanks to the Passive DNS developers who contributed to the document. 308 5. IANA Considerations 310 This memo includes no request to IANA. 312 6. Privacy Considerations 314 Passive DNS Servers capture DNS answers from multiple collecting 315 points ("sensors") which are located on the Internet-facing side of 316 DNS recursors ("post-recursor passive DNS"). In this process, they 317 intentionally omit the source IP, source port, destination IP and 318 destination port from the captured packets. Since the data is 319 captured "post-recursor", the timing information (who queries what) 320 is lost, since the recursor will cache the results. Furthermore, 321 since multiple sensors feed into a passive DNS server, the resulting 322 data gets mixed together, reducing the likelihood that Passive DNS 323 Servers are able to find out much about the actual person querying 324 the DNS records nor who actually sent the query. In this sense, 325 passive DNS Servers are similar to keeping an archive of all previous 326 phone books - if public DNS records can be compared to phone numbers 327 - as they often are. Nevertheless, the authors strongly encourage 328 Passive DNS implementors to take special care of privacy issues. 329 bortzmeyer-dnsop-dns-privacy is an excellent starting point for this. 330 Finally, the overall recommendations in RFC6973 [RFC6973] should be 331 taken into consideration when designing any application which uses 332 Passive DNS data. 334 7. Security Considerations 336 In some cases, Passive DNS output might contain confidential 337 information and its access might be restricted. When a user is 338 querying multiple Passive DNS and aggregating the data, the 339 sensitivity of the data must be considered. 341 8. References 343 8.1. Normative References 345 [RFC1034] Mockapetris, P., "Domain names - concepts and facilities", 346 STD 13, RFC 1034, DOI 10.17487/RFC1034, November 1987, 347 . 349 [RFC1035] Mockapetris, P., "Domain names - implementation and 350 specification", STD 13, RFC 1035, DOI 10.17487/RFC1035, 351 November 1987, . 353 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 354 Requirement Levels", BCP 14, RFC 2119, 355 DOI 10.17487/RFC2119, March 1997, 356 . 358 [RFC2234] Crocker, D., Ed. and P. Overell, "Augmented BNF for Syntax 359 Specifications: ABNF", RFC 2234, DOI 10.17487/RFC2234, 360 November 1997, . 362 [RFC3597] Gustafsson, A., "Handling of Unknown DNS Resource Record 363 (RR) Types", RFC 3597, DOI 10.17487/RFC3597, September 364 2003, . 366 [RFC3912] Daigle, L., "WHOIS Protocol Specification", RFC 3912, 367 DOI 10.17487/RFC3912, September 2004, 368 . 370 [RFC3986] Berners-Lee, T., Fielding, R., and L. Masinter, "Uniform 371 Resource Identifier (URI): Generic Syntax", STD 66, 372 RFC 3986, DOI 10.17487/RFC3986, January 2005, 373 . 375 [RFC4627] Crockford, D., "The application/json Media Type for 376 JavaScript Object Notation (JSON)", RFC 4627, 377 DOI 10.17487/RFC4627, July 2006, 378 . 380 [RFC5001] Austein, R., "DNS Name Server Identifier (NSID) Option", 381 RFC 5001, DOI 10.17487/RFC5001, August 2007, 382 . 384 [RFC6648] Saint-Andre, P., Crocker, D., and M. Nottingham, 385 "Deprecating the "X-" Prefix and Similar Constructs in 386 Application Protocols", BCP 178, RFC 6648, 387 DOI 10.17487/RFC6648, June 2012, 388 . 390 [RFC6973] Cooper, A., Tschofenig, H., Aboba, B., Peterson, J., 391 Morris, J., Hansen, M., and R. Smith, "Privacy 392 Considerations for Internet Protocols", RFC 6973, 393 DOI 10.17487/RFC6973, July 2013, 394 . 396 8.2. References 398 [BAILIWICK] 399 "Passive DNS Hardening", 2010, 400 . 403 [CACHEPOISONING] 404 "Black ops 2008: It's the end of the cache as we know 405 it.", 2008, . 407 [DNSDB] "DNSDB API", 2013, . 409 [PDNSCERTAT] 410 "pDNS presentation at 4th Centr R&D workshop Frankfurt Jun 411 5th 2012", 2012, 412 . 415 [PDNSCIRCL] 416 "CIRCL Passive DNS", 2012, . 419 [PDNSCLIENT] 420 "Queries 5 major Passive DNS databases: BFK, CERTEE, 421 DNSParse, ISC, and VirusTotal.", 2013, 422 . 424 [PDNSCOF] "Passive DNS server interface using the common output 425 format", 2013, . 428 [REST] "Representational State Transfer (REST)", 2000, 429 . 432 [WEIMERPDNS] 433 "Passive DNS Replication", 2005, 434 . 437 8.3. Informative References 439 [I-D.narten-iana-considerations-rfc2434bis] 440 Narten, T. and H. Alvestrand, "Guidelines for Writing an 441 IANA Considerations Section in RFCs", draft-narten-iana- 442 considerations-rfc2434bis-09 (work in progress), March 443 2008. 445 [RFC3552] Rescorla, E. and B. Korver, "Guidelines for Writing RFC 446 Text on Security Considerations", BCP 72, RFC 3552, 447 DOI 10.17487/RFC3552, July 2003, 448 . 450 Appendix A. Examples 452 The JSON output are represented on multiple lines for readability but 453 each JSON object should on a single line. 455 If you query a passive DNS for the rrname www.ietf.org, the passive 456 dns common output format can be: 458 {"count": 102, "time_first": 1298412391, "rrtype": "AAAA", 459 "rrname": "www.ietf.org", "rdata": "2001:1890:1112:1::20", 460 "time_last": 1302506851} 461 {"count": 59, "time_first": 1384865833, "rrtype": "A", 462 "rrname": "www.ietf.org", "rdata": "4.31.198.44", 463 "time_last": 1389022219} 465 If you query a passive DNS for the rrname ietf.org, the passive dns 466 common output format can be: 468 {"count": 109877, "time_first": 1298398002, "rrtype": "NS", 469 "rrname": "ietf.org", "rdata": "ns1.yyz1.afilias-nst.info", 470 "time_last": 1389095375} 471 {"count": 4, "time_first": 1298495035, "rrtype": "A", 472 "rrname": "ietf.org", "rdata": "64.170.98.32", 473 "time_last": 1298495035} 474 {"count": 9, "time_first": 1317037550, "rrtype": "AAAA", 475 "rrname": "ietf.org", "rdata": "2001:1890:123a::1:1e", 476 "time_last": 1330209752} 478 Please note that in the examples above, any backslashes "\" can be 479 ignored and are an artefact of the tools which produced this 480 document. 482 Authors' Addresses 484 Alexandre Dulaunoy 485 CIRCL 486 41, avenue de la gare 487 Luxembourg L-1611 488 Luxembourg 490 Phone: (+352) 247 88444 491 Email: alexandre.dulaunoy@circl.lu 492 URI: http://www.circl.lu/ 494 L. Aaron Kaplan 495 CERT.at 496 Karlsplatz 1/2/9 497 Vienna A-1010 498 Austria 500 Phone: +43 1 5056416 78 501 Email: kaplan@cert.at 502 URI: http://www.cert.at/ 504 Paul Vixie 505 Farsight Security, Inc. 506 11400 La Honda Road 507 Woodside, California 94062 508 U.S.A. 510 Email: paul@redbarn.org 511 URI: https://www.farsightsecurity.com/ 512 Henry Stern 513 Farsight Security, Inc. 514 11400 La Honda Road 515 Woodside, California 94062 516 U.S.A. 518 Phone: +1 650 542-7836 519 Email: henry@stern.ca 520 URI: https://www.farsightsecurity.com/