idnits 2.17.1 draft-dulaunoy-dnsop-passive-dns-cof-04.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** There is 1 instance of too long lines in the document, the longest one being 7 characters in excess of 72. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (June 21, 2018) is 2136 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Missing Reference: 'WEIMERPDNS' is mentioned on line 437, but not defined == Missing Reference: 'REST' is mentioned on line 433, but not defined == Missing Reference: 'DNSDB' is mentioned on line 412, but not defined == Missing Reference: 'PDNSCERTAT' is mentioned on line 414, but not defined == Missing Reference: 'PDNSCIRCL' is mentioned on line 420, but not defined == Missing Reference: 'PDNSCOF' is mentioned on line 429, but not defined == Missing Reference: 'PDNSCLIENT' is mentioned on line 424, but not defined == Missing Reference: 'BAILIWICK' is mentioned on line 403, but not defined == Missing Reference: 'CACHEPOISONING' is mentioned on line 408, but not defined == Unused Reference: 'RFC5001' is defined on line 385, but no explicit reference was found in the text == Unused Reference: 'I-D.narten-iana-considerations-rfc2434bis' is defined on line 444, but no explicit reference was found in the text == Unused Reference: 'RFC3552' is defined on line 450, but no explicit reference was found in the text ** Obsolete normative reference: RFC 2234 (Obsoleted by RFC 4234) ** Obsolete normative reference: RFC 4627 (Obsoleted by RFC 7158, RFC 7159) Summary: 3 errors (**), 0 flaws (~~), 13 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Domain Name System Operations A. Dulaunoy 3 Internet-Draft CIRCL 4 Intended status: Informational A. Kaplan 5 Expires: December 23, 2018 CERT.at 6 P. Vixie 7 H. Stern 8 Farsight Security, Inc. 9 June 21, 2018 11 Passive DNS - Common Output Format 12 draft-dulaunoy-dnsop-passive-dns-cof-04 14 Abstract 16 This document describes a common output format of Passive DNS Servers 17 which clients can query. The output format description includes also 18 in addition a common semantic for each Passive DNS system. By having 19 multiple Passive DNS Systems adhere to the same output format for 20 queries, users of multiple Passive DNS servers will be able to 21 combine result sets easily. 23 Status of This Memo 25 This Internet-Draft is submitted in full conformance with the 26 provisions of BCP 78 and BCP 79. 28 Internet-Drafts are working documents of the Internet Engineering 29 Task Force (IETF). Note that other groups may also distribute 30 working documents as Internet-Drafts. The list of current Internet- 31 Drafts is at https://datatracker.ietf.org/drafts/current/. 33 Internet-Drafts are draft documents valid for a maximum of six months 34 and may be updated, replaced, or obsoleted by other documents at any 35 time. It is inappropriate to use Internet-Drafts as reference 36 material or to cite them other than as "work in progress." 38 This Internet-Draft will expire on December 23, 2018. 40 Copyright Notice 42 Copyright (c) 2018 IETF Trust and the persons identified as the 43 document authors. All rights reserved. 45 This document is subject to BCP 78 and the IETF Trust's Legal 46 Provisions Relating to IETF Documents 47 (https://trustee.ietf.org/license-info) in effect on the date of 48 publication of this document. Please review these documents 49 carefully, as they describe your rights and restrictions with respect 50 to this document. Code Components extracted from this document must 51 include Simplified BSD License text as described in Section 4.e of 52 the Trust Legal Provisions and are provided without warranty as 53 described in the Simplified BSD License. 55 Table of Contents 57 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 58 1.1. Requirements Language . . . . . . . . . . . . . . . . . . 3 59 2. Limitation . . . . . . . . . . . . . . . . . . . . . . . . . 3 60 3. Common Output Format . . . . . . . . . . . . . . . . . . . . 3 61 3.1. Overview . . . . . . . . . . . . . . . . . . . . . . . . 4 62 3.2. ABNF grammar . . . . . . . . . . . . . . . . . . . . . . 4 63 3.3. Mandatory Fields . . . . . . . . . . . . . . . . . . . . 4 64 3.3.1. rrname . . . . . . . . . . . . . . . . . . . . . . . 5 65 3.3.2. rrtype . . . . . . . . . . . . . . . . . . . . . . . 5 66 3.3.3. rdata . . . . . . . . . . . . . . . . . . . . . . . . 5 67 3.3.4. time_first . . . . . . . . . . . . . . . . . . . . . 5 68 3.3.5. time_last . . . . . . . . . . . . . . . . . . . . . . 6 69 3.4. Optional Fields . . . . . . . . . . . . . . . . . . . . . 6 70 3.4.1. count . . . . . . . . . . . . . . . . . . . . . . . . 6 71 3.4.2. bailiwick . . . . . . . . . . . . . . . . . . . . . . 6 72 3.5. Additional Fields . . . . . . . . . . . . . . . . . . . . 6 73 3.5.1. sensor_id . . . . . . . . . . . . . . . . . . . . . . 6 74 3.5.2. zone_time_first . . . . . . . . . . . . . . . . . . . 6 75 3.5.3. zone_time_last . . . . . . . . . . . . . . . . . . . 7 76 3.5.4. origin . . . . . . . . . . . . . . . . . . . . . . . 7 77 3.6. Additional Fields Registry . . . . . . . . . . . . . . . 7 78 4. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 7 79 5. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 7 80 6. Privacy Considerations . . . . . . . . . . . . . . . . . . . 7 81 7. Security Considerations . . . . . . . . . . . . . . . . . . . 8 82 8. References . . . . . . . . . . . . . . . . . . . . . . . . . 8 83 8.1. Normative References . . . . . . . . . . . . . . . . . . 8 84 8.2. References . . . . . . . . . . . . . . . . . . . . . . . 9 85 8.3. Informative References . . . . . . . . . . . . . . . . . 10 86 Appendix A. Examples . . . . . . . . . . . . . . . . . . . . . . 10 87 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 11 89 1. Introduction 91 Passive DNS is a technique described by Florian Weimer in 2005 in 92 Passive DNS replication, F Weimer - 17th Annual FIRST Conference on 93 Computer Security [WEIMERPDNS]. Since then multiple Passive DNS 94 implementations were created and evolved over time. Users of these 95 Passive DNS servers may query a server (often via WHOIS [RFC3912] or 96 HTTP REST [REST]), parse the results and process them in other 97 applications. 99 There are multiple implementations of Passive DNS software. Users of 100 passive DNS query each implementation and aggregate the results for 101 their search. This document describes the output format of four 102 Passive DNS Systems ([DNSDB], [PDNSCERTAT], [PDNSCIRCL] and 103 [PDNSCOF]) which are in use today and which already share a nearly 104 identical output format. As the format and the meaning of output 105 fields from each Passive DNS need to be consistent, we propose in 106 this document a solution to commonly name each field along with their 107 corresponding interpretation. The format follows a simple key-value 108 structure in JSON [RFC4627] format. The benefit of having a 109 consistent Passive DNS output format is that multiple client 110 implementations can query different servers without having to have a 111 separate parser for each individual server. passivedns-client 112 [PDNSCLIENT] currently implements multiple parsers due to a lack of 113 standardization. The document does not describe the protocol (e.g. 114 WHOIS [RFC3912], HTTP REST [REST]) nor the query format used to query 115 the Passive DNS. Neither does this document describe "pre-recursor" 116 Passive DNS Systems. Both of these are separate topics and deserve 117 their own RFC document. The document describes the current best 118 practices implemented in various Passive DNS server implementations. 120 1.1. Requirements Language 122 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 123 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 124 document are to be interpreted as described in RFC 2119 [RFC2119]. 126 2. Limitation 128 As a Passive DNS servers can include protection mechanisms for their 129 operation, results might be different due to those protection 130 measures. These mechanisms filter out DNS answers if they fail some 131 criteria. The bailiwick algorithm [BAILIWICK] protects the Passive 132 DNS Database from cache poisoning attacks [CACHEPOISONING]. Another 133 limitation that clients querying the database need to be aware of is 134 that each query simply gets a snapshot-answer of the time of 135 querying. Clients MUST NOT rely on consistent answers. Nor must 136 they assume that answers must be identical across multiple Passive 137 DNS Servers. 139 3. Common Output Format 140 3.1. Overview 142 The formatting of the answer follows the JSON [RFC4627] format. In 143 fact, it is a subset of the full JSON language. Notable differences 144 are the modified definition of whitespace ("ws"). The order of the 145 fields is not significant for the same resource type. 147 The intent of this output format is to be easily parsable by scripts. 148 Each JSON object is expressed on a single line to be processed by the 149 client line-by-line. Every implementation MUST support the JSON 150 output format. 152 Examples of JSON (Appendix A) output are in the appendix. 154 3.2. ABNF grammar 156 Formal grammar as defined in ABNF [RFC2234] 158 answer = entries 159 entries = * ( entry CR) 160 entry = "{" keyvallist "}" 161 keyvallist = [ member *( value-separator member ) ] 162 member = qm field qm name-separator value 163 name-separator = ws %x3A ws ; a ":" colon 164 value = value ; as defined in the JSON RFC 165 value-separator = ws %x2C ws ; , comma. As defined in JSON 166 field = "rrname" | "rrtype" | "rdata" | "time_first" | 167 "time_last" | "count" | "bailiwick" | "sensor_id" | 168 "zone_time_first" | "zone_time_last" | "origin" | futureField 169 futureField = string 170 CR = %x0D 171 qm = %x22 ; " a quotation mark 172 ws = *( 173 %x20 | ; Space 174 %x09 ; Horizontal tab 175 ) 177 Note that value is defined in JSON [RFC4627] and has the exact same 178 specification as there. The same goes for the definition of string. 180 3.3. Mandatory Fields 182 Implementation MUST support all the mandatory fields. 184 Uniqueness property: the tuple (rrname,rrtype,rdata) will always be 185 unique within one answer per server. While rrname and rrtype are 186 always individual JSON primitive types (strings, numbers, booleans or 187 null), rdata MAY return multiple resource records or a single record. 188 When multiple resource records are returned, rdata MUST be a JSON 189 array. In the case of a single resource record is returned, rdata 190 MUST be a JSON string. 192 3.3.1. rrname 194 This field returns the name of the queried resource. 196 3.3.2. rrtype 198 This field returns the resource record type as seen by the passive 199 DNS. The key is rrtype and the value is in the interpreted record 200 type represented as a JSON [RFC4627] string. If the value cannot be 201 interpreted the decimal value is returned following the principle of 202 transparency as described in RFC 3597 [RFC3597]. Then the decimal 203 value is represented as a JSON [RFC4627] number. The resource record 204 type can be any values as described by IANA in the DNS parameters 205 document in the section 'Resource Record (RR) TYPEs' 206 (http://www.iana.org/assignments/dns-parameters). Currently known 207 and supported textual descriptions of rrtypes are: A, AAAA, CNAME, 208 PTR, SOA, TXT, DNAME, NS, SRV, RP, NAPTR, HINFO, A6. A client MUST 209 be able to understand these textual rrtype values represented as a 210 JSON [RFC4627] string. In addition, a client MUST be able to handle 211 a decimal value (as mentioned above) as answer represented as a JSON 212 [RFC4627] number. 214 3.3.3. rdata 216 This field returns the resource records of the queried resource. 217 When multiple resource records are returned, rdata MUST be a JSON 218 array. In the case of a single resource record is returned, rdata 219 MUST be a JSON string. Each resource record is represented as a JSON 220 [RFC4627] string. Each resource record MUST be escaped as defined in 221 section 2.6 of RFC4627 [RFC4627]. Depending on the rrtype, this can 222 be an IPv4 or IPv6 address, a domain name (as in the case of CNAMEs), 223 an SPF record, etc. A client MUST be able to interpret any value 224 which is legal as the right hand side in a DNS master file RFC 1035 225 [RFC1035] and RFC 1034 [RFC1034]. If the rdata came from an unknown 226 DNS resource records, the server must follow the transparency 227 principle as described in RFC 3597 [RFC3597]. 229 3.3.4. time_first 231 This field returns the first time that the record / unique tuple 232 (rrname, rrtype, rdata) has been seen by the passive DNS. The date 233 is expressed in seconds (decimal) since 1st of January 1970 (Unix 234 timestamp). The time zone MUST be UTC. This field is represented as 235 a JSON [RFC4627] number. 237 3.3.5. time_last 239 This field returns the last time that the unique tuple (rrname, 240 rrtype, rdata) record has been seen by the passive DNS. The date is 241 expressed in seconds (decimal) since 1st of January 1970 (Unix 242 timestamp). The time zone MUST be UTC. This field is represented as 243 a JSON [RFC4627] number. 245 3.4. Optional Fields 247 Implementations SHOULD support one or more fields. 249 3.4.1. count 251 Specifies how many authoritative DNS answers were received at the 252 Passive DNS Server's collectors with exactly the given set of values 253 as answers (i.e. same data in the answer set - compare with the 254 uniqueness property in "Mandatory Fields"). The number of requests 255 is expressed as a decimal value. This field is represented as a JSON 256 [RFC4627] number. 258 3.4.2. bailiwick 260 The bailiwick is the best estimate of the apex of the zone where this 261 data is authoritative. 263 3.5. Additional Fields 265 Implementations MAY support the following fields: 267 3.5.1. sensor_id 269 This field returns the sensor information where the record was seen. 270 It is represented as a JSON [RFC4627] string. 272 If the data originate from sensors or probes which are part of a 273 publicly-known gathering or measurement system (e.g. RIPE Atlas), a 274 JSON [RFC4627] string SHOULD be prefixed. 276 3.5.2. zone_time_first 278 This field returns the first time that the unique tuple (rrname, 279 rrtype, rdata) record has been seen via master file import. The date 280 is expressed in seconds (decimal) since 1st of January 1970 (Unix 281 timestamp). The time zone MUST be UTC. This field is represented as 282 a JSON [RFC4627] number. 284 3.5.3. zone_time_last 286 This field returns the last time that the unique tuple (rrname, 287 rrtype, rdata) record has been seen via master file import. The date 288 is expressed in seconds (decimal) since 1st of January 1970 (Unix 289 timestamp). The time zone MUST be UTC. This field is represented as 290 a JSON [RFC4627] number. 292 3.5.4. origin 294 Specifies the resource origin of the Passive DNS response. This 295 field is represented as a Uniform Resource Identifier [RFC3986] 296 (URI). 298 3.6. Additional Fields Registry 300 In accordance with [RFC6648], designers of new passive DNS 301 applications that would need additional fields can request and 302 register new field name at https://github.com/adulau/pdns-qof/wiki/ 303 Additional-Fields. 305 4. Acknowledgements 307 Thanks to the Passive DNS developers who contributed to the document. 309 5. IANA Considerations 311 This memo includes no request to IANA. 313 6. Privacy Considerations 315 Passive DNS Servers capture DNS answers from multiple collecting 316 points ("sensors") which are located on the Internet-facing side of 317 DNS recursors ("post-recursor passive DNS"). In this process, they 318 intentionally omit the source IP, source port, destination IP and 319 destination port from the captured packets. Since the data is 320 captured "post-recursor", the timing information (who queries what) 321 is lost, since the recursor will cache the results. Furthermore, 322 since multiple sensors feed into a passive DNS server, the resulting 323 data gets mixed together, reducing the likelihood that Passive DNS 324 Servers are able to find out much about the actual person querying 325 the DNS records nor who actually sent the query. In this sense, 326 passive DNS Servers are similar to keeping an archive of all previous 327 phone books - if public DNS records can be compared to phone numbers 328 - as they often are. Nevertheless, the authors strongly encourage 329 Passive DNS implementors to take special care of privacy issues. 330 bortzmeyer-dnsop-dns-privacy is an excellent starting point for this. 331 Finally, the overall recommendations in RFC6973 [RFC6973] should be 332 taken into consideration when designing any application which uses 333 Passive DNS data. 335 In the scope of the General Data Protection Regulation (GDPR - 336 Directive 95/46/EC), operators of Passive DNS Server needs to ensure 337 the legal ground and lawfulness of its operation. 339 7. Security Considerations 341 In some cases, Passive DNS output might contain confidential 342 information and its access might be restricted. When a user is 343 querying multiple Passive DNS and aggregating the data, the 344 sensitivity of the data must be considered. 346 8. References 348 8.1. Normative References 350 [RFC1034] Mockapetris, P., "Domain names - concepts and facilities", 351 STD 13, RFC 1034, DOI 10.17487/RFC1034, November 1987, 352 . 354 [RFC1035] Mockapetris, P., "Domain names - implementation and 355 specification", STD 13, RFC 1035, DOI 10.17487/RFC1035, 356 November 1987, . 358 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 359 Requirement Levels", BCP 14, RFC 2119, 360 DOI 10.17487/RFC2119, March 1997, 361 . 363 [RFC2234] Crocker, D., Ed. and P. Overell, "Augmented BNF for Syntax 364 Specifications: ABNF", RFC 2234, DOI 10.17487/RFC2234, 365 November 1997, . 367 [RFC3597] Gustafsson, A., "Handling of Unknown DNS Resource Record 368 (RR) Types", RFC 3597, DOI 10.17487/RFC3597, September 369 2003, . 371 [RFC3912] Daigle, L., "WHOIS Protocol Specification", RFC 3912, 372 DOI 10.17487/RFC3912, September 2004, 373 . 375 [RFC3986] Berners-Lee, T., Fielding, R., and L. Masinter, "Uniform 376 Resource Identifier (URI): Generic Syntax", STD 66, 377 RFC 3986, DOI 10.17487/RFC3986, January 2005, 378 . 380 [RFC4627] Crockford, D., "The application/json Media Type for 381 JavaScript Object Notation (JSON)", RFC 4627, 382 DOI 10.17487/RFC4627, July 2006, 383 . 385 [RFC5001] Austein, R., "DNS Name Server Identifier (NSID) Option", 386 RFC 5001, DOI 10.17487/RFC5001, August 2007, 387 . 389 [RFC6648] Saint-Andre, P., Crocker, D., and M. Nottingham, 390 "Deprecating the "X-" Prefix and Similar Constructs in 391 Application Protocols", BCP 178, RFC 6648, 392 DOI 10.17487/RFC6648, June 2012, 393 . 395 [RFC6973] Cooper, A., Tschofenig, H., Aboba, B., Peterson, J., 396 Morris, J., Hansen, M., and R. Smith, "Privacy 397 Considerations for Internet Protocols", RFC 6973, 398 DOI 10.17487/RFC6973, July 2013, 399 . 401 8.2. References 403 [BAILIWICK] 404 "Passive DNS Hardening", 2010, 405 . 408 [CACHEPOISONING] 409 "Black ops 2008: It's the end of the cache as we know 410 it.", 2008, . 412 [DNSDB] "DNSDB API", 2013, . 414 [PDNSCERTAT] 415 "pDNS presentation at 4th Centr R&D workshop Frankfurt Jun 416 5th 2012", 2012, 417 . 420 [PDNSCIRCL] 421 "CIRCL Passive DNS", 2012, 422 . 424 [PDNSCLIENT] 425 "Queries 5 major Passive DNS databases: BFK, CERTEE, 426 DNSParse, ISC, and VirusTotal.", 2013, 427 . 429 [PDNSCOF] "Passive DNS server interface using the common output 430 format", 2013, 431 . 433 [REST] "Representational State Transfer (REST)", 2000, 434 . 437 [WEIMERPDNS] 438 "Passive DNS Replication", 2005, 439 . 442 8.3. Informative References 444 [I-D.narten-iana-considerations-rfc2434bis] 445 Narten, T. and H. Alvestrand, "Guidelines for Writing an 446 IANA Considerations Section in RFCs", draft-narten-iana- 447 considerations-rfc2434bis-09 (work in progress), March 448 2008. 450 [RFC3552] Rescorla, E. and B. Korver, "Guidelines for Writing RFC 451 Text on Security Considerations", BCP 72, RFC 3552, 452 DOI 10.17487/RFC3552, July 2003, 453 . 455 Appendix A. Examples 457 The JSON output are represented on multiple lines for readability but 458 each JSON object should on a single line. 460 If you query a passive DNS for the rrname www.ietf.org, the passive 461 dns common output format can be: 463 {"count": 102, "time_first": 1298412391, "rrtype": "AAAA", 464 "rrname": "www.ietf.org", "rdata": "2001:1890:1112:1::20", 465 "time_last": 1302506851} 466 {"count": 59, "time_first": 1384865833, "rrtype": "A", 467 "rrname": "www.ietf.org", "rdata": "4.31.198.44", 468 "time_last": 1389022219} 469 If you query a passive DNS for the rrname ietf.org, the passive dns 470 common output format can be: 472 {"count": 109877, "time_first": 1298398002, "rrtype": "NS", 473 "rrname": "ietf.org", "rdata": "ns1.yyz1.afilias-nst.info", 474 "time_last": 1389095375} 475 {"count": 4, "time_first": 1298495035, "rrtype": "A", 476 "rrname": "ietf.org", "rdata": "64.170.98.32", 477 "time_last": 1298495035} 478 {"count": 9, "time_first": 1317037550, "rrtype": "AAAA", 479 "rrname": "ietf.org", "rdata": "2001:1890:123a::1:1e", 480 "time_last": 1330209752} 482 Please note that in the examples above, any backslashes "\" can be 483 ignored and are an artefact of the tools which produced this 484 document. 486 Authors' Addresses 488 Alexandre Dulaunoy 489 CIRCL 490 16, bd d'Avranches 491 Luxembourg L-1160 492 Luxembourg 494 Phone: (+352) 247 88444 495 Email: alexandre.dulaunoy@circl.lu 496 URI: http://www.circl.lu/ 498 L. Aaron Kaplan 499 CERT.at 500 Karlsplatz 1/2/9 501 Vienna A-1010 502 Austria 504 Phone: +43 1 5056416 78 505 Email: kaplan@cert.at 506 URI: http://www.cert.at/ 507 Paul Vixie 508 Farsight Security, Inc. 509 11400 La Honda Road 510 Woodside, California 94062 511 U.S.A. 513 Email: paul@redbarn.org 514 URI: https://www.farsightsecurity.com/ 516 Henry Stern 517 Farsight Security, Inc. 518 11400 La Honda Road 519 Woodside, California 94062 520 U.S.A. 522 Phone: +1 650 542-7836 523 Email: henry@stern.ca 524 URI: https://www.farsightsecurity.com/