idnits 2.17.1 draft-ietf-dnsop-dns-capture-format-08.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** There are 11 instances of too long lines in the document, the longest one being 9 characters in excess of 72. == There are 3 instances of lines with non-RFC6890-compliant IPv4 addresses in the document. If these are example addresses, they should be changed. -- The document has examples using IPv4 documentation addresses according to RFC6890, but does not use any IPv6 documentation addresses. Maybe there should be IPv6 examples, too? Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (August 10, 2018) is 2058 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) -- Looks like a reference, but probably isn't: '2' on line 2204 -- Looks like a reference, but probably isn't: '1' on line 2202 -- Looks like a reference, but probably isn't: '3' on line 2206 -- Looks like a reference, but probably isn't: '4' on line 2208 -- Looks like a reference, but probably isn't: '5' on line 2211 -- Looks like a reference, but probably isn't: '6' on line 2777 -- Looks like a reference, but probably isn't: '7' on line 2783 -- Looks like a reference, but probably isn't: '8' on line 2826 -- Looks like a reference, but probably isn't: '9' on line 2836 -- Looks like a reference, but probably isn't: '10' on line 2849 -- Looks like a reference, but probably isn't: '11' on line 2876 -- Looks like a reference, but probably isn't: '12' on line 2877 -- Looks like a reference, but probably isn't: '13' on line 2879 -- Looks like a reference, but probably isn't: '14' on line 2882 -- Looks like a reference, but probably isn't: '15' on line 2884 -- Looks like a reference, but probably isn't: '16' on line 2886 -- Looks like a reference, but probably isn't: '17' on line 3071 -- Looks like a reference, but probably isn't: '18' on line 3073 ** Obsolete normative reference: RFC 7049 (Obsoleted by RFC 8949) == Outdated reference: A later version (-08) exists of draft-ietf-cbor-cddl-03 Summary: 2 errors (**), 0 flaws (~~), 3 warnings (==), 20 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 dnsop J. Dickinson 3 Internet-Draft J. Hague 4 Intended status: Standards Track S. Dickinson 5 Expires: February 11, 2019 Sinodun IT 6 T. Manderson 7 J. Bond 8 ICANN 9 August 10, 2018 11 C-DNS: A DNS Packet Capture Format 12 draft-ietf-dnsop-dns-capture-format-08 14 Abstract 16 This document describes a data representation for collections of DNS 17 messages. The format is designed for efficient storage and 18 transmission of large packet captures of DNS traffic; it attempts to 19 minimize the size of such packet capture files but retain the full 20 DNS message contents along with the most useful transport metadata. 21 It is intended to assist with the development of DNS traffic 22 monitoring applications. 24 Status of This Memo 26 This Internet-Draft is submitted in full conformance with the 27 provisions of BCP 78 and BCP 79. 29 Internet-Drafts are working documents of the Internet Engineering 30 Task Force (IETF). Note that other groups may also distribute 31 working documents as Internet-Drafts. The list of current Internet- 32 Drafts is at http://datatracker.ietf.org/drafts/current/. 34 Internet-Drafts are draft documents valid for a maximum of six months 35 and may be updated, replaced, or obsoleted by other documents at any 36 time. It is inappropriate to use Internet-Drafts as reference 37 material or to cite them other than as "work in progress." 39 This Internet-Draft will expire on February 11, 2019. 41 Copyright Notice 43 Copyright (c) 2018 IETF Trust and the persons identified as the 44 document authors. All rights reserved. 46 This document is subject to BCP 78 and the IETF Trust's Legal 47 Provisions Relating to IETF Documents 48 (http://trustee.ietf.org/license-info) in effect on the date of 49 publication of this document. Please review these documents 50 carefully, as they describe your rights and restrictions with respect 51 to this document. Code Components extracted from this document must 52 include Simplified BSD License text as described in Section 4.e of 53 the Trust Legal Provisions and are provided without warranty as 54 described in the Simplified BSD License. 56 Table of Contents 58 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 59 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4 60 3. Data collection use cases . . . . . . . . . . . . . . . . . . 5 61 4. Design considerations . . . . . . . . . . . . . . . . . . . . 7 62 5. Choice of CBOR . . . . . . . . . . . . . . . . . . . . . . . 8 63 6. C-DNS format conceptual overview . . . . . . . . . . . . . . 9 64 6.1. Block Parameters . . . . . . . . . . . . . . . . . . . . 13 65 6.2. Storage Parameters . . . . . . . . . . . . . . . . . . . 13 66 6.2.1. Optional data items . . . . . . . . . . . . . . . . . 13 67 6.2.2. Optional RRs and OPCODEs . . . . . . . . . . . . . . 14 68 6.2.3. Storage flags . . . . . . . . . . . . . . . . . . . . 15 69 6.2.4. IP Address storage . . . . . . . . . . . . . . . . . 15 70 7. C-DNS format detailed description . . . . . . . . . . . . . . 15 71 7.1. Map quantities and indexes . . . . . . . . . . . . . . . 15 72 7.2. Tabular representation . . . . . . . . . . . . . . . . . 16 73 7.3. "File" . . . . . . . . . . . . . . . . . . . . . . . . . 17 74 7.4. "FilePreamble" . . . . . . . . . . . . . . . . . . . . . 17 75 7.4.1. "BlockParameters" . . . . . . . . . . . . . . . . . . 18 76 7.4.2. "CollectionParameters" . . . . . . . . . . . . . . . 21 77 7.5. "Block" . . . . . . . . . . . . . . . . . . . . . . . . . 22 78 7.5.1. "BlockPreamble" . . . . . . . . . . . . . . . . . . . 23 79 7.5.2. "BlockStatistics" . . . . . . . . . . . . . . . . . . 24 80 7.5.3. "BlockTables" . . . . . . . . . . . . . . . . . . . . 25 81 7.6. "QueryResponse" . . . . . . . . . . . . . . . . . . . . . 31 82 7.6.1. "ResponseProcessingData" . . . . . . . . . . . . . . 33 83 7.6.2. "QueryResponseExtended" . . . . . . . . . . . . . . . 33 84 7.7. "AddressEventCount" . . . . . . . . . . . . . . . . . . . 34 85 7.8. "MalformedMessage" . . . . . . . . . . . . . . . . . . . 35 86 8. Versioning . . . . . . . . . . . . . . . . . . . . . . . . . 36 87 9. C-DNS to PCAP . . . . . . . . . . . . . . . . . . . . . . . . 36 88 9.1. Name compression . . . . . . . . . . . . . . . . . . . . 37 89 10. Data collection . . . . . . . . . . . . . . . . . . . . . . . 38 90 10.1. Matching algorithm . . . . . . . . . . . . . . . . . . . 39 91 10.2. Message identifiers . . . . . . . . . . . . . . . . . . 41 92 10.2.1. Primary ID (required) . . . . . . . . . . . . . . . 41 93 10.2.2. Secondary ID (optional) . . . . . . . . . . . . . . 42 94 10.3. Algorithm parameters . . . . . . . . . . . . . . . . . . 42 95 10.4. Algorithm requirements . . . . . . . . . . . . . . . . . 42 96 10.5. Algorithm limitations . . . . . . . . . . . . . . . . . 42 97 10.6. Workspace . . . . . . . . . . . . . . . . . . . . . . . 43 98 10.7. Output . . . . . . . . . . . . . . . . . . . . . . . . . 43 99 10.8. Post processing . . . . . . . . . . . . . . . . . . . . 43 100 11. Implementation guidance . . . . . . . . . . . . . . . . . . . 43 101 11.1. Optional data . . . . . . . . . . . . . . . . . . . . . 44 102 11.2. Trailing bytes . . . . . . . . . . . . . . . . . . . . . 44 103 11.3. Limiting collection of RDATA . . . . . . . . . . . . . . 44 104 12. Implementation status . . . . . . . . . . . . . . . . . . . . 44 105 12.1. DNS-STATS Compactor . . . . . . . . . . . . . . . . . . 45 106 13. IANA considerations . . . . . . . . . . . . . . . . . . . . . 45 107 14. Security considerations . . . . . . . . . . . . . . . . . . . 46 108 15. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 46 109 16. Changelog . . . . . . . . . . . . . . . . . . . . . . . . . . 46 110 17. References . . . . . . . . . . . . . . . . . . . . . . . . . 49 111 17.1. Normative References . . . . . . . . . . . . . . . . . . 49 112 17.2. Informative References . . . . . . . . . . . . . . . . . 49 113 17.3. URIs . . . . . . . . . . . . . . . . . . . . . . . . . . 50 114 Appendix A. CDDL . . . . . . . . . . . . . . . . . . . . . . . . 51 115 Appendix B. DNS Name compression example . . . . . . . . . . . . 61 116 B.1. NSD compression algorithm . . . . . . . . . . . . . . . . 62 117 B.2. Knot Authoritative compression algorithm . . . . . . . . 62 118 B.3. Observed differences . . . . . . . . . . . . . . . . . . 63 119 Appendix C. Comparison of Binary Formats . . . . . . . . . . . . 63 120 C.1. Comparison with full PCAP files . . . . . . . . . . . . . 66 121 C.2. Simple versus block coding . . . . . . . . . . . . . . . 66 122 C.3. Binary versus text formats . . . . . . . . . . . . . . . 67 123 C.4. Performance . . . . . . . . . . . . . . . . . . . . . . . 67 124 C.5. Conclusions . . . . . . . . . . . . . . . . . . . . . . . 67 125 C.6. Block size choice . . . . . . . . . . . . . . . . . . . . 68 126 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 68 128 1. Introduction 130 There has long been a need to collect DNS queries and responses on 131 authoritative and recursive name servers for monitoring and analysis. 132 This data is used in a number of ways including traffic monitoring, 133 analyzing network attacks and "day in the life" (DITL) [ditl] 134 analysis. 136 A wide variety of tools already exist that facilitate the collection 137 of DNS traffic data, such as DSC [dsc], packetq [packetq], dnscap 138 [dnscap] and dnstap [dnstap]. However, there is no standard exchange 139 format for large DNS packet captures. The PCAP [pcap] or PCAP-NG 140 [pcapng] formats are typically used in practice for packet captures, 141 but these file formats can contain a great deal of additional 142 information that is not directly pertinent to DNS traffic analysis 143 and thus unnecessarily increases the capture file size. 145 There has also been work on using text based formats to describe DNS 146 packets such as [I-D.daley-dnsxml], [I-D.hoffman-dns-in-json], but 147 these are largely aimed at producing convenient representations of 148 single messages. 150 Many DNS operators may receive hundreds of thousands of queries per 151 second on a single name server instance so a mechanism to minimize 152 the storage size (and therefore upload overhead) of the data 153 collected is highly desirable. 155 The format described in this document, C-DNS (Compacted-DNS), 156 focusses on the problem of capturing and storing large packet capture 157 files of DNS traffic with the following goals in mind: 159 o Minimize the file size for storage and transmission. 161 o Minimize the overhead of producing the packet capture file and the 162 cost of any further (general purpose) compression of the file. 164 This document contains: 166 o A discussion of some common use cases in which DNS data is 167 collected, see Section 3. 169 o A discussion of the major design considerations in developing an 170 efficient data representation for collections of DNS messages, see 171 Section 4. 173 o A description of why CBOR [RFC7049] was chosen for this format, 174 see Section 5. 176 o A conceptual overview of the C-DNS format, see Section 6. 178 o The definition of the C-DNS format for the collection of DNS 179 messages, see Section 7. 181 o Notes on converting C-DNS data to PCAP format, see Section 9. 183 o Some high level implementation considerations for applications 184 designed to produce C-DNS, see Section 10. 186 2. Terminology 188 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 189 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 190 document are to be interpreted as described in [RFC2119]. 192 "Packet" refers to an individual IPv4 or IPv6 packet. Typically 193 packets are UDP datagrams, but may also be part of a TCP data stream. 194 "Message", unless otherwise qualified, refers to a DNS payload 195 extracted from a UDP datagram or a TCP data stream. 197 The parts of DNS messages are named as they are in [RFC1035]. 198 Specifically, the DNS message has five sections: Header, Question, 199 Answer, Authority, and Additional. 201 Pairs of DNS messages are called a Query and a Response. 203 3. Data collection use cases 205 In an ideal world, it would be optimal to collect full packet 206 captures of all packets going in or out of a name server. However, 207 there are several design choices or other limitations that are common 208 to many DNS installations and operators. 210 o DNS servers are hosted in a variety of situations: 212 * Self-hosted servers 214 * Third party hosting (including multiple third parties) 216 * Third party hardware (including multiple third parties) 218 o Data is collected under different conditions: 220 * On well-provisioned servers running in a steady state 222 * On heavily loaded servers 224 * On virtualized servers 226 * On servers that are under DoS attack 228 * On servers that are unwitting intermediaries in DoS attacks 230 o Traffic can be collected via a variety of mechanisms: 232 * Within the name server implementation itself 234 * On the same hardware as the name server itself 236 * Using a network tap on an adjacent host to listen to DNS 237 traffic 239 * Using port mirroring to listen from another host 241 o The capabilities of data collection (and upload) networks vary: 243 * Out-of-band networks with the same capacity as the in-band 244 network 246 * Out-of-band networks with less capacity than the in-band 247 network 249 * Everything being on the in-band network 251 Thus, there is a wide range of use cases from very limited data 252 collection environments (third party hardware, servers that are under 253 attack, packet capture on the name server itself and no out-of-band 254 network) to "limitless" environments (self hosted, well provisioned 255 servers, using a network tap or port mirroring with an out-of-band 256 networks with the same capacity as the in-band network). In the 257 former, it is infeasible to reliably collect full packet captures, 258 especially if the server is under attack. In the latter case, 259 collection of full packet captures may be reasonable. 261 As a result of these restrictions, the C-DNS data format is designed 262 with the most limited use case in mind such that: 264 o data collection will occur on the same hardware as the name server 265 itself 267 o collected data will be stored on the same hardware as the name 268 server itself, at least temporarily 270 o collected data being returned to some central analysis system will 271 use the same network interface as the DNS queries and responses 273 o there can be multiple third party servers involved 275 Because of these considerations, a major factor in the design of the 276 format is minimal storage size of the capture files. 278 Another significant consideration for any application that records 279 DNS traffic is that the running of the name server software and the 280 transmission of DNS queries and responses are the most important jobs 281 of a name server; capturing data is not. Any data collection system 282 co-located with the name server needs to be intelligent enough to 283 carefully manage its CPU, disk, memory and network utilization. This 284 leads to designing a format that requires a relatively low overhead 285 to produce and minimizes the requirement for further potentially 286 costly compression. 288 However, it is also essential that interoperability with less 289 restricted infrastructure is maintained. In particular, it is highly 290 desirable that the collection format should facilitate the re- 291 creation of common formats (such as PCAP) that are as close to the 292 original as is realistic given the restrictions above. 294 4. Design considerations 296 This section presents some of the major design considerations used in 297 the development of the C-DNS format. 299 1. The basic unit of data is a combined DNS Query and the associated 300 Response (a "Q/R data item"). The same structure will be used 301 for unmatched Queries and Responses. Queries without Responses 302 will be captured omitting the response data. Responses without 303 queries will be captured omitting the Query data (but using the 304 Question section from the response, if present, as an identifying 305 QNAME). 307 * Rationale: A Query and Response represents the basic level of 308 a client's interaction with the server. Also, combining the 309 Query and Response into one item often reduces storage 310 requirements due to commonality in the data of the two 311 messages. 313 In the context of generating a C-DNS file it is assumed that only 314 those DNS payloads which can be parsed to produce a well-formed 315 DNS message are stored in the C-DNS format and that all other 316 messages will be (optionally) recorded as malformed messages. 317 Parsing a well-formed message means as a minimum: 319 * The packet has a well-formed 12 byte DNS Header with a 320 recognised OPCODE. 322 * The section counts are consistent with the section contents. 324 * All of the resource records can be fully parsed. 326 2. All top level fields in each Q/R data item will be optional. 328 * Rationale: Different users will have different requirements 329 for data to be available for analysis. Users with minimal 330 requirements should not have to pay the cost of recording full 331 data, though this will limit the ability to perform certain 332 kinds of data analysis and also to reconstruct packet 333 captures. For example, omitting the resource records from a 334 Response will reduce the C-DNS file size; in principle 335 responses can be synthesized if there is enough context. 337 3. Multiple Q/R data items will be collected into blocks in the 338 format. Common data in a block will be abstracted and referenced 339 from individual Q/R data items by indexing. The maximum number 340 of Q/R data items in a block will be configurable. 342 * Rationale: This blocking and indexing provides a significant 343 reduction in the volume of file data generated. Although this 344 introduces complexity, it provides compression of the data 345 that makes use of knowledge of the DNS message structure. 347 * It is anticipated that the files produced can be subject to 348 further compression using general purpose compression tools. 349 Measurements show that blocking significantly reduces the CPU 350 required to perform such strong compression. See 351 Appendix C.2. 353 * Examples of commonality between DNS messages are that in most 354 cases the QUESTION RR is the same in the query and response, 355 and that there is a finite set of query signatures (based on a 356 subset of attributes). For many authoritative servers there 357 is very likely to be a finite set of responses that are 358 generated, of which a large number are NXDOMAIN. 360 4. Traffic metadata can optionally be included in each block. 361 Specifically, counts of some types of non-DNS packets (e.g. 362 ICMP, TCP resets) sent to the server may be of interest. 364 5. The wire format content of malformed DNS messages may optionally 365 be recorded. 367 * Rationale: Any structured capture format that does not capture 368 the DNS payload byte for byte will be limited to some extent 369 in that it cannot represent malformed DNS messages. Only 370 those messages that can be fully parsed and transformed into 371 the structured format can be fully represented. Note, 372 however, this can result in rather misleading statistics. For 373 example, a malformed query which cannot be represented in the 374 C-DNS format will lead to the (well formed) DNS responses with 375 error code FORMERR appearing as 'unmatched'. Therefore it can 376 greatly aid downstream analysis to have the wire format of the 377 malformed DNS messages available directly in the C-DNS file. 379 5. Choice of CBOR 381 This document presents a detailed format description using CBOR, the 382 Concise Binary Object Representation defined in [RFC7049]. 384 The choice of CBOR was made taking a number of factors into account. 386 o CBOR is a binary representation, and thus is economical in storage 387 space. 389 o Other binary representations were investigated, and whilst all had 390 attractive features, none had a significant advantage over CBOR. 391 See Appendix C for some discussion of this. 393 o CBOR is an IETF standard and familiar to IETF participants. It is 394 based on the now-common ideas of lists and objects, and thus 395 requires very little familiarization for those in the wider 396 industry. 398 o CBOR is a simple format, and can easily be implemented from 399 scratch if necessary. More complex formats require library 400 support which may present problems on unusual platforms. 402 o CBOR can also be easily converted to text formats such as JSON 403 ([RFC8259]) for debugging and other human inspection requirements. 405 o CBOR data schemas can be described using CDDL 406 [I-D.ietf-cbor-cddl]. 408 6. C-DNS format conceptual overview 410 The following figures show purely schematic representations of the 411 C-DNS format to convey the high-level structure of the C-DNS format. 412 Section 7 provides a detailed discussion of the CBOR representation 413 and individual elements. 415 Figure 1 shows the C-DNS format at the top level including the file 416 header and data blocks. The Query/Response data items, Address/Event 417 Count data items and Malformed Message data items link to various 418 Block tables. 420 +-------+ 421 + C-DNS | 422 +-------+--------------------------+ 423 | File type identifier | 424 +----------------------------------+ 425 | File preamble | 426 | +--------------------------------+ 427 | | Format version info | 428 | +--------------------------------+ 429 | | Block parameters | 430 +-+--------------------------------+ 431 | Block | 432 | +--------------------------------+ 433 | | Block preamble | 434 | +--------------------------------+ 435 | | Block statistics | 436 | +--------------------------------+ 437 | | Block tables | 438 | +--------------------------------+ 439 | | Query/Response data items | 440 | +--------------------------------+ 441 | | Address/Event Count data items | 442 | +--------------------------------+ 443 | | Malformed Message data items | 444 +-+--------------------------------+ 445 | Block | 446 | +--------------------------------+ 447 | | Block preamble | 448 | +--------------------------------+ 449 | | Block statistics | 450 | +--------------------------------+ 451 | | Block tables | 452 | +--------------------------------+ 453 | | Query/Response data items | 454 | +--------------------------------+ 455 | | Address/Event Count data items | 456 | +--------------------------------+ 457 | | Malformed Message data items | 458 +-+--------------------------------+ 459 | Further Blocks... | 460 +----------------------------------+ 462 Figure 1: The C-DNS format. 464 Figure 2 shows some more detailed relationships within each block, 465 specifically those between the Query/Response data item and the 466 relevant Block tables. 468 +----------------+ 469 | Query/Response | 470 +-------------------------+ 471 | Time offset | 472 +-------------------------+ +------------------+ 473 | Client address |------------>| IP address array | 474 +-------------------------+ +------------------+ 475 | Client port | 476 +-------------------------+ +------------------+ 477 | Transaction ID | +------>| Name/RDATA array |<------+ 478 +-------------------------+ | +------------------+ | 479 | Query signature |--+ | | 480 +-------------------------+ | | +-----------------+ | 481 | Client hoplimit (q) | +--)------>| Query Signature | | 482 +-------------------------+ | +-----------------+------+ | 483 | Response delay (r) | | | Server address | | 484 +-------------------------+ | +------------------------+ | 485 | Query name (q) |--+--+ | Server port | | 486 +-------------------------+ | +------------------------+ | 487 | Query size (q) | | | Transport flags | | 488 +-------------------------+ | +------------------------+ | 489 | Response size (r) | | | QR type | | 490 +-------------------------+ | +------------------------+ | 491 | Response processing (r) | | | QR signature flags | | 492 | +-----------------------+ | +------------------------+ | 493 | | Bailiwick index |--+ | Query OPCODE (q) | | 494 | +-----------------------+ +------------------------+ | 495 | | Flags | | QR DNS flags | | 496 +-+-----------------------+ +------------------------+ | 497 | Extra query info (q) | | Query RCODE (q) | | 498 | +-----------------------+ +------------------------+ | 499 | | Question |--+---+ +--+-Query Class/Type (q) | | 500 | +-----------------------+ | | +------------------------+ | 501 | | Answer |--+ | | | Query QD count (q) | | 502 | +-----------------------+ | | | +------------------------+ | 503 | | Authority |--+ | | | Query AN count (q) | | 504 | +-----------------------+ | | | +------------------------+ | 505 | | Additional |--+ | | | Query NS count (q) | | 506 +-+-----------------------+ | | | +------------------------+ | 507 | Extra response info (r) | |-+ | | | Query EDNS version (q) | | 508 | +-----------------------+ | | | | +------------------------+ | 509 | | Answer |--+ | | | | EDNS UDP size (q) | | 510 | +-----------------------+ | | | | +------------------------+ | 511 | | Authority |--+ | | | | Query Opt RDATA (q) | | 512 | +-----------------------+ | | | | +------------------------+ | 513 | | Additional |--+ | | | | Response RCODE (r) | | 514 +-+-----------------------+ | | | +------------------------+ | 515 | | | | 516 | | | | 517 + -----------------------------+ | +----------+ | 518 | | | | 519 | + -----------------------------+ | | 520 | | +---------------+ +----------+ | | 521 | +->| Question list |->| Question | | | 522 | | array | | array | | | 523 | +---------------+ +----------+--+ | | 524 | | Name |--+------)------------------+ 525 | +-------------+ | | +------------+ 526 | | Class/type |--)---+--+->| Class/Type | 527 | +-------------+ | | | array | 528 | | | +------------+--+ 529 | | | | Class | 530 | +---------------+ +----------+ | | +---------------+ 531 +--->| RR list array |->| RR array | | | | Type | 532 +---------+-----+ +----------+--+ | | +---------------+ 533 | Name |--+ | 534 +-------------+ | 535 | Class/type |------+ 536 +-------------+ 538 Figure 2: The Query/Response data item and subsidiary tables. 540 In Figure 2 data items annotated (q) are only present when a query/ 541 response has a query, and those annotated (r) are only present when a 542 query/response response is present. 544 A C-DNS file begins with a file header containing a File Type 545 Identifier and a File Preamble. The File Preamble contains 546 information on the file Format Version and an array of Block 547 Parameters items (the contents of which include Collection and 548 Storage Parameters used for one or more blocks). 550 The file header is followed by a series of data Blocks. 552 A Block consists of a Block Preamble item, some Block Statistics for 553 the traffic stored within the Block and then various arrays of common 554 data collectively called the Block Tables. This is then followed by 555 an array of the Query/Response data items detailing the queries and 556 responses stored within the Block. The array of Query/Response data 557 items is in turn followed by the Address/Event Counts data items (an 558 array of per-client counts of particular IP events) and then 559 Malformed Message data items (an array of malformed messages that 560 stored in the Block). 562 The exact nature of the DNS data will affect what block size is the 563 best fit, however sample data for a root server indicated that block 564 sizes up to 10,000 Q/R data items give good results. See 565 Appendix C.6 for more details. 567 6.1. Block Parameters 569 The details of the Block Parameters items are not shown in the 570 diagrams but are discussed here for context. 572 An array of Block Parameters items is stored in the File Preamble 573 (with a minimum of one item at index 0); a Block Parameters item 574 consists of a collection of Storage and Collection Parameters that 575 applies to any given Block. An array is used in order to support use 576 cases such as wanting to merge C-DNS files from different sources. 577 The Block Preamble item then contains an optional index for the Block 578 Parameters item that applies for that Block; if not present the index 579 defaults to 0. Hence, in effect, a global Block Parameters item is 580 defined which can then be overridden per Block. 582 6.2. Storage Parameters 584 The Block Parameters item includes a Storage Parameters item - this 585 contains information about the specific data fields stored in the 586 C-DNS file. 588 These parameters include: 590 o The sub-second timing resolution used by the data. 592 o Information (hints) on which optional data are omitted. See 593 Section 6.2.1. 595 o Recorded OPCODES and RR types. See Section 6.2.2. 597 o Flags indicating, for example, whether the data is sampled or 598 anonymised. See Section 6.2.3. 600 o Client and server IPv4 and IPv6 address prefixes. See 601 Section 6.2.4 603 6.2.1. Optional data items 605 To enable implementations to store data to their precise requirements 606 in as space-efficient manner as possible, all fields in the following 607 arrays are optional: 609 o Query/Response 611 o Query Signature 612 o Malformed messages 614 In other words, an implementation can choose to omit any data item 615 that is not required for its use case. In addition, implementations 616 may be configured to not record all RRs, or only record messages with 617 certain OPCODES. 619 This does, however, mean that a consumer of a C-DNS file faces two 620 problems: 622 1. How can it quickly determine if a file definitely does not 623 contain the data items it requires to complete a particular task 624 (e.g. reconstructing query traffic or performing a specific piece 625 of data analysis)? 627 2. How can it determine if a data item is not present because it 628 was: 630 * explicitly not recorded or 632 * the data item was not available/present. 634 For example, capturing C-DNS data from within a nameserver 635 implementation makes it unlikely that the Client Hoplimit can be 636 recorded. Or, if there is no query ARCount recorded and no query OPT 637 RDATA recorded, is that because no query contained an OPT RR, or 638 because that data was not stored? 640 The Storage Parameters therefore also contains a Storage Hints item 641 which specifies which items the encoder of the file omits from the 642 stored data. An implementation decoding that file can then use these 643 to quickly determine whether the input data is rich enough for its 644 needs. 646 6.2.2. Optional RRs and OPCODEs 648 Also included in the Storage Parameters are explicit arrays listing 649 the RR types and the OPCODEs to be recorded. These remove any 650 ambiguity over whether messages containing particular OPCODEs or RR 651 types are not present because they did not occur, or because the 652 implementation is not configured to record them. 654 In the case of OPCODEs, for a message to be fully parsable, the 655 OPCODE must be known to the collecting implementation. Any message 656 with an OPCODE unknown to the collecting implementation cannot be 657 validated as correctly formed, and so must be treated as malformed. 658 Messages with OPCODES known to the recording application but not 659 listed in the Storage Parameters are discarded (regardless of whether 660 they are malformed or not). 662 In the case of RR records, each record in a message must be fully 663 parsable, including parsing the record RDATA, as otherwise the 664 message cannot be validated as correctly formed. Any RR record with 665 an RR type not known to the collecting implementation cannot be 666 validated as correctly formed, and so must be treated as malformed. 668 Once a message is correctly parsed, an implementation is free to 669 record only a subset of the RR records present. 671 6.2.3. Storage flags 673 The Storage Parameters contains flags that can be used to indicate 674 if: 676 o the data is anonymised, 678 o the data is produced from sample data, or 680 o names in the data have been normalised (converted to uniform 681 case). 683 The Storage Parameters also contains optional fields holding details 684 of the sampling method used and the anonymisation method used. It is 685 RECOMMENDED these fields contain URIs pointing to resources 686 describing the methods used. 688 6.2.4. IP Address storage 690 The format contains fields to indicate if only IP prefixes were 691 stored. If IP address prefixes are given, only the prefix bits of 692 addresses are stored. For example, if a client IPv4 prefix of 16 is 693 specified, a client address of 192.0.2.1 will be stored as 0xc000 694 (192.0), reducing address storage space requirements. 696 7. C-DNS format detailed description 698 The CDDL definition for the C-DNS format is given in Appendix A. 700 7.1. Map quantities and indexes 702 All map keys are integers with values specified in the CDDL. String 703 keys would significantly bloat the file size. 705 All key values specified are positive integers under 24, so their 706 CBOR representation is a single byte. Positive integer values not 707 currently used as keys in a map are reserved for use in future 708 standard extensions. 710 Implementations may choose to add additional implementation-specific 711 entries to any map. Negative integer map keys are reserved for these 712 values. Key values from -1 to -24 also have a single byte CBOR 713 representation, so such implementation-specific extensions are not at 714 any space efficiency disadvantage. 716 An item described as an index is the index of the data item in the 717 referenced array. Indexes are 0-based. 719 7.2. Tabular representation 721 The following sections present the C-DNS specification in tabular 722 format with a detailed description of each item. 724 In all quantities that contain bit flags, bit 0 indicates the least 725 significant bit, i.e. flag "n" in quantity "q" is on if "(q & (1 << 726 n)) != 0". 728 For the sake of readability, all type and field names defined in the 729 CDDL definition are shown in double quotes. Type names are by 730 convention camel case (e.g. "BlockTable"), field names are lower- 731 case with hyphens (e.g. "block-tables"). 733 For the sake of brevity, the following conventions are used in the 734 tables: 736 o The column O marks whether items in a map are optional. 738 * O - Optional. The item may be omitted. 740 * M - Mandatory. The item must be present. 742 o The column T gives the CBOR data type of the item. 744 * U - Unsigned integer 746 * I - Signed integer 748 * B - Byte string 750 * T - Text string 752 * M - Map 754 * A - Array 756 In the case of maps and arrays, more information on the type of each 757 value, include the CDDL definition name if applicable, is given in 758 the description. 760 7.3. "File" 762 A C-DNS file has an outer structure "File", a map that contains the 763 following: 765 +---------------+---+---+-------------------------------------------+ 766 | Field | O | T | Description | 767 +---------------+---+---+-------------------------------------------+ 768 | file-type-id | M | T | String "C-DNS" identifying the file type. | 769 | | | | | 770 | file-preamble | M | M | Version and parameter information for the | 771 | | | | whole file. Map of type "FilePreamble", | 772 | | | | see Section 7.4. | 773 | | | | | 774 | file-blocks | M | A | Array of items of type "Block", see | 775 | | | | Section 7.5. The array may be empty if | 776 | | | | the file contains no data. | 777 +---------------+---+---+-------------------------------------------+ 779 7.4. "FilePreamble" 781 Information about data in the file. A map containing the following: 783 +----------------------+---+---+------------------------------------+ 784 | Field | O | T | Description | 785 +----------------------+---+---+------------------------------------+ 786 | major-format-version | M | U | Unsigned integer '1'. The major | 787 | | | | version of format used in file. | 788 | | | | | 789 | minor-format-version | M | U | Unsigned integer '0'. The minor | 790 | | | | version of format used in file. | 791 | | | | | 792 | private-version | O | U | Version indicator available for | 793 | | | | private use by implementations. | 794 | | | | | 795 | block-parameters | M | A | Array of items of type | 796 | | | | "BlockParameters", see Section | 797 | | | | 7.4.1. The array must contain at | 798 | | | | least one entry. (The "block- | 799 | | | | parameters-index" item in each | 800 | | | | "BlockPreamble" indicates which | 801 | | | | array entry applies to that | 802 | | | | "Block".) | 803 +----------------------+---+---+------------------------------------+ 805 7.4.1. "BlockParameters" 807 Parameters relating to data storage and collection which apply to one 808 or more items of type "Block". A map containing the following: 810 +-----------------------+---+---+-----------------------------------+ 811 | Field | O | T | Description | 812 +-----------------------+---+---+-----------------------------------+ 813 | storage-parameters | M | M | Parameters relating to data | 814 | | | | storage in a "Block" item. Map | 815 | | | | of type "StorageParameters", see | 816 | | | | Section 7.4.1.1. | 817 | | | | | 818 | collection-parameters | O | M | Parameters relating to collection | 819 | | | | of the data in a "Block" item. | 820 | | | | Map of type | 821 | | | | "CollectionParameters", see | 822 | | | | Section 7.4.2. | 823 +-----------------------+---+---+-----------------------------------+ 825 7.4.1.1. "StorageParameters" 827 Parameters relating to how data is stored in the items of type 828 "Block". A map containing the following: 830 +------------------+---+---+----------------------------------------+ 831 | Field | O | T | Description | 832 +------------------+---+---+----------------------------------------+ 833 | ticks-per-second | M | U | Sub-second timing is recorded in | 834 | | | | ticks. This specifies the number of | 835 | | | | ticks in a second. | 836 | | | | | 837 | max-block-items | M | U | The maximum number of items stored in | 838 | | | | any of the arrays in a "Block" item | 839 | | | | (Q/R items, address event counts or | 840 | | | | malformed messages). An indication to | 841 | | | | a decoder of the resources needed to | 842 | | | | process the file. | 843 | | | | | 844 | storage-hints | M | M | Collection of hints as to which fields | 845 | | | | are omitted in the arrays that have | 846 | | | | optional fields. Map of type | 847 | | | | "StorageHints", see Section 7.4.1.1.1. | 848 | | | | | 849 | opcodes | M | A | Array of OPCODES (unsigned integers) | 850 | | | | recorded by the collection | 851 | | | | implementation. See Section 6.2.2. | 852 | | | | | 853 | rr-types | M | A | Array of RR types (unsigned integers) | 854 | | | | recorded by the collection | 855 | | | | implementation. See Section 6.2.2. | 856 | | | | | 857 | storage-flags | O | U | Bit flags indicating attributes of | 858 | | | | stored data. | 859 | | | | Bit 0. 1 if the data has been | 860 | | | | anonymised. | 861 | | | | Bit 1. 1 if the data is sampled data. | 862 | | | | Bit 2. 1 if the names have been | 863 | | | | normalised (converted to uniform | 864 | | | | case). | 865 | | | | | 866 | client-address | O | U | IPv4 client address prefix length. If | 867 | -prefix-ipv4 | | | specified, only the address prefix | 868 | | | | bits are stored. | 869 | | | | | 870 | client-address | O | U | IPv6 client address prefix length. If | 871 | -prefix-ipv6 | | | specified, only the address prefix | 872 | | | | bits are stored. | 873 | | | | | 874 | server-address | O | U | IPv4 server address prefix length. If | 875 | -prefix-ipv4 | | | specified, only the address prefix | 876 | | | | bits are stored. | 877 | | | | | 878 | server-address | O | U | IPv6 server address prefix length. If | 879 | -prefix-ipv6 | | | specified, only the address prefix | 880 | | | | bits are stored. | 881 | | | | | 882 | sampling-method | O | T | Information on the sampling method | 883 | | | | used. See Section 6.2.3. | 884 | | | | | 885 | anonymisation | O | T | Information on the anonymisation | 886 | -method | | | method used. See Section 6.2.3. | 887 +------------------+---+---+----------------------------------------+ 889 7.4.1.1.1. "StorageHints" 891 An indicator of which fields the collecting implementation omits in 892 the arrays with optional fields. A map containing the following: 894 +------------------+---+---+----------------------------------------+ 895 | Field | O | T | Description | 896 +------------------+---+---+----------------------------------------+ 897 | query-response | M | U | Hints indicating which "QueryResponse" | 898 | -hints | | | fields are omitted, see section | 899 | | | | Section 7.6. If the field is omitted | 900 | | | | the bit is unset. | 901 | | | | Bit 0. time-offset | 902 | | | | Bit 1. client-address-index | 903 | | | | Bit 2. client-port | 904 | | | | Bit 3. transaction-id | 905 | | | | Bit 4. qr-signature-index | 906 | | | | Bit 5. client-hoplimit | 907 | | | | Bit 6. response-delay | 908 | | | | Bit 7. query-name-index | 909 | | | | Bit 8. query-size | 910 | | | | Bit 9. response-size | 911 | | | | Bit 10. response-processing-data | 912 | | | | Bit 11. query-question-sections | 913 | | | | Bit 12. query-answer-sections | 914 | | | | Bit 13. query-authority-sections | 915 | | | | Bit 14. query-additional-sections | 916 | | | | Bit 15. response-answer-sections | 917 | | | | Bit 16. response-authority-sections | 918 | | | | Bit 17. response-additional-sections | 919 | | | | | 920 | query-response | M | U | Hints indicating which | 921 | -signature-hints | | | "QueryResponseSignature" fields are | 922 | | | | omitted, see section Section 7.5.3.2. | 923 | | | | If the field is omitted the bit is | 924 | | | | unset. | 925 | | | | Bit 0. server-address | 926 | | | | Bit 1. server-port | 927 | | | | Bit 2. qr-transport-flags | 928 | | | | Bit 3. qr-type | 929 | | | | Bit 4. qr-sig-flags | 930 | | | | Bit 5. query-opcode | 931 | | | | Bit 6. dns-flags | 932 | | | | Bit 7. query-rcode | 933 | | | | Bit 8. query-class-type | 934 | | | | Bit 9. query-qdcount | 935 | | | | Bit 10. query-ancount | 936 | | | | Bit 11. query-nscount | 937 | | | | Bit 12. query-arcount | 938 | | | | Bit 13. query-edns-version | 939 | | | | Bit 14. query-udp-size | 940 | | | | Bit 15. query-opt-rdata | 941 | | | | Bit 16. response-rcode | 942 | | | | | 943 | rr-hints | M | U | Hints indicating which optional "RR" | 944 | | | | fields are omitted, see Section | 945 | | | | 7.5.3.4. If the field is omitted the | 946 | | | | bit is unset. | 947 | | | | Bit 0. ttl | 948 | | | | Bit 1. rdata-index | 949 | other-data-hints | M | U | Hints indicating which other data | 950 | | | | types are are omitted. If the data | 951 | | | | type is are omitted the bit is unset. | 952 | | | | Bit 0. malformed-messages | 953 | | | | Bit 1. address-event-counts | 954 +------------------+---+---+----------------------------------------+ 956 7.4.2. "CollectionParameters" 958 Parameters relating to how data in the file was collected. 960 These parameters have no default. If they do not appear, nothing can 961 be inferred about their value. 963 A map containing the following items: 965 +------------------+---+---+----------------------------------------+ 966 | Field | O | T | Description | 967 +------------------+---+---+----------------------------------------+ 968 | query-timeout | O | U | To be matched with a query, a response | 969 | | | | must arrive within this number of | 970 | | | | seconds. | 971 | | | | | 972 | skew-timeout | O | U | The network stack may report a | 973 | | | | response before the corresponding | 974 | | | | query. A response is not considered to | 975 | | | | be missing a query until after this | 976 | | | | many micro-seconds. | 977 | | | | | 978 | snaplen | O | U | Collect up to this many bytes per | 979 | | | | packet. | 980 | | | | | 981 | promisc | O | U | 1 if promiscuous mode was enabled on | 982 | | | | the interface, 0 otherwise. | 983 | | | | | 984 | interfaces | O | A | Array of identifiers (of type text | 985 | | | | string) of the interfaces used for | 986 | | | | collection. | 987 | | | | | 988 | server-addresses | O | A | Array of server collection IP | 989 | | | | addresses (of type byte string). Hint | 990 | | | | for downstream analysers; does not | 991 | | | | affect collection. | 992 | | | | | 993 | vlan-ids | O | A | Array of identifiers (of type unsigned | 994 | | | | integer) of VLANs selected for | 995 | | | | collection. | 996 | | | | | 997 | filter | O | T | "tcpdump" [pcap] style filter for | 998 | | | | input. | 999 | | | | | 1000 | generator-id | O | T | String identifying the collection | 1001 | | | | method. | 1002 | | | | | 1003 | host-id | O | T | String identifying the collecting | 1004 | | | | host. Empty if converting an existing | 1005 | | | | packet capture file. | 1006 +------------------+---+---+----------------------------------------+ 1008 7.5. "Block" 1010 Container for data with common collection and and storage parameters. 1011 A map containing the following: 1013 +--------------------+---+---+--------------------------------------+ 1014 | Field | O | T | Description | 1015 +--------------------+---+---+--------------------------------------+ 1016 | block-preamble | M | M | Overall information for the "Block" | 1017 | | | | item. Map of type "BlockPreamble", | 1018 | | | | see Section 7.5.1. | 1019 | | | | | 1020 | block-statistics | O | M | Statistics about the "Block" item. | 1021 | | | | Map of type "BlockStatistics", see | 1022 | | | | Section 7.5.2. | 1023 | | | | | 1024 | block-tables | O | M | The arrays containing data | 1025 | | | | referenced by individual | 1026 | | | | "QueryResponse" or | 1027 | | | | "MalformedMessage" items. Map of | 1028 | | | | type "BlockTables", see Section | 1029 | | | | 7.5.3. | 1030 | | | | | 1031 | query-responses | O | A | Details of individual DNS Q/R data | 1032 | | | | items. Array of items of type | 1033 | | | | "QueryResponse", see Section 7.6. If | 1034 | | | | present, the array must not be | 1035 | | | | empty. | 1036 | | | | | 1037 | address-event | O | A | Per client counts of ICMP messages | 1038 | -counts | | | and TCP resets. Array of items of | 1039 | | | | type "AddressEventCount", see | 1040 | | | | Section 7.7. If present, the array | 1041 | | | | must not be empty. | 1042 | | | | | 1043 | malformed-messages | O | A | Details of malformed DNS messages. | 1044 | | | | Array of items of type | 1045 | | | | "MalformedMessage", see Section 7.8. | 1046 | | | | If present, the array must not be | 1047 | | | | empty. | 1048 +--------------------+---+---+--------------------------------------+ 1050 7.5.1. "BlockPreamble" 1052 Overall information for a "Block" item. A map containing the 1053 following: 1055 +------------------+---+---+----------------------------------------+ 1056 | Field | O | T | Description | 1057 +------------------+---+---+----------------------------------------+ 1058 | earliest-time | O | A | A timestamp (2 unsigned integers, | 1059 | | | | "Timestamp") for the earliest record | 1060 | | | | in the "Block" item. The first integer | 1061 | | | | is the number of seconds since the | 1062 | | | | Posix epoch ("time_t"). The second | 1063 | | | | integer is the number of ticks since | 1064 | | | | the start of the second. This | 1065 | | | | timestamp can only be omitted if all | 1066 | | | | block items containing a time offset | 1067 | | | | from the start of the block also omit | 1068 | | | | that time offset. | 1069 | | | | | 1070 | block-parameters | O | U | The index of the item in the "block- | 1071 | -index | | | parameters" array (in the "file- | 1072 | | | | premable" item) applicable to this | 1073 | | | | block. If not present, index 0 is | 1074 | | | | used. See Section 7.4.1. | 1075 +------------------+---+---+----------------------------------------+ 1077 7.5.2. "BlockStatistics" 1079 Basic statistical information about a "Block" item. A map containing 1080 the following: 1082 +---------------------+---+---+-------------------------------------+ 1083 | Field | O | T | Description | 1084 +---------------------+---+---+-------------------------------------+ 1085 | processed-messages | O | U | Total number of DNS messages | 1086 | | | | processed from the input traffic | 1087 | | | | stream during collection of data in | 1088 | | | | this "Block" item. | 1089 | | | | | 1090 | qr-data-items | O | U | Total number of Q/R data items in | 1091 | | | | this "Block" item. | 1092 | | | | | 1093 | unmatched-queries | O | U | Number of unmatched queries in this | 1094 | | | | "Block" item. | 1095 | | | | | 1096 | unmatched-responses | O | U | Number of unmatched responses in | 1097 | | | | this "Block" item. | 1098 | | | | | 1099 | discarded-opcode | O | U | Number of DNS messages processed | 1100 | | | | from the input traffic stream | 1101 | | | | during collection of data in this | 1102 | | | | "Block" item but not recorded | 1103 | | | | because their OPCODE is not in the | 1104 | | | | list to be collected. | 1105 | | | | | 1106 | malformed-items | O | U | Number of malformed messages found | 1107 | | | | in input for this "Block" item. | 1108 +---------------------+---+---+-------------------------------------+ 1110 7.5.3. "BlockTables" 1112 Arrays containing data referenced by individual "QueryResponse" or 1113 "MalformedMessage" items in this "Block". Each element is an array 1114 which, if present, must not be empty. 1116 An item in the "qlist" array contains indexes to values in the "qrr" 1117 array. Therefore, if "qlist" is present, "qrr" must also be present. 1118 Similarly, if "rrlist" is present, "rr" must also be present. 1120 The map contains the following items: 1122 +-------------------+---+---+---------------------------------------+ 1123 | Field | O | T | Description | 1124 +-------------------+---+---+---------------------------------------+ 1125 | ip-address | O | A | Array of IP addresses, in network | 1126 | | | | byte order (of type byte string). If | 1127 | | | | client or server address prefixes are | 1128 | | | | set, only the address prefix bits are | 1129 | | | | stored. Each string is therefore up | 1130 | | | | to 4 bytes long for an IPv4 address, | 1131 | | | | or up to 16 bytes long for an IPv6 | 1132 | | | | address. See Section 7.4.1.1. | 1133 | | | | | 1134 | classtype | O | A | Array of RR class and type | 1135 | | | | information. Type is "ClassType", see | 1136 | | | | Section 7.5.3.1. | 1137 | | | | | 1138 | name-rdata | O | A | Array where each entry is the | 1139 | | | | contents of a single NAME or RDATA | 1140 | | | | (of type byte string). Note that | 1141 | | | | NAMEs, and labels within RDATA | 1142 | | | | contents, are full domain names or | 1143 | | | | labels; no DNS style name compression | 1144 | | | | is used on the individual | 1145 | | | | names/labels within the format. | 1146 | | | | | 1147 | qr-sig | O | A | Array Q/R data item signatures. Type | 1148 | | | | is "QueryResponseSignature", see | 1149 | | | | Section 7.5.3.2. | 1150 | | | | | 1151 | qlist | O | A | Array of type "QuestionList". A | 1152 | | | | "QuestionList" is an array of | 1153 | | | | unsigned integers, indexes to | 1154 | | | | "Question" items in the "qrr" array. | 1155 | | | | | 1156 | qrr | O | A | Array of type "Question". Each entry | 1157 | | | | is the contents of a single question, | 1158 | | | | where a question is the second or | 1159 | | | | subsequent question in a query. See | 1160 | | | | Section 7.5.3.3. | 1161 | | | | | 1162 | rrlist | O | A | Array of type "RRList". An "RRList" | 1163 | | | | is an array of unsigned integers, | 1164 | | | | indexes to "RR" items in the "rr" | 1165 | | | | array. | 1166 | | | | | 1167 | rr | O | A | Array of type "RR". Each entry is the | 1168 | | | | contents of a single RR. See Section | 1169 | | | | 7.5.3.4. | 1170 | | | | | 1171 | malformed-message | O | A | Array of the contents of malformed | 1172 | -data | | | messages. Array of type | 1173 | | | | "MalformedMessageData", see Section | 1174 | | | | 7.5.3.5. | 1175 +-------------------+---+---+---------------------------------------+ 1177 7.5.3.1. "ClassType" 1179 RR class and type information. A map containing the following: 1181 +-------+---+---+--------------+ 1182 | Field | O | T | Description | 1183 +-------+---+---+--------------+ 1184 | type | M | U | TYPE value. | 1185 | | | | | 1186 | class | M | U | CLASS value. | 1187 +-------+---+---+--------------+ 1189 7.5.3.2. "QueryResponseSignature" 1191 Elements of a Q/R data item that are often common between multiple 1192 individual Q/R data items. A map containing the following: 1194 +--------------------+---+---+--------------------------------------+ 1195 | Field | O | T | Description | 1196 +--------------------+---+---+--------------------------------------+ 1197 | server-address | O | U | The index in the item in the "ip- | 1198 | -index | | | address" array of the server IP | 1199 | | | | address. See Section 7.5.3. | 1200 | | | | | 1201 | server-port | O | U | The server port. | 1202 | | | | | 1203 | qr-transport-flags | O | U | Bit flags describing the transport | 1204 | | | | used to service the query. | 1205 | | | | Bit 0. IP version. 0 if IPv4, 1 if | 1206 | | | | IPv6 | 1207 | | | | Bit 1-4. Transport. 4 bit unsigned | 1208 | | | | value where 0 = UDP, 1 = TCP, 2 = | 1209 | | | | TLS, 3 = DTLS. Values 4-15 are | 1210 | | | | reserved for future use. | 1211 | | | | Bit 5. 1 if trailing bytes in query | 1212 | | | | packet. See Section 11.2. | 1213 | | | | | 1214 | qr-type | O | U | Type of Query/Response transaction. | 1215 | | | | 0 = Stub. A query from a stub | 1216 | | | | resolver. | 1217 | | | | 1 = Client. An incoming query to a | 1218 | | | | recursive resolver. | 1219 | | | | 2 = Resolver. A query sent from a | 1220 | | | | recursive resolver to an authorative | 1221 | | | | resolver. | 1222 | | | | 3 = Authorative. A query to an | 1223 | | | | authorative resolver. | 1224 | | | | 4 = Forwarder. A query sent from a | 1225 | | | | recursive resolver to an upstream | 1226 | | | | recursive resolver. | 1227 | | | | 5 = Tool. A query sent to a server | 1228 | | | | by a server tool. | 1229 | | | | | 1230 | qr-sig-flags | O | U | Bit flags explicitly indicating | 1231 | | | | attributes of the message pair | 1232 | | | | represented by this Q/R data item | 1233 | | | | (not all attributes may be recorded | 1234 | | | | or deducible). | 1235 | | | | Bit 0. 1 if a Query was present. | 1236 | | | | Bit 1. 1 if a Response was present. | 1237 | | | | Bit 2. 1 if a Query was present and | 1238 | | | | it had an OPT Resource Record. | 1239 | | | | Bit 3. 1 if a Response was present | 1240 | | | | and it had an OPT Resource Record. | 1241 | | | | Bit 4. 1 if a Query was present but | 1242 | | | | had no Question. | 1243 | | | | Bit 5. 1 if a Response was present | 1244 | | | | but had no Question (only one query- | 1245 | | | | name-index is stored per Q/R item). | 1246 | | | | | 1247 | query-opcode | O | U | Query OPCODE. | 1248 | | | | | 1249 | qr-dns-flags | O | U | Bit flags with values from the Query | 1250 | | | | and Response DNS flags. Flag values | 1251 | | | | are 0 if the Query or Response is | 1252 | | | | not present. | 1253 | | | | Bit 0. Query Checking Disabled (CD). | 1254 | | | | Bit 1. Query Authenticated Data | 1255 | | | | (AD). | 1256 | | | | Bit 2. Query reserved (Z). | 1257 | | | | Bit 3. Query Recursion Available | 1258 | | | | (RA). | 1259 | | | | Bit 4. Query Recursion Desired (RD). | 1260 | | | | Bit 5. Query TrunCation (TC). | 1261 | | | | Bit 6. Query Authoritative Answer | 1262 | | | | (AA). | 1263 | | | | Bit 7. Query DNSSEC answer OK (DO). | 1264 | | | | Bit 8. Response Checking Disabled | 1265 | | | | (CD). | 1266 | | | | Bit 9. Response Authenticated Data | 1267 | | | | (AD). | 1268 | | | | Bit 10. Response reserved (Z). | 1269 | | | | Bit 11. Response Recursion Available | 1270 | | | | (RA). | 1271 | | | | Bit 12. Response Recursion Desired | 1272 | | | | (RD). | 1273 | | | | Bit 13. Response TrunCation (TC). | 1274 | | | | Bit 14. Response Authoritative | 1275 | | | | Answer (AA). | 1276 | | | | | 1277 | query-rcode | O | U | Query RCODE. If the Query contains | 1278 | | | | OPT, this value incorporates any | 1279 | | | | EXTENDED_RCODE_VALUE. | 1280 | | | | | 1281 | query-classtype | O | U | The index to the item in the the | 1282 | -index | | | "classtype" array of the CLASS and | 1283 | | | | TYPE of the first Question. See | 1284 | | | | Section 7.5.3. | 1285 | | | | | 1286 | query-qd-count | O | U | The QDCOUNT in the Query, or | 1287 | | | | Response if no Query present. | 1288 | | | | | 1289 | query-an-count | O | U | Query ANCOUNT. | 1290 | | | | | 1291 | query-ns-count | O | U | Query NSCOUNT. | 1292 | | | | | 1293 | query-ar-count | O | U | Query ARCOUNT. | 1294 | | | | | 1295 | edns-version | O | U | The Query EDNS version. | 1296 | | | | | 1297 | udp-buf-size | O | U | The Query EDNS sender's UDP payload | 1298 | | | | size. | 1299 | | | | | 1300 | opt-rdata-index | O | U | The index in the "name-rdata" array | 1301 | | | | of the OPT RDATA. See Section 7.5.3. | 1302 | | | | | 1303 | response-rcode | O | U | Response RCODE. If the Response | 1304 | | | | contains OPT, this value | 1305 | | | | incorporates any | 1306 | | | | EXTENDED_RCODE_VALUE. | 1307 +--------------------+---+---+--------------------------------------+ 1309 7.5.3.3. "Question" 1311 Details on individual Questions in a Question section. A map 1312 containing the following: 1314 +-----------------+---+---+-----------------------------------------+ 1315 | Field | O | T | Description | 1316 +-----------------+---+---+-----------------------------------------+ 1317 | name-index | M | U | The index in the "name-rdata" array of | 1318 | | | | the QNAME. See Section 7.5.3. | 1319 | | | | | 1320 | classtype-index | M | U | The index in the "classtype" array of | 1321 | | | | the CLASS and TYPE of the Question. See | 1322 | | | | Section 7.5.3. | 1323 +-----------------+---+---+-----------------------------------------+ 1325 7.5.3.4. "RR" 1327 Details on individual Resource Records in RR sections. A map 1328 containing the following: 1330 +-----------------+---+---+-----------------------------------------+ 1331 | Field | O | T | Description | 1332 +-----------------+---+---+-----------------------------------------+ 1333 | name-index | M | U | The index in the "name-rdata" array of | 1334 | | | | the NAME. See Section 7.5.3. | 1335 | | | | | 1336 | classtype-index | M | U | The index in the "classtype" array of | 1337 | | | | the CLASS and TYPE of the RR. See | 1338 | | | | Section 7.5.3. | 1339 | | | | | 1340 | ttl | O | U | The RR Time to Live. | 1341 | | | | | 1342 | rdata-index | O | U | The index in the "name-rdata" array of | 1343 | | | | the RR RDATA. See Section 7.5.3. | 1344 +-----------------+---+---+-----------------------------------------+ 1346 7.5.3.5. "MalformedMessageData" 1348 Details on malformed message items in this "Block" item. A map 1349 containing the following: 1351 +--------------------+---+---+--------------------------------------+ 1352 | Field | O | T | Description | 1353 +--------------------+---+---+--------------------------------------+ 1354 | server-address | O | U | The index in the "ip-address" array | 1355 | -index | | | of the server IP address. See | 1356 | | | | Section 7.5.3. | 1357 | | | | | 1358 | server-port | O | U | The server port. | 1359 | | | | | 1360 | mm-transport-flags | O | U | Bit flags describing the transport | 1361 | | | | used to service the query. Bit 0 is | 1362 | | | | the least significant bit. | 1363 | | | | Bit 0. IP version. 0 if IPv4, 1 if | 1364 | | | | IPv6 | 1365 | | | | Bit 1-4. Transport. 4 bit unsigned | 1366 | | | | value where 0 = UDP, 1 = TCP, 2 = | 1367 | | | | TLS, 3 = DTLS. Values 4-15 are | 1368 | | | | reserved for future use. | 1369 | | | | | 1370 | mm-payload | O | B | The payload (raw bytes) of the DNS | 1371 | | | | message. | 1372 +--------------------+---+---+--------------------------------------+ 1374 7.6. "QueryResponse" 1376 Details on individual Q/R data items. 1378 Note that there is no requirement that the elements of the "query- 1379 responses" array are presented in strict chronological order. 1381 A map containing the following items: 1383 +----------------------+---+---+------------------------------------+ 1384 | Field | O | T | Description | 1385 +----------------------+---+---+------------------------------------+ 1386 | time-offset | O | U | Q/R timestamp as an offset in | 1387 | | | | ticks from "earliest-time". The | 1388 | | | | timestamp is the timestamp of the | 1389 | | | | Query, or the Response if there is | 1390 | | | | no Query. | 1391 | | | | | 1392 | client-address-index | O | U | The index in the "ip-address" | 1393 | | | | array of the client IP address. | 1394 | | | | See Section 7.5.3. | 1395 | | | | | 1396 | client-port | O | U | The client port. | 1397 | | | | | 1398 | transaction-id | O | U | DNS transaction identifier. | 1399 | | | | | 1400 | qr-signature-index | O | U | The index in the "qr-sig" array of | 1401 | | | | the "QueryResponseSignature" item. | 1402 | | | | See Section 7.5.3. | 1403 | | | | | 1404 | client-hoplimit | O | U | The IPv4 TTL or IPv6 Hoplimit from | 1405 | | | | the Query packet. | 1406 | | | | | 1407 | response-delay | O | I | The time difference between Query | 1408 | | | | and Response, in ticks. Only | 1409 | | | | present if there is a query and a | 1410 | | | | response. The delay can be | 1411 | | | | negative if the network | 1412 | | | | stack/capture library returns | 1413 | | | | packets out of order. | 1414 | | | | | 1415 | query-name-index | O | U | The index in the "name-rdata" | 1416 | | | | array of the item containing the | 1417 | | | | QNAME for the first Question. See | 1418 | | | | Section 7.5.3. | 1419 | | | | | 1420 | query-size | O | U | DNS query message size (see | 1421 | | | | below). | 1422 | | | | | 1423 | response-size | O | U | DNS query message size (see | 1424 | | | | below). | 1425 | | | | | 1426 | response-processing | O | M | Data on response processing. Map | 1427 | -data | | | of type "ResponseProcessingData", | 1428 | | | | see Section 7.6.1. | 1429 | | | | | 1430 | query-extended | O | M | Extended Query data. Map of type | 1431 | | | | "QueryResponseExtended", see | 1432 | | | | Section 7.6.2. | 1433 | | | | | 1434 | response-extended | O | M | Extended Response data. Map of | 1435 | | | | type "QueryResponseExtended", see | 1436 | | | | Section 7.6.2. | 1437 +----------------------+---+---+------------------------------------+ 1439 The "query-size" and "response-size" fields hold the DNS message 1440 size. For UDP this is the size of the UDP payload that contained the 1441 DNS message. For TCP it is the size of the DNS message as specified 1442 in the two-byte message length header. Trailing bytes in UDP queries 1443 are routinely observed in traffic to authoritative servers and this 1444 value allows a calculation of how many trailing bytes were present. 1446 7.6.1. "ResponseProcessingData" 1448 Information on the server processing that produced the response. A 1449 map containing the following: 1451 +------------------+---+---+----------------------------------------+ 1452 | Field | O | T | Description | 1453 +------------------+---+---+----------------------------------------+ 1454 | bailiwick-index | O | U | The index in the "name-rdata" array of | 1455 | | | | the owner name for the response | 1456 | | | | bailiwick. See Section 7.5.3. | 1457 | | | | | 1458 | processing-flags | O | U | Flags relating to response processing. | 1459 | | | | Bit 0. 1 if the response came from | 1460 | | | | cache. | 1461 +------------------+---+---+----------------------------------------+ 1463 7.6.2. "QueryResponseExtended" 1465 Extended data on the Q/R data item. 1467 Each item in the map is present only if collection of the relevant 1468 details is configured. 1470 A map containing the following items: 1472 +------------------+---+---+----------------------------------------+ 1473 | Field | O | T | Description | 1474 +------------------+---+---+----------------------------------------+ 1475 | question-index | O | U | The index in the "qlist" array of the | 1476 | | | | entry listing any second and | 1477 | | | | subsequent Questions in the Question | 1478 | | | | section for the Query or Response. See | 1479 | | | | Section 7.5.3. | 1480 | | | | | 1481 | answer-index | O | U | The index in the "rrlist" array of the | 1482 | | | | entry listing the Answer Resource | 1483 | | | | Record sections for the Query or | 1484 | | | | Response. See Section 7.5.3. | 1485 | | | | | 1486 | authority-index | O | U | The index in the "rrlist" array of the | 1487 | | | | entry listing the Authority Resource | 1488 | | | | Record sections for the Query or | 1489 | | | | Response. See Section 7.5.3. | 1490 | | | | | 1491 | additional-index | O | U | The index in the "rrlist" array of the | 1492 | | | | entry listing the Additional Resource | 1493 | | | | Record sections for the Query or | 1494 | | | | Response. See Section 7.5.3. Note that | 1495 | | | | Query OPT RR data can be optionally | 1496 | | | | stored in the QuerySignature. | 1497 +------------------+---+---+----------------------------------------+ 1499 7.7. "AddressEventCount" 1501 Counts of various IP related events relating to traffic with 1502 individual client addresses. A map containing the following: 1504 +------------------+---+---+----------------------------------------+ 1505 | Field | O | T | Description | 1506 +------------------+---+---+----------------------------------------+ 1507 | ae-type | M | U | The type of event. The following | 1508 | | | | events types are currently defined: | 1509 | | | | 0. TCP reset. | 1510 | | | | 1. ICMP time exceeded. | 1511 | | | | 2. ICMP destination unreachable. | 1512 | | | | 3. ICMPv6 time exceeded. | 1513 | | | | 4. ICMPv6 destination unreachable. | 1514 | | | | 5. ICMPv6 packet too big. | 1515 | | | | | 1516 | ae-code | O | U | A code relating to the event. | 1517 | | | | | 1518 | ae-address-index | M | U | The index in the "ip-address" array of | 1519 | | | | the client address. See Section 7.5.3. | 1520 | | | | | 1521 | ae-count | M | U | The number of occurrences of this | 1522 | | | | event during the block collection | 1523 | | | | period. | 1524 +------------------+---+---+----------------------------------------+ 1526 7.8. "MalformedMessage" 1528 Details of malformed messages. A map containing the following: 1530 +----------------------+---+---+------------------------------------+ 1531 | Field | O | T | Description | 1532 +----------------------+---+---+------------------------------------+ 1533 | time-offset | O | U | Message timestamp as an offset in | 1534 | | | | ticks from "earliest-time". | 1535 | | | | | 1536 | client-address-index | O | U | The index in the "ip-address" | 1537 | | | | array of the client IP address. | 1538 | | | | See Section 7.5.3. | 1539 | | | | | 1540 | client-port | O | U | The client port. | 1541 | | | | | 1542 | message-data-index | O | U | The index in the "malformed- | 1543 | | | | message-data" array of the message | 1544 | | | | data for this message. See Section | 1545 | | | | 7.5.3. | 1546 +----------------------+---+---+------------------------------------+ 1548 8. Versioning 1550 The C-DNS file preamble includes a file format version; a major and 1551 minor version number are required fields. The document defines 1552 version 1.0 of the C-DNS specification. This section describes the 1553 intended use of these version numbers in future specifications. 1555 It is noted that version 1.0 includes many optional fields and 1556 therefore consumers of version 1.0 should be inherently robust to 1557 parsing files with variable data content. 1559 Within a major version, a new minor version MUST be a strict superset 1560 of the previous minor version, with no semantic changes to existing 1561 fields. New keys MAY be added to existing maps, and new maps MAY be 1562 added. A consumer capable of reading a particular major.minor 1563 version MUST also be capable of reading all previous minor versions 1564 of the same major version. It SHOULD also be capable of parsing all 1565 subsequent minor versions ignoring any keys or maps that it does not 1566 recognise. 1568 A new major version indicates changes to the format that are not 1569 backwards compatible with previous major versions. A consumer 1570 capable of only reading a particular major version (greater than 1) 1571 is not required to and has no expectation to be capable of reading a 1572 previous major version. 1574 9. C-DNS to PCAP 1576 It is possible to re-construct PCAP files from the C-DNS format in a 1577 lossy fashion. Some of the issues with reconstructing both the DNS 1578 payload and the full packet stream are outlined here. 1580 The reconstruction depends on whether or not all the optional 1581 sections of both the query and response were captured in the C-DNS 1582 file. Clearly, if they were not all captured, the reconstruction 1583 will be imperfect. 1585 Even if all sections of the response were captured, one cannot 1586 reconstruct the DNS response payload exactly due to the fact that 1587 some DNS names in the message on the wire may have been compressed. 1588 Section 9.1 discusses this is more detail. 1590 Some transport information is not captured in the C-DNS format. For 1591 example, the following aspects of the original packet stream cannot 1592 be re-constructed from the C-DNS format: 1594 o IP fragmentation 1595 o TCP stream information: 1597 * Multiple DNS messages may have been sent in a single TCP 1598 segment 1600 * A DNS payload may have be split across multiple TCP segments 1602 * Multiple DNS messages may have be sent on a single TCP session 1604 o Malformed DNS messages if the wire format is not recorded 1606 o Any Non-DNS messages that were in the original packet stream e.g. 1607 ICMP 1609 Simple assumptions can be made on the reconstruction: fragmented and 1610 DNS-over-TCP messages can be reconstructed into single packets and a 1611 single TCP session can be constructed for each TCP packet. 1613 Additionally, if malformed messages and Non-DNS packets are captured 1614 separately, they can be merged with packet captures reconstructed 1615 from C-DNS to produce a more complete packet stream. 1617 9.1. Name compression 1619 All the names stored in the C-DNS format are full domain names; no 1620 DNS style name compression is used on the individual names within the 1621 format. Therefore when reconstructing a packet, name compression 1622 must be used in order to reproduce the on the wire representation of 1623 the packet. 1625 [RFC1035] name compression works by substituting trailing sections of 1626 a name with a reference back to the occurrence of those sections 1627 earlier in the message. Not all name server software uses the same 1628 algorithm when compressing domain names within the responses. Some 1629 attempt maximum recompression at the expense of runtime resources, 1630 others use heuristics to balance compression and speed and others use 1631 different rules for what is a valid compression target. 1633 This means that responses to the same question from different name 1634 server software which match in terms of DNS payload content (header, 1635 counts, RRs with name compression removed) do not necessarily match 1636 byte-for-byte on the wire. 1638 Therefore, it is not possible to ensure that the DNS response payload 1639 is reconstructed byte-for-byte from C-DNS data. However, it can at 1640 least, in principle, be reconstructed to have the correct payload 1641 length (since the original response length is captured) if there is 1642 enough knowledge of the commonly implemented name compression 1643 algorithms. For example, a simplistic approach would be to try each 1644 algorithm in turn to see if it reproduces the original length, 1645 stopping at the first match. This would not guarantee the correct 1646 algorithm has been used as it is possible to match the length whilst 1647 still not matching the on the wire bytes but, without further 1648 information added to the C-DNS data, this is the best that can be 1649 achieved. 1651 Appendix B presents an example of two different compression 1652 algorithms used by well-known name server software. 1654 10. Data collection 1656 This section describes a non-normative proposed algorithm for the 1657 processing of a captured stream of DNS queries and responses and 1658 production of a stream of query/response items, matching queries/ 1659 responses where possible. 1661 For the purposes of this discussion, it is assumed that the input has 1662 been pre-processed such that: 1664 1. All IP fragmentation reassembly, TCP stream reassembly, and so 1665 on, has already been performed. 1667 2. Each message is associated with transport metadata required to 1668 generate the Primary ID (see Section 10.2.1). 1670 3. Each message has a well-formed DNS header of 12 bytes and (if 1671 present) the first Question in the Question section can be parsed 1672 to generate the Secondary ID (see below). As noted earlier, this 1673 requirement can result in a malformed query being removed in the 1674 pre-processing stage, but the correctly formed response with 1675 RCODE of FORMERR being present. 1677 DNS messages are processed in the order they are delivered to the 1678 implementation. 1680 It should be noted that packet capture libraries do not necessarily 1681 provide packets in strict chronological order. This can, for 1682 example, arise on multi-core platforms where packets arriving at a 1683 network device are processed by different cores. On systems where 1684 this behaviour has been observed, the timestamps associated with each 1685 packet are consistent; queries always have a timestamp prior to the 1686 response timestamp. However, the order in which these packets appear 1687 in the packet capture stream is not necessarily strictly 1688 choronological; a response can appear in the capture stream before 1689 the query that provoked the response. For this discussion, this non- 1690 chronological delivery is termed "skew". 1692 In the presence of skew, a response packets can arrive for matching 1693 before the corresponding query. To avoid generating false instances 1694 of responses without a matching query, and queries without a matching 1695 response, the matching algorithm must take account of the possibility 1696 of skew. 1698 10.1. Matching algorithm 1700 A schematic representation of the algorithm for matching Q/R data 1701 items is shown in Figure 3. It takes individual DNS query or 1702 response messages as input, and outputs matched Q/R items. The 1703 numbers in the figure identify matching operations listed in Table 1. 1704 Specific details of the algorithm, for example queues, timers and 1705 identifiers, are given in the following sections. 1707 .----------------------. 1708 | Process next message |<------------------+ 1709 `----------------------' | 1710 | | 1711 +------------------------------+ | 1712 | Generate message identifiers | | 1713 +------------------------------+ | 1714 | | 1715 Response | Query | 1716 +--------------< >---------------+ | 1717 | | | 1718 +--------------------+ +--------------------+ | 1719 | Find earliest QR | | Create QR item [2] | | 1720 | item in OFIFO [1] | +--------------------+ | 1721 +--------------------+ | | 1722 | +---------------+ | 1723 Match | No match | Append new QR | | 1724 +--------< >------+ | item to OFIFO | | 1725 | | +---------------+ | 1726 +-----------+ +--------+ | | 1727 | Update QR | | Add to | +-------------------+ | 1728 | item [3] | | RFIFO | | Find earliest QR | | 1729 +-----------+ +--------+ | item in RFIFO [1] | | 1730 | | +-------------------+ | 1731 +-----------------+ | | 1732 | | | 1733 | +----------------+ Match | No match | 1734 | | Remove R |-------< >-----+ | 1735 | | from RFIFO [3] | | | 1736 | +----------------+ | | 1737 | | | | 1738 +--------------+-----------------------+ | 1739 | | 1740 +----------------------------------------------+ | 1741 | Update all timed out (QT) OFIFO QR items [4] | | 1742 +----------------------------------------------+ | 1743 | | 1744 +--------------------------------+ | 1745 | Remove all timed out (ST) R | | 1746 | from RFIFO, create QR item [5] | | 1747 +--------------------------------+ | 1748 ____________________|_______________________ | 1749 / / | 1750 / Remove all consecutive done entries from /-------+ 1751 / front of OFIFO for further processing / 1752 /____________________________________________/ 1754 Figure 3: Query/Response matching algorithm 1756 +-----+-------------------------------------------+ 1757 | Ref | Operation | 1758 +-----+-------------------------------------------+ 1759 | [1] | Find earliest QR item in FIFO where: | 1760 | | * QR.done = false | 1761 | | * QR.Q.PrimaryID == R.PrimaryID | 1762 | | and, if both QR.Q and R have SecondaryID: | 1763 | | * QR.Q.SecondaryID == R.SecondaryID | 1764 | | | 1765 | [2] | Set: | 1766 | | QR.Q := Q | 1767 | | QR.R := nil | 1768 | | QR.done := false | 1769 | | | 1770 | [3] | Set: | 1771 | | QR.R := R | 1772 | | QR.done := true | 1773 | | | 1774 | [4] | Set: | 1775 | | QR.done := true | 1776 | | | 1777 | [5] | Set: | 1778 | | QR.Q := nil | 1779 | | QR.R := R | 1780 | | QR.done := true | 1781 +-----+-------------------------------------------+ 1783 Table 1: Operations used in the matching algorithm 1785 10.2. Message identifiers 1787 10.2.1. Primary ID (required) 1789 A Primary ID is constructed for each message. It is composed of the 1790 following data: 1792 1. Source IP Address 1794 2. Destination IP Address 1796 3. Source Port 1798 4. Destination Port 1800 5. Transport 1802 6. DNS Message ID 1804 10.2.2. Secondary ID (optional) 1806 If present, the first Question in the Question section is used as a 1807 secondary ID for each message. Note that there may be well formed 1808 DNS queries that have a QDCOUNT of 0, and some responses may have a 1809 QDCOUNT of 0 (for example, responses with RCODE=FORMERR or NOTIMP). 1810 In this case the secondary ID is not used in matching. 1812 10.3. Algorithm parameters 1814 1. Query timeout, QT. A query arrives with timestamp t1. If no 1815 response matching that query has arrived before other input 1816 arrives timestamped later than (t1 + QT), a query/response item 1817 containing only a query item is recorded. The query timeout 1818 value is typically of the order of 5 seconds. 1820 2. Skew timeout, ST. A response arrives with timestamp t2. If a 1821 response has not been matched by a query before input arrives 1822 timestamped later than (t2 + ST), a query/response item 1823 containing only a response is recorded. The skew timeout value 1824 is typically a few microseconds. 1826 10.4. Algorithm requirements 1828 The algorithm is designed to handle the following input data: 1830 1. Multiple queries with the same Primary ID (but different 1831 Secondary ID) arriving before any responses for these queries are 1832 seen. 1834 2. Multiple queries with the same Primary and Secondary ID arriving 1835 before any responses for these queries are seen. 1837 3. Queries for which no later response can be found within the 1838 specified timeout. 1840 4. Responses for which no previous query can be found within the 1841 specified timeout. 1843 10.5. Algorithm limitations 1845 For cases 1 and 2 listed in the above requirements, it is not 1846 possible to unambiguously match queries with responses. This 1847 algorithm chooses to match to the earliest query with the correct 1848 Primary and Secondary ID. 1850 10.6. Workspace 1852 The algorithm employs two FIFO queues: 1854 o OFIFO, an output FIFO containing Q/R items in chronological order, 1856 o RFIFO, a FIFO holding responses without a matching query in order 1857 of arrival. 1859 10.7. Output 1861 The output is a list of Q/R data items. Both the Query and Response 1862 elements are optional in these items, therefore Q/R data items have 1863 one of three types of content: 1865 1. A matched pair of query and response messages 1867 2. A query message with no response 1869 3. A response message with no query 1871 The timestamp of a list item is that of the query for cases 1 and 2 1872 and that of the response for case 3. 1874 10.8. Post processing 1876 When ending capture, all items in the responses FIFO are timed out 1877 immediately, generating response-only entries to the Q/R data item 1878 FIFO. These and all other remaining entries in the Q/R data item 1879 FIFO should be treated as timed out queries. 1881 11. Implementation guidance 1883 Whilst this document makes no specific recommendations with respect 1884 to Canonical CBOR (see Section 3.9 of [RFC7049]) the following 1885 guidance may be of use to implementors. 1887 Adherence to the first two rules given in Section 3.9 of [RFC7049] 1888 will minimise file sizes. 1890 Adherence to the last two rules given in Section 3.9 of [RFC7049] for 1891 all maps and arrays would unacceptably constrain implementations, for 1892 example, in the use case of real-time data collection in constrained 1893 environments. 1895 11.1. Optional data 1897 When decoding C-DNS data some of the items required for a particular 1898 function that the consumer wishes to perform may be missing. 1899 Consumers should consider providing configurable default values to be 1900 used in place of the missing values in their output. 1902 11.2. Trailing bytes 1904 A DNS query message in a UDP or TCP payload can be followed by some 1905 additional (spurious) bytes, which are not stored in C-DNS. 1907 When DNS traffic is sent over TCP, each message is prefixed with a 1908 two byte length field which gives the message length, excluding the 1909 two byte length field. In this context, trailing bytes can occur in 1910 two circumstances with different results: 1912 1. The number of bytes consumed by fully parsing the message is less 1913 than the number of bytes given in the length field (i.e. the 1914 length field is incorrect and too large). In this case, the 1915 surplus bytes are considered trailing bytes in an analogous 1916 manner to UDP and recorded as such. If only this case occurs it 1917 is possible to process a packet containing multiple DNS messages 1918 where one or more has trailing bytes. 1920 2. There are surplus bytes between the end of a well-formed message 1921 and the start of the length field for the next message. In this 1922 case the first of the surplus bytes will be processed as the 1923 first byte of the next length field, and parsing will proceed 1924 from there, almost certainly leading to the next and any 1925 subsequent messages in the packet being considered malformed. 1926 This will not generate a trailing bytes record for the processed 1927 well-formed message. 1929 11.3. Limiting collection of RDATA 1931 Implementations should consider providing a configurable maximum 1932 RDATA size for capture, for example, to avoid memory issues when 1933 confronted with large XFR records. 1935 12. Implementation status 1937 [Note to RFC Editor: please remove this section and reference to 1938 [RFC7942] prior to publication.] 1940 This section records the status of known implementations of the 1941 protocol defined by this specification at the time of posting of this 1942 Internet-Draft, and is based on a proposal described in [RFC7942]. 1944 The description of implementations in this section is intended to 1945 assist the IETF in its decision processes in progressing drafts to 1946 RFCs. Please note that the listing of any individual implementation 1947 here does not imply endorsement by the IETF. Furthermore, no effort 1948 has been spent to verify the information presented here that was 1949 supplied by IETF contributors. This is not intended as, and must not 1950 be construed to be, a catalog of available implementations or their 1951 features. Readers are advised to note that other implementations may 1952 exist. 1954 According to [RFC7942], "this will allow reviewers and working groups 1955 to assign due consideration to documents that have the benefit of 1956 running code, which may serve as evidence of valuable experimentation 1957 and feedback that have made the implemented protocols more mature. 1958 It is up to the individual working groups to use this information as 1959 they see fit". 1961 12.1. DNS-STATS Compactor 1963 ICANN/Sinodun IT have developed an open source implementation called 1964 DNS-STATS Compactor. The Compactor is a suite of tools which can 1965 capture DNS traffic (from either a network interface or a PCAP file) 1966 and store it in the Compacted-DNS (C-DNS) file format. PCAP files 1967 for the captured traffic can also be reconstructed. See Compactor 1968 [1]. 1970 This implementation: 1972 o covers the whole of the specification described in the -03 draft 1973 with the exception of support for malformed messages and pico 1974 second time resolution. (Note: this implementation does allow 1975 malformed messages to be recorded separately in a PCAP file). 1977 o is released under the Mozilla Public License Version 2.0. 1979 o has a users mailing list available, see dns-stats-users [2]. 1981 There is also some discussion of issues encountered during 1982 development available at Compressing Pcap Files [3] and Packet 1983 Capture [4]. 1985 This information was last updated on 3rd of May 2018. 1987 13. IANA considerations 1989 None 1991 14. Security considerations 1993 Any control interface MUST perform authentication and encryption. 1995 Any data upload MUST be authenticated and encrypted. 1997 15. Acknowledgements 1999 The authors wish to thank CZ.NIC, in particular Tomas Gavenciak, for 2000 many useful discussions on binary formats, compression and packet 2001 matching. Also Jan Vcelak and Wouter Wijngaards for discussions on 2002 name compression and Paul Hoffman for a detailed review of the 2003 document and the C-DNS CDDL. 2005 Thanks also to Robert Edmonds, Jerry Lundstroem, Richard Gibson, 2006 Stephane Bortzmeyer and many other members of DNSOP for review. 2008 Also, Miek Gieben for mmark [5] 2010 16. Changelog 2012 draft-ietf-dnsop-dns-capture-format-08 2014 o Convert diagrams to ASCII 2016 o Describe versioning 2018 o Fix unused group warning in CDDL 2020 draft-ietf-dnsop-dns-capture-format-07 2022 o Resolve outstanding questions and TODOs 2024 o Make RR RDATA optional 2026 o Update matching diagram and explain skew 2028 o Add count of discarded messages to block statistics 2030 o Editorial clarifications and improvements 2032 draft-ietf-dnsop-dns-capture-format-06 2034 o Correct BlockParameters type to map 2036 o Make RR ttl optional 2038 o Add storage flag indicating name normalisation 2039 o Add storage parameter fields for sampling and anonymisation 2040 methods 2042 o Editorial clarifications and improvements 2044 draft-ietf-dnsop-dns-capture-format-05 2046 o Make all data items in Q/R, QuerySignature and Malformed Message 2047 arrays optional 2049 o Re-structure the FilePreamble and ConfigurationParameters into 2050 BlockParameters 2052 o BlockParameters has separate Storage and Collection Parameters 2054 o Storage Parameters includes information on what optional fields 2055 are present, and flags specifying anonymisation or sampling 2057 o Addresses can now be stored as prefixes. 2059 o Switch to using a variable sub-second timing granularity 2061 o Add response bailiwick and query response type 2063 o Add specifics of how to record malformed messages 2065 o Add implementation guidance 2067 o Improve terminology and naming consistency 2069 draft-ietf-dnsop-dns-capture-format-04 2071 o Correct query-d0 to query-do in CDDL 2073 o Clarify that map keys are unsigned integers 2075 o Add Type to Class/Type table 2077 o Clarify storage format in section 7.12 2079 draft-ietf-dnsop-dns-capture-format-03 2081 o Added an Implementation Status section 2083 draft-ietf-dnsop-dns-capture-format-02 2085 o Update qr_data_format.png to match CDDL 2086 o Editorial clarifications and improvements 2088 draft-ietf-dnsop-dns-capture-format-01 2090 o Many editorial improvements by Paul Hoffman 2092 o Included discussion of malformed message handling 2094 o Improved Appendix C on Comparison of Binary Formats 2096 o Now using C-DNS field names in the tables in section 8 2098 o A handful of new fields included (CDDL updated) 2100 o Timestamps now include optional picoseconds 2102 o Added details of block statistics 2104 draft-ietf-dnsop-dns-capture-format-00 2106 o Changed dnstap.io to dnstap.info 2108 o qr_data_format.png was cut off at the bottom 2110 o Update authors address 2112 o Improve wording in Abstract 2114 o Changed DNS-STAT to C-DNS in CDDL 2116 o Set the format version in the CDDL 2118 o Added a TODO: Add block statistics 2120 o Added a TODO: Add extend to support pico/nano. Also do this for 2121 Time offset and Response delay 2123 o Added a TODO: Need to develop optional representation of malformed 2124 messages within C-DNS and what this means for packet matching. 2125 This may influence which fields are optional in the rest of the 2126 representation. 2128 o Added section on design goals to Introduction 2130 o Added a TODO: Can Class be optimised? Should a class of IN be 2131 inferred if not present? 2133 draft-dickinson-dnsop-dns-capture-format-00 2134 o Initial commit 2136 17. References 2138 17.1. Normative References 2140 [RFC1035] Mockapetris, P., "Domain names - implementation and 2141 specification", STD 13, RFC 1035, DOI 10.17487/RFC1035, 2142 November 1987, . 2144 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 2145 Requirement Levels", BCP 14, RFC 2119, 2146 DOI 10.17487/RFC2119, March 1997, . 2149 [RFC7049] Bormann, C. and P. Hoffman, "Concise Binary Object 2150 Representation (CBOR)", RFC 7049, DOI 10.17487/RFC7049, 2151 October 2013, . 2153 17.2. Informative References 2155 [ditl] DNS-OARC, "DITL", 2016, . 2158 [dnscap] DNS-OARC, "DNSCAP", 2016, . 2161 [dnstap] dnstap.info, "dnstap", 2016, . 2163 [dsc] Wessels, D. and J. Lundstrom, "DSC", 2016, 2164 . 2166 [I-D.daley-dnsxml] 2167 Daley, J., Morris, S., and J. Dickinson, "dnsxml - A 2168 standard XML representation of DNS data", draft-daley- 2169 dnsxml-00 (work in progress), July 2013. 2171 [I-D.hoffman-dns-in-json] 2172 Hoffman, P., "Representing DNS Messages in JSON", draft- 2173 hoffman-dns-in-json-16 (work in progress), May 2018. 2175 [I-D.ietf-cbor-cddl] 2176 Birkholz, H., Vigano, C., and C. Bormann, "Concise data 2177 definition language (CDDL): a notational convention to 2178 express CBOR data structures", draft-ietf-cbor-cddl-03 2179 (work in progress), July 2018. 2181 [packetq] .SE - The Internet Infrastructure Foundation, "PacketQ", 2182 2014, . 2184 [pcap] tcpdump.org, "PCAP", 2016, . 2186 [pcapng] Tuexen, M., Risso, F., Bongertz, J., Combs, G., and G. 2187 Harris, "pcap-ng", 2016, . 2190 [RFC7942] Sheffer, Y. and A. Farrel, "Improving Awareness of Running 2191 Code: The Implementation Status Section", BCP 205, 2192 RFC 7942, DOI 10.17487/RFC7942, July 2016, 2193 . 2195 [RFC8259] Bray, T., Ed., "The JavaScript Object Notation (JSON) Data 2196 Interchange Format", STD 90, RFC 8259, 2197 DOI 10.17487/RFC8259, December 2017, . 2200 17.3. URIs 2202 [1] https://github.com/dns-stats/compactor/wiki 2204 [2] https://mm.dns-stats.org/mailman/listinfo/dns-stats-users 2206 [3] https://www.sinodun.com/2017/06/compressing-pcap-files/ 2208 [4] https://www.sinodun.com/2017/06/more-on-debian-jessieubuntu- 2209 trusty-packet-capture-woes/ 2211 [5] https://github.com/miekg/mmark 2213 [6] https://www.nlnetlabs.nl/projects/nsd/ 2215 [7] https://www.knot-dns.cz/ 2217 [8] https://avro.apache.org/ 2219 [9] https://developers.google.com/protocol-buffers/ 2221 [10] http://cbor.io 2223 [11] https://github.com/kubo/snzip 2225 [12] http://google.github.io/snappy/ 2227 [13] http://lz4.github.io/lz4/ 2229 [14] http://www.gzip.org/ 2231 [15] http://facebook.github.io/zstd/ 2233 [16] http://tukaani.org/xz/ 2235 [17] https://github.com/dns-stats/draft-dns-capture- 2236 format/blob/master/file-size-versus-block-size.png 2238 [18] https://github.com/dns-stats/draft-dns-capture- 2239 format/blob/master/file-size-versus-block-size.svg 2241 Appendix A. CDDL 2243 This appendix gives a CDDL [I-D.ietf-cbor-cddl] specification for 2244 C-DNS. 2246 CDDL does not permit a range of allowed values to be specified for a 2247 bitfield. Where necessary, those values are given as a CDDL group, 2248 but the group definition is commented out to prevent CDDL tooling 2249 from warning that the group is unused. 2251 ; CDDL specification of the file format for C-DNS, 2252 ; which describes a collection of DNS messages and 2253 ; traffic meta-data. 2255 ; 2256 ; The overall structure of a file. 2257 ; 2258 File = [ 2259 file-type-id : tstr .regexp "C-DNS", 2260 file-preamble : FilePreamble, 2261 file-blocks : [* Block], 2262 ] 2264 ; 2265 ; The file preamble. 2266 ; 2267 FilePreamble = { 2268 major-format-version => uint .eq 1, 2269 minor-format-version => uint .eq 0, 2270 ? private-version => uint, 2271 block-parameters => [+ BlockParameters], 2272 } 2273 major-format-version = 0 2274 minor-format-version = 1 2275 private-version = 2 2276 block-parameters = 3 2277 BlockParameters = { 2278 storage-parameters => StorageParameters, 2279 ? collection-parameters => CollectionParameters, 2280 } 2281 storage-parameters = 0 2282 collection-parameters = 1 2284 StorageParameters = { 2285 ticks-per-second => uint, 2286 max-block-items => uint, 2287 storage-hints => StorageHints, 2288 opcodes => [+ uint], 2289 rr-types => [+ uint], 2290 ? storage-flags => StorageFlags, 2291 ? client-address-prefix-ipv4 => uint, 2292 ? client-address-prefix-ipv6 => uint, 2293 ? server-address-prefix-ipv4 => uint, 2294 ? server-address-prefix-ipv6 => uint, 2295 ? sampling-method => tstr, 2296 ? anonymisation-method => tstr, 2297 } 2298 ticks-per-second = 0 2299 max-block-items = 1 2300 storage-hints = 2 2301 opcodes = 3 2302 rr-types = 4 2303 storage-flags = 5 2304 client-address-prefix-ipv4 = 6 2305 client-address-prefix-ipv6 = 7 2306 server-address-prefix-ipv4 = 8 2307 server-address-prefix-ipv6 = 9 2308 sampling-method = 10 2309 anonymisation-method = 11 2311 ; A hint indicates if the collection method will output the 2312 ; item or will ignore the item if present. 2313 StorageHints = { 2314 query-response-hints => QueryResponseHints, 2315 query-response-signature-hints => QueryResponseSignatureHints, 2316 rr-hints => RRHints, 2317 other-data-hints => OtherDataHints, 2318 } 2319 query-response-hints = 0 2320 query-response-signature-hints = 1 2321 rr-hints = 2 2322 other-data-hints = 3 2324 QueryResponseHintValues = &( 2325 time-offset : 0, 2326 client-address-index : 1, 2327 client-port : 2, 2328 transaction-id : 3, 2329 qr-signature-index : 4, 2330 client-hoplimit : 5, 2331 response-delay : 6, 2332 query-name-index : 7, 2333 query-size : 8, 2334 response-size : 9, 2335 response-processing-data : 10, 2336 query-question-sections : 11, ; Second & subsequent questions 2337 query-answer-sections : 12, 2338 query-authority-sections : 13, 2339 query-additional-sections : 14, 2340 response-answer-sections : 15, 2341 response-authority-sections : 16, 2342 response-additional-sections : 17, 2343 ) 2344 QueryResponseHints = uint .bits QueryResponseHintValues 2346 QueryResponseSignatureHintValues = &( 2347 server-address : 0, 2348 server-port : 1, 2349 qr-transport-flags : 2, 2350 qr-type : 3, 2351 qr-sig-flags : 4, 2352 query-opcode : 5, 2353 dns-flags : 6, 2354 query-rcode : 7, 2355 query-class-type : 8, 2356 query-qdcount : 9, 2357 query-ancount : 10, 2358 query-arcount : 11, 2359 query-nscount : 12, 2360 query-edns-version : 13, 2361 query-udp-size : 14, 2362 query-opt-rdata : 15, 2363 response-rcode : 16, 2364 ) 2365 QueryResponseSignatureHints = uint .bits QueryResponseSignatureHintValues 2367 RRHintValues = &( 2368 ttl : 0, 2369 rdata-index : 1, 2370 ) 2371 RRHints = uint .bits RRHintValues 2372 OtherDataHintValues = &( 2373 malformed-messages : 0, 2374 address-event-counts : 1, 2375 ) 2376 OtherDataHints = uint .bits OtherDataHintValues 2378 StorageFlagValues = &( 2379 anonymised-data : 0, 2380 sampled-data : 1, 2381 normalised-names : 2, 2382 ) 2383 StorageFlags = uint .bits StorageFlagValues 2385 CollectionParameters = { 2386 ? query-timeout => uint, 2387 ? skew-timeout => uint, 2388 ? snaplen => uint, 2389 ? promisc => uint, 2390 ? interfaces => [+ tstr], 2391 ? server-addresses => [+ IPAddress], ; Hint for later analysis 2392 ? vlan-ids => [+ uint], 2393 ? filter => tstr, 2394 ? generator-id => tstr, 2395 ? host-id => tstr, 2396 } 2397 query-timeout = 0 2398 skew-timeout = 1 2399 snaplen = 2 2400 promisc = 3 2401 interfaces = 4 2402 server-addresses = 5 2403 vlan-ids = 6 2404 filter = 7 2405 generator-id = 8 2406 host-id = 9 2408 ; 2409 ; Data in the file is stored in Blocks. 2410 ; 2411 Block = { 2412 block-preamble => BlockPreamble, 2413 ? block-statistics => BlockStatistics, ; Much of this could be derived 2414 ? block-tables => BlockTables, 2415 ? query-responses => [+ QueryResponse], 2416 ? address-event-counts => [+ AddressEventCount], 2417 ? malformed-messages => [+ MalformedMessage], 2418 } 2419 block-preamble = 0 2420 block-statistics = 1 2421 block-tables = 2 2422 query-responses = 3 2423 address-event-counts = 4 2424 malformed-messages = 5 2426 ; 2427 ; The (mandatory) preamble to a block. 2428 ; 2429 BlockPreamble = { 2430 ? earliest-time => Timestamp, 2431 ? block-parameters-index => uint .default 0, 2432 } 2433 earliest-time = 0 2434 block-parameters-index = 1 2436 ; Ticks are subsecond intervals. The number of ticks in a second is file/block 2437 ; metadata. Signed and unsigned tick types are defined. 2438 ticks = int 2439 uticks = uint 2441 Timestamp = [ 2442 timestamp-secs : uint, 2443 timestamp-uticks : uticks, 2444 ] 2446 ; 2447 ; Statistics about the block contents. 2448 ; 2449 BlockStatistics = { 2450 ? processed-messages => uint, 2451 ? qr-data-items => uint, 2452 ? unmatched-queries => uint, 2453 ? unmatched-responses => uint, 2454 ? discarded-opcode => uint, 2455 ? malformed-items => uint, 2456 } 2457 processed-messages = 0 2458 qr-data-items = 1 2459 unmatched-queries = 2 2460 unmatched-responses = 3 2461 discarded-opcode = 4 2462 malformed-items = 5 2464 ; 2465 ; Tables of common data referenced from records in a block. 2466 ; 2467 BlockTables = { 2468 ? ip-address => [+ IPAddress], 2469 ? classtype => [+ ClassType], 2470 ? name-rdata => [+ bstr], ; Holds both Name RDATA and RDATA 2471 ? qr-sig => [+ QueryResponseSignature], 2472 ? QuestionTables, 2473 ? RRTables, 2474 ? malformed-message-data => [+ MalformedMessageData], 2475 } 2476 ip-address = 0 2477 classtype = 1 2478 name-rdata = 2 2479 qr-sig = 3 2480 qlist = 4 2481 qrr = 5 2482 rrlist = 6 2483 rr = 7 2484 malformed-message-data = 8 2486 IPv4Address = bstr .size 4 2487 IPv6Address = bstr .size 16 2488 IPAddress = IPv4Address / IPv6Address 2490 ClassType = { 2491 type => uint, 2492 class => uint, 2493 } 2494 type = 0 2495 class = 1 2497 QueryResponseSignature = { 2498 ? server-address-index => uint, 2499 ? server-port => uint, 2500 ? qr-transport-flags => QueryResponseTransportFlags, 2501 ? qr-type => QueryResponseType, 2502 ? qr-sig-flags => QueryResponseFlags, 2503 ? query-opcode => uint, 2504 ? qr-dns-flags => DNSFlags, 2505 ? query-rcode => uint, 2506 ? query-classtype-index => uint, 2507 ? query-qd-count => uint, 2508 ? query-an-count => uint, 2509 ? query-ns-count => uint, 2510 ? query-ar-count => uint, 2511 ? edns-version => uint, 2512 ? udp-buf-size => uint, 2513 ? opt-rdata-index => uint, 2514 ? response-rcode => uint, 2515 } 2516 server-address-index = 0 2517 server-port = 1 2518 qr-transport-flags = 2 2519 qr-type = 3 2520 qr-sig-flags = 4 2521 query-opcode = 5 2522 qr-dns-flags = 6 2523 query-rcode = 7 2524 query-classtype-index = 8 2525 query-qd-count = 9 2526 query-an-count = 10 2527 query-ns-count = 12 2528 query-ar-count = 12 2529 edns-version = 13 2530 udp-buf-size = 14 2531 opt-rdata-index = 15 2532 response-rcode = 16 2534 ; Transport gives the values that may appear in bits 1..4 of 2535 ; TransportFlags. There is currently no way to express this in 2536 ; CDDL, so Transport is unused. To avoid confusion when used 2537 ; with CDDL tools, it is commented out. 2538 ; 2539 ; Transport = &( 2540 ; udp : 0, 2541 ; tcp : 1, 2542 ; tls : 2, 2543 ; dtls : 3, 2544 ; ) 2546 TransportFlagValues = &( 2547 ip-version : 0, ; 0=IPv4, 1=IPv6 2548 ) / (1..4) 2549 TransportFlags = uint .bits TransportFlagValues 2551 QueryResponseTransportFlagValues = &( 2552 query-trailingdata : 5, 2553 ) / TransportFlagValues 2554 QueryResponseTransportFlags = uint .bits QueryResponseTransportFlagValues 2556 QueryResponseType = &( 2557 stub : 0, 2558 client : 1, 2559 resolver : 2, 2560 auth : 3, 2561 forwarder : 4, 2562 tool : 5, 2563 ) 2564 QueryResponseFlagValues = &( 2565 has-query : 0, 2566 has-reponse : 1, 2567 query-has-opt : 2, 2568 response-has-opt : 3, 2569 query-has-no-question : 4, 2570 response-has-no-question: 5, 2571 ) 2572 QueryResponseFlags = uint .bits QueryResponseFlagValues 2574 DNSFlagValues = &( 2575 query-cd : 0, 2576 query-ad : 1, 2577 query-z : 2, 2578 query-ra : 3, 2579 query-rd : 4, 2580 query-tc : 5, 2581 query-aa : 6, 2582 query-do : 7, 2583 response-cd: 8, 2584 response-ad: 9, 2585 response-z : 10, 2586 response-ra: 11, 2587 response-rd: 12, 2588 response-tc: 13, 2589 response-aa: 14, 2590 ) 2591 DNSFlags = uint .bits DNSFlagValues 2593 QuestionTables = ( 2594 qlist => [+ QuestionList], 2595 qrr => [+ Question] 2596 ) 2598 QuestionList = [+ uint] ; Index of Question 2600 Question = { ; Second and subsequent questions 2601 name-index => uint, ; Index to a name in the name-rdata table 2602 classtype-index => uint, 2603 } 2604 name-index = 0 2605 classtype-index = 1 2607 RRTables = ( 2608 rrlist => [+ RRList], 2609 rr => [+ RR] 2610 ) 2611 RRList = [+ uint] ; Index of RR 2613 RR = { 2614 name-index => uint, ; Index to a name in the name-rdata table 2615 classtype-index => uint, 2616 ? ttl => uint, 2617 ? rdata-index => uint, ; Index to RDATA in the name-rdata table 2618 } 2619 ; Other map key values already defined above. 2620 ttl = 2 2621 rdata-index = 3 2623 MalformedMessageData = { 2624 ? server-address-index => uint, 2625 ? server-port => uint, 2626 ? mm-transport-flags => TransportFlags, 2627 ? mm-payload => bstr, 2628 } 2629 ; Other map key values already defined above. 2630 mm-transport-flags = 2 2631 mm-payload = 3 2633 ; 2634 ; A single query/response pair. 2635 ; 2636 QueryResponse = { 2637 ? time-offset => uticks, ; Time offset from start of block 2638 ? client-address-index => uint, 2639 ? client-port => uint, 2640 ? transaction-id => uint, 2641 ? qr-signature-index => uint, 2642 ? client-hoplimit => uint, 2643 ? response-delay => ticks, 2644 ? query-name-index => uint, 2645 ? query-size => uint, ; DNS size of query 2646 ? response-size => uint, ; DNS size of response 2647 ? response-processing-data => ResponseProcessingData, 2648 ? query-extended => QueryResponseExtended, 2649 ? response-extended => QueryResponseExtended, 2650 } 2651 time-offset = 0 2652 client-address-index = 1 2653 client-port = 2 2654 transaction-id = 3 2655 qr-signature-index = 4 2656 client-hoplimit = 5 2657 response-delay = 6 2658 query-name-index = 7 2659 query-size = 8 2660 response-size = 9 2661 response-processing-data = 10 2662 query-extended = 11 2663 response-extended = 12 2665 ResponseProcessingData = { 2666 ? bailiwick-index => uint, 2667 ? processing-flags => ResponseProcessingFlags, 2668 } 2669 bailiwick-index = 0 2670 processing-flags = 1 2672 ResponseProcessingFlagValues = &( 2673 from-cache : 0, 2674 ) 2675 ResponseProcessingFlags = uint .bits ResponseProcessingFlagValues 2677 QueryResponseExtended = { 2678 ? question-index => uint, ; Index of QuestionList 2679 ? answer-index => uint, ; Index of RRList 2680 ? authority-index => uint, 2681 ? additional-index => uint, 2682 } 2683 question-index = 0 2684 answer-index = 1 2685 authority-index = 2 2686 additional-index = 3 2688 ; 2689 ; Address event data. 2690 ; 2691 AddressEventCount = { 2692 ae-type => &AddressEventType, 2693 ? ae-code => uint, 2694 ae-address-index => uint, 2695 ae-count => uint, 2696 } 2697 ae-type = 0 2698 ae-code = 1 2699 ae-address-index = 2 2700 ae-count = 3 2702 AddressEventType = ( 2703 tcp-reset : 0, 2704 icmp-time-exceeded : 1, 2705 icmp-dest-unreachable : 2, 2706 icmpv6-time-exceeded : 3, 2707 icmpv6-dest-unreachable: 4, 2708 icmpv6-packet-too-big : 5, 2709 ) 2711 ; 2712 ; Malformed messages. 2713 ; 2714 MalformedMessage = { 2715 ? time-offset => uticks, ; Time offset from start of block 2716 ? client-address-index => uint, 2717 ? client-port => uint, 2718 ? message-data-index => uint, 2719 } 2720 ; Other map key values already defined above. 2721 message-data-index = 3 2723 Appendix B. DNS Name compression example 2725 The basic algorithm, which follows the guidance in [RFC1035], is 2726 simply to collect each name, and the offset in the packet at which it 2727 starts, during packet construction. As each name is added, it is 2728 offered to each of the collected names in order of collection, 2729 starting from the first name. If labels at the end of the name can 2730 be replaced with a reference back to part (or all) of the earlier 2731 name, and if the uncompressed part of the name is shorter than any 2732 compression already found, the earlier name is noted as the 2733 compression target for the name. 2735 The following tables illustrate the process. In an example packet, 2736 the first name is example.com. 2738 +---+-------------+--------------+--------------------+ 2739 | N | Name | Uncompressed | Compression Target | 2740 +---+-------------+--------------+--------------------+ 2741 | 1 | example.com | | | 2742 +---+-------------+--------------+--------------------+ 2744 The next name added is bar.com. This is matched against example.com. 2745 The com part of this can be used as a compression target, with the 2746 remaining uncompressed part of the name being bar. 2748 +---+-------------+--------------+--------------------+ 2749 | N | Name | Uncompressed | Compression Target | 2750 +---+-------------+--------------+--------------------+ 2751 | 1 | example.com | | | 2752 | 2 | bar.com | bar | 1 + offset to com | 2753 +---+-------------+--------------+--------------------+ 2755 The third name added is www.bar.com. This is first matched against 2756 example.com, and as before this is recorded as a compression target, 2757 with the remaining uncompressed part of the name being www.bar. It 2758 is then matched against the second name, which again can be a 2759 compression target. Because the remaining uncompressed part of the 2760 name is www, this is an improved compression, and so it is adopted. 2762 +---+-------------+--------------+--------------------+ 2763 | N | Name | Uncompressed | Compression Target | 2764 +---+-------------+--------------+--------------------+ 2765 | 1 | example.com | | | 2766 | 2 | bar.com | bar | 1 + offset to com | 2767 | 3 | www.bar.com | www | 2 | 2768 +---+-------------+--------------+--------------------+ 2770 As an optimization, if a name is already perfectly compressed (in 2771 other words, the uncompressed part of the name is empty), then no 2772 further names will be considered for compression. 2774 B.1. NSD compression algorithm 2776 Using the above basic algorithm the packet lengths of responses 2777 generated by NSD [6] can be matched almost exactly. At the time of 2778 writing, a tiny number (<.01%) of the reconstructed packets had 2779 incorrect lengths. 2781 B.2. Knot Authoritative compression algorithm 2783 The Knot Authoritative [7] name server uses different compression 2784 behavior, which is the result of internal optimization designed to 2785 balance runtime speed with compression size gains. In brief, and 2786 omitting complications, Knot Authoritative will only consider the 2787 QNAME and names in the immediately preceding RR section in an RRSET 2788 as compression targets. 2790 A set of smart heuristics as described below can be implemented to 2791 mimic this and while not perfect it produces output nearly, but not 2792 quite, as good a match as with NSD. The heuristics are: 2794 1. A match is only perfect if the name is completely compressed AND 2795 the TYPE of the section in which the name occurs matches the TYPE 2796 of the name used as the compression target. 2798 2. If the name occurs in RDATA: 2800 * If the compression target name is in a query, then only the 2801 first RR in an RRSET can use that name as a compression 2802 target. 2804 * The compression target name MUST be in RDATA. 2806 * The name section TYPE must match the compression target name 2807 section TYPE. 2809 * The compression target name MUST be in the immediately 2810 preceding RR in the RRSET. 2812 Using this algorithm less than 0.1% of the reconstructed packets had 2813 incorrect lengths. 2815 B.3. Observed differences 2817 In sample traffic collected on a root name server around 2-4% of 2818 responses generated by Knot had different packet lengths to those 2819 produced by NSD. 2821 Appendix C. Comparison of Binary Formats 2823 Several binary serialisation formats were considered, and for 2824 completeness were also compared to JSON. 2826 o Apache Avro [8]. Data is stored according to a pre-defined 2827 schema. The schema itself is always included in the data file. 2828 Data can therefore be stored untagged, for a smaller serialisation 2829 size, and be written and read by an Avro library. 2831 * At the time of writing, Avro libraries are available for C, 2832 C++, C#, Java, Python, Ruby and PHP. Optionally tools are 2833 available for C++, Java and C# to generate code for encoding 2834 and decoding. 2836 o Google Protocol Buffers [9]. Data is stored according to a pre- 2837 defined schema. The schema is used by a generator to generate 2838 code for encoding and decoding the data. Data can therefore be 2839 stored untagged, for a smaller serialisation size. The schema is 2840 not stored with the data, so unlike Avro cannot be read with a 2841 generic library. 2843 * Code must be generated for a particular data schema to to read 2844 and write data using that schema. At the time of writing, the 2845 Google code generator can currently generate code for encoding 2846 and decoding a schema for C++, Go, Java, Python, Ruby, C#, 2847 Objective-C, Javascript and PHP. 2849 o CBOR [10]. Defined in [RFC7049], this serialisation format is 2850 comparable to JSON but with a binary representation. It does not 2851 use a pre-defined schema, so data is always stored tagged. 2853 However, CBOR data schemas can be described using CDDL 2854 [I-D.ietf-cbor-cddl] and tools exist to verify data files conform 2855 to the schema. 2857 * CBOR is a simple format, and simple to implement. At the time 2858 of writing, the CBOR website lists implementations for 16 2859 languages. 2861 Avro and Protocol Buffers both allow storage of untagged data, but 2862 because they rely on the data schema for this, their implementation 2863 is considerably more complex than CBOR. Using Avro or Protocol 2864 Buffers in an unsupported environment would require notably greater 2865 development effort compared to CBOR. 2867 A test program was written which reads input from a PCAP file and 2868 writes output using one of two basic structures; either a simple 2869 structure, where each query/response pair is represented in a single 2870 record entry, or the C-DNS block structure. 2872 The resulting output files were then compressed using a variety of 2873 common general-purpose lossless compression tools to explore the 2874 compressibility of the formats. The compression tools employed were: 2876 o snzip [11]. A command line compression tool based on the Google 2877 Snappy [12] library. 2879 o lz4 [13]. The command line compression tool from the reference C 2880 LZ4 implementation. 2882 o gzip [14]. The ubiquitous GNU zip tool. 2884 o zstd [15]. Compression using the Zstandard algorithm. 2886 o xz [16]. A popular compression tool noted for high compression. 2888 In all cases the compression tools were run using their default 2889 settings. 2891 Note that this draft does not mandate the use of compression, nor any 2892 particular compression scheme, but it anticipates that in practice 2893 output data will be subject to general-purpose compression, and so 2894 this should be taken into consideration. 2896 "test.pcap", a 662Mb capture of sample data from a root instance was 2897 used for the comparison. The following table shows the formatted 2898 size and size after compression (abbreviated to Comp. in the table 2899 headers), together with the task resident set size (RSS) and the user 2900 time taken by the compression. File sizes are in Mb, RSS in kb and 2901 user time in seconds. 2903 +-------------+-----------+-------+------------+-------+-----------+ 2904 | Format | File size | Comp. | Comp. size | RSS | User time | 2905 +-------------+-----------+-------+------------+-------+-----------+ 2906 | PCAP | 661.87 | snzip | 212.48 | 2696 | 1.26 | 2907 | | | lz4 | 181.58 | 6336 | 1.35 | 2908 | | | gzip | 153.46 | 1428 | 18.20 | 2909 | | | zstd | 87.07 | 3544 | 4.27 | 2910 | | | xz | 49.09 | 97416 | 160.79 | 2911 | | | | | | | 2912 | JSON simple | 4113.92 | snzip | 603.78 | 2656 | 5.72 | 2913 | | | lz4 | 386.42 | 5636 | 5.25 | 2914 | | | gzip | 271.11 | 1492 | 73.00 | 2915 | | | zstd | 133.43 | 3284 | 8.68 | 2916 | | | xz | 51.98 | 97412 | 600.74 | 2917 | | | | | | | 2918 | Avro simple | 640.45 | snzip | 148.98 | 2656 | 0.90 | 2919 | | | lz4 | 111.92 | 5828 | 0.99 | 2920 | | | gzip | 103.07 | 1540 | 11.52 | 2921 | | | zstd | 49.08 | 3524 | 2.50 | 2922 | | | xz | 22.87 | 97308 | 90.34 | 2923 | | | | | | | 2924 | CBOR simple | 764.82 | snzip | 164.57 | 2664 | 1.11 | 2925 | | | lz4 | 120.98 | 5892 | 1.13 | 2926 | | | gzip | 110.61 | 1428 | 12.88 | 2927 | | | zstd | 54.14 | 3224 | 2.77 | 2928 | | | xz | 23.43 | 97276 | 111.48 | 2929 | | | | | | | 2930 | PBuf simple | 749.51 | snzip | 167.16 | 2660 | 1.08 | 2931 | | | lz4 | 123.09 | 5824 | 1.14 | 2932 | | | gzip | 112.05 | 1424 | 12.75 | 2933 | | | zstd | 53.39 | 3388 | 2.76 | 2934 | | | xz | 23.99 | 97348 | 106.47 | 2935 | | | | | | | 2936 | JSON block | 519.77 | snzip | 106.12 | 2812 | 0.93 | 2937 | | | lz4 | 104.34 | 6080 | 0.97 | 2938 | | | gzip | 57.97 | 1604 | 12.70 | 2939 | | | zstd | 61.51 | 3396 | 3.45 | 2940 | | | xz | 27.67 | 97524 | 169.10 | 2941 | | | | | | | 2942 | Avro block | 60.45 | snzip | 48.38 | 2688 | 0.20 | 2943 | | | lz4 | 48.78 | 8540 | 0.22 | 2944 | | | gzip | 39.62 | 1576 | 2.92 | 2945 | | | zstd | 29.63 | 3612 | 1.25 | 2946 | | | xz | 18.28 | 97564 | 25.81 | 2947 | | | | | | | 2948 | CBOR block | 75.25 | snzip | 53.27 | 2684 | 0.24 | 2949 | | | lz4 | 51.88 | 8008 | 0.28 | 2950 | | | gzip | 41.17 | 1548 | 4.36 | 2951 | | | zstd | 30.61 | 3476 | 1.48 | 2952 | | | xz | 18.15 | 97556 | 38.78 | 2953 | | | | | | | 2954 | PBuf block | 67.98 | snzip | 51.10 | 2636 | 0.24 | 2955 | | | lz4 | 52.39 | 8304 | 0.24 | 2956 | | | gzip | 40.19 | 1520 | 3.63 | 2957 | | | zstd | 31.61 | 3576 | 1.40 | 2958 | | | xz | 17.94 | 97440 | 33.99 | 2959 +-------------+-----------+-------+------------+-------+-----------+ 2961 The above results are discussed in the following sections. 2963 C.1. Comparison with full PCAP files 2965 An important first consideration is whether moving away from PCAP 2966 offers significant benefits. 2968 The simple binary formats are typically larger than PCAP, even though 2969 they omit some information such as Ethernet MAC addresses. But not 2970 only do they require less CPU to compress than PCAP, the resulting 2971 compressed files are smaller than compressed PCAP. 2973 C.2. Simple versus block coding 2975 The intention of the block coding is to perform data de-duplication 2976 on query/response records within the block. The simple and block 2977 formats above store exactly the same information for each query/ 2978 response record. This information is parsed from the DNS traffic in 2979 the input PCAP file, and in all cases each field has an identifier 2980 and the field data is typed. 2982 The data de-duplication on the block formats show an order of 2983 magnitude reduction in the size of the format file size against the 2984 simple formats. As would be expected, the compression tools are able 2985 to find and exploit a lot of this duplication, but as the de- 2986 duplication process uses knowledge of DNS traffic, it is able to 2987 retain a size advantage. This advantage reduces as stronger 2988 compression is applied, as again would be expected, but even with the 2989 strongest compression applied the block formatted data remains around 2990 75% of the size of the simple format and its compression requires 2991 roughly a third of the CPU time. 2993 C.3. Binary versus text formats 2995 Text data formats offer many advantages over binary formats, 2996 particularly in the areas of ad-hoc data inspection and extraction. 2997 It was therefore felt worthwhile to carry out a direct comparison, 2998 implementing JSON versions of the simple and block formats. 3000 Concentrating on JSON block format, the format files produced are a 3001 significant fraction of an order of magnitude larger than binary 3002 formats. The impact on file size after compression is as might be 3003 expected from that starting point; the stronger compression produces 3004 files that are 150% of the size of similarly compressed binary 3005 format, and require over 4x more CPU to compress. 3007 C.4. Performance 3009 Concentrating again on the block formats, all three produce format 3010 files that are close to an order of magnitude smaller that the 3011 original "test.pcap" file. CBOR produces the largest files and Avro 3012 the smallest, 20% smaller than CBOR. 3014 However, once compression is taken into account, the size difference 3015 narrows. At medium compression (with gzip), the size difference is 3016 4%. Using strong compression (with xz) the difference reduces to 2%, 3017 with Avro the largest and Protocol Buffers the smallest, although 3018 CBOR and Protocol Buffers require slightly more compression CPU. 3020 The measurements presented above do not include data on the CPU 3021 required to generate the format files. Measurements indicate that 3022 writing Avro requires 10% more CPU than CBOR or Protocol Buffers. It 3023 appears, therefore, that Avro's advantage in compression CPU usage is 3024 probably offset by a larger CPU requirement in writing Avro. 3026 C.5. Conclusions 3028 The above assessments lead us to the choice of a binary format file 3029 using blocking. 3031 As noted previously, this draft anticipates that output data will be 3032 subject to compression. There is no compelling case for one 3033 particular binary serialisation format in terms of either final file 3034 size or machine resources consumed, so the choice must be largely 3035 based on other factors. CBOR was therefore chosen as the binary 3036 serialisation format for the reasons listed in Section 5. 3038 C.6. Block size choice 3040 Given the choice of a CBOR format using blocking, the question arises 3041 of what an appropriate default value for the maximum number of query/ 3042 response pairs in a block should be. This has two components; what 3043 is the impact on performance of using different block sizes in the 3044 format file, and what is the impact on the size of the format file 3045 before and after compression. 3047 The following table addresses the performance question, showing the 3048 impact on the performance of a C++ program converting "test.pcap" to 3049 C-DNS. File size is in Mb, resident set size (RSS) in kb. 3051 +------------+-----------+--------+-----------+ 3052 | Block size | File size | RSS | User time | 3053 +------------+-----------+--------+-----------+ 3054 | 1000 | 133.46 | 612.27 | 15.25 | 3055 | 5000 | 89.85 | 676.82 | 14.99 | 3056 | 10000 | 76.87 | 752.40 | 14.53 | 3057 | 20000 | 67.86 | 750.75 | 14.49 | 3058 | 40000 | 61.88 | 736.30 | 14.29 | 3059 | 80000 | 58.08 | 694.16 | 14.28 | 3060 | 160000 | 55.94 | 733.84 | 14.44 | 3061 | 320000 | 54.41 | 799.20 | 13.97 | 3062 +------------+-----------+--------+-----------+ 3064 Increasing block size, therefore, tends to increase maximum RSS a 3065 little, with no significant effect (if anything a small reduction) on 3066 CPU consumption. 3068 The following figure plots the effect of increasing block size on 3069 output file size for different compressions. 3071 Figure showing effect of block size on file size (PNG) [17] 3073 Figure showing effect of block size on file size (SVG) [18] 3075 From the above, there is obviously scope for tuning the default block 3076 size to the compression being employed, traffic characteristics, 3077 frequency of output file rollover etc. Using a strong compression, 3078 block sizes over 10,000 query/response pairs would seem to offer 3079 limited improvements. 3081 Authors' Addresses 3082 John Dickinson 3083 Sinodun IT 3084 Magdalen Centre 3085 Oxford Science Park 3086 Oxford OX4 4GA 3087 United Kingdom 3089 Email: jad@sinodun.com 3091 Jim Hague 3092 Sinodun IT 3093 Magdalen Centre 3094 Oxford Science Park 3095 Oxford OX4 4GA 3096 United Kingdom 3098 Email: jim@sinodun.com 3100 Sara Dickinson 3101 Sinodun IT 3102 Magdalen Centre 3103 Oxford Science Park 3104 Oxford OX4 4GA 3105 United Kingdom 3107 Email: sara@sinodun.com 3109 Terry Manderson 3110 ICANN 3111 12025 Waterfront Drive 3112 Suite 300 3113 Los Angeles CA 90094-2536 3115 Email: terry.manderson@icann.org 3117 John Bond 3118 ICANN 3119 12025 Waterfront Drive 3120 Suite 300 3121 Los Angeles CA 90094-2536 3123 Email: john.bond@icann.org