idnits 2.17.1 draft-ietf-dnsop-dns-capture-format-10.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- == There are 2 instances of lines with non-RFC6890-compliant IPv4 addresses in the document. If these are example addresses, they should be changed. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (December 12, 2018) is 1961 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) -- Looks like a reference, but probably isn't: '2' on line 2575 -- Looks like a reference, but probably isn't: '1' on line 2573 -- Looks like a reference, but probably isn't: '3' on line 2577 -- Looks like a reference, but probably isn't: '4' on line 2579 -- Looks like a reference, but probably isn't: '5' on line 2582 -- Looks like a reference, but probably isn't: '6' on line 2585 -- Looks like a reference, but probably isn't: '7' on line 3168 -- Looks like a reference, but probably isn't: '8' on line 3174 -- Looks like a reference, but probably isn't: '9' on line 3217 -- Looks like a reference, but probably isn't: '10' on line 3228 -- Looks like a reference, but probably isn't: '11' on line 3241 -- Looks like a reference, but probably isn't: '12' on line 3267 -- Looks like a reference, but probably isn't: '13' on line 3268 -- Looks like a reference, but probably isn't: '14' on line 3270 -- Looks like a reference, but probably isn't: '15' on line 3273 -- Looks like a reference, but probably isn't: '16' on line 3275 -- Looks like a reference, but probably isn't: '17' on line 3277 == Outdated reference: A later version (-08) exists of draft-ietf-cbor-cddl-06 ** Obsolete normative reference: RFC 7049 (Obsoleted by RFC 8949) == Outdated reference: A later version (-02) exists of draft-bortzmeyer-dprive-rfc7626-bis-01 Summary: 1 error (**), 0 flaws (~~), 4 warnings (==), 18 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 dnsop J. Dickinson 3 Internet-Draft J. Hague 4 Intended status: Standards Track S. Dickinson 5 Expires: June 15, 2019 Sinodun IT 6 T. Manderson 7 J. Bond 8 ICANN 9 December 12, 2018 11 C-DNS: A DNS Packet Capture Format 12 draft-ietf-dnsop-dns-capture-format-10 14 Abstract 16 This document describes a data representation for collections of DNS 17 messages. The format is designed for efficient storage and 18 transmission of large packet captures of DNS traffic; it attempts to 19 minimize the size of such packet capture files but retain the full 20 DNS message contents along with the most useful transport metadata. 21 It is intended to assist with the development of DNS traffic 22 monitoring applications. 24 Status of This Memo 26 This Internet-Draft is submitted in full conformance with the 27 provisions of BCP 78 and BCP 79. 29 Internet-Drafts are working documents of the Internet Engineering 30 Task Force (IETF). Note that other groups may also distribute 31 working documents as Internet-Drafts. The list of current Internet- 32 Drafts is at https://datatracker.ietf.org/drafts/current/. 34 Internet-Drafts are draft documents valid for a maximum of six months 35 and may be updated, replaced, or obsoleted by other documents at any 36 time. It is inappropriate to use Internet-Drafts as reference 37 material or to cite them other than as "work in progress." 39 This Internet-Draft will expire on June 15, 2019. 41 Copyright Notice 43 Copyright (c) 2018 IETF Trust and the persons identified as the 44 document authors. All rights reserved. 46 This document is subject to BCP 78 and the IETF Trust's Legal 47 Provisions Relating to IETF Documents 48 (https://trustee.ietf.org/license-info) in effect on the date of 49 publication of this document. Please review these documents 50 carefully, as they describe your rights and restrictions with respect 51 to this document. Code Components extracted from this document must 52 include Simplified BSD License text as described in Section 4.e of 53 the Trust Legal Provisions and are provided without warranty as 54 described in the Simplified BSD License. 56 Table of Contents 58 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 59 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 5 60 3. Data collection use cases . . . . . . . . . . . . . . . . . . 5 61 4. Design considerations . . . . . . . . . . . . . . . . . . . . 7 62 5. Choice of CBOR . . . . . . . . . . . . . . . . . . . . . . . 9 63 6. C-DNS format conceptual overview . . . . . . . . . . . . . . 9 64 6.1. Block Parameters . . . . . . . . . . . . . . . . . . . . 13 65 6.2. Storage Parameters . . . . . . . . . . . . . . . . . . . 13 66 6.2.1. Optional data items . . . . . . . . . . . . . . . . . 14 67 6.2.2. Optional RRs and OPCODEs . . . . . . . . . . . . . . 15 68 6.2.3. Storage flags . . . . . . . . . . . . . . . . . . . . 15 69 6.2.4. IP Address storage . . . . . . . . . . . . . . . . . 16 70 7. C-DNS format detailed description . . . . . . . . . . . . . . 16 71 7.1. Map quantities and indexes . . . . . . . . . . . . . . . 16 72 7.2. Tabular representation . . . . . . . . . . . . . . . . . 17 73 7.3. "File" . . . . . . . . . . . . . . . . . . . . . . . . . 18 74 7.4. "FilePreamble" . . . . . . . . . . . . . . . . . . . . . 18 75 7.4.1. "BlockParameters" . . . . . . . . . . . . . . . . . . 19 76 7.4.2. "CollectionParameters" . . . . . . . . . . . . . . . 22 77 7.5. "Block" . . . . . . . . . . . . . . . . . . . . . . . . . 24 78 7.5.1. "BlockPreamble" . . . . . . . . . . . . . . . . . . . 24 79 7.5.2. "BlockStatistics" . . . . . . . . . . . . . . . . . . 25 80 7.5.3. "BlockTables" . . . . . . . . . . . . . . . . . . . . 26 81 7.6. "QueryResponse" . . . . . . . . . . . . . . . . . . . . . 32 82 7.6.1. "ResponseProcessingData" . . . . . . . . . . . . . . 34 83 7.6.2. "QueryResponseExtended" . . . . . . . . . . . . . . . 34 84 7.7. "AddressEventCount" . . . . . . . . . . . . . . . . . . . 35 85 7.8. "MalformedMessage" . . . . . . . . . . . . . . . . . . . 36 86 8. Versioning . . . . . . . . . . . . . . . . . . . . . . . . . 37 87 9. C-DNS to PCAP . . . . . . . . . . . . . . . . . . . . . . . . 37 88 9.1. Name compression . . . . . . . . . . . . . . . . . . . . 38 89 10. Data collection . . . . . . . . . . . . . . . . . . . . . . . 39 90 10.1. Matching algorithm . . . . . . . . . . . . . . . . . . . 40 91 10.2. Message identifiers . . . . . . . . . . . . . . . . . . 42 92 10.2.1. Primary ID (required) . . . . . . . . . . . . . . . 42 93 10.2.2. Secondary ID (optional) . . . . . . . . . . . . . . 43 94 10.3. Algorithm parameters . . . . . . . . . . . . . . . . . . 43 95 10.4. Algorithm requirements . . . . . . . . . . . . . . . . . 43 96 10.5. Algorithm limitations . . . . . . . . . . . . . . . . . 43 97 10.6. Workspace . . . . . . . . . . . . . . . . . . . . . . . 44 98 10.7. Output . . . . . . . . . . . . . . . . . . . . . . . . . 44 99 10.8. Post processing . . . . . . . . . . . . . . . . . . . . 44 100 11. Implementation guidance . . . . . . . . . . . . . . . . . . . 44 101 11.1. Optional data . . . . . . . . . . . . . . . . . . . . . 45 102 11.2. Trailing bytes . . . . . . . . . . . . . . . . . . . . . 45 103 11.3. Limiting collection of RDATA . . . . . . . . . . . . . . 45 104 11.4. Timestamps . . . . . . . . . . . . . . . . . . . . . . . 45 105 12. Implementation status . . . . . . . . . . . . . . . . . . . . 46 106 12.1. DNS-STATS Compactor . . . . . . . . . . . . . . . . . . 46 107 13. IANA considerations . . . . . . . . . . . . . . . . . . . . . 47 108 13.1. Transport types . . . . . . . . . . . . . . . . . . . . 47 109 13.2. Data storage flags . . . . . . . . . . . . . . . . . . . 48 110 13.3. Response processing flags . . . . . . . . . . . . . . . 48 111 13.4. AddressEvent types . . . . . . . . . . . . . . . . . . . 49 112 14. Security considerations . . . . . . . . . . . . . . . . . . . 49 113 15. Privacy considerations . . . . . . . . . . . . . . . . . . . 50 114 16. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 50 115 17. Changelog . . . . . . . . . . . . . . . . . . . . . . . . . . 51 116 18. References . . . . . . . . . . . . . . . . . . . . . . . . . 54 117 18.1. Normative References . . . . . . . . . . . . . . . . . . 54 118 18.2. Informative References . . . . . . . . . . . . . . . . . 55 119 18.3. URIs . . . . . . . . . . . . . . . . . . . . . . . . . . 57 120 Appendix A. CDDL . . . . . . . . . . . . . . . . . . . . . . . . 58 121 Appendix B. DNS Name compression example . . . . . . . . . . . . 68 122 B.1. NSD compression algorithm . . . . . . . . . . . . . . . . 69 123 B.2. Knot Authoritative compression algorithm . . . . . . . . 70 124 B.3. Observed differences . . . . . . . . . . . . . . . . . . 70 125 Appendix C. Comparison of Binary Formats . . . . . . . . . . . . 70 126 C.1. Comparison with full PCAP files . . . . . . . . . . . . . 73 127 C.2. Simple versus block coding . . . . . . . . . . . . . . . 74 128 C.3. Binary versus text formats . . . . . . . . . . . . . . . 74 129 C.4. Performance . . . . . . . . . . . . . . . . . . . . . . . 74 130 C.5. Conclusions . . . . . . . . . . . . . . . . . . . . . . . 75 131 C.6. Block size choice . . . . . . . . . . . . . . . . . . . . 75 132 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 76 134 1. Introduction 136 There has long been a need for server operators to collect DNS 137 queries and responses on authoritative and recursive name servers for 138 monitoring and analysis. This data is used in a number of ways 139 including traffic monitoring, analyzing network attacks and "day in 140 the life" (DITL) [ditl] analysis. 142 A wide variety of tools already exist that facilitate the collection 143 of DNS traffic data, such as DSC [dsc], packetq [packetq], dnscap 144 [dnscap] and dnstap [dnstap]. However, there is no standard exchange 145 format for large DNS packet captures. The PCAP [pcap] or PCAP-NG 146 [pcapng] formats are typically used in practice for packet captures, 147 but these file formats can contain a great deal of additional 148 information that is not directly pertinent to DNS traffic analysis 149 and thus unnecessarily increases the capture file size. Additionally 150 these tools and formats typically have no filter mechanism to 151 selectively record only certain fields at capture time, requiring 152 post-processing for anonymization or pseudonymization of data to 153 protect user privacy. 155 There has also been work on using text based formats to describe DNS 156 packets such as [I-D.daley-dnsxml], [RFC8427], but these are largely 157 aimed at producing convenient representations of single messages. 159 Many DNS operators may receive hundreds of thousands of queries per 160 second on a single name server instance so a mechanism to minimize 161 the storage and transmission size (and therefore upload overhead) of 162 the data collected is highly desirable. 164 The format described in this document, C-DNS (Compacted-DNS), 165 focusses on the problem of capturing and storing large packet capture 166 files of DNS traffic with the following goals in mind: 168 o Minimize the file size for storage and transmission. 170 o Minimize the overhead of producing the packet capture file and the 171 cost of any further (general purpose) compression of the file. 173 This document contains: 175 o A discussion of some common use cases in which DNS data is 176 collected, see Section 3. 178 o A discussion of the major design considerations in developing an 179 efficient data representation for collections of DNS messages, see 180 Section 4. 182 o A description of why CBOR [RFC7049] was chosen for this format, 183 see Section 5. 185 o A conceptual overview of the C-DNS format, see Section 6. 187 o The definition of the C-DNS format for the collection of DNS 188 messages, see Section 7. 190 o Notes on converting C-DNS data to PCAP format, see Section 9. 192 o Some high level implementation considerations for applications 193 designed to produce C-DNS, see Section 10. 195 2. Terminology 197 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 198 "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and 199 "OPTIONAL" in this document are to be interpreted as described in BCP 200 14 [RFC2119] [RFC8174] when, and only when, they appear in all 201 capitals, as shown here. 203 "Packet" refers to an individual IPv4 or IPv6 packet. Typically 204 packets are UDP datagrams, but may also be part of a TCP data stream. 205 "Message", unless otherwise qualified, refers to a DNS payload 206 extracted from a UDP datagram or a TCP data stream. 208 The parts of DNS messages are named as they are in [RFC1035]. 209 Specifically, the DNS message has five sections: Header, Question, 210 Answer, Authority, and Additional. 212 Pairs of DNS messages are called a Query and a Response. 214 3. Data collection use cases 216 From a purely server operator perspective, collecting full packet 217 captures of all packets going in or out of a name server provides the 218 most comprehensive picture of network activity. However, there are 219 several design choices or other limitations that are common to many 220 DNS installations and operators. 222 o DNS servers are hosted in a variety of situations: 224 * Self-hosted servers 226 * Third party hosting (including multiple third parties) 228 * Third party hardware (including multiple third parties) 230 o Data is collected under different conditions: 232 * On well-provisioned servers running in a steady state 234 * On heavily loaded servers 236 * On virtualized servers 238 * On servers that are under DoS attack 239 * On servers that are unwitting intermediaries in DoS attacks 241 o Traffic can be collected via a variety of mechanisms: 243 * Within the name server implementation itself 245 * On the same hardware as the name server itself 247 * Using a network tap on an adjacent host to listen to DNS 248 traffic 250 * Using port mirroring to listen from another host 252 o The capabilities of data collection (and upload) networks vary: 254 * Out-of-band networks with the same capacity as the in-band 255 network 257 * Out-of-band networks with less capacity than the in-band 258 network 260 * Everything being on the in-band network 262 Thus, there is a wide range of use cases from very limited data 263 collection environments (third party hardware, servers that are under 264 attack, packet capture on the name server itself and no out-of-band 265 network) to "limitless" environments (self hosted, well provisioned 266 servers, using a network tap or port mirroring with an out-of-band 267 networks with the same capacity as the in-band network). In the 268 former, it is infeasible to reliably collect full packet captures, 269 especially if the server is under attack. In the latter case, 270 collection of full packet captures may be reasonable. 272 As a result of these restrictions, the C-DNS data format is designed 273 with the most limited use case in mind such that: 275 o data collection will occur on the same hardware as the name server 276 itself 278 o collected data will be stored on the same hardware as the name 279 server itself, at least temporarily 281 o collected data being returned to some central analysis system will 282 use the same network interface as the DNS queries and responses 284 o there can be multiple third party servers involved 285 Because of these considerations, a major factor in the design of the 286 format is minimal storage size of the capture files. 288 Another significant consideration for any application that records 289 DNS traffic is that the running of the name server software and the 290 transmission of DNS queries and responses are the most important jobs 291 of a name server; capturing data is not. Any data collection system 292 co-located with the name server needs to be intelligent enough to 293 carefully manage its CPU, disk, memory and network utilization. This 294 leads to designing a format that requires a relatively low overhead 295 to produce and minimizes the requirement for further potentially 296 costly compression. 298 However, it is also essential that interoperability with less 299 restricted infrastructure is maintained. In particular, it is highly 300 desirable that the collection format should facilitate the re- 301 creation of common formats (such as PCAP) that are as close to the 302 original as is realistic given the restrictions above. 304 4. Design considerations 306 This section presents some of the major design considerations used in 307 the development of the C-DNS format. 309 1. The basic unit of data is a combined DNS Query and the associated 310 Response (a "Q/R data item"). The same structure will be used 311 for unmatched Queries and Responses. Queries without Responses 312 will be captured omitting the response data. Responses without 313 queries will be captured omitting the Query data (but using the 314 Question section from the response, if present, as an identifying 315 QNAME). 317 * Rationale: A Query and Response represents the basic level of 318 a client's interaction with the server. Also, combining the 319 Query and Response into one item often reduces storage 320 requirements due to commonality in the data of the two 321 messages. 323 In the context of generating a C-DNS file it is assumed that only 324 those DNS payloads which can be parsed to produce a well-formed 325 DNS message are stored in the C-DNS format and that all other 326 messages will be (optionally) recorded as malformed messages. 327 Parsing a well-formed message means as a minimum: 329 * The packet has a well-formed 12 byte DNS Header with a 330 recognised OPCODE. 332 * The section counts are consistent with the section contents. 334 * All of the resource records can be fully parsed. 336 2. All top level fields in each Q/R data item will be optional. 338 * Rationale: Different operators will have different 339 requirements for data to be available for analysis. Operators 340 with minimal requirements should not have to pay the cost of 341 recording full data, though this will limit the ability to 342 perform certain kinds of data analysis and also to reconstruct 343 packet captures. For example, omitting the resource records 344 from a Response will reduce the C-DNS file size; in principle 345 responses can be synthesized if there is enough context. 346 Operators may have different policies for collecting user data 347 and can choose to omit or anonymize certain fields at capture 348 time e.g. client address. 350 3. Multiple Q/R data items will be collected into blocks in the 351 format. Common data in a block will be abstracted and referenced 352 from individual Q/R data items by indexing. The maximum number 353 of Q/R data items in a block will be configurable. 355 * Rationale: This blocking and indexing provides a significant 356 reduction in the volume of file data generated. Although this 357 introduces complexity, it provides compression of the data 358 that makes use of knowledge of the DNS message structure. 360 * It is anticipated that the files produced can be subject to 361 further compression using general purpose compression tools. 362 Measurements show that blocking significantly reduces the CPU 363 required to perform such strong compression. See 364 Appendix C.2. 366 * Examples of commonality between DNS messages are that in most 367 cases the QUESTION RR is the same in the query and response, 368 and that there is a finite set of query signatures (based on a 369 subset of attributes). For many authoritative servers there 370 is very likely to be a finite set of responses that are 371 generated, of which a large number are NXDOMAIN. 373 4. Traffic metadata can optionally be included in each block. 374 Specifically, counts of some types of non-DNS packets (e.g. 375 ICMP, TCP resets) sent to the server may be of interest. 377 5. The wire format content of malformed DNS messages may optionally 378 be recorded. 380 * Rationale: Any structured capture format that does not capture 381 the DNS payload byte for byte will be limited to some extent 382 in that it cannot represent malformed DNS messages. Only 383 those messages that can be fully parsed and transformed into 384 the structured format can be fully represented. Note, 385 however, this can result in rather misleading statistics. For 386 example, a malformed query which cannot be represented in the 387 C-DNS format will lead to the (well formed) DNS responses with 388 error code FORMERR appearing as 'unmatched'. Therefore it can 389 greatly aid downstream analysis to have the wire format of the 390 malformed DNS messages available directly in the C-DNS file. 392 5. Choice of CBOR 394 This document presents a detailed format description using CBOR, the 395 Concise Binary Object Representation defined in [RFC7049]. 397 The choice of CBOR was made taking a number of factors into account. 399 o CBOR is a binary representation, and thus is economical in storage 400 space. 402 o Other binary representations were investigated, and whilst all had 403 attractive features, none had a significant advantage over CBOR. 404 See Appendix C for some discussion of this. 406 o CBOR is an IETF specification and familiar to IETF participants. 407 It is based on the now-common ideas of lists and objects, and thus 408 requires very little familiarization for those in the wider 409 industry. 411 o CBOR is a simple format, and can easily be implemented from 412 scratch if necessary. More complex formats require library 413 support which may present problems on unusual platforms. 415 o CBOR can also be easily converted to text formats such as JSON 416 ([RFC8259]) for debugging and other human inspection requirements. 418 o CBOR data schemas can be described using CDDL 419 [I-D.ietf-cbor-cddl]. 421 6. C-DNS format conceptual overview 423 The following figures show purely schematic representations of the 424 C-DNS format to convey the high-level structure of the C-DNS format. 425 Section 7 provides a detailed discussion of the CBOR representation 426 and individual elements. 428 Figure 1 shows the C-DNS format at the top level including the file 429 header and data blocks. The Query/Response data items, Address/Event 430 Count data items and Malformed Message data items link to various 431 Block tables. 433 +-------+ 434 + C-DNS | 435 +-------+--------------------------+ 436 | File type identifier | 437 +----------------------------------+ 438 | File preamble | 439 | +--------------------------------+ 440 | | Format version info | 441 | +--------------------------------+ 442 | | Block parameters | 443 +-+--------------------------------+ 444 | Block | 445 | +--------------------------------+ 446 | | Block preamble | 447 | +--------------------------------+ 448 | | Block statistics | 449 | +--------------------------------+ 450 | | Block tables | 451 | +--------------------------------+ 452 | | Query/Response data items | 453 | +--------------------------------+ 454 | | Address/Event Count data items | 455 | +--------------------------------+ 456 | | Malformed Message data items | 457 +-+--------------------------------+ 458 | Block | 459 | +--------------------------------+ 460 | | Block preamble | 461 | +--------------------------------+ 462 | | Block statistics | 463 | +--------------------------------+ 464 | | Block tables | 465 | +--------------------------------+ 466 | | Query/Response data items | 467 | +--------------------------------+ 468 | | Address/Event Count data items | 469 | +--------------------------------+ 470 | | Malformed Message data items | 471 +-+--------------------------------+ 472 | Further Blocks... | 473 +----------------------------------+ 475 Figure 1: The C-DNS format. 477 Figure 2 shows some more detailed relationships within each block, 478 specifically those between the Query/Response data item and the 479 relevant Block tables. 481 +----------------+ 482 | Query/Response | 483 +-------------------------+ 484 | Time offset | 485 +-------------------------+ +------------------+ 486 | Client address |------------>| IP address array | 487 +-------------------------+ +------------------+ 488 | Client port | 489 +-------------------------+ +------------------+ 490 | Transaction ID | +------>| Name/RDATA array |<------+ 491 +-------------------------+ | +------------------+ | 492 | Query signature |--+ | | 493 +-------------------------+ | | +-----------------+ | 494 | Client hoplimit (q) | +--)------>| Query Signature | | 495 +-------------------------+ | +-----------------+------+ | 496 | Response delay (r) | | | Server address | | 497 +-------------------------+ | +------------------------+ | 498 | Query name |--+--+ | Server port | | 499 +-------------------------+ | +------------------------+ | 500 | Query size (q) | | | Transport flags | | 501 +-------------------------+ | +------------------------+ | 502 | Response size (r) | | | QR type | | 503 +-------------------------+ | +------------------------+ | 504 | Response processing (r) | | | QR signature flags | | 505 | +-----------------------+ | +------------------------+ | 506 | | Bailiwick index |--+ | Query OPCODE (q) | | 507 | +-----------------------+ +------------------------+ | 508 | | Flags | | QR DNS flags | | 509 +-+-----------------------+ +------------------------+ | 510 | Extra query info (q) | | Query RCODE (q) | | 511 | +-----------------------+ +------------------------+ | 512 | | Question |--+---+ +--+-Query Class/Type (q) | | 513 | +-----------------------+ | | +------------------------+ | 514 | | Answer |--+ | | | Query QD count (q) | | 515 | +-----------------------+ | | | +------------------------+ | 516 | | Authority |--+ | | | Query AN count (q) | | 517 | +-----------------------+ | | | +------------------------+ | 518 | | Additional |--+ | | | Query NS count (q) | | 519 +-+-----------------------+ | | | +------------------------+ | 520 | Extra response info (r) | |-+ | | | Query EDNS version (q) | | 521 | +-----------------------+ | | | | +------------------------+ | 522 | | Answer |--+ | | | | EDNS UDP size (q) | | 523 | +-----------------------+ | | | | +------------------------+ | 524 | | Authority |--+ | | | | Query Opt RDATA (q) | | 525 | +-----------------------+ | | | | +------------------------+ | 526 | | Additional |--+ | | | | Response RCODE (r) | | 527 +-+-----------------------+ | | | +------------------------+ | 528 | | | | 529 | | | | 530 + -----------------------------+ | +----------+ | 531 | | | | 532 | + -----------------------------+ | | 533 | | +---------------+ +----------+ | | 534 | +->| Question list |->| Question | | | 535 | | array | | array | | | 536 | +---------------+ +----------+--+ | | 537 | | Name |--+------)------------------+ 538 | +-------------+ | | +------------+ 539 | | Class/type |--)---+--+->| Class/Type | 540 | +-------------+ | | | array | 541 | | | +------------+--+ 542 | | | | Class | 543 | +---------------+ +----------+ | | +---------------+ 544 +--->| RR list array |->| RR array | | | | Type | 545 +---------+-----+ +----------+--+ | | +---------------+ 546 | Name |--+ | 547 +-------------+ | 548 | Class/type |------+ 549 +-------------+ 551 Figure 2: The Query/Response data item and subsidiary tables. 553 In Figure 2 data items annotated (q) are only present when a query/ 554 response has a query, and those annotated (r) are only present when a 555 query/response response is present. 557 A C-DNS file begins with a file header containing a File Type 558 Identifier and a File Preamble. The File Preamble contains 559 information on the file Format Version and an array of Block 560 Parameters items (the contents of which include Collection and 561 Storage Parameters used for one or more blocks). 563 The file header is followed by a series of data Blocks. 565 A Block consists of a Block Preamble item, some Block Statistics for 566 the traffic stored within the Block and then various arrays of common 567 data collectively called the Block Tables. This is then followed by 568 an array of the Query/Response data items detailing the queries and 569 responses stored within the Block. The array of Query/Response data 570 items is in turn followed by the Address/Event Counts data items (an 571 array of per-client counts of particular IP events) and then 572 Malformed Message data items (an array of malformed messages that 573 stored in the Block). 575 The exact nature of the DNS data will affect what block size is the 576 best fit, however sample data for a root server indicated that block 577 sizes up to 10,000 Q/R data items give good results. See 578 Appendix C.6 for more details. 580 This design exploits data commonality and block based storage to 581 minimise the C-DNS file size. As a result C-DNS cannot be streamed 582 below the level of a block. 584 6.1. Block Parameters 586 The details of the Block Parameters items are not shown in the 587 diagrams but are discussed here for context. 589 An array of Block Parameters items is stored in the File Preamble 590 (with a minimum of one item at index 0); a Block Parameters item 591 consists of a collection of Storage and Collection Parameters that 592 applies to any given Block. An array is used in order to support use 593 cases such as wanting to merge C-DNS files from different sources. 594 The Block Preamble item then contains an optional index for the Block 595 Parameters item that applies for that Block; if not present the index 596 defaults to 0. Hence, in effect, a global Block Parameters item is 597 defined which can then be overridden per Block. 599 6.2. Storage Parameters 601 The Block Parameters item includes a Storage Parameters item - this 602 contains information about the specific data fields stored in the 603 C-DNS file. 605 These parameters include: 607 o The sub-second timing resolution used by the data. 609 o Information (hints) on which optional data are omitted. See 610 Section 6.2.1. 612 o Recorded OPCODES [opcodes] and RR types [rrtypes]. See 613 Section 6.2.2. 615 o Flags indicating, for example, whether the data is sampled or 616 anonymized. See Section 6.2.3 and Section 15. 618 o Client and server IPv4 and IPv6 address prefixes. See 619 Section 6.2.4 621 6.2.1. Optional data items 623 To enable implementations to store data to their precise requirements 624 in as space-efficient manner as possible, all fields in the following 625 arrays are optional: 627 o Query/Response 629 o Query Signature 631 o Malformed messages 633 In other words, an implementation can choose to omit any data item 634 that is not required for its use case. In addition, implementations 635 may be configured to not record all RRs, or only record messages with 636 certain OPCODES. 638 This does, however, mean that a consumer of a C-DNS file faces two 639 problems: 641 1. How can it quickly determine if a file definitely does not 642 contain the data items it requires to complete a particular task 643 (e.g. reconstructing query traffic or performing a specific piece 644 of data analysis)? 646 2. How can it determine if a data item is not present because it 647 was: 649 * explicitly not recorded or 651 * the data item was not available/present. 653 For example, capturing C-DNS data from within a nameserver 654 implementation makes it unlikely that the Client Hoplimit can be 655 recorded. Or, if there is no query ARCount recorded and no query OPT 656 RDATA [RFC6891] recorded, is that because no query contained an OPT 657 RR, or because that data was not stored? 659 The Storage Parameters therefore also contains a Storage Hints item 660 which specifies which items the encoder of the file omits from the 661 stored data and will therefore never be present. (This approach is 662 taken because a flag that indicated which items were included for 663 collection would not guarantee that the item was present, only that 664 it might be.) An implementation decoding that file can then use 665 these to quickly determine whether the input data is rich enough for 666 its needs. 668 6.2.2. Optional RRs and OPCODEs 670 Also included in the Storage Parameters are explicit arrays listing 671 the RR types and the OPCODEs to be recorded. These remove any 672 ambiguity over whether messages containing particular OPCODEs or are 673 not present because they did not occur, or because the implementation 674 is not configured to record them. 676 In the case of OPCODEs, for a message to be fully parsable, the 677 OPCODE must be known to the collecting implementation. Any message 678 with an OPCODE unknown to the collecting implementation cannot be 679 validated as correctly formed, and so must be treated as malformed. 680 Messages with OPCODES known to the recording application but not 681 listed in the Storage Parameters are discarded by the recording 682 application during C-DNS capture (regardless of whether they are 683 malformed or not). 685 In the case of RR records, each record in a message must be fully 686 parsable, including parsing the record RDATA, as otherwise the 687 message cannot be validated as correctly formed. Any RR record with 688 an RR type not known to the collecting implementation cannot be 689 validated as correctly formed, and so must be treated as malformed. 691 Once a message is correctly parsed, an implementation is free to 692 record only a subset of the RR records present. 694 6.2.3. Storage flags 696 The Storage Parameters contains flags that can be used to indicate 697 if: 699 o the data is anonymized, 701 o the data is produced from sample data, or 703 o names in the data have been normalized (converted to uniform 704 case). 706 The Storage Parameters also contains optional fields holding details 707 of the sampling method used and the anonymization method used. It is 708 RECOMMENDED these fields contain URIs [RFC3986] pointing to resources 709 describing the methods used. See Section 15 for further discussion 710 of anonymization and normalization. 712 6.2.4. IP Address storage 714 The format can store either full IP addresses or just IP prefixes, 715 the Storage Parameters contains fields to indicate if only IP 716 prefixes were stored. 718 If the IP address prefixes are absent, then full addresses are 719 stored. In this case the IP version can be directly inferred from 720 the stored address length and the fields "qr-transport-flags" in 721 QueryResponseSignature and "mm-transport-flags" in 722 MalformedMessageData (which contain the IP version bit) are optional. 724 If IP address prefixes are given, only the prefix bits of addresses 725 are stored. In this case the fields "qr-transport-flags" in 726 QueryResponseSignature and "mm-transport-flags" in 727 MalformedMessageData MUST be present, so that the IP version can be 728 determined. See Section 7.5.3.2 and Section 7.5.3.5. 730 As an example of storing only IP prefixes, if a client IPv6 prefix of 731 48 is specified, a client address of 2001:db8:85a3::8a2e:370:7334 732 will be stored as 0x20010db885a3, reducing address storage space 733 requirements. Similarly, if a client IPv4 prefix of 16 is specified, 734 a client address of 192.0.2.1 will be stored as 0xc000 (192.0). 736 7. C-DNS format detailed description 738 The CDDL definition for the C-DNS format is given in Appendix A. 740 7.1. Map quantities and indexes 742 All map keys are integers with values specified in the CDDL. String 743 keys would significantly bloat the file size. 745 All key values specified are positive integers under 24, so their 746 CBOR representation is a single byte. Positive integer values not 747 currently used as keys in a map are reserved for use in future 748 standard extensions. 750 Implementations may choose to add additional implementation-specific 751 entries to any map. Negative integer map keys are reserved for these 752 values. Key values from -1 to -24 also have a single byte CBOR 753 representation, so such implementation-specific extensions are not at 754 any space efficiency disadvantage. 756 An item described as an index is the index of the data item in the 757 referenced array. Indexes are 0-based. 759 7.2. Tabular representation 761 The following sections present the C-DNS specification in tabular 762 format with a detailed description of each item. 764 In all quantities that contain bit flags, bit 0 indicates the least 765 significant bit, i.e. flag "n" in quantity "q" is on if "(q & (1 << 766 n)) != 0". 768 For the sake of readability, all type and field names defined in the 769 CDDL definition are shown in double quotes. Type names are by 770 convention camel case (e.g. "BlockTable"), field names are lower- 771 case with hyphens (e.g. "block-tables"). 773 For the sake of brevity, the following conventions are used in the 774 tables: 776 o The column M marks whether items in a map are mandatory. 778 * X - Mandatory items. 780 * C - Conditionally mandatory item. Such items are usually 781 optional but may be mandatory in some configurations. 783 * If the column is empty, the item is optional. 785 o The column T gives the CBOR data type of the item. 787 * U - Unsigned integer 789 * I - Signed integer (i.e. CBOR unsigned or negative integer) 791 * B - Boolean 793 * S - Byte string 795 * T - Text string 797 * M - Map 799 * A - Array 801 In the case of maps and arrays, more information on the type of each 802 value, include the CDDL definition name if applicable, is given in 803 the description. 805 7.3. "File" 807 A C-DNS file has an outer structure "File", a map that contains the 808 following: 810 +---------------+---+---+-------------------------------------------+ 811 | Field | M | T | Description | 812 +---------------+---+---+-------------------------------------------+ 813 | file-type-id | X | T | String "C-DNS" identifying the file type. | 814 | | | | | 815 | file-preamble | X | M | Version and parameter information for the | 816 | | | | whole file. Map of type "FilePreamble", | 817 | | | | see Section 7.4. | 818 | | | | | 819 | file-blocks | X | A | Array of items of type "Block", see | 820 | | | | Section 7.5. The array may be empty if | 821 | | | | the file contains no data. | 822 +---------------+---+---+-------------------------------------------+ 824 7.4. "FilePreamble" 826 Information about data in the file. A map containing the following: 828 +----------------------+---+---+------------------------------------+ 829 | Field | M | T | Description | 830 +----------------------+---+---+------------------------------------+ 831 | major-format-version | X | U | Unsigned integer '1'. The major | 832 | | | | version of format used in file. | 833 | | | | See Section 8. | 834 | | | | | 835 | minor-format-version | X | U | Unsigned integer '0'. The minor | 836 | | | | version of format used in file. | 837 | | | | See Section 8. | 838 | | | | | 839 | private-version | | U | Version indicator available for | 840 | | | | private use by implementations. | 841 | | | | | 842 | block-parameters | X | A | Array of items of type | 843 | | | | "BlockParameters", see Section | 844 | | | | 7.4.1. The array must contain at | 845 | | | | least one entry. (The "block- | 846 | | | | parameters-index" item in each | 847 | | | | "BlockPreamble" indicates which | 848 | | | | array entry applies to that | 849 | | | | "Block".) | 850 +----------------------+---+---+------------------------------------+ 852 7.4.1. "BlockParameters" 854 Parameters relating to data storage and collection which apply to one 855 or more items of type "Block". A map containing the following: 857 +-----------------------+---+---+-----------------------------------+ 858 | Field | M | T | Description | 859 +-----------------------+---+---+-----------------------------------+ 860 | storage-parameters | X | M | Parameters relating to data | 861 | | | | storage in a "Block" item. Map | 862 | | | | of type "StorageParameters", see | 863 | | | | Section 7.4.1.1. | 864 | | | | | 865 | collection-parameters | | M | Parameters relating to collection | 866 | | | | of the data in a "Block" item. | 867 | | | | Map of type | 868 | | | | "CollectionParameters", see | 869 | | | | Section 7.4.2. | 870 +-----------------------+---+---+-----------------------------------+ 872 7.4.1.1. "StorageParameters" 874 Parameters relating to how data is stored in the items of type 875 "Block". A map containing the following: 877 +------------------+---+---+----------------------------------------+ 878 | Field | M | T | Description | 879 +------------------+---+---+----------------------------------------+ 880 | ticks-per-second | X | U | Sub-second timing is recorded in | 881 | | | | ticks. This specifies the number of | 882 | | | | ticks in a second. | 883 | | | | | 884 | max-block-items | X | U | The maximum number of items stored in | 885 | | | | any of the arrays in a "Block" item | 886 | | | | (Q/R items, address event counts or | 887 | | | | malformed messages). An indication to | 888 | | | | a decoder of the resources needed to | 889 | | | | process the file. | 890 | | | | | 891 | storage-hints | X | M | Collection of hints as to which fields | 892 | | | | are omitted in the arrays that have | 893 | | | | optional fields. Map of type | 894 | | | | "StorageHints", see Section 7.4.1.1.1. | 895 | | | | | 896 | opcodes | X | A | Array of OPCODES [opcodes] (unsigned | 897 | | | | integers, each in the range 0 to 15 | 898 | | | | inclusive) recorded by the collection | 899 | | | | implementation. See Section 6.2.2. | 900 | | | | | 901 | rr-types | X | A | Array of RR types [rrtypes] (unsigned | 902 | | | | integers, each in the range 0 to 65535 | 903 | | | | inclusive) recorded by the collection | 904 | | | | implementation. See Section 6.2.2. | 905 | | | | | 906 | storage-flags | | U | Bit flags indicating attributes of | 907 | | | | stored data. | 908 | | | | Bit 0. 1 if the data has been | 909 | | | | anonymized. | 910 | | | | Bit 1. 1 if the data is sampled data. | 911 | | | | Bit 2. 1 if the names have been | 912 | | | | normalized (converted to uniform | 913 | | | | case). | 914 | | | | | 915 | client-address | | U | IPv4 client address prefix length, in | 916 | -prefix-ipv4 | | | the range 1 to 32 inclusive. If | 917 | | | | specified, only the address prefix | 918 | | | | bits are stored. | 919 | | | | | 920 | client-address | | U | IPv6 client address prefix length, in | 921 | -prefix-ipv6 | | | the range 1 to 128 inclusive. If | 922 | | | | specified, only the address prefix | 923 | | | | bits are stored. | 924 | | | | | 925 | server-address | | U | IPv4 server address prefix length, in | 926 | -prefix-ipv4 | | | the range 1 to 32 inclusive. If | 927 | | | | specified, only the address prefix | 928 | | | | bits are stored. | 929 | | | | | 930 | server-address | | U | IPv6 server address prefix length, in | 931 | -prefix-ipv6 | | | the range 1 to 128 inclusive. If | 932 | | | | specified, only the address prefix | 933 | | | | bits are stored. | 934 | | | | | 935 | sampling-method | | T | Information on the sampling method | 936 | | | | used. See Section 6.2.3. | 937 | | | | | 938 | anonymization | | T | Information on the anonymization | 939 | -method | | | method used. See Section 6.2.3. | 940 +------------------+---+---+----------------------------------------+ 942 7.4.1.1.1. "StorageHints" 944 An indicator of which fields the collecting implementation omits in 945 the maps with optional fields. A map containing the following: 947 +------------------+---+---+----------------------------------------+ 948 | Field | M | T | Description | 949 +------------------+---+---+----------------------------------------+ 950 | query-response | X | U | Hints indicating which "QueryResponse" | 951 | -hints | | | fields are candidates for capture or | 952 | | | | omitted, see section Section 7.6. If a | 953 | | | | bit is unset, the field is omitted | 954 | | | | from the capture. | 955 | | | | Bit 0. time-offset | 956 | | | | Bit 1. client-address-index | 957 | | | | Bit 2. client-port | 958 | | | | Bit 3. transaction-id | 959 | | | | Bit 4. qr-signature-index | 960 | | | | Bit 5. client-hoplimit | 961 | | | | Bit 6. response-delay | 962 | | | | Bit 7. query-name-index | 963 | | | | Bit 8. query-size | 964 | | | | Bit 9. response-size | 965 | | | | Bit 10. response-processing-data | 966 | | | | Bit 11. query-question-sections | 967 | | | | Bit 12. query-answer-sections | 968 | | | | Bit 13. query-authority-sections | 969 | | | | Bit 14. query-additional-sections | 970 | | | | Bit 15. response-answer-sections | 971 | | | | Bit 16. response-authority-sections | 972 | | | | Bit 17. response-additional-sections | 973 | | | | | 974 | query-response | X | U | Hints indicating which | 975 | -signature-hints | | | "QueryResponseSignature" fields are | 976 | | | | candidates for capture or omitted, see | 977 | | | | section Section 7.5.3.2. If a bit is | 978 | | | | unset, the field is omitted from the | 979 | | | | capture. | 980 | | | | Bit 0. server-address | 981 | | | | Bit 1. server-port | 982 | | | | Bit 2. qr-transport-flags | 983 | | | | Bit 3. qr-type | 984 | | | | Bit 4. qr-sig-flags | 985 | | | | Bit 5. query-opcode | 986 | | | | Bit 6. dns-flags | 987 | | | | Bit 7. query-rcode | 988 | | | | Bit 8. query-class-type | 989 | | | | Bit 9. query-qdcount | 990 | | | | Bit 10. query-ancount | 991 | | | | Bit 11. query-nscount | 992 | | | | Bit 12. query-arcount | 993 | | | | Bit 13. query-edns-version | 994 | | | | Bit 14. query-udp-size | 995 | | | | Bit 15. query-opt-rdata | 996 | | | | Bit 16. response-rcode | 997 | | | | | 998 | rr-hints | X | U | Hints indicating which optional "RR" | 999 | | | | fields are candidates for capture or | 1000 | | | | omitted, see Section 7.5.3.4. If a bit | 1001 | | | | is unset, the field is omitted from | 1002 | | | | the capture. | 1003 | | | | Bit 0. ttl | 1004 | | | | Bit 1. rdata-index | 1005 | other-data-hints | X | U | Hints indicating which other data | 1006 | | | | types are omitted. If a bit is unset, | 1007 | | | | the the data type is omitted from the | 1008 | | | | capture. | 1009 | | | | Bit 0. malformed-messages | 1010 | | | | Bit 1. address-event-counts | 1011 +------------------+---+---+----------------------------------------+ 1013 7.4.2. "CollectionParameters" 1015 Parameters providing information to how data in the file was 1016 collected (applicable for some, but not all collection environments). 1017 The values are informational only and serve as hints to downstream 1018 analysers as to the configuration of a collecting implementation. 1019 They can provide context when interpreting what data is present/ 1020 absent from the capture but cannot necessarily be validated against 1021 the data captured. 1023 These parameters have no default. If they do not appear, nothing can 1024 be inferred about their value. 1026 A map containing the following items: 1028 +------------------+---+---+----------------------------------------+ 1029 | Field | M | T | Description | 1030 +------------------+---+---+----------------------------------------+ 1031 | query-timeout | | U | To be matched with a query, a response | 1032 | | | | must arrive within this number of | 1033 | | | | seconds. | 1034 | | | | | 1035 | skew-timeout | | U | The network stack may report a | 1036 | | | | response before the corresponding | 1037 | | | | query. A response is not considered to | 1038 | | | | be missing a query until after this | 1039 | | | | many micro-seconds. | 1040 | | | | | 1041 | snaplen | | U | Collect up to this many bytes per | 1042 | | | | packet. | 1043 | | | | | 1044 | promisc | | B | "true" if promiscuous mode | 1045 | | | | [pcap-options] was enabled on the | 1046 | | | | interface, "false" otherwise. | 1047 | | | | | 1048 | interfaces | | A | Array of identifiers (of type text | 1049 | | | | string) of the interfaces used for | 1050 | | | | collection. | 1051 | | | | | 1052 | server-addresses | | A | Array of server collection IP | 1053 | | | | addresses (of type byte string). Hint | 1054 | | | | for downstream analysers; does not | 1055 | | | | affect collection. | 1056 | | | | | 1057 | vlan-ids | | A | Array of identifiers (of type unsigned | 1058 | | | | integer, each in the range 1 to 4094 | 1059 | | | | inclusive) of VLANs [IEEE802.1Q] | 1060 | | | | selected for collection. VLAN IDs are | 1061 | | | | unique only within an administrative | 1062 | | | | domain. | 1063 | | | | | 1064 | filter | | T | "tcpdump" [pcap-filter] style filter | 1065 | | | | for input. | 1066 | | | | | 1067 | generator-id | | T | Implementation specific human-readable | 1068 | | | | string identifying the collection | 1069 | | | | method. | 1070 | | | | | 1071 | host-id | | T | String identifying the collecting | 1072 | | | | host. Empty if converting an existing | 1073 | | | | packet capture file. | 1074 +------------------+---+---+----------------------------------------+ 1076 7.5. "Block" 1078 Container for data with common collection and storage parameters. A 1079 map containing the following: 1081 +--------------------+---+---+--------------------------------------+ 1082 | Field | M | T | Description | 1083 +--------------------+---+---+--------------------------------------+ 1084 | block-preamble | X | M | Overall information for the "Block" | 1085 | | | | item. Map of type "BlockPreamble", | 1086 | | | | see Section 7.5.1. | 1087 | | | | | 1088 | block-statistics | | M | Statistics about the "Block" item. | 1089 | | | | Map of type "BlockStatistics", see | 1090 | | | | Section 7.5.2. | 1091 | | | | | 1092 | block-tables | | M | The arrays containing data | 1093 | | | | referenced by individual | 1094 | | | | "QueryResponse" or | 1095 | | | | "MalformedMessage" items. Map of | 1096 | | | | type "BlockTables", see Section | 1097 | | | | 7.5.3. | 1098 | | | | | 1099 | query-responses | | A | Details of individual DNS Q/R data | 1100 | | | | items. Array of items of type | 1101 | | | | "QueryResponse", see Section 7.6. If | 1102 | | | | present, the array must not be | 1103 | | | | empty. | 1104 | | | | | 1105 | address-event | | A | Per client counts of ICMP messages | 1106 | -counts | | | and TCP resets. Array of items of | 1107 | | | | type "AddressEventCount", see | 1108 | | | | Section 7.7. If present, the array | 1109 | | | | must not be empty. | 1110 | | | | | 1111 | malformed-messages | | A | Details of malformed DNS messages. | 1112 | | | | Array of items of type | 1113 | | | | "MalformedMessage", see Section 7.8. | 1114 | | | | If present, the array must not be | 1115 | | | | empty. | 1116 +--------------------+---+---+--------------------------------------+ 1118 7.5.1. "BlockPreamble" 1120 Overall information for a "Block" item. A map containing the 1121 following: 1123 +------------------+---+---+----------------------------------------+ 1124 | Field | M | T | Description | 1125 +------------------+---+---+----------------------------------------+ 1126 | earliest-time | C | A | A timestamp (2 unsigned integers, | 1127 | | | | "Timestamp") for the earliest record | 1128 | | | | in the "Block" item. The first integer | 1129 | | | | is the number of seconds since the | 1130 | | | | POSIX epoch [posix-time] ("time_t"), | 1131 | | | | excluding leap seconds. The second | 1132 | | | | integer is the number of ticks (see | 1133 | | | | Section 7.4.1.1) since the start of | 1134 | | | | the second. This field is mandatory | 1135 | | | | unless all block items containing a | 1136 | | | | time offset from the start of the | 1137 | | | | block also omit that time offset. | 1138 | | | | | 1139 | block-parameters | | U | The index of the item in the "block- | 1140 | -index | | | parameters" array (in the "file- | 1141 | | | | premable" item) applicable to this | 1142 | | | | block. If not present, index 0 is | 1143 | | | | used. See Section 7.4.1. | 1144 +------------------+---+---+----------------------------------------+ 1146 7.5.2. "BlockStatistics" 1148 Basic statistical information about a "Block" item. A map containing 1149 the following: 1151 +---------------------+---+---+-------------------------------------+ 1152 | Field | M | T | Description | 1153 +---------------------+---+---+-------------------------------------+ 1154 | processed-messages | | U | Total number of DNS messages | 1155 | | | | processed from the input traffic | 1156 | | | | stream during collection of data in | 1157 | | | | this "Block" item. | 1158 | | | | | 1159 | qr-data-items | | U | Total number of Q/R data items in | 1160 | | | | this "Block" item. | 1161 | | | | | 1162 | unmatched-queries | | U | Number of unmatched queries in this | 1163 | | | | "Block" item. | 1164 | | | | | 1165 | unmatched-responses | | U | Number of unmatched responses in | 1166 | | | | this "Block" item. | 1167 | | | | | 1168 | discarded-opcode | | U | Number of DNS messages processed | 1169 | | | | from the input traffic stream | 1170 | | | | during collection of data in this | 1171 | | | | "Block" item but not recorded | 1172 | | | | because their OPCODE is not in the | 1173 | | | | list to be collected. | 1174 | | | | | 1175 | malformed-items | | U | Number of malformed messages found | 1176 | | | | in input for this "Block" item. | 1177 +---------------------+---+---+-------------------------------------+ 1179 7.5.3. "BlockTables" 1181 Map of arrays containing data referenced by individual 1182 "QueryResponse" or "MalformedMessage" items in this "Block". Each 1183 element is an array which, if present, must not be empty. 1185 An item in the "qlist" array contains indexes to values in the "qrr" 1186 array. Therefore, if "qlist" is present, "qrr" must also be present. 1187 Similarly, if "rrlist" is present, "rr" must also be present. 1189 The map contains the following items: 1191 +-------------------+---+---+---------------------------------------+ 1192 | Field | M | T | Description | 1193 +-------------------+---+---+---------------------------------------+ 1194 | ip-address | | A | Array of IP addresses, in network | 1195 | | | | byte order (of type byte string). If | 1196 | | | | client or server address prefixes are | 1197 | | | | set, only the address prefix bits are | 1198 | | | | stored. Each string is therefore up | 1199 | | | | to 4 bytes long for an IPv4 address, | 1200 | | | | or up to 16 bytes long for an IPv6 | 1201 | | | | address. See Section 7.4.1.1. | 1202 | | | | | 1203 | classtype | | A | Array of RR class and type | 1204 | | | | information. Type is "ClassType", see | 1205 | | | | Section 7.5.3.1. | 1206 | | | | | 1207 | name-rdata | | A | Array where each entry is the | 1208 | | | | contents of a single NAME or RDATA in | 1209 | | | | wire format (of type byte string). | 1210 | | | | Note that NAMEs, and labels within | 1211 | | | | RDATA contents, are full domain names | 1212 | | | | or labels; no [RFC1035] name | 1213 | | | | compression is used on the individual | 1214 | | | | names/labels within the format. | 1215 | | | | | 1216 | qr-sig | | A | Array Q/R data item signatures. Type | 1217 | | | | is "QueryResponseSignature", see | 1218 | | | | Section 7.5.3.2. | 1219 | | | | | 1220 | qlist | | A | Array of type "QuestionList". A | 1221 | | | | "QuestionList" is an array of | 1222 | | | | unsigned integers, indexes to | 1223 | | | | "Question" items in the "qrr" array. | 1224 | | | | | 1225 | qrr | | A | Array of type "Question". Each entry | 1226 | | | | is the contents of a single question, | 1227 | | | | where a question is the second or | 1228 | | | | subsequent question in a query. See | 1229 | | | | Section 7.5.3.3. | 1230 | | | | | 1231 | rrlist | | A | Array of type "RRList". An "RRList" | 1232 | | | | is an array of unsigned integers, | 1233 | | | | indexes to "RR" items in the "rr" | 1234 | | | | array. | 1235 | | | | | 1236 | rr | | A | Array of type "RR". Each entry is the | 1237 | | | | contents of a single RR. See Section | 1238 | | | | 7.5.3.4. | 1239 | | | | | 1240 | malformed-message | | A | Array of the contents of malformed | 1241 | -data | | | messages. Array of type | 1242 | | | | "MalformedMessageData", see Section | 1243 | | | | 7.5.3.5. | 1244 +-------------------+---+---+---------------------------------------+ 1246 7.5.3.1. "ClassType" 1248 RR class and type information. A map containing the following: 1250 +-------+---+---+--------------------------+ 1251 | Field | M | T | Description | 1252 +-------+---+---+--------------------------+ 1253 | type | X | U | TYPE value [rrtypes]. | 1254 | | | | | 1255 | class | X | U | CLASS value [rrclasses]. | 1256 +-------+---+---+--------------------------+ 1258 7.5.3.2. "QueryResponseSignature" 1260 Elements of a Q/R data item that are often common between multiple 1261 individual Q/R data items. A map containing the following: 1263 +--------------------+---+---+--------------------------------------+ 1264 | Field | M | T | Description | 1265 +--------------------+---+---+--------------------------------------+ 1266 | server-address | | U | The index in the item in the "ip- | 1267 | -index | | | address" array of the server IP | 1268 | | | | address. See Section 7.5.3. | 1269 | | | | | 1270 | server-port | | U | The server port. | 1271 | | | | | 1272 | qr-transport-flags | C | U | Bit flags describing the transport | 1273 | | | | used to service the query. Same | 1274 | | | | definition as "mm-transport-flags" | 1275 | | | | in Section 7.5.3.5, with an | 1276 | | | | additional indicator for trailing | 1277 | | | | bytes, see Appendix A. | 1278 | | | | Bit 0. IP version. 0 if IPv4, 1 if | 1279 | | | | IPv6. See Section 6.2.4. | 1280 | | | | Bit 1-4. Transport. 4 bit unsigned | 1281 | | | | value where 0 = UDP, 1 = TCP, 2 = | 1282 | | | | TLS, 3 = DTLS [RFC7858], 4 = DoH | 1283 | | | | [RFC8484]. Values 5-15 are reserved | 1284 | | | | for future use. | 1285 | | | | Bit 5. 1 if trailing bytes in query | 1286 | | | | packet. See Section 11.2. | 1287 | | | | | 1288 | qr-type | | U | Type of Query/Response transaction. | 1289 | | | | 0 = Stub. A query from a stub | 1290 | | | | resolver. | 1291 | | | | 1 = Client. An incoming query to a | 1292 | | | | recursive resolver. | 1293 | | | | 2 = Resolver. A query sent from a | 1294 | | | | recursive resolver to an authorative | 1295 | | | | resolver. | 1296 | | | | 3 = Authorative. A query to an | 1297 | | | | authorative resolver. | 1298 | | | | 4 = Forwarder. A query sent from a | 1299 | | | | recursive resolver to an upstream | 1300 | | | | recursive resolver. | 1301 | | | | 5 = Tool. A query sent to a server | 1302 | | | | by a server tool. | 1303 | | | | | 1304 | qr-sig-flags | | U | Bit flags explicitly indicating | 1305 | | | | attributes of the message pair | 1306 | | | | represented by this Q/R data item | 1307 | | | | (not all attributes may be recorded | 1308 | | | | or deducible). | 1309 | | | | Bit 0. 1 if a Query was present. | 1310 | | | | Bit 1. 1 if a Response was present. | 1311 | | | | Bit 2. 1 if a Query was present and | 1312 | | | | it had an OPT Resource Record. | 1313 | | | | Bit 3. 1 if a Response was present | 1314 | | | | and it had an OPT Resource Record. | 1315 | | | | Bit 4. 1 if a Query was present but | 1316 | | | | had no Question. | 1317 | | | | Bit 5. 1 if a Response was present | 1318 | | | | but had no Question (only one query- | 1319 | | | | name-index is stored per Q/R item). | 1320 | | | | | 1321 | query-opcode | | U | Query OPCODE. | 1322 | | | | | 1323 | qr-dns-flags | | U | Bit flags with values from the Query | 1324 | | | | and Response DNS flags. Flag values | 1325 | | | | are 0 if the Query or Response is | 1326 | | | | not present. | 1327 | | | | Bit 0. Query Checking Disabled (CD). | 1328 | | | | Bit 1. Query Authenticated Data | 1329 | | | | (AD). | 1330 | | | | Bit 2. Query reserved (Z). | 1331 | | | | Bit 3. Query Recursion Available | 1332 | | | | (RA). | 1333 | | | | Bit 4. Query Recursion Desired (RD). | 1334 | | | | Bit 5. Query TrunCation (TC). | 1335 | | | | Bit 6. Query Authoritative Answer | 1336 | | | | (AA). | 1337 | | | | Bit 7. Query DNSSEC answer OK (DO). | 1338 | | | | Bit 8. Response Checking Disabled | 1339 | | | | (CD). | 1340 | | | | Bit 9. Response Authenticated Data | 1341 | | | | (AD). | 1342 | | | | Bit 10. Response reserved (Z). | 1343 | | | | Bit 11. Response Recursion Available | 1344 | | | | (RA). | 1345 | | | | Bit 12. Response Recursion Desired | 1346 | | | | (RD). | 1347 | | | | Bit 13. Response TrunCation (TC). | 1348 | | | | Bit 14. Response Authoritative | 1349 | | | | Answer (AA). | 1350 | | | | | 1351 | query-rcode | | U | Query RCODE. If the Query contains | 1352 | | | | OPT [RFC6891], this value | 1353 | | | | incorporates any | 1354 | | | | EXTENDED_RCODE_VALUE [rcodes]. | 1355 | | | | | 1356 | query-classtype | | U | The index to the item in the the | 1357 | -index | | | "classtype" array of the CLASS and | 1358 | | | | TYPE of the first Question. See | 1359 | | | | Section 7.5.3. | 1360 | | | | | 1361 | query-qd-count | | U | The QDCOUNT in the Query, or | 1362 | | | | Response if no Query present. | 1363 | | | | | 1364 | query-an-count | | U | Query ANCOUNT. | 1365 | | | | | 1366 | query-ns-count | | U | Query NSCOUNT. | 1367 | | | | | 1368 | query-ar-count | | U | Query ARCOUNT. | 1369 | | | | | 1370 | edns-version | | U | The Query EDNS version. | 1371 | | | | | 1372 | udp-buf-size | | U | The Query EDNS sender's UDP payload | 1373 | | | | size. | 1374 | | | | | 1375 | opt-rdata-index | | U | The index in the "name-rdata" array | 1376 | | | | of the OPT RDATA. See Section 7.5.3. | 1377 | | | | | 1378 | response-rcode | | U | Response RCODE. If the Response | 1379 | | | | contains OPT [RFC6891], this value | 1380 | | | | incorporates any | 1381 | | | | EXTENDED_RCODE_VALUE [rcodes]. | 1382 +--------------------+---+---+--------------------------------------+ 1384 7.5.3.3. "Question" 1386 Details on individual Questions in a Question section. A map 1387 containing the following: 1389 +-----------------+---+---+-----------------------------------------+ 1390 | Field | M | T | Description | 1391 +-----------------+---+---+-----------------------------------------+ 1392 | name-index | X | U | The index in the "name-rdata" array of | 1393 | | | | the QNAME. See Section 7.5.3. | 1394 | | | | | 1395 | classtype-index | X | U | The index in the "classtype" array of | 1396 | | | | the CLASS and TYPE of the Question. See | 1397 | | | | Section 7.5.3. | 1398 +-----------------+---+---+-----------------------------------------+ 1400 7.5.3.4. "RR" 1402 Details on individual Resource Records in RR sections. A map 1403 containing the following: 1405 +-----------------+---+---+-----------------------------------------+ 1406 | Field | M | T | Description | 1407 +-----------------+---+---+-----------------------------------------+ 1408 | name-index | X | U | The index in the "name-rdata" array of | 1409 | | | | the NAME. See Section 7.5.3. | 1410 | | | | | 1411 | classtype-index | X | U | The index in the "classtype" array of | 1412 | | | | the CLASS and TYPE of the RR. See | 1413 | | | | Section 7.5.3. | 1414 | | | | | 1415 | ttl | | U | The RR Time to Live. | 1416 | | | | | 1417 | rdata-index | | U | The index in the "name-rdata" array of | 1418 | | | | the RR RDATA. See Section 7.5.3. | 1419 +-----------------+---+---+-----------------------------------------+ 1421 7.5.3.5. "MalformedMessageData" 1423 Details on malformed message items in this "Block" item. A map 1424 containing the following: 1426 +--------------------+---+---+--------------------------------------+ 1427 | Field | M | T | Description | 1428 +--------------------+---+---+--------------------------------------+ 1429 | server-address | | U | The index in the "ip-address" array | 1430 | -index | | | of the server IP address. See | 1431 | | | | Section 7.5.3. | 1432 | | | | | 1433 | server-port | | U | The server port. | 1434 | | | | | 1435 | mm-transport-flags | C | U | Bit flags describing the transport | 1436 | | | | used to service the query, see | 1437 | | | | Section 6.2.4. | 1438 | | | | Bit 0. IP version. 0 if IPv4, 1 if | 1439 | | | | IPv6 | 1440 | | | | Bit 1-4. Transport. 4 bit unsigned | 1441 | | | | value where 0 = UDP, 1 = TCP, 2 = | 1442 | | | | TLS, 3 = DTLS [RFC7858], 4 = DoH | 1443 | | | | [RFC8484]. Values 5-15 are reserved | 1444 | | | | for future use. | 1445 | | | | | 1446 | mm-payload | | S | The payload (raw bytes) of the DNS | 1447 | | | | message. | 1448 +--------------------+---+---+--------------------------------------+ 1450 7.6. "QueryResponse" 1452 Details on individual Q/R data items. 1454 Note that there is no requirement that the elements of the "query- 1455 responses" array are presented in strict chronological order. 1457 A map containing the following items: 1459 +----------------------+---+---+------------------------------------+ 1460 | Field | M | T | Description | 1461 +----------------------+---+---+------------------------------------+ 1462 | time-offset | | U | Q/R timestamp as an offset in | 1463 | | | | ticks (see Section 7.4.1.1) from | 1464 | | | | "earliest-time". The timestamp is | 1465 | | | | the timestamp of the Query, or the | 1466 | | | | Response if there is no Query. | 1467 | | | | | 1468 | client-address-index | | U | The index in the "ip-address" | 1469 | | | | array of the client IP address. | 1470 | | | | See Section 7.5.3. | 1471 | | | | | 1472 | client-port | | U | The client port. | 1473 | | | | | 1474 | transaction-id | | U | DNS transaction identifier. | 1475 | | | | | 1476 | qr-signature-index | | U | The index in the "qr-sig" array of | 1477 | | | | the "QueryResponseSignature" item. | 1478 | | | | See Section 7.5.3. | 1479 | | | | | 1480 | client-hoplimit | | U | The IPv4 TTL or IPv6 Hoplimit from | 1481 | | | | the Query packet. | 1482 | | | | | 1483 | response-delay | | I | The time difference between Query | 1484 | | | | and Response, in ticks (see | 1485 | | | | Section 7.4.1.1). Only present if | 1486 | | | | there is a query and a response. | 1487 | | | | The delay can be negative if the | 1488 | | | | network stack/capture library | 1489 | | | | returns packets out of order. | 1490 | | | | | 1491 | query-name-index | | U | The index in the "name-rdata" | 1492 | | | | array of the item containing the | 1493 | | | | QNAME for the first Question. See | 1494 | | | | Section 7.5.3. | 1495 | | | | | 1496 | query-size | | U | DNS query message size (see | 1497 | | | | below). | 1498 | | | | | 1499 | response-size | | U | DNS response message size (see | 1500 | | | | below). | 1501 | | | | | 1502 | response-processing | | M | Data on response processing. Map | 1503 | -data | | | of type "ResponseProcessingData", | 1504 | | | | see Section 7.6.1. | 1505 | | | | | 1506 | query-extended | | M | Extended Query data. Map of type | 1507 | | | | "QueryResponseExtended", see | 1508 | | | | Section 7.6.2. | 1509 | | | | | 1510 | response-extended | | M | Extended Response data. Map of | 1511 | | | | type "QueryResponseExtended", see | 1512 | | | | Section 7.6.2. | 1513 +----------------------+---+---+------------------------------------+ 1515 The "query-size" and "response-size" fields hold the DNS message 1516 size. For UDP this is the size of the UDP payload that contained the 1517 DNS message. For TCP it is the size of the DNS message as specified 1518 in the two-byte message length header. Trailing bytes in UDP queries 1519 are routinely observed in traffic to authoritative servers and this 1520 value allows a calculation of how many trailing bytes were present. 1522 7.6.1. "ResponseProcessingData" 1524 Information on the server processing that produced the response. A 1525 map containing the following: 1527 +------------------+---+---+----------------------------------------+ 1528 | Field | M | T | Description | 1529 +------------------+---+---+----------------------------------------+ 1530 | bailiwick-index | | U | The index in the "name-rdata" array of | 1531 | | | | the owner name for the response | 1532 | | | | bailiwick. See Section 7.5.3. | 1533 | | | | | 1534 | processing-flags | | U | Flags relating to response processing. | 1535 | | | | Bit 0. 1 if the response came from | 1536 | | | | cache. | 1537 +------------------+---+---+----------------------------------------+ 1539 7.6.2. "QueryResponseExtended" 1541 Extended data on the Q/R data item. 1543 Each item in the map is present only if collection of the relevant 1544 details is configured. 1546 A map containing the following items: 1548 +------------------+---+---+----------------------------------------+ 1549 | Field | M | T | Description | 1550 +------------------+---+---+----------------------------------------+ 1551 | question-index | | U | The index in the "qlist" array of the | 1552 | | | | entry listing any second and | 1553 | | | | subsequent Questions in the Question | 1554 | | | | section for the Query or Response. See | 1555 | | | | Section 7.5.3. | 1556 | | | | | 1557 | answer-index | | U | The index in the "rrlist" array of the | 1558 | | | | entry listing the Answer Resource | 1559 | | | | Record sections for the Query or | 1560 | | | | Response. See Section 7.5.3. | 1561 | | | | | 1562 | authority-index | | U | The index in the "rrlist" array of the | 1563 | | | | entry listing the Authority Resource | 1564 | | | | Record sections for the Query or | 1565 | | | | Response. See Section 7.5.3. | 1566 | | | | | 1567 | additional-index | | U | The index in the "rrlist" array of the | 1568 | | | | entry listing the Additional Resource | 1569 | | | | Record sections for the Query or | 1570 | | | | Response. See Section 7.5.3. Note that | 1571 | | | | Query OPT RR data can be optionally | 1572 | | | | stored in the QuerySignature. | 1573 +------------------+---+---+----------------------------------------+ 1575 7.7. "AddressEventCount" 1577 Counts of various IP related events relating to traffic with 1578 individual client addresses. A map containing the following: 1580 +------------------+---+---+----------------------------------------+ 1581 | Field | M | T | Description | 1582 +------------------+---+---+----------------------------------------+ 1583 | ae-type | X | U | The type of event. The following | 1584 | | | | events types are currently defined: | 1585 | | | | 0. TCP reset. | 1586 | | | | 1. ICMP time exceeded. | 1587 | | | | 2. ICMP destination unreachable. | 1588 | | | | 3. ICMPv6 time exceeded. | 1589 | | | | 4. ICMPv6 destination unreachable. | 1590 | | | | 5. ICMPv6 packet too big. | 1591 | | | | | 1592 | ae-code | | U | A code relating to the event. For ICMP | 1593 | | | | or ICMPv6 events, this MUST be the | 1594 | | | | ICMP [RFC0792] or ICMPv6 [RFC4443] | 1595 | | | | code. For other events the contents | 1596 | | | | are undefined. | 1597 | | | | | 1598 | ae-address-index | X | U | The index in the "ip-address" array of | 1599 | | | | the client address. See Section 7.5.3. | 1600 | | | | | 1601 | ae-count | X | U | The number of occurrences of this | 1602 | | | | event during the block collection | 1603 | | | | period. | 1604 +------------------+---+---+----------------------------------------+ 1606 7.8. "MalformedMessage" 1608 Details of malformed messages. A map containing the following: 1610 +----------------------+---+---+------------------------------------+ 1611 | Field | M | T | Description | 1612 +----------------------+---+---+------------------------------------+ 1613 | time-offset | | U | Message timestamp as an offset in | 1614 | | | | ticks (see Section 7.4.1.1) from | 1615 | | | | "earliest-time". | 1616 | | | | | 1617 | client-address-index | | U | The index in the "ip-address" | 1618 | | | | array of the client IP address. | 1619 | | | | See Section 7.5.3. | 1620 | | | | | 1621 | client-port | | U | The client port. | 1622 | | | | | 1623 | message-data-index | | U | The index in the "malformed- | 1624 | | | | message-data" array of the message | 1625 | | | | data for this message. See Section | 1626 | | | | 7.5.3. | 1627 +----------------------+---+---+------------------------------------+ 1629 8. Versioning 1631 The C-DNS file preamble includes a file format version; a major and 1632 minor version number are required fields. The document defines 1633 version 1.0 of the C-DNS specification. This section describes the 1634 intended use of these version numbers in future specifications. 1636 It is noted that version 1.0 includes many optional fields and 1637 therefore consumers of version 1.0 should be inherently robust to 1638 parsing files with variable data content. 1640 Within a major version, a new minor version MUST be a strict superset 1641 of the previous minor version, with no semantic changes to existing 1642 fields. New keys MAY be added to existing maps, and new maps MAY be 1643 added. A consumer capable of reading a particular major.minor 1644 version MUST also be capable of reading all previous minor versions 1645 of the same major version. It SHOULD also be capable of parsing all 1646 subsequent minor versions ignoring any keys or maps that it does not 1647 recognise. 1649 A new major version indicates changes to the format that are not 1650 backwards compatible with previous major versions. A consumer 1651 capable of only reading a particular major version (greater than 1) 1652 is not required to and has no expectation to be capable of reading a 1653 previous major version. 1655 9. C-DNS to PCAP 1657 It is possible to re-construct PCAP files from the C-DNS format in a 1658 lossy fashion. Some of the issues with reconstructing both the DNS 1659 payload and the full packet stream are outlined here. 1661 The reconstruction depends on whether or not all the optional 1662 sections of both the query and response were captured in the C-DNS 1663 file. Clearly, if they were not all captured, the reconstruction 1664 will be imperfect. 1666 Even if all sections of the response were captured, one cannot 1667 reconstruct the DNS response payload exactly due to the fact that 1668 some DNS names in the message on the wire may have been compressed. 1669 Section 9.1 discusses this is more detail. 1671 Some transport information is not captured in the C-DNS format. For 1672 example, the following aspects of the original packet stream cannot 1673 be re-constructed from the C-DNS format: 1675 o IP fragmentation 1676 o TCP stream information: 1678 * Multiple DNS messages may have been sent in a single TCP 1679 segment 1681 * A DNS payload may have been split across multiple TCP segments 1683 * Multiple DNS messages may have been sent on a single TCP 1684 session 1686 o Malformed DNS messages if the wire format is not recorded 1688 o Any Non-DNS messages that were in the original packet stream e.g. 1689 ICMP 1691 Simple assumptions can be made on the reconstruction: fragmented and 1692 DNS-over-TCP messages can be reconstructed into single packets and a 1693 single TCP session can be constructed for each TCP packet. 1695 Additionally, if malformed messages and Non-DNS packets are captured 1696 separately, they can be merged with packet captures reconstructed 1697 from C-DNS to produce a more complete packet stream. 1699 9.1. Name compression 1701 All the names stored in the C-DNS format are full domain names; no 1702 [RFC1035] name compression is used on the individual names within the 1703 format. Therefore when reconstructing a packet, name compression 1704 must be used in order to reproduce the on the wire representation of 1705 the packet. 1707 [RFC1035] name compression works by substituting trailing sections of 1708 a name with a reference back to the occurrence of those sections 1709 earlier in the message. Not all name server software uses the same 1710 algorithm when compressing domain names within the responses. Some 1711 attempt maximum recompression at the expense of runtime resources, 1712 others use heuristics to balance compression and speed and others use 1713 different rules for what is a valid compression target. 1715 This means that responses to the same question from different name 1716 server software which match in terms of DNS payload content (header, 1717 counts, RRs with name compression removed) do not necessarily match 1718 byte-for-byte on the wire. 1720 Therefore, it is not possible to ensure that the DNS response payload 1721 is reconstructed byte-for-byte from C-DNS data. However, it can at 1722 least, in principle, be reconstructed to have the correct payload 1723 length (since the original response length is captured) if there is 1724 enough knowledge of the commonly implemented name compression 1725 algorithms. For example, a simplistic approach would be to try each 1726 algorithm in turn to see if it reproduces the original length, 1727 stopping at the first match. This would not guarantee the correct 1728 algorithm has been used as it is possible to match the length whilst 1729 still not matching the on the wire bytes but, without further 1730 information added to the C-DNS data, this is the best that can be 1731 achieved. 1733 Appendix B presents an example of two different compression 1734 algorithms used by well-known name server software. 1736 10. Data collection 1738 This section describes a non-normative proposed algorithm for the 1739 processing of a captured stream of DNS queries and responses and 1740 production of a stream of query/response items, matching queries/ 1741 responses where possible. 1743 For the purposes of this discussion, it is assumed that the input has 1744 been pre-processed such that: 1746 1. All IP fragmentation reassembly, TCP stream reassembly, and so 1747 on, has already been performed. 1749 2. Each message is associated with transport metadata required to 1750 generate the Primary ID (see Section 10.2.1). 1752 3. Each message has a well-formed DNS header of 12 bytes and (if 1753 present) the first Question in the Question section can be parsed 1754 to generate the Secondary ID (see below). As noted earlier, this 1755 requirement can result in a malformed query being removed in the 1756 pre-processing stage, but the correctly formed response with 1757 RCODE of FORMERR being present. 1759 DNS messages are processed in the order they are delivered to the 1760 implementation. 1762 It should be noted that packet capture libraries do not necessarily 1763 provide packets in strict chronological order. This can, for 1764 example, arise on multi-core platforms where packets arriving at a 1765 network device are processed by different cores. On systems where 1766 this behaviour has been observed, the timestamps associated with each 1767 packet are consistent; queries always have a timestamp prior to the 1768 response timestamp. However, the order in which these packets appear 1769 in the packet capture stream is not necessarily strictly 1770 chronological; a response can appear in the capture stream before the 1771 query that provoked the response. For this discussion, this non- 1772 chronological delivery is termed "skew". 1774 In the presence of skew, a response packets can arrive for matching 1775 before the corresponding query. To avoid generating false instances 1776 of responses without a matching query, and queries without a matching 1777 response, the matching algorithm must take account of the possibility 1778 of skew. 1780 10.1. Matching algorithm 1782 A schematic representation of the algorithm for matching Q/R data 1783 items is shown in Figure 3. It takes individual DNS query or 1784 response messages as input, and outputs matched Q/R items. The 1785 numbers in the figure identify matching operations listed in Table 1. 1786 Specific details of the algorithm, for example queues, timers and 1787 identifiers, are given in the following sections. 1789 .----------------------. 1790 | Process next message |<------------------+ 1791 `----------------------' | 1792 | | 1793 +------------------------------+ | 1794 | Generate message identifiers | | 1795 +------------------------------+ | 1796 | | 1797 Response | Query | 1798 +--------------< >---------------+ | 1799 | | | 1800 +--------------------+ +--------------------+ | 1801 | Find earliest QR | | Create QR item [2] | | 1802 | item in OFIFO [1] | +--------------------+ | 1803 +--------------------+ | | 1804 | +---------------+ | 1805 Match | No match | Append new QR | | 1806 +--------< >------+ | item to OFIFO | | 1807 | | +---------------+ | 1808 +-----------+ +--------+ | | 1809 | Update QR | | Add to | +-------------------+ | 1810 | item [3] | | RFIFO | | Find earliest QR | | 1811 +-----------+ +--------+ | item in RFIFO [1] | | 1812 | | +-------------------+ | 1813 +-----------------+ | | 1814 | | | 1815 | +----------------+ Match | No match | 1816 | | Remove R |-------< >-----+ | 1817 | | from RFIFO [3] | | | 1818 | +----------------+ | | 1819 | | | | 1820 +--------------+-----------------------+ | 1821 | | 1822 +----------------------------------------------+ | 1823 | Update all timed out (QT) OFIFO QR items [4] | | 1824 +----------------------------------------------+ | 1825 | | 1826 +--------------------------------+ | 1827 | Remove all timed out (ST) R | | 1828 | from RFIFO, create QR item [5] | | 1829 +--------------------------------+ | 1830 ____________________|_______________________ | 1831 / / | 1832 / Remove all consecutive done entries from /-------+ 1833 / front of OFIFO for further processing / 1834 /____________________________________________/ 1836 Figure 3: Query/Response matching algorithm 1838 +-----+-------------------------------------------+ 1839 | Ref | Operation | 1840 +-----+-------------------------------------------+ 1841 | [1] | Find earliest QR item in FIFO where: | 1842 | | * QR.done = false | 1843 | | * QR.Q.PrimaryID == R.PrimaryID | 1844 | | and, if both QR.Q and R have SecondaryID: | 1845 | | * QR.Q.SecondaryID == R.SecondaryID | 1846 | | | 1847 | [2] | Set: | 1848 | | QR.Q := Q | 1849 | | QR.R := nil | 1850 | | QR.done := false | 1851 | | | 1852 | [3] | Set: | 1853 | | QR.R := R | 1854 | | QR.done := true | 1855 | | | 1856 | [4] | Set: | 1857 | | QR.done := true | 1858 | | | 1859 | [5] | Set: | 1860 | | QR.Q := nil | 1861 | | QR.R := R | 1862 | | QR.done := true | 1863 +-----+-------------------------------------------+ 1865 Table 1: Operations used in the matching algorithm 1867 10.2. Message identifiers 1869 10.2.1. Primary ID (required) 1871 A Primary ID is constructed for each message. It is composed of the 1872 following data: 1874 1. Source IP Address 1876 2. Destination IP Address 1878 3. Source Port 1880 4. Destination Port 1882 5. Transport 1884 6. DNS Message ID 1886 10.2.2. Secondary ID (optional) 1888 If present, the first Question in the Question section is used as a 1889 secondary ID for each message. Note that there may be well formed 1890 DNS queries that have a QDCOUNT of 0, and some responses may have a 1891 QDCOUNT of 0 (for example, responses with RCODE=FORMERR or NOTIMP). 1892 In this case the secondary ID is not used in matching. 1894 10.3. Algorithm parameters 1896 1. Query timeout, QT. A query arrives with timestamp t1. If no 1897 response matching that query has arrived before other input 1898 arrives timestamped later than (t1 + QT), a query/response item 1899 containing only a query item is recorded. The query timeout 1900 value is typically of the order of 5 seconds. 1902 2. Skew timeout, ST. A response arrives with timestamp t2. If a 1903 response has not been matched by a query before input arrives 1904 timestamped later than (t2 + ST), a query/response item 1905 containing only a response is recorded. The skew timeout value 1906 is typically a few microseconds. 1908 10.4. Algorithm requirements 1910 The algorithm is designed to handle the following input data: 1912 1. Multiple queries with the same Primary ID (but different 1913 Secondary ID) arriving before any responses for these queries are 1914 seen. 1916 2. Multiple queries with the same Primary and Secondary ID arriving 1917 before any responses for these queries are seen. 1919 3. Queries for which no later response can be found within the 1920 specified timeout. 1922 4. Responses for which no previous query can be found within the 1923 specified timeout. 1925 10.5. Algorithm limitations 1927 For cases 1 and 2 listed in the above requirements, it is not 1928 possible to unambiguously match queries with responses. This 1929 algorithm chooses to match to the earliest query with the correct 1930 Primary and Secondary ID. 1932 10.6. Workspace 1934 The algorithm employs two FIFO queues: 1936 o OFIFO, an output FIFO containing Q/R items in chronological order, 1938 o RFIFO, a FIFO holding responses without a matching query in order 1939 of arrival. 1941 10.7. Output 1943 The output is a list of Q/R data items. Both the Query and Response 1944 elements are optional in these items, therefore Q/R data items have 1945 one of three types of content: 1947 1. A matched pair of query and response messages 1949 2. A query message with no response 1951 3. A response message with no query 1953 The timestamp of a list item is that of the query for cases 1 and 2 1954 and that of the response for case 3. 1956 10.8. Post processing 1958 When ending capture, all items in the responses FIFO are timed out 1959 immediately, generating response-only entries to the Q/R data item 1960 FIFO. These and all other remaining entries in the Q/R data item 1961 FIFO should be treated as timed out queries. 1963 11. Implementation guidance 1965 Whilst this document makes no specific recommendations with respect 1966 to Canonical CBOR (see Section 3.9 of [RFC7049]) the following 1967 guidance may be of use to implementors. 1969 Adherence to the first two rules given in Section 3.9 of [RFC7049] 1970 will minimise file sizes. 1972 Adherence to the last two rules given in Section 3.9 of [RFC7049] for 1973 all maps and arrays would unacceptably constrain implementations, for 1974 example, in the use case of real-time data collection in constrained 1975 environments where outputting block tables after query/response data 1976 and allowing indefinite length maps and arrays could reduce memory 1977 requirements. 1979 11.1. Optional data 1981 When decoding C-DNS data some of the items required for a particular 1982 function that the consumer wishes to perform may be missing. 1983 Consumers should consider providing configurable default values to be 1984 used in place of the missing values in their output. 1986 11.2. Trailing bytes 1988 A DNS query message in a UDP or TCP payload can be followed by some 1989 additional (spurious) bytes, which are not stored in C-DNS. 1991 When DNS traffic is sent over TCP, each message is prefixed with a 1992 two byte length field which gives the message length, excluding the 1993 two byte length field. In this context, trailing bytes can occur in 1994 two circumstances with different results: 1996 1. The number of bytes consumed by fully parsing the message is less 1997 than the number of bytes given in the length field (i.e. the 1998 length field is incorrect and too large). In this case, the 1999 surplus bytes are considered trailing bytes in an analogous 2000 manner to UDP and recorded as such. If only this case occurs it 2001 is possible to process a packet containing multiple DNS messages 2002 where one or more has trailing bytes. 2004 2. There are surplus bytes between the end of a well-formed message 2005 and the start of the length field for the next message. In this 2006 case the first of the surplus bytes will be processed as the 2007 first byte of the next length field, and parsing will proceed 2008 from there, almost certainly leading to the next and any 2009 subsequent messages in the packet being considered malformed. 2010 This will not generate a trailing bytes record for the processed 2011 well-formed message. 2013 11.3. Limiting collection of RDATA 2015 Implementations should consider providing a configurable maximum 2016 RDATA size for capture, for example, to avoid memory issues when 2017 confronted with large XFR records. 2019 11.4. Timestamps 2021 The preamble to each block includes a timestamp of the earliest 2022 record in the block. As described in Section 7.5.1, the timestamp is 2023 an array of 2 unsigned integers. The first is a POSIX "time_t" 2024 [posix-time]. Consumers of C-DNS should be aware of this as it 2025 excludes leap seconds and therefore may cause minor anomalies in the 2026 data e.g. when calculating query throughput. 2028 12. Implementation status 2030 [Note to RFC Editor: please remove this section and reference to 2031 [RFC7942] prior to publication.] 2033 This section records the status of known implementations of the 2034 protocol defined by this specification at the time of posting of this 2035 Internet-Draft, and is based on a proposal described in [RFC7942]. 2036 The description of implementations in this section is intended to 2037 assist the IETF in its decision processes in progressing drafts to 2038 RFCs. Please note that the listing of any individual implementation 2039 here does not imply endorsement by the IETF. Furthermore, no effort 2040 has been spent to verify the information presented here that was 2041 supplied by IETF contributors. This is not intended as, and must not 2042 be construed to be, a catalog of available implementations or their 2043 features. Readers are advised to note that other implementations may 2044 exist. 2046 According to [RFC7942], "this will allow reviewers and working groups 2047 to assign due consideration to documents that have the benefit of 2048 running code, which may serve as evidence of valuable experimentation 2049 and feedback that have made the implemented protocols more mature. 2050 It is up to the individual working groups to use this information as 2051 they see fit". 2053 12.1. DNS-STATS Compactor 2055 ICANN/Sinodun IT have developed an open source implementation called 2056 DNS-STATS Compactor. The Compactor is a suite of tools which can 2057 capture DNS traffic (from either a network interface or a PCAP file) 2058 and store it in the Compacted-DNS (C-DNS) file format. PCAP files 2059 for the captured traffic can also be reconstructed. See Compactor 2060 [1]. 2062 This implementation: 2064 o covers the whole of the specification described in the -03 draft 2065 with the exception of support for malformed messages and pico 2066 second time resolution. (Note: this implementation does allow 2067 malformed messages to be recorded separately in a PCAP file). 2069 o is released under the Mozilla Public License Version 2.0. 2071 o has a users mailing list available, see dns-stats-users [2]. 2073 There is also some discussion of issues encountered during 2074 development available at Compressing Pcap Files [3] and Packet 2075 Capture [4]. 2077 This information was last updated on 3rd of May 2018. 2079 13. IANA considerations 2081 IANA is requested to create a registry "C-DNS DNS Capture Format" 2082 containing the subregistries defined in sections Section 13.1 to 2083 Section 13.4 inclusive. 2085 In all cases, new entries may be added to the subregistries by Expert 2086 Review as defined in [RFC8126]. Experts are expected to exercise 2087 their own expert judgement, and should consider the following general 2088 guidelines in addition to any guidelines given particular to a 2089 subregistry. 2091 o There should be a real and compelling use for any new value. 2093 o Values assigned should be carefully chosen to minimise storage 2094 requirements for common cases. 2096 13.1. Transport types 2098 IANA is requested to create a registry "C-DNS Transports" of C-DNS 2099 transport type identifiers. The primary purpose of this registry is 2100 to provide unique identifiers for all transports used for DNS 2101 queries. 2103 The following note is included in this registry: "In version 1.0 of 2104 C-DNS [[this RFC]], there is a field to identify the type of DNS 2105 transport. This field is 4 bits in size." 2107 The initial contents of the registry are as follows - see sections 2108 Section 7.5.3.2 and Section 7.5.3.5 of [[this RFC]]: 2110 +------------+------------+--------------+ 2111 | Identifier | Name | Reference | 2112 +------------+------------+--------------+ 2113 | 0 | UDP | [[this RFC]] | 2114 | 1 | TCP | [[this RFC]] | 2115 | 2 | TLS | [[this RFC]] | 2116 | 3 | DTLS | [[this RFC]] | 2117 | 4 | DoH | [[this RFC]] | 2118 | 5-15 | Unassigned | | 2119 +------------+------------+--------------+ 2121 Expert reviewers should take the following points into consideration: 2123 o Is the requested DNS transport described by a Standards Track RFC? 2125 13.2. Data storage flags 2127 IANA is requested to create a registry "C-DNS Storage Flags" of C-DNS 2128 data storage flags. The primary purpose of this registry is to 2129 provide indicators giving hints on processing of the data stored. 2131 The following note is included in this registry: "In version 1.0 of 2132 C-DNS [[this RFC]], there is a field describing attributes of the 2133 data recorded. The field is a CBOR [RFC7049] unsigned integer 2134 holding bit flags." 2136 The initial contents of the registry are as follows - see section 2137 Section 7.4.1.1 of [[this RFC]]: 2139 +------+------------------+-----------------------------+-----------+ 2140 | Bit | Name | Description | Reference | 2141 +------+------------------+-----------------------------+-----------+ 2142 | 0 | anonymised-data | The data has been | [[this | 2143 | | | anonymised. | RFC]] | 2144 | 1 | sampled-data | The data is sampled data. | [[this | 2145 | | | | RFC]] | 2146 | 2 | normalized-names | Names in the data have been | [[this | 2147 | | | normalized. | RFC]] | 2148 | 3-63 | Unassigned | | | 2149 +------+------------------+-----------------------------+-----------+ 2151 13.3. Response processing flags 2153 IANA is requested to create a registry "C-DNS Response Flags" of 2154 C-DNS response processing flags. The primary purpose of this 2155 registry is to provide indicators giving hints on the generation of a 2156 particular response. 2158 The following note is included in this registry: "In version 1.0 of 2159 C-DNS [[this RFC]], there is a field describing attributes of the 2160 responses recorded. The field is a CBOR [RFC7049] unsigned integer 2161 holding bit flags." 2163 The initial contents of the registry are as follows - see section 2164 Section 7.6.1 of [[this RFC]]: 2166 +------+------------+-------------------------------+--------------+ 2167 | Bit | Name | Description | Reference | 2168 +------+------------+-------------------------------+--------------+ 2169 | 0 | from-cache | The response came from cache. | [[this RFC]] | 2170 | 1-63 | Unassigned | | | 2171 +------+------------+-------------------------------+--------------+ 2173 13.4. AddressEvent types 2175 IANA is requested to create a registry "C-DNS Address Event Types" of 2176 C-DNS AddressEvent types. The primary purpose of this registry is to 2177 provide unique identifiers of different types of C-DNS address 2178 events, and so specify the contents of the optional companion field 2179 "ae-code" for each type. 2181 The following note is included in this registry: "In version 1.0 of 2182 C-DNS [[this RFC]], there is a field identify types of the events 2183 related to client addresses. This field is a CBOR [RFC7049] unsigned 2184 integer. There is a related optional field "ae-code", which, if 2185 present, holds an additional CBOR unsigned integer giving additional 2186 information specific to the event type." 2188 The initial contents of the registry are as follows - see section 2189 Section 7.7: 2191 +------------+----------------------+-------------------+-----------+ 2192 | Identifier | Event Type | ae-code contents | Reference | 2193 +------------+----------------------+-------------------+-----------+ 2194 | 0 | TCP reset | None | [[this | 2195 | | | | RFC]] | 2196 | 1 | ICMP time exceeded | ICMP code | [[this | 2197 | | | [icmpcodes] | RFC]] | 2198 | 2 | ICMP destination | ICMP code | [[this | 2199 | | unreachable | [icmpcodes] | RFC]] | 2200 | 3 | ICMPv6 time exceeded | ICMPv6 code | [[this | 2201 | | | [icmp6codes] | RFC]] | 2202 | 4 | ICMPv6 destination | ICMPv6 code | [[this | 2203 | | unreachable | [icmp6codes] | RFC]] | 2204 | 5 | ICMPv6 packet too | ICMPv6 code | [[this | 2205 | | big | [icmp6codes] | RFC]] | 2206 | >5 | Unassigned | | | 2207 +------------+----------------------+-------------------+-----------+ 2209 Expert reviewers should take the following points into consideration: 2211 o "ae-code" contents must be defined for a type, or if not 2212 appropriate specified as "None". A specification of "None" 2213 requires less storage, and is therefore preferred. 2215 14. Security considerations 2217 Any control interface MUST perform authentication and encryption. 2219 Any data upload MUST be authenticated and encrypted. 2221 15. Privacy considerations 2223 Storage of DNS traffic by operators in PCAP and other formats is a 2224 long standing and widespread practice. Section 2.5 of 2225 [I-D.bortzmeyer-dprive-rfc7626-bis] is an analysis of the risks to 2226 Internet users of the storage of DNS traffic data in servers 2227 (recursive resolvers, authoritative and rogue servers). 2229 Section 5.2 of [I-D.dickinson-dprive-bcp-op] describes mitigations 2230 for those risks for data stored on recursive resolvers (but which 2231 could by extension apply to authoritative servers). These include 2232 data handling practices and methods for data minimization, IP address 2233 pseudonymization and anonymization. Appendix B of that document 2234 presents an analysis of 7 published anonymization processes. In 2235 addition, RSSAC have recently published RSSAC04: [5] " 2236 Recommendations on Anonymization Processes for Source IP Addresses 2237 Submitted for Future Analysis". 2239 The above analyses consider full data capture (e.g using PCAP) as a 2240 baseline for privacy considerations and therefore this format 2241 specification introduces no new user privacy issues beyond those of 2242 full data capture (which are quite severe). It does provides 2243 mechanisms to selectively record only certain fields at the time of 2244 data capture to improve user privacy and to explicitly indicate that 2245 data is sampled and or anonymized. It also provide flags to indicate 2246 if data normalization has been performed; data normalization 2247 increases user privacy by reducing the potential for fingerprinting 2248 individuals, however, a trade-off is potentially reducing the 2249 capacity to identify attack traffic via query name signatures. 2250 Operators should carefully consider their operational requirements 2251 and privacy policies and SHOULD capture at source the minimum user 2252 data required to meet their needs. 2254 16. Acknowledgements 2256 The authors wish to thank CZ.NIC, in particular Tomas Gavenciak, for 2257 many useful discussions on binary formats, compression and packet 2258 matching. Also Jan Vcelak and Wouter Wijngaards for discussions on 2259 name compression and Paul Hoffman for a detailed review of the 2260 document and the C-DNS CDDL. 2262 Thanks also to Robert Edmonds, Jerry Lundstroem, Richard Gibson, 2263 Stephane Bortzmeyer and many other members of DNSOP for review. 2265 Also, Miek Gieben for mmark [6] 2267 17. Changelog 2269 draft-ietf-dnsop-dns-capture-format-10 2271 o Add IANA Considerations 2273 o Convert graph in C.6 to table 2275 draft-ietf-dnsop-dns-capture-format-09 2277 o Editorial changes arising from IESG review 2279 o *-transport-flags and may be mandatory in some configurations 2281 o Mark fields that are conditionally mandatory 2283 o Change `promisc' flag CDDL data type to boolean 2285 o Add ranges to configuration quantities where appropriate 2287 draft-ietf-dnsop-dns-capture-format-08 2289 o Convert diagrams to ASCII 2291 o Describe versioning 2293 o Fix unused group warning in CDDL 2295 draft-ietf-dnsop-dns-capture-format-07 2297 o Resolve outstanding questions and TODOs 2299 o Make RR RDATA optional 2301 o Update matching diagram and explain skew 2303 o Add count of discarded messages to block statistics 2305 o Editorial clarifications and improvements 2307 draft-ietf-dnsop-dns-capture-format-06 2309 o Correct BlockParameters type to map 2311 o Make RR ttl optional 2313 o Add storage flag indicating name normalization 2314 o Add storage parameter fields for sampling and anonymization 2315 methods 2317 o Editorial clarifications and improvements 2319 draft-ietf-dnsop-dns-capture-format-05 2321 o Make all data items in Q/R, QuerySignature and Malformed Message 2322 arrays optional 2324 o Re-structure the FilePreamble and ConfigurationParameters into 2325 BlockParameters 2327 o BlockParameters has separate Storage and Collection Parameters 2329 o Storage Parameters includes information on what optional fields 2330 are present, and flags specifying anonymization or sampling 2332 o Addresses can now be stored as prefixes. 2334 o Switch to using a variable sub-second timing granularity 2336 o Add response bailiwick and query response type 2338 o Add specifics of how to record malformed messages 2340 o Add implementation guidance 2342 o Improve terminology and naming consistency 2344 draft-ietf-dnsop-dns-capture-format-04 2346 o Correct query-d0 to query-do in CDDL 2348 o Clarify that map keys are unsigned integers 2350 o Add Type to Class/Type table 2352 o Clarify storage format in section 7.12 2354 draft-ietf-dnsop-dns-capture-format-03 2356 o Added an Implementation Status section 2358 draft-ietf-dnsop-dns-capture-format-02 2360 o Update qr_data_format.png to match CDDL 2361 o Editorial clarifications and improvements 2363 draft-ietf-dnsop-dns-capture-format-01 2365 o Many editorial improvements by Paul Hoffman 2367 o Included discussion of malformed message handling 2369 o Improved Appendix C on Comparison of Binary Formats 2371 o Now using C-DNS field names in the tables in section 8 2373 o A handful of new fields included (CDDL updated) 2375 o Timestamps now include optional picoseconds 2377 o Added details of block statistics 2379 draft-ietf-dnsop-dns-capture-format-00 2381 o Changed dnstap.io to dnstap.info 2383 o qr_data_format.png was cut off at the bottom 2385 o Update authors address 2387 o Improve wording in Abstract 2389 o Changed DNS-STAT to C-DNS in CDDL 2391 o Set the format version in the CDDL 2393 o Added a TODO: Add block statistics 2395 o Added a TODO: Add extend to support pico/nano. Also do this for 2396 Time offset and Response delay 2398 o Added a TODO: Need to develop optional representation of malformed 2399 messages within C-DNS and what this means for packet matching. 2400 This may influence which fields are optional in the rest of the 2401 representation. 2403 o Added section on design goals to Introduction 2405 o Added a TODO: Can Class be optimised? Should a class of IN be 2406 inferred if not present? 2408 draft-dickinson-dnsop-dns-capture-format-00 2409 o Initial commit 2411 18. References 2413 18.1. Normative References 2415 [I-D.ietf-cbor-cddl] 2416 Birkholz, H., Vigano, C., and C. Bormann, "Concise data 2417 definition language (CDDL): a notational convention to 2418 express CBOR and JSON data structures", draft-ietf-cbor- 2419 cddl-06 (work in progress), November 2018. 2421 [pcap-filter] 2422 tcpdump.org, "Manpage of PCAP-FILTER", 2017, 2423 . 2425 [pcap-options] 2426 tcpdump.org, "Manpage of PCAP", 2018, 2427 . 2429 [posix-time] 2430 The Open Group, "Section 4.16, Base Definitions, Standard 2431 for Information Technology - Portable Operating System 2432 Interface (POSIX(R)) Base Specifications, Issue 7", IEEE 2433 Standard 1003.1 2017 Edition, 2434 DOI 10.1109/IEEESTD.2018.8277153, 2017. 2436 [RFC0792] Postel, J., "Internet Control Message Protocol", STD 5, 2437 RFC 792, DOI 10.17487/RFC0792, September 1981, 2438 . 2440 [RFC1035] Mockapetris, P., "Domain names - implementation and 2441 specification", STD 13, RFC 1035, DOI 10.17487/RFC1035, 2442 November 1987, . 2444 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 2445 Requirement Levels", BCP 14, RFC 2119, 2446 DOI 10.17487/RFC2119, March 1997, 2447 . 2449 [RFC3986] Berners-Lee, T., Fielding, R., and L. Masinter, "Uniform 2450 Resource Identifier (URI): Generic Syntax", STD 66, 2451 RFC 3986, DOI 10.17487/RFC3986, January 2005, 2452 . 2454 [RFC4443] Conta, A., Deering, S., and M. Gupta, Ed., "Internet 2455 Control Message Protocol (ICMPv6) for the Internet 2456 Protocol Version 6 (IPv6) Specification", STD 89, 2457 RFC 4443, DOI 10.17487/RFC4443, March 2006, 2458 . 2460 [RFC6891] Damas, J., Graff, M., and P. Vixie, "Extension Mechanisms 2461 for DNS (EDNS(0))", STD 75, RFC 6891, 2462 DOI 10.17487/RFC6891, April 2013, 2463 . 2465 [RFC7049] Bormann, C. and P. Hoffman, "Concise Binary Object 2466 Representation (CBOR)", RFC 7049, DOI 10.17487/RFC7049, 2467 October 2013, . 2469 [RFC7858] Hu, Z., Zhu, L., Heidemann, J., Mankin, A., Wessels, D., 2470 and P. Hoffman, "Specification for DNS over Transport 2471 Layer Security (TLS)", RFC 7858, DOI 10.17487/RFC7858, May 2472 2016, . 2474 [RFC8126] Cotton, M., Leiba, B., and T. Narten, "Guidelines for 2475 Writing an IANA Considerations Section in RFCs", BCP 26, 2476 RFC 8126, DOI 10.17487/RFC8126, June 2017, 2477 . 2479 [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2480 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, 2481 May 2017, . 2483 [RFC8484] Hoffman, P. and P. McManus, "DNS Queries over HTTPS 2484 (DoH)", RFC 8484, DOI 10.17487/RFC8484, October 2018, 2485 . 2487 18.2. Informative References 2489 [ditl] DNS-OARC, "DITL", 2016, 2490 . 2492 [dnscap] DNS-OARC, "DNSCAP", 2016, 2493 . 2495 [dnstap] dnstap.info, "dnstap", 2016, . 2497 [dsc] Wessels, D. and J. Lundstrom, "DSC", 2016, 2498 . 2500 [I-D.bortzmeyer-dprive-rfc7626-bis] 2501 Bortzmeyer, S. and S. Dickinson, "DNS Privacy 2502 Considerations", draft-bortzmeyer-dprive-rfc7626-bis-01 2503 (work in progress), July 2018. 2505 [I-D.daley-dnsxml] 2506 Daley, J., Morris, S., and J. Dickinson, "dnsxml - A 2507 standard XML representation of DNS data", draft-daley- 2508 dnsxml-00 (work in progress), July 2013. 2510 [I-D.dickinson-dprive-bcp-op] 2511 Dickinson, S., Overeinder, B., Rijswijk-Deij, R., and A. 2512 Mankin, "Recommendations for DNS Privacy Service 2513 Operators", draft-dickinson-dprive-bcp-op-01 (work in 2514 progress), July 2018. 2516 [icmp6codes] 2517 IANA, "ICMPv6 "Code" Fields", 2018, 2518 . 2521 [icmpcodes] 2522 IANA, "Code Fields", 2018, 2523 . 2526 [IEEE802.1Q] 2527 IEEE, "IEEE Standard for Local and metropolitan area 2528 networks -- Bridges and Bridged Networks", 2529 DOI 10.1109/IEEESTD.2014.6991462, 2014. 2531 [opcodes] IANA, "DNS OpCodes", 2018, 2532 . 2535 [packetq] .SE - The Internet Infrastructure Foundation, "PacketQ", 2536 2014, . 2538 [pcap] tcpdump.org, "PCAP", 2016, . 2540 [pcapng] Tuexen, M., Risso, F., Bongertz, J., Combs, G., and G. 2541 Harris, "pcap-ng", 2016, 2542 . 2544 [rcodes] IANA, "DNS RCODEs", 2018, 2545 . 2548 [RFC7942] Sheffer, Y. and A. Farrel, "Improving Awareness of Running 2549 Code: The Implementation Status Section", BCP 205, 2550 RFC 7942, DOI 10.17487/RFC7942, July 2016, 2551 . 2553 [RFC8259] Bray, T., Ed., "The JavaScript Object Notation (JSON) Data 2554 Interchange Format", STD 90, RFC 8259, 2555 DOI 10.17487/RFC8259, December 2017, 2556 . 2558 [RFC8427] Hoffman, P., "Representing DNS Messages in JSON", 2559 RFC 8427, DOI 10.17487/RFC8427, July 2018, 2560 . 2562 [rrclasses] 2563 IANA, "DNS CLASSes", 2018, 2564 . 2567 [rrtypes] IANA, "Resource Record (RR) TYPEs", 2018, 2568 . 2571 18.3. URIs 2573 [1] https://github.com/dns-stats/compactor/wiki 2575 [2] https://mm.dns-stats.org/mailman/listinfo/dns-stats-users 2577 [3] https://www.sinodun.com/2017/06/compressing-pcap-files/ 2579 [4] https://www.sinodun.com/2017/06/more-on-debian-jessieubuntu- 2580 trusty-packet-capture-woes/ 2582 [5] https://www.icann.org/en/system/files/files/rssac- 2583 040-07aug18-en.pdf 2585 [6] https://github.com/miekg/mmark 2587 [7] https://www.nlnetlabs.nl/projects/nsd/ 2589 [8] https://www.knot-dns.cz/ 2591 [9] https://avro.apache.org/ 2593 [10] https://developers.google.com/protocol-buffers/ 2595 [11] http://cbor.io 2597 [12] https://github.com/kubo/snzip 2599 [13] http://google.github.io/snappy/ 2601 [14] http://lz4.github.io/lz4/ 2603 [15] http://www.gzip.org/ 2605 [16] http://facebook.github.io/zstd/ 2607 [17] http://tukaani.org/xz/ 2609 Appendix A. CDDL 2611 This appendix gives a CDDL [I-D.ietf-cbor-cddl] specification for 2612 C-DNS. 2614 CDDL does not permit a range of allowed values to be specified for a 2615 bitfield. Where necessary, those values are given as a CDDL group, 2616 but the group definition is commented out to prevent CDDL tooling 2617 from warning that the group is unused. 2619 ; CDDL specification of the file format for C-DNS, 2620 ; which describes a collection of DNS messages and 2621 ; traffic meta-data. 2623 ; 2624 ; The overall structure of a file. 2625 ; 2626 File = [ 2627 file-type-id : "C-DNS", 2628 file-preamble : FilePreamble, 2629 file-blocks : [* Block], 2630 ] 2632 ; 2633 ; The file preamble. 2634 ; 2635 FilePreamble = { 2636 major-format-version => 1, 2637 minor-format-version => 0, 2638 ? private-version => uint, 2639 block-parameters => [+ BlockParameters], 2640 } 2641 major-format-version = 0 2642 minor-format-version = 1 2643 private-version = 2 2644 block-parameters = 3 2645 BlockParameters = { 2646 storage-parameters => StorageParameters, 2647 ? collection-parameters => CollectionParameters, 2648 } 2649 storage-parameters = 0 2650 collection-parameters = 1 2652 IPv6PrefixLength = 1..128 2653 IPv4PrefixLength = 1..32 2654 OpcodeRange = 0..15 2655 RRTypeRange = 0..65535 2657 StorageParameters = { 2658 ticks-per-second => uint, 2659 max-block-items => uint, 2660 storage-hints => StorageHints, 2661 opcodes => [+ OpcodeRange], 2662 rr-types => [+ RRTypeRange], 2663 ? storage-flags => StorageFlags, 2664 ? client-address-prefix-ipv4 => IPv4PrefixLength, 2665 ? client-address-prefix-ipv6 => IPv6PrefixLength, 2666 ? server-address-prefix-ipv4 => IPv4PrefixLength, 2667 ? server-address-prefix-ipv6 => IPv6PrefixLength, 2668 ? sampling-method => tstr, 2669 ? anonymisation-method => tstr, 2670 } 2671 ticks-per-second = 0 2672 max-block-items = 1 2673 storage-hints = 2 2674 opcodes = 3 2675 rr-types = 4 2676 storage-flags = 5 2677 client-address-prefix-ipv4 = 6 2678 client-address-prefix-ipv6 = 7 2679 server-address-prefix-ipv4 = 8 2680 server-address-prefix-ipv6 = 9 2681 sampling-method = 10 2682 anonymisation-method = 11 2684 ; A hint indicates if the collection method will output the 2685 ; item or will ignore the item if present. 2686 StorageHints = { 2687 query-response-hints => QueryResponseHints, 2688 query-response-signature-hints => 2689 QueryResponseSignatureHints, 2690 rr-hints => RRHints, 2691 other-data-hints => OtherDataHints, 2692 } 2693 query-response-hints = 0 2694 query-response-signature-hints = 1 2695 rr-hints = 2 2696 other-data-hints = 3 2698 QueryResponseHintValues = &( 2699 time-offset : 0, 2700 client-address-index : 1, 2701 client-port : 2, 2702 transaction-id : 3, 2703 qr-signature-index : 4, 2704 client-hoplimit : 5, 2705 response-delay : 6, 2706 query-name-index : 7, 2707 query-size : 8, 2708 response-size : 9, 2709 response-processing-data : 10, 2710 query-question-sections : 11, ; Second & subsequent 2711 ; questions 2712 query-answer-sections : 12, 2713 query-authority-sections : 13, 2714 query-additional-sections : 14, 2715 response-answer-sections : 15, 2716 response-authority-sections : 16, 2717 response-additional-sections : 17, 2718 ) 2719 QueryResponseHints = uint .bits QueryResponseHintValues 2721 QueryResponseSignatureHintValues = &( 2722 server-address : 0, 2723 server-port : 1, 2724 qr-transport-flags : 2, 2725 qr-type : 3, 2726 qr-sig-flags : 4, 2727 query-opcode : 5, 2728 dns-flags : 6, 2729 query-rcode : 7, 2730 query-class-type : 8, 2731 query-qdcount : 9, 2732 query-ancount : 10, 2733 query-arcount : 11, 2734 query-nscount : 12, 2735 query-edns-version : 13, 2736 query-udp-size : 14, 2737 query-opt-rdata : 15, 2738 response-rcode : 16, 2739 ) 2740 QueryResponseSignatureHints = 2741 uint .bits QueryResponseSignatureHintValues 2743 RRHintValues = &( 2744 ttl : 0, 2745 rdata-index : 1, 2746 ) 2747 RRHints = uint .bits RRHintValues 2749 OtherDataHintValues = &( 2750 malformed-messages : 0, 2751 address-event-counts : 1, 2752 ) 2753 OtherDataHints = uint .bits OtherDataHintValues 2755 StorageFlagValues = &( 2756 anonymised-data : 0, 2757 sampled-data : 1, 2758 normalized-names : 2, 2759 ) 2760 StorageFlags = uint .bits StorageFlagValues 2762 ; Hints for later analysis. 2763 VLANIdRange = 1..4094 2765 CollectionParameters = { 2766 ? query-timeout => uint, 2767 ? skew-timeout => uint, 2768 ? snaplen => uint, 2769 ? promisc => bool, 2770 ? interfaces => [+ tstr], 2771 ? server-addresses => [+ IPAddress], 2772 ? vlan-ids => [+ VLANIdRange], 2773 ? filter => tstr, 2774 ? generator-id => tstr, 2775 ? host-id => tstr, 2776 } 2777 query-timeout = 0 2778 skew-timeout = 1 2779 snaplen = 2 2780 promisc = 3 2781 interfaces = 4 2782 server-addresses = 5 2783 vlan-ids = 6 2784 filter = 7 2785 generator-id = 8 2786 host-id = 9 2788 ; 2789 ; Data in the file is stored in Blocks. 2790 ; 2791 Block = { 2792 block-preamble => BlockPreamble, 2793 ? block-statistics => BlockStatistics, ; Much of this 2794 ; could be derived 2795 ? block-tables => BlockTables, 2796 ? query-responses => [+ QueryResponse], 2797 ? address-event-counts => [+ AddressEventCount], 2798 ? malformed-messages => [+ MalformedMessage], 2799 } 2800 block-preamble = 0 2801 block-statistics = 1 2802 block-tables = 2 2803 query-responses = 3 2804 address-event-counts = 4 2805 malformed-messages = 5 2807 ; 2808 ; The (mandatory) preamble to a block. 2809 ; 2810 BlockPreamble = { 2811 ? earliest-time => Timestamp, 2812 ? block-parameters-index => uint .default 0, 2813 } 2814 earliest-time = 0 2815 block-parameters-index = 1 2817 ; Ticks are subsecond intervals. The number of ticks in a second is 2818 ; file/block metadata. Signed and unsigned tick types are defined. 2819 ticks = int 2820 uticks = uint 2822 Timestamp = [ 2823 timestamp-secs : uint, 2824 timestamp-uticks : uticks, 2825 ] 2827 ; 2828 ; Statistics about the block contents. 2829 ; 2830 BlockStatistics = { 2831 ? processed-messages => uint, 2832 ? qr-data-items => uint, 2833 ? unmatched-queries => uint, 2834 ? unmatched-responses => uint, 2835 ? discarded-opcode => uint, 2836 ? malformed-items => uint, 2838 } 2839 processed-messages = 0 2840 qr-data-items = 1 2841 unmatched-queries = 2 2842 unmatched-responses = 3 2843 discarded-opcode = 4 2844 malformed-items = 5 2846 ; 2847 ; Tables of common data referenced from records in a block. 2848 ; 2849 BlockTables = { 2850 ? ip-address => [+ IPAddress], 2851 ? classtype => [+ ClassType], 2852 ? name-rdata => [+ bstr], ; Holds both Names 2853 ; and RDATA 2854 ? qr-sig => [+ QueryResponseSignature], 2855 ? QuestionTables, 2856 ? RRTables, 2857 ? malformed-message-data => [+ MalformedMessageData], 2858 } 2859 ip-address = 0 2860 classtype = 1 2861 name-rdata = 2 2862 qr-sig = 3 2863 qlist = 4 2864 qrr = 5 2865 rrlist = 6 2866 rr = 7 2867 malformed-message-data = 8 2869 IPv4Address = bstr .size 4 2870 IPv6Address = bstr .size 16 2871 IPAddress = IPv4Address / IPv6Address 2873 ClassType = { 2874 type => uint, 2875 class => uint, 2876 } 2877 type = 0 2878 class = 1 2880 QueryResponseSignature = { 2881 ? server-address-index => uint, 2882 ? server-port => uint, 2883 ? qr-transport-flags => QueryResponseTransportFlags, 2884 ? qr-type => QueryResponseType, 2885 ? qr-sig-flags => QueryResponseFlags, 2886 ? query-opcode => uint, 2887 ? qr-dns-flags => DNSFlags, 2888 ? query-rcode => uint, 2889 ? query-classtype-index => uint, 2890 ? query-qd-count => uint, 2891 ? query-an-count => uint, 2892 ? query-ns-count => uint, 2893 ? query-ar-count => uint, 2894 ? edns-version => uint, 2895 ? udp-buf-size => uint, 2896 ? opt-rdata-index => uint, 2897 ? response-rcode => uint, 2898 } 2899 server-address-index = 0 2900 server-port = 1 2901 qr-transport-flags = 2 2902 qr-type = 3 2903 qr-sig-flags = 4 2904 query-opcode = 5 2905 qr-dns-flags = 6 2906 query-rcode = 7 2907 query-classtype-index = 8 2908 query-qd-count = 9 2909 query-an-count = 10 2910 query-ns-count = 12 2911 query-ar-count = 12 2912 edns-version = 13 2913 udp-buf-size = 14 2914 opt-rdata-index = 15 2915 response-rcode = 16 2917 ; Transport gives the values that may appear in bits 1..4 of 2918 ; TransportFlags. There is currently no way to express this in 2919 ; CDDL, so Transport is unused. To avoid confusion when used 2920 ; with CDDL tools, it is commented out. 2921 ; 2922 ; Transport = &( 2923 ; udp : 0, 2924 ; tcp : 1, 2925 ; tls : 2, 2926 ; dtls : 3, 2927 ; doh : 4, 2928 ; ) 2930 TransportFlagValues = &( 2931 ip-version : 0, ; 0=IPv4, 1=IPv6 2932 ) / (1..4) 2933 TransportFlags = uint .bits TransportFlagValues 2934 QueryResponseTransportFlagValues = &( 2935 query-trailingdata : 5, 2936 ) / TransportFlagValues 2937 QueryResponseTransportFlags = 2938 uint .bits QueryResponseTransportFlagValues 2940 QueryResponseType = &( 2941 stub : 0, 2942 client : 1, 2943 resolver : 2, 2944 auth : 3, 2945 forwarder : 4, 2946 tool : 5, 2947 ) 2949 QueryResponseFlagValues = &( 2950 has-query : 0, 2951 has-reponse : 1, 2952 query-has-opt : 2, 2953 response-has-opt : 3, 2954 query-has-no-question : 4, 2955 response-has-no-question: 5, 2956 ) 2957 QueryResponseFlags = uint .bits QueryResponseFlagValues 2959 DNSFlagValues = &( 2960 query-cd : 0, 2961 query-ad : 1, 2962 query-z : 2, 2963 query-ra : 3, 2964 query-rd : 4, 2965 query-tc : 5, 2966 query-aa : 6, 2967 query-do : 7, 2968 response-cd: 8, 2969 response-ad: 9, 2970 response-z : 10, 2971 response-ra: 11, 2972 response-rd: 12, 2973 response-tc: 13, 2974 response-aa: 14, 2975 ) 2976 DNSFlags = uint .bits DNSFlagValues 2978 QuestionTables = ( 2979 qlist => [+ QuestionList], 2980 qrr => [+ Question] 2981 ) 2982 QuestionList = [+ uint] ; Index of Question 2984 Question = { ; Second and subsequent questions 2985 name-index => uint, ; Index to a name in the 2986 ; name-rdata table 2987 classtype-index => uint, 2988 } 2989 name-index = 0 2990 classtype-index = 1 2992 RRTables = ( 2993 rrlist => [+ RRList], 2994 rr => [+ RR] 2995 ) 2997 RRList = [+ uint] ; Index of RR 2999 RR = { 3000 name-index => uint, ; Index to a name in the 3001 ; name-rdata table 3002 classtype-index => uint, 3003 ? ttl => uint, 3004 ? rdata-index => uint, ; Index to RDATA in the 3005 ; name-rdata table 3006 } 3007 ; Other map key values already defined above. 3008 ttl = 2 3009 rdata-index = 3 3011 MalformedMessageData = { 3012 ? server-address-index => uint, 3013 ? server-port => uint, 3014 ? mm-transport-flags => TransportFlags, 3015 ? mm-payload => bstr, 3016 } 3017 ; Other map key values already defined above. 3018 mm-transport-flags = 2 3019 mm-payload = 3 3021 ; 3022 ; A single query/response pair. 3023 ; 3024 QueryResponse = { 3025 ? time-offset => uticks, ; Time offset from 3026 ; start of block 3027 ? client-address-index => uint, 3028 ? client-port => uint, 3029 ? transaction-id => uint, 3030 ? qr-signature-index => uint, 3031 ? client-hoplimit => uint, 3032 ? response-delay => ticks, 3033 ? query-name-index => uint, 3034 ? query-size => uint, ; DNS size of query 3035 ? response-size => uint, ; DNS size of response 3036 ? response-processing-data => ResponseProcessingData, 3037 ? query-extended => QueryResponseExtended, 3038 ? response-extended => QueryResponseExtended, 3039 } 3040 time-offset = 0 3041 client-address-index = 1 3042 client-port = 2 3043 transaction-id = 3 3044 qr-signature-index = 4 3045 client-hoplimit = 5 3046 response-delay = 6 3047 query-name-index = 7 3048 query-size = 8 3049 response-size = 9 3050 response-processing-data = 10 3051 query-extended = 11 3052 response-extended = 12 3054 ResponseProcessingData = { 3055 ? bailiwick-index => uint, 3056 ? processing-flags => ResponseProcessingFlags, 3057 } 3058 bailiwick-index = 0 3059 processing-flags = 1 3061 ResponseProcessingFlagValues = &( 3062 from-cache : 0, 3063 ) 3064 ResponseProcessingFlags = uint .bits ResponseProcessingFlagValues 3066 QueryResponseExtended = { 3067 ? question-index => uint, ; Index of QuestionList 3068 ? answer-index => uint, ; Index of RRList 3069 ? authority-index => uint, 3070 ? additional-index => uint, 3071 } 3072 question-index = 0 3073 answer-index = 1 3074 authority-index = 2 3075 additional-index = 3 3077 ; 3078 ; Address event data. 3079 ; 3080 AddressEventCount = { 3081 ae-type => &AddressEventType, 3082 ? ae-code => uint, 3083 ae-address-index => uint, 3084 ae-count => uint, 3085 } 3086 ae-type = 0 3087 ae-code = 1 3088 ae-address-index = 2 3089 ae-count = 3 3091 AddressEventType = ( 3092 tcp-reset : 0, 3093 icmp-time-exceeded : 1, 3094 icmp-dest-unreachable : 2, 3095 icmpv6-time-exceeded : 3, 3096 icmpv6-dest-unreachable: 4, 3097 icmpv6-packet-too-big : 5, 3098 ) 3100 ; 3101 ; Malformed messages. 3102 ; 3103 MalformedMessage = { 3104 ? time-offset => uticks, ; Time offset from 3105 ; start of block 3106 ? client-address-index => uint, 3107 ? client-port => uint, 3108 ? message-data-index => uint, 3109 } 3110 ; Other map key values already defined above. 3111 message-data-index = 3 3113 Appendix B. DNS Name compression example 3115 The basic algorithm, which follows the guidance in [RFC1035], is 3116 simply to collect each name, and the offset in the packet at which it 3117 starts, during packet construction. As each name is added, it is 3118 offered to each of the collected names in order of collection, 3119 starting from the first name. If labels at the end of the name can 3120 be replaced with a reference back to part (or all) of the earlier 3121 name, and if the uncompressed part of the name is shorter than any 3122 compression already found, the earlier name is noted as the 3123 compression target for the name. 3125 The following tables illustrate the process. In an example packet, 3126 the first name is foo.example. 3128 +---+-------------+--------------+--------------------+ 3129 | N | Name | Uncompressed | Compression Target | 3130 +---+-------------+--------------+--------------------+ 3131 | 1 | foo.example | | | 3132 +---+-------------+--------------+--------------------+ 3134 The next name added is bar.example. This is matched against 3135 foo.example. The example part of this can be used as a compression 3136 target, with the remaining uncompressed part of the name being bar. 3138 +---+-------------+--------------+-----------------------+ 3139 | N | Name | Uncompressed | Compression Target | 3140 +---+-------------+--------------+-----------------------+ 3141 | 1 | foo.example | | | 3142 | 2 | bar.example | bar | 1 + offset to example | 3143 +---+-------------+--------------+-----------------------+ 3145 The third name added is www.bar.example. This is first matched 3146 against foo.example, and as before this is recorded as a compression 3147 target, with the remaining uncompressed part of the name being 3148 www.bar. It is then matched against the second name, which again can 3149 be a compression target. Because the remaining uncompressed part of 3150 the name is www, this is an improved compression, and so it is 3151 adopted. 3153 +---+-----------------+--------------+-----------------------+ 3154 | N | Name | Uncompressed | Compression Target | 3155 +---+-----------------+--------------+-----------------------+ 3156 | 1 | foo.example | | | 3157 | 2 | bar.example | bar | 1 + offset to example | 3158 | 3 | www.bar.example | www | 2 | 3159 +---+-----------------+--------------+-----------------------+ 3161 As an optimization, if a name is already perfectly compressed (in 3162 other words, the uncompressed part of the name is empty), then no 3163 further names will be considered for compression. 3165 B.1. NSD compression algorithm 3167 Using the above basic algorithm the packet lengths of responses 3168 generated by NSD [7] can be matched almost exactly. At the time of 3169 writing, a tiny number (<.01%) of the reconstructed packets had 3170 incorrect lengths. 3172 B.2. Knot Authoritative compression algorithm 3174 The Knot Authoritative [8] name server uses different compression 3175 behavior, which is the result of internal optimization designed to 3176 balance runtime speed with compression size gains. In brief, and 3177 omitting complications, Knot Authoritative will only consider the 3178 QNAME and names in the immediately preceding RR section in an RRSET 3179 as compression targets. 3181 A set of smart heuristics as described below can be implemented to 3182 mimic this and while not perfect it produces output nearly, but not 3183 quite, as good a match as with NSD. The heuristics are: 3185 1. A match is only perfect if the name is completely compressed AND 3186 the TYPE of the section in which the name occurs matches the TYPE 3187 of the name used as the compression target. 3189 2. If the name occurs in RDATA: 3191 * If the compression target name is in a query, then only the 3192 first RR in an RRSET can use that name as a compression 3193 target. 3195 * The compression target name MUST be in RDATA. 3197 * The name section TYPE must match the compression target name 3198 section TYPE. 3200 * The compression target name MUST be in the immediately 3201 preceding RR in the RRSET. 3203 Using this algorithm less than 0.1% of the reconstructed packets had 3204 incorrect lengths. 3206 B.3. Observed differences 3208 In sample traffic collected on a root name server around 2-4% of 3209 responses generated by Knot had different packet lengths to those 3210 produced by NSD. 3212 Appendix C. Comparison of Binary Formats 3214 Several binary serialisation formats were considered, and for 3215 completeness were also compared to JSON. 3217 o Apache Avro [9]. Data is stored according to a pre-defined 3218 schema. The schema itself is always included in the data file. 3220 Data can therefore be stored untagged, for a smaller serialisation 3221 size, and be written and read by an Avro library. 3223 * At the time of writing, Avro libraries are available for C, 3224 C++, C#, Java, Python, Ruby and PHP. Optionally tools are 3225 available for C++, Java and C# to generate code for encoding 3226 and decoding. 3228 o Google Protocol Buffers [10]. Data is stored according to a pre- 3229 defined schema. The schema is used by a generator to generate 3230 code for encoding and decoding the data. Data can therefore be 3231 stored untagged, for a smaller serialisation size. The schema is 3232 not stored with the data, so unlike Avro cannot be read with a 3233 generic library. 3235 * Code must be generated for a particular data schema to read and 3236 write data using that schema. At the time of writing, the 3237 Google code generator can currently generate code for encoding 3238 and decoding a schema for C++, Go, Java, Python, Ruby, C#, 3239 Objective-C, Javascript and PHP. 3241 o CBOR [11]. Defined in [RFC7049], this serialisation format is 3242 comparable to JSON but with a binary representation. It does not 3243 use a pre-defined schema, so data is always stored tagged. 3244 However, CBOR data schemas can be described using CDDL 3245 [I-D.ietf-cbor-cddl] and tools exist to verify data files conform 3246 to the schema. 3248 * CBOR is a simple format, and simple to implement. At the time 3249 of writing, the CBOR website lists implementations for 16 3250 languages. 3252 Avro and Protocol Buffers both allow storage of untagged data, but 3253 because they rely on the data schema for this, their implementation 3254 is considerably more complex than CBOR. Using Avro or Protocol 3255 Buffers in an unsupported environment would require notably greater 3256 development effort compared to CBOR. 3258 A test program was written which reads input from a PCAP file and 3259 writes output using one of two basic structures; either a simple 3260 structure, where each query/response pair is represented in a single 3261 record entry, or the C-DNS block structure. 3263 The resulting output files were then compressed using a variety of 3264 common general-purpose lossless compression tools to explore the 3265 compressibility of the formats. The compression tools employed were: 3267 o snzip [12]. A command line compression tool based on the Google 3268 Snappy [13] library. 3270 o lz4 [14]. The command line compression tool from the reference C 3271 LZ4 implementation. 3273 o gzip [15]. The ubiquitous GNU zip tool. 3275 o zstd [16]. Compression using the Zstandard algorithm. 3277 o xz [17]. A popular compression tool noted for high compression. 3279 In all cases the compression tools were run using their default 3280 settings. 3282 Note that this draft does not mandate the use of compression, nor any 3283 particular compression scheme, but it anticipates that in practice 3284 output data will be subject to general-purpose compression, and so 3285 this should be taken into consideration. 3287 "test.pcap", a 662Mb capture of sample data from a root instance was 3288 used for the comparison. The following table shows the formatted 3289 size and size after compression (abbreviated to Comp. in the table 3290 headers), together with the task resident set size (RSS) and the user 3291 time taken by the compression. File sizes are in Mb, RSS in kb and 3292 user time in seconds. 3294 +-------------+-----------+-------+------------+-------+-----------+ 3295 | Format | File size | Comp. | Comp. size | RSS | User time | 3296 +-------------+-----------+-------+------------+-------+-----------+ 3297 | PCAP | 661.87 | snzip | 212.48 | 2696 | 1.26 | 3298 | | | lz4 | 181.58 | 6336 | 1.35 | 3299 | | | gzip | 153.46 | 1428 | 18.20 | 3300 | | | zstd | 87.07 | 3544 | 4.27 | 3301 | | | xz | 49.09 | 97416 | 160.79 | 3302 | | | | | | | 3303 | JSON simple | 4113.92 | snzip | 603.78 | 2656 | 5.72 | 3304 | | | lz4 | 386.42 | 5636 | 5.25 | 3305 | | | gzip | 271.11 | 1492 | 73.00 | 3306 | | | zstd | 133.43 | 3284 | 8.68 | 3307 | | | xz | 51.98 | 97412 | 600.74 | 3308 | | | | | | | 3309 | Avro simple | 640.45 | snzip | 148.98 | 2656 | 0.90 | 3310 | | | lz4 | 111.92 | 5828 | 0.99 | 3311 | | | gzip | 103.07 | 1540 | 11.52 | 3312 | | | zstd | 49.08 | 3524 | 2.50 | 3313 | | | xz | 22.87 | 97308 | 90.34 | 3314 | | | | | | | 3315 | CBOR simple | 764.82 | snzip | 164.57 | 2664 | 1.11 | 3316 | | | lz4 | 120.98 | 5892 | 1.13 | 3317 | | | gzip | 110.61 | 1428 | 12.88 | 3318 | | | zstd | 54.14 | 3224 | 2.77 | 3319 | | | xz | 23.43 | 97276 | 111.48 | 3320 | | | | | | | 3321 | PBuf simple | 749.51 | snzip | 167.16 | 2660 | 1.08 | 3322 | | | lz4 | 123.09 | 5824 | 1.14 | 3323 | | | gzip | 112.05 | 1424 | 12.75 | 3324 | | | zstd | 53.39 | 3388 | 2.76 | 3325 | | | xz | 23.99 | 97348 | 106.47 | 3326 | | | | | | | 3327 | JSON block | 519.77 | snzip | 106.12 | 2812 | 0.93 | 3328 | | | lz4 | 104.34 | 6080 | 0.97 | 3329 | | | gzip | 57.97 | 1604 | 12.70 | 3330 | | | zstd | 61.51 | 3396 | 3.45 | 3331 | | | xz | 27.67 | 97524 | 169.10 | 3332 | | | | | | | 3333 | Avro block | 60.45 | snzip | 48.38 | 2688 | 0.20 | 3334 | | | lz4 | 48.78 | 8540 | 0.22 | 3335 | | | gzip | 39.62 | 1576 | 2.92 | 3336 | | | zstd | 29.63 | 3612 | 1.25 | 3337 | | | xz | 18.28 | 97564 | 25.81 | 3338 | | | | | | | 3339 | CBOR block | 75.25 | snzip | 53.27 | 2684 | 0.24 | 3340 | | | lz4 | 51.88 | 8008 | 0.28 | 3341 | | | gzip | 41.17 | 1548 | 4.36 | 3342 | | | zstd | 30.61 | 3476 | 1.48 | 3343 | | | xz | 18.15 | 97556 | 38.78 | 3344 | | | | | | | 3345 | PBuf block | 67.98 | snzip | 51.10 | 2636 | 0.24 | 3346 | | | lz4 | 52.39 | 8304 | 0.24 | 3347 | | | gzip | 40.19 | 1520 | 3.63 | 3348 | | | zstd | 31.61 | 3576 | 1.40 | 3349 | | | xz | 17.94 | 97440 | 33.99 | 3350 +-------------+-----------+-------+------------+-------+-----------+ 3352 The above results are discussed in the following sections. 3354 C.1. Comparison with full PCAP files 3356 An important first consideration is whether moving away from PCAP 3357 offers significant benefits. 3359 The simple binary formats are typically larger than PCAP, even though 3360 they omit some information such as Ethernet MAC addresses. But not 3361 only do they require less CPU to compress than PCAP, the resulting 3362 compressed files are smaller than compressed PCAP. 3364 C.2. Simple versus block coding 3366 The intention of the block coding is to perform data de-duplication 3367 on query/response records within the block. The simple and block 3368 formats above store exactly the same information for each query/ 3369 response record. This information is parsed from the DNS traffic in 3370 the input PCAP file, and in all cases each field has an identifier 3371 and the field data is typed. 3373 The data de-duplication on the block formats show an order of 3374 magnitude reduction in the size of the format file size against the 3375 simple formats. As would be expected, the compression tools are able 3376 to find and exploit a lot of this duplication, but as the de- 3377 duplication process uses knowledge of DNS traffic, it is able to 3378 retain a size advantage. This advantage reduces as stronger 3379 compression is applied, as again would be expected, but even with the 3380 strongest compression applied the block formatted data remains around 3381 75% of the size of the simple format and its compression requires 3382 roughly a third of the CPU time. 3384 C.3. Binary versus text formats 3386 Text data formats offer many advantages over binary formats, 3387 particularly in the areas of ad-hoc data inspection and extraction. 3388 It was therefore felt worthwhile to carry out a direct comparison, 3389 implementing JSON versions of the simple and block formats. 3391 Concentrating on JSON block format, the format files produced are a 3392 significant fraction of an order of magnitude larger than binary 3393 formats. The impact on file size after compression is as might be 3394 expected from that starting point; the stronger compression produces 3395 files that are 150% of the size of similarly compressed binary 3396 format, and require over 4x more CPU to compress. 3398 C.4. Performance 3400 Concentrating again on the block formats, all three produce format 3401 files that are close to an order of magnitude smaller that the 3402 original "test.pcap" file. CBOR produces the largest files and Avro 3403 the smallest, 20% smaller than CBOR. 3405 However, once compression is taken into account, the size difference 3406 narrows. At medium compression (with gzip), the size difference is 3407 4%. Using strong compression (with xz) the difference reduces to 2%, 3408 with Avro the largest and Protocol Buffers the smallest, although 3409 CBOR and Protocol Buffers require slightly more compression CPU. 3411 The measurements presented above do not include data on the CPU 3412 required to generate the format files. Measurements indicate that 3413 writing Avro requires 10% more CPU than CBOR or Protocol Buffers. It 3414 appears, therefore, that Avro's advantage in compression CPU usage is 3415 probably offset by a larger CPU requirement in writing Avro. 3417 C.5. Conclusions 3419 The above assessments lead us to the choice of a binary format file 3420 using blocking. 3422 As noted previously, this draft anticipates that output data will be 3423 subject to compression. There is no compelling case for one 3424 particular binary serialisation format in terms of either final file 3425 size or machine resources consumed, so the choice must be largely 3426 based on other factors. CBOR was therefore chosen as the binary 3427 serialisation format for the reasons listed in Section 5. 3429 C.6. Block size choice 3431 Given the choice of a CBOR format using blocking, the question arises 3432 of what an appropriate default value for the maximum number of query/ 3433 response pairs in a block should be. This has two components; what 3434 is the impact on performance of using different block sizes in the 3435 format file, and what is the impact on the size of the format file 3436 before and after compression. 3438 The following table addresses the performance question, showing the 3439 impact on the performance of a C++ program converting "test.pcap" to 3440 C-DNS. File size is in Mb, resident set size (RSS) in kb. 3442 +------------+-----------+--------+-----------+ 3443 | Block size | File size | RSS | User time | 3444 +------------+-----------+--------+-----------+ 3445 | 1000 | 133.46 | 612.27 | 15.25 | 3446 | 5000 | 89.85 | 676.82 | 14.99 | 3447 | 10000 | 76.87 | 752.40 | 14.53 | 3448 | 20000 | 67.86 | 750.75 | 14.49 | 3449 | 40000 | 61.88 | 736.30 | 14.29 | 3450 | 80000 | 58.08 | 694.16 | 14.28 | 3451 | 160000 | 55.94 | 733.84 | 14.44 | 3452 | 320000 | 54.41 | 799.20 | 13.97 | 3453 +------------+-----------+--------+-----------+ 3455 Increasing block size, therefore, tends to increase maximum RSS a 3456 little, with no significant effect (if anything a small reduction) on 3457 CPU consumption. 3459 The following table demonstrates the effect of increasing block size 3460 on output file size for different compressions. 3462 +------------+--------+-------+-------+-------+-------+-------+ 3463 | Block size | None | snzip | lz4 | gzip | zstd | xz | 3464 +------------+--------+-------+-------+-------+-------+-------+ 3465 | 1000 | 133.46 | 90.52 | 90.03 | 74.65 | 44.78 | 25.63 | 3466 | 5000 | 89.85 | 59.69 | 59.43 | 46.99 | 37.33 | 22.34 | 3467 | 10000 | 76.87 | 50.39 | 50.28 | 38.94 | 33.62 | 21.09 | 3468 | 20000 | 67.86 | 43.91 | 43.90 | 33.24 | 32.62 | 20.16 | 3469 | 40000 | 61.88 | 39.63 | 39.69 | 29.44 | 28.72 | 19.52 | 3470 | 80000 | 58.08 | 36.93 | 37.01 | 27.05 | 26.25 | 19.00 | 3471 | 160000 | 55.94 | 35.10 | 35.06 | 25.44 | 24.56 | 19.63 | 3472 | 320000 | 54.41 | 33.87 | 33.74 | 24.36 | 23.44 | 18.66 | 3473 +------------+--------+-------+-------+-------+-------+-------+ 3475 There is obviously scope for tuning the default block size to the 3476 compression being employed, traffic characteristics, frequency of 3477 output file rollover etc. Using a strong compression scheme, block 3478 sizes over 10,000 query/response pairs would seem to offer limited 3479 improvements. 3481 Authors' Addresses 3483 John Dickinson 3484 Sinodun IT 3485 Magdalen Centre 3486 Oxford Science Park 3487 Oxford OX4 4GA 3488 United Kingdom 3490 Email: jad@sinodun.com 3492 Jim Hague 3493 Sinodun IT 3494 Magdalen Centre 3495 Oxford Science Park 3496 Oxford OX4 4GA 3497 United Kingdom 3499 Email: jim@sinodun.com 3500 Sara Dickinson 3501 Sinodun IT 3502 Magdalen Centre 3503 Oxford Science Park 3504 Oxford OX4 4GA 3505 United Kingdom 3507 Email: sara@sinodun.com 3509 Terry Manderson 3510 ICANN 3511 12025 Waterfront Drive 3512 Suite 300 3513 Los Angeles CA 90094-2536 3515 Email: terry.manderson@icann.org 3517 John Bond 3518 ICANN 3519 12025 Waterfront Drive 3520 Suite 300 3521 Los Angeles CA 90094-2536 3523 Email: john.bond@icann.org