idnits 2.17.1 draft-murchison-nntp-compress-06.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The expression 'MAY NOT', while looking like RFC 2119 requirements text, is not defined in RFC 2119, and should not be used. Consider using 'MUST NOT' instead (if that is what you mean). Found 'MAY NOT' in this paragraph: o Reworded a sentence wrongly using "MAY NOT" (not a key word defined in [RFC2119]). -- The document date (October 26, 2016) is 2732 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Missing Reference: 'C' is mentioned on line 453, but not defined == Missing Reference: 'S' is mentioned on line 454, but not defined -- Looks like a reference, but probably isn't: '1' on line 234 ** Downref: Normative reference to an Informational RFC: RFC 1951 ** Obsolete normative reference: RFC 5226 (Obsoleted by RFC 8126) -- Obsolete informational reference (is this intentional?): RFC 5246 (Obsoleted by RFC 8446) -- Obsolete informational reference (is this intentional?): RFC 7525 (Obsoleted by RFC 9325) Summary: 2 errors (**), 0 flaws (~~), 4 warnings (==), 4 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Independent Submission K. Murchison 3 Internet-Draft Carnegie Mellon University 4 Intended status: Standards Track J. Elie 5 Expires: April 29, 2017 October 26, 2016 7 Network News Transfer Protocol (NNTP) Extension for Compression 8 draft-murchison-nntp-compress-06 10 Abstract 12 This document defines an extension to the Network News Transport 13 Protocol (NNTP) that allows a connection to be effectively and 14 efficiently compressed between an NNTP client and server. 16 Status of This Memo 18 This Internet-Draft is submitted in full conformance with the 19 provisions of BCP 78 and BCP 79. 21 Internet-Drafts are working documents of the Internet Engineering 22 Task Force (IETF). Note that other groups may also distribute 23 working documents as Internet-Drafts. The list of current Internet- 24 Drafts is at http://datatracker.ietf.org/drafts/current/. 26 Internet-Drafts are draft documents valid for a maximum of six months 27 and may be updated, replaced, or obsoleted by other documents at any 28 time. It is inappropriate to use Internet-Drafts as reference 29 material or to cite them other than as "work in progress." 31 This Internet-Draft will expire on April 29, 2017. 33 Copyright Notice 35 Copyright (c) 2016 IETF Trust and the persons identified as the 36 document authors. All rights reserved. 38 This document is subject to BCP 78 and the IETF Trust's Legal 39 Provisions Relating to IETF Documents 40 (http://trustee.ietf.org/license-info) in effect on the date of 41 publication of this document. Please review these documents 42 carefully, as they describe your rights and restrictions with respect 43 to this document. Code Components extracted from this document must 44 include Simplified BSD License text as described in Section 4.e of 45 the Trust Legal Provisions and are provided without warranty as 46 described in the Simplified BSD License. 48 Table of Contents 50 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 51 1.1. About TLS-level Compression . . . . . . . . . . . . . . . 3 52 1.2. Conventions Used in This Document . . . . . . . . . . . . 4 53 1.3. Authors' Note . . . . . . . . . . . . . . . . . . . . . . 4 54 2. The COMPRESS Extension . . . . . . . . . . . . . . . . . . . 4 55 2.1. Advertising the COMPRESS Extension . . . . . . . . . . . 4 56 2.2. COMPRESS Command . . . . . . . . . . . . . . . . . . . . 5 57 2.2.1. Usage . . . . . . . . . . . . . . . . . . . . . . . . 5 58 2.2.2. Description . . . . . . . . . . . . . . . . . . . . . 6 59 2.2.3. Examples . . . . . . . . . . . . . . . . . . . . . . 7 60 3. Compression Efficiency . . . . . . . . . . . . . . . . . . . 10 61 4. DEFLATE Specificities . . . . . . . . . . . . . . . . . . . . 12 62 5. Augmented BNF Syntax for the COMPRESS Extension . . . . . . . 12 63 5.1. Commands . . . . . . . . . . . . . . . . . . . . . . . . 13 64 5.2. Capability Entries . . . . . . . . . . . . . . . . . . . 13 65 5.3. General Non-terminals . . . . . . . . . . . . . . . . . . 13 66 6. Summary of Response Codes . . . . . . . . . . . . . . . . . . 13 67 7. Security Considerations . . . . . . . . . . . . . . . . . . . 13 68 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 15 69 8.1. NNTP Compression Algorithm Registry . . . . . . . . . . . 15 70 8.1.1. Algorithm Name Registration Procedure . . . . . . . . 16 71 8.1.2. Comments on Algorithm Registrations . . . . . . . . . 16 72 8.1.3. Change Control . . . . . . . . . . . . . . . . . . . 17 73 8.2. Registration of the DEFLATE Compression Algorithm . . . . 17 74 8.3. Registration of the NNTP COMPRESS Extension . . . . . . . 18 75 9. References . . . . . . . . . . . . . . . . . . . . . . . . . 19 76 9.1. Normative References . . . . . . . . . . . . . . . . . . 19 77 9.2. Informative References . . . . . . . . . . . . . . . . . 20 78 Appendix A. Acknowledgments . . . . . . . . . . . . . . . . . . 22 79 Appendix B. Document History (to be removed by RFC Editor before 80 publication) . . . . . . . . . . . . . . . . . . . . 22 81 B.1. Changes since -05 . . . . . . . . . . . . . . . . . . . . 22 82 B.2. Changes since -04 . . . . . . . . . . . . . . . . . . . . 23 83 B.3. Changes since -03 . . . . . . . . . . . . . . . . . . . . 23 84 B.4. Changes since -02 . . . . . . . . . . . . . . . . . . . . 24 85 B.5. Changes since -01 . . . . . . . . . . . . . . . . . . . . 24 86 B.6. Changes since -00 . . . . . . . . . . . . . . . . . . . . 25 87 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 25 89 1. Introduction 91 The goal of COMPRESS is to reduce the bandwidth usage of NNTP. 93 Compared to PPP compression [RFC1962] and modem-based compression 94 ([MNP] and [V42bis]), COMPRESS offers greater compression efficiency. 95 COMPRESS can be used together with Transport Layer Security (TLS) 97 [RFC5246], Simple Authentication and Security Layer (SASL) encryption 98 [RFC4422], Virtual Private Networks (VPNs), etc. 100 The point of COMPRESS as an NNTP extension is to act as a compression 101 layer, similarly to a security layer like the one negotiated by 102 STARTTLS [RFC4642]. Compression can therefore benefit to all NNTP 103 commands sent or received after the use of COMPRESS. This facility 104 responds to a long-standing need for NNTP to compress data, that has 105 partially been addressed by unstandardized commands like XZVER, 106 XZHDR, XFEATURE COMPRESS, or MODE COMPRESS. These commands are not 107 wholly satisfactory because they enable compression only for the 108 responses sent by the news server. On the contrary, the COMPRESS 109 command permits to compress data sent by both the client and the 110 server, and removes the constraint of having to implement compression 111 separately in each NNTP command. Besides, the compression level can 112 be dynamically adjusted and optimized at any time during the 113 connection, which even allows to disable compression for certain 114 commands, if need be. If the news client wants to stop compression 115 on a particular connection, it can simply use QUIT ([RFC3977] 116 Section 5.4), and establish a new connection. For these reasons, 117 using other NNTP commands than COMPRESS to enable compression is 118 discouraged once COMPRESS is supported. 120 In order to increase interoperability, it is desirable to have as few 121 different compression algorithms as possible, so this document 122 specifies only one. The DEFLATE algorithm (defined in [RFC1951]) 123 MUST be implemented as part of this extension. This compression 124 algorithm is standard, widely available, and fairly efficient. 126 This specification should be read in conjunction with the NNTP base 127 specification [RFC3977]. In the case of a conflict between these two 128 documents, [RFC3977] takes precedence. 130 1.1. About TLS-level Compression 132 Though lossless data compression is already possible via the use of 133 TLS with NNTP [RFC4642], the best current practice is to disable TLS- 134 level compression as explained in Section 3.3 of [RFC7525]. The 135 COMPRESS command will permit to keep the compression facility in 136 NNTP, and control when it is available during a connection. 138 Compared to TLS-level compression [RFC3749], NNTP COMPRESS has the 139 following advantages: 141 o COMPRESS can be implemented easily both by NNTP servers and 142 clients. 144 o COMPRESS benefits from an intimate knowledge of the NNTP 145 protocol's state machine, allowing for dynamic and aggressive 146 optimization of the underlying compression algorithm's parameters. 148 o COMPRESS can be activated after authentication has completed, thus 149 reducing the chances that authentication credentials can be leaked 150 via for instance a CRIME attack ([RFC7457] Section 2.6). 152 1.2. Conventions Used in This Document 154 The notational conventions used in this document are the same as 155 those in [RFC3977], and any term not defined in this document has the 156 same meaning as it does in that one. 158 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 159 "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and 160 "OPTIONAL" in this document are to be interpreted as described in 161 [RFC2119]. 163 In the examples, commands from the client are indicated with [C], and 164 responses from the server are indicated with [S]. The client is the 165 initiator of the NNTP connection; the server is the other endpoint. 167 1.3. Authors' Note 169 Please write the first letter of "Elie" with an acute accent wherever 170 possible -- it is U+00C9 ("É" in XML). The third letter of 171 "Stephane" and the penultimate letter of "allee" similarly have an 172 acute accent (U+00E9, "é" in XML). Also, the letters "ae" in 173 "Baeuerle" should be written as an a-umlaut (U+00E4, "ä" in 174 XML), and the first letter of "Angel" as well as the fifth letter of 175 "Gonzalez" should be written with an acute accent (respectively 176 U+00C1 and U+00E1, that is to say "Á" and "á" in XML). 178 2. The COMPRESS Extension 180 The COMPRESS extension is used to enable lossless data compression on 181 an NNTP connection. 183 This extension provides a new COMPRESS command and has capability 184 label COMPRESS. 186 2.1. Advertising the COMPRESS Extension 188 A server supporting the COMPRESS command as defined in this document 189 will advertise the "COMPRESS" capability label in response to the 190 CAPABILITIES command ([RFC3977] Section 5.2). However, this 191 capability MUST NOT be advertised once a compression layer is active 192 (see Section 2.2.2). This capability MAY be advertised both before 193 and after any use of the MODE READER command ([RFC3977] Section 5.3), 194 with the same semantics. 196 The COMPRESS capability label contains a whitespace-separated list of 197 available compression algorithms. This document defines one 198 compression algorithm: DEFLATE. This algorithm is mandatory to 199 implement; it MUST be supported and listed in the advertisement of 200 the COMPRESS extension. 202 Future extensions may add additional compression algorithms to this 203 capability. Unrecognized algorithms MUST be ignored by the client. 205 Example: 207 [C] CAPABILITIES 208 [S] 101 Capability list: 209 [S] VERSION 2 210 [S] READER 211 [S] IHAVE 212 [S] COMPRESS DEFLATE SHRINK 213 [S] LIST ACTIVE NEWSGROUPS 214 [S] . 216 As the COMPRESS command is related to security because it can weaken 217 encryption, cached results of CAPABILITIES from a previous session 218 MUST NOT be relied on, as per Section 12.6 of [RFC3977]. 220 2.2. COMPRESS Command 222 2.2.1. Usage 224 This command MUST NOT be pipelined. 226 Syntax 227 COMPRESS algorithm 229 Responses 230 206 Compression active 231 403 Unable to activate compression 232 502 Command unavailable [1] 234 [1] If a compression layer is already active, COMPRESS is not a valid 235 command (see Section 2.2.2). 237 Parameters 238 algorithm = Name of compression algorithm (e.g., "DEFLATE") 240 2.2.2. Description 242 The COMPRESS command instructs the server to use the named 243 compression algorithm ("DEFLATE" is the only one defined in this 244 document) for all commands and responses after COMPRESS. 246 The client MUST NOT send any further commands until it has seen the 247 result of COMPRESS. 249 If the requested compression algorithm is syntactically incorrect, 250 the server MUST reject the COMPRESS command with a 501 response code 251 ([RFC3977] Section 3.2.1). If the requested compression algorithm is 252 invalid (e.g., is not supported), the server MUST reject the COMPRESS 253 command with a 503 response code ([RFC3977] Section 3.2.1). If the 254 server is unable to activate compression for any reason (e.g., a 255 server configuration or resource problem), the server MUST reject the 256 COMPRESS command with a 403 response code ([RFC3977] Section 3.2.1). 257 Otherwise, the server issues a 206 response code and the compression 258 layer takes effect for both client and server immediately following 259 the CRLF of the success reply. 261 Additionally, the client MUST NOT issue a MODE READER command after 262 activating a compression layer, and a server MUST NOT advertise the 263 MODE-READER capability. 265 Both the client and the server MUST know if there is a compression 266 layer active (for instance via the previous use of the COMPRESS 267 command or the negotiation of a TLS-level compression method 268 [RFC3749]). A client MUST NOT attempt to activate compression (for 269 instance via the COMPRESS command) or negotiate a TLS security layer 270 (because STARTTLS [RFC4642] may activate TLS-level compression) if a 271 compression layer is already active. A server MUST NOT return the 272 COMPRESS or STARTTLS capability labels in response to a CAPABILITIES 273 command received after a compression layer is active, and a server 274 MUST reply with a 502 response code if a syntactically valid COMPRESS 275 or STARTTLS command is received while a compression layer is already 276 active. 278 In order to help mitigate leaking authentication credentials via for 279 instance a CRIME attack [CRIME], authentication MUST NOT be attempted 280 after a successful use of the COMPRESS command. Consequently, a 281 server MUST either list the AUTHINFO capability with no arguments or 282 not advertise it at all, in response to a CAPABILITIES command 283 received from an unauthenticated client after a successful use of the 284 COMPRESS command, and such a client MUST NOT attempt to utilize any 285 AUTHINFO [RFC4643] commands. It implies that a server MUST reply 286 with a 502 response code if a syntactically valid AUTHINFO command is 287 received after a successful use of the COMPRESS command. (Note that 288 this specification does not change the behaviour of AUTHINFO as 289 described in [RFC4643] independently of TLS-level compression. 290 Authentication is therefore still allowed, even though TLS-level 291 compression is active.) 293 For DEFLATE [RFC1951] (as for many other compression algorithms), the 294 sending compressor can trade speed against compression ratio. The 295 receiving decompressor MUST automatically adjust to the parameters 296 selected by the sender. Consequently, the client and server are both 297 free to pick the best reasonable rate of compression for the data 298 they send. Besides, all data that was submitted for compression MUST 299 be included in the compressed output, and appropriately flushed so as 300 to ensure that the receiving decompressor can completely decompress 301 it. 303 When COMPRESS is combined with TLS [RFC5246] or SASL [RFC4422] 304 security layers, the processing order of the three layers MUST be 305 first COMPRESS, then SASL, and finally TLS. That is, before data is 306 transmitted, it is first compressed. Second, if a SASL security 307 layer has been negotiated, the compressed data is then signed and/or 308 encrypted accordingly. Third, if a TLS security layer has been 309 negotiated, the data from the previous step is signed and/or 310 encrypted accordingly (with a possible additional TLS-level 311 compression). When receiving data, the processing order MUST be 312 reversed. This ensures that before sending, data is compressed 313 before it is encrypted. 315 When compression is active and either the client or the server 316 receives invalid or corrupted compressed data, the receiving end 317 immediately closes the connection, in response to which the sending 318 end will do the same. 320 2.2.3. Examples 322 Example of layering a TLS security layer and NNTP compression: 324 [C] CAPABILITIES 325 [S] 101 Capability list: 326 [S] VERSION 2 327 [S] READER 328 [S] STARTTLS 329 [S] AUTHINFO 330 [S] COMPRESS DEFLATE 331 [S] LIST ACTIVE NEWSGROUPS 332 [S] . 333 [C] STARTTLS 334 [S] 382 Continue with TLS negotiation 335 [TLS negotiation without compression occurs here] 336 [Following successful negotiation, all traffic is encrypted] 337 [C] CAPABILITIES 338 [S] 101 Capability list: 339 [S] VERSION 2 340 [S] READER 341 [S] AUTHINFO USER 342 [S] COMPRESS DEFLATE 343 [S] LIST ACTIVE NEWSGROUPS 344 [S] . 345 [C] AUTHINFO USER fred 346 [S] 381 Enter passphrase 347 [C] AUTHINFO PASS flintstone 348 [S] 281 Authentication accepted 349 [C] CAPABILITIES 350 [S] 101 Capability list: 351 [S] VERSION 2 352 [S] READER 353 [S] POST 354 [S] COMPRESS DEFLATE 355 [S] LIST ACTIVE NEWSGROUPS 356 [S] . 357 [C] COMPRESS DEFLATE 358 [S] 206 Compression active 359 [Henceforth, all traffic is compressed before being encrypted] 360 [C] CAPABILITIES 361 [S] 101 Capability list: 362 [S] VERSION 2 363 [S] READER 364 [S] POST 365 [S] LIST ACTIVE NEWSGROUPS 366 [S] . 368 Example of a server failing to activate compression: 370 [C] CAPABILITIES 371 [S] 101 Capability list: 372 [S] VERSION 2 373 [S] IHAVE 374 [S] COMPRESS DEFLATE 375 [S] . 376 [C] COMPRESS DEFLATE 377 [S] 403 Unable to activate compression 379 Example of attempting to use an unsupported compression algorithm: 381 [C] CAPABILITIES 382 [S] 101 Capability list: 383 [S] VERSION 2 384 [S] IHAVE 385 [S] COMPRESS DEFLATE 386 [S] . 387 [C] COMPRESS SHRINK 388 [S] 503 Compression algorithm not supported 390 Example of a server refusing to compress twice: 392 [C] CAPABILITIES 393 [S] 101 Capability list: 394 [S] VERSION 2 395 [S] IHAVE 396 [S] STARTTLS 397 [S] COMPRESS DEFLATE 398 [S] . 399 [C] STARTTLS 400 [S] 382 Continue with TLS negotiation 401 [TLS negotiation with compression occurs here] 402 [Following successful negotiation, all traffic is encrypted] 403 [C] CAPABILITIES 404 [S] 101 Capability list: 405 [S] VERSION 2 406 [S] IHAVE 407 [S] . 408 [C] COMPRESS DEFLATE 409 [S] 502 Compression already active via TLS 411 Example of a server refusing to negotiate a TLS security layer after 412 compression has been activated: 414 [C] CAPABILITIES 415 [S] 101 Capability list: 416 [S] VERSION 2 417 [S] IHAVE 418 [S] STARTTLS 419 [S] COMPRESS DEFLATE 420 [S] . 421 [C] COMPRESS DEFLATE 422 [S] 206 Compression active 423 [Henceforth, all traffic is compressed] 424 [C] CAPABILITIES 425 [S] 101 Capability list: 426 [S] VERSION 2 427 [S] IHAVE 428 [S] . 429 [C] STARTTLS 430 [S] 502 DEFLATE compression already active 432 Example of a server not advertising AUTHINFO arguments after 433 compression has been activated: 435 [C] CAPABILITIES 436 [S] 101 Capability list: 437 [S] VERSION 2 438 [S] READER 439 [S] AUTHINFO USER 440 [S] COMPRESS DEFLATE 441 [S] LIST ACTIVE NEWSGROUPS 442 [S] . 443 [C] COMPRESS DEFLATE 444 [S] 206 Compression active 445 [Henceforth, all traffic is compressed] 446 [C] CAPABILITIES 447 [S] 101 Capability list: 448 [S] VERSION 2 449 [S] READER 450 [S] AUTHINFO 451 [S] LIST ACTIVE NEWSGROUPS 452 [S] . 453 [C] AUTHINFO USER fred 454 [S] 502 DEFLATE compression already active 456 3. Compression Efficiency 458 This section is informative, not normative. 460 NNTP poses some unusual problems for a compression layer. 462 Upstream traffic is fairly simple. Most NNTP clients send the same 463 few commands again and again, so any compression algorithm that can 464 exploit repetition works efficiently. The article posting and 465 transfer commands (e.g., POST, IHAVE, and TAKETHIS [RFC4644]) are 466 exceptions; clients that send many article posting or transfer 467 commands may want to surround large multi-line data blocks with a 468 dictionary flush and/or, depending on the compression algorithm, a 469 change of compression level in the same way as is recommended for 470 servers later in this document (Section 4). 472 Downstream traffic has the unusual property that several kinds of 473 data are sent, possibly confusing a dictionary-based compression 474 algorithm. 476 One type is NNTP responses not related to article header/body 477 retrieval. Compressing NNTP simple responses (e.g., in answer to 478 CHECK [RFC4644], DATE, GROUP, LAST, NEXT, STAT, etc.) generally does 479 not save many bytes, unless repeated several times in the same NNTP 480 session. On the contrary, most of NNTP multi-line responses (e.g., 481 in answer to LIST, LISTGROUP, NEWGROUPS, NEWNEWS, etc.) are highly 482 compressible; zlib using its least CPU-intensive setting compresses 483 typical responses to 25-40% of their original size. 485 Another type is article headers (as retrieved for instance via the 486 HEAD, HDR, OVER, or ARTICLE commands). These are equally 487 compressible, and benefit from using the same dictionary as the NNTP 488 responses. 490 A third type is article body text (as retrieved for instance via the 491 BODY or ARTICLE commands). Text is usually fairly short and includes 492 much ASCII, so the same compression dictionary will do a good job 493 here, too. When multiple messages in the same thread are read at the 494 same time, quoted lines, etc. can often be compressed almost to zero. 496 Finally, non-text article bodies or attachments (as retrieved for 497 instance via the BODY or ARTICLE commands) are transmitted in encoded 498 form, usually Base64 [RFC4648], UUencode [IEEE.1003-2.1992], or yEnc 499 [yEnc]. 501 When already compressed articles or attachments are retrieved, a 502 compression algorithm may be able to compress them, but the format of 503 their encoding is usually not NNTP-like, so the dictionary built 504 while compressing NNTP does not help much. The compressor has to 505 adapt its dictionary from NNTP to the attachment's encoding format, 506 and then back. 508 When attachments are retrieved in Base64 or UUencode form, the 509 Huffman coding usually compresses those to approximatively only 75% 510 of their encoding size. 8-bit compression algorithms such as DEFLATE 511 work well on 8-bit file formats; however, both Base64 and UUencode 512 transform a file into something resembling 6-bit bytes, hiding most 513 of the 8-bit file format from the compressor. 515 On the other end, attachments encoded using a compression algorithm 516 that retains the full 8-bit spectrum, like yEnc, are much more likely 517 to be incompressible. 519 4. DEFLATE Specificities 521 When using the zlib library (see [RFC1951]), the functions 522 deflateInit2(), deflate(), inflateInit2(), and inflate() suffice to 523 implement this extension. 525 The windowBits value MUST be in the range -8 to -15 for 526 deflateInit2(), or else it will use the wrong format. The windowBits 527 value SHOULD be -15 for inflateInit2(), or else it will not be able 528 to decompress a stream with a larger window size, thus reducing 529 interoperability. deflateParams() can be used to improve compression 530 rate and resource use. Regarding flush operations, the Z_FULL_FLUSH 531 argument to deflate() permits to clear the dictionary, which 532 generally results in compression that is less effective than 533 performing a Z_PARTIAL_FLUSH. As a matter of fact, keeping the 32kB 534 dictionary from previous data, no matter how unrelated, can be of 535 help (if there are no matching strings in there, then it is simply 536 not referenced). 538 A server can improve downstream compression and the CPU efficiency 539 both of the server and the client if it adjusts the compression level 540 (e.g., using the deflateParams() function in zlib) at the start and 541 end of large non-text multi-line data blocks (before and after 542 'content-lines' in the definition of 'multi-line-data-block' in 543 [RFC3977] Section 9.8). It permits to avoid trying to compress 544 incompressible attachments. 546 A very simple strategy is to change the compression level to 0 at the 547 start of an incompressible multi-line data block, for instance when 548 encoded using yEnc [yEnc], and to keep it at 1-5 the rest of the 549 time. More complex strategies are of course possible, and 550 encouraged. 552 5. Augmented BNF Syntax for the COMPRESS Extension 554 This section describes the formal syntax of the COMPRESS extension 555 using ABNF [RFC7405] [RFC5234]. It extends the syntax in Section 9 556 of [RFC3977], and non-terminals not defined in this document are 557 defined there. The [RFC3977] ABNF should be imported first, before 558 attempting to validate these rules. 560 5.1. Commands 562 This syntax extends the non-terminal , which represents an 563 NNTP command. 565 command =/ compress-command 567 compress-command = "COMPRESS" WS algorithm 569 5.2. Capability Entries 571 This syntax extends the non-terminal , which 572 represents a capability that may be advertised by the server. 574 capability-entry =/ compress-capability 576 compress-capability = "COMPRESS" 1*(WS algorithm) 578 5.3. General Non-terminals 580 algorithm = %s"DEFLATE" / 1*20alg-char ; case-sensitive 581 alg-char = UPPER / DIGIT / "-" / "_" 583 6. Summary of Response Codes 585 This section defines the following new response code. It is not 586 multi-line and has no arguments. 588 Response code 206 589 Generated by: COMPRESS 590 Meaning: compression layer activated 592 7. Security Considerations 594 Security issues are discussed throughout this document. 596 In general, the security considerations of the NNTP core 597 specification ([RFC3977] Section 12) and the DEFLATE compressed data 598 format specification ([RFC1951] Section 6) are applicable here. 600 Implementers should be aware that combining compression with 601 encryption like TLS can sometimes reveal information that would not 602 have been revealed without compression, as explained in Section 6 of 603 [RFC3749]. As a matter of fact, adversaries that observe the length 604 of the compressed data might be able to derive information about the 605 corresponding uncompressed data. The CRIME and the BREACH attacks 606 ([RFC7457] Section 2.6) are examples of such case. 608 In order to help mitigate leaking authentication credentials, this 609 document states in Section 2.2.2 that authentication MUST NOT be 610 attempted after a successful use of COMPRESS. Therefore, when a 611 client wants to authenticate, compress data, and negotiate a TLS 612 security layer (without TLS-level compression) in the same NNTP 613 connection, it MUST use the STARTTLS, AUTHINFO, and COMPRESS commands 614 in that order. Of course, instead of using the STARTTLS command, a 615 client can also use implicit TLS, that is to say it begins the TLS 616 negotiation immediately upon connection on a separate port dedicated 617 to NNTP over TLS. 619 NNTP commands other than AUTHINFO are not believed to divulge 620 confidential information as long as only public Netnews newsgroups 621 and articles are accessed. That is why this specification only 622 prohibits the use of AUTHINFO after COMPRESS. In case confidential 623 articles are accessed in private newsgroups, special care is needed: 624 implementations SHOULD NOT compress confidential data together with 625 public data when a TLS [RFC5246] or SASL [RFC4422] security layer is 626 active. As a matter of fact, adversaries that observe the length of 627 the compressed data might be able to derive information about it, 628 when public data (that adversaries know is read) and confidential 629 data are compressed in the same compress session. 631 Additionally, it is preferable not to compress the contents of two 632 distinct confidential articles together if it can be avoided, as 633 adversaries might be able to derive information about them (for 634 instance if they have a few header fields or body lines in common). 635 This can be achieved for instance with DEFLATE by clearing the 636 compression dictionary each time a confidential article is sent. 637 More complex implementations are of course possible, and encouraged. 639 Implementations are encouraged to unconditionally allow compression 640 when no security layer is active, and to support an option to enable 641 or disable compression when a security layer is active. Such an 642 option could for instance be with global scope or server/connection 643 based. Besides, as compression may in general weaken the 644 confidentiality of a security layer, implementations SHOULD NOT 645 automatically enable compression when a security layer is active 646 unless the user explicitly enabled it with this knowledge. 648 Future extensions to NNTP that define commands conveying confidential 649 data SHOULD ensure to state that these confidential data SHOULD NOT 650 be compressed together with public data when a security layer is 651 active. 653 Last but not least, careful consideration should be given to 654 protections against implementation errors that introduce security 655 risks with regards to compression algorithms. See for instance the 656 part of Section 6 of [RFC3749] about compression algorithms that can 657 occasionally expand, rather than compress, input data. 659 8. IANA Considerations 661 8.1. NNTP Compression Algorithm Registry 663 The NNTP Compression Algorithm registry will be maintained by IANA. 664 The registry will be available at . 667 The purpose of this registry is not only to ensure uniqueness of 668 values used to name NNTP compression algorithms, but also to provide 669 a definitive reference to technical specifications detailing each 670 NNTP compression algorithm available for use on the Internet. 672 An NNTP compression algorithm is either a private algorithm, or its 673 name is included in the IANA NNTP Compression Algorithm registry (in 674 which case it is a "registered NNTP compression algorithm"). 675 Different entries in the registry MUST use different names. 677 Private algorithms with unregistered names are allowed, but SHOULD 678 NOT be used because it is difficult to achieve interoperability with 679 them. 681 The 206, 403, and 502 response codes that a news server answers to 682 the COMPRESS command using a private compression algorithm MUST have 683 the same meaning as the one documented in Section 2.2 of this 684 document. 686 The procedure detailed in Section 8.1.1 is to be used for 687 registration of a value naming a specific individual compression 688 algorithm. 690 Any name that conforms to the syntax of an NNTP compression algorithm 691 name (Section 5.3) can be used. Especially, NNTP compression 692 algorithms are named by strings, from 1 to 20 characters in length, 693 consisting of upper-case letters, digits, hyphens, and/or 694 underscores. 696 Comments may be included in the registry as discussed in 697 Section 8.1.2 and may be changed as discussed in Section 8.1.3. 699 8.1.1. Algorithm Name Registration Procedure 701 IANA will register new NNTP compression algorithm names on a First 702 Come First Served basis, as defined in BCP 26 [RFC5226]. IANA has 703 the right to reject obviously bogus registration requests, but will 704 perform no review of claims made in the registration form. 706 Registration of an NNTP compression algorithm is requested by filling 707 in the following template and sending it via electronic mail to IANA 708 at : 710 Subject: Registration of NNTP compression algorithm Z 712 NNTP compression algorithm name: 714 Security considerations: 716 Published specification (recommended): 718 Contact for further information: 720 Intended usage: (One of COMMON, LIMITED USE, or OBSOLETE) 722 Owner/Change controller: 724 Note: (Any other information that the author deems relevant may be 725 added here.) 727 While this registration procedure does not require expert review, 728 authors of NNTP compression algorithms are encouraged to seek 729 community review and comment whenever that is feasible. Authors may 730 seek community review by posting a specification of their proposed 731 algorithm as an Internet-Draft. NNTP compression algorithms intended 732 for widespread use should be standardized through the normal IETF 733 process, when appropriate. 735 8.1.2. Comments on Algorithm Registrations 737 Comments on a registered NNTP compression algorithm should first be 738 sent to the "owner" of the algorithm and/or to the mailing list for 739 the now concluded IETF NNTPEXT working group 740 (). 742 Submitters of comments may, after a reasonable attempt to contact the 743 owner and/or the above mailing list, request IANA to attach their 744 comment to the NNTP compression algorithm registration itself by 745 sending mail to . At IANA's sole discretion, IANA may 746 attach the comment to the NNTP compression algorithm's registration. 748 8.1.3. Change Control 750 Once an NNTP compression algorithm registration has been published by 751 IANA, the owner may request a change to its definition. The change 752 request follows the same procedure as the initial registration 753 request. 755 The owner of an NNTP compression algorithm may pass responsibility 756 for the algorithm to another person or agency by informing IANA; this 757 can be done without discussion or review. 759 The IESG may reassign responsibility for an NNTP compression 760 algorithm. The most common case of this will be to enable changes to 761 be made to algorithms where the owner of the registration has died, 762 has moved out of contact, or is otherwise unable to make changes that 763 are important to the community. 765 NNTP compression algorithm registrations MUST NOT be deleted; 766 algorithms that are no longer believed appropriate for use can be 767 declared OBSOLETE by a change to their "intended usage" field; such 768 algorithms will be clearly marked in the registry published by IANA. 770 The IESG is considered to be the owner of all NNTP compression 771 algorithms that are on the IETF standards track. 773 8.2. Registration of the DEFLATE Compression Algorithm 775 This section gives a formal definition of the DEFLATE compression 776 algorithm as required by Section 8.1.1 for the IANA registry. 778 NNTP compression algorithm name: DEFLATE 780 Security considerations: See Section 7 of this document 782 Published specification: This document 784 Contact for further information: Authors of this document 786 Intended usage: COMMON 788 Owner/Change controller: IESG 790 Note: This algorithm is mandatory to implement 792 This registration will appear as follows in the NNTP Compression 793 Algorithm registry: 795 +----------------+----------------+---------------+ 796 | Algorithm Name | Intended Usage | Reference | 797 +----------------+----------------+---------------+ 798 | DEFLATE | COMMON | [ RFC-to-be ] | 799 +----------------+----------------+---------------+ 801 8.3. Registration of the NNTP COMPRESS Extension 803 This section gives a formal definition of the COMPRESS extension as 804 required by Section 3.3.3 of [RFC3977] for the IANA registry. 806 o The COMPRESS extension allows an NNTP connection to be effectively 807 and efficiently compressed. 809 o The capability label for this extension is "COMPRESS", whose 810 arguments list the available compression algorithms. 812 o This extension defines one new command, COMPRESS, whose behavior, 813 arguments, and responses are defined in Section 2.2. 815 o This extension does not associate any new responses with pre- 816 existing NNTP commands. 818 o This extension does affect the overall behavior of both server and 819 client, in that after successful use of the COMPRESS command, all 820 communication is transmitted in a compressed format. 822 o This extension does not affect the maximum length of commands or 823 initial response lines. 825 o This extension does not alter pipelining, but the COMPRESS command 826 cannot be pipelined. 828 o Use of this extension does alter the capabilities list; once the 829 COMPRESS command has been used successfully, the COMPRESS 830 capability can no longer be advertised by CAPABILITIES. 831 Additionally, the STARTTLS and MODE-READER capabilities MUST NOT 832 be advertised, and the AUTHINFO capability label MUST either be 833 listed with no arguments or not advertised at all after a 834 successful execution of the COMPRESS command. 836 o This extension does not cause any pre-existing command to produce 837 a 401, 480, or 483 response code. 839 o This extension is unaffected by any use of the MODE READER 840 command; however, the MODE READER command MUST NOT be used in the 841 same session following a successful execution of the COMPRESS 842 command. 844 o The STARTTLS and AUTHINFO commands MUST NOT be used in the same 845 session following a successful execution of the COMPRESS command. 847 o Published Specification: This document. 849 o Contact for Further Information: Authors of this document. 851 o Change Controller: IESG . 853 This registration will appear as follows in the NNTP capability 854 labels registry contained in the Network News Transfer Protocol 855 (NNTP) Parameters registry: 857 +----------+----------------------------------+---------------+ 858 | Label | Meaning | Reference | 859 +----------+----------------------------------+---------------+ 860 | COMPRESS | Supported compression algorithms | [ RFC-to-be ] | 861 +----------+----------------------------------+---------------+ 863 9. References 865 9.1. Normative References 867 [RFC1951] Deutsch, P., "DEFLATE Compressed Data Format Specification 868 version 1.3", RFC 1951, DOI 10.17487/RFC1951, May 1996, 869 . 871 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 872 Requirement Levels", BCP 14, RFC 2119, 873 DOI 10.17487/RFC2119, March 1997, 874 . 876 [RFC3977] Feather, C., "Network News Transfer Protocol (NNTP)", 877 RFC 3977, DOI 10.17487/RFC3977, October 2006, 878 . 880 [RFC5226] Narten, T. and H. Alvestrand, "Guidelines for Writing an 881 IANA Considerations Section in RFCs", BCP 26, RFC 5226, 882 DOI 10.17487/RFC5226, May 2008, 883 . 885 [RFC5234] Crocker, D., Ed. and P. Overell, "Augmented BNF for Syntax 886 Specifications: ABNF", STD 68, RFC 5234, 887 DOI 10.17487/RFC5234, January 2008, 888 . 890 [RFC7405] Kyzivat, P., "Case-Sensitive String Support in ABNF", 891 RFC 7405, DOI 10.17487/RFC7405, December 2014, 892 . 894 9.2. Informative References 896 [CRIME] Rizzo, J. and T. Duong, "The CRIME Attack", Ekoparty 897 Security Conference, 2012. 899 [IEEE.1003-2.1992] 900 Institute of Electrical and Electronics Engineers, 901 "Information Technology - Portable Operating System 902 Interface (POSIX) - Part 2: Shell and Utilities (Vol. 1)", 903 IEEE Standard 1003.2, 1992. 905 [MNP] Held, G., "The Complete Modem Reference", Second 906 Edition, Wiley Professional Computing, May 1994. 908 [RFC1962] Rand, D., "The PPP Compression Control Protocol (CCP)", 909 RFC 1962, DOI 10.17487/RFC1962, June 1996, 910 . 912 [RFC3749] Hollenbeck, S., "Transport Layer Security Protocol 913 Compression Methods", RFC 3749, DOI 10.17487/RFC3749, May 914 2004, . 916 [RFC4422] Melnikov, A., Ed. and K. Zeilenga, Ed., "Simple 917 Authentication and Security Layer (SASL)", RFC 4422, 918 DOI 10.17487/RFC4422, June 2006, 919 . 921 [RFC4642] Murchison, K., Vinocur, J., and C. Newman, "Using 922 Transport Layer Security (TLS) with Network News Transfer 923 Protocol (NNTP)", RFC 4642, DOI 10.17487/RFC4642, October 924 2006, . 926 [RFC4643] Vinocur, J. and K. Murchison, "Network News Transfer 927 Protocol (NNTP) Extension for Authentication", RFC 4643, 928 DOI 10.17487/RFC4643, October 2006, 929 . 931 [RFC4644] Vinocur, J. and K. Murchison, "Network News Transfer 932 Protocol (NNTP) Extension for Streaming Feeds", RFC 4644, 933 DOI 10.17487/RFC4644, October 2006, 934 . 936 [RFC4648] Josefsson, S., "The Base16, Base32, and Base64 Data 937 Encodings", RFC 4648, DOI 10.17487/RFC4648, October 2006, 938 . 940 [RFC4978] Gulbrandsen, A., "The IMAP COMPRESS Extension", RFC 4978, 941 DOI 10.17487/RFC4978, August 2007, 942 . 944 [RFC5246] Dierks, T. and E. Rescorla, "The Transport Layer Security 945 (TLS) Protocol Version 1.2", RFC 5246, 946 DOI 10.17487/RFC5246, August 2008, 947 . 949 [RFC7457] Sheffer, Y., Holz, R., and P. Saint-Andre, "Summarizing 950 Known Attacks on Transport Layer Security (TLS) and 951 Datagram TLS (DTLS)", RFC 7457, DOI 10.17487/RFC7457, 952 February 2015, . 954 [RFC7525] Sheffer, Y., Holz, R., and P. Saint-Andre, 955 "Recommendations for Secure Use of Transport Layer 956 Security (TLS) and Datagram Transport Layer Security 957 (DTLS)", BCP 195, RFC 7525, DOI 10.17487/RFC7525, May 958 2015, . 960 [V42bis] International Telecommunications Union, "Data compression 961 procedures for data circuit-terminating equipment (DCE) 962 using error correction procedures", ITU-T Recommendation 963 V.42bis, January 1990. 965 [yEnc] Helbing, J., "yEnc - Efficient encoding for Usenet and 966 eMail", March 2002, . 968 Appendix A. Acknowledgments 970 This document draws heavily on ideas in [RFC4978] by Arnt 971 Gulbrandsen; a large portion of this text was borrowed from that 972 specification. 974 The authors would like to thank the following individuals for 975 contributing their ideas and reviewing this specification: Mark 976 Adler, Russ Allbery, Stephane Bortzmeyer, Francis Dupont, Angel 977 Gonzalez, Barry Leiba, John Levine, and Brian Peterson. 979 Special thanks to our Document Shepherd, Michael Baeuerle, who 980 significantly helped to increase the quality of this specification. 982 Many thanks to the Responsible Area Director, Alexey Melnikov, for 983 reviewing and sponsoring this document. 985 Appendix B. Document History (to be removed by RFC Editor before 986 publication) 988 B.1. Changes since -05 990 o Take into account all the remarks sent during IETF Last Call. 992 o Do not prevent the registration of compression algorithm names 993 beginning with "X" (in conformance with RFC6648). Also, in the 994 examples, use "SHRINK" instead of "X-SHRINK". 996 o Separate Section 3 and Section 4 because the latter uses normative 997 keywords. Also improve and simplify the wording of these two 998 Sections, notably by distinguishing NNTP simple responses and NNTP 999 multi-line responses, and by not being categorical that 1000 Z_PARTIAL_FLUSH is the best and only flush operation to use. 1002 o Do not declare non-compliant an implementation that only supports 1003 COMPRESS when there is no security layer. 1005 o Move [RFC1951] reference to normative, and [RFC4642] reference to 1006 informative. 1008 o Explain why STARTTLS is not allowed after COMPRESS. 1010 o Improve security by stating that authentication MUST NOT be 1011 attempted after COMPRESS has been successfully executed. Do not 1012 change, though, the behaviour of AUTHINFO as described in 1013 [RFC4643], allowing authentication even though TLS-level 1014 compression is active. 1016 o Require that all data that was submitted for compression MUST be 1017 included in the compressed output, and appropriately flushed. 1019 o Mention possible security risks with regards to compression 1020 algorithms. 1022 o Mention that using unregistered algorithms decrease 1023 interoperability. 1025 o Mention the exact contents of the IANA registrations asked by this 1026 document. 1028 o NNTP compression algorithm names really are case-sensitive. 1029 Update ABNF syntax accordingly. 1031 o Minor other wording improvements. 1033 B.2. Changes since -04 1035 o Reworded a sentence wrongly using "MAY NOT" (not a key word 1036 defined in [RFC2119]). 1038 o Uppercased a "must" and a "should" in Section 4. 1040 B.3. Changes since -03 1042 o Added a naming convention for NNTP compression algorithms. 1043 Improve the wording of registered vs private compression 1044 algorithms. 1046 o If a registered NNTP compression algorithm is advertised, it MUST 1047 fully conform with its related specification. 1049 o Fixed the wording of security considerations to reflect that the 1050 threat appears when public and confidential data are compressed 1051 together inside a security layer. Thanks to Angel Gonzalez for 1052 pointing that. 1054 o The default configuration SHOULD be disabled compression when a 1055 security layer is active. 1057 o COMPRESS acts as a compression layer, not a transport layer. 1059 o Minor editorial changes. 1061 B.4. Changes since -02 1063 o Added text stating that the receiving end SHOULD terminate the 1064 connection when receiving invalid or corrupted compressed data. 1066 o Explained why COMPRESS permits to do better than existing 1067 unstandardized commands like XZVER, XZHDR, MODE COMPRESS, and 1068 XFEATURE GZIP. 1070 o Added an example of AUTHINFO command when compression is active. 1072 o The LIST capability label was missing in the examples when READER 1073 was also advertised. 1075 o Improved an example to send CAPABILITIES after successful 1076 authentication. 1078 o Mentioned that COMPRESS is related to security. CAPABILITIES is 1079 therefore sent again after COMPRESS. 1081 o Re-added discussion of attachments in binary form and 1082 incompressible file formats. Improve the discussion about 1083 flushes, and add a specific section about DEFLATE. 1085 o Changed a MUST NOT to SHOULD NOT for the use of AUTHINFO after 1086 COMPRESS. 1088 o Algorithm names are case-insensitive. 1090 o Mentioned the use of the 501 response code. 1092 o Added the Security Considerations Section. 1094 o Added Julien Elie as co-author of this document. 1096 o Minor editorial changes. 1098 B.5. Changes since -01 1100 o Switched to using 206 response code when compression has been 1101 activated. 1103 o Added text stating that TLS-level compression is susceptible to 1104 CRIME attack and current BCP is to disable it. 1106 o Added text stating that AUTHINFO shouldn't be advertised or used 1107 after COMPRESS to prevent possible CRIME attack (with example). 1109 o Added text stating that a windowBits value of -15 should be used 1110 for inflateInit2(). 1112 o Minor editorial changes. 1114 B.6. Changes since -00 1116 o Made DEFLATE the mandatory to implement compression algorithm. 1118 o Removed the requirement that clients/servers implementing COMPRESS 1119 also implement TLS compression. 1121 o Added an example of a client trying to use an unsupported 1122 compression algorithm. 1124 o Rewrote Compression Efficiency (Section 3) as follows: 1126 * Included a sample listing of which NNTP commands produce which 1127 type of data to be compressed. 1129 * Removed discussion of attachments in binary form and 1130 incompressible file formats. 1132 * Mentioned UUencode and yEnc encoding of attachments. 1134 o Added IANA registry of NNTP compression algorithms. 1136 o Miscellaneous editorial changes submitted by Julien Elie. 1138 Authors' Addresses 1140 Kenneth Murchison 1141 Carnegie Mellon University 1142 5000 Forbes Avenue 1143 Pittsburgh, PA 15213 1144 USA 1146 Phone: +1 412 268 1982 1147 EMail: murch@andrew.cmu.edu 1149 Julien Elie 1150 10 allee Clovis 1151 Noisy-le-Grand 93160 1152 France 1154 EMail: julien@trigofacile.com 1155 URI: http://www.trigofacile.com/