idnits 2.17.1 draft-ietf-usefor-usepro-05.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** It looks like you're using RFC 3978 boilerplate. You should update this to the boilerplate described in the IETF Trust License Policy document (see https://trustee.ietf.org/license-info), which is required now. -- Found old boilerplate from RFC 3978, Section 5.1 on line 17. -- Found old boilerplate from RFC 3978, Section 5.5 on line 2751. -- Found old boilerplate from RFC 3979, Section 5, paragraph 1 on line 2722. -- Found old boilerplate from RFC 3979, Section 5, paragraph 2 on line 2729. -- Found old boilerplate from RFC 3979, Section 5, paragraph 3 on line 2735. ** This document has an original RFC 3978 Section 5.4 Copyright Line, instead of the newer IETF Trust Copyright according to RFC 4748. ** This document has an original RFC 3978 Section 5.5 Disclaimer, instead of the newer disclaimer which includes the IETF Trust according to RFC 4748. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- ** Missing expiration date. The document expiration date should appear on the first and last page. ** The document seems to lack a 1id_guidelines paragraph about Internet-Drafts being working documents. == No 'Intended status' indicated for this document; assuming Proposed Standard Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** There is 1 instance of too long lines in the document, the longest one being 2 characters in excess of 72. ** The abstract seems to contain references ([USEFOR], [NNTP], [USEAGE]), which it shouldn't. Please replace those with straight textual mentions of the documents in question. == There are 12 instances of lines with non-RFC2606-compliant FQDNs in the document. == There are 1 instance of lines with non-RFC6890-compliant IPv4 addresses in the document. If these are example addresses, they should be changed. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the RFC 3978 Section 5.4 Copyright Line does not match the current year == The document seems to lack the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords. (The document does seem to have the reference to RFC 2119 which the ID-Checklist requires). -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (January 2006) is 6674 days in the past. Is this intentional? -- Found something which looks like a code comment -- if you have code sections in the document, please surround them with '' and '' lines. Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Missing Reference: 'FWS' is mentioned on line 1270, but not defined == Missing Reference: 'ARTICLE' is mentioned on line 2760, but not defined ** Obsolete normative reference: RFC 2048 (Obsoleted by RFC 4288, RFC 4289) ** Obsolete normative reference: RFC 2822 (Obsoleted by RFC 5322) -- No information found for draft-ietf-usefor-useage- - is the name correct? -- Possible downref: Normative reference to a draft: ref. 'USEAGE' -- No information found for draft-ietf-usefor-usefor- - is the name correct? -- Possible downref: Normative reference to a draft: ref. 'USEFOR' -- Possible downref: Non-RFC (?) normative reference: ref. 'USEPRO' -- No information found for draft-ietf-nntpext-base- - is the name correct? -- Obsolete informational reference (is this intentional?): RFC 1036 (Obsoleted by RFC 5536, RFC 5537) -- Obsolete informational reference (is this intentional?): RFC 2298 (Obsoleted by RFC 3798) -- Obsolete informational reference (is this intentional?): RFC 2616 (Obsoleted by RFC 7230, RFC 7231, RFC 7232, RFC 7233, RFC 7234, RFC 7235) -- Obsolete informational reference (is this intentional?): RFC 2821 (Obsoleted by RFC 5321) Summary: 9 errors (**), 0 flaws (~~), 7 warnings (==), 18 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 INTERNET-DRAFT Charles H. Lindsey 2 Usenet Format Working Group University of Manchester 3 January 2006 5 News Article Architecture and Protocols 6 8 Status of this Memo 10 By submitting this Internet-Draft, each author represents that any 11 applicable patent or other IPR claims of which he or she is aware 12 have been or will be disclosed, and any of which he or she becomes 13 aware will be disclosed, in accordance with Section 6 of BCP 79. 14 .QP Internet-Drafts are working documents of the Internet 15 Engineering Task Force (IETF), its areas, and its working groups. 16 Note that other groups may also distribute working documents as 17 Internet-Drafts. 19 Internet-Drafts are draft documents valid for a maximum of six 20 months and may be updated, replaced, or obsoleted by other 21 documents at any time. It is inappropriate to use Internet-Drafts 22 as reference material or to cite them other than as "work in 23 progress." 25 The list of current Internet-Drafts can be accessed at 26 http://www.ietf.org/ietf/1id-abstracts.html. 28 The list of Internet-Draft Shadow Directories can be accessed at 29 http://www.ietf.org/shadow.html. 31 This Internet-Draft will expire in July 2006. 33 Abstract 35 This Draft, together with its companion draft [USEFOR], are 36 intended as standards track documents, together obsoleting RFC 37 1036, which itself dates from 1987. 39 This Standard defines the architecture of Netnews systems and 40 specifies the requirements to be met by software which originates, 41 distributes, stores and displays Netnews articles. 43 Backward compatibility has been a major goal of this endeavour, but 44 where this standard and earlier documents or practices conflict, this 45 standard should be followed. In most such cases, current practice is 46 already compatible with these changes. 48 A companion Best Current Practice document [USEAGE], addressing 49 requirements which are present for Social rather than Normative 50 reasons is in preparation. 52 [The use of the words "this standard" within this document when 53 referring to itself does not imply that this draft yet has pretensions 54 to be a standard, but rather indicates what will become the case if and 55 News Article Architecture and Protocols January 2006 57 when it is accepted as an RFC with the status of a proposed or draft 58 standard.] 60 [Remarks enclosed in square brackets and aligned with the left margin, 61 such as this one, are not part of this draft, but are editorial notes to 62 explain matters amongst ourselves, or to point out alternatives, or to 63 assist the RFC Editor.] 65 [In this draft, references to [NNTP] are to be replaced by references to 66 the RFC arising from the series of drafts draft-ietf-nntpext-base-*.txt, 67 which has now passed its IETF last call.] 69 Table of Contents 71 1. Introduction .................................................. 4 72 1.1. Basic Concepts ............................................ 4 73 1.2. Objectives ................................................ 4 74 1.3. Historical Outline ........................................ 5 75 2. Definitions, Notations and Conventions ........................ 5 76 2.1. Definitions ............................................... 5 77 2.2. Defining the Architecture ................................. 6 78 2.3. Identification of news servers ............................ 7 79 2.4. Variant Header Fields ..................................... 8 80 2.5. Textual Notations ......................................... 8 81 3. Changes to the existing protocols ............................. 9 82 3.1. Protocol Changes .......................................... 9 83 3.2. Transitional Arrangements ................................. 10 84 4. Transport ..................................................... 11 85 5. Definition of new Media Types ................................. 12 86 5.1. Application/news-transmission ............................. 12 87 5.2. Message/news obsoleted .................................... 13 88 5.3. Application/news-groupinfo ................................ 13 89 5.4. Application/news-checkgroups .............................. 14 90 6. Control Messages .............................................. 15 91 6.1. Digital Signature of Header Fields ........................ 16 92 6.2. Group Control Messages .................................... 16 93 6.2.1. The 'newgroup' Control Message ........................ 16 94 6.2.1.1. The Body of the 'newgroup' Control Message ........ 17 95 6.2.1.2. Initial Articles .................................. 17 96 6.2.1.3. Example ........................................... 18 97 6.2.2. The 'rmgroup' Control Message ......................... 19 98 6.2.2.1. Example ........................................... 19 99 6.2.3. The 'mvgroup' Control Message ......................... 19 100 6.2.3.1. Example ........................................... 21 101 6.2.4. The 'checkgroups' Control Message ..................... 21 102 6.3. Cancel .................................................... 23 103 6.4. Ihave, sendme ............................................. 23 104 6.5. Obsolete control messages. ............................... 25 105 7. Duties of Various Agents ...................................... 25 106 7.1. General principles to be followed ......................... 26 107 7.2. Duties of an Injecting Agent .............................. 26 108 7.2.1. Proto-articles ........................................ 27 109 7.2.2. Procedure to be followed by Injecting Agents .......... 27 110 News Article Architecture and Protocols January 2006 112 7.2.3. Procedure for Forwarding to a Moderator ............... 29 113 7.3. Duties of a Relaying Agent ................................ 30 114 7.3.1. Path Header Field Example ............................. 33 115 7.4. Duties of a Serving Agent ................................. 34 116 7.5. Duties of a Posting Agent ................................. 35 117 7.6. Duties of a Followup Agent ................................ 36 118 7.6.1. Construction of the References header field ........... 36 119 7.7. Duties of a Reading Agent ................................. 37 120 7.8. Duties of a Moderator ..................................... 37 121 7.9. Duties of a Gateway ....................................... 39 122 7.9.1. Duties of an Outgoing Gateway ......................... 40 123 7.9.2. Duties of an Incoming Gateway ......................... 41 124 7.9.3. Example ............................................... 43 125 8. Security and Related Considerations ........................... 44 126 8.1. Leakage ................................................... 44 127 8.2. Attacks ................................................... 44 128 8.2.1. Denial of Service ..................................... 44 129 8.2.2. Compromise of System Integrity ........................ 46 130 8.3. Liability ................................................. 47 131 9. IANA Considerations ........................................... 47 132 10. References ................................................... 47 133 10.1. Normative References ..................................... 47 134 10.2. Informative References ................................... 48 135 11. Acknowledgements ............................................. 49 136 12. Contact Address .............................................. 49 137 Appendix A - Obsolete Control Messages ............................ 49 138 Appendix B - Notices .............................................. 50 139 Appendix C - Change Log ........................................... 51 140 News Article Architecture and Protocols January 2006 142 1. Introduction 144 1.1. Basic Concepts 146 "Netnews" is a set of protocols for generating, storing and 147 retrieving news "articles" (which resemble email messages) and for 148 exchanging them amongst a readership which is potentially widely 149 distributed. It is organized around "newsgroups", with the 150 expectation that each reader will be able to see all articles posted 151 to each newsgroup in which he participates. These protocols most 152 commonly use a flooding algorithm which propagates copies throughout 153 a network of participating servers. Typically, only one copy is 154 stored per server, and each server makes it available on demand to 155 readers able to access that server. 157 "Usenet" is a particular worldwide publicly accessible network based 158 upon the Netnews protocols, with the newsgroups being organized into 159 recognized "hierarchies". Anybody can join (it is simply necessary 160 to negotiate an exchange of articles with one or more other 161 participating hosts). 163 An important characteristic of Usenet is the lack of any requirement 164 for a central administration or for the establishment of any 165 controlling host to manage the network. Nevertheless, administrative 166 agencies do exists with varying degrees of authority to establish 167 "policies" applicable to particular parts of Usenet. 169 A "policy" is a rule intended to facilitate the smooth operation of a 170 network by establishing parameters which restrict behaviour that, 171 whilst technically unexceptionable, would nevertheless contravene 172 some accepted standard of "Good Netkeeping". Since the ultimate 173 beneficiaries of a network are its human readers, who will be less 174 tolerant of poorly designed interfaces than mere computers, articles 175 in breach of established policy can cause considerable annoyance to 176 their recipients. 177 [Could omit that last sentence.] 179 1.2. Objectives 181 The purpose of this present standard is to define the overall 182 architecture and the protocols to be used for Netnews in general, and 183 for Usenet in particular, and to set standards to be followed by 184 software that implements those protocols. A companion standard 185 [USEFOR] sets out the canonical format of news articles exchanged 186 between the various agents comprising that architecture. In this 187 standard, references to individual sections in the companion [USEFOR] 188 are prefixed with "F-". 190 A set of hosts within a network which, by mutual arrangement, 191 operates some variant (whether more or less restrictive) of the 192 Netnews protocols is a "cooperating subnet". 193 [It is not clear whether we still need that definition.] 194 News Article Architecture and Protocols January 2006 196 It is NOT the purpose of this standard to settle matters of policy, 197 nor aspects of software behaviour which do not impinge upon the 198 generation, transmission, storage and reception of articles, nor how 199 the authority of various agencies to create such policies and to 200 exercise control or oversight of the various parts of Usenet is 201 established. For these purposes, a separate Best Current Practice 202 document [USEAGE] is being provided. 204 Nevertheless, it is assumed that such agencies with the necessary 205 authority will exist, and tools are provided within the protocols for 206 their use. 208 1.3. Historical Outline 210 Network news originated as the medium of communication for Usenet, 211 circa 1980. Since then, Usenet has grown explosively, and many 212 Internet and non-Internet sites participate in it. In addition, the 213 news technology is now in widespread use for other purposes, on the 214 Internet and elsewhere. 216 For an account of the earlier formats used in Netnews prior to [RFC 217 1036], see Henry Spencer's 1994 draft, popularly referred to as "Son 218 of 1036" [Son-of-1036], which has recently been republished as an 219 Informational RFC. 220 [That is a tentative statement, which may need revision.] 222 Although never adopted as a formal standard, [Son-of-1036] had a 223 considerable effect on the development of Netnews and hence on these 224 present standards, and it is hoped that we have followed its spirit 225 and intentions. 227 2. Definitions, Notations and Conventions 229 2.1. Definitions 231 All the technical terms defined in F-1.5 are to be considered as 232 defined also, with the same meaning, in this standard. In addition, 233 some further terms are defined here, and in the following section. 235 A "hierarchy" is the set of all newsgroups whose names share a first 236 (as defined in F-3.1.5). The term "sub-hierarchy" is 237 also used where several initial components are shared. 239 The "semantic content" (often abbreviated to just "content" when the 240 context is clear) of a header field is its semantic interpretation; 241 i.e. what remains after unfolding it and removing its field name with 242 its colon and any leading and trailing whitespace and, in the case of 243 structured header fields only, ignoring comments and other 244 semantically invisible items and replacing white space by a single 245 SP. 247 News Article Architecture and Protocols January 2006 249 2.2. Defining the Architecture 251 A Netnews system is a distributed database composed of agents of 252 various types which, acting together according to the protocols 253 defined in section 7 of this standard, causes articles to be 254 propagated throughout the system and to be made available to its 255 readers. The protocols ensure that all copies of a given article, 256 wherever stored, are identical apart from those header fields defined 257 as variant (2.4). For explaining the working of the protocols, it is 258 convenient to define particular sub-categories of agent as follows: 260 A "posting agent" is the software that assists posters to prepare 261 proto-articles in compliance with [USEFOR]. The proto-article is 262 then passed on to an "injecting agent" for final checking and 263 injection into the news stream. If the article is not compliant, or 264 is rejected by the injecting agent, then the posting agent informs 265 the poster with an explanation of the error. 267 A "reading agent" is software which presents articles to a reader. 269 A "followup agent" is a combination of reading agent and posting 270 agent that aids in the preparation and posting of a followup. 272 An "injecting agent" takes the finished article from the posting 273 agent (often via the NNTP "POST" command), performs some final checks 274 and passes it on to a "relaying agent" for general distribution. 276 A "relaying agent" is software which receives allegedly compliant 277 articles from injecting agents and/or other relaying agents, and 278 possibly passes copies on to other relaying agents and "serving 279 agents". 281 A "serving agent" receives an article from a relaying agent and files 282 it in a "news database". It also provides an interface for reading 283 agents to access the news database. 284 [There is a suggestion that "serving agent" should be changed to 285 "storage agent" throughout.] 287 A "news database" is the set of articles and related structural 288 information stored by a serving agent and made available for access 289 by reading agents. 291 A "gateway" is software which receives news articles and converts 292 them to messages of some other kind (e.g. mail to a mailing list), or 293 vice versa; in essence it is a translating relaying agent that 294 straddles boundaries between different methods of message exchange. 295 The most common type of gateway connects newsgroup(s) to mailing 296 list(s), either unidirectionally or bidirectionally, but there are 297 also gateways between news networks using the [USEFOR] news format 298 and those using other formats. 300 Posting, reading and followup agents (which are usually just 301 different services provided by the same piece of software) together 302 comprise the "user agents" defined in F-1.5. 304 News Article Architecture and Protocols January 2006 306 Likewise, injecting, relaying and serving agents (which are often 307 just different services provided by the same piece of software) 308 together comprise the "news servers". 310 2.3. Identification of news servers 312 [The format of the Path header is still under discussion (ticket #1047). 313 Hence the following texts are tentative, and will need to be changed (as 314 will the associated protocols in 7.3). Moreover, there are two 315 alternative texts which have been proposed:] 317 In order to record the passage of articles through the network, news 318 servers need to identify themselves by means of a 319 (F-3.1.6), which can appear in Path, Injection-Info and Xref header 320 fields. Whatever is used in the Path header field 321 SHOULD be used also in any Injection-Info header field (and it would 322 be normal to use it in any Xref header field also). 323 [Maybe that last sentence moves elsewhere.] 325 NOTE: Such s may also be suitable for sending 326 email to news server administrators (see [USEAGE]). 328 [1st alternative] 330 s can take the following forms (in decreasing order of 331 preference): 333 1. 1. A fully qualified domain name (FQDN) that SHOULD be resolvable 334 in the DNS (whether via an A, AAAA or MX record or an equivalent 335 CNAME), thus guaranteeing a unique identity. Ideally, it will also 336 provide a means to contact the administrators by email (according 337 to [RFC 2142], the forms "usenet@server" and "news@server" are 338 common addresses for a news server administrator). 340 2. Some other (arbitrary) name believed to be unique and registered 341 at least with all other news servers sending articles directly to 342 the given one. This option SHOULD NOT be used unless the earlier 343 option is unavailable (e.g. because the server in question is not 344 connected to the Internet), or unless it is of longstanding usage 345 and cessation would be unduly disruptive, or unless the earlier 346 option is provided as well. 348 [2nd alternative] 350 s can take the following forms (in decreasing order of 351 preference): 353 1. A fully qualified domain name (FQDN) that can be resolved to an 354 email server via an MX, A or AAAA record according to the 355 procedures of [RFC 2821]; this guarantees that the name is unique, 356 and makes it easy to contact the administrators if needed. 358 News Article Architecture and Protocols January 2006 360 2. A fully qualified domain name (FQDN) that is guaranteed to be 361 unique by the administrators of the domain; for instance, the 362 uniqueness of "server.example.org" could be guaranteed by the 363 administrator of "example.org" even if nothing is stored in the 364 DNS for that name. 366 3. Some other (arbitrary) name believed to be unique and registered 367 at least with all other news-servers sending articles directly to 368 the given one. This option SHOULD NOT be used unless the earlier 369 options are unavailable, or unless the name is of longstanding 370 usage and cessation would be unduly disruptive, or unless one of 371 the earlier options is provided as well. 373 According to [RFC 2142]], the forms "usenet@server" and "news@server" 374 are common addresses for a news server administrator. 375 [end of alternatives] 377 NOTE: A news server administrator who chooses a name which turns 378 out not to be unique will have to bear the consequences. 380 NOTE: The syntax permits the colon character (which, prior to 381 this standard, was a ) within any which is in the form of an . It would 383 therefore be unwise to choose, as such a name, anything composed 384 solely from four (or less) hexadecimal digits. 386 2.4. Variant Header Fields 388 Header fields with the variant property may differ between (or even 389 be completely absent from) copies of the same article as stored or 390 relayed throughout a Netnews system. The manner of the difference (or 391 absence) MUST be as specified in this (or some future) standard. 392 Typically, these header fields are modified as articles are 393 propagated, or they reflect the status of the article on a particular 394 serving agent, or cooperating group of such agents. A variant header 395 field MAY be placed anywhere within the header fields (though placing 396 it first is recommended). 398 The following header fields are classified as "variant": 399 o Path (F-3.1.6) - augmented at each relaying agent that an article 400 passes through. 401 o Xref (F-3.2.11) - used to keep track of the s of 402 crossposted articles so that reading agents serviced by a 403 particular serving agent can mark such articles as read. 404 o Injection-Info (F-3.2.14) is also considered variant in some 405 special situations involving reinjection (7.2 and 7.2.2). 407 2.5. Textual Notations 409 This standard contains explanatory NOTEs using the following format. 410 These may be skipped by persons interested solely in the content of 411 the specification. The purpose of the notes is to explain why choices 412 were made, to place them in context, or to suggest possible 413 implementation techniques. 415 News Article Architecture and Protocols January 2006 417 NOTE: While such explanatory notes may seem superfluous in 418 principle, they often help the less-than-omniscient reader grasp 419 the purpose of the specification and the constraints involved. 420 Given the limitations of natural language for descriptive 421 purposes, this improves the probability that implementors and 422 users will understand the true intent of the specification in 423 cases where the wording is not entirely clear. 425 "US-ASCII" is short for "the ANSI X3.4 character set" [ANSI X3.4]. 426 US-ASCII is a 7 bit character set. Please note that this standard 427 requires that all agents be 8 bit clean; that is, they must accept 428 and transmit data without changing or omitting the 8th bit. 430 Certain words, when capitalized, are used to define the significance 431 of individual requirements. The key words "MUST", "REQUIRED", 432 "SHOULD", "RECOMMENDED", "MAY" and "OPTIONAL", and any of those words 433 associated with the word "NOT", are to be interpreted as described in 434 [RFC 2119]. 436 NOTE: A requirement imposed on a relaying or serving agent 437 regarding some particular article should be understood as 438 applying only if that article is actually accepted for 439 processing (since any agent may always reject any article 440 entirely, for reasons of site policy). 442 Wherever the context permits, use of the masculine includes the 443 feminine and use of the singular includes the plural, and vice versa. 445 Throughout this standard we will give examples of various 446 definitions, header fields and other specifications. It needs to be 447 remembered that these samples are for the aid of the reader only and 448 do NOT define any specification themselves. In order to prevent 449 possible conflict with "Real World" entities and people the top level 450 domain ".example" is used in all sample domains and addresses. The 451 hierarchy "example.*" is also used as a sample hierarchy. 452 Information on the ".example" top level domain is in [RFC 2606]. 454 3. Changes to the existing protocols 456 This standard prescribes many changes, clarifications and new 457 features since the protocols described in [RFC 1036] and [Son-of- 458 1036]. It is the intention that they can be assimilated into Usenet 459 as it presently operates without major interruption to the service 460 (3.2), though some of the new features may not begin to show benefit 461 until they become widely implemented. Changes in the syntax and 462 format are documented in F-Appendix B and changes to control messages 463 and the protocols are documented below. 465 3.1. Protocol Changes 467 o There is a new Control message 'mvgroup' to facilitate moving a 468 group to a different place (name) in a hierarchy. 469 o Certain Control messages (Appendix A) have been made obsolete, 470 and the special significance of "cmsg" when at the start of a 471 News Article Architecture and Protocols January 2006 473 Subject header field has been removed (section 6). 474 o Additional media types are defined for better structuring of 475 control messages (5.3 and 5.4). 476 o Distributions are expected to be checked at the receiving end, as 477 well as the sending end, of a relaying link. 478 o There are numerous other small changes, clarifications and 479 enhancements. 481 3.2. Transitional Arrangements 483 An important distinction must be made between news servers, which are 484 responsible for the distribution and storage of news articles, and 485 user agents, which are responsible for interactions with users. It is 486 important that the former should be upgraded to conform to this 487 standard as soon as possible to provide the benefit of the enhanced 488 facilities. Fortunately, the number of distinct implementations of 489 such servers is rather small, at least so far as the main "backbone" 490 of Usenet is concerned, and many of the new features are already 491 supported. Contrariwise, there are a great number of implementations 492 of user agents, installed on a vastly greater number of small sites. 493 Therefore, the new functionality has been designed so that existing 494 user agents may continue to be used, although the full benefits may 495 not be realised until a substantial proportion of them have been 496 upgraded. 498 In the list which follows, care has been taken to distinguish the 499 implications for both kinds of agent. 501 o [RFC 2822] style s have been prohibited in the case of 502 those header fields of particular concern to news servers. They 503 are unlikely to hinder their proper display in existing reading 504 agents except in the case of the References header field in 505 agents which thread articles. [USEFOR] therefore provides that 506 they SHOULD NOT be generated in that case. 507 o Because of its importance to all serving agents, the whitespace 508 and folding in Newsgroups header fields newly permitted by 509 [USEFOR] SHOULD NOT be generated (though it MUST be accepted); 510 this restriction may well be removed in a future version of this 511 standard. 512 [That last bit needs discussion. It should probably be moved to USEFOR 513 if it is to be retained.] 514 o The new style of Path header field, using "!!" as a , is already consistent with the previous standards. 516 However, the intention is that relaying agents should eventually 517 reject articles in the old style, and so this possibility should 518 be offered as a configurable option in relaying agents. User 519 agents are unaffected. 520 o The introduction by [USEFOR] of MIME reflects a practice that is 521 already widespread. Articles in strict compliance with the 522 previous standards (using strict US-ASCII) will be unaffected. 523 Many user agents already support it, at least to the extent of 524 widely used charsets such as ISO-8859-1. Users expecting to read 525 articles using other charsets will need to acquire suitable 526 reading agents. It is not intended, in general, that any single 527 News Article Architecture and Protocols January 2006 529 user agent will be able to display every charset known to IANA, 530 but all such agents MUST support US-ASCII. Serving and relaying 531 agents are not affected. 532 o The new Control: mvgroup command will need to be implemented in 533 serving agents. For the benefit of older serving agents it is 534 therefore RECOMMENDED that it be followed shortly by a 535 corresponding newgroup command and it MUST always be followed by 536 a rmgroup command for the old group after a reasonable overlap 537 period. An implementation of the mvgroup command as an alias for 538 the newgroup command would thus be minimally conforming. User 539 agents are unaffected. 540 o Provision is made for relaying and serving agents to use the Date 541 header field in the case of articles injected through existing 542 agents which do not yet provide an Injection-Date header field. 543 o All the header fields newly introduced by [USEFOR] can safely be 544 ignored by existing software, albeit with loss of the new 545 functionality. 547 4. Transport 549 As in this standard's predecessors, the exact means used to transmit 550 articles from one host to another is not specified. NNTP [NNTP] is 551 the most common transmission method on the Internet, but much 552 transmission takes place entirely independent of the Internet. Other 553 methods in use include the UUCP protocol [RFC 976] extensively used 554 in the early days of Usenet, FTP, downloading via satellite, tape 555 archives, and physically delivered magnetic and optical media. 557 Transmission paths for news articles MUST treat news articles as 558 uninterpreted sequences of octets, excluding the values 0 (US-ASCII 559 NUL) and 13 and 10 (US-ASCII CR and LF, which MUST ONLY appear in the 560 combination CRLF which denotes a line separator). 562 NOTE: this corresponds to the range of octets permitted for MIME 563 "8bit data" [RFC 2045]. Thus raw binary data cannot be 564 transmitted in an article body except by the use of a Content- 565 Transfer-Encoding such as base64. 567 In particular, transmission paths MUST convey all header fields 568 (including body part header fields and header fields within 569 message/rfc822 objects) intact, even if they contain octets in the 570 range 128 to 255. Furthermore, relaying agents MUST, and other 571 agents SHOULD, convey lines even if they exceed 998 characters in 572 length, especially in article bodies. These requirements include the 573 transmissiom paths between posting agents, injecting agents, relaying 574 agents, serving agents and reading agents, but NOT the paths 575 traversed by Netnews articles that have been gatewayed into Email 576 (7.9.1). 577 [At some point it will be necessary for the IMAP standards to catch up 578 with these requirements.] 579 News Article Architecture and Protocols January 2006 581 5. Definition of new Media Types 583 This standard defines (or redefines) several new Media Types, which 584 require to be registered with IANA as provided for in [RFC 2048]. 586 5.1. Application/news-transmission 588 The Media Type "application/news-transmission" is intended for the 589 encapsulation of complete news articles where the intention is that 590 the recipient should then inject them into Netnews. This Application 591 type provides one of the methods for mailing articles to moderators 592 (see 7.2.2) and it is also the preferred method when sending to an 593 email-to-news gateway (see 7.9.2). 595 NOTE: The benefit of such encapsulation is that it removes 596 possible conflict between news and email header fields and it 597 provides a convenient way of "tunnelling" a news article through 598 a transport medium that does not support 8bit characters. 600 The MIME Media Type definition of "application/news-transmission" is: 602 MIME type name: application 603 MIME subtype name: news-transmission 604 Required parameters: none 605 Optional parameters: usage=moderate 606 usage=inject 607 usage=relay 608 Encoding considerations: A transfer-encoding (such as Quoted- 609 Printable or Base64) different from that of 610 the article transmitted MAY be supplied 611 (perhaps en route) to ensure correct 612 transmission over some 7bit transport 613 medium. 614 Security considerations: A news article may be a "control message", 615 which could have effects on the recipient 616 host's system beyond just storage of the 617 article. However, such control messages 618 also occur in normal news flow, so most 619 hosts will already be suitably defended 620 against undesired effects. 621 Published specification: [USEPRO] 622 Body part: A complete article or proto-article, ready 623 for injection into Netnews, or a batch of 624 such articles in the batch format described 625 in section 6.4. 627 NOTE: It is likely that the recipient of an "application/news- 628 transmission" will be a specialized gateway (e.g. a moderator's 629 submission address) able to accept articles with only one of the 630 three usage parameters "moderate", "inject" and "relay", hence 631 the reason why they are optional, being redundant in most 632 situations. Nevertheless, they MAY be used to signify the 633 originator's intention with regard to the transmission, so 634 removing any possible doubt. 636 News Article Architecture and Protocols January 2006 638 When the parameter "relay" is used, or implied, the body part MAY be 639 a batch of articles to be transmitted together, in which case the 640 batch format defined in section 6.4 MUST be used. 642 5.2. Message/news obsoleted 644 The Media Type "message/news", as previously registered with IANA, is 645 hereby declared obsolete. It was never widely implemented, and its 646 default treatment as "application/octet-stream" by agents that did 647 not recognize it was counter productive. The Media Type 648 "message/rfc822" SHOULD be used in its place. 650 5.3. Application/news-groupinfo 652 The "application/news-groupinfo" is used in conjunction with the 653 "newgroup" (6.2.1) and "mvgroup" (6.2.3) control messages. The 654 in the MUST agree with the 655 in the "newgroup" or "mvgroup" control message. The 656 Media Type "application/news-groupinfo" MUST NOT be used except as a 657 part of such control messages. 659 The "application/news-groupinfo" body part contains brief information 660 about a newsgroup, i.e. the group's name, it's and the . 663 NOTE: The presence of the "For your newsgroups 664 file:" is intended to make the whole newgroup message compatible 665 with current practice as described in [Son-of-1036]. 667 The MIME Media Type definition of "application/news-groupinfo" is: 669 MIME type name: application 670 MIME subtype name: news-groupinfo 671 Required parameters: none 672 Disposition: by default, inline 673 Encoding considerations: "7bit" or "8bit" is sufficient and MUST be 674 used to maintain compatibility. 675 Security considerations: this type MUST NOT be used except as part 676 of a control message for the creation or 677 modification of a Netnews newsgroup 678 Published specification: [USEPRO] 680 The content of the "application/news-groupinfo" body part is defined 681 as: 683 groupinfo-body = [ newsgroups-tag CRLF ] 684 newsgroups-line CRLF 685 newsgroups-tag = %x46.6F.72 SP %x79.6F.75.72 SP 686 %x6E.65.77.73.67.72.6F.75.70.73 SP 687 %x66.69.6C.65.3A 688 ; case sensitive 689 ; "For your newsgroups file:" 691 News Article Architecture and Protocols January 2006 693 newsgroups-line = newsgroup-name 694 [ 1*HTAB newsgroup-description ] 695 [ 1*WSP moderation-flag ] 696 newsgroup-description 697 = utext *( *WSP utext ) 698 moderation-flag = %x28.4D.6F.64.65.72.61.74.65.64.29 699 ; case sensitive "(Moderated)" 701 The MUST NOT contain any occurrence of the 702 string "(Moderated)" within it. Although optional, the SHOULD be included until such time as this standard has been 704 widely adopted, to ensure compatibility with present practice. 706 Moderated newsgroups MUST be marked by appending the case sensitive 707 text " (Moderated)" at the end. It is NOT recommended that the 708 moderator's email address be included in the 709 as has sometimes been done. 711 NOTE: There is no provision for the use of charsets other than 712 US-ASCII within a . Such a facility may 713 be provided in a future extension to this standard. 714 [That may seem harsh, but if we make any such provision now, it will 715 make life more complicated and restrict our freedom when it comes to the 716 proposed I18N extension. Therefore I resisted the temptation to include 717 any charset parameter with this Media Type. Note that this also applies 718 to the checkgroups message further on.] 720 5.4. Application/news-checkgroups 722 The "application/news-checkgroups" Media Type is used in conjunction 723 with the "checkgroups" control message (6.2.4). It MUST NOT be used 724 except as a part of such control messages. 726 The "application/news-checkgroups" body part contains a complete list 727 of all the newsgroups in a (sub)hierarchy, their s and their moderation status. 730 The MIME Media Type definition of "application/news-checkgroups" is: 732 MIME type name: application 733 MIME subtype name: news-checkgroups 734 Required parameters: none 735 Disposition: by default, inline 736 Encoding considerations: "7bit" or "8bit" is sufficient and MUST be 737 used to maintain compatibility. 738 Security considerations: this type MUST NOT be used except as part 739 of a checkgroups control message 740 Published specification: [USEPRO] 742 The content of the "application/news-checkgroups" body part is 743 defined as: 745 News Article Architecture and Protocols January 2006 747 checkgroups-body = *( valid-group CRLF ) 748 valid-group = newsgroups-line ; see 5.3 750 6. Control Messages 752 The following sections document the control messages. "Message" is 753 used herein as a synonym for "article" unless context indicates 754 otherwise. 756 Each comprises a , which indicates the action 757 to be taken, and (s), which supply the details (see F- 758 3.2.5). The following sections contain syntactic definitions for the 759 , s, and possibly the body, for each type of control 760 message. 761 [The term is now used to denote the syntactic object 762 within the Control header field, to distinguish it from "control 763 message", which refers to the whole article.] 765 The Newsgroups header field of each control message SHOULD include 766 the (s) for the group(s) affected (i.e. groups to be 767 created, modified or removed, or containing articles to be canceled). 768 This is to ensure that the message propagates to all sites which 769 receive (or would receive) that group(s). It MAY include other 770 s so as to improve propagation (but this practice may 771 cause the control message to propagate also to places where it is 772 unwanted, or even cause it not to propagate where it should, so it 773 should not be used without good reason). 775 NOTE: Propagation is controlled by relaying agents, and it may 776 be necessary for relaying agents to take special steps to ensure 777 that control messages such as newgroup messages for not-yet- 778 existent newsgroups are propagated correctly (see 7.3). 780 The presence of a Subject header field whose content starts with the 781 string "cmsg " followed by a was construed under 782 [RFC 1036] as a request to perform that control action (even if no 783 genuine Control header field was present). Indeed, some 784 implementations went further and added the implied Control header 785 field before injecting. Likewise, the presence of a 786 ending in ".ctl" in the Newsgroups header field caused the Subject 787 header field content (not starting with "cmsg" in this case) to be 788 interpreted as a . 790 All these practices, which have already largely fallen into disuse, 791 are now declared to be Obsolete, and Subject header fields MUST NOT 792 now be interpreted as s under any circumstances. 794 [Possible addtional text:] 796 In order to prevent continuing interpretation of Subject header 797 fields in this way by existing agents, posting and injecting agents 798 SHOULD detect and decline to post articles in which the Subject 799 header field starts with the word "cmsg" and in which there is no 800 Control header field. 802 News Article Architecture and Protocols January 2006 804 The descriptions below set out REQUIREMENTS to be followed by sites 805 that receive control messages and choose to honour them. However, 806 nothing in these descriptions should be taken as overriding the right 807 of any such site, in accordance with its local policy, to refuse to 808 honour any particular control message, or to refer it to an 809 administrator for approval (either as a class or on a case-by-case 810 basis). 812 6.1. Digital Signature of Header Fields 814 It is most desirable that group control messages (6.2) in particular 815 be authenticated by incorporating them within some digital signature 816 scheme that encompasses other header fields closely associated with 817 them (including at least Approved, Message-ID and Date). At the time 818 of writing, this is usually done by means of a protocol known as 819 "PGPverify" ([PGPVERIFY]), and continued usage of this is encouraged 820 at least as an interim measure. 822 However, PGPverify is not considered suitable for standardization in 823 its present form, for various technical reasons. It is therefore 824 expected that an early extension to this standard will provide a 825 robust and general purpose digital authentication mechanism with 826 applicability to all situations requiring protection against 827 malicious use of, or interference with, header fields. That 828 extension would also address other Netnews security issues. 830 6.2. Group Control Messages 832 "Group control messages" are the sub-class of control messages that 833 request some update to the configuration of the groups known to a 834 serving agent, namely "newgroup", "rmgroup", "mvgroup" and 835 "checkgroups", plus any others created by extensions to this 836 standard. 838 Group control messages that attempt to create groups with names that 839 are deprecated or reserved according to F-3.1.5 MUST NOT be issued, 840 except by prior agreement within some cooperating subnet. Moreover, 841 sites receiving such control messages SHOULD check them for 842 conformance before honouring them. 844 All of the group control messages MUST have an Approved header field 845 (F-3.2.9) which, in those hierarchies where appropriate 846 administrative agencies exist (see 1.1), identifies the appropriate 847 person or entity as authorized by those agencies. The authorized 848 person or entity SHOULD adhere to any conventions and restrictions on 849 the format of s established for those hierarchies 850 [USEAGE]. 852 6.2.1. The 'newgroup' Control Message 854 control-command =/ Newgroup-command 855 Newgroup-command = "newgroup" Newgroup-arguments 856 Newgroup-arguments = FWS newsgroup-name [ FWS newgroup-flag ] 857 newgroup-flag = "moderated" 858 News Article Architecture and Protocols January 2006 860 The "newgroup" control message requests that the specified group be 861 created or have its moderation status or changed. 862 When the request is honoured, if the "moderated" is 863 present then the status of the group SHOULD be marked as moderated, 864 and vice versa. "Moderated" is the only such flag defined by this 865 standard; other flags MAY be defined for use in cooperating subnets, 866 but newgroup messages containing them MUST NOT be acted on outside of 867 those subnets. 869 NOTE: Specifically, some alternative flags such as "y" and "m", 870 which are sent and recognized by some current software, are NOT 871 part of this standard. Moreover, some existing implementations 872 treat any flag other than "moderated" as indicating an 873 unmoderated newsgroup. Both of these usages are contrary to this 874 standard and control messages with such non-standard flags 875 should be ignored. 877 6.2.1.1. The Body of the 'newgroup' Control Message 879 The body of the newgroup message contains the following subparts, 880 preferably in the order shown: 882 1. An "application/news-groupinfo" part (5.3) containing the name and 883 (5.3) of the group. This part MUST be present 884 and SHOULD be used to update any copy of the 885 maintained by the serving agent. 887 2. Other parts containing useful information about the background of 888 the newgroup message (typically of type "text/plain"). 890 3. Parts containing initial articles for the newsgroup. See section 891 6.2.1.2 for details. 893 In the event that there is only the single (i.e. application/news- 894 groupinfo) subpart present, it will suffice to include a "Content- 895 Type: application/news-groupinfo" amongst the header fields of the 896 control message. Otherwise, a "Content-Type: multipart/mixed" header 897 field will be needed, and each separate part will then need its own 898 Content-Type header field. 900 6.2.1.2. Initial Articles 902 Some subparts of a "newgroup" or "mvgroup" control message MAY 903 contain an initial set of articles to be posted to the affected 904 newsgroup as soon as it has been created or modified. These parts are 905 identified by having the Media Type "application/news-transmission", 906 possibly with the parameter "usage=inject". The body of each such 907 part should be a complete proto-article, ready for posting. This 908 feature is intended for the posting of charters, initial FAQs and the 909 like to the newly formed group. 911 The Newsgroups header field of the proto-article MUST include the 912 of the newly created or modified group. It MAY 913 include other s. If the proto-article includes a 914 News Article Architecture and Protocols January 2006 916 Message-ID header field, the message identifier in it MUST be 917 different from that of any existing article and from that of the 918 control message as a whole. Alternatively such a message identifier 919 MAY be derived by the injecting agent when the proto-article is 920 posted. The proto-article SHOULD include the header field 921 "Distribution: local". 923 The proto-article SHOULD be injected at the serving agent that 924 processes the control message AFTER the newsgroup in question has 925 been created or modified. It MUST NOT be injected if the newsgroup 926 is not, in fact, created (for whatever reason). It MUST NOT be 927 submitted to any relaying agent for transmission beyond the serving 928 agent(s) upon which the newsgroup creation has just been effected (in 929 other words, it is to be treated as having a "Distribution: local" 930 header field, whether such a field is actually present or not). 932 NOTE: It is not precluded that the proto-article is itself a 933 control message or other type of special article, to be 934 activated only upon creation of the new newsgroup. However, 935 except as might arise from that possibility, any 936 "application/news-transmission" within some nested "multipart/*" 937 structure within the proto-article is not to be activated. 939 6.2.1.3. Example 941 A "newgroup" with its charter: 943 From: "example.all Administrator" 944 Newsgroups: example.admin.info,example.admin.announce 945 Date: 27 Feb 2002 12:50:22 +0200 946 Subject: cmsg newgroup example.admin.info moderated 947 Approved: admin@noc.example 948 Control: newgroup example.admin.info moderated 949 Message-ID: 950 MIME-Version: 1.0 951 Content-Type: multipart/mixed; boundary="nxtprt" 952 Content-Transfer-Encoding: 8bit 954 This is a MIME control message. 955 --nxtprt 956 Content-Type: application/news-groupinfo 958 For your newsgroups file: 959 example.admin.info About the example.* groups (Moderated) 961 --nxtprt 962 Content-Type: application/news-transmission 964 Newsgroups: example.admin.info 965 From: "example.all Administrator" 966 Subject: Charter for example.admin.info 967 Message-ID: 968 Distribution: local 969 News Article Architecture and Protocols January 2006 971 Content-Type: text/plain; charset=us-ascii 972 Content-Transfer-Encoding: 7bit 974 The group example.admin.info contains regularly posted 975 information on the example.* hierarchy. 977 --nxtprt-- 979 6.2.2. The 'rmgroup' Control Message 981 control-command =/ Rmgroup-command 982 Rmgroup-command = "rmgroup" Rmgroup-arguments 983 Rmgroup-arguments = FWS newsgroup-name 985 The "rmgroup" control message requests that the specified group be 986 removed from the list of valid groups. The Media Type of the body is 987 unspecified; it MAY contain anything, usually an explanatory text. 989 NOTE: It is entirely proper for a serving agent to retain the 990 group until all the articles in it have expired, provided that 991 it ceases to accept new articles. 993 6.2.2.1. Example 995 From: "example.all Administrator" 996 Newsgroups: example.admin.obsolete, example.admin.announce 997 Date: 4 Apr 2002 22:04 -0900 (PST) 998 Subject: cmsg rmgroup example.admin.obsolete 999 Message-ID: 1000 Approved: admin@noc.example 1001 Control: rmgroup example.admin.obsolete 1003 The group example.admin.obsolete is obsolete. Please remove it 1004 from your system. 1006 6.2.3. The 'mvgroup' Control Message 1008 control-command =/ Mvgroup-command 1009 Mvgroup-command = "mvgroup" Mvgroup-arguments 1010 Mvgroup-arguments = FWS newsgroup-name FWS newsgroup-name 1011 [ FWS newgroup-flag ] 1013 The "mvgroup" control message requests that the group specified by 1014 the first <(old-)newsgroup-name> be moved to that specified by the 1015 second <(new-)newsgroup-name>. Thus it is broadly equivalent to a 1016 "newgroup" control message for the second group followed by a 1017 "rmgroup" control message for the first group. 1019 The message body contains an "application/news-groupinfo" part (5.3) 1020 containing machine- and human-readable information about the new 1021 group, and possibly other subparts as for a "newgroup" control 1022 message. The information conveyed in the "application/news-groupinfo" 1023 body part, notably its (5.3), is applied to the new 1024 group. 1026 News Article Architecture and Protocols January 2006 1028 When this message is received, the new group is created (if it does 1029 not exist already) as for a "newgroup" control message, and SHOULD in 1030 any case be made moderated if a "moderated" is 1031 present, and vice versa. At the same time, arrangements SHOULD be 1032 made to remove the old group (as with a "rmgroup" control message), 1033 but only after a suitable overlap period to allow the network to 1034 adjust to the new arrangement. 1036 At the same time as a serving agent acts upon this message, all 1037 injecting agents associated with that serving agent SHOULD inhibit 1038 the posting of new articles to the old group (preferably with some 1039 indication to the poster that the new group should have been used). 1040 Relaying agents, however, MUST continue to propagate such articles 1041 during the overlap period. 1043 NOTE: It is to be expected that different serving agents will 1044 act on this message at different points of time, users of the 1045 old group will have to become accustomed to the new arrangement, 1046 and followups to already established threads will likely 1047 continue under the old group. Therefore, there needs to be an 1048 overlap period during which articles may continue to be accepted 1049 by relaying and serving agents in either group. This standard 1050 does not specify any standard period of overlap (though it would 1051 be expected to be expressed in days rather than in months). The 1052 inhibition of injection of new articles to the old group may 1053 seem draconian, but it is the surest way to prevent the 1054 changeover from dragging on indefinitely. 1056 Since the "mvgroup" control message is newly introduced in this 1057 standard and may not be widely implemented initially, it SHOULD be 1058 followed shortly afterwards by a corresponding "newgroup" control 1059 message; and again, after a reasonable overlap period, it MUST be 1060 followed by a "rmgroup" control message for the old group. 1062 In order to facilitate a smooth changeover, serving agents MAY 1063 arrange to service requests for access to the old group by providing 1064 access to the new group, which would then contain, or appear to 1065 contain, all articles posted to either group (including, ideally, the 1066 pre-changeover articles from the old one). Nevertheless, if this 1067 feature is implemented, the articles themselves, as supplied to 1068 reading agents, MUST NOT be altered in any way (and, in particular, 1069 their Newsgroups header fields MUST contain exactly those newsgroups 1070 present when they were injected). On the other hand, the Xref header 1071 field (F-3.2.11) MAY contain entries for either group (or even both). 1073 NOTE: Some serving agents that use an "active" file permit an 1074 entry of the form "oldgroup xxx yyy =newgroup", which enables 1075 any articles arriving for oldgroup to be diverted to newgroup, 1076 thus providing a simple implementation of this feature. However, 1077 it is known that not all current serving agents will find 1078 implementation so easy (especially in the short term) which is 1079 why it is not mandated by this standard. Nevertheless, its 1080 eventual implementation in all serving agents is to be 1081 considered highly desirable. 1083 News Article Architecture and Protocols January 2006 1085 On the other hand, it is recognized that this feature would 1086 likely not be implementable if the new group was already in 1087 existence with existing articles in it. This situation should 1088 not normally arise except when there is already some confusion 1089 as to which groups are, or are not, supposed to exist in that 1090 hierarchy. Note that the "mvgroup" control message is not really 1091 intended to be used for merging two existing groups. 1093 6.2.3.1. Example 1095 From: "example.all Administrator" 1096 Newsgroups: example.oldgroup,example.newgroup,example.admin.announce 1097 Date: 30 Apr 2002 22:04 -0500 (EST) 1098 Subject: cmsg mvgroup example.oldgroup example.newgroup moderated 1099 Message-ID: 1100 Approved: admin@noc.example 1101 Control: mvgroup example.oldgroup example.newgroup moderated 1102 MIME-Version: 1.0 1103 Content-Type: multipart/mixed; boundary=nxt 1105 --nxt 1106 Content-Type: application/news-groupinfo 1108 For your newsgroups file: 1109 example.newgroup The new replacement group (Moderated) 1111 --nxt 1113 The moderated group example.oldgroup is replaced by 1114 example.newgroup. Please update your configuration, and please, 1115 if possible, arrange to file articles arriving for 1116 example.oldgroup as if they were in example.newgroup. 1118 --nxt 1119 Content-Type: application/news-transmission 1121 Newsgroups: example.admin.info 1122 From: "example.all Administrator" 1123 Subject: Charter for example.newgroup 1124 Message-ID: 1125 Distribution: local 1126 Content-Type: text/plain; charset=us-ascii 1127 Content-Transfer-Encoding: 7bit 1129 This group (formerly known as example.oldgroup) is for the 1130 discussion of examples. 1132 --nxt-- 1134 6.2.4. The 'checkgroups' Control Message 1136 The "checkgroups" control message contains a list of all the valid 1137 groups in a complete hierarchy. 1139 News Article Architecture and Protocols January 2006 1141 control-command =/ Checkgroup-command 1142 Checkgroup-command = "checkgroups" Checkgroup-arguments 1143 Checkgroup-arguments= [ chkscope ] [ chksernr ] 1144 chkscope = 1*( FWS ["!"] newsgroup-name ) 1145 chksernr = FWS "#" 1*DIGIT 1147 A "checkgroups" message applies to any (sub-)hierarchy with a prefix 1148 listed in the argument, provided that the rightmost 1149 matching in the list is not immediately preceded by 1150 a "!". If no argument is given, it applies to all 1151 hierarchies for which group statements appear in the body of the 1152 message. 1154 NOTE: Some existing software does not support the 1155 argument. Thus a "checkgroups" message SHOULD also contain the 1156 groups of other subhierarchies the sender is not responsible 1157 for. "New" software MUST ignore groups which do not fall within 1158 the argument of the "checkgroups" message. 1160 The argument is a serial number, which can be any positive 1161 integer (e.g. just numbered or the date in YYYYMMDD). It SHOULD 1162 increase by an arbitrary value with every change to the group list 1163 and MUST NOT ever decrease. 1165 NOTE: This was added to circumvent security problems in 1166 situations where the Date header field cannot be authenticated. 1168 Example: 1170 Control: checkgroups de !de.alt #248 1172 which includes the whole of the 'de.*' hierarchy, with the exception 1173 of its 'de.alt.*' sub-hierarchy. 1175 The body of the message has the Media Type "application/news- 1176 checkgroups" (5.4). It asserts that the s it lists are 1177 the only newsgroups in the specified hierarchies. 1179 NOTE: The "checkgroups" message is intended to synchronize the 1180 list of newsgroups stored by a serving agent, and their 1181 s, with the lists stored by other serving 1182 agents throughout the network. However, it might be inadvisable 1183 for the serving agent actually to create or delete any 1184 newsgroups without first obtaining the approval of its 1185 administrators for such proposed actions. 1187 NOTE: The possibility of removing a complete hierarchy by means 1188 of an "invalidation" line beginning with a '!' in the 1189 checkgroups-body is no longer provided by this standard. The 1190 intent of the feature was widely misunderstood and it was 1191 misused more often than it was used correctly. The same effect, 1192 if required, can now be obtained by the use of an appropriate 1193 argument in conjunction with an empty . 1196 News Article Architecture and Protocols January 2006 1198 6.3. Cancel 1200 The "cancel" message requests that a target article be "canceled", 1201 i.e. be withdrawn from circulation or access. 1203 control-command =/ Cancel-command 1204 Cancel-command = "cancel" Cancel-arguments 1205 Cancel-arguments = FWS msg-id [FWS] 1207 The argument identifies the article to be cancelled by its message 1208 identifier. The body SHOULD contain an indication of why the 1209 cancellation was requested. The "cancel" message SHOULD be posted to 1210 the same newsgroup(s), with the same distribution(s), as the article 1211 it is attempting to cancel. 1213 A serving agent that elects to honour a "cancel" message SHOULD make 1214 the article unavailable for relaying or serving (perhaps by deleting 1215 it completely). If the target article is unavailable, and the 1216 acceptability of the "cancel" message cannot be established without 1217 it, activation of the "cancel" message SHOULD be delayed until the 1218 target article has been seen. See also sections 7.3 and 7.4. 1220 NOTE: It is expected that the security extension envisaged in 1221 section 6.1 will make more detailed provisions for establishing 1222 whether honouring a particular "cancel" message is in order. In 1223 particular, it is likely that there will be provision for the 1224 digital signature of 3rd party cancels (i.e. those issued other 1225 than by the sender, the moderator, or the injector). 1227 NOTE: A cancel submitted by the poster for an article in a 1228 moderated group will be forwarded to the moderator of that 1229 group, and it is up to that moderator to act upon it (7.8). 1231 NOTE: The former requirement [RFC 1036] that the From and/or 1232 Sender header fields of the "cancel" message should match those 1233 of the original article has been removed from this standard, 1234 since it only encouraged cancel issuers to conceal their true 1235 identity, and it was not usually checked or enforced by 1236 canceling software. Therefore, both the From and/or Sender 1237 header fields and any Approved header field should now relate to 1238 the entity responsible for issuing the "cancel" message. 1240 6.4. Ihave, sendme 1242 The "ihave" and "sendme" control messages implement a crude batched 1243 predecessor of the NNTP [NNTP] protocol. They are largely obsolete on 1244 the Internet, but still see use in conjunction with some transport 1245 protocols such as UUCP, especially for backup feeds that normally are 1246 active only when a primary feed path has failed. There is no 1247 requirement for relaying agents that do not support such transport 1248 protocols to implement them. 1250 News Article Architecture and Protocols January 2006 1252 NOTE: The ihave and sendme messages defined here have ABSOLUTELY 1253 NOTHING TO DO WITH NNTP, despite similarities of terminology. 1255 The two messages share the same syntax: 1257 control-command =/ Ihave-command 1258 Ihave-command = "ihave" Ihave-argument 1259 Ihave-argument = relayer-name 1260 control-command =/ Sendme-command 1261 Sendme-command = "sendme" Sendme-argument 1262 Sendme-argument = Ihave-argument 1263 relayer-name = path-identity ; see F-3.1.6 1264 ihave-body = *( msg-id CRLF ) 1265 sendme-body = ihave-body 1267 The body of the message consists of a list of s, one per 1268 line. [RFC 1036] also permitted the list of s to appear in 1269 the or with the syntax 1270 Ihave-argument = [FWS] *( msg-id FWS ) [relayer-name] 1271 but this form SHOULD NOT now be used, though relaying agents MAY 1272 recognize and process it for backward compatibility. 1274 The "ihave" message states that the named relaying agent has received 1275 articles with the specified message identifiers, which may be of 1276 interest to the relaying agents receiving the ihave message. The 1277 "sendme" message requests that the agent receiving it send the 1278 articles having the specified message identifiers to the named 1279 relaying agent. 1281 Upon receipt of the sendme message, the receiving agent sends the 1282 article(s) requested, often (especially when the transport protocol 1283 is UUCP) in the form of one or more batches, each containing several 1284 articles. The usual form of a is defined by the following 1285 syntax (which is also used in the application/news transmission media 1286 type (5.1)). 1288 batch = 1*( batch-header article ) 1289 batch-header = "#!" SP rnews SP article-size CRLF 1290 rnews = %x72.6E.65.77.73 ; case sensitive "rnews" 1291 article-size = 1*DIGIT 1293 Thus a is a sequence of articles, each prefixed by a header 1294 line that includes its size. The is a decimal count of 1295 the octets in the article, counting each CRLF as one octet regardless 1296 of how it is actually represented. 1298 NOTE: Despite the similarity of this format to an executable 1299 UNIX script, it is EXTREMELY unwise to feed such a batch into a 1300 command interpreter in anticipation of it running a command 1301 named "rnews"; the security implications of so doing would be 1302 disastrous. 1304 News Article Architecture and Protocols January 2006 1306 These control messages are normally sent essentially as point-to- 1307 point messages, by using s in the Newsgroups header 1308 field of the form "to." followed by one (or possibly more) 1309 s in the form of a (see section F-3.1.5 1310 which forbids "to" as the first of a ). 1311 The control message SHOULD then be delivered ONLY to the relaying 1312 agent(s) identified by that , and any relaying agent 1313 receiving such a message which includes its own MUST 1314 NOT propagate it further. Each pair of relaying agent(s) sending and 1315 receiving these messages MUST be immediate neighbours, exchanging 1316 news directly with each other. Each relaying agent advertises its new 1317 arrivals to the other using "ihave" messages, and each uses "sendme" 1318 messages to request the articles it lacks. 1320 To reduce overhead, ihave and sendme messages SHOULD be sent 1321 relatively infrequently and SHOULD contain reasonable numbers of 1322 message identifiers. If ihave and sendme are being used to implement 1323 a backup feed, it may be desirable to insert a delay between 1324 reception of an ihave and generation of a sendme, so that a slightly 1325 slow primary feed will not cause large numbers of articles to be 1326 requested unnecessarily via sendme. 1328 6.5. Obsolete control messages. 1330 The following control messages (as described in Appendix A) are 1331 declared obsolete by this standard: 1333 sendsys 1334 version 1335 whogets 1336 senduuname 1338 7. Duties of Various Agents 1340 The following section sets out the duties of various agents involved 1341 in the creation, relaying and serving of Netnews articles. Insofar as 1342 these duties are described as sequences of steps to be followed, it 1343 should be understood that it is the effect of these sequences that is 1344 important, and implementations may use any method that gives rise to 1345 that same effect. 1347 In this section, the word "trusted", as applied to the source of some 1348 article, means that an agent processing that article has verified, by 1349 some means, the identity of that source (which may be another agent 1350 or a poster). 1352 NOTE: In many implementations, a single agent may perform 1353 various combinations of the injecting, relaying and serving 1354 functions. Its duties are then the union of the various duties 1355 concerned. 1357 News Article Architecture and Protocols January 2006 1359 7.1. General principles to be followed 1361 There are two important principles that news implementors (and 1362 administrators) need to keep in mind. The first is the well-known 1363 Internet Robustness Principle: 1365 Be liberal in what you accept, and conservative in what you 1366 send. 1368 However, in the case of news there is an even more important 1369 principle, derived from a much older code of practice, the 1370 Hippocratic Oath (we may thus call this the Hippocratic Principle): 1372 First, do no harm. 1374 It is VITAL to realize that decisions which might be merely 1375 suboptimal in a smaller context can become devastating mistakes when 1376 amplified by the actions of thousands of hosts within a few minutes. 1378 In the case of gateways, the primary corollary to this is: 1380 Cause no loops. 1382 7.2. Duties of an Injecting Agent 1384 An Injecting Agent is responsible for taking a (proto-)article from a 1385 posting (or other) agent and either forwarding it to a moderator or 1386 injecting it into the relaying system for access by readers. 1388 As such, an injecting agent is considered responsible for ensuring 1389 that any article it injects conforms with the rules of [USEFOR]. It 1390 is also expected to bear some responsibility towards the rest of the 1391 network for the behaviour of its posters. 1393 In the normal course of events, an article that has already been 1394 injected into a Netnews network will never pass through another 1395 injecting agent. So, if an injecting agent receives an otherwise 1396 valid article that has already been injected (as evidenced by the 1397 presence of an Injection-Date header field, an Injection-Info header 1398 field, or more than one "POSTED" in a Path header field) it MAY 1399 choose to reject it, but otherwise SHOULD cause it to be relayed, as 1400 it stands, by a relaying agent (7.3). 1402 In exceptional circumstances (e.g. as part of some complex gatewaying 1403 process, or where a relaying agent considers it essential for 1404 fulfilling its responsibility towards the rest of the network) an 1405 already injected article MAY be "reinjected" into the network. This 1406 standard does not prescribe any such circumstance; rather this is a 1407 matter of policy to be determined by the administrators of each 1408 injecting agent, who have the responsibility to ensure that no harm 1409 arises. In all other circumstances, unintented reinjection is to be 1410 avoided (see 7.9). Nevertheless, in order to preserve the integrity 1411 of the network in these special cases, this standard does set out the 1412 correct way to reinject (see special provisions in 7.2.2 Steps 3, 7 1413 News Article Architecture and Protocols January 2006 1415 and 9). 1417 It is usual for an injecting agent to be closely associated with a 1418 serving agent, thus giving it access to the list (7.4) showing the 1419 moderation status of the newsgroups it is likely to handle. In the 1420 event that it does not have such an associated serving agent, it MUST 1421 maintain that list itself. 1423 7.2.1. Proto-articles 1425 A proto-article SHOULD NOT be propagated in that form to other than 1426 injecting agents. 1428 A proto-article has the same format as a normal article except that 1429 some of the following mandatory header fields MAY be omitted: 1430 Message-Id, Date, Path (and even From if the particular injecting 1431 agent can derive that information from other sources). However, if 1432 it is intended to offer the proto-article to two or more injecting 1433 agents in parallel, then it is only the Path header field that MAY be 1434 omitted. The header fields that can be omitted MUST NOT contain 1435 invalid values; they MUST either be correct or not present at all. 1436 [Maybe omit that last sentence.] 1438 NOTE: An article that is offered for reinjection has, by 1439 definition, already been injected once, and is not therefore to 1440 be considered as a proto-article. Hence a genuine proto-article 1441 will not contain any Injection-Date header field nor any 1442 "POSTED" in its Path header field, though that header field MAY 1443 contain s corresponding to machines traversed 1444 between the posting agent and the injecting agent proper. 1446 7.2.2. Procedure to be followed by Injecting Agents 1448 An injecting agent receives (proto-)articles from posting and 1449 followup agents. It verifies them, adds header fields where required, 1450 and then either forwards them to a moderator or injects them by 1451 passing them to serving or relaying agents. It MUST NOT forward an 1452 already injected article to a moderator. 1454 An injecting agent processes articles as follows: 1456 1. It MUST remove any Injection-Info header field already present 1457 (though it might be useful to copy it to a suitable "X-" header 1458 field). It SHOULD likewise remove any NNTP-Posting-Host, X-Trace, 1459 or other non-standard tracing header field. 1461 2. It SHOULD verify that the article is from a trusted source, and 1462 MAY reject articles in which header fields contain unverified 1463 email addresses, that is, addresses which are not known to be 1464 valid for the trusted source, though it would be perverse to 1465 reject intentionally unverifiable addresses such as those ending 1466 in ".invalid" (7.5). 1468 News Article Architecture and Protocols January 2006 1470 3. It SHOULD reject any article whose Date header field (F-3.1.2) is 1471 more than 24 hours into the future (and MAY use a margin less than 1472 that 24 hours). It MUST (except when reinjecting) reject any 1473 article with an Injection-Date header field already present (and 1474 SHOULD do likewise with any NNTP-Posting-Date header field). When 1475 reinjecting it MAY, in the absence of any Injection-Date header 1476 field, reject any article whose Date header field appears to be 1477 stale (e.g. more than 72 hours into the past). 1479 4. It MUST reject any article that does not have the proper mandatory 1480 header fields for a proto-article or which contains any header 1481 field that does not have legal contents. It SHOULD reject any 1482 article which contains any header field deprecated for Netnews 1483 (e.g. as in [RFC 2298]). It SHOULD reject any article whose 1484 Newsgroups header field does not contain at least one for an existing group (as listed by its associated serving 1486 agent) and it MAY reject any which violates one 1487 of the restrictions in F-3.1.5 or which, although otherwise 1488 correct, violates a policy restriction established, for some 1489 (sub-)hierarchy, by an agency with the appropriate authority 1490 (1.2). Observe that crossposting to unknown newsgroups is not 1491 precluded provided at least one of those in the Newsgroups header 1492 field is listed. 1494 NOTE: This ability to reject s in breach of 1495 established policy does not extend to relaying agents, though it 1496 might be reasonable for posting agents to do it. 1498 5. If the article is rejected (for reasons given above, or for other 1499 formatting errors or matters of site policy) the posting agent 1500 SHOULD be informed (such as via an NNTP 44x response code) that 1501 posting has failed and the article MUST NOT then be processed 1502 further. 1504 6. The Message-ID, Date and From header fields (with appropriate 1505 contents) MUST be added when not already present. A User-Agent 1506 header field MAY be added (or an already present User-Agent header 1507 field MAY be augmented) so as to identify the software (e.g. 1508 "INN/1.7.2") used by the injecting agent. 1509 [That last sentence may need to be reconsidered (in which case see 1510 consequential change in 7.3).] 1512 NOTE: The Message-ID, Date and From fields will already be 1513 present during reinjection. 1515 7. The injecting agent MUST NOT alter the body of the article in any 1516 way (including any change of Content-Transfer-Encoding). It MAY 1517 (except when reinjecting) add other header fields not already 1518 provided by the poster, but SHOULD NOT alter, delete, or reorder 1519 any existing header field, with the specific exception of the 1520 "tracing" header field Injection-Info, which is to be removed as 1521 already mentioned. 1523 News Article Architecture and Protocols January 2006 1525 8. If the Newsgroups header field contains one or more moderated 1526 groups and the article does NOT contain an Approved header field, 1527 the injecting agent MUST forward it to a moderator as specified in 1528 section 7.2.3 below. 1530 9. Otherwise, a Path header field with a (F-3.1.6) MUST 1531 be correctly added if not already present. During reinjection, the 1532 existing Path header field SHOULD be retained. 1534 10.It MUST then prepend the of the injecting agent, 1535 followed by a '!', the "POSTED" and a further "!" 1536 (or "!!" if appropriate) to the content of the Path header field; 1537 this header field SHOULD then be folded if it would otherwise 1538 result in a header line of excessive length. 1539 [This may need further changes depending on the resolution of ticket 1540 #1047.] 1542 NOTE: This could result in more that one "POSTED" 1543 in the case of reinjection. 1545 11.An Injection-Info header field (F-3.2.14) SHOULD be added, 1546 identifying the trusted source of the article and possibly an 1547 address for mailing complaints to. Each injecting agent SHOULD 1548 use a consistent form of the Injection-Info header field for all 1549 articles emanating from the same or similar origins. 1551 NOTE: The step above is the only place in which an Injection- 1552 Info header field is to be created. It follows that this header 1553 field MUST NOT be created, replaced, changed or deleted by any 1554 other agent (except during reinjection, in which case it will 1555 always relate to the latest injection and is, to that extent, 1556 regarded as a variant header field). 1558 12.The injecting agent MUST then add an Injection-Date header field 1559 (F-3.2.1) if one is not already present, but it MUST NOT alter, or 1560 delete, an already present Injection-Date header field (and 1561 likewise SHOULD NOT alter, or delete, an already present NNTP- 1562 Posting-Date header field). Finally, it forwards the article to 1563 one or more relaying or serving agents, and the injection process 1564 is to be considered complete. 1566 NOTE: The step above is the only place where an Injection-Date 1567 header field is to be created It follows that it MUST NOT 1568 subsequently be replaced, changed or deleted by any other agent, 1569 even during reinjection. 1571 7.2.3. Procedure for Forwarding to a Moderator 1573 An injecting agent forwards an article to a moderator as follows: 1575 1. It MUST forward it to the moderator of the first (leftmost) 1576 moderated group listed in the Newsgroups header field, customarily 1577 via email, (see 7.8 for how that moderator may forward it to 1578 further moderators). There are two possibilities for doing this: 1580 News Article Architecture and Protocols January 2006 1582 (a) The complete article is encapsulated (header fields and all) 1583 within the email, preferably using the Content-Type 1584 "application/news-transmission" (5.1) with any usage 1585 parameter set to "moderate". Moreover, there SHOULD NOT be 1586 more than one encapsulated article within the one email. 1587 This method has the advantage of removing any possible 1588 conflict between Netnews and Email header fields, or of 1589 changes to those fields during transport through email. 1591 (b) The article is sent as an email as it stands, with the 1592 addition of such extra header fields (e.g. a To header field) 1593 as are necessary for an email. The existing Message-ID header 1594 field SHOULD be retained. 1596 Although both of these methods have seen use in the past, the 1597 preponderance of current usage on Usenet has been for method (b) 1598 and many moderators are ill-prepared to deal with method (a). 1599 Therefore, method (a) SHOULD NOT be used until such time as the 1600 majority of moderators are able to accept it. 1602 2. This standard does not prescribe how the email address of the 1603 moderator is to be determined, that being a matter of policy to be 1604 arranged by the agency responsible for the oversight of each 1605 hierarchy. Nevertheless, there do exist various agents worldwide 1606 which provide the service of forwarding to moderators, and the 1607 address to use with them is obtained as follows: 1609 (a) Each '.' in the is replaced with a '-'. 1611 (b) The result of these operations is used as the of 1612 the of the agent. For example, articles intended 1613 for "news.announce.important" would be emailed to "news- 1614 announce-important@forwardingagent.example". 1616 7.3. Duties of a Relaying Agent 1618 A Relaying Agent accepts injected articles from injecting and other 1619 relaying agents and passes them on to relaying or serving agents 1620 according to mutually agreed policy. Relaying agents SHOULD accept 1621 articles ONLY from trusted agents. 1623 An article SHOULD NOT be relayed unless the sending agent has been 1624 configured to supply and the receiving agent to receive at least one 1625 of the s in its Newsgroups header field and at least 1626 one of the s in its Distribution header field, if any. 1627 Exceptionally, ALL relaying agents are deemed willing to supply or 1628 accept the "world", and NO relaying agent should supply 1629 or accept the "local". 1631 However, if the particular implementation does not relay non-existent 1632 newsgroups, even when included in the Newsgroups header field and 1633 implied (e.g. by some "wild card" notation) in the configuration 1634 tables, then the agent MUST examine all group control messages (6.2) 1635 in order to ensure that relaying of those messages proceeds normally. 1637 News Article Architecture and Protocols January 2006 1639 NOTE: Although it would seem redundant to filter out unwanted 1640 distributions at both ends of a relaying link (and it is clearly 1641 more efficient to do so at the sending end), many sending sites 1642 have been reluctant, historically speaking, to apply such 1643 filters (except to ensure that distributions local to their own 1644 site or cooperating subnet did not escape); moreover they tended 1645 to configure their filters on an "all but those listed" basis, 1646 so that new and hitherto unheard of distributions would not be 1647 caught. Indeed many "hub" sites actually wanted to receive all 1648 possible distributions so that they could feed on to their 1649 clients in all possible geographical (or organizational) 1650 regions. 1652 Therefore, it is desirable to provide facilities for rejecting 1653 unwanted distributions at the receiving end. Indeed, it may be 1654 simpler to do so locally than to inform each sending site of 1655 what is required, especially in the case of specialized 1656 distributions (for example for control messages, such as cancels 1657 from certain issuers) which might need to be added at short 1658 notice. A similar possibility for reading agents to filter 1659 distributions is also suggested in [USEAGE]) for the same 1660 reason. 1662 In order to avoid unnecessary relaying, an article SHOULD NOT be 1663 relayed if the of the receiving agent (or some known 1664 alias thereof) appears as a (excluding within the 1665 ) in its Path header field. 1667 A relaying agent processes articles as follows: 1669 1. It MUST establish the trusted identity of the source of the 1670 article and compare it with the leftmost of the 1671 Path header field's content. If it matches it MUST then prepend 1672 its own and a '!!' to that 1673 content. If it does not match then it prepends instead two entries 1674 to that content; firstly the true established of 1675 the source followed by a '!', the "MISMATCH" and a 1676 further '!', and then, to the left of that, its own followed by a '!!' as usual. This 1678 prepending of two entries SHOULD NOT be done if the provided and 1679 established identities match. This header field SHOULD then be 1680 folded if it would otherwise result in a header line of excessive 1681 length. 1682 [This may need further changes depending on the resolution of ticket 1683 #1047.] 1684 [It has been suggested that relaying agents should be permitted to 1685 prepend more than the one or two entries permitted above.] 1686 [something like the following from Diablo might also be useful: 1687 >>> NOTE <<< you should grep through newly created spool directories 1688 every so often looking for .MISMATCH in the spool files to locate 1689 incoming feeds with improperly configured I found that four of my 80+ 1690 feeds were misconfigured. ] 1691 News Article Architecture and Protocols January 2006 1693 NOTE: In order to prevent overloading, relaying agents should 1694 not routinely query an external entity (such as a DNS-server) in 1695 order to verify an article (though a local cache of the required 1696 information might usefully be consulted). 1698 2. It MUST examine the Injection-Date header field (or, if that is 1699 absent, the Date header field) and reject the article as stale 1700 (F-3.2.1) if that predates the earliest articles of which it 1701 normally keeps record, or if it is more than 24 hours into the 1702 future (the margin MAY be less than that 24 hours). 1704 3. It SHOULD reject any article that does not include all the 1705 mandatory header fields (section F-3.1). 1707 4. It MAY reject any article whose header fields do not have legal 1708 contents. 1710 5. It SHOULD reject any article that has already been sent to it (a 1711 database of message identifiers of recent messages is usually kept 1712 and matched against). 1714 NOTE: Even though commonly derived from the domain name of the 1715 originating site (and domain names are case-insensitive), a 1716 message identifier MUST NOT be altered in any way during 1717 transport, or when copied (as when forming a References header 1718 field), and thus a simple (case-sensitive) comparison of octets 1719 will always suffice to recognize that same message identifier 1720 wherever it subsequently reappears. 1722 NOTE: These requirements are to be contrasted with those of the 1723 un-normalized msg-ids defined by [RFC 2822], which may perfectly 1724 legitimately become normalized (or vice versa) during transport 1725 or copying in email systems. 1727 6. It SHOULD reject any article that matches an already received 1728 cancel message (or an equivalent Supersedes header field) issued 1729 by its poster or by some other trusted entity. 1731 7. It MAY reject any article without an Approved header field posted 1732 to newsgroups known to be moderated (this practice is strongly 1733 recommended, but the information necessary to do so may not be 1734 available to all agents). 1736 8. It MAY delete any Xref header field that is present. 1738 9. Finally, it passes the articles on to neighbouring relaying and 1739 serving agents. 1741 If the article is rejected as being invalid, unwanted or unacceptable 1742 due to site policy, the agent that passed the article to the relaying 1743 agent SHOULD be informed (such as via an NNTP 43x response code) that 1744 relaying failed. In order to prevent a large number of error messages 1745 being sent to one location, relaying agents MUST NOT inform any other 1746 external entity that an article was not relayed UNLESS that external 1747 News Article Architecture and Protocols January 2006 1749 entity has explicitly requested that it be informed of such errors. 1751 Relaying agents MUST NOT alter, delete or rearrange any part of an 1752 article except for header fields designated as variant (2.4). In 1753 particular 1755 o they MUST NOT create or augment a User-Agent header field in 1756 order to identify themselves; 1757 o they MUST NOT rewrite the Newsgroups header field in any way, 1758 even if some supposedly non-existent newsgroup is included; 1759 o they MUST NOT refold any header field (i.e. they must pass on the 1760 folding as received); 1761 o they MUST NOT alter the Date header field or the Injection-Date 1762 header field; 1763 o they MUST NOT delete any unrecognized header field whose field 1764 name is syntactically correct (whether or not it is registered 1765 with IANA [RFC 3864]); 1766 o they MUST NOT change the Content-Transfer-Encoding of the body or 1767 any body part; 1768 o they MUST transmit lines of arbitrary length and articles of 1769 arbitrary size. 1771 7.3.1. Path Header Field Example 1773 Path: foo.isp.example!!foo-server!!bar.isp.example!MISMATCH! 1774 2001:DB8:0:0:8:800:200C:417A!!old.site.example!barbaz!! 1775 baz.isp.example!POSTED!!dialup123.baz.isp.example!not-for-mail 1777 NOTE: That article was injected into the news stream by 1778 baz.isp.example, as indicated by the "POSTED" 1779 (complaints may be addressed to abuse@baz.isp.example). The 1780 injector has chosen to record that it got it from 1781 dialup123.baz.isp.example. "not-for-mail" is a dummy , though sometimes a real userid is put there. 1784 The article was relayed, perhaps by UUCP, to the machine known, 1785 at least to old.site.example, as "barbaz". 1787 Barbaz relayed it to old.site.example, which does not yet 1788 conform to this standard (hence the '!' ) to have verified that it came from 1794 old.site.example. 1796 [2001:DB8:0:0:8:800:200C:417A] relayed it to "foo-server" which, 1797 not being convinced that it truly came from 1798 [2001:DB8:0:0:8:800:200C:417A], inserted the 1799 "MISMATCH" and then did a reverse lookup on the actual source 1800 and concluded it was known as bar.isp.example (that is not to 1801 say that [2001:DB8:0:0:8:800:200C:417A] was not a correct IPv6 1802 address for bar.isp.example, but simply that that connection 1803 News Article Architecture and Protocols January 2006 1805 could not be substantiated by foo-server). Observe that foo- 1806 server has now added two entries to the Path. 1808 "foo-server" is a locally significant name within the complex 1809 site of many machines run by foo.isp.example, so the latter 1810 should have no problem recognizing foo-server and using a '!!' 1811 . Presumably foo.isp.example then delivered the 1812 article to its direct clients. 1814 It appears that foo-server and barbaz decided to fold the line, 1815 on the grounds that it seemed to be getting a little too long. 1817 7.4. Duties of a Serving Agent 1819 A Serving Agent takes an article from a relaying or injecting agent 1820 and files it in a "news database". It also provides an interface for 1821 reading agents to access the news database. This database is normally 1822 indexed by newsgroup with articles in each newsgroup identified by an 1823 (usually in the form of a decimal number - see F- 1824 3.2.11). 1826 A serving agent MUST maintain a list of the newsgroups it stores in 1827 its news database showing the moderation status of each one (see 1828 6.2.1), and SHOULD include in that list all groups likely to be 1829 crossposted to from those groups (e.g. all other groups in the same 1830 hierarchy(ies)). 1832 NOTE: Since control messages are often of interest, but should 1833 not be displayed as normal articles in regular newsgroups, it is 1834 common for serving agents to make them available in a pseudo- 1835 newsgroup named "control" or in a pseudo-newsgroup in a sub- 1836 hierarchy under "control." (e.g. "control.cancel"). 1838 A serving agent MAY decline to accept an article if the Path header 1839 field contains some whose articles the serving agent 1840 does not want, as a matter of local policy. 1842 NOTE: This last facility is sometimes used to detect and decline 1843 control messages (notably cancel messages) which have been 1844 deliberately seeded with a to be "aliased out" 1845 by sites not wishing to act upon them. 1846 [INN at least does this. It might be argued that it is not necessary to 1847 mention it here.] 1849 A serving agent processes articles as follows: 1851 1. It MUST establish the trusted identity of the source of the 1852 article and modify the Path header field as for a relaying agent 1853 (7.3). 1855 2. It MUST examine the Injection-Date header field (or, if that is 1856 absent, the Date header field) and reject the article as stale 1857 (F-3.2.1) if that predates the earliest articles of which it 1858 normally keeps record, or if it is more than 24 hours into the 1859 News Article Architecture and Protocols January 2006 1861 future (the margin MAY be less than that 24 hours). 1863 3. It MUST reject any article that does not include all the mandatory 1864 header fields (section F-3.1), or which contains any header field 1865 that does not have legal contents. 1867 4. It SHOULD reject any article that has already been sent to it (a 1868 database of message identifiers of recent articles is usually kept 1869 and matched against). 1871 5. It SHOULD reject any article that matches an already received 1872 cancel message (or an equivalent Supersedes header field) issued 1873 by its poster or by some other trusted entity. 1875 6. It MUST reject any article without an Approved header field posted 1876 to any newsgroup listed as moderated. 1878 7. It MUST (exept when specially configured to preserve the 1879 s set by the sending site) remove any Xref header 1880 field (F-3.2.11) from each article. It then MAY (and usually 1881 will) generate a fresh Xref header field. 1883 8. Finally, it stores the article in its news database. 1885 Serving agents MUST NOT create new newsgroups simply because an 1886 unrecognized occurs in a Newsgroups header field 1887 (see 6.2.1 for the correct method of newsgroup creation). 1889 Serving agents MUST NOT alter, delete or rearrange any part of an 1890 article in any other way. The list of particular cases given for 1891 relaying agents (7.3) applies here also. 1893 7.5. Duties of a Posting Agent 1895 A Posting Agent is used to assist the poster in creating a valid 1896 proto-article and forwarding it to an injecting agent. 1898 Postings agents SHOULD ensure that proto-articles they create are 1899 valid according to [USEFOR] and other applicable policies. In 1900 particular, they MUST NOT create any Injection-Date or Injection-Info 1901 header field. 1903 Contrary to [RFC 2822], which implies that the mailbox(es) in the 1904 From header field should be that of the poster(s), a poster who does 1905 not, for whatever reason, wish to use his own mailbox MAY use any 1906 mailbox ending in the top level domain ".invalid" [RFC 2606]. 1908 Posting agents meant for use by ordinary posters SHOULD reject any 1909 attempt to post an article which cancels or Supersedes another 1910 article of which the poster is not the author or sender. 1912 News Article Architecture and Protocols January 2006 1914 7.6. Duties of a Followup Agent 1916 A Followup Agent is a special case of a posting agent, and as such is 1917 bound by all the posting agent's requirements. Followup agents MUST 1918 create valid followups and are subject to special requirements 1919 involving the Newsgroups, Subject, Distribution and References header 1920 fields. Wherever in the following it is stated that, "by default", a 1921 header field is to be "inherited" from one of those header fields in 1922 the precursor, it means that its initial (semantic) content is to be 1923 a copy of the content of that precursor header field. However, 1924 posters MAY then override that default before posting if they so 1925 wish. 1927 NOTE: The Keywords header field is not inheritable, though some 1928 older newsreaders treated it as such. 1930 1. The Newsgroups header field (F-3.1.5) SHOULD by default be 1931 inherited from the precursor's Followup-To header field if 1932 present, and otherwise from the precursor's Newsgroups header 1933 field. However, if the content of that Followup-To header field 1934 consists of "poster" (and the user does not override it), then the 1935 followup MUST NOT be posted but, rather, is to be emailed to the 1936 precursor's poster. 1938 2. The Subject header field SHOULD by default be inherited from that 1939 of the precursor. The case sensitive string "Re: " MAY be 1940 prepended to the content of its Subject header field, unless it 1941 already begins with that string. 1943 3. The Distribution header field (F-3.2.7) SHOULD by default be 1944 inherited from the precursor's Distribution header field, if any. 1946 4. The followup MUST (in accordance with the definition of that term) 1947 have a References header field referring to its precursor, 1948 constructed in accordance with section 7.6.1 below. 1950 NOTE: That "MUST" is to be contrasted with the weaker 1951 recomendation using "SHOULD" applied, in [RFC 2822], to the 1952 generation of "replies" in email. Moreover, in Netnews, there is 1953 no expectation of any In-Reply-To header field in a followup. 1955 7.6.1. Construction of the References header field 1957 The following procedure is to be used whenever some previous article 1958 (the "parent") is to be referred to in the References header field 1959 (F-3.2.2) of a new article, whether in the course of generating a 1960 followup or for some other reason (e.g. the later parts of a 1961 multipart posting such as a FAQ, or the later parts of a 1962 message/partial as suggested in [RFC 2046]). 1964 The (semantic) content of the new article's References header field 1965 consists of the content of the Message-ID header field of the parent 1966 preceded, if the parent had a References header field, by the content 1967 of that References header field and a SP (subject to trimming as 1968 News Article Architecture and Protocols January 2006 1970 described below). 1972 If the resulting References header field would, after unfolding, 1973 exceed 998 characters in length (including its field name but not the 1974 final CRLF), it MUST be trimmed (and otherwise it MAY be trimmed). 1975 Trimming involves removing any number of message identifiers from its 1976 content, except that the first message identifier and the last two 1977 MUST NOT be removed. 1979 NOTE: There is no provision in this standard for an article to 1980 have more than one parent. The essential property of the 1981 References header field, guaranteed by the procedure above and 1982 to be preserved in any future extension, is that no article can 1983 ever precede one of its own parents. 1985 7.7. Duties of a Reading Agent 1987 A reading agent downloads articles from a serving agent, as directed 1988 by the reader, and displays them to the reader (or processes them in 1989 some other manner). It SHOULD also have the capability to show the 1990 raw article exactly as received. 1992 It MAY present lists of articles available for display, and MAY 1993 structure those lists so as to show the relationships between the 1994 articles, as determined by the References, Subject, Date and other 1995 header fields (see [USEAGE] for some usual methods of doing this). 1996 [This whole section may yet get omitted] 1998 7.8. Duties of a Moderator 2000 A Moderator receives news articles, customarily by email, decides 2001 whether to approve them and, if so, either injects them into the news 2002 stream or forwards them to further moderators. 2004 Articles will be received by the moderator either encapsulated as an 2005 object of Content-Type application/news-transmission (or possibly 2006 encapsulated but without an explicit Content-Type header field), or 2007 else directly as an email already containing all the header fields 2008 appropriate for a Netnews article (see 7.2.2). Moderators SHOULD be 2009 prepared to accept articles in either format. 2011 A moderator processes an article, as submitted to any newsgroup that 2012 he moderates, as follows: 2014 1. He decides, on the basis of whatever moderation policy applies to 2015 his group, whether to approve or reject the article. He MAY do 2016 this manually, or else partially or wholly with the aid of 2017 appropriate software for whose operation he is then responsible. 2018 If the article is a cancel nessage (6.3) issued by the poster of 2019 an earlier article, then he is expected to cancel that earlier 2020 article (in which case there is no more to be done). He MAY 2021 modify the article if that is in accordance with the applicable 2022 moderation policy (and in particular he MAY remove redundant 2023 header fields and add Comments and other informational header 2024 News Article Architecture and Protocols January 2006 2026 fields). He also needs to be aware if any change he makes to the 2027 article will invalidate some authentication check provided by the 2028 poster or by an earlier moderator. 2030 If the article is rejected, then it normally fails for all the 2031 newsgroups for which it was intended. If it is approved, the 2032 moderator proceeds with the following steps. 2034 2. If the Newsgroups header field contains further moderated 2035 newsgroups for which approval has not already been given, he adds 2036 an indication (identifying both himself and the name of the group) 2037 that he approves the article, and then forwards it to the 2038 moderator of the leftmost unapproved group (which, if this 2039 standard has been followed correctly, will generally be the next 2040 moderated group to the right of his own). There are two ways to do 2041 this: 2043 (a) He emails it to the submission address of the next moderator 2044 (see section 7.2.2 for the proper method of doing this), or 2046 (b) he rotates the s in the Newsgroups header 2047 field to the left so that the targeted group is the leftmost 2048 moderated group in that field, and injects it again (thus 2049 causing the injecting agent to forward it to the correct 2050 moderator). However, he MUST first ensure that the article 2051 contains no Approved header field. 2053 NOTE: This standard does not prescribe how a moderator's 2054 approval is to be indicated (though a future standard may do 2055 so). Possible methods include adding an Approved header field 2056 (or a similar but differently named header field if method (b) 2057 is being used) listing all the approvals made so far, or adding 2058 a separate header field for each individual approval (the header 2059 field X-Auth is sometimes used for this purpose). The approval 2060 may also be confirmed with some form of digital signature (6.1). 2062 3. If the Newsgroups header field contains no further unapproved 2063 moderated groups, he adds an Approved header field (F-3.2.9) 2064 identifying himself and, insofar as is possible, all the other 2065 moderators who have approved the article. He thus assumes 2066 responsibility for having ensured that the article was approved by 2067 the moderators of all the moderated groups involved. 2069 4. The Date header field SHOULD be retained. Any Injection-Date 2070 header field already present (though there should be none) MUST be 2071 removed. Exceptionally, if it is known that the injecting agent 2072 does not yet support the Injection-Date header field and the Date 2073 header field appears to be stale (F-3.2.1) for reasons understood 2074 by the moderator (e.g. delays in the moderation process) he MAY 2075 substitute the current date. The Message-ID header field SHOULD 2076 also be retained unless it is obviously non-compliant with this 2077 standard. 2079 News Article Architecture and Protocols January 2006 2081 NOTE: A message identifier created by a conforming posting or 2082 injecting agent, or even by a mail user agent conforming to [RFC 2083 2822], may reasonably be supposed to be conformant (and will, in 2084 any case, be caught by the injecting agent if it is not). 2086 5. Any variant header fields (2.4) MUST be removed, except that a 2087 Path header field MAY be truncated to only those entries following 2088 its "POSTED" . Any Injection-Info header field (F- 2089 3.2.14) SHOULD be removed (and if not, the injecting agent will do 2090 so, as required in 7.2.2). 2092 6. He then causes the article to be injected, having first observed 2093 all the duties of a posting agent. 2095 NOTE: This standard does not prescribe how the moderator or 2096 moderation policy for each newsgroup is established; rather it 2097 assumes that whatever agencies are responsible for the relevant 2098 network or hierarchy (1.1) will have made appropriate 2099 arrangements in that regard. 2101 7.9. Duties of a Gateway 2103 A Gateway transforms an article into the native message format of 2104 another medium, or translates the messages of another medium into 2105 news articles. Encapsulation of a news article into a message of MIME 2106 type application/news-transmission, or the subsequent undoing of that 2107 encapsulation, is not gatewaying, since it involves no transformation 2108 of the article. 2110 There are two basic types of gateway, the Outgoing Gateway that 2111 transforms a news article into a different type of message, and the 2112 Incoming Gateway that transforms a message from another medium into a 2113 news article and injects it into a news system. These are handled 2114 separately below. 2116 The primary diktat for a gateway is: 2118 Above all, prevent loops. 2120 Transformation of an article into another medium stands a very high 2121 chance of discarding or interfering with the protection inherent in 2122 the news system against duplicate articles. The most common problem 2123 caused by gateways is "spews", gateway loops that cause previously 2124 posted articles to be reinjected repeatedly into Usenet. To prevent 2125 this, a gateway MUST take precautions against loops, as detailed 2126 below. 2128 If bidirectional gatewaying (both an incoming and an outgoing 2129 gateway) is being set up between Netnews and some other medium, the 2130 incoming and outgoing gateways SHOULD be coordinated to avoid 2131 unintended reinjection of gated articles. Circular gatewaying 2132 (gatewaying a message into another medium and then back into Netnews) 2133 SHOULD NOT be done; encapsulation of the article SHOULD be used 2134 instead where this is necessary. 2136 News Article Architecture and Protocols January 2006 2138 A second general principal of gatewaying is that the transformations 2139 applied to the message SHOULD be as minimal as possible while still 2140 accomplishing the gatewaying. Every change made by a gateway 2141 potentially breaks a property of one of the media or loses 2142 information, and therefore only those transformations made necessary 2143 by the differences between the media should be applied. 2145 It is worth noting that safe bidirectional gatewaying between a 2146 mailing list and a newsgroup is far easier if the newsgroup is 2147 moderated. Posts to the moderated group and submissions to the 2148 mailing list can then go through a single point that does the 2149 necessary gatewaying and then sends the message out to both the 2150 newsgroup and the mailing list at the same time, eliminating most of 2151 the possibility of loops. Bidirectional gatewaying between a mailing 2152 list and an unmoderated newsgroup, in contrast, is difficult to do 2153 correctly and is far more fragile. 2155 Newsgroups intended to be bidirectionally gated to a mailing list 2156 SHOULD therefore be moderated where possible, even if the moderator 2157 is a simple gateway and injecting agent that correctly handles 2158 crossposting to other moderated groups and otherwise passes all 2159 traffic. 2161 7.9.1. Duties of an Outgoing Gateway 2163 From the perspective of Netnews, an outgoing gateway is just a 2164 special type of reading agent. The exact nature of what the outgoing 2165 gateway will need to do to articles depends on the medium to which 2166 the articles are being gated. The operation of the outgoing gateway 2167 is subject to additional constraints due to the possibility of one or 2168 more corresponding incoming gateways back from that medium to 2169 Netnews, since this opens the possibility of loops. 2171 In general, the following practices are recommended for all outgoing 2172 gateways, regardless of whether there is known to be a related 2173 incoming gateway, both as a precautionary measure and as a guideline 2174 to quality of implementation. 2176 1. The message identifier of the news article should be preserved if 2177 at all possible, preferably as or within the corresponding unique 2178 identifier of the other medium, but if not at least as a comment 2179 in the message. This helps greatly with preventing loops. 2181 2. The Date and Injection-Date of the news article should also be 2182 preserved if possible, for similar reasons. 2184 3. The message should be tagged in some way so as to prevent its 2185 reinjection into Netnews. This may be impossible to do without 2186 knowledge of potential incoming gateways, but it is better to try 2187 to provide some indication even if not successful; at the least, a 2188 human-readable indication that the article should not be gated 2189 back to Netnews can help locate a human problem. 2191 News Article Architecture and Protocols January 2006 2193 4. Netnews control messages should not be gated to another medium 2194 unless they would somehow be meaningful in that medium. 2196 5. Changes MAY be made to the Content-Transfer-Encoding of some or 2197 all parts of the body, and even to the charsets specified in 2198 s or in Content-Type header fields, but such changes 2199 SHOULD NOT be made unless absolutely necessary. 2201 7.9.2. Duties of an Incoming Gateway 2203 The incoming gateway has the serious responsibility of ensuring that 2204 all of the requirements of this standard are met by the articles that 2205 it forms. In addition to its special duties as a gateway, it bears 2206 all of the duties and responsibilities of an injecting agent as well, 2207 and additionally has the same responsibility of a relaying agent to 2208 reject articles that it has already gatewayed. 2210 An incoming gateway MUST NOT gate the same message twice. It may not 2211 be possible to ensure this in the face of mangling or modification of 2212 the message, but at the very least a gateway, when given a copy of a 2213 message it has already gated identical except for trace header fields 2214 (like Received in Email or Path in Netnews) MUST NOT gate the message 2215 again. An incoming gateway SHOULD take precautions against having 2216 this rule bypassed by modifications of the message that can be 2217 anticipated. 2219 News articles prepared by gateways MUST be legal news articles. In 2220 particular, they MUST include all of the mandatory header fields, 2221 MUST fully conform to the restrictions on those fields, and SHOULD 2222 exclude any deprecated header fields (e.g. as in [RFC 2298]). This 2223 often requires that a gateway function not only as a relaying agent, 2224 but also partly as a posting agent, aiding in the synthesis of a 2225 conforming article from non-conforming input. 2227 Incoming gateways MUST NOT pass control messages (articles containing 2228 a Control or Supersedes header field) without removing or renaming 2229 that header field. Gateways MAY, however, generate their own cancel 2230 messages, under the general allowance for injecting agents to cancel 2231 their own messages ([USEAGE]). If a gateway receives a message that 2232 it can determine is a valid equivalent of a cancel message in the 2233 medium it is gatewaying, it SHOULD discard that message without 2234 gatewaying it, generate a corresponding cancel message of its own, 2235 and inject that cancel message. 2237 Incoming gateways MUST NOT inject control messages other than 2238 cancels. Encapsulation SHOULD be used instead of gatewaying, when 2239 direct posting is not possible or desirable. 2241 NOTE: It is not unheard of for mail-to-news gateways to be used 2242 to post control messages, but encapsulation should be used for 2243 these cases instead. Gateways by their very nature are 2244 particularly prone to loops. Spews of normal articles are bad 2245 enough; spews of control messages with special significance to 2246 the news system, possibly resulting in high processing load or 2247 News Article Architecture and Protocols January 2006 2249 even email sent for every message received, are catastrophic. It 2250 is far preferable to construct a system specifically for posting 2251 control messages that can do appropriate consistency checks and 2252 authentication of the originator of the message. 2254 If there is a message identifier that fills a role similar to that of 2255 the Message-ID header field in news, it SHOULD be used in the 2256 formation of the message identifier of the news article, perhaps with 2257 transformations required to meet the uniqueness requirement of 2258 Netnews and with the removal of any comments so as to comply with the 2259 syntax in section F-3.1.3. Such transformations SHOULD be designed so 2260 that two messages with the same identifier generate the same 2261 Message-ID header field. 2263 NOTE: Message identifiers play a central role in the prevention 2264 of duplicates, and their correct use by gateways will do much to 2265 prevent loops. Netnews does, however, require that message 2266 identifiers be unique, and therefore message identifiers from 2267 other media may not be suitable for use without modification. A 2268 balance must be struck by the gateway between preserving 2269 information used to prevent loops and generating unique message 2270 identifiers. 2272 Exceptionally, if there are multiple incoming gateways for a 2273 particular set of messages, each to a different newsgroup(s), each 2274 one SHOULD generate a message identifier unique to that gateway. Each 2275 incoming gateway nonetheless MUST ensure that it does not gate the 2276 same message twice. 2278 NOTE: Consider the example of two gateways of a given mailing 2279 list into the world-wide Usenet newsgroups, both of which 2280 preserve the email message identifier. Each newsgroup may then 2281 receive a portion of the messages (different sites seeing 2282 different portions). In these cases, where there is no one 2283 "official" gateway, some other method of generating message 2284 identifiers has to be used to avoid collisions. It would 2285 obviously be preferable for there to be only one gateway which 2286 crossposts, but this may not be possible to coordinate. 2288 If no date information is available, the gateway MAY supply a Date 2289 header field with the gateway's current date. If no injection-date 2290 information is available, the gateway MUST supply an Injection-Date 2291 header field with whatever date information is available, and 2292 otherwise with the gateway's current date. If only partial 2293 information is available (e.g. date but not time), this SHOULD be 2294 fleshed out to a full Date and/or Injection-Date header field by 2295 adding default values rather than discarding this information. Only 2296 in very exceptional circumstances should Date information be 2297 discarded, as it plays an important role in preventing reinjection of 2298 old messages. 2300 An incoming gateway MUST add a Sender header field to the news 2301 article it forms containing the of the administrator of the 2302 gateway. Problems with the gateway may be reported to this 2303 News Article Architecture and Protocols January 2006 2305 . The portion of this SHOULD 2306 indicate that the entity responsible for injection of the message is 2307 a gateway. If the original message already had a Sender header field, 2308 it SHOULD be renamed so that its contents can be preserved. 2310 7.9.3. Example 2312 To illustrate the type of precautions that should be taken against 2313 loops, here is an example of the measures taken by one particular 2314 combination of mail-to-news and news-to-mail gateways at Stanford 2315 University designed to handle bidirectional gatewaying between 2316 mailing lists and unmoderated groups. 2318 1. The news-to-mail gateway preserves the message identifier of the 2319 news article in the generated email message. The mail-to-news 2320 gateway likewise preserves the email message identifier provided 2321 that it is syntactically valid for Netnews. This allows the news 2322 system's built-in suppression of duplicates to serve as the first 2323 line of defense against loops. 2325 2. The news-to-mail gateway adds an X-Gateway header field to all 2326 messages it generates. The mail-to-news gateway discards any 2327 incoming messages containing this header field. This is robust 2328 against mailing list managers that replace the message identifier, 2329 and against any number of email hops, provided that the other 2330 message header fields are preserved. 2332 3. The mail-to-news gateway prepends the host name from which it 2333 received the email message to the content of the Path header 2334 field. The news-to-mail gateway refuses to gateway any message 2335 that contains the list server name in its Path header field. This 2336 is robust against any amount of munging of the message header 2337 fields by the mailing list, provided that the email only goes 2338 through one hop. 2340 4. The mail-to-news gateway is designed never to generate bounces to 2341 the envelope sender. Instead, articles that are rejected by the 2342 news server (for reasons not warranting silent discarding of the 2343 message) result in a bounce message sent to an errors address 2344 known not to forward to any mailing lists, so that they can be 2345 handled by the news administrators. 2347 These precautions have proven effective in practice at preventing 2348 loops for this particular application (bidirectional gatewaying 2349 between mailing lists and locally distributed newsgroups where both 2350 gateways can be designed together). General gatewaying to world-wide 2351 newsgroups poses additional difficulties; one must be very wary of 2352 strange configurations, such as a newsgroup gated to a mailing list 2353 which is in turn gated to a different newsgroup. 2355 News Article Architecture and Protocols January 2006 2357 8. Security and Related Considerations 2359 There is no security. Don't fool yourself. Usenet is a prime example 2360 of an Internet Adhocratic-Anarchy; that is, an environment in which 2361 trust forms the basis of all agreements. It works. 2363 See also F-5 for further security considerations related to the 2364 format of articles. 2365 [And a similar pointer from there to here might be in order.] 2367 8.1. Leakage 2369 Articles which are intended to have restricted distribution are 2370 dependent on the goodwill of every site receiving them. The 2371 "Archive: no" header field (F-3.2.12) is available as a signal to 2372 automated archivers not to file an article, but that cannot be 2373 guaranteed. 2375 The Distribution header field makes provision for articles which 2376 should not be propagated beyond a cooperating subnet. The key 2377 security word here is "cooperating". When a machine is not configured 2378 properly, it may become uncooperative and tend to distribute all 2379 articles. 2381 The flooding algorithm is extremely good at finding any path by which 2382 articles can leave a subnet with supposedly restrictive boundaries, 2383 and substantial administrative effort is required to avoid this. 2384 Organizations wishing to control such leakage are strongly advised to 2385 designate a small number of official gateways to handle all news 2386 exchange with the outside world (however, making such gateways too 2387 restrictive can also encourage the setting up of unofficial paths 2388 which can be exceedingly hard to track down). 2390 The sendme control message (6.4), insofar as it is still used, can be 2391 used to request articles with a given message identifier, even one 2392 that is not supposed to be supplied to the requestor. 2394 8.2. Attacks 2396 8.2.1. Denial of Service 2398 The proper functioning of individual newsgroups can be disrupted by 2399 the massive posting of "noise" articles, by the repeated posting of 2400 identical or near identical articles, by posting followups unrelated 2401 to their precursors, or which quote their precursors in full with the 2402 addition of minimal extra material (especially if this process is 2403 iterated), and by crossposting to, or setting followups to, totally 2404 unrelated newsgroups. 2406 Many have argued that "spam", massively multiposted (and to a lesser 2407 extent massively crossposted) articles, usually for advertising 2408 purposes, also constitutes a DoS attack in its own regard. This may 2409 be so. 2411 News Article Architecture and Protocols January 2006 2413 Such articles intended to deny service, or other articles of an 2414 inflammatory nature, may also have their From or Reply-To addresses 2415 set to valid but incorrect email addresses, thus causing large 2416 volumes of email to descend on the true owners of those addresses. 2418 Similar effects could be caused by any email header field which could 2419 cause every reading agent receiving it to take some externally 2420 visible action. For example, the Disposition-Notification-To header 2421 field defined in [RFC 2298] could cause huge numbers of 2422 acknowledgements to be emailed to an unsuspecting third party (for 2423 which reason [RFC 2298] declares that that header field SHOULD NOT be 2424 used in Netnews). 2426 It is a violation of this standard for a poster to use as his address 2427 a which he is not entitled to use. Even addresses with an 2428 invalid but a valid can cause disruption to the 2429 administrators of such domains. Posters who wish to remain anonymous 2430 or to prevent automated harvesting of their addresses, but who do not 2431 care to take the additional precautions of using more sophisticated 2432 anonymity measures, should avoid that violation by the use of 2433 addresses ending in the ".invalid" top-level-domain (see 7.5). 2435 A malicious poster may also prevent his article being seen at a 2436 particular site by preloading that site into the Path header field 2437 (F-3.1.6) and may thus prevent the true owner of a forged From or 2438 Reply-To address from ever seeing it. 2440 A malicious complainer may submit a modified copy of an article (e.g. 2441 with an altered Injection-Info header field) to the administrator of 2442 an injecting agent in an attempt to discredit the author of that 2443 article and even to have his posting privileges removed. 2444 Administrators should therefore obtain a genuine copy of the article 2445 from their own serving agent before taking such precipitate action. 2447 Administrative agencies with responsibility for establishing policies 2448 in particular hierarchies can and should set bounds upon the 2449 behaviour that is considered acceptable within those hierarchies (for 2450 example by promulgating charters for individual newsgroups, and other 2451 codes of conduct). 2453 Whilst this standard places an onus upon injecting agents to bear 2454 responsibility for the misdemeanours of their posters (which includes 2455 non-adherence to established policies of the relevant hierarchies as 2456 provided in section 7.2), and to provide assistance to the rest of 2457 the network by making proper use of the Injection-Info (F-3.2.14) 2458 header field, it makes no provision for enforcement, which may in 2459 consequence be patchy. Nevertheless, injecting sites which 2460 persistently fail to honour their responsibilities or to comply with 2461 generally accepted standards of behaviour are likely to find 2462 themselves blacklisted, with their articles refused propagation and 2463 even subject to cancellation, and other relaying sites would be well 2464 advised to withdraw peering arrangements from them. 2466 News Article Architecture and Protocols January 2006 2468 8.2.2. Compromise of System Integrity 2470 The posting of unauthorized (as determined by the policies of the 2471 relevant hierarchy) control messages can cause unwanted newsgroups to 2472 be created, or wanted ones removed, from serving agents. 2473 Administrators of such agents SHOULD therefore take steps to verify 2474 the authenticity of such control messages, either by manual 2475 inspection (particularly of the Approved header field) or by checking 2476 any digital signatures that may be provided (see 6.1). In addition, 2477 they SHOULD periodically compare the newsgroups carried against any 2478 regularly issued checkgroups messages, or against lists maintained by 2479 trusted servers and accessed by out-of-band protocols such as FTP or 2480 HTTP. 2482 Malicious cancel messages (6.3) can cause valid articles to be 2483 removed from serving agents. Administrators of such agents SHOULD 2484 therefore take steps to verify that they originated from the 2485 (apparent) poster, the injector or the moderator of the article, or 2486 that in other cases they came from a place that is trusted to work 2487 within established policies and customs. Such steps SHOULD include 2488 the checking of any digital signatures, or other security devices, 2489 that may be provided (see 6.1). Articles containing Supersedes 2490 header fields (F-3.2.6) are effectively cancel messages, and SHOULD 2491 be subject to the same checks. Currently, many sites choose to 2492 ignore all cancel messages on account of the difficulty of conducting 2493 such checks. 2495 Improperly configured serving agents can allow articles posted to 2496 moderated groups onto the net without first being approved by the 2497 moderator. Injecting agents SHOULD verify that moderated articles 2498 were received from one of the entities given in their Approved header 2499 fields and/or check any digital signatures that may be provided (see 2500 6.1). 2502 There may be weaknesses in particular implementations that are 2503 subject to malicious exploitation. In particular, it has not been 2504 unknown for complete shell scripts to be included within Control 2505 header fields. Implementors need to be aware of this. 2507 Reading agents should be chary of acting automatically upon MIME 2508 objects with an "application" Content-Type that could change the 2509 state of that agent, except in contexts where such applications are 2510 specifically expected (as in 5). Even the Content-Type "text/html" 2511 could have unexpected side effects on account of embedded objects, 2512 especially embedded executable code or URIs that invoke non-news 2513 protocols such as HTTP [RFC 2616]. It is therefore generally 2514 recommended that reading agents do not enable the execution of such 2515 code (since it is extremely unlikely to have a valid application 2516 within Netnews) and that they only honour URIs referring to other 2517 parts of the same article. 2519 Non-printable characters embedded in article bodies may have 2520 surprising effects on printers or terminals, notably by reconfiguring 2521 them in undesirable ways which may become apparent only after the 2522 News Article Architecture and Protocols January 2006 2524 reading agent has terminated. 2526 8.3. Liability 2528 There is a presumption that a poster who sends an article to Usenet 2529 intends it to be stored on a multitude of serving agents, and has 2530 therefore given permission for it to be copied to that extent. 2531 Nevertheless, Usenet is not exempt from the Copyright laws, and it 2532 should not be assumed that permission has been given for the article 2533 to be copied outside of Usenet, nor for its permanent archiving 2534 contrary to any Archive header field that may be present. 2536 Posters also need to be aware that they are responsible if they 2537 breach Copyright, Libel, Harassment or other restrictions relating to 2538 material that they post, and that they may possibly find themselves 2539 liable for such breaches in jurisdictions far from their own. Serving 2540 agents may also be liable in some jurisdictions, especially if the 2541 breach has been explicitly drawn to their attention. 2543 Users who are concerned about such matters should seek advice from 2544 competent legal authorities. 2546 9. IANA Considerations 2548 IANA is requested to register the following media types, described 2549 elsewhere in this standard, for use with the Content-Type header 2550 field, in the IETF tree in accordance with the procedures set out in 2551 [RFC 2048]. 2553 application/news-transmission (5.1) 2554 application/news-groupinfo (5.3) 2555 application/news-checkgroups (5.4) 2557 IANA is also requested to change the status of the following media 2558 type to "OBSOLETE". 2560 message/news (5.2) 2562 NOTE: "Application/news-transmission" is an update, with 2563 clarification and additional optional parameters, to an existing 2564 registration. "Message/rfc822" should now be used in place of 2565 the obsoleted "message/news". 2567 10. References 2569 10.1. Normative References 2571 [ANSI X3.4] "American National Standard for Information Systems - 2572 Coded Character Sets - 7-Bit American National Standard Code for 2573 Information Interchange (7-Bit ASCII)", ANSI X3.4, 1986. 2575 News Article Architecture and Protocols January 2006 2577 [RFC 2048] N. Freed, J. Klensin, and J. Postel, "Multipurpose 2578 Internet Mail Extensions (MIME) Part Four: Registration 2579 Procedures", RFC 2048, November 1996. 2581 [RFC 2119] S. Bradner, "Key words for use in RFCs to Indicate 2582 Requirement Levels", RFC 2119, March 1997. 2584 [RFC 2606] D. Eastlake and A. Panitz, "Reserved Top Level DNS Names", 2585 RFC 2606, June 1999. 2587 [RFC 2822] P. Resnick, "Internet Message Format", RFC 2822, April 2588 2001. 2590 [RFC 3864] G. Klyne, M. Nottingham, and J. Mogul, "Registration 2591 procedures for message header fields", RFC 3864. 2593 [USEAGE] draft-ietf-usefor-useage-*.txt. 2595 [USEFOR] K. Murchison et al, "News Article Format", draft-ietf- 2596 usefor-usefor-*.txt. 2598 [USEPRO] This Standard. 2600 10.2. Informative References 2602 [NNTP] Clive D.W. Feather, "Network News Transport Protocol", draft- 2603 ietf-nntpext-base-*.txt. 2605 [PGPVERIFY] David Lawrence, 2606 . 2608 [RFC 1036] M. Horton and R. Adams, "Standard for Interchange of 2609 USENET Messages", RFC 1036, December 1987. 2611 [RFC 2045] N. Freed and N. Borenstein, "Multipurpose Internet Mail 2612 Extensions (MIME) Part One: Format of Internet Message Bodies", 2613 RFC 2045, November 1996. 2615 [RFC 2046] N. Freed and N. Borenstein, "Multipurpose Internet Mail 2616 Extensions (MIME) Part Two: Media Types", RFC 2046, November 2617 1996. 2619 [RFC 2142] D. Crocker, "Mailbox Names for Common Services, Roles and 2620 Functions", RFC 2142, May 1997. 2622 [RFC 2298] R. Fajman, "An Extensible Message Format for Message 2623 Disposition Notifications", RFC 2298, March 1998. 2625 [RFC 2616] R. Fielding, J. Gettys, J. Mogul, H. Frystyk, L. Masinter, 2626 P. Leach, and T. Berners-Lee, "Hypertext Transfer Protocol -- 2627 HTTP/1.1", RFC 2616, June 1999. 2629 News Article Architecture and Protocols January 2006 2631 [RFC 2821] John C. Klensin and Dawn P. Mann, "Simple Mail Transfer 2632 Protocol", RFC 2821, April 2001. 2634 [RFC 976] Mark R. Horton, "UUCP mail interchange format standard", 2635 RFC 976, February 1986. 2637 [Son-of-1036] Henry Spencer, "News article format and transmission", 2638 , June 1994. 2640 11. Acknowledgements 2642 As this document is the result of an eight year effort, the number of 2643 people that have contributed to its content are too numerous to 2644 mention individually. Many thanks go out to all past and present 2645 members of the USEFOR Working Group of the Internet Engineering Task 2646 Force (IETF) and the accompanying mailing list. 2648 12. Contact Address 2650 Editor 2652 Charles. H. Lindsey 2653 5 Clerewood Avenue 2654 Heald Green 2655 Cheadle 2656 Cheshire SK8 3JU 2657 United Kingdom 2658 Phone: +44 161 436 6131 2659 Email: chl@clerew.man.ac.uk 2661 [ 2663 Working group chairs 2665 Alexey Melnikov 2666 Harald Tveit Alvestrand 2667 ] 2669 Comments on this draft should preferably be sent to the mailing list 2670 of the Usenet Format Working Group at 2672 ietf-usefor@imc.org. 2674 Appendix A - Obsolete Control Messages 2676 This present standard obsoletes certain control messages defined in 2677 [RFC 1036] (see 6.5), all of which had the effect of requesting a 2678 description of a relaying or serving agent's software, or its peering 2679 arrangements with neighbouring sites, to be emailed to the article's 2680 reply address. Whilst of some utility when Usenet was much smaller 2681 than it is now, they had become no more than a tool for the malicious 2682 sending of mailbombs. Moreover, many organizations now consider 2683 information about their internal connectivity to be confidential. 2685 News Article Architecture and Protocols January 2006 2687 version 2688 sendsys 2689 whogets 2690 senduuname 2692 "Version" requested details of the transport software in use at a 2693 site. "Sendsys" requested the full list of newsgroups taken, and the 2694 peering arrangements. "Whogets" was similar, but restricted to a 2695 named newsgroup. "Senduuname" resembled "sendsys" but restricted to 2696 the list of peers connected by UUCP. 2698 Historically, a checkgroups body consisting of one or two lines, the 2699 first of the form "-n newsgroup", caused check-groups to apply to 2700 only that single newsgroup. 2702 Historically, an article posted to a newsgroup whose name had exactly 2703 three components of which the third was "ctl" signified that article 2704 was to be taken as a control message. The Subject header field 2705 specified the actions, in the same way the Control header field does 2706 now. 2708 These forms are documented for archaeological purposes only; they 2709 MUST NO LONGER be used. 2711 Appendix B - Notices 2713 Intellectual Property 2715 The IETF takes no position regarding the validity or scope of any 2716 Intellectual Property Rights or other rights that might be claimed to 2717 pertain to the implementation or use of the technology described in 2718 this document or the extent to which any license under such rights 2719 might or might not be available; nor does it represent that it has 2720 made any independent effort to identify any such rights. Information 2721 on the procedures with respect to rights in RFC documents can be 2722 found in BCP 78 and BCP 79. 2724 Copies of IPR disclosures made to the IETF Secretariat and any 2725 assurances of licenses to be made available, or the result of an 2726 attempt made to obtain a general license or permission for the use of 2727 such proprietary rights by implementers or users of this 2728 specification can be obtained from the IETF on-line IPR repository at 2729 http://www.ietf.org/ipr. 2731 The IETF invites any interested party to bring to its attention any 2732 copyrights, patents or patent applications, or other proprietary 2733 rights that may cover technology that may be required to implement 2734 this standard. Please address the information to the IETF at ietf- 2735 ipr@ietf.org. 2737 News Article Architecture and Protocols January 2006 2739 Full Copyright Statement 2741 Copyright (C) The Internet Society (2006). This document is subject 2742 to the rights, licenses and restrictions contained in BCP 78, and 2743 except as set forth therein, the authors retain all their rights. 2745 This document and the information contained herein are provided on an 2746 "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS 2747 OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET 2748 ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, 2749 INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE 2750 INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED 2751 WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. 2753 Appendix C - Change Log 2755 [This Appendix is to be removed prior to final publication.] 2757 For version 01 2759 1 Numerous texts describing protocol features related to 2760 particular header fields in parts of [ARTICLE] which were 2761 destined to become part of [USEFOR] have been moved to 2762 appropriate locations within section 7 of this document. Such 2763 revised texts will be found in sections 2764 7.2.2 Steps 4, 6, 7, 10, 11, 12; 2765 7.2.3 Step 1(b); 2766 7.3 introductory paragraphs, Steps 1, 4, 8, 9, and some final 2767 paragraphs; 2768 7.4 introductory and final paragraphs; 2769 7.9.1 Step 5. 2771 2 A section on "Duties of a Reading Agent" (7.8) has been added. 2773 3 Some demotions MUST -> SHOULD -> MAY, as noted in pseudo- 2774 comments, have been made or proposed in sections 2775 7.3 2776 7.3 Step 4. 2778 4 Part of the procedure for examining Path header fields by 2779 relaying agents has been moved to serving agents, as explained 2780 in pseudo-comments in section 7.4. 2782 5 Some renumbering of sections and minor textual clarifications. 2784 For version 02 2786 1 2nd para. of a-7 temporarily reinstated in section 6. 2788 2 Para. in section 6 relating to propagation of control messages 2789 and local policy removed to [USEAGE].] 2790 News Article Architecture and Protocols January 2006 2792 3 Requirement for some relaying agents to examine control messages 2793 for non-existent groups 2794 6 2795 7.3 2797 4 Text regarding "aliasing out" brought into line with actual 2798 practice. 2799 7.3 2801 5 More realistic wording regarding the expectations of reading 2802 agents 2803 7.7 2804 7.4 2806 6 "Precursor" is now defined for all cases in which a References 2807 header field may be used (even though "followup" is not always 2808 defined under Alternative-1). 2809 2.1 2811 7 Provision is made for a poster to use a mailbox ending in 2812 ".invalid" in a From header field (formerly in a-5.2). 2813 7.5 2815 8 "Inheritable" and "Variant" header fields defined (formerly in 2816 a-4.2.5). 2817 2.3 2819 9 Additional wording regarding function of verb/arguments/body in 2820 control messages (formerly in a-6.13). 2821 6 2823 10 NOTE regarding not altering message indentifiers during 2824 transport or copying added (formerly in a-5.3). 2825 7.3 2827 11 All mention of MIME-style parameters and extension-parameters 2828 removed. 2829 3.1 2830 7.6 2832 For version 03 2834 1 The term "inheritable header" is no longer defined. Instead, the 2835 term "inherited' is used in place of "taken" when defining the 2836 actions of a followup agent. 2837 7.6 2839 2 Consequent changes to "variant header field", and also mention 2840 of Injection-Info as sometimes variant. 2841 2.3 2843 3 The term "reply address" is no longer defined. 2845 News Article Architecture and Protocols January 2006 2847 4 References now made to sections within USEFOR using "F-..." 2848 notation. 2850 5 Cross-references to sections within USEFOR added. Consistent use 2851 of <...> around all mentions of syntactic objects. All 2852 occurrences of "Foobar-header" changed to "Foobar header". Many 2853 other minor textual changes. 2855 6 changed to , to avoid 2856 confusion with "control message", which signifies the complete 2857 article containing the . 2859 7 has been changed to (since 2860 the earlier practice of multiple arguments is now deprecated). 2861 Likewise . 2863 For version 04 2865 1 References divided into Normative and Informational. 2867 2 All mention of the Mail-Copies-To, Posted-And-Mailed and 2868 Complaints-To header fields removed. 2870 3 NOTE added to contrast MUST for References header field with 2871 SHOULD in RFC 2822. 2872 7.6 2874 4 Changes arising from the new syntax of s and 2875 s. 2876 7.3 2878 5 Changes to clarify the construction of the References header 2879 field. 2880 7.6.1 2882 6 Changes due to removal of s from further header fields. 2884 7 New section on Identification of news-servers describing 2885 acceptable forms for s. 2886 2.3 2888 8 Definition of "semantic content" of a header field. 2889 2.1 2891 9 Systematic replacement of "header" by "header field". 2893 10 More stringent rules for checking s in control 2894 messages for compliance with USEFOR. 2895 6.2 2897 For version 05 2898 News Article Architecture and Protocols January 2006 2900 1 Historical Appendices A.1 and A.2 removed, in anticipation of 2901 republication of Son-of-1036 as an Informational RFC. 2902 1.3 2904 2 Definitions of technical terms adopted from USEFOR rather than 2905 defining them separately. 2907 3 Discussion of rewritten to reflect recent 2908 developments (but still awaiting further work in USEFOR). 2909 2.3 2911 4 Items now included in Appendix A of USEFOR have been removed 2912 from section 3, and the "Transitional Arrangements" (which still 2913 cover the USEFOR changes) have been modified to reflect this.