idnits 2.17.1 draft-yergeau-rfc2279bis-04.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** Looks like you're using RFC 2026 boilerplate. This must be updated to follow RFC 3978/3979, as updated by RFC 4748. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- == No 'Intended status' indicated for this document; assuming Proposed Standard Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- -- The abstract seems to indicate that this document obsoletes RFC2279, but the header doesn't have an 'Obsoletes:' line to match this. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the RFC 3978 Section 5.4 Copyright Line does not match the current year -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (February 17, 2003) is 7731 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) ** Obsolete normative reference: RFC 2234 (Obsoleted by RFC 4234) -- Possible downref: Non-RFC (?) normative reference: ref. 'UNICODE' Summary: 2 errors (**), 0 flaws (~~), 2 warnings (==), 4 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group F. Yergeau 3 Internet-Draft Alis Technologies 4 Expires: August 18, 2003 February 17, 2003 6 UTF-8, a transformation format of ISO 10646 7 draft-yergeau-rfc2279bis-04 9 Status of this Memo 11 This document is an Internet-Draft and is in full conformance with 12 all provisions of Section 10 of RFC2026. 14 Internet-Drafts are working documents of the Internet Engineering 15 Task Force (IETF), its areas, and its working groups. Note that other 16 groups may also distribute working documents as Internet-Drafts. 18 Internet-Drafts are draft documents valid for a maximum of six months 19 and may be updated, replaced, or obsoleted by other documents at any 20 time. It is inappropriate to use Internet-Drafts as reference 21 material or to cite them other than as "work in progress." 23 The list of current Internet-Drafts can be accessed at 24 http://www.ietf.org/ietf/1id-abstracts.txt. 26 The list of Internet-Draft Shadow Directories can be accessed at 27 http://www.ietf.org/shadow.html. 29 This Internet-Draft will expire on August 18, 2003. 31 Copyright Notice 33 Copyright (C) The Internet Society (2003). All Rights Reserved. 35 Abstract 37 ISO/IEC 10646-1 defines a large character set called the Universal 38 Character Set (UCS) which encompasses most of the world's writing 39 systems. The originally proposed encodings of the UCS, however, were 40 not compatible with many current applications and protocols, and this 41 has led to the development of UTF-8, the object of this memo. UTF-8 42 has the characteristic of preserving the full US-ASCII range, 43 providing compatibility with file systems, parsers and other software 44 that rely on US-ASCII values but are transparent to other values. 45 This memo obsoletes and replaces RFC 2279. 47 Table of Contents 49 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 50 2. Notational conventions . . . . . . . . . . . . . . . . . . . . 4 51 3. UTF-8 definition . . . . . . . . . . . . . . . . . . . . . . . 4 52 4. Syntax of UTF-8 Byte Sequences . . . . . . . . . . . . . . . . 6 53 5. Versions of the standards . . . . . . . . . . . . . . . . . . 6 54 6. Byte order mark (BOM) . . . . . . . . . . . . . . . . . . . . 7 55 7. Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 56 8. MIME registration . . . . . . . . . . . . . . . . . . . . . . 9 57 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 10 58 10. Security Considerations . . . . . . . . . . . . . . . . . . . 10 59 11. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 11 60 12. Changes from RFC 2279 . . . . . . . . . . . . . . . . . . . . 11 61 Normative references . . . . . . . . . . . . . . . . . . . . . 12 62 Informative references . . . . . . . . . . . . . . . . . . . . 12 63 Author's Address . . . . . . . . . . . . . . . . . . . . . . . 13 64 Intellectual Property and Copyright Statements . . . . . . . . 14 66 1. Introduction 68 ISO/IEC 10646 [ISO.10646] defines a large character set called the 69 Universal Character Set (UCS), which encompasses most of the world's 70 writing systems. The same set of characters is defined by the Unicode 71 standard [UNICODE], which further defines additional character 72 properties and other application details of great interest to 73 implementers. Up to the present time, changes in Unicode and 74 amendments and additions to ISO/IEC 10646 have tracked each other, so 75 that the character repertoires and code point assignments have 76 remained in sync. The relevant standardization committees have 77 committed to maintain this very useful synchronism. 79 ISO/IEC 10646 and Unicode define several encoding forms of their 80 common repertoire: UTF-8, UCS-2, UTF-16, UCS-4 and UTF-32. In an 81 encoding form, each character is represented as one or more encoding 82 units. All standard UCS encoding forms except UTF-8 have an encoding 83 unit larger than one octet, making them hard to use in many current 84 applications and protocols that assume 8 or even 7 bit characters. 86 UTF-8, the object of this memo, has a one-octet encoding unit. It 87 uses all bits of an octet, but has the quality of preserving the full 88 US-ASCII [US-ASCII] range: US-ASCII characters are encoded in one 89 octet having the normal US-ASCII value, and any octet with such a 90 value can only stand for a US-ASCII character, and nothing else. 92 UTF-8 encodes UCS characters as a varying number of octets, where the 93 number of octets, and the value of each, depend on the integer value 94 assigned to the character in ISO/IEC 10646 (the character number, 95 a.k.a. code point or Unicode scalar value). This encoding form has 96 the following characteristics (all values are in hexadecimal): 98 o Character numbers from U+0000 to U+007F (US-ASCII repertoire) 99 correspond to octets 00 to 7F (7 bit US-ASCII values). A direct 100 consequence is that a plain ASCII string is also a valid UTF-8 101 string. 103 o US-ASCII octet values do not appear otherwise in a UTF-8 encoded 104 character stream. This provides compatibility with file systems 105 or other software (e.g. the printf() function in C libraries) that 106 parse based on US-ASCII values but are transparent to other 107 values. 109 o Round-trip conversion is easy between UTF-8 and other encoding 110 forms. 112 o The first octet of a multi-octet sequence indicates the number of 113 octets in the sequence. 115 o The octet values C0, C1, FE and FF never appear. If the range of 116 character numbers is restricted to U+0000..U+10FFFF (the UTF-16 117 accessible range), then the octet values F5..FD also never appear. 119 o Character boundaries are easily found from anywhere in an octet 120 stream. 122 o The lexicographic sorting order of UTF-8 strings is the same as if 123 ordered by character numbers. Of course this is of limited 124 interest since a sort order based on character numbers is not 125 culturally valid. 127 o The Boyer-Moore fast search algorithm can be used with UTF-8 data. 129 o UTF-8 strings can be fairly reliably recognized as such by a 130 simple algorithm, i.e. the probability that a string of characters 131 in any other encoding appears as valid UTF-8 is low, diminishing 132 with increasing string length. 134 UTF-8 was originally a project of the X/Open Joint 135 Internationalization Group XOJIG with the objective to specify a File 136 System Safe UCS Transformation Format [FSS_UTF] that is compatible 137 with UNIX systems, supporting multilingual text in a single encoding. 138 The original authors were Gary Miller, Greger Leijonhufvud and John 139 Entenmann. Later, Ken Thompson and Rob Pike did significant work for 140 the formal definition of UTF-8. 142 2. Notational conventions 144 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 145 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 146 document are to be interpreted as described in [RFC2119]. 148 UCS characters are designated by the U+HHHH notation, where HHHH is a 149 string of from 4 to 6 hexadecimal digits representing the character 150 number in ISO/IEC 10646. 152 3. UTF-8 definition 154 UTF-8 is defined by the Unicode Standard [UNICODE]. Descriptions and 155 formulae can also be found in Annex D of ISO/IEC 10646-1 [ISO.10646] 157 In UTF-8, characters from the U+0000..U+10FFFF range (the UTF-16 158 accessible range) are encoded using sequences of 1 to 4 octets. The 159 only octet of a "sequence" of one has the higher-order bit set to 0, 160 the remaining 7 bits being used to encode the character number. In a 161 sequence of n octets, n>1, the initial octet has the n higher-order 162 bits set to 1, followed by a bit set to 0. The remaining bit(s) of 163 that octet contain bits from the number of the character to be 164 encoded. The following octet(s) all have the higher-order bit set to 165 1 and the following bit set to 0, leaving 6 bits in each to contain 166 bits from the character to be encoded. 168 The table below summarizes the format of these different octet types. 169 The letter x indicates bits available for encoding bits of the 170 character number. 172 Char. number range | UTF-8 octet sequence 173 (hexadecimal) | (binary) 174 --------------------+--------------------------------------------- 175 0000 0000-0000 007F | 0xxxxxxx 176 0000 0080-0000 07FF | 110xxxxx 10xxxxxx 177 0000 0800-0000 FFFF | 1110xxxx 10xxxxxx 10xxxxxx 178 0001 0000-0010 FFFF | 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx 180 Encoding a character to UTF-8 proceeds as follows: 182 1. Determine the number of octets required from the character number 183 and the first column of the table above. It is important to note 184 that the rows of the table are mutually exclusive, i.e. there is 185 only one valid way to encode a given character. 187 2. Prepare the high-order bits of the octets as per the second 188 column of the table. 190 3. Fill in the bits marked x from the bits of the character number, 191 expressed in binary. Start by putting the lowest-order bit of the 192 character number in the lowest-order position of the last octet 193 of the sequence, then put the next higher-order bit of the 194 character number in the next higher-order position of that octet, 195 etc. When the x bits of the last octet are filled in, move on to 196 the next to last octet, then to the preceding one, etc. until all 197 x bits are filled in. 199 The definition of UTF-8 prohibits encoding character numbers between 200 U+D800 and U+DFFF, which are reserved for use with the UTF-16 201 encoding form (as surrogate pairs) and do not directly represent 202 characters. When encoding in UTF-8 from UTF-16 data, it is necessary 203 to first decode the UTF-16 data to obtain character numbers, which 204 are then encoded in UTF-8 as described above. This contrasts with 205 CESU-8 [CESU-8], which is a UTF-8-like encoding that is not meant for 206 use on the Internet. CESU-8 operates similarly to UTF-8 but encodes 207 the UTF-16 code values (16-bit quantities) instead of the character 208 number (code point). This leads to different results for character 209 numbers above 0xFFFF; the CESU-8 encoding of those characters is NOT 210 valid UTF-8. 212 Decoding a UTF-8 character proceeds as follows: 214 1. Initialize a binary number with all bits set to 0. Up to 21 bits 215 may be needed. 217 2. Determine which bits encode the character number from the number 218 of octets in the sequence and the second column of the table 219 above (the bits marked x). 221 3. Distribute the bits from the sequence to the binary number, first 222 the lower-order bits from the last octet of the sequence and 223 proceeding to the left until no x bits are left. The binary 224 number is now equal to the character number. 226 Implementations of the decoding algorithm above MUST protect against 227 decoding invalid sequences. For instance, a naive implementation may 228 decode the overlong UTF-8 sequence C0 80 into the character U+0000, 229 or the surrogate pair ED A1 8C ED BE B4 into U+233B4. Decoding 230 invalid sequences may have security consequences or cause other 231 problems. See Security Considerations (Section 10) below. 233 4. Syntax of UTF-8 Byte Sequences 235 A UTF-8 string is a sequence of octets representing a sequence of UCS 236 characters. An octet sequence is valid UTF-8 only if it matches the 237 following syntax, which is derived from the rules for encoding UTF-8 238 and is expressed in the ABNF of [RFC2234]. 240 UTF8-octets = *( UTF8-char ) 241 UTF8-char = UTF8-1 / UTF8-2 / UTF8-3 / UTF8-4 242 UTF8-1 = %x00-7F 243 UTF8-2 = %xC2-DF UTF8-tail 244 UTF8-3 = %xE0 %xA0-BF UTF8-tail / %xE1-EC 2( UTF8-tail ) / 245 %xED %x80-9F UTF8-tail / %xEE-EF 2( UTF8-tail ) 246 UTF8-4 = %xF0 %x90-BF 2( UTF8-tail ) / %xF1-F3 3( UTF8-tail ) / 247 %xF4 %x80-8F 2( UTF8-tail ) 248 UTF8-tail = %x80-BF 250 5. Versions of the standards 252 ISO/IEC 10646 is updated from time to time by publication of 253 amendments and additional parts; similarly, new versions of the 254 Unicode standard are published over time. Each new version obsoletes 255 and replaces the previous one, but implementations, and more 256 significantly data, are not updated instantly. 258 In general, the changes amount to adding new characters, which does 259 not pose particular problems with old data. In 1996, Amendment 5 to 260 the 1993 edition of ISO/IEC 10646 and Unicode 2.0 moved and expanded 261 the Korean Hangul block, thereby making any previous data containing 262 Hangul characters invalid under the new version. Unicode 2.0 has the 263 same difference from Unicode 1.1. The justification for allowing such 264 an incompatible change was that there were no major implementations 265 and no significant amounts of data containing Hangul. The incident 266 has been dubbed the "Korean mess", and the relevant committees have 267 pledged to never, ever again make such an incompatible change (see 268 Unicode Consortium Policies [1]). 270 New versions, and in particular any incompatible changes, have 271 consequences regarding MIME charset labels, to be discussed in MIME 272 registration (Section 8). 274 6. Byte order mark (BOM) 276 The UCS character U+FEFF "ZERO WIDTH NO-BREAK SPACE" is also known 277 informally as "BYTE ORDER MARK" (abbreviated "BOM"). This character 278 can be used as a genuine "ZERO WIDTH NO-BREAK SPACE" within text, but 279 the BOM name hints at a second possible usage of the character: to 280 prepend a U+FEFF character to a stream of UCS characters as a 281 "signature". A receiver of such a serialized stream may then use the 282 initial character as a hint that the stream consists of UCS 283 characters and also to recognize which UCS encoding is involved and, 284 with encodings having a multi-octet encoding unit, as a way to 285 recognize the serialization order of the octets. UTF-8 having a 286 single-octet encoding unit, this last function is useless and the BOM 287 will always appear as the octet sequence EF BB BF. 289 It is important to understand that the character U+FEFF appearing at 290 any position other than the beginning of a stream MUST be interpreted 291 with the semantics for the zero-width non-breaking space, and MUST 292 NOT be interpreted as a signature. When interpreted as a signature, 293 the Unicode standard suggests than an initial U+FEFF character may be 294 stripped before processing the text. Such stripping is necessary in 295 some cases (e.g. when concatenating two strings, because otherwise 296 the resulting string may contain an unintended "ZERO WIDTH NO-BREAK 297 SPACE" at the connection point), but might affect an external process 298 at a different layer (such as a digital signature or a count of the 299 characters) that is relying on the presence of all characters in the 300 stream. It is therefore RECOMMENDED to avoid stripping an initial 301 U+FEFF interpreted as a signature without a good reason, to ignore it 302 instead of stripping it when appropriate (such as for display) and to 303 strip it only when really necessary. 305 U+FEFF in the first position of a stream MAY be interpreted as a 306 zero-width non-breaking space, and is not always a signature. In an 307 attempt at diminishing this uncertainty, Unicode 3.2 adds a new 308 character, U+2060 "WORD JOINER", with exactly the same semantics and 309 usage as U+FEFF except for the signature function, and strongly 310 recommends its exclusive use for expressing word-joining semantics. 311 Eventually, following this recommendation will make it all but 312 certain that any initial U+FEFF is a signature, not an intended "ZERO 313 WIDTH NO-BREAK SPACE". 315 In the meantime, the uncertainty unfortunately remains and may affect 316 Internet protocols. Protocol specifications MAY restrict usage of 317 U+FEFF as a signature in order to reduce or eliminate the potential 318 ill effects of this uncertainty. In the interest of striking a 319 balance between the advantages (reduction of uncertainty) and 320 drawbacks (loss of the signature function) of such restrictions, it 321 is useful to distinguish a few cases: 323 o A protocol SHOULD forbid use of U+FEFF as a signature for those 324 textual protocol elements that the protocol mandates to be always 325 UTF-8, the signature function being totally useless in those 326 cases. 328 o A protocol SHOULD also forbid use of U+FEFF as a signature for 329 those textual protocol elements for which the protocol provides 330 character encoding identification mechanisms, when it is expected 331 that implementations of the protocol will be in a position to 332 always use the mechanisms properly. This will be the case when 333 the protocol elements are maintained tightly under the control of 334 the implementation from the time of their creation to the time of 335 their (properly labeled) transmission. 337 o A protocol SHOULD NOT forbid use of U+FEFF as a signature for 338 those textual protocol elements for which the protocol does not 339 provide character encoding identification mechanisms, when a ban 340 would be unenforceable, or when it is expected that 341 implementations of the protocol will not be in a position to 342 always use the mechanisms properly. The latter two cases are 343 likely to occur with larger protocol elements such as MIME 344 entities, especially when implementations of the protocol will 345 obtain such entities from file systems, from protocols that do not 346 have encoding identification mechanisms for payloads (such as FTP) 347 or from other protocols that do not guarantee proper 348 identification of character encoding (such as HTTP). 350 When a protocol forbids use of U+FEFF as a signature for a certain 351 protocol element, then any initial U+FEFF in that protocol element 352 MUST be interpreted as a "ZERO WIDTH NO-BREAK SPACE". When a protocol 353 does NOT forbid use of U+FEFF as a signature for a certain protocol 354 element, then implementations SHOULD be prepared to handle a 355 signature in that element and react appropriately: using the 356 signature to identify the character encoding as necessary and 357 stripping or ignoring the signature as appropriate. 359 7. Examples 361 The character sequence U+0041 U+2262 U+0391 U+002E "A." is encoded in UTF-8 as follows: 364 --+--------+-----+-- 365 41 E2 89 A2 CE 91 2E 366 --+--------+-----+-- 368 The character sequence U+D55C U+AD6D U+C5B4 (Korean "hangugeo", 369 meaning "the Korean language") is encoded in UTF-8 as follows: 371 --------+--------+-------- 372 ED 95 9C EA B5 AD EC 96 B4 373 --------+--------+-------- 375 The character sequence U+65E5 U+672C U+8A9E (Japanese "nihongo", 376 meaning "the Japanese language") is encoded in UTF-8 as follows: 378 --------+--------+-------- 379 E6 97 A5 E6 9C AC E8 AA 9E 380 --------+--------+-------- 382 The character U+233B4 (a Chinese character meaning 'stump of tree'), 383 prepended with a UTF-8 BOM, is encoded in UTF-8 as follows: 385 --------+----------- 386 EF BB BF F0 A3 8E B4 387 --------+----------- 389 8. MIME registration 391 This memo serves as the basis for registration of the MIME charset 392 parameter for UTF-8, according to [RFC2978]. The charset parameter 393 value is "UTF-8". This string labels media types containing text 394 consisting of characters from the repertoire of ISO/IEC 10646 395 including all amendments at least up to amendment 5 of the 1993 396 edition (Korean block), encoded to a sequence of octets using the 397 encoding scheme outlined above. UTF-8 is suitable for use in MIME 398 content types under the "text" top-level type. 400 It is noteworthy that the label "UTF-8" does not contain a version 401 identification, referring generically to ISO/IEC 10646. This is 402 intentional, the rationale being as follows: 404 A MIME charset label is designed to give just the information needed 405 to interpret a sequence of bytes received on the wire into a sequence 406 of characters, nothing more (see [RFC2045], section 2.2). As long as 407 a character set standard does not change incompatibly, version 408 numbers serve no purpose, because one gains nothing by learning from 409 the tag that newly assigned characters may be received that one 410 doesn't know about. The tag itself doesn't teach anything about the 411 new characters, which are going to be received anyway. 413 Hence, as long as the standards evolve compatibly, the apparent 414 advantage of having labels that identify the versions is only that, 415 apparent. But there is a disadvantage to such version-dependent 416 labels: when an older application receives data accompanied by a 417 newer, unknown label, it may fail to recognize the label and be 418 completely unable to deal with the data, whereas a generic, known 419 label would have triggered mostly correct processing of the data, 420 which may well not contain any new characters. 422 Now the "Korean mess" (ISO/IEC 10646 amendment 5) is an incompatible 423 change, in principle contradicting the appropriateness of a version 424 independent MIME charset label as described above. But the 425 compatibility problem can only appear with data containing Korean 426 Hangul characters encoded according to Unicode 1.1 (or equivalently 427 ISO/IEC 10646 before amendment 5), and there is arguably no such data 428 to worry about, this being the very reason the incompatible change 429 was deemed acceptable. 431 In practice, then, a version-independent label is warranted, provided 432 the label is understood to refer to all versions after Amendment 5, 433 and provided no incompatible change actually occurs. Should 434 incompatible changes occur in a later version of ISO/IEC 10646, the 435 MIME charset label defined here will stay aligned with the previous 436 version until and unless the IETF specifically decides otherwise. 438 9. IANA Considerations 440 The entry for UTF-8 in the IANA charset registry should be updated to 441 point to this memo. 443 10. Security Considerations 445 Implementers of UTF-8 need to consider the security aspects of how 446 they handle illegal UTF-8 sequences. It is conceivable that in some 447 circumstances an attacker would be able to exploit an incautious 448 UTF-8 parser by sending it an octet sequence that is not permitted by 449 the UTF-8 syntax. 451 A particularly subtle form of this attack can be carried out against 452 a parser which performs security-critical validity checks against the 453 UTF-8 encoded form of its input, but interprets certain illegal octet 454 sequences as characters. For example, a parser might prohibit the 455 NUL character when encoded as the single-octet sequence 00, but 456 erroneously allow the illegal two-octet sequence C0 80 and interpret 457 it as a NUL character. Another example might be a parser which 458 prohibits the octet sequence 2F 2E 2E 2F ("/../"), yet permits the 459 illegal octet sequence 2F C0 AE 2E 2F. This last exploit has actually 460 been used in a widespread virus attacking Web servers in 2001; the 461 security threat is thus very real. 463 Another security issue occurs when encoding to UTF-8: the ISO/IEC 464 10646 description of UTF-8 allows encoding character numbers up to 465 U+7FFFFFFF, yielding sequences of up to 6 bytes. There is therefore 466 a risk of buffer overflow if the range of character numbers is not 467 explicitly limited to U+10FFFF or if buffer sizing doesn't take into 468 account the possibility of 5- and 6-byte sequences. 470 11. Acknowledgements 472 The following have participated in the drafting and discussion of 473 this memo: James E. Agenbroad, Harald Alvestrand, Andries Brouwer, 474 Mark Davis, Martin J. Duerst, Patrick Faltstrom, Ned Freed, David 475 Goldsmith, Tony Hansen, Edwin F. Hart, Paul Hoffman, David Hopwood, 476 Simon Josefsson, Kent Karlsson, Dan Kohn, Markus Kuhn, Michael Kung, 477 Alain LaBonte, Ira McDonald, Alexey Melnikov, MURATA Makoto, John 478 Gardiner Myers, Dan Oscarsson, Roozbeh Pournader, Murray Sargent, 479 Markus Scherer, Keld Simonsen, Arnold Winkler, Kenneth Whistler and 480 Misha Wolf. 482 12. Changes from RFC 2279 484 o Restricted the range of characters to 0000-10FFFF (the UTF-16 485 accessible range). 487 o Made Unicode the source of the normative definition of UTF-8, 488 keeping ISO/IEC 10646 as the reference for characters. 490 o Straightened out terminology. UTF-8 now described in terms of an 491 encoding form of the character number. UCS-2 and UCS-4 almost 492 disappeared. 494 o Turned the note warning against decoding of invalid sequences into 495 a normative MUST NOT. 497 o Added a new section about the UTF-8 BOM, with advice for 498 protocols. 500 o Removed suggested UNICODE-1-1-UTF-8 MIME charset registration. 502 o Added an ABNF syntax for valid UTF-8 octet sequences 504 Normative references 506 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 507 Requirement Levels", BCP 14, RFC 2119, March 1997. 509 [RFC2234] Crocker, D. and P. Overell, "Augmented BNF for Syntax 510 Specifications: ABNF", RFC 2234, November 1997. 512 [ISO.10646] 513 International Organization for Standardization, 514 "Information Technology - Universal Multiple-octet coded 515 Character Set (UCS)", ISO/IEC Standard 10646, comprised 516 of ISO/IEC 10646-1:2000, "Information technology -- 517 Universal Multiple-Octet Coded Character Set (UCS) -- Part 518 1: Architecture and Basic Multilingual Plane", ISO/IEC 519 10646-2:2001, "Information technology -- Universal 520 Multiple-Octet Coded Character Set (UCS) -- Part 2: 521 Supplementary Planes" and ISO/IEC 10646-1:2000/Amd 1:2002, 522 "Mathematical symbols and other characters". 524 [UNICODE] The Unicode Consortium, "The Unicode Standard -- Version 525 3.2", defined by The Unicode Standard, Version 3.0 526 (Reading, MA, Addison-Wesley, 2000. ISBN 0-201-61633-5), 527 as amended by the Unicode Standard Annex #27: Unicode 3.1 528 (see http://www.unicode.org/reports/tr27) and by the 529 Unicode Standard Annex #28: Unicode 3.2 (see 530 http://www.unicode.org/reports/tr28), March 2002, 531 . 534 Informative references 536 [CESU-8] Phipps, T., "Compatibility Encoding Scheme for UTF-16: 537 8-Bit (CESU-8)", UTR 26, April 2002, 538 . 540 [FSS_UTF] X/Open Company Ltd., "X/Open CAE Specification C501 -- 541 File System Safe UCS Transformation Format (FSS_UTF)", 542 ISBN 1-85912-082-2, April 1995. 544 [RFC2045] Freed, N. and N. Borenstein, "Multipurpose Internet Mail 545 Extensions (MIME) Part One: Format of Internet Message 546 Bodies", RFC 2045, November 1996. 548 [RFC2978] Freed, N. and J. Postel, "IANA Charset Registration 549 Procedures", BCP 19, RFC 2978, October 2000. 551 [US-ASCII] 552 American National Standards Institute, "Coded Character 553 Set - 7-bit American Standard Code for Information 554 Interchange", ANSI X3.4, 1986. 556 URIs 558 [1] 560 Author's Address 562 Francois Yergeau 563 Alis Technologies 564 100, boul. Alexis-Nihon, bureau 600 565 Montreal, QC H4M 2P2 566 Canada 568 Phone: +1 514 747 2547 569 Fax: +1 514 747 2561 570 EMail: fyergeau@alis.com 572 Intellectual Property Statement 574 The IETF takes no position regarding the validity or scope of any 575 intellectual property or other rights that might be claimed to 576 pertain to the implementation or use of the technology described in 577 this document or the extent to which any license under such rights 578 might or might not be available; neither does it represent that it 579 has made any effort to identify any such rights. Information on the 580 IETF's procedures with respect to rights in standards-track and 581 standards-related documentation can be found in BCP-11. Copies of 582 claims of rights made available for publication and any assurances of 583 licenses to be made available, or the result of an attempt made to 584 obtain a general license or permission for the use of such 585 proprietary rights by implementors or users of this specification can 586 be obtained from the IETF Secretariat. 588 The IETF invites any interested party to bring to its attention any 589 copyrights, patents or patent applications, or other proprietary 590 rights which may cover technology that may be required to practice 591 this standard. Please address the information to the IETF Executive 592 Director. 594 Full Copyright Statement 596 Copyright (C) The Internet Society (2003). All Rights Reserved. 598 This document and translations of it may be copied and furnished to 599 others, and derivative works that comment on or otherwise explain it 600 or assist in its implementation may be prepared, copied, published 601 and distributed, in whole or in part, without restriction of any 602 kind, provided that the above copyright notice and this paragraph are 603 included on all such copies and derivative works. However, this 604 document itself may not be modified in any way, such as by removing 605 the copyright notice or references to the Internet Society or other 606 Internet organizations, except as needed for the purpose of 607 developing Internet standards in which case the procedures for 608 copyrights defined in the Internet Standards process must be 609 followed, or as required to translate it into languages other than 610 English. 612 The limited permissions granted above are perpetual and will not be 613 revoked by the Internet Society or its successors or assignees. 615 This document and the information contained herein is provided on an 616 "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING 617 TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING 618 BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION 619 HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF 620 MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. 622 Acknowledgement 624 Funding for the RFC Editor function is currently provided by the 625 Internet Society.