idnits 2.17.1 draft-yergeau-utf8-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** Cannot find the required boilerplate sections (Copyright, IPR, etc.) in this document. Expected boilerplate is as follows today (2024-04-20) according to https://trustee.ietf.org/license-info : IETF Trust Legal Provisions of 28-dec-2009, Section 6.a: This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. IETF Trust Legal Provisions of 28-dec-2009, Section 6.b(i), paragraph 2: Copyright (c) 2024 IETF Trust and the persons identified as the document authors. All rights reserved. IETF Trust Legal Provisions of 28-dec-2009, Section 6.b(i), paragraph 3: This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- ** Missing expiration date. The document expiration date should appear on the first and last page. ** The document seems to lack a 1id_guidelines paragraph about Internet-Drafts being working documents. ** The document seems to lack a 1id_guidelines paragraph about 6 months document validity -- however, there's a paragraph with a matching beginning. Boilerplate error? ** The document seems to lack a 1id_guidelines paragraph about the list of current Internet-Drafts. ** The document seems to lack a 1id_guidelines paragraph about the list of Shadow Directories. == No 'Intended status' indicated for this document; assuming Proposed Standard Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack an IANA Considerations section. (See Section 2.2 of https://www.ietf.org/id-info/checklist for how to handle the case when there are no actions for IANA.) Miscellaneous warnings: ---------------------------------------------------------------------------- -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (1 October 1996) is 10063 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) -- Missing reference section? 'UNICODE' on line 245 looks like a reference -- Missing reference section? 'ISO-10646' on line 226 looks like a reference -- Missing reference section? 'US-ASCII' on line 250 looks like a reference -- Missing reference section? 'RFC1642' on line 241 looks like a reference -- Missing reference section? 'FSS-UTF' on line 102 looks like a reference -- Missing reference section? 'UNI-CODE' on line 108 looks like a reference -- Missing reference section? 'RFC1521' on line 233 looks like a reference -- Missing reference section? 'RFC1641' on line 238 looks like a reference Summary: 7 errors (**), 0 flaws (~~), 1 warning (==), 10 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 Network Working Group F. Yergeau 2 Internet Draft Alis Technologies 3 27 March 1996 4 Expires 1 October 1996 6 UTF-8, a transformation format of Unicode and ISO 10646 8 Status of this Memo 10 This document is an Internet-Draft. Internet-Drafts are working doc- 11 uments of the Internet Engineering Task Force (IETF), its areas, and 12 its working groups. Note that other groups may also distribute work- 13 ing documents as Internet-Drafts. 15 Internet-Drafts are draft documents valid for a maximum of six 16 months. Internet-Drafts may be updated, replaced, or obsoleted by 17 other documents at any time. It is not appropriate to use Internet- 18 Drafts as reference material or to cite them other than as a "working 19 draft" or "work in progress". 21 To learn the current status of any Internet-Draft, please check the 22 1id-abstracts.txt listing contained in the Internet-Drafts Shadow 23 Directories on ds.internic.net (US East Coast), nic.nordu.net 24 (Europe), ftp.isi.edu (US West Coast), or munnari.oz.au (Pacific 25 Rim). 27 Distribution of this document is unlimited. 29 Abstract 31 The Unicode Standard, version 1.1, and ISO/IEC 10646-1:1993 jointly 32 define a 16 bit character set which encompasses most of the world's 33 writing systems. 16-bit characters, however, are not compatible with 34 many current applications and protocols, and this has led to the 35 development of a few so-called UCS transformation formats (UTF), each 36 with different characteristics. UTF-8, the object of this memo, has 37 the characteristic of preserving the full US-ASCII range: US-ASCII 38 characters are encoded in one octet having the usual US-ASCII value, 39 and any octet with such a value can only be an US-ASCII character. 40 This provides compatibility with file systems, parsers and other 41 software that rely on US-ASCII values but are transparent to other 42 values. 44 1. Introduction 46 The Unicode Standard, version 1.1 [UNICODE], and ISO/IEC 10646-1:1993 47 [ISO-10646] jointly define a 16 bit character set, UCS-2, which 48 encompasses most of the world's writing systems. ISO 10646 further 49 defines a 32 bit character set, UCS-4, with currently no assignments 50 outside of the region corresponding to UCS-2 (the Basic Multilingual 51 Plane, BMP). The UCS-2 and UCS-4 encodings, however, are hard to use 52 in many current applications and protocols that assume 8 or even 7 53 bit characters. Even newer systems able to deal with 16 bit charac- 54 ters cannot process UCS-4 data. This situation has led to the devel- 55 opment of so-called UCS transformation formats (UTF), each with dif- 56 ferent characteristics. 58 UTF-1 has only historical interest, having been removed from ISO 59 10646. UTF-7 has the quality of encoding the full Unicode repertoire 60 using only octets with the high-order bit clear (7 bit US-ASCII val- 61 ues, [US-ASCII]), and is thus deemed a mail-safe encoding 62 ([RFC1642]). UTF-8, the object of this memo, uses all bits of an 63 octet, but has the quality of preserving the full US-ASCII range: US- 64 ASCII characters are encoded in one octet having the normal US-ASCII 65 value, and any octet with such a value can only stand for an US-ASCII 66 character, and nothing else. 68 UTF-16 is a scheme for transforming a subset of the UCS-4 repertoire 69 into a pair of UCS-2 values from a reserved range. UTF-16 impacts 70 UTF-8 in that UCS-2 values from the reserved range must be treated 71 specially in the UTF-8 transformation. 73 UTF-8 encodes UCS-2 or UCS-4 characters as a varying number of 74 octets, where the number of octets and the value of each depend on 75 the integer value assigned to the character in ISO 10646. This 76 transformation format has the following characteristics (all values 77 are in hexadecimal): 79 - Character values from 0000 0000 to 0000 007E (US-ASCII repertoire) 80 correspond to octets 00 to 7E (7 bit US-ASCII values). 82 - US-ASCII values do not appear otherwise in a UTF-8 encoded charac- 83 ter stream. This provides compatibility with file systems or 84 other software that parse based on US-ASCII values but are trans- 85 parent to other values. 87 - Round-trip conversion is easy between UTF-8 and either of UCS-4, 88 UCS-2 or Unicode. 90 - The first octet of a multi-octet sequence indicates the number of 91 octets in the sequence. 93 - Character boundaries are easily found from anywhere in an octet 94 stream. 96 - The lexicographic sorting order of UCS-4 strings is preserved. 98 - The octet values FE and FF never appear. 100 UTF-8 was originally a project of the X/Open Joint Internationaliza- 101 tion Group XOJIG with the objective to specify a File System Safe UCS 102 Transformation Format [FSS-UTF] that is compatible with UNIX systems, 103 supporting multilingual text in a single encoding. The original 104 authors were Gary Miller, Greger Leijonhufvud and John Entenmann. 105 Later, Ken Thompson and Rob Pike did significant work for the formal 106 UTF-8. 108 A description can also be found in Unicode Technical Report #4 [UNI- 109 CODE]. The definitive reference, including provisions for UTF-16 110 data within UTF-8, will be Annex P of ISO/IEC 10646-1 [ISO-10646]. 112 2. UTF-8 definition 114 In UTF-8, characters are encoded using sequences of 1 to 6 octets. 115 The only octet of a "sequence" of one has the higher-order bit set to 116 0, the remaining 7 bits being used to encode the character value. In 117 a sequence of n octets, n>1, the initial octet has the n higher-order 118 bits set to 1, followed by a bit set to 0. The remaining bit(s) of 119 that octet is(are) used to encode bits from the initial character. 120 The following octet(s) all have the higher-order bit set to 1 and the 121 following bit set to 0, leaving 6 bits in each to encode bits from 122 the initial character. 124 The table below shows the format of these different octet types. The 125 letter x indicates bits available for encoding bits of the initial 126 character. 128 Octet Binary Free bits Max. character value 129 1st of 1 0xxxxxxx 7 0000 007F 130 1st of 2 110xxxxx 5 0000 07FF 131 1st of 3 1110xxxx 4 0000 FFFF 132 1st of 4 11110xxx 3 001F FFFF 133 1st of 5 111110xx 2 03FF FFFF 134 1st of 6 1111110x 1 7FFF FFFF 135 2nd to 6th 10xxxxxx 6 137 Encoding from UCS-4 to UTF-8 proceeds as follows: 139 1) determine the number of octets required from the character value 140 and the fourth column of the table above. 142 2) Prepare the high-order bits of the octets as per the second column 143 of the table. 145 3) Fill in the bits marked x from the bits of the character value, 146 starting from the lower-order bits of the character value and 147 putting them first in the last octet of the sequence, then the 148 next to last, etc. until all x bits are filled in. 150 The algorithm for encoding UCS-2 (or Unicode) to UTF-8 can be 151 obtained from the above, in principle, by simply extending each 152 UCS-2 character with two zero-valued octets. However, UCS-2 val- 153 ues between D800 and DFFF, being actually UCS-4 characters trans- 154 formed trough UTF-16, need special treatment: the UTF-16 transfor- 155 mation must be undone, yielding a UCS-4 character that is then 156 transformed as above. 158 Decoding from UTF-8 to UCS-4 proceeds as follows: 160 1) Initialize the 4 octets of the UCS-4 character with all bits set 161 to 0. 163 2) Determine which bits encode the character value from the number of 164 octets in the sequence and the second column of the table above 165 (the bits marked x). 167 3) Distribute the bits from the sequence to the UCS-4 character, 168 first the lower-order bits from the last octet of the sequence and 169 proceeding to the left until no x bits are left. 171 If the UTF-8 sequence is no more than three octets long, decoding 172 can proceed directly to UCS-2 (or equivalently Unicode). 174 A more detailed algorithm and formulae can be found in [FSS_UTF], 175 [UNICODE] or the forthcoming Annex P to [ISO-10646]. 177 3. Examples 179 The Unicode sequence "A." (0041, 2262, 0391, 180 002E) may be encoded as follows: 182 41 E2 89 A2 CE 91 2E 184 The Unicode sequence "Hi Mom !" (0048, 0069, 185 0020, 004D, 006F, 004D, 0020, 263A, 0021) may be encoded as follows: 187 48 69 20 4D 6F 4D 20 E2 98 BA 21 189 The Unicode sequence representing the Han characters for the Japanese 190 word "nihongo" (65E5, 672C, 8A9E) may be encoded as follows: 192 E6 97 A5 E6 9C AC E4 AA 9E 194 MIME registrations 196 This memo is meant to serve as the basis for registration of a MIME 197 character encoding (charset) as per [RFC1521]. The proposed charset 198 parameter value is "UTF-8". This string would label media types con- 199 taining text consisting of characters from the repertoire of ISO 200 10646-1 encoded to a sequence of octets using the encoding scheme 201 outlined above. 203 Security Considerations 205 Security issues are not discussed in this memo. 207 Acknowledgments 209 The following have participated in the drafting and discussion of 210 this memo: 212 Andries Brouwer Martin J. Drst 213 David Goldsmith Edwin F. Hart 214 Markus Kuhn Michael Kung 215 Alain LaBont Murray Sargent 216 Keld Simonsen Arnold Winkler 218 Bibliography 220 [FSS_UTF] X/Open CAE Specification C501 ISBN 1-85912-082-2 28cm. 221 22p. pbk. 172g. 4/95, X/Open Company Ltd., "File Sys- 222 tem Safe UCS Transformation Format (FSS_UTF)", X/Open 223 Preleminary Specification, Document Number P316. Also 224 published in Unicode Technical Report #4. 226 [ISO-10646] ISO/IEC 10646-1:1993. International Standard -- Infor- 227 mation technology -- Universal Multiple-Octet Coded 228 Character Set (UCS) -- Part 1: Architecture and Basic 229 Multilingual Plane. UTF-8 is described in Annex P, 230 adopted but not yet published. UTF-16 is described in 231 Annex O, adopted but not yet published. 233 [RFC1521] N. Borenstein and N. Freed, "MIME (Multipurpose Inter- 234 net Mail Extensions) Part One: Mechanisms for Specify- 235 ing and Describing the Format of Internet Message Bod- 236 ies", RFC 1521, Bellcore, Innosoft, September 1993. 238 [RFC1641] D. Goldsmith, M.Davis, "Using Unicode with MIME", RFC 239 1641, Taligent inc., July 1994. 241 [RFC1642] D. Goldsmith, M. Davis, "UTF-7: A Mail-safe Transfor- 242 mation Format of Unicode", RFC 1642, Taligent inc., 243 July 1994. 245 [UNICODE] The Unicode Consortium, "The Unicode Standard -- 246 Worldwide Character Encoding -- Version 1.0", Addison- 247 Wesley, Volume 1, 1991, Volume 2, 1992. UTF-8 is 248 described in Unicode Technical Report #4. 250 [US-ASCII] Coded Character Set--7-bit American Standard Code for 251 Information Interchange, ANSI X3.4-1986. 253 Author's Address 255 Franois Yergeau 256 Alis Technologies 257 100, boul. Alexis-Nihon 258 Suite 600 259 Montral QC H4M 2P2 260 Canada 262 Tel: +1 (514) 747-2547 263 Fax: +1 (514) 747-2561 264 EMail: fyergeau@alis.com