Internet Draft Paul Hoffman
draft-hoffman-i18n-terms-01.txt IMC & VPNC
January 17, 2001
Expires in six months
Terminology Used in Internationalization in the IETF
Status of this memo
This document is an Internet-Draft and is in full conformance with all
provisions of Section 10 of RFC2026.
Internet-Drafts are working documents of the Internet Engineering Task
Force (IETF), its areas, and its working groups. Note that other groups
may also distribute working documents as Internet-Drafts.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference material
or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html.
Abstract
This document provides a glossary of terms used in the IETF when
discussing internationalization. The purpose is to help frame
discussions of internationalization in the various areas of the IETF and
to help introduce the main concepts to IETF participants.
1. Introduction
As [RFC2277] summarizes: "Internationalization is for humans. This means
that protocols are not subject to internationalization; text strings
are." Many protocols throughout the IETF use text strings that are
entered by, or are visible to, humans. It should be possible to make
these text strings readable to anyone, which means that the text must be
able to be displayed in any human language. This is the challenge of
internationalization.
1.1 About this document
Internationalization is discussed in many working groups of the IETF.
However, few working groups have internationalization experts. When
designing or updating protocols, the question often comes up "should we
internationalize this" (or, more likely, "do we have to internationalize
this").
This document gives an overview of internationalization as it applies to
IETF standards work by covering lightly the many aspects of
internationalization and the vocabulary associated with those topics. It
is not meant to be a complete description of internationalization. The
definitions in this document come from many earlier IETF documents and
books.
As in many fields, there is disagreement in the internationalization
community on definitions for many words. The topic of language brings up
particularly passionate opinions for experts and non-experts alike. This
document attempts to define terms that will be most useful to the IETF
audience.
Note that this is a very early draft of the document. Many definitions
here will likely change, and some topics may be added. Discussion of
this document is encouraged. Information on the mailing list for this
document can be found at .
1.2 Foundations for internationalization
language
A language is a way that humans interact. The use of language occurs in
many forms, the most common of which are writing, vocal, and visual.
Each language form is independent: some languages have a close
relationship between the written and vocal forms, while others have a
looser relationship. [RFC1766] and [RFC1766bis] discuss languages in
more detail.
script
A script is a collection of symbols used to represent textual
information in one or more writing systems. A script can have many
attributes, such as the number of written characters and the rules for
combining written characters. Most IETF protocols that deal with
languages deal only with scripts, not spoken or visual forms. [RFC2277]
discusses scripts in more detail.
It is common for internationalization novices to mix up the terms
"language" and "script". This can be a problem in protocols that
differentiate the two, such as mail content protocols. Almost all
internationalized protocols deal with scripts (the written systems),
while fewer deal with languages. Many languages can be expressed using
different scripts.
grapheme, phoneme, alphabet
A grapheme is an abstract, atomic written entity of a script. A phoneme
is an abstract, atomic spoken entity of a spoken language. An alphabet
is a script that maps between graphemes and phonemes.
internationalization
In the IETF, the verb "internationalize" means to add or improve the
handling of international information in a protocol. Many protocols that
handle text only handle one script, the one that contains the letters
used in English text. Internationalizing such a protocol allows
the protocol to handle more scripts, hopefully all of the ones
useful to anyone in the world.
localization
Internationalized applications can handle a wide variety of languages.
Typical users only understand a small number of languages, so the
program must be tailored to interact with users in just the languages
they know. Localization involves not only changing the language
interaction, but also other relevant changes such as display of numbers,
dates, currency, and so on.
i18n, l10n
These are abbreviations for "internationalization" and "localization".
"18" is the number of characters between the "i" and the "n" in
"internationalization", and "10" is the number of characters between the
"l" and the "n" in "localization".
2. Fundamental Terms
This section covers basic topics that are needed for almost anyone who
is involved with internationalization of IETF protocols. Many terms in
this section are based on [IDN-REQ].
characters
A character is a member of a set of elements used for organization,
control, or representation of data. In written form, a language is
expressed in characters. The same set of characters can often be used
to express many languages.
In ISO 10646 [ISO10646], a character is identified by its name, not by
its shape. A name may suggest a meaning, but the character may be used
for representing other meanings as well. A name may suggest a shape, but
that does not imply that just that is commonly used in print.
coded character
A coded character is a character with its coded representation. A coded
character set (CCS) is a set of unambiguous rules that establish a
character set and the relationship between the characters of the set and
their coded representation. The specified set of characters in a CCS is
often called the "repertoire".
character encoding scheme
A character encoding scheme (CES) is a mapping from one or more coded
character sets to a set of octets. Some CESs are associated with a
single CCS; for example, UTF-8 [RFC2279] applies only to ISO 10646.
Other CESs, such as ISO 2022, are associated with many CCSs.
charset
A charset is a method of mapping a sequence of octets to a sequence of
abstract characters. A charset is, in effect, a combination of one or
more CCS with a CES. Charset names are registered by the IANA according
to procedures documented in [RFC2278]. A particular charset may have
different glyphs depending on the language being used.
transfer encoding syntax
A transfer encoding syntax (TES) (sometimes called a transfer encoding
scheme) is a reversible transform of already-encoded data represented in
one or more character encoding schemes. TESs are useful for encoding
types of character data into an another format, usually for allowing new
types of data to be transmitted over legacy protocols. Examples of TESs
used in the IETF include Base64 and quoted-printable (described in more
detail in Chapter 6).
3. Standards Bodies and Standards
This section describes some of the standards bodies and standards that
appear in discussions of internationalization in the IETF. This is an
incomplete and possibly over-full list; listing too few bodies or
standards can be just as politically dangerous as listing too many. Note
that there are many other bodies that deal with internationalization;
however, none of them appear commonly in IETF standards work.
ISO
The International Organization for Standardization has been involved
with standards for scripts since before the IETF was started. ISO is a
non-governmental group made up of national bodies. ISO has many diverse
standards in the international scripts area; the one that is most used
in the IETF is commonly referred to as "ISO 10646", although its
official name has more qualifications. ISO 10646 describes a CCS that
covers almost all known written characters in use today.
ISO 10646 is controlled by the group known as "ISO/IEC JTC 1/SC 2 WG2",
often called "WG2" for short. ISO standards go through many steps before
being finished, and years often go by between changes to ISO 10646.
Information on WG2, and its work products, can be found at
.
Unicode Consortium
The second important group for international character standards is the
Unicode Consortium. The Unicode Consortium is a trade association of
companies and governments interested in promoting the Unicode Standard
[Unicode3]. The Unicode Standard is a CCS whose repertoire is identical
to ISO 10646. The Unicode Consortium has added features to the base CCS
which make it more useful in protocols, such as defining attributes for
each character.
The Unicode Consortium publishes addenda to the Unicode Standard as
Unicode Technical Reports. There are many types of technical reports at
various stages of maturity. The Unicode Standard and affiliated
technical reports can be found at
encodings and transformations of ISO 10646
Characters in the ISO 10646 CCS can be expressed in many ways. Encoding
forms are direct addressing methods, while transformation formats are
methods for expressing encoding forms as bits on the wire. Two encoding
forms are defined for ISO 10646:
UCS-2 addresses the first 2^15 characters as 16-bit values. This range
is also called the "Basic Multilingual Plane" (BMP), and is also called
"plane 0".
UCS-4 addresses the entire range of 2^31 characters as 32-bit values.
Many transformation formats of the CCS are used in IETF standards. The
two most common are:
UTF-8, defined in [RFC2279], is the preferred encoding for IETF
protocols. Characters in the BMP are encoded as one, two, or three
octets.
UTF-16 (BE & LE), defined in [RFC2781], is used much less often than
UTF-8. Characters in the BMP are always encoded as two octets, and many
characters outside the BMP as four octets.
native CCSs and charsets
Before ISO 10646 was developed, many countries developed their own CCSs
and charsets. Many dozen of these are in common use on the Internet
today. Examples include ISO 8859-5 (Russia) and Shift-JIS (Japan). The
official list of the registered charset tags is maintained by IANA.
ASCII
Probably the most well-known native CCS is ASCII [US-ASCII]. This CCS is
used in numerous IETF protocols that have not yet been
internationalized.
local and regional standards organizations
Just as there are many native CCSs and charsets, there are many local
and regional standards organizations to create and support them. Common
examples of these are ANSI (United States), JIS (Japan), GB (China), and
CEN/ISSS (Europe).
4. Linguistic Issues
This section contains terms and topics that are commonly used in
linguistics and therefore are of concern to people internationalizing
protocols. These topics are standardized outside the IETF.
composition and decomposition
In some CCSs, some characters consist of combinations of other
characters. For example, the letter "a with acute" might be a
combination of the two characters "a" and "combining acute". The rules
for combining two or more characters are called "composition", and the
rules for taking apart a character into other characters is called
"decomposition".
normalization and canonicalization
These two terms are often used interchangeably in internationalization.
Generally, they both mean to convert a string of one or more characters
into another string based on standardized rules. In internationalized
text, these rules are usually based on decomposing combined characters
or composing characters with combining characters. [UTR15] describes the
process and many forms of normalization in detail. Normalization is
important when comparing strings to see if they are the same.
locale and region
Because languages differ from country to country (and even region to
region within a country), the locale of the user of internationalized
text can often be an important factor. Typically, the locale information
for a user includes the language(s) used. Locale issues go beyond
character use, and can include things such as the display format for
currency, dates, and times.
case
In many scripts, particularly those from or based on European alphabets,
there are two forms for letters: uppercase and lowercase. There is
usually (but not always) a one-to-one mapping between the same letter in
the two cases. However, there are many examples of characters which
exist in one case but for which there is no corresponding character in
the other case. Case conversion can even be dependant on locale.
Converting between the two cases is sometimes called "folding".
sorting
Many processes have a need to order strings in a consistent sequence
(sorted). For some CCS/CES combinations, there is an obvious sort order
that can be done without reference to the linguistic meaning of the
characters: the codepoint order is sufficient. For other CCS/CES (such
as the ISO 2022 family) there is no such order that works well.
Codepoint order is usually not how any human educated by a local school
system expects to see strings ordered; if one orders to the expectations
of a human, one has a localized sort.
Sorting to codepoint order will seem inconsistent if the strings are not
normalized before sorting because different representations of the same
character will sort differently. This problem may be smaller with a
localized sort.
glyph
A graphic character or glyph is a character, other than a control
function, that has a visual representation (normally handwritten,
printed, or displayed). It is a recognizable abstract graphic symbol
which is independent of any specific design. There are many types of
glyphs, including alphabetic characters, digits, punctuation,
diacritics, and symbols.
font
A collection of glyph images having the same basic design.
code table or code page
A tabular representation of a coded character set, also showing the
coded representations.
code space
The numeric domain occupied by all bit combinations used for the coding
of a coded character set.
4.1 Types of characters
alphabetic
Characters from the spoken part of phonetic or syllabic scripts.
Examples include Latin letters a through z, Arabic letters, and Katakana
characters from Japanese.
ideographic
Characters that, by themselves, represent words or concepts (as compared
to phonetic sounds). Examples include Han characters used in Chinese,
Japanese, and Korean.
punctuation
Non-alphabetic characters that are used to delimit sounds, sentences,
and phrases. Examples include the period, comma, and hyphen.
symbols
Characters that represent pictures or icons (as compared to ones that
represent letters or punctuation). Examples include characters for
arrows, faces, and geometric shapes.
spacing characters
Characters that represent horizontal or vertical spaces in written text.
Examples include the space character, the tab character, and the
paragraph mark.
diacritics
Characters that combine with alphabetic characters, usually to change
the spoken pronunciation of the base alphabetic character. Examples
include the combining acute accent, combining tilde, and combining ring
above.
combining character
Characters that visually change the characters that precede them.
Examples include combining diacritics, many Arabic alphabetic letters,
and many letters from the Indic scripts.
control character
Non-displaying characters that cause changes in the systems in which
they are entered. Examples include the null character, the delete
character, and the bell character.
formatting character
Non-displaying character that has an effect on the surrounding
characters. Examples include characters for specifying the direction of
text, letter-spacing characters, and characters for specifying joining.
5. User interface for internationalized text
Although the IETF does not standardize user interfaces, many protocols
make assumptions about how a user will enter or see text that is used in
the protocol. Many protocols make inherent assumptions such as that text
will be typed on a standard (that is, US-centric) keyboard, or that text
will be displayed on a character-based monitor. Internationalization
challenges assumptions like these, and it is therefore useful to
consider how users typically interact with internationalized text.
input methods
Text can be entered into a computer in many ways. Keyboards are by far
the most common method used, but many characters cannot be entered on
typical computer keyboards. Many operating systems come with system
software that lets users input characters outside the range of what is
allowed by keyboards.
For example, there are dozens of different input method for Han
characters in Chinese, Japanese, and Korean. Some start with phonetic
input through the keyboard, while others use the number of strokes in
the character. Input methods are also needed for scripts that have many
diacritics, such as European characters that have two or three
diacritics on a single alphabetic character.
display methods
Many scripts can be directly displayed with fonts, where each character
from an input stream can simply be copied from a font system and put on
the screen. Other scripts need rules that are based on the input stream
in order to display text. Some examples of these display rules include:
- Scripts such as Arabic (and many others), where the form of the letter
changes depending on whether the letter is standing alone, at the
beginning of a word, in the middle of a word, or at the end of a word
- Scripts such as the Indic scripts, where consonants may change their
form if they are adjacent to certain other consonants
- Arabic and Hebrew script, where the order of the characters displayed
can be changed with right-to-left and left-to-right ordering marks
rendering characters
Combining characters modify the display of the character (or, in some
cases, characters) that precede them. When rendering such text, the
display engine must either find the character in the font that contains
the base character and all of the combining characters, or it must
render the combination itself. Such rendering can be straight-forward,
but it is sometimes complicated when the combining marks interact with
each other, such as when there are two combining marks that would appear
above one character.
bidirectional scripts
Most of the world's written languages are displayed left-to-right.
However, many widely-used written languages such as Hebrew and ones
based on the Arabic script are displayed right-to-left. Right-to-left
text often confounds protocol writers because they have to keep thinking
in terms of the order of characters in a string in memory, and that
order might be different than what they see on the screen.
Bidirectional text can cause even more confusion because there are
formatting characters in ISO 10646 which cause the order of display of
text to change. These formatting characters are needed so that one can
properly display right-to-left characters with left-to-right characters
at the same time without forcing the one of the strings to be stored in
reverse order. [Unicode3] has a long and incredibly detailed discussion
of bidirectional text.
undisplayable characters
Some characters have no displayable form. For instance, the zero-width
space (U+200B) cannot be displayed because it takes up no horizontal
space. Formatting characters such as those for setting the direction of
text are also undisplayable.
6. Text in current IETF protocols
Many IETF protocols started off being fully internationalized, while
others have been internationalized as they were revised. In this
process, IETF members have seen patterns in the way that many protocols
use text. This section describes some specific protocol interactions
with text.
protocol elements
Almost every protocol has named elements, such as "source port" in TCP.
In some protocols, the names of the elements
(or text aliases for the names) are transmitted within the protocol.
For example, in SMTP, the names of the verbs are part of the
command stream. The names of protocol elements are not normally seen
by end users.
on-the-wire encoding
Characters exist as codepoints in a charset. Before being transmitted
in a protocol, they must first be encoded. Similarly, when characters
are received in a transmission, they have been encoded, and a protocol
that needs to process the individual characters needs to decode them
before processing. The encoding and decoding used before and after
transmission is often called the "on-the-wire" (or sometimes just "wire")
format.
name spaces
Many items in Internet protocols use names to identify content.
The field of names for particular item is called its name space.
The names in a name space may be controlled centrally (such as by IANA)
or may have distributed control, such as the names in the DNS.
parsed text
In some protocols, free text in text fields might be parsed. For
example, many mail user agents will parse the words in the text of
the Subject: field to attempt to thread based on the "Re:" prefix.
charset tagging
Protocols that allow more than one charset to be used on text that
is meant for human consumption usually require
that the text be tagged with the appropriate charset. Without this
tagging, a program looking at the text cannot definitively discern the
charset of the text.
language tagging
Some protocols allow text that is meant for machine processing
to be tagged with the language used in the text. Such tagging is important
for machine-processing of the text, such as by systems that "display"
the text by speaking it.
MIME
MIME (Multipurpose Internet Mail Extensions) is a message format that
allows for textual message bodies and headers in character sets other
than US-ASCII in formats that require ASCII (most notably, RFC 822).
MIME is described in RFCs 2045 through 2049.
Base64
Base64 is a transfer encoding syntax that allows binary data to be
represented by the ASCII characters A through Z, a through z, 0
through 9, +, /, and =. It is described in RFC 2045.
quoted printable
The original design of the quoted printable transfer encoding syntax was
to allow strings that had some non-ASCII printable characters mixed in
with mostly ASCII printable characters to be human readable. It is
described in RFC 2047. It is generally considered to be somewhat of a
failure at being readable.
ASN.1 text formats
The ASN.1 data description language has many formats for text items. The
formats allow for different repertoires and different encodings. Some of
the formats that appear in IETF standards based on ASN.1 include
IA5String (all ASCII characters), PrintableString (most ASCII
characters, but missing many punctuation characters), BMPString
(characters from ISO 10646 plane 0 in UTF-16BE format), UTF8String (just
as the name implies), and TeletexString (also called T61String; the
repertoire changes over time).
ASCII-compatible encoding
Starting in 2000, many ASCII-compatible encoding schemes (which are
actually transfer encoding syntaxes) have been proposed as possible
solutions for internationalizing host names. Their goal is to be able to
encode any string of ISO 10646 characters as legal DNS host names (as
described in STD 13). At the time of this writing, non ACE has become an
IETF standard.
7. Other Common Terms In Internationalization
This is a hodge-podge of other terms that have appeared in
internationalization discussions in the IETF. It is likely that
additional terms will be added as this document matures.
Latin
"Latin characters" is a not-precise term for characters derived from
Greek script and used throughout the world. The base Latin characters
make up the ASCII repertoire and have been augmented by many single and
multiple diacritics.
romanization
Because of the widespread use of Latin characters, people have tried to
represent many languages that are not based on a Latin repertoire in
Latin. This process is called "romanization". For example, there are two
popular romanizations of Chinese: Wade-Giles and Pinyin, the latter of
which is by far more common today. Most romanization systems are inexact
and do not give perfect round trip mappings between the native script
and the Latin characters.
CJK and Han
The ideographic characters used in Chinese, Japanese, Korean, and
traditional Vietnamese scripts are often called "CJK" characters after
the initial letters of the script names in English. They are also called
"Han" characters, after the romanized translation of the term in Chinese
that is often used for these characters. Note that CJK and Han
characters do not include the phonetic characters of the Japanese or
Korean alphabets.
translation
The process of converting one language to another, or one script to
another, is called translation. Most language translation systems are
inexact and do not give one-to-one round trip mappings between the
languages. Most script translations are exact, and many have perfect
round-trip mappings.
regular expressions
Pattern matching for text involves being able to represent one or more
code points in an abstract notation, such as searching for all capital
Latin letters or all punctuation. The most common mechanism in IETF
protocols for naming such patterns is the use of regular expressions.
private use characters
Many CCSs have code points that are defined as existing characters that
have not defined semantics. The use of these "private use" characters is
defined by the parties who transmit and receive them, and is thus not
appropriate for standardization.
8. Security Considerations
Security is not discussed in this document.
9. References
[IDN-REQ] "Requirements of Internationalized Domain Names", work in
progress (draft-ietf-idn-requirements), Z. Wenzel and J. Seng.
[ISO10646] ISO/IEC 10646-1:2000. International Standard -- Information
technology -- Universal Multiple-Octet Coded Character Set (UCS) -- Part
1: Architecture and Basic Multilingual Plane.
[RFC1766] "Tags for the Identification of Languages", RFC 1766, H.
Alvestrand.
[RFC1766bis] "Tags for the Identification of Languages", work in
progress (draft-alvestrand-lang-tag-v2), H. Alvestrand.
[RFC2277] "IETF Policy on Character Sets and Languages", RFC 2277, H.
Alvestrand.
[RFC2279] "UTF-8, a transformation format of ISO 10646", RFC 2279, F.
Yergeau.
[RFC2781] "UTF-16, an encoding of ISO 10646", RFC 2781,
P. Hoffman and F. Yergeau.
[Unicode3] The Unicode Consortium, "The Unicode Standard -- Version
3.0", ISBN 0-201-61633-5. Described at
.
[US-ASCII] Coded Character Set -- 7-bit American Standard Code for
Information Interchange, ANSI X3.4-1986.
[UTR15] "Unicode Normalization Forms", Unicode Technical Report
#15, M. Davis & M. Duerst.
10. Additional Interesting Reading
ALA-LC Romanization Tables, Randall Barry (ed.), ISBN 0844409405
Blackwell Encyclopedia of Writing Systems, Florian Coulmas, ISBN
063121481X
Unicode Standard version 3.0, Unicode Consortium, ISBN 0201616335
Writing Systems of the World, Akira Nakanishi, ISBN 0804816549
A. Acknowledgements
The definitions in this document come from many sources, including
a wide variety of IETF documents.
James Seng contributed to the initial outline of this document.
Others who contributed to the development include:
Jacob Palme
Susan Harris
Harald Alvestrand
Johan van Wingen
Yuri Demchenko
B. Author Contact Information
Paul Hoffman
Internet Mail Consortium and VPN Consortium
127 Segre Place
Santa Cruz, CA 95060 USA
paul.hoffman@imc.org and paul.hoffman@vpnc.org