< draft-crispin-collation-unicasemap-06.txt		draft-crispin-collation-unicasemap-07.txt >
	Network Working Group M. Crispin		Network Working Group M. Crispin
	Internet-Draft University of Washington		Internet-Draft University of Washington
	Intended status: Proposed Standard August 8, 2007		Intended status: Proposed Standard August 30, 2007
	Expires: February 8, 2008		Expires: February 30, 2008
	Document: internet-drafts/draft-crispin-collation-unicasemap-06.txt		Document: internet-drafts/draft-crispin-collation-unicasemap-07.txt
	i;unicode-casemap - Simple Unicode Collation Algorithm		i;unicode-casemap - Simple Unicode Collation Algorithm
	Status of this Memo		Status of this Memo
	By submitting this Internet-Draft, each author represents that		By submitting this Internet-Draft, each author represents that
	any applicable patent or other IPR claims of which he or she is		any applicable patent or other IPR claims of which he or she is
	aware have been or will be disclosed, and any of which he or she		aware have been or will be disclosed, and any of which he or she
	becomes aware will be disclosed, in accordance with Section 6 of		becomes aware will be disclosed, in accordance with Section 6 of
	BCP 79.		BCP 79.
	Internet-Drafts are working documents of the Internet Engineering		Internet-Drafts are working documents of the Internet Engineering
	Task Force (IETF), its areas, and its working groups. Note that		Task Force (IETF), its areas, and its working groups. Note that
	other groups may also distribute working documents as		other groups may also distribute working documents as
	Internet-Drafts.		Internet-Drafts.
	Internet-Drafts are draft documents valid for a maximum of six months		Internet-Drafts are draft documents valid for a maximum of six months
	and may be updated, replaced, or obsoleted by other documents at any		and may be updated, replaced, or obsoleted by other documents at any
	time. It is inappropriate to use Internet-Drafts as reference		time. It is inappropriate to use Internet-Drafts as reference
	material or to cite them other than as "work in progress."		material or to cite them other than as "work in progress."
	The list of current Internet-Drafts can be accessed at		The list of current Internet-Drafts can be accessed at
	http://www.ietf.org/ietf/1id-abstracts.txt		http://www.ietf.org/ietf/1id-abstracts.txt
	The list of Internet-Draft Shadow Directories can be accessed at		The list of Internet-Draft Shadow Directories can be accessed at
	http://www.ietf.org/shadow.html.		http://www.ietf.org/shadow.html.
	A revised version of this document will be submitted to the RFC		A revised version of this document will be submitted to the RFC
	editor as an Informational Document for the Internet Community.		editor as an Informational Document for the Internet Community.
	A revised version of this draft document will be submitted to the RFC		A revised version of this draft document will be submitted to the RFC
	editor as a Proposed Standard for the Internet Community. Discussion		editor as a Proposed Standard for the Internet Community. Discussion
	and suggestions for improvement are requested, and should be sent to		and suggestions for improvement are requested, and should be sent to
	ietf-imapext@IMC.ORG.		ietf-imapext@IMC.ORG.
	Distribution of this memo is unlimited.		Distribution of this memo is unlimited.
	Abstract		Abstract
	This document describes "i;unicode-casemap", a simple		This document describes "i;unicode-casemap", a simple
	case-insensitive collation for Unicode strings. It provides		case-insensitive collation for Unicode strings. It provides
	equality, substring and ordering operations.		equality, substring and ordering operations.
	Introduction		Introduction
	The "i;ascii-casemap" collation described in [COMPARATOR] is quite		The "i;ascii-casemap" collation described in [COMPARATOR] is quite
	simple to implement and provides case-independent comparisons for the		simple to implement and provides case-independent comparisons for the
	26 Latin alphabetics. It is specified as the default and/or baseline		26 Latin alphabetics. It is specified as the default and/or baseline
	comparator in some application protocols, e.g., [IMAP-SORT].		comparator in some application protocols, e.g., [IMAP-SORT].
	However, the "i;ascii-casemap" collation does not produce		However, the "i;ascii-casemap" collation does not produce
	satisfactory results with non-ASCII characters. It is possible, with		satisfactory results with non-ASCII characters. It is possible, with
	a modest extension, to provide a more sophisticated collation with		a modest extension, to provide a more sophisticated collation with
	greater multilingual applicability than "i;ascii-casemap". This		greater multilingual applicability than "i;ascii-casemap". This
	extension provides case-independent comparisons for a much greater		extension provides case-independent comparisons for a much greater
	number of characters. It also collates characters with diacriticals		number of characters. It also collates characters with diacriticals
	with the non-diacritical character forms.		with the non-diacritical character forms.
	This collation, "i;unicode-casemap", is intended to be an alternative		This collation, "i;unicode-casemap", is intended to be an alternative
	to, and preferred over, "i;ascii-casemap". It does not replace the		to, and preferred over, "i;ascii-casemap". It does not replace the
	"i;basic" collation described in [BASIC].		"i;basic" collation described in [BASIC].
	1. Unicode Casemap Collation Description		1. Unicode Casemap Collation Description
	The "i;unicode-casemap" collation is a simple collation which is		The "i;unicode-casemap" collation is a simple collation which is
	case-insensitive in its treatment of characters. It provides		case-insensitive in its treatment of characters. It provides
	equality, substring and ordering operations. The validity test		equality, substring and ordering operations. The validity test
	operation returns "valid" for any input.		operation returns "valid" for any input.
	This collation allows strings in arbitrary (and mixed) character		This collation allows strings in arbitrary (and mixed) character
	sets, as long as the character set for each string is identified and		sets, as long as the character set for each string is identified and
	it is possible to convert the string to Unicode. Strings which have		it is possible to convert the string to Unicode. Strings which have
	an unidentified character set and/or can not be converted to Unicode		an unidentified character set and/or can not be converted to Unicode
	are not rejected, but are treated as binary.		are not rejected, but are treated as binary.
	Each input string is prepared by converting it to a "titlecased		Each input string is prepared by converting it to a "titlecased
	canonicalized UTF-8" string according to the following steps, using		canonicalized UTF-8" string according to the following steps, using
	using UnicodeData.txt ([UNICODE-DATA]):		UnicodeData.txt ([UNICODE-DATA]):
	(1) A Unicode codepoint is obtained from the input string.		(1) A Unicode codepoint is obtained from the input string.
	(a) If the input string is in a known charset that can be		(a) If the input string is in a known charset that can be
	converted to Unicode, a sequence in the string's charset		converted to Unicode, a sequence in the string's charset
	is read and checked for validity according to the rules of		is read and checked for validity according to the rules of
	that charset. If the sequence is valid, it is converted		that charset. If the sequence is valid, it is converted
	to a Unicode codepoint. Note that for input strings in		to a Unicode codepoint. Note that for input strings in
	UTF-8, the UTF-8 sequence must be valid according to the		UTF-8, the UTF-8 sequence must be valid according to the
	rules of [UTF-8]; e.g., overlong UTF-8 sequences are		rules of [UTF-8]; e.g., overlong UTF-8 sequences are
	invalid.		invalid.
	(b) If the input string is in an unknown charset, or an		(b) If the input string is in an unknown charset, or an
	invalid sequence occurs in step (1)(a), conversion ceases.		invalid sequence occurs in step (1)(a), conversion ceases.
	No further preparation is performed, and any partial		No further preparation is performed, and any partial
	preparation results are discarded. The original string is		preparation results are discarded. The original string is
	used unchanged with the i;octet comparator.		used unchanged with the i;octet comparator.
	(2) The following steps, using UnicodeData.txt ([UNICODE-DATA]),		(2) The following steps, using UnicodeData.txt ([UNICODE-DATA]),
	are performed on the resulting codepoint from step (1)(a).		are performed on the resulting codepoint from step (1)(a).
	(a) If the codepoint has a titlecase property in		(a) If the codepoint has a titlecase property in
	UnicodeData.txt (this is normally the same as the		UnicodeData.txt (this is normally the same as the
	uppercase property), the codepoint is converted to the		uppercase property), the codepoint is converted to the
	codepoints in the titlecase property.		codepoints in the titlecase property.
	(b) If the resulting codepoint from (2)(a) has a decomposition		(b) If the resulting codepoint from (2)(a) has a decomposition
	property of any type in UnicodeData.txt, the codepoint is		property of any type in UnicodeData.txt, the codepoint is
	converted to the codepoints in the decomposition property.		converted to the codepoints in the decomposition property.
	This step is recursively applied to each of the resulting		This step is recursively applied to each of the resulting
	codepoints until no more decomposition is possible		codepoints until no more decomposition is possible
	(effectively Normalization Form KD).		(effectively Normalization Form KD).
	Example: codepoint U+01C4 (LATIN CAPITAL LETTER DZ WITH CARON)		Example: codepoint U+01C4 (LATIN CAPITAL LETTER DZ WITH CARON)
	has a titlecase property of U+01C5 (LATIN CAPITAL LETTER D		has a titlecase property of U+01C5 (LATIN CAPITAL LETTER D
	WITH SMALL LETTER Z WITH CARON). Codepoint U+01C5 has a		WITH SMALL LETTER Z WITH CARON). Codepoint U+01C5 has a
	decomposition property of U+0044 (LATIN CAPITAL LETTER D)		decomposition property of U+0044 (LATIN CAPITAL LETTER D)
	U+017E (LATIN SMALL LETTER Z WITH CARON). U+017E has a		U+017E (LATIN SMALL LETTER Z WITH CARON). U+017E has a
	decomposition property of U+007A (LATIN SMALL LETTER Z) U+030c		decomposition property of U+007A (LATIN SMALL LETTER Z) U+030c
	(COMBINING CARON). Neither U+0044, U+007A, nor U+030C have		(COMBINING CARON). Neither U+0044, U+007A, nor U+030C have
	any decomposition properties. Therefore, U+01C4 is converted		any decomposition properties. Therefore, U+01C4 is converted
	to U+0044 U+007A U+030C by this step.		to U+0044 U+007A U+030C by this step.
	(3) The resulting codepoint(s) from step (2) is/are appended, in		(3) The resulting codepoint(s) from step (2) is/are appended, in
	UTF-8 format, to the "titledcased canonicalized UTF-8" string.		UTF-8 format, to the "titlecased canonicalized UTF-8" string.
	(4) Repeat from step (1) until there is no more data in the input		(4) Repeat from step (1) until there is no more data in the input
	string.		string.
	Following the above preparation process on each string, the equality,		Following the above preparation process on each string, the equality,
	ordering and substring operations are as for i;octet.		ordering and substring operations are as for i;octet.
	It is permitted to use an alternative implementation of the above		It is permitted to use an alternative implementation of the above
	preparation process if it produces the same results. For example, it		preparation process if it produces the same results. For example, it
	may be more convenient for an implementation to convert all input		may be more convenient for an implementation to convert all input
	strings to a sequence of UTF-16 or UTF-32 values prior to performing		strings to a sequence of UTF-16 or UTF-32 values prior to performing
	any of the step (2) actions. Similarly, if all input strings are (or		any of the step (2) actions. Similarly, if all input strings are (or
	are convertable to) Unicode, it may be possible to use UTF-32 as an		are convertible to) Unicode, it may be possible to use UTF-32 as an
	alternative to UTF-8 in step (3).		alternative to UTF-8 in step (3).
	Note: UTF-16 is unsuitable as an alternative to UTF-8 in step (3),		Note: UTF-16 is unsuitable as an alternative to UTF-8 in step (3),
	because UTF-16 surrogates will cause i;octet to collate codepoints		because UTF-16 surrogates will cause i;octet to collate codepoints
	U+E0000 through U+FFFF after non-BMP codepoints.		U+E0000 through U+FFFF after non-BMP codepoints.
	This collation is not locale sensitive. Consequently, care should be		This collation is not locale sensitive. Consequently, care should be
	taken when using OS-supplied functions to implement this collation.		taken when using OS-supplied functions to implement this collation.
	Functions such as strcasecmp and toupper are sometimes locale		Functions such as strcasecmp and toupper are sometimes locale
	sensitive and may inconsistently casemap letters.		sensitive and may inconsistently casemap letters.
	The i;unicode-casemap collation is well suited to use with many		The i;unicode-casemap collation is well suited to use with many
	Internet protocols and computer languages. Use with natural language		Internet protocols and computer languages. Use with natural language
	is often inappropriate; even though the collation apparently supports		is often inappropriate; even though the collation apparently supports
	languages such as Swahili and English, in real-world use it tends to		languages such as Swahili and English, in real-world use it tends to
	mis-sort a number of types of string:		mis-sort a number of types of string:
	o people and place names containing scripts that are not collated		o people and place names containing scripts that are not collated
	according to "alphabetical order".		according to "alphabetical order".
	o words with characters that have diacriticals. However,		o words with characters that have diacriticals. However,
	i;unicode-casemap generally does a better job than i;ascii-casemap		i;unicode-casemap generally does a better job than i;ascii-casemap
	for most (but not all) languages. For example, German umlaut		for most (but not all) languages. For example, German umlaut
	letters will sort correctly, but some Scandinavian letters will		letters will sort correctly, but some Scandinavian letters will
	not.		not.
	o names such as "Lloyd" (which in Welsh sorts after "Lyon", unlike		o names such as "Lloyd" (which in Welsh sorts after "Lyon", unlike
	in English),		in English),
	o strings containing other non-letter symbols; e.g., euro and pound		o strings containing other non-letter symbols; e.g., euro and pound
	sterling symbols, quotation marks other than '"', dashes/hyphens,		sterling symbols, quotation marks other than '"', dashes/hyphens,
	etc.		etc.
	2. Unicode Casemap Collation Registration		2. Unicode Casemap Collation Registration
	<?xml version='1.0'?>		<?xml version='1.0'?>
	<!DOCTYPE collation SYSTEM 'collationreg.dtd'>		<!DOCTYPE collation SYSTEM 'collationreg.dtd'>
	<collation rfc="XXXX" scope="global" intendedUse="common">		<collation rfc="XXXX" scope="global" intendedUse="common">
	<identifier>i;unicode-casemap</identifier>		<identifier>i;unicode-casemap</identifier>
	<title>Unicode Casemap</title>		<title>Unicode Casemap</title>
	<operations>equality order substring</operations>		<operations>equality order substring</operations>
	<specification>RFC XXXX</specification>		<specification>RFC XXXX</specification>
	<owner>IETF</owner>		<owner>IETF</owner>
	<submitter>mrc@cac.washington.edu</submitter>		<submitter>mrc@cac.washington.edu</submitter>
	</collation>		</collation>
	3. Security Considerations		3. Security Considerations
	The security considerations for [UTF-8], [STRINGPREP] and		The security considerations for [UTF-8], [STRINGPREP] and
	[UNICODE-SECURITY] apply and are normative to this specification.		[UNICODE-SECURITY] apply and are normative to this specification.
			The results from this comparator will vary depending upon the
			implementation for several reasons. Implementations MUST consider
			whether these possibilities are a problem for their use case:
			1) New characters added in Unicode may have decomposition or
			titlecase properties that will not be known to an implementation
			based upon an older revision of Unicode. This impacts Step (2).
			2) Step (2)(b) defines a subset of Normalization Form KD that does
			not require normalization of out-of-order diacriticals. However,
			an implementation MAY use an NFKD library routine that does such
			normalization. This impacts step (2)(b) and possibly also step
			(1)(a), and is an issue only with ill-formed UTF-8 input.
			3) The set of charsets handled in step (1)(a) is open-ended. UTF-8
			(and, by extension, US-ASCII) are the only mandatory-to-implement
			charsets. This impacts step (1)(a).
			Implementations SHOULD, as far as feasible, support all the
			charsets they are likely to encounter in the input data, in order
			to avoid poor collation caused by the fall through to the (1)(b)
			rule.
			4) Other charsets may have revisions which add new characters that
			are not known to an implementation based upon an older revision.
			This impacts step (1)(a) and possibly also step (1)(b).
			An attacker may create input that is ill-formed or in an unknown
			charset, with the intention of impacting the results of this
			comparator or exploiting other parts of the system which process this
			input in different ways. Note, however, that even well-formed data
			in a known charset can impact the result of this comparator in
			unexpected ways. For example, an attacker can substitute U+0041
			(LATIN CAPITAL LETTER A) with U+0391 (GREEK CAPITAL LETTER ALPHA) or
			U+0410 (CYRILLIC CAPITAL LETTER A) in the intention of a non-match of
			strings which visually appear the same and/or to cause the string to
			appear elsewhere in a sort.
	4. IANA Considerations		4. IANA Considerations
	The i;unicode-casemap collation defined in section 2 should be added		The i;unicode-casemap collation defined in section 2 should be added
	to the registry of collations defined in [COMPARATOR].		to the registry of collations defined in [COMPARATOR].
	5. Normative References		5. Normative References
	The following documents are normative to this document:		The following documents are normative to this document:
	[COMPARATOR] Newman, C., "Internet Appplication Protocol		[COMPARATOR] Newman, C., "Internet Application Protocol
	Collation Registry", RFC 4790, February 2007.		Collation Registry", RFC 4790, February 2007.
	[STRINGPREP] Hoffman, P. and M. Blanchet, "Preparation of		[STRINGPREP] Hoffman, P. and M. Blanchet, "Preparation of
	Internationalized Strings ("stringprep")",		Internationalized Strings ("stringprep")",
	RFC 3454, December 2002.		RFC 3454, December 2002.
	[UTF-8] Yergeau, F., "UTF-8, a transformation format		[UTF-8] Yergeau, F., "UTF-8, a transformation format
	of ISO 10646", STD 63, RFC 3629, November 2003.		of ISO 10646", STD 63, RFC 3629, November 2003.
	[UNICODE-DATA] <http://www.unicode.org/Public/UNIDATA/		[UNICODE-DATA] <http://www.unicode.org/Public/UNIDATA/
	UnicodeData.txt>		UnicodeData.txt>
	Although the UnicodeData.txt file referenced		Although the UnicodeData.txt file referenced
	here is part of the Unicode standard, it is		here is part of the Unicode standard, it is
	subject to change as new characters are added		subject to change as new characters are added
	to Unicode and errors are corrected in Unicode		to Unicode and errors are corrected in Unicode
	revisions. As a result, it may be less stable		revisions. As a result, it may be less stable
	than might otherwise be implied by the		than might otherwise be implied by the
	standards status of this specification.		standards status of this specification.
	[UNICODE-SECURITY] Davis, M. and M. Suignard, "Unicode Security		[UNICODE-SECURITY] Davis, M. and M. Suignard, "Unicode Security
	Considerations", February 2006,		Considerations", February 2006,
	<http://www.unicode.org/reports/tr36/>.		<http://www.unicode.org/reports/tr36/>.
	6. Informative References:		6. Informative References:
	[BASIC] Newman, C., Duerst, M., and Gulbrandsen, A.,		[BASIC] Newman, C., Duerst, M., and Gulbrandsen, A.,
	"i;basic - the Unicode Collation Algorithm",		"i;basic - the Unicode Collation Algorithm",
	draft-gulbrandsen-collation-basic, Work in		draft-gulbrandsen-collation-basic, Work in
	Progress.		Progress.
	[IMAP-SORT] Crispin, M. "Internet Message Access Protocol -		[IMAP-SORT] Crispin, M. "Internet Message Access Protocol -
	SORT and THREAD Extensions",		SORT and THREAD Extensions",
	draft-ietf-imapext-sort, Work in Progress (in		draft-ietf-imapext-sort, Work in Progress (in
	RFC Editor queue).		RFC Editor queue).
	Appendices		Appendices
	Author's Address		Author's Address
	Mark R. Crispin		Mark R. Crispin
	Networks and Distributed Computing		Networks and Distributed Computing
	University of Washington		University of Washington
	4545 15th Avenue NE		4545 15th Avenue NE
	Seattle, WA 98105-4527		Seattle, WA 98105-4527
	Phone: +1 (206) 543-5762		Phone: +1 (206) 543-5762
	EMail: MRC@CAC.Washington.EDU		EMail: MRC@CAC.Washington.EDU
	Full Copyright Statement		Full Copyright Statement
	Copyright (C) The IETF Trust (2007).		Copyright (C) The IETF Trust (2007).
	This document is subject to the rights, licenses and restrictions		This document is subject to the rights, licenses and restrictions
	contained in BCP 78, and except as set forth therein, the authors		contained in BCP 78, and except as set forth therein, the authors
	retain all their rights.		retain all their rights.
	This document and the information contained herein are provided on an		This document and the information contained herein are provided on an
	"AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS		"AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
	OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND		OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND
	THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS		THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS
	OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF		OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF
	THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED		THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
	WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.		WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
	Intellectual Property		Intellectual Property
	The IETF takes no position regarding the validity or scope of any		The IETF takes no position regarding the validity or scope of any
	Intellectual Property Rights or other rights that might be claimed to		Intellectual Property Rights or other rights that might be claimed to
	pertain to the implementation or use of the technology described in		pertain to the implementation or use of the technology described in
	this document or the extent to which any license under such rights		this document or the extent to which any license under such rights
	might or might not be available; nor does it represent that it has		might or might not be available; nor does it represent that it has
	made any independent effort to identify any such rights. Information		made any independent effort to identify any such rights. Information
	on the procedures with respect to rights in RFC documents can be		on the procedures with respect to rights in RFC documents can be
	found in BCP 78 and BCP 79.		found in BCP 78 and BCP 79.
	Copies of IPR disclosures made to the IETF Secretariat and any		Copies of IPR disclosures made to the IETF Secretariat and any
	assurances of licenses to be made available, or the result of an		assurances of licenses to be made available, or the result of an
	attempt made to obtain a general license or permission for the use of		attempt made to obtain a general license or permission for the use of
	such proprietary rights by implementers or users of this		such proprietary rights by implementers or users of this
	specification can be obtained from the IETF on-line IPR repository at		specification can be obtained from the IETF on-line IPR repository at
	http://www.ietf.org/ipr.		http://www.ietf.org/ipr.
	The IETF invites any interested party to bring to its attention any		The IETF invites any interested party to bring to its attention any
	copyrights, patents or patent applications, or other proprietary		copyrights, patents or patent applications, or other proprietary
	rights that may cover technology that may be required to implement		rights that may cover technology that may be required to implement
	this standard. Please address the information to the IETF at ietf-		this standard. Please address the information to the IETF at ietf-
	ipr@ietf.org.		ipr@ietf.org.
	Acknowledgement		Acknowledgement
	Funding for the RFC Editor function is currently provided by the		Funding for the RFC Editor function is currently provided by the
	Internet Society.		Internet Society.
End of changes. 54 change blocks.
	208 lines changed or deleted		246 lines changed or added
This html diff was produced by rfcdiff 1.48. The latest version is available from http://tools.ietf.org/tools/rfcdiff/