Network Working Group L. Daigle Internet-Draft T. Hardie Expires: August 16, 2004 Editor Internet Architecture Board IAB February 16, 2004 Considerations on Increasing Character Repertoires for Protocol Actionable Elements draft-iab-char-rep-01 Status of this Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. This Internet-Draft will expire on August 16, 2004. Copyright Notice Copyright (C) The Internet Society (2004). All Rights Reserved. Abstract This document describes a set of considerations and strategies to use in increasing the character repertoire available in a protocol actionable element or suite of protocol actionable elements. This document is not meant to provide normative instruction to protocol designers, but does hope to provide guidance on common issues arising from this task. Feedback should be sent to the editors or the IAB. Daigle, et. al. Expires August 16, 2004 [Page 1] Internet-Draft draft-iab-char-rep-01 February 2004 Table of Contents 1. Definitions . . . . . . . . . . . . . . . . . . . . . . . . . 3 2. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4 3. Avoidance mechanisms . . . . . . . . . . . . . . . . . . . . . 5 3.1 Choosing a large initial character repertoire . . . . . . . . 5 3.2 Choosing opaque protocol tokens . . . . . . . . . . . . . . . 5 3.3 Expansion mechanisms . . . . . . . . . . . . . . . . . . . . . 6 3.4 Replace . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 3.5 Subsume . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 3.6 Map . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 4. Layering a presentation element on a new protocol element . . 9 5. Selecting a strategy . . . . . . . . . . . . . . . . . . . . . 10 6. Case Studies . . . . . . . . . . . . . . . . . . . . . . . . . 11 6.1 Uniform and Internationalized Resource Identifiers . . . . . . 11 7. Security Considerations . . . . . . . . . . . . . . . . . . . 12 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 13 9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 14 References . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . 15 Full Copyright Statement . . . . . . . . . . . . . . . . . . . 16 Daigle, et. al. Expires August 16, 2004 [Page 2] Internet-Draft draft-iab-char-rep-01 February 2004 1. Definitions Protocol (actionable) element: A protocol actionable element, or protocol element is any portion of a message which affects processing of that message by the protocol in question. In general, protocol elements are bound to specific processing choices by membership in a set of predetermined tokens or by explicit structure. Protocol elements are context dependent in that the processing for a token is specific to a protocol. To IP, for example, a TCP port number is payload; to TCP it is a protocol element. Similarly, to TCP a Content-encoding: header is payload; to HTTP, it is a protocol element. Character repertoire: A character repertoire is the set of all characters in all permitted encodings which may be used in a protocol element. Each element in a character repertoire is a tuple of a code point and an encoding. Thus the glyph "a" would appear three times in a character repertoire that permitted ASCII, iso-8859-1, and iso- 8859-7. Character set: As it says, "a set of characters", but more particularly a set of characters as represented by code points in a particular encoding. Daigle, et. al. Expires August 16, 2004 [Page 3] Internet-Draft draft-iab-char-rep-01 February 2004 2. Introduction After a protocol's initial deployment, changes in the use of the protocol sometimes neccesitate revisiting the character repertoire originally chosen for one or more of the elements which make up the protocol. On rare occasions, this occurs because the protocol designers need to increase the number of tokens available in a fixed- length field and choose to do so by increasing the number of characters which may be used. More commonly, the motive for the increase of a character repertoire is the exposure of a protocol element to a user community. Once this leakage occurs, there is often pressure to expand the permitted character repertoire of the protocol element to match the character repertoire in use in that community. Though increasing a character repertoire may appear to be a relatively simple matter, there are a number of protocol processing functions which may be affected. First among these is matching. Many encodings have very specific matching rules or equivalence tables; increasing a character repertoire to include a new encoding implies that the protocol must specify how matching works in that encoding. Like matching, sorting works in different ways in different encoding schemes, and including a new encoding means specifying sorting algorithms for use with it. Transformation presents some unique issues, as it may be possible for some systems to map only unidirectionally from one encoding to another. Any of these, and more, can present problems to a protocol designer who must post-facto retrofit an increased character repertoire into a deployed protocol. Daigle, et. al. Expires August 16, 2004 [Page 4] Internet-Draft draft-iab-char-rep-01 February 2004 3. Avoidance mechanisms To avoid the need to increase character repertoires at some later date, protocol designers can either start with a character repertoire which is large enough to encompass that in use in the target user community or use protocol elements that are sufficiently opaque to a human user that their leakage is unlikely to present later pressure. Both strategies, unfortunately, have been notororiously difficult to get right. 3.1 Choosing a large initial character repertoire In this avoidance strategy, the protocol designers presume that their protocol elements will leak in the future and provide a character repertoire which is sufficiently rich to match the user community's needs. Increasing use of a protocol, however, often changes the target user community beyond the intial designers projections. A character repetoire which looks large to one user community may be completely wrong or very limited to another. When protocol designers attempt to avoid the issue by using a character repertoire with a very large number of code points in a very large number of encodings, they incurr real costs in parser complexity, processing overhead, and bloat. They also risk that misconfiguration of these complex parsers will result in incorrect protocol processing. 3.2 Choosing opaque protocol tokens In the second case, designers who choose to use tokens or structure which are not human-readable can resist later pressure to increase the character repertoire available. As those who have used encodings like ASN.1 can attest, there is, however, an increased development cost, as those working with the protocol must develop an understanding of the use of the tokens or structure without the aid of readability. This avenue may also be blocked or narrowed to protocol designers who will need to pass the new elements among different protocols; in those cases, the new protocol is either constrained by the previous choices or must provide a normative mapping to them. When designers use tokens or structures which are not human readable, it is common to create a presentation format or layer which is mapped to the tokens or structures. One of the advantage to this approach is that new mappings can be defined as new user communities express the need for them. It is important, however, that these are always retained as mappings to the protocol elements, and are not treated as protocol elements themselves. Daigle, et. al. Expires August 16, 2004 [Page 5] Internet-Draft draft-iab-char-rep-01 February 2004 3.3 Expansion mechanisms For designers who must increase the character repertoire for a particular protocol element, there are three basic strategies available: they may replace the existing protocol element with a new one; they may subsume the character repertoire of the existing protocol element in a new one; they may map the new character repertoire into the existing repertoire. For each of the following strategies, consider the following example: a protocol element called "POSTAL" used to name the U.S. zip code in which the network element is placed cannot handle postal codes containing characters outside (0,1,2,3,4,5,6,7,8,9) encoded in a subset of US-ASCII. We will refer to this character set as (NUM- ASCII). The original character repertoire for this protocol element has NUM-ASCII as its single member character set. 3.4 Replace Replacing an existing protocol element with an entirely new protocol element with a different character repertoire is by far the cleanest solution from a design perspective. A new protocol element may have its own matching and sorting rules, without regard to any previous deployment. This means that the new element will have as little baggage as is possible when updating parsers and setting forth how it fits into the protocol's semantics. Unfortunately, this method presents a raft of deployment problems. Since existing protocol implementations will know nothing about it, they cannot be interoperable with any entirely new protocol element. At best, they can ignore it gracefully; at worst, they will fail. A protocol designer can react to this by changing the revision number on a protocol, by using some form of feature negotiation, or by using heuristics (including failure!) to determine whether or not a new protocol element may be used. All of these are difficult to get right, especially in hop-by-hop protocols, in which it may not be possible to determine whether all hops support specific features or versions. A protocol designer tackling this problem for the protocol element naming the postal code in which a network element is placed might replace "POSTAL" with "NEW_POSTAL" and create a new character repertoire for "NEW_POSTAL" which contained the single entry (ISO- 8859-1). [This is merely an example; the choice of which character set or sets to use would be made in this instance by reference to the relevant international postal standards.] Obviously, any system which did not understand "NEW_POSTAL" would need to be upgraded to handle the new character set. Depending on the transition mechanism, Daigle, et. al. Expires August 16, 2004 [Page 6] Internet-Draft draft-iab-char-rep-01 February 2004 systems communicating postal codes which were numeric-only might well include both "POSTAL" and "NEW_POSTAL" protocol elements. 3.5 Subsume Rather than completely replacing an existing protocol element, another strategy is to create a protocol element which subsumes the character repertoire of the existing protocol element. When this option is chosen, the new protocol element retains all the character sets and the related matching and sorting rules which were originally present. These become a strict subset of the new character repertoire. This strategy limits the functionality of the new protocol element both by forcing it to include specific character sets and by requiring that the semantics of the new protocol element exactly match the existing protocol element. This strategy also retains many of the deployment problems of the replacement strategy, though it offers some opportunities to mitigate the issues. Like the replacement strategy, there may need to be negotiation mechanisms capable of handling both protocol elements, though new implementations can sometimes treat the old protocol element as a degenerate case of new protocol element. If our "POSTAL" protocol design team took this strategy, they might replace the (NUM-ASCII) character repertoire of "POSTAL" with a new protocol element "BIG_POSTAL" for which the character repertoire is (NUM-ASCII, US-ASCII). Because NUM-ASCII is a strict subset of US- ASCII, the protocol can treat all "POSTAL" protocol elements as if they were "BIG_POSTAL" protocol elements. Note that this is the simplest possible example of this particular strategy, as there is no need to mark which character set from the character repertoire is in use. More complex examples may require much more complex processing to achieve the same results. 3.6 Map In some instances it may be possible and desirable to map an expanded character repertoire onto the existing code points specified by a protocol. In this case, the code points are themselves retained but the character encoding portion of the tuple is changed to create an expanded character repertoire. This strategy can only work when some marker is used to indicate which character encoding applies to a specific instance of the protocol. This marker must be something which is non-operative in the original protocol processing, or the strategy will incur the negotiation costs mentioned above. This strategy will tend to increase the size of protocol elements unless the original code points were radically under-used. It also carries Daigle, et. al. Expires August 16, 2004 [Page 7] Internet-Draft draft-iab-char-rep-01 February 2004 the near-certainty that there will be occasions in which protocol elements encoded with the new character encoding are mis-identified as being encoded with the original character encoding. This strategy has somewhat unique deployment consequences, in that it is both easier to get initial deployment and harder to get complete penetration. Because the same code points are used throughout, there is no requirement that all systems upgrade for the increased character repertoire to be available to a subset of users. There is also, however, almost no incentive for upgrade of systems which do not themselves require the increased repertoire. This is particularly true in hop-by-hop and commonly proxied protocols, because the on-path intermediate systems will pass the elements of the expanded repertoire by virtue of their being legitimate code points in the original repertoire; they do not need to upgrade and they probably never will. For our protocol design team to tackle "POSTAL" using this strategy they must develop or discover an encoding which allows them to represent all the needed characters using just (NUM-ASCII). If, for example, the character repertoire needed to add a character set which included (A-Z), but no others, the team could use US-ASCII's three digit decimal encoding for each included character. A postal code like "KLHSW1" would then be encoded as "075076071083087049". Provided that the original POSTAL protocol element had a field length sufficient to handle the new encoding, it could carry the new values without any difficulty. The difficulty would be determining whether the new encoding or the old should be assumed; in this limited case, length alone could be made a marker by padding any short alphabetic postal codes with the ASCII null character,"OOO", until they reached a length sufficient to trigger treatment as non-ZIP code postal codes. In other cases more complex triggers would be required. Daigle, et. al. Expires August 16, 2004 [Page 8] Internet-Draft draft-iab-char-rep-01 February 2004 4. Layering a presentation element on a new protocol element It is noted above that designers using non-human readable tokens may provide a mapping to a presentation element which can be used by humans working with the protocol. In employing any of the strategies above, it is useful for protocol designers to consider introducing a presentation element at the same time. This is almost a required part of the mapping strategy, as using an encoding based on the original set of code points does not help the user community unless it can also be mapped to an encoding in common use for presentation. It may be used with any of them, though, and given the potential for the introduction of new character encodings, it must be considered carefully as a method of ensuring that the same problem does not face the protocol in a few years time. Daigle, et. al. Expires August 16, 2004 [Page 9] Internet-Draft draft-iab-char-rep-01 February 2004 5. Selecting a strategy The first step in selecting a strategy is identifying the protocol processing choices which depend on the protocol element. If a protocol element is passed among different protocols, this set of choices must be identified for each of the protocols which depend on the element. After those have been identified, the available methods for passing the protocol elements from one protocol to another must be considered. If at all possible, a single strategy should be selected for use with a specific protocol element, even when that protocol element will be passed among different protocols. Since protocol processing is context-specific, it is technically possible to use different methods in different contexts, but this increase in complexity rarely has a corresponding gain. Whether the protocol element will be used in one protocol or several,the core question to consider is how best to maintain interoperability while increasing the character repertoire. For example, if creating a new protocol element as a fully fledged replacement, are there available mechanisms to handle the negotiation and/or versioning? Alternatively, are there methods which would allow both protocol elements to coexist? The second question to consider is the cost of implementation. If, for example, a choice is made to introduce a protocol element which subsumes the original character repertoire in a larger character repertoire, how expensive will the increase in parsing complexity be? The third question to consider is likely deployment patterns. For a client/server protocol, will it be feasible to update both client and server? For a hop-by-hop protocol, will there be any pressure for interemdiate servers to upgrade? A related question is whether this change will be tied to other changes which will drive adoption, or whether this change will be unrelated to other updates to the protocol. Daigle, et. al. Expires August 16, 2004 [Page 10] Internet-Draft draft-iab-char-rep-01 February 2004 6. Case Studies 6.1 Uniform and Internationalized Resource Identifiers Uniform Resource Names ([1]) make use of the 7-bit US-ASCII character repertoire. The syntax of the URI permits other encodings to be mapped into that repertoire, by defining a hex-encoding framework. Increasingly, new URI schemes are using UTF-8 to for characters beyond US-ASCII. In recognition of this, and to provide a means to handle such identifiers in a more straightforward manner, the "Internationalized Resource Identifier" (IRI) has been introduced. From [2]: "This document defines a new protocol element, the Internationalized Resource Identifier (IRI), as a complement to the URI [RFCYYYY]. An IRI is a sequence of characters from the Universal Character Set [ISO10646]. A mapping from IRIs to URIs is defined, which means that IRIs can be used instead of URIs where appropriate to identify resources." The IRI specification applies the "replace", "map" and "subsume" strategies for expansion outlined above. As noted in the quoted text from the IRI document, IRIs are defined as a new protocol element ("replace"). Therefore, any protocol or message format defined in the future may use an IRI protocol element and not a URI protocol element. However, as URIs are ubiquitous and IRIs would face steep deployment challenges without the possibility of relating to URIs. Therefore, [2] defines a mapping strategy to ensure IRIs can be mapped onto URIs and vice versa. The IRI document also goes on to note that there are specifications already designated to handle IRIs -- "anyURI" in XML Schema. This is an example of subsumption. While the IRI document is clear that conversions between IRI and URI formats must be made when transitioning from systems that understand IRIs to ones that do not, it is unclear how message parsers that detect and interpret "http://" as a URI will recognize IRIs as distinct from (malformed) URIs. Daigle, et. al. Expires August 16, 2004 [Page 11] Internet-Draft draft-iab-char-rep-01 February 2004 7. Security Considerations Any protocol processing which depends on a specific set of tokens or structure is at risk when the matching and sorting rules for the set is indeterminate. In some cases, this can result in a denial of service, as legitimate tokens are not recognized; in other cases, inappropriate access may be granted by matching incorrectly. Daigle, et. al. Expires August 16, 2004 [Page 12] Internet-Draft draft-iab-char-rep-01 February 2004 8. IANA Considerations There are no IANA considerations defined in this memo. Daigle, et. al. Expires August 16, 2004 [Page 13] Internet-Draft draft-iab-char-rep-01 February 2004 9. Acknowledgements The authors would like to thank Martin Duerst for his attention and expertise. Daigle, et. al. Expires August 16, 2004 [Page 14] Internet-Draft draft-iab-char-rep-01 February 2004 References [1] Berners-Lee, T., Fielding, R. and L. Masinter, "Uniform Resource Identifiers (URI): Generic Syntax", RFC 2396, August 1998. [2] Duerst, M. and M. Suignard, "Internationalized Resource Identifiers (IRIs)", draft-duerst-iri-05.txt (work in progress), October 2003. Authors' Addresses Leslie Daigle Editor Ted Hardie Editor Internet Architecture Board IAB EMail: iab@iab.org Daigle, et. al. Expires August 16, 2004 [Page 15] Internet-Draft draft-iab-char-rep-01 February 2004 Full Copyright Statement Copyright (C) The Internet Society (2004). All Rights Reserved. This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English. The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns. This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Acknowledgement Funding for the RFC Editor function is currently provided by the Internet Society. Daigle, et. al. Expires August 16, 2004 [Page 16]