INTERNET-DRAFT Yoram Bernet Diffserv Working Group Microsoft Expires: November 1998 James Binder 3Com Steven Blake IBM Mark Carlson Redcape Software Elwyn Davies Nortel UK Borje Ohlman Ericsson Dinesh Verma IBM Zheng Wang Bell Labs Lucent Technologies Walter Weiss Lucent Technologies May 1998 A Framework for Differentiated Services Status of This Memo This document is an Internet-Draft. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." To view the entire list of current Internet-Drafts, please check the "1id-abstracts.txt" listing contained in the Internet-Drafts Shadow Directories on ftp.is.co.za (Africa), ftp.nordu.net (Northern Europe), ftp.nis.garr.it (Southern Europe), munnari.oz.au (Pacific Rim), ftp.ietf.org (US East Coast), or ftp.isi.edu (US West Coast). Abstract This document provides a general description of issues related to the definition, configuration, and management of services enabled by the differentiated services architecture [DSARCH]. It should be considered as a work-in-progress. Bernet, et. al. Expires: November 1998 [Page 1] INTERNET-DRAFT A Framework for Differentiated Services May 1998 Sec. 1 provides a motivation for the deployment of differentiated services. Sec. 2 examines the range of services that are enabled by the differentiated services architecture. Sec. 3 examines service instantiation, configuration, and management issues. Sec. 4 discusses issues relating to the deployment of differentiated services across multiple provider boundaries or end-to-end. Sec. 5 addresses interoperability with the Integrated Services. 1. Motivation A service is a contract between a network provider and its customer, which outlines the characteristics and behavior of network connectivity offered by the provider to the customer. The network provider may be an Internet Service Provider whose customers include individual users or corporations, an I/T department within an enterprise, or a networking company to which the operations of an enterprise network has been outsourced. Individual users would typically access the network at a single point, while businesses may have multiple access-points to the network. A service level agreement (SLA) may specify different aspects of network behavior, such as the security to be expected by the customer, constraints on the type and amount of data that can be sent on the network, and the performance aspects of communication. The SLA would typically also include a payment/billing scenario as well as the performance (both up time as well as throughput) aspects expected with the associated contract. While services can be differentiated from each other by various aspects, we concentrate on service differentiation from the performance aspect only in this document. Traditionally, network providers have tended to provide all of their customers with the same type of performance (a best-effort service), with most differentiation resulting from the pricing structure (business rates versus individual rates) or the connectivity structure (dial-in access versus leased line T1 access, etc.). However, the size of the Internet continues to grow at an astounding rate, and network capacity has become a scarce resource. Given new types of media and the further reliance on the network as a mission critical resource, there is a need felt by the network providers to offer different types or grades of performance to customers, with network providers offering better performance to customers who are willing to pay more. The key aspects determining service performance are availability and responsiveness. Availability refers to the ability of a customer to maintain connectivity between its access points on the provider network, while responsiveness refers to the round-trip delay and throughputs seen by the customer on its communication. This document presents a framework for service definition and service differentiation in the context of a differentiated service domain as Bernet, et. al. Expires: November 1998 [Page 2] INTERNET-DRAFT A Framework for Differentiated Services May 1998 described in [DSARCH]. 2. Services 2.1 Service Categorization In order to provide the notion of differentiated services, the network provider needs to be able to categorize traffic entering its network into multiple categories. Each of the categories of traffic is marked with a codepoint as described in [DSARCH]. These codepoints (or per-hop behaviors (PHBs)) in turn can be used to build a specific service offered by the Network Provider to the customer. Service differentiation may be made along multiple dimensions. A provider may offer a customer a service which provides different performance assurances to packets being received by the customer from the provider network, or to packets being sent from the customer network into the provider network, or a combination of the above. A business hosting a web-site might have a contract with its ISP, which enables the responses out from the web-server to be carried at a better performance level than the default. One business may not want to pay for a better performance level for requests trying to reach its server, while a third business may be willing to pay more both for its requests as well as responses. Depending on the type of service contract between the customer and the provider, the network may control packets being sent into it, packets being sent out of it, or both. A service may be defined which is dependent on the time-of-day. A customer may want a better performance of service during business hours, and may choose to revert to the default behavior during off- hours. A customer may require that a different path be offered to its packets which satisfy certain constraints (e.g., a minimum of T1 capacity along any link connecting the service), or meet certain geographical or topological constraints (e.g., packets must not leave the boundaries of the United States). A service may be defined based on the performance level to be expected between a pair of customer access points to the network. Thus, one may define a service, which would offer statistical bounds on the delay or loss rate to be experienced between two access- points. While such performance bounds may be different for different access-points, some providers may choose to offer a service, which would provide the minimum acceptable performance among any two points on the customer network. Such performance assurances may be coupled with constraints on how much traffic a customer may inject into the provider's network. A service may be defined on the basis of the fault tolerance properties that are offered to the customer. The service may specify Bernet, et. al. Expires: November 1998 [Page 3] INTERNET-DRAFT A Framework for Differentiated Services May 1998 resumption of the service within a certain limit of time, or promise upper bounds on the total connectivity downtime over a period of time or the amount of traffic which is dropped due to congestion over a given period of time. Services, as may be obvious from the preceding paragraphs, could be quite varied and rich in context. Different ISP's or vendors may offer different types of services to their customers over the network. These diverse services need to be supported and mapped onto a few PHBs within the DS domain of the network provider. In Section 2.3, we offer some examples of the services that may be offered by a network provider, and illustrate how they could be supported using a few simple PHBs. A service agreement between a customer and a provider is usually captured in the form of a SLA. The SLA is normally in the form of a binding business contract, which specifies the features and performance requirements of the service provided, as well as traffic profiles and/or corresponding packet marking requirements that are captured in a traffic conditioning agreement (TCA) [DSARCH]. SLA's may be static or dynamic. Static SLAs, that are the norm at the present time, are defined at service initiation time, and do not change frequently. Static SLAs may include the provision of specific performance levels to a selected set of applications, time-of-day changes in performance requirements, and changes dependent on network load, as long as the specific agreement between the customer and provider does not change. Dynamic SLAs, on the other hand, are possibly subject to frequent changes, and may require an automated protocol such as the bandwidth broker [BB] or other methods between the customer and the network. 2.1.1 Sample SLA The following example illustrates a boundary between two networks and a simple static SLA: The customer's network sends traffic to the provider's network via the customer's egress router. The provider's network receives traffic from the customer's network via the provider's ingress router. The SLA takes the following form: Service Level Profile Premium R = 100 Kbps, b = 1000 bytes, r = 100 Kbps Assured Gold R = 1000 Kbps, b = 1000 bytes, r = 250 Kbps (where R: peak rate, b: burst size, r: average rate) Some of the characteristics that might be used to describe an SLA between a customer and a service provider are discussed in the following paragraphs. Bernet, et. al. Expires: November 1998 [Page 4] INTERNET-DRAFT A Framework for Differentiated Services May 1998 2.1.1.1 Qualitative Performance and Quantitative Performance SLAs may characterize Performance levels in qualitative (i.e., relative) terms, or in absolute quantitative terms. An example of a qualitative performance guarantee would be "Traffic from stream A will be given priority over traffic from stream B". An example of a quantitative performance guarantee would be "Traffic from stream A, if provisioned at an average rate of 100 Kbps, with bursts not exceeding 64 KB allowed at a peak rate of 1 Mbps, will then have a loss rate of no more than 0.001%, when averaged over the interval of 1 hour". 2.1.1.2 Throughput Characteristics Throughput characteristics (sometimes referred to as Token Bucket Models) are another means to define a quantitative SLA. This model could be used to describe a service that guarantees to deliver traffic at a certain sustained rate and to accommodate bursts of a limited size up to a certain peak rate. 2.1.1.3 Latency Parameters A SLA may define maximum latency (or delay) as well as jitter bounds for any packet within a specifically marked traffic stream within a certain traffic profile. 2.1.1.4 Packet Loss (or Drop) Probability A service may guarantee a limit on the percentage of packets dropped from a certain flow. Typically, such a service is defined using token bucket parameters and a drop probability. This is typically a relative contract. For example - "When traffic from flow A conforms to X token bucket parameters, it is 90% less likely to be dropped than traffic from flow B. If it does not conform to the profile..." 2.1.1.5 Additional Performance Parameters Additional performance parameters may be offered using specific PHBs. differentiated services does not dictate a specific set of performance parameters. 2.1.2 Service Constraints SLAs may be offered which meet certain constraints in addition to those listed above. Two of the most obvious constraints are Locality and Time. 2.1.2.1 Flow Locality-based Constraints SLAs may be offered only for communication between specific ingress Bernet, et. al. Expires: November 1998 [Page 5] INTERNET-DRAFT A Framework for Differentiated Services May 1998 and egress points. Others may allow combinations of various ingress or egress points. For example, services may be offered: a. For all traffic between ingress point A and egress point B. b. For all traffic originating at ingress A, to any egress point. c. For all traffic originating at any ingress point, to egress point B. d. For traffic from the group of ingress points A to the group of egress points B. 2.1.2.2 Time based Constraints These SLAs would typically specify a reduced or improved performance service for certain hours of the day and/or days of the week or month. 2.1.3 Other Service Characteristics Although differentiated services focuses primarily on performance related characteristics, other non-performance characteristics may be offered as part of a contract. Examples of these non-performance-based characteristics might include: availability and reliability, security (e.g., encryption). The SLA shown in Sec. 2.1.1 defines two levels of service to the customer at the specific boundary [CLARK]. 2.2 Service Provisioning In order to support a range of different services, a network would map all traffic into one or more of the PHBs that it supports. One of the preconditions for satisfactory performance of the network is that the provider provision its network so as to be able to meet the performance needs of the customers under normal operating conditions according to the negotiated SLAs. Thus, adequate attention must be given to the right selection of the speed of the network links available within the DS domain of the network provider. The provisioning of the network would be done on the basis of the predicted (or contracted) traffic, which the network provider would expect from its customers. The provisioning decision has to take into account the needs for traffic belonging to different PHBs. Let us examine two simple cases of PHBs that can be supported in the provider network. [DSHEAD] specifies two PHBs, an Expedited Forwarding PHB and a Default PHB. Packets marked with the Expedited Bernet, et. al. Expires: November 1998 [Page 6] INTERNET-DRAFT A Framework for Differentiated Services May 1998 Forwarding PHB are put into a queue which is serviced frequently, and packets belonging to the Default PHB are serviced less frequently when the Expedited Forwarding queue is non-empty. Within this context, the network may expect a certain amount of traffic marked with the Expedited Forwarding PHB between its various access points, and a different amount of traffic marked by the Default PHB between the different access-points. In order to meet the desired performance goals of the network, the network provider may decide to provision the links in its network, or conversely, the SLAs, so that the expected load due to Expedited Forwarding traffic on any node/ link in the network does not exceed a fixed percentage (e.g., 10%), and the expected load due to the combined (Expedited Forwarding and Default) traffic on any other element does not exceed another (higher) percentage (e.g., 90%). There are well-established tools and algorithms, which would enable the network provider to obtain an optimum configuration of the network to satisfy these constraints. Another PHB case may consist of defining a limited number of output queues (e.g., four) at each of the routers in the DS domain. A packet marked with one of the distinct codepoints is placed into one of the four queues in the network. The queues are served using a algorithm according to weights which are configured by the network provider at the different routers. The network provider would compute the expected distribution of traffic at each of the network elements belonging to each codepoint. It would then assign weights at the different routers so that they match the ratio of the different traffic load expected at the specific router. With this provisioning of the network, the provider can expect to satisfy the average load on the network in a satisfactory manner. In either of the two cases (or any other network provisioning case) described above, the network provider's predictions of traffic may not match the actual usage of the network. When the actual usage of traffic exceeds the provisioned traffic for any specified codepoint, the network provider may choose to regulate the amount of traffic that could be allowed into the network for a specific codepoint. The excess traffic may be discarded, smoothed, or converted to another codepoint, or just billed at a different rate depending on the policies adopted by the network provider. A network provider would typically run metering and accounting software on the access points to estimate the amount of traffic flowing between its different customer access-points. An estimation of the these traffic flows can then be used in order to provide admission control of traffic belonging to different codepoints in the network (assuming some form of admission control protocol or policy is in place). When the network provider opts to use admission control to limit the amount of traffic belonging to different codepoints (and hence enforce or validate a given SLA), each access-point has a limit on the amount of traffic it can inject into the network. The limit may Bernet, et. al. Expires: November 1998 [Page 7] INTERNET-DRAFT A Framework for Differentiated Services May 1998 be enforced as an aggregate depending only on the codepoint of the traffic entering the network, or enforced on separate traffic streams which may be defined by the codepoint of the traffic as well as by other header field values, including the destination access-point (prefix) of the stream. Access control on the basis of separate traffic flows or streams is better from the perspective of efficient network resource usage, but requires the ingress access-point to maintain more routing information to determine the egress access- point. The shaping of the traffic based on some form of admission control could be distributed to the edges of the network (i.e., host or first-hop router) as needed. If this is done, then minimal state is needed within the core of the network while still maintaining the SLA. In either of the two modes of access-control defined above, the DS network needs to determine the amount of bandwidth to be assigned to a specific codepoint at each access-node. The amount of bandwidth may be obtained by management software in the network which determines the routing topology of the network, obtains the expected traffic at each access-point, and determines the correct admission control limits for the traffic at each access-point. A distributed version with admission control daemons that track resource usage at each of the intermediate routers and hosts participating within the DS domain can also be used. 2.3 Service Examples In this section, we describe a few service examples, and show how some specific PHBs could support them. We use two simple sets of PHBs. The first set of PHBs consist of two codepoints, one specifying Expedited Forwarding behavior and the other specifying a Default behavior [DSHEAD]. The second set of PHBs consist of four codepoints, each specifying a different queue at each router, the queues being serviced on a weighed basis as configured by the network provider [DSPREC]. 2.3.1 Services Enabled by First set of PHBs. The ISP offers three levels of service to the businesses that are hosting web-servers on its DS domain. The "Preferred Service" enables users trying to access the business sites with better performance for both the request and response of the web-server. The "Special Service" enables users trying to access the business sites with better performance for the web-responses, while the "Normal Service" provides the default performance for both requests and responses. In order to provide the services using the Expedited Forwarding and Default PHBs, the network provider maintains a list of the addresses and ports at which the preferred and special clients' web-servers are operational. For all packets in which either the source host/port Bernet, et. al. Expires: November 1998 [Page 8] INTERNET-DRAFT A Framework for Differentiated Services May 1998 combination or the destination host/port combination identifies a preferred web-server, the packets are marked with the Expedited Forwarding codepoint. If the source host/port combination indicates a special service web-server, the packets are also marked expedited. All other packets are marked with the default codepoint. Using this scheme, the right type of performance is given to the customers hosting a preferred web-server. A second type of classification service may be provided by an ISP to its business clients using this PHB set. For each of the business customers, some set of web-sites are identified as being more relevant to their business. The ISP would provide expedited forwarding to traffic from the web-sites that are considered relevant to its business by the customer. As opposed to the preferred/special service definition presented above, the customer accessing the web- site, not the person hosting the web-site, drives the classification. Such a marking can be done by supporting a PICS capable web-proxy at each ISP access-point which uses a business rating system to classify the web-pages into those relevant/irrelevant for a specific customer, and marking the packets as Expedited Forwarding/Default appropriately. Another example of service that can be provided using this is preferential service that may be given to "Premium Customers" when they use the network provider to provide connectivity between different customer premises. The packets, which contain the source or destination addresses of the premium customers, are marked as Expedited Forwarding packets while the other packets are marked with the default codepoint. For example, an enterprise with an out- sourced VPN which is replicating multiple databases or directories across different geographies or other mission critical functions across the VPN. In this example, all data being transmitted between the mission critical replicated servers would be classified as "Preferred Service" and given priority across the VPN (similar to a virtual leased line). User traffic from outside the network destined to specific servers within the network (i.e., web-servers) might be given a different level of access called "Response Service". This service would have other specific characteristics such as minimum response time, latency and availability requirements. Any other traffic on the VPN would be marked as "Normal Service" and would be considered a best effort service, associated with the default PHB. 2.3.2 Services Enabled by the Second Set of PHBs An ISP offers two services to its customer. The "voice" traffic results in a low bandwidth low-delay communication across the ISP network. The "video" traffic results in a high-bandwidth low-delay communication across the ISP network. The "data" traffic provides behavior equivalent to that found in Bernet, et. al. Expires: November 1998 [Page 9] INTERNET-DRAFT A Framework for Differentiated Services May 1998 current IP networks. The ISP supports these three modes of services by mapping them into the three different queues at each router. The ISP periodically recomputes the load and routing patterns of its voice and video connections, regenerates the weights on each backbone router to support this set of services, and reconfigures the router. A second service offered by the same ISP is that of a fixed bandwidth pipe. The ISP offers a specific bandwidth between two access-points of its customer. The ISP adjusts the weights of the different queues to meet the bandwidth requirements of all the customers passing through a router in the appropriate queue. These weights are only recomputed when a new pipe is added or an existing pipe modified or deleted. 3. Service Control Mechanisms 3.1 Service Allocation and Configuration Each access-point to a DS domain needs to be configured with the appropriate packet classification rules. At each of the access- points, the provider of a specific DS-domain is either a provider to a customer, or may be a customer of yet another provider. The support of a SLA requires that the configuration of functional components at the boundary be done with parameters that are designed to support the required SLA performance levels. In order to support specific services, specific functionality may be required of the classifiers, meters, markers, shapers and policers at an access- point [DSARCH]. Boundaries between two DS networks and between a DS and non-DS network, are of particular interest. At each such boundary, at least one network is a customer and one is a provider. The provider agrees to carry certain subsets of the customer's traffic at certain service levels. This agreement is captured in the form of an SLA. Specific functionality is required at such boundaries, to serve the needs of the customer and the provider, subject to the SLA. This section describes the methods by which such functionality may be configured. 3.1.1 Service Allocation Two types of SLA may exist, static and dynamic. A static SLA changes infrequently (typically weekly, or monthly) and typically is renegotiated by human interaction. Dynamic SLA's change frequently and typically are renegotiated via a machine to machine negotiation protocol of some sort. Note that a static SLA may call for different service levels to be supported at different times of day, in which case, the service covered by the SLA changes automatically and somewhat frequently, however, the agreement itself does not change frequently. Bernet, et. al. Expires: November 1998 [Page 10] INTERNET-DRAFT A Framework for Differentiated Services May 1998 In order to adhere to a SLA, it is useful to configure certain functional components at access-points. These may include classifiers, meters, markers, shapers, and policers, as explained below. An access-point to a DS domain can run in one of two modes of classification, as a microflow or traffic stream classifier (MFC), or a bandwidth aggregate classifier (BAC). As a MFC, the access- point classifies microflows (identified by the 5-tuple of src/dest IP addresses, src/dest ports and the protocol) or traffic streams (identified by subsets of the 5-tuple) and marks the DS field according to the classification rules that are configured for it. As a BAC, the access-point expects that the customer has already marked the DS field appropriately after running a MFC function on its own behalf, and only looks at the DS field. Even as a BAC, the access- point may choose to remark the DS field as it may check the aggregated traffic level associated with the DS field to make sure it fits into limits defined for it by the appropriate SLA. In either of the two modes, the access-point can perform the policing function on the aggregate stream as a result of the classification process. In the first mode, the customer applies no traffic conditioning. Instead, the customer contracts with the provider to do per-microflow or per-traffic stream classification, marking and policing. In the second mode, the customer applies per-microflow/stream classification and marking. In this case, the provider applies aggregate (per- service level) classification and policing, to assure compliance with the SLA. It is assumed that all traffic from a single customer arrives at an interface dedicated to the customer. If multiple customers share a single interface, it will be necessary to apply additional finer-grain classification for the purpose of conditioning. Provider marking will typically be applied at the periphery of the DS domains, where a DS network connects to non-DS, stub networks. Shaping is not included in the above discussion, but may be recommended in the customer's network, or the provider's network or both. Typically, flows will be shaped within the customer's network. By shaping at the microflow level, the customer can maintain traffic separation, ensuring that no microflow will seize more than its share of the aggregate DS domain resources guaranteed by the SLA. By shaping at the aggregate stream level, the customer can assure aggregate compliance with the SLA and for example can avoid charges that may have been negotiated as part of the SLA for excess traffic. However, by shaping at the aggregate stream level only, the customer cannot assure traffic separation once the traffic is injected into the service provider's network. Devices that do full microflow classification will be more complex than those that just offer bandwidth aggregation classification Bernet, et. al. Expires: November 1998 [Page 11] INTERNET-DRAFT A Framework for Differentiated Services May 1998 support. The costs associated with these devices will should be considered when determining where the classification and flow shaping should occur within the DS domain. In certain cases, the customer may contract to have the provider apply microflow or aggregate shaping. This is a form of value-add which providers may offer customers that are incapable of providing their own shaping. Obviously, wherever shaping is applied at a microflow level, microflow classifiers will be required. Wherever shaping is applied at the behavior aggregate (BA) level, BA classifiers will be required. In general, classifiers are required only to support other traffic conditioning functionality such as policing, marking, and/or shaping. Subsequent discussion of specific functionality implies the co-existence of the appropriate classifier. A static SLA with marking performed by customer requires that the customer provide an MFC at or before its egress, and the provider provide a BA policer at the ingress access-point. A static SLA with provider marking would require that the access-point support a MFC and BA policer at the ingress access-point. 3.1.2 Service Configuration Configuration requirements may vary depending on several parameters: Variations in the distribution of the configured functionality, e.g., use of the MFC mode versus the BA mode, specific functional components being configured (policers versus markers), and the nature of the SLAs (static versus dynamic). The functionality and operational mode of access-points (egress as well as ingress) need to be specified within a DS-domain. In order for the classification to work efficiently, it must be simple and easy to configure. Furthermore, the configuration of different access-points must be consistent. Otherwise, the performance of the service provided to the different customers may be erratic. For example, assuming the preferred ISP service described in Section 2.3.1 is being supported, all the access-points must have an entry classifying the same set of sites into its "Preferred" class. Note that consistency does not imply that the configuration information is identical for all the access-points. In the special ISP service described in Section 2.3.1, the configuration entry for each web-server needs to be only present at the access-point to which the web-server is attached. The simplest approach to the configuration problem is to ignore it. The assumption would be that the ISP would configure each router manually, and in a static manner. This solution requires extensive manual upgrade to each ISP router whenever a new service or a customer is added or deleted. The solution would not work well in practice. Bernet, et. al. Expires: November 1998 [Page 12] INTERNET-DRAFT A Framework for Differentiated Services May 1998 Another option might be for the DS field to be marked with the correct PHB service request by hosts or first-hop routers within the CPE and then expedited through the ISP's (and billed based on this service) network as described. The ideal solution should provide a single administration point where the ISP can enter the configuration rules from a centralized site. The configuration information should also permit scalability, in that a large number of different ingress routers should be capable of receiving the configuration information. At the same time, the configuration information should not become a single point of failure, which can bring the entire network to a halt. One solution to the configuration problem is a protocol, which permits the administration at a single master site, but allows replication to several slave/mirror sites, which can be looked up by a large set of access-points to permit a scalable operation. Another desirable attribute for scalability would be the ability to cache a limited set of classification rules and to query other rules as and when needed. Several alternatives exist for such a configuration policy. Some of these alternatives are outlined below: The SNMP MIB approach: One may define a diffserv-specific SNMP MIB that needs to be supported at each of the access-routers, and can be populated using a SNMP manager. The MIB definition approach would enable a centralized administration of the configuration information. However, the MIB approach does not enable caching in an efficient manner. Since the MIB for each access-point would have distinct entries, consistency would have to be enforced using a layer above the SNMP manager. The LDAP approach: One can store the configuration entries in a directory accessible using the LDAP protocol. The directory is looked up at the initialization of the access-point. Directories permit centralized administration at one master site, and support replication to different slave sites. The access-point can also cache portions of the classification rules, and can query the appropriate rule when they detect the initiation of a new session (e.g. seeing a TCP header with the SYN flag). The directory schemata also permit a way to provide configuration information for a group of access-point in a single entry, which can provide consistent configuration across several access-points. The one drawback of the directory protocol is inadequate support for asynchronous notification, which is currently being discussed, in the various working groups. A tentative schema for classification rules using this schema has been proposed in Bernet, et. al. Expires: November 1998 [Page 13] INTERNET-DRAFT A Framework for Differentiated Services May 1998 [Ellesson]. The COPS approach: One can extend the policy query protocol for RSVP to provide classification information for differentiated services. COPS supports asynchronous notification in a better way but has not yet addressed issues of replication of classification rules at different sites [COPS]. COPS extensions that support diffserv semantics would need to be developed in order to use this approach. When dynamic SLA's are being supported, the above mentioned configuration approaches need to be augmented with a negotiation protocol between the customer and provider network to negotiate the current details of the service levels. To this end, a standard protocol is required between customers and providers. Such a protocol would enable customers to present requests to providers that would result in changes to the SLA. Entities within the provider's network would respond to these requests by determining if the requested SLA could be met, adjusting the policers accordingly and responding to the customer's request. The bandwidth broker, as described in [BB] is such an entity. One realization of such a protocol is RSVP [RSVP]. Other protocols could also be devised for this purpose. 3.1.2.1 Functionality Required at Boundary Equipment In the interest of adhering to the SLA, it is useful to configure certain functional components at DS boundary nodes. These may include classifiers, markers, meters, shapers and policers as described in the following section. The following basic combinations of functional components are possible: Mode Customer's Egress Provider's Ingress Provider marking None MFC, BAC, M, P Note: Provider classifies microflows to aggregated service level and marks DS field accordingly. Provider polices based on aggregate. Customer marking MFC, M BAC, P Note: Customer classifies microflows to aggregated service level and marks DS field accordingly. Provider polices based on aggregate. MFC - MF Classifier Bernet, et. al. Expires: November 1998 [Page 14] INTERNET-DRAFT A Framework for Differentiated Services May 1998 BAC - BA Classifier M - Marker P - Policer (policing shaper) In the first example, the customer applies no traffic conditioning. Instead, the customer contracts with the provider to do per-microflow classification, marking and policing. In the second example, the customer applies per-microflow classification and marking. In this case, the provider applies aggregate (per-service level) classification and policing, to assure compliance with the SLA. It is assumed that all traffic from a single customer arrives at an interface dedicated to the customer. If multiple customers share a single interface, it will be necessary to apply additional finer- grain classification for the purpose of policing. Provider marking will typically be applied at the periphery of the DS domains, where a DS network connects to a non-DS, stub network. Note that shapers are absent from the table above. Shapers are not strictly required but are recommended, at the customer's network, the provider's network, or both. Typically, flows will be shaped within the customer's network. By shaping at the microflow level, the customer can maintain traffic separation, ensuring that no microflow will seize more than its share of the aggregate DS domain resources guaranteed by the SLA. By shaping at the aggregate stream level, the customer can assure aggregate compliance with the SLA and for example can avoid charges that may have been negotiated as part of the SLA, for excess traffic. However, by shaping at the aggregate level only, the customer cannot assure traffic separation. In certain cases, the customer may contract to have the provider apply microflow or aggregate shaping. This is a form of value-add which providers may offer customers that are incapable of providing their own shaping. Obviously, wherever shaping is applied at a microflow level, MF classifiers will be required. Wherever shaping is applied at the behavior aggregate level, BA classifiers will be required. In general, classifiers are required only to support other functionality such as meters, markers, or shapers. Subsequent discussion of specific functionality implies the co-existence of the appropriate classifier. 3.1.2.2 Configuration of Functionality at Boundary Equipment Configuration requirements may vary depending on the following parameters: 1. Variations in the distribution of configured functionality, as described in the previous section. Bernet, et. al. Expires: November 1998 [Page 15] INTERNET-DRAFT A Framework for Differentiated Services May 1998 2. The functional component configured (for example, policing vs. shapers vs. markers). 3. The nature of the SLA (static vs. dynamic). This section describes configuration methods subject to the variations listed. 3.1.2.3 Static SLA, Provider Marking In this scenario, it is necessary to configure the following functional components: 1. A MF classifier + marker at the provider's ingress node. 2. A BA policer at the provider's ingress node. The SLA specifies the traffic profiles that should be configured per- service level for the policer at the ingress node of the provider's network. The format of the information to be configured is tabulated in Sec. 2.1.1. Since the SLA is static, this information changes infrequently and likely requires human intervention. Therefore, the policer will typically be configured remotely via SNMP or via a command-line interface. The marker configuration is actually independent of the SLA. The customer may contract the provider to mark any service level based on the results of the MF classification. Configuring the marker is similar to configuring access lists on standard switches and routers. Such lists take the form of "n-tuple: service level", where "-tuple" refers to some combination of possibly masked values in packet headers. Such information is lengthy and error prone. In addition, it is subject to change more frequently than a static SLA and, does not require the approval of the provider. While it is possible to configure markers using the same static methods as used to configure the policers (SNMP and/or command-line interfaces), there are strong incentives to provide dynamic marker configuration via a standard protocol that is available to the customer. RSVP may be used for this purpose. 3.1.3 Receiver Control In differentiated services there are two possible types of receiver control. One, which has been addressed in a previous section, is when the receiver is allowed to control how the marking of the DS field is performed at the sender side. This can be done either in accordance with a static SLA that the user has with the provider, or by use of application- or session-level signaling which induces the Bernet, et. al. Expires: November 1998 [Page 16] INTERNET-DRAFT A Framework for Differentiated Services May 1998 sender (or a DS edge node) to mark the packets in accordance with the receiver's preferences. Another type is when the receivers need to control the priority of packets that arrive from the network onto his/her access link. There are two reasons why this is important. One is that it provides protection from certain types of denial-of-service attacks. The other is that this is important on low bandwidth access links, in particular for mobile IP hosts. At the last-hop router, before the egress link, traffic from different sources are merged onto the access link of the receiver (destination). Some packets might have been marked with respect to the SLA the receiver has with the ISP, others might not. To allow the receiver to control which traffic gets priority on its access link it should be allowed to direct the egress node re-mark (or use an alternative PHB than what is in accordance with the marking that was made with respect to the sender's SLA) packets at the egress node, in accordance with a "receiver marking policy". This can be thought of as a special case of a static or dynamic SLA between a downstream network (the ISP) and an upstream network (the receiver). Such a "receiver marking policy" could e.g. state that all IP telephony traffic (e.g., identified by a particular source port number) should be given priority to all other traffic. 3.1.4 Policy Protocols Two types of requests have been described which can result in allocation of resources to particular users (at the expense of others), and which may result in charges to customers. The two types of requests are requests to mark traffic for prioritized treatment (subject to the terms of an existing SLA), and requests to change the existing SLA. These requests should be governed by policy decisions. Information required to make policy decisions must be conveyed to policy servers. These must reply with approval or rejection of the request. COPS is an example of a policy protocol suitable for carrying such requests. 3.2 Requirements for Measurement and Management. In order to validate conformance to the specific SLA's within a DS domain, the network operator should measure the performance of the traffic belong to specific codepoints within its domain. The measurement of traffic can be done in one of two modes: Single Point Measurements: These measurements would monitor the traffic in the network at a single point in the network. Such a measurement can be done at the access-point of the network after classification, and reported to the network management tools using RMON MIBs [RMON]. A network operator can also station real-time flow Bernet, et. al. Expires: November 1998 [Page 17] INTERNET-DRAFT A Framework for Differentiated Services May 1998 monitors, and collect a more comprehensive record of the type of traffic flowing in the network [RTFlow]. Dual Point Measurements: These measurements would monitor the performance of the traffic between ingress access-points and the egress access-point within the DS domain. The measurement of performance could be done by means of network performance monitoring protocols stationed at the two egress points. Network performance monitoring protocols tend to be relatively heavyweight in their use of network bandwidth, and these protocols should be used only when a potential violation of the SLA is suspected between the two access- points. It would be desirable for a DS domain operator to have a continuous low-overhead monitoring of the service-levels obtained by packets within its domain using a passive monitoring scheme. When the passive monitoring scheme suspects a potential problem, the more heavyweight performance monitor can be activated. A SNMP manager may be used as an intermediary to trigger this activation, with the passive monitoring protocol generating an SNMP trap on suspicion of SLA violation and the manager subsequently activating active performance monitoring. 4. Inter-Domain Considerations and End-to-End Services While the differentiated services architecture works very well to enable different service level agreements within a single ISP domain or in a corporate Intranet, comprehensive service differentiation in the Internet requires that there is support for service differentiation in a uniform manner across several ISP's. In general, communication across the Internet traverses several ISP boundaries. When a browser accesses a web-server, the connection traverses the client (browser) network; multiple ISP networks, and eventually reaches the server network. The response from the server traverses a (possibly asymmetric) path back to the client host. In these cases, SLAs are usually only specified between the client and its ISP, the server and its ISP, and between the neighboring pairs of ISP's. Within this context, one must enable the definition of services across the Internet using only bilateral SLAs between the adjacent providers. In order to appreciate the issues in service definition; let us consider the "Preferred Service" web-service described in Section 2.3.1. When the preferred service is provided, any client accessing the web-server must be provided Expedited Forwarding service for its requests as well as its response stream. As long as the client browser connects to the same ISP as the server, the preferred service is straightforward to provide. However, when the client browser is connecting to a different ISP, expedited processing for request packets is not enabled until they reach the domain of the ISP Bernet, et. al. Expires: November 1998 [Page 18] INTERNET-DRAFT A Framework for Differentiated Services May 1998 immediately connected to the server. Similarly, the expedited processing for response packets may not be honored once the response packets leave the domain of the ISP. Some of the methods by which the service concept can be extended across multiple ISP's are outlined in the following sections. 4.1 Service Provisioning Across Boundaries There are two basic schemes which can be used to provide service levels across ISP boundaries. The first scheme consists of negotiating an aggregated bandwidth classification to one neighbor, and the second scheme consists of negotiating a microflow or traffic stream classification to a neighboring ISP. An aggregated bandwidth negotiation is useful in preserving the desired service levels of packets that are leaving an ISP DS domain. If an ISP A is sending packets into the DS domain of an ISP B, ISP A would negotiate a SLA which would honor the DS field marking created by ISP A. As packets enter the domain of ISP A, they would be marked with a specific codepoint. Before the packets leave the domain of ISP A and cross over to ISP B, they would be remarked so that DS field encoding matches the ones expected by ISP B as per the SLA between A and B. For example, let us assume that the SLA between A and B specifies that A would treat all packets marked for Expedited Forwarding with the same PHB as long as the aggregate inbound packet rate is less than a certain limit. For the service category of "special service" as described in Section 2.3.1, ISP A would mark all response packets originating from the special web-servers for Expedited Forwarding. It would also negotiate the SLA with ISP B so as to obtain a rate for Expedited packets, which would be adequate to meet the demand for its servers. Since ISP B would carry the packets at expedited level through its domain, the special service would be provided to all clients accessing either ISP A or B. A transitive agreement between B and other service clients would lead to the service being special to all clients in the Internet. For boundaries where the aggregated bandwidth is negotiated, one ISP can simply act as the customer of another ISP. The marking and policing scheme would work as for any other DS domain boundary situation. The aggregated bandwidth negotiation is not adequate for providing the preferred service described in Section 2.3.1. The packets originating from a client who is connecting to ISP B and trying to access a preferred web-server (as defined in ISP B's SLA with ISP A) will not receive the Expedited Forwarding treatment in Bernet, et. al. Expires: November 1998 [Page 19] INTERNET-DRAFT A Framework for Differentiated Services May 1998 the domain of ISP B since ISP B has no information that allows it to treat these packets specially. The client may have no SLA with ISP B. One of the ways in which ISP A can manage to extend the preferred service agreement to clients in the domain of ISP B is to exchange a microflow- or traffic stream-level SLA with ISP B. In this negotiation, ISP A would inform ISP B about the set of destination addresses (or prefixes) which need to be treated preferentially. In return ISP A would configure all of its access-points to mark the packets headed to the preferred service for Expedited Forwarding. An ISP may want to choose some other codepoint for marking packets belonging to the microflows/streams, which have a SLA with some other ISP. There is also an associated scaling problem with too many microflow/stream classifier entries floating across multiple ISP boundaries. The ISP may want to export the microflows/streams classifier designations only across a certain number of administrative domains, and pay different amounts depending on how far the microflow/stream classification negotiation would be carried. An ISP may also want to provide different levels of preferred service to its web-server customers. A service where preferred service is extended to all the clients in the Internet may cost substantially more than a service where preferred service is only extended across one or two ISP domains. 4.1.1 Requirements for Service Level Agreements Although the differentiated service architecture focuses on differentiated treatment within a single domain, it is expected that different DS domains will be linked to each other through mutual agreements and the DS region will expand in an incremental fashion. How two different DS domains are interconnected is governed by the SLA between two-DS domains. The SLA will specify details as to how each other's traffic is conditioned at the boundary of the two domains, and how packets are re-marked to be promoted or demoted within a PHB group. A SLA is normally a contract between two DS domains, which specifies details of treatment for traffic crossing the boundary of the two domains. A SLA may contain the following: Performance, Security, Availability between the domains: A bilateral agreement may dictate when the service is available and what type of performance and security are expected when the service is running. Rules for traffic conditioning: A SLA should specify how traffic from one DS domain will be conditioned crossing the boundary, including classification rules (MF classification or BA classification), profiles for each PHB, actions for packets out-of-profile (marked with a different PHB or shaped/discarded). Bernet, et. al. Expires: November 1998 [Page 20] INTERNET-DRAFT A Framework for Differentiated Services May 1998 Mapping of PHBs between two domains: If the two DS domains use two distinct PHB groups, it may be necessary to establish some rules for mapping one PHB to another. Location of traffic conditioning: The packets can be conditioned by the egress node of one DS domain before the traffic crosses to the next DS domain, or by the ingress node of the next domain. 4.1.2 Promotion and Demotion within a PHB group When two DS domains use similar PHB groups, packets crossing the boundary may be promoted or demoted to use a different PHB with the same PHB group. Such promotion and demotion take place when there is a mis-match between the traffic rate specified in the SLA and the actual traffic rate. For example, two ISP's both offer three traffic classes: premium, standard and best effort. If the amount of premium traffic crossing the boundary is greater than the rate specified in the SLA, some of the premium traffic may be demoted to standard class. On the other hand, if there is less premium traffic than specified in the SLA, standard traffic may be promoted to premium. 4.1.3 Mapping of PHBs at Boundaries When two DS domains are based on very different PHB groups, the SLA has to specify how one PHB in one DS domain is mapped to a PHB in another domain. If the PHBs have different values but similar semantics, the mapping may be simple. However, in some case, the mapping can be very difficult. For example, one DS domain uses queueing priorities while the other is based on discard priorities. However, in general, a "better" treatment in one PHB group should be mapped to another "better" treatment. 4.2 Host Interactions In the previous sections, we discuss the differentiated services primarily from the perspective of a service provider. In this section we look at how the users can make use of services offered by DS-capable networks. A host which is DS aware is just another DS egress device; not a special case. 4.2.1 Traffic Conditioning for Directly Connected Users A user can gain access to a DS-capable network directly or through a customer domain. Users who are directly connected to a DS-capable ISP (e.g., a dial-up user) should have a SLA with the ISP. Complex traffic conditioning functions ideally should ideally be performed by the hosts of the users rather than the access nodes, which may have to handle tens of thousands of users. However, mechanisms have to be in place so that the ISPs are assured that they can trust the self-policing by the users. Bernet, et. al. Expires: November 1998 [Page 21] INTERNET-DRAFT A Framework for Differentiated Services May 1998 Users may have to install DS control software from ISPs. When a user dials in, the authentication process also configures the DS control module in the user's host according to the SLA between the user and the ISP. Random checking can also be done to ensure that the SLA is maintained. 4.2.2 Service Allocation within Customer Domain When users are part of a customer domain, there is an issue of local resource allocation. The customer domain should have a policy as to what packets can be set to use a particular PHB group. Some form of allocation, either static or dynamic, is needed in order to share the profiles that the customer domain has obtained. The egress node of the customer domain may perform traffic conditioning on the aggregated traffic to ensure that the packets crossing the boundary to the ISP conforms to the SLA between the customer domain and the ISP. A number of mechanisms for local allocation have been suggested. One is bandwidth broker, which acts as a central allocation server for the customer domain and as a client to request additional resources from the ISP [BB]. RSVP may also be adopted as a mechanism for requesting bandwidth with the customer domain and the resources to the ISP [RSVP]. 4.2.3 End-to-End Services The differentiated services architecture is designed to be deployed incrementally. ISPs are likely to offer differentiated services first within their own networks, and conduct experiments. Over time, ISPs will design business models for the new services and start to extend the services by signing SLAs with other peering ISPs. Since the differentiated services model is biased towards long term allocation rather than on-demand reservation, the guarantees for any differentiated services come through proper provisioning. Unless there is a clear traffic pattern, provisioning becomes much harder when more DS domains are linked together. The level of provisioning depends on the economic factors in the differentiated services and varies from ISP to ISP. In general, when the packets leave the ISP to which the customer has a SLA, the level of service may degrade. For example, if the traffic aggregate for a particular PHB were over the rate the next ISP would accept, some packets may be demoted to a PHB with worse treatment. The more ISPs a packet traverses, the less control the originating ISP has. The extensibility of the differentiated services model allows many different schemes to be tried out and developed further. One possible approach is to establish an ISP-to-ISP pipe across multiple ISPs. So, the originating ISP and terminating ISP know exactly how traffic between them is provisioned. This is similar to Bernet, et. al. Expires: November 1998 [Page 22] INTERNET-DRAFT A Framework for Differentiated Services May 1998 the VPN services that an ISP offers for connecting different sites of an organization. 5. Interoperability with Integrated Services In this section, we discuss three possible options for the interoperability between the differentiated services and Integrated Services [IntServ]. 5.1 Parallel Operation Both differentiated services and Integrated Services may operate in parallel over the same network infrastructure. It may be necessary to set limits on the amount of bandwidth available for Integrated Services reservations so that the bandwidth provisioning for differentiated services traffic is protected. In general, differentiated services provides an overall provisioning to aggregates of traffic while Integrated Services can be used to make reservation for long-lasting and high-bandwidth flows. This is similar to the current telephone networks. One does not need to make a prior reservation in order to make a phone call. The likelihood of blocking inside the network is very small as capacity is guaranteed through long-term provisioning. However, if one would like to make a conference call or video call, an advanced reservation may be necessary. 5.2 Int-serv Over Diff-serv Alternatively, differentiated services and Integrated Services may be tightly coupled by means of a hierarchical bandwidth allocation scheme. Differentiated services may be used to allocate bandwidth in bulk quantity to individual customer networks. Each customer network may use Integrated Services mechanisms to allocate the bandwidth among individual hosts and application flows. This model works particularly well for VPNs when a corporation can purchase "virtual pipes" across the Internet and then run Integrated Services over the over-laid virtual network. Examples of this model include [E2EQOS]. 5.3 Aggregated Int-serv State In this approach, Integrated Services flow signaling state (e.g., RSVP) is aggregated between border nodes within a cooperating network region. Hop-by-hop signaling messages within this region convey the aggregated state of a number of Integrated Services flows. Packets within an aggregated Integrated Services class are marked by a special DS codepoint to convey aggregated classification state (the service class, and possibly an in- or out-of-profile indicator). Examples of this model include [GBH97, CLASSY]. Bernet, et. al. Expires: November 1998 [Page 23] INTERNET-DRAFT A Framework for Differentiated Services May 1998 6. Acknowledgements The authors would like to acknowledge the helpful comments and suggestions of the following individuals: Kathleen Nichols, Brian Carpenter, David Black, Konstantinos Dovrolis, Shivkumar Kalyana, Wu-chang Feng, Marty Borden, and Ronald Bonica. 7. References [BB] K. Nichols, V. Jacobson, and L. Zhang, "A Two-bit Differentiated Services Architecture for the Internet", Internet Draft , November 1997. [CLARK] D. Clark and J. Wroclawski, "An Approach to Service Allocation in the Internet", Internet Draft , July 1997. [CLASSY] S. Berson and S. Vincent, "Aggregation of Internet Integrated Services State", Internet Draft , November 1997. [COPS] J. Boyle, R. Cohen, D. Durham, S. Herzog, R. Rajan, and A. Sastry, "COPS (Common Open Policy Service) Protocol", , March 1998. [DSARCH] D. Black, S. Blake, M. Carlson, E. Davies, Z. Wang, and W. Weiss, "An Architecture for Differentiated Services", Internet Draft , May 1998. [DSHEAD] K. Nichols and S. Blake, "Definition of the Differentiated Services Field (DS Byte) in the IPv4 and IPv6 Headers", Internet Draft , May 1998. [DSPREC] F. Baker, S. Brim, T. Li, F. Kastenholz, S. Jagannath, and J. Renwick, "IP Precedence in Differentiated Services Using the Assured Service", Internet Draft , April 1998. [Ellesson] E. Ellesson and S. Blake, "A Proposal for the Format and Semantics of the TOS Byte and Traffic Class Byte in IPv4 and IPv6", Internet Draft , November 1997. [E2EQOS] Y. Bernet, R. Yavatkar, P. Ford, F. Baker, and L. Zhang, "A Framework for End-to-End QoS Combining RSVP/Intserv and Differentiated Services", Internet Draft , March 1998. Bernet, et. al. Expires: November 1998 [Page 24] INTERNET-DRAFT A Framework for Differentiated Services May 1998 [GBH97] R. Guerin, S. Blake, and S. Herzog, "Aggregating RSVP- based QoS Requests", Internet Draft , November 1997. [IntServ] R. Braden, D. Clark, and S. Shenker, "Integrated Services in the Internet Architecture: An Overview", Internet RFC 1633, July 1994. [RMON] S. Waldbusser, "Remote Network Monitoring Management Information Base Version 2 using SMIv2", Internet RFC 2021, January 1997. [RSVP] B. Braden et. al., "Resource ReSerVation Protocol (RSVP) -- Version 1 Functional Specification", Internet RFC 2205, September 1997. [RTFlow] N. Brownlee, C. Mills, and G. Ruth, "Traffic Flow Measurement: Architecture", Internet Draft , Dec. 1997. Authors' Addresses Yoram Bernet Microsoft One Microsoft Way, Redmond, WA 98052 Phone: +1-425-936-9568 E-mail: yoramb@microsoft.com James Binder 3Com Corp. 5400 Bayfront Plaza Santa Clara, CA 95052 Phone: +1-408-326-6051 E-mail: james_binder@3com.com Steven Blake IBM Corporation 800 Park Offices Drive Research Triangle Park, NC 27709 Phone: +1-919-254-2030 E-mail: slblake@raleigh.ibm.com Mark A. Carlson Redcape Software, Inc. 2990 Center Green Court South Boulder, CO 80301 Phone: +1-303-448-0048 x115 E-mail: mac@redcape.com Bernet, et. al. Expires: November 1998 [Page 25] INTERNET-DRAFT A Framework for Differentiated Services May 1998 Elwyn Davies Nortel UK London Road Harlow, Essex CM17 9NA, UK Phone: +44-1279-405498 E-mail: elwynd@nortel.co.uk Borje Ohlman Ericsson Radio Dialoggatan 1 (Kungens Kurva) S-126 25 Stockholm Sweden Phone: +46-8-719 3187 E-mail: Borje.Ohlman@ericsson.com Dinesh C. Verma IBM TJ Watson Research Center PO Box 218 Yorktown Heights, NY 10598 E-mail: dverma@watson.ibm.com Zheng Wang Bell Labs Lucent Tech 101 Crawfords Corner Road Holmdel, NJ 07733 E-mail: zhwang@bell-labs.com Walter Weiss Lucent Technologies 300 Baker Avenue, Suite 100, Concord, MA 01742-2168 E-mail: wweiss@lucent.com Bernet, et. al. Expires: November 1998 [Page 26]