Internet Draft Gerald R. Ash AT&T Labs November 2000 Expires: May 2001 Traffic Engineering & QoS Methods for IP-, ATM-, & TDM-Based Multiservice Networks STATUS OF THIS MEMO: This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. Distribution of this memo is unlimited. ABSTRACT The draft describes, analyzes, and recommends traffic engineering (TE) methods which control a network's response to traffic demands and other stimuli, such as link failures or node failures. These TE methods include: * traffic management through control of routing functions, which include call routing (number/name translation to routing address), connection routing, QoS resource management, routing table management, and dynamic transport routing. * capacity management through control of network design. * TE operational requirements for traffic management and capacity management, including forecasting, performance monitoring, and short-term network adjustment. These TE methods are recommended for application across network types based on established practice and experience. ***************************************************************************** NOTE: A PDF VERSION OF THIS DRAFT (WITH THE FIGURES IS AVAILABLE ON REQUEST ***************************************************************************** Ash [Page MAIN-1] Internet Draft TE & QoS Methods for IP,ATM,TDM-Based Networks 2000 TABLE OF CONTENTS ABSTRACT 1.0 Introduction 1.1 Scope 1.2 Definitions 1.3 Abbreviations 1.4 Traffic Engineering Model 1.5 Traffic Models 1.6 Traffic Management Functions 1.7 Capacity Management Functions 1.8 Traffic Engineering Operational Requirements 1.9 Traffic Engineering Modeling & Analysis 1.10 Conclusions/Recommendations 1.10.1 Conclusions/Recommendations on Call Routing & Connection Routing Methods (ANNEX 2) 1.10.2 Conclusions/Recommendations on QoS Resource Management (ANNEX 3) 1.10.3 Conclusions/Recommendations on Routing Table Management Methods & Requirements (ANNEX 4) 1.10.4 Conclusions/Recommendations on Dynamic Transport Routing Methods (ANNEX 5) 1.10.5 Conclusions/Recommendations on Capacity Management Methods (ANNEX 6) 1.10.6 Conclusions/Recommendations on TE Operational Requirements (ANNEX 7) 1.11 Authors' Addresses 1.12 Copyright Statement ANNEX 1. Bibliography ANNEX 2. Call Routing & Connection Routing Methods 2.1 Introduction 2.2 Call Routing Methods 2.3 Connection (Bearer-Path) Routing Methods 2.4 Hierarchical Fixed Routing (FR) Path Selection 2.5 Time-Dependent Routing (TDR) Path Selection 2.6 State-Dependent Routing (SDR) Path Selection 2.7 Event-Dependent Routing (EDR) Path Selection 2.8 Interdomain Routing 2.9 Modeling of Traffic Engineering Methods 2.9.1 Network Design Comparisons 2.9.2 Network Performance Comparisons 2.9.3 Network Modeling Conclusions 2.10 Conclusions/Recommendations ANNEX 3. QoS Resource Management Methods 3.1 Introduction 3.2 Class-of-Service Identification, Policy-Based Routing Table Derivation, & QoS Resource Management Steps 3.2.1 Class-of-Service Identification 3.2.2 Policy-Based Routing Table Derivation 3.2.3 QoS Resource Management Steps 3.3 Dynamic Bandwidth Allocation, Protection, and Reservation Principles 3.4 Per-Virtual-Network Bandwidth Allocation, Protection, and Reservation 3.4.1 Per-VNET Bandwidth Allocation/Reservation - Meshed Network Case 3.4.2 Per-VNET Bandwidth Allocation/Reservation - Sparse Network Case 3.5 Per-Flow Bandwidth Allocation, Protection, and Reservation 3.5.1 Per-Flow Bandwidth Allocation/Reservation - Meshed Network Case 3.5.2 Per-Flow Bandwidth Allocation/Reservation - Sparse Network Case 3.6 Priority Queuing Ash [Page MAIN-2] Internet Draft TE & QoS Methods for IP,ATM,TDM-Based Networks November 2000 3.7 Other QoS Resource Management Constraints 3.8 Interdomain QoS Resource Management 3.9 Modeling of Traffic Engineering Methods 3.9.1 Performance of Bandwidth Reservation Methods 3.9.2 Multiservice Network Performance: Per-VNET vs. Per-Flow Bandwidth Allocation 3.9.3 Multiservice Network Performance: Multi-Area 2-Level Hierarchical vs. Single-Area Flat Topology 3.9.4 Multiservice Network Performance: Need for MPLS & DiffServ 3.10 Conclusions/Recommendations ANNEX 4. Routing Table Management Methods & Requirements 4.1 Introduction 4.2 Routing Table Management for IP-Based Networks 4.3 Routing Table Management for ATM-Based Networks 4.4 Routing Table Management for TDM-Based Networks 4.5 Signaling and Information Exchange Requirements 4.5.1 Call Routing (Number Translation to Routing Address) Information-Exchange Parameters 4.5.2 Connection Routing Information-Exchange Parameters 4.5.3 QoS Resource Management Information-Exchange Parameters 4.5.4 Routing Table Management Information-Exchange Parameters 4.5.5 Harmonization of Information-Exchange Standards 4.5.6 Open Routing Application Programming Interface (API) 4.6 Examples of Internetwork Routing 4.6.1 Internetwork E Uses a Mixed Path Selection Method 4.6.2 Internetwork E Uses a Single Path Selection Method 4.7 Modeling of Traffic Engineering Methods 4.8 Conclusions/Recommendations ANNEX 5. Dynamic Transport Routing Methods 5.1 Introduction 5.2 Dynamic Transport Routing Principles 5.3 Dynamic Transport Routing Examples 5.4 Modeling of Traffic Engineering Methods 5.4.1 Dynamic Transport Routing Capacity Design 5.4.2 Performance for Network Failures 5.4.3 Performance for General Traffic Overloads 5.4.4 Performance for Unexpected Overloads 5.4.5 Performance for Peak-Day Traffic Loads 5.5 Conclusions/Recommendations ANNEX 6. Capacity Management Methods 6.1 Introduction 6.2 Link Capacity Design Models 6.3 Shortest Path Selection Models 6.4 Multihour Network Design Models 6.4.1 Discrete Event Flow Optimization (DEFO) Models 6.4.2 Traffic Load Flow Optimization (TLFO) Models 6.4.3 Virtual Trunking Flow Optimization (VTFO) Models 6.5 Day-to-day Load Variation Design Models 6.6 Forecast Uncertainty/Reserve Capacity Design Models 6.7 Meshed, Sparse, and Dynamic-Transport Design Models 6.8 Modeling of Traffic Engineering Methods 6.9 Conclusions/Recommendations ANNEX 7. Traffic Engineering Operational Requirements 7.1 Introduction Ash [Page MAIN-3] Internet Draft TE & QoS Methods for IP,ATM,TDM-Based Networks November 2000 7.2 Traffic Management 7.2.1 Real-time Performance Monitoring 7.2.2 Network Control 7.2.3 Work Center Functions 7.2.3.1 Automatic controls 7.2.3.2 Code Controls 7.2.3.3 Reroute Controls 7.2.3.4 Peak-Day Control 7.2.4 Traffic Management on Peak Days 7.2.5 Interfaces to Other Work Centers 7.3 Capacity Management---Forecasting 7.3.1 Load forecasting 7.3.1.1 Configuration Database Functions 7.3.1.2 Load Aggregation, Basing, and Projection Functions 7.3.1.3 Load Adjustment Cycle and View of Business Adjustment Cycle 7.3.2 Network Design 7.3.3 Work Center Functions 7.3.4 Interfaces to Other Work Centers 7.4 Capacity Management---Daily and Weekly Performance Monitoring 7.4.1 Daily Congestion Analysis Functions 7.4.2 Study-week Congestion Analysis Functions 7.4.3 Study-period Congestion Analysis Functions 7.5 Capacity Management---Short-Term Network Adjustment 7.5.1 Network Design Functions 7.5.2 Work Center Functions 7.5.3 Interfaces to Other Work Centers 7.6 Comparison of Off-line (TDR) versus On-line (SDR/EDR) TE Methods 7.7 Conclusions/Recommendations 1.1 Introduction Traffic engineering (TE) is an indispensable network function which controls a network's response to traffic demands and other stimuli, such as network failures. TE encompasses * traffic management through control of routing functions, which include number/name translation to routing address, connection routing, routing table management, QoS resource management, and dynamic transport routing. * capacity management through control of network design. Current and future networks are rapidly evolving to carry a multitude of voice/ISDN services and packet data services on internet protocol (IP), asynchronous transfer mode (ATM), and time division multiplexing (TDM) networks. The long awaited data revolution is occurring, with the extremely rapid growth of data services such as IP-multimedia and frame-relay services. Within these categories of networks and services supported by IP, ATM, and TDM protocols have evolved various TE methods. The TE mechanisms are covered in the draft, and a comparative analysis and performance evaluation of various TE alternatives is presented. Finally, operational requirements for TE implementation are covered. We begin this draft with a general model for TE functions, which include traffic management and capacity management functions responding to traffic demands on the network. We then present a traffic-variations model which these TE functions are responding to. Next we outline traffic management functions which include call routing (number/name translation to routing address), connection or bearer-path routing, QoS resource management, routing table management, and dynamic transport routing. These traffic Ash [Page MAIN-4] Internet Draft TE & QoS Methods for IP,ATM,TDM-Based Networks November 2000 management functions are further developed in ANNEXs 2, 3, 4, and 5. We then outline capacity management functions, which are further developed in ANNEX 6. Finally we briefly summarize TE operational requirements, which are further developed in ANNEX 7. In ANNEX 2, we present models for call routing, which entails number/name translation to a routing address associated with service requests, and also compare various connection (bearer-path) routing methods. In ANNEX 3, we examine QoS resource management methods in detail, and illustrate per-flow versus per-virtual-network (or per-traffic-trunk or per-bandwidth-pipe) resource management and the realization of multiservice integration with priority routing services. In ANNEX 4, we identify and discuss routing table management approaches. This includes a discussion of TE signaling and information exchange requirements needed for interworking across network types, so that the information exchange at the interface is compatible across network types. In ANNEX 5 we describe methods for dynamic transport routing, which is enabled by the capabilities such as optical cross-connect devices, to dynamically rearrange transport network capacity. In ANNEX 6 we describe principles for TE capacity management, and in ANNEX 7 we present TE operational requirements. 1.2 Definitions Alternate Path Routing: a routing technique where multiple paths, rather than just the shortest path, between a source node and a destination node are utilized to route traffic, which is used to distribute load among multiple paths in the network; Autonomous System: a routing domain which has a common administrative authority and consistent internal routing policy. An AS may employ multiple intradomain routing protocols and interfaces to other ASs via a common interdomain routing protocol; Blocking: refers to the denial or non-admission of a call or connection-request, based for example on the lack of available resources on a particular link (e.g., link bandwidth or queuing resources); Call: generic term to describe the establishment, utilization, and release of a connection (bearer path) or data flow; Call Routing: number (or name) translation to routing address(es), perhaps involving use of network servers or intelligent network (IN) databases for service processing; Class of Service characteristics of a service such as described by service identity, virtual network, link capability requirements, QoS & traffic threshold parameters; Connection: bearer path, label switched path, virtual circuit, and/or virtual path established by call routing and connection routing; Ash [Page MAIN-5] Internet Draft TE & QoS Methods for IP,ATM,TDM-Based Networks November 2000 Connection Admission a process by which it is determined whether a link or a node has sufficient resources Control (CAC) to satisfy the QoS required for a connection or flow. CAC is typically applied by each node in the path of a connection or flow during set-up to check local resource availability; Connection Routing: connection establishment through selection of one path from path choices governed by the routing table; Crankback a technique where a connection or flow setup is backtracked along the call/connection/flow path up to the first node that can determine an alternative path to the destination node; Flow: bearer traffic associated with a given connection or connectionless stream having the same originating node, destination node, class of service, and session identification; Integrated Services: a model which allows for integration of services with various QoS classes, such as key-priority, normal-priority, & best-effort priority services; Link: a bandwidth transmission medium between nodes that is engineered as a unit; Logical Transport Link: a bandwidth transmission medium established over physical transport links and switched, for example, through optical cross-connect devices; Destination Node: terminating node within a given network; Node: a network element (switch, router, exchange) providing switching and routing capabilities, or an aggregation of such network elements representing a network; Multiservice Network a network in which various classes of service share the transmission, switching, queuing, management, and other resources of the network; O-D pair: an originating node to destination node pair for a given connection/bandwidth-allocation request; Originating Node: originating node within a given network; Path: a concatenation of links providing a connection/bandwidth-allocation between an O-D pair; Physical Transport Link:a bandwidth transmission medium established over a physical path such as on a fiber transmission link; Policy-Based Routing network function which involves the application of rules applied to input parameters to derive a routing table and its associated parameters; QoS a set of service requirements to be met by the network while transporting a connection or flow; QoS Resource network functions which include class-of-service identification, routing table derivation, Management connection admission, bandwidth allocation, bandwidth protection, bandwidth reservation, priority routing, and priority Ash [Page MAIN-6] Internet Draft TE & QoS Methods for IP,ATM,TDM-Based Networks November 2000 queuing; QoS Routing see QoS Resource Management; Route: a set of paths connecting the same originating node-destination node pair; Routing Table: describes the path choices and selection rules to select one path out of the route for a connection/bandwidth-allocation request; Traffic Engineering encompasses traffic management, capacity management, traffic measurement and modeling, network modeling, and performance analysis; Traffic Engineering network functions which support traffic engineering and include call routing, Methods connection routing, QoS resource management, routing table management, and capacity management; Traffic Stream: a class of connection requests with the same traffic characteristics; Traffic Trunk: an aggregation of traffic flows of the same class which are routed on the same path Via node: an intermediate node in a path within a given network; 1.3 Abbreviations AAR Automatic Alternate Routing ABR Available Bit Rate ADR Address AESA ATM End System Address AFI Authority and Format Identifier AINI ATM Inter-Network Interface ALB Available Link Bandwidth ARR Automatic Rerouting AS Autonomous System ATM Asynchronous Transfer Mode B Busy BBP Bandwidth Broker Processor BGP Border Gateway Protocol BICC Bearer Independent Call Control B-ISDN Broadband Integrated Services Digital Network BNA Bandwidth Not Available BW Bandwidth BWIP Bandwidth in Progress BWOF Bandwidth Offered BWOV Bandwidth Overflow BWPC Bandwidth Peg Count CAC Call (or Connection) Admission Control CBK Crankback CBR Constant Bit Rate CCS Common Channel Signaling CIC Call Identification Code CRLDP Constraint-Based Routing Label Distribution Protocol CRLSP Constraint-Based Routing Label Switched Path DADR Distributed Adaptive Dynamic Routing DAR Dynamic Alternate Routing DCC Data Country Code DCR Dynamically Controlled Routing Ash [Page MAIN-7] Internet Draft TE & QoS Methods for IP,ATM,TDM-Based Networks November 2000 DIFFSERV Differentiated Services DN Destination Node DNHR Dynamic Nonhierarchical Routing DoS Depth-of-Search DSP Domain Specific Part DTL Designated Transit List EDR Event Dependent Routing ER Explicit Route FR Fixed Routing GCAC Generic Call Admission Control GOS Grade of Service HL Heavily Loaded IAM Initial Address Message ICD International Code Designator IDI Initial Domain Identifier IDP Initial Domain Part IE Information Element IETF Internet Engineering Task Force II Information Interchange ILBW Idle Link Bandwidth INRA International Network Routing Address IP Internet Protocol IPDC Internet Protocol Device Control LBL Link Blocking Level LC Link capability LDP Label Distribution Protocol LL Lightly Loaded LLR Least Loaded Routing LSA Link State Advertisement LSP Label Switched Path MEGACO Media Gateway Control MOD Modify MPLS Multiprotocol Label Switching NANP North American Numbering Plan N-ISDN Narrowband Integrated Services Digital Network NSAP Network Service Access Point ODR Optimized Dynamic Routing ON Originating Node OSPF Open Shortest Route First PAR Parameters PNNI Private Network-to-Network Interface PSTN Public Switched Telephone Network PTSE PNNI Topology State Elements QoS Quality of Service R Reserved RQE Routing Query Element RSE Routing State Element RRE Routing Recommendation Element RSVP Resource Reservation Protocol RTNR Real-Time Network Routing SCP Service Control Point SDR State-Dependent Routing SI Service Identity SIP Session Initiation Protocol SS7 Signaling System 7 STR State- and Time-Dependent Routing SVC Switched Virtual Circuit SVP Switched Virtual Path Ash [Page MAIN-8] Internet Draft TE & QoS Methods for IP,ATM,TDM-Based Networks November 2000 TBW Total Bandwidth TBWIP Total Bandwidth In Progress TDR Time-Dependent Routing TIPHON Telecommunications and Internet Protocol Harmonization Over Networks TLV Type/Length/Value ToS Type of Service TR Trunk Reservation TRAF Traffic TSE Topology State Element UBR Unassigned Bit Rate UNI User-Network Interface VBR Variable Bit Rate VC Virtual Circuit VCI Virtual Circuit Identifier VN Via Node VNET Virtual Network VPI Virtual Path Identifier WIN Worldwide Intelligent Network (Routing) 1.4 Traffic Engineering Model Figure 1.1 illustrates a model for network traffic engineering. The central box represents the network, which can have various architectures and configurations, and the routing tables used within the network. Network configurations could include metropolitan area networks, national intercity networks, and global international networks, which support both hierarchical and nonhierarchical structures and combinations of the two. Routing tables describe the path choices from an originating node to a terminating node, for a connection request for a particular service. Hierarchical and nonhierarchical traffic routing tables are possible, as are fixed routing tables and dynamic routing tables. Routing tables are used for a multiplicity of traffic and transport services on the telecommunications network. Figure 1.1 Traffic Engineering Model Terminology used in the draft, as illustrated in Figure 1.2, is that a link is a transmission medium (logical or physical) which connects two nodes, a path is a sequence of links connecting an origin and destination node, and a route is the set of different paths between the origin and destination that a call might be routed on within a particular routing discipline. Here a call is a generic term used to describe the establishment, utilization, and release of a connection, or data flow. In this context a call can refer to a voice call established perhaps using the SS7 signaling protocol, or to a web-based data flow session, established perhaps by the HTTP and associated IP-based protocols. Various implementations of routing tables are discussed in ANNEX 2. Figure 1.2 Terminology Traffic engineering functions include traffic management, capacity management, and network planning. Traffic management ensures that network performance is maximized under all conditions including load shifts and failures. Capacity management ensures that the network is designed and provisioned to meet performance objectives for network demands at minimum cost. Network planning ensures that node and transport capacity is planned and deployed in advance of forecasted traffic growth. Figure 1.1 illustrates Ash [Page MAIN-9] Internet Draft TE & QoS Methods for IP,ATM,TDM-Based Networks November 2000 traffic management, capacity management, and network planning as three interacting feedback loops around the network. The input driving the network ("system") is a noisy traffic load ("signal"), consisting of predictable average demand components added to unknown forecast error and load variation components. The load variation components have different time constants ranging from instantaneous variations, hour-to-hour variations, day-to-day variations, and week-to-week or seasonal variations. Accordingly, the time constants of the feedback controls are matched to the load variations, and function to regulate the service provided by the network through capacity and routing adjustments. Traffic management functions include a) call routing, which entails number/name translation to routing address, b) connection or bearer-path routing methods, c) QoS resource management, d) routing table management, and e) dynamic transport routing. These functions can be a) decentralized and distributed to the network nodes, b) centralized and allocated to a centralized controller such as a bandwidth broker, or c) performed by a hybrid combination of these approaches. Capacity management plans, schedules, and provisions needed capacity over a time horizon of several months to one year or more. Under exceptional circumstances, capacity can be added on a shorter-term basis, perhaps one to several weeks, to alleviate service problems. Network design embedded in capacity management encompasses both routing design and capacity design. Routing design takes account of the capacity provided by capacity management, and on a weekly or possibly real-time basis adjusts routing tables as necessary to correct service problems. The updated routing tables are provisioned (configured) in the switching systems either directly or via an automated routing update system. Network planning includes node planning and transport planning, operates over a multiyear forecast interval, and drives network capacity expansion over a multiyear period based on network forecasts. The scope of the TE methods includes the establishment of connections for narrowband, wideband, and broadband multimedia services within multiservice networks and between multiservice networks. Here a multiservice network refers to one in which various classes of service share the transmission, switching, management, and other resources of the network. These classes of services can include constant bit rate (CBR), variable bit rate (VBR), unassigned bit rate (UBR), and available bit rate (ABR) traffic classes. There are quantitative performance requirements that the various classes of service normally are required to meet, such as end-to-end blocking, delay, and/or delay-jitter objectives. These objectives are achieved through a combination of traffic management and capacity management. Figure 1.3 illustrates the functionality for setting up a connection from an originating node in one network to a destination node in another network, using one or more routing methods across networks of various types. The Figure illustrates a multimedia connection between two PCs which carries traffic for a combination of voice, video, and image applications. For this purpose a logical point-to-point connection is established from the PC served by node a1 to the PC served by node c2. The connection could be a CBR ISDN connection across TDM-based network A and ATM-based network C, or it might be a VBR connection via IP-based network B. Gateway nodes a3, b1, b4, and c1 provide the interworking capabilities between the TDM-, ATM-, and IP-based networks. The actual multimedia connection might be routed, for example, on a path consisting of nodes a1-a2-a3-b1-b4-c1-c2, or possibly on a different path through different gateway nodes. Ash [Page MAIN-10] Internet Draft TE & QoS Methods for IP,ATM,TDM-Based Networks November 2000 Figure 1.3 Example of Multimedia Connection across TDM-, ATM-, and IP-Based Networks We now briefly describe the traffic model, the traffic management functions, the capacity management functions, and the TE operational requirements, which are further developed in ANNEXs 2-7 of the draft. 1.5 Traffic Models In this section we discuss load variation models which drive traffic engineering functions, that is traffic management, capacity management, and network planning. Table 1.1 summarizes examples of models that could be used to represent the different traffic variations under consideration. Traffic models for both voice and data traffic need to be reflected. Work has been done on measurement and characterization of data traffic, such as web-based traffic [FGLRRT00, FGHW99, LTWW94]. Some of the analysis suggests that web-based traffic can be self-similar, or fractal, with very large variability and extremely long tails of the associated traffic distributions. Characterization studies of such data traffic have investigated various traditional models, such as the Markov modulated Poisson Process (MMPP), in which it is shown that MMPP with two parameters can suitably capture the essential nature of the data traffic [H99, BCHLL99]. Modeling work has been done to investigate the causes of the extreme variability of web-based traffic. In [HM00], the congestion-control mechanisms for web-based traffic, such as window flow control for transport-control-protocol (TCP) traffic appear to be at the root cause of its extreme variability. [HM00] suggests that the regular flow control dynamics are more useful to model than the self-similar traffic itself. Considerable work has been done on modeling of broadband and other data traffic, in which two-parameter models that capture the mean and burstiness of the traffic have proven to be quite adequate. See [E.716] for a good reference on this. Much work has also been done on measurement and characterization of voice traffic, and two-parameter models reflecting mean and variance (the ratio of the variance to the mean is sometimes called the peakedness parameter) of traffic have proven to be accurate models. Here we reflect initial, two-parameter, multiservice traffic models, which are manageable from a modeling and analysis aspect and which attempt to capture essential aspects of data and voice traffic variability for purposes of traffic engineering and QoS methods. Table 1.1 Traffic Models for Load Variations For instantaneous traffic load variations, the load is typically modeled as a stationary random process over a given period (normally within each hourly period) characterized by a fixed mean and variance. From hour to hour, the mean traffic loads are modeled as changing deterministically; for example, according to their 20-day average values. From day to day, for a fixed hour, the mean load can be modeled, for example, as a random variable having a gamma distribution with a mean equal to the 20-day average load. From week to week, the load variation is modeled as a random process in the network design procedure. The random component of the realized week-to-week load is the forecast error, which is equal to the forecast load minus the realized load. Forecast error is accounted for in short-term capacity management. Ash [Page MAIN-11] Internet Draft TE & QoS Methods for IP,ATM,TDM-Based Networks November 2000 In traffic management, traffic load variations such as instantaneous variations, hour-to-hour variations, day-to-day traffic variations, and week-to-week variations are responded to in traffic management by appropriately controlling number translation/routing, path selection, routing table management, and/or QoS resource management. Traffic management provides monitoring of network performance through collection and display of traffic and performance data, and allows traffic management controls, such as destination-address per-connection blocking, per-connection gapping, routing table modification, and path selection/reroute controls, to be inserted when circumstances warrant. For example, a focused overload might lead to application of connection gapping controls in which a connection request to a particular destination address or set of addresses is admitted only once every x seconds, and connections arriving after an accepted call are rejected for the next x seconds. In that way call gapping throttles the calls and prevents overloading the network to a particular focal point. Routing table modification and reroute control are illustrated in ANNEXs 2, 3, 5, and 7. Capacity management must provide sufficient capacity to carry the expected traffic variations so as to meet end-to-end blocking/delay objective levels. Here the term blocking refers to the denial or non-admission of a call or connection request, based for example on the lack of available resources on a particular link (e.g., link bandwidth or queuing resources). Traffic load variations lead in direct measure to capacity increments and can be categorized as (1) minute-to-minute instantaneous variations and associated busy-hour traffic load capacity, (2) hour-to-hour variations and associated multihour capacity, (3) day-to-day variations and associated day-to-day capacity, and (4) week-to-week variations and associated reserve capacity. Design methods within the capacity management procedure account for the mean and variance of the within-the-hour variations of the offered and overflow loads. For example, classical methods [e.g., Wil56] are used to size links for these two parameters of load. Multihour dynamic route design accounts for the hour-to-hour variations of the load and, hour-to-hour capacity can vary from zero to 20 percent or more of network capacity. Hour-to-hour capacity can be reduced by multihour dynamic routing design models such as the discrete event flow optimization, traffic load flow optimization, and virtual trunking flow optimization models described in ANNEX 6. As noted in Table 1.1, capacity management excludes non-recurring traffic such as caused by overloads (focused or general overloads), or failures. This process is described further in ANNEX 7. It is known that some daily variations are systematic (for example, Monday morning business traffic is usually higher than Friday morning); however, in some day-to-day variation models these systematic changes are ignored and lumped into the stochastic model. For instance, the traffic load between Los Angeles and New Brunswick is very similar from one day to the next, but the exact calling levels differ for any given day. This load variation can be characterized in network design by a stochastic model for the daily variation, which results in additional capacity called day-to-day capacity. Day-to-day capacity is needed to meet the average blocking/delay objective when the load varies according to the stochastic model. Day-to-day capacity is nonzero due to the nonlinearities in link blocking and/or link queuing delay levels as a function of load. When the load on a link fluctuates about a mean value, because of day-to-day variation, the mean blocking/delay is higher than the blocking/delay produced by the mean load. Therefore, additional capacity is provided to maintain the blocking/delay probability grade-of-service objective in the presence of day-to-day load variation. Ash [Page MAIN-12] Internet Draft TE & QoS Methods for IP,ATM,TDM-Based Networks November 2000 Typical day-to-day capacity required is 4--7 percent of the network cost for medium to high day-to-day variations, respectively. Reserve capacity, like day-to-day capacity, comes about because load uncertainties---in this case forecast errors---tend to cause capacity buildup in excess of the network design that exactly matches the forecast loads. Reluctance to disconnect and rearrange link and transport capacity contributes to this reserve capacity buildup. At a minimum, the currently measured mean load is used to adjust routing and capacity design, as needed. In addition, the forecast-error variance component in used in some models to build in so-called protective capacity. Reserve or protective capacity can provide a cushion against overloads and failures, and generally benefits network performance. However, provision for reserve capacity is not usually built into the capacity management design process, but arises because of sound administrative procedures. These procedures attempt to minimize total cost, including both network capital costs and operations costs. Studies have shown that reserve capacity in some networks to be in the range of 15 to 25 percent or more of network cost [FHH79]. This is further described in ANNEXs 5 and 6. 1.6 Traffic Management Functions In ANNEXs 2-5, traffic management functions are discussed: a) Call Routing Methods (ANNEX 2). Call routing involves the translation of a number or name to a routing address. We describe how number (or name) translation should result in the E.164 ATM end-system addresses (AESA), network routing addresses (NRAs), and/or IP addresses. These addresses are used for routing purposes and therefore must be carried in the connection-setup information element (IE). b) Connection/Bearer-Path Routing Methods (ANNEX 2). Connection or bearer-path routing involves the selection of a path from the originating node to the destination node in a network. We discuss bearer-path selection methods, which are categorized into the following four types: fixed routing (FR), time-dependent routing (TDR), state-dependent routing (SDR), and event-dependent routing (EDR). These methods are associated with routing tables, which consist of a route and rules to select one path from the route for a given connection or bandwidth-allocation request. c) QoS Resource Management Methods (ANNEX 3). QoS resource management functions include class-of-service derivation, policy-based routing table derivation, connection admission, bandwidth allocation, bandwidth protection, bandwidth reservation, priority routing, priority queuing, and other related resource management functions. d) Routing Table Management Methods (ANNEX 4). Routing table management information, such as topology update, status information, or routing recommendations, is used for purposes of applying the routing table design rules for determining path choices in the routing table. This information is exchanged between one node and another node, such as between the ON and DN, for example, or between a node and a network element such as a bandwidth-broker processor (BBP). This information is used to generate the routing table, and then the routing table is used to determine the path choices used in the selection of a path. e) Dynamic Transport Routing Methods (ANNEX 5). Dynamic transport routing combines with dynamic traffic routing to shift transport bandwidth among node pairs and services through use of flexible transport switching Ash [Page MAIN-13] Internet Draft TE & QoS Methods for IP,ATM,TDM-Based Networks November 2000 technology, such as optical cross-connects (OXCs). Dynamic transport routing offers advantages of simplicity of design and robustness to load variations and network failures, and can provide automatic link provisioning, diverse link routing, and rapid link restoration for improved transport capacity utilization and performance under stress. OXCs can reconfigure logical transport capacity on demand, such as for peak day traffic, weekly redesign of link capacity, or emergency restoration of capacity under node or transport failure. MPLS control capabilities are proposed for the setup of layer 2 logical transport links through OXCs [ARDC99]. 1.7 Capacity Management Functions In ANNEX 6, we discuss capacity management methods, as follows: a) Link Capacity Design Models. These models find the optimum tradeoff between traffic carried on a shortest network path (perhaps a direct link) versus traffic carried on alternate (longer, less efficient) network paths. b) Shortest Path Selection Models. These models enable the determination of shortest paths in order to provide a more efficient and flexible routing plan. c) Multihour Network Design Models. Three models are described including i) discrete event flow optimization (DEFO) models, ii) traffic load flow optimization (TLFO) models, and iii) virtual trunking flow optimization (VTFO) models. d) Day-to-day Load Variation Design Models. These models describe techniques for handling day-to-day variations in capacity design. e) Forecast Uncertainty/Reserve Capacity Design Models. These models describe the means for accounting for errors in projecting design traffic loads in the capacity design of the network. 1.8 Traffic Engineering Operational Requirements In ANNEX 7, we discuss traffic engineering operational requirements, as follows: a) Traffic Management. We discuss requirements for real-time performance monitoring, network control, and work center functions. The latter includes automatic controls, manual controls, code controls, cancel controls, reroute controls, peak-day controls, traffic management on peak days, and interfaces to other work centers. b) Capacity Management - Forecasting. We discuss requirements for load forecasting, including configuration database functions, load aggregation, basing, and projection functions, and load adjustment cycle and view of business adjustment cycle. We also discuss network design, work center functions, and interfaces to other work centers. c) Capacity Management - Daily and Weekly Performance Monitoring. We discuss requirements for daily congestion analysis, study-week congestion analysis, and study-period congestion analysis. d) Capacity Management - Short-Term Network Adjustment. We discuss requirements for network design, work center functions, and interfaces to other work centers. e) Comparison of off-line (TDR) versus on-line (SDR/EDR) TE methods. We contrast off-line TE methods, such as in a TDR-based network, with on-line TE methods, such as in an SDR- or EDR-based network. 1.9 Traffic Engineering Modeling & Analysis In ANNEXs 2-6 we use network models to illustrate the traffic engineering methods developed in the ANNEXs. The details of the models are presented in each ANNEX in accordance with the TE functions being illustrated. Ash [Page MAIN-14] Internet Draft TE & QoS Methods for IP,ATM,TDM-Based Networks November 2000 In the draft, a full-scale 135-node national network node model is used together with a multiservice traffic demand model to study various TE scenarios and tradeoffs. Typical voice/ISDN traffic loads are used to model the various network alternatives. These voice/ISDN loads are further segmented in the model into eight constant-bit-rate (CBR) virtual networks (VNETs), including business voice, consumer voice, international voice in and out, key-service voice, normal and key-service 64-kbps ISDN data, and 384-kbps ISDN data. The data services traffic model incorporates typical traffic load patterns and comprises three additional VNET load patterns. These include a) a variable bit rate real-time (VBR-RT) VNET, representing services such as IP-telephony and compressed voice, b) a variable bit rate non-real-time (VBR-NRT) VNET, representing services such as WWW multimedia and credit card check, and c) an unassigned bit rate (UBR) VNET, representing best-effort services such as email, voice mail, and file transfer multimedia applications. The cost model represents typical switching and transport costs, and illustrates the economies-of-scale for costs projected for high capacity network elements in the future. Many different alternatives and tradeoffs are examined in the models, including: 1. centralized routing table control versus distributed control 2. off-line, pre-planned (e.g.,TDR-based) routing table control versus on-line routing table control (e.g., SDR- or EDR-based) 3. per-flow traffic management versus per-virtual-network (or per-traffic-trunk or per-bandwidth-pipe) traffic management 4. sparse logical topology versus meshed logical topology 5. FR versus TDR versus SDR versus EDR path selection 6. multilink path selection versus two-link path selection 7. path selection using local status information versus global status information 8. global status dissemination alternatives including status flooding, distributed query for status, and centralized status in a bandwidth-broker processor Table 1.2 summarizes brief comparisons and observations, based on the modeling, in each of the above alternatives and tradeoffs (further details are contained in ANNEXs 2-6). Table 1.2 Tradeoff Categories and Comparisons (Based on Modeling in ANNEXs 2-6) 1.10 Conclusions/Recommendations Following is a summary of the main conclusions/recommendations reached in the draft. 1.10.1 Conclusions/Recommendations on Call Routing & Connections Routing Methods (ANNEX 2) * In all cases of the TE methods being applied, network performance is always better and usually substantially better than when no TE methods are applied * Sparse-topology multilink-routing networks provide better overall performance under overload than meshed-topology networks, but performance under failure may favor the 2-link STT-EDR/DC-SDR meshed-topology options Ash [Page MAIN-15] Internet Draft TE & QoS Methods for IP,ATM,TDM-Based Networks November 2000 with more alternate routing choices. * State information as used by the SDR options provides essentially equivalent performance to the EDR options. * Various path selection methods can interwork with each other in the same network, as required for multi-vendor network operation. * EDR TE methods are shown to an important class of TE algorithms. EDR TE methods are distinct from the TDR and SDR TE methods in how the paths (e.g., MPLS label switched paths, or LSPs) are selected. In the SDR TE case, the available link bandwidth (based on LSA flooding of ALB information) is typically used to compute the path. In the EDR TE case, the ALB information is not needed to compute the path, therefore the ALB flooding does not need to take place (reducing the overhead). * EDR TE algorithms are adaptive and distributed in nature and typically use learning models to find good paths for TE in a network. For example, in a success-to-the-top (STT) EDR TE method, if the LSR-A to LSR-B bandwidth needs to be modified, say increased by delta-BW, the primary LSP-p is tried first. If delta-BW is not available on one or more links of LSP-p, then the currently successful LSP-s is tried next. If delta-BW is not available on one or more links of LSP-s, then a new LSP is searched by trying additional candidate paths until a new successful LSP-n is found or the candidate paths are exhausted. LSP-n is then marked as the currently successful path for the next time bandwidth needs to be modified. The performance of distributed EDR TE methods is shown to be equal to or better than SDR methods, centralized or distributed. * While SDR TE models typically use available-link-bandwidth (ALB) flooding for TE path selection, EDR TE methods do not require ALB flooding. Rather, EDR TE methods typically search out capacity by learning models, as in the STT method above. ALB flooding can be very resource intensive, since it requires link bandwidth to carry LSAs, processor capacity to process LSAs, and the overhead can limit area/autonomous system (AS) size. Modeling results show EDR TE methods can lead to a large reduction in ALB flooding overhead without loss of network throughput performance [as shown in ANNEX 4]. * interdomain routing methods can be considered to extend the intradomain call routing and connection routing concepts, such as flexible path selection and per-class-of-service bandwidth selection, to routing between network domains. 1.10.2 Conclusions/Recommendations on QoS Resource Management Methods (ANNEX 3) * Bandwidth reservation is critical to the stable and efficient performance of TE methods in a network, and to ensure the proper operation of multiservice bandwidth allocation, protection, and priority treatment. * Per-VNET bandwidth allocation is essentially equivalent to per-flow bandwidth allocation in network performance and efficiency. Because of the much lower routing table management overhead requirements, as discussed and modeled in ANNEX 4, per-VNET bandwidth allocation is preferred to per-flow allocation. * Single-area flat topologies exhibit better network performance and, Ash [Page MAIN-16] Internet Draft TE & QoS Methods for IP,ATM,TDM-Based Networks November 2000 as discussed and modeled in ANNEX 6, greater design efficiencies in comparison with multi-area hierarchical topologies. As illustrated in ANNEX 4, larger administrative areas can be achieved through use of EDR-based TE methods as compared to SDR-based TE methods. * QoS resource management is shown to be effective in achieving key service, normal service, and best effort service differentiation. * Both MPLS QoS and bandwidth management and DiffServ priority queuing management are important for ensuring that multiservice network performance objectives are met under a range of network conditions. Both mechanisms operate together to ensure QoS resource allocation mechanisms (bandwidth allocation, protection, and priority queuing) are achieved. 1.10.3 Conclusions/Recommendations on Routing Table Management Methods & Requirements (ANNEX 4) * Because of the much lower routing table management overhead requirements, per-VNET bandwidth allocation is preferred to per-flow allocation. Per-VNET bandwidth allocation is essentially equivalent to per-flow bandwidth allocation in network performance and efficiency, as discussed in ANNEX 3. * Modeling results show EDR TE methods can lead to a large reduction in ALB flooding overhead without loss of network throughput performance. While SDR TE models typically use ALB flooding for TE path selection, EDR TE methods do not require ALB flooding. Rather, EDR TE methods typically search out capacity by learning models, as in the STT method. ALB flooding can be very resource intensive, since it requires link bandwidth to carry LSAs, processor capacity to process LSAs, and the overhead can limit area/autonomous system (AS) size. * Because of lower routing table management overhead requirements, larger administrative areas can be achieved through use of EDR-based TE methods as compared to SDR-based TE methods. This can help achieve single-area flat topologies which, as discussed in ANNEX 3, exhibit better network performance and, as discussed in ANNEX 6, greater design efficiencies in comparison with multi-area hierarchical topologies. 1.10.4 Conclusions/Recommendations on Dynamic Transport Routing Methods (ANNEX 5) * Dynamic transport routing network design improves network performance in comparison with fixed transport routing for all network conditions simulated, which include abnormal and unpredictable traffic load patterns. * The ability of the dynamic transport routing network design to enhance network performance under failure arises from automatic inter-backbone-router and access logical-transport-link diversity in combination with the dynamic traffic routing and transport restoration of logical transport links. * Higher network throughput and enhanced revenue should accrue from deployment of a dynamic transport routing network, and at the same time capital savings should result, as discussed in ANNEX 6. Ash [Page MAIN-17] Internet Draft TE & QoS Methods for IP,ATM,TDM-Based Networks November 2000 1.10.5 Conclusions/Recommendations on Capacity Management Methods (ANNEX 6) * Discrete event flow optimization (DEFO) design models are shown to be able to capture very complex routing behavior through the equivalent of a simulation model provided in software in the routing design module. By this means, very complex routing networks have been designed by the model, which include all of the routing methods discussed in ANNEX 2 (FR, TDR, SDR, and EDR methods) and the multiservice QoS resource allocation models discussed in ANNEX 3. * Capital cost advantages may be attributed to the sparse topology options, such as the multilink STT-EDR/DC-SDR/DP-SDR options, but may not be significant compared to operational costs, and are subject to the particular switching and transport cost assumptions. Operational issues are further detailed in ANNEX 7. * Voice and data integration can provide capital cost advantages, but may be more important in achieving operational simplicity and cost reduction. * Single-area flat topologies exhibit greater design efficiencies and, as discussed and modeled in ANNEX 3, better network performance in comparison with multi-area hierarchical topologies. As illustrated in ANNEX 4, larger administrative areas can be achieved through use of EDR-based TE methods as compared to SDR-based TE methods. * Dynamic transport routing networks achieve capital savings by concentrating capacity on fewer, high-capacity physical fiber links and, as discussed in ANNEX 5, achieve higher network throughput and enhanced revenue by their ability to flexibly allocate bandwidth on the logical transport links serving the access and inter-node traffic. * If IP-telephony takes hold and a significant portion of voice calls use voice compression technology, this could lead to more efficient networks. 1.10.6 Conclusions/Recommendations on TE Operational Requirements (ANNEX 7) * Monitoring of traffic and performance data is required for traffic management, capacity forecasting, daily and weekly performance monitoring, and short-term network adjustment. * Traffic management is required which provides monitoring of network performance through collection and display of real-time traffic and performance data and allows traffic management controls such as code blocks, connection request gapping, and reroute controls to be inserted when circumstances warrant. * Capacity management is required which includes capacity forecasting, daily and weekly performance monitoring, and short-term network adjustment. * Forecasting is required which operates over a multiyear forecast interval and drives network capacity expansion. * Daily and weekly performance monitoring is required to identify any service problems in the network. If service problems are detected, short-term network adjustment can include routing table updates and, if necessary, short-term capacity additions to alleviate service problems. Ash [Page MAIN-18] Internet Draft TE & QoS Methods for IP,ATM,TDM-Based Networks November 2000 Updated routing tables are sent to the switching systems either directly or via an automated routing update system. * Short-term capacity additions are required, but as an exception, whereas most capacity changes are normally forecasted, planned, scheduled, and managed over a period of months or a year or more. * Network design is required, which is embedded in capacity management and includes routing design and capacity design. * Network planning is required, which includes longer-term node planning and transport network planning, and which operates over a horizon of months to years to plan and implement new node and transport capacity. 1.11 Authors' Addresses Gerald R. Ash AT&T Labs Room MT D5-2A01 200 Laurel Avenue Middletown, NJ 07748 Phone: 732-420-4578 Fax: 732-368-8659 Email: gash@att.com 1.12 Full Copyright Statement Copyright (C) The Internet Society (1998). All Rights Reserved. This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English. The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns. This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Ash [Page MAIN-19] Internet Draft TE & QoS Methods for IP,ATM,TDM-Based Networks November 2000 ANNEX 1. Bibliography [A98] Ash, G. R., Dynamic Routing in Telecommunications Networks, McGraw-Hill, 1998. [A99a] Ash, G. R., Routing Guidelines for Efficient Routing Methods, IETF Draft draft-ash-itu-sg2-routing-guidelines-00.txt, October 1999. [A99b] Awduche, D. O., MPLS and Traffic Engineering in IP Networks, IEEE Communications Magazine, December 1999. [A99c] Awduche, D. O., MPLS and Traffic Engineering in IP Networks, IEEE Communications Magazine, December 1999. [A99d] Apostolopoulos, G., On the Cost and Performance Trade-offs of Quality of Service Routing", Ph.D. thesis, University of Maryland, 1999. [A00] Armitage, G., Quality of Service in IP Networks: Foundations for a Multi-Service Internet, Macmillan, April 2000. [AAFJLLS00] Ash, G. R., Ashwood-Smith, P., Fedyk, D., Jamoussi, B., Lee, Y., Li, L., Skalecki, D., LSP Modification Using CRLDP, draft-ietf-mpls-crlsp-modify-00.txt, February 2000. [ABGLSS00] Awduche, D., Berger, L., Gan, D., Li, T., Swallow, G., Srinivasan, V., RSVP-TE: Extension to RSVP for LSP Tunnels, IETF Draft draft-ietf-mpls-rsvp-lsp-tunnel-05.txt>, February 2000. [ACEWX00] Awduche, D. O., Chiu, A., Elwalid, A., Widjaja, I., Xiao, X., A Framework for Internet Traffic Engineering, draft-ietf-tewg-framework-00.txt, May 2000. [ACFM99] Ash, G. R., Chen, J., Fishman, S. D., Maunder, A., Routing Evolution in Multiservice Integrated Voice/Data Networks, International Teletraffic Congress ITC-16, Edinburgh, Scotland, June 1999. [ADFFT98] Anderson, L., Doolan, P., Feldman, N., Fredette, A., Thomas, B., LDP Specification, IETF Draft, draft-ietf-mpls-ldp-00.txt, August 1998. [AGK99] Apostolopoulos, G., Guerin, R., Kamat, S., Implementation and Performance Measurements of QoS Routing Extensions toOSPF, Proceedings of INFOCOM '99, April 1999. [AGKOT99] Apostolopoulos, G., Guerin, R., Kamat, S., Orda, A., Tripathi, S. K., Intra-Domain QoS Routing in IP Networks: A Feasibility and Cost/Benefit Analysis, IEEE Network Magazine, 1999. [AJF00] Ashwood-Smith, P., Jamoussi, B., Fedyk, D., Improving Topology Data Base Accuracy with LSP Feedback via CR-LDP, IETF Draft draft-ietf-mpls-te-feed-00.txt, January 2000. [Aki83] Akinpelu, J. M., "The Overload Performance of Engineered Networks with Nonhierarchical and Hierarchical Routing," Proceedings of the Tenth International Teletraffic Congress, Montreal, Canada, June 1983. [Aki84] Akinpelu, J. M., "The Overload Performance of Engineered Networks with Nonhierarchical and Hierarchical Routing," Bell System Technical Ash [Page ANNEX1-1] Internet Draft TE & QoS Methods for IP,ATM,TDM-Based Networks November 2000 Journal, Vol. 63, 1984. [AL99] Ash, G. R., Lee, Y., Routing of Multimedia Connections Across TDM-, ATM-, and IP-Based Networks, IETF Draft, draft-ash-itu-sg2-qos-routing-02.txt, October 1999. [AM98] Ash, G. R., Maunder, A., Routing of Multimedia Connections when Interworking with PSTN, ATM, and IP Networks, AF-98-0927, Nashville TN, December 1998. [AAJL99] Ash, G. R., Aboul-Magd, O. S., Jamoussi, B., Lee, Y., QoS Resource Management in MPLS-Based Networks, IETF Draft, draft-ash-qos-routing-00.txt, Minneapolis MN, March 1999. [AM99] Ash, G. R., Maunder, A., QoS Resource Management in ATM Networks, AF-99-, Rome Italy, April 1999. [ARDC99] Awduche, D. O., Rekhter, Y., Drake, J., Coltun, R., Multiprotocol Lambda Switching: Combined MPLS Traffic Engineering Control with Optical Crossconnects, IETF Draft, draft-awduche-mpls-te-optical-00txt, October 1999. [ATM950013] ATM Forum Technical Committee, B-ISDN Inter Carrier Interface (B-ICI) Specification Version 2.0 (Integrated), af-bici-0013.003, December 1995. [ATM960055] ATM Forum Technical Committee, Private Network-Network Interface Specification Version 1.0 (PNNI 1.0), af-pnni-0055.000, March 1996. [ATM960056] ATM Forum Technical Committee, Traffic Management Specification Version 4.0, af-tm0056.000, April 1996. [ATM960061] ATM Forum Technical Committee, ATM User-Network Interface (UNI) Signaling Specification Version 4.0, af-sig-0061.000, July 1996. [ATM980103] ATM Forum Technical Committee, Specification of the ATM Inter-Network Interface (AINI) (Draft), ATM Forum/BTD-CS-AINI-01.03, July 1998. [ATM990097] ATM Signaling Requirements for IP Differentiated Services and IEEE 802.1D, ATM Forum, Atlanta, GA, February 1999. [ATM000102] ATM Forum Technical Committee, Priority Services Support in ATM Networks, V1.0, ltd-cs-priority-01.02, May 2000. [ATM000146] ATM Forum Technical Committee, Operation of BICC with SIG 4.0/PNNI 1.0/AINI, fb-cs-vmoa-0146.000, May 2000. [ATM000148] Modification of the ATM Traffic Descriptor of an Active Connection, V1.0, fb-cs-0148.000, May 2000. [ATM000213] Noorchashm, M., Ash, G. R., Comely, T., Dianda, R. B., Hartani, R., Proposed Revised Text for the Introduction and Scope Sections of ltd-cs-priority-01.02, May 2000. [B00] Brown, A., ENUM Requirements, IETF Draft Ash [Page ANNEX1-2] Internet Draft TE & QoS Methods for IP,ATM,TDM-Based Networks November 2000 draft-ietf-enum-rqmts-00.txt, June 2000. [B00a] Bernet, Y., The Complementary Rles of RSVP and Differentiated Services in the Full-Service QoS Network, IEEE Communications Magazine, February 2000. [BCHLL99] Bolotin, V., Coombs-Reyes, Heyman, D., Levy, Y., Liu, D., IP Traffic Characterization for Planning and Control, Teletraffic Engineering in a Competitive World, P. Key and D. Smith (Eds.), Elsevier, Amsterdam, 1999. [Bur61] Burke, P. J., "Blocking Probabilities Associated with Directional Reservation," unpublished memorandum, 1961. [C97] Crovella, M. E., Self-Similarity in WWW Traffic: Evidence and Possible Causes, IEEE Transactions on Networking, December 1997. [CED91] Chao, C-W., Eslambolchi, H., Dollard, P., Nguyen, L., Weythman, J., "FASTAR---A Robust System for Fast DS3 Restoration," Proceedings of GLOBECOM 1991, Phoenix, Arizona, December 1991, pp. 1396--1400. [CDFFSV99] Callon, R., Doolan, P., Feldman, N., Fredette, A., Swallow, G., Viswanathan, A., IETF Network Working Group Draft, A Framework for Multiprotocol Label Switching, draft-ietf-mpls-framework-05.txt, September 1999. [CHY00] Chaudhuri, S., Hjalmtysson, G., Yates, J., Control of Lightpaths in an Optical Network, IETF Draft draft-chaudhuri-ip-olxc-control-00.txt, February 2000. [COM 2-39-E] ANNEX, Draft New Recommendation E.ip, Report of Joint Meeting of Questions 1/2 and 10/2, Torino, Italy, July 1998. [D99] Dvorak, C., IP-Related Impacts on End-to-End Transmission Performance, ITU-T Liaison to Study Group 2, Temporary Document TD GEN-22, Geneva Switzerland, May 1999. [Dij59] Dijkstra, E. W., "A Note on Two Problems in Connection with Graphs," Numerical Mathematics, Vol. 1, 1959, pp. 269--271. [DN99] Dianda, R. B., Noorchashm, M., Bandwidth Modification for UNI, PNNI, AINI, and BICI, ATM Forum Technical Working Group, April 1999. [DPW99] Doverspike, R. D., Phillips, S., Westbrook, J. R., Future Transport Network Architectures, IEEE Communications Magazine, August 1999. [DR00] Davie, B. S., Rekhter, Y., MPLS: Technology and Applications, Morgan Kaufmann Publishers, May 2000. [DY00] Doverspike, R., Yates, J., Challenges for MPLS Protocols in the Optical Network Control Plane, submitted for publication. [E.41IP] ITU-T Draft Recommendation, Framework for the Traffic Management of IP-Based Networks, March 2000. [E.106] ITU-T Recommendation, Description of International Emergency Preference System (IEPS). Ash [Page ANNEX1-3] Internet Draft TE & QoS Methods for IP,ATM,TDM-Based Networks November 2000 [E.164] ITU-T Recommendation, The International Telecommunications Numbering Plan. [E.170] ITU-T Recommendation, Traffic Routing. [E.177] ITU-T Recommendation, B-ISDN Routing. [E.191] ITU-T Recommendation, B-ISDN Numbering and Addressing, October 1996. [E.350] ITU-T Recommendation, Dynamic Routing Interworking. [E.351] ITU-T Recommendation, Routing of Multimedia Connections Across TDM-, ATM-, and IP-Based Networks. [E.352] ITU-T Recommendation, Routing Guidelines for Efficient Routing Methods. [E.353] ITU-T Draft Recommendation, Routing of Calls when Using International Network Routing Addresses [E.412] ITU-T Recommendation, Network Management Controls. [E.525] ITU-T Recommendation, Designing Networks to Control GOS, May 1992. [E.529] ITU-T Recommendation, Network Dimensioning using End-to-End GOS Objectives, May 1997. [E.600] ITU-T Recommendation, Terms and Definitions of Traffic Engineering, March 1993. [E.651] ITU-T Recommendation, Reference Connections for Traffic Engineering of IP Access Networks. [E.716] User Demand Modeling in Broadband-ISDN, October 1996. [E.734] ITU-T Recommendation, Methods for Allocation and Dimensioning Intelligent Network (IN) Resources, October 1996. [E.735] ITU-T Recommendation, Framework for Traffic Control and Dimensioning in B-ISDN, May 1997. [E.736] ITU-T Recommendation, Methods for Cell Level Traffic Control in the B-ISDN, May 1997. [E.737] ITU-T Recommendation, Dimensioning Methods for B-ISDN, May 1997. [E.800] ITU-T Recommendation, Terms and Definitions Related to Quality of Service and Network Performance Including Dependability, August 1994. [E.TE] ITU-T Draft Recommendation, Traffic Engineering and QoS Methods for IP-, ATM- and TDM-Based Multiservice Networks, March 2000. [ETSIa] ETSI Secretariat, Telecommunications and Internet Protocol Harmonization over Networks (TIPHON); Naming and Addressing; Scenario 2, DTS/TIPHON-04002 v1.1.64, 1998 Ash [Page ANNEX1-4] Internet Draft TE & QoS Methods for IP,ATM,TDM-Based Networks November 2000 [ETSIb] ETSI STF, Request for Information (RFI): Requirements for Very Large Scale E.164 -> IP Database, TD35, ETSI EP TIPHON 9, Portland, September 1998. [ETSIc] TD290, ETSI Working Party Numbering and Routing, Proposal to Study IP Numbering, Addressing, and Routing Issues, Sophia, September 1998. [F00] Faltstrom, P., E.164 Number and DNS, IETF Draft draft-ietf-enum-e164-dns-00, April 2000. [FASOAKCGRBASEWDH00] Fan, Y., Ashwood-Smith, P., Sharma, V., Ash, G. R., Krishnaswamy, M., Cao, Y., Girish, M. K., Ruck, H. M., Bernstein, S., Ahluwalia, S., Sjostrand, H., Eriksson, K., Wang, L., Doria, A., Hummel, H., Extensions to CR-LDP and RSVP-TE for Optical Path Set-up, IETF Draft draft-fan-mpls-lambda-signaling-00.txt, March 2000. [FGHW99] Feldman, A., Gilbert, A., Huang, P., Willinger, W., Dynamic of IP Traffic: A Study of the Role of Variability and the Impact of Control, Proceedings of the ACM SIGCOMM, September 1999. [FGLRRT00] Feldman, A., Greenberg, A., Lund, C., Reingold, N., Rexford, J., True, F., Deriving Traffic Demands for Operational IP Networks: Methodology and Experience, work in progress. [FGLRR99] Feldman, A., Greenberg, A., Lund, C., Reingold, N., Rexford, J., True, F., Netscope: Traffic Engineering for IP Networks, IEEE Network Magazine, March 2000. [FH98] Ferguson, P., Huston, G., Quality of Service: Delivering QoS on the Internet and in Corporate Networks, John Wiley & Sons, 1998. [FHH79] Franks, R. L., Heffes, H., Holtzman, J. M., Horing, S., Messerli, E. J., "A Model Relating Measurements and Forecast Errors to the Provisioning of Direct Final Trunk Groups," Bell System Technical Journal, Vol. 58, No. 2, February 1979. [FIA99] Fujita, N., Iwata, A., Ash, G. R., Crankback Routing Extensions for CR-LDP, IETF Draft draft-fujita-mpls-crldp-crankback-00.txt, March 2000. [FJ93] Floyd, S., Jacobson, V., Random Early Detection Gateways for Congestion Avoidance, IEEE/ACM Transactions on Networking, August 1993. [FO00] Folts, H., Ohno, H., Functional Requirements for Priority Services to Support Critical Communications, draft-folts-ohno-ieps-considerations-00.txt, June 2000. [FRC98] Feldman, A., Rexford, J., Caceres, R., Efficient Policies for Carrying Web Traffic Over Flow-Switched Networks, IEEE/ACM Transactions on Networking, December 1998. [FT00] Fortz, B., Thorup, M., Internet Traffic Engineering by Optimizing OSPF Weights, Proceedings of IEEE INFOCOM, March 2000. [G.723.1] ITU-T Recommendation, Dual Rate Speech Coder for Multimedia Communications Transmitting at 5.3 and 6.3 kbit/s, March 1996. Ash [Page ANNEX1-5] Internet Draft TE & QoS Methods for IP,ATM,TDM-Based Networks November 2000 [G99a] Glossbrenner, K., Elements Relevant to Routing of ATM Connections, ITU-T Liaison to Study Group 2, Temporary Document 1/2-8, Geneva Switzerland, May 1999. [G99b] Glossbrenner, K., IP Performance Studies, ITU-T Liaison to Study Group 2, Temporary Document GEN-27, Geneva Switzerland, May 1999. [GDW00] Ghani, N., Duxit, S., Wang, T., On IP-Over-WDM Integration, IEEE Communications Magazine, March 2000. [GJFALF99] Ghanwani, A., Jamoussi, B., Fedyk, D., Ashwood-Smith, P., Li, L., Feldman, N., Traffic Engineering Standards in IP Networks using MPLS, IEEE Communications Magazine, December 1999. [GWA97] Gray, E., Wang, Z., Armitage, G., Generic Label Distribution Protocol Specification, IETF Draft, draft-gray-mpls-generic-ldp-spec-00.txt, November 1997. [GR99] Greene, N., Ramalho, M., Media Gateway Control Protocol Architecture and Requirements, IETF Draft, draft-ietf-megaco-reqs-00.txt, January 1999. [H95] Huitema, C., Routing in the Internet, Prentice Hall, 1995. [H97] Halabi, B., Internet Routing Architectures, Cisco Press, 1997. [H99] Heyman, D. P., Estimation of MMPP Models of IP Traffic, unpublished work. [H.225.0] ITU-T Recommendation, Media Stream Packetization and Synchronization on Non-Guaranteed Quality of Service LANs, November 1996. [H.245] ITU-T Recommendation, Control Protocol for Multimedia Communication, March 1996. [H.246] Draft ITU-T Recommendation, Interworking of H.Series Multimedia Terminals with H.Series Multimedia Terminals and Voice/Voiceband Terminals on GSTN and ISDN, September 1997. [H.323] ITU-T Recommendation, Visual Telephone Systems and Equipment for Local Area Networks which Provide a Non-Guaranteed Quality of Service, November 1996. [HCC00] Huston, G., Cerf, V. G., Chapin, L., Internet Performance Survival Guide: QoS Strategies for Multi-Service Networks, John Wiley & Sons, February 2000. [HiN76] Hill, D. W., Neal, S. R., "The Traffic Capacity of a Probability Engineered Trunk Group," Bell System Technical Journal, Vol. 55, No. 7, September 1976. [HL96] Heyman, D. P., Lakshman, T. V., What are the Implications of Long-Range Dependence for VBR-Video Traffic Engineering?, IEEE Transactions on Networking, June 1996. [HSMOA00] Huang, C., Sharma, V., Makam, S., Owens, K., A Path Protection/Restoration Mechanism for MPLS Networks, draft-chang-mpls-path-protection-00.txt, March 2000. Ash [Page ANNEX1-6] Internet Draft TE & QoS Methods for IP,ATM,TDM-Based Networks November 2000 [HM00] Heyman, D. P., Mang, X., Why Modeling Broadband Traffic is Difficult, and Potential Ways of Doing It, Fifth INFORMS Telecommunications Conference, Boca Raton, FL, March 2000. [HSMO00] Huang, C., Sharma, V., Makam, S., Owens, K., A Path Protection/Restoration Mechanism for MPLS Networks, draft-chang-mpls-path-protection-00.txt, March 2000. [HSMOA00] Huang, C., Sharma, V., Makam, S., Owens, K., Akyol, B., Extensions to RSVP-TE for MPLS Path Protection, draft-chang-mpls-rsvpte-path-protection-ext-00.txt, June 2000. [HY00] Hjalmtysson, G., Yates, J., Smart Routers - Simple Optics, An Architecture for the Optical Internet, submitted for publication. [I.211] ITU-T Recommendation, B-ISDN Service Aspects, March 1993. [I.324] ITU-T Recommendation, ISDN Network Architecture, 1991. [I.327] ITU-T Recommendation, B-ISDN Functional Architecture, March 1993. [I.356] ITU-T Recommendation, B-ISDN ATM Layer Cell Transfer Performance, October 1996. [J00] Jamoussi, B., Editor, Constraint-Based LSP Setup using LDP, IETF draft-ietf-mpls-cr-ldp-03.txt, September 2000. [K99] Kilkki, K., Differentiated Services for the Internet, Macmillan, 1999. [KAHRSYB00] Kankkunen, A., Ash, G., Hopkins, J., Rosen, B., Stacey, D., Yelundur, A., Berger, L., Voice over MPLS Framework, IETF Draft draft-kankkunen-vompls-fw-00.txt, March 2000. [Kne73] Knepley, J. E., "Minimum Cost Design for Circuit Switched Networks," Technical Note Numbers 36--73, Defense Communications Engineering Center, System Engineering Facility, Reston, Virginia, July 1973. [KR00] Kurose, J. F., Ross, K. W., Computer Networking, A Top-Down Approach Featuring the Internet, Addison-Wesley, 2000. [KR00a] Kompella, K., Rekhter, Y., LSP Hierarchy with MPLS TE, Internet Draft draft-kompella-lsp-hierarchy-00.txt, June 2000. [Kru37] Kruithof, J., "Telefoonverkeersrekening," De Ingenieur, Vol. 52, No. 8, February 1937. [Kru79] Krupp, R. S., "Properties of Kruithof's Projection Method," Bell System Technical Journal, Vol. 58, No. 2, February 1979. [Kru82] Krupp, R. S., "Stabilization of Alternate Routing Networks," IEEE International Communications Conference, Philadelphia, Pennsylvania, 1982. [L00] Lai, W., Capacity Engineering of IP-Based Networks with MPLS, IETF Draft draft-wlai-tewg-cap-eng-00.txt, March 2000. Ash [Page ANNEX1-7] Internet Draft TE & QoS Methods for IP,ATM,TDM-Based Networks November 2000 [L99] Li, T., MPLS and the Evolving Internet Architecture, IEEE Communications Magazine, December 1999. [LRACJ00] Luciani, J., Rajagopalan, B., Awduche, D., Cain, B., Jamoussi, B., IP over Optical Networks - A Framework, IETF Draft draft-ip-optical-framework-00.txt, March 2000. [LS00] Lazer, M., Strand, J., Some Routing Constraints, Optical Interworking Forum contribution OIF2000.109, May 2000. [LTWW94] Leland, W., Taqqu, M., Willinger, W., Wilson, D., On the Self-Similar Nature of Ethernet Traffic, IEEE/ACM Transactions on Networking, February 1994. [LDVKCH00] Le Faucheur, F., Davari, S., Vaananen, P., Krishnan, R., Cheval, P., Heinanen, J., MPLS Support of Differentiated Services, IETF Draft draft-ietf-mpls-diff-ext-05.txt, June 2000. [M98] Metz, C., IP Switching: Protocols and Architecture, McGraw-Hill, 1998. [M99] Moy, J., OSPF: Anatomy of an Internet Routing Protocol, Addison Wesley, 1999. [M99a] McDysan, D., QoS and Traffic Management in IP and ATM Networks, McGraw-Hill, 1999. [MRMRBMSOAPLFEK00] Marshall, W., Ramakrishnan, K., Miller, E., Russell, G., Beser, B., Mannette, M., Steinbrenner, K., Oran, D., Andreasen, F., Pickens, J., Lalwaney, P., Fellows, J., Evans, D., Kelly, K., Architectural Considerations for Providing Carrier Class Telephony Services Utilizing SIP-based Distributed Call Control Mechanisms, IETF Draft draft-dcsgroup-sip-arch-00.txt, March 2000. [Mum76] Mummert, V. S., "Network Management and Its Implementation on the No. 4ESS," International Switching Symposium, Japan, 1976. [NaM73] Nakagome, Y., Mori, H., "Flexible Routing in the Global Communication Network," Proceedings of the Seventh International Teletraffic Congress, Stockholm, Sweden, 1973. [NWRH99] Neilson, R., Wheeler, J., Reichmeyer, F., Hares, S., A Discussion of Bandwidth Broker Requirements for Internet2 Qbone Deployment, August 1999. [PARLAY] Parlay API Specification 1.2, September 10, 1999. [PaW82] Pack, C. D., Whitaker, B. A., "Kalman Filter Models for Network Forecasting," Bell System Technical Journal, Vol. 61, No. 1, January 1982. [PL99] Faltstrom, P., Larson, B., E.164 Number and DNS, IETF draft-faltstrom-e164-03.txt, September 1999. [PW00] Park, K., Willinger, W., Self-Similar Network Traffic and Performance Evaluation, John Wiley & Sons, August 2000. [Q.71] ITU-T Recommendation, ISDN Circuit Mode Switched Bearer Services. Ash [Page ANNEX1-8] Internet Draft TE & QoS Methods for IP,ATM,TDM-Based Networks November 2000 [Q.765.5] ITU-T Recommendation, Application Transport Mechanism - Bearer Independent Call Control (BICC), December 1999. [Q.1901] ITU-T Recommendation, Bearer Independent Call Control Protocol, February 2000. [Q.2761] ITU-T Recommendation, Broadband Integrated Services Digital Network (B-ISDN) Functional Description of the B-ISDN User Part (B-ISUP) of Signaling System Number 7. [Q.2931] ITU-T Recommendation, Broadband Integrated Services Digital Network (B-ISDN) - Digital Subscriber Signalling System No. 2 (DSS 2) - User-Network Interface (UNI) Layer 3 Specification for Basic Call/Connection Control, February 1995. [R99] Roberts, J. W., Engineering for Quality of Service, Chapter appearing in [PW00]. [RFC1633] Braden, R., Clark, D., Shenker, S., Integrated Services in the Internet Architecture: an Overview, June 1994. [RFC1889] Schulzrinne, H., Casner, S., Frederick, R., Jacobson, V., RTP: A Transport Protocol for Real-Time Applications, January 1996. [RFC1940] Estrin, D., Li, T., Rekhter, Y., Varadhan, K., Zappala, D., Source Demand Routing: Packet Format and Forwarding Specification (Version 1), May 1996. [RFC2205] Bradem. R., Zhang, L., Berson, S., Herzog, S., Jamin, S., Resource ReSerVation Protocol (RSVP) - Version 1 Functional Specification, September 1997. [RFC2328] Moy, J, OSPF Version 2, April 1998. [RFC2332] Luciani, J., Katz, D., Piscitello, D., Cole, B., Doraswamy, N., NBMA Next Hop Resolution Protocol (NHRP), April 1998. [RFC2386] Crawley, E., Nair, R., Rajagopalan, B., Sandick, H., A Framework for QoS-based Routing in the Internet, August 1998. [RFC2475] Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z., Weiss, W., An Architecture for Differentiated Services, December 1998. [RFC2543] Handley, M., Schulzrinne, H., Schooler, E. Rosenberg, J. SIP: Session Initiation Protocol, March 1999. [RFC2702] Awduche, D., Malcolm, J., Agogbua, J., O'Dell, M., McManus, J. Requirements for Traffic Engineering over MPLS, September 1999. [RFC2805] Greene, N., Ramalho, M., Rosen, B., Media Gateway Control Protocol Architecture and Requirements, April 2000. [RL00] Rekhter, Y., Li, T., A Border Gateway Protocol 4 (BGP-4), IETF Draft draft-ietf-idr-bgp4-10.txt, April 2000. [RO00] Roberts, J. W., Oueslati-Boulahia, S., Quality of Service by Flow Ash [Page ANNEX1-9] Internet Draft TE & QoS Methods for IP,ATM,TDM-Based Networks November 2000 Aware Networking, work in progress. [RVC99] Rosen, E., Viswanathan, A., Callon, R., Multiprotocol Label Switching Architecture, IETF draft-ietf-mpls-arch-06.txt, August 1999. [S94] Stevens, W. R., TCP/IP Illustrated, Volume 1, The Protocols, Addison-Wesley, 1994. [S95] Steenstrup, M., Editor, Routing in Communications Networks, Prentice-Hall, 1995. [S99] Swallow, G., MPLS Advantages for Traffic Engineering, IEEE Communications Magazine, December 1999. [SAHG00] Slutsman, L., Ash, G., Haerens, F., Gurbani, V. K., Framework and Requirements for the Internet Intelligent Network (IIN), IETF Draft draft-lslutsman-sip-iin-framework-00.txt, March 2000. [SC00] Strand, J., Chiu, A. L., What's Different About the Optical Layer Control Plane?, submitted for publication. [SL99] Schwefel, H-P., Lipsky, L., Performance Results for Analytic Models of Traffic in Telecommunication Systems, Based on Multiple ON-OFF Sources with Self-Similar Behavior, 16th International Teletraffic Congress, Edinburgh, June 1999. [ST98] Sikora, J., Teitelbaum, B., Differentiated Services for Internet2, Internet2: Joint Applications/Engineering QoS Workshop, Santa Clara, CA, May 1998. [ST99] Sahinoglu, Z., Tekinay, S., On Multimedia Networks: Self-Similar Traffic and Network Performance, IEEE Communication Magazine, January 1999. [STB99] Suryaputra, S., Touch, J. D., Bannister, J., Simple Wavelength Assignment Protocol, USC Information Sciences Institute ISC/ISI RR-99-473, October 1999. [TRQ3000] Supplement to ITU-T Recommendation Q.1901, Operation of the Bearer Independent Call Control (BICC) Protocol with Digital Subscriber Signaling System No. 2 (DSS2), December 1999. [TRQ3010] Supplement to ITU-T Recommendation Q.1901, Operation of the Bearer Independent Call Control (BICC) Protocol with AAL Type 2 Signaling Protocol (CS1), December 1999. [TRQ3020] Supplement to ITU-T Recommendation Q.1901, Operation of the Bearer Independent Call Control (BICC) Protocol with Broadband ISDN User Part (B-ISUP) Protocol for AAL Type 1 Adaptation, December 1999. [Tru54] Truitt, C. J., "Traffic Engineering Techniques for Determining Trunk Requirements in Alternate Routed Networks," Bell System Technical Journal, Vol. 31, No. 2, March 1954. [V99] Villamizar, C., MPLS Optimized Multipath, draft-villamizar-mpls-omp-01, February 1999. [WBP00] Wright, G., Ballarte, S., Pearson, T., CR-LDP Extensions for Ash [Page ANNEX1-10] Internet Draft TE & QoS Methods for IP,ATM,TDM-Based Networks November 2000 Interworking with RSVP-TE, Internet Draft draft-wright-mpls-crldp-rsvpte-iw-00.txt, March 2000. [Wei63] Weintraub, S., Tables of Cumulative Binomial Probability Distribution for Small Values of p, London: Collier-Macmillan Limited, 1963. [WE99] Widjaja, I., Elwalid, A., MATE: MPLS Adaptive Traffic Engineering, draft-widjaja-mpls-mate-00.txt, October 1999. [WHJ00] Wright, S., Herzog, S., Jaeger, R., Requirements for Policy Enabled MPLS, draft-wright-policy-mpls-00.txt, March 2000. [Wil56] Wilkinson, R. I., "Theories of Toll Traffic Engineering in the U.S.A.," Bell System Technical Journal, Vol. 35, No. 6, March 1956. [Wil58] Wilkinson, R. I., "A Study of Load and Service Variations in Toll Alternate Route Systems," Proceedings of the Second International Teletraffic Congress, The Hague, Netherlands, July 1958, Document No. 29. [Wil71] Wilkinson, R. I., "Some Comparisons of Load and Loss Data with Current Teletraffic Theory," Bell System Technical Journal, Vol. 50, October 1971, pp. 2807--2834. [XHBN00] Xiao, X., Hannan, A., Bailey, B., Ni, L. M., Traffic Engineering with MPLS in the Internet, IEEE Network Magazine, March/April 2000. [XN99] Xiao, X., Ni, L. M., Internet QoS: A Big Picture, IEEE Network Magazine, March/April, 1999. [Yag71] Yaged, B., Jr., "Long Range Planning for Communications Networks," Polytechnic Institute of Brooklyn, Ph.D. Thesis, 1971. [Yag73] Yaged, B., "Minimum Cost Design for Circuit Switched Networks," Networks, Vol. 3, 1973, pp. 193--224. [YR99] Yates, J. M., Rumsewicz, M. P. Lacey, J. P. R., Wavelength Converters in Dynamically-Reconfigurable WDM Networks, IEEE Communications Society Survey Paper, 1999. [ZSSC97] Zhang, Sanchez, Salkewicz, Crawley, Quality of Service Extensions to OSPF or Quality of Service Route First Routing (QOSPF), IETF Draft, draft-shang-qos-ospf-00.txt, September 1997. Ash [Page ANNEX1-11] Internet Draft TE & QoS Methods for IP,ATM,TDM-Based Networks November 2000 ANNEX 2 Call Routing & Connection Routing Methods Traffic Engineering & QoS Methods for IP-, ATM-, & TDM-Based Multiservice Networks 2.1 Introduction In the draft we assume the separation of "call routing" and signaling for call establishment from "connection (or bearer-path) routing" and signaling for bearer-channel establishment. Call routing protocols primarily translate a number or a name, which is given to the network as part of a call setup, to a routing address needed for the connection (bearer-path) establishment. Call routing protocols are described for example in [Q.2761] for the Broadband ISDN Used Part (B-ISUP) call signaling, [ATM990048] for bearer-independent call control (BICC), or virtual trunking, call signaling, [H.323] for H.323 call signaling, [GR99] for the media gateway control [RFC2805] call signaling, and in [HSSR99] for the session initiation protocol (SIP) call signaling. Connection routing protocols include for example [Q.2761] for B-ISUP signaling, [ATM960055] for PNNI signaling, [ATM960061] for UNI signaling, [DN99] for switched virtual path (SVP) signaling, and [J00] for MPLS constraint-based routing label distribution protocol (CRLDP) signaling. A specific connection or bearer-path routing method is characterized by the routing table used in the method. The routing table consists of a set of paths and rules to select one path from the route for a given connection request. When a connection request arrives at its originating node (ON), the ON implementing the routing method executes the path selection rules associated with the routing table for the connection to determine a selected path from among the path candidates in the route for the connection request. In a particular routing method, the path selected for the connection request is governed by the connection routing, or path selection, rules. Various path selection methods are discussed: fixed routing (FR) path selection, time-dependent routing (TDR) path selection, state-dependent routing (SDR) path selection, and event-dependent routing (EDR) path selection. 2.2 Call Routing Methods Call routing entails number (or name) translation to a routing address, which is then used for connection establishment. Routing addresses can consist, for example, of a) E.164 ATM end system addresses (AESAs) [E.191], b) network routing addresses (NRAs) [E.353], and/or c) IP addresses [S94]. As discussed in ANNEX 4, a TE requirement is the need for carrying E.164-AESA addresses, NRAs, and IP addresses in the connection-setup information element (IE). In that case, E.164-AESA addresses, NRAs, and IP addresses become the standard addressing method for interworking across IP-, ATM-, and TDM-based networks. Another TE requirement is that a call identification code (CIC) be carried in the call-control and bearer-control connection-setup IEs in order to correlate the call-control setup with the bearer-control setup [Q.1901, ATM990048]. Carrying these additional parameters in the Signaling System 7 (SS7) ISDN User Part (ISUP) connection-setup IEs is referred to as the bearer independent call control (BICC) protocol. Number (or name) translation, then, should result in the E.164-AESA Ash [Page ANNEX2-1] Internet Draft TE & QoS Methods for IP,ATM,TDM-Based Networks November 2000 addresses, NRAs, and/or IP addresses. NRA formats are covered in [E.353], and IP-address formats in [S94]. The AESA address has a 20-byte format as shown in Figure 2.1a below [E.191]. Figure 2.1a AESA Address STructure The IDP is the initial domain part and the DSP is the domain specific part. The IDP is further subdivided into the AFI and IDI. The IDI is the initial domain identifier and can contain the 15-digit E.164 address if the AFI is set to 45. AFI is the authority and format identifier and determines what kind of addressing method is followed, and based on the 1 octet AFI value, the length of the IDI and DSP fields can change. The E.164-AESA address is used to determine the path to the destination endpoint. E.164-AESA addressing for B-ISDN services is supported in ATM networks using PNNI, through use of the above AESA format. In this case the E.164 part of the AESA address occupies the 8 octet IDI, and the 11 octet DSP can be used at the discretion of the network operator (perhaps for sub-addresses). The above AESA structure also supports AESA DCC (data country code) and AESA ICD (international code designator) addressing formats. Within the IP network, routing is performed using IP addresses. Translation databases, such as based on domain name system (DNS) technology [F00], are used to translate the E.164 numbers/names for calls to IP addresses for routing over the IP network. The IP address is a 4-byte address structure as shown below: Figure 2.1b IP Address Structure There are five classes of IP addresses. Different classes have different field lengths for the network identification field. Classless inter-domain routing (CIDR) allows blocks of addresses to be given to service providers in such a manner as to provide efficient address aggregation. This is accompanied by capabilities in the BGP4.0 protocol for efficient address advertisements [RL00, S94]. 2.3 Connection (Bearer-Path) Routing Methods Connection routing is characterized by the routing table used in the method and rules to select one path from the route for a given connection or bandwidth-allocation request. When a connection/bandwidth-allocation request is initiated by an ON, the ON implementing the routing method executes the path selection rules associated with the routing table for the connection/bandwidth-allocation to find an admissible path from among the paths in the route that satisfies the connection/bandwidth-allocation request. In a particular routing method, the selected path is determined according to the rules associated with the routing table. In a network with originating connection/bandwidth-allocation control, the ON maintains control of the connection/bandwidth-allocation request. If crankback/bandwidth-not-available is used, for example, at a via node (VN), the preceding node maintains control of the connection/bandwidth-allocation request even if the request is blocked on all the links outgoing from the VN. Here we are discussing network-layer connection routing (sometimes referred to as "layer-3" routing), as opposed to the link-layer logical-transport-link ("layer-2") routing or physical-layer ("layer-1") routing. In the draft the term "link" will normally mean Ash [Page ANNEX2-2] Internet Draft TE & QoS Methods for IP,ATM,TDM-Based Networks November 2000 "logical-transport-link." In ANNEX 5 we address logical-transport-link routing. The network-layer (layer-3) connection routing methods addressed include those discussed in * Open Shortest Path First (OSPF), Border Gateway Protocol (BGP), and Multiprotocol Label Switching (MPLS) for IP-based routing methods, * User-to-Network Interface (UNI), Private Network-to-Network Interface (PNNI), ATM Inter-Network Interface (AINI), and Bandwidth Modify for ATM-based routing methods, and * Recommendations E.170, E.350, and E.351 for TDM-based routing methods. In an IP network, logical links called traffic trunks can be defined which consist of MPLS label switched paths (LSPs) between the IP nodes. Traffic trunks are used to allocate the bandwidth of the logical transport links to various node pairs. In an ATM network, logical links called virtual paths (VPs) (the equivalent of traffic trunks) can be defined between the ATM nodes, and VPs can be used to allocate the bandwidth of the logical transport links to various node pairs. In a TDM network, the logical transport links consist of trunk groups between the TDM nodes. A sparse logical transport link network is typically used with IP and ATM technology, as illustrated in Figure 2.2, and FR, TDR, SDR, and EDR can be used in combination with multilink shortest path selection. Figure 2.2 Sparse Logical Network Topology with Connections Routed on Multilink Paths A meshed logical-transport-link network is typically used with TDM technology, but can be used also with IP or ATM technology as well, and selected paths are normally limited to 1 or 2 logical transport links, or trunk groups, as illustrated in Figure 2.3. Figure 2.3 Mesh Logical Network Topology with Connections Routed on 1- and 2-Link Paths Paths may be set up on individual connections (or "per flow") for each call request, such as on a switched virtual circuits (SVC). Paths may also be set up for bandwidth-allocation requests associated with "bandwidth pipes" or traffic trunks, such as on switched virtual paths (SVPs) in ATM-based networks or constraint-based routing label switched paths (CRLSPs) in IP-based networks. Paths are determined by (normally proprietary) algorithms based on the network topology and reachable address information. These paths can cross multiple peer groups in ATM-based networks, and multiple autonomous systems (ASs) in IP-based networks. An ON may select a path from the routing table based on the routing rules and the QoS resource management criteria, described in ANNEX 3, which must be satisfied on each logical-transport-link in the path. If a link is not allowed based on the QoS criteria, then a release with crankback/bandwidth-not-available parameter is used to signal that condition to the ON in order to return the connection/bandwidth-allocation request to the ON, which may then select an alternate path. In addition to controlling bandwidth allocation, the QoS resource management procedures can check end-to-end transfer delay, delay variation, and transmission quality considerations such as loss, echo, and noise. Ash [Page ANNEX2-3] Internet Draft TE & QoS Methods for IP,ATM,TDM-Based Networks November 2000 When source routing is used, setup of a connection/bandwidth-allocation request is achieved by having the ON identify the entire selected path including all VNs and DN in the path in a designated-transit-list (DTL) or explicit-route (ER) parameter in the connection-setup IE. If the QoS or traffic parameters cannot be realized at any of the VNs in the connection setup request, then the VN generates a crankback (CBK)/bandwidth-not-available (BNA) parameter in the connection-release IE which allows a VN to return control of the connection request to the ON for further alternate routing. In ANNEX 4, the DTL/ER and CBK/BNA elements are identified as being required for interworking across IP-, ATM-, and TDM--based networks. As noted earlier, connection routing, or path selection, methods are categorized into the following four types: fixed routing (FR), time-dependent routing (TDR), state-dependent routing (SDR), and event-dependent routing (EDR). We discuss each of these methods in the following paragraphs. Examples of each of these path selection methods are illustrated in Figures 2.4a and 2.4b and discussed in the following sections. Figure 2.4a TDR Dynamic Path Selection Methods Figure 2.4b EDR & SDR Dynamic Path Selection Methods Dynamic routing allows routing tables to be changed dynamically, either in an off-line, preplanned, time-varying manner, as in TDR, or on-line, in real time, as in SDR or EDR. With off-line, pre-planned TDR path selection methods, routing patterns contained in routing tables might change every hour or at least several times a day to respond to measured hourly shifts in traffic loads, and in general TDR routing tables change with a time constant normally greater than a call/traffic-flow holding time. A typical TDR routing method may change routing tables every hour, which is longer than a typical voice call/traffic-flow holding time of a few minutes. Three implementations of TDR dynamic path selection are illustrated in Figure 2.4a, which shows multilink path routing, 2-link path routing, and progressive routing. TDR routing tables are preplanned, preconfigured, and recalculated perhaps each week within the capacity management network design function. Real-time dynamic path selection does not depend on precalculated routing tables. Rather, the node or centralized bandwidth broker senses the immediate traffic load and if necessary searches out new paths through the network possibly on a per-traffic-flow basis. With real-time path selection methods, routing tables change with a time constant on the order of or less than a call/traffic-flow holding time. As illustrated in Figure 2.4b, on-line, real-time path selection methods include EDR and SDR. 2.4 Hierarchical Fixed Routing (FR) Path Selection Hierarchical fixed routing (FR) is an important routing topology employed in all types of networks, including IP-, ATM-, and TDM-based networks. In IP-based networks, there is often a hierarchical relationship among different "areas", or sub-networks. Hierarchical multi-domain (or multi-area or multi-autonomous-system) topologies are normally used with IP routing protocols (OSPF, BGP) and ATM routing protocols (PNNI), as well as within almost all TDM-based network routing topologies. Ash [Page ANNEX2-4] Internet Draft TE & QoS Methods for IP,ATM,TDM-Based Networks November 2000 Figure 2.4c Hierachical Fixed Routing Path Selection Methods (2-Level Hierarchical Network) For example, in Figure 2.4c, BB1 and BB2 could be backbone nodes in a "backbone area", and AN1 and AN2 could be access nodes in separate "access areas" distinct from the backbone area. Routing between the areas follows a hierarchical routing pattern, while routing within an area follows an interior gateway protocol (IGP), such as OSPF plus MPLS. Similarly, in ATM-based networks the same concept exists, but here the "areas" are called "peer-groups", and for example, the IGP used within peer-groups could be PNNI. In TDM-based networks, the routing between sub-networks, for example, metropolitan-area-networks and long-distance networks, is normally hierarchical, as in IP- and ATM-based networks, and the IGP in TDM-based networks could be either hierarchical or dynamic routing. We now discuss more specific attributes and methods for hierarchical FR path selection. In a FR method, a routing pattern is fixed for a connection request. A typical example of fixed routing is a conventional, TDM-based, hierarchical alternate routing pattern where the route and route selection sequence are determined on a preplanned basis and maintained over a long period of time. Hierarchical FR is illustrated below. FR is more efficiently applied, however, when the network is nonhierarchical, or flat, as compared to the hierarchical structure [A98]. The aim of hierarchical fixed routing is to carry as much traffic as is economically feasible over direct links between pairs of nodes low in the hierarchy. This is accomplished by application of routing procedures to determine where sufficient load exists to justify high-usage logical-transport-links, and then by application of alternate-routing principles that effectively pool the capacities of high-usage links with those of final links, to the end that all traffic is carried efficiently. The routing of connection requests in a hierarchical network involves an originating ladder, a terminating ladder, and links interconnecting the two ladders. In a two-level network, for example, the originating ladder is the final link from lower level-1 node to the upper level-2 node, and the terminating ladder is the final link from upper level-2 node to the lower level-1 node. Links AN1-BB2, AN2-BB1, and BB1--BB2 in Figure 2.4c are examples of interladder links. The identification of the proper interladder link for the routing of a given connection request identifies the originating ladder "exit" point and the terminating ladder "entry" point. Once these exit and entry points are identified and the intraladder links are known, a first-choice path from originating to terminating location can be determined. Various levels of traffic concentration are used to achieve an appropriate balance between transport and switching. The generally preferred routing sequence for the AN1 to AN2 connections is 1. A connection request involving no via nodes: path AN1-AN2 (if the link existed). 2. A connection request involving one via node: path AN1-BB2-AN2, AN1-BB1-AN2, in that order. Ash [Page ANNEX2-5] Internet Draft TE & QoS Methods for IP,ATM,TDM-Based Networks November 2000 3. A connection request involving two via nodes: path AN1-BB1-BB2-AN2 This procedure provides only the first-choice interladder link from AN1 to AN2. Connection requests from AN2 to AN1 often route differently. To determine the AN2-to-AN1 route requires reversing the diagram, making AN2-BB2 the originating ladder and AN1-BB1 the terminating ladder. In Figure 2.4c the preferred path from AN2 to AN1 is AN2-AN1, AN2-BB1-AN1, AN2-BB2-AN1, and AN2-BB2-BB1-AN1, in that order. The alternate path for any high-usage link is the path the node-to-node traffic load between the nodes would follow if the high-usage link did not exist. In Figure 2.4c, this is AN2-BB1-AN1. 2.5 Time-Dependent Routing (TDR) Path Selection TDR methods are a type of dynamic routing in which the routing tables are altered at a fixed point in time during the day or week. TDR routing tables are determined on an off-line, preplanned basis and are implemented consistently over a time period. The TDR routing tables are determined considering the time variation of traffic load in the network, for example based on measured hourly load patterns. Several TDR time periods are used to divide up the hours on an average business day and weekend into contiguous routing intervals sometimes called load set periods. Typically, the TDR routing tables used in the network are coordinated by taking advantage of noncoincidence of busy hours among the traffic loads. In TDR, the routing tables are preplanned and designed off-line using a centralized bandwidth broker, which employs a TDR network design model. Such models are discussed in ANNEX 6. The off-line computation determines the optimal routes from a very large number of possible alternatives, in order to maximize network throughput and/or minimize the network cost. The designed routing tables are loaded and stored in the various nodes in the TDR network, and periodically recomputed and updated (e.g., every week) by the bandwidth broker. In this way an ON does not require additional network information to construct TDR routing tables, once the routing tables have been loaded. This is in contrast to the design of routing tables on-line in real time, such as in the SDR and EDR methods described below. Paths in the TDR routing table may consist of time varying routing choices and use a subset of the available paths. Paths used in various time periods need not be the same. Paths in the TDR routing table may consist of the direct link, a 2-link path through a single VN, or a multiple-link path through multiple VNs. Path routing implies selection of an entire path between originating and terminating nodes before a connection is actually attempted on that path. If a connection on one link in a path is blocked (e.g., because of insufficient bandwidth), the connection request then attempts another complete path. Implementation of such a routing method can be done through control from the originating node, plus a multiple-link crankback capability to allow paths of two, three, or more links to be used. Crankback is an information-exchange message capability that allows a connection request blocked on a link in a path to return to the originating node for further alternate routing on other paths. Path-to-path routing is nonhierarchical and allows the choice of the most economical paths rather than being restricted to hierarchical paths. Path selection rules employed in TDR routing tables, for example, may Ash [Page ANNEX2-6] Internet Draft TE & QoS Methods for IP,ATM,TDM-Based Networks November 2000 consist of simple sequential routing. In the sequential method all traffic in a given time period is offered to a single route, and lets the first path in the route overflow to the second path which overflows to the third path, and so on. Thus, traffic is routed sequentially from path to path, and the route is allowed to change from hour to hour to achieve the preplanned dynamic, or time varying, nature of the TDR method. Other TDR path selection rules can employ probabilistic techniques to select each path in the route and thus influence the realized flows. One such method of implementing TDR multilink path selection is to allocate fractions of the traffic to routes and to allow the fractions to vary as a function of time. One approach is cyclic path selection, illustrated in Figure 2.4a, which has as its first route (1, 2, ..., M), where the notation (i, j, k) means that all traffic is offered first to path i, which overflows to path j, which overflows to path k. The second route of a cyclic route choice is a cyclic permutation of the first route: (2, 3, ..., M, 1). The third route is likewise (3, 4, ..., M, 1, 2), and so on. This approach has computational advantages because its cyclic structure requires considerably fewer calculations in the design model than does a general collection of paths. The route congestion level of cyclic routes are identical; what varies from route to route is the proportion of flow on the various links. Two-link TDR path selection is illustrated in Figure 2.4a. An example implementation is 2-link sequential TDR (2S-TDR) path selection. By using the crankback signal, 2S-TDR limits path connections to at most two links, and, in meshed network topologies, such TDR 2-link sequential path selection allows nearly as much network utilization and performance improvement as TDR multilink path selection. This is because in the design of multilink path routing in meshed networks, about 98 percent of the traffic is routed on one- and 2-link paths, even though paths of greater length are allowed. Because of switching costs, paths with one or two links are usually less expensive than paths with more links. Therefore, as illustrated in Figure 2.4a, 2-link path routing uses the simplifying restriction that paths can have only one or two links, which requires only single-link crankback to implement and uses no common links as is possible with multilink path routing. Alternative 2-link path selection methods include the cyclic routing method described above and sequential routing. In sequential routing, all traffic in a given hour is offered to a single route, and the first path is allowed to overflow to the second path, which overflows to the third path, and so on. Thus, traffic is routed sequentially from path to path with no probabilistic methods being used to influence the realized flows. The reason that sequential routing works well is that permuting path order provides sufficient flexibility to achieve desired flows without the need for probabilistic routing. In 2S-TDR, the sequential route is allowed to change from hour to hour. The TDR nature of the dynamic path selection method is achieved by introducing several route choices, which consist of different sequences of paths, and each path has one or, at most, two links in tandem. Paths in the routing table are subject to depth-of-search (DoS) restrictions for QoS resource management, which is discussed in ANNEX 3. DoS requires that the bandwidth capacity available on each link in the path be sufficient to meet a DoS bandwidth threshold level, which is passed to each node in the path in the setup message. DoS restrictions prevent connections that path on the first-choice or primary (often the shortest) ON-DN path, for example, from being swamped by alternate routed multiple-link connections. Ash [Page ANNEX2-7] Internet Draft TE & QoS Methods for IP,ATM,TDM-Based Networks November 2000 A TDR connection set-up example is now given. The first step is for the ON to identify the DN and routing table information to the DN. The ON then tests for spare capacity on the first or shortest path, and in doing this supplies the VNs and DN on this path, along with the DoS parameter, to all nodes in the path. Each VN tests the available bandwidth capacity on each link in the path against the DoS threshold. If there is sufficient capacity, the VN forwards the connection setup to the next node, which performs a similar function. If there is insufficient capacity, the VN sends a release message with crankback/bandwidth-not-available parameter back to the ON, at which point the ON tries the next path in the route as determined by the routing table rules. As described above, the TDR routes are preplanned off-line, and then loaded and stored in each ON. Allocating traffic to the optimum path choice during each time period leads to design benefits due to the noncoincidence of loads. Since in many network applications traffic demands change with time in a reasonably predictable manner, the routing also changes with time to achieve maximum link utilization and minimum network cost. Several TDR routing time periods are used to divide up the hours on an average business day and weekend into contiguous routing intervals. The network design is performed in an off-line, centralized computation in the bandwidth broker that determines the optimal routing tables from a very large number of possible alternatives in order to minimize the network cost. In TDR path selection, rather than determine the optimal routing tables based on real-time information, a centralized bandwidth broker design system employs a design model, such as described in ANNEX 6. The effectiveness of the design depends on how accurately we can estimate the traffic load on the network. Forecast errors are corrected in the short-term capacity management process, which allows routing table updates to replace link augments whenever possible, as described in ANNEX 7. 2.6 State-Dependent Routing (SDR) Path Selection In SDR, the routing tables are altered automatically according to the state of the network. For a given SDR method, the routing table rules are implemented to determine the path choices in response to changing network status, and are used over a relatively short time period. Information on network status may be collected at a central bandwidth broker processor or distributed to nodes in the network. The information exchange may be performed on a periodic or on-demand basis. SDR methods use the principle of routing connections on the best available path on the basis of network state information. For example, in the least loaded routing (LLR) method, the residual capacity of candidate paths is calculated, and the path having the largest residual capacity is selected for the connection. Various relative levels of link occupancy can be used to define link load states, such as lightly-loaded, heavily-loaded, or bandwidth-not-available states. Methods of defining these link load states are discussed in ANNEX 3. In general, SDR methods calculate a path cost for each connection request based on various factors such as the load-state or congestion state of the links in the network. In SDR, the routing tables are designed on-line by the ON or a central bandwidth broker processor (BBP) through the use of network status and topology information obtained through information exchange with other nodes and/or a centralized BBP. There are various implementations of SDR distinguished by Ash [Page ANNEX2-8] Internet Draft TE & QoS Methods for IP,ATM,TDM-Based Networks November 2000 a) whether the computation of the routing tables is distributed among the network nodes or centralized and done in a centralized BBP, and b) whether the computation of the routing tables is done periodically or connection by connection. This leads to three different implementations of SDR: a) centralized periodic SDR (CP-SDR) -- here the centralized BBP obtains link status and traffic status information from the various nodes on a periodic basis (e.g., every 10 seconds) and performs a computation of the optimal routing table on a periodic basis. To determine the optimal routing table, the BBP executes a particular routing table optimization procedure such as LLR and transmits the routing tables to the network nodes on a periodic basis (e.g., every 10 seconds). b) distributed periodic SDR (DP-SDR) -- here each node in the SDR network obtains link status and traffic status information from all the other nodes on a periodic basis (e.g., every 5 minutes) and performs a computation of the optimal routing table on a periodic basis (e.g., every 5 minutes). To determine the optimal routing table, the ON executes a particular routing table optimization procedure such as LLR. c) distributed connection-by-connection (DC-SDR) SDR -- here an ON in the SDR network obtains link status and traffic status information from the DN, and perhaps from selected VNs, on a connection by connection basis and performs a computation of the optimal routing table for each connection. To determine the optimal routing table, the ON executes a particular routing table optimization procedure such as LLR. In DP-SDR path selection, nodes may exchange status and traffic data, for example, every five minutes, between traffic management processors, and based on analysis of this data, the traffic management processors can dynamically select alternate paths to optimize network performance. This method is illustrated in Figure 2.4b. Flooding is a common technique for distributing the status and traffic data, however other techniques with less overhead are also available, such as a query-for-status method, as discussed in ANNEX 4. Figure 2.4b illustrates a CP-SDR path selection method with periodic updates based on periodic network status. CP-SDR path selection provides near-real-time routing decisions by having an update of the number of idle trunks in each link sent to a network database every five seconds. Routing tables are determined from analysis of the status data using a path selection method which provides that the shortest path choice is used if the bandwidth is available. If the shortest path is busy (e.g., bandwidth is unavailable on one or more links), the second path is selected from the list of feasible paths on the basis of having the greatest level of idle bandwidth at the time; the current second path choice becomes the third, and so on. This path update is performed, for example, every five seconds. The CP-SDR model uses dynamically activated bandwidth reservation and other controls to automatically modify routing tables during network overloads and failures. CP-SDR requires the use of network status and routing recommendation information-exchange messages. Figure 2.4b also illustrates an example of a DC-SDR path selection method. In DC-SDR, the routing computations are distributed among all the nodes in Ash [Page ANNEX2-9] Internet Draft TE & QoS Methods for IP,ATM,TDM-Based Networks November 2000 the network. DC-SDR uses real-time exchange of network status information, such as with query and status messages, to determine an optimal path from a very large number of possible choices. With DC-SDR, the originating node first tries the primary path and if it is not available finds an optimal alternate path by querying the destination node and perhaps several via nodes through query-for-status network signaling for the busy-idle load status of all links connected on the alternate paths to the destination node. The originating node then finds the least loaded alternate path to route the connection request. DC-SDR computes required bandwidth allocations by virtual network from node-measured traffic flows and uses this capacity allocation to reserve capacity when needed for each virtual network. Any excess traffic above the expected flow is routed to temporarily idle capacity borrowed from capacity reserved for other loads that happen to be below their expected levels. Idle link capacity is communicated to other nodes via the query-status information-exchange messages, as illustrated in Figure 2.4b, and the excess traffic is dynamically allocated to the set of allowed paths that are identified as having temporarily idle capacity. DC-SDR controls the sharing of available capacity by using dynamic bandwidth reservation, as described in ANNEX 3, to protect the capacity required to meet expected loads and to minimize the loss of traffic for classes-of-service which exceed their expected load and allocated capacity. Paths in the SDR routing table may consist of the direct link, a 2-link path through a single VN, or a multiple-link path through multiple VNs. Paths in the routing table are subject to DoS restrictions on each link. 2.7 Event-Dependent Routing (EDR) Path Selection In EDR, the routing tables are updated locally on the basis of whether connections succeed or fail on a given path choice. In the EDR learning approaches, the path last tried, which is also successful, is tried again until blocked, at which time another path is selected at random and tried on the next connection request. EDR path choices can also be changed with time in accordance with changes in traffic load patterns. Success-to-the-top (STT) EDR path selection, illustrated in Figure 2.4b, is a decentralized, on-line path selection method with update based on random routing. STT-EDR uses a simplified decentralized learning method to achieve flexible adaptive routing. The primary path path-p is used first if available, and a currently successful alternate path path-s is used until it is blocked. In the case that path-s is blocked (e.g., bandwidth is not available on one or more links), a new alternate path path-n is selected at random as the alternate path choice for the next connection request overflow from the primary path. As described in ANNEX 3, dynamically activated bandwidth reservation is used under congestion conditions to protect traffic on the primary path. STT-EDR uses crankback when an alternate path is blocked at a via node, and the connection request advances to a new random path choice. In STT-EDR, many path choices can be tried by a given connection request before the request is blocked. In the EDR learning approaches, the current alternate path choice can be updated randomly, cyclically, or by some other means, and may be maintained as long as a connection can be established successfully on the path. Hence the routing table is constructed with the information determined during connection setup, and no additional information is required by the ON. Paths in the EDR routing table may consist of the direct link, a 2-link path through a single VN, or a multiple-link path through multiple VNs. Paths in Ash [Page ANNEX2-10] Internet Draft TE & QoS Methods for IP,ATM,TDM-Based Networks November 2000 the routing table are subject to DoS restrictions on each link. Note that for either SDR or EDR, as in TDR, the alternate path for a connection request may be changed in a time-dependent manner considering the time-variation of the traffic load. 2.8 Interdomain Routing In current practice, interdomain routing protocols generally do not incorporate standardized path selection or per class-of-service resource management. For example, in IP-based networks BGP [RL00] is used for interdomain routing but does not incorporate per class-of-service resource allocation as described in this Section. Also, MPLS techniques have not yet been addressed for interdomain applications. Extensions to interdomain routing methods discussed in this Section therefore can be considered to extend the call routing and connection routing concepts to routing between network domains. Many of the principles described for intradomain routing can be extended to interdomain routing. As illustrated in Figure 2.5, interdomain routing paths can be divided into three types: * a primary shortest path between the originating domain and destination domain, * alternate paths with all nodes in the origination domain and destination domain, and * alternate or transit paths through other transit domains. Interdomain routing can support a multiple ingress/egress capability, as illustrated in Figure 2.5 in which a connection request is routed either on the shortest path or, if not available, via an alternate path through any one of the other nodes from an originating node to a gateway node. Figure 2.5 Multiple Ingress/Egress Interdomain Routing Within an originating network, a destination network could be served by more than one gateway node, such as OGN1 and OGN2 in Figure 2.5, in which case multiple ingress/egress routing is used. As illustrated in Figure 2.5, with multiple ingress/egress routing, a connection request from the originating node N1 destined for the destination gateway node DGN1 tries first to access the links from originating gateway node OGN2 to DGN1. In doing this it is possible that the connection request could be routed from N1 to OGN2 directly or via N2. If no bandwidth is available from OGN2 to DGN1, the control of the connection request can be returned to N1 with a crankback/bandwidth-not-available indicator, after which the connection request is routed to OGN1 to access the OGN1-to-DGN1 bandwidth. If the connection request cannot be completed on the link connecting gateway node OGN1 to DGN1, the connection request can return to the originating node N1 through use of a crankback/band-not-available indicator for possible further routing to another gateway node (not shown). In this manner all ingress/egress connectivity is utilized to a connecting network, maximizing connection request completion and reliability. Once the connection request reaches an originating gateway node (such as OGN1 or OGN2), this node determines the routing to the destination gateway node DGN1 and routes the connection request accordingly. In completing the connection request to DGN1, an originating gateway node can dynamically select a direct shortest path, an alternate path through an alternate node Ash [Page ANNEX2-11] Internet Draft TE & QoS Methods for IP,ATM,TDM-Based Networks November 2000 in the destination network, or perhaps an alternate path through an alternate node in another network domain. Hence, with interdomain routing, connection requests are routed first to a shortest primary path between the originating and destination domain, then to a list of alternate paths through alternate nodes in the terminating network domain, then to a list of alternate paths through alternate nodes in the originating network domain (e.g., OGN1 and OGN2 in Figure 2.5), and finally to a list of alternate paths through nodes in other transit network domains. Examples of alternate paths which might be selected through a transit network domain are N1-OGN1-VGN1-DGN1, N1-OGN1-VGN2-DGN1, or N1-N2-OGN2-VGN2-DGN1 in Figure 2.5. Such paths through transit network domains may be tried last in the example network configuration in the Figure. For example, flexible interdomain routing may try to find an available alternate path based on link load states, where known, and connection request completion performance, where it can be inferred. That is, the originating gateway node (e.g., node OGN1 in Figure 2.5) may use its link status to a via node in a transit domain (e.g., links OGN1-VGN1 and OGN1-VGN2) in combination with the connection request completion performance from the candidate via node to the destination node in the destination network domain, in order to find the most available path to route the connection request over. For each path, a load state and a completion state are tracked. The load state indicates whether the link bandwidth from the gateway node to the via node is lightly loaded, heavily loaded, reserved, or busy. The completion state indicates whether a path is achieving above-average completion, average completion, or below-average completion. The selection of a via path, then, is based on the load state and completion state. Alternate paths in the same destination network domain and in a transit network domain are each considered separately. During times of congestion, the link bandwidth to a candidate via node may be in a reserved state, in which case the remaining link bandwidth is reserved for traffic routing directly to the candidate via node. During periods of no congestion, capacity not needed by one virtual network is made available to other virtual networks that are experiencing loads above their allocation. Similar to intradomain routing, interdomain routing can use discrete load states for interdomain links terminating in the originating domain (e.g., links OGN1-VGN1, OGN1-DGN1, OGN2-DGN1). As described in ANNEX 3, these link load states could may include lightly-loaded, heavily-loaded, reserved, and busy/bandwidth-not-available, in which the idle link bandwidth is compared with the load state thresholds for the link to determine its load condition. Completion rate is tracked on the various via paths (such as the path through via node VGN1 or VGN2 to destination node DGN1 in Figure 2.5) by taking account of the information relating either the successful completion or non-completion of a connection request through the via node. A non-completion, or failure, is scored for the connection request if a signaling release message is received from the far end after the connection request seizes an egress link, indicating a network in-completion cause value. If no such signaling release message is received after the connection request seizes an egress trunk, then the connection request is scored as a success. Each gateway node keeps a connection request completion history of the success or failure, for example, of the last 10 connection requests using a particular via path, and it drops the oldest record and adds the connection request completion for the newest connection request on that path. Based on the number of connection request completions relative to the total number of connection requests, a completion state is computed. Ash [Page ANNEX2-12] Internet Draft TE & QoS Methods for IP,ATM,TDM-Based Networks November 2000 Based on the completion states, connection requests are normally routed on the first path with a high completion state with a lightly loaded egress link. If such a path does not exist, then a path having an average completion state with a lightly loaded egress link is selected, followed by a path having a low completion state with a lightly loaded egress link. If no path with a lightly loaded egress link is available, and if the search depth permits the use of a heavily loaded egress link, the paths with heavily loaded egress links are searched in the order of high completion, average completion, and low completion. If no such paths are available, paths with reserved egress links are searched in the same order, based on the connection request completion state, if the search depth permits the use of a reserved egress link. The rules for selecting primary shortest paths and alternate paths for a connection request are governed by the availability of shortest path bandwidth and node-to-node congestion. The path sequence consists of the primary shortest path, lightly loaded alternate paths, heavily loaded alternate paths, and reserved alternate paths. Alternate paths are first selected which include nodes only in the originating and destination domains, and then selected through transit domains if necessary. Thus we have illustrated that interdomain routing methods can be considered to extend the intradomain call routing and connection routing concepts, such as flexible path selection and per-class-of-service bandwidth selection, to routing between network domains. 2.9 Modeling of Traffic Engineering Methods In the draft, a full-scale national network node model is used together with a multiservice traffic demand model to study various TE scenarios and tradeoffs. The 135-node national model is illustrated in Figure 2.6. Figure 2.6 135-Node National Network Model Typical voice/ISDN traffic loads are used to model the various network alternatives, which are based on 72 hours of a full-scale national network loading. Table 2.1 summarizes the multiservice traffic model used for the TE studies. Here the traffic loads are dynamically varying and tracked by the exponential smoothing models discussed in ANNEX 3. Three levels of traffic priority - key, normal, and best-effort -- are given to the various class-of-service categories, or virtual networks (VNETs), illustrated in Table 2.1. Class-of-service, traffic priority, and QoS resource management are all discussed further in ANNEX 3. The voice/ISDN loads are further segmented in the model into eight constant-bit-rate (CBR) VNETs, including business voice, consumer voice, international voice in and out, key-service voice, normal and key-service 64-kbps ISDN data, and 384-kbps ISDN data. For the CBR voice services, the mean data rate is assumed to be 64 kbps for all VNETs except the 384-kbps ISDN data VNET-8, for which the mean data rate is 384 kbps. Table 2.1 Virtual Network (VNET) Traffic Model used for TE Studies The data services traffic model incorporates typical traffic load patterns and comprises three additional VNET load patterns. These data services VNETs include Ash [Page ANNEX2-13] Internet Draft TE & QoS Methods for IP,ATM,TDM-Based Networks November 2000 * variable bit rate real-time (VBR-RT) VNET-9, representing services such as IP-telephony and compressed voice, * variable bit rate non-real-time (VBR-NRT) VNET-10, representing services such as WWW multimedia and credit card check, and