Network Working Group Lily Cheng Internet Draft Expiration Date: January 2002 (Editor) Yang Cao (Sycamore) John Ellson (Lucent) Anwar Elwalid (Lucent) Takeshi Hashimoto (Japan Telecom) Hirokazu Ishimatsu (Japan Telecom) Admela Jukan (Vienna U. of Tech) Li Mo (Metra Network) Ananth Nagarajan (Sprint) Yoshihito Oyama (Japan Telecom) Lijun Qian (Lucent) Iraj Saniee (Lucent) Ed Snyder (PhotonEx) Shinya Tanaka (Japan Telecom) Maarten Vissers (Lucent) Indra Widjaja (Lucent) Yangguang Xu (Lucent) Susumu Yoneda (Japan Telecom) July 2001 A Framework for Internet Network Engineering draft-cheng-network-engineering-framework-01.txt Status of this Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six Months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as ``work in progress.'' The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. Cheng et. al. [Page 1] Abstract This document outlines an early draft of a Network Engineering (NE) framework. It defines the Network Engineering function and discusses the differences between Traffic Engineering (TE) and Network Engineering. It identifies network architecture and major functional capabilities for Network Engineering. Network Engineering provides a new set of capabilities that is utilized in conjunction with Traffic Engineering to optimize the network efficiency, operation, and utilization. Table of Contents 1.0 Introduction and Background 1.1 Long-term and Short-term Internet Engineering Processes 1.2 Conventional IP Traffic Engineering and Limitation 1.3 Layered Network 1.4 Conventional IP Network Planning and Limitation 1.5 Terminology 2.0 Network Engineering 2.1 Why 2.2 What 2.3 How 2.4 When 2.5 Where 3.0 Architectural Elements of Network Engineering 3.1 Architectural Overview 3.2 Traffic Information 3.3 Network Topology Information 3.3.1 IP Network Topology Information 3.3.2 Transport Network Topology Information 3.4 Triggering Mechanisms 3.5 Optimization Module 3.6 Link Operations 4.0 Network Engineering and Traffic Engineering - A Closed-loop relation 5.0 Summary 6.0 Security 7.0 Acknowledgement 8.0 Authors Addresses 9.0 References Cheng et. al. [Page 2] 1. Introduction and Background 1.1 Long-term and Short-term Internet Engineering Processes Figure 1 illustrates the long-term and short-term Internet Engineering model for a network. Traditionally, Network Planning is used to build a physical network with static topology. Once a network is installed, it is expected to stay for a long period of time. Traffic Engineering is then used to optimize network resource utilization for traffic demands [TE-ash]. It is the optimization of network resources based on a fixed network topology. It allows traffic to utilize network resources in an efficient way. Network Planning is a long-term process to optimize network resources for long-term traffic growth, while Traffic Engineering is a short-term process to optimize network resources for short-term traffic fluctuation. Traffic NE, NE, fiber, fiber +-------- data Rsc. Pool w. uninstalled NEs & fiber | | | | | | | +----------v----------+ +---------v---------+ ^ | Traffic Engineering | | Network Planning | | | (Short-term) | | (Long-Term) | | +----------+----------+ +---------+---------+ | | | | | | | v v | ---------------------------------------------- +--<----- Network w. Resources (capacity) Physical Topology ---------------------------------------------- Figure 1 Long-term and Short-term Internet Engineering Processes 1.2 Conventional IP Traffic Engineering and Limitation Traffic Engineering, in short, is putting traffic where the capacity is [ASTN-LN01]. This is contrasted with Network Engineering, which is establishing capacity where the traffic needs it. [TE-FRWK] has a detailed definition of Internet Traffic Engineering. "Internet traffic engineering is defined as that aspect of Internet Network Engineering dealing with the issues of performance evaluation and performance optimization of operational IP networks. Traffic Engineering encompasses the application of technology and scientific principles to the measurement, characterization, modeling, and control of Internet traffic [rfc2702]." Cheng et. al. [Page 3] In traditional IP traffic engineering, the underlying network topology is assumed to be relatively static [rfc2702, te-MPLS-diff]. In particular, the links connecting the IP/MPLS routers in the backbone are typically provisioned for a long period of time, due to the difficulties of rapid reconfiguration of physical links. The main objective of IP/MPLS traffic engineering is efficient mapping of traffic demands onto the network topology to maximize resource utilization while meeting QoS constraints such as delay and packet loss. The traffic demands may be obtained from measurement, projection, customer prospects, Service Level Agreement (SLA), or combination of the above. The mapping may be done in a multi-period fashion corresponding to diurnal or weekly patterns. In a MPLS network, the mapping is facilitated by establishing explicit label switched paths (LSP). In a connectionless IP network, the mapping can be attempted by adjusting IGP weights. The key assumption of current Internet Traffic Engineering is that the total network resources for IP routers to use are fixed. This assumption may be changed by leveraging configurable circuit-switched networks. Instead of a fixed amount of resources, the allocation of total network resources for IP routers can now be flexible. The IP network can get more resources from its provider network, which may be a transport network that carries IP traffic. It can also return resources to its provider network. The configurable circuit-switched network is a flexible transport network, which provides a flexible network resource pool. However, the resources (bandwidth, links, ports) of the pool are limited (if there were unlimited, there would be no point of switching). IP traffic has the following characteristics: - IP traffic is variable-speed. - IP traffic is bursty. - Destination address is frequently changed (e.g., net surfing). - A variety of QoS is needed depending on application. A fixed topology with a fixed amount of bandwidth for Traffic Engineering does not offer sufficient flexibility to optimally deal with the characteristics mentioned above, because IP traffic characteristics change dynamically from moment to moment. The Network Engineering function described herein offers the potential to overcome this limitation via dynamic reconfiguration of links according to the status of the networks, a property that is supported by dynamically configurable circuit-switched networks. 1.3 Layered Network Before we discuss the solutions that this paper identifies, we briefly discuss the notion of a layered network architecture. The concept of layering enables a separation between a network's logical topology and its physical routes and resources. In particular, the process for setting up connections becomes generic to all layers. Cheng et. al. [Page 4] Layering refers to a logical decomposition of a network into a number of distinct layers, which have a well-defined set of associated characteristics. A layer network may be defined in terms of a -tuple (or signal type). For example, STS-1 is a different layer network from STS-3c is a different layer network from STS-12c. Flexibility can occur at each of these layers, and is usually discussed in terms of a network's "switching" granularity. For example, an IP router's switching granularity is an IP packet; a SONET cross-connectËs switching granularity might be an STS-1; a photonic cross-connect's switching granularity is a wavelength (optical channel). Figure 3 shows some switching hierarchies, where the highest is SDM (Space Division Multiplexing), the next one WDM (Wavelength Division Multiplexing), etc. There are different types of Network Elements (NEs) that can include components of different network layers depending on the targeted application (refer to Figure 2). | | +-------+ ___/ \___ __/ \___ / \ +-------------------------+ | +--STS-1-+ | | | \ / | | | /|/ \/ \ |\ | | | || /\ |\| | | | /|/| || _/ \_ | | |\|\ | || / \|+--------+ |/ | || || | +-STS-3c-+ | || =====| | /|| \ / | |\ | |======= =====| |-| || \/ --| |-| |======= || | | || /\ | | | | || || | \|| _/ \_ | |/ | || || | +--------+ | || || | +-STS-12c+ | || | \|\ /|| \ / | |\ /|/ | | | || \/ |/| | | | | |\ /\ / | | | | \|| _/ \_ | |/ | | +--------+ | +-------------------------+ +--------+ | | Figure 2 An example of NE Cheng et. al. [Page 5] PDM (packet/cell) ____________________________________________________________ TDM LO (DS-1, VT1.5 etc.) | | | ______________________________V_______ | | TDM HO (DS-3, STS-1 etc.) | | | __________________________________V______V____ | WDM | | ______________________________________V_____________V___ SDM | __________________________________________V________________ Figure 3 Switching Hierarchies Another way of describing Network Engineering is that it is the process of obtaining connections from a provider network (note that in this discussion we make no assumption that the user and provider are from different organizations) in support of aggregates of traffic units (e.g., packets) in the user network. When the resources of the provider network are switched 1:1 with the traffic units in the user network, then no Network Engineering function is involved. At this point, we should introduce and distinguish between the concepts of layering and aggregation. When layering involves an adaptation of more than one client signal into a server signal (i.e., multiplexing), then it is an aggregate of client signals and the layer relationship can be managed by Network Engineering. When layering is an adaptation of a single client into a server signal (e.g., DS3 into STS-1) then it is not a spacial aggregate. If it is also not a temporal aggregate then the server connections are set-up and torn down one-to-one with client traffic units, even though thereËs a layering relationship between them. When there is no layering relationship, or when the client/server relationship is 1:1, there is no spacial aggregation, but there may still be temporal aggregation. Temporal aggregation is when one provider connection is kept up for the multiple user traffic units. Thus, Network Engineering is used to control spacial and/or temporal aggregates of traffic. Another way to identify an aggregation boundary is to observe bandwidth time product (BTP) mismatches between traffic units in the user and provider network. The BTP of the traffic units in the user network is of finer granularity than the BTP of the traffic units in the provider network. Multiple IP traffic units are interleaved in time into a single provider connection, and the holding time of an IP traffic unit is shorter than the holding time of an optical connection. Tearing down one or more IP associations may only result in the reduction of bandwidth utilization in an optical connection. Tearing down an optical connection can only be done if all carried user IP associations are terminated or re-routed through another optical connection. Cheng et. al. [Page 6 Note that the establishment of an IP link usually incurs a certain cost to the user IP network. It is important that the cost is within a planned budget. This may involve re-optimization and tearing down some IP links. The provider network has limited resources, so it is important that the total value of the links established from the provider network is maximized. As another consideration, databases of resources lower than OSI layer 3 are traditionally easy to manage since those resources are mostly static and changed on a order-by-order basis. However, for configurable circuit-switched networks, where resources lower than layer 3 are dynamically changed, it brings a completely new paradigm in how to manage resource databases. Easy and manageable database systems are desired. 1.4 Conventional IP Network Planning and Limitations In conventional IP network, static topology is normally set up at the network planning time according to the most stringent traffic demand patterns in a given duration. Because of forecasting mismatch and the inability of changing the network topology fast enough, Traffic Engineering limits itself to achieving higher network resource utilization. Observe that the static topology of the IP network introduces limitations. Consider first the case when traffic demands are well estimated a priori. In this case, provisioning needs to be done according to the most stringent traffic demand patterns in a given duration (even under the assumption that the best TE plan, which can take advantage of multi-period traffic patterns if used). Therefore, network resources remain under-utilized when traffic demands are light (e.g., during weekends or evenings) since provisioned links cannot be released easily. When the traffic demands cannot be estimated accurately, network planning may not be done efficiently. For example, if link provisioning is inadequate, traffic demands can exceed the required network resources. This may occur because of forecasting mismatch or the inability of provisioning the network fast enough to meet the growth in resource requirements. On the other hand, it may be necessary to over- provision the network due to the difficulties of forecasting the traffic growth accurately. Furthermore, if the provisioning cycle (i.e., the time it takes to add resources to the network) is long, resource provisioning needs to take into account the projected traffic growth until the beginning of the next cycle. Consequently the extra network resources may be significantly under-utilized at the early part of a cycle if the projected growth is large. Cheng et. al. [Page 7 With the advent of circuit-switched networks, the client IP networks can leverage the dynamic nature of the underlying transport topology. It can achieve higher network resource utilization by dynamically setting up and tearing down IP links in response to critical traffic conditions (e.g., overloaded ports, overloaded routers, etc.). This overcomes the limitation associated with Traffic Engineering, which is based on a given network topology. Note that the leverage of a configurable circuit-switched network is complementary to the use of effective Traffic Engineering schemes for the IP/MPLS networks. Indeed, it is best to use an effective Traffic Engineering scheme to optimize the traffic distribution over the existing network resources before attempting to dynamically change the underlying transport network resources. We present a first-draft description of Internet Network Engineering functionality and capabilities to address the limitation of classic IP Traffic Engineering and Network Planning. 1.5 Terminology In this section we present a glossary of terminology that we use in this paper and follow it up with a description of what we term ŸNetwork Engineering÷ as a solution. Traffic Engineering “ Putting traffic where the capacity is [ASTN-LN01]. Network Planning “ Network Planning function makes decisions on installing fiber and Network Equipment (NE) for a network based on forecast long-term traffic demands. Network Engineering “ Establishing capacity where the traffic needs it. Link-connection “ one channel between subnetworks. That is to say one unit of capacity between subnetworks. A link connection may or may not be carrying traffic. Link “ a set of all link connections between a pair of subnetworks that are equivalent for the purposes of routing. This definition allows a link to have zero link connections. An IP link “ a unidirectional physical connection between two IP routers. Note that usually IP links are bi-directional. Any request for a bi-directional link can then be accomplished by two requests, each for an uni-directional link. It could be an optical network connection through the underlying circuit switched network. In this case, an IP router has logical adjacency with its IP router peer and physical adjacency with the circuit switch that it is connected to. A configurable IP link-connection “ an IP link, which can be dynamically setup and released within the circuit-switched layer, thus may actively change the IP network topology. Cheng et. al [Page 8 A non-configurable IP link-connection “ an IP link, which cannot be dynamically setup and released within the circuit-switched layer. User Network “ an IP network, which can request for dynamical setup and release a link within the circuit-switched Network. Provider Network “ a circuit-switched network, which can dynamically setup and release a link for the user network. Capacity “ The ability of a link or link connection to carry traffic; this doesnËt imply the traffic is present or not present. Traffic “ the presence of traffic in the capacity of a link or link connection. Overloaded port “ A port which has more traffic than the port capacity Overloaded router “ An IP router which has no more port capacity for additional traffic Configurability “ the ability of the IP layer to add a link, delete a link, and modify the parameter(s) of a link. Cheng et. al [Page9] 2. Network Engineering In this section, we will elaborate - ŸWhy÷ a network needs a Network Engineering function, which serves as the motivation of our work, - ŸWhat÷ Network Engineering is, - ŸHow÷ a Network Engineering function adds, deletes, modifies circuit-switched links, - ŸWhen÷ to activate a Network Engineering function, and - ŸWhere÷ in the network the NE modifies the user network topology via modifying circuit-switched link configurations. 2.1 Why How can Traffic Engineering, which was designed for static topology, cope with conventional dynamic network topology? We will introduce here, Network Engineering, to dynamically Ÿadjust÷ network topology for the traditional Traffic Engineering. The network topology considered by the Network Engineering function consists of a user network and a provider network. Hence, it is a logical network topology. Figure 4 illustrates the engineering hierarchy of Internet Engineering Processes for a network to satisfy traffic demands from both short-term and long-term perspectives. Traffic links from Physical Resrc pool +---- data Provider Network NE, fiber, etc. | | | | | | | | | +-----v----+ +-------v--------+ +-----v-----+ ^ | TE | unresol |Network Engnring| unresol | NP | | |(Shrt-trm)|--------->| (real-time) |---------->|(Long-term | | +-----+----+ traffic +-------+--------+ traffic +-----+-----+ | | | | | | | | | v v v | ----------------------------------------------------------- +--<----- User Network w. Resources (capacity) logical Topology ----------------------------------------------------------- Figure 4 Engineering Hierarchy Cheng et. al. [Page 10] We consider a network consisting of a set of IP routers that are inter- connected either through fixed links or through a set of circuit switches. That is, there are two logical layers, i.e., IP layer and the underlying circuit-switched layer. These two logical layers interact in a user-provider relationship. The circuit-switched layer can be thought of as merely providing dynamically configurable links for the user network. To address the short-term fluctuation of more/less capacity needs for traffic demands, the Network Engineering function takes/deletes more links from the provider network, in a near real-time fashion. It is a function defines how the user network uses the provider network. It changes the logical user network topology by adding/deleting circuit- switched links. It supports better and flexible network topology for a classic Traffic Engineering function, without having to change existing Traffic Engineering functions. 2.2 What Network Engineering is a part of a larger Network Planning function, which makes decisions on installing fiber and Network Equipment (NE) for a network based on forecast traffic demands. The user network and the provider network will have independent Network Planning processes. The computation for Network Planning is typically done offline since it involves searching on multi-dimensional solution spaces for long-term traffic growth. Network Engineering (NE) is an automation of the part of Network Planning of the user network that can be supported from a provider network. NE provides a more real-time topology changes according to existing and near-term traffic demands. It assists current Traffic Engineering function to better optimize the network resource utilization via topology changes. Network Engineering that we describe here can be thought of as an automatic network provisioning procedure. It is to allocate/de-allocate provider network resources to be used by the user network. In other words, the intent of Network Engineering is to put the capacity where it is needed by the traffic. Or conversely, to remove the capacity where the traffic has diminished. Internet Network Engineering process is concerned with the management of the (virtual) links in the IP network based on the traffic demands that are expected between any two routers in the network (pro-active) and the actual traffic demands (re-active). The fundamental issue is design, ranking, and choice of an IP link, which is to be established by the circuit-switched layer. It is also worth being able to calculate the number of current open connections and the maximum number of open connections allowed (due to limits in capacity). Network Engineering may also add a link or modify a link (via adding or deleting link connections) on request of a traffic engineering process (if compliant with policy), which is not able to complete a connection in the absence of available bandwidth. Cheng et. al. [Page 11] 2.3 How To data network, conventional transport networks are "static". They are provisioned as leased line services. Classic IP TE considers a fixed IP layer topology. A distributed transport control plane can provide an on-demand automatic choice and configuration of underlying circuits to be used in the IP layer. The new control plane of the transport network transforms the "dummy pipes" into a foundation for switched connection services, a service platform that can provide an automatic link provisioning function rather than simple transmission. This new function enables numerous network services. This new paradigm of adding and deleting capacity of the transport network has profound impacts on conventional data networking. It changes the data network's basic assumption that the transport network consists of fixed pipes, i.e., a static network. In the new paradigm, the transport network can be considered as a configurable backbone. Packet switches can dynamically adjust the network topology according to end network traffic to optimize overall network performance. Much effort has been devoted to the control plane design. That is, Ÿhow÷ to automatically provision a link. Note that Network Engineering considers user link configuration within a provider domain, hence the choice of the signaling protocol, which will carry parameters needed for a Network Engineering function, is not relevant for the time being. Our work, on the other hand, focuses on Ÿwhen÷ and Ÿwhere÷ to automatically provision a link. In this sense, this work complements the work that is progressing in the CCAMP working group and sub-IP area in general. 2.4 When In order to change the logical network topology, Network Engineering has some basic operations, which include adding a link, deleting a link, and modifying a link from the circuit-switched layer. We will elaborate these NE operations in detail later in section 3.6. Network Engineering also monitors the layered network to determine - when a new link (or link connection) should be added to the IP network, - when an existing link (or link connection) should be modified (i.e. increased or reduced), or - when a link (or link connection) should be released. Network Engineering is a new functional module in addition to conventional Traffic Engineering. This function is independent of any network models (e.g., peers, overlay, or augmented)). Cheng at. al. [Page 12 When should a Network Engineering be invoked to perform the aforementioned operations? We will define a set of trigger mechanisms to invoke the Network Engineering function in section 3.4. 2.5 Where The connection resources available from the provider network are limited. Network Engineering must choose where the most beneficial links to add (or to delete/modify). Network Engineering should also monitor the bandwidth demands from a source/destination relationship in order to identify the traffic patterns in the network. Where necessary, a user network's topology can be optimized by e.g. adding direct links between two heavily overloaded nodes. Decisions are taken based on the SLA between the user and the provider and the policies set by the provider network operators. We will address how Network Engineering optimally choosing Ÿwhere÷ to add/delete/modify links in section 3.5 (Decision-Making Process). Cheng et. al. [Page 13] 3. Architectural Element of Network Engineering 3.1 Architectural Overview Figure 5 shows an architecture for Network Engineering. A decision- making functional module is triggered by some triggering mechanisms (e.g., proactive input or reactive input), and takes the current network status (e.g., traffic information and topology information) to compute the output. The input of the Network Engineering process is the current measured state of the user network. Proactive information makes possible traffic shaping for particular purposes, e.g., busy hour, network overload threshold, etc. The proactive mechanism might also use the bandwidth advertising from the circuit network in order to pre-estimate the links to establish. The bandwidth advertising might be invoked periodically. Reactive information can trigger the establishment of new links upon measured performance. The output of the Network Engineering phase is a target IP link, which can be configured as a switched circuit. In other words, the result of the Network Engineering is to add a link, delete a link, or modify parameters of a link (i.e., via adding/deleting link connections). +------------------------+ | IP Traffic Information | +------------|-----------+ +-----------+ | | Proactive | +------------v-----------+ +--------------+ | Input | Triggers | | | Operations | +-----------+--------->| IP Network Engineering +-->|(Getting res. | | Reactive | | (decision-making) | | from Provider| | Input | +------------^-----------+ | Networks) | +-----------+ | +--------------+ +------------|------------+ | IP Topology Information | +-------------------------+ Figure 5 Network Engineering Architecture 3.2 Traffic Information An IP network has network internal information and network external information, that can be applied to proactively or reactively triggering an IP Network Engineering function. Cheng et. al. [Page 14] The network internal information deals with the traffic statistics collected over a certain period of time, based on which particular traffic flows can be established according to the predictable traffic patterns. The internal information can also trigger the reconfiguration of the existing traffic patterns (i.e., Traffic Engineering) for more efficient service-level guarantees or network throughput. If this fails, then Network Engineering is attempted. This information is reactive but can be also used predictively by assuming cyclic traffic patterns. The network-external information deals with edge-to-edge traffic requests. If such a request identifies an amount of resources that is needed to assess the need for new IP links. This information can be neither estimated nor measured with complete accuracy. An example of such information might be a new contract or a new traffic request. In this case, for example, the predictable traffic patterns as defined by the long-term traffic measurements have to be revised. Traffic statistics maybe stationary or non-stationary. For stationary traffic statistics data, using periodically sampled data to construct a probability distribution function (PDF) may be useful. This PDF function is then a good input to the Network Engineering function. For non-stationary traffic, more data (e.g., bandwidth utilization, etc.) is needed to perform the Network Engineering function. Examples of internal information are: packet loss [rfc2680], router overload, and port overload [newsome01]. Examples of external information are: SLA violations, or new SLAs. Such information triggers the network design to automatically configure an IP link. Depending on the triggering information, configuration can be adding, deleting circuit links, or modifying existing traffic parameters. Both external and internal information may result in connection requests. The network-internal information are obtained based on the measurements within the user network, and the network-external information can be estimated based on the service contract information related to the new traffic. 3.3 Network Topology Information 3.3.1 IP Network Topology Information In order to perform Network Engineering function, IP routers need to have the following topology information: (1) IP routes These are conventional routes to IP peers. They are standard IP routes to various destinations learned through conventional IGPs (e.g., ISIS or OSPF) and EGPs (e.g., BGP). Cheng et. al. [Page 15] (2) Forwarding Adjacencies (FA) Forwarding Adjacencies are IP (i.e., Layer 3) neighbors that are connected by an underlying circuit-LSP through a provider network. These connections are dynamic in nature and can be set up and torn down as needed. Both IP routes and Forwarding Adjacencies are used for packet forwarding. Such information is disseminated using IGP or EGP routing protocols. A provider network treats traffic carrying this kind of information as user traffic. (3) Potential FAs (PFA) PFAs are remote CAGs [BGP/GMPLS]. They represent NEs that belong to the same client network or to different client networks, which allow NEs in the local network to connect to it. The NEs represented by a PFA are not connected yet but can be connected via the provider network. Such a connection is a direct connection at the IP layer with an underlying circuit connection that spans multiple ASËs. Information about PFAs can be disseminated through BGP extensions in the provider network, directory service, or yellow page mechanisms. (4) Accessible IP Routes through the PFAs This provides information about all the IP routers reachable through a PFA endpoint. Such information can be used for the IP Network Engineering function to determine the source and destination of the circuit connections. Although this document does not address the issue of how this information is disseminated, we note here that various options exist for doing so. These options include dissemination through established user network connections or a dedicated user network control plane network. 3.3.2 Server Network Topology Information For the purpose of IP Network Engineering, it understands the IP network topology, and does not require circuit network topology information to compute the output. Since it does not require network topology information, Network Engineering is independent of any network models. Existing network models are all able to support a Network Engineering function. Cheng et. al. [Page 16] 3.4 Triggering Mechanisms There is a set of triggering mechanisms, which defines Ÿwhen÷ to activate the Network Engineering function. 1. Direct user request “ User should be able to request a Network Engineering function. For example, an operator may need to do on- line or off-line capacity planning or business modeling for his network. Network Engineering provides a most efficient network topology as a basic standing point for Traffic Engineering to consider. 2. Response to adding a link “ It is among Network Engineering capability to add a link to the network. The request maybe a result of future Network Planning of the network or an unexpected traffic loads. Such a request can also come from Traffic Engineering. 3. Response to deleting a link “ It is among Network Engineering capability to delete a link from the network. The request maybe a result of future planning to an expected reduced traffic loads. Also such a request can come from Traffic Engineering. 4. Response to modifying a link “ It is among Network Engineering capability to modify a link to the network. If the request is to increase the capacity of a link, it maybe a result of future planning of the network or an unexpected traffic loads. If the request is to decrease the capacity of a link, it maybe a result of future planning or reduced traffic loads. 5. Congestion condition “ Congestion problem is probably one of the most studied problems in contemporary Internet networks. When a congestion condition is not properly handled, it will significantly degrade network performance. Traffic Engineering should be able to detect a congestion condition and re-engineers the network. Upon its limit, then activate Network Engineering function. 6. Router overload “ Upon a router overload condition, it is desirable to find an alternative route for traffic to bypass the overloaded router. 7. Link overload “ Upon a link overload condition, it is desirable to add more capacity parallel to the congested link. If this is not feasible, it is desirable to increase capacity over a different route and redirect some of the existing traffic to the newly added link to relief the overloaded condition. However, this has to be done carefully as existing traffic should not get all redirected on the newly added link, leading to a mere shift in the location of the link overload condition. 8. Maintenance “ In the maintenance phase, it is also desirable to run Network Engineering function. 9. Failure “ Upon a failure situation (e.g., link failure, network failure, etc.), it is desirable to run Network Engineering function. Cheng et. al. [Page 17] 10. Routine “ If none of the above condition occurs, it is desirable to run the Network Engineering function on a regular basis. Routinely running Network Engineering function can insure that the network is making the most efficient use of it network resources. The interval of the routine depends on the network sizes and traffic loads of the network. 3.5 The Decision Making Process The decision-making process decides Ÿwhere÷ to add/delete/modify circuit-switched links. The Network Engineering decision making module will be involved in the path computation depending on the knowledge of the layered network. The problem of selecting a new link that best satisfies demands for bandwidth is a multi-dimensional optimization problem, i.e. one that is mathematically "hard" and probably not possible to compute perfectly, even in principle. This problem can be decomposed into two parts: link design, and link choice. The design phase identifies a set of links that would satisfy some needs; whereas the choice phase chooses a final set of links based on maximizing the total value. First of all, link design provides a set of potential links that meets the following criteria: 1. links that are configurable in the circuit-switched layer (e.g., with free ports, and free capacities), and 2. links that has enough capacity to meet traffic demands. During the design phase, we distinguish different types of links based on the number of hops the link bridges. The output of link design can be zero or more candidate links to be Ÿconfigured÷ (e.g., add, delete, modify) in the circuit-switched layer. If no link is selected, it means blocking in the circuit layer. This can be unavailable ports or insufficient capacity. Secondly in the choice phase, a set of criteria, for choosing a final link amongst a set of candidate links, is needed. The criteria are based on the consideration of bandwidth, distance, and duration. Link choice ranks the set of potential links, selected from the above link design, according to the following set of parameters: 1. number of hops bridged, 2. bandwidth utilization of the new link, 3. value (or more correctly, rate of return on investment), 4. number of switched-ports spared, or 5. etc. Cheng et. al. [Page 18] It then requests the reduced set of maximum value from the potential links. One implication of the link choice is that a link design may get refused for reasons of insufficient value. The value of any link requested can be expressed by 1. multiple attributes (as listed for link choice) or 2. a single metric function of multiple parameters (e.g. bandwidth, time, distance, etc.). That is, a single valued function of the parameters. To avoid the problems of multi-dimensional optimization as described above, a single metric can be chosen as precedent, based on which a choice is made if all other requirements are satisfied. Administrative policies, constraints, directives, and guidance can also be weighted and contributed to the single metric. Some will be simple boolean, go-nogo parameters, so if we're converting to cost, nogo parameters might have to be represented as an infinite cost value to force rejection of the option. Other parameters can be given weights and factored in to the single metric for the multi-dimensional tradeoffs. So far we have been focussing on shortest-path costing only, but we realize that Network Engineering must support additional administrative inputs. Within the circuit layer a choice between candidate switched circuits can be made for the IP network in order to have the maximum advantages of a dynamic configuration. All circuits, which may be configured, are expected to be within the bounds requested by the IP layer, otherwise the request is rejected. 3.6 Link Operations As we have mentioned earlier, in order to change the network topology, Network Engineering has to be able to add, delete, modify circuit- switched links. We distinguish the following four types of link operations: (1) add a link Adding a link is to add a potential route with zero capacity. This type of link bridges two or more hops of existing IP links. Note that it changes the IP network topology without adding new capacity. Routing tables will need to be updated. Consider an IP (user) over an circuit (provider) network, where traffic flows from the source node A to the destination node Z. The following figure shows adding a new link p-s. Cheng et. al. [Page 19] A p q r s Z O------O......O......O......O-------O :....................: (2) add a link connection Adding a link connection is to increase the capacity of an existing link. This new link (connection) bridges a single hop of IP link. It parallels an existing link and increases capacity of this link without changing the network topology. The following figure shows adding a new link connection r-s to an existing circuit-switched link r-s. A p q r s Z O------O......O......O:::::::O-------O (3) delete a link connection This operation is only allowed when the traffic of that link connection is zero. Deleting a link connection is to decrease the capacity of a link. It may reduce the capacity of a link to zero. (4) delete a link This operation is only allowed when the number of link connections in the link is zero. That is, when the capacity of the link is zero. To delete a link means to remove a potential route with zero capacity. For the purpose of Network Engineering, if we need a new link with some capacity, we need to add a link (with zero capacity), and then add a new link connection to increase the capacity of this new link. If we need to delete a link, we will need to delete all the link connections in that link before deleting the link. Cheng et. al. [Page 20] 4 Network Engineering and Traffic Engineering - A Closed-Loop Relation This section illustrates a closed-loop relationship between Network Engineering and Traffic Engineering processes (Figure 6). A closed-loop link provisioning process would automatically configure circuits in the circuit-switched network [close-loop, oif-197]. The cycle is closed because changes in bandwidth or topology affect the state of the user network. --------------------------------------- +--> the state of the user network | ---+---------^------------------+------ | | | | | topology& | topology & | traffic route Triggers | updates updates | | | | | topology | | | updates +-v----------+ +------v------+ | | Traffic | traffic | Network | | | Engineering|--->------| Engineering |-+ ^ +------------+ problems +-------------+ | | | +----+------+ | | topology | | | discovery | | +----+------+ User Network --------+---------------------------------------------+--------- | Provider Network | | Operation Operations results requests | +------------+ | +---<----------| Operations |-----<-----------+ +------|-----+ ^ | | v ---------------|-------------------- the state of the provider network ------------------------------------ Figure 6 A Closed-loop Procedure for NE and TE A user network may consist of multiple sub-networks. For the purpose of discussion the relationship between Network Engineering and Traffic Engineering, we treat these sub-networks as one user-network. Hence the Ÿuser network÷ in Figure 5 can be single or multiple sub-network. We do not wish to further complicate our discussion here. Cheng et. al. [Page 21 Note that in Figure 5, Traffic Engineering still performs its function without having to interact with Network Engineering. Traffic Engineering gets traffic demands and optimally maps them into the existing network topology. Upon reaching the Traffic Engineering limit when traffic problems occur (e.g., insufficient capacity, etc.), Traffic Engineering then turns the problem to Network Engineering for dynamic topology reconfiguration. Network Engineering can also be run routinely. Traffic Engineering mainly decides Ÿwhen÷ to use Network Engineering. Network Engineering computes Ÿwhere÷ to change the topology. 5. Summary In this paper, we have outlined an early draft for an Internet Network Engineering framework. Traffic problems, which occur due to the limitations of Traffic Engineering, can trigger Network Engineering processes to further optimize the network performance. Network Engineering is to automatically select a link for addition, deletion, and modification to address Traffic Engineering limitation. Network Engineering and Traffic Engineering form a closed-loop process to optimize the network resources. We also note that the specific choice of technologies is not essential to a Network Engineering process. It is important to recognize the problem is recursive. The closed-loop process for Network Engineering and Traffic Engineering is applicable to any user-provider relationship networks. 6. Security Issues This document raises no new security concerns for MPLS signaling. 7. Acknowledgement The authors would like to express thanks to Siva Sankaranarayanan for his fruitful comments and technical discussion. Cheng et. al. [Page22] 8. Authors Addresses Lily Cheng John Ellson Lucent Technologies, Inc. 101 Crawfords Corner Rd Holmdel, NJ 07733 Email {lilycheng, ellson}@lucent.com Admela Jukan Vienna University of Technology Institute of Communication Networks Favoritenstrasse 9/388 A-1040 Vienna, Austria Email admela.jukan@tuwien.ac.at Maarten Vissers Lucent Technologies, Inc. Botterstraat 45, Postbus 18 Huizen, 1270 AA Netherlands Email mvissers@lucent.com Yangguang Xu Lucent Technologies, Inc. 1600 Osgood St. North Andover, MA08145 Email xuyg@lucent.com Iraj Saniee Lijun Qian Anwar Elwalid Indra Widjaja Lucent Technologies, Inc. 600-700 Mountain Ave. Murry Hill, NJ07974 Email {iis, ljqian, anwar, iwidjaja}@lucent.com Yang Cao 10 Elizabeth Dr. Chelmsford, MA 01824 Sycamore Networks Email yang.cao@sycamorenet.com Susumu Yoneda (yone@japan-telecom.co.jp) Hirokazu Ishimatsu (hirokazu@japan-telecom.co.jp) Takeshi Hashimoto (take_h@nts.japan-telecom.co.jp) Yoshihito Oyama (oyama@jtlab.net) Shinya Tanaka (t@nts.japan-telecom.co.jp) Japan Telecom Co., Ltd. 2-9-1 Hatchobori, Chuo-ku Tokyo, 104-0032 Japan Cheng et. al. [Page 23 Li Mo Metera Networks Inc. 1202 Richardson Drive, Suite 100 Richardson, TX 75080 Email li@metera.com Ed Snyder PhotonEx Corporation 200 MetroWest Technology Drive Maynard, MA01754 Email esnyder@photonex.com Ananth Nagarajan Sprint 9300 Metcalf Ave Overland Park, KS 66212. ananth.nagarajan@mail.sprint.com 9. References [newsome01] Newsome, G. "IP Traffic Engineering resulting in Optical Layer Connections", IETF draft, draft-newsome-mgmtplanerqmts-00.txt, November 2000. [TE-ash] Ash J. ŸTraffic Engineering & QoS Methods for IP-, ATM-, & TDM-based Multiservice Networks÷, IETF draft, draft-ietf-tewg-qos- routing-01.txt, March, 2001. [rfc2702] Awduche D. et. al. "Requirements for Traffic Engineering over MPLS", RFC2702, September 1999. [ASTN-LN01] Maarten Vissers, ŸASTN Layer Network Architecture÷, ITU-T CD, SG-15, CD-ASTN-LN01, May,2001. [te-MPLS-diff] Le Faucheur et. al. "Requirements for support of Diff- serv-aware MPLS Traffic Engineering", IETF draft, draft-ietf-mpls-diff- te-reqts-00.txt, November 2000. [te-frame] Awduche, D. et. al. "A framework for Internet Traffic Engineering" IETF draft, draft-ietf-tewg-framework-02.txt, July 2000. [rfc2680] Almes, G. et. al. "A One-way Packet Loss Metric for IPPM," RFC2680, September 1999. [closed-loop] Ellson, Cheng, Jukan, et. al. ŸClosed-Loop Automatic Link Provisioning÷, IETF draft, draft-ellson-ipo-te-link-provisioning- 00.txt, March, 2001. [oif-197] Ellson, Cheng, Jukan ŸClosed-loop Link Provisioning for Bandwidth Brokering÷, OIF-197, May, 2001. Cheng et. al. [Page 24]