Internet Draft Alex Audu Expiration: September 2003 Alcatel USA Inc. File: draft-gopal-forces-fact-03.txt Working Group: ForCES Ram Gopal Nokia Hormuzd Khosravi Intel Chaoping Wu Azanda Network Devices March 2003 ForwArding and Control ElemenT protocol (FACT) draft-gopal-forces-fact-03.txt Status of this Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026[1]. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as ``work in progress.'' The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. Conventions used in this document The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [2]. Abstract This document defines a FACT protocol that is suitable for communicating between Forwarding Element and Control Elements inside a network element. This protocol addresses all the requirements described in Forces [3] requirements document. This document also describes some of the architecture that FACT may leverage during the protocol operation. Gopal,Audu,Wu,Khosravi Expires September 2003 [Page 1] Internet Draft Forwarding and Control Element protocol March 2003 Table of Content 1. Definitions.....................................................3 2. Introduction....................................................5 3. Protocol Overview...............................................6 3.1. Independence from Interconnect/Transport Layer................7 3.2. Reliability And Fail-Over Model...............................7 3.3. Retransmission................................................9 4. Message Overview................................................9 4.1. Protocol Message Header structure............................10 4.1.1. Version....................................................10 4.1.2. Message Classes and Types..................................10 4.1.3. Length.....................................................12 4.1.4. CE Tag.....................................................12 4.1.5. FE Identifier..............................................13 4.1.6. Priority (P) Bits..........................................13 4.1.7. Transaction Sequence Number (TSN)..........................13 4.2. Service Data or Payload Structure............................13 5. FACT Messages..................................................15 5.1. Association and Connection (CA) Messages.....................15 5.1.1. Join Request...............................................15 5.1.2. Join Response..............................................16 5.1.3. Leave Request..............................................17 5.1.4. Leave Response.............................................18 5.2. Capabilities Control (CAPCO) Messages........................19 5.2.1. Capabilities Request.......................................19 5.2.2. Capabilities Response......................................19 5.2.3. Configure Logic Components.................................20 5.2.4. Configure Logic Components Response........................22 5.2.5. Topology Request...........................................23 5.2.6. Topology Response..........................................23 5.2.7. Query Request..............................................24 5.2.8. Query Response.............................................25 5.3. PE Maintenance Messages......................................26 5.3.1. Protocol Element UP (PEUP).................................26 5.3.2. Protocol Element Up Acknowledge............................27 5.3.3. Protocol Element Active (ACT)..............................27 5.3.4. Protocol Element Active Acknowledgement (ACT-ACK)..........28 5.3.5. Protocol Element Inactive (INACT)..........................28 5.3.6. CE or FE Inactive Acknowledgement (INACT-ACK)..............29 5.3.7. Protocol Element Down (DOWN)...............................29 5.3.8. Protocol Element Down Ack (PEDN-ACK).......................30 Gopal,Audu,Wu,Khosravi Expires September 2003 [Page 2] Internet Draft Forwarding and Control Element protocol March 2003 5.3.9. Heartbeat (or Health check)................................30 5.3.10. Heartbeat Acknowledgement (FE response to heart beat request)..........................................................31 5.4. PE Traffic Maintenance (PETM) Messages.......................31 5.4.1. Control packet Redirect to CE for processing...............31 5.4.2. Control Packet Redirect Acknowledgement....................32 5.4.3. Control Packet Forwarding through specific output ports....32 5.4.4. Control Packet Forwarding Acknowledgement..................33 5.5. Event Notification Messages..................................33 5.5.1. Event Register.............................................34 5.5.2. Event Register Acknowledgement.............................34 5.5.3. Event De-Register..........................................35 5.5.4. Event De-Register Acknowledgement..........................35 5.5.5. Port-Link Status Change and Event Notify...................35 5.5.6. Generic FE Status Change and Event Notify..................36 5.5.7. Error Event Report.........................................37 5.6. Vendor Specific Function Message Handling....................37 5.6.1. Vendor Specific Data (VS-DATA Request).....................38 5.6.2. Vendor Specific Data Ack (VS-Data Ack).....................38 6. Procedures for FACT Protocol...................................38 6.1. CE and FE State Maintenance..................................38 6.1.1. CE and FE States...........................................39 6.1.2. NE States..................................................40 6.2. State Maintenance Procedures.................................42 6.2.1. PE-UP (Protocol Element Up)................................42 6.2.2. PE-DOWN (Protocol Element Down)............................43 6.2.3. FACT Version Control.......................................43 6.2.4. PE-ACTIVE..................................................44 6.2.5. PE Inactive................................................44 7. Example Scenarios..............................................45 7.1. Establishment of Association between CEs and FEs.............45 7.2. Steady State Communication...................................46 7.3. Two FEs in NE (1+1 Over-ride Redundancy Mode)................47 7.4. Two FEs in NE (1+1 sparing, load-sharing Mode)...............47 7.5. CE Fail-over Support for 1+1 Redundancy......................47 8. Architecture support for FACT protocol.........................48 8.1. Security.....................................................48 8.2. High availability support....................................48 8.3. Access Control to FE.........................................49 8.4. Configurable parameters......................................49 8.5. Management Interface to FE and CE through FE-Manager and CE- Manager...........................................................49 9. References.....................................................50 10. Acknowledgments...............................................50 11. Authors' Addresses............................................50 1. Definitions Gopal,Audu,Wu,Khosravi Expires September 2003 [Page 3] Internet Draft Forwarding and Control Element protocol March 2003 The following definitions are taken from [3] Forwarding Element (FE) - A logical entity that implements the ForCES protocol. FEs use the underlying hardware to provide per- packet processing and handling as directed by a CE via the ForCES protocol. FEs may use PFE partitions, whole PFEs, or multiple PFEs. Control Element (CE) - A logical entity that implements the ForCES protocol and uses it to instruct one or more FEs how to process packets. CEs handle functionality such as the execution of control and signaling protocols. CEs may consist of PCE partitions or whole PCEs. Pre-association Phase - The period of time during which a FE Manager (see below) and a CE Manager (see below) are determining which FE and CE should be part of the same network element. Post-association Phase - The period of time during which a FE does know which CE is to control it and vice versa, including the time during which the CE and FE are establishing communication with one another. ForCES Protocol - While there may be multiple protocols used within the overall ForCES architecture, the term "ForCES protocol" refers only to the ForCES post-association phase protocol (see below). ForCES Post-Association Phase Protocol - The protocol used for post- association phase communication between CEs and FEs. This protocol does not apply to CE-to-CE communication, FE-to-FE communication, or to communication between FE and CE managers. The ForCES protocol is a master-slave protocol in which FEs are slaves and CEs are masters. This protocol includes both the management of the communication channel (e.g., connection establishment, heartbeats) and the control messages themselves. The term ForCES protocol may refer to a suite of protocols that are used to exchange control information as well as redirect data packets between the CEs and FEs. FE Model û Modeling of logical functions in a Forwarding element line card. FE Manager - A logical entity that operates in the pre-association phase and is responsible for determining to which CE(s) a FE should communicate. This process is called CE discovery and may involve the FE manager learning the capabilities of available CEs. A FE manager may use anything from a static configuration to a pre- association phase protocol (see below) to determine which CE(s) to use. Being a logical entity, a FE manager might be physically Gopal,Audu,Wu,Khosravi Expires September 2003 [Page 4] Internet Draft Forwarding and Control Element protocol March 2003 combined with any of the other logical entities mentioned in this section. CE Manager - A logical entity that operates in the pre-association phase and is responsible for determining to which FE(s) a CE should communicate. This process is called FE discovery and may involve the CE manager learning the capabilities of available FEs. A CE manager may use anything from a static configuration to a pre- association phase protocol (see below) to determine which FE to use. Being a logical entity, a CE manager might be physically combined with any of the other logical entities mentioned in this section. Pre-association Phase Protocol - A protocol between FE managers and CE managers that is used to determine which CEs or FEs to use. A pre-association phase protocol may include a CE and/or FE capability discovery mechanism. Note that this capability discovery process is wholly separate from (and does not replace) that used within the ForCES protocol (see Section 7, requirement #1). However, the two capability discovery mechanisms may utilize the same FE model (see Section 6). Pre-association phase protocols are not discussed further in this document (see Section 11.3). ForCES Network Element (NE) - An entity composed of one or more CEs and one or more FEs. To entities outside a NE, the NE represents a single point of management. Similarly, a NE usually hides its internal organization from external entities. ForCES Protocol Element (PE) - A Forwarding Element or a Control Element. Logical Component û Components in the forwarding data path like filter, meter, forwarder, shaper etc. FE Model (FEM) û Organization of logical components in the Forwarding plane. 2. Introduction Network elements such as routers play an important role in transporting IP packets in the Internet. QoS aware router, policy based routing and middle-box functions such as firewall, NAT, proxies put heavy requirements on per-hop behavior treatment for IP packets. This complicates the network element. Routers have emerged from simple monolithic software to a distributed routing complex that interconnects different networks and distributes the routing and forwarding logic to line cards and Gopal,Audu,Wu,Khosravi Expires September 2003 [Page 5] Internet Draft Forwarding and Control Element protocol March 2003 control cards. A line card does basic forwarding function and a control card runs all the control and management functions. Forces [3][5] defines both architectural and protocol requirements for the communication between CE and FE. Forwarding and Control ElemenT (FACT) protocol addresses all the requirement of Forces protocol and is described in this document. 3. Protocol Overview ForCES is a framework consisting of set of protocols and data structure representing the forwarding and control elements in the form of an extensible model [6][5][4]. CEs handle control, signaling and management protocols, while FEs perform forwarding functions. CEs control the behavior of FEs in a master/slave fashion. FACT protocol is designed to communicate between FE and CE, and/or between CE and CE. Since CE-CE communication is outside the scope of Forces activity, we will not be discussing CE-CE communication in this document FACT protocol satisfies all the Forces protocol requirements. The FACT protocol is logically separated into base control protocol and logical components service functions. This is similar to SNMP where SNMP protocol provides a set of message to exchange messages between SNMP manager and agent(s)_ MIB is the actual payload communicated in the form of OID between them. Similarly FACT protocol has fixed header and a variable size payload field; the payload field carries data that may contain one of the following * Control packets which are received by FE through external ports or Packets that are sent to the external egress port * Logical components details which are represented in FE Modeling Document [4] * Other configuration information required according to [3], [5] (a) Base control functions FACT protocol depends upon certain information to facilitate communication between FE and CE; this information is sent to either FE or CE by FE-Manager and CE-Manager [sec 8.4] components respectively. After the pre-association phase, FACT protocol provides mechanism Gopal,Audu,Wu,Khosravi Expires September 2003 [Page 6] Internet Draft Forwarding and Control Element protocol March 2003 that carries traffic between FE and CE components. This includes support for dynamic association, high availability, security, topology discovery of FE and CE, Control packet redirection etc. (b) Service specific functions Using base Forces protocol, any logical components inside the FE can be configured or managed. Examples of service specific functions in FACT protocol include messages on capability, logical component configuration, port status, etc. FE modeling is flexible enough to allow any arbitrary queries. These queries can be encoded using simple TLVs, OIDs, or XML. FACT protocol is independent of type of encoding. FACT protocol supports different types of messages, including synchronous messages like simple request/reply, asynchronous messages to notify PEs of status changes, and transaction oriented synchronous messages that support 2 phase command bundling functions operations. The FACT protocol provides a notion of distributed IPC mechanism by providing support functions required for replication, high availability and fail-over support for the ForCES distributed network element environment. All these features are optional and can be configured through FE-Manager and CE-Managers for FEs and CEs respectively during the pre-association phase. 3.1.Independence from Interconnect/Transport Layer The FACT protocol is independent of the Interconnect or Transport Layer. It makes no assumptions about the transport or interconnect layer and can work over different transport protocols such as TCP, UDP or SCTP or can also work directly over IP. 3.2.Reliability And Fail-Over Model FACT protocol supports FE and/or CE fail-over functions in order to support a high availability of the network element. All control or data messages exchanged between a CE and a FE are assigned a tag for identification purposes. The CE-SET is a list of CEs that reside within a Network Element (NE) as a cooperating unit. Likewise, the FE-SET is a list of FEs that reside in an NE as a cooperating unit. Gopal,Audu,Wu,Khosravi Expires September 2003 [Page 7] Internet Draft Forwarding and Control Element protocol March 2003 The following is a list of high-availability mechanisms, which can be supported by FACT protocol. Note this list is not exhaustive and is provided for illustrative purpose. (1) Strong Consistency: More than one CE are active and only one CE is engaged in communicating with a FE. The other CEs may be simply snooping the traffic and updating their internal states (assume shared LAN). If CEs are far apart, (may be by a few hops), it is then responsibility of the CEs to synchronize their states. (2) Weak Consistency (Fail-over): FE can communicate directly with the designated CE and if the designated CE fails, FE will start communicating with the backup CE. The selection of designated and backup CEs is dependent on their connection topologies and is done during pre-association phase. (3) Load Sharing: Different CEs can be configured to support part of the functions. For example, OSPF may be running in one CE, and BGP running in another CE, thus distributing the functionalities across the CEs in a CE-set. FEs can forward the packets to different CE based on the CEÆs capabilities and functions. This way, the load and processing on the CEs is distributed evenly. In all the above cases, CE (including designated and backup CEs) and FE are pre-configured to perform such activities and are done as part of pre-association phase. The fail-over model supports n+k redundancy, where n FEs are the Minimum number of redundant FEs required to handle configured traffic and k FEs are available to take over for a failed or unavailable FE. This is also true for CEs. Note that 1+1 active/standby redundancy is a subset of this model, as is a simplex 1+0 (no redundancy). To avoid a single point of failure, it is recommended that a minimum of a pair of two FEs be in the list, resident in separate hosts and available over different associations with the CEs. Host1 ******** ************** * *-----------------------------------------* ******** * * * _________* * FE1 * * * CE1 * Associations +---------* ******** * * *-----------------------+ | * ******** * ******** | | * * FE2 * * | | * ******** * Gopal,Audu,Wu,Khosravi Expires September 2003 [Page 8] Internet Draft Forwarding and Control Element protocol March 2003 ******** | | ************** * *-------------------------------+ * * | * CE2 * Associations | * *------------+ | * * | | Host2 ******** | | ************** | +-----------------* ******** * |____________________________* * FE1 * * +----------------------------* ******** * * ******** * * * FE2 * * * ******** * ************** . Figure 1 û Illustration of Logical Association between CE-FE 3.3.Retransmission FACT protocol uses optional retransmission procedures to enhance its reliability. Even if a reliable underlying transport layer is used for FACT messages and the messages are properly delivered to the destination PE, the destination PE may fail to respond or acknowledge original messages. In these scenarios where a FACT message sender has established a logical connection with the receiver PE and is expecting a response or an acknowledgment to the original FACT message, retransmission may take place. Unless specified otherwise, the original FACT message is retransmitted if the expected reply message does not arrive in a pre-configured retransmission time. The retransmission stops after the sender receives a reply or detects terminal conditions, such as breaking of the logical connection or leaving of the NE group. A system-wide retransmission timer can be applied, by default, to all FACT messages that require response. In case different timers are required for messages in various connection topologies, new timers and relevant handling procedures can be added in future revisions of the protocol. 4. Message Overview FACT protocol messages are made up of two parts: the header part, and the message body or service payload part. This section describes the details of each message and its format. Gopal,Audu,Wu,Khosravi Expires September 2003 [Page 9] Internet Draft Forwarding and Control Element protocol March 2003 4.1. Protocol Message Header structure FACT protocol Header contains the following fields; some of the fields are optional and it is interpreted based on the Type. All these messages are based on TLV format and is conforms to network byte ordering. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |Version| MsgCls|Msg-Type |P| Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | CE-Tag | FE-Identifier | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Transaction sequence Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 4.1.1. Version The version field contains the version of the FACT protocol supported by the implementation. The current supported versions are : value 0x01 4.1.2. Message Classes and Types The messages can be grouped into classes, with each class consisting Of a set of related message types. In the following, ôPEö stands for Protocol Element and represents a CE or a FE in a network element. The valid message classes are: Message Class: 4 bits (unsigned integer) 0 Reserved 1 PE Connection and association (CA) Messages 2 Capabilities Control (CAPCO) Messages 3 PE State Maintenance (PESM ) Message 4 PE Traffic Maintenance (PETM) Messages 5 Event Notification (EN) Messages 6 Vendor Specific (VS) Messages 7-15 Reserved by IETF Gopal,Audu,Wu,Khosravi Expires September 2003 [Page 10] Internet Draft Forwarding and Control Element protocol March 2003 The message types for the defined message classes are as follows: Message Type for Connection and Association (CA) Message Class 0 Reserved 1 Join Request 2 Join Response 3 Leave Request 4 Leave Response 5-127 Reserved by IETF Message Type for Capabilities Control (CAPCO) Message Class 0 Reserved 1 Capabilities Request 2 Capabilities Response 3 Configure Logic Components 4 Configure Logic Components Response. 5 Topology Request 6 Topology Response 7 Query Request 8 Query Response 9-127 Reserved by IETF Message Types for PE State Maintenance (PESM) Message Class 0 Reserved 1 PE Up 2 PE Up Ack 3 PE Down 4 PE Down Ack 5 PE Active 6 PE Inactive 7 PE Active ACK 8 PE Inactive ACK 9 Heartbeat 10 Heartbeat Ack 11-127 Reserved by IETF Message Types for PE Traffic Maintenance (PETM) Message Class 0 Reserved 1 Control Packet Redirect 2 Control Packet Redirect Acknowledgement 3 Control Packet Forward Gopal,Audu,Wu,Khosravi Expires September 2003 [Page 11] Internet Draft Forwarding and Control Element protocol March 2003 4 Control Packet Forward Acknowledgement 5-127 Reserved by IETF Message Types for Event Notification (EN) Message Class 0 Reserved 1 Event Register 2 Event Register Acknowledgement 3 Event De-register 4 Event De-register Acknowledgement 5 Generic Status Change and Event Notify from FE to CE 6 Port and Link Status Change and Event Notify from FE to CE 8 Error Event Report 9-127 Reserved by IETF Message Types for Vendor Specific Function (VSF) Message Class 0 Reserved 1 VS-Data Request 2 VS-Data Ack 3-127 Reserved for other vendor specific messages Vendor specific function types interpretation is beyond the scope of this protocol. 4.1.3. Length The Message Length denotes the length of the message in octets, including the Message Header. 4.1.4. CE Tag During a pre-association phase, CEs can be configured using CE- Manager component. In a network element, there may be many CEs; one or more CEs can be grouped together to form a CE-set. A CE-set is unique in one network element and is identified by an 8-bit number. To identify a CE inside a CE-set, the CE identifier is used which is also an 8-bit field. Gopal,Audu,Wu,Khosravi Expires September 2003 [Page 12] Internet Draft Forwarding and Control Element protocol March 2003 Figure below shows the CE-Tag fields, CE-Tag is a 16-bit integer which has two portions, the higher order 8-bit describes the CE-set number, and the lower 8-bit describes the CE-identification number. The main advantage of having such naming is to uniquely identify the Communicating elements and their logical association inside a network element. This field is mandatory for all message types. +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | CE-Set Number | CE-Identifier | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ CE-Tag fields 4.1.5. FE Identifier This is a 16-bit field to uniquely identify the FE in one network element. The main advantage is that FE or CE may use different interconnect technologies, they can be identified by using the address itself (for example, portions of IP address or MAC address etc). This unique naming scheme helps to manage the CEs and FEs together and support some features, which are required for back-up recovery, configuration update process, and high availability support. This identifier is always present in all type of messages. 4.1.6. Priority (P) Bits If this bit is set then the message should be treated as a high- priority message by the receiving end-point. If this is set to zero then the message is of normal priority. 4.1.7. Transaction Sequence Number (TSN) This 32-bit field uniquely identifies the transaction between the FE and CE. When a request is made by one endpoint (say CE) it generates TSN number that is sent in the request message; the other endpoint (Say FE) can copy this same TSN number in its reply message. 4.2. Service Data or Payload Structure FACT protocol messages consist of the Common Message Header described in the previous section, followed by zero or more variable length parameters, as defined by the message type. This constitutes Gopal,Audu,Wu,Khosravi Expires September 2003 [Page 13] Internet Draft Forwarding and Control Element protocol March 2003 the Payload or Service Data. This service data represents one of the following: (1) Logical component configuration (2) Logical component statistics (3) Logical component status and events (4) FE capabilities and topology information (5) Command data which FE wants to send through a particular logical component output interface (6) Command data which CE wants to receive from a particular logical component input interface The variable length parameters in the payload are defined in a Tag- Length-Value (TLV) format as shown below: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Parameter Tag | Parameter Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ \ \ / Parameter Value / \ \ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Mandatory parameters MUST be placed before optional parameters in a message. Parameter Tag: 16 bits (unsigned integer) The Tag field is a 16-bit unique identifier of the type of the parameter. It takes a value of 0 to 65534. Appendix-1 lists all used values of the Tag and related messages. The value of 65535 is reserved for IETF-defined extensions. Values other than those defined in specific parameter description are reserved for use by the IETF. Parameter Length: 16 bits (unsigned integer) The Parameter Length field contains the size of the parameter in bytes, including the Parameter Tag, Parameter Length, and Parameter Value fields. The Parameter Length does not include any padding bytes. Parameter Value: variable-length Gopal,Audu,Wu,Khosravi Expires September 2003 [Page 14] Internet Draft Forwarding and Control Element protocol March 2003 The Parameter Value field contains the actual information to be transferred in the parameter. The total length of a parameter (including Tag, Parameter Length and Value fields) MUST be a multiple of 4 bytes. If the length of the parameter is not a multiple of 4 bytes, the sender pads the parameter at the end (i.e., after the Parameter Value field) with all zero bytes. The length of the padding is NOT included in the parameter length field. A sender SHOULD NEVER pad with more than 3 bytes. The receiver MUST ignore the padding bytes. 5. FACT Messages This section defines the messages and their parameter contents. 5.1. Association and Connection (CA) Messages 5.1.1. Join Request After the pre-association phase [8.4], the FEs can join or leave any CE in a CE-set. A FE uses the Join Request message to initiate association with a CE-set. The message will contain the requesterÆs identity that was configured during pre-association After a successful join process, FEs can report their capabilities to the CE. Other information contained in the request message include indication of whether the FE can support features like command bundling, 2 phase command operation, or support for high availability etc.[section 8] At a given point, CEs from one CE set can communicate with a FE. FE has to know which CEÆs request it can accept. This information is configured during the pre-association phase. FE uses this CE-list to send the join request. It first tries one of the CEÆs in the list and if it not successful, it tries the next CE in the list. If all of the CEs in the list are tried without success, the FE should start over again until Retry timer expires. The JOIN Message is not subject to retransmission, since there is no logical connection established yet. In case of load-sharing, the FE have to send join requests to multiple CEs in a CE set. The format of the JOIN Message payload is as follows: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Tag (0x10) | Length (8) | Gopal,Audu,Wu,Khosravi Expires September 2003 [Page 15] Internet Draft Forwarding and Control Element protocol March 2003 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Address | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | HAS |C| Reserved | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 2 Join request HAS bits: These 3 bits are optionally used to communicate whether the FE can participate in high availability mechanisms. [section 8] HAS - High availability Support (Also called Traffic Mode) -------------------------------- 000 û No support 001 û Fail over support (Weak Consistency) 011 û Replicate the control packets to designated CEÆs in a CE set (Strong consistency). NOTE: This may also depend upon the type of interconnects technology being used. For example, if FE and CEÆs are connected through the shared bus or segment, this feature all the CEÆs may receive the copies of the packet. 100 û Retransmission support (as described in Section 3.3) Bit C: This bit set to zero means it does not provide Persistence to command sets which is required to perform 2 phase command operations. This bit set to ONE means that this FE can participate in 2-phase command operation. 5.1.2. Join Response CE after receiving a join request message performs the following operations. (1) Checks the FE association ID (FE identifier) in the request message and if it equals zero, then the CE generates a unique identifier for that FE and allocates resources for its attributes. (2) If the FE association ID (FE identifier) field is not zero, then the CE checks up its previously stored Gopal,Audu,Wu,Khosravi Expires September 2003 [Page 16] Internet Draft Forwarding and Control Element protocol March 2003 configuration for that FE. If it has any, it will use that to configure the FE in subsequent messages. This is warm restart operation. (3) If the CE needs to reject the join request for some reason, it sends a Leave Response Message as indicated later. The format for the JOIN RESPONSE message payload is as follows: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Tag (0x11) | Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | FE identifier | Previous configuration if any +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ After a successful JOIN operation, the CE should initiate the next phase of the association establishment process by querying the FE for its capabilities, its topologies, etc, and then configuring the FE for the functions it is to perform in the NE. 5.1.3. Leave Request The FE can leave by sending Leave request to CE. The CEÆs upon receiving such request releases the associated resources assigned for the FE. Similarly CE can also send a leave request to other CEÆs in the CE-set and receiving CEÆs releases the associated resources assigned for the FEÆs. The LEAVE message contains the following parameters: Reason Info String (optional) The format for the LEAVE Message parameters is same as follows: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Tag (0xa) | Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Gopal,Audu,Wu,Khosravi Expires September 2003 [Page 17] Internet Draft Forwarding and Control Element protocol March 2003 | Reason | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Tag (0x4) | Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ > INFO String* < +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ The format and description of the optional Info String parameter is the same as for the PE Up message (See Section 4.3.2.1.). The reason parameter indicates the reason the PE is leaving the NE. Valid values are as follows: Value Description 0x1 Management Inhibit (Manual Removal) 0x2 Invalid NE group 0x3 FE or CE not ready 0x4 Max number of FEs reached (for leave response only) 0x5 Physical Layer fault 0x6 Fail-over switching (refer to Section 7.5) Note: Message 0x3 if generated by CE means CE is not ready. If generated by FE means FE is not ready. The optional INFO String parameter can be any meaningful 8-bit character string, up to 255 characters in length. This may be used for debugging purposes 5.1.4. Leave Response When a FE is leaving the CE generate an acknowledgment to the FE. IF a CE is leaving one of the CEÆs in the CE-set will generate a response to the leaving CE. Note CE-CE communication is currently out of scope. If a CE in a CE-set has generated leave request, the CE with low CE-identifier will generate a leave response. This mechanism will avoid contention for generating the leave response. If a CE needs to reject a join request from a FE for some reason, it sends a Leave Response Message to the FE as well (Refer to Section 5.1.2). The LEAVE Response message contains the following parameters: Reason Info String (optional) Gopal,Audu,Wu,Khosravi Expires September 2003 [Page 18] Internet Draft Forwarding and Control Element protocol March 2003 The format for the LEAVE Message parameters is same as in LEAVE Request message (See 5.1.3). 5.2.Capabilities Control (CAPCO) Messages This is the next phase of the JOIN process. After a FE has been successfully accepted into a CE-SET, the CE initiates the next phase of the association establishment process by querying the FE for its capabilities, its topologies, etc, and then configuring the FE for the functions it is to perform in the NE. 5.2.1. Capabilities Request FE may have one or more logical components on the forwarding plane like meter, shaper, egress port etc. CE may configure or query these components and their status at any time. In order to do this, CE needs to know the logical components placement and sequence in the forwarding data path. FE Model [4] describes the arrangement and the relationship of those components. For understanding purposes, we provide the following summary : A FE identifier identifies FE; FE may contain one or more parallel data path(s). On each parallel data path, there may be one or more logical components and each one is connected to another either in series or in parallel fashion. The logical component is uniquely identified by a number (which needs to be IANA assigned). Examples of logical components are filter, shaper, egress port etc. 5.2.2. Capabilities Response This is used by the FE to report its capabilities to the CE as per CEÆs request. The format for the Capability Response is as follows: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Gopal,Audu,Wu,Khosravi Expires September 2003 [Page 19] Internet Draft Forwarding and Control Element protocol March 2003 | Tag (0x1b) | Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Port Count | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ > Port Info < +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Logical Component Count | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ > Logical Component Info < +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ The format for the Port Info is as follows: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Port ID | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Port Type | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Port Speed | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Port Address | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ The format for the Logical Component Info is as follows: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 | Logical Component Handle | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Logical Component Type | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | DownStream L.Comp. Count | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | DownStream L.Comp. Handle | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ . . | DownStream L.Comp. Handle | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 5.2.3. Configure Logic Components This message is used by the CE to configure the FE. The FE consists of functional or logic elements that could be configured to achieve Gopal,Audu,Wu,Khosravi Expires September 2003 [Page 20] Internet Draft Forwarding and Control Element protocol March 2003 a desired behavior of the FE in the network. Some configurable attributes of the FE include: 1) parameters of a logic elementÆs logic function 2) relative position of a logic element in the data path 3) port attributes (direction, bandwidth,..) 4) routing table functions 5) High Touch functions 6) Off-Load functions 7) Data Path Security Package (e.g. use IPSEC, etc) 8) Filter and Matching functions. The CE might also have received a command (either through CLI or through SNMP etc) to setup some tunnels or path or some configuration. This may sometimes involve configuring more than one FE. For example, packet may come through one line card and leave through another. In this situation, CE has to configure both FEsÆ logical components. If one of the logical components fails, it has to perform rollback operation and issue a command failure notification to the management station or CLI or SNMP etc. This operation is called command bundling. To perform this, CE sends series of commands to each FE with command bundling bit set. Each FE after receiving the command will have to save the current configuration and check whether it can program the requested configuration. A status message should be sent back to the CE. Once CE receives all the status messages, it can then send an execute command with same transaction sequence number, signaling the FEs to now switch to the new configuration. This operation will get too complicated when more than one CE try to generate different configuration on the logical components; If FE tries to do parallel activities, it may get into a deadlock situation. It is up to the FE to react with the appropriate status message. The mechanism to perform locking operation is implementation dependent but the operation is considered atomic. The format of the Configure Logic command is as follows: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Tag (0x1c) | Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | C-Operation| Configuration Command | reserved | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Logical Component Handle | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ > Command Data < Gopal,Audu,Wu,Khosravi Expires September 2003 [Page 21] Internet Draft Forwarding and Control Element protocol March 2003 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ C-Operation: FEÆs and CEÆs may engage in two-phase commit operation. This field provides the stage of such transaction. 0x00 - Command is single operation no need to generate a status message to FE 0x01 û This command is a two-phase operation, FE needs to save the previous state if rollback operation may be performed later by FE. 0x10 û Rollback to the previous state. 0x11 û Execute and complete the command. During this operation the TSN value is same and used to identify the transaction. The same CE should generate this TSN otherwise, FE will treat this as different query from other CE. Configuration command: This field defines the command type. The valid values for this field are 0 Reserved 1 NULL 2 Add 3 Update 4 Delete 5 Delete All (Flush or Purge) Logical Component Handle: This field defines the logical component handle or identifier for which this command is being issued. Command Data: This is the variable length configuration data for the logical component. This can be encapsulated using TLV or OID or XML formats. 5.2.4. Configure Logic Components Response This is sent by the FE to CE to acknowledge Logic Components configuration as requested by CE. The format of the Configure Logic Response is as follows: 0 1 2 3 Gopal,Audu,Wu,Khosravi Expires September 2003 [Page 22] Internet Draft Forwarding and Control Element protocol March 2003 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Tag (0x1d) | Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | C-Operation| Configuration Command | G.Result | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Logical Component Handle | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ > Command Result < +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ The response contains the information from the command, but with ôGlobal Resultö field filled in to indicate success or failure. Result Value Meaning CONFIG-OK 0x0 Success CONFIG-BADCMD 0x1 Bad or unsupported configuration command. CONFIG-BADPRM 0x2 Bad configuration parameter. Logical Component Handle: This field defines the logical component handle or identifier for which this command is being issued. Command Result: This is the variable length configuration result for the logical component. This can be encapsulated using TLV or OID or XML formats. 5.2.5. Topology Request CE wants to know how each FE is connected or configured during the pre-association phase. This may be used by the CE to coordinate their activities for high availability, health check etc. 5.2.6. Topology Response FE manages the logical components and it requires some minimal information to communicate with CE. During pre-association phase, the FE-Manager may configure the following FE environment and attributes: (1) List of CEÆs with which it can communicate and their IP address or MAC address etc. (2) May designate one CE as primary and other as backup for asynchronous notification of events. (3) May also configure other parameters which are needed for high availability operations Gopal,Audu,Wu,Khosravi Expires September 2003 [Page 23] Internet Draft Forwarding and Control Element protocol March 2003 (4) The FEs may be connected in different kinds of topology as directed by the FE Manager to allow for different data flow paths through the FEs. These FE attributes are represented in the FE Model as OID or XML or any other suitable format in the FE table. The format for the Topology Response is as follows: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Tag (0x1e) | Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ > Topology Info < +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ The format of the topology Info is as follows: It consists of the list of FEs (FEIDs) directly connected to the communicating FE. (FE Addresses are not required) 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | #of neighbor FEs (MAX 4) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | FE1ID | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | FE1 ADDRESS | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | FE2ID | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | FE2 ADDRESS | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | FE3ID | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | FE3 ADDRESS | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 5.2.7.Query Request Gopal,Audu,Wu,Khosravi Expires September 2003 [Page 24] Internet Draft Forwarding and Control Element protocol March 2003 CE may be interested in querying the properties of Logical Components or collecting statistical information from FE, including that of its logical components. In this case, a CE sends a Query Request Message to a FE and expects a Query Response Message from it. The format for the Query Request Message parameters is as follows: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Tag (0x13) | Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Type of Info | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Logical component Handle | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Query Specific Data | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Logical component Handle2 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ : : | Logical component HandleN | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ There can be multiple logical component IDs in one message. The number of IDs is derived from the Length field. Valid values of the Info Type are: GEN-STATS (0x0) û General Statistics PORT-STATS (0x1) û Port Statistics LINK-STATS (0x2) û Link Statistics LComp-STATS (0x3) û Logical Component Statistics LComp-PROPS (0x4) û Logical Component Properties PORT-PROPS (0x5) û Port Properties Query Specific Data: This is query specific data, which can be in encapsulated as TLV or OID or XML. 5.2.8.Query Response After receiving a Query Request Message, a FE replies with the Query Response Message. The format of the Query Response Message is as follows: Gopal,Audu,Wu,Khosravi Expires September 2003 [Page 25] Internet Draft Forwarding and Control Element protocol March 2003 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Tag (0x14) | Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Info Type | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Logical component Handle | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ > Logical component Data < +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ The Logical Component data is encapsulated similar to the query specific data in the respective format. 5.3. PE Maintenance Messages 5.3.1.Protocol Element UP (PEUP) The UP message is sent by a PE to indicate to its master (or slave) that it is UP (in-service) and ready to be used. The format of the PEUP message is as follows: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Tag (0xb) | Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Association ID | Traffic Mode Type | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Tag (0xyy) | Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | PortID1 | PortID1OpStatus | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | PortID2 | PortID2OpStatus | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ : : +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | PortIDN | PortIDNOpStatus | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ The UP message contains the following parameters Association ID (Mandatory) Gopal,Audu,Wu,Khosravi Expires September 2003 [Page 26] Internet Draft Forwarding and Control Element protocol March 2003 Traffic Mode Type (Mandatory) Interface Ids or ports available for use on the FE (Optional) 5.3.2. Protocol Element Up Acknowledge The PEUP Acknowledgement message is used to acknowledge a PE-Up message received from a remote PE slave or master peer. The PEUP Acknowledgement message contains the following parameters: Association ID (Mandatory) Traffic Mode Type (Mandatory) Interface Ids or ports usable on the FE (Optional) The format for the PEUP Acknowledgement message is the same as in PE UP Message. 5.3.3. Protocol Element Active (ACT) The ACT message is sent by a controlling CE to ask its slave FE to go ACTIVE and start handling traffic. The FE must be UP and must have sent the PEUP message to the CE. The ACT message contains the following parameters Association ID (Mandatory) Traffic Mode Type (Mandatory) INFO String (Optional) The format for the ACT message payload Is as follows: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Tag (0xb) | Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Association ID | Traffic Mode Type | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Tag (0x4) | Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ > INFO String* < +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ The Traffic Mode Type parameter identifies the traffic mode of Gopal,Audu,Wu,Khosravi Expires September 2003 [Page 27] Internet Draft Forwarding and Control Element protocol March 2003 operation of the PE within an NE. The valid values for Type are shown in the following table: Value Description 0x1 Over-ride-strong-consistency 0x2 Over-ride-weak-consistency 0x3 Load-share Within a particular Association Identifier, only one Traffic Mode Type can be used. The Over-ride value indicates that the PE is operating in Over-ride mode, where the PE takes over all traffic in an NE i.e., primary/back-up operation), over-riding any currently active PEs in the NE. In Load-share mode, the PE will share in the traffic distribution with any other currently active PEs. The format and description of the optional Info String parameter is the same as for the PE Up message (See Section 5.3.1.). 5.3.4. Protocol Element Active Acknowledgement (ACT-ACK) The ACT Acknowledgement message is used to acknowledge a PE-Active message received from a remote CE master. The ACT Acknowledgement message contains the following parameters: Association ID (Mandatory) Traffic Mode Type (Mandatory) INFO String (Optional) The format for the ACT Acknowledgement message is the same as in PE Active Message (See 5.3.3) 5.3.5. Protocol Element Inactive (INACT) The INACT message is sent by a CE to indicate to its other CEÆs in the CE-set that it is no longer an active CE to be used from within a list of CEÆs. Similarly, FE can send this message to indicate that it is no longer active to its designated CE. The receiver will respond with an INACT Ack message and either discard incoming messages meant for the outgoing CE or FE , or buffer them for a timed period and then discard (if no other PEs become active). The INACT message contains the following parameters Traffic Mode Type (Mandatory) Gopal,Audu,Wu,Khosravi Expires September 2003 [Page 28] Internet Draft Forwarding and Control Element protocol March 2003 INFO String (Optional) The format for the CE/FE Inactive message parameters is as shown for FE/CE Active Message(See 5.3.3) 5.3.6. CE or FE Inactive Acknowledgement (INACT-ACK) The CE or FE Inactive Acknowledgement (INACT-ACK) message is used to acknowlege a CE or FE that has generated the inactive message. When a FE has generated the request to its designated CE. The designated CE will generate this response. IF a CE in a CE-set has generated a request, one of the CEÆs in the CE-set will generate this INACT- ACK response message. Note the CE-CE communication is currently out of scope. If all the CEÆs in the CE-set are in a shared bus(or LAN). If a CE in a CE-set has generated INACT request, the CE with low CE- identifier will generate a leave response. This mechanism will avoid contention for generating response message. The INACT-ACK message contains the following parameters: Traffic Mode Type (Mandatory) INFO String (Optional) The format for the FE or CE Inactive Acknowledgement message parameters is same as CE or FE Active Message (See 5.3.3) 5.3.7. Protocol Element Down (DOWN) Due to failure or maintenance operation, FE can send generate this DOWN message to its designated CE. Upon receiving this request, designated CE may reassign the responsibility to other FEÆs (if possible). Similarly CE in a CE-set can generate the same message to all other CEÆs in the same CE-set. The DOWN message contains the following parameters: Reason - (Mandatory) reason for going down Info String û (Optional) information to augment reason The format for the DOWN Message is as follows: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 Gopal,Audu,Wu,Khosravi Expires September 2003 [Page 29] Internet Draft Forwarding and Control Element protocol March 2003 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Tag (0xc) | Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Reason | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Tag (0x4) | Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ > INFO String* < +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ The reason parameter indicates the reason the PE is leaving the NE. Valid values are as follows: Value Description 0x1 Management Inhibit (Manual Removal) 0x2 Device Fault The format and description of the optional Info String parameter is the same as for the PE Up message (See Section 5.3.1.). 5.3.8. Protocol Element Down Ack (PEDN-ACK) The PEDN-ACK message is used by the designated CE to acknowledge the PEDN message sent by the outgoing PE. The format of this message is the same as for the PEDN message. The DOWN ACK message contains the following parameters: Reason - (Mandatory) reason for going down Info String û (Optional) information to augment reason The format for the DOWN ACK Message is same as for DOWN message. 5.3.9. Heartbeat (or Health check) CE periodically polls each FE to ensure that it is operational. A CE generates this message. There may be more than one CE in a CE-set; to avoid duplicate requests, one of the ACTIVE CEs in the CE set can generate this request. Then it is up to the CEs to update the status of the FE. Note: - By default, the designated ACTIVE CE should initiate the health check. An optional Heartbeat Data parameter may be sent in the heart beat message. Its contents are defined by the sending node. The Heartbeat Gopal,Audu,Wu,Khosravi Expires September 2003 [Page 30] Internet Draft Forwarding and Control Element protocol March 2003 Data could include, for example, a Heartbeat Sequence Number and, or Timestamp. The receiver of a Heartbeat message does not process this field as it is only of significance to the sender. The receiver MUST respond with a Heartbeat Acknowledgement message. The format for the Heartbeat Message is as follows: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Tag (0x9) | Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ > Heartbeat Data < +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 5.3.10. Heartbeat Acknowledgement (FE response to heart beat request) After verifying the CE-Tag, FE simply echoÆs the original heartbeat message. 5.4. PE Traffic Maintenance (PETM) Messages 5.4.1. Control packet Redirect to CE for processing When a Router receives both control and data packets through a physical port, any of the following scenarios may occur: (a) Forwarding blade receives IP packet that is not destined for it; these packets are forwarded to the CE by the forwarding plane component. (b) Forwarding blade receives IP packet that is destined for it. These packets are not forwarded to the Control plane; rather they are processed by the forwarding plane control logic (stack in the forwarding plane). Example of such packet is ping request.. (c) Forwarding blade receives IP packets that may be routing protocol packets or packets which cannot be processed by the stack in the line card. Such packets have to be forwarded to the control plane by the FE. FE encapsulates the control packet along with logical component information (refer to FE Modeling [4]) in the FACT header and forwards it to CE. FE may be pre- configured during the pre-association phase or during the Join response message from the FE. For example, consider a scenario Gopal,Audu,Wu,Khosravi Expires September 2003 [Page 31] Internet Draft Forwarding and Control Element protocol March 2003 where each CE is running one routing protocol say OSPF, BGP, or IS-IS. When a FE receives routing protocol packet, FE encapsulates that packet inside the Forces protocol and forwards it to the respective CE. The format of the Control Packet Redirect is as follows: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Tag (0xe) | Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Port ID on which packet arrived | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Tag (0xe) | Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ > Control Packet < +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Control Packet: The control packet that the network element received through a particular IP interface. 5.4.2. Control Packet Redirect Acknowledgement This may be used by the CE to acknowledge the Control Packet Redirect message received from the source FE. The format of this message is same as for the Control Redirect Request. 5.4.3. Control Packet Forwarding through specific output ports CE may generate a packet and want FE to forward that packet through a particular or multiple egress port(s). Examples of such packets are routing protocol updates, discoveries, etc. Before generating such request, CE has to know the FEÆs logical components and the list of available port and the configuration status. (reference snapshot of Logical components ) 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Tag (0xf) | Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Egress Port ID through which to forward packet | Gopal,Audu,Wu,Khosravi Expires September 2003 [Page 32] Internet Draft Forwarding and Control Element protocol March 2003 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Tag (0x30) | Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ > Packet to be forwarded < +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Packet to be forwarded is preceded by the egress port through which the packet must be forwarded by the FE. To forward to multiple egress ports (such as multicast), the format is as follows: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Tag (0x2f) | Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Start Egress Port ID1 through which to forward packet | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | End Egress Port ID1 through which to forward packet | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ > .. < | Start Egress Port IDn through which to forward packet | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | End Egress Port IDn through which to forward packet | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Tag (0x30) | Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ > Packet to be forwarded < +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 5.4.4. Control Packet Forwarding Acknowledgement This is used by the FE to acknowledge the Control Packet Forwarding Request initiated by the CE. The format of this message is the same as for the Control Packet Forwarding Request message. 5.5. Event Notification Messages Various events in the data path can be monitored for by the FE and reported to the CE. The CE must first inform the FE which of these events it is interested in through a registration process. The following events can be reported to the CE: Gopal,Audu,Wu,Khosravi Expires September 2003 [Page 33] Internet Draft Forwarding and Control Element protocol March 2003 1) Packet received in error 2) Port Error/Failure/OK transitions 3) Port Up/Down transitions 4) Port is in (or out of) congestion (high-water mark of bandwidth utilization reached) 5) Packet flow rate through port reached set limit 6) Buffer overflow (onset, abatement) 7) Link Error/Clear transitions 8) Link Down/Up transitions 9) CPU Usage High (above 70%) and other generic errors in FE. 5.5.1. Event Register This is sent by the CE to the FE to request that FE notify the CE when the indicated events occur on the FE. The format of the payload is as follows: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Tag (0xe) | Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Event Type | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ > Event Specific Data < +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Event Type is the type of the event to report. Valid values are as follows: NO-EVENTS (0x0) û No Events reported PKT-EVENTS (0x1) û Packet events PORT-EVENTS (0x2) - Port Events LINK-EVENTS (0x3) û Link related events GFE-EVENTS (0x4) û Generic (internal) FE events RSRC-USAGE (0x5) û Resource utilization levels LC-EVENTS (0x6) û Logical Component events ALL-EVENTS (0x8) û All events reported Event Specific Data: This is the variable length event specific data, which can be encapsulated using TLV or OID or XML formats. For example, this could be used to specify what packets should be redirected to the CE. 5.5.2. Event Register Acknowledgement Gopal,Audu,Wu,Khosravi Expires September 2003 [Page 34] Internet Draft Forwarding and Control Element protocol March 2003 This is used by the FE to acknowledge CEÆs event registration request. The format of this payload is same as in the Event Register Request. 5.5.3. Event De-Register This is sent by the CE to the FE to indicate that it no longer is interested in receiving notifies for the events indicated in message. The format of this payload is same as in the Event Register Request. 5.5.4. Event De-Register Acknowledgement The FE sends this message to the CE to acknowledge CEÆs request not get any more notifies for events indicated in message. The format for this payload is same as in the Event Register Request. 5.5.5. Port-Link Status Change and Event Notify This is used to report asynchronous changes in port status, link status, etc, to the CE. The message contains the following: Event Type - Type of event Event ID - ID of event within type Port ID - port on which event occurred Diagnostic Info û Optional. The format of the notify message is as follows: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Tag (0xe) | Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Event Type | Event ID | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Port ID | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Tag (0x7) | Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ > Diagnostic Info < +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Valid values of the Event ID are as follows: Gopal,Audu,Wu,Khosravi Expires September 2003 [Page 35] Internet Draft Forwarding and Control Element protocol March 2003 PORT-DOWN (0x1) û Port is down. PORT-UP (0x2) û Port is up. PORT-ERROR (0x3) - Port is in non-catastrophic error state PORT-OK (0x4) û Port is no longer in error state LINK-DOWN (0x5) û Link attached to port is down; port is up. LINK-UP (0x6) û Link attached to port is up; port is up. LINK-ERROR (0x7) û Link attached to port is in error state. LINK-OK (0x8) û Link attached to port is no longer in error. BW-UTIL (0x9) û Bandwidth usage more than 75% of capacity PKT-ERROR (0xa) û Packet received in error at port 5.5.6. Generic FE Status Change and Event Notify This is used to report non-port or non-link related asynchronous events occurring in the FE. These could be overall FE errors or Logical Component specific events. The message contains the following: Event Type Event ID Logical Component Handle Diagnostic Info - optional The format of the GFENTFY is as follows: 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Tag (0x1e) | Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Event Type | Event ID | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Logical Component Handle | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Tag (0x7) | Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ > Diagnostic Info < +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Valid values of the Event ID are as follows: BUFF-OVFL (0x1) û Internal output buffer overflow onset BUFF-OK (0x2) û Internal output buffer overflow abated CPU-USAGE (0x3) û CPU usage more than 75% of capacity FE-ERROR (0x4) û FE in non-catastrophic error state FE-OK (0x5) û FE no longer in error state ALT-FE-ACT (0x6) û Alternate FE has become active (override) Gopal,Audu,Wu,Khosravi Expires September 2003 [Page 36] Internet Draft Forwarding and Control Element protocol March 2003 NO-FE-ACT (0x7) û No active FE available notify 5.5.7.Error Event Report The Error message is used to notify a master or slave peer or local management of an error event associated with an incoming message. For example, the message might be unexpected, given the current state, or a parameter value might be invalid. The Error message contains the following parameters: Error Code Diagnostic Information (Optional) The format of the Error Message is as follows: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Tag (0xc) | Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Error Code | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Tag (0x7) | Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ > Diagnostic Information < +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ The Error Code parameter indicates the reason for the error message. Possible error parameter values include: INV-PROT (0x01) - Invalid Protocol Version INV-ASSOC (0x02) û Invalid Association ID BAD-MSGCLS (0x03) û Unsupported message class BAD-MSGTYP (0x04) û Unsupported message type UNEX-MSG (0x05) û Unexpected message PROT-ERROR (0x06) û Protocol Error TWOPHASE-ERR (0x07) û Could not complete two-phase command COMM-FAIL (0x08) û Communication lost between CE and FE BAD-TRAF-MODE (0x09) û Unsupported Traffic Mode The optional Diagnostic information can be any information that can assist in identifying the error condition. To enhance debugging, the Diagnostic information could contain the first 40 bytes of offending message, for example. 5.6.Vendor Specific Function Message Handling Gopal,Audu,Wu,Khosravi Expires September 2003 [Page 37] Internet Draft Forwarding and Control Element protocol March 2003 This allows extensions to the FE functions so that new, currently unknown FE functionality (outside of those already specified) can be expressed. These messages will be transported transparently by the FACT protocol. Interpretation of the transported messages will be left solely to the application layers sitting on FACT in the CE and FE. 5.6.1. Vendor Specific Data (VS-DATA Request) Separated PEs may use this message to pass any information that is not to be consumed by FACT to each other. This message is not destined outside the involved PEs either. Application layers sitting on top of the FACT protocol layer can exchange information with this message between PEs. An example is the SNMP management layer making use of this message to perform necessary duties. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Tag (0x30) | Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ > Data to be transported < +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ As this message is opaque to FACT, the content is vendor-specific. FACT does not parse the content of this message. However, the data must be transported in chuncks of not larger than 256 octects. 5.6.2. Vendor Specific Data Ack (VS-Data Ack) This is used by the PE that receives an Inter-PE communication message to acknowledge the reception to the original sender. The format of this message is same as for VS-DATA Request. 6. Procedures for FACT Protocol 6.1.CE and FE State Maintenance FACT layer on the CE needs to maintain the states of the FEs it communicates with. Likewise, the FACT layer on the FE needs to maintain the states of the CEs the FE communicates with. The state of the (logical) NE also needs to be maintained. Since the NE is comprised of CEs and FEs, the NE state will be determined by the Gopal,Audu,Wu,Khosravi Expires September 2003 [Page 38] Internet Draft Forwarding and Control Element protocol March 2003 states of the contained FE and CE elements. Figure 6.1 below shows a hypothetical NE with its set of CEs and FEs |--------------------------------------------| | NE | | |------| |-------| | | | CE1 | | CE2 | | | |(active) | |(standby) | | | |------| |-------| | | ^ ^ ^ ^ | | | | | | | | | \-----| |-------/ | | | | |-------|-/ |---------/ | | v v v v | | |------| |-----| |-----| |-----| | | | FE1 | | FE2 | | FE3 | | FE4 | | | |(active) |--->|(active)| |(standby)| |(standby)| | | |------| |-----| |-----| |-----| | | ^ | | | | | | |---|------------|---------------------------| | v Figure 2. Showing logical NE and its components 6.1.1.CE and FE States The state of each configured FE and CE is maintained by the FACT layer. The state of each CE or FE element can change due to the following events: . Reception of management messages by CE. . Reception of management messages by FE. . Reception of control messages by FE from CE. . Loss of communication between CE and FE (e.g. due to faults). The CE and FE state transition diagram is shown in fig 3. Gopal,Audu,Wu,Khosravi Expires September 2003 [Page 39] Internet Draft Forwarding and Control Element protocol March 2003 +-------------+ +---------------------| | | Alternate +-------| PE-ACTIVE | | PE | +-------------+ | Takeover | ^ | | | CE/FE | | CE/FE | | Active | | Inactive | | | v | | +-------------+ | | | | | +------>| PE-INACT | | +-------------+ | ^ | CE/FE Down/ | CE/FE | | CE/FE Down / CE-FE COMM | Up | | CE-FE COMM FAIL FAIL | | v | +-------------+ +-------------------->| | | PE-DOWN | +-------------+ LEGEND: PE û Protocol Element (CE or FE) Figure 3. CE and FE State Transition Diagram The possible states of a protocol element (CE or FE) are: PE-DOWN: CE or FE is unavailable for service and/or the related CE-FE association is down. Initially, all CEs and FEs will be in this state. A CE or FE in this state should not be sent any traffic messages. PE-INACTIVE: The CE or FE is available for service and the related CE-FE association is up, but application traffic is stopped (the CE Or FE could be in a standby state for example). In this state, the CE or FE involved can be sent management, control, and non-traffic related messages. PE-ACTIVE: The CE or FE is available, and actively carrying application traffic. 6.1.2.NE States It may be necessary to track the state of the NE itself. Since the NE consists of distributed CEs and FEs, the state of the NE will be dependent on the states of its CEs and FEs. The state of the NE is maintained by FACT in both FE and CE. Gopal,Audu,Wu,Khosravi Expires September 2003 [Page 40] Internet Draft Forwarding and Control Element protocol March 2003 The state of an NE can be changed due to events including: . CE or FE state transitions . Recovery timer triggers The possible states of a NE are as follows: NE-DOWN: The network element is not available for service. This implies all related CEs and FEs are in the PE-DOWN state. Initially, the NE will be in this state. NE-INACTIVE: The network element is available but no application traffic is active. Here, one or more protocol elements (CE or FE) are in the PE-INACTIVE state, but none in the PE-ACTIVE state. Also, the recovery timer is not running, or has expired. This may be the state of standby NE if redundancy is provided at logical NE level. NE-ACTIVE: The network element is available and it is carrying application traffic. This implies that at least, one CE-FE communicating pair is in PE-ACTIVE state. NE-PENDING: An active CE or FE has transitioned into inactive or down state, and it was the last remaining active CE or FE in the NE. A recovery timer T(r), will be started, and the source FE or CE will queue up messages meant for the inactive target. If another target CE or FE becomes active (depending on which went inactive), before T(r) expires, the queued up messages are directed to the newly active CE or FE, and T(r) timer is cancelled. In this case, NE will move back to the NE-ACTIVE state. However, if T(r) expires before an alternate CE or FE becomes active, the queued up messages are discarded, and the NE will move to NE-DOWN state. +----------+ one PE goes ACTIVE +-------------+ | |------------------------>| | | NE-INACT | | NE-ACTIVE | | | | | | |< | | +----------+ \ +-------------+ ^ | \ Tr fires; ^ | | | \ at least one | | | | \ PE is UP | | | | \ | | | | \ | | | | \ | | one PE | | \ one PE | | Last ACTIVE PE Gopal,Audu,Wu,Khosravi Expires September 2003 [Page 41] Internet Draft Forwarding and Control Element protocol March 2003 goes | | all PEs \------\ goes to | | goes INACT to | | go DOWN \ ACTIVE | | or DOWN INACT | | \ | | (start Tr timer) | | \ | | | | \ | | | | \ | | | v \ | v +----------+ \ +-------------+ | | -| | | NE-DOWN | | NE-PENDING | | | | (queueing) | | |<------------------------| | +----------+ Tr Expiry and all +-------------+ PEs in DOWN state Tr = Recovery Timer PE = ProtocolElement Figure 3: NE State Transition Diagram 6.2.State Maintenance Procedures Before the establishment of a CE-FE association, the CE must be in- service and active but the FE is Down. Local management (CeMgr or FeMgr) can be used to effect appropriate state transitions of CEs and FEs. 6.2.1.PE-UP (Protocol Element Up) After an FE has successfully established an association with a CE, the FE sends a PE-UP message to indicate to the CE that it has finished all its internal configuration and is available. When the CE gets the PE-UP message, and the FE is not locked out for local management reasons, FACT at the CE will mark the FE as UP but ôInactiveö. The CE responds with a PE-UP Ack message in acknowledgement. If for any reason the CE cannot respond with a PE- UP, it will respond with a PE-DOWN Ack message with an appropriate reason parameter. The CE can also generate the PE-UP message. The last ACTIVE CE may have gone DOWN after establishing an association with a FE. In this case, the NE would first transition into the PENDING state for a duration of T(r ), and then to DOWN state. The first CE that transitions to UP state will send a PE-UP to the FE to notify it of Gopal,Audu,Wu,Khosravi Expires September 2003 [Page 42] Internet Draft Forwarding and Control Element protocol March 2003 its status, assuming the link between them is up. The FE will acknowledge with a PE-UP Ack. If the source PE does not receive a response from the target PE, or if a PE-DOWN Ack is received, source PE MAY resend PE-UP message every 2 seconds until it receives a PE-UP Ack from the target. After a few tries without an appropriate ACK message, the sending PE MAY reduce the frequency (for example, to every 5 seconds). 6.2.2.PE-DOWN (Protocol Element Down) The FE will send a PE-DOWN to the CE when the FE is to be removed from the list of FEs in an NE that it is a member of, that is eligible to receive application traffic or management messages. FACT at the CE marks the FE as ôDownö and returns a PE-DOWN Ack message to the FE if one of the following events occur: - a PE-DOWN message is received from the FE - another state message is received from an FE but the FE is locked out by management for some reason. The CE sends a PE-DOWN Ack message in response this message. If the FE does not receive a response from the CE, the FE MAY send PE-DOWN messages every 2 seconds until it receives a PE-DOWN Ack message from the CE or the association goes down. The FE may decide to reduce the frequency (to say, once every 5 seconds) if the PE-DOWN Ack is not received after a few tries. The CE may also send a PE-DOWN messaged to the FE. This occurs when the CE is about to be removed from service, and it is the ACTIVE CE. On getting this notification, the FE will respond with a PE-DOWN Ack, and stop sending any more messages to the out-going CE. The whole mechanism allows for a graceful removal of CEs or FEs. 6.2.3.FACT Version Control If a PE-UP message with an unsupported version is received, the receiving end responds with an error message indicating the version the receiving node supports and notifies its layer management. This is useful when protocol version upgrades are being performed in a network. A node upgraded to a newer version SHOULD support the older versions used on other nodes it is communicating with. Gopal,Audu,Wu,Khosravi Expires September 2003 [Page 43] Internet Draft Forwarding and Control Element protocol March 2003 6.2.4.PE-ACTIVE Any time after the CE has sent a PE-UP Ack to the FE, the CE can send a PE-Active (PEACT) to the FE, to activate the FE to start processing traffic. When a PEACT message is received, the FE responds with a PEACT Ack message, after which it starts handling traffic messages. The FE must wait for the PEACT message from the CE before handling traffic data. The CE only sends the PEACT message if it intends to transition the FE to ACTIVE state. There are different traffic modes that the NE can exist in: Over-ride-strong-consistency Over-ride-weak-consistency Load-shared The TYPE parameter in the PEACT message indicates the mode used in a particular NE. If the mode indicated in the PACT message is incompatible with the traffic handling mode currently used in the FE, the FE responds with an error message indicating ôUnsupported Traffic Modeö. In case of an OVER-RIDE mode FE, reception of a PEACT ACK message at the CE causes the re-direction of all subsequent control and data traffic to the FE that sent the PEACT ACK. Any previously active FE is now considered INACTIVE and will no longer receive traffic from the CE within the NE. The CE sends a Notify (Alternate FE-Active) message to the previously active FE in the NE. In the case of a LOAD-SHARE mode FE, reception of a PEACT ACK message by the CE causes the direction of traffic to the FE sending the PEACT ACK, in addition to all the other FEs that are currently active in the NE. The algorithm at the CE for load-sharing traffic within an NE to all the active FEs is implementation dependent. It could be based for example, on the destination interface IDs (ports), or on some application requirements. The FE responds to the PEACT with a PEACT Ack message to the CE. 6.2.5. PE Inactive When an FE wishes to withdraw from receiving traffic within an NE, the FE sends a PE-Inactive (PEINACT) to the NE object (or CE). The message should indicate which interface IDs (ports) on the FE are affected. It should also include the traffic mode of operation (e.g. LOAD_SHARED). If the CE determines that the mode included in the Gopal,Audu,Wu,Khosravi Expires September 2003 [Page 44] Internet Draft Forwarding and Control Element protocol March 2003 message is invalid, it responds with an Error message indicating ôUnsupported Traffic Modeö. In the case of an Over-ride mode FE, where normally another FE has already taken over the traffic within the NE with an Over-ride PEACT, the FE which sent the PEINACT is already considered inactive by the NE. A PEINACT Ack message is sent to the FE, after ensuring all traffic to the FE is stopped. In the case of a Load-share mode FE, FACT in the NE moves the FE to the ôInactiveö state, and its traffic re-allocated across the remaining ôactiveö FEs per the load distribution algorithm currently used within the NE. A PEINACT Ack message is sent to the FE after all traffic to it has been halted. A PEACT request may be sent to any FE in UP state, if required (in the case when too few FEs are active to guarantee service reliability). If no other FEs are active in the NE, the CE sends a NOTIFY (PENDING) to all inactive FEs in the NE, and either discards all incoming messages for the FEs or starts buffering the incoming messages for T( r) seconds, after which messages will be discarded. If the CE receives a PEACT from an FE in the NE before T( r) expires, the buffered messages are directed to the FE and the timer is cancelled. If T( r) expires, the NE moves to the ôInactiveö state. 7. Example Scenarios 7.1.Establishment of Association between CEs and FEs The associations among CEs and FEs are established via join request and response messages. If a join request is granted, a join response message is replied to the join request message. If a join request is denied, a Leave response message is replied to the join request message. This association process may be followed by security exchanges, capability query, topology query,. If the capabilities match, the FE sends a PE-UP message to the CE. CE responds with PE-UP Ack. According to the configuration, the CE can send a PE-ACTIVE to inform the FE to go active. FE acknowledges it with a PE-ACTIVE Ack to indicate it is ready to take traffic. The sequences of messages are illustrated in the Figure below. FE CE | | Gopal,Audu,Wu,Khosravi Expires September 2003 [Page 45] Internet Draft Forwarding and Control Element protocol March 2003 | Join REQ | 1 |---------------------->| | | | Join RESP | 2 |<----------------------| | | | CAPABILITY Request | 3 |<----------------------| | | | CAPABILITY Response | 4 |---------------------->| | | | TOPOLOGY Request | 5 |---------------------->| | | | CONFIG FE (RTE/LOGIC) | 6 |<----------------------| | | | CONFIG FE ACK | 7 |---------------------->| | | | PE UP | 8 |---------------------->| | | | PE UP ACK | 9 |---------------------->| | | | PE ACT | 10|<----------------------| | | | PE-ACT ACK | 11|---------------------->| Figure 4: Showing Establishment messages between CE and FE 7.2.Steady State Communication Once CE and FE establish their association and exchange initial configuration information, they enter a phase of steady state communication, with the following example messages exchanging. FE CE | | |Heart Beat | 1 |<--------------------->| | | |Heart Beat ACK | 2 |<--------------------->| | | |Query Request | Gopal,Audu,Wu,Khosravi Expires September 2003 [Page 46] Internet Draft Forwarding and Control Element protocol March 2003 3 |<----------------------| | | |Query Response | 4 |---------------------->| | | |Port Event Notice | 5 |---------------------->| | | |Port Event Notice ACK | 6 |<----------------------| | | |Configure logic components (new configuration) 7 |<----------------------| | | |Configure logic components ACK 8 |---------------------->| | | |Control Packet Redirect| 9 |---------------------->| | | |Control Packet Redirect ACK 10|<----------------------| | | When transferring forwarding information to FE, the CE uses the configure logical component message with a 16-bit length field indicating the number of bytes of configuration information. If the CE has a very large amount of database that exceeds 64K bytes in length, it should send multiple configure logical component messages to complete the configuration. 7.3.Two FEs in NE (1+1 Over-ride Redundancy Mode) 7.4.Two FEs in NE (1+1 sparing, load-sharing Mode) 7.5. CE Fail-over Support for 1+1 Redundancy This scenario refers to the case that FEs can start communication with a backup (standby) CE in the event that a designated (active) CE fails. Both the designated and backup CEs are pre-configured during the pre-association phase. FEs can detect communication failure via reception of leave request/response messages and loss of heart beat messages. Gopal,Audu,Wu,Khosravi Expires September 2003 [Page 47] Internet Draft Forwarding and Control Element protocol March 2003 Once a FE that has the fail-over capability detects a break-down of communication with a designated CE, the FE can send a leave request message to the CE optionally. The leave reason code should be 0x06 for fail-over switching. In the event that the break-down is uni- directional, the designated CE may receive the leave request and gracefully prepare for the de-association with the FE. The FE should send a join request message to the standby CE to establish an association with it. In the join messages, the High Availability Support (HAS) field indicates the capability of fail- over support in FEs. The FE should use the original FE identifier in the join request message so that the standby CE may upload proper configuration to the FE, if the CE has any, via a join response message. 8. Architecture support for FACT protocol Pre-association phase is used to configure certain key attributes. FE-Manager and CE-Manager are responsible for providing that information. 8.1.Security If FE or CE is going to communicate over the untrust domain, then it is the responsibility of the administrator to choose appropriate lower layer security protocols. For example if the FE and CE are communicating in a shared LAN or Ethernet etc, then layer 2 encryption is used. If it is going be above IP layer either TLS or IPSec can be used. FACT protocol is just a payload for those protocols and does not carry any explicitly fields for authentication or encryption. 8.2. High availability support Fail-over, load sharing etc, are part of high-availability treatment. Since CE-CE communication is out of scope (but FACT protocol can be applied there also). FE-Manager can configure FE to perform one of the following functions - Strong consistency: Send all asynchronous notification to the entire CE in the CE set. - Weak consistency: Send all the asynchronous notification events only to one CE in the CE set, that CE may be acting as primary CE, when that CE is not active use the next one in the list or a backup CE. Gopal,Audu,Wu,Khosravi Expires September 2003 [Page 48] Internet Draft Forwarding and Control Element protocol March 2003 This type of requirement does not affect the protocol; the protocol has provision and bit fields to communicate the FE capabilities to CE during the join message request. 8.3. Access Control to FE It is up to the implementer to configure the number of CE that can access FE, as such protocol is flexible and has 16-bit fields to identify a CE. 8.4.Configurable parameters The following are the currently identified configurable parameters that can be done through FE-Manager and CE-Manager for FE and CEÆs respectively (1) Load Sharing (optional) (2) Fail over configuration (optional) (3) Load Balancing (optional) (4) Security Keys (optional) (5) Number of CE to which it has to communicate (6) Maximum number of CE (7) Whether FE should notify to CEÆs regarding the change in configuration (which happened via SNMP or CLI) (8) Timer for health check (9) Data format support. If FE and CE can support different data formats say TLV, OID, XML etc. Then this has to established during this phase. (10) Maximum numbers of FEs that each CE can support in a NE (11) Retransmission Timer 8.5.Management Interface to FE and CE through FE-Manager and CE- Manager. Previous section describes the FE and CE minimal configuration and information that are to be supported as part of pre-association phase. During the normal operation, it may be required to know the association and dynamic change of such attributes and capability configuration of FE and CE. This can be done either through CLI or SNMP or any other suitable mechanism. This information can be modeled as MIB for Forces protocol and their control elements. The following are the key attributes that are needed to support such operation. Gopal,Audu,Wu,Khosravi Expires September 2003 [Page 49] Internet Draft Forwarding and Control Element protocol March 2003 (1) Add new CE endpoint to the FE (2) Add new FE endpoint to the CE (3) (Re)Establish an association with CE or FE (4) Leave (Terminate) an association with CE or FE (5) Update the capabilities of FE or CE 9. References 1. S. Bradner, "The Internet Standards Process -Revision 3", RFC 2026, October 1996. 2. S. Bradner, "Keywords for use in RFCs to Indicate Requirement Levels", RFC2119 (BCP), IETF, March 1997. 3. Anderson, et. al., öRequirements for Separation of IP Control and Forwardingö, work in progress, February 2002,,IETF. 4. Ram Gopal, öForwarding Element Modellingö,work in progress, February 2002, , IETF 5. L. Yang, et. al, ö ForCES Architectural Frameworkö,work in progressö, June 2002, 6. L. Yang, et. al, ö ForCES Forwarding Element Functional Modelö, work in progressö, June 2002,< draft-yang-forces-model-00.txt> 7. Morneault, K,. Rengasami, S,. Kalla, M., and G. Sidebottom, "ISDN Q.921-User Adaptation Layer", RFC 3057, February 2001 10. Acknowledgments I would like to thank Man Li, Nokia Research Center for her suggestions and comments. 11. Authors' Addresses Ram Gopal Nokia Research Center 5, Wayside Road, Burlington, MA 01803 Phone: 1-781-993-3685 Email: ram.gopal@nokia.com Alex Audu Alcatel R&I Gopal,Audu,Wu,Khosravi Expires September 2003 [Page 50] Internet Draft Forwarding and Control Element protocol March 2003 1000 Coit Road Plano, TX 75075 Phone: 1-972-477-7809 Email: alex.audu@alcatel.com Chaoping Wu Azanda Network Devices 250 Santa Ana Court Sunnyvale, CA 94085 Phone: 1-408-720-3117 Email: chaoping_wu@yahoo.com Hormuzd Khosravi Intel 2111 NE 25th Avenue Hillsboro, OR 97124 Phone: 1-503-264-0334 Email: hormuzd.m.khosravi@intel.com Appendix-1: Tag (Hex) Values Used in FACT Messages +-----+--------------------+---------------------------------------+ |Tag | Meaning | Messages | +-----+--------------------+---------------------------------------+ |000A |Leave Reason | Leave Request/Response | +-----+--------------------+---------------------------------------+ |0004 |Info Stream |Leave Request/Response, PE (IN)ACT/ACK | +-----+--------------------+---------------------------------------+ |000F |Release Reason | Release Request/Response | +-----+--------------------+---------------------------------------+ |000B |PE ACT-Traffic Mode | PE (IN)ACT/ACK | +-----+--------------------+---------------------------------------+ |0010 |Join Capability | Join Request | +-----+--------------------+---------------------------------------+ |0011 |Join Configuration | Join Response | +-----+--------------------+---------------------------------------+ |0013 |Logic Compo ID List | Statistics Request | +-----+--------------------+---------------------------------------+ |0014 |Logic Compo Stats | Statistics Response | +-----+--------------------+---------------------------------------+ |0015 |Abnormality Code | PE Abnormal Notification | +-----+--------------------+---------------------------------------+ |0016 |Abnormal Action Code| PE Abnormal Notification ACK | +-----+--------------------+---------------------------------------+ Appendix-2: Interfaces To Local Management As part of any normal protocol operation, management interface either CLI or EMS or NMS or other suitable entity would like to send commands to either FE or Gopal,Audu,Wu,Khosravi Expires September 2003 [Page 51] Internet Draft Forwarding and Control Element protocol March 2003 CE. This section identifies minimal command set that may be required for such operation. This may be implemented by each vendor in various ways and is beyound the scope of ForCES protocol. This section describes the minimal message and how FE and CE may use FACT to inform the local management operation to each other. +----------------------+ (NMS or EMS act as CE) <--FACT---> CE (FEÆs w.r.to NMS)| | ^ | | | |Network Element | | FACT | | | | | | | | V | | FE . . . FE | +----------------------+ M-PE-STATUS The status of CEs and FEs are stored in and tracked by FACT. This primitive is used to request, confirm, and indicate the status of a PE (FE or CE) to local management. M-ASSOCIATION-STATUS This is used to request and indicate the status of the association between a CE and an FE. M-ERROR This is used to indicate an error with a received FACT message to local management. M-PE-UP This primitive can be used by local management to request that a PE be restored into in-service (UP) state by FACT. This could be triggered by a RESTORE request from the CLI (Command Line Interface) into Local Management. (CLI)-----Restore(COLD)--->(LM)----- M-PE-UP(COLD)--->(FACT) OR other possible configuration is +----------------------+ (NMS or EMS act as CE) <--FACT---> CE (FEÆs w.r.to NMS)| | ^ | | | | | NE | FACT | | | | Gopal,Audu,Wu,Khosravi Expires September 2003 [Page 52] Internet Draft Forwarding and Control Element protocol March 2003 | | | | V | | FE . . . FE | +----------------------+ FACT can also use this primitive to indicate or acknowledge to local management (LM) that a PE is UP (with a M-PE-UP.indicate and M-PE-UP.confirm primitives respectively). Valid parameters to M-PE-UP.req are: COLD û initialize all attributes of PE during restore process. WARM û use previous attributes in memory to restore PE (assumes those Attributes have not been lost). CONFIG - Restore PE with new configuration attributes. M-PE-DOWN This can be used by local management to request that a PE be taken to the DOWN State. It could be triggered for example, by CLI sending a Remove request to the local management layer. (CLI) ----Remove(BOOT)--->(LM)----- M-PE-DOWN(BOOT) --->(FACT) FACT can in turn indicate a PE-DOWN event or confirm the DOWN request with a M-PE-DOWN.ind or M-PE-DOWN.con respectively. Valid parameters to M-PE-DOWN.req are: NO-BOOT û previous (static) contents in memory are preserved. BOOT û all previous contents in memory are zeroÆd out. CONFIG û previous configuration information between FE and CE are lost and new ones are being established. M-PE-ACTIVE This is used by FACT to inform local management that PE has gone ACTIVE. M-PE-INACTIVE This is used by FACT to inform local management that PE has gone INACTIVE. Gopal,Audu,Wu,Khosravi Expires September 2003 [Page 53]