idnits 2.17.1 draft-ietf-forces-framework-05.txt: -(80): Line appears to be too long, but this could be caused by non-ascii characters in UTF-8 encoding -(413): Line appears to be too long, but this could be caused by non-ascii characters in UTF-8 encoding -(971): Line appears to be too long, but this could be caused by non-ascii characters in UTF-8 encoding -(996): Line appears to be too long, but this could be caused by non-ascii characters in UTF-8 encoding -(1332): Line appears to be too long, but this could be caused by non-ascii characters in UTF-8 encoding -(1333): Line appears to be too long, but this could be caused by non-ascii characters in UTF-8 encoding -(1345): Line appears to be too long, but this could be caused by non-ascii characters in UTF-8 encoding -(1372): Line appears to be too long, but this could be caused by non-ascii characters in UTF-8 encoding -(1373): Line appears to be too long, but this could be caused by non-ascii characters in UTF-8 encoding -(1418): Line appears to be too long, but this could be caused by non-ascii characters in UTF-8 encoding -(1472): Line appears to be too long, but this could be caused by non-ascii characters in UTF-8 encoding -(1547): Line appears to be too long, but this could be caused by non-ascii characters in UTF-8 encoding -(1552): Line appears to be too long, but this could be caused by non-ascii characters in UTF-8 encoding -(1581): Line appears to be too long, but this could be caused by non-ascii characters in UTF-8 encoding -(1601): Line appears to be too long, but this could be caused by non-ascii characters in UTF-8 encoding -(1604): Line appears to be too long, but this could be caused by non-ascii characters in UTF-8 encoding -(1607): Line appears to be too long, but this could be caused by non-ascii characters in UTF-8 encoding -(1613): Line appears to be too long, but this could be caused by non-ascii characters in UTF-8 encoding -(1616): Line appears to be too long, but this could be caused by non-ascii characters in UTF-8 encoding -(1619): Line appears to be too long, but this could be caused by non-ascii characters in UTF-8 encoding -(1622): Line appears to be too long, but this could be caused by non-ascii characters in UTF-8 encoding -(1628): Line appears to be too long, but this could be caused by non-ascii characters in UTF-8 encoding -(1631): Line appears to be too long, but this could be caused by non-ascii characters in UTF-8 encoding -(1634): Line appears to be too long, but this could be caused by non-ascii characters in UTF-8 encoding -(1637): Line appears to be too long, but this could be caused by non-ascii characters in UTF-8 encoding -(1640): Line appears to be too long, but this could be caused by non-ascii characters in UTF-8 encoding -(1643): Line appears to be too long, but this could be caused by non-ascii characters in UTF-8 encoding -(1646): Line appears to be too long, but this could be caused by non-ascii characters in UTF-8 encoding -(1649): Line appears to be too long, but this could be caused by non-ascii characters in UTF-8 encoding Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** Looks like you're using RFC 2026 boilerplate. This must be updated to follow RFC 3978/3979, as updated by RFC 4748. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- ** Missing document type: Expected "INTERNET-DRAFT" in the upper left hand corner of the first page ** The document seems to lack a 1id_guidelines paragraph about Internet-Drafts being working documents. ** The document seems to lack a 1id_guidelines paragraph about 6 months document validity -- however, there's a paragraph with a matching beginning. Boilerplate error? == There are 42 instances of lines with non-ascii characters in the document. == No 'Intended status' indicated for this document; assuming Proposed Standard Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack an IANA Considerations section. (See Section 2.2 of https://www.ietf.org/id-info/checklist for how to handle the case when there are no actions for IANA.) ** There are 594 instances of too long lines in the document, the longest one being 33 characters in excess of 72. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the RFC 3978 Section 5.4 Copyright Line does not match the current year -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (June 2003) is 7619 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) -- Looks like a reference, but probably isn't: 'RFC-2119' on line 103 == Unused Reference: '12' is defined on line 1637, but no explicit reference was found in the text == Unused Reference: '16' is defined on line 1649, but no explicit reference was found in the text -- Possible downref: Non-RFC (?) normative reference: ref. '3' == Outdated reference: A later version (-13) exists of draft-ietf-idr-restart-06 -- Obsolete informational reference (is this intentional?): RFC 2246 (ref. '13') (Obsoleted by RFC 4346) -- Obsolete informational reference (is this intentional?): RFC 2401 (ref. '14') (Obsoleted by RFC 4301) -- Obsolete informational reference (is this intentional?): RFC 2409 (ref. '15') (Obsoleted by RFC 4306) Summary: 6 errors (**), 0 flaws (~~), 6 warnings (==), 7 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Internet Draft L. Yang 3 Expiration: Dec 2003 Intel Corp. 4 File: draft-ietf-forces-framework-05.txt R. Dantu 5 Working Group: ForCES Univ. of North Texas 6 T. Anderson 7 Intel Corp. 8 R. Gopal 9 Nokia 10 June 2003 12 Forwarding and Control Element Separation (ForCES) Framework 14 draft-ietf-forces-framework-05.txt 16 Status of this Memo 18 This document is an Internet-Draft and is in full conformance with 19 all provisions of Section 10 of RFC2026. Internet-Drafts are 20 working documents of the Internet Engineering Task Force (IETF), its 21 areas, and its working groups. Note that other groups may also 22 distribute working documents as Internet-Drafts. 24 Internet-Drafts are draft documents valid for a maximum of six 25 months and may be updated, replaced, or obsoleted by other documents 26 at any time. It is inappropriate to use Internet-Drafts as 27 reference material or to cite them other than as ``work in 28 progress.'' 30 The list of current Internet-Drafts can be accessed at 31 http://www.ietf.org/ietf/1id-abstracts.txt. 33 The list of Internet-Draft Shadow Directories can be accessed at 34 http://www.ietf.org/shadow.html. 36 Copyright Notice 38 Copyright (C) The Internet Society (2002). All Rights Reserved. 40 Abstract 42 This document defines the architectural framework for the ForCES 43 (Forwarding and Control Element Separation) network elements, and 44 identifies the associated entities and the interaction among them. 46 Table of Contents 47 1. Definitions.....................................................3 48 2. Introduction to Forwarding and Control Element Separation 49 (ForCES)...........................................................5 50 3. Architecture....................................................8 51 3.1. Control Elements and Fr Reference Point....................9 52 3.2. Forwarding Elements and Fi reference point................10 53 3.3. CE Managers...............................................13 54 3.4. FE Managers...............................................13 55 4. Operational Phases.............................................13 56 4.1. Pre-association Phase.....................................13 57 4.1.1. Fl Reference Point...................................13 58 4.1.2. Ff Reference Point...................................14 59 4.1.3. Fc Reference Point...................................15 60 4.2. Post-association Phase and Fp reference point.............15 61 4.2.1. Proximity and Interconnect between CEs and FEs.......15 62 4.2.2. Association Establishment............................16 63 4.2.3. Steady-state Communication...........................17 64 4.2.4. Data Packets across Fp reference point...............18 65 4.2.5. Proxy FE.............................................19 66 4.3. Association Re-establishment..............................19 67 4.3.1. CE graceful restart..................................19 68 4.3.2. FE restart...........................................20 69 5. Applicability to RFC1812.......................................21 70 5.1. General Router Requirements...............................22 71 5.2. Link Layer................................................23 72 5.3. Internet Layer Protocols..................................23 73 5.4. Internet Layer Forwarding.................................24 74 5.5. Transport Layer...........................................24 75 5.6. Application Layer -- Routing Protocols....................25 76 5.7. Application Layer -- Network Management Protocol..........25 77 6. Summary........................................................25 78 7. Security Considerations........................................26 79 7.1. Analysis of Potential Threats Introduced by ForCES........26 80 7.1.1. ��Join�� or ��Remove�� Flooding on CEs...................26 81 7.1.2. Impersonation Attack.................................27 82 7.1.3. Replay Attack........................................27 83 7.1.4. Attack during Fail Over..............................27 84 7.1.5. Data Integrity.......................................27 85 7.1.6. Data Confidentiality.................................27 86 7.1.7. Sharing security parameters..........................28 87 7.1.8. Denial of Service Attack via External Interface......28 88 7.2. Security Recommendations for ForCES.......................28 89 7.2.1. Security Configuration...............................29 90 7.2.2. Using TLS with ForCES................................29 91 7.2.3. Using IPsec with ForCES..............................30 92 8. Normative References...........................................31 93 9. Informative References.........................................32 94 10. Acknowledgement...............................................32 95 11. Authors' Addresses............................................33 96 12. Intellectual Property Right...................................33 97 13. Full Copyright Statement......................................33 98 Conventions used in this document 100 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 101 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in 102 this document are to be interpreted as described in [RFC-2119]. 104 1. Definitions 106 A set of terminology associated with the ForCES requirements is 107 defined in [3] and we only include the ones that are most relevant 108 to this document here. 110 Addressable Entity (AE) - - An entity that is directly addressable 111 given some interconnect technology. For example, on IP networks, it 112 is a device to which we can communicate using an IP address; and on 113 a switch fabric, it is a device to which we can communicate using a 114 switch fabric port number. 116 Physical Forwarding Element (PFE) - An AE that includes hardware 117 used to provide per-packet processing and handling. This hardware 118 may consist of (but is not limited to) network processors, ASICs, or 119 general processors, installed on line cards, daughter boards, 120 mezzanine cards, or in stand-alone boxes. 122 PFE Partition - A logical partition of a PFE consisting of some 123 subset of each of the resources (e.g., ports, memory, forwarding 124 table entries) available on the PFE. This concept is analogous to 125 that of the resources assigned to a virtual switching element as 126 described in [8]. 128 Physical Control Element (PCE) - An AE that includes hardware used 129 to provide control functionality. This hardware typically includes 130 a general-purpose processor. 132 PCE Partition - A logical partition of a PCE consisting of some 133 subset of each of the resources available on the PCE. 135 Forwarding Element (FE) - A logical entity that implements the 136 ForCES protocol. FEs use the underlying hardware to provide per- 137 packet processing and handling as directed by a CE via the ForCES 138 protocol. FEs may happen to be a single blade (or PFE), a partition 139 of a PFE or multiple PFEs. 141 Control Element (CE) - A logical entity that implements the ForCES 142 protocol and uses it to instruct one or more FEs how to process 143 packets. CEs handle functionality such as the execution of control 144 and signaling protocols. CEs may consist of PCE partitions or whole 145 PCEs. 147 ForCES Network Element (NE) - An entity composed of one or more CEs 148 and one or more FEs. To entities outside a NE, the NE represents a 149 single point of management. Similarly, a NE usually hides its 150 internal organization from external entities. 152 Pre-association Phase - The period of time during which a FE Manager 153 (see below) and a CE Manager (see below) are determining which FE 154 and CE should be part of the same network element. 156 Post-association Phase - The period of time during which a FE does 157 know which CE is to control it and vice versa, including the time 158 during which the CE and FE are establishing communication with one 159 another. 161 ForCES Protocol - While there may be multiple protocols used within 162 the overall ForCES architecture, the term "ForCES protocol" refers 163 only to the ForCES post-association phase protocol (see below). 165 ForCES Post-Association Phase Protocol - The protocol used for post- 166 association phase communication between CEs and FEs. This protocol 167 does not apply to CE-to-CE communication, FE-to-FE communication, or 168 to communication between FE and CE managers. The ForCES protocol is 169 a master-slave protocol in which FEs are slaves and CEs are masters. 170 This protocol includes both the management of the communication 171 channel (e.g., connection establishment, heartbeats) and the control 172 messages themselves. This protocol could be a single protocol or 173 could consist of multiple protocols working together. 175 FE Manager - A logical entity that operates in the pre-association 176 phase and is responsible for determining to which CE(s) a FE should 177 communicate. This process is called CE discovery and may involve 178 the FE manager learning the capabilities of available CEs. A FE 179 manager may use anything from a static configuration to a pre- 180 association phase protocol (see below) to determine which CE(s) to 181 use, however this is currently out of scope. Being a logical 182 entity, a FE manager might be physically combined with any of the 183 other logical entities mentioned in this section. 185 CE Manager - A logical entity that operates in the pre-association 186 phase and is responsible for determining to which FE(s) a CE should 187 communicate. This process is called FE discovery and may involve 188 the CE manager learning the capabilities of available FEs. A CE 189 manager may use anything from a static configuration to a pre- 190 association phase protocol (see below) to determine which FE to use, 191 however this is currently out of scope. Being a logical entity, a 192 CE manager might be physically combined with any of the other 193 logical entities mentioned in this section. 195 Pre-association Phase Protocol - A protocol between FE managers and 196 CE managers that is used to determine which CEs or FEs to use. A 197 pre-association phase protocol may include a CE and/or FE capability 198 discovery mechanism. Note that this capability discovery process is 199 wholly separate from (and does not replace) that used within the 200 ForCES protocol. However, the two capability discovery mechanisms 201 may utilize the same FE model. 203 FE Model - A model that describes the logical processing functions 204 of a FE. 206 ForCES Protocol Element - A FE or CE. 208 FE Topology -- Representation of how the multiple FEs in a single NE 209 are interconnected. Sometimes it is called inter-FE topology, to be 210 distinguished from intra-FE topology (or FE block topology) used by 211 FE model. 213 Inter-FE topology - see FE Topology. 215 2. Introduction to Forwarding and Control Element Separation (ForCES) 217 An IP network element (NE) appears to external entities as a 218 monolithic piece of network equipment, e.g., a router, NAT, 219 firewall, or load balancer. Internally, however, an IP network 220 element (NE) (such as a router) is composed of numerous logically 221 separated entities that cooperate to provide a given functionality 222 (such as routing). Two types of network element components exist: 223 control element (CE) in control plane and forwarding element (FE) in 224 forwarding plane (or data plane). Forwarding elements typically are 225 ASIC, network-processor, or general-purpose processor-based devices 226 that handle data path operations for each packet. Control elements 227 are typically based on general-purpose processors that provide 228 control functionality like routing and signaling protocols. 230 ForCES aims to define a framework and associated protocol(s) to 231 standardize information exchange between the control and forwarding 232 plane. Having standard mechanisms allows CEs and FEs to become 233 physically separated standard components. This physical separation 234 accrues several benefits to the ForCES architecture. Separate 235 components would allow component vendors to specialize in one 236 component without having to become experts in all components. 237 Standard protocol also allows the CEs and FEs from different 238 component vendors to interoperate with each other and hence it 239 becomes possible for system vendors to integrate together the CEs 240 and FEs from different component suppliers. This interoperability 241 translates into a lot more design choices and flexibility to the 242 system vendors. Overall, ForCES will enable rapid innovation in 243 both the control and forwarding planes while maintaining 244 interoperability. Scalability is also easily provided by this 245 architecture in that additional forwarding or control capacity can 246 be added to existing network elements without the need for forklift 247 upgrades. 249 ------------------------- ------------------------- 250 | Control Blade A | | Control Blade B | 251 | (CE) | | (CE) | 252 ------------------------- ------------------------- 253 ^ | ^ | 254 | | | | 255 | V | V 256 --------------------------------------------------------- 257 | Switch Fabric Backplane | 258 --------------------------------------------------------- 259 ^ | ^ | ^ | 260 | | | | � | | 261 | V | V | V 262 ------------ ------------ ------------ 263 |Router | |Router | |Router | 264 |Blade #1 | |Blade #2 | |Blade #N | 265 | (FE) | | (FE) | | (FE) | 266 ------------ ------------ ------------ 267 ^ | ^ | ^ | 268 | | | | � | | 269 | V | V | V 271 Figure 1. A router configuration example with separate blades. 273 One example of such physical separation is at the blade level. 274 Figure 1 shows such an example configuration of a router, with two 275 control blades and multiple router (forwarding) blades, all 276 interconnected into a switch fabric backplane. In such chassis 277 configuration, the control blades are the CEs while the router 278 blades are FEs, and the switch fabric backplane provides the 279 physical interconnect for all the blades. Control blade A may be 280 the primary CE while control blade B is the backup CE providing 281 redundancy. It is also possible to have a redundant switch fabric 282 for high availability support. Routers today with this kind of 283 configuration use proprietary interface for messaging between CEs 284 and FEs. The goal of ForCES is to replace such proprietary 285 interface with a standard protocol. With a standard protocol like 286 ForCES implemented on all blades, it becomes possible for control 287 blades from vendor X and routing blades from vendor Y to work 288 seamlessly together in one chassis. 290 ------- ------- 291 | CE1 | | CE2 | 292 ------- ------- 293 ^ ^ 294 | | 295 V V 296 ============================================ Ethernet 297 ^ ^ � ^ 298 | | | 299 V V V 300 ------- ------- -------- 301 | FE#1| | FE#2| | FE#n | 302 ------- ------- -------- 303 ^ | ^ | ^ | 304 | | | | | | 305 | V | V | V 307 Figure 2. A router configuration example with separate boxes. 309 Another level of physical separation between the CEs and FEs can be 310 at the box level. In such configuration, all the CEs and FEs are 311 physically separated boxes, interconnected with some kind of high 312 speed LAN connection (like Gigabit Ethernet). These separated CEs 313 and FEs are only one hop away from each other within a local area 314 network. The CEs and FEs communicate to each other by running 315 ForCES, and the collection of these CEs and FEs together become one 316 routing unit to the external world. Figure 2 shows such an example. 318 In both examples shown here, the same physical interconnect is used 319 for both CE-to-FE and FE-to-FE communication. However, that does 320 not have to be the case. One reason to use different interconnect 321 is that CE-to-FE interconnect does not have to be as fast as the FE- 322 to-FE interconnect, so the more expensive fast connections can be 323 saved for FE-to-FE only. The separate interconnects may also 324 provide reliability and redundancy benefits for the NE. 326 Some examples of control functions that can be implemented in the CE 327 include routing protocols like RIP, OSPF and BGP, control and 328 signaling protocols like RSVP (Resource Reservation Protocol), LDP 329 (Label Distribution Protocol) for MPLS, etc. Examples of forwarding 330 functions in FE include LPM (longest prefix match) forwarder, 331 classifiers, traffic shaper, meter, NAT, etc. Figure 3 shows a 332 diagram with examples in both CE and FE. Any given NE may contain 333 one or many of these CE and FE functions in it. The diagram also 334 shows that ForCES protocol is used to transport both the control 335 messages for ForCES itself and the data packets that are 336 originated/destined from/to the control functions in CE (e.g., 337 routing packets). Section 4.2.4 provides more detail on this. 339 ------------------------------------------------- 340 | | | | | | | 341 |OSPF |RIP |BGP |RSVP |LDP |� | 342 | | | | | | | 343 ------------------------------------------------- 344 | ForCES Interface | 345 ------------------------------------------------- 346 ^ ^ 347 ForCES | |data 348 control | |packets 349 messages| |(e.g., routing packets) 350 v v 351 ------------------------------------------------- 352 | ForCES Interface | 353 ------------------------------------------------- 354 | | | | | | | 355 |LPM Fwd|Meter |Shaper |NAT |Classi-|� | 356 | | | | |fier | | 357 ------------------------------------------------- 358 | FE resources | 359 ------------------------------------------------- 360 Figure 3. Examples of CE and FE functions 362 A set of requirements for control and forwarding separation is 363 identified in [3]. This document describes a ForCES architecture 364 that satisfies the architectural requirements of that document and 365 defines a framework for ForCES network elements and the associated 366 entities to facilitate protocol definition. Whenever necessary, 367 this document uses many examples to illustrate the issues and/or 368 possible solutions in ForCES. These examples are intended to be 369 just examples, and should not be taken as the only or definite ways 370 of doing certain things. It is expected that separate document will 371 be produced by the ForCES working group to specify the ForCES 372 protocol(s). 374 3. Architecture 376 This section defines the ForCES architectural framework and the 377 associated logical components. This ForCES framework defines 378 components of ForCES NEs including several ancillary components. 379 These components may be connected in different kinds of topologies 380 for flexible packet processing. 382 --------------------------------------- 383 | ForCES Network Element | 384 -------------- Fc | -------------- -------------- | 385 | CE Manager |---------+-| CE 1 |------| CE 2 | | 386 -------------- | | | Fr | | | 387 | | -------------- -------------- | 388 | Fl | | | Fp / | 389 | | Fp| |----------| / | 390 | | | |/ | 391 | | | | | 392 | | | Fp /|----| | 393 | | | /--------/ | | 394 -------------- Ff | -------------- -------------- | 395 | FE Manager |---------+-| FE 1 | Fi | FE 2 | | 396 -------------- | | |------| | | 397 | -------------- -------------- | 398 | | | | | | | | | | 399 ----+--+--+--+----------+--+--+--+----- 400 | | | | | | | | 401 | | | | | | | | 402 Fi/f Fi/f 404 Figure 4. ForCES Architectural Diagram 406 The diagram in Figure 4 shows the logical components of the ForCES 407 architecture and their relationships. There are two kinds of 408 components inside a ForCES network element: control element (CE) and 409 forwarding element (FE). The framework allows multiple instances of 410 CE and FE inside one NE. Each FE contains one or more physical 411 media interfaces for receiving and transmitting packets from/to the 412 external world. The aggregation of these FE interfaces becomes the 413 NE�s external interfaces. In addition to the external interfaces, 414 there must also exist some kind of interconnect within the NE so 415 that the CE and FE can communicate with each other, and one FE can 416 forward packets to another FE. The diagram also shows two entities 417 outside of the ForCES NE: CE Manager and FE Manager. These two 418 entities provide configuration to the corresponding CE or FE in the 419 pre-association phase (see Section 5.1). There is no defined role 420 for FE Manager and CE Manager in post-association phase, thus these 421 logical components are not considered part of the ForCES NE. 423 For convenience, the logical interactions between these components 424 are labeled by reference points Fp, Fc, Ff, Fr, Fl, and Fi, as shown 425 in Figure 4. The FE external interfaces are labeled as Fi/f. More 426 detail is provided in Section 4 and 5 for each of these reference 427 points. All these reference points are important in understanding 428 the ForCES architecture, however, the ForCES protocol is only 429 defined over one reference point -- Fp. 431 The interface between two ForCES NEs is identical to the interface 432 between two conventional routers and these two NEs exchange the 433 protocol packets through the external interfaces at Fi/f. ForCES 434 NEs connect to existing routers transparently. 436 3.1. Control Elements and Fr Reference Point 438 It is not necessary to define any protocols across the Fr reference 439 point to enable control and forwarding separation for simple 440 configurations like single CE and multiple FEs. However, this 441 architecture permits multiple CEs to be present in a network 442 element. In cases where an implementation uses multiple CEs, the 443 invariant that the CEs and FEs together appear as a single NE must 444 be maintained. 446 Multiple CEs may be used for redundancy, load sharing, distributed 447 control, or other purposes. Redundancy is the case where one or 448 more CEs are prepared to take over should an active CE fail. Load 449 sharing is the case where two or more CEs are concurrently active 450 and any request that can be serviced by one of the CEs can also be 451 serviced by any of the other CEs. For both redundancy and load 452 sharing, the CEs involved are equivalently capable. The only 453 difference between these two cases is in terms of how many active 454 CEs there are. Distributed control is the case where two or more 455 CEs are concurrently active but certain requests can only be 456 serviced by certain CEs. 458 When multiple CEs are employed in a ForCES NE, their internal 459 organization is considered an implementation issue that is beyond 460 the scope of ForCES. CEs are wholly responsible for coordinating 461 amongst themselves via the Fr reference point to provide consistency 462 and synchronization. However, ForCES does not define the 463 implementation or protocols used between CEs, nor does it define how 464 to distribute functionality among CEs. Nevertheless, ForCES will 465 support mechanisms for CE redundancy or fail over, and it is 466 expected that vendors will provide redundancy or fail over solutions 467 within this framework. 469 3.2. Forwarding Elements and Fi reference point 471 FE is a logical entity that implements the ForCES protocol and uses 472 the underlying hardware to provide per-packet processing and 473 handling as directed by a CE. It is possible to partition one 474 physical FE into multiple logical FEs. It is also possible for one 475 FE to use multiple physical FEs. The mapping between physical FE(s) 476 and the logical FE(s) is beyond the scope of ForCES. For example, a 477 logical partition of a physical FE can be created by assigning some 478 portion of each of the resources (e.g., ports, memory, forwarding 479 table entries) available on the physical FE to each of the logical 480 FEs. Such concept of FE virtualization is analogous to a virtual 481 switching element as described in [8]. FE virtualization should 482 occur only in the pre-association phase and hence has no impact on 483 ForCES. 485 FEs perform all packet processing functions as directed by CEs. 486 FEs have no initiative of their own. Instead, FEs are slaves and 487 only do as they are told. FEs may communicate with one or more CEs 488 concurrently across reference point Fp. FEs have no notion of CE 489 redundancy, load sharing, or distributed control. Instead, FEs 490 accept commands from any CE authorized to control them, and it is up 491 to the CEs to coordinate among themselves to achieve redundancy, 492 load sharing or distributed control. The idea is to keep FEs as 493 simple and dumb as possible so that FEs can focus its resource on 494 the packet processing functions. 496 ------- Fr ------- 497 | CE1 | ------| CE2 | 498 ------- ------- 499 | \ / | 500 | \ / | 501 | \ / | 502 | \/Fp | 503 | /\ | 504 | / \ | 505 | / \ | 506 ------- Fi ------- 507 | FE1 |<----->| FE2 | 508 ------- ------- 510 Figure 5. CE redundancy example. 512 For example, in Figure 5, FE1 and FE2 can be configured to accept 513 commands from both the primary CE (CE1) and the backup CE (CE2). 514 Upon detection of CE1 failure, perhaps across the Fr or Fp reference 515 point, CE2 is configured to take over activities of CE1. This is 516 beyond the scope of ForCES and is not discussed further. 518 Distributed control can be achieved in the similar fashion, without 519 much intelligence on the part of FEs. For example, FEs can be 520 configured to detect RSVP and BGP protocol packets, and forward RSVP 521 packets to one CE and BGP packets to another CE. Hence, FEs may 522 need to do packet filtering for forwarding packets to specific CEs. 524 This architecture permits multiple FEs to be present in a NE. [3] 525 dictates that the ForCES protocol must be able to scale to at least 526 hundreds of FEs (see [3] Section 5, requirement #11). Each of these 527 FEs may potentially have a different set of packet processing 528 functions, with different media interfaces. FEs are responsible for 529 basic maintenance of layer-2 connectivity with other FEs and with 530 external entities. Many layer-2 media include sophisticated control 531 protocols. The FORCES protocol (over the Fp reference point) will 532 be able to carry messages for such protocols so that, in keeping 533 with the "dumb FE�� model, the CE can provide appropriate 534 intelligence and control over these media. 536 When multiple FEs are present, ForCES requires that packets must be 537 able to arrive at the NE by one FE and leave the NE via a different 538 FE (See [3], Section 5, Requirement #3). Packets that enter the NE 539 via one FE and leave the NE via a different FE are transferred 540 between FEs across the Fi reference point. Fi reference point could 541 be used by FEs to discovery their (inter-FE) topology, perhaps 542 during pre-association phase. The Fi reference point is a separate 543 protocol from the Fp reference point and is not currently defined by 544 the ForCES architecture. 546 ----------------- 547 | CE | 548 ----------------- 549 ^ ^ ^ 550 / | \ 551 / v \ 552 / ------- \ 553 / +->| FE3 |<-+ \ 554 / | | | | \ 555 v | ------- | v 556 ------- | | ------- 557 | FE1 |<-+ +->| FE2 | 558 | |<--------------->| | 559 ------- ------- 560 ^ | ^ | 561 | | | | 562 | v | v 564 (a) Full mesh among FE1, FE2 and FE3. 566 ----------- 567 | CE | 568 ----------- 569 ^ ^ ^ ^ 570 / | | \ 571 /------ | | ------\ 572 v v v v 573 ------- ------- ------- ------- 574 | FE1 |<->| FE2 |<->| FE3 |<->| FE4 | 575 ------- ------- ------- ------- 576 ^ | ^ | ^ | ^ | 577 | | | | | | | | 578 | v | v | v | v 580 (b) Multiple FEs in a daisy chain 582 ^ | 583 | v 584 ----------- 585 | FE1 |<-----------------------| 586 ----------- | 587 ^ ^ | 588 / \ | 589 | ^ / \ ^ | V 590 v | v v | v ---------- 591 --------- --------- | | 592 | FE2 | | FE3 |<------------>| CE | 593 --------- --------- | | 594 ^ ^ ^ ---------- 595 | \ / ^ ^ 596 | \ / | | 597 | v v | | 598 | ----------- | | 599 | | FE4 |<----------------------| | 600 | ----------- | 601 | | ^ | 602 | v | | 603 | | 604 |----------------------------------------| 606 (c) Multiple FEs connected by a ring 608 Figure 6. Some examples of FE topology. 610 FEs could be connected in different kinds of topologies and packet 611 processing may spread across several FEs in the topology. Hence, 612 logical packet flow may be different from physical FE topology. 613 Figure 6 provides some topology examples. When it is necessary to 614 forward packets between FEs, the CE needs to understand the FE 615 topology. The FE topology can be queried from the FEs by CEs. 617 3.3. CE Managers 619 CE managers are responsible for determining which FEs a CE should 620 control. It is legitimate for CE managers to be hard-coded with the 621 knowledge of with which FEs its CEs should communicate. A CE 622 manager may also be physically embedded into a CE and be implemented 623 as a simple keypad or other direct configuration mechanism on the 624 CE. Finally, CE managers may be physically and logically separate 625 entities that configure the CE with FE information via such 626 mechanisms as COPS-PR [6] or SNMP [4]. 628 3.4. FE Managers 630 FE managers are responsible for determining to which CE any 631 particular FE should initially communicate. Like CE managers, no 632 restrictions are placed on how a FE manager decides to which CE its 633 FEs should communicate, nor are restrictions placed on how FE 634 managers are implemented. Each FE should have one and only one FE 635 manager, while different FEs may have the same or different FE 636 manager(s). Each manager can choose to exist and operate 637 independent of other manager. 639 4. Operational Phases 641 Both FEs and CEs require some configuration in place before they can 642 start information exchange and function as a coherent network 643 element. Two operational phases are identified in this framework -- 644 pre-association and post-association. 646 4.1.Pre-association Phase 648 Pre-association phase is the period of time during which an FE 649 Manager and a CE Manager are determining which FE and CE should be 650 part of the same network element. The protocols used during this 651 phase may include all or some of the message exchange over Fl, Ff 652 and Fc reference points. However, all these may be optional and 653 none of this is within the scope of ForCES protocol. 655 4.1.1. Fl Reference Point 657 CE managers and FE managers may communicate across the Fl reference 658 point in the pre-association phase in order to determine which CEs 659 and FEs should communicate with each other. Communication across 660 the Fl reference point is optional in this architecture. No 661 requirements are placed on this reference point. 663 CE managers and FE managers may be operated by different entities. 664 The operator of the CE manager may not want to divulge, except to 665 specified FE managers, any characteristics of the CEs it manages. 666 Similarly, the operator of the FE manager may not want to divulge FE 667 characteristics, except to authorized entities. As such, CE 668 managers and FE managers may need to authenticate one another. 669 Subsequent communication between CE managers and FE managers may 670 require other security functions such as privacy, non-repudiation, 671 freshness, and integrity. 673 FE Manager FE CE Manager CE 674 | | | | 675 | | | | 676 |(security exchange) | | 677 1|<------------------------------>| | 678 | | | | 679 |(a list of CEs and their attributes) | 680 2|<-------------------------------| | 681 | | | | 682 |(a list of FEs and their attributes) | 683 3|------------------------------->| | 684 | | | | 685 | | | | 686 |<----------------Fl------------>| | 688 Figure 7. An example of message exchange over Fl reference point 690 Once the necessary security functions have been performed, the CE 691 and FE managers communicate to determine which CEs and FEs should 692 communicate with each other. At the very minimum, the CE and FE 693 managers need to learn of the existence of available FEs and CEs 694 respectively. This discovery process may or may not entail one or 695 both managers learning the capabilities of the discovered ForCES 696 protocol elements. Figure 7 shows an example of possible message 697 exchange between CE manager and FE manager over Fl reference point. 699 4.1.2. Ff Reference Point 701 The Ff reference point is used to inform forwarding elements of the 702 association decisions made by the FE manager in pre-association 703 phase. Only authorized entities may instruct a FE with respect to 704 which CE should control it. Therefore, privacy, integrity, 705 freshness, and authentication are necessary between the FE manager 706 and FEs when the FE manager is remote to the FE. Once the 707 appropriate security has been established, the FE manager instructs 708 the FEs across this reference point to join a new NE or to 709 disconnect from an existing NE. The FE Manager could also assign 710 unique FE identifiers to the FEs using this reference point. The FE 711 identifiers are useful in post association phase to express FE 712 topology. Figure 8 shows example of message exchange over Ff 713 reference point. 715 Note that the FE manager function may be co-located with the FE 716 (such as by manual keypad entry of the CE IP address), in which case 717 this reference point is reduced to a built-in function. 719 FE Manager FE CE Manager CE 720 | | | | 721 | | | | 722 |(security exchange) |(security exchange) 723 1|<------------>|authentication 1|<----------->|authentication 724 | | | | 725 |(FE ID, attributes) |(CE ID, attributes) 726 2|<-------------|request 2|<------------|request 727 | | | | 728 3|------------->|response 3|------------>|response 729 |(corresponding CE ID) |(corresponding FE ID) 730 | | | | 731 | | | | 732 |<-----Ff----->| |<-----Fc---->| 734 Figure 8. Examples of message exchange 735 over Ff and Fc reference points. 737 4.1.3. Fc Reference Point 739 The Fc reference point is used to inform control elements of the 740 association decisions made by CE managers in pre-association phase. 741 When the CE manager is remote, only authorized entities may instruct 742 a CE to control certain FEs. Privacy, integrity, freshness and 743 authentication are also required across this reference point in such 744 a configuration. Once appropriate security has been established, 745 the CE manager instructs CEs as to which FEs they should control and 746 how they should control them. Figure 7 shows example of message 747 exchange over Fc reference point. 749 As with the FE manager and FEs, configurations are possible where 750 the CE manager and CE are co-located and no protocol is used for 751 this function. 753 4.2. Post-association Phase and Fp reference point 755 Post-association phase is the period of time during which a FE and 756 CE have been configured with information necessary to contact each 757 other and includes both association establishment and steady-state 758 communication. The communication between CE and FE is performed 759 across the Fp ("p" meaning protocol) reference point. ForCES 760 protocol is exclusively used for all communication across the Fp 761 reference point. 763 4.2.1. Proximity and Interconnect between CEs and FEs 765 The ForCES Working Group has made a conscious decision that the 766 first version of ForCES will not be designed to support 767 configurations where the CE and FE are located arbitrarily in the 768 network. In particular, ForCES is intended for "very close" CE/FE 769 localities in IP networks, as defined by ForCES Applicability 770 Statement ([7]). Very Close localities consist of control and 771 forwarding elements that either are components in the same physical 772 box, or are separated at most by one local network hop. 774 CEs and FEs can be connected by a variety of interconnect 775 technologies, including Ethernet connections, backplanes, ATM (cell) 776 fabrics, etc. ForCES should be able to support each of these 777 interconnects (see [3] Section 5, requirement #1). ForCES will make 778 use of an existing RFC2914 ([2]) compliant L4 protocol with adequate 779 reliability, security and congestion control (e.g. TCP, SCTP) for 780 transport purposes. 782 4.2.2. Association Establishment 784 FE CE 785 | | 786 |(Security exchange.) | 787 1|<--------------------->| 788 | | 789 |(Let me join the NE please.) 790 2|---------------------->| 791 | | 792 |(What kind of FE are you? -- capability query) 793 3|<----------------------| 794 | | 795 |(Here is my FE functions/state: use model to describe) 796 4|---------------------->| 797 | | 798 |(How are you connected with other FEs?) 799 5|<----------------------| 800 | | 801 |(Here is the FE topology info) 802 6|---------------------->| 803 | | 804 |(Initial config for FE -- optional) 805 7|<----------------------| 806 | | 807 |(I am ready to go. Shall I?) 808 8|---------------------->| 809 | | 810 |(Go ahead!) | 811 9|<----------------------| 812 | | 814 Figure 9. Example of message exchange between CE and FE 815 over Fp to establish NE association 817 As an example, figure 9 shows some of the message exchange that may 818 happen before the association between the CE and FE is fully 819 established. Either the CE or FE can initiate the connection. 820 Security handshake is necessary to authenticate the two 821 communication endpoints to each other before any further message 822 exchange can happen. The exact details of the security handshake 823 depend on the security solution chosen by ForCES protocol. It is 824 most likely that either IPSec or TLS will be used. Section 9 825 provides more details on the security considerations for ForCES. 826 After the successful security handshake, the FE needs to inform the 827 CE of its own capability and its topology in relation to other FEs. 828 The capability of the FE is represented by the FE model, described 829 in a separate document. The model would allow a FE to describe what 830 kind of packet processing functions it contains, in what order the 831 processing happens, what kinds of configurable parameters it allows, 832 what statistics it collects and what events it might throw, etc. 833 Once such information is available to the CE, the CE may choose to 834 send some initial or default configuration to the FE so that the FE 835 can start receiving and processing packets correctly. Such 836 initialization may not be necessary if the FE already obtains the 837 information from its own bootstrap process. Once FE starts 838 accepting packets for processing, we say the association of this FE 839 with its CE is now established. From then on, the CE and FE enter 840 steady-state communication. 842 4.2.3. Steady-state Communication 844 Once an association is established between the CE and FE, the ForCES 845 protocol is used by the CE and FE over Fp reference point to 846 exchange information to facilitate packet processing. 848 FE CE 849 | | 850 |(Add these new routes.)| 851 1|<----------------------| 852 | | 853 |(Successful.) | 854 2|---------------------->| 855 | | 856 | | 857 |(Query some stats.) | 858 1|<----------------------| 859 | | 860 |(Reply with stats collected.) 861 2|---------------------->| 862 | | 863 | | 864 |(My port is down, with port #.) 865 1|---------------------->| 866 | | 867 |(Here is a new forwarding table) 868 2|<----------------------| 869 | | 870 | | 872 Figure 10. Examples of message exchange between CE and FE 873 over Fp during steady-state communication 874 Based on the information acquired through CEs' control processing, 875 CEs will frequently need to manipulate the packet-forwarding 876 behaviors of their FE(s) by sending instructions to FEs. For 877 example, Figure 10 shows message exchange examples in which the CE 878 sends new routes to the FE so that the FE can add them to its 879 forwarding table. The CE may query the FE for statistics collected 880 by the FE and the FE may notify the CE of important events such as 881 port failure. 883 4.2.4. Data Packets across Fp reference point 885 Control plane protocol packets (such as RIP, OSPF messages) 886 addressed to any of NE's interfaces are typically redirected by the 887 receiving FE to its CE, and CE may originate packets and have its FE 888 deliver them to other NEs. Therefore, ForCES protocol over Fp not 889 only transports the ForCES protocol messages between CEs and FEs, 890 but also encapsulates the data packets from control plane protocols. 891 Moreover, one FE may be controlled by multiple CEs for distributed 892 control. In this configuration, the control protocols supported by 893 the FORCES NEs may spread across multiple CEs. For example, one CE 894 may support routing protocols like OSPF and BGP, while a signaling 895 and admission control protocol like RSVP is supported in another CE. 896 FEs are configured to recognize and filter these protocol packets 897 and forward them to the corresponding CE. 899 --------------------- ---------------------- 900 | | | | 901 | +--------+ | | +--------+ | 902 | |CE(BGP) | | | |CE(BGP) | | 903 | +--------+ | | +--------+ | 904 | | | | ^ | 905 | |Fp | | |Fp | 906 | v | | | | 907 | +--------+ | | +--------+ | 908 | | FE | | | | FE | | 909 | +--------+ | | +--------+ | 910 | | | | ^ | 911 | Router | | | Router | | 912 | A | | | B | | 913 ---------+----------- -----------+---------- 914 v ^ 915 | | 916 | | 917 ------------------->--------------- 919 Figure 11. Example to show data packet flow between two NEs. 921 Figure 11 shows one example of how the BGP packets originated by 922 router A are passed to router B. In this example, the ForCES 923 protocol is used to transport the packets from the CE to the FE 924 inside router A, and then from the FE to the CE inside router B. In 925 light of the fact that the ForCES protocol is responsible for 926 transporting both the control messages and the data packets between 927 the CE and FE over Fp reference point, it is possible to use either 928 a single protocol or multiple protocols to achieve this. 930 4.2.5. Proxy FE 932 In the case where a physical FE cannot implement (e.g., due to the 933 lack of a general purpose CPU) the ForCES protocol directly, a proxy 934 FE can be used in the middle of Fp reference point. This allows the 935 CE communicate to the physical FE via the proxy by using ForCES, 936 while the proxy manipulates the physical FE using some intermediary 937 form of communication (e.g., a non-ForCES protocol or DMA). In such 938 an implementation, the combination of the proxy and the physical FE 939 becomes one logical FE entity. 941 4.3. Association Re-establishment 943 FEs and CEs may join and leave NEs dynamically (see [3] Section 5, 944 requirements #12). When a FE or CE leaves the NE, the association 945 with the NE is broken. If the leaving party rejoins a NE later, to 946 re-establish the association, it may or may not need to re-enter the 947 pre-association phase. Loss of association can also happen 948 unexpectedly due to loss of connection between the CE and the FE. 949 Therefore, the framework allows the bi-directional transition 950 between these two phases, but the ForCES protocol is only applicable 951 for the post-association phase. However, the protocol should 952 provide mechanisms to support association re-establishment. This 953 includes the ability for CEs and FEs to determine when there is a 954 loss of association between them, ability to restore association and 955 efficient state (re)synchronization mechanisms (see [3] Section 5, 956 requirement #7). Note that security association and state must be 957 also re-established to guarantee the same level of security exists 958 before and after the association re-establishment. 960 4.3.1. CE graceful restart 962 The failure and restart of the CE in a router can potentially cause 963 much stress and disruption on the control plane throughout a 964 network. Because when a CE has to restart for any reason, the 965 router loses routing adjacencies or sessions with its routing 966 neighbors. Neighbors who detect the lost adjacency normally re- 967 compute new routes and then send routing updates to their own 968 neighbors to communicate the lost adjacency. Their neighbors do the 969 same thing to propagate throughout the network. In the meantime, 970 the restarting router cannot receive traffic from other routers 971 because the neighbors have stopped using the router�s previously 972 advertised routes. When the restarting router restores adjacencies, 973 neighbors must once again re-compute new routes and send out 974 additional routing updates. The restarting router is unable to 975 forward packets until it has re-established routing adjacencies with 976 neighbors, received route updates through these adjacencies, and 977 computed new routes. Until convergence takes place throughout the 978 network, packets may be lost in transient black holes or forwarding 979 loops. 981 A high availability mechanism known as the "graceful restart" has 982 been used by the IP routing protocols (OSPF [10], BGP [11], BGP 983 [11]) and MPLS label distribution protocol (LDP [9]) to help 984 minimize the negative effects on routing throughout an entire 985 network caused by a restarting router. Route flap on neighboring 986 routers is avoided, and a restarting router can continue to forward 987 packets that would otherwise be dropped. 989 While the details differ from protocol to protocol, the general idea 990 behind the graceful restart mechanism remains the same. With the 991 graceful restart, a restarting router can inform its neighbors when 992 it restarts. The neighbors may detect the lost adjacency but do not 993 recompute new routes or send routing updates to their neighbors. 994 The neighbors also hold on to the routes received from the 995 restarting router before restart and assume they are still valid for 996 a limited time. By doing so, the restarting router�s FEs can also 997 continue to receive and forward traffic from other neighbors for a 998 limited time by using the routes they already have. The restarting 999 router then re-establishes routing adjacencies, downloads updated 1000 routes from all its neighbors, recomputes new routes and uses them 1001 to replace the older routes it was using. It then sends these 1002 updated routes to its neighbors and signals the completion of the 1003 graceful restart process. 1005 Non-stop forwarding is a requirement for graceful restart. It is 1006 necessary so a router can continue to forward packets while it is 1007 downloading routing information and recomputing new routes. This 1008 ensures that packets will not be dropped. As one can see, one of 1009 the benefits afforded by the separation of CE and FE is exactly the 1010 ability of non-stop forwarding in the face of the CE failure and 1011 restart. The support of dynamic changes to CE/FE association in 1012 ForCES also makes it compatible with high availability mechanisms 1013 such as graceful restart. 1015 ForCES should be able to support CE graceful restart easily. When 1016 the association is established the first time, the CE must inform 1017 the FEs what to do in the case of CE failure. If graceful restart 1018 is not supported, the FEs may be told to stop packet processing all 1019 together if its CE fails. If graceful restart is supported, the FEs 1020 should be told to cache and hold on to its FE state including the 1021 forwarding tables across the restarts. A timer must be included so 1022 that the timeout causes such cached state to expire eventually. 1023 Those timers should be settable by the CE. 1025 4.3.2. FE restart 1027 In the same example in Figure 5, assuming CE1 is the working CE for 1028 the moment, what would happen if one of the FEs, say FE1, leaves the 1029 NE temporarily? FE1 may voluntarily decide to leave the 1030 association. Alternatively, FE1 may stop functioning simply due to 1031 unexpected failure. In former case, CE1 receives a "leave- 1032 association request" from FE1. In the latter, CE1 detects the 1033 failure of FE1 by some other means. In both cases, CE1 must inform 1034 the routing protocols of such an event, most likely prompting a 1035 reachability and SPF (Shortest Path First) recalculation and 1036 associated downloading of new FIBs from CE1 to the other remaining 1037 FEs (only FE2 in this example). Such recalculation and FIB update 1038 will also be propagated from the CE1 to its neighbors that are 1039 affected by the connectivity of FE1. 1041 When FE1 decides to rejoin again, or when it restarts again from the 1042 failure, FE1 needs to re-discover its master (CE). This can be 1043 achieved by several means. It may re-enter the pre-association 1044 phase and get that information from its FE manager. It may retrieve 1045 the previous CE information from its cache, if it can validate the 1046 information freshness. Once it discovers its CE, it starts message 1047 exchange with CE to re-establish the association just as outlined in 1048 Figure 9, with the possible exception that it might be able to 1049 bypass the transport of the complete initial configuration. Suppose 1050 that FE1 still has its routing table and other state information 1051 from the last association, instead of sending all the information 1052 again from scratch, it may be able to use more efficient mechanism 1053 to re-sync up the state with its CE if such mechanism is supported 1054 by the ForCES protocol. For example, CRC-32 of the state might give 1055 a quick indication of whether or not the state is in-sync with its 1056 CE. By comparing its state with CE first, it sends information 1057 update only if it is needed. ForCES protocol may choose to 1058 implement similar optimization mechanisms, but it may also choose 1059 not to, as this is not a requirement. 1061 5. Applicability to RFC1812 1063 [3] Section 5, requirement #9 dictates that "All proposed ForCES 1064 architecture must explain how that architecture supports all of the 1065 router functions as defined in RFC1812." RFC1812 discusses many 1066 important requirements for IPv4 routers from the link layer to the 1067 application layer. This section addresses the relevant requirements 1068 in RFC1812 for implementing IPv4 routers based on ForCES 1069 architecture and explains how ForCES satisfies these requirements by 1070 providing guidelines on how to separate the functionalities required 1071 into forwarding plane and control plane. 1073 In general, the forwarding plane carries out the bulk of the per- 1074 packet processing that is required at line speed, while the control 1075 plane carries most of the computationally complex operations that 1076 are typical of the control and signaling protocols. However, it is 1077 impossible to draw a rigid line to divide the processing into CEs 1078 and FEs cleanly. Nor should the ForCES architecture limit the 1079 innovative approaches in control and forwarding plane separation. 1080 As more and more processing power is available in the FEs, some of 1081 the control functions that traditionally are performed by CEs may 1082 now be moved to FEs for better performance and scalability. Such 1083 offloaded functions may include part of ICMP or TCP processing, or 1084 part of routing protocols. Once off-loaded onto the forwarding 1085 plane, such CE functions, even though logically belonging to the 1086 control plane, now become part of the FE functions. Just like the 1087 other logical functions performed by FEs, such off-loaded functions 1088 must be expressed as part of the FE model so that the CEs can decide 1089 how to best take advantage of these off-loaded functions when 1090 present on the FEs. 1092 5.1. General Router Requirements 1094 Routers have at least two or more logical interfaces. When CEs and 1095 FEs are separated by ForCES within a single NE, some additional 1096 interfaces are needed for intra-NE communications. Figure 12 shows 1097 an example to illustrate that. This NE contains one CE and two FEs. 1098 Each FE has four interfaces; two of them are used for receiving and 1099 transmitting packets to the external world, while the other two are 1100 for intra-NE connections. CE has two logical interfaces #9 and #10, 1101 connected to interfaces #3 and #6 from FE1 and FE2, respectively. 1102 Interface #4 and #5 are connected for FE1-FE2 communication. 1103 Therefore, this router NE provides four external interfaces (#1, 2, 1104 7 and 8). 1106 --------------------------------- 1107 | router NE | 1108 | ----------- ----------- | 1109 | | FE1 | | FE2 | | 1110 | ----------- ----------- | 1111 | 1| 2| 3| 4| 5| 6| 7| 8| | 1112 | | | | | | | | | | 1113 | | | | +----+ | | | | 1114 | | | | | | | | 1115 | | | 9| 10| | | | 1116 | | | -------------- | | | 1117 | | | | CE | | | | 1118 | | | -------------- | | | 1119 | | | | | | 1120 -----+--+----------------+--+---- 1121 | | | | 1122 | | | | 1124 Figure 12. A router NE example with four interfaces. 1126 IPv4 routers must implement IP to support its packet forwarding 1127 function, which is driven by its FIB (Forwarding Information Base). 1128 This Internet layer forwarding (see RFC1812 [1] Section 5) 1129 functionality naturally belongs to FEs in the ForCES architecture. 1131 A router may implement transport layer protocols (like TCP and UDP) 1132 that are required to support application layer protocols (see 1133 RFC1812 [1] Section 6). One important class of application 1134 protocols is routing protocols (see RFC1812 [1] Section 7). In 1135 ForCES architecture, routing protocols are naturally implemented by 1136 CEs. Routing protocols require routers communicate with each other. 1138 This communication between CEs in different routers is supported in 1139 ForCES by FEs' ability to redirect data packets addressed to routers 1140 (i.e., NEs) and CEs' ability to originate packets and have them 1141 delivered by their FEs. This communication occurs across Fp 1142 reference point inside each router and between neighboring routers' 1143 external interfaces, as illustrated in Figure 11. 1145 5.2.Link Layer 1147 Since FEs own all the external interfaces for the router, FEs need 1148 to conform to the link layer requirements in RFC1812. Arguably, ARP 1149 support may be implemented in either CEs or FEs. As we will see 1150 later, a number of behaviors that RFC1812 mandates fall into this 1151 category -- they may be performed by the FE and may be performed by 1152 the CE. A general guideline is needed to ensure interoperability 1153 between separated control and forwarding planes. The guideline we 1154 offer here is that in general CEs are required to be capable of 1155 these kind of operations while FEs may or may not choose to 1156 implement them. FE model should indicate its capabilities in this 1157 regard so that CEs can decide where these functions are implemented. 1159 Interface parameters, including MTU, IP address, etc., must be 1160 configurable by CEs via ForCES. CEs must be able to determine 1161 whether a physical interface in an FE is available to send packets 1162 or not. FEs must also inform CEs the status change of the 1163 interfaces (like link up/down) via ForCES. 1165 5.3.Internet Layer Protocols 1167 Both FEs and CEs must implement IP protocol and all mandatory 1168 extensions as RFC1812 specified. CEs should implement IP options 1169 like source route and record route while FEs may choose to implement 1170 those as well. Timestamp option should be implemented by FEs to 1171 insert the timestamp most accurately. FE must interpret the IP 1172 options that it understands and preserve the rest unchanged for use 1173 by CEs. Both FEs and CEs might choose to silently discard packets 1174 without sending ICMP errors, but such events should be logged and 1175 counted. FEs may report statistics for such events to CEs via 1176 ForCES. 1178 When multiple FEs are involved to process packets, the appearance of 1179 single NE must be strictly maintained. For example, Time-To-Live 1180 (TTL) must be decremented only once within a single NE. For 1181 example, it can be always decremented by the last FE with egress 1182 function. 1184 FEs must receive and process normally any packets with a broadcast 1185 destination address or a multicast destination address that the 1186 router has asked to receive. When IP multicast is supported in 1187 routers, IGMP is implemented in CEs. CEs are also required of ICMP 1188 support, while it is optional for FEs to support ICMP. Such an 1189 option can be communicated to CEs as part of the FE model. 1190 Therefore, FEs can always rely upon CEs to send out ICMP error 1191 messages, but FEs also have the option to generate ICMP error 1192 messages themselves. 1194 5.4.Internet Layer Forwarding 1196 IP forwarding is implemented by FEs. When the routing table is 1197 updated at CEs, ForCES is used to send the new route entries from 1198 CEs to FEs. Each FE has its own forwarding table and uses this 1199 table to direct packets to the next hop interface. 1201 Upon receiving IP packets, FE verifies the IP header and processes 1202 most of the IP options. Some options cannot be processed until the 1203 routing decision has been made. Routing decision is made after 1204 examining the destination IP address. If the destination address 1205 belongs to the router itself, the packets are filtered and either 1206 processed locally or forwarded to CE, depending upon the 1207 instructions set-up by CE. Otherwise, FE determines the next hop IP 1208 address by looking up in its forwarding table. FE also determines 1209 the network interface it uses to send the packets. Sometimes FE may 1210 need to forward the packets to another FE before packets can be 1211 forwarded out to the next hop. Right before packets are forwarded 1212 out to the next hop, FE decrements TTL by 1 and processes any IP 1213 options that cannot be processed before. FE performs any IP 1214 fragmentation if necessary, determines link layer address (e.g., by 1215 ARP), and encapsulates the IP datagram (or each of the fragments 1216 thereof) in an appropriate link layer frame and queues it for output 1217 on the interface selected. 1219 Other options mentioned in RFC1812 for IP forwarding may also be 1220 implemented at FEs, for example, packet filtering. 1222 FEs typically forward packets destined locally to CEs. FEs may also 1223 forward exceptional packets (packets that FEs do not know how to 1224 handle) to CEs. CEs are required to handle packets forwarded by FEs 1225 for whatever different reasons. It might be necessary for ForCES to 1226 attach some meta-data with the packets to indicate the reasons of 1227 forwarding from FEs to CEs. Upon receiving packets with meta-data 1228 from FEs, CEs can decide to either process the packets themselves, 1229 or pass the packets to the upper layer protocols including routing 1230 and management protocols. If CEs are to process the packets by 1231 themselves, CEs may choose to discard the packets, or modify and re- 1232 send the packets. CEs may also originate new packets and deliver 1233 them to FEs for further forwarding. 1235 Any state change during router operation must also be handled 1236 correctly according to RFC1812. For example, when an FE ceases 1237 forwarding, the entire NE may continue forwarding packets, but it 1238 needs to stop advertising routes that are affected by the failed FE. 1240 5.5.Transport Layer 1242 Transport layer is typically implemented at CEs to support higher 1243 layer application protocols like routing protocols. In practice, 1244 this means that most CEs implement both the Transmission Control 1245 Protocol (TCP) and the User Datagram Protocol (UDP). 1247 Both CEs and FEs need to implement ForCES protocol. If some layer-4 1248 transport is used to support ForCES, then both CEs and FEs need to 1249 implement the L4 transport and ForCES protocols. 1251 5.6. Application Layer -- Routing Protocols 1253 Interior and exterior routing protocols are implemented on CEs. The 1254 routing packets originated by CEs are forwarded to FEs for delivery. 1255 The results of such protocols (like forwarding table updates) are 1256 communicated to FEs via ForCES. 1258 For performance or scalability reasons, portions of the control 1259 plane functions that need faster response may be moved from the CEs 1260 and off-loaded onto the FEs. For example in OSPF, the Hello 1261 protocol packets are generated and processed periodically. When 1262 done at CEs, the inbound Hello packets have to traverse from the 1263 external interfaces at the FEs to the CEs via the internal CE-FE 1264 channel. Similarly, the outbound Hello packets have to go from the 1265 CEs to the FEs and to the external interfaces. Frequent Hello 1266 updates place heavy processing overhead on the CEs and can overwhelm 1267 the CE-FE channel as well. Since typically there are far more FEs 1268 than CEs in a router, the off-loaded Hello packets are processed in 1269 a much more distributed and scalable fashion. By expressing such 1270 off-loaded functions in the FE model, we can ensure 1271 interoperability. 1273 5.7. Application Layer -- Network Management Protocol 1275 RFC1812 also dictates "Routers MUST be manageable by SNMP." (see 1276 [4] Section 8) In general, for post-association phase, most external 1277 management tasks (including SNMP) should be done through interaction 1278 with the CE in order to support the appearance of a single 1279 functional device. Therefore, it is recommended that SNMP 1280 management agent be implemented by CEs and the SNMP messages 1281 received by FEs be redirected to their CEs. AgentX framework 1282 defined in RFC2741 ([5]) may be applied here such that CEs act in 1283 the role of master agent to process SNMP protocol messages while FEs 1284 act in the role of subagent to provide access to the MIB objects 1285 residing on FEs. AgentX protocol messages between the master agent 1286 (CE) and the subagent (FE) are encapsulated and transported via 1287 ForCES, just like data packets from any other application layer 1288 protocols. 1290 6. Summary 1292 This document defines an architectural framework for ForCES. It 1293 identifies the relevant components for a ForCES network element, 1294 including (one or more) FEs, (one or more) CEs, one optional FE 1295 manager, and one optional CE manager. It also identifies the 1296 interaction among these components and discusses all the major 1297 reference points. It is important to point out that, among all the 1298 reference points, only the Fp interface between CEs and FEs is 1299 within the scope of ForCES. ForCES alone may not be enough to 1300 support all desirable NE configurations. However, we believe that 1301 ForCES over Fp interface is the most important element in realizing 1302 physical separation and interoperability of CEs and FEs, and hence 1303 the first interface that ought to be standardized. Simple and 1304 useful configurations can still be implemented with only CE-FE 1305 interface being standardized, e.g., single CE with full-meshed FEs. 1307 7. Security Considerations 1309 In general, the physical separation of two entities usually results 1310 in a potentially insecure link between the two entities and hence 1311 much stricter security measurements are required. For example, we 1312 pointed out in Section 4.1 that authentication becomes necessary 1313 between CE manager and FE manager, between CE and CE manager, 1314 between FE and FE manager in some configurations. The physical 1315 separation of CE and FE also imposes serious security requirement 1316 for ForCES protocol over Fp interface. This section first attempts 1317 to describe the security threats that may be introduced by the 1318 physical separation of the FEs and the CEs, and then it provides 1319 recommendation and guidelines for secure operation and management of 1320 ForCES protocol over Fp interface. 1322 7.1. Analysis of Potential Threats Introduced by ForCES 1324 This section provides the threat analysis for ForCES, with a focus 1325 on Fp interface. Each threat is described in details with the 1326 effects on the ForCES protocol entities or/and the NE as a whole, 1327 and the required functionalities that need to be in place to defend 1328 the threat. 1330 7.1.1. ��Join�� or ��Remove�� Flooding on CEs 1332 Threats: A malicious node could send a stream of false ��join NE�� or 1333 ��remove from NE�� requests on behalf of non-existent or unauthorized 1334 FE to legitimate CEs at a very rapid rate and thereby create 1335 unnecessary state in the CEs. 1337 Effects: If by maintaining state for non-existent or unauthorized 1338 FEs, a CE may become unavailable for other processing and hence 1339 suffer from denial of service (DoS) attack similar to the TCP SYN 1340 DoS. If multiple CEs are used, the unnecessary state information may 1341 also be conveyed to multiple CEs via Fr interface (e.g., from the 1342 active CE to the stand-by CE) and hence subject multiple CEs to DoS 1343 attack. 1345 Requirement: A CE that receives a ��join�� or ��remove�� request should 1346 not create any state information until it has authenticated the FE 1347 endpoint. 1349 7.1.2. Impersonation Attack 1351 Threats: A malicious node can impersonate a CE or FE and send out 1352 false messages. 1354 Effects: The whole NE could be compromised. 1356 Requirement: The CE or FE must authenticate the message before 1357 accepting and processing it. 1359 7.1.3. Replay Attack 1361 Threat: A malicious node could replay the entire message previously 1362 sent by a FE or CE entity to get around authentication. 1364 Effect: The NE could be compromised. 1366 Requirement: Replay protection mechanism needs to be part of the 1367 security protocol to defend this attack. 1369 7.1.4. Attack during Fail Over 1371 Threat: A malicious node may exploit the CE fail-over mechanism to 1372 take over the control of NE. For example, suppose two CE�s, say CE-A 1373 and CE-B, are controlling several FE�s. CE-A is active and CE-B is 1374 stand-by. When CE-A fails, CE-B is taking over the active CE 1375 position. The FEs already had a trusted relationship with CE-A, but 1376 the FEs may not have the same trusted relationship established with 1377 CE-B prior to the fail-over. A malicious node can take over as CE-B 1378 if such trusted relationship is not established during the fail- 1379 over. 1381 Effect: The NE may be compromised after such insecure fail-over. 1383 Requirement: The level of trust relationship between the stand-by 1384 CE and the FEs must be as strong as the one between the active CE 1385 and the FEs, and such security association must be re-established 1386 during fail-over. 1388 7.1.5. Data Integrity 1390 Threats: A malicious node may inject false messages to legitimate CE 1391 or FE. 1393 Effect: An FE or CE receives the fabricated packet and performs 1394 incorrect or catastrophic operation. 1396 Requirement: Protocol messages require integrity protection. 1398 7.1.6. Data Confidentiality 1399 Threat: When FE and CE are physically separated, a malicious node 1400 may eavesdrop the messages in transit. Some of the messages are 1401 critical to the functioning of the whole network, while others may 1402 contain confidential business data. Leaking of such information may 1403 result in compromise even beyond the immediate CE or FE. 1405 Effect: Sensitive information might be exposed between CE and FE. 1407 Requirement: Data confidentiality between FE and CE must be 1408 available for sensitive information. 1410 7.1.7. Sharing security parameters 1412 Threat: Consider a scenario where several FEs communicating to the 1413 same CE share the same authentication keys for the Fp interface. If 1414 any FE or the CE is compromised, all other entities are compromised. 1416 Effect: The whole NE is compromised. 1418 Requirement: To avoid this side effect, it�s better to configure 1419 different security parameters for each FE-CE communication over Fp 1420 interface. 1422 7.1.8. Denial of Service Attack via External Interface 1424 Threat: When an FE receives a packet that is destined for its CE, 1425 the FE forwards the packet over the Fp interface. Malicious node can 1426 generate huge message storm like routing protocol packets etc. 1427 through the external Fi/f interface so that the FE has to process 1428 and forward all packets to CE through Fp interface. 1430 Effect: CE encounters resource exhaustion and bandwidth starvation 1431 on Fp interface due to an overwhelming number of packets from FEs. 1433 Requirement: Rate limiting mechanism needs to be in place at both FE 1434 and CE. Rate Limiter can be configured at FE for each message type 1435 that are being received through Fi/F interface. 1437 7.2. Security Recommendations for ForCES 1439 The requirements document [3] suggested that ForCES protocol should 1440 support reliability over Fp interface, but no particular transport 1441 protocol is yet specified for ForCES. This framework document does 1442 not intend to specify the particular transport either, and so we 1443 only provide recommendations and guidelines based on the existing 1444 standard security protocols that can work with the common transport 1445 candidates suitable for ForCES. 1447 We highlight two existing security protocol solutions, namely IPsec 1448 (IP Security) [14] or TLS (Transport Layer Security) [13]. TLS works 1449 with reliable transports such as TCP or SCTP, while IPsec can be 1450 used with any transport (UDP, TCP, SCTP). Both TLS and IPsec can be 1451 used potentially to satisfy all of the security requirements for 1452 ForCES protocol. 1454 It is important to realize that even if the NE is in a single-box, 1455 the DoS attacks can still be launched through Fi/f interfaces. 1456 Therefore, it is still important to have counter-measurement as 1457 stated in 1.1.9 for DoS while authentication, confidentiality and 1458 integrity can be provided by the physical security of the box. 1460 7.2.1. Security Configuration 1462 The NE administrator has the freedom to determine the exact security 1463 configuration that is needed for the specific deployment. For 1464 example, ForCES may be deployed between CEs and FEs connected to 1465 each other inside a box over a backplane. In such scenario, 1466 physical security of the box ensures that most of the attacks such 1467 as man-in-the-middle, snooping, and impersonation are not possible, 1468 and hence ForCES architecture may rely on the physical security of 1469 the box to defend against these attacks and protocol mechanisms may 1470 be turned off. However, it is also shown that denial of service 1471 attack via external interface as described in Section 7.1.8 is still 1472 a potential threat even for such ��all-in-one-box�� deployment 1473 scenario and hence the rate limiting mechanism is still necessary. 1474 This is just one example to show that it is important to assess the 1475 security needs of the ForCES-enabled network elements under 1476 different deployment scenarios. It should be possible for the 1477 administrator to configure the level of security needed for the 1478 ForCES protocol. 1480 7.2.2. Using TLS with ForCES 1482 TLS [13] can be used if a reliable transport such as TCP or SCTP is 1483 used for ForCES over Fp interface. The TLS handshake protocol is 1484 used during association establishment or re-establishment phase to 1485 negotiate a TLS session between the CE and FE. Once the session is 1486 in place, the TLS record protocol is used to secure ForCES 1487 communication messages between the CE and FE. 1489 A basic outline of how TLS can be used with ForCES is described 1490 below. Steps 1) till 7) complete the security handshake as 1491 illustrated in Figure 9 while step 8) is for all the further 1492 communication between the CE and FE, including the rest of messages 1493 after the security handshake shown in Figure 9 and the steady-state 1494 communication shown in Figure 10. 1496 1) During Pre-association phase all FEs are configured with the CEs 1497 (including both the active CE and the standby CE). 1498 2) The FE establishes a TLS connection with the CE (master) and 1499 negotiates a cipher suite. 1500 3) The FE (slave) gets the CE certificate, validates the signature, 1501 checks the expiration date, checks if the certificate has been 1502 revoked. 1504 4) The CE (master) gets the FE certificate and performs the same 1505 validation as the FE in step 4). 1506 5) If any of the check fails in step 5 or step 6, endpoint must 1507 generate an error message and abort. 1508 6) After successful mutual authentication, a TLS session is 1509 established between CE and FE. 1510 7) The FE sends a ��join NE�� message to the CE. 1511 8) The FE and CE use TLS session for further communication. 1513 Note that there are different ways for the CE and FE to validate a 1514 received certificate. One way is to configure the FE Manager or CE 1515 Manager or other central component as CA, so that the CE or FE can 1516 query this pre-configured CA to validate that the certificate has 1517 not been revoked. Another way is to have the CE and the FE 1518 configured directly a list of valid certificates in the pre- 1519 association phase. 1521 In the case of fail-over, it is the responsibility of the active CE 1522 and the standby CE to synchronize ForCES states including the TLS 1523 states to minimize the state reestablishment during fail-over. Care 1524 must be taken to ensure that the standby CE is also authenticated in 1525 the same way as the active CE, either before or during the fail- 1526 over. 1528 7.2.3. Using IPsec with ForCES 1530 IPsec [14] can be used with any transport protocol, such as UDP, 1531 SCTP and TCP over Fp interface for ForCES. We recommend using ESP 1532 in transport mode for ForCES because message confidentiality is 1533 required for ForCES and the communication between the CE and FE is 1534 point-to-point. 1536 IPsec can be used with both manual and automated SA and 1537 cryptographic key management. But Ipsec�s replay protection 1538 mechanisms are not available if manual key management is used. 1539 Hence, automatic key management is recommended if replay protection 1540 is deemed important. Otherwise, manual key management might be 1541 sufficient for some deployment scenarios, esp. when the number of 1542 CEs and FEs is relatively small. It is recommended that the keys be 1543 changed periodically even for manual key management. 1545 Unlike TLS, IPsec provides security services between the CE and FE 1546 at IP level, and so the security handshake as illustrated in Figure 1547 9 amounts to a ��no-op�� when manual key management is used. The 1548 following outline the steps taken for ForCES in such a case. 1550 1) During Pre-association phase all FEs are configured with the CEs 1551 (including active CE and standby CE) and SA parameters manually. 1552 2) The FE sends a ��join NE�� message to the CE. This message and all 1553 others that follow are afforded security service according to the 1554 manually configured IPsec SA parameters, but replay protection is 1555 not available. 1557 It is up to the administrator to decide whether to share the same 1558 key across multiple FE-CE communication, but it is recommended that 1559 different keys be used. Similarly, it is recommended that different 1560 keys be used for inbound and outbound traffic. 1562 If automatic key management is needed, IKE [15] can be used for that 1563 purpose. Other automatic key distribution techniques such as 1564 Kerberos may be used as well. The key exchange process constitutes 1565 the security handshake as illustrated in Figure 9. The following 1566 shows the steps involved in using IKE with IPsec for ForCES. Steps 1567 1) to 6) constitute the security handshake in Figure 9. 1569 1) During Pre-association phase all FEs are configured with the CEs 1570 (including active CE and standby CE), IPsec policy etc. 1571 2) The FE kicks off IKE process and tries to establish an IPsec SA 1572 with the CE (master). The FE (Slave) gets the CE certificate as part 1573 of the IKE negotiation. The FE validates signature, checks the 1574 expiration date, checks if the certificate has been revoked. 1575 3) The CE (master) gets the FE certificate and performs the same 1576 check as the FE in step 3). 1577 4) If any of the check fails in step 3 or step 4, the endpoint must 1578 generate an error message and abort. 1579 5) After successful mutual authentication, IPsec session is 1580 established between the CE and FE. 1581 6) The FE sends a ��join NE�� message to CE. No SADB entry is created 1582 in FE yet. 1583 7) The FE and CE use the IPsec session for further communication. 1585 FE Manager or CE Manager or other central component can be used as 1586 CA for validating CE and FE certificates during the IKE process. 1587 Alternatively, during the pre-association phase, the CE and FE can 1588 be configured directly with the required information such as 1589 certificates or passwords etc depending upon the type of 1590 authentication that administrator wants to configure. 1592 In the case of fail-over, it is the responsibility of active CE and 1593 standby CE to synchronize ForCES states and IPsec states to minimize 1594 the state reestablishment during fail-over. Alternatively, the FE 1595 needs to establish different IPsec SA during the startup operation 1596 itself with each CE�s. This will minimize the periodic state 1597 transfer across IPsec layer though Fr (CE-CE) Interface. 1599 8. Normative References 1601 [1] F. Baker, ��Requirements for IP Version 4 Routers", RFC1812, June 1602 1995. 1604 [2] S. Floyd, ��Congestion Control Principles", RFC2914, September 1605 2000. 1607 [3] H. Khosravi, et. al., ��Requirements for Separation of IP Control 1608 and Forwarding", work in progress, May 2003, . 1611 9. Informative References 1613 [4] J. Case, et. al., ��A Simple Network Management Protocol (SNMP)", 1614 RFC1157, May 1990. 1616 [5] M. Daniele, et. al., ��Agent Extensibility (AgentX) Protocol 1617 Version 1", RFC2741, Jan 2000. 1619 [6] K. Chan, et. al., ��COPS Usage for Policy Provisioning (COPS- 1620 PR)", RFC3084, March 2001. 1622 [7] A. Crouch, et. al., ��ForCES Applicability Statement", work in 1623 progress, June 2002, . 1625 [8] T. Anderson, J. Buerkle, ��Requirements for the Dynamic 1626 Partitioning of Switching Elements", RFC3532, May 2003. 1628 [9] M. Leelanivas, et. al., ��Graceful Restart Mechanism for Label 1629 Distribution Protocol��, RFC 3478, Feb. 2003. 1631 [10] J. Moy, et. al., ��Graceful OSPF Restart��, work in progress, 1632 March 2003, . 1634 [11] S. R. Sangli, et. al., ��Graceful Restart Mechanism for BGP��, 1635 work in progress, Jan 203, < draft-ietf-idr-restart-06.txt>. 1637 [12] M. Shand and L. Ginsberg, ��Restart Signaling for IS-IS��, work 1638 in progress, March 2003, . 1640 [13] T. Dierks and C. Allen, ��The TLS Protocol, version 1.0��, 1641 RFC2246, Jan. 1999. 1643 [14] S. Kent and R. Atkinson, ��Security Architecture for the 1644 Internet Protocol��, RFC2401, Nov. 1998. 1646 [15] D. Harkins and D. Carrel, ��The Internet Key Exchange (IKE)��, 1647 RFC2409, Nov. 1998. 1649 [16] S. M. Bellovin, ��Guidelines for Mandating the Use of Ipsec��, 1650 work in progress, Oct. 2002, . 1652 10. Acknowledgement 1654 Joel M. Halpern gave us many insightful comments and suggestions and 1655 pointed out several major issues. T. Sridhar suggested that the 1656 AgentX protocol could be used with SNMP to manage the ForCES network 1657 elements. Many of our colleagues and people in the ForCES mailing 1658 list also provided valuable feedback. 1660 11. Authors' Addresses 1662 Lily L. Yang 1663 Intel Corp., MS JF3-206, 1664 2111 NE 25th Avenue 1665 Hillsboro, OR 97124 USA 1666 Phone: +1 503 264 8813 1667 Email: lily.l.yang@intel.com 1669 Ram Dantu 1670 Department of Computer Science, 1671 University of North Texas, 1672 Denton, Texas, 76203 1673 Phone: +1 940 565 2822 1674 Email: rdantu@unt.edu 1676 Todd A. Anderson 1677 Intel Corp. 1678 2111 NE 25th Avenue 1679 Hillsboro, OR 97124 USA 1680 Phone: +1 503 712 1760 1681 Email: todd.a.anderson@intel.com 1683 Ram Gopal 1684 Nokia Research Center 1685 5, Wayside Road, 1686 Burlington, MA 01803 1687 Phone: +1 781 993 3685 1688 Email: ram.gopal@nokia.com 1690 12. Intellectual Property Right 1692 The authors are not aware of any intellectual property right issues 1693 pertaining to this document. 1695 13. Full Copyright Statement 1697 Copyright (C) The Internet Society (2002). All Rights Reserved. 1699 This document and translations of it may be copied and furnished to 1700 others, and derivative works that comment on or otherwise explain it 1701 or assist in its implementation may be prepared, copied, published 1702 and distributed, in whole or in part, without restriction of any 1703 kind, provided that the above copyright notice and this paragraph 1704 are included on all such copies and derivative works. However, this 1705 document itself may not be modified in any way, such as by removing 1706 the copyright notice or references to the Internet Society or other 1707 Internet organizations, except as needed for the purpose of 1708 developing Internet standards in which case the procedures for 1709 copyrights defined in the Internet Standards process must be 1710 followed, or as required to translate it into languages other than 1711 English. 1713 The limited permissions granted above are perpetual and will not be 1714 revoked by the Internet Society or its successors or assigns. 1715 This document and the information contained herein is provided on an 1716 "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING 1717 TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING 1718 BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION 1719 HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF 1720 MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE."