idnits 2.17.1 draft-ogawa-forces-ceha-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (May 21, 2010) is 5089 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Unused Reference: 'RFC5812' is defined on line 613, but no explicit reference was found in the text Summary: 0 errors (**), 0 flaws (~~), 2 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group K. Ogawa 3 Internet-Draft NTT Corporation 4 Intended status: Standards Track W. Wang 5 Expires: November 22, 2010 Zhejiang Gongshang University 6 E. Haleplidis 7 University of Patras 8 May 21, 2010 10 ForCES Intra-NE High Availability 11 draft-ogawa-forces-ceha-00 13 Abstract 15 This document discusses CE High Availability within a ForCES NE. 17 Status of this Memo 19 This Internet-Draft is submitted in full conformance with the 20 provisions of BCP 78 and BCP 79. 22 Internet-Drafts are working documents of the Internet Engineering 23 Task Force (IETF). Note that other groups may also distribute 24 working documents as Internet-Drafts. The list of current Internet- 25 Drafts is at http://datatracker.ietf.org/drafts/current/. 27 Internet-Drafts are draft documents valid for a maximum of six months 28 and may be updated, replaced, or obsoleted by other documents at any 29 time. It is inappropriate to use Internet-Drafts as reference 30 material or to cite them other than as "work in progress." 32 This Internet-Draft will expire on November 22, 2010. 34 Copyright Notice 36 Copyright (c) 2010 IETF Trust and the persons identified as the 37 document authors. All rights reserved. 39 This document is subject to BCP 78 and the IETF Trust's Legal 40 Provisions Relating to IETF Documents 41 (http://trustee.ietf.org/license-info) in effect on the date of 42 publication of this document. Please review these documents 43 carefully, as they describe your rights and restrictions with respect 44 to this document. Code Components extracted from this document must 45 include Simplified BSD License text as described in Section 4.e of 46 the Trust Legal Provisions and are provided without warranty as 47 described in the Simplified BSD License. 49 Table of Contents 51 1. Definitions . . . . . . . . . . . . . . . . . . . . . . . . . 3 52 2. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4 53 2.1. Document Scope . . . . . . . . . . . . . . . . . . . . . . 5 54 2.2. Quantifying Problem Scope . . . . . . . . . . . . . . . . 5 55 3. CE HA Framework . . . . . . . . . . . . . . . . . . . . . . . 6 56 3.1. Current CE High Availability Support . . . . . . . . . . . 6 57 3.1.1. Cold Standby Interaction with ForCES Protocol . . . . 7 58 3.1.2. Responsibilities for HA . . . . . . . . . . . . . . . 9 59 4. CE HA Hot Standby . . . . . . . . . . . . . . . . . . . . . . 10 60 5. CE Fr Interface Communication . . . . . . . . . . . . . . . . 11 61 5.1. Basic Scope for Fr Interface . . . . . . . . . . . . . . . 12 62 5.1.1. Fr Interface Operational Approach . . . . . . . . . . 12 63 5.1.2. Fr Interface Liveliness Protocol . . . . . . . . . . . 13 64 5.1.3. Fr Interface Data Synchronization . . . . . . . . . . 13 65 5.1.4. Fr Interface Election . . . . . . . . . . . . . . . . 14 66 6. Contributors . . . . . . . . . . . . . . . . . . . . . . . . . 14 67 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 14 68 8. Security Considerations . . . . . . . . . . . . . . . . . . . 14 69 9. References . . . . . . . . . . . . . . . . . . . . . . . . . . 14 70 9.1. Normative References . . . . . . . . . . . . . . . . . . . 14 71 9.2. Informative References . . . . . . . . . . . . . . . . . . 14 72 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 15 74 1. Definitions 76 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 77 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 78 document are to be interpreted as described in RFC 2119. 80 The following definitions are taken from [RFC3654]and [RFC3746]: 82 Logical Functional Block (LFB) -- A template that represents a fine- 83 grained, logically separate aspects of FE processing. 85 ForCES Protocol -- The protocol used at the Fp reference point in the 86 ForCES Framework in [RFC3746]. 88 ForCES Protocol Layer (ForCES PL) -- A layer in the ForCES 89 architecture that embodies the ForCES protocol and the state transfer 90 mechanisms as defined in [RFC5810]. 92 ForCES Protocol Transport Mapping Layer (ForCES TML) -- A layer in 93 ForCES protocol architecture that specifically addresses the protocol 94 message transportation issues, such as how the protocol messages are 95 mapped to different transport media (like SCTP, IP, TCP, UDP, ATM, 96 Ethernet, etc), and how to achieve and implement reliability, 97 security, etc. 99 2. Introduction 101 Figure 1 illustrates a ForCES NE controlled by a set of redundant CEs 102 with CE1 being active and CE2 and CEn-1 being a backup. 104 ----------------------------------------- 105 | ForCES Network Element | 106 | +-----------+ | 107 | | CEn-1 | | 108 | | (Backup) | | 109 -------------- Fc | +------------+ +------------+ | | 110 | CE Manager |--------+-| CE1 |------| CE2 |-+ | 111 -------------- | | (Active) | Fr | (Backup) | | 112 | | +-------+--+-+ +---+---+----+ | 113 | Fl | | | Fp / | | 114 | | | +---------+ / | | 115 | | Fp| |/ |Fp | 116 | | | | | | 117 | | | Fp /+--+ | | 118 | | | +-------+ | | | 119 | | | | | | | 120 -------------- Ff | --------+--+-- ----+---+----+ | 121 | FE Manager |--------+-| FE1 | Fi | FE2 | | 122 -------------- | | |------| | | 123 | -------------- -------------- | 124 | | | | | | | | | | 125 ----+--+--+--+----------+--+--+--+------- 126 | | | | | | | | 127 | | | | | | | | 128 Fi/f Fi/f 130 Fp: CE-FE interface 131 Fi: FE-FE interface 132 Fr: CE-CE interface 133 Fc: Interface between the CE Manager and a CE 134 Ff: Interface between the FE Manager and an FE 135 Fl: Interface between the CE Manager and the FE Manager 136 Fi/f: FE external interface 138 Figure 1: ForCES Architecture 140 The ForCES architecture allows FEs to be aware of multiple CEs but 141 enforces that only one CE be the master controller. This is known in 142 the industry as 1+N redundancy [refxxxx]. The master CE controls the 143 FEs via the ForCES protocol operating in the Fp interface. If the 144 master CE becomes faulty, a backup CE takes over and NE operation 145 continues. By definition, the current documented setup is known as 146 cold-standby [refxxxx]. The CE set is static and is passed to the FE 147 by the FE Manager (FEM) via the Ff interface and to each CE by the CE 148 Manager (CEM) in the Fc interface during the pre-association phase. 150 From an FE perspective, the knobs of control for a CE set are defined 151 by the FEPO LFB in [RFC5810], Appendix B. Section 3.1 details these 152 knobs further. 154 2.1. Document Scope 156 By current definition, the Fr interface is out of scope for the 157 ForCES architecture. However, it is expected that organizations 158 implementing a set of CEs may need to have the CEs communicate to 159 each other via the Fr interface in order to achieve the 160 synchronization necessary for controlling the FEs. 162 The problem scope addressed by this document falls into 3 areas: 164 1. To describe with more clarity (than [RFC5810]) how current cold- 165 standby approach operates within the NE cluster. 167 2. To describe how to evolve the cold-standby setup to a hot-standby 168 redundancy setup so as to improve the failover time and NE 169 availability. 171 3. To describe a minimalist approach for Fr plane communication 172 which interacting CEs MAY use for both cold and hot standby. 174 2.2. Quantifying Problem Scope 176 The NE recovery and availability is dependent on several time- 177 sensitive metrics: 179 1. How fast the CE plane failure is detected. 181 2. How fast a backup CE becomes operational. 183 3. How fast the FEs associate with the new master CE. 185 4. How fast the FEs recover their state and become operational. 187 The design goals of the current [RFC5810] choices to meet the above 188 goals are driven by desire for simplicity. 190 To quantify the above criteria with the current prescribed ForCES CE 191 setup: 193 1. How fast the CE side detects a CE failure is left undefined. To 194 illustrate an extreme scenario, we could have a human operator 195 acting as the monitoring entity to detect faulty CEs. How fast 196 such detection happens could be in the range of seconds to days. 197 A more active monitor on the Fr interface could improve this 198 detection. In Section 5 we define a behavior on Fr interface to 199 detect CE failures in order to improve things. 201 2. How fast the backup CE becomes operational is also currently out 202 of scope. In the current setup, a backup CE need not be 203 operational at all (for example, to save power) and therefore it 204 is feasible for a monitoring entity to boot up a backup CE after 205 it detects the failure of the master CE. In this document 206 Section 4 we suggest that at least one backup CE be online so as 207 to improve this metric. 209 3. How fast an FE associates with new master CE is also currently 210 undefined. The cost of an FE connecting and associating adds to 211 the recovery overhead. As mentioned above we suggest having at 212 least one backup CE online. In Section 4 we propose to zero out 213 the connection and association cost on failover by having each FE 214 associate with all online backup CEs after associating to the 215 active CE. Note that if an FE pre-associates with backup CEs, 216 then the system will be technically operating in hot-standby 217 mode. 219 4. And last: How fast an FE recovers its state depends on how much 220 NE state exists. By ForCES current definition, the new master CE 221 assumes zero state on the FE and starts from scratch to update 222 the FE. So the larger the state, the longer the recovery. In 223 Section 5 we propose to improve this metric by having the master 224 CE and backup CEs synchronizing in the Fr plane. 226 3. CE HA Framework 228 To achieve CE High Availability, FEs and CEs MUST inter-operate per 229 [RFC5810] definition which is repeated for contextual reasons in 230 Section 3.1. It should be noted that in this default setup, which 231 MUST be implemented by CEs and FEs needing HA, the Fr plane is out of 232 scope (and if available is proprietary to an implementation). 234 3.1. Current CE High Availability Support 236 As mentioned earlier, there can be multiple redundant CEs controlling 237 FEs in a ForCES NE (although in practice there may be only one backup 238 CE). At any one time only one master CE can control the FEs. In 239 addition, the FE connects and associates to only the master CE. The 240 FE and the CE PL are aware of the primary and secondary CEs. This 241 information (primary, secondary CEs) is configured on the FE and the 242 CE PLs during pre-association by the FEM and the CEM respectively. 244 Figure 2 below illustrates the Forces message sequences that the FE 245 uses to recover the connection in current defined cold-standby 246 scheme. 248 FE CE Primary CE Secondary 249 | | | 250 | Asso Estb,Caps exchg | | 251 1 |<--------------------->| | 252 | | | 253 | state update | | 254 2 |<--------------------->| | 255 | | | 256 | | | 257 | FAILURE | 258 | | 259 | Asso Estb,Caps exchange | 260 3 |<------------------------------------------>| 261 | | 262 | Event Report (pri CE down) | 263 4 |------------------------------------------->| 264 | | 265 | state update from scratch | 266 5 |<------------------------------------------>| 268 Figure 2: CE Failover for Cold Standby 270 3.1.1. Cold Standby Interaction with ForCES Protocol 272 High Availability parameterization in an FE is driven by configuring 273 the FE Protocol Object (FEPO) LFB. 275 The FEPO CEID component identifies the current master CE and the 276 component table BackupCEs identifies the backup CEs. The FEPO FE 277 Heartbeat Interval, CE Heartbeat Dead Interval, and CE Heartbeat 278 policy help in detecting connectivity problems between an FE and CE. 279 The CE Failover policy defines how the FE should react on a detected 280 failure. 282 Figure 3 illustrates the defined state machine that facilitates 283 connection recovery. 285 The FE connects to the CE specified on FEPO CEID component. If it 286 fails to connect to the defined CE, it moves it to the bottom of 287 table BackupCEs and sets its CEID component to be the first CE 288 retrieved from table BackupCEs. The FE then attempts to associate 289 with the CE designated as the new primary CE. The FE continues 290 through this procedure until it successfully connects to one of the 291 CEs. 293 (CE issues Teardown || +-----------------+ 294 Lost association) && | Pre-Association | 295 CE failover policy = 0 | (Association | 296 +------------------>| in |<----+ 297 | | progress) | | 298 | CE Issues +--------+--------+ | 299 | Association | | CFTI 300 | Setup Response = Success | | timer 301 | +----------------------+ | expires 302 | | | 303 | V | 304 +-+-----------+ +----+--------+ 305 | | | Not | 306 | | (CE issues Teardown || | Associated | 307 | | Lost association) && | | 308 | Associated | CE Failover Policy = 1 | (May | 309 | | | Continue | 310 | +------------------------->| Forwarding)| 311 | | | | 312 +-------------+ +-----+-------+ 313 ^ | 314 | | 315 | CE Issues | 316 | Association | 317 | Setup Response = Success | 318 +-----------------------------------------+ 320 Figure 3: FE State Machine considering HA 322 When communication fails between the FE and CE (which can be caused 323 by either the CE or link failure but not FE related), either the TML 324 on the FE will trigger the FE PL regarding this failure or it will be 325 detected using the HB messages between FEs and CEs. The 326 communication failure, regardless of how it is detected, MUST be 327 considered as a loss of association between the CE and corresponding 328 FE. 330 If the FE's FEPO CE Failover Policy is configured to mode 0 (the 331 default), it will immediately transition to the pre-association 332 phase. This means that if association is again established, all FE 333 state will need to be re-established. 335 If the FE's FEPO CE Failover Policy is configured to mode 1, it 336 indicates that the FE is capable of HA restart recovery. In such a 337 case, the FE transitions to the not associated state and the CEFTI 338 timer is started. The FE MAY continue to forward packets during this 339 state. It MAY also recycle through any configured backup CEs in a 340 round-robin fashion. It first adds its primary CE to the bottom of 341 table BackupCEs and sets its CEID component to be the first secondary 342 retrieved from table BackupCEs. The FE then attempts to associate 343 with the CE designated as the new primary CE. If it fails to re- 344 associate with any CE and the CEFTI expires, the FE then transitions 345 to the pre-association state. 347 If the FE, while in the not associated state, manages to reconnect to 348 a new primary CE before CEFTI expires it transitions to the 349 Associated state. Once re-associated, the FE tries to recover any 350 state that may have been lost during the not associated state. How 351 the FE achieves to re-synchronize its state is out of scope for the 352 current ForCES architecture. 354 An explicit message (a Config message setting Primary CE component in 355 ForCES Protocol object) from the primary CE, can also be used to 356 change the Primary CE for an FE during normal protocol operation. 358 Also note that the FEs in a ForCES NE could also use a multicast CE 359 ID, i.e., they could be associated with a group of CEs (this assumes 360 the use of a CE-CE synchronization protocol, which is out of scope 361 for this specification). In this case, the loss of association would 362 mean that communication with the entire multicast group of CEs has 363 been lost. The mechanisms described above will apply for this case 364 as well during the loss of association. If, however, the secondary 365 CE was also using the multicast CE ID that was lost, then the FE will 366 need to form a new association using a different CE ID. If the 367 capability exists, the FE MAY first attempt to form a new association 368 with original primary CE using a different non multicast CE ID. 370 3.1.2. Responsibilities for HA 372 XXX: we may remove this section (not much value to overall 373 discussion) 375 TML Level: 377 1. The TML controls logical connection availability and failover. 379 2. The TML also controls peer HA management. 381 At this level, control of all lower layers, for example transport 382 level (such as IP addresses, MAC addresses etc) and associated links 383 going down are the role of the TML. 385 PL Level: 386 All other functionality, including configuring the HA behavior during 387 setup, the CE IDs used to identify primary and secondary CEs, 388 protocol messages used to report CE failure (Event Report), Heartbeat 389 messages used to detect association failure, messages to change the 390 primary CE (Config), and other HA related operations described 391 before, are the PL responsibility. 393 To put the two together, if a path to a primary CE is down, the TML 394 would take care of failing over to a backup path, if one is 395 available. If the CE is totally unreachable then the PL would be 396 informed and it would take the appropriate actions described before. 398 4. CE HA Hot Standby 400 In this section we make some small extensions to the existing scheme 401 to enable it to achieve hot standby HA. With these suggested changes 402 we achieve some of the goals defined in Section 2.2, namely: 404 o How fast a backup CE becomes operational. 406 o How fast the FEs associate with the new master CE. 408 As described in Section 3.1, the FEM configures the FE to make it 409 aware of all the CEs in the NE. The FEM also configures the FE to 410 make it aware of which CE is the master and which are backup(s). The 411 FE's FEPO LFB CEID component identifies the current master CE and 412 table BackupCEs identifies the backup CEs. The FE only connects to 413 the master CE and then proceeds to associate with it. The master 414 thereafter controls the FE and receives events from it. This 415 continues until there is communication failure between the FE and CE 416 at which point the FE attempts to connect to a CE from the BackupCEs 417 table until it succeeds to connect and associate with one listed CE. 419 It is recommended that at least one backup CE should be online. 420 Doing so will improve how fast the backup CE will take to be 421 operational (as opposed to bringing up a backup CE when we detect a 422 master CE fault). If we assume that a CE implementation does state 423 synchronization between CEs (proprietary or as discussed in 424 Section 5), then we can zero out the cost of making the backup CE 425 operational and ready to serve FEs; in such a case an associating FE 426 could immediately become operational. 428 If we assume the presence of at least one backup CE online, we can 429 improve how fast the FEs associate with a new master CE by making two 430 changes: 432 The first change that needs to be made is to have the FE, soon after 433 successfully connecting and associating with the master CE, to 434 proceed and connect as well as associate with the rest of the CEs 435 listed in the BackupCEs table. 437 By virtue of having multiple CE connections, the FE switchover to a 438 new master CE will be relatively much faster. The overall effect is 439 improving the NE recovery time in case of communication failure or 440 faults of the master CE. 442 The second change is to have the FE respond to messages issued by any 443 CE (including a backup CE) it is associated with. This keeps the FE 444 simple and as dumb as it is in the current definition. 446 Again for the sake of simplicity, asynchronous events and packet 447 redirects continue to be sent only to the master CE. XXXX: We need 448 to rethink perhaps and discuss possibility of events being sent to 449 ALLCEIDs CEID (which the TML can translate to mean send-to-all- 450 online-CES). 452 XXXX: We need to have an extra state for each CE (master, connected, 453 associated, stats etc) on the FEPO - so probably another change to 454 current FEPO components. 456 XXXX: What about FEs each assuming a different master CE - is that a 457 problem? It doesnt seem to be because what matters is how the CEs 458 agree between themselves who the master is. The FE responds to all 459 CEs. 461 XXXX: What other kind of traffic needs to be running between FE and 462 backup CEs? Heartbeats? 464 5. CE Fr Interface Communication 466 In this section, we define activities in the Fr interface in order to 467 achieve the other two goals defined Section 2.2 469 o How fast the CE plane failure is detected. 471 o How fast the FEs recover their state and become operational. 473 5.1. Basic Scope for Fr Interface 475 In the Fr plane we expect to see liveliness detection and 476 configuration. 478 In the case of a fault of a master CE being detected by liveliness, 479 we expect there is going to be an election to choose a new master CE. 481 It is also expected that the master CE will be updating the backup 482 CEs via configuration on any necessary NE state changes. 484 Our goal is to keep the Fr interface simple. For this reason, our 485 scope is not very ambitious and tries as much as possible to maintain 486 current ForCES architecture: 488 o We keep the number of active CEs at 1 and backup CEs at an 489 arbitrary number, N. This is also known as 1+N setup which is also 490 the currently defined ForCES architectural setup. So no changes 491 there. 493 o Define that the protocol for the Fr interface for both liveliness 494 and synchronization be the current ForCES protocol. If there are 495 any changes to be made they should be very minimal. 497 o Define the use of the ForCES model as the way to describe what 498 data and events are synchronized in the Fr interface. The LFB 499 model is sufficient to describe components that the ForCES 500 protocol could act on. We keep the state synchronization between 501 CEs limited only to what the CE-FE (Fp) plane exchange and not 502 anything else. 504 o Keep the CE set static and known at FEM/CEM configuration time and 505 build a very simple CE master election process. 507 In this section, we start by assuming the ForCES architecture 508 (protocol and model) and then extend it when necessary. 510 5.1.1. Fr Interface Operational Approach 512 Each CE on bootup knows the NE CE set as configured by the CEM. This 513 static approach greatly simplifies discovery. It is expected in most 514 operational setups, there will be one active and one backup CE. 516 Each backup CE does a ForCES association to the listed master CE. 518 The master CE updates backup CEs with configuration necessary to 519 mantain ForCES related NE state. 521 5.1.2. Fr Interface Liveliness Protocol 523 The ForCES protocol already has built-in heartbeats for liveliness 524 detection. If we define a CEPO LFB, in the same spirit as the FEPO 525 LFB, it should be sufficient to have ForCES act as the liveliness 526 protocol in the Fr plane. 528 XXX: We need to be very clear on what is needed and reused from 529 ForCES protocol. XXX: What details does the CEPO carry? Example 530 that seems to make sense: What CE type (eg master/slave), Status 531 (connected etc), Connectivity parameters, Dead intervals etc 533 5.1.3. Fr Interface Data Synchronization 535 Most existing NE implementations in the industry run some hot standby 536 proprietary scheme. They synchronize many things using such a 537 scheme. Example they keep protocol state of things like OSPF, BGP, 538 IKE etc. We dont want to do that. 540 We focus on a scope that specifies only the need to migrate state and 541 maybe configuration that is maintained by the CE on behalf of CE-FE 542 plane. Not anything else. To be specific: A master CE synchronizes 543 to backup CEs any state updates that happen on the CE-FE plane that 544 it controls. 546 One challenge that will require an extension to the ForCES protocol 547 is on how to communicate (from the master CE to a backup CE) the 548 details about an LFB component state change that happened in a 549 specific FE. 551 We propose to introduce a new protocol TLV at the same hierarchy 552 level as LFB selector. Operationally, this TLV will define that a 553 set of state changes that happened apply to a specific FE. For this 554 reason it will encompass the FEID on which the update happened on. 555 We call it the applies-to TLV. 557 Lets say an update has happened (or depending on update scheme needs 558 to happen) on FE z, LFB-a/instance-b/path-c from the controlling CE 559 x, then the synchronization method to backup CE y will be in the form 560 of a config message from master CE x to backup CE y that will have a 561 message source CEID of x and destination CEID of y. The applies-to 562 TLV will contain FEID z. The rest of the message will be exactly as 563 if the CE x had sent a config message to FE z and will contain the 564 path LFB-a/instance-b/path-c 566 XXX: Refer to IETF 77 presentation slide 11 for choices on how to do 567 CECE synchronization in conjunction with FEs. The consensus seems to 568 lean on the second scheme.. 570 5.1.4. Fr Interface Election 572 Upon failure detection of the master CE, a very simple election 573 occurs. The CE with the lowest CEID wins. Operationally, all CEs 574 associate to the next lowest CEID. This is easy to execute since the 575 static CE list never changes. 577 XXX: Optimize - the master CE could keep tabs on which backup CEs are 578 alive and update the associated CEs CEPO table with status info so 579 this way if the next lowest CE is not alive, theres no point in 580 connecting to it when the master fails... 582 6. Contributors 584 Jamal Hadi Salim has contributed to discussions that created this 585 document. 587 7. IANA Considerations 589 TBA 591 8. Security Considerations 593 TBA 595 9. References 597 9.1. Normative References 599 [RFC5810] Doria, A., Hadi Salim, J., Haas, R., Khosravi, H., Wang, 600 W., Dong, L., Gopal, R., and J. Halpern, "Forwarding and 601 Control Element Separation (ForCES) Protocol 602 Specification", RFC 5810, March 2010. 604 9.2. Informative References 606 [RFC3654] Khosravi, H. and T. Anderson, "Requirements for Separation 607 of IP Control and Forwarding", RFC 3654, November 2003. 609 [RFC3746] Yang, L., Dantu, R., Anderson, T., and R. Gopal, 610 "Forwarding and Control Element Separation (ForCES) 611 Framework", RFC 3746, April 2004. 613 [RFC5812] Halpern, J. and J. Hadi Salim, "Forwarding and Control 614 Element Separation (ForCES) Forwarding Element Model", 615 RFC 5812, March 2010. 617 Authors' Addresses 619 Kentaro Ogawa 620 NTT Corporation 621 3-9-11 Midori-cho 622 Musashino-shi, Tokyo 180-8585 623 Japan 625 Email: ogawa.kentaro@lab.ntt.co.jp 627 Weiming Wang 628 Zhejiang Gongshang University 629 18, Xuezheng Str., Xiasha University Town 630 Hangzhou 310018 631 P.R.China 633 Email: wmwang@mail.zjgsu.edu.cn 635 Evangelos Haleplidis 636 University of Patras 637 Patras 638 Greece 640 Email: ehalep@ece.upatras.gr