idnits 2.17.1 draft-ietf-forces-ceha-01.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** There are 4 instances of too long lines in the document, the longest one being 30 characters in excess of 72. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (February 22, 2011) is 4805 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Missing Reference: 'RFC 5810' is mentioned on line 330, but not defined == Unused Reference: 'RFC5812' is defined on line 522, but no explicit reference was found in the text Summary: 1 error (**), 0 flaws (~~), 3 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group K. Ogawa 3 Internet-Draft NTT Corporation 4 Intended status: Standards Track W. M. Wang 5 Expires: August 26, 2011 Zhejiang Gongshang University 6 E. Haleplidis 7 University of Patras 8 J. Hadi Salim 9 Mojatatu Networks 10 February 22, 2011 12 ForCES Intra-NE High Availability 13 draft-ietf-forces-ceha-01 15 Abstract 17 This document discusses CE High Availability within a ForCES NE. 19 Status of this Memo 21 This Internet-Draft is submitted in full conformance with the 22 provisions of BCP 78 and BCP 79. 24 Internet-Drafts are working documents of the Internet Engineering 25 Task Force (IETF). Note that other groups may also distribute 26 working documents as Internet-Drafts. The list of current Internet- 27 Drafts is at http://datatracker.ietf.org/drafts/current/. 29 Internet-Drafts are draft documents valid for a maximum of six months 30 and may be updated, replaced, or obsoleted by other documents at any 31 time. It is inappropriate to use Internet-Drafts as reference 32 material or to cite them other than as "work in progress." 34 This Internet-Draft will expire on August 26, 2011. 36 Copyright Notice 38 Copyright (c) 2011 IETF Trust and the persons identified as the 39 document authors. All rights reserved. 41 This document is subject to BCP 78 and the IETF Trust's Legal 42 Provisions Relating to IETF Documents 43 (http://trustee.ietf.org/license-info) in effect on the date of 44 publication of this document. Please review these documents 45 carefully, as they describe your rights and restrictions with respect 46 to this document. Code Components extracted from this document must 47 include Simplified BSD License text as described in Section 4.e of 48 the Trust Legal Provisions and are provided without warranty as 49 described in the Simplified BSD License. 51 Table of Contents 53 1. Definitions . . . . . . . . . . . . . . . . . . . . . . . . . 3 54 2. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4 55 2.1. Document Scope . . . . . . . . . . . . . . . . . . . . . . 5 56 2.2. Quantifying Problem Scope . . . . . . . . . . . . . . . . 5 57 3. RFC5810 CE HA Framework . . . . . . . . . . . . . . . . . . . 6 58 3.1. Current CE High Availability Support . . . . . . . . . . . 6 59 3.1.1. Cold Standby Interaction with ForCES Protocol . . . . 7 60 3.1.2. Responsibilities for HA . . . . . . . . . . . . . . . 9 61 4. CE HA Hot Standby . . . . . . . . . . . . . . . . . . . . . . 10 62 4.1. Changes to the FEPO model . . . . . . . . . . . . . . . . 12 63 5. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 12 64 6. Security Considerations . . . . . . . . . . . . . . . . . . . 12 65 7. References . . . . . . . . . . . . . . . . . . . . . . . . . . 12 66 7.1. Normative References . . . . . . . . . . . . . . . . . . . 12 67 7.2. Informative References . . . . . . . . . . . . . . . . . . 12 68 Appendix 1. Appendix I - New FEPO version . . . . . . . . . . . . 13 69 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 20 71 1. Definitions 73 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 74 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 75 document are to be interpreted as described in RFC 2119. 77 The following definitions are taken from [RFC3654]and [RFC3746]: 79 Logical Functional Block (LFB) -- A template that represents a fine- 80 grained, logically separate aspects of FE processing. 82 ForCES Protocol -- The protocol used at the Fp reference point in the 83 ForCES Framework in [RFC3746]. 85 ForCES Protocol Layer (ForCES PL) -- A layer in the ForCES 86 architecture that embodies the ForCES protocol and the state transfer 87 mechanisms as defined in [RFC5810]. 89 ForCES Protocol Transport Mapping Layer (ForCES TML) -- A layer in 90 ForCES protocol architecture that specifically addresses the protocol 91 message transportation issues, such as how the protocol messages are 92 mapped to different transport media (like SCTP, IP, TCP, UDP, ATM, 93 Ethernet, etc), and how to achieve and implement reliability, 94 security, etc. 96 2. Introduction 98 Figure 1 illustrates a ForCES NE controlled by a set of redundant CEs 99 with CE1 being active and CE2 and CEn-1 being a backup. 101 ----------------------------------------- 102 | ForCES Network Element | 103 | +-----------+ | 104 | | CEn-1 | | 105 | | (Backup) | | 106 -------------- Fc | +------------+ +------------+ | | 107 | CE Manager |--------+-| CE1 |------| CE2 |-+ | 108 -------------- | | (Active) | Fr | (Backup) | | 109 | | +-------+--+-+ +---+---+----+ | 110 | Fl | | | Fp / | | 111 | | | +---------+ / | | 112 | | Fp| |/ |Fp | 113 | | | | | | 114 | | | Fp /+--+ | | 115 | | | +-------+ | | | 116 | | | | | | | 117 -------------- Ff | --------+--+-- ----+---+----+ | 118 | FE Manager |--------+-| FE1 | Fi | FE2 | | 119 -------------- | | |------| | | 120 | -------------- -------------- | 121 | | | | | | | | | | 122 ----+--+--+--+----------+--+--+--+------- 123 | | | | | | | | 124 | | | | | | | | 125 Fi/f Fi/f 127 Fp: CE-FE interface 128 Fi: FE-FE interface 129 Fr: CE-CE interface 130 Fc: Interface between the CE Manager and a CE 131 Ff: Interface between the FE Manager and an FE 132 Fl: Interface between the CE Manager and the FE Manager 133 Fi/f: FE external interface 135 Figure 1: ForCES Architecture 137 The ForCES architecture allows FEs to be aware of multiple CEs but 138 enforces that only one CE be the master controller. This is known in 139 the industry as 1+N redundancy [refxxxx]. The master CE controls the 140 FEs via the ForCES protocol operating in the Fp interface. If the 141 master CE becomes faulty, a backup CE takes over and NE operation 142 continues. By definition, the current documented setup is known as 143 cold-standby [refxxxx]. The CE set is static and is passed to the FE 144 by the FE Manager (FEM) via the Ff interface and to each CE by the CE 145 Manager (CEM) in the Fc interface during the pre-association phase. 147 From an FE perspective, the knobs of control for a CE set are defined 148 by the FEPO LFB in [RFC5810], Appendix B. Section 3.1 of this 149 document details these knobs further. 151 2.1. Document Scope 153 By current definition, the Fr interface is out of scope for the 154 ForCES architecture. However, it is expected that organizations 155 implementing a set of CEs will need to have the CEs communicate to 156 each other via the Fr interface in order to achieve the 157 synchronization necessary for controlling the FEs. 159 The problem scope addressed by this document falls into 2 areas: 161 1. To describe with more clarity (than [RFC5810]) how current cold- 162 standby approach operates within the NE cluster. 164 2. To describe how to evolve the cold-standby setup to a hot-standby 165 redundancy setup so as to improve the failover time and NE 166 availability. 168 2.2. Quantifying Problem Scope 170 The NE recovery and availability is dependent on several time- 171 sensitive metrics: 173 1. How fast the CE plane failure is detected the FE. 175 2. How fast a backup CE becomes operational. 177 3. How fast the FEs associate with the new master CE. 179 4. How fast the FEs recover their state and become operational. 181 The design goals of the current [RFC5810] choices to meet the above 182 goals are driven by desire for simplicity. 184 To quantify the above criteria with the current prescribed ForCES CE 185 setup in [RFC5810]: 187 1. How fast the CE side detects a CE failure is left undefined. To 188 illustrate an extreme scenario, we could have a human operator 189 acting as the monitoring entity to detect faulty CEs. How fast 190 such detection happens could be in the range of seconds to days. 191 A more active monitor on the Fr interface could improve this 192 detection. 194 2. How fast the backup CE becomes operational is also currently out 195 of scope. In the current setup, a backup CE need not be 196 operational at all (for example, to save power) and therefore it 197 is feasible for a monitoring entity to boot up a backup CE after 198 it detects the failure of the master CE. In this document 199 Section 4 we suggest that at least one backup CE be online so as 200 to improve this metric. 202 3. How fast an FE associates with new master CE is also currently 203 undefined. The cost of an FE connecting and associating adds to 204 the recovery overhead. As mentioned above we suggest having at 205 least one backup CE online. In Section 4 we propose to zero out 206 the connection and association cost on failover by having each FE 207 associate with all online backup CEs after associating to the 208 active CE. Note that if an FE pre-associates with backup CEs, 209 then the system will be technically operating in hot-standby 210 mode. 212 4. And last: How fast an FE recovers its state depends on how much 213 NE state exists. By ForCES current definition, the new master CE 214 assumes zero state on the FE and starts from scratch to update 215 the FE. So the larger the state, the longer the recovery. 217 3. RFC5810 CE HA Framework 219 To achieve CE High Availabilty, FEs and CEs MUST inter-operate per 220 [RFC5810] definition which is repeated for contextual reasons in 221 Section 3.1. It should be noted that in this default setup, which 222 MUST be implemented by CEs and FEs needing HA, the Fr plane is out of 223 scope (and if available is proprietary to an implementation). 225 3.1. Current CE High Availability Support 227 As mentioned earlier, although there can be multiple redundant CEs, 228 only one CE actively controls FEs in a ForCES NE. In practice there 229 may be only one backup CE. At any moment in time only one master CE 230 can control the FEs. In addition, the FE connects and associates to 231 only the master CE. The FE and the CE PL are aware of the primary 232 and one or more secondary CEs. This information (primary, secondary 233 CEs) is configured on the FE and the CE PLs during pre-association by 234 the FEM and the CEM respectively. 236 Figure 2 below illustrates the Forces message sequences that the FE 237 uses to recover the connection in current defined cold-standby 238 scheme. 240 FE CE Primary CE Secondary 241 | | | 242 | Asso Estb,Caps exchg | | 243 1 |<--------------------->| | 244 | | | 245 | state update | | 246 2 |<--------------------->| | 247 | | | 248 | | | 249 | FAILURE | 250 | | 251 | Asso Estb,Caps exchange | 252 3 |<------------------------------------------>| 253 | | 254 | Event Report (pri CE down) | 255 4 |------------------------------------------->| 256 | | 257 | state update from scratch | 258 5 |<------------------------------------------>| 260 Figure 2: CE Failover for Cold Standby 262 3.1.1. Cold Standby Interaction with ForCES Protocol 264 High Availability parameterization in an FE is driven by configuring 265 the FE Protocol Object (FEPO) LFB. 267 The FEPO CEID component identifies the current master CE and the 268 component table BackupCEs identifies the backup CEs. The FEPO FE 269 Heartbeat Interval, CE Heartbeat Dead Interval, and CE Heartbeat 270 policy help in detecting connectivity problems between an FE and CE. 271 The CE Failover policy defines how the FE should react on a detected 272 failure. 274 Figure 3 illustrates the defined state machine that facilitates 275 connection recovery. 277 The FE connects to the CE specified on FEPO CEID component. If it 278 fails to connect to the defined CE, it moves it to the bottom of 279 table BackupCEs and sets its CEID component to be the first CE 280 retrieved from table BackupCEs. The FE then attempts to associate 281 with the CE designated as the new primary CE. The FE continues 282 through this procedure until it successfully connects to one of the 283 CEs. 285 (CE issues Teardown || +-----------------+ 286 Lost association) && | Pre-Association | 287 CE failover policy = 0 | (Association | 288 +------------>-->-->| in +<----+ 289 | | progress) | | 290 | CE Issues +--------+--------+ | 291 | Association | | CEFTI 292 | Response V | timer 293 | ___________________+ | expires 294 | | | 295 | V ^ 296 +-+-----------+ +-------+-----+ 297 | | | Not | 298 | | (CE issues Teardown || | Associated | 299 | | Lost association) && | | 300 | Associated | CE Failover Policy = 1 | (May | 301 | | | Continue | 302 | |---------->------->------>| Forwarding)| 303 | | | | 304 +-------------+ +-------------+ 305 ^ V 306 | | 307 | CE Issues | 308 | Association | 309 | Setup | 310 +_________________________________________+ 312 Figure 3: FE State Machine considering HA 314 When communication fails between the FE and CE (which can be caused 315 by either the CE or link failure but not FE related), either the TML 316 on the FE will trigger the FE PL regarding this failure or it will be 317 detected using the HB messages between FEs and CEs. The 318 communication failure, regardless of how it is detected, MUST be 319 considered as a loss of association between the CE and corresponding 320 FE. 322 If the FE's FEPO CE Failover Policy is configured to mode 0 (the 323 default), it will immediately transition to the pre-association 324 phase. This means that if association is again established, all FE 325 state will need to be re-established. 327 If the FE's FEPO CE Failover Policy is configured to mode 1, it 328 indicates that the FE is capable of HA restart recovery. In such a 329 case, the FE transitions to the not associated state and the CEFTI 330 timer[RFC 5810] is started. The FE MAY continue to forward packets 331 during this state. It MAY also recycle through any configured backup 332 CEs in a round-robin fashion. It first adds its primary CE to the 333 bottom of table BackupCEs and sets its CEID component to be the first 334 secondary retrieved from table BackupCEs. The FE then attempts to 335 associate with the CE designated as the new primary CE. If it fails 336 to re-associate with any CE and the CEFTI expires, the FE then 337 transitions to the pre-association state. 339 If the FE, while in the not associated state, manages to reconnect to 340 a new primary CE before CEFTI expires it transitions to the 341 Associated state. Once re-associated, the FE tries to recover any 342 state that may have been lost during the not associated state. How 343 the FE achieves to re-synchronize its state is out of scope for the 344 current ForCES architecture. 346 An explicit message (a Config message setting Primary CE component in 347 ForCES Protocol object) from the primary CE, can also be used to 348 change the Primary CE for an FE during normal protocol operation. In 349 this case, the FE transitions to the Not Associated State and 350 attempts to Associate with the new CE. 352 3.1.2. Responsibilities for HA 354 XXX: we may remove this section (not much value to overall 355 discussion) 357 TML Level: 359 1. The TML controls logical connection availability and failover. 361 2. The TML also controls peer HA management. 363 At this level, control of all lower layers, for example transport 364 level (such as IP addresses, MAC addresses etc) and associated links 365 going down are the role of the TML. 367 PL Level: 368 All other functionality, including configuring the HA behavior during 369 setup, the CE IDs used to identify primary and secondary CEs, 370 protocol messages used to report CE failure (Event Report), Heartbeat 371 messages used to detect association failure, messages to change the 372 primary CE (Config), and other HA related operations described 373 before, are the PL responsibility. 375 To put the two together, if a path to a primary CE is down, the TML 376 would take care of failing over to a backup path, if one is 377 available. If the CE is totally unreachable then the PL would be 378 informed and it would take the appropriate actions described before. 380 4. CE HA Hot Standby 382 In this section we make some small extensions to the existing scheme 383 to enable it to achieve hot standby HA. With these suggested changes 384 we achieve some of the goals defined in Section 2.2, namely: 386 o How fast a backup CE becomes operational. 388 o How fast the FEs associate with the new master CE. 390 As described in Section 3.1, in the pre-association phase the FEM 391 configures the FE to make it aware of all the CEs in the NE. The FEM 392 MUST configure the FE to make it aware of which CE is the master and 393 MAY specify any backup CE(s). 395 The FE's FEPO LFB version 2 AllCEs table (previously BackupCEs) 396 contains all the CEIDs that the FE may connect and associate with. 397 The sequence of the CE IDs is also the conncetion priority for the 398 FE. In the pre-association phase, the first CE ID in the AllCEs 399 table MUST be the first CE ID that the FE will attempt to connect and 400 associate with. If the FE fails to connect and associate with the 401 first CE ID it will attempt to connect to the second CE ID and so 402 forth, until there is a connection and an association or the list 403 ends. The FEPO's CEID component identifies the current associated 404 master CE. 406 Once the FE has associated with a master CE it moves to the post- 407 association phase. In the post-association phase, the master CE MAY 408 update the list of backup CEs. It MAY also instruct the FE to use a 409 different master CE. It is assumed that the master CE will 410 communicate with other CEs within the NE for the purpose of 411 synchronization via the CE-CE interface. The CE-CE interface is out 412 of scope for this document. 414 While in the post-association phase, if the CE Failover Policy is set 415 to 2 (High Availability without Graceful Restart) or 3 (High 416 Availability with Graceful Restart) then the FE, after succesfully 417 associating with the master CE, MUST attempt to connect and associate 418 with all the CEs that it becomes aware of. If it fails to connect or 419 associate with some CEs, the FE MAY flag them as unreachable to avoid 420 continuous attempts to connect. 422 When the master CE for any reason is considered to be down, then the 423 FE will try to find the first associated CE from the list of all CEs 424 in a round-robin fashion. 426 If the FE is unable to find an associated FE in its list of CEs, then 427 it will attempt to connect and associate with the first from the list 428 of all CEs and continue in a round-robin fashion until it connects 429 and associates with a CE. 431 "XXX: We need to discuss what should happen to CEs in the AllCEs list 432 which an FE has attempted to connect or associate to but failed." 434 Once connected and associated it assumes that the new associated CE 435 is the new master CE and sets the FEPO CEID component's value with 436 the new associated CE's ID. 438 The FE then sends the Primary CE Down Event Notification to all 439 associated CEs to notify them that the FE considers this CE as the 440 new master CE. 442 The new master CE MUST configure the CEID component of the FE within 443 the time limit defined in the FEPO Failover Timeout as a confirmation 444 that the FE made the right choice. 446 XXX: We need to discuss what happenes if a CE doesn't respond within 447 a FEPO Failover Timeout. 449 If the CE the FE assumed to be the master discovers that it should 450 not be the new master CE, then it will configure the CEID with the ID 451 of the proper master CE. How the CE decides who the new master CE 452 is, is also out of scope of this document and is assumed to be done 453 via a CE-CE communication protocol. 455 In most High Availability architectures the split-brain issue is 456 present. However, since the FE will never accept any configuration 457 messages from any other than the master CE, we consider the FE as 458 fenced against data corruption from the other CEs that consider 459 themselves as the master. The split-brain issue is mostly a CE-CE 460 communication problem and is considered to be out of scope. 462 By virtue of having multiple CE connections, the FE switchover to a 463 new master CE will be relatively much faster. The overall effect is 464 improving the NE recovery time in case of communication failure or 465 faults of the master CE. 467 For the sake of simplicity, the FE MUST respond to messages issued by 468 only the master CE. This simplifies the synchronization and avoids 469 the concept of locking FE state. The FE MUST drop any messages from 470 backup CEs. However, asynchronous events that the master CE has 471 subscribed to and heartbeats are sent to all associated-to CEs. 472 Packet redirects continue to be sent only to the master CE. The 473 Heartbeat Interval, the CEHB Policy and the FEHB Policy MUST apply to 474 all CEs. 476 4.1. Changes to the FEPO model 478 In order for the above to be achievable there is a need to make a few 479 changes in the FEPO model. Appendix I contains the xml of the new 480 version of the FEPO. 482 Changes from the previous version are: 484 1. Addition of a new datatype, status (unsigned char) with special 485 values 0 (Disconnected), 1 (Connected), 2 (Associated), 3 486 (Lost_Connection) and 4 (Unreachable). 488 2. Change Component BackupCEs (9) to AllCEs and instead of an Array 489 of unsigned integers, it MUST be an Array of unsigned integers 490 (CEID) and unsigned char (status) for each CE. 492 3. Add two special values to the CEFailoverPolicyValues. 2 (High 493 availability without Graceful restart) and 3 (High availability 494 with Graceful restart). 496 5. IANA Considerations 498 TBA 500 6. Security Considerations 502 TBA 504 7. References 506 7.1. Normative References 508 [RFC5810] Doria, A., Hadi Salim, J., Haas, R., Khosravi, H., Wang, 509 W., Dong, L., Gopal, R., and J. Halpern, "Forwarding and 510 Control Element Separation (ForCES) Protocol 511 Specification", RFC 5810, March 2010. 513 7.2. Informative References 515 [RFC3654] Khosravi, H. and T. Anderson, "Requirements for Separation 516 of IP Control and Forwarding", RFC 3654, November 2003. 518 [RFC3746] Yang, L., Dantu, R., Anderson, T., and R. Gopal, 519 "Forwarding and Control Element Separation (ForCES) 520 Framework", RFC 3746, April 2004. 522 [RFC5812] Halpern, J. and J. Hadi Salim, "Forwarding and Control 523 Element Separation (ForCES) Forwarding Element Model", 524 RFC 5812, March 2010. 526 1. Appendix I - New FEPO version 528 XXX: Describe this to conform to LFB extensions as prescribed in the 529 model 531 535 536 537 538 CEHBPolicyValues 539 540 The possible values of CE heartbeat policy 541 542 543 uchar 544 545 546 CEHBPolicy0 547 548 The CE heartbeat policy 0 549 550 551 552 CEHBPolicy1 553 554 The CE heartbeat policy 1 555 556 557 558 559 560 561 FEHBPolicyValues 562 563 The possible values of FE heartbeat policy 564 565 566 uchar 567 568 569 FEHBPolicy0 570 571 The FE heartbeat policy 0 572 573 574 575 FEHBPolicy1 576 577 The FE heartbeat policy 1 578 579 580 581 582 583 584 FERestartPolicyValues 585 586 The possible values of FE restart policy 587 588 589 uchar 590 591 592 FERestartPolicy0 593 594 The FE restart policy 0 595 596 597 598 599 600 601 CEFailoverPolicyValues 602 603 The possible values of CE failover policy 604 605 606 uchar 607 608 609 CEFailoverPolicy0 610 611 The CE failover policy 0 612 No High Availability or Graceful Restart. 613 614 615 616 CEFailoverPolicy1 617 618 Graceful Restart 619 620 621 622 CEFailoverPolicy2 623 624 High Availability without Graceful Restart 625 626 627 628 CEFailoverPolicy3 629 630 High Availability with Graceful Restart 631 632 633 634 635 636 637 FEHACapab 638 639 The supported HA features 640 641 642 uchar 643 644 645 GracefullRestart 646 647 The FE supports Graceful Restart 648 649 650 651 HA 652 653 The FE supports HA 654 655 656 657 658 659 660 CEStatusType 661 662 Status values. Status for each CE. 663 664 665 uchar 666 667 668 Disconnected 669 670 No connection attempt with the CE yet. 671 672 673 674 Connected 675 676 The FE has connected with the CE. 677 678 679 680 Associated 681 682 The FE has associated with the CE. 683 684 685 686 Lost_Connection 687 688 The FE was associated with the CE 689 but lost the connection. 690 691 692 693 Unreachable 694 695 The CE is deemed as unreachable by the FE. 696 697 698 699 700 701 702 AllCEType 703 704 Table Type for AllCE component. 705 706 707 708 CEID 709 ID of the CE 710 uint32 711 712 713 CEStatus 714 Status of the CE 715 CEStatusType 716 717 718 719 720 721 722 FEPO 723 724 The FE Protocol Object 725 726 2.0 727 728 729 CurrentRunningVersion 730 Currently running ForCES version 731 u8 732 733 734 FEID 735 Unicast FEID 736 uint32 737 738 739 MulticastFEIDs 740 741 the table of all multicast IDs 742 743 744 uint32 745 746 747 748 CEHBPolicy 749 750 The CE Heartbeat Policy 751 752 CEHBPolicyValues 753 754 755 CEHDI 756 757 The CE Heartbeat Dead Interval in millisecs 758 759 uint32 760 761 762 FEHBPolicy 763 764 The FE Heartbeat Policy 765 766 FEHBPolicyValues 767 768 769 FEHI 770 771 The FE Heartbeat Interval in millisecs 772 773 uint32 774 775 776 CEID 777 778 The Primary CE this FE is associated with 779 780 uint32 781 782 783 AllCEs 784 785 The table of all CEs. 786 787 788 AllCEType 789 790 791 792 CEFailoverPolicy 793 794 The CE Failover Policy 795 796 CEFailoverPolicyValues 797 798 799 CEFTI 800 801 The CE Failover Timeout Interval in millisecs 802 803 uint32 804 805 806 FERestartPolicy 807 808 The FE Restart Policy 810 811 FERestartPolicyValues 812 813 814 LastCEID 815 816 The Primary CE this FE was last associated with 817 818 uint32 819 820 821 822 823 SupportableVersions 824 825 the table of ForCES versions that FE supports 826 827 828 u8 829 830 831 832 HACapabilities 833 834 the table of HA capabilities the FE supports 835 836 837 FEHACapab 838 839 840 841 842 843 PrimaryCEDown 844 845 The pimary CE has changed 846 847 848 LastCEID 849 850 851 852 853 LastCEID 854 855 856 857 859 860 861 863 Authors' Addresses 865 Kentaro Ogawa 866 NTT Corporation 867 3-9-11 Midori-cho 868 Musashino-shi, Tokyo 180-8585 869 Japan 871 Email: ogawa.kentaro@lab.ntt.co.jp 873 Weiming Wang 874 Zhejiang Gongshang University 875 149 Jiaogong Road 876 Hangzhou 310035 877 P.R.China 879 Phone: +86-571-88057712 880 Email: wmwang@mail.zjgsu.edu.cn 882 Evangelos Haleplidis 883 University of Patras 884 Patras 885 Greece 887 Email: ehalep@ece.upatras.gr 889 Jamal Hadi Salim 890 Mojatatu Networks 891 Ottawa, Ontario 892 Canada 894 Email: hadi@mojatatu.com