idnits 2.17.1 draft-ietf-forces-ceha-07.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** There are 6 instances of too long lines in the document, the longest one being 1 character in excess of 72. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 344 has weird spacing: '... |try v...' == Line 595 has weird spacing: '...ociated v...' == Line 596 has weird spacing: '...)|CE or retry...' -- The document date (May 08, 2013) is 4006 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Unused Reference: 'RFC2119' is defined on line 708, but no explicit reference was found in the text Summary: 1 error (**), 0 flaws (~~), 5 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group K. Ogawa 3 Internet-Draft NTT Corporation 4 Intended status: Standards Track W. M. Wang 5 Expires: November 09, 2013 Zhejiang Gongshang University 6 E. Haleplidis 7 University of Patras 8 J. Hadi Salim 9 Mojatatu Networks 10 May 08, 2013 12 ForCES Intra-NE High Availability 13 draft-ietf-forces-ceha-07 15 Abstract 17 This document discusses Control Element High Availability within a 18 ForCES Network Element. 20 Status of This Memo 22 This Internet-Draft is submitted in full conformance with the 23 provisions of BCP 78 and BCP 79. 25 Internet-Drafts are working documents of the Internet Engineering 26 Task Force (IETF). Note that other groups may also distribute 27 working documents as Internet-Drafts. The list of current Internet- 28 Drafts is at http://datatracker.ietf.org/drafts/current/. 30 Internet-Drafts are draft documents valid for a maximum of six months 31 and may be updated, replaced, or obsoleted by other documents at any 32 time. It is inappropriate to use Internet-Drafts as reference 33 material or to cite them other than as "work in progress." 35 This Internet-Draft will expire on November 09, 2013. 37 Copyright Notice 39 Copyright (c) 2013 IETF Trust and the persons identified as the 40 document authors. All rights reserved. 42 This document is subject to BCP 78 and the IETF Trust's Legal 43 Provisions Relating to IETF Documents 44 (http://trustee.ietf.org/license-info) in effect on the date of 45 publication of this document. Please review these documents 46 carefully, as they describe your rights and restrictions with respect 47 to this document. Code Components extracted from this document must 48 include Simplified BSD License text as described in Section 4.e of 49 the Trust Legal Provisions and are provided without warranty as 50 described in the Simplified BSD License. 52 Table of Contents 54 1. Definitions . . . . . . . . . . . . . . . . . . . . . . . . . 2 55 2. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 56 2.1. Document Scope . . . . . . . . . . . . . . . . . . . . . 5 57 2.2. Quantifying Problem Scope . . . . . . . . . . . . . . . . 5 58 3. RFC5810 CE HA Framework . . . . . . . . . . . . . . . . . . . 6 59 3.1. RFC 5810 CE HA Support . . . . . . . . . . . . . . . . . 6 60 3.1.1. Cold Standby Interaction with ForCES Protocol . . . . 7 61 3.1.2. Responsibilities for HA . . . . . . . . . . . . . . . 9 62 4. CE HA Hot Standby . . . . . . . . . . . . . . . . . . . . . . 10 63 4.1. Changes to the FEPO model . . . . . . . . . . . . . . . . 10 64 4.2. FEPO processing . . . . . . . . . . . . . . . . . . . . . 12 65 5. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 15 66 6. Security Considerations . . . . . . . . . . . . . . . . . . . 15 67 7. References . . . . . . . . . . . . . . . . . . . . . . . . . 16 68 7.1. Normative References . . . . . . . . . . . . . . . . . . 16 69 7.2. Informative References . . . . . . . . . . . . . . . . . 16 70 Appendix A. New FEPO version . . . . . . . . . . . . . . . . . . 16 71 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 25 73 1. Definitions 75 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 76 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 77 document are to be interpreted as described in RFC 2119. 79 The following definitions are taken from [RFC3654]and [RFC3746]: 81 o Logical Functional Block (LFB) -- A template that represents a 82 fine-grained, logically separate aspects of FE processing. 84 o ForCES Protocol -- The protocol used at the Fp reference point in 85 the ForCES Framework in [RFC3746]. 87 o ForCES Protocol Layer (ForCES PL) -- A layer in the ForCES 88 architecture that embodies the ForCES protocol and the state 89 transfer mechanisms as defined in [RFC5810]. 91 o ForCES Protocol Transport Mapping Layer (ForCES TML) -- A layer in 92 ForCES protocol architecture that specifically addresses the 93 protocol message transportation issues, such as how the protocol 94 messages are mapped to different transport media (like SCTP, IP, 95 TCP, UDP, ATM, Ethernet, etc), and how to achieve and implement 96 reliability, security, etc. 98 o Forwarding Element (FE) - A logical entity that implements the 99 ForCES Protocol. FEs use the underlying hardware to provide per- 100 packet processing and handling as directed by a CE via the ForCES 101 Protocol. 103 o Control Element (CE) - A logical entity that implements the ForCES 104 Protocol and uses it to instruct one or more FEs on how to process 105 packets. CEs handle functionality such as the execution of 106 control and signaling protocols. 108 o ForCES Network Element (NE) - An entity composed of one or more 109 CEs and one or more FEs. An NE usually hides its internal 110 organization from external entities and represents a single point 111 of management to entities outside the NE. 113 o FE Manager (FEM) - A logical entity that operates in the pre- 114 association phase and is responsible for determining to which 115 CE(s) an FE should communicate. This process is called CE 116 discovery and may involve the FE manager learning the capabilities 117 of available CEs. 119 o CE Manager - A logical entity that operates in the pre-association 120 phase and is responsible for determining to which FE(s) a CE 121 should communicate. This process is called FE discovery and may 122 involve the CE manager learning the capabilities of available FEs. 124 2. Introduction 126 Figure 1 illustrates a ForCES NE controlled by a set of redundant CEs 127 with CE1 being active and CE2 and CEN being a backup. 129 ----------------------------------------- 130 | ForCES Network Element | 131 | +-----------+ | 132 | | CEN | | 133 | | (Backup) | | 134 -------------- Fc | +------------+ +------------+ | | 135 | CE Manager |--------+-| CE1 |------| CE2 |-+ | 136 -------------- | | (Active) | Fr | (Backup) | | 137 | | +-------+--+-+ +---+---+----+ | 138 | Fl | | | Fp / | | 139 | | | +---------+ / | | 140 | | Fp| |/ |Fp | 141 | | | | | | 142 | | | Fp /+--+ | | 143 | | | +-------+ | | | 144 | | | | | | | 145 -------------- Ff | --------+--+-- ----+---+----+ | 146 | FE Manager |--------+-| FE1 | Fi | FE2 | | 147 -------------- | | |------| | | 148 | -------------- -------------- | 149 | | | | | | | | | | 150 ----+--+--+--+----------+--+--+--+------- 151 | | | | | | | | 152 | | | | | | | | 153 Fi/f Fi/f 155 Fp: CE-FE interface 156 Fi: FE-FE interface 157 Fr: CE-CE interface 158 Fc: Interface between the CE Manager and a CE 159 Ff: Interface between the FE Manager and an FE 160 Fl: Interface between the CE Manager and the FE Manager 161 Fi/f: FE external interface 163 Figure 1: ForCES Architecture 165 The ForCES architecture allows FEs to be aware of multiple CEs but 166 enforces that only one CE be the master controller. This is known in 167 the industry as 1+N redundancy. The master CE controls the FEs via 168 the ForCES protocol operating in the Fp interface. If the master CE 169 becomes faulty, a backup CE takes over and NE operation continues. 170 By definition, the current documented setup is known as cold-standby. 171 The set of CEs controlling an FE is static and is passed to the FE by 172 the FE Manager (FEM) via the Ff interface and to each CE by the CE 173 Manager (CEM) in the Fc interface during the pre-association phase. 175 From an FE perspective, the knobs of control for a CE set are defined 176 by the FEPO LFB in [RFC5810], Appendix B. In Section 3.1 of this 177 document we discuss further details of these knobs. 179 2.1. Document Scope 181 It is assumed that the reader is aware of the ForCES architecture to 182 make sense of the changes being described in this document. This 183 document provides background information to set the context of the 184 discussion in Section 4. 186 At the time this document is being written, the Fr interface is out 187 of scope for the ForCES architecture. However, it is expected that 188 organizations implementing a set of CEs will need to have the CEs 189 communicate to each other via the Fr interface in order to achieve 190 the synchronization necessary for controlling the FEs. 192 The problem scope addressed by this document falls into 2 areas: 194 1. To describe with more clarity (than [RFC5810]) how current cold- 195 standby approach operates within the NE cluster. 197 2. To describe how to evolve the [RFC5810] cold-standby setup to a 198 hot-standby redundancy setup to improve the failover time and NE 199 availability. 201 2.2. Quantifying Problem Scope 203 The NE recovery and availability is dependent on several time- 204 sensitive metrics: 206 1. How fast the CE plane failure is detected by the FE. 208 2. How fast a backup CE becomes operational. 210 3. How fast the FEs associate with the new master CE. 212 4. How fast the FEs recover their state and become operational. 214 The design intent of the current [RFC5810] as well as this document 215 to meet the above goals are driven by desire for simplicity. 217 To quantify the above criteria with the current prescribed ForCES CE 218 setup in [RFC5810]: 220 1. How fast the FE side detects a CE failure is left undefined. To 221 illustrate an extreme scenario, we could have a human operator 222 acting as the monitoring entity to detect faulty CEs. How fast 223 such detection happens could be in the range of seconds to days. 224 A more active monitor on the Fr interface could improve this 225 detection. 227 2. How fast the backup CE becomes operational is also currently out 228 of scope. In the current setup, a backup CE need not be 229 operational at all (for example, to save power) and therefore it 230 is feasible for a monitoring entity to boot up a backup CE after 231 it detects the failure of the master CE. In this document 232 Section 4 we suggest that at least one backup CE be online so as 233 to improve this metric. 235 3. How fast an FE associates with new master CE is also currently 236 undefined. The cost of an FE connecting and associating adds to 237 the recovery overhead. As mentioned above we suggest having at 238 least one backup CE online. In Section 4 we propose to zero out 239 the connection and association cost on failover by having each FE 240 associate with all online backup CEs after associating to an 241 active/master CE. Note that if an FE pre-associates with at 242 least one backup CE, then the system will be technically 243 operating in hot-standby mode. 245 4. And last: How fast an FE recovers its state depends on how much 246 NE state exists. By ForCES current definition, the new master CE 247 assumes zero state on the FE and starts from scratch to update 248 the FE. So the larger the state, the longer the recovery. 250 3. RFC5810 CE HA Framework 252 To achieve CE High Availability (HA), FEs and CEs MUST inter-operate 253 per [RFC5810] definition which is repeated for contextual reasons in 254 Section 3.1. It should be noted that in this default setup, which 255 MUST be implemented by CEs and FEs requiring HA, the Fr plane is out 256 of scope (and if available is proprietary to an implementation). 258 3.1. RFC 5810 CE HA Support 260 As mentioned earlier, although there can be multiple redundant CEs, 261 only one CE actively controls FEs in a ForCES NE. In practice there 262 may be only one backup CE. At any moment in time, only one master CE 263 can control an FE. In addition, the FE connects and associates to 264 only the master CE. The FE and the CE are aware of the primary and 265 one or more secondary CEs. This information (primary, secondary CEs) 266 is configured on the FE and the CE during pre-association by the FEM 267 and the CEM respectively. 269 Figure 2 below illustrates the Forces message sequences that the FE 270 uses to recover the connection in current defined cold-standby 271 scheme. 273 FE CE Primary CE Secondary 274 | | | 275 | Asso Estb,Caps exchg | | 276 1 |<--------------------->| | 277 | | | 278 | state update | | 279 2 |<--------------------->| | 280 | | | 281 | | | 282 | FAILURE | 283 | | 284 | Asso Estb,Caps exchange | 285 3 |<------------------------------------------>| 286 | | 287 | Event Report (pri CE down) | 288 4 |------------------------------------------->| 289 | | 290 | state update | 291 5 |<------------------------------------------>| 293 Figure 2: CE Failover for Cold Standby 295 3.1.1. Cold Standby Interaction with ForCES Protocol 297 HA parameterization in an FE is driven by configuring the FE Protocol 298 Object (FEPO) LFB. 300 The FEPO CEID component identifies the current master CE and the 301 component table BackupCEs identifies the configured backup CEs. The 302 FEPO FE Heartbeat Interval, CE Heartbeat Dead Interval, and CE 303 Heartbeat policy help in detecting connectivity problems between an 304 FE and CE. The CE Failover policy defines how the FE should react on 305 a detected failure. The FEObject FEState component [RFC5812] defines 306 the operational forwarding status and control. The CE can turn off 307 the FE's forwarding operations by setting the FEState to AdminDisable 308 and can turn it on by setting it to OperEnable. Note: [RFC5812] 309 section 5.1 has an erratta which describes the FEState as read-only 310 when it should be read-write. 312 Figure 3 illustrates the defined state machine that facilitates the 313 recovery of connection state. 315 The FE connects to the CE specified on FEPO CEID component. If it 316 fails to connect to the defined CE, it moves it to the bottom of 317 table BackupCEs and sets its CEID component to be the first CE 318 retrieved from table BackupCEs. The FE then attempts to associate 319 with the CE designated as the new primary CE. The FE continues 320 through this procedure until it successfully connects to one of the 321 CEs. 323 FE tries to associate 324 +-->-----+ 325 | | 326 (CE changes master || | | 327 CE issues Teardown || +---+--------v----+ 328 Lost association) && | Pre-Association | 329 CE failover policy = 0 | (Association | 330 +------------>-->-->| in +<----+ 331 | | progress) | | 332 | | | | 333 | +--------+--------+ | 334 | CE Association | | CEFTI 335 | Response V | timer 336 | +------------------+ | expires 337 | |FE issue CEPrimaryDown ^ 338 | V | 339 +-+-----------+ +------+-----+ 340 | | (CE changes master || | Not | 341 | | CE issues Teardown || | Associated | 342 | | Lost association) && | +->---+ 343 | Associated | CE Failover Policy = 1 |(May | FE | 344 | | | Continue |try v 345 | |-------->------->------>| Forwarding)|assn | 346 | | Start CEFTI timer | |-<---+ 347 | | | | 348 +-------------+ +-------+-----+ 349 ^ | 350 | Successful V 351 | Association | 352 | Setup | 353 | (Cancel CEFTI Timer) | 354 +_________________________________________+ 355 FE issue CEPrimaryDown event 357 Figure 3: FE State Machine considering HA 359 There are several events that trigger mastership changes: The master 360 CE may issue a mastership change (by changing the CEID value), or 361 teardown an existing association; and last, connectivity may be lost 362 between the CE and FE. 364 When communication fails between the FE and CE (which can be caused 365 by either the CE or link failure but not FE related), either the TML 366 on the FE will trigger the FE PL regarding this failure or it will be 367 detected using the HB messages between FEs and CEs. The 368 communication failure, regardless of how it is detected, MUST be 369 considered as a loss of association between the CE and corresponding 370 FE. 372 If the FE's FEPO CE Failover Policy is configured to mode 0 (the 373 default), it will immediately transition to the pre-association 374 phase. This means that if association is later re-established with a 375 CE, all FE state will need to be re-created. 377 If the FE's FEPO CE Failover Policy is configured to mode 1, it 378 indicates that the FE will run in HA restart recovery. In such a 379 case, the FE transitions to the Not Associated state and the CEFTI 380 timer [RFC5810] is started. The FE MAY continue to forward packets 381 during this state. The FE recycles through any configured backup CEs 382 in a round-robin fashion. It first adds its primary CE to the bottom 383 of table BackupCEs and sets its CEID component to be the first 384 secondary retrieved from table BackupCEs. The FE then attempts to 385 associate with the CE designated as the new primary CE. If it fails 386 to re-associate with any CE and the CEFTI expires, the FE then 387 transitions to the pre-association state and FE will operationally 388 bring down its forwarding path (and set the [RFC5812] FEObject 389 FEState component to OperDisable). 391 If the FE, while in the not associated state, manages to reconnect to 392 a new primary CE before CEFTI expires it transitions to the 393 Associated state. Once re-associated, the CE may try to synchronize 394 any state that the FE may have lost during disconnection. How the CE 395 re-synchronizes such state is out of scope for the current ForCES 396 architecture but would typically constitute the issuing of new 397 configs and queries. 399 An explicit message (a Config message setting Primary CE component in 400 ForCES Protocol object) from the primary CE, can also be used to 401 change the Primary CE for an FE during normal protocol operation. In 402 this case, the FE transitions to the Not Associated State and 403 attempts to Associate with the new CE. 405 3.1.2. Responsibilities for HA 407 TML Level: 409 1. The TML controls logical connection availability and failover. 411 2. The TML also controls peer HA management. 413 At this level, control of all lower layers, for example transport 414 level (such as IP addresses, MAC addresses etc) and associated links 415 going down are the role of the TML. 417 PL Level: 418 All other functionality, including configuring the HA behavior during 419 setup, the CE IDs used to identify primary and secondary CEs, 420 protocol messages used to report CE failure (Event Report), Heartbeat 421 messages used to detect association failure, messages to change the 422 primary CE (Config), and other HA related operations described in 423 Section 3.1, are the PL's responsibility. 425 To put the two together, if a path to a primary CE is down, the TML 426 would help recover from a failure by switching over to a backup path, 427 if one is available. If the CE is totally unreachable then the PL 428 would be informed and it would take the appropriate actions described 429 before. 431 4. CE HA Hot Standby 433 In this section we describe small extensions to the existing scheme 434 to enable hot standby HA. To achieve hot standby HA, we target to 435 improve the specific goals defined in Section 2.2, namely: 437 o How fast a backup CE becomes operational. 439 o How fast the FEs associate with the new master CE. 441 As described in Section 3.1, in the pre-association phase the FEM 442 configures the FE to make it aware of all the CEs in the NE. The FEM 443 MUST configure the FE to make it aware which CE is the master and MAY 444 specify any backup CE(s). 446 4.1. Changes to the FEPO model 448 In order for the above to be achievable there is a need to make a few 449 changes in the FEPO model. Appendix A contains the xml definition of 450 the new version 1.1 of the FEPO LFB. 452 Changes from the version 1 of FEPO are: 454 1. Added four new datatypes: 456 1. CEStatusType an unsigned char to specify status of a 457 connection with a CE. Special values are: 459 + 0 (Disconnected) represents that no connection attempt has 460 been made with the CE yet 462 + 1 (Connected) represents that the FE connection with the 463 CE at the TML has completed successfully 465 + 2 (Associated) represents that the FE has successfully 466 associated with the CE 468 + 3 (IsMaster) represents that the FE has associated with 469 the CE and is the master of the FE 471 + 4 (LostConnection) represents that the FE was associated 472 with the CE at one point but lost the connection 474 + 5 (Unreachable) represents the FE deems this CE 475 unreachable. i.e the FE has tried over a period to 476 connect to it but has failed. 478 2. HAModeValues an unsigned char to specify selected HA mode. 479 Special values are: 481 + 0 (No HA Mode) represents that the FE is not running in HA 482 mode 484 + 1 (HA Mode - Cold Standby) represents that the FE is in HA 485 mode cold Standby 487 + and 2 (HA Mode - Hot Standby) represents that the FE is in 488 HA mode hot Standby 490 3. Statistics, a complex structure, representing the 491 communication statistics between the FE and CE. The 492 components are: 494 + RecvPackets representing the packet count received from 495 the CE 497 + RecvBytes representing the byte count received from the CE 499 + RecvErrPackets representing the erronous packets received 500 from the CE. This component logs badly formatted packets 501 as well as good packets sent to the FE by the CE to set 502 components whilst that CE is not the master. Erronous 503 packets are dropped(i.e not responded to). 505 + RecvErrBytes representing the RecvErrPackets byte count 506 received from the CE 508 + TxmitPackets representing the packet count transmitted to 509 the CE 511 + TxmitErrPackets representing the error packet count 512 transmitted to the CE. Typically these would be failures 513 due to communication. 515 + TxmitBytes representing the byte count transmitted to the 516 CE 518 + TxmitErrBytes representing the byte count of errors from 519 transmit to the CE 521 4. AllCEType, a complex structure constituing the CE ID, 522 Statistics and CEStatusType to reflect connection information 523 for one CE. Used in the AllCEs component array. 525 2. Appended two new components: 527 1. Read-only AllCEs to hold status for all CEs. AllCEs is an 528 Array of the AllCEType. 530 2. Read-write HAMode of type HAModeValues to carry the HA mode 531 used by the FE. 533 3. Added one additional Event, PrimaryCEChanged, reporting the new 534 master CEID when there is a mastership change. 536 Since no component from the FEPO v1 has been changed FEPO v1.1 537 retains backwards compatibility with CEs that know only version 1.0. 538 These CEs however cannot make use of the HA options that the new FEPO 539 provides. 541 4.2. FEPO processing 543 The FE's FEPO LFB version 1.1 AllCEs table contains all the CEIDs 544 that the FE may connect and associate with. The ordering of the CE 545 IDs in this table defines the priority order in which an FE will 546 connect to the CEs. This table is provisioned initially from the 547 configuration plane (FEM). In the pre-association phase, the first 548 CE (lowest table index) in the AllCEs table MUST be the first CE that 549 the FE will attempt to connect and associate with. If the FE fails 550 to connect and associate with the first listed CE, it will attempt to 551 connect to the second CE and so forth, and cycles back to the 552 beggining of the list until there is a successful association. The 553 FE MUST associate with at least one CE. Upon a successful 554 association, the FEPO's CEID component identifies the current 555 associated master CE. 557 While it would be much simpler to have the FE not respond to any 558 messages from a CE other than the master, in practise it has been 559 found to be useful to respond to queries and hearbeats from backup 560 CEs. For this reason, we allow backup CEs to issues queries to the 561 FE. Configuration messages (SET/DEL) from backup CEs MUST be dropped 562 by the FE and logged as received errors. 564 Asynchronous events that the master CE has subscribed to, as well as 565 heartbeats are sent to all associated-to CEs. Packet redirects 566 continue to be sent only to the master CE. The Heartbeat Interval, 567 the CEHB Policy and the FEHB Policy are global for all CEs(and 568 changed only by the master CE). 570 Figure 4 illustrates the state machine that facilitates connection 571 recovery with HA enabled. 573 FE tries to associate 574 +-->-----+ 575 | | 576 (CE changes master || | | 577 CE issues Teardown || +---+--------v----+ 578 Lost association) && | Pre-Association | 579 CE failover policy = 0 | (Association | 580 +------------>-->-->| in +<----+ 581 | | progress) | | 582 | | | | 583 | +--------+--------+ | 584 | CE Association | | CEFTI 585 | Response V | timer 586 | +------------------+ | expires 587 | |FE issue CEPrimaryDown ^ 588 | |FE issue PrimaryCEChanged ^ 589 | V | 590 +-+-----------+ +------+-----+ 591 | | (CE changes master || | Not | 592 | | CE issues Teardown || | Associated | 593 | | Lost association) && | +->-----------+ 594 | Associated | CE Failover Policy = 1 |(May |find first | 595 | | | Continue |associated v 596 | |-------->------->------>| Forwarding)|CE or retry | 597 | | Start CEFTI timer | |associating | 598 | | | |-<-----------+ 599 | | | | 600 +----+--------+ +-------+----+ 601 | | 602 ^ Found | associated CE 603 | or newly | associated CE 604 | V 605 | (Cancel CEFTI Timer) | 606 +_________________________________________+ 607 FE issue CEPrimaryDown event 608 FE issue PrimaryCEChanged event 610 Figure 4: FE State Machine considering HA 612 Once the FE has associated with a master CE it moves to the post- 613 association phase (Associated state). It is assumed that the master 614 CE will communicate with other CEs within the NE for the purpose of 615 synchronization via the CE-CE interface. The CE-CE interface is out 616 of scope for this document. An election result amongst CEs may 617 result in desire to change mastership to a different associated CE; 618 at which point current assumed master CE will instruct the FE to use 619 a different master CE. 621 FE CE#1 CE#2 ... CE#N 622 | | | | 623 | Asso Estb,Caps exchg | | | 624 1 |<-------------------->| | | 625 | | | | 626 | state update | | | 627 2 |<-------------------->| | | 628 | | | | 629 | Asso Estb,Caps exchg | | 630 3I|<--------------------------------->| | 631 ... ... ... ... 632 | Asso Estb,Caps exchg | 633 3N|<------------------------------------------>| 634 | | | | 635 4 |<-------------------->| | | 636 . . . . 637 4x|<-------------------->| | | 638 | FAILURE | | 639 | | | | 640 | Event Report (LastCEID changed) | | 641 5 |---------------------------------->|------->| 642 | Event Report (CE#2 is new master) | | 643 6 |---------------------------------->|------->| 644 | | | 645 7 |<--------------------------------->| | 646 . . . . 647 7x|<--------------------------------->| | 648 . . . . 650 Figure 5: CE Failover for Hot Standby 652 While in the post-association phase, if the CE Failover Policy is set 653 to 1 and HAMode set to 2 (HotStandby) then the FE, after succesfully 654 associating with the master CE, MUST attempt to connect and associate 655 with all the CEs that it is aware of. Figure 5 steps #1 and #2 656 illustrates the FE associating with CE#1 as the master and then 657 proceeding to steps #3I to #3N the association with backup CEs CE#2 658 to CE#N. If the FE fails to connect or associate with some CEs, the 659 FE MAY flag them as unreachable to avoid continuous attempts to 660 connect. The FE MAY retry to reassociate with unreachable CEs when 661 possible. 663 When the master CE for any reason is considered to be down, then the 664 FE MUST try to find the first associated CE from the list of all CEs 665 in a round-robin fashion. 667 If the FE is unable to find an associated FE in its list of CEs, then 668 it MUST attempt to connect and associate with the first from the list 669 of all CEs and continue in a round-robin fashion until it connects 670 and associates with a CE. 672 Once the FE selects an associated CE to use as the new master, the FE 673 issues a PrimaryCEDown Event Notification to all associated CEs to 674 notify them that the last primary CE went down (and what its identity 675 was); a second event PrimaryCEChanged identifying the new master CE 676 is sent as well to identify which CE the reporting FE considers to be 677 the new master. 679 In most HA architectures there exists the possibility of split-brain. 680 However, since in our setup the FE will never accept any 681 configuration messages from any other than the master CE, we consider 682 the FE as fenced against data corruption from the other CEs that 683 consider themselves as the master. The split-brain issue becomes 684 mostly a CE-CE communication problem which is considered to be out of 685 scope. 687 By virtue of having multiple CE connections, the FE switchover to a 688 new master CE will be relatively much faster. The overall effect is 689 improving the NE recovery time in case of communication failure or 690 faults of the master CE. This satisfies the requirement we set to 691 achieve. 693 5. IANA Considerations 695 XXX: This document updates an IANA registered FE Protocol object 696 Logical Functional Block (LFB). At minimal when it becomes RFC we 697 should update https://www.iana.org/assignments/forces/forces.xml 698 section on FEPO. 700 6. Security Considerations 702 Security consideration as defined in section 9 of [RFC5810] applies. 704 7. References 706 7.1. Normative References 708 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 709 Requirement Levels", BCP 14, RFC 2119, March 1997. 711 [RFC5810] Doria, A., Hadi Salim, J., Haas, R., Khosravi, H., Wang, 712 W., Dong, L., Gopal, R., and J. Halpern, "Forwarding and 713 Control Element Separation (ForCES) Protocol 714 Specification", RFC 5810, March 2010. 716 7.2. Informative References 718 [RFC3654] Khosravi, H. and T. Anderson, "Requirements for Separation 719 of IP Control and Forwarding", RFC 3654, November 2003. 721 [RFC3746] Yang, L., Dantu, R., Anderson, T., and R. Gopal, 722 "Forwarding and Control Element Separation (ForCES) 723 Framework", RFC 3746, April 2004. 725 [RFC5812] Halpern, J. and J. Hadi Salim, "Forwarding and Control 726 Element Separation (ForCES) Forwarding Element Model", RFC 727 5812, March 2010. 729 Appendix A. New FEPO version 731 The xml has been validated against the schema defined in [RFC5812]. 733 736 737 738 739 CEHBPolicyValues 740 741 The possible values of CE heartbeat policy 742 743 744 uchar 745 746 747 CEHBPolicy0 748 749 The CE will send heartbeats to the FE 750 every CEHDI timeout if no other messages 751 have been sent since. 753 754 755 756 CEHBPolicy1 757 758 The CE will not send heartbeats to the FE 759 760 761 762 763 764 765 FEHBPolicyValues 766 767 The possible values of FE heartbeat policy 768 769 770 uchar 771 772 773 FEHBPolicy0 774 775 The FE will not generate any heartbeats to the CE 776 777 778 779 FEHBPolicy1 780 781 The FE generates heartbeats to the CE every FEHI 782 if no other messages have been sent to the CE. 783 784 785 786 787 788 789 FERestartPolicyValues 790 791 The possible values of FE restart policy 792 793 794 uchar 795 796 797 FERestartPolicy0 798 799 The FE restart restats its state from scratch 800 802 803 804 805 806 807 HAModeValues 808 809 The possible values of HA modes 810 811 812 uchar 813 814 815 NoHA 816 817 The FE is not running in HA mode 818 819 820 821 ColdStandby 822 823 The FE is running in HA mode cold Standby 824 825 826 827 HotStandby 828 829 The FE is running in HA mode hot Standby 830 831 832 833 834 835 836 CEFailoverPolicyValues 837 838 The possible values of CE failover policy 839 840 841 uchar 842 843 844 CEFailoverPolicy0 845 846 The FE should stop functioning immediate and 847 transition to the FE OperDisable state 848 849 850 851 CEFailoverPolicy1 852 853 The FE should continue forwarding even without an 854 associated CE for CEFTI. The FE goes to FE 855 OperDisable when the CEFTI expires and no 856 association. Requires graceful restart support. 857 858 859 860 861 862 863 FEHACapab 864 865 The supported HA features 866 867 868 uchar 869 870 871 GracefullRestart 872 873 The FE supports Graceful Restart 874 875 876 877 HA 878 879 The FE supports HA 880 881 882 883 884 885 886 CEStatusType 887 Status values. Status for each CE 888 889 uchar 890 891 892 Disconnected 893 No connection attempt with the CE yet 894 895 896 897 Connected 898 The FE connection with the CE at the TML 899 has been completed 900 901 902 903 Associated 904 The FE has associated with the CE 905 906 907 908 IsMaster 909 The CE is the master (and associated) 910 911 912 913 LostConnection 914 The FE was associated with the CE but 915 lost the connection 916 917 918 919 Unreachable 920 The CE is deemed as unreachable by the FE 921 922 923 924 925 926 927 StatisticsType 928 Statistics Definition 929 930 931 RecvPackets 932 Packets Received 933 uint64 934 935 936 RecvErrPackets 937 Packets Received from CE with errors 938 939 uint64 940 941 942 RecvBytes 943 Bytes Received from CE 944 uint64 945 946 947 RecvErrBytes 948 Bytes Received from CE in Error 949 uint64 950 951 952 TxmitPackets 953 Packets Transmitted to CE 954 uint64 955 956 957 TxmitErrPackets 958 959 Packets Transmitted to CE that incurred 960 errors 961 962 uint64 963 964 965 TxmitBytes 966 Bytes Transmitted to CE 967 uint64 968 969 970 TxmitErrBytes 971 Bytes Transmitted to CE incurring errors 972 973 uint64 974 975 976 977 978 AllCEType 979 Table Type for AllCE component 980 981 982 CEID 983 ID of the CE 984 uint32 985 986 987 Statistics 988 Statistics per CE 989 StatisticsType 990 991 992 CEStatus 993 Status of the CE 994 CEStatusType 995 996 997 998 999 1000 1001 FEPO 1002 1003 The FE Protocol Object, with new CEHA 1004 1005 1.1 1006 1007 1008 CurrentRunningVersion 1009 Currently running ForCES version 1010 uchar 1011 1012 1013 FEID 1014 Unicast FEID 1015 uint32 1016 1017 1018 MulticastFEIDs 1019 1020 the table of all multicast IDs 1021 1022 1023 uint32 1024 1025 1026 1027 CEHBPolicy 1028 1029 The CE Heartbeat Policy 1030 1031 CEHBPolicyValues 1032 1033 1034 CEHDI 1035 1036 The CE Heartbeat Dead Interval in millisecs 1037 1038 uint32 1039 1040 1041 FEHBPolicy 1042 1043 The FE Heartbeat Policy 1044 1045 FEHBPolicyValues 1046 1047 1048 FEHI 1049 1050 The FE Heartbeat Interval in millisecs 1051 1052 uint32 1053 1054 1055 CEID 1056 1057 The Primary CE this FE is associated with 1058 1059 uint32 1060 1061 1062 BackupCEs 1063 1064 The table of all backup CEs other than the 1065 primary 1066 1067 1068 uint32 1069 1070 1071 1072 CEFailoverPolicy 1073 1074 The CE Failover Policy 1075 1076 CEFailoverPolicyValues 1077 1078 1079 CEFTI 1080 1081 The CE Failover Timeout Interval in millisecs 1082 1083 uint32 1084 1085 1086 FERestartPolicy 1087 1088 The FE Restart Policy 1089 1090 FERestartPolicyValues 1091 1092 1093 LastCEID 1094 1095 The Primary CE this FE was last associated 1096 with 1097 1098 uint32 1099 1100 1101 HAMode 1102 1103 The HA mode used 1104 1105 HAModeValues 1106 1107 1108 AllCEs 1109 The table of all CEs 1110 1111 AllCEType 1112 1113 1114 1115 1116 1117 SupportableVersions 1118 1119 the table of ForCES versions that FE supports 1120 1121 1122 uchar 1123 1124 1125 1126 HACapabilities 1127 1128 the table of HA capabilities the FE supports 1129 1130 1131 FEHACapab 1132 1133 1134 1135 1136 1137 PrimaryCEDown 1138 1139 The primary CE has changed 1140 1141 1142 LastCEID 1143 1144 1145 1146 1147 LastCEID 1148 1149 1150 1151 1152 PrimaryCEChanged 1153 A New primary CE has been selected 1154 1155 1156 CEID 1157 1158 1159 1160 1161 CEID 1162 1163 1164 1165 1166 1167 1168 1170 Authors' Addresses 1172 Kentaro Ogawa 1173 NTT Corporation 1174 3-9-11 Midori-cho 1175 Musashino-shi, Tokyo 180-8585 1176 Japan 1178 Email: ogawa.kentaro@lab.ntt.co.jp 1179 Weiming Wang 1180 Zhejiang Gongshang University 1181 149 Jiaogong Road 1182 Hangzhou 310035 1183 P.R.China 1185 Phone: +86-571-88057712 1186 Email: wmwang@mail.zjgsu.edu.cn 1188 Evangelos Haleplidis 1189 University of Patras 1190 Panepistimioupoli Patron 1191 Patras 26504 1192 Greece 1194 Email: ehalep@ece.upatras.gr 1196 Jamal Hadi Salim 1197 Mojatatu Networks 1198 Suite 400, 303 Moodie Dr. 1199 Ottawa, Ontario K2H 9R4 1200 Canada 1202 Email: hadi@mojatatu.com