Internet Draft S.Hares Intended status: Informational Huawei Expires: January 6, 2013 July 6, 2012 Analysis of Comparisons between OpenFlow and ForCES draft-hares-forces-vs-openflow-00.txt Status of this Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet- Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html This Internet-Draft will expire on January 6, 2013. Copyright Notice Copyright (c) 2012 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Abstract Hares Expires March 12, 2013 [Page 1] Internet-Draft OpenFlow vs. ForCES July 12, 2012 While both ForCES and OpenFlow follow the basic idea of separations of forwarding plane and control plane in network elements, they are technically different. ForCES specification contains both a modeling language [RFC5812] which allows flexible definition of the Flow tables and flow logic. ForCES flow logic include Logical Functional Blocks (LFBs) connected in flow logic that is described in logic of direct graphs augmented by passage of Metadata and grouping concepts. OpenFlow's specifications contain a specific instantiation of Flow tables and flow logic which has emerged from the research community theories. OpenFlow's logic varies based on the revision of the specification (OpenFlow-Paper [McKeown2008], OpenFlow Switch Specification 1.0 [OpenFlow1-0], OpenFlow 1.1 [OpenFlow-1.1] Open Configuration 1.0 [OpenFlowConfig-1.0]). Table of Contents 1. Introduction...................................................3 1.1. ForcES Introduction.......................................4 1.2. OpenFlow Introduction.....................................4 2. Definitions....................................................5 2.1. New Common Configurations.................................6 2.2. Forces Definitions........................................8 2.3. Open Flow Definitions....................................10 3. Comparisons between ForCES and OpenFlow.......................12 3.1. Difference in Historical setting.........................12 3.2. Difference in Goals......................................14 3.3. Difference in Architectural Requirements.................14 3.3.1. ForCES System Building Blocks.......................15 3.3.2. OpenFlow Building blocks............................17 3.3.2.1. Match Fields in OFS............................18 3.3.2.2. Flow Logic - Flow Table and Group tables.......18 3.3.3. ForCES FE types.....................................22 ForCES and OpenFlow FEs can operate either new switching entities or integrated with existing processing as a hybrid. In OFS-1.2, the Ships-in-the-Night (SIN) mode divides existing ports into groups controlled by specific ports (see figure x) or VLANs (figure-x).........................22 3.3.4. ForCES Pre-Association..............................23 3.3.5. Architectural requirements..........................25 3.3.6. ForCES versus OpenFlow - A Central Controller.......33 3.4. Difference in Forwarding Model...........................33 3.4.1. Looping.............................................34 3.4.2. Handling unicast and multicast......................35 3.5. Difference in Logical Forwarding Block Libraries.........36 Hares, et al. Expires November, 6 2012 [Page 2] Internet-Draft OpenFlow vs. ForCES July 12, 2012 3.6. Difference in Protocol Interface.........................36 3.6.1 Secure Transport..................................37 4. Use of ForCES and OFS in Applications.........................41 5. The use of ForCES or OpenFlow in S(D)N or CSO/SOP.............41 6. Security Considerations.......................................42 7. IANA Considerations...........................................42 8. Conclusions...................................................42 9. References....................................................43 9.1. Normative References.....................................43 9.2. Informative References...................................43 10. Acknowledgments..............................................44 1. Introduction This document analyzes the differences between OpenFlow and ForCES technically from the aspects of goals, architecture, forwarding models, forwarding policy, control plane interaction, configuration of nodes, and applications (firewalls, load- balancers, High-availability nodes). This informational document compares OpenFlow and Forces as of March, 2012 seeking to provide clarity for other discussions of controller/forwarding split, Software Defined Networking, Software Driven Networking, Cloud Service Oriented networking (CSO), and a host of orchestrators for virtualized network devices. A fellow Engineer provided inspiration for this deeper comparison by saying: "OpenFlow-0 is the Diff-Serv Tspec, OpenFlow-1.0 is Forces--, and OpenFlow 1.1 is Forces++." Jamal Salim suggests Open Flow 1.1 does not have the same functionality[Jamal-01]. While this summary brings the expert listener quickly into heart of the issues, this document examines: - Is OpenFlow Switch 1.1.0 really "ForCES++", and "is the group table safe ++ logic? What direction does Open Flow Switch 1.2 and 1.3 take us? - Where does Open Flow's Config fit in the picture? - How does this help us get to Clouds Service Oriented Networks (CSO) enable by Software defined networks (SDN) or software driven networks (SDN)? And that, as the saying goes is the "rest of the story" of this draft. Here's hoping the readers of this document will decide and argue with the author to refine the next-generation of hardware devices. Hares, et al. Expires November, 6 2012 [Page 3] Internet-Draft OpenFlow vs. ForCES July 12, 2012 1.1. ForcES Introduction ForCES (Forwarding and Control Element Separation) work in IETF has defined a new environment to build network devices that split the network devices into control plane and forwarding plane into units. For example, a router could be considered a network element (NE) with a control plane running router protocol and a data plane forwarding IP traffic. The drive to have ForCES NE device split arose from the desire to build hardware forwarding blades out of flexible hardware components. These hardware devices included Network Processors and network specific ASICs. The ForCES environment defines requirements [RFC3654], goals [RFC3565], architecture and protocol requirements [RFC3654], a controller-forwarder communication protocol [RFC 5810]. ForCES also describes a policy on how to building the forwarding engine out of a set of logical functional blocks (LFBs)which are connected as a directed graph [RFC5812]. ForCES allows many different Forwarding Engines (FE) to linked to Controller Engines (CE) via the protocols. ForCES provides a modeling language [RFC- 5812] to describe these FE devices so that controllers can load control the devices, load forwarding tables, and keep track of statistics. ForCES RFCs also define how the ForCES protocol runs over SCTP [RFC5811. 1.2. OpenFlow Introduction OpenFlow[McKeown2001, p. 1]] arose out of the frustration that network research projects felt at not being able to experiment with new protocols on large-scale networks. Experimentation on research networks did not have a large enough scale to provide a reasonable test-bed for new research ideas for the Internet. Pure commercial networks would not allow experimental protocols, and commercial router vendors took 3-5 years to create a new protocol features. The OpenFlow researchers suggested an alternative to allow the research to creating a slice out of commercial network to try out new ideas for network. OpenFlow's initial paper grew into OpenFlow Switch Specification versions 1.0 [OFS-1.0], 1.1 [OFS-1.1], 1.2[OFS-1.2], 1.3[OFS-1.3], and (likely) 1.4 [OFS-1.4]. Additionally a Config and Management Protocol version 1.0[OFC-1.0] and version 1.1 [OFC-1.1], and a set of papers and application notes on implementations. A hybrid Specification [OFHy-1.0] suggests how Open Flow may be combined with existing network OpenFlow switches which mix existing Hares, et al. Expires November, 6 2012 [Page 4] Internet-Draft OpenFlow vs. ForCES July 12, 2012 network devices (routers/switches) with OpenFlow controlled switches in either a Ships-in-the Night (SIN) or a hybrid model OpenFlow's host of specifications and ForCES flexible reprogramming of the network element architecture can be considered the first wave of Software-Defined Networking (SDfN) where software can alter how the forwarding engine's logic operates. This work arises out of the GENI research [GENI][McKeown2008]. The current industry discussion regarding Software-Defined Networking (SDfN) or Software-Driven Networking (SDrN), Network Virtualized Overlays [NVO3], Service-Oriented Protocols (SoP)[SoP] or network orchestrators of Cloud Service Oriented Network (CSO)forwarding [CSO-Arch] have a great deal confusions as they apply the terms to ForCES and OpenFlow. Rather than carefully define each of these terms explicitly at the outset, we will give brief expansions of the abbreviations and return to the definitions later in the draft after examining the FORCES draft. This document will examine ForCES and OpenFlow goals, architecture, forwarding conceptual models, Controller-forwarder communication mechanisms and protocols, the policies in the loading of forwarding state, configurations of nodes, and sanity checking of the forwarding. These basic concepts will be then examined in terms of specific implementations (switch, hybrid router/switch, wireless, load- balancer, firewall) as described by ForCES and OpenFlow reports. Finally, the document will return to defining S[*]N, SOP, and Cloud-Oriented Services (CSO). 2. Definitions Definitions used in this document The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC-2119 [RFC2119 The following RFC2119 definitions used in this document are: The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC-2119 [RFC2119]. Hares, et al. Expires November, 6 2012 [Page 5] Internet-Draft OpenFlow vs. ForCES July 12, 2012 ForCES definitions relevant to this documents discussion are taken from [RFC3654][RFC3746][RFC5810] as noted below. The quoted italized definitions come from the ForCES RFC, and the non-quoted text applies ForCES RFC text to this document. 2.1. New Common Configurations Controlling Entity (CE): is defined as an entity which remote controls the forwarding engine. This Entity can be either a ForCES CE or a Open Flow Controller. Controlling Entity Manager (CE-MGR): This documents loosens the ForCES CE-Manager definition to allow Open Flow and ForCES to be compared. This document defines the CE manager as a logical entity (distributed or located in physical or virtual device) which controls which controllers attach to which logical Forwarding Entities. The Controllers can be in the same physical switch/device in the control plane or other logical software. A CE-Mgr may also be within a VM hypervisor, a VM hypervisor manager, or other virtual software. The CE-Mgr logical function may be distributed across many CE as a defined function. This definitional Allows both ForCES CE-Mgrs and Open Flow Controller collaboration/management via coordinated remote configuration of OF Capable Switches. Controlled Router-Switch (CRSW): A Controlled Switch is a network entity performing switching capability that is controlled by remotely by either the ForCES protocol (FP) or the Open Flow Protocol (OFP). This switch can perform IP routing, MPLS switching, Trill Switching or Layer 2 Switching. Forwarding Entity (FE): is defined as an entity which forwarding packets or frames under the control of the CE. This entity can be an ForCES FE (F-FE) or an Open Flow Capable Switch (OF-CS). An Open Flow Capable switch can either be a hybrid switch or a Open- Flow Only switch. FE Manager (FE-MGR): The FE-Manager controls the FEs assignments. This document defines FE-Manager's logical entity may be a logical software process residing within local switch/device in the control plane or management plane. The FE-Mgr can also within a VM hypervisor, or a VM hypervisor manager, or other virtual software. The FE-Mgr can be a remote service managing the forwarding engine. The Open Flow Configuration [OFC-1.0] Configuration Service point with its logical configuration function may also have a FE-MGR function. This FE-Mgr capability is an capability outside the [OFC-1.0] specification. Hares, et al. Expires November, 6 2012 [Page 6] Internet-Draft OpenFlow vs. ForCES July 12, 2012 Pre-Association Phase (Pre-A): This document defines a Pre- Association Phase (Pre-A) as the period during which a CE- Management(Forces CE-Mgr or OF controller groups) and FE-Managers (Forces FE-MGR (F-FE-MGR)or OF-CS management) determines which Controlling entity (CE)controls which Forwarding Entity (FE). Hares, et al. Expires November, 6 2012 [Page 7] Internet-Draft OpenFlow vs. ForCES July 12, 2012 2.2. Forces Definitions Force Forwarding Element (F-FE) - "A logical entity that implements the ForCES Protocol. FEs use the underlying hardware to provide per-packet processing and handling as directed by a CE via the ForCES Protocol." [RFC3654] ForCES forwarding FE supports forwarding rules insertion. ForCES Control Element (F-CE) - "A logical entity that implements the ForCES Protocol and uses it to instruct one or more FEs on how to process packets. CEs handle functionality such as the execution of control and signaling protocols." [RFC3654] The ForCES CE controller may be located within the same hardware box on a different blade or across an Ethernet connection, or across a L3 Link (if security used). ForCES Network Element(F-NE)- "An entity composed of one or more CEs and one or more FEs. To entities outside a NE, the NE represents a single point of management. An NE usually hides its internal organization from external entities and represents a single point of management to entities outside the NE." [RFC3654] The NE's single point of management can be at the IP layer, the Ethernet layer, and at a virtual layer. In this document, the network element is examined as being the set of network functions in the hardware that collaborates to act like a switch. This less strict definition allows ForCES to be compared with the Open Flow work. ForCES Pre-Association Phase (F-Pre-A): ForCES defines the Pre- Association Phase (F-Pre-A) as "the period of time during which a FE Manager(see below)and a CE Manager (see below) are determining which FE is a part of the network element" [RFC3654]. FE Manager(F-FE-Mgr)- ForCES (F-FE-Mgr)is "A logical entity that operates in the Pre-Association Phase and is responsible for determining to which CE(s) a FE should communicate. This process is called CE Discovery and may involve the FE manager learning capabilities of available CEs." [RFC3654] CE Manager (CE-Mgr) - Forces CE-MGR[F-CE-Mgr] is "A logical entity that operates in the pre-associaation phase and is responsible for determining to which FE(s) a CE should communicate. This process is called FE discovery and may involve the CE manager learning the capabilities of available FEs. The CE manager may use anything from statics configuration to a pre- association phase protocol." [RFC3654] Hares, et al. Expires November, 6 2012 [Page 8] Internet-Draft OpenFlow vs. ForCES July 12, 2012 ForCES Protocol (ForCES-Proto) - "While there may be multiple protocols used within the overall ForCES architecture, the term "ForCES protocol" refers to only the post-association phase protocol." [RFC3654] The ForCES protocol operates between the "Fp reference points" of the ForCES architecture (as shown in figure 1) [RFC5810]. "Basically, the ForCES protocol works in a master- slave mode in which the FEs are slaves and the CEs are masters." [RFC5810] The location and exact instantiation of the CE logical entities associated with the FE logical entity is flexible. The CE entities could reside on a process on a local switch communicating to other process off the local switch. ForCES Protocol Layer (ForCES PL) - "A layer in the ForCES protocol architecture that defines the ForCES protocol messages, the protocol state transfer scheme, and the ForCES protocol architecture itself (including requirements of ForCES TML)" [RFC5810] This layer is defined in RFC5810. ForCES Protocol Transport Mapping Layer (ForCES TML) - "A layer in ForCES protocol architecture that uses the capabilities of existing transport protocols to specifically address protocol message transportation issues, such as how the protocol messages are mapped to different transport media (like TCP, IP, ATM, Ethernet, etc.), and how to achieve and implement reliability, multicast, ordering, etc. The ForCES TML specifications are detailed in separate ForCES documents, one for each TML." [RFC5810, p. x]. The ForCES TMLs focused on are STCP [STCP] and SSL[SSL]. TM handles transport of messages [reliable or non-reliable], "congestion control", "multicast", ordering, and other things [RFC5810, p. 14]. LFB (Logical Function Block) - The basic building block that is operated on by the ForCES protocol. The LFB is a well-defined, logically separable functional block that resides in an FE and is controlled by the CE via the ForCES protocol. The LFB may reside at the FE's data path and process packets or may be purely an FE control or configuration entity that is operated on by the CE. Note that the LFB is a functionally accurate abstraction of the FE's processing capabilities, but not a hardware-accurate representation of the FE implementation. LFB Class and LFB Instance - LFBs are categorized by LFB classes. An LFB instance represents an LFB class (or type) existence. There may be multiple instances of the same LFB class (or type) in an FE. An LFB class is represented by an LFB class ID, and an LFB instance is represented by an LFB instance ID. As a result, Hares, et al. Expires November, 6 2012 [Page 9] Internet-Draft OpenFlow vs. ForCES July 12, 2012 an LFB class ID associated with an LFB instance ID uniquely specifies an LFB existence. Physical Forwarding Element - The physical element that forwards the packets. 2.3. Open Flow Definitions Open Flow (OF): [McKeown-xx] defines OF as a "way for researchers to run experiments in networks they use every day"[McKeown-2008, p.1]. Open Flow Action [OF-Act]: [OF-1.1.0] defines an OF-Act as an action that may: "forward the packet to a port", or modifies the packet". Actions may be specified in "Open Flow Instruction" [OF-Inst] in Flow Table Entry or "action buckets in Group Table Entry"[OF-1.1.0,p.4]. Open Flow Action Set [OF-ActSet]: [OF-1.1] defines an OF-ActSet as "a set of actions associated with the packet that are accumulated while the packet is processed by each table, and are executed when the packet exits the processing Pipeline."[OF-1.1.0, p. 5]. Open Flow Capable switch [OF-CS]: [One or more physical or virtual switch device which can act as an operational context for an Open Flow Logical Switch [OF-LS]. A OF-CS hosts an Open Flow Data Path [OF-DP]. Open Flow Configuration and Management Protocol [OFCMP]: [OFC-1.0] states the [OFCMP-1.0]enables the remote configuration of Open Flow Data Path (OF-DP). The OFCMP allows a controller to configure the OF-DP on the Open Flow Logical Switch (OF-LS) "so that the controller can communicate and contro" the OF-LS via Open Flow Protocol (OFP). The OFCMP allows dynamic association of resources with OF-LS in an OF-CS. [OFC-1.0] defines the protocol, but not the resource allocation mechanisms. Open Flow Configuration Point (OFCPT): [OFC-1.0] defines an OFCPT as "a service" which sends OFCMP messages to a OF-CS with an OF- LS inside. Open Flow Controller [OF-CTLER]: [McKeown-2008] defines the OF- CTLER as a "controller that adds and deletes Flow entries on behalf of experiments" [McKeown-2008, p. 3]. Hares, et al. Expires November, 6 2012 [Page 10] Internet-Draft OpenFlow vs. ForCES July 12, 2012 Open Flow Datapath [OF-DP]: [OFC-1.0] defines OF-DP as an abstraction called Open Flow Logical Switch [OF-LS]. Open Flow Dedicated Switches [OF-DS]: [McKeown-2008] defines OF- DS as a "dumb datapath element that forwards packets between ports as defined by a remote process" [McKeown-2008, p3.] The Open Flow process programs the forwarding engine for this dumb datapath switch. Open Flow Enable Switches [OF-ES]: [McKeown-2008] defines OF-ES as "commercial switches, routers or access points" enhanced by adding the OF feature Open Flow Feature [OF-Feature]: Open Flow[McKeown2008] and [OF- 1.0.0] defines the OF-Feature as adding the features of an Open Flow Logical Switch [OF-LS]. These features are the Open Flow "Flow Tables", "Secure Channel that connects the switch to the controller", and "the Open Flow Protocol" [McKeown-2008, p. 3][OF-1.0.0, p.2]. Open Flow's Flow Table (OF-FT): [McKeown-2008] defines a Flow table in OF as having "an action with every flow table entry to tell the switch how to process the flow" [McKeown-2008, p. 2] Open Flow's Flow Table Entry (OF-FTE): [McKeown-2008], [OF-1.0.0], [OF-1.1.1], [OF-1.1.2], and [OF-1.1.3] define the specific of an single entry in a flow table. See Section x.x for a detailed comparison of this entry. Open Flow Group (OF-G): [OF-1.2] defines an OF group (OF-G) as a list of Open Flow "action buckets" and "a means to choose one or more buckets to apply on a per-packet basis" [OF-1.2, p. 5]. Open Flow Group Table (OF-GT): Open Flow Logical Switch[OF-LS]: OFC-1.0 defines the OF-LS as an abstraction of the "open flow datapath". Open Flow Packet (OF-Pkt): [OF-1.1.0] defines OF-Pkt as "ethernet frame including header and payload" [OF-1.1.0, p. 4]. Open Flow Pipeline [OF-PipeLine]: [OF-1.2] defines OF-Pipeline as "a set of linked tables that matching, forwarding, and Open Flow Port [OF-Port]: [OF-1.2] defines an Open Flow port as a place where packets enter and exit the Open Flow Pipeline. Hares, et al. Expires November, 6 2012 [Page 11] Internet-Draft OpenFlow vs. ForCES July 12, 2012 Open Flow Protocol (OFP): OF 1.0 defines "an open protocol to program the flow table in switches and routers" in which "a controller communicates with a switch"[McKeown-2008,p. 2-3]. Open Flow Switch (OFS): [McKeown-2008] defines an Open Flow Switch as a ethernet switch with "at least" the following three functions: "(1) a Flow Table","(2) "a secure channel that connects the" controller with the switch over which the open flow protocol runs, and (3)an "open flow protocol" [McKeown-2008,p 2.][OF-1.0.0,p.2]. [OF-1.1.0] defines an OFS as "one or more flow tables and a group table which perform packet look-ups and forwarding", and an open flow channel to an external controller"[OF-1.1.0,p.3]. The external controller controls the switch via the Open Flow protocol (OF-Proto)[OF-1.1.0, p.3]. [OF-1.3.0] adds that the "switch communicates with the controller, and the controller controls the switch"[OF-1.3.0, Section 2, paragraph 1.] 3. Comparisons between ForCES and OpenFlow ForCES and OpenFlow are very similar in the following aspects: o Both ForCES and OpenFlow are efforts to separate control plane from forwarding plane; o Both ForCES and OpenFlow protocols standardize information exchange between the control and forwarding plane. Although both ForCES and OpenFlow can be considered as the solutions for forwarding and control plane separation, they are different in many aspects. This section compares them in their history, goals, architecture, forwarding model and protocol interface. 3.1. Difference in Historical setting ForCES work began during the 1995-2000 timeframe with the development of netlink [RFC3549]. The linux netlink began its linux driver history as first a "character device /dev/netlink for Linux kernel 1.3.31" but was superceded by "Alexey Kunzetsov's socket-based af_netlink.c in Linux v 2.1.15" [Englehardt-2010]. The rtnetlink brought configuration and router queries to links. The netlink socket allowed messages between kernel and user space regarding routes, firewalls, and monitoring. Hares, et al. Expires November, 6 2012 [Page 12] Internet-Draft OpenFlow vs. ForCES July 12, 2012 The historical context of the 1995-2002 timeframe saw the initial growth of the US Internet and spread into non-US sites. The historical changes included changes that began the split of tight binding of control plane to data plane. Forwarding plane elements (ASICs, TCAMs, Network processors) and backplanes began the growth of non-stop forwarding with high-availability. ForCES notes that the data processors have "forwarding tables", "per- flow Qos Tables and access control lists" [RFC365]. ForCES had control processors were general-purpose processors that support route calculations. ForCES began in quasi-commercial realm of linux development where linux developers used routing software with netlink to build early Internet networks. Alexey's early work was deployed in Russian and European networks to turn linux boxes into Routers. By early 2000, this work had migrated to router boxes seeking to harden routers to provide non-stop forwarding. Netlink implementations were provided with many commercial OEM standards for switches and routers. ForCES work came out of the desire to expand the basic netlink protocol into an architecture that allow quick modeling of new forwarding hardware and an extensible control-plane to forwarding plane communication. Early discussions in ForCES look to allow coordination of multiple control planes as well as control plane to forwarding plane functionality. However, the IETF decision was to restrict the first versions of ForCES to architecture, CE-FE communication and FE modeling. OpenFlow arises out of the academic's communities realization that the hardening of commercial of network infrastructure [1995- 2006] to support businesses, caused a "reluctance to experiment with production traffic"[McKeown-2008, p. 1]. The GENI (Global Environments for Network Research)[2006-2007] suggested that: a)Internet's infrastructure faced "serious problems" in "security, reliability, manageability, and evolvability" and "possible solutions" existed in research, but there were "severe experimental barriers" to test new ideas in wide-spread deployments [GENI-2007, p. vii]. A new US research network would allow slices of routers to be used for researcher's experiments in network protocols. McKeown and colleagues work examined how these experiments could be extended to run [McKeown-2008] on local campuses. McKeown and colleagues examined persuading commercial routers to provide an open interface for experimentation or using existing open source solutions (linux, XORP[XORP-2008]). Their conclusion was that "commercial solutions are too closed", and "research solutions have insufficient Hares, et al. Expires November, 6 2012 [Page 13] Internet-Draft OpenFlow vs. ForCES July 12, 2012 performance or fanout, and are too expensive." [McKeown-2008, p. 2]. 3.2. Difference in Goals RFC3654 lists the the architectural goals of ForCES. OpenFlow Switch (OFS) Specification version 1.0.0[OFS-1.0.0] and OFS version 1.1.0 [OF-1.1.0] refer to [McKeown-2008] as the basis of these specifications. This document's goal comparison compares the four goals [McKeown-2008] sets against [RFC3654]. The goal ForCES is to define the "architecture", "architectural modeling" and protocols to "logically separate control and data forwarding plans of an IP (IPv4, IPv6, etc.) networking device" [RFC3654, p. 1]. ForCES network device (aka. network element (NE)) can be a router, IP switch, firewall, switch. ForCES redefines network elements to be logical relationships placed on physical devices. McKeown et al. state the goals of the OpenFlow approach was to find "high-performance and lost-cost implementations" of new network algorithms, capable of being used in "broad range of research", adaptable to commercial "close platforms", and able to "isolate experimental traffic from production traffic [McKeown- 2008, p. 2]. Difference in goals underscores the original commercial focus of ForCES and the experimental focus of OpenFlow. 3.3. Difference in Architectural Requirements Architecture sets the building blocks for system, and architectural requirements sets rules for interconnecting the building blocks. Building blocks for ForCES include the CE(s), FE(s), ForCES protocol (CE to/from FE), FE-Manager, CE-Manager, and logical input flows. Within the FEs there are Logical Forwarding Blocks connected together in a directed graph. The flow processing passes along input port, (modified)frame, metadata (which may include actions). The flow stream may be output to interfaces (logical or physical). Building blocks for OpenFlow include Controllers (~CEs) and Forwarding Units (~FEs) with OpenFlow processing. OpenFlow logic is designed in terms of Flow Processing controlled by Flow Tables (McKeown-2008][OFS-1.0][OFS-1.1], and Group tables [OFS-1.1]which Hares, et al. Expires November, 6 2012 [Page 14] Internet-Draft OpenFlow vs. ForCES July 12, 2012 operate on the modified frame, metadata, or group of actions via actions or instructions (a group of actions and forwarding commands). Both flow streams Flow processing may cause the flow to multiple into several streams or combine multiple streams into one. 3.3.1. ForCES System Building Blocks The building blocks within the ForCES architecture are the CEs (controller elements), FEs (forwarding elements), and an interconnect protocol between CE(s) and FE(s). ForCES also recognized the logical functions of a FE-Manager and a CE- Manager. Figure 1 shows a diagram from RFC5810 that details interaction between all these components. The ForCES CE controls switching, signaling, routing, and management protocols. Each CE is a logical unit which may be located within the same box, different boxes, or across the network. ForCES architecture [RFC3746] allows CEs to control forwarding in multiple FEs. ForCES defines logical Forwarding Elements (FEs) that reside on a variety of physical forwarding elements (PFE) such as a "single blade (PFE)", partition within blade, or multiple PFEs in a single box, or among multiple boxes [RFC3746, p. 2]. The ForCES logical FEs could also be run within Virtual Machines (VMs) within a single box or a set of boxes or a cloud. A single FE may be connected to multiple CEs providing strong redundancy. FE internal processing is described in terms of Logical Forwarding Blocks (LFBs) connected together in a directed graph that "receive, process, modify and transmit packets along with metadata"[RFC5810, p. 6]. The FE model determines the LFBs, the topological description, the operational characteristics, and the configuration parameters of each FE. The Forces Logical Forwarding Block (LFBs) Library [FORCES-LFB] provides the class descriptions for Ethernet, IP Packet Validation, IP Forwarding LFBs, and Redirection, MetaData, and Scheduling. Forces-LFB document demonstrates how these logical blocks can be placed within a machine to support IP Forwarding (IPv4/IPv6)for unicast & multicast and ARP processing[Forces-LFB, p. 17]. ForCES architecture [RFC3746] allows CEs to control forwarding in multiple FEs. ForCES also recognized the logical functions of a FE-Manager and a CE-Manager. The FE manager determines the CE(s) Hares, et al. Expires November, 6 2012 [Page 15] Internet-Draft OpenFlow vs. ForCES July 12, 2012 each FE should communicate with. The CE manager determines which FEs each CE should communicate with. The ForCES defines the FE- Manager and CE-Manager to operate in a "pre-association" phase of the communication to set-up the FORCES communication path. Similarities between the functions of the CE-Mangers and FE managers of ForCES and modern hypervisors may come from the creative interplay of early open source communities [1995-2005]. Applications directly interacting with ForCES components (CEs or CE-Managers or FE-Manager) could be described as interactions with the CEs or CE-Managers. Figure 1 shows ForCES Architectural Diagram [RFC5810]. --------------------------------------- | ForCES Network Element | -------------- Fc | -------------- -------------- | | CE Manager |---------+-| CE 1 |------| CE 2 | | -------------- | | | Fr | | | | | -------------- -------------- | | Fl | | | Fp / | | | Fp| |----------| / | | | | |/ | | | | | | | | | Fp /|----| | | | | /--------/ | | -------------- Ff | -------------- -------------- | | FE Manager |---------+-| FE 1 | Fi | FE 2 | | -------------- | | |------| | | | -------------- -------------- | | | | | | | | | | | ----+--+--+--+----------+--+--+--+----- | | | | | | | | | | | | | | | | Fi/f Fi/f Fp: CE-FE interface Fi: FE-FE interface Fr: CE-CE interface Fc: Interface between the CE manager and a CE Ff: Interface between the FE manager and an FE Fl: Interface between the CE manager and the FE manager Fi/f: FE external interface Figure 1: ForCES Architectural Diagram [RFC5810] Hares, et al. Expires November, 6 2012 [Page 16] Internet-Draft OpenFlow vs. ForCES July 12, 2012 3.3.2. OpenFlow Building blocks OpenFlow architecture consists of set of OpenFlow Switches with a Flow Table, a Secure Channel between controller and switch, and an Open flow protocol. OpenFlow switches can either be only controlled by OpenFlow, enabled by Open Flow [OFS 1.0] (shared), or hybrid Switches (OFS 1.2). ---------------- ---------------- | Application1 | | Application2 | ...... ---------------- ---------------- | APIs | ---------------------------------------- CE | --------------- --------------- | | | OpenFlow |------| OpenFlow | | | | Controller | | Controller | | | --------------- --------------- | ---------------------------------------- | | | | OpenFlow Protocol | | | | NE&FE | | | NE&FE -------------- | -------------- | OpenFlow | | | OpenFlow | | Switch | | | Switch | -------------- | -------------- | | | | | | | | | | | | | | | | | | | | | | | | | | | Fi/f | NE&FE | | Fi/f | -------------- | | | OpenFlow | | | | Switch | | | -------------- | | | | | | | | | | | | | -------- | | -------- Fi/f Fi/f: FE external interface Figure 2: OpenFlow Architectural Diagram by Using the terms NEs, FEs, CEs The Flow table provides entries on how to process a flow whose header fields match a pattern in the header field [OFS-1.0.0] or Hares, et al. Expires November, 6 2012 [Page 17] Internet-Draft OpenFlow vs. ForCES July 12, 2012 a set of meta data generated from pipeline processing of a header[OFS-1.1.0, figure 4][OFS-1.3]. The matching of a packet in OFS-1.0.0 based on first exact match of header and/or meta data, and secondly wild-card entries. The wild-card entries contain a priority field to order the process of matching. For example, a priority of "1" will be the first wild card processed. 3.3.2.1. Match Fields in OFS The match field has been expanding as the ONF specifications evolve - from a 10-tuple [McKowen-2008], to 12-tuple [OFS-1.0], and to a OFS-1.1.0 a 15-tuple, to 39-tuple in OFS-1.3[OFS-1.3] (14 required and 25 optional). The original 10-tuple includes ingress port, VLAN ID, Ethernet source address, destination address and type, the IPv4 source/destination address, and IP protocol, TCP/UDP source & destination port. The 12-tuple adds VLAN priority, and IP Tos bits. The 15-tuple adds: metadata, MPLS label, MPLS traffic class. The TCP/UDP source & destination port have redefined for ICMP packets to have instead he the ICMP Type and ICMP code. [OFS-1.3-pre]required matches include the 10-tuple plus IPv6 source and destination addresses and UDP source and destination ports. 3.3.2.2. Flow Logic - Flow Table and Group tables The flow table operation and data structures vary based on the version of OpenFlow Switch specification. OFS in versions [McKeown-2008] and [OFS-1.0.0] operate on logic in flow tables which are executed in ascending order. Each Flow Table ID must be greater than the current Flow table id. [OFS- 1.1.0][OFS-1.3] flow logic operates on flow tables and group tables which allow jumps based on "Goto table" logic or combinations of flows. |=============| |==============| |=============| |Flow table 0 |---.|Flow Table 1 | _ -. |Flow Table n | |=============| |==============| |=============| Hares, et al. Expires November, 6 2012 [Page 18] Internet-Draft OpenFlow vs. ForCES July 12, 2012 Figure 3: Flow Table [OFS-0.9][OFS-1.0] All OFS Flow tables match on data and perform actions [OFS- 0.8][OFS-0.9][OFS-1.0][OFS-1.1][OFS-1.2][OFS-1.3]. Later versions [OFS-1.1.0][OFS-1.2][OFS-1.3] use instructions to immediately perform actions or to queue specific actions for later processing. These same later versions also allow metadata to be stored to be passed along to additional processing. |----------------| | Group table 1 | |------. |action-bucket1 |----. egress-port-1 | | action-bucket2 ] | |----------------| | |=============| |==============| |=============| |Flow table 0 |---.|Flow Table 1 | _ -. |Flow Table n | |=============| |==============| |=============| Figure 3: Flow Table [OFS-1.1] 1) Action Specifics [McKeown-2008] has three actions in the flow tables: forwarding of a matched packet to a specific port or ports, sending the packet to the controller, or dropping the packet. This simply processing is why some engineers suggest that OFS-2008 is similar to the RSVP T-Spec [RFC2870]. [OFS 1.0.0] actions direct switch processing to forward packet, drop packet, enqueue packet (optional), and modify-field in packet (optional). Forwarding packets can be sent to all ports, the controller, local switches forwarding stack, send out input port, and specific ports after performing table actions & send out specified port, send via normal (L2/L3/VLAN) processing, and Hares, et al. Expires November, 6 2012 [Page 19] Internet-Draft OpenFlow vs. ForCES July 12, 2012 flood (via minimum spanning tree) ports. This causes some engineers to consider OFS 1.0 equivalent to be Forces--. [OFS-1.1.0] uses instructions within each flow entry to determine how a packet and associated data is processed. These associated data includes: ingress port packet came in on, the generated meta-data and an action set. The action set is a set of commands to execute prior to sending the packet out. The [OFS-1.1.0] instructions are: "Apply-Actions", "Clear-Actions", "Write- Actions", "Write-metadata", "Goto Table" [OFS-1.1.0, p. 14]. Actions are either applied immediately with apply action command, or stored (via an Write-Action)in action sets for later processing in the action-set via a Write-Action. The existing actions can be cleared with a "Clear-Action". [OFS-1.1.0] actions are output packet, set queue, drop packet, process via group table, push/pop tags, and set-field. [OFS-1.1.0] action sets include single entries for any of the following: copy TTL inward in packet, pop/push actions to packet, copy TTL outwards, decrement TTL, set fields in packet, apply QoS Actions (e.g. set queue), and apply group actions, and output packet. 2) Flow Logic Encoding [OFS-1.0.0] encodes the ForCES LFB in table sequences. The LFB directed graph of ForCES modeling is encoded in sequences of Flow Table. OFS 1.0 specifies that the Ethernet header (similar to ForCES Ethernet II) is the basic frame for all input. A fixed processing ARP, IPv4, TCP/UDP, and ICMP packets are specified based matches beyond the Ethernet header. The Forces LFB library provides building blocks for matches beyond the Ethernet to ARP, IPv4 and IPv6 packets, and Meta data. The OFS-1.0.0 does not have Metadata. [OFS-1.1.0] match of a flow goes against specific table 15-tuple header. If the frame/packet matches, the flow table can alter the packet (via immediate actions), add metadata (for later handling), set an actions in action set, pass the processing to a specific table (Flow Table or Group Table), or pass the processing to the next table in the sequence. The information passed on to the next processing is [ingress port, (post- modification) frame/packet, metadata, and action set. [OFS-1.1.0] allows processing between tables to carry the ingress port, the packet, generated metadata, and an action set. An action set is a set of commands to execute prior to sending the packet out. [OFS-1.1.0] uses Flow table with instruction. The instructions can be which can be "Apply-Actions", "Clear- Hares, et al. Expires November, 6 2012 [Page 20] Internet-Draft OpenFlow vs. ForCES July 12, 2012 Actions", "Write-Actions", "Write-metadata", "Goto Table" [OFS-1.1.0, p. 14]. Actions are either applied immediately with apply action command, or stored (via an Write-Action) for later processing in the action-set via a Write-Action. The existing actions can be cleared with a Clear-Action. [OFS-1.1.0] supports two types of tables: flow tables, and group tables. The group table structure has a group identifier, group type, counters, and an order list action buckets. Action buckets contain a set of actions with parameters. If the group table has "zero" action buckets, then no processing occurs. [OFS-1.1.0] group type field specifies how the action buckets operate on the packet. The group can be "all", "select", "indirect", and "fast-failover" [p.7]. The "all" bucket provides multicast and/or broadcast support execute all buckets. The packet is effectively cloned and sent out. The select, indirect, and fast-failover execute one bucket. In select, the switch determines which port via internal algorithm (e.g. round-robin). In "indirect", the group table logic selects the bucket. The "fast-failover", the first live bucket is chosen. The single- bucket choses may restrict flows if the specified buckets and/or output ports are down. This new sequential logic is the basis of some engineers comment that OpenFlow 1.1 is ForCES++. ForCES people indicate that that the group table logic plus "Go to" logic is simply a LFB model of a specific type. [Haleplidis- 2012] demonstrates on the LFB library concept can be used to capture all of the OFS-1.1.0 specification. Hares, et al. Expires November, 6 2012 [Page 21] Internet-Draft OpenFlow vs. ForCES July 12, 2012 3.3.3. ForCES FE types ForCES and OpenFlow FEs can operate either new switching entities or integrated with existing processing as a hybrid. In OFS-1.2, the Ships-in-the-Night (SIN) mode divides existing ports into groups controlled by specific ports (see figure x) or VLANs (figure-x) |=============| |==============| |=============| | standard | | Open Flow | | Forces | | CE | | controller | | CE | |=============| |==============| |=============| || || || || || || (forwarding) |----------------------------------------------------------- | |==========| |==============| |=============| | | | standard | | Open Flow | | Forces FE | | | | FE board | | FE | | | | | |==|==|====| |====|==|======| |====|===|====| | |-----|--|--------------|--|------------------|---|---------- Standard ports OFS ports ForCES ports Figure x - Hybrid mode per port Hares, et al. Expires November, 6 2012 [Page 22] Internet-Draft OpenFlow vs. ForCES July 12, 2012 |=============| |==============| |=============| | standard | | Open Flow | | Forces | | CE | | controller | | CE | |=============| |==============| |=============| || || || || || (forwarding) || |-----------------------------------------------------------| | |==========| |==============| |=============| | | | standard | | Open Flow | | Forces FE | | | | FE board | | FE | | | | | |==|==|====| |====|==|======| |====|===|====| | | | | | | | | | | vlan1 vlan10 vlan2 vlan15 vlan3 vlan7 | | | | | | | | | | | | | | | |---|-|---|-|---------|-|---|-|-------------|-|---|-|-------| S OFS ports ForCES ports Figure x - Hybrid mode per port 3.3.4. ForCES Pre-Association Neither the ForCES protocol nor the OFS protocols [McKeown- 2008/OFS-0, OFS-1.0.0, OFS-1.1.1] specify how the CEs/controllers and FEs/forwarding switch meet. Do CEs go to the FEs meeting place and CEs pick up the FE that delights their forwarding fancy? How do they found out where Hares, et al. Expires November, 6 2012 [Page 23] Internet-Draft OpenFlow vs. ForCES July 12, 2012 eligible FEs are meeting? Do FEs have choice on which CEs select them, and if so what are the criteria? The forming of intimate relationships between CEs and FEs remains to the readers of the specification as mysterious as the pre- association stages of human group dating or clique forming. ForCES specifications specifically call this phase a pre- association phase. The ForCES architecture names the entity that coordinates the forming of associations (like a Jewish match- maker) for CEs and for FEs. The CE-Manager determines which FEs each CE should talk to. The FE-Manager determines which CEs each FE should talk to. Only when the associations between each CE and its FEs, and each FE and its CEs are complete, does the system complete pass from pre-association phase to association phase. It is assumed that some protocol interactions within the logical ForCES network entity (or entities) determine how CEs will coordinate their work. However, the IETF work specifically denoted this CE-CE coordination work as a second phase of work. OpenFlow Switch specifications ([McKeown-200][OFS-0.8][OFS- 0.9][OFS- 1.0] OFS-1.1] ignore the concept of this dynamic meeting processing. Either OFS specification missed this concept. Perhaps the OFS specifications assumed a static configuration as part of a boot process of the hybrid switch will set-up some basics. It is possible that the lack specification may come from the sponsors of the specification wanting proprietary pre-association interactions. If so, this provides an interesting line of demarcation between standards and OFS standard. In any case, this oddity from the OFS proponents leaves one to ask "What is the rest of the story on the pre-association phase?" Hares, et al. Expires November, 6 2012 [Page 24] Internet-Draft OpenFlow vs. ForCES July 12, 2012 3.3.5. Architectural requirements RFC3654 specifies 15 architectural requirements. Table 15 provides summary of this requirements and possible OFS (McKeown- 2008/OFS 0, OFS 1.0, OFS 1.1). ForCES Architectural Requirement OpenFlow Switch Architecture -------------------------------- ---------------------------- 1. CE/FEs connect via variety of Controllers/switch communicate interconnect technologies. over a secure connection [RFC3654, p. 5] [McKeown-2008][OFS-1.0][OFS-1.1] FE can use different technology not specified than CE/FE topologies. 2. "FEs MUST support a minimal [McKeown-2008][OFS-1.0][OFS-1.1] set of capabilities necessary specify a set of required for establishing network actions, instructions, Flow connectivity (e.g. interface actions, and an implied set of discovery, port up/down port functions(e.g.interface functions.) discovery, port up/down). Beyond this minimal set, the These OFS specifications ForCES architecture MUST also declare some set of NOT restrict the types of features optional. numbers of capabiliti8es that FEs may contain. [RF3654, p.5] 3. "Packets MUST be able to arrive [OFS-1.0][OFS-1.1]specifies at the NE by one FE and leave in ofp_port the OFPP_IN_PORT the NE via different FE." flag which allows the port to [RFC3654, p.5] explicitly send it back out the input port [OFS-1.0, p. 18] [OFS-1.1, p. 26] Hares, et al. Expires November, 6 2012 [Page 25] Internet-Draft OpenFlow vs. ForCES July 12, 2012 ForCES Architectural Requirement OpenFlow Switch Architecture -------------------------------- ---------------------------- 4. "A NE must support the [OFS-1.0][OFS-1.1] describes appearance of a single the devices as an individual functional device." switch, and provides (e.g. single IP TTL reduction.) the capability to reduce TTL [RFC3654,p. 5] [OFS-1.1,section 4.7, p. 12] such as push/pop of VLAN IDs and/or MPLS headers [OFS-1.1, p. 12] 4b. However, external entities 4b. No pre-association logic has (e.g. FE managers and been defined. CE managers) MAY have direct Access to individual ForCES protocol elements for providing information to transition [RFC3654, p. 5] 5."The architecture MUST provide Beyond a secure channel a way to prevent unauthorized between some "controller FORCES protocol elements entity and switch" from joining an NE." no prevention of unauthorized [RFC3654, p. 5] access has been encoded. 6. A FE must be able to [OFS-1.0][OFS-1.1] have asynchronously inform the CE "three message types: of a failure or controller-to-switch, increase/decrease in available asynchronous, resources or capabilities symmetric [OFS-1.0, p. 10] on the FE. [OFS-1.1, p. 16]. Asynchronous messages Hares, et al. Expires November, 6 2012 [Page 26] Internet-Draft OpenFlow vs. ForCES July 12, 2012 ForCES Architectural Requirement OpenFlow Switch Architecture -------------------------------- ---------------------------- Switches send a controller (CE) Messages with a received packet (packet-in), notification of a removed flow table entry (flow-removed), changes in port status (port-status), and error conditions (error) [OFS-1.0, pp-10-12)][OFS-1.1, p. 16- 17] "Thus, the FE MUST support The change in port status error monitoring and reporting" is covered,but memory status (e.g. "number of physical ports is not covered Or "memory changes"). [RFC3654,p. 5] 7. "The Architecture MUST support [McKeown-2008], [OFS-1.0] mechanisms for CE redundancy [OFS-1.1]do not specifically or CE failover. provide CE Redundancy or CE failover. 1. This includes the The OFS protocol supports ability for CE and FEs echo request/reply to determine when there message [OFS-1.0, p. 41] is a loss of association [OFS-1.1, p. 55-56]. between them, ability to restore the association The OFS-1.1 provides a and efficient state barrier message that (re)synchronization provides synchronization mechanisms. [OFS-1.1, p. 50]. Hares, et al. Expires November, 6 2012 [Page 27] Internet-Draft OpenFlow vs. ForCES July 12, 2012 ForCES Architectural Requirement OpenFlow Switch Architecture -------------------------------- ---------------------------- The OFS-1.1 protocol supports query by the controller of the switch features, capabilities, configuration, flow table configuration, flow table entries, group table entries, port configuration (pp. 36-42). FS-1.1 also provides Statistics on description of Switch (OFPST_DESC), Flow table status (OFPST_FLOW),aggregate flow statistics (OFPST_AGGREGATE), port statistics (OFPST_PORT), queue statistics (OFSPST_QUEUE), group statistics (OFSPST_GROUP), and experimenter extension (OFPST_EXPERIMENTER) (p.43-49). 7. (CE redundancy - continued). 2. This also includes the OFS-1.1 states: ability to preset the "In the case the switch loses Hares, et al. Expires November, 6 2012 [Page 28] Internet-Draft OpenFlow vs. ForCES July 12, 2012 actions an FE will take contact with the current in reaction to loss of controller, as a result of an association to its CE echo request timeout, (e.g., whether the FE TLS session timeout, or other will continue to forward disconnection, it should packets or whether it attempt to contact one or more will halt operations." backup controllers. [RFC3654, p. 6.] The ordering of the backup Controllers is not specified by the protocol." "The switch should immediately enter either "fail secure mode", or "fail standalone mode" if it loses connection to the controller, depending on the switch implementation and configuration." [OFS-1.1, p.18] In "fail secure mode", the FE behavior remains the same except FE drops "packets and messages" destined for CE. In "fail-standalone mode", FE processes all packets acts as a "legacy switch or router." [OFS-1.1, p. 18] ForCES Architectural Requirement OpenFlow Switch Architecture Hares, et al. Expires November, 6 2012 [Page 29] Internet-Draft OpenFlow vs. ForCES July 12, 2012 -------------------------------- ---------------------------- Flow entries persist over Failure in either fail secure Mode or fail standalone mode. 8. "FEs must be able to redirect [McKeown-2008][OFS-1.0][OFS-1.1] control packets addressed to allow packets to forward to their interfaces to the CE. Controller for processing. The (FE) MUST also redirect other relevant packets (E.g., such as those with Router Alert Option set) to their CE. The CEs MUST be able [OFS-1.0][OFS-1.1] Flow tables to configure the packet allow packet redirection filters redirections information/filters on the FEs with action Forward on the FEs. controller (OFS-1.0, p. 6), (OFS-1.1, p. 13). The CEs MUST also be able [OFS-1.0][OFS-1.1] allow a to create packets and have CE to FE message (packet-out) its FEs deliver them. to send frames/packets out a specific port [OFS-1.0, p. 10], [OFS-1.1,p.17] 9."Any proposed ForCES [OFS-1.0][OFS-1.1] do not architecture MUST explain consider this requirement. how that architecture supports Future OFS work may consider all the router functions as this set of features. defined in [RFC1812]." 1. Includes: IPv4 Forwarding Options 2. Should include: IPv6 forwarding options Hares, et al. Expires November, 6 2012 [Page 30] Internet-Draft OpenFlow vs. ForCES July 12, 2012 ForCES Architectural Requirement OpenFlow Switch Architecture -------------------------------- ---------------------------- 10. "In a ForCES NE, the CE(s) [OFS-1.0][OFS-1.1] do not MUST be able to learn the consider this requirement. topology by which the FEs in the NE are connected." 11. The ForCES NE architecture [McKeown-2008], [OFS-1.0] MUST be capable of supporting [OFS-1.1] do not consider (i.e. must scale to) at least scale of CE/FE communications. hundred of FEs and tens of thousands of ports. 12. "The ForCES NE architecture [McKeown-2008], [OFS-1.0], MUST allow FEs and CEs to join [OFS-1.1] do not consider and leave NEs dynamically." issues relating to active join/leaving of CEs and FEs in communication. 13. "The ForCES architecture [McKeown-2008],[OFS-1.0], MUST support multiple CEs [OFS-1.1] do not directly and FEs. However, coordination discuss how multiple CEs between CEs is out of scope will attach to FE. Or FEs of ForCES. attach to a CE. [Historical note: The restriction of CE coordination was a desired phase 2 work of the ForCES group.] Hares, et al. Expires November, 6 2012 [Page 31] Internet-Draft OpenFlow vs. ForCES July 12, 2012 ForCES Architectural Requirement OpenFlow Switch Architecture -------------------------------- ---------------------------- 14. For pre-association phase [McKeown-2008], [OFS-1.0], set-up, monitoring, [OFS-1.1] do not consider configuration issues, it MAY issues relating setting up be useful to use standard the links between CEs and management mechanisms for FEs. Some "magic" occurs CEs and FEs. and the CE is talking to a particular FE. The ForCES architecture and requirements do not preclude this. [OFS-1.0][OFS-1.1] give In general, for no discussion on other post-association phase, management process (SNMP) most management tasks SHOULD outside the [OFC-1.0] be done through interactions using netconf and beep with the CE. In certain conditions (e.g., CE/FE disconnection), it may be useful to allow management tools (E.g., SNMP) to diagnose and repair problems. The following guidelines MUST be observed: 1. The ability for a management tool (e.g., SNMP) to be used to read (but not change) the state of FE SHOULD NOT be precluded. 2. IT MUST NOT be possible Hares, et al. Expires November, 6 2012 [Page 32] Internet-Draft OpenFlow vs. ForCES July 12, 2012 for management tools (e.g., SNMP, etc) to change the state of an FE in a manner that affects overall NE behavior without the CE being notified. 3.3.6. ForCES versus OpenFlow - A Central Controller ForCES and OpenFlow seek to split the control plane and the forwarding engine. Both protocols using a secure connection can be used to interact with a central controller. ForCES has spent more time determining how CEs and FEs might find one or more central controllers. OpenFlow Specifications are just beginning to rediscover the need for this work. Both Forces an OpenFlow can provide the ability of a logically centralized controller to: o Collect the network view and make decisions according to control logics (or applications); o Interact with forwarding hardware (FE) to install forwarding policy and state, o Provide open APIs to users to add new features. ForCES has considered security issues (such as Denial of Service (DOS)) and the mechanisms for grouping CEs with an FE, or FEs with a CE, Forwarding Models, and Forwarding Libraries. OFS specifications have focused on defining one simple functionality that can be implemented in specific networks. For example, many discussions point to code deployed in Google. 3.4. Difference in Forwarding Model ForCES and OFS pipeline processing of frames/packets include the basic steps of matching framing, processing frame, and outputting/dropping frame. The processing of a frame occurs in a pipeline of processes where initially processing adds the metadata and actions that subsequent processing will use to create the final packet that will be sent via output port or dropped. Hares, et al. Expires November, 6 2012 [Page 33] Internet-Draft OpenFlow vs. ForCES July 12, 2012 Key ingredients of a good pipeline process are: 1) a deterministic logic based that doesn't loop, 2) handles both unicast and multicast traffic, 3) Flexible matching that can growth with new features, 4) Metadata to allow passing of results to subsequent stages, 5) Logic that allows some stages to be skipped, and 6) Allows for no match (Table Miss). 3.4.1. Looping In ForCES, [RFC5812] defines the FE (Forwarding Element) model based on an abstraction of Logical Functional Blocks (LFBs). In this model, each FE is composed of multiple LFBs that are interconnected in a directed graph, which is represented by the LFB topology model. The directed graph model prevents the recycle of processing in a loop. Each LFB defines a set of processing on handling frames/packets. For example, typical LFBs include IPv4/IPv6 Longest Prefix Matching, etc. XML is used to describe LFB model formally. In [OFS-0.8][OFS-0.9][OFS-1.0] the forwarding model has been static defining specific functions for early experimentation of the switch. Loops have been prevent [OFS-1.0] defines the Flow tables identified by a sequential number [0,1,2_n]. Processing loops are prevent by defining that a flow table can only transfer to a higher flow table. [OFS-1.1] provides a Group table to augment the Flow Table logic described above. If a flow matches, instructions in the flow table may direct the packet toward specific table (group table or flow table). This jumping provides skips in sequential process unlike [OFS-1.0]. In addition, the "goto" action allows skips between tables. Hares, et al. Expires November, 6 2012 [Page 34] Internet-Draft OpenFlow vs. ForCES July 12, 2012 3.4.2. Handling unicast and multicast [OFS-0.9][OFS-1.0] do not provide an easy to provide cloning for multicast. The group table in [OFS-1.1] provides the necessary cloning for multiple outputs of a single packet. A ForCES IPv4MultiLPB and IPv6MultiLPB could be defined beyond today's ForCES standards. These LFBs would use an LPM to match the multicast address, and generate a list of "L3PortID" metadata to identify a set of ports the cloned packet could be sent out. This metadata could be passed to the EthernetEncap LFB. 3.4.3. Flexible matching All OFS specifications ([OFS-0.8][OFS-0.9][OFS- 1.0][OFS1.1][OFS1.3-pre] seeks to match the header data against the flow table's match field. Ranging from a 10-29 possible matches in the header and metadata, the OFS provides flexible matching within the data packet (Ethernet, MPLS, IP, TCP, UDP). [Heleplidis-Forces-LFB] shows the matching capability in OFS-1.1 can be implemented in ForCES LFBs. 3.4.4. Metadata to allow passing of results to subsequent stages, Both ForCES and OFS ([OFS-1.1][OFS-1.3-pre]) allow metadata to be passed to subsequent spaces. 3.4.5. Optionally skipping logic [OFS-1.1] provides a Group table to augment the Flow Table logic described above. If a flow matches, instructions in the flow table may direct the packet toward specific table (group table or flow table). This jumping provides skips in sequential process unlike [OFS-1.0]. The Group Table concepts provides the ability to group flows for execution, single out a single flow for additional processing, and use port liveness mechanisms for fast- failover. [OFS-1.1,p, 7]. The ForCES directed graph model can also allow the skips in processing by having multiple exits from. 3.4.6. Table Miss A frame may not match any table in the forwarding pipeline. Hares, et al. Expires November, 6 2012 [Page 35] Internet-Draft OpenFlow vs. ForCES July 12, 2012 [OFS-0.9] states "if no matching entry can be found for a packet, the packet will be sent to the controller over the secure channel"[p. 8]. Experience has taught the OpenFlow community this can be problematic. [OFS-1.0.0-errta] states the following exceptions: "Sending the packet to the controller on table-miss may overload the switch or the controller, and some of those packets may have to be dropped, and this may be an issue in some deployments" [p., 3]. The OFS-1.0.0-errta suggests that the vendor extension may allow the packet to be dropped or forwarded via pipeline. However, due to many application use of table-miss to do topology discovery or watch traffic - this feature is continued. ForCES does not define a global Table-Miss, but allows the LFB model to define these issues. 3.5. Difference in Logical Forwarding Block Libraries The Open-Flow group is beginning to consider flexible description of the next OFS switches using a modeling language. No modeling language has been approved as yet. The Force LFB Library [ForCES-LFB-Lib] has been defined and implemented. [Heleplidis-Forces-LFB] shows the modeling language can support OFS-1.1 definitions. 3.6. Difference in Protocol Interface The OFS protocol and the ForCES protocol both use: . Secure transport protocols over which they operate (3.6.1) . Messages to establish the Controller/CE - FE connections, . Messages to Loading of forwarding logic (3.6.3) . Messages to Configuration the box (3.6.4), . Error handling messages (3.6.5), . Liveness protocols (3.6.6), and Hares, et al. Expires November, 6 2012 [Page 36] Internet-Draft OpenFlow vs. ForCES July 12, 2012 . Sending packets for processing to/from controller (3.6.8). 3.6.1 Secure Transport ForCES defines two layers of protocols: ForCES Protocol Layer (ForCES PL) and ForCES Protocol Transport Mapping Layer (ForCES TML). ForCES PL defines Protocol between FEs and CEs (Fp Reference Point). ForCES Protocol Transport Mapping Layer (ForCES TML) is defined to transport the PL messages. It is expected that more than one TML will be standardized and interoperability is guaranteed as long as both endpoints support the same TML. [RFC5811] has defined a SCTP-based TML for ForCES. OpenFlow defines the protocol between controller and OpenFlow switches, i.e. OpenFlow protocol. OFS-1.1 states that the data channel is "usually encrypted using TLS, but may be run over TCP"[OFS-1.1.0,p. 16] 3.6.2 Types of Messages As Table-x shows, ForCES and OFS protocol are remarkably similar. Many OFS authors indicate the influence of the ForCES protocol on the OFS work. Due to the IETF review, ForCES protocol's top level is carefully designed with orthogonal features of association setup, association teardown, config, query, events, packet redirect, and heartbeat. The OFS protocols [OFS-0.8][OFS-0.9][OFS-1.0][OFS-1.1] provide similar features, but have some overlapping functions. Similarly, the OFS protocol has o Hello (initial association), o Reading of switch Features (read/response), o config of switch and flow pipeline's via Flow tables [OFPT_FLOW_MOD), group tables [OFPT_GROUP_MOD], ports [OFPT_PORT_MOD]. o Flow Removed [OFPT_FLOW_REMOVED] - Flow entry is removed due to either an idle timer or a hard timeout. o Query of statistics (OFPT_STATS_REQUEST, OFPT_STATS_RESPONSE), Hares, et al. Expires November, 6 2012 [Page 37] Internet-Draft OpenFlow vs. ForCES July 12, 2012 o Packet-OUT- redirect of packet from controller out a port, o PACKET-IN - redirect packet inbound port to controller, o Echo request/reply - heartbeat from OFS switches. In addition, the OFS protocol has a Barrier Request/reply message that allows the controller to synchronize message processing. This general (but lose) definitional has allowed experimentation with OFS switches. Due to IETF review, the similar ForCES protocol has a clear orthogonal set of actions described in terms of execution and transaction models. The CE can set execution flags on sets on transactions (groups of functions). The execution of transactions can be: execute-all-or-none, continue-execute-on- failure, and execute-until failure. A transaction set is must me the ACDity test (atomicity, consistency, isolation, and durability)[RFC5810,section 4.3.1.2]. Transaction sets have an start, middle, and end. The transaction can also signal an abort. The notification messages for Error and Port status are uniquely specified in the OFS as a notification. In ForCES this is included with the general notification category. Table-x ForCES vs. OFS messages Message: ForCES OFS-0.9 OFS-1.0 OFS-1.1 ================ ====================================== Associate ----------Hello---------- CE/FE (Req/Rsp) X X X X Association -----Feature Request/Response--- Stop (Req/Rsp) X X X X Query/ X ----Feature or STATS(Req/Rsp)--- Query Response X X X X Config [Switch] ---Config (non-flow table)----- (Req/Rsp) X X X X Hares, et al. Expires November, 6 2012 [Page 38] Internet-Draft OpenFlow vs. ForCES July 12, 2012 Config (Req/Rsp) X ------ FLOW_MOD------- X X X X Table-x ForCES vs. OFS messages Message: ForCES OFS-0.9 OFS-1.0 OFS-1.1 ================ ====================================== Heartbeat(Forces) X ---Echo-Request/Response--- /Echo-Request(OFS) X X X X Redirect ---Packet_in & Packet-Out) X X X Execution flags X ----Barrier-Set-Queue X X Notification x X X 3.6.3 Loading of forwarding logic ForCES and OFS both use TLVS to add, modify, and delete the flow entry. In addition, ForCES has a concept of "commit" to a set of changes to allow multiple stages of set. OFS has the concept of modifying or deleting only strictly matching flows (OFPFC_MODIFY_STRICT, OFPFC_DELETE_STRICT). This is different that the OFS default of modifying all flows with that match (with wildcard). 3.6.4 Configuration ForCES defines changing configuration of the switching Forwarding pipeline within the protocol. OF-Config-1.0 has provided a protocol to use an OpenFlow Configuration Point (logical mode) that can configure one or more OFS via the OpenFlow configuration protocol. The configuration protocol runs on top of TLS or TCP. The configuration protocol sets the following information: o Failure standby mode (fail secure or fail standalone) Hares, et al. Expires November, 6 2012 [Page 39] Internet-Draft OpenFlow vs. ForCES July 12, 2012 o Encryption mode (TLS or not), o Queue configurations (min-rate, max-rate, experimenter), o Ports (speed, no-receive, no forward, no packet-in, link- down, blocked, life) and optionally (duplex-mode, copper- medium, fiber-medium auto-negotiation, pause, asymmetric- pause), o Data path id of switch. 3.6.5 Error handling and sanity checking The error handling indicates errors that occur within the protocol. The Error handling includes message form and action failure. For OFS the action failure includes all interactions such as: hello failure, bad request, bad flow action, bad flow instruction, bad match, cannot modify entry in flow table, cannot modify entry in group table, cannot modify port. Due to IETF review, the ForCES errors and notifications are define to contain all cases within the protocol. The error processing contains sanity checking. 3.6.6 Failure of CE/FE connection Heart beat messages in both ForCES and OFS insure "liveness" of the CE/FE connection. The ForCES heartbeats are traffic sensitive, and are only sent if no traffic has occurred. OFS predefines that switches should enter the following based on losing connections with controller: "Fail secure mode" or "fail standalone mode" [OFS-1.1.0,section 5.3]. In Fail secure mode, forwarding continues as previous with the only change that no packets can be uploaded to the processor. In fail standalone mode, the OSF switch drops into the Ethernet legacy mode [OFPP_NORMAL]. If the ForCES protocol is supporting the high-availability function, the begins the engage the high-availability statement machine. OpenFlow specifications have not yet described how High- availability will work in Open-Flow. Hares, et al. Expires November, 6 2012 [Page 40] Internet-Draft OpenFlow vs. ForCES July 12, 2012 4. Use of ForCES and OFS in Applications ForCES and OFS [OFS-1.0][OFS-1.1] have been encoding in a variety of applications. These application include: . Firewalls, . Loads balancers, . Switches . High-availability routers, . Wireless devices. . Table-x ForCES vs. OFS messages 5. The use of ForCES or OpenFlow in S(D)N or CSO/SOP This section will contain a summary of the common capabilities of ForCES and OpenFLow in environments of centralized controllers, distributed controllers, and hybrid (centralized/distributed control) suggested by open flow. 5.1 - Centralized controller logic ForCES and OFS have been designed for centralized controller logic. ForCES has considered the pre-association and association phase of the CE-FE relationship with all the timing issues. The execution and transaction model provide a strongly reviewed model to provide roll-forward and roll-back of transactions. The high- availability drafts for ForCES provide a clear case on how to keep high-availability of forwarding and CE processing while distributing the flows. ForCES has a clear body of work developed over years of implementation experience. OFS specifications do not deal with how controllers find FEs. However, numerous companies are developing centralized controllers. The standardization efforts for Hybrid (OFS-1.2) and the next generation OFS switch (OFS-1.3) indicate an effort to capture this growing body of wisdom. 5.2 - Distributed controller logic Hares, et al. Expires November, 6 2012 [Page 41] Internet-Draft OpenFlow vs. ForCES July 12, 2012 ForCES was built to distribute the controller logic to automonomous network elements that operate either as ForCES controlled or as integrated hybrid controller. OFS has created distributed logic per switch, but considers grouping of these switches outside the OFS specifications. The Hybrid [OFS-1.2] provides use cases for Ships-in-the-Night and integrated. The Ships-in-the-Night provide per port allocation to either OFS or standard processing. The Integrated seeks to run both on a set of ports. 5.3 - Hybrid controllers ForCES was built for the hybrid environment where routing and switching protocol. OFS is now entering the processing of standardizing for hybrid controllers [OFS-1.2]. 6. Security Considerations No security considerations. This is an informational comparison used to inform clarify ForCES work. 7. IANA Considerations No IANA considerations. 8. Conclusions Both ForCES and OpenFlow follow the basic idea of separations of forwarding plane and control plane in network elements. Both are capable of operating for centralized control, distributed control, and hybrid control. [OFS-1.1] Flow Table Logic with the instructions and Group Tables is the major difference between the ForCES RFCs. As this paper has shown, the full ramifications of this difference need to be considered in terms of differences in capability of implementation. The author welcomes any additional implementation experience. Hares, et al. Expires November, 6 2012 [Page 42] Internet-Draft OpenFlow vs. ForCES July 12, 2012 [OFS-1.0][OFS-1.1] lacks a forwarding model, a standardized LFB library and the concepts of FE-CE associations (FE-Manger, CE- Manager, pre/post association phase). It appears the OpenFlow work is starting to invent the equivalent of existing ForCES work as OpenFlow work. The guide of this reinventing seems to be the Google code snippets passed to the OpenFlow Forum as examples of "running code" to provide rough consensus. 9. References 9.1. Normative References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. [RFC5810] Doria, A., Hadi Salim, J., Haas, R., Khosravi, H., Wang, W., Dong, L., Gopal, R., and J. Halpern, "Forwarding and Control Element Separation (ForCES) Protocol Specification", RFC 5810, March 2010. [RFC5812] Halpern, J. and J. Hadi Salim, "Forwarding and Control Element Separation (ForCES) Forwarding Element Model", RFC 5812, March 2010. [RFC5811] Hadi Salim, J. and K. Ogawa, "SCTP-Based Transport Mapping Layer (TML) for the Forwarding and Control Element Separation (ForCES) Protocol", RFC 5811, March 2010 9.2. Informative References [McKeown2008] McKeown, N., Anderson, T., Balakrishnan, H., et al, "Openflow: enabling innovation in campus networks", ACM SIGCOMM Computer Communication Review. 2008, 38(2):69- 74. [OFS-1.0.0] OpenFlow Switch Specification - Version 1.0.0 (Wire Protocol 0x01), December 31, 2009. [OFS-1.0.1-rc3] OpenFlow Switch Errata - Version 1.0.1-r3, June 12, 2012. Hares, et al. Expires November, 6 2012 [Page 43] Internet-Draft OpenFlow vs. ForCES July 12, 2012 [OFS-1.1.0] OpenFlow Switch Specification Version 1.1.0 (Wire Protocol 0x02). February 2011. (http://www.openflow.org/documents/openflow-spec- v1.1.0.pdf) [OpenFlow-1.2] Open Flow 1.2 - Hybrid Switch (Terminology, Use cases, etc), April-May notes, work-in-progress. [OFS-1.3-rc4[ OpenFlow Switch Specification 1.3 (version 1.3-rc4) (Wire Protocol 0x04) April4, 2012. [OpenFlow members only] [OFS-1.1.0] OpenFlow Switch Specification Version 1.1.0 (Wire Protocol 0x02). February 2011. (http://www.openflow.org/documents/openflow-spec- v1.1.0.pdf) [OFC-1.0] OpenFlow Configuration and Management Protocol OF- CONFIG 1.0 (January 15, 2012). [RFC3654] Khosravi, H. and T. Anderson, "Requirements for Separation of IP Control and Forwarding", RFC 3654, November 2003. [RFC3746] Yang, L., Dantu, R., Anderson, T., and R. Gopal, "Forwarding and Control Element Separation (ForCES) Framework", RFC 3746, April 2004. [LFB-Lib] Wang W., Haleplidis E., Ogawa K., Li C., J. Halpern, "ForCES Logical Function Block (LFB) Library", draft- ietf-forces-lfb-lib-06, Work in Progress. 10. Acknowledgments The author would like to thank Tina Tsou, Zhiliang Wang,Jing Huang, Xingang Shi, and Xia Yin. Their earlier draft comparing the FORCES and ONF Technology inspired this draft providing an opposing viewpoint. It is said that "iron sharpens iron" so that in our debates we sharpen our understandings. Hares, et al. Expires November, 6 2012 [Page 44] Internet-Draft OpenFlow vs. ForCES July 12, 2012 The author also acknowledges Edward Crabbe's succinct comments which also inspired the in-depth comparison found in this draft. I only hope that this "wordy" draft proves worth of his pithy and concise insights. Hares, et al. Expires November, 6 2012 [Page 45] Internet-Draft OpenFlow vs. ForCES July 12, 2012 Authors' Addresses Susan Hares Huawei Technologies (USA) 2330 Central Expressway Santa Clara, CA 95050 USA Email: Susan Hares Susan.Hares@huawei.com Hares, et al. Expires November, 6 2012 [Page 46]