idnits 2.17.1 draft-irtf-nfvrg-nfv-policy-arch-04.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document doesn't use any RFC 2119 keywords, yet seems to have RFC 2119 boilerplate text. -- The document date (September 7, 2016) is 2787 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- No issues found here. Summary: 0 errors (**), 0 flaws (~~), 2 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 NFV Research Group N. Figueira 3 Internet-Draft Brocade 4 Intended status: Informational R. Krishnan 5 Expires: March 11, 2017 Dell 6 D. Lopez 7 Telefonica 8 S. Wright 9 AT&T 10 D. Krishnaswamy 11 IBM 12 September 7, 2016 14 Policy Architecture and Framework for NFV Infrastructures 15 draft-irtf-nfvrg-nfv-policy-arch-04 17 Abstract 19 A policy architecture and framework is discussed to support NFV 20 environments, where policies are used to enforce business rules and 21 to specify resource constraints in a number of subsystems. This 22 document approaches the policy framework and architecture from the 23 perspective of overall orchestration requirements for services 24 involving multiple subsystems. The framework extends beyond common 25 orchestration constraints across compute, network, and storage 26 subsystems to include energy conservation. This document also 27 analyses policy scope, global versus local policies, static versus 28 dynamic versus autonomic policies, policy actions and translations, 29 policy conflict detection and resolution, interactions among policies 30 engines, and a hierarchical policy architecture/framework to address 31 the demanding and growing requirements of NFV environments. These 32 findings may also be applicable to cloud infrastructures in general. 34 Status of this Memo 36 This Internet-Draft is submitted to IETF in full conformance with the 37 provisions of BCP 78 and BCP 79. 39 Internet-Drafts are working documents of the Internet Engineering 40 Task Force (IETF), its areas, and its working groups. Note that other 41 groups may also distribute working documents as Internet-Drafts. 43 Internet-Drafts are draft documents valid for a maximum of six months 44 and may be updated, replaced, or obsoleted by other documents at any 45 time. It is inappropriate to use Internet-Drafts as reference 46 material or to cite them other than as "work in progress." 47 The list of current Internet-Drafts can be accessed at 48 http://www.ietf.org/ietf/1id-abstracts.txt. 50 The list of Internet-Draft Shadow Directories can be accessed at 51 http://www.ietf.org/shadow.html. 53 This Internet-Draft will expire in May 2015. 55 Copyright Notice 57 Copyright (c) 2016 IETF Trust and the persons identified as the 58 document authors. All rights reserved. 60 This document is subject to BCP 78 and the IETF Trust's Legal 61 Provisions Relating to IETF Documents 62 (http://trustee.ietf.org/license-info) in effect on the date of 63 publication of this document. Please review these documents 64 carefully, as they describe your rights and restrictions with respect 65 to this document. 67 Conventions used in this document 69 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 70 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 71 document are to be interpreted as described in RFC 2119. 73 Table of Contents 75 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 76 2. Policy Intent Statement versus Subsystem Actions and 77 Configurations . . . . . . . . . . . . . . . . . . . . . . . . 4 78 3. Global vs Local Policies . . . . . . . . . . . . . . . . . . . 5 79 4. Static vs Dynamic vs Autonomic Policies . . . . . . . . . . . . 7 80 5. Hierarchical Policy Framework . . . . . . . . . . . . . . . . . 7 81 6. Policy Conflicts and Resolution . . . . . . . . . . . . . . . . 9 82 6.1. Soft vs Hard Policy Constraints . . . . . . . . . . . . . . 11 83 7. Policy Pub/Sub Bus . . . . . . . . . . . . . . . . . . . . . . 12 84 7.1 Pub/Sub Bus Name Space . . . . . . . . . . . . . . . . . . . 16 85 8. Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 86 8.1 Establishment of a Multipoint Ethernet Service . . . . . . . 17 87 8.2 Policy-Based NFV Placement and Scheduling . . . . . . . . . 20 88 8.2.1 Policy Engine Role in NFV Placement and Scheduling . . . 20 89 8.2.2 Policy-based NFV Placement and Scheduling with 90 OpenStack . . . . . . . . . . . . . . . . . . . . . . . 21 91 9. Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 92 10. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 25 93 11. Security Considerations . . . . . . . . . . . . . . . . . . . 25 94 12. Contributors . . . . . . . . . . . . . . . . . . . . . . . . . 26 95 13. References . . . . . . . . . . . . . . . . . . . . . . . . . . 26 96 13.1. Normative References . . . . . . . . . . . . . . . . . . . 26 97 13.2. Informative References . . . . . . . . . . . . . . . . . . 26 98 Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . 28 99 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 28 101 1. Introduction 103 This document discusses the policy architecture and framework to 104 support Network Function Virtualization (NFV) [11] infrastructures. 105 In these environments, policies are used to enforce business rules 106 and to specify resource constraints, e.g., energy constraints, in a 107 number of subsystems, e.g., compute, storage, network, and etc., and 108 across subsystems. These subsystems correspond to the different 109 "infrastructure domains" identified by the NFV ISG Infrastructure 110 Working Group [8][10][7]. 112 The current work in the area of policy for NFV is mostly considered 113 in the framework of general cloud services, and typically focused on 114 individual subsystems and addressing very specific use cases or 115 environments. For example, [11] addresses network subsystem policy 116 for network virtualization, [19] and [20] are open source projects in 117 the area of network policy as part of the OpenDaylight [21] software 118 defined networking (SDN) controller framework, [18] specifies an 119 information model for network policy, [13] focuses on placement and 120 migration policies for distributed virtual computing, [24] is an open 121 source project in OpenStack [22] to address policy for general cloud 122 environments. 124 This document approaches policy, policy framework, and policy 125 architecture for NFV services from the perspective of overall 126 orchestration requirements for services involving multiple 127 subsystems, and can be applied to the general case of any cloud-based 128 service. The analysis extends beyond common orchestration constraints 129 across compute, network, and storage subsystems to also include 130 energy conservation constraints applicable to NFV and other 131 environments. The analysis in this document also extends beyond a 132 single virtual Point of Presence (vPoP) or administrative domain to 133 include multiple data centers and networks forming hierarchical 134 domain architectures [12]. The focus of this document is not general 135 policy theory, which has already been intensively studied and 136 documented on numerous publications over the past 10 to 15 years (see 137 [18], [27], [17], [28], and [3] to name a few). This document's 138 purpose is to discuss and document a policy architecture that uses 139 known policy concepts and theories to address the unique requirements 140 of NFV services including multiple vPoPs and networks forming 141 hierarchical domain architectures [12]. 143 With the above goals, this document analyses policy scope, global 144 versus local policies, static versus dynamic versus autonomic 145 policies, policy actions and translations of actions, policy conflict 146 detection and resolution (which can be relevant to resource 147 management in service chains [16]), the interactions among policies 148 engines from the different vPoPs and network subsystems, and a 149 hierarchical policy architecture/framework to address the demanding 150 and growing requirements of NFV environments. These findings may also 151 be applicable to cloud infrastructures in general. 153 2. Policy Intent Statement versus Subsystem Actions and Configurations 155 Policies define which states of deployment are in compliance with the 156 policy, and, by logic negation, which ones are not. The compliance 157 statement in a policy may define specific actions, e.g., "a given 158 customer is [not allowed to deploy VNF X]", where VNF refers to a 159 Virtual Network Function, or quasi-specific actions, e.g., "a given 160 customer [must be given platinum treatment]." Quasi-specific actions 161 differ from the specific ones in that the former requires an 162 additional level of translation or interpretation, which will depend 163 on the subsystems where the policy is being evaluated, while the 164 latter does not require further translation or interpretation. 166 In the previous examples, "VNF X" defines a specific VNF type, i.e., 167 "X" in this case, while "platinum treatment" could be translated to 168 an appropriate resource type depending on the subsystem. For example, 169 in the compute subsystem this could be translated to servers of a 170 defined minimum performance specification, while in the network 171 subsystem this could be translated to a specific Quality of Service 172 (QoS) level treatment. 174 The actions defined in a policy may be translated to subsystem 175 configurations. For example, when "platinum treatment" is translated 176 to a specific QoS level treatment in a networking subsystem, one of 177 the outcomes (there can be multiple ones) of the policy could be the 178 configuration of network elements (physical or virtual) to mark that 179 customer's traffic to a certain DSCP (DiffServ Code Point) level 180 (Figure 1). Some may refer to the QoS configuration above as a policy 181 in itself, e.g., [27], [28], [4], and [17]. In this document, such 182 domain configurations are called policy enforcement technologies to 183 set them apart from the actual policy intent, e.g., "a given customer 184 must be given platinum treatment" as in the above example. 186 Describing intent using a high-level policy language instead of 187 directly describing configuration details allows for the decoupling 188 of the desired intent from the actual configurations, which are 189 subsystem dependent, as shown in the previous example (Figure 1). The 190 translation of a policy into appropriate subsystem configurations 191 requires additional information that is usually subsystem and 192 technology dependent. Therefore, policies should not be written in 193 terms of policy enforcement technologies. Policies should be 194 translated at the subsystems using the appropriate policy provides a 195 few examples where the policy "a given customer must be given 196 platinum treatment" is translated to appropriate configurations at 197 the respective subsystems. 199 The above may sound like a discussion about "declarative" versus 200 "imperative" policies. We are actually postulating that "imperative 201 policy" is just a derived subsystem configuration using an 202 appropriate policy enforcement technology to support an actually 203 intended policy. 205 +----------------------------------------------------------------+ 206 | Policy: "a given customer must be given Platinum treatment" | 207 +----------------------------------------------------------------+ 208 ^ ^ ^ ^ 209 | | | | 210 V V V V 211 +-------------+ +-------------+ +-------------+ +-------------+ 212 |Compute | |Network | |Storage | |Whatever | 213 |Subsystem | |Subsystem | |Subsystem | |Subsystem | 214 | | | | | | | | 215 |Policy | |Policy | |Policy | |Policy | 216 |translation: | |translation: | |translation: | |translation: | 217 | | | | | | | | 218 |Install | |Give customer| |Give customer| | ... | 219 |customer VMs | |the best QoS,| |the fastest | | | 220 |on servers | |which | |SSD storage. | | | 221 |with 3GHz | |translates | | | | | 222 |16-core Xeon | |here to set | | | | | 223 |processors, | |DSCP to xx, | | | | | 224 |and etc. | |and etc. | | | | | 225 +-------------+ +-------------+ +-------------+ +-------------+ 227 Figure 1: Example of Subsystem Translations of Policy Actions 229 3. Global vs Local Policies 231 Some policies may be subsystem specific in scope, while others may 232 have broader scope and interact with multiple subsystems. For 233 example, a policy constraining certain customer types (or specific 234 customers) to only use certain server types for VNF or Virtual 235 Machine (VM) deployment would be within the scope of the compute 236 subsystem. A policy dictating that a given customer type (or specific 237 customers) must be given "platinum treatment" could have different 238 implications on different subsystems. As shown in Figure 1, that 239 "platinum treatment" could be translated to servers of a given 240 performance specification in a compute subsystem and storage of a 241 given performance specification in a storage subsystem. 243 Policies with broader scope, or global policies, would be defined 244 outside affected subsystems and enforced by a global policy engine 245 (Figure 2), while subsystem-specific policies or local policies, 246 would be defined and enforced at the local policy engines of the 247 respective subsystems. 249 Examples of sub-system policies can include thresholds for 250 utilization of sub-system resources, affinity/anti-affinity 251 constraints with regard to utilization or mapping of sub-system 252 resources for specific tasks, network services, or workloads, or 253 monitoring constraints regarding under-utilization or over- 254 utilization of sub-system resources. 256 +----------------------------------------------------------------+ 257 | +----------------------------------------------+ | 258 | | Global Policy Engine | | 259 | +----------------------------------------------+ | 260 | | 261 | +----------------------------------------------+ | 262 | | Global Policies | | 263 | +----------------------------------------------+ | 264 +----------------------------------------------------------------+ 265 ^ ^ ^ ^ 266 | | | | 267 V V V V 268 +-------------+ +-------------+ +-------------+ +-------------+ 269 |Compute | |Network | |Storage | |Whatever | 270 |Subsystem | |Subsystem | |Subsystem | |Subsystem | 271 | | | | | | | | 272 |Local Policy | |Local Policy | |Local Policy | |Local Policy | 273 |Engine | |Engine | |Engine | |Engine | 274 | | | | | | | | 275 |Local | |Local | |Local | |Local | 276 |Policies: | |Policies | |Policies | |Policies | 277 | P0, P1, | | P0, P1, | | P0, P1, | | P0, P1, | 278 | | | | | | | | 279 +-------------+ +-------------+ +-------------+ +-------------+ 281 Figure 2: Global versus Local Policy Engines 283 4. Static vs Dynamic vs Autonomic Policies 285 Policies can be defined based on a diverse set of constraints and 286 sources, e.g., an operator's energy cost reduction policy based on 287 the time-varying energy rates imposed by a utility company supplying 288 energy to a given region or an operator's "green" policy based on the 289 operator's green operational requirements, which may drive a 290 reduction in energy consumption despite of potentially increased 291 energy costs. 293 While "static" policies can be imposed based on past learned behavior 294 in the systems, "dynamic" policies can be define that would override 295 "static" policies due to dynamically varying constraints. For 296 example, if a system needs to provided significantly additional 297 scaling of users in a given geographic area due to a sporting or a 298 concert event at that location, then a dynamic policy can be 299 superimposed to temporarily override a static policy to support 300 additional users at that location for a certain period of time. 301 Alternatively, if energy costs significantly rise on a particular 302 day, then an energy cost threshold could be dynamically raised to 303 avoid policy violation on that day. 305 Support for autonomic policies may also be required, such as an auto- 306 scaling policy that allows a resource (compute/storage/network/energy 307 resource) to be scaled up or down as needed. For example, a policy 308 specifying that a VNF can be scaled up or down to accommodate traffic 309 needs. 311 5. Hierarchical Policy Framework 313 So far, we have referenced compute, network, and storage as 314 subsystems examples. However, the following subsystems may also 315 support policy engines and subsystem specific policies: 317 - SDN Controllers, e.g., OpenDaylight [21]. 318 - OpenStack [22] components such as, Neutron, Cinder, Nova, and 319 etc. 320 - Directories, e.g., LDAP, ActiveDirectory, and etc. 321 - Applications in general, e.g., standalone or on top of 322 OpenDaylight or OpenStack. 323 - Physical and virtual network elements, e.g., routers, firewalls, 324 application delivery controllers (ADCs), and etc. 325 - Energy subsystems, e.g., OpenStack Neat [25]. 327 Therefore, a policy framework may involve a multitude of subsystems. 328 Subsystems may include other lower level subsystems, e.g., Neutron 329 [26] would be a lower level subsystem in the OpenStack subsystem. In 330 other words, the policy framework is hierarchical in nature, where 331 the policy engine of a subsystem may be viewed as a higher level 332 policy engine by lower level subsystems. In fact, the global policy 333 engine in Figure 2 could be the policy engine of a Data Center 334 subsystem and multiple Data Center subsystems could be grouped in a 335 region containing a region global policy engine. In addition, one 336 could define regions inside regions, hierarchically, as shown in 337 Figure 3. 339 Metro and wide-area network (WAN) used to interconnect data centers 340 would also be independent subsystems with their own policy engines. 342 To higher level domain 343 ^ 344 Region 1 | 345 Domain V 346 +-------------------+ +-------------------+ 347 | +---------------+ | | +---------------+ | 348 | |Region 1 Global| |<------>| |WAN 1 Global | | 349 | |Policy Engine | | | |Policy Engine | | 350 | +---------------+ | | +---------------+ | 351 | | | | 352 | +---------------+ | | +---------------+ | 353 | |Whatever | | | |Whatever | | 354 | |Subsystems | | | |Subsystems | | 355 | | | | | | | | 356 | |Local Policy | | | |Local Policy | | 357 | |Engines | | | |Engines | | 358 | +---------------+ | | +---------------+ | 359 +-------------------+ +-------------------+ 360 ^ ^ 361 | | 362 | +-------------------------+ 363 | | 364 DC 1 Domain V DC N Domain V 365 +-------------------+ +-------------------+ 366 | +---------------+ | | +---------------+ | 367 | |DC 1 Global | | | |DC N Global | | 368 | |Policy Engine | | | |Policy Engine | | 369 | +---------------+ | | +---------------+ | 370 | | | | 371 | +---------------+ | | +---------------+ | 372 | |Whatever | | | |Whatever | | 373 | |Subsystems | | | |Subsystems | | 374 | | | | | | | | 375 | |Local Policy | | | |Local Policy | | 376 | |Engines | | | |Engines | | 377 | +---------------+ | | +---------------+ | 378 +-------------------+ +-------------------+ 380 Figure 3: A Hierarchical Policy Framework 382 6. Policy Conflicts and Resolution 384 Policies should be stored in databases accessible by the policy 385 engines. For example, the local policies defined for the Compute 386 subsystem in Figure 2 would be stored in a database accessible by the 387 local policy engine in that subsystem. 389 As a new policy is added to a subsystem, the subsystem's policy 390 engine should perform conflict checks. For example, a simple conflict 391 would be created if a new policy states that "customer A must not be 392 allowed to use VNF X", while an already existing policy states that 393 "customer A is allowed to use VNF X". In this case, the conflict 394 should be detected and an appropriate policy conflict resolution 395 mechanism should be initiated. 397 The nature of the policy conflict resolution mechanism would depend 398 on how the new policy is being entered into the database. If an 399 administrator is manually attempting to enter that policy, the 400 conflict resolution could entail a warning message and rejection of 401 the new policy. The administrator would then decide whether or not to 402 replace the existing policy with the new one. 404 When policies are batched for later inclusion in the database, the 405 administrator should run a preemptive conflict resolution check on 406 those policies before committing to include them in the database at a 407 future time. However, running a preemptive conflict resolution check 408 does not guarantee that there will be no conflicts at the time the 409 batched policies are actually included in the database, since other 410 policies could have been added in the interim that cause conflicts 411 with those batched policies. 413 To avoid conflicts between batched policies waiting for later 414 inclusion in the database and new policies being immediately added to 415 the database, one could run a preemptive conflict resolution check 416 against database policies and also batched policies every time new 417 policies are added to the database. However, this may not be 418 sufficient in case of separate administrative domains. A region 419 administration could define batched polices to be pushed to the 420 Compute subsystem of a Data Center at a later time. However, the 421 Compute subsystem may be a separate administrative domain from that 422 of the region administrative domain. In this case, the Compute 423 subsystem may not be allowed to run preemptive policy conflict checks 424 against the batched policies defined at the region administrative 425 domain. Thus, there is a need for a reactive policy conflict 426 resolution mechanism besides preemptive techniques. 428 The above discussions implicitly assumed that policies are 429 individually evaluated for conflicts and individually committed 430 without regard to other policies. However, a set of policies could be 431 labeled as part of a same "Commit Group", where the whole set of 432 policies in the Commit Group must be committed for a desired result 433 to be obtained. In this case, the conflict resolution mechanism would 434 need to verify that none of the policies in the Commit Group 435 conflicts with currently committed policies before the Commit Group 436 is added (in other words, committed) to the policy database. 438 The Commit Group conflict detection mechanism and subsequent addition 439 to the database should be implemented as an atomic process, i.e., no 440 changes to the policy database should be allowed by other processes 441 until either the whole Commit Group is checked and committed or a 442 conflict is detected and the process stopped, to avoid multiple 443 writers issues. 445 The above described atomic Commit Group conflict detection and policy 446 commit mechanism would eliminate the need for Commit Group rollback. 447 A rollback could be required if policies in a Commit Group were to be 448 checked for conflicts and committed one by one, since the detection 449 of a subsequent policy conflict in the Commit Group would require the 450 rollback of previously committed policies in that group. 452 6.1. Soft vs Hard Policy Constraints 454 Policies at any level in the policy hierarchy can be either soft or 455 hard. A soft policy imposes a soft constraint in a system that can be 456 violated without causing any catastrophic failure in the system. A 457 hard policy imposes a hard constraint in the system that must not be 458 violated. An example of a soft constraint is the degree of 459 underutilization (for example, a 40% utilization threshold could be 460 used as a soft constraint) of compute servers with regard to CPU 461 utilization. In such a case, when this soft constraint is violated, 462 the system continues to function, although it may consume more energy 463 (due to a non-linear dependence of the energy utilization as a 464 function of the CPU utilization) compared to a task allocation across 465 multiple servers after workload consolidation is performed. 466 Alternatively, a soft constraint could be violated if the network 467 bandwidth exceeds a certain fraction (say 80%) of the available 468 bandwidth, the energy utilization exceeds a certain value, or the 469 memory utilization falls below a certain value (say 30%). An example 470 of a hard constraint could be to disallow a desired mean CPU 471 utilization across allocated workloads in a compute subsystem to 472 exceed a certain high threshold (such as 90%), or to disallow the 473 number of CPUs allocated for a set of workloads to exceed a certain 474 value, or to disallow the network bandwidth across workloads to 475 exceed a certain value (say 98% of the maximum available bandwidth). 477 When considering policy conflicts, violation of policies across hard 478 policy constraints is undesirable, and must be avoided. A conflict 479 resolution could be possible by relaxing one or more of the hard 480 constraints to an extent that is mutually satisfactory to the 481 imposers of the policies. Alternatively, as discussed earlier, a new 482 hard policy that conflicts with existing policies is not admitted 483 into the system. Violation of policies across soft policy constraints 484 or between one or more hard policy constraints and one or more soft 485 policy constraints can be allowed, such that one or more soft policy 486 constraints are violated without hard constraints being violated. 487 Despite soft constraints being violated, it is desirable to have a 488 region of operating conditions that would allow the system to 489 operate. For admission of new policy constraints, whether hard or 490 soft, one should ensure that the overall system has a feasible region 491 for operation given the existing constraints, and the new constraints 492 under consideration. When such a feasible region is not possible, one 493 can consider relaxing one or more of the existing or new constraints 494 to allow such policies to be admitted. 496 7. Policy Pub/Sub Bus 498 In the previous section, we considered policy conflicts within a same 499 level subsystem. For example, new local policies added to the Compute 500 subsystem conflicting with existing local policies at that subsystem. 501 However, more subtle conflicts are possible between global and local 502 policies. 504 A global policy may conflict with subsystems' local policies. 505 Consider the following Compute subsystem local policy: "Platinum 506 treatment must be provided using server of type A." 508 The addition of the Global policy "Platinum treatment must be 509 provided using server subtype A-1" would intrude into the Compute 510 subsystem by redefining the type of server to be used for a 511 particular service treatment. While one could argue that such global 512 policy should not be permitted, this is an event that requires 513 detection and proper resolution. A possible resolution is for the 514 Compute subsystem to import the more restrictive policy into its 515 local database. The original local policy would remain in the 516 database as is along with the new restrictive policy. The local 517 policy engine would then enforce the more restricted form of the 518 policy after this policy change, which could make already existing 519 resource allocations non-compliant and requiring corrective actions, 520 e.g., Platinum treatment being currently provided by a server of type 521 A instead of a server of type A-1. 523 If the new Global policy read "Platinum treatment must be provided 524 using server of types A or B" instead, the Compute subsystem would 525 not need to do anything different, since the Compute subsystem has a 526 more restrictive local policy in place, i.e., "Platinum treatment 527 must be provided using server of type A." 529 The above examples demonstrate the need for subsystems to subscribe 530 to policy updates at the Global policy level. A policy 531 publication/subscription (pub/sub) bus would be required as shown in 532 Figure 4. 534 +----------------------------------------------------------------+ 535 | +----------------------------------------------+ | 536 | | Global Policy Engine | | 537 | +----------------------------------------------+ | 538 | | 539 | +----------------------------------------------+ | 540 | | Global Policies | | 541 | +----------------------------------------------+ | 542 +----------------------------------------------------------------+ 543 ^ 544 | 545 | 546 Policy Pub/Sub Bus V 547 -------------------------------------------------------------- 548 ^ ^ ^ ^ 549 | | | | 550 | | | | 551 V V V V 552 +-------------+ +-------------+ +-------------+ +-------------+ 553 |Compute | |Network | |Storage | |Whatever | 554 |Subsystem | |Subsystem | |Subsystem | |Subsystem | 555 | | | | | | | | 556 |Local Policy | |Local Policy | |Local Policy | |Local Policy | 557 |Engine | |Engine | |Engine | |Engine | 558 | | | | | | | | 559 |Local | |Local | |Local | |Local | 560 |Policies: | |Policies | |Policies | |Policies | 561 | P0, P1, | | P0, P1, | | P0, P1, | | P0, P1, | 562 | | | | | | | | 563 +-------------+ +-------------+ +-------------+ +-------------+ 565 Figure 4: A Policy Pub/Sub Bus 567 A policy conflict may force policies to change scope. Consider the 568 following existing policies in a Data Center: 570 Compute subsystem policy: "Platinum treatment requires a server of 571 type A or B." 573 Storage subsystem policy: "Platinum treatment requires a server 574 storage of type X or Y." 576 Now consider the outcome of adding the following new Global policy: 577 "Platinum treatment requires a server of type A when storage of type 578 X is used or a server of type B when storage of type Y is used." 580 This new Global policy intrudes into the Compute and Storage 581 subsystems. Again, one could argue that such global policy should not 582 be permitted. Nevertheless, this is an event that would require 583 detection and proper resolution. This Global policy causes a conflict 584 because the Compute and Storage subsystems can no longer 585 independently define whether to use a server of type A or B or 586 storage of type X or Y, respectively. If the Compute subsystem 587 selects server of type A for a customer and the Storage subsystem 588 selects storage of type Y for that same customer service the Global 589 policy is violated. In conclusion, if such global policy is 590 permitted, the Compute and Storage subsystems can no longer make such 591 selections. A possible conflict resolution is for the Compute and 592 Storage subsystems to relegate policy enforcement for such resources 593 to the Global policy engine. In this example, the Global Policy 594 engine would need to coordinate with the Compute and Storage 595 subsystems the selection of appropriate resource types to satisfy 596 that policy. 598 That suggests that the policy pub/sub bus should in fact be an 599 integral part of the northbound service interfaces (NBI) of the 600 subsystems in the hierarchy. Such issue was analyzed in [12], where 601 the concepts of service capability, service availability, and service 602 instantiation were introduced to enable a higher-level subsystem to 603 properly select services and resources from lower-level subsystems to 604 satisfy existing policies. 606 The above example demonstrates again the need for subsystems to 607 subscribe to policy updates at the higher policy level (the Global 608 policy level in this example) as shown in Figure 4. 610 If, as demonstrated, a Global policy may "hijack" or "nullify" local 611 policies of subsystems, what exactly makes the scope of a policy 612 local versus global then? 614 Proposition: A Local Policy does not affect the compliance state 615 imposed by global Policies or the local policies of other subsystems. 617 The above non-exhaustive examples demonstrate that global and local 618 policies may conflict in subtle ways. Policy conflicts will also 619 policy framework requires a policy pub/sub bus between all levels to 620 allow for conflict detection, conflict information propagation, and 621 conflict resolution (Figure 5). 623 Pub/Sub bus to higher level 624 ^ 625 | 626 Region 1 Domain V 627 +-------------------+ 628 | +---------------+ | 629 | |Region 1 Global| | 630 | |Policy Engine | | +-------------------+ 631 | +---------------+ | | | +---------------+ | 632 | | |<-->| |WAN 1 Global | | 633 | +---------------+ | | | |Policy Engine | | 634 | |Whatever | | | | +---------------+ | 635 | |Subsystems | | | | | 636 | | | | | | +---------------+ | 637 | |Local Policy | | | | |Whatever | | 638 | |Engines | | | | |Subsystems | | 639 | +---------------+ | | | | | | 640 +-------------------+ | | |Local Policy | | 641 ^ | | |Engines | | 642 Region | | | +---------------+ | 643 Pub/Sub Bus V | +-------------------+ 644 ----------------------+ 645 ^ ^ 646 | +-------------------------+ 647 | | 648 DC 1 Domain V DC N Domain V 649 +-------------------+ +-------------------+ 650 | +---------------+ | | +---------------+ | 651 | |DC 1 Global | | | |DC N Global | | 652 | |Policy Engine | | | |Policy Engine | | 653 | +---------------+ | | +---------------+ | 654 | | | | 655 | +---------------+ | | +---------------+ | 656 | |Whatever | | | |Whatever | | 657 | |Subsystems | | | |Subsystems | | 658 | | | | | | | | 659 | |Local Policy | | | |Local Policy | | 660 | |Engines | | | |Engines | | 661 | +---------------+ | | +---------------+ | 662 +-------------------+ +-------------------+ 664 Figure 5: Pub/Sub Bus - Hierarchical Policy Framework 666 7.1 Pub/Sub Bus Name Space 668 As described above, a higher tier policy engine would communicate 669 policies to lower tier policy engines using a policy pub/sub bus. 670 Conversely, lower tier policy engines would communicate their 671 configured policies and services to the higher tier policy engine 672 using the same policy pub/sub bus. Such communications require each 673 policy pub/sub bus to have a pre-defined/pre-configured policy "name 674 space". For example, a pub/sub bus could define services using the 675 name space "Platinum", "Gold", and "Silver". A policy could then be 676 communicated over that pub/sub bus specifying a Silver service 677 requirement. 679 In a hierarchical policy framework, a policy engine may use more than 680 one policy pub/sub bus, e.g., a policy pub/sub bus named "H" to 681 communicate with a higher tier policy engine and a policy pub/sub bus 682 named "L" to communicate with lower tier policy engines. As the name 683 spaces of policy pub/sub buses H and L may be different, the policy 684 engine would translate policies defined using the policy pub/sub bus 685 H name space to policies defined using the policy pub/sub bus L name 686 space, and vice-versa. For example, suppose that the policy pub/sub 687 bus H name space defines service levels named Platinum, Gold, and 688 Silver and that the policy pub/sub bus L name space does not define 689 such service levels, but defines QoS levels High, Medium, and Low. 690 The policy engine would translate a policy to support Silver service, 691 which is written using the policy pub/sub bus H name space, to an 692 appropriate policy (or set of policies) written using the policy 693 pub/sub bus L name space, e.g., QoS level Low. 695 The described policy framework does not preclude use of a single/same 696 name space throughout the hierarchy. However, to promote scalability 697 and limit complexity, the name spaces of higher tier policy pub/sub 698 buses should be limited to support higher level policies, since the 699 higher the degree of specificity allowed at the higher tiers of the 700 policy hierarchy the higher the operational complexity. 702 8. Examples 704 8.1 Establishment of a Multipoint Ethernet Service 706 Consider a service provider with an NFV infrastructure (NFVI) with 707 multiple vPoPs, where each vPoP is a separate administrative domain. 708 A customer "Z" requests the creation of a "multipoint Silver Ethernet 709 service" between three of its sites, which are connected to service 710 provider's vPoPs A, B, and C. The customer request is carried out 711 using a service provider self-service web portal, which offers 712 customers multiple service type options, e.g., point-to-point and 713 multipoint Ethernet services, and multiple service levels per service 714 type, e.g., Platinum, Gold, and Silver Ethernet services, where the 715 different service levels may represent different service 716 specifications in terms of QoS, latency, and etc. The web portal 717 relays the request to a service provider's OSS/BSS. The service 718 request is stored as a service policy that reads as: "multipoint 719 Silver Ethernet service between vPoPs A, B, and C for customer Z". 721 The OSS/BSS subsystem would communicate the service request and 722 requirements as a policy to a global NFV Orchestrator (NFVO) 723 subsystem using the name space of the pub/sub bus between these two 724 subsystems (see Section 7.1). For example, the OSS/BSS could 725 translate "Silver" service level into a policy defined using a 726 Network Service (NS) Flavor ID, as defined by the name space of 727 pub/sub bus between the OSS/BSS and the NFVO. 729 The service provider's vPoP NFV infrastructure architecture may vary 730 depending on the size of each vPoP and other specific needs of the 731 service provider. For example, a vPoP may have a local NFVO subsystem 732 and one or more local Virtual Infrastructure Manager (VIM) subsystems 733 (as in Figure 6). In this case, the global NFVO subsystem would 734 communicate the service request and requirements as a policy to the 735 local NFVOs of vPoPs A, B, and C. 737 At each vPoP, the local NFVO (and VNF Managers) would carry out the 738 requested service policy based on the local configurations of 739 respective subsystems and current availability of resources. For 740 example, the requested service may translate in vPoP A to use a 741 specific vCE (virtual customer edge) VNF type, say vCE_X, while in 742 vPoP B it may translate to use a different vCPE VNF type, say vCPE_Y, 743 due to local subsystem configurations (refer to Section 2 for a 744 discussion on subsystem actions and configurations). Similarly, the 745 local VIM interaction with the vPoP's compute, network, and storage 746 subsystems may lead to local configurations of these subsystems 747 driven by the translation of the policies received by the respective 748 subsystems (see Section 3 for a discussion on global versus local 749 policies). Note that the original policy at the OSS/BSS level is 750 translated throughout the policy hierarchy by respective policy 751 engines to fit the name spaces of the associated pub/sub buses in the 752 hierarchy. 754 The global NFVO subsystem could also communicate a policy defining 755 the requirements to create a multipoint Ethernet service between 756 vPoPs A, B, and C to a WAN infrastructure management (WIM) subsystem 757 (not shown in Figure 6). The WIM subsystem could oversee a hierarchy 758 of other subsystems, e.g., SDN multi-domain architecture of 759 controllers deployed as a hierarchy of network regions (see [12]). 760 Network subsystems would translate locally received policies to local 761 configurations (again, refer to Section 2 for a discussion on 762 subsystem actions and configurations). 764 As depicted in Figure 6, policy communications would employ a policy 765 pub/sub bus between the subsystems' policy engines in the policy 766 hierarchy (see Section 7). The global NFVO subsystem should have 767 visibility into the policies defined locally at each vPoP to be able 768 to detect any potential global policy conflicts, e.g., a local vPoP 769 administrator could add a local policy that violates or conflicts 770 with a global policy. In addition, the global NFVO subsystem would 771 benefit from being able to import the currently configured services 772 at each vPoP. The global NFVO would use such information to monitor 773 global policy conformance and also to facilitate detection of policy 774 violations when new global policies are created, e.g., a global level 775 administrator is about to add a new global policy that, if committed, 776 would make certain already configured services a violation of the 777 policy. The publication of subsystem service tables for consumption 778 by a global policy engine is a concept used in the Congress [24] 779 OpenStack [22] project. 781 Customers 782 | 783 V 784 +--------------+ 785 | Web Portal | 786 +--------------+ 787 ^ 788 | 789 V 790 +-----------------+ +-----------------+ 791 | OSS/BSS | | Global NFVO | 792 | +-------------+ | | +-------------+ | 793 | |OSS/BSS | | Policy | |NFVO | | 794 | |Policy Engine|<---------->|Policy Engine| | 795 | +-------------+ | | +-------------+ | 796 | | | ^ | 797 | ... | | | ... | 798 +-----------------+ +--------|--------+ 799 | 800 Policy (Pub/Sub Bus) V 801 ------------------------------------------- 802 ^ ^ ^ 803 | | | 804 +-------|-------+ +-------|-------+ +-------|-------+ 805 |vPoP A | | |vPoP B | | |vPoP C | | 806 | V | | V | | V | 807 | +-----------+ | | +-----------+ | | +-----------+ | 808 | |Local NFVO | | | |Local NFVO | | | |Local NFVO | | 809 | | +-------+ | | | | +-------+ | | | | +-------+ | | 810 | | |Policy | | | | | |Policy | | | | | |Policy | | | 811 | | |Engine | | | | | |Engine | | | | | |Engine | | | 812 | | +-------+ | | | | +-------+ | | | | +-------+ | | 813 | +-----------+ | | +-----------+ | | +-----------+ | 814 | ^ | | ^ | | ^ | 815 | | | | | | | | | 816 | V | | V | | V | 817 | +-----------+ | | +-----------+ | | +-----------+ | 818 | |VIM | | | |VIM | | | |VIM | | 819 | | +-------+ | | | | +-------+ | | | | +-------+ | | 820 | | |Policy | | | | | |Policy | | | | | |Policy | | | 821 | | |Engine | | | | | |Engine | | | | | |Engine | | | 822 | | +-------+ | | | | +-------+ | | | | +-------+ | | 823 | +-----------+ | | +-----------+ | | +-----------+ | 824 | ... | | ... | | ... | 825 +---------------+ +---------------+ +---------------+ 827 Figure 6: Simplified view of a service provider's NFV Architecture: 828 Multipoint Ethernet Service Example 830 8.2 Policy-Based NFV Placement and Scheduling 832 One of the goals of NFV is to allow a Service Provider (SP) offer the 833 NFV infrastructure as a service to other Service Providers as 834 Customers - this is called NFVIaaS [6]. In this context, it may be 835 desirable for a Service Provider to run virtual network elements 836 (e.g., virtual routers, virtual firewalls, and etc.) as virtual 837 machine instances inside the infrastructure of another Service 838 Provider. In this document, we call the former a "customer SP" and 839 the latter an "NFVIaaS SP." 841 There are many reasons for a customer SP to require the services of 842 an NFVIaaS SP, including: to meet performance requirements (e.g., 843 latency or throughput) in locations where the customer SP does not 844 have physical data center presence, to allow for expanded customer 845 reach, regulatory requirements, and etc. 847 As VNFs are virtual machines, their deployment in such NFVIaaS SPs 848 would share some of the same placement restrictions (i.e., placement 849 policies) as those intended for Cloud Services. However, VNF 850 deployment will drive support for unique placement policies, given 851 VNF's stringent service level specifications (SLS) required/imposed 852 by customer SPs. Additionally, NFV DCs or NFV PoPs [8] often have 853 capacity, energy and other constraints - thus, optimizing the overall 854 resource usage based on policy is an important part of the overall 855 solution. 857 This section describes an example [15] of a global policy written in 858 Datalog [3] applicable to compute to promote energy conservation for 859 the NFVIaaS use case in an OpenStack framework. The goal of that 860 global policy is to address the energy efficiency requirements 861 described in the ETSI NFV Virtualization Requirements [9]. 863 A related energy efficiency use case using analytics-driven policies 864 in the context of OpenStack Congress [24] policy as a service was 865 presented and demonstrated at the Vancouver OpenStack summit [14], 866 where the Congress policy engine delegated VM placement to a VM 867 placement engine that migrated under-utilized VMs to save energy. 869 8.2.1 Policy Engine Role in NFV Placement and Scheduling 871 A policy engine may facilitate policy-based resource placement and 872 scheduling of VMs in an NFVIaaS environment. In this role, a policy 873 engine (Figure 7) would determine optimized placement and scheduling 874 choices based on the constraints specified by current resource 875 placement and scheduling policies. The policy engine would evaluate 876 such policies based on event triggers or programmable timers. 878 In one instantiation, a policy engine would interface with a 879 "Measurement Collector" (e.g., OpenStack Ceilometer [23]) to 880 periodically retrieve instantaneous per-server CPU utilization, in 881 order to compute a table of per-server average CPU utilization. In an 882 alternative instantiation, the Measurement Collector could itself 883 compute per-server average CPU utilization and provide that 884 information to the policy engine. The latter approach would reduce 885 overhead, since it would avoid too frequent pulling of stats from the 886 Measurement Collector. 888 Other average utilization parameters such as VM CPU utilization, VM 889 Memory utilization, VM disk read IOPS, Network utilization/latency, 890 and etc. could also be used by the policy engine to enforce other 891 types of placement policies. 893 +---------------------------------------+ 894 | Policy Engine | 895 | Performs resource placement | 896 | and scheduling function (proactive | 897 | and dynamic policy enforcement) | 898 +-------------------+-------------------+ 899 | 900 | 901 +-------------+-------------+ 902 | Measurement Collector | 903 | VM DB - CPU | 904 | Utilization, Network | 905 | Utilization/Latency etc. | 906 +---------------------------+ 908 Figure 7: NFVIaaS Architecture for Policy Based Resource 909 Placement and Scheduling 911 In the ETSI NFV Architectural Framework [7], the Policy Engine is 912 part of the Orchestrator and the Measurement Collector is part of the 913 Virtual Infrastructure Manager (VIM). 915 8.2.2 Policy-based NFV Placement and Scheduling with OpenStack 917 Consider an NFVIaaS SP that owns a multitude of mini NFV data centers 918 managed by OpenStack [22] where: 920 - The Policy Engine function is performed by OpenStack Congress 921 [24]. 923 - The Measurement Collector function is performed by OpenStack 924 Celiometer [23]. 926 - The Policy Engine has access to the OpenStack Nova database that 927 stores records of mapping of virtual machines to physical servers. 929 An exemplary mini NFV DC configuration is used in this example, as 930 described below: 932 - 210 physical servers in 2U rack server configuration spread over 933 10 racks. 935 - 4 types of physical servers each with a different system 936 configuration and from a particular manufacturer. It is possible that 937 the servers are all from the same or different manufacturers. For the 938 purpose of this example, a server "type 1" is described. Server type 939 1 has 32 virtual CPUs and 128GB DRAM from manufacturer x. Assume 55 940 physical servers of type 1 per mini NFV DC. 942 - 2 types of instances large.2 and large.3, which are described in 943 Table 1. Each parameter has a minimum guarantee and a maximum usage 944 limit. 946 +--------+------------------+-------------------+---------------+ 947 |Instance|Virtual CPU Units |Memory (GB) |Minimum/Maximum| 948 |Type |Minimum Guarantee |Minimum Guarantee |Physical Server| 949 | |/Maximum Usage |/Maximum Usage |Utilization (%)| 950 +--------+------------------+-------------------+---------------+ 951 |large.2 | 0/4 | 0/16 | 0/12.5 | 952 |large.3 | 0/8 | 0/32 | 0/25 | 953 +--------+------------------+-------------------+---------------+ 955 Table 1: NFVIaaS Instance Types 957 For the purpose of this example, the Mini NFV DC topology is 958 considered static -- the above topology, including the network 959 interconnection, is available through a simple file-based interface. 961 Policy 1 (an exemplary NFV policy): In a descriptive language, Policy 962 1 is as follows - "For physical servers of type 1, there can be at 963 most only one active physical server with average overall utilization 964 less than 50%." Policy 1 is an example of reactive enforcement. The 965 goal of this policy is to address the energy efficiency requirements 966 described in the ETSI NFV Virtualization Requirements [9]. 968 Policy 2 (another exemplary NFV policy): Policy 2 is designed to 969 protect NFV instances from physical server failures. Policy 2 reads 970 as follows in a descriptive language - "Not more than one VM of the 971 same high availability (HA) group must be deployed on the same 972 physical server". Policy 2 is an example of proactive and reactive 973 enforcement. 975 Note that there may be cases where there may not be any placement 976 solution respecting both policies given the current DC load. To avoid 977 such cases, Policy 1 could be relaxed to: "Minimize the number of 978 physical servers with average overall utilization less than 50%". 980 Policy 1 calls for the identification of servers by type. OpenStack 981 Congress would need to support server type, average CPU utilization, 982 and be able to support additional performance parameters (in the 983 future) to support additional types of placement policies. OpenStack 984 Congress would run the policy check periodically or based on trigger 985 events, e.g., deleting/adding VMs. In case OpenStack Congress detects 986 a violation, it would determine optimized placement and scheduling 987 choices so that current placement and scheduling policy are not 988 violated. 990 A key goal of Policy 1 is to ensure that servers are not kept under 991 low utilization, since servers have a non-linear power profile and 992 exhibit relatively higher power consumption at lower utilization. 993 For example, in the active idle state as much as 30% of peak power is 994 consumed. At the physical server level, instantaneous energy 995 consumption can be accurately measured through IPMI standard. At a 996 customer instance level, instantaneous energy consumption can be 997 approximately measured using an overall utilization metric, which is 998 a combination of CPU utilization, memory usage, I/O usage, and 999 network usage. Hence, Policy 1 is written in terms of overall 1000 utilization and not power usage. 1002 The following example addressed the combined effect of Policy 1 and 1003 Policy 2. 1005 For an exemplary maximum usage scenario, 53 physical servers could be 1006 under peak utilization (100%), 1 server (server-a) could be under 1007 partial utilization (62.5%) with 2 instances of type large.3 and 1 1008 instance of type large.2 (this instance is referred as large.2.X1), 1009 and 1 server (server-b) could be under partial utilization (37.5%) 1010 with 3 instances of type large.2. Call these three instances 1011 large.2.X2, large.2.Y and large.2.Z 1013 One HA-group has been configured and two large.2 instances belong to 1014 this HA-group. To enforce Policy 2 large.2.X1 and large2.X2 that 1015 belong to the HA-group have been deployed in different physical 1016 servers, one in server-a and a second in server-b. 1018 When one of the large.3 instances mapped to server-a is deleted from 1019 physical server type 1, Policy 1 will be violated, since the overall 1020 utilization of server-a falls to 37.5%, since two servers are 1021 underutilized (below 50%). 1023 OpenStack Congress, on detecting the policy violation, would use 1024 various constraint based placement techniques to find placements for 1025 physical server type 1 to address Policy 1 violation without breaking 1026 Policy 2. Constraint based placement involves a convex optimization 1027 framework [5]. Some of the algorithms that could be considered 1028 include linear programming [1], branch and bound [2], interior point 1029 methods, equality constrained minimization, non-linear optimization, 1030 and etc. 1032 Various new placements are described below: 1034 1) New placement 1: Move 2 of three instances of large.2 running on 1035 server-b to server-a. Overall utilization of server-a - 62.5%. 1036 Overall utilization of server-b - 25%. large.2.X2 must not be one of 1037 the migrated instances. 1039 2) New placement 2: Move 1 instance of large.3 to server-b. Overall 1040 utilization of server-a - 12.5%. Overall utilization of server-b - 1041 62.5%. 1043 A third solution consisting of moving 3 large.2 instances to server-a 1044 cannot be adopted, since this violates Policy 2. Another policy 1045 minimizing the number of migrations could allow choosing between 1046 solutions (1) and (2) above. 1048 New placements 2 and 3 could be considered optimal, since they 1049 achieve maximal bin packing and open up the door for turning off 1050 server-a or server-b and maximizing energy efficiency. 1052 To detect violations of Policy 1, an example of a classification rule 1053 is expressed below in Datalog, the policy language used by OpenStack 1054 Congress. 1056 The database table exported by the Resource Placement and Scheduler 1057 for Policy 1 is described below: 1059 - server_utilization (physical_server, overall_util): Each 1060 database entry has the physical server and the calculated average 1061 overall utilization. 1063 - vm_host_mapping(vm, server): Each database entry gives the 1064 physical server on which VM is deployed. 1066 - anti-affinity_group(vm, group): Each entry gives the anti- 1067 affinity group to which a VM belongs. 1069 Policy 1 in a Datalog [3] policy language is as follows: 1071 error (physical_server) :- 1072 nova: node (physical_server, "type1"), 1073 resource placement and scheduler: 1074 server_utilization (physical_server, overall_util < 50) 1076 Policy 2 (in Datalog policy language): 1078 error(vm) :- 1079 anti-affinity_group(vm1, grp1), 1080 anti-affinity_group(vm2, grp2), 1081 grp1 != grp2, 1082 nova: vm host mapping(vm1, server-1), 1083 nova: vm host mapping(vm2,server-2), 1084 server-1 == server-2 1086 9. Summary 1088 This document approached the policy framework and architecture from 1089 the perspective of overall orchestration requirements for services 1090 involving multiple subsystems. The analysis extended beyond common 1091 orchestration for compute, network, and storage subsystems to also 1092 include energy conservation constraints. This document also analyzed 1093 policy scope, global versus local policies, policy actions and 1094 translations, policy conflict detection and resolution, interactions 1095 among policies engines, and a hierarchical policy 1096 architecture/framework to address the demanding and growing 1097 requirements of NFV environments, applicable as well to general cloud 1098 infrastructures. 1100 The concept of NFV and the proposed policy architecture is applicable 1101 to service providers and also enterprises. For example, an enterprise 1102 branch office could have capacity and energy constraints similar to 1103 that of many service provider NFV vPoPs in constrained environments. 1104 This is an aspect that would be worth examining in detail in future 1105 work. 1107 10. IANA Considerations 1109 This draft does not have any IANA considerations. 1111 11. Security Considerations 1113 Security issues due to exchanging policies across different 1114 administrative domains are an aspect for further study. 1116 12. Contributors 1118 In addition to the authors listed on the front page, the following 1119 individuals contributed to the content of Section 8.2 ("Policy-Based 1120 NFV Placement and Scheduling"): 1122 Tim Hinrichs 1123 Styra 1124 tim@styra.com 1126 Ruby Krishnaswamy 1127 Orange 1128 ruby.krishnaswamy@orange.com 1130 Arun Yerra 1131 Dell Inc. 1132 arun.yerra@dell.com 1134 13. References 1136 13.1. Normative References 1138 13.2. Informative References 1140 [1] Alevras, D. and Padberg, M. "Linear Optimization and Extensions: 1141 Problems and Solutions," Universitext, Springer-Verlag, 2001. 1143 [2] Brassard, G. and Bratley, P., "Fundamentals of Algorithmics," . 1145 [3] Ceri, S. et al., "What you always wanted to know about Datalog 1146 (and never dared to ask)," IEEE Transactions on Knowledge and Data 1147 Engineering, (Volume: 1, Issue: 1), August 2002 1149 [4] "Common Information Model (CIM)," DTMF, 1150 http://www.dmtf.org/standards/cim 1152 [5] "Convex Optimization," 1153 https://web.stanford.edu/~boyd/cvxbook/bv_cvxbook.pdf 1155 [6] ETSI GS NFV 001 v1.1.1 (2013-10): "Network Function 1156 Virtualisation (NFV); Use Cases," http://www.etsi.org/deliver/ 1157 etsi_gs/NFV/001_099/001/01.01.01_60/gs_NFV001v010101p.pdf 1159 [7] ETSI GS NFV 002 v1.2.1 (2014-12): "Network Function 1160 Virtualisation (NFV); Architectural Framework," 1161 http://www.etsi.org/deliver/etsi_gs/NFV/001_099/002/ 1162 01.02.01_60/gs_nfv002v010201p.pdf 1164 [8] ETSI GS NFV 003 V1.2.1 (2014-12): "ETSI NFV: Terminology for Main 1165 Concepts in NFV," http://www.etsi.org/deliver/etsi_gs/ 1166 NFV/001_099/003/01.02.01_60/gs_NFV003v010201p.pdf 1168 [9] ETSI GS NFV 004 v1.1.1 (2013-10): "Network Function 1169 Virtualisation (NFV); Virtualization Requirements," 1170 http://www.etsi.org/deliver/etsi_gs/NFV/001_099/004/01.01.01_60/ 1171 gs_NFV004v010101p.pdf 1173 [10] ETSI GS NFV-INF 001 v.1.1.1 (2015-01): "Network Function 1174 Virtualisation (NFV); Infrastructure Overview," 1175 http://www.etsi.org/deliver/etsi_gs/NFV- 1176 INF/001_099/001/01.01.01_60/gs_NFV-INF001v010101p.pdf 1178 [11] ETSI NFV White Paper: "Network Functions Virtualisation, An 1179 Introduction, Benefits, Enablers, Challenges, & Call for Action," 1180 http://portal.etsi.org/NFV/NFV_White_Paper.pdf 1182 [12] Figueira, N. and Krishnan, R., "SDN Multi-Domain Orchestration 1183 and Control: Challenges and Innovative Future Directions," CNC VIII: 1184 Cloud and Multimedia Applications, IEEE International Conference on 1185 Computing (ICNC), February 2015 1187 [13] Grit, L. et al., "Virtual Machine Hosting for Networked 1188 Clusters: Building the Foundations for "Autonomic" Orchestration," 1189 Virtualization Technology in Distributed Computing, 2006. VTDC 2006. 1191 [14] Krishnan, R. et al., "Helping Telcos go Green and save OpEx via 1192 Policy", Talk and demo at the Vancouver OpenStack summit. Video Link: 1193 https://www.openstack.org/summit/vancouver-2015/summit-videos/ 1194 presentation/helping-telcos-go-green-and-save-opex-via-policy 1196 [15] Krishnan, R. et al., "NFVIaaS Architectural Framework for Policy 1197 Based Resource Placement and Scheduling," 1198 https://datatracker.ietf.org/doc/draft-krishnan-nfvrg-policy-based- 1199 rm-nfviaas/ 1201 [16] Lee, S. et al., "Resource Management in Service Chaining", 1202 https://datatracker.ietf.org/doc/draft-irtf-nfvrg-resource- 1203 management-service-chain/ 1205 [17] Moore, B., et al., "Information Model for Describing Network 1206 Device QoS Datapath Mechanisms," RFC 3670, January 2004 1208 [18] Moore, B. et al., "Policy Core Information Model -- Version 1 1209 Specification," RFC 3060, February 2001 1211 [19] "OpenDaylight Group Based Policy," 1212 https://wiki.opendaylight.org/view/Project_Proposals:Group_Based_ 1213 Policy_Plugin 1215 [20] "OpenDaylight Network Intent Composition Project," 1216 https://wiki.opendaylight.org/index.php?title=Network_Intent_ 1217 Composition:Main#Friday_8AM_Pacific_Time 1219 [21] "OpenDaylight SDN Controller," http://www.opendaylight.org/ 1221 [22] "OpenStack," http://www.openstack.org/ 1223 [23] "OpenStack Celiometer," http://docs.openstack.org/ 1224 developer/ceilometer/measurements.html 1226 [24] "OpenStack Congress," https://wiki.openstack.org/wiki/Congress 1228 [25] "OpenStack Neat," http://openstack-neat.org/ 1230 [26] "OpenStack Neutron," https://wiki.openstack.org/wiki/Neutron 1232 [27] "Policy Framework Working Group," IETF, 1233 http://www.ietf.org/wg/concluded/policy.html 1235 [28] Westerinen, A. et al., "Terminology for Policy-Based 1236 Management," RFC 3198, November 2001 1238 Acknowledgements 1240 The authors would like to thank the following individuals for 1241 valuable discussions on some of the topics addressed in this 1242 document: Uwe Michel, Klaus Martiny, Pedro Andres Aranda Gutierrez, 1243 Tim Hinrichs, Juergen Schoenwaelder, and Tina TSOU. 1245 Authors' Addresses 1247 Norival Figueira 1248 Brocade Communications Systems, Inc. 1249 nfigueir@Brocade.com 1251 Ram (Ramki) Krishnan 1252 Dell 1253 Ramki_Krishnan@Dell.com 1255 Diego R. Lopez 1256 Telefonica I+D 1257 diego.r.lopez@telefonica.com 1258 Steven Wright 1259 AT&T 1260 sw3588@att.com 1262 Dilip Krishnaswamy 1263 IBM Research 1264 dilikris@in.ibm.com