idnits 2.17.1 draft-irtf-nfvrg-policy-based-resource-management-02.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document doesn't use any RFC 2119 keywords, yet seems to have RFC 2119 boilerplate text. -- The document date (October 30, 2016) is 2728 days in the past. Is this intentional? -- Found something which looks like a code comment -- if you have code sections in the document, please surround them with '' and '' lines. Checking references for intended status: Informational ---------------------------------------------------------------------------- == Unused Reference: 'ETSI-NFV-PER001' is defined on line 1155, but no explicit reference was found in the text == Unused Reference: 'I-D.ietf-bmwg-virtual-net' is defined on line 1169, but no explicit reference was found in the text == Unused Reference: 'I-D.liu-bmwg-virtual-network-benchmark' is defined on line 1186, but no explicit reference was found in the text == Outdated reference: A later version (-05) exists of draft-ietf-bmwg-virtual-net-04 Summary: 0 errors (**), 0 flaws (~~), 6 warnings (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 nfvrg R. Szabo, Ed. 3 Internet-Draft Ericsson 4 Intended status: Informational S. Lee, Ed. 5 Expires: May 3, 2017 ETRI 6 N. Figueira 7 Brocade 8 October 30, 2016 10 Policy-Based Resource Management 11 draft-irtf-nfvrg-policy-based-resource-management-02 13 Abstract 15 abstract to be defined 17 Status of This Memo 19 This Internet-Draft is submitted in full conformance with the 20 provisions of BCP 78 and BCP 79. 22 Internet-Drafts are working documents of the Internet Engineering 23 Task Force (IETF). Note that other groups may also distribute 24 working documents as Internet-Drafts. The list of current Internet- 25 Drafts is at http://datatracker.ietf.org/drafts/current/. 27 Internet-Drafts are draft documents valid for a maximum of six months 28 and may be updated, replaced, or obsoleted by other documents at any 29 time. It is inappropriate to use Internet-Drafts as reference 30 material or to cite them other than as "work in progress." 32 This Internet-Draft will expire on May 3, 2017. 34 Copyright Notice 36 Copyright (c) 2016 IETF Trust and the persons identified as the 37 document authors. All rights reserved. 39 This document is subject to BCP 78 and the IETF Trust's Legal 40 Provisions Relating to IETF Documents 41 (http://trustee.ietf.org/license-info) in effect on the date of 42 publication of this document. Please review these documents 43 carefully, as they describe your rights and restrictions with respect 44 to this document. Code Components extracted from this document must 45 include Simplified BSD License text as described in Section 4.e of 46 the Trust Legal Provisions and are provided without warranty as 47 described in the Simplified BSD License. 49 Table of Contents 51 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 52 1.1. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . 3 53 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3 54 3. Definitions . . . . . . . . . . . . . . . . . . . . . . . . . 3 55 4. Requirements . . . . . . . . . . . . . . . . . . . . . . . . 4 56 5. Architecture Considerations . . . . . . . . . . . . . . . . . 4 57 5.1. MANO Architecture . . . . . . . . . . . . . . . . . . . . 5 58 5.2. Policies in the MANO Architecture . . . . . . . . . . . . 8 59 5.3. Global vs Local Policies . . . . . . . . . . . . . . . . 9 60 5.4. Hierarchical Policy Framework . . . . . . . . . . . . . . 10 61 5.4.1. Mapping to Hierarchical Resource Orchestration . . . 12 62 5.5. Policy Pub/Sub Bus . . . . . . . . . . . . . . . . . . . 13 63 5.5.1. Pub/sub bus in the hierarchical framework . . . . . . 15 64 5.6. Policy Intent Statement versus Subsystem Actions and 65 Configurations . . . . . . . . . . . . . . . . . . . . . 17 66 5.7. Static vs Dynamic vs Autonomic Policies . . . . . . . . . 17 67 5.8. Policy Conflicts and Resolution . . . . . . . . . . . . . 17 68 5.9. Soft vs Hard Policy Constraints . . . . . . . . . . . . . 17 69 5.10. Operational Policies for Resource management . . . . . . 17 70 5.10.1. Operational Policies at NFVO . . . . . . . . . . . . 19 71 5.10.2. Operational Policies at VIM/WIM . . . . . . . . . . 19 72 6. Policy-Based Resource Management Examples . . . . . . . . . . 20 73 6.1. Policy-Based Multipoint Ethernet Service . . . . . . . . 20 74 6.2. Policy-Based NFV Placement . . . . . . . . . . . . . . . 20 75 6.3. Policy-Based VNF-FG Management . . . . . . . . . . . . . 20 76 6.4. Policy-Based Fault Management . . . . . . . . . . . . . . 22 77 7. Implementation Examples . . . . . . . . . . . . . . . . . . . 28 78 8. Gaps and Open Questions . . . . . . . . . . . . . . . . . . . 28 79 9. Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . 28 80 9.1. Relation to other IETF/IRTF activities . . . . . . . . . 28 81 10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 28 82 11. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 28 83 12. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 29 84 13. Security Considerations . . . . . . . . . . . . . . . . . . . 29 85 14. References . . . . . . . . . . . . . . . . . . . . . . . . . 29 86 14.1. Normative References . . . . . . . . . . . . . . . . . . 29 87 14.2. Informative References . . . . . . . . . . . . . . . . . 29 88 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 32 90 1. Introduction 92 NFV "Point of Presence" (PoP) will be likely constrained in compute 93 and storage capacity. Since practically all NFV PoPs are foreseen to 94 be distributed, inter-datacenter network capacity is also a 95 constraint. Additionally, energy is also a constraint, both as a 96 general concern for NFV operators, and in particular for specific- 97 purpose NFV PoPs such as those in mobile base stations. This draft 98 focuses on the optimized resource management and workload 99 distribution based on policy to address such contraints. 101 1.1. Scope 103 For the first version of the draft, only the research group currently 104 adopted drafts (i.e., [I-D.norival-nfvrg-nfv-policy-arch], 105 [I-D.irtf-nfvrg-resource-management-service-chain], and 106 [I-D.unify-nfvrg-recursive-programming]) are considered as inputs to 107 this document. The initial goal is to summarize these inputs and to 108 assess gaps and open questions. 110 2. Terminology 112 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 113 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 114 document are to be interpreted as described in [RFC2119]. 116 3. Definitions 118 This document uses the terms of [ETSI-NFV-TERM]: 120 o MANO - Management and Orchestration: Describes the architecture 121 framework to manage NFVI and orchestrate the allocation of 122 resources needed by the NSs and VNFs. 124 o NF - Network Functions: A functional building block within a 125 network infrastructure, which has well-defined external interfaces 126 and a well-defined functional behavior. 128 o NFV Framework: The totality of all entities, reference points, 129 information models and other constructs defined by the 130 specifications published by the ETSI ISG NFV. 132 o NFVI - NFV Infrastructure: The NFV-Infrastructure is the totality 133 of all hardware and software components which build up the 134 environment in which VNFs are deployed. 136 o NFVI-PoP: A location or point of presence that hosts NFV 137 infrastructure 139 o NFVO - Network Function Virtualization Orchestrator: The NFV 140 Orchestrator is in charge of the network wide orchestration and 141 management of NFV (infrastructure and software) resources, and 142 realizing NFV service topology on the NFVI. 144 o NS - Network service: A composition of network functions and 145 defined by its functional and behavioural specification. 147 o VNF - Virtualized Network Function: An implementation of an NF 148 that can be deployed on a Network Function Virtualization 149 Infrastructure (NFVI). 151 o VNF-FG - VNF Forwarding Graph: A NF forwarding graph where at 152 least one node is a VNF. 154 Additionally, we use the following terms: 156 o NFP - Network Forwarding Path: The sequence of hardware/software 157 switching ports and operations in the NFV network infrastructure 158 as configured by management and orchestration that implements a 159 logical VNF forwarding graph "link" connecting VNF "node" logical 160 interfaces. 162 o Virtual Link: A set of connection points along with the 163 connectivity relationship between them and any associated target 164 performance metrics (e.g. bandwidth, latency, QoS). The Virtual 165 Link can interconnect two or more entities (VNF components, VNFs, 166 or PNFs). 168 o Scaling: Ability to dynamically extend/reduce resources granted to 169 the Virtual Network function (VNF) as needed. 171 o NFVIaaS: NFV infrastructure as a service to other SP customers. 173 o SDN: Software Defined Networking. 175 o BSS: Business Support Systems 177 o OSS: Operation Support Systems 179 o DC: Data Center 181 o VM: Virtual machine 183 4. Requirements 185 tbd 187 5. Architecture Considerations 188 5.1. MANO Architecture 190 According to the ETSI MANO framework [ETSI-NFV-MANO], an NFVO is 191 split into two functions (see Figure 1): 193 o The orchestration of NFVI resources across multiple VIMs, 194 fulfilling the Resource Orchestration functions. The NFVO uses 195 the Resource Orchestration functionality to provide services that 196 support accessing NFVI resources in an abstracted manner 197 independently of any VIMs, as well as governance of VNF instances 198 sharing resources of the NFVI infrastructure 200 o The lifecycle management of Network Services, fulfilling the 201 network Service Orchestration functions. 203 Similarly, a VIM is split into two functions (see Figure 1): 205 o Orchestrating the allocation/upgrade/release/reclamation of NFVI 206 resources (including the optimization of such resources usage), 207 and 209 o managing the association of the virtualised resources to the 210 physical compute, storage, networking resources. 212 +-------------------+ 213 |NVFO | 214 | +--------------+ | 215 | |NFVO: | | 216 | |Service | | 217 | |Lifecycle | | 218 | |Management | | 219 | +------+-------+ | 220 | | | 221 | +------+-------+ | 222 | |NFVO: | | 223 | |Resource | | 224 | |Orchestration | | 225 | +--+---+----+--+ | 226 +-----|---|----|----+ 227 / | \ 228 /---------/ | \------------\ 229 / | \ 230 +-------------|-----+ +--------|----------+ +------|------------+ 231 |VIM | | |VIM | | |VIM | | 232 | +----------+---+ | | +-----+--------+ | | +---+----------+ | 233 | |VIM: | | | |VIM: | | | |VIM: | | 234 | |Orchestration | | | |Orchestration | | | |Orchestration | | 235 | |& | | | |& | | | |& | | 236 | |Optimization | | | |Optimization | | | |Optimization | | 237 | +------+-------+ | | +------+-------+ | | +------+-------+ | 238 | | | | | | | | | 239 | +------+-------+ | | +------+-------+ | | +------+-------+ | 240 | |VIM: | | | |VIM: | | | |VIM: | | 241 | |Virtualized 2 | | | |Virtualized 2 | | | |Virtualized 2 | | 242 | |Pys mapping | | | |Pys mapping | | | |Pys mapping | | 243 | +--------------+ | | +--------------+ | | +--------------+ | 244 +-------------------+ +-------------------+ +-------------------+ 246 Figure 1: Functional decomposition of the NFVO and the VIM according 247 to the ETSI MANO 249 In Figure 2 we show various policies mapped to the MANO architecture 250 (see Section 5.2 for more dicussions on policies in the MANO 251 architeture): 253 o Tenant Policies: Tenant policies exist whenever a domain offers a 254 virtualization service to more than one consumer. User tenants 255 may exists at the northbound of the NFVO. Additionally, if a VIM 256 exposes resource services to more than one NFVO, then each NFVO 257 may appear as a tenant (virtualization consumer) at the northbound 258 of the VIM. 260 o Wherever virtualization services are produced or consumed 261 corresponding export and import policies may exist. Export 262 policies govern the details of resources, capabilities, costs, 263 etc. exposed to consumers. In turn, consumers (tenants) apply 264 import policies to filter, tweak, annotate resources and services 265 received from their southbound domains. An entity may at the same 266 time consume and produce virtualization services hence apply both 267 import and export policies. 269 o Operational policies support the business logic realized by the 270 domain's ownership. They are often associated with Operations or 271 Business Support Systems (OSS or BSS) and frequently determine 272 operational objectives like energy optimization, utilization 273 targets, offered services, charging models, etc. Operational 274 policies may be split according to different control plane layers, 275 for example, i) lifecycle and ii) resource management layers 276 within the NFVO. 278 T1 T2...Tn 279 | | | 280 +-----|--|---|------+ 281 |NVFO | | | | Tenant 282 | +--+--+---+----+ | <- Policies 283 | |NFVO: | | 284 Operational | |Service | | 285 Policies-> | |Lifecycle | | 286 | |Management | | 287 | +------+-------+ | 288 | | | 289 | +------+-------+ | 290 | |NFVO: | | 291 Operational | |Resource | | 292 Policies-> | |Orchestration | | ^ 293 | +--+---+----+--+ | |Import 294 to +-----|---|---------+ |Policies 295 other NFVO / \ 296 \ +-------+ \ 297 \ / \ to NFVO ^ 298 +------\------|-----+ \ / |Export 299 |VIM \ | | \ / |Policies 300 | +-----+----+---+ | +--------|----|-----+ 301 | |VIM: | | |VIM | | | Tenant 302 | |Orchestration | | | +-----+----+---+ | <- Policies 303 | |& | | | |VIM: | | 304 | |Optimization | | . | |Orchestration | | 305 | +------+-------+ | . | |& | | <- Operational 306 | | | | |Optimization | | Policies 307 | +------+-------+ | | +------+-------+ | 308 | |VIM: | | | | | 309 | |Virtualized 2 | | | +------+-------+ | 310 | |Pys mapping | | | |VIM: | | <- Operational 311 | +--------------+ | | |Virtualized 2 | | Policies 312 +-------------------+ | |Pys mapping | | 313 | +--------------+ | 314 +-------------------+ 316 Figure 2: Policies within the MANO framework 318 5.2. Policies in the MANO Architecture 320 The current industry work in the area of policy for NFV is mostly 321 considered in the framework of general cloud services, and typically 322 focused on individual subsystems and addressing very specific use 323 cases or environments. For example, [ETSI-NFV-WHITE-PAPER] addresses 324 network subsystem policy for network virtualization, [ODL-GB-POLICY] 325 and [ODL-NIC-PROJECT] are open source projects in the area of network 326 policy as part of the OpenDaylight [ODL-SDN-CONTROLLER] software 327 defined networking (SDN) controller framework, [RFC3060] specifies an 328 information model for network policy, [VM-HOSTING-NET-CLUSTER] 329 focuses on placement and migration policies for distributed virtual 330 computing, [OPENSTACK-CONGRESS] is an open source project proposal in 331 OpenStack [OPENSTACK] to address policy for general cloud 332 environments. 334 A policy framework applicable to the MANO architure must consider NFV 335 services from the perspective of overall orchestration requirements 336 for services involving multiple subsystems (e.g., Figure 1 and 337 Figure 2). 339 While this document discusses policy atributes as applicable to the 340 MANO architecture, the general topic of policy has already been 341 intensively studied and documented on numerous publications over the 342 past 10 to 15 years (see [RFC3060], [POLICY-FRAMEWORK-WG], [RFC3670], 343 [RFC3198], and [CERI-DATALOG] to name a few). This document's 344 purpose is to discuss and document a policy framework applicable to 345 the MANO architecture using known policy concepts and theories to 346 address the unique requirements of NFV services including multiple 347 PoPs and networks forming hierarchical domain architectures 348 [SDN-MULTI-DOMAIN]. 350 With the above goals, this document analyses "global versus local 351 policies" (Section 5.3), a "hierarchical policy framework" 352 (Section 5.4) to address the demanding and growing requirements of 353 NFV environments, a "policy pub/sub bus in the hierarchical 354 framework" (Section 5.5), "policy intent versus subsystem actions" 355 (Section 5.6), "static versus dynamic versus autonomic policies" 356 (Section 5.7), "policy conflict detection and resolution" 357 (Section 5.8), and "soft versus hard policy constraints" 358 (Section 5.9), which can be relevant to resource management in 359 service chains [RESOURCE-MGMT-SERVICE-CHAIN]. 361 5.3. Global vs Local Policies 363 Some policies may be subsystem specific in scope, while others may 364 have broader scope and interact with multiple subsystems. For 365 example, a policy constraining certain customer types (or specific 366 customers) to only use certain server types for VNF or Virtual 367 Machine (VM) deployment would be within the scope of the compute 368 subsystem. A policy dictating that a given customer type (or 369 specific customers) must be given "platinum treatment" could have 370 different implications on different subsystems. As shown in 371 Figure 8, that "platinum treatment" could be translated to servers of 372 a given performance specification in a compute subsystem and storage 373 of a given performance specification in a storage subsystem. 375 Policies with broader scope, or global policies, would be defined 376 outside affected subsystems and enforced by a global policy engine 377 (Figure 3), while subsystem-specific policies or local policies, 378 would be defined and enforced at the local policy engines of the 379 respective subsystems. 381 Examples of sub-system policies can include thresholds for 382 utilization of sub-system resources, affinity/anti-affinity 383 constraints with regard to utilization or mapping of sub-system 384 resources for specific tasks, network services, or workloads, or 385 monitoring constraints regarding under-utilization or over- 386 utilization of sub-system resources. 388 +----------------------------------------------------------------+ 389 | +----------------------------------------------+ | 390 | | Global Policy Engine | | 391 | +----------------------------------------------+ | 392 | | 393 | +----------------------------------------------+ | 394 | | Global Policies | | 395 | +----------------------------------------------+ | 396 +----------------------------------------------------------------+ 397 ^ ^ ^ ^ 398 | | | | 399 V V V V 400 +-------------+ +-------------+ +-------------+ +-------------+ 401 |Compute | |Network | |Storage | |Whatever | 402 |Subsystem | |Subsystem | |Subsystem | |Subsystem | 403 | | | | | | | | 404 |Local Policy | |Local Policy | |Local Policy | |Local Policy | 405 |Engine | |Engine | |Engine | |Engine | 406 | | | | | | | | 407 |Local | |Local | |Local | |Local | 408 |Policies: | |Policies | |Policies | |Policies | 409 | P0, P1, | | P0, P1, | | P0, P1, | | P0, P1, | 410 | | | | | | | | 411 +-------------+ +-------------+ +-------------+ +-------------+ 413 Figure 3: Global versus Local Policy Engines 415 5.4. Hierarchical Policy Framework 417 So far, we have referenced compute, network, and storage as 418 subsystems examples. However, the following subsystems may also 419 support policy engines and subsystem specific policies: 421 o SDN Controllers, e.g., OpenDaylight [ODL-SDN-CONTROLLER]. 423 o OpenStack [OPENSTACK] components such as, Neutron, Cinder, Nova, 424 and etc. 426 o Directories, e.g., LDAP, ActiveDirectory, and etc. 428 o Applications in general, e.g., standalone or on top of 429 OpenDaylight or OpenStack. 431 o Physical and virtual network elements, e.g., routers, firewalls, 432 application delivery controllers (ADCs), and etc. 434 o Energy subsystems, e.g., OpenStack Neat [OPENSTACK-NEAT]. 436 Therefore, a policy framework may involve a multitude of subsystems. 437 Subsystems may include other lower level subsystems, e.g., Neutron 438 [OPENSTACK-NEUTRON] would be a lower level subsystem in the OpenStack 439 subsystem. In other words, the policy framework is hierarchical in 440 nature, where the policy engine of a subsystem may be viewed as a 441 higher level policy engine by lower level subsystems. In fact, the 442 global policy engine in Figure 3 could be the policy engine of a Data 443 Center subsystem and multiple Data Center subsystems could be grouped 444 in a region containing a region global policy engine. In addition, 445 one could define regions inside regions, hierarchically, as shown in 446 Figure 4. 448 Metro and wide-area network (WAN) used to interconnect data centers 449 would also be independent subsystems with their own policy engines. 451 To higher level domain 452 ^ 453 Region 1 | 454 Domain V 455 +-------------------+ +-------------------+ 456 | +---------------+ | | +---------------+ | 457 | |Region 1 Global| |<------>| |WAN 1 Global | | 458 | |Policy Engine | | | |Policy Engine | | 459 | +---------------+ | | +---------------+ | 460 | | | | 461 | +---------------+ | | +---------------+ | 462 | |Whatever | | | |Whatever | | 463 | |Subsystems | | | |Subsystems | | 464 | | | | | | | | 465 | |Local Policy | | | |Local Policy | | 466 | |Engines | | | |Engines | | 467 | +---------------+ | | +---------------+ | 468 +-------------------+ +-------------------+ 469 ^ ^ 470 | | 471 | +-------------------------+ 472 | | 473 DC 1 Domain V DC N Domain V 474 +-------------------+ +-------------------+ 475 | +---------------+ | | +---------------+ | 476 | |DC 1 Global | | | |DC N Global | | 477 | |Policy Engine | | | |Policy Engine | | 478 | +---------------+ | | +---------------+ | 479 | | | | 480 | +---------------+ | | +---------------+ | 481 | |Whatever | | | |Whatever | | 482 | |Subsystems | | | |Subsystems | | 483 | | | | | | | | 484 | |Local Policy | | | |Local Policy | | 485 | |Engines | | | |Engines | | 486 | +---------------+ | | +---------------+ | 487 +-------------------+ +-------------------+ 489 Figure 4: A Hierarchical Policy Framework 491 5.4.1. Mapping to Hierarchical Resource Orchestration 493 If the MANO framework is extended to multi layer hierarchies 494 [I-D.unify-nfvrg-recursive-programming], then a potential mapping of 495 the hierarchical policies to the MANO architecture is shown in 496 Figure 5 497 T1 T2...Tn 498 +--|--|----|---+ 499 | | 500 |Service | 501 Domain 4 | Orchestrator| 502 ************************************ +--+-----------+ 503 T1 T2...Tn *******|****************** 504 Tenant Policies -> +--|--|----|---+ | 505 | NFVO: | | 506 | Service | | <-Domain Policies 507 | Lifecycle | | 508 | Orchestrator | | 509 +-------+------+ / 510 Domain 3 | / 511 ******************** +-+---------+--+ <-Tenant Policies 512 * |NFVO: | 513 ****************** * |Rersource | <-Domain Policies 514 T1 T2...Tn * * |Orchestration | 515 +--|--|----|---+ * * +--+---+-------+ 516 | NFVO: | * ********|***|************************ 517 | Service | * / | 518 | Lifecycle | /---------/ | <-Domain Policies 519 | Orchestrator | / * | 520 +---------+----+ | * | 521 | | * | 522 +--+-------+---+* | <-Tenant Policies 523 | |* | 524 |NFVO: |* | <-Domain Policies 525 |Rersource |* | 526 |Orchestration |* | 527 +------+-------+* | 528 /|\ * *********|********** 529 +------+-------+* * +------+-------+ * <-Tenant Policies 530 +--|a/VIM: |* * |VIM: | * 531 +--|b |Resource |* * |Resource | * <-Domain Policies 532 |c | |Orchestrationn|* * |Orchestration | * 533 | | +--------------+* * +--------------+ * 534 Domain 1 * * Domain 2 * 535 ************************ * * 537 Figure 5: Policies in a Hierarchical Orchestration View 539 5.5. Policy Pub/Sub Bus 541 In [I-D.irtf-nfvrg-nfv-policy-arch] the authors argued for the need 542 of policy subsystems to subscribe to policy updates at a higher 543 policy level. A policy publication/subscription (pub/sub) bus would 544 be required as shown in Figure 6. 546 +----------------------------------------------------------------+ 547 | +----------------------------------------------+ | 548 | | Global Policy Engine | | 549 | +----------------------------------------------+ | 550 | | 551 | +----------------------------------------------+ | 552 | | Global Policies | | 553 | +----------------------------------------------+ | 554 +----------------------------------------------------------------+ 555 ^ 556 | 557 | 558 Policy Pub/Sub Bus V 559 -------------------------------------------------------------- 560 ^ ^ ^ ^ 561 | | | | 562 | | | | 563 V V V V 564 +-------------+ +-------------+ +-------------+ +-------------+ 565 |Compute | |Network | |Storage | |Whatever | 566 |Subsystem | |Subsystem | |Subsystem | |Subsystem | 567 | | | | | | | | 568 |Local Policy | |Local Policy | |Local Policy | |Local Policy | 569 |Engine | |Engine | |Engine | |Engine | 570 | | | | | | | | 571 |Local | |Local | |Local | |Local | 572 |Policies: | |Policies | |Policies | |Policies | 573 | P0, P1, | | P0, P1, | | P0, P1, | | P0, P1, | 574 | | | | | | | | 575 +-------------+ +-------------+ +-------------+ +-------------+ 577 Figure 6: A Policy Pub/Sub Bus 579 A higher tier policy engine would communicate policies to lower tier 580 policy engines using a policy pub/sub bus. Conversely, lower tier 581 policy engines would communicate their configured policies and 582 services to the higher tier policy engine using the same policy pub/ 583 sub bus. Such communications require each policy pub/sub bus to have 584 a pre-defined/pre-configured policy "name space". For example, a 585 pub/sub bus could define services using the name space "Platinum", 586 "Gold", and "Silver". A policy could then be communicated over that 587 pub/sub bus specifying a Silver service requirement. 589 In a hierarchical policy framework, a policy engine may use more than 590 one policy pub/sub bus, e.g., a policy pub/sub bus named "H" to 591 communicate with a higher tier policy engine and a policy pub/sub bus 592 named "L" to communicate with lower tier policy engines. As the name 593 spaces of policy pub/sub buses H and L may be different, the policy 594 engine would translate policies defined using the policy pub/sub bus 595 H name space to policies defined using the policy pub/sub bus L name 596 space, and vice-versa. 598 5.5.1. Pub/sub bus in the hierarchical framework 600 Figure 7 shows the Pub/sub bus in the hierarchical MANO framework. 601 Policy communications would employ a policy pub/sub bus between the 602 subsystems' policy engines in the policy hierarchy (see Section 5.4). 603 The global NFVO subsystem should have visibility into the policies 604 defined locally at each PoP to be able to detect any potential global 605 policy conflicts, e.g., a local PoP administrator could add a local 606 policy that violates or conflicts with a global policy. In addition, 607 the global NFVO subsystem would benefit from being able to import the 608 currently configured services at each PoP. The global NFVO would use 609 such information to monitor global policy conformance and also to 610 facilitate detection of policy violations when new global policies 611 are created, e.g., a global level administrator is about to add a new 612 global policy that, if committed, would make certain already 613 configured services a violation of the policy. The publication of 614 subsystem service tables for consumption by a global policy engine is 615 a concept used in the Congress [OPENSTACK-CONGRESS] OpenStack 616 [OPENSTACK] project. 618 Customers 619 | 620 V 621 +--------------+ 622 | Web Portal | 623 +--------------+ 624 ^ 625 | 626 V 627 +-----------------+ +-----------------+ 628 | OSS/BSS | | Global NFVO | 629 | +-------------+ | | +-------------+ | 630 | |OSS/BSS | | Policy | |NFVO | | 631 | |Policy Engine|<---------->|Policy Engine| | 632 | +-------------+ | | +-------------+ | 633 | | | ^ | 634 | ... | | | ... | 635 +-----------------+ +--------|--------+ 636 | 637 Policy (Pub/Sub Bus) V 638 ------------------------------------------- 639 ^ ^ ^ 640 | | | 641 +-------|-------+ +-------|-------+ +-------|-------+ 642 | PoP A | | | PoP B | | | PoP C | | 643 | V | | V | | V | 644 | +-----------+ | | +-----------+ | | +-----------+ | 645 | |Local NFVO | | | |Local NFVO | | | |Local NFVO | | 646 | | +-------+ | | | | +-------+ | | | | +-------+ | | 647 | | |Policy | | | | | |Policy | | | | | |Policy | | | 648 | | |Engine | | | | | |Engine | | | | | |Engine | | | 649 | | +-------+ | | | | +-------+ | | | | +-------+ | | 650 | +-----------+ | | +-----------+ | | +-----------+ | 651 | ^ | | ^ | | ^ | 652 | | | | | | | | | 653 | V | | V | | V | 654 | +-----------+ | | +-----------+ | | +-----------+ | 655 | |VIM | | | |VIM | | | |VIM | | 656 | | +-------+ | | | | +-------+ | | | | +-------+ | | 657 | | |Policy | | | | | |Policy | | | | | |Policy | | | 658 | | |Engine | | | | | |Engine | | | | | |Engine | | | 659 | | +-------+ | | | | +-------+ | | | | +-------+ | | 660 | +-----------+ | | +-----------+ | | +-----------+ | 661 | ... | | ... | | ... | 662 +---------------+ +---------------+ +---------------+ 664 Figure 7: Pub/sub bus in the hierarchical MANO framework 666 5.6. Policy Intent Statement versus Subsystem Actions and 667 Configurations 669 Content to be merged 671 +----------------------------------------------------------------+ 672 | Policy: "a given customer must be given Platinum treatment" | 673 +----------------------------------------------------------------+ 674 ^ ^ ^ ^ 675 | | | | 676 V V V V 677 +-------------+ +-------------+ +-------------+ +-------------+ 678 |Compute | |Network | |Storage | |Whatever | 679 |Subsystem | |Subsystem | |Subsystem | |Subsystem | 680 | | | | | | | | 681 |Policy | |Policy | |Policy | |Policy | 682 |translation: | |translation: | |translation: | |translation: | 683 | | | | | | | | 684 |Install | |Give customer| |Give customer| | ... | 685 |customer VMs | |the best QoS,| |the fastest | | | 686 |on servers | |which | |SSD storage. | | | 687 |with 3GHz | |translates | | | | | 688 |16-core Xeon | |here to set | | | | | 689 |processors, | |DHCP to xx, | | | | | 690 |and etc. | |and etc. | | | | | 691 +-------------+ +-------------+ +-------------+ +-------------+ 693 Figure 8: Example of Subsystem Translations of Policy Actions 695 5.7. Static vs Dynamic vs Autonomic Policies 697 Content to be merged 699 5.8. Policy Conflicts and Resolution 701 Content to be merged 703 5.9. Soft vs Hard Policy Constraints 705 Content to be merged 707 5.10. Operational Policies for Resource management 708 +-------------------+ 709 |NFVO | <- AAA Policy 710 | +--------------+ | 711 | |NFVO: | | 712 | |Service | | <- RD Policy 713 | |Lifecycle | | 714 | |Management | | 715 | +------+-------+ | 716 | | | 717 | +------+-------+ | 718 | |NFVO: | | 719 | |Resource | | <- RS Policy 720 | |Orchestration | | 721 | +---+--+----+--+ | 722 +------|--|----|----+ 723 / \ \ 724 / \ \ +-------+ 725 / \ ---+ VNFM | 726 / \ +---+---+ 727 / \ | 728 / \ | 729 / \ | 730 +---------+---------+ +-----+------+------+ 731 |WIM | |VIM | 732 | +--------------+ | | +--------------+ | 733 | |WIM: | | | |VIM: | | 734 | |Orchestration | | | |Orchestration | | <- RA Policy 735 | |& | | | |& | | 736 | |Optimization | | | |Optimization | | 737 | +------+-------+ | | +------+-------+ | 738 | | | | | | 739 | +------+-------+ | | +------+-------+ | 740 | |WIM: | | | |VIM: | | 741 | |Virtualized 2 | | | |Virtualized 2 | | <- RE Policy 742 | |Pys mapping | | | |Pys mapping | | 743 | +--------------+ | | +--------------+ | 744 +-------------------+ +-------------------+ 746 Figure 9: Operational policies for resource management 748 The use of NFVI resources for multiple network services can be 749 optimized in various objectives as defined in the operational 750 policies (as described in Section 5.2). 752 The operational policies can be split to different layers of NFVO and 753 VIM/WIM and they include 1) resource scheduling (RS) policy, resource 754 adaptation (RD) policy and authentication, authorization, accounting 755 (AAA) policy at NFVO, and 2) resource allocation (RA) policy and 756 resource embedding (RE) policy at VIM/WIM. They can be mapped to the 757 MANO architecture as shown in Figure 9. 759 5.10.1. Operational Policies at NFVO 761 During NS/VNF lifecycles, states of NFVI/WAN resources or the 762 performance of VNF and VL instances may vary in time (e.g., the 763 performance degradation due to incorrect placement or incorrect 764 forwarding action). Another concern for such dynamic changes is 765 fail-over as a fundamental consideration, i.e., physical resources or 766 virtualized resources in NFVI may fail during network services. 767 These dynamic changes significantly could affect the overall 768 performance for NS. Therefore, such dynamic changes triggered during 769 NS/VNF lifecycles should be coped with for guaranteeing the NS 770 performance and the optimized resource usage. Figure 9 shows that 771 NFVO needs to enforce resource adaptation (RD) policy as an 772 operational policy at NFVO. RD policy supports how NFVO adapts the 773 allocated NFVI/WAN resources (e.g., VM migration, scaling) by dealing 774 with triggered variations. RD policy engine can detect the changes 775 from measurement and diagnosis from VNFM and/or VIM/WIM. 777 Figure 9 also shows that NFVO needs to enforce resource scheduling 778 (RS) policy. RS policy determines the locations of VNF and VL 779 instances that constitute NS across multiple PoPs and WANs while 780 optimally allocating NFVI and WAN resources to the instances. 782 In particular, RD and RA policies would consider a business model 783 from OSS/BSS which specifies operational (or business) objectives 784 (e.g., overall energy consumption and NFVI resource utilization) 785 within its domain and with taking account of (on-boarded) network 786 service descriptor (NSD) as an NS policy including the virtualization 787 aspects of application feature, QoS parameters, affinity, anti- 788 affinity rules, and so on. 790 On the one hand, for the user authorization, authentication, 791 authorization, accounting (AAA) policy may be needed. Authentication 792 policy provides a way of identifying a user while the authorization 793 policy determines whether the user has the authority for virtualized 794 resources (i.e., NFVI/WAN resources) to receive the network service 795 or not. Accounting policy measures the resources the user consumes 796 during the network service. This can include the amount of system 797 time/data, and so on. 799 5.10.2. Operational Policies at VIM/WIM 801 As shown in Figure 9, RA policy supports how each subsystem (e.g., 802 compute, storage subsystem) in NFVI is allocated depending on the 803 placement information from NFVO to further optimize the resource 804 usage. Moreover, the assigned NFVI resources are embedded (or 805 allocated) to physical resources in VIM/WIM depending on states and 806 usage of resources by means of resource embedding (RE) policy as 807 shown in Figure 9. In other words, RE policy determines and 808 coordinates how the allocated virtual resources are mapped to 809 physical resources. For example, RE policy may be updated when some 810 physical resources are failed or a virtualization technique is 811 changed. 813 6. Policy-Based Resource Management Examples 815 6.1. Policy-Based Multipoint Ethernet Service 817 Content to be merged 819 6.2. Policy-Based NFV Placement 821 Content to be merged 823 6.3. Policy-Based VNF-FG Management 824 +-------------------+ 825 | NFVO | 826 | +---------------+ | 827 | |NFVO | | 828 | |RS Policy | | 829 | |Engine | | 830 | +---------------+ | 831 | ^ | 832 | | ... | 833 +---------|---------+ 834 | 835 V Allocation information 836 ----------------------------------------------- 837 ^ ^ ^ 838 | | | 839 +-------|---------+ +-------|---------+ +-------|---------+ 840 |vPoP A | | |WAN | | |vPoP B | | 841 | V | | V | | V | 842 | +-------------+ | | +-------------+ | | +-------------+ | 843 | |VIM A | | | |WIM | | | |VIM B | | 844 | | +---------+ | | | | +---------+ | | | | +---------+ | | 845 | | |RA Policy| | | | | |RA Policy| | | | | |RA Policy| | | 846 | | |Engine | | | | | |Engine | | | | | |Engine | | | 847 | | +---------+ | | | | +---------+ | | | | +---------+ | | 848 | +-------------+ | | +-------------+ | | +-------------+ | 849 | ... | | ... | | ... | 850 | | | | | | 851 | +-----+ | | | | | 852 | |VNF-A| | | | | | 853 | +-----+ | | | | | 854 | # +-----+ | | | | +-----+ | 855 | #####|VNF-B|#############################|VNF-C| | 856 | VL1 +-----+ | | VL2 | | +-----+ | 857 | | | | | | 858 +-----------------+ +-----------------+ +-----------------+ 860 ### NFP 862 Figure 10: Policy-based VNF-FG Management 864 Another subsystem example for the policy framework is VNF-FG. When 865 VNF-FGs of end-to-end network services are realized, NFVI resources 866 across multiple NFVI-PoPs and WAN resources that connect among them 867 should be allocated to the VNF-FGs. It depends on the target KPIs of 868 individual VNF and VL instances that constitute VNF-FGs. In 869 particular, in case of VNF-FG, chained performances and capabilities 870 of VNF and VL instances need to be considered together with on VL 871 instances the inter-connectivity between different NFVI-PoPs. For 872 example, if one of the VNF instances or VL instances along the VNF-FG 873 gets overloaded, the end-to-end network service may also get 874 affected. Therefore, while features of such VNF-FG are carefully 875 considered, proper operational policies for resource management (see 876 Section 5.10) are required. 878 As shown in Figure 10, consider a scenario where a user requests a 879 VNF-FG composed of "VNF A-VL 1-VNF B-VL 2-VNF C". For the VNF-FG, an 880 RA policy is enforced in which it is designed to avoid over- 881 utilization of PoP A and to reduce latency on VL 1. Therefore, NFVO 882 places VNF A, VNF B, and VL 1 on PoP A by consuming its computing and 883 network resources to achieve low latency. On the other hand, VL 2 884 and VNF C is allocated to the resources of WAN and PoP B, 885 respectively to avoid over-utilization of PoP A. 887 On the one hand, dynamic changes such as a VNF failure significantly 888 affect on the overall performance of VNF-FG since VNF-FG is a chain 889 of VNF and VL instances. Thus, such dynamic changes should be coped 890 with by RD policy for guaranteeing the VNF-FG performance and the 891 optimized resource usage. A fault management for VNF-FG based on 892 policy example is shown in Section 6.4. 894 6.4. Policy-Based Fault Management 895 +-------------------+ 896 | NFVO | 897 | +---------------+ | 898 | |NFVO | | 899 | |RS Policy | | 900 | |Engine | | 901 | +---------------+ | 902 | ^ | 903 | | ... | 904 +---------|---------+ 906 V 907 ---------------------------------------------- 908 ^ ^ ^ 909 | | | 910 +-------|---------+ +-------|---------+ +-------|---------+ 911 |vPoP A | | |vPoP B | | |vPoP C | | 912 | V | | V | | V | 913 | +-------------+ | | +-------------+ | | +-------------+ | 914 | |VIM A | | | |VIM B | | | |VIM C | | 915 | | +---------+ | | | | +---------+ | | | | +---------+ | | 916 | | |RA Policy| | | | | |RA Policy| | | | | |RA Policy| | | 917 | | |Engine | | | | | |Engine | | | | | |Engine | | | 918 | | +---------+ | | | | +---------+ | | | | +---------+ | | 919 | +-------------+ | | +-------------+ | | +-------------+ | 920 | ... | | ... | | ... | 921 | | | | | | 922 | +-----+ | | +-----+ | | +-----+ | 923 | |VNF-A|##############|VNF-B|#############|VNF-C| | 924 | +-----+ VL-1 | | +-----+ VL-2| | +-----+ | 925 | | | ^ | | | 926 | | | | | | | 927 | | | (failure) | | | 928 | | | | | | 929 +-----------------+ +-----------------+ +-----------------+ 931 ### NFP 933 Figure 11: Failure Scenario for VNF-FG 935 +------------------------------------+ 936 |NFVO | 937 | +--------------------------------+ | 938 | | RD policy engine | | 939 | |Adapts resources to the failed | | 940 | |elements while guaranteeing | | 941 | |performance | | 942 | +--------------------------------+ | 943 +------------------+-----------------+ 944 | 945 | 946 +-------------------+---------------------+ 947 |VNFM/VIM/WIM | 948 | +-------------------------------------+ | 949 | | Diagnosis / Measurement | | 950 | |A failure event | | 951 | |Throughputs of VNF and VL instances | | 952 | |Topological location... | | 953 | +-------------------------------------+ | 954 +-----------------------------------------+ 956 Figure 12: Architecture for policy-based fault management 958 As shown in Figure 11, consider a scenario that a VM related to VNF-B 959 (i.e., a VNF-B instance) is failed in the given VNF-FG composed VNF- 960 A, VNF-B, VNF-C in order. Note that the NFVI and WAN resources are 961 already allocated to the instances by RS policy. For service 962 continuity, failure of the VNF-B instance needs to be detected based 963 on diagnosis function in VIM/VNFM and the failed one needs to be 964 replaced with a new instance or to be assigned to the existing 965 instance which is available. The diagnosis and measurement function 966 may collect current performance measures and location for instances 967 as well as such a failure event. 969 +-------------------+ 970 | NFVO | 971 | +---------------+ | 972 | |NFVO | | 973 | |RD Policy | | 974 | |Engine | | 975 | +---------------+ | 976 | ^ | 977 | | ... | 978 +---------|---------+ 979 | 980 V 981 ---------------------------------------------- 982 ^ ^ ^ 983 | | | 984 +-------|---------+ +-------|---------+ +-------|---------+ 985 |vPoP A | | |vPoP B | | |vPoP C | | 986 | V | | V | | V | 987 | +-------------+ | | +-------------+ | | +-------------+ | 988 | |VIM A | | | |VIM B | | | |VIM C | | 989 | | +---------+ | | | | +---------+ | | | | +---------+ | | 990 | | |RA Policy| | | | | |RA Policy| | | | | |RA Policy| | | 991 | | |Engine | | | | | |Engine | | | | | |Engine | | | 992 | | +---------+ | | | | +---------+ | | | | +---------+ | | 993 | +-------------+ | | +-------------+ | | +-------------+ | 994 | ... | | ... | | ... | 995 | | | | | | 996 | +-----+ | | +-----+ | | +-----+ | 997 | |VNF-A| | | |VNF-B| | | |VNF-C| | 998 | +-----+ | | +-----+ | | +-----+ | 999 | # | | ^ | | # | 1000 | # New | | | | | # | 1001 | # VL-1 | | (failure) | # | 1002 | +-------+ | | | | # | 1003 | |New |#################################### | 1004 | |VNF-B | | | | | New VL-2 | 1005 | +-------+ | | | | | 1006 | | | | | | 1007 +-----------------+ +-----------------+ +-----------------+ 1009 ### NFP 1011 Figure 13: Re-instantiation for VNF-FG 1013 In the first case where a VNF instantiation is needed, a new VNF 1014 instantiation is determined by the RD policy engine in NFVO. For 1015 example, NFVO may avoid replacement of VNF B on NFVI-PoP B owing to 1016 high possibility of failure. Therefore, NFVO could instantiate VNF B 1017 on NFVI-PoP A or NFVI-PoP C with the setup of new connection points 1018 (CPs) while guaranteeing performance as shown in Figure 13. 1020 +-------------------+ 1021 | NFVO | 1022 | +---------------+ | 1023 | |NFVO | | 1024 | |RD Policy | | 1025 | |Engine | | 1026 | +---------------+ | 1027 | ^ | 1028 | | ... | 1029 +---------|---------+ 1030 | 1031 V 1032 ---------------------------------------------- 1033 ^ ^ ^ 1034 | | | 1035 +-------|---------+ +-------|---------+ +-------|---------+ 1036 |vPoP A | | |vPoP B | | |vPoP C | | 1037 | V | | V | | V | 1038 | +-------------+ | | +-------------+ | | +-------------+ | 1039 | |VIM A | | | |VIM B | | | |VIM C | | 1040 | | +---------+ | | | | +---------+ | | | | +---------+ | | 1041 | | |RA Policy| | | | | |RA Policy| | | | | |RA Policy| | | 1042 | | |Engine | | | | | |Engine | | | | | |Engine | | | 1043 | | +---------+ | | | | +---------+ | | | | +---------+ | | 1044 | +-------------+ | | +-------------+ | | +-------------+ | 1045 | ... | | ... | | ... | 1046 | | | | | | 1047 | +-----+ | | +-----+ | | +-----+ | 1048 | |VNF-A| | | |VNF-B| | | |VNF-C| | 1049 | +-----+ | | +-----+ | | +-----+ | 1050 | # | | ^ | | # | 1051 | # | | | | | # | 1052 | # New | | (failure) | | # | 1053 | # VL-1 | | | | # New | 1054 | # | | +-------+ | | # VL-2 | 1055 | ##################| VNF-B1|############### | 1056 | | | +-------+ | | | 1057 +-----------------+ +-----------------+ +-----------------+ 1059 ### NFP 1061 Figure 14: No Re-instantiation for VNF-FG 1063 In the second case where no VNF instantiation is needed since a 1064 redundant VNF exists, the available VNF-B instance can used by the 1065 VNF-FG. For example, a redundant VNF B instance exists in NFVI-PoP 1066 B. Therefore, NFVO selects the instance and re-constructs two VLs as 1067 shown in Figure 14, and the corresponding NS can be continued without 1068 re-instantiation. 1070 7. Implementation Examples 1072 tbd 1074 8. Gaps and Open Questions 1076 tbd 1078 9. Conclusions 1080 tbd 1082 9.1. Relation to other IETF/IRTF activities 1084 tbd 1086 10. Acknowledgements 1088 The research leading to some of the results described in this 1089 document has received funding from the European Union Seventh 1090 Framework Programme (FP7/2007-2013) under grant agreement no. 619609 1091 - the UNIFY project. The views expressed here are those of the 1092 authors only. The European Commission is not liable for any use that 1093 may be made of the information in this document. 1095 11. Contributors 1097 This document is the result of merging multiple drafts. This section 1098 acknowledges those who provided important ideas and text into this 1099 document. 1101 o Z. Qiang (Ericsson), M. Kind (DT-AG) from 1102 [I-D.unify-nfvrg-recursive-programming] 1104 o R. Krishnan (Dell), D. Lopez (Telefonica) and S. Wright (AT&T) 1105 from [I-D.irtf-nfvrg-nfv-policy-arch] 1107 o S. Lee (ETRI), S. Pack (KU), M-K. Shin (ETRI) and E. Paik (KT) 1108 from [I-D.irtf-nfvrg-resource-management-service-chain] 1110 12. IANA Considerations 1112 tbd 1114 13. Security Considerations 1116 tbd 1118 14. References 1120 14.1. Normative References 1122 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1123 Requirement Levels", BCP 14, RFC 2119, 1124 DOI 10.17487/RFC2119, March 1997, 1125 . 1127 [RFC3060] Moore, B., Ellesson, E., Strassner, J., and A. Westerinen, 1128 "Policy Core Information Model -- Version 1 1129 Specification", RFC 3060, DOI 10.17487/RFC3060, February 1130 2001, . 1132 [RFC3198] Westerinen, A., Schnizlein, J., Strassner, J., Scherling, 1133 M., Quinn, B., Herzog, S., Huynh, A., Carlson, M., Perry, 1134 J., and S. Waldbusser, "Terminology for Policy-Based 1135 Management", RFC 3198, DOI 10.17487/RFC3198, November 1136 2001, . 1138 [RFC3670] Moore, B., Durham, D., Strassner, J., Westerinen, A., and 1139 W. Weiss, "Information Model for Describing Network Device 1140 QoS Datapath Mechanisms", RFC 3670, DOI 10.17487/RFC3670, 1141 January 2004, . 1143 14.2. Informative References 1145 [CERI-DATALOG] 1146 Ceri, S. and others, "What you always wanted to know about 1147 Datalog (and never dared to ask", IEEE Transactions on 1148 Knowledge and Data Engineering, (Volume: 1, Issue: 1), 1149 August 2002. 1151 [ETSI-NFV-MANO] 1152 ETSI, "Network Function Virtualization (NFV) Management 1153 and Orchestration V0.6.3", October 2014. 1155 [ETSI-NFV-PER001] 1156 ETSI, "Network Function Virtualization: Performance and 1157 Portability Best Practices v1.1.1", June 2014. 1159 [ETSI-NFV-TERM] 1160 ETSI, "NFV Terminology for Main Concepts in NFV", October 1161 2013. 1163 [ETSI-NFV-WHITE-PAPER] 1164 ETSI NFV White Paper, "Network Functions Virtualisation, 1165 An Introduction, Benefits, Enablers, Challenges, & Call 1166 for Action", 1167 . 1169 [I-D.ietf-bmwg-virtual-net] 1170 Morton, A., "Considerations for Benchmarking Virtual 1171 Network Functions and Their Infrastructure", draft-ietf- 1172 bmwg-virtual-net-04 (work in progress), August 2016. 1174 [I-D.irtf-nfvrg-nfv-policy-arch] 1175 Figueira, N., Krishnan, R., Lopez, D., Wright, S., and D. 1176 Krishnaswamy, "Policy Architecture and Framework for NFV 1177 Infrastructures", draft-irtf-nfvrg-nfv-policy-arch-04 1178 (work in progress), September 2016. 1180 [I-D.irtf-nfvrg-resource-management-service-chain] 1181 Lee, S., Pack, S., Shin, M., Paik, E., and R. Browne, 1182 "Resource Management in Service Chaining", draft-irtf- 1183 nfvrg-resource-management-service-chain-03 (work in 1184 progress), March 2016. 1186 [I-D.liu-bmwg-virtual-network-benchmark] 1187 Liu, V., Liu, D., Mandeville, B., Hickman, B., and G. 1188 Zhang, "Benchmarking Methodology for Virtualization 1189 Network Performance", draft-liu-bmwg-virtual-network- 1190 benchmark-00 (work in progress), July 2014. 1192 [I-D.norival-nfvrg-nfv-policy-arch] 1193 Figueira, N., Krishnan, R., Lopez, D., and S. Wright, 1194 "Policy Architecture and Framework for NFV 1195 Infrastructures", draft-norival-nfvrg-nfv-policy-arch-04 1196 (work in progress), June 2015. 1198 [I-D.unify-nfvrg-recursive-programming] 1199 Szabo, R., Qiang, Z., and M. Kind, "Towards recursive 1200 virtualization and programming for network and cloud 1201 resources", draft-unify-nfvrg-recursive-programming-02 1202 (work in progress), October 2015. 1204 [ODL-GB-POLICY] 1205 "OpenDaylight Group Based Policy", 1206 . 1209 [ODL-NIC-PROJECT] 1210 "OpenDaylight Network Intent Composition Project", 1211 . 1214 [ODL-SDN-CONTROLLER] 1215 "OpenDaylight SDN Controller", 1216 . 1218 [OPENSTACK] 1219 "OpenStack", . 1221 [OPENSTACK-CONGRESS] 1222 "OpenStack Congress", . 1225 [OPENSTACK-NEAT] 1226 "OpenStack Neat", . 1228 [OPENSTACK-NEUTRON] 1229 "OpenStack Neutron", . 1232 [POLICY-FRAMEWORK-WG] 1233 "Policy Framework Working Group (IETF)", 1234 . 1236 [RESOURCE-MGMT-SERVICE-CHAIN] 1237 Lee, S. and others, "Resource Management in Service 1238 Chaining", . 1241 [SDN-MULTI-DOMAIN] 1242 Figueira, N. and R. Krishnan, "SDN Multi-Domain 1243 Orchestration and Control: Challenges and Innovative 1244 Future Directions", IEEE International Conference on 1245 Computing (ICNC), February 2015. 1247 [VM-HOSTING-NET-CLUSTER] 1248 Grit, L. and others, "Virtual Machine Hosting for 1249 Networked Clusters: Building the Foundations for 1250 "Autonomic" Orchestration", Virtualization Technology in 1251 Distributed Computing (VTDC), 2006. 1253 Authors' Addresses 1255 Robert Szabo (editor) 1256 Ericsson 1257 Konyves Kaman krt. 11 1258 Budapest, EMEA 1097 1259 Hungary 1261 Phone: +36703135738 1262 Email: robert.szabo@ericsson.com 1264 Seungik Lee (editor) 1265 ETRI 1266 218 Gajeong-ro Yuseung-Gu 1267 Daejeon 305-700 1268 Korea 1270 Phone: +82 42 860 1483 1271 Email: seungiklee@etri.re.kr 1273 Norival Figueira 1274 Brocade 1276 Email: nfigueir@Brocade.com