idnits 2.17.1 draft-lachosrothenberg-alto-brokermdo-02.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document seems to lack the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords. (The document does seem to have the reference to RFC 2119 which the ID-Checklist requires). -- The document date (March 11, 2019) is 1844 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- -- Looks like a reference, but probably isn't: '1' on line 792 -- Looks like a reference, but probably isn't: '2' on line 795 -- Looks like a reference, but probably isn't: '3' on line 798 -- Looks like a reference, but probably isn't: '4' on line 801 == Missing Reference: 'TBD' is mentioned on line 534, but not defined == Unused Reference: 'RFC2119' is defined on line 676, but no explicit reference was found in the text == Outdated reference: A later version (-08) exists of draft-li-sfc-hhsfc-05 == Outdated reference: A later version (-24) exists of draft-ietf-alto-unified-props-new-03 == Outdated reference: A later version (-25) exists of draft-ietf-alto-path-vector-03 Summary: 0 errors (**), 0 flaws (~~), 7 warnings (==), 5 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 ALTO WG D. Lachos 3 Internet-Draft C. Rothenberg 4 Intended status: Informational Unicamp 5 Expires: September 12, 2019 March 11, 2019 7 ALTO-based Broker-assisted Multi-domain Orchestration 8 draft-lachosrothenberg-alto-brokermdo-02 10 Abstract 12 Evolving networking scenarios (e.g., 5G) demand new multiple 13 administrative domain (aka multi-domain) orchestration models. This 14 document proposes a new broker-plane approach working on top of per- 15 domain management and orchestration functions to coordinate the 16 delivery of a multi-operator End-to-End Network Service (E2ENS). 17 This proposed design resorts to the Application-Layer Traffic 18 Optimization (ALTO) protocol to offer topology and resource 19 information from different administrative domains. The ALTO services 20 with the proposed protocol extension offer aggregated views on 21 various types of resources contributing to a more simple and scalable 22 solution. 24 Status of This Memo 26 This Internet-Draft is submitted in full conformance with the 27 provisions of BCP 78 and BCP 79. 29 Internet-Drafts are working documents of the Internet Engineering 30 Task Force (IETF). Note that other groups may also distribute 31 working documents as Internet-Drafts. The list of current Internet- 32 Drafts is at https://datatracker.ietf.org/drafts/current/. 34 Internet-Drafts are draft documents valid for a maximum of six months 35 and may be updated, replaced, or obsoleted by other documents at any 36 time. It is inappropriate to use Internet-Drafts as reference 37 material or to cite them other than as "work in progress." 39 This Internet-Draft will expire on September 12, 2019. 41 Copyright Notice 43 Copyright (c) 2019 IETF Trust and the persons identified as the 44 document authors. All rights reserved. 46 This document is subject to BCP 78 and the IETF Trust's Legal 47 Provisions Relating to IETF Documents 48 (https://trustee.ietf.org/license-info) in effect on the date of 49 publication of this document. Please review these documents 50 carefully, as they describe your rights and restrictions with respect 51 to this document. Code Components extracted from this document must 52 include Simplified BSD License text as described in Section 4.e of 53 the Trust Legal Provisions and are provided without warranty as 54 described in the Simplified BSD License. 56 Table of Contents 58 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 59 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3 60 3. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 61 4. Problem Statement and Challenges . . . . . . . . . . . . . . 5 62 5. Proposed Approach . . . . . . . . . . . . . . . . . . . . . . 6 63 5.1. Inter-domain Resource (IdR) Component . . . . . . . . . . 7 64 5.2. Inter-domain Topology (IdT) Component . . . . . . . . . . 7 65 5.3. ALTO Server Functionalities . . . . . . . . . . . . . . . 7 66 5.4. Filtered Cost Map Extension . . . . . . . . . . . . . . . 8 67 5.4.1. Accept Input Parameters . . . . . . . . . . . . . . . 8 68 5.4.2. Response . . . . . . . . . . . . . . . . . . . . . . 9 69 5.5. Examples of Message Exchange . . . . . . . . . . . . . . 9 70 5.5.1. Property Map Service . . . . . . . . . . . . . . . . 9 71 5.5.2. Filtered Cost Map Service . . . . . . . . . . . . . . 10 72 6. Discussion . . . . . . . . . . . . . . . . . . . . . . . . . 13 73 6.1. Benefits . . . . . . . . . . . . . . . . . . . . . . . . 14 74 6.2. Open Issues . . . . . . . . . . . . . . . . . . . . . . . 14 75 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 15 76 8. Security Considerations . . . . . . . . . . . . . . . . . . . 15 77 9. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 15 78 10. References . . . . . . . . . . . . . . . . . . . . . . . . . 15 79 10.1. Normative References . . . . . . . . . . . . . . . . . . 15 80 10.2. Informative References . . . . . . . . . . . . . . . . . 16 81 10.3. URIs . . . . . . . . . . . . . . . . . . . . . . . . . . 18 82 Appendix A. Proof of Concept Use Case Implementation . . . . . . 18 83 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 22 85 1. Introduction 87 Envisioned 5G network architectures and related service models 88 consider broader cooperation between stakeholders in order to provide 89 flexible multi-operator multi-domain services. These multi-provider 90 orchestration operations will require the information exchange across 91 Multi-domain Orchestrators (MdOs). The key information to be 92 exchanged between MdOs includes the abstract network topology, 93 resource availability (e.g., CPUs, Memory, and Storage) and 94 capability (e.g., supported network functions). 96 This document presents a federation networking paradigm where a 97 broker-plane works on top of the management and orchestration plane 98 to assist and coordinate the creation of an End-to-End Network 99 Service (E2ENS) spanning over multi-operator multi-domain networks. 100 Our design resorts to the Application-Layer Traffic Optimization 101 (ALTO) protocol [RFC7285] to address the lack of abstractions to 102 discover and adequately represent in confidentiality-preserving 103 fashion the resource and topology information from different 104 administrative domains. Moreover, this draft introduces an extension 105 to the ALTO base protocol for inter-domain connectivity information 106 discovery. 108 2. Terminology 110 We use the following definitions, as established in [ETSI-NFV-DEF]: 112 Administrative Domain: Collection of systems and networks operated 113 by a single organization or administrative authority. 115 Network Function (NF): Functional block within a network 116 infrastructure that has well-defined external interfaces and well- 117 defined functional behaviour. 119 Network Functions Virtualisation (NFV): The principle of separating 120 network functions from the hardware they run on by using virtual 121 hardware abstraction. 123 NF Forwarding Graph: (NFFG): Graph of logical links connecting NF 124 nodes for the purpose of describing traffic flow between these 125 network functions. 127 Network Service Orchestration (NSO): Function responsible for 128 network service lifecycle management. 130 Resource Orchestration (RO): Function responsible for global 131 resource management governance. 133 Our proof of concept implementation follows the architectural 134 proposal of the 5GEx project [H2020.5GEX]. Some additional 5GEx 135 terms commonly used in this document are defined below: 137 Domain Orchestrator (DO): Performs Resource Orchestration and/or 138 Service Orchestration within the same administrative domain. 140 Multi-domain Orchestrator (MdO): Coordinates resource and/or service 141 orchestration at multi-domain level, where multi-domain may refer 142 to multiple DOs or multiple administrative domains. 144 Resource Topology (RT): Functional module that is responsible for 145 keeping an updated global view of the underlying infrastructure 146 topology exposed by DOs. 148 Service Graph (SG): A high-level data model for defining flexible 149 network services (including traffic steering primitives). 151 Service Access Point (SAP): A named/tagged port supporting stitching 152 (service to service, domain to domain, etc.) 154 3. Scope 156 Envisioned 5G scenarios are expected to work not only with 157 heterogeneous technologies but also across different network 158 operators. Many ongoing standardization activities and research 159 projects are addressing the multi-provider multi-domain orchestration 160 challenges under different approaches. 162 For example, within the IETF, [RFC8459] proposes a hierarchical 163 Service Function Chaining (SFC) for multiple domains under the same 164 administrative entity, and the document "Hybrid Hierarchical Multi- 165 Domain SFC [DRAFT-HHSFC] describes SFC crossing different domains 166 owned by various organizations or by a single organization with 167 administration partitions. In the NFVRG, the draft "Multi-domain 168 Network Virtualization" [DRAFT-MD-VIRT] envisions a complete E2E 169 logical network as stitching services offered by multiple domains 170 from multiple providers. Another initiative is the ETSI Industry 171 Specification Group for Network Functions Virtualization (ETSI NFV 172 ISG), the document [ETSI-NFV-IFA028] reports different NFV MANO 173 architectural approaches with use cases related to network services 174 provided using multiple administrative domains. 176 In case of research projects, [H2020.5GEX] [H2020-5G-TRANSFORMER] 177 seek to integrate multiple administrations and technologies through 178 the collaboration between operations. Other studies are envisioned 179 to use a centralized approach, where each domain advertises its 180 capabilities to a federation layer which will act as a 181 broker [VITAL][T-NOVA]. The proposed architecture in [ICAF] allows 182 the creation of cloud services from different administrative domains, 183 however, it is not related to the provisioning of NFV-based cross- 184 domain network services. 186 All such proposals described above envision the potential 187 introduction of new business model approaches, including federation 188 models [PPP-5:2013] among administrative domains. In this context, 189 this document considers each network operator involved in the 190 community advertises its abstracted capabilities (e.g., software/ 191 hardware resources, physical/virtual network functions, etc.) to a 192 broker (i.e., 3rd party). This broker, in its turn, provides or 193 assists coordinate E2E network services spanning multi-domain 194 networks. 196 4. Problem Statement and Challenges 198 The provision of a complete E2E network service requires chaining 199 services provided by multiple network operators with multiple 200 technologies. In this multi-domain environment, the orchestration 201 process will require an advertisement mechanism through which 202 individual domains can describe their capabilities, resources, and 203 VNFs in an interoperable manner. Moreover, a discovery mechanism is 204 also necessary so that source domains can obtain candidate domains 205 (with the corresponding connectivity information) which can provide a 206 part of the service and/or slice in an E2ENS requirement. 208 In order that the advertising and discovery process works in a proper 209 way, a number of challenges can be identified: 211 Lack of Abstractions: Multiple vendors with heterogeneous 212 technologies need an information model to adequately represent 213 in confidentiality-preserving fashion the resource and topology 214 information. 216 Scalability: Involves the distribution of topology and resource 217 information in a peer-to-peer fashion (MdO-to-MdO). Multi- 218 operator multi-domain environments where the information 219 distribution is advertised in a peer-to-peer model scales 220 linearly. It means more MdO interconnections one has, the more 221 it "costs" to distribute. 223 Flexibility: Considers that a distributed approach does not allow 224 domains without physical infrastructure (e.g., without BGP or 225 BGP-LS) to advertise resource capabilities and networking 226 resources. Such procedures consist in deploying and configuring 227 physical peering points for these domains. 229 Complexity: Refers to the discovery mechanism to pre-select 230 candidate domains, accounting for resources and capabilities, 231 necessary for an E2E network service deployment. An intrinsic 232 complexity exists in the process of assembling, logically 233 organizing, and enabling abstraction views of different 234 resources and capabilities in multi-domain scenarios. 236 5. Proposed Approach 238 The primary design goal for ALTO-based Broker-assisted Multi-domain 239 Orchestration is to discover resource and topology information from 240 different administrative domains involved in the federation, while 241 also safeguarding the privacy and autonomy of every domain. 243 In the architectural proposal shown in Figure 1, a broker component 244 is conceived to be working as coordinator of a set of MdOs. In 245 particular, the broker-assisted design consists of the following key 246 components: (i) Inter-domain Resource (IdR), (ii) Inter-domain 247 Topology (IdT), and (iii) ALTO Server. 249 BROKER COMPONENT 250 +--------------------------------------------+ 251 | | 252 | +-----------------+ | 253 | | | | 254 XXXXXXXXXXXXXXXXXXX ALTO SERVER(s) | | 255 X | | + | 256 X | +----------------+\ | 257 X | / \ | 258 X | / \ | 259 X | +------+/+-------+ +---------------+ | 260 X | | Inter-domain | | Inter-domain | | 261 X | | Topology (IdT) | | Resource (IdR)| | 262 X | +-^------^-------+ +---^-------^---+ | 263 X | . . * * | 264 X +----.------.-----------------*-------*------+ 265 X . . * * 266 X . . * * 267 +--X--------.---+********************* * 268 | | . * 269 | | .............+------------*---+ 270 | MdO1 | | | 271 | +<------------->+ +---+ 272 +---------------+ | MdO2 | | 273 | | | 274 Legend: +-+--------------+ | 275 XXX ALTO Protocol | | 276 ... BGP/BGP-LS/REST | MdO(n) | 277 *** UNIFY/TOSCA/ETSI-NFV +------------------+ 279 Figure 1: Broker-assisted Multi-operator Network Architecture 281 5.1. Inter-domain Resource (IdR) Component 283 It creates a hierarchical database that contains inter-domain 284 resource information such as resource availability (i.e., CPU, 285 memory, and storage), Virtual Network Functions (VNFs) and Physical 286 Network Functions (PNFs) supported and Service Access Points (SAPs) 287 to access those resources. UNIFY [UNIFY.NFFG], TOSCA [TOSCA], ETSI- 288 NFV [ETSI-NFV-MANO], among other data models can be used to create 289 the interface between IdR and MdOs. 291 5.2. Inter-domain Topology (IdT) Component 293 A hierarchical TED (Traffic Engineering Database) that contains 294 inter-domain network topology information including additional key 295 parameters (e.g., throughput and latency of links). This information 296 can be retrieved from each MdO through BGP-LS or REST interfaces. 298 5.3. ALTO Server Functionalities 300 The ALTO server component is the core of the broker layer. Multiple 301 logically centralized ALTO servers use the information collected from 302 IdR and IdT components to create and provide abstract maps with a 303 simplified view, yet enough information about MdOs involved in the 304 federation. This information includes domain-level topology, 305 resource availability (i.e., CPU, memory, and storage), PNF/VNF 306 capabilities, and SAPs. 308 As an ALTO client, each MdO sends ALTO service queries to the ALTO 309 server. This server provides aggregated inter-domain information 310 exposed as set ALTO base services defined in [RFC7285], e.g., Network 311 Map, Cost Map and ALTO extension services, e.g., Property 312 Map [DRAFT-PM], Multi-Cost Map [RFC8189], Path Vector [DRAFT-PV]. 314 For example, when a source MdO receives a customer service request, 315 it checks whether or not it can deliver the full service. If it is 316 unable to do so, the MdO consumes from the ALTO Server the Property 317 Map service to have a clear global view of the resource information 318 offered by other MdOs. This information allows discovering which 319 candidate MdOs may be contacted to deliver the remaining requirements 320 of a requested end-to-end service deployment. The connectivity 321 information among discovered MdOs can be retrieved by a Cost Map 322 service, responding, for instance, a path vector with the AS-level 323 topology distance between the source MdO and candidate MdOs. 325 5.4. Filtered Cost Map Extension 327 The ALTO server MUST provide connectivity information for every SG 328 link in the SG path for an E2E requirement. This information is the 329 AS-level topology distance in the form of path vector, and it 330 includes all possible ways for each (source node, destination node) 331 pair in the SG link. 333 In this section, we introduce a non-normative overview of the 334 Filtered Cost Map defined in Section 6.1 of [DRAFT-PV] [1]. 336 The specifications for the "Media Types", "HTTP method", 337 "Capabilities" and "Uses" (described in Section 6.1 of [DRAFT-PV] 338 [2]) are unchanged. 340 5.4.1. Accept Input Parameters 342 The ReqFilteredCostMap object in Section 6.1.2 of [DRAFT-PV] [3] is 343 extended as follow: 345 object { 346 NFFG sg; 347 } ReqFilteredCostMap; 349 object { 350 JSONString nfs<1..*>; 351 JSONString saps<1..*>; 352 NextHops sg_links<1..*>; 353 REQs reqs<1..*>; 354 } NFFG; 356 object { 357 JSONNumber id; 358 JSONString src-node; 359 JSONString dst-node; 360 } NextHops; 362 object { 363 JSONString id; 364 JSONString src-node; 365 JSONString dst-node; 366 JSONNumber sg-path<1..*>; 367 } REQs; 369 sg: If present, the ALTO Server MUST allow the request input to 370 include an SG with a formatted body as an NFFG object. An NFFG 371 object contains NFs, SAPs, SG links representing logical 372 connections between NFs, SAPs or both and E2E requirements as a 373 list of ids of SG links. 375 It is worth noting that further versions of this draft will define a 376 more elaborated NFFG object to support extended parameters such as 377 monitoring parameters, resource requirements, etc. 379 5.4.2. Response 381 If the ALTO client includes the path vector cost mode in the "cost- 382 type" or "multi-cost-types" field of the input parameter, the 383 response for each SG link in each E2E requirement MUST be encoded as 384 a JSONArray of JSONArrays of JSONStrings. Anyone of the sub-arrays 385 indicates a potential candidate path calculated as the per-domain 386 topological distance corresponding to the amount of traversing 387 domains. 389 Moreover, as defined in Section 6.3.6 of [DRAFT-PV] [4], If an ALTO 390 client sends a request of the media type "application/alto- 391 costmapfilter+json" and accepts "multipart/related", the ALTO server 392 MUST provide path vector information along with the associated 393 Property Map information (e.g., entry points of the corresponding 394 foreign domains), in the same body of the response. 396 Section 5.5.2 gives an example of the Filtered Cost Map query and the 397 corresponding responses. 399 5.5. Examples of Message Exchange 401 This section list a couple of examples of the Property Map and 402 Filtered Cost Map queries and the corresponding responses. These 403 responses are based on the information in Table 1 and Table 2 of a 404 use case implementation described in Appendix A. 406 5.5.1. Property Map Service 408 In this example, the ALTO client wants to retrieve the entire 409 Property Map for PID entities with the "entry-point", "cpu", "mem", 410 "storage", "port" and "nf" properties. 412 o HTTP Request: 414 GET /propmap/full/inet-ucmspn HTTP/1.1 415 Host: alto.example.com 416 Accept: application/alto-propmap+json,application/alto-error+json 418 o HTTP Response: 420 HTTP/1.1 200 OK 421 Content-Length: ### 422 Content-Type: application/alto-propmap+json 423 { 424 "property-map": { 425 "pid:AS1": { 426 "entry-point": [ "http://172.25.0.10:8888/escape" ], 427 "cpu": [ "50.0" ], 428 "mem": [ "60.0" ], 429 "storage": [ "70.0" ], 430 "port": [ "SAP1" ], 431 "nf": [ "NF1", "NF3" ] 432 }, 433 "pid:AS2": { 434 "entry-point": [ "http://172.26.0.10:8888/escape" ], 435 "cpu": [ "10.0" ], 436 "mem": [ "20.0" ], 437 "storage": [ "30.0" ], 438 "nf": [ "NF2" ] 439 }, 440 "pid:AS3": { 441 "entry-point": [ "http://172.27.0.10:8888/escape" ], 442 "cpu": [ "80.0" ], 443 "mem": [ "90.0" ], 444 "storage": [ "100.0" ], 445 "port": [ "SAP2" ], 446 "nf": [ "NF1", "NF3" ] 447 } 448 } 449 } 451 5.5.2. Filtered Cost Map Service 453 The following example uses the Filtered Cost Map service to request 454 the path vector for a given E2E requirement. The SG request 455 information in Table 2 is used to describe the service, and it is 456 composed of three NFs (NF1, NF2, and NF3) and two SAPs (SAP1 and 457 SAP2). Links connecting the NFs and SAPs ("sg_links" tag) are also 458 included, followed by an E2E requirement ("reqs" tag) with 459 information about the order in which NFs are traversed from SAP1 to 460 SAP2. 462 Note that the request accepts "multipart/related" media type. This 463 means the ALTO server will include associated property information in 464 the same response. 466 o HTTP Request: 468 POST /costmap/pv HTTP/1.1 469 Host: alto.example.com 470 Accept: multipart/related, application/alto-costmap+json, 471 application/alto-propmap+json, application/alto-error+json 472 Content-Length: [TBD] 473 Content-Type: application/alto-costmapfilter+json 475 { 476 "cost-type": { 477 "cost-mode": "array", 478 "cost-metric": "ane-path" 479 }, 480 "sg": { 481 "nfs": [ "NF1", "NF2", "NF3" ], 482 "saps": [ "SAP1", "SAP2" ], 483 "sg_links":[ 484 { 485 "id": 2, 486 "src-node": "SAP1", 487 "dst-node": "NF1", 489 }, 490 { 491 "id": 2, 492 "src-node": "NF1", 493 "dst-node": "NF2", 494 }, 495 { 496 "id": 3, 497 "src-node": "NF2", 498 "dst-node": "NF3", 499 }, 500 { 501 "id": 4, 502 "src-node": "NF3", 503 "dst-node": "SAP2", 504 } 506 ], 507 "reqs": [ 508 { 509 "id": 1, 510 "src-node": "SAP1", 511 "dst-node": "SAP2", 512 "sg-path": [ 1, 2, 3, 4 ] 513 } 514 ] 515 } 516 } 518 o HTTP Response: The ALTO server returns connectivity information 519 for the E2E requirement provided by the ALTO Client request of the 520 above example. 522 For each SG link in the E2E requirement (SAP1->NF1, NF1->NF2, 523 NF2->NF3, NF3->SAP2), the ALTO server returns sub-arrays 524 indicating potential candidate paths calculated as the AS-level 525 topological distance corresponding to the amount of traversing 526 domains. 528 Also, the response includes Property Map information for each 529 element in the path vector. In this case, it is retrieved a 530 Property Map with the "entry-point" property, i.e., the URL of the 531 MdO entry point for the corresponding network. 533 HTTP/1.1 200 OK 534 Content-Length: [TBD] 535 Content-Type: multipart/related; boundary=example 537 --example 538 Content-Type: application/alto-endpointcost+json 540 { 541 "meta": { 542 "cost-type": { 543 "cost-mode": "array", 544 "cost-metric": "ane-path" 545 }, 546 }, 548 "cost-map": { 549 "SAP1": { 550 "SAP2": { 551 "SAP1": { 552 "NF1": [ 553 [ "AS1" ], [ "AS1", "AS2", "AS3" ] 554 ] 555 }, 556 "NF1": { 557 "NF2": [ 558 [ "AS1", "AS2" ], [ "AS3", "AS2" ] 559 ] 560 }, 561 "NF2": { 562 "NF3": [ 563 [ "AS2", "AS1" ], [ "AS2", "AS3" ] 564 ] 565 }, 566 "NF3": { 567 "SAP2": [ 568 [ "AS1", "AS2", "AS3" ], [ "AS3" ] 569 ] 570 } 571 } 572 } 573 } 574 } 576 --example 577 Content-Type: application/alto-propmap+json 579 { 580 "property-map": { 581 "pid:AS1": { "entry-point": "http://172.25.0.10:8888/escape" }, 582 "pid:AS2": { "entry-point": "http://172.26.0.10:8888/escape" }, 583 "pid:AS3": { "entry-point": "http://172.27.0.10:8888/escape" } 584 } 585 } 587 --example-- 589 6. Discussion 591 In this section, we analyze the benefits and open issues in our 592 broker-assisted architecture. 594 6.1. Benefits 596 The broker-assisted orchestration has numerous benefits, such as: 598 o Avoid the distribution of topology and resource information in a 599 peer-to-peer fashion (MdO-to-MdO) 601 o The (abstracted) information and offered resources, services are 602 maintained in each local MdO. 604 o Allow domains without physical infrastructure (hence without BGP 605 or BGP-LS) to advertise their capabilities. 607 o An ALTO-based privacy-preserving information model to provide 608 computing, storage, and networking resource info. 610 o An MdO discovery method to determine the underlying network graph 611 and a potential set of paths before bilateral negotiation between 612 MdOs is started. 614 6.2. Open Issues 616 Although the broker-assisted information exchange has several 617 advantages, it also raises some questions which we try to answer from 618 our lessons learned. 620 o What kind of organization will manage and support the operation of 621 a broker entity? If a broker is used to exchange information, 622 then how does one ensure that the data delivered amongst the 623 operators by this 3rd party has not been changed? 625 * The broker entity must be trusted by each operator since it 626 stores and handles sensitive information. For example, future 627 deployment of SDN at IXPs can be used as a trusted third-party 628 platform to support rich business models between different 629 operators [DRAFT-HHSFC]. 631 o In the case of peer-to-peer information exchange model, an MdO 632 failure concerns only the domain where the failure occurs, other 633 peers can perform the information exchange without any limitation. 634 However, If any error occurs in the broker entity the information 635 exchange among all involved ASes will be impacted. How avoid this 636 single point of failure? 638 * The broker entity maintains a centralized database. Local 639 restoration/replication options may be applied. 641 o The MdO information exchange depends on the policies. Operators 642 have a preference to share a different view about its compute and 643 network resources towards different operators. For example, a 644 detailed view for the operators that are belonging to same 645 operator group and a high-level information towards the other 646 operators. How is the fine-grained/coarse-grained information 647 exchange handled?. 649 * It requires much more complex database handling and information 650 exchange with the MdOs depending on the policies. 652 7. IANA Considerations 654 This document includes no request to IANA. 656 8. Security Considerations 658 TBD. 660 9. Acknowledgments 662 This work is supported by the Innovation Center of Ericsson S.A., 663 Brazil (grant agreement UNI.64). 665 Thank you to Robert Szabo (Ericsson Research, Hungary) for the 666 contribution and substantial feedback and suggestions in this 667 document. 669 Many thanks to Richard Yang, Dawn Chan, Jensen Zhang, Shawn Lin, Qiao 670 Xiang, Sabine Randriamasy for their feedback on this draft. 672 10. References 674 10.1. Normative References 676 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 677 Requirement Levels", BCP 14, RFC 2119, March 1997, 678 . 680 [RFC7285] Alimi, R., Ed., Penno, R., Ed., Yang, Y., Ed., Kiesel, S., 681 Previdi, S., Roome, W., Shalunov, S., and R. Woundy, 682 "Application-Layer Traffic Optimization (ALTO) Protocol", 683 RFC 7285, DOI 10.17487/RFC7285, September 2014, 684 . 686 [RFC8189] Randriamasy, S., Roome, W., and N. Schwan, "Multi-Cost 687 Application-Layer Traffic Optimization (ALTO)", RFC 8189, 688 DOI 10.17487/RFC8189, October 2017, 689 . 691 [RFC8459] Dolson, D., Homma, S., Lopez, D., and M. Boucadair, 692 "Hierarchical Service Function Chaining (hSFC)", RFC 8459, 693 DOI 10.17487/RFC8459, September 2018, 694 . 696 10.2. Informative References 698 [DRAFT-HHSFC] 699 Li, G., Li, G., Li, T., Xu, Q., and H. Zhou, "Hybrid 700 Hierarchical Multi-Domain Service Function chaining", 701 draft-li-sfc-hhsfc-05 (work in progress), September 2018. 703 [DRAFT-MD-VIRT] 704 Bernardos, C., Contreras, L., Vaishnavi, I., Szabo, R., 705 Li, X., Paolucci, F., Sgambelluri, A., Martini, B., 706 Valcarenghi, L., Landi, G., Andrushko, D., and A. Mourad, 707 "Multi-domain Network Virtualization", draft-bernardos- 708 nfvrg-multidomain-05 (work in progress), September 2018. 710 [DRAFT-PM] 711 Roome, W., Chen, S., Randriamasy, S., Yang, Y., and J. 712 Zhang, "Unified Properties for the ALTO Protocol", draft- 713 ietf-alto-unified-props-new-03 (work in progress), March 714 2018. 716 [DRAFT-PV] 717 Bernstein, G., Chen, S., Gao, K., Lee, Y., Roome, W., 718 Scharf, M., Yang, Y., and J. Zhang, "ALTO Extension: Path 719 Vector Cost Type", draft-ietf-alto-path-vector-03 (work in 720 progress), March 2018. 722 [ETSI-NFV-DEF] 723 ETSI, "Network Functions Virtualisation (NFV); Terminology 724 for Main Concepts in NFV V1.3.1", Jan 2018, 725 . 729 [ETSI-NFV-IFA028] 730 ETSI, "Report on architecture options to support multiple 731 administrative domains V3.1.1", Jan 2018, 732 . 735 [ETSI-NFV-MANO] 736 ETSI, "Network Functions Virtualisation (NFV) Management 737 and Orchestration V1.1.1", Dec 2014, 738 . 741 [H2020-5G-TRANSFORMER] 742 H2020, "5G-Transformer -- 5G Mobile Transport Platform for 743 Vertical", 2017, . 745 [H2020.5GEX] 746 Bernardos, C., Dugeon, O., Galis, A., Morris, D., Simon, 747 C., and R. Szabo, "5G Exchange (5GEx)--Multi-domain 748 Orchestration for Software Defined Infrastructures", 749 focus vol. 4, no.5, p.2, 2015. 751 [H2020.5GEX.ESCAPE] 752 5GEx Project, "ESCAPE: Extensible Service ChAin 753 Prototyping Environment", 2015, 754 . 756 [ICAF] Demchenko, Y., Makkes, M., Strijkers, R., Ngo, C., and C. 757 Laat, "Intercloud Architecture Framework for Heterogeneous 758 Multi-Provider Cloud based Infrastructure Services 759 Provisioning", International Journal of Next-Generation 760 Computing vol. 4, no.2, 2013. 762 [PPP-5:2013] 763 5G-PPP, "Advanced 5G Network Infrastructure for the Future 764 Internet", 2013, . 768 [T-NOVA] FP7 project T-NOVA, "T-NOVA Project, Network Functions as 769 a Service over Virtualised Infrastructures", 2014, 770 . 772 [TELEFONICA.NET.TOPO] 773 Telefonica I+D, "Netphony-Topology", 2016, 774 . 776 [TOSCA] OASIS, "TOSCA: Topology and Orchestration Specification 777 for Cloud Applications V1.0", 2013, . 780 [UNIFY.NFFG] 781 UNIFY Deliverable D3.2a, "Network Function Forwarding 782 Graph specification", 2015, . 786 [VITAL] VITAL PROJECT H2020, "VITAL -- VIrtualized hybrid 787 satellite-TerrestriAl systems for resilient and fLexible 788 future networks", 2015, . 790 10.3. URIs 792 [1] https://tools.ietf.org/html/draft-ietf-alto-path-vector- 793 02#section-6.1 795 [2] https://tools.ietf.org/html/draft-ietf-alto-path-vector- 796 02#section-6.1 798 [3] https://tools.ietf.org/html/draft-ietf-alto-path-vector- 799 02#section-6.1.2 801 [4] https://tools.ietf.org/html/draft-ietf-alto-path-vector- 802 02#section-6.3.6 804 Appendix A. Proof of Concept Use Case Implementation 806 A strawman use case scenario has been implemented following the 807 architectural proposal of the 5GEx project [H2020.5GEX]. It refers 808 to an E2ENS orchestration involving three administrative domains. 810 As shown in Figure 2, each administrative domain has an MdO (MdO-AS1, 811 MdO-AS2, and MdO-AS3) to coordinate resource and/or service 812 orchestration at multi-operator level via interface I2 APIs. For the 813 orchestration within the same administrative domain, each MdO uses 814 emulated DOs with emulated I3 interfaces, since no data-plane is 815 present. DOs use static configuration files to load local 816 information about resources (I3-RC) and topology (I3-RT). The 817 different MdO components are based on existing open source tools such 818 as ESCAPE [H2020.5GEX.ESCAPE] (Service/Resource Orchestrator) and 819 Netphony-topology [TELEFONICA.NET.TOPO] (Resource Topology) and run 820 in Docker containers on a single computer. Besides, MdOs expose I1 821 interfaces to the tenants who request services and/or slices which 822 should follow a Network Function Forwarding Graph (NFFG) [UNIFY.NFFG] 823 format. 825 In case of the broker layer, the IdR and IdT components use a UNIFY 826 Virtualizer API [UNIFY.NFFG] (broker-based I2-RC API) and a REST API 827 (broker-based I2-RT API) respectively, in order to create the 828 hierarchical databases. Regarding the IdT, the administrative domain 829 2 is a transit provider so that the domain-level topology computed 830 is: AS1-AS2-AS3. From the inter-domain information are created the 831 two different ALTO Map Services: (i) Property Map and (ii) Cost Map. 833 +----------------------------------------+ 834 | +---------------+ BROKER LAYER| 835 XXXXXXXXXXXXXX ALTO Server | | 836 X | +--------+----+-+ | 837 X | / \ | 838 X | +-----------+/+--+ +-\-------------+ | 839 X | | Inter-domain | | Inter-domain | | 840 +-----------+ X | | Topology (IdT) | | Resource (IdR)| | 841 | Service | X | +----------------+ +---------------+ | 842 | Graph (SG)| X +---------^-^----------------^--^--------+ 843 | Request | X * * ............. . 844 +-----+-----+ X * * . .............. 845 | X * * . . MdO-AS3 846 I1| X * * . . +--------------+ 847 | X MdO-AS1 * * . . | | 848 +-----|---------X-----------+ * * . . | MdO-AS2 | 849 | | | * * . . +---------------+ | 850 | +---v-------------------+ | * * . . | +-----------+ | | 851 | | | | * * . . | | | | | 852 | | Network Service Orch.| | * * . . | | NSO | | | 853 | | (NSO) | | * * . . | | | | | 854 | +-----------------------+ | * * . . | +-----------+ | | 855 | | * * . . | | | 856 | +---------+ | * * . . | +---+ | | 857 | | Resource........................... . | | | | | 858 | | Topoloy | | * * .......RT | | | 859 | | (RT) | +-----------+ | * * | | | | | 860 | +---------+ |Resource | | * * | +---+ +---+ | | 861 | |Orch. | | * ********************** | | | 862 | |(RO) ****** | |RO | +-+ 863 | +-----------+ | | | | | 864 | |<------------->| +---+ | 865 +---------------------------+ I2 +-----+---------+ 866 / \ | 867 I3/ \ |I3 868 +-------+---+ +-----------+ +-----------+ 869 | Domain | | Domain | | Domain | 870 | Orch (DO) | | Orch (DO) | | Orch (DO) | 871 +-----------+ +-----------+ +-----------+ 872 Legend: 873 XXX ALTO Protocol 874 ... broker-based I2-RT API 875 *** broker-based I2-RC API 877 Figure 2: Broker-assisted 5GEx Info Exchange 879 The Property Map includes property values grouped by Autonomous 880 System (AS). Such values are SAPs, NFs and the 5GEx Entry Point 881 (e.g., the URL of the ESCAPE orchestrator). An example of the 882 Property Map in our prototype is: 884 +-----+------------+-------+--------------+-----+-----+-------+-----+ 885 | | Entry | Port | Capabilities | CPU | MEM | Stor | ... | 886 | | Point | SAP | | | | age | | 887 +-----+------------+-------+--------------+-----+-----+-------+-----+ 888 | AS1 | http://... | SAP1 | {NF1, NF3} | 50 | 60 | 70 | ... | 889 | AS2 | http://... | - | {NF2} | 10 | 20 | 30 | ... | 890 | AS3 | http://... | SAP2 | {NF1, NF3} | 80 | 90 | 100 | ... | 891 +-----+------------+-------+--------------+-----+-----+-------+-----+ 893 Table 1: ALTO Property Map 895 The Cost Map defines a path vector as an array of ASes, representing 896 the AS-level topological distance for a given E2ENS request. Path 897 vector constraints (as described in the Multi-Cost Map [RFC8189]) can 898 be applied to restricts the response to costs that satisfy a list of 899 simple predicates. 901 Table 2 below shows a brief example of an SG request and its path 902 vector response containing a list of potential providers to be 903 traversed to deliver such service. Every AS path is computed from 904 the inter-domain topology information in the IdT module. In our 905 scenario, MdO-AS2 is a transit provider, so that the domain-level 906 topology map is AS1<->AS2<->AS3. 908 +--------------------+----------------------------------------------+ 909 | Service Graph (SG) | Path(s) Vector | 910 | Request | | 911 +--------------------+----------------------------------------------+ 912 | SAP1->NF1->NF2->NF | 1:{AS1:SAP1->AS1:NF1->AS2:NF2->AS3:NF3->AS3: | 913 | 3->SAP2 | SAP2} | 914 | | 2:{AS1:SAP1->AS1:NF1->AS2:NF2->AS1:NF3->AS2- | 915 | | >AS3:SAP2} | 916 | | 3:{AS1:SAP1->AS2->AS3:NF1->AS2:NF2->AS3:NF3- | 917 | | AS3:SAP2} | 918 | | 4:{AS1:SAP1->AS2->AS3:NF1->AS2:NF2->AS1:NF3- | 919 | | >AS2->AS3:SAP2} | 920 +--------------------+----------------------------------------------+ 922 Table 2: ALTO Cost Map 924 Authors' Addresses 926 Danny Alex Lachos Perez 927 University of Campinas 928 Av. Albert Einstein 400 929 Campinas, Sao Paulo 13083-970 930 Brazil 932 Email: dlachosp@dca.fee.unicamp.br 933 URI: https://intrig.dca.fee.unicamp.br/danny-lachos/ 935 Christian Esteve Rothenberg 936 University of Campinas 937 Av. Albert Einstein 400 938 Campinas, Sao Paulo 13083-970 939 Brazil 941 Email: chesteve@dca.fee.unicamp.br 942 URI: https://intrig.dca.fee.unicamp.br/christian/