idnits 2.17.1 draft-dalela-orchestration-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack a Security Considerations section. == There are 3 instances of lines with non-RFC2606-compliant FQDNs in the document. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (January 4, 2012) is 4495 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Missing Reference: 'Switzerland' is mentioned on line 1169, but not defined Summary: 1 error (**), 0 flaws (~~), 3 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 Network Working Group A. Dalela 2 Internet Draft Cisco Systems 3 Intended status: Informational M. Hammer 4 Expires: July 2012 January 4, 2012 6 Service Orchestration Protocol (SOP) Requirements 7 draft-dalela-orchestration-00.txt 9 Status of this Memo 11 This Internet-Draft is submitted in full conformance with the 12 provisions of BCP 78 and BCP 79. 14 Internet-Drafts are working documents of the Internet Engineering 15 Task Force (IETF), its areas, and its working groups. Note that 16 other groups may also distribute working documents as Internet- 17 Drafts. 19 Internet-Drafts are draft documents valid for a maximum of six months 20 and may be updated, replaced, or obsoleted by other documents at any 21 time. It is inappropriate to use Internet-Drafts as reference 22 material or to cite them other than as "work in progress." 24 The list of current Internet-Drafts can be accessed at 25 http://www.ietf.org/ietf/1id-abstracts.txt 27 The list of Internet-Draft Shadow Directories can be accessed at 28 http://www.ietf.org/shadow.html 30 This Internet-Draft will expire on July 4, 2012. 32 Copyright Notice 34 Copyright (c) 2012 IETF Trust and the persons identified as the 35 document authors. All rights reserved. 37 This document is subject to BCP 78 and the IETF Trust's Legal 38 Provisions Relating to IETF Documents 39 (http://trustee.ietf.org/license-info) in effect on the date of 40 publication of this document. Please review these documents 41 carefully, as they describe your rights and restrictions with respect 42 to this document. 44 Abstract 46 Cloud services need to interoperate across cloud providers, service 47 vendors and private/public domains. To enable this interoperability, 48 there is need for a standard protocol for exchanging service 49 information. This draft describes requirements for such a protocol. 50 Current cloud implementations expose application level APIs, which 51 are not syntactically and semantically compatible with each other. 52 One approach to interoperate cloud services is to standardize the 53 protocol while leaving the API definition implementation specific. 54 Standard protocols have been used widely in the Internet and can be 55 extended to cloud. Use of such protocols is compatible with existing 56 cloud APIs, which can exchange information in a standard protocol. 57 New APIs may also be developed using a standard protocol. By this, it 58 would be possible to interoperate diverse APIs across service 59 providers, service vendors and service users. 61 Table of Contents 63 1. Introduction...................................................3 64 2. Conventions used in this document..............................4 65 3. Terms and Acronyms.............................................4 66 4. Interoperability Scenarios.....................................6 67 5. Cloud Open Source and Open Standards..........................10 68 6. Is Cloud Control an Internet Problem?.........................11 69 7. Overview of Standard Work.....................................13 70 8. Deficiencies of Current Models................................14 71 8.1. Service Discovery........................................15 72 8.2. Service Publishing.......................................15 73 8.3. Persistent Identities....................................15 74 8.4. Blocking Calls...........................................16 75 8.5. Transaction Support......................................16 76 8.6. Interactive Behaviors....................................17 77 9. Extensibility Considerations..................................17 78 9.1. Service-Independent Components...........................17 79 9.2. Service-Dependent Components.............................19 80 10. Protocol Requirements........................................19 81 11. Separating Control and Policy Planes.........................20 82 12. Service Management Policies..................................23 83 12.1. Routing Policies........................................23 84 12.2. Security Policies.......................................24 85 12.3. Service Policies........................................24 86 13. Architecture Requirements....................................25 87 14. IANA Considerations..........................................26 88 15. Conclusions..................................................26 89 16. References...................................................26 90 16.1. Normative References....................................26 91 16.2. Informative References..................................26 92 17. Acknowledgments..............................................27 94 1. Introduction 96 Cloud computing has become important for an on-demand delivery of a 97 variety of services, broadly called XaaS, such as Infrastructure, 98 Platform and Software as a Service [NIST]. Users of such services may 99 be individuals, enterprises, content providers, or other cloud 100 providers. These users need to be able to request and manage services 101 seamlessly across private, public, hybrid, or community clouds. Lack 102 of interoperability across these domains will lead to new kinds of 103 cloud silos, which will in turn hinder economies of scale. 105 Current cloud deployments use web-services (SOAP or REST) to deliver 106 services over the Internet. Each provider exposes different APIs that 107 generally do not interoperate, because each API has different syntax 108 and semantics. To interoperate, we must either converge on one API 109 format, or translate between them. Both alternatives are hard. API 110 translations are difficult because APIs have different semantics. 111 Converging to one API means current services may be broken. We want 112 to maintain diverse APIs, while enabling interoperability. 114 Historically, in Internet, different APIs have interoperated through 115 use of standard protocols. Basically, we separate the network view of 116 information from the application view. Network carries information 117 via protocols while applications consume information via APIs. 119 Web-services equate the network view of information with the 120 application view. Basically, each API has its own packet format which 121 is derived from the API, and changes to API syntax or semantics will 122 change the packet format. This is at the root of interoperability 123 issues. As applications proliferate, each API will project its view 124 of information into the network. As a result, there will be as many 125 communications "protocols" as there are applications. This is 126 contrary to the (unstated) assumption in Internet that there are far 127 fewer protocols than there are applications, so that many 128 applications can communicate using the same protocols. 130 To remedy this problem we should separate the network and application 131 views of information and design them independently. Applications may 132 design APIs in many ways and two applications should communicate 133 using a standard protocol whether or not they use the same API. This 134 document describes requirements for such a standard protocol. 136 2. Conventions used in this document 138 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 139 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 140 document are to be interpreted as described in RFC-2119 [RFC2119]. 142 In this document, these words will appear with that interpretation 143 only when in ALL CAPS. Lower case uses of these words are not to be 144 interpreted as carrying RFC-2119 significance. 146 3. Terms and Acronyms 148 +------------+ +--------------------------------------+ 149 | Customer |<-------->| Provider | 150 | +--------+ | | +----------------------------------+ | 151 | | User | | | | Service=X +--------------------+ | | 152 | | |**************| |Product-1 (Vendor-A)| | | 153 | | | | | | +--------------------+ | | 154 | | | | | +----------------------------------+ | 155 | | | | | +----------------------------------+ | 156 | | |**************| Service=Y +--------------------+ | | 157 | | | | | | |Product-2 (Vendor-A)| | | 158 | | | | | | +--------------------+ | | 159 | | | | | | +--------------------+ | | 160 | | | | | | |Product-3 (Vendor-B)| | | 161 | | | | | | +--------------------+ | | 162 | +--------+ | | +----------------------------------+ | 163 +------------+ +--------------------------------------+ 165 Fig-1: Cloud Ecosystem and Relations 167 Provider: A Provider is a supplier of cloud services who offers these 168 services to cloud Customers and Users, per some business agreement. 170 Service: Any virtual instance of a hardware or software product that 171 can be owned by a Customer or User for their personal use. 173 Vendor: A Vendor is a hardware/software product vendor who provides 174 the technology implementation of a service. In some cases, Providers 175 and Vendors may be the same business entity. 177 Product: A unit of software or hardware entity that is sold by the 178 Vendor to the Provider to be made available as a service. 180 Customer: A business entity that enters into an agreement with a 181 Provider to source cloud services for their users. A customer would 182 be an enterprise that buys cloud services. A customer would define 183 policies for service and may authenticate its users. Customers may 184 also be called Subscribers of a cloud provider's services. 186 User: A user is the end consumer of cloud services. Users belong to 187 the customer and place service requests on the provider. These 188 requests are controlled by Customers who could be enterprises, 189 individuals or other cloud providers who source services from one 190 cloud and provide them to another. 192 Virtual Provider: A Provider who does not host or manage services, 193 but redirects requests to other providers who do that. A Virtual 194 Provider has customers but does not operate services. 196 Orchestration: This is the act of creating, modifying, moving or 197 deleting services. It may involve one or more actions performed in 198 sequence or in parallel. These actions could be invoked on hardware 199 and software services, or even on other cloud providers. 201 Service Domain Name (SDN): This is a dotted-decimal notation to 202 represent service names hierarchically. For example, a virtual 203 machine can be represented as iaas.compute.virtual. Each SDN will be 204 associated with a set of service specific attributes. 206 4. Interoperability Scenarios 208 The following interoperability scenarios should be covered by the 209 protocol. We list them here because depending on the context 210 interoperability may mean different things. 212 Scenario S-1. Users Interoperate Across Cloud Providers. Users must 213 be able to use cloud services from different cloud providers in the 214 same way. This allows a user to move across providers, or source the 215 same service in a different geography from a different provider. In 216 Figure-2, a user accesses the compute service across various 217 providers in the same way. 219 | | | | 220 Customer | Virtual | Provider 1 | Provider 2 | 221 | Provider | | | 222 +--------------------------------------S-1-------+ | 223 | | | | | | 224 +-----S-1-----+ | | | | 225 | | | | | | | 226 +-------+ | +-------+ | +-------+ | +--------+ | 227 | Cloud | | | Cloud |--------| Cloud | | | Cloud | | 228 |Control| | |Control| | |Control| | | Control| | 229 +-------+ | +-------+ | +-------+ | +--------+ | 230 | | | | | | | 231 +-------+ | | +---------+ | +---------+ | 232 | User | | | | Compute | | | Compute | | 233 +-------+ | | +---------+ | +---------+ | 234 * | | * | * | 235 * | service usage| * | * | 236 ************************************************** | 237 | | | | 239 Fig-2: Scenario S-1 - User Interoperates Across Providers 241 Scenario S-2. Users Interoperate Between Private and Public Clouds. 242 Users should be able to interoperate private clouds with those in the 243 provider domain. This could involve moving workloads between private 244 and public clouds. It also means creating virtual services in the 245 same way in the public cloud as that in the private cloud. In Figure- 246 3, a user creates compute in the private and public clouds in same 247 way. Private storage is accessed from public and private compute. 249 | | | | 250 Customer | Virtual | Provider 1 | Provider 2 | 251 | Provider | | | 252 | | | | 253 +--------------------------------------S-2---------+ | 254 | | | | | | 255 +-----S-2-----+ | | | | 256 | | | | | | | 257 +-------+ | +-------+ | +-------+ | +-------+ | 258 | Cloud | | | Cloud |--------| Cloud | | | Cloud | | 259 +--|Control| | |Control| | |Control| | |Control| | 260 | +-------+ | +-------+ | +-------+ | +-------+ | 261 | | | | | | | | 262 | +-------+ | | | | | | 263 | | User |********************************************* | | 264 | | | | | * | | * | | 265 | +-------+ | | * | | * | | 266 | * | | * | | * | | 267 +-S-2--+ * | | * | | * | | 268 | | * | | * | | * | | 269 | +-------+ | | +---------+ | +---------+ | 270 | |Compute|<-- move workload -->| Compute |<-move->| Compute | | 271 | +-------+ | | +---------+ | +---------+ | 272 +-S-2--+ * | | * | * | 273 | * | | * | * | 274 +-------+ | | * | * | 275 |Storage|********************************************* | 276 +-------+ | | | | 278 Fig-3: Scenario S-2 - User Interoperates Across Private and Public 280 Scenario S-3. Providers Interoperate With Other Providers. Providers 281 should be able to interoperate their services with other providers. 282 This could mean sourcing each other services when the demand suddenly 283 grows, or using one vendor's services as backup or for disaster 284 recovery under an outage. Providers might agree to host services in 285 each other clouds in a follow the sun models, where workload moves 286 between providers located in different geographies. There could be 287 "provider of providers" - a virtual provider that sources services 288 across different providers by using interoperability. In Figure-4, a 289 provider sources the storage service to complement their compute 290 service, and offers compute and storage as a bundle to the user. 292 | | | | 293 Customer | Virtual | Provider 1 | Provider 2 | 294 | Provider | | | 295 | | | | 296 +-------+ | +-------+ | +-------+ | +--------+ | 297 | Cloud |-----| Cloud |--S-3---| Cloud |---S-3---| Cloud | | 298 |Control| | |Control| | |Control| | | Control| | 299 +-------+ | +-------+ | +-------+ | +--------+ | 300 | | | | | | | 301 | | | | | | | 302 | | | | | | | 303 +-------+ | | +---------+ | +---------+ | 304 | User |*********************| Compute |*******| Storage | | 305 | | | | +---------+ | +---------+ | 306 +-------+ | | | | 307 | | | | 309 Fig-4: Scenario S-3 - Providers Interoperate Amongst Providers 311 Scenario S-4. Providers Interoperate Services Across Service Tiers. 312 A cloud provider may deliver many kinds of services, layered on top 313 of one another. For instance, SaaS may use PaaS, which in turn may 314 use IaaS, network and security services, etc. Since cloud providers 315 build services incrementally, it should be possible to interoperate 316 services across these tiers, without having to build a new IaaS 317 system for every new PaaS, or a new PaaS for every SaaS. In Figure-5, 318 a provider sources IaaS services for their PaaS from another provider 319 in the same way as they source them internally. 321 | | | | 322 Customer | Virtual | Provider 1 | Provider 2 | 323 | Provider | | | 324 | | | | 325 +-------+ | +-------+ | +-------+ | +--------+ | 326 | Cloud |-----| Cloud |--------| Cloud |---S-4---| Cloud | | 327 |Control| | |Control| | |Control| | | Control| | 328 +-------+ | +-------+ | +-------+ | +--------+ | 329 | | | | | | | | 330 | | | S-4 | | S-4 | 331 | | | | | | | | 332 +-------+ | | +------+ | | +------+ | 333 | User |*******************| SaaS |**************| IaaS | | 334 | | | | +------+ | | +------+ | 335 +-------+ | | * | | * | 336 | | * S-4 | * | 337 | | +------+ | | * | 338 | | | PaaS |--+ | * | 339 | | +------+****************** | 340 | | * | | | 341 | | * S-4 | | 342 | | +------+ | | | 343 | | | IaaS |--+ | | 344 | | +------+ | | 346 Fig-5: Scenario S-4 - Providers Interoperate Across Tiers 348 Scenario S-5. Providers Interoperate Across Service Vendors. A cloud 349 provider may source a service from more than one vendor. Examples of 350 these include compute virtualization, storage, network, security, 351 etc. A customer's existing orchestration solution should be able to 352 orchestrate multi-vendor products and services. In Figure-6, 353 providers deliver a service using offerings from multiple vendors in 354 the same way. These inter-vendor services may also be connected. 356 | | | | 357 Customer | Virtual | Provider 1 | Provider 2 | 358 | Provider | | | 359 | | | | 360 +-------+ | +-------+ | +-------+ | +-------+ | 361 | Cloud |-----| Cloud |--------| Cloud |---S-4---| Cloud | | 362 |Control| | |Control| | |Control| | |Control| | 363 +-------+ | +-------+ | +-------+ | +-------+ | 364 | | | | | | | | | 365 | | | / S-5 | S-5 \ | 366 | | | / | | | \ | 367 +-------+ | | + +-------+ | +-------+ + | 368 | User |********************|**|Compute|********|Compute| | | 369 | | | | | |Vendor1|** | **|Vendor3| | | 370 +-------+ | | | +-------+ * | * +-------+ | | 371 | | S-5 +-------+ * | * +-------+ S-5 | 372 | | | |Storage| * | * |Storage| | | 373 | | +--|Vendor2|** | **|Vendor4|--+ | 374 | | +-------+ | +-------+ | 376 Fig-6: Scenario S-5 - Providers Interoperate Across Service Vendors 378 The above scenarios are illustrative and non-exhaustive. There could 379 be many permutations of the above scenarios. Standardization will 380 benefit users, vendors and providers - the total cloud ecosystem. 382 5. Cloud Open Source and Open Standards 384 Some efforts towards cloud openness today are focused on Open Source 385 implementations of cloud services. This leads to the question of the 386 relation between Open Source and Open Standards, as different ways to 387 achieve interoperability. Obviously, cloud will not be totally open 388 or closed source. The key problem in cloud is not the ability to 389 inspect and modify code, which open source enables, but to integrate 390 services, both open and closed. To integrate Open and Closed Source 391 services, Open Standards are required, which may be implemented as 392 Open Source. Open standards don't detract us from open source. 394 On the other hand, lack of open standards can make open source less 395 attractive because there can be many open source implementations that 396 are incompatible. Within an implementation, various versions may be 397 incompatible. This means that Open Source alone cannot solve problems 398 of interoperability unless everyone converges to a common code base 399 and contributes their private changes back into the common base. This 400 is impossible to mandate and unlikely to happen. 402 Open Standard implementations on the other hand will be interoperable 403 even when implementations are enhanced in different ways. So Open 404 Standards enhance rather than detract from benefits of Open Source. 405 The key problem for cloud is service integration across vendors, 406 providers and customers. These services will be Open Source, Closed 407 Source, or multiple variations of Open Source. Integrating the 408 variety of services is best done through Open Standards. 410 6. Is Cloud Control an Internet Problem? 412 Given that problems of cloud interoperability need to be addressed 413 through standards, it may not be obvious that they need to be 414 addressed by IETF. Why is cloud control an IETF problem? 416 First, to create, modify or move a distributed system, orchestrators 417 need to know network topology. For instance, if firewall rules have 418 to be installed for a VM, they must be installed on a device that 419 lies in the "path" to the VM. To know which firewall lies on the path 420 to a given VM, topology needs to be known. Similarly, if bandwidth 421 needs to be provisioned between two sites, it is necessary to know 422 which routers are at the edge of the two sites so that bandwidth can 423 be provisioned between those routers. Likewise, if a VM is moved from 424 one location to another, all associated network port configuration 425 (such as VLAN or policy) needs to be dragged along with the VM. That 426 requires the orchestrator to know which port the VM was attached on, 427 and where it is going to move next. In some cases, the VLAN and 428 policy may need to be provisioned not just in the access but also on 429 the trunk ports to permit the packet flow. That requires the 430 orchestrator to know which access is mapped to which trunks. To 431 ensure that performance of a VM does not degrade after a move, it may 432 be necessary to determine whether sufficient bandwidth is available 433 at the destination location before the move is made. That requires 434 knowledge of the paths that will be used and if those paths are 435 congested. An orchestrator may need to assess the "distance" between 436 the compute and network storage and between the user's location and 437 the service's location for optimal performance. 439 There are also cases when knowledge of topology is needed for network 440 optimization. For example, the network paths may not be optimal after 441 a VM move, and the paths may need to be re-provisioned. Such things 442 are common with multicast and broadcast traffic that uses trees. 443 During outages, network topologies are dynamically reconfigured. 444 Recovery procedures must be aware of this network reconfiguration. 445 The above examples illustrate a close relation between network 446 information and orchestration of services. These two are currently 447 treated as separate domains, and they need to be linked. 449 Second, cloud service discovery is about knowing the capability of 450 devices in the Internet. Today, IP routing allows us to discover the 451 location of IP addresses, but not their capabilities. For instance, 452 the same IP address can belong to a PC, a router, a storage array, an 453 IP-TV, a mobile phone, etc. Network protocols don't tell us the 454 "semantics" of the IP, namely what that IP can "do". This of course 455 is not a new problem, but cloud makes this problem very important. 456 Cloud is about the ability to know which capabilities are available 457 where in the network. This would be achieved if some protocol 458 advertizes capabilities of IP addresses. Ideally, the systems that 459 advertize addresses and those that advertise capability should be 460 linked because the capability is of the address. To reach that 461 capability, we need to translate it into an address. 463 When a service is yet to be created, it needs to be referred by its 464 capability because the DN or IP for that service is yet to be 465 created. This capability can be advertized by some service 466 orchestrator that can create the service based on a request. In the 467 Internet, a service naming mechanism is needed to advertize and 468 request services by their "type" instead of DN or IP (DN and IP are 469 useful for advertizing and requesting services that exist). These 470 names can have a similar structure like DNS or IP addresses (dotted- 471 decimal) but need to belong to a separate address space. We can call 472 these "type" names Service Domain Names or SDNs. 474 Third, a cloud user may not care about the IP or DN of a service. 475 What users care about is the "type" of service they are looking for. 476 This service may be fulfilled anywhere in the network. The user will 477 issue a request referencing the SDN, and would expect the request to 478 be automatically routed to its correct destination. This is possible 479 if SDNs have been advertised in the network. A user can forward a 480 request to service aware router, and the router will map the request 481 to destination. Mappings between service types and addresses can be 482 done at the edges of the Internet allowing users to be unaware of IP 483 addresses while the Internet to be unaware of services. A variety of 484 policy controls can be built at the network edges to determine how a 485 service "prefix" is mapped into an IP "prefix". 487 The problem of routing based on "types" is similar to routing based 488 on IP addresses. In both cases, addresses need to be discovered, 489 aggregated by some meaningful prefix, and advertized to routers 490 upstream. These similarities imply that service routing can be 491 implemented in ways similar to Internet routing in the past. 493 Fourth, thus far the link between capability and address has been 494 done for services that are already created, generally within an 495 administrative domain. For instance, it is possible to use DNS to 496 discover the address of a printer or email server. Cloud deals with 497 creation of services on-demand. This discovery over the Internet 498 needs a somewhat different ability, such as policy control, routing, 499 billing of services, authentication, security from denial of service, 500 SLA announcements, etc. There is a greater amount of complexity in 501 advertizing service information, publishing service interest, 502 policies to control per-user services, etc. However, these issues are 503 similar to things that have been done in IETF earlier. 505 In summary, orchestration needs to know network topology. The network 506 can learn and advertize service capabilities like IP addresses. A 507 mapping between addresses and capability is needed to perform service 508 request routing. Such mappings have been created in the past, but 509 just not to the extent required for cloud. The problem is both 510 relevant for IETF and optimally solved within IETF. 512 7. Overview of Standard Work 514 To run the service exchange network over the current Internet, three 515 important enhancements to the current schemes are proposed. 517 First, we need a service naming convention that addresses services by 518 their "types" rather than by their DN or IP addresses. This naming 519 system should also be hierarchical, in order to aggregate service 520 types into "classes" of services. For instance, virtual machines may 521 be referred by the name iaas.compute.virtual and firewalls by the 522 name iaas.network.services.firewall. Each class of service may be 523 associated with one or more attributes, or may be further divided 524 into sub-classes, or sub-sub-classes, with suitable names. We can 525 refer to these names as Service Domain Names or SDNs. 527 Second, we need a protocol that advertizes SDNs and routes service 528 requests based upon these SDNs. This protocol will facilitate service 529 aggregation based on names, service discovery, advertisement, 530 selective publishing and indication of service interest, besides 531 mechanisms to route the request based on where it can be fulfilled. 532 We can refer to this as a Service Routing System (SRS). 534 The SRS needs to map service "prefixes" into IP "prefixes" and will 535 interact with a policy based control system, where users, customers 536 or providers can define rules for routing requests to a destination. 537 The SRS discovers services and their locations and provides the 538 mapping between Service Names and IP or DN. Using this mapping it is 539 possible to identify the service by its name as well as type. The SN, 540 DN and IP names are orthogonal name spaces. That is, any SN may map 541 to any DN, which may in turn map to any IP. 543 Third, there is need for a common format to specify service 544 attributes. This common format can be XML and it is necessary to 545 define cross-service-domain orchestration rules. For example, in a L3 546 network, the IP of a host must belong to the subnet configured on the 547 switch. The IP access-list on the switch must permit the IP address 548 on the host. The ports open on a host must also be open on the 549 firewall. The file systems accessible to a VLAN must align with the 550 VLAN configured on the host access interface. The user-ids 551 provisioned on the server must be available to authentication on the 552 network storage. The speed of the virtual host interface must be 553 equal to the bandwidth allowed to the host on the virtual or physical 554 network interface. The virtual MAC allocated to a VM must not clash 555 with any other virtual or physical MACs allocated anywhere else on 556 the VLAN. The authentication system must use a combination of the 557 tenant-id on the network in addition to the user-id on the host. 559 These relations represent semantic "rules" of orchestration. Today, 560 we can't express these rules because information schemas across 561 domains are incompatible. In effect it requires us to map some 562 parameter in some CLI to some OID in another MIB. Or, some attribute 563 in some XML schema to some TLV in another Protocol. Or, the value of 564 a resource in a GUI to a range specified via another API. If all 565 services are described in a common format (such as XML) then 566 orchestration rules can be easily specified. This will allow rapid 567 customization of services by defining orchestration rules in a high- 568 level language rather than programming in a low-level language. 570 8. Deficiencies of Current Models 572 Cloud deployments today use HTTP web-services (SOAP and REST) to 573 distribute service information and manage services. Web-services were 574 designed for distributed application objects, where one object 575 executes requests on other objects. This leads to the question if 576 treating cloud orchestration as a distributed application object is 577 the right approach to thinking about cloud services. In this section 578 we will describe limitations of the web-service model. The web- 579 service model is constrained by the capabilities of HTTP in service 580 discovery, publishing and transaction management. 582 8.1. Service Discovery 584 HTTP was designed to connect clients to servers, but not designed for 585 clients and servers to discover each other. HTTP assumes that client- 586 server discovery happens through other mechanisms. The Universal 587 Description Discovery and Integration (UDDI) web-service standard for 588 instance defines registries where providers could publish their 589 services but this mechanism is manual and not widely used. 591 In the cloud network, operators require services to be automatically 592 discovered and advertized to consumers. Dynamic service discovery is 593 also needed because as services are allocated or de-allocated, 594 capacity dynamically changes. Manually detecting these changes would 595 be nearly impossible for any large deployment. 597 HTTP does not have procedures by which a network of clients and 598 servers can DISCOVER others and ADVERTISE their presence. HTTP allows 599 a client to connect to a server after it has been discovered. 601 8.2. Service Publishing 603 With millions of possible services, users may rarely be interested in 604 all such services. They may instead define selected types of service 605 "interest" and expect to be "notified" when new services of interest 606 are available. HTTP does not support SUBSCRIBE and PUBLISH mechanisms 607 by which a client can SUBSCRIBE to select interests and would be 608 notified of new services through a PUBLISH. 610 To know of the existence of new services, a Client must query a 611 registry periodically. This makes service publishing a synchronous 612 phenomenon and can be very hard to scale if millions of users query 613 available services at regular intervals. To scale service publishing, 614 it is necessary to make publishing an asynchronous phenomenon. HTTP 615 is not designed to deal with asynchronous publishing. 617 8.3. Persistent Identities 619 HTTP loses the identity of a client after a transaction (such as GET 620 or POST) has been completed. This means that every new transaction 621 has to be authenticated and may require a new key-exchange. When 622 millions of service instances have to advertize their presence or 623 publish capabilities periodically, it is imperative that the 624 underlying control protocol can maintain identity information 625 persistently across these multiple transactions. 627 For instance in Session Initiation Protocol [SIP] users REGISTER with 628 a SIP Proxy, at which time they are authenticated. Subsequent session 629 initiations don't require authentication. The identity established at 630 the time of registration can be used across all transactions. This 631 mechanism can be very useful as a single sign-on capability because 632 after registering once, every other service does not require the user 633 to be authenticated. The user can interact with all services by using 634 the identity established during the registration. HTTP does not 635 enable this because authentication is done by the server. 637 8.4. Blocking Calls 639 In a web-service call, a client blocks waiting for a response from 640 the server. There is no mechanism for the client to timeout on a 641 request, or cancel the request midway. If the server fails to respond 642 to the request, the client must separately terminate the connection. 643 This is not ideal because the server may in fact be taking a longer 644 period of time to fulfill the request. When requests are used to 645 orchestrate complex services, a server needs to send provisional 646 responses indicating that a "session is in progress". 648 When a service involves multiple independent but related components 649 (such as network, storage and compute), failure in one component may 650 render the entire service unusable. In such cases, it is necessary to 651 cancel the request midway. HTTP blocks for the server to respond and 652 cannot cancel on-going transactions. The only mechanism to terminate 653 the transaction mid-way is to close the HTTP connection, which can 654 then result in leaked resources or incomplete actions. 656 8.5. Transaction Support 658 Complex orchestration scenarios need to treat multiple operations as 659 a single atomic "transaction". For instance, an orchestration request 660 may allocate compute, storage, network and security resources in a 661 single request. Unless all of these operations have succeeded, the 662 resulting service is not useful and must be cancelled as a whole. If 663 all operations have succeeded, then they must be committed as a 664 whole. Complex orchestrations thus need transaction support. 666 There are two ways to build this transaction support. First, each 667 service can have its own transactions and cancelations. Second, 668 transactions can be available natively in the orchestration protocol. 669 Obviously, the first approach is very complex, and the preferred 670 route is to have transaction support in the protocol. 672 HTTP does not have the ability to create transactions. HTTP request- 673 response is atomic and considered complete individually. One HTTP 674 request-response is independent of prior or subsequent request- 675 response even to the same server, let alone another server. 677 Orchestration requires the ability to correlate request-responses 678 across multiple servers and commit or cancel them as a whole. 680 If an orchestrator that uses HTTP web-services fails after making a 681 request, the client will believe that the transaction has failed, 682 while the service nodes continue to allocate resources towards 683 completion. The client cannot be billed for the service, although the 684 services would be created. To address reliability issues, each 685 service must build application level transactions, and these will 686 rapidly grow as services are modified. A native mechanism at the 687 protocol level is required to address this. 689 8.6. Interactive Behaviors 691 Incompatibilities between a cloud request and cloud policies or 692 partial failures in service orchestration may require an orchestrator 693 to prompt a user with questions and/or confirmation before 694 proceeding. For example, if a VM has been allocated but the requested 695 amount of network storage is not available, the orchestrator may need 696 to prompt the user to allocate a reduced amount of storage. Such 697 interactive behaviors need to pause a transaction waiting for a 698 confirmation from a user. HTTP does not allow a server to make 699 another client connection to ask this question during an on-going 700 transaction. Also, if the question is passed as a provisional 701 response to the user, a user's response would be treated as a new 702 request. HTTP has no schemes to tie a request to another request in 703 the past, as all requests are independent. 705 9. Extensibility Considerations 707 One of the key issues in standardizing service orchestration is how 708 this standard can be extended for service variety. To make the 709 orchestration standard extensible to many services, we need to 710 separate things that are service independent from those that are 711 service dependant. Through this separation, it would be possible to 712 extend a service protocol to transmit information about a variety of 713 diverse services. This separation is described below. 715 9.1. Service-Independent Components 717 - Orchestration Verbs. Regardless of the kind of service that is 718 being offered, there is need for service Discovery, Creation, 719 Modification, Deletion, Migration, etc. There is also need for 720 Confirming and Canceling requests midway through a transaction or 721 indicating Successes and Failures upstream. Cloud involves many 722 such useful "verbs" which are service independent. Whether we are 723 creating a VM, VPN or Disk, the "CREATE" verb can be used to 724 indicate the operation of service creation. This common "CREATE" 725 can be used for a variety of create tasks, and its meaning can 726 depend on the receiver. Defining the verb once eliminates the need 727 to redefine the same operation for each new service. A collection 728 of such verbs can be standardized for any service to use. 730 - Transaction Nouns. To construct orchestration message 731 transactions, there is need to address messages to destinations and 732 identify their source, match requests with responses, bundle 733 multiple such messages into a single complex exchange, sequence 734 requests in the correct order with sequence numbers, have message 735 fields to identify type of content and content lengths, common 736 procedures for challenge and authentication of requestors, and many 737 other such transaction level functions. Like orchestration verbs, 738 these are service independent and can be standardized, without 739 limiting service diversity and flexibility. 741 - Workflow and Task Language. Different users will request different 742 combinations of services. One user might request a VM with only an 743 IP address, but another user may also require storage allocation, 744 bandwidth reservation, a secure firewall and a VPN to be setup 745 automatically when a VM is allocated. To accommodate variety of 746 service requests, a generic mechanism to define Workflows is 747 required. A Workflow identifies a set of tasks to be performed for 748 service orchestration. Users or providers may define Workflows at 749 various levels of abstractions. Hence, it is important to 750 distinguish Workflows from actual Tasks. A Workflow might equal to 751 one Task, or a Workflow might comprise of several Tasks bundled as 752 a single request. A service independent language to describe Tasks 753 and Workflows is needed. A User should be able to refer to 754 Workflows and Tasks using unique identifiers. 756 - Service Domain Names. To name services, a classification scheme is 757 required. Classification allows us to combine attributes across 758 similar types of services. We can take an object oriented approach 759 for defining service domains. For example, "network" can be a root 760 domain, "switching", "routing" and "network-services" can be child 761 domains of the root "network" domain, "security" and "packet 762 inspection" can be child domains of the "network-services" domain, 763 etc. Child domains may inherit properties of the parent domain. A 764 child domain may override the parent domain's attributes by 765 redefining them in the child domain. Once a domain naming is well 766 understood, service Proxies only need to advertize domains, with 767 references to well-understood domain schemas. Users who request 768 services will know what they are requesting based on domain name of 769 the service. They will also know each domain's attributes. This 770 abstracts a service implementation from the service user. 772 9.2. Service-Dependent Components 774 - Service Domain Parameters. Each service domain can have its own 775 service specific parameters. They can reuse existing parameters by 776 inheriting an existing domain. Domain parameters are inputs into a 777 request, and effectively can be used like parameters being passed 778 into APIs. Each domain may be associated with its own schema so 779 that an orchestrator that does not understand a domain can still 780 validate the request before forwarding it. The parameters of a 781 domain can be defined in a sufficiently generalized way to apply to 782 a wide variety of services in that domain. 784 - Vendor Specific Domains. Some service might not be standardized 785 through well-defined domain definitions. These definitions cannot 786 be understood by all clients or users. These may however be 787 understood between select network end-points that choose to use 788 such definitions. Using Vendor Specific Domains, experimental or 789 customized domains may be defined. 791 10. Protocol Requirements 793 A protocol that supports service variety must separate service- 794 independent and service-dependant parts of information. The service- 795 dependant and service-independent information may be carried in the 796 same message. This section describes needed capabilities for various 797 service-independent and service-dependant functions. 799 P-1. N-way transactions - an orchestration controller will need to 800 perform multi-domain (e.g. storage, compute, network, etc.) service 801 operations. The protocol should be able to stitch these varieties of 802 service domains into a single context. All transactions in the 803 client-server model are 2-way, so this needs a new protocol. 805 P-2. It should be possible to sequence and parallelize messages 806 within a single context. Sequences or parallelization would depend on 807 the specific needs of a particular kind of service. For instance, 808 compute and network services may be provisioned in parallel, while 809 workload movement across geographical regions must take place 810 sequentially. Accordingly, the responses to such requests may also be 811 received in sequential or parallel fashion. 813 P-3. When using requests in a parallel or sequential fashion, it 814 should be possible to "commit" these operations as a whole. If errors 815 are encountered in any one of the transactions, it should be possible 816 to "cancel" the entire service context as a whole. 818 P-4. For reliability, the protocol should support timers and 819 timeouts on requests. These timers may be used to expect a response 820 to a request within the specified timeframe. When the timer expires, 821 recovery actions should be possible. This is also useful in case of 822 network failures, and on-going transactions can be automatically 823 reversed. Through use of timers, and automated reversal, failures 824 would not result in leaked resources, incorrect accounting, etc. 826 P-5. The protocol should support explicit mechanisms to advertize 827 services and discover other service agents in a network. That is, 828 configuration of service agents should be minimized and the protocol 829 should facilitate automated discovery and advertisement. 831 P-6. The protocol should support selective propagation of service 832 information through use of publish-subscribe mechanisms. It should be 833 possible for a client to request specific kinds of service 834 information that it supports and expects to know about. 836 P-7. It should be possible to define workflows and tasks at various 837 levels of abstraction. Some users will prefer abstract requests that 838 are translated to concrete requests at some point before fulfillment. 839 Others may prefer that they define every service parameter. The 840 protocol must be able to support both these cases. 842 P-8. The protocol must support the CRUD (Create, Read, Update and 843 Delete) operations to transact services, after discovery of agents 844 and selective service exchange. These operations are part of HTTP and 845 should be present in the new protocol as well. 847 P-9. It should be possible to refer to services using standard 848 names. Use of standard names establishes convention on how services 849 will be referred to, which in turn facilitates interoperable service 850 publishing, advertizing, discovery and requests. 852 P-10. It should be possible to associate each service name with 853 service-specific properties. These properties may be mandatory or 854 optional. It should be possible to re-use these properties by 855 inheriting a service name into another service name. 857 11. Separating Control and Policy Planes 859 Each service may be customized according to a variety of needs such 860 as customer profile, user roles, location awareness, service design, 861 SLAs, etc. The set of rules that are used to customize a service 862 represent the "policy plane" as they specify how a service must be 863 designed. This policy must obviously interact with the protocol 864 messages ("control plane") to control service orchestration. 866 There are two broad approaches in which policy and control can 867 interact. First, we might collapse the difference between control and 868 policy, and just have a single plane that is designed for specific 869 services. Second, we might separate control and policy planes, and 870 allow independent evolution of policy and control planes. These 871 options and their relative merits are discussed below. 873 In many orchestration schemes, the policy and control planes are 874 collapsed into one. The orchestrator is designed and pre-programmed 875 to automate a few types of services. This scheme works well if the 876 desired service variety is small. Basically, for a small number of 877 service types, a few service templates can be hardcoded and published 878 to users. Users may choose from amongst available service templates 879 to create services on-demand. A service template defines a set of 880 business rules using which services would be created, deleted, 881 modified or moved. If pre-defined rules meet the requirements of 882 users, this is a huge simplification over manual service creation, 883 and a good starting point for service automation. 885 +----------+ +----------+ 886 | Policy | | Policy | 887 +----------+ +----------+ 888 | | 889 | | 890 +----------+ +----------+ +----------+ +----------+ 891 | Client |<--->| Server | | Client |<--->| Server | 892 +----------+ +----------+ +----------+ +----------+ 893 Option (a) Policy at Clients: Option (b) Policy at Servers: 894 Client Mgmt Complexity Server Mgmt Complexity 896 +----------+ +----------+ +----------+ 897 | Client |<-----------| Policy |---------->| Server | 898 +----------+ +----------+ +----------+ 899 Option (c) In-Band Policies - Complexity Centralized 901 Figure-7 Policy Deployment Models 903 However, as the service variety grows, this approach cannot scale 904 because the number of orchestrators will increase linearly with the 905 number of service types, and the complexity in each orchestrator will 906 increase exponentially with customization of business rules. Now, it 907 is necessary to separate definition of business rules ("policy") from 908 execution of rules ("control"). Interoperable control requires a 909 protocol and interoperable policy requires an abstract high-level 910 language to define orchestration rules. If the language of rules and 911 protocol have been separated and standardized, then the hurdles to 912 deploying new services have been significantly reduced. 914 There are still multiple policy deployment options where policy is 915 deployed at different points in the network, and these options can 916 make important differences to the ease of service management. 917 Different policy deployment options are shown in Figure-7. 919 First, policy may be attached to the user, such that users tune their 920 personalized policies about services. Second, policy may be attached 921 to each service, and the hardware-software vendor must give a 922 configurable system for policy controlling each service, which the 923 provider will have to customize to suit the needs of their 924 deployment. Third, policy may be attached to the orchestrator, which 925 may be defined either by provider or customer or jointly. The key 926 difference between these options is who controls the service. 928 Client-based policies are totally in control of clients. Server-based 929 policies are in provider control, but require the provider to 930 individually manage policies on each service instance. When services 931 are created dynamically, these service instances may have to download 932 policies dynamically and refresh them when policies change. Dynamic 933 changes to policies may disrupt existing services unless each server 934 has the intelligence to process policy rules per request. If common 935 policies have to be implemented across a set of clients, then these 936 clients must be updated with the new policy rules. There must also be 937 intelligence in client or server to deal with policy inconsistencies 938 across client and servers. All this entails a significant amount of 939 complexity in implementing and managing services. 941 Orchestrator based policies in contrast are easy to manage because 942 they can be controlled at few network points. When policies change, 943 the client and server don't have to be updated because policies are 944 enforced run-time. Orchestrator policies can also be controlled 945 either by provider or customer or jointly. It is architecturally 946 important to place this control in the right point in the network to 947 facilitate the best control scenarios. Obviously, orchestrator based 948 policy control is more flexible and easier than others. 950 When policies are attached to orchestrators, clients and servers 951 remain unaware of policy. Policy is now enforced at a small number of 952 customer and provider edges. While the total number of policy rules 953 remains unchanged, the complexity in managing these rules is reduced 954 by centralizing the intelligence to define and apply policies. 955 Challenges related to policy consistency are also addressed. 957 To apply these policies, client requests must be intercepted, policy 958 transformed and policy routed before they reach the server. The 959 clients and servers don't need to be aware of this behavior. The 960 rules for controlling service requests can be defined through 961 configuration in a policy server. Now, an orchestrator can download 962 policy rules for a service, and execute those rules in real-time. 964 The separation of the control and policy planes allows the same 965 control plane to be re-used for a variety of policies. Policies can 966 be defined through configuration instead of being programmed in the 967 orchestrator. And a common control plane can be used to orchestrate 968 variety of services. Through this separation, a service orchestrator 969 becomes a "Programmable Orchestrator", because it does not hardcode 970 service logic. Rather, orchestrators can be "programmed" through 971 policies defined by users in a user-friendly language. This approach 972 eases service creation and customization of existing services while 973 reducing overall management complexity. 975 12. Service Management Policies 977 This section describes different types of policies that might be used 978 in cloud services. A few of these policies are currently being 979 employed in the industry today, while many of them are desired 980 features of cloud services in future. The totality of these policy 981 types create a level of complexity that cannot be deployed by 982 embedding policy in client or server. These policies should exist in 983 a separate policy plane that interacts with the control plane. 985 12.1. Routing Policies 987 A service may be sourced from multiple destinations and to route a 988 request to the correct destination, various types of routing policies 989 may be applied. For example, a service request may be routed to the 990 geographically nearest provider. Or, it might be routed to a location 991 that offers the cheapest service rate or, to a different location 992 based on time of day. There might be routing rules based on SLAs. 993 Each user's request may be routed differently based on their roles. 994 There could be rules specific to a type of service, or routing may be 995 determined by the locations that have the necessary capacity. Routing 996 may be determined by legal or governmental regulations. 998 These rules may be dynamically changed, and different rules may apply 999 to different types of services, users, locations, roles etc. The 1000 provider and customer may independently or jointly define these 1001 policies, and enforce them at customer edge, provider edge, or both. 1003 12.2. Security Policies 1005 Security in the context of services encompasses a broad spectrum of 1006 issues spanning authentication, authorization and accounting (AAA). 1007 For instance, a customer may authenticate its users based on internal 1008 user-databases, while a provider owns the authorization and 1009 accounting of the service request. Or, a customer may own user- 1010 specific authorization and authentication while the provider owns the 1011 accounting. As users join or leave a customer, the provider may not 1012 own user-specific authentication and policies. 1014 The AAA functions are best performed at the provider or customer 1015 edges. First, each service should not be required to do AAA; it is 1016 inefficient and complex. Second, service nodes must be protected from 1017 DoS attacks by preventing unauthorized requests from entering the 1018 network. Third, services may only be accounted as a bundle (e.g. 1019 network, compute and storage form a single usable service bundle) and 1020 not individually. Fourth, request logging for business analytics is 1021 best done at the network edges and not in individual services. 1023 A provider may also wish to hide network topology of services, and 1024 may abstract locations from user-visibility. For instance, a provider 1025 may publish one interface to access all services although these 1026 services are orchestrated by service-specific orchestrators. And 1027 these orchestrators may be situated in different locations. 1029 12.3. Service Policies 1031 Complex services require coordination of multiple resources. A VM for 1032 instance may need network attached storage, network based security 1033 and network quality of service. The VM service may be regarded 1034 incomplete without the combination of all services. But, much of this 1035 is a matter of policy. Some VMs may require network attached storage, 1036 while others don't. Some VMs may need firewalls, while others may 1037 just need encryption of data. Some services may need a specific 1038 amount of network bandwidth to be available. 1040 Policies associated with services can be abstracted from clients and 1041 servers. Accordingly, when a client requests for a VM, the request 1042 may be modified to include storage, security and quality of service 1043 requests before it reaches the server. Likewise, if a user is not 1044 authorized to request high-end services, their requests might be 1045 automatically downgraded to the appropriate grade of service. This is 1046 a function of policies that a provider and customer define. 1048 This means that an AllocateVM request may do different things for 1049 different classes of users. Users may be upgraded or downgraded in 1050 the level of services, while using the same AllocateVM request. This 1051 means that the syntax and semantics of a request is not fixed in 1052 advance. Rather, it is determined based on context, and different 1053 factors may be used to modify these requests in transit. 1055 It is important to restrict the syntax and semantics of a request 1056 from an end-user perspective. It is also important to offload this 1057 restriction from the service itself. Thus, a server should be able to 1058 support a superset of request parameters, to allow any user to access 1059 the service in different ways. But each client may only request a 1060 well-defined subset of those parameters, based on prior customer or 1061 provider defined policies or SLAs. The validation and tweaking of 1062 request parameters in a user-specific manner should be controlled by 1063 policy in transit. In effect, the requests that a client makes and 1064 the requests that a server receives can be very different based upon 1065 the policies that modify the request in the middle. 1067 13. Architecture Requirements 1069 The general principle embedded in the following requirements is sub- 1070 system re-use by identifying common requirements and avoiding 1071 duplication for every new service (XaaS) needing to be deployed. 1073 A-1. To ease the creation of varied services, there SHOULD be a 1074 separation between policy and protocol. Policy MUST deal with 1075 abstract rules about which components make up a service, and how 1076 those individual components must be created, deleted, modified or 1077 moved. Protocol MUST deal with the execution of these rules. 1079 A-2. The interaction between policy and protocol SHOULD take place 1080 at the service orchestrators. Embedding this interaction in the 1081 client and server increases complexity and makes it harder to deploy 1082 new services or customize existing ones. 1084 A-3. The Policy control MUST contain rules for service Authorizing 1085 and Accounting. That is, it must have rules about which users are 1086 allowed to access which services, or how services are customized for 1087 users and the user-specific charging rules to be applied. 1089 A-4. Orchestration MUST be able to use the same Identity Management 1090 infrastructure for all services. Authentication should be performed 1091 by a coherent system across current and new applications. That is, 1092 each new service should not require new sets of mechanisms. Rather 1093 existing support systems should be extensible. Note, this may also 1094 span both provisioning and use of any particular service. 1096 A-5. Orchestration MUST be able to utilize the same Accounting 1097 system across multiple services. New accounting systems should not be 1098 required for each service. Rather, the orchestrator MUST be able to 1099 use the same accounting system to create charging records. 1101 A-6. Orchestration MUST be able to integrate with existing Fault 1102 management systems. Orchestrators MAY offload and/or automate 1103 intelligence to recover from failures. 1105 A-7. Orchestration MUST be able to integrate with existing 1106 Performance management systems. Orchestrators MAY offload and/or 1107 automate intelligence to recover from performance issues. 1109 A-8. Orchestration MUST be able to use common Operational Support 1110 Systems (OSS) such as DNS, DHCP and BOOTP systems. 1112 A-9. Orchestration MUST be able to integrate with existing customer 1113 support and billing systems and/or provisioning new customers (BSS). 1114 This is to enable a single customer interface for all services. 1116 14. IANA Considerations 1118 Not applicable. 1120 15. Conclusions 1122 Interoperable ways of creating, delivering and consuming services is 1123 essential for cloud. To create this interoperability, there is need 1124 for an open standard protocol for exchanging service information. 1125 This document captures the requirements for such a protocol. 1127 We envision that such a protocol can be an essential ingredient of 1128 Cloud Controllers / Proxies to exchange services across multiple 1129 private, public, hosted, community and other clouds. 1131 16. References 1133 16.1. Normative References 1135 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1136 Requirement Levels", BCP 14, RFC 2119, March 1997. 1138 16.2. Informative References 1140 [NIST] DRAFT Cloud Computing Synopsis and Recommendations 1141 http://csrc.nist.gov/publications/drafts/800-146/Draft- 1142 NIST-SP800-146.pdf 1144 [SIP] Session Initiation Protocol 1145 http://www.ietf.org/rfc/rfc3261.txt 1147 17. Acknowledgments 1149 This document was prepared using 2-Word-v2.0.template.dot. 1151 Authors' Addresses 1153 Ashish Dalela 1154 Cisco Systems 1155 Cessna Business Park 1156 Bangalore 1157 India 560037 1159 Email: adalela@cisco.com 1161 Mike Hammer 1162 Reston 1163 Virginia 1164 USA 20190 1166 Email: mphmmr@gmail.com 1168 Monique Morrow 1169 Cisco Systems [Switzerland] GmbH 1170 Richistrasse 7 1171 CH-8304 1172 Walllisellen 1173 Switzerland 1175 Email: mmorrow@cisco.com 1177 Peter Tomsu 1178 Cisco Systems Austria GmbH 1179 30 Floor, Millennium Tower 1180 Handelskai 94-96 1181 A-1200 Vienna 1182 Austria 1184 Email: ptomsu@cisco.com