Network Working Group Y. Gu Internet-Draft Huawei Intended status: Standards Track Y. Fan Expires: December 10, 2011 China Telecom June 12, 2011 Policies and dynamic information migration in DC draft-gu-opsa-policies-migration-00 Abstract Virtualization and Virtual Machine (VM) migration provide Data Center with feasibility and improves the utilization of limited physical resource, e.g. switches/routers, servers and links. Meanwhile, a variety of policies (e.g. ACL, firewalls, load balancers, IPS and QoS) are deployed in Data Center to improve system security and gurantee SLA. Those polices are executed by rules configured or generated on network devices. E.g. packet filtering policies are executed by Access Control List on switches or firewalls. Another example is Load balancer (LB) who extablishes TCP/HTTP connections with external clients and balances connections among server farm. During this process, TCP connection tables are dynamically generated on LB. When VM migrates, the network devices that processing and forward VM's packets may change. In order to keep VM's running serives and guanrantee security on new place, VM-relevant policies, including static policies as well as the dynamically generated information, need to migrate with VM. This draft describes some examples of the policies that need to migrate with VM, the problems that need to consider when migrate polices in Data Center. The goal is to justify that it is necessary for IETF to make new effort on management of virtualized Data Center. Status of this Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at http://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." Gu & Fan Expires December 10, 2011 [Page 1] Internet-Draft policy and dynamic information migration June 2011 This Internet-Draft will expire on December 10, 2011. Copyright Notice Copyright (c) 2011 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Gu & Fan Expires December 10, 2011 [Page 2] Internet-Draft policy and dynamic information migration June 2011 Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4 2. Terminologies and concepts . . . . . . . . . . . . . . . . . . 5 3. Policies Classification . . . . . . . . . . . . . . . . . . . 6 3.1. Static Policies . . . . . . . . . . . . . . . . . . . . . 7 3.1.1. Use Cases . . . . . . . . . . . . . . . . . . . . . . 7 3.2. Dynamic Information . . . . . . . . . . . . . . . . . . . 9 3.2.1. Use Cases . . . . . . . . . . . . . . . . . . . . . . 10 4. Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . 22 4.1. Policy migration within a single site . . . . . . . . . . 22 4.2. Polices migration in asymmetric architecture . . . . . . . 23 4.3. Policies migration between DC sites . . . . . . . . . . . 24 5. General Considerations . . . . . . . . . . . . . . . . . . . . 25 5.1. Time-sensitive . . . . . . . . . . . . . . . . . . . . . . 26 5.2. Notification . . . . . . . . . . . . . . . . . . . . . . . 27 5.3. Functionality . . . . . . . . . . . . . . . . . . . . . . 27 5.4. Resource Capability . . . . . . . . . . . . . . . . . . . 29 5.5. Migration Feedback . . . . . . . . . . . . . . . . . . . . 29 6. Security Considerations . . . . . . . . . . . . . . . . . . . 29 7. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 29 8. References . . . . . . . . . . . . . . . . . . . . . . . . . . 29 8.1. Normative Reference . . . . . . . . . . . . . . . . . . . 29 8.2. Informative Reference . . . . . . . . . . . . . . . . . . 30 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 30 Gu & Fan Expires December 10, 2011 [Page 3] Internet-Draft policy and dynamic information migration June 2011 1. Introduction Data centers can host tens or even thousands of different applications. Some are simple applications such as web servers providing static content, while some may be very complex, e.g. e-commerce, that requiring all around privacy protection and data security. Clients of Data Center, unlike server hosting clients, raise more strict QoS and Security requirements. To satisfy different level of security requirements and to manage and improve the performance of these applications, data centers typically deploy a large variety of policies on network devices, e.g. interface security functions on switches and routers, and middleboxes, including firewalls, load balancers, SSL offloaders, web caches and intrusion prevention boxes. To satisfy QoS requirements, Data Center also implement QoS mechanism as ISP network. IEEE 802.1 DCB working group defines a series of standards to guarantee quality of service. 802.1Qau - Congestion Notification 802.1Qaz - Enhanced Transmission Selection 802.1Qbb - Priority-based Flow Control Without regard to mobile network, the existing DC network management has a pre-assumption that end hosts will not move. If an end host moves, because the physical link has to break down and the service also has to break down, the network can treat it as two separated parts: one host leave the network and another host join the network. Server Virtualization and Virtual Machine Migration changes the situation and disable the preassumption. With server virtualization, providers can reduce networking cost. To support the same volume of services, fewer network devices, servers and links are required than before. Multiple Virtual Machines (VMs) are established within a single physical server and the VMs are allowed to relocate to a different servers within the same subnet of Data Center, or even among different sites of a Data Center. This is so called VM Migration. VM Migration brings flexibility to Data Center, meanwhile it makes network management more complex and challenging. Unlike servers power off and power on again, in which case, servers know and can bear services disruption, VM Migration requires that VM's IP Address shouldn't change and running service mustn't be disrupted. Meawhile, security level should be guaranteed. In order to satisfy the above requirements, the VM relevant polices need to Gu & Fan Expires December 10, 2011 [Page 4] Internet-Draft policy and dynamic information migration June 2011 migrate with VM. IEEE 802.1Qbg has developed VSI Discovery and Configuration Protocol (VDP) to enable ajacent bridge to discover VM and configure corresponding static polices. The author has presented the problem to IEEE 802.1 DCB task group. DCB has recognized this problem and make extension to VDP to notify VM status, which will be introduced in Section 5.2. IEEE 802.1 DCB would like to know furture progress of this problem in IETF and ask the author to let know the furthur progress. In the following sections, detail examples and senarios are given to explain why and which polices need to migrate with VM. 2. Terminologies and concepts The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119]. Source Network Device, Source switch, or Source device: the network device/switch/device from where the VM migrates. I.E. VM is originally located under the source network device/switch/device. Destination Network Device, Destination switch, or Destination device: the network device/switch/device to where the VM migrates. I.E. VM is relocated to the destination network device/switch/device. TCP connection table: A table containing TCP connection-specific information. VM: Virtual Machine; A completely isolated operation system which is installed by software on a normal operation system. An normal operation system can be virtualized into several VM. ACL: Access Control List; A list of rules that specifies which packets can be permited, which should be denied. QoS: Quality of Service; FW: Firewall; a policy based security function, typically used for restricting access to/from specific devices and applications. LB: Load Balancer; A methodology to distribute workload across multiple computers or a computer cluster, network links, central processing units, disk drives, or other resources, to achieve optimal Gu & Fan Expires December 10, 2011 [Page 5] Internet-Draft policy and dynamic information migration June 2011 resource utilization, maximize throughput, minimize response time, and avoid overload. NAT: Network Address Translation. Refer to RFC3303. 3. Policies Classification In this section, we use the following figure as an example network architecture. Some redundant links are omitted for simplicity. the new VM1 under Server4 represents the VM1 after migration. VM1 and new VM1 don't exist at the same time. In the real world, the networking of DC could be different. ------------------------------------------------- | Gateway | ------------------------------------------------- / \ / \ / \ / \ ----- -------- -------- -------- -------- ------ |FW-A|--| AGG1 |----| AGG2 | | AGG3 |----| AGG4 |---|FW-B| ----- -------- -------- -------- -------- ------ | | | | | | | | | \ / | | \ / | | \ / | | \ / | | \/ | | \/ | | /\ | | /\ | | / \ | | / \ | | / \ | | / \ | | | | | | | | | ---------- ---------- ---------- ---------- |Switch1 | |Switch2 | |Switch3 | |Switch4 | ---------- ---------- ---------- ---------- | \ / | | \ / | | \/ | | \/ | | /\ | | /\ | | / \ | | / \ | ---------- ---------- ---------- ---------- |Server1 | |Server2 | |Server3 | |Server4 | ---------- ---------- ---------- ---------- | | | | VM1 VM2 VM3 new VM1 Figure 1: Basic networking for discussion in this draft. One key feature of VM migration is that VM's IP Address shouldn't change, otherwise the running service could be disrupted. In order to achieve this, additional mechanism need to be developed. However, these mechanism are out of the scope of this draft. In policy Gu & Fan Expires December 10, 2011 [Page 6] Internet-Draft policy and dynamic information migration June 2011 migration problem statement, we assume that there already exist an effective mechanism to guarantee VM's IP Address unchanged. Two kinds of polices are introduced in this draft, i.e. static polices and dynamic polices (or dynamic information). 3.1. Static Policies Administrators of DC establish VM profile, which may include static policies for the VM, e.g. VLAN Name, bandwidth indication, and Qos parameters, etc. And they are suppposed to be consistent during the lifetime of the VM. No matter where VM migrates, the static policies need to move with VM for the sake of security, privacy and QoS. Static polices can be described by natural languages or mathematics fomula. They are implementated by specific configurations on different network elements, e.g. a ACL item on physical ports of switches enables a packet filtering policies. In this draft, we discuss the migration of policies, but the policies migration also implies the migration of configurations on network devices, because configurations are embodiment of policies. Policies that need to migrate with VM are those can influence VM's running services or are related to security and billing&accounting. Different network devices will contain different kind of policies. For example, it can be static Access Control Lists on Access switches, QoS on switches and routers, security rules on Firewalls, and load balancing mechanism on Load Balancer, etc. 3.1.1. Use Cases It's hard to enumerate all the VM-related static polices. In this section, we just give two examples of static policies to show the necessity of static polices migration. 3.1.1.1. Access Control List shows the influence of lack of ACLs on destination switch. There is an Default ACL on source switch (Switch1) deny all packets from IP subnet 10.138.3.0 to Internet. And another ACL 101 allows IP Address 10.138.3.1, VM1's IP Address, to send packets to Internet. VM1 has a running service on it. During service provisioning, VM1 is migrated to Server4 under Switch4, Where there is default ACL to deny all packets. Since ACL 101 on Switch1 is not migrated to Switch4, VM1's IP Address falls into a default ACL which deny all unmatching packets. As a result, packets belonging to the running services are dropped, hence the running service is disrupted. Gu & Fan Expires December 10, 2011 [Page 7] Internet-Draft policy and dynamic information migration June 2011 ------------------------------------------------- | Gateway | ------------------------------------------------- //\ \ / \ / * \ / \ ------ -------- -------- -------- -------- ------ |FW-A|----| AGG1 |----| AGG2 | | AGG3 |----| AGG4 |---|FW-B| ------ -------- -------- -------- -------- ------ |* | | | | | | | |* \ / | | \ / | |* \ / | | \ / | |* \/ | | \/ | ACL 101: |* /\ | | /\ | permit VM1 |* / \ | | / \ |Default ACL: Default ACL:|* / \ | | / \ |deny all deny all |* | | | | | | |Packets dropped ---------- ---------- ---------- ---------- |Switch1 | |Switch2 | |Switch3 | |Switch4 | ---------- ---------- ---------- ---------- |* \ / | | \ / |/\ \/ |* \/ | | \/ |* /\ |* /\ | | /\ |* |* / \ | | / \ |* ---------- ---------- ---------- ---------- |Server1 | |Server2 | |Server3 | |Server4 | ---------- ---------- ---------- ---------- |* | | |* |* | | |* VM1 VM2 VM3 new VM1 Figure 2: VM migration without ACL migration 3.1.1.2. VLAN VMs with similar properties could be grouped into a VLAN, e.g. end hosts providing Web service are grouped into Web-VLAN. Each VLAN has a unique VLAN ID. Only when a VM is configured into a specific VLAN, can it receives broadcast packets from other VMs in the same VLAN. In , VM1 belongs to VLAN1. Assuming VLAN1 is not configured on the port on AGG4 that attaching Switch4, and VM1 migrates to new VM1, Broadcast pakcets from another VM belonging to VLAN1 are forwarded to AGG4, however they can not be forwarded to Switch4, since broadcast packets are sending only to VLAN1-enabled port. The *** line represents VLAN1 broadcast, which can not access Server4. New VM1 can not receive VLAN1 broadcast. Gu & Fan Expires December 10, 2011 [Page 8] Internet-Draft policy and dynamic information migration June 2011 ----------------------------------- | Gateway | ----------------------------------- | | ------------------------------------------ | Core Switch | ------------------------------------------ /* \ / *\ /* \ / \/\ ------ ------- ------- -------- ------- ------ |FW-A|----|AGG1 |----| AGG2| | AGG3 |--| AGG4|---|Fw-B| ------ ------- ------- -------- ------- ------ |* *| | | | | | | \/ No VLAN1 |* *\ / | | \ / | /\ |* *\ / | | \ / | |* *\/ | | \/ | |* /*\ | | /\ | |* / * \ | | / \ | |* / * \ | | / \ | |* | * | | | | | | --------- --------- --------- ---------- |Switch1| |Switch2| |Switch3| |Switch4 | --------- --------- --------- ---------- VLAN1 |* \ / * |VLAN1 | \ / | |* \/ * | | \/ | |* /\ * | | /\ | |* / \ \/ | | / \ | --------- --------- --------- ---------- |Server1| |Server2| |Server3| |Server4 | --------- --------- --------- ---------- |* | | | |\/ | | | VM1 VM2 VM3 new VM1 Figure 3: VM migration without VLAN migration 3.2. Dynamic Information Except for the static policies, some dynamic information could also be generated by network devices to assist packet processing. TCP connection table on proxy is an obvious example. Proxy intervenes the direct connection between external client and internal server, which can protect internal server from attacking. Proxy establishes TCP connection with external client for internal server, a TCP connection Table being generated on proxy. Then proxy establishes TCP connection with internal server for external client, another TCP connection table being generated. These TCP Connection table can not Gu & Fan Expires December 10, 2011 [Page 9] Internet-Draft policy and dynamic information migration June 2011 be configured by network manager, but is generated by network devices, e.g. Firewalls and proxies, according to real connections. Another example is cumulative data, e.g. how many packets/TCP connecition requests an end host has sent. This information can only be generated by network devices according to real traffic. Configurations could be generated dynamically by network devices themselves according to the dynamic informaiton, e.g. Dynamic ACLs. In following section, we enumerate several kinds of dynamic information. A dynamic information could be generated on different network devices, meanwhile a specific network devices could hold more than one kind of dynamic information. In following section, we introduce possible dynamic informaiton on specific network devices. 3.2.1. Use Cases 3.2.1.1. Switches We use switch to represent Top of Racks, Bridges and switches. These devices have slightly different definition and functions, and may be used or combinedly used in differenct scenarios. 3.2.1.1.1. Dynamic ACL In , assuming a default ACL is configured on Switch1 that all traffic is denied unless the end host is authorized and authenticated. VM1 has been authenticated on source device and a dynamic ACL has been generated which allows VM1's traffic to pass through. A 'Deny all' default ACL is configured on Switch4 too. VM1 migrates to destination device which attaches to Swtich4. Because Switch4 regards new VM1 as an unathenticated end host, according to default ACL, all data packets from VM1 will be dropped. In order to continue service, VM1 has to authenticate again. In this case, running service is disrupted. If authentication and dynamic ACL, generated on Switch1, can be migrated to Switch4, data packets from VM1 are permitted, hence, without regard to influence caused by features, running service can continue. Gu & Fan Expires December 10, 2011 [Page 10] Internet-Draft policy and dynamic information migration June 2011 ------------------------------------------------- | Gateway | ------------------------------------------------- //\ \ / \ / * \ / \ ------ ------- ------- ------- ------ ------ |FW-A|-----| AGG1 |---| AGG2 | | AGG3|----| AGG4|---|FW-B| ------ ------- ------- ------- ------ ------ |* | | | | | | | |* \ / | | \ / | |* \ / | | \ / | |* \/ | | \/ | |* /\ | | /\ | |* / \ | | / \ | |* / \ | | / \ | |* | | | | | | | Default ACL: --------- --------- --------- --------- Default ACL: deny all |Switch1| |Switch2| |Switch3| |Switch4| deny all --------- --------- --------- --------- |* \ / | | \ / /\| \/ VM1's pakcets Dynamic ACL: |* \/ | | \/ *| /\ are dropped. Permit VM! |* /\ | | /\ *| |* / \ | | / \ *| --------- --------- --------- --------- |Server1| |Server2| |Server3| |Server4| --------- --------- --------- --------- |* | | *| |* | | *| VM1 VM2 VM3 new VM1 Figure 4: VM migration without dynamic ACL migration 3.2.1.1.2. DHCP snooping Assuming Switch1 is DHCP Snooping Enabled and a DHCP Snooping mapping item is created for VM1: (IP-VM1: MAC-VM1). This mapping is created dynamically by listening to DHCP Response message. If VM1 migrate to destination device, since the IP Address of VM1 doesn't change, there is no DHCP Request sent by VM1 before lease time expires. So on Switch4, no mapping information can be generated by listening to DHCP response, and all traffic from VM1 will be dropped. If DHCP snooping table on Switch1 can be migrated to Switch4, Switch4 learns from the migrated DHCP snooping talbe that packet with IP-VM1 and MAC-VM1 can be regularly forwarded. Gu & Fan Expires December 10, 2011 [Page 11] Internet-Draft policy and dynamic information migration June 2011 ------------------------------------------------- | Gateway | ------------------------------------------------- //\ \ / \ / * \ / \ ------ ------- ------ -------- ------- ------ |FW-A|---|AGG1 |----| AGG2| | AGG3 |---|AGG4 |---|FW-B| ------ -------- ------ -------- ------- ------ |* | | | | | | | DHCP |* \ / | | \ / | Response |* \ / | | \ / | \ |* \/ | | \/ | \ |* /\ | | /\ | \ |* / \ | | / \ |DHCP snooping enabled \ |* / \ | | / \ |No DHCP request DHCP >|* | | | | | | |from new VM1 snooping --------- --------- --------- ---------- enabled |Switch1| |Switch2| |Switch3| |Switch4 | --------- --------- --------- ---------- ---------- |* \ / | | \ / /\| \/ VM1's packets |Snooping| |* \/ | | \/ *| /\ are dropped |Table | |* /\ | | /\ *| ---------- |* / \ | | / \ *| -------- --------- --------- ---------- |Server1| |Server2| |Server3| |Server4 | -------- --------- --------- ---------- |* | | *| |* | | *| VM1 VM2 VM3 new VM1 Figure 5: VM migration without DHCP snooping table migration 3.2.1.1.3. IGMP snooping IGMP snooping is similar to DHCP Snooping. Multicast membership is created on ports by listening to IGMP JOIN or membership report messages. When a switch receives a multicast packet, it forwards the packet to the port which has joined the multicast group. In , assuming VM1 sends an IGMP JOIN Group1 message to Multicast server, port1 and port-uplink on Switch1 check the JOIN message and tag on port1 and port-uplink that they should forward Multicast packets from Group1. If VM1 migrates to destination, since migration is transparent to VM, VM1 will not send IGMP membership report until next IGMP General Query is received. If port2 on Swtich4 didn't join Group1, multicast packet from Group1 won't be forward to port2, then VM1 won't be able to receive Multicast packets. Gu & Fan Expires December 10, 2011 [Page 12] Internet-Draft policy and dynamic information migration June 2011 ------------------------------------------------- | Gateway | ------------------------------------------------- /* \ / *\ /* \ / *\ ------ -------- -------- -------- -------- ------ |FW-A|-----| AGG1 |----| AGG2 | | AGG3 |---| AGG4 |--- |FW-B| ------ -------- -------- -------- -------- ------ |* | | | | | | *| IGMP |* \ / | | \ / *| JOIN Snooping|* \ / | | \ / *| \ |* \/ | | \/ *| \ |* /\ | | /\ *| \ |* / \ | | / \ *|IGMP enabled \ |* / \ | | / \ *|No IGMP JOIN >|* | | | | | | \/|from new VM1 IGMP ---------- ---------- ---------- ---------- \/ enabled |Switch1 | |Switch2 | |Switch3 | |Switch4 | /\ ---------- ---------- ---------- ---------- |* \ / | | \ / | ---------- |* \/ | | \/ | |Group1: | |* /\ | | /\ | |Port1 | |* / \ | | / \ | ---------- ---------- --------- --------- ---------- |Server1 | |Server2| |Server3| |Server4 | ---------- --------- --------- ---------- |* | | | |\/ | | | VM1 VM2 VM3 new VM1 Figure 6: VM migration without IGMP snooping table migration 3.2.1.1.4. Cumulative Data To ensure VMs consume no more than assigned bandwidth and to enable QoS control, billing and accounting, bridges need to cumulate packets number and calculate packet rate. Lack of these cumulative data and statistic data on destination bridge decrease the accuracy. 3.2.1.2. Firewall Fireall (FW) is a filtering device that separates LAN segments, giving each segment a different security level and establishing a security perimeter that controls the traffic flow between segments. There are different types of firewalls based on their packet- processing capabilities and their awareness of application-level information: Gu & Fan Expires December 10, 2011 [Page 13] Internet-Draft policy and dynamic information migration June 2011 Packet-filetering firewalls Proxy firewalls Stateful firewalls Hybrid firewalls 3.2.1.2.1. Dynamic ACL Packet filtering Firewall filters packet according to ACL. Dynamic ACL could be generated to protect internal network/servers from attacks, similar to dynamic ACL on Switches. 3.2.1.2.2. TCP Connection Table Proxy Firewall proxies TCP connections between internal servers and external clients. It establishes TCP connection with external client and then it establishes TCP connection with internal server. TCP connection tables are cached on Proxy Firewall to indicate how to transform following packets belonging to this TCP connection. External Client Firewall Internal server |----- TCP SYN ----->| |<-----SYN/ACK ------| |-------ACK ----->| |--------TCP SYN ------>| |<-------SYN/ACK -------| |--------ACK ------>| |-------DATA ------>|--------DATA ------>| |<------DATA -------|<-------DATA -------| Figure 7: Proxy Firewall Gu & Fan Expires December 10, 2011 [Page 14] Internet-Draft policy and dynamic information migration June 2011 ------------------------------------------------- | Gateway | ------------------------------------------------- ---------------- /* \ / *\ |TCP Conn Table| /* \ / *\ No corresponding ---------------- /* \ / *\ TCP Conn Table ------ **** -------- -------- -------- -------- ****>------ |FW-A|-----| AGG1 |----| AGG2 | | AGG3 |----| AGG4 |----- |FW-B| ------ **** -------- -------- -------- -------- \/ ------ |* | | | | | | | /\ |* \ / | | \ / | |* \ / | | \ / | |* \/ | | \/ | |* /\ | | /\ | |* / \ | | / \ | |* / \ | | / \ | |* | | | | | | | ---------- ---------- ---------- ---------- |Switch1 | |Switch2 | |Switch3 | |Switch4 | ---------- ---------- ---------- ---------- |* \ / | | \ / | |* \/ | | \/ | |* /\ | | /\ | |* / \ | | / \ | ---------- ---------- ---------- ---------- |Server1 | |Server2 | |Server3 | |Server4 | ---------- ---------- ---------- ---------- |* | | | |\/ | | | VM1 VM2 VM3 new VM1 Figure 8: VM migration without TCP connection table migration A typical TCP Connection Table includes the following data: tcpConnState: The state of this TCP connection. tcpConnLocalAddress: The local IP address for this TCP connection. tcpConnLocalPort: The local port number for this TCP connection. tcpConnRemAddress: The remote IP address for this TCP connection. tcpConnRemPort: The remote port number for this TCP connection. A TCP Connectin Table could also includes the following information: Gu & Fan Expires December 10, 2011 [Page 15] Internet-Draft policy and dynamic information migration June 2011 Sequence Number: the sequence number in the packet header the sender is going to send. Acknowledgement Number: the sequence number in the packet header the receiver is hoping to receive. Idle time: the time that the tcp connection table hasn't been updated. Assuming TCP Connection Table item is generated for VM1 on Fw-A, the information is as follows: tcpConnState == Established tcpConnLocalAddress == 10.138.3.1 tcpConnLocalPort == 1234 tcpConnRemAddress: == 192.167.22.3 tcpConnRemPort == 4321 VM1 migrates to Server4 under Switch4, without TCP Connection Table items migration. The packets belonging to this TCP Connection will continue coming, which will pass FW-B, instead of FW-A. Because there is no TCP Connection Table for VM1 on FW-B, the following packets belonging to the TCP Connection will be dropped, hence the running service is broken down. 3.2.1.2.3. Cumulative Data One example for cumulative data is unfinished TCP Connection established by a specific VM. In order to avoid TCP SYN flood, Firewall may control the unfinished TCP Connection established by a single end host by setting a threshold. For example, the NM set the threshold to 50, and VM1 has established 30 unfinished TCP connections. If the cumulated TCP connection number isn't migrated to destination devices, the destination device will allow VM1 to establish up to 50 unfinished TCP connections. For the single destination device, the unfinished TCP connections established by VM1 is under control, but for the whole DC, VM1 has established 80 unfinished TCP connections. So VM1 has consume more resoureces than allowed. 3.2.1.2.4. NAT Address Mapping Data center may implement network address translation. Sometimes NAT function is enabled on Firewall. Private IP Add. is assigned to Gu & Fan Expires December 10, 2011 [Page 16] Internet-Draft policy and dynamic information migration June 2011 internal VM. When VM communicate with external client, NAT function assigns a public IP Add. to the communicating internal VM. A mapping between private IP Add. and public IP Add. is recorded on NAT function. Assuming a mapping from Pri-IP-VM1 to Pub-IP-VM1 is generated on FW-A. When VM1 migrates to Server4 without the address mapping, following packets from external client can not be translated to correct Pri-IP-VM1, and vice versa. ------------------------------------------------- | Gateway | ------------------------------------------------- //\ \ / \* Dst: Pub-IP-VM1 Src: Pub-IP-VM1 /* \ / \* No Address mapping ----------------- /* \ / \* from Pub-IP-VM1 |Address Mapping| /* \ / \* to Private-IP-VM1 ----------------- /* \ / \*Packets are dropped ------- ****>-------- -------- ------- ------- ****>------ |NAT-A|------| AGG1 |---| AGG2 | |AGG3 |----| AGG4 |-----|NAT-B| ------- <****-------- -------- ------- -------\/ ------- |* | | | | | | | /\ |* \ / | | \ / | |* \ / | | \ / | |* \/ | | \/ | |* /\ | | /\ | |* / \ | | / \ | |* / \ | | / \ | |* | | | | | | | --------- --------- --------- ---------- |Switch1| |Switch2| |Switch3| |Switch4 | --------- --------- --------- ---------- |* \ / | | \ / | Src: |* \/ | | \/ | Pri-IP-VM1 |* /\ | | /\ | |* / \ | | / \ | --------- --------- --------- ---------- |Server1| |Server2| |Server3| |Server4 | --------- --------- --------- ---------- |* | | | |\/ | | | VM1 VM2 VM3 new VM1 Figure 9: VM migration without NAT Address Mapping migration Gu & Fan Expires December 10, 2011 [Page 17] Internet-Draft policy and dynamic information migration June 2011 3.2.1.3. Load Balancer When a cluster of servers are organized to provide same services, Load Balancer (LB) is used to balance service load between servers. All external requests are sent to LB, before they are redirected to a specific server. First, external clients extablish TCP connection with LB. Then LB decides which specific server should take care of the requests. Two ways to redirect the requests, i.e. transparent LB and proxy LB. The following dynamic information applies for both ways. 3.2.1.3.1. Connection Table Layer 4 LB identifies and makes load-balancing decesion by Layer 4 Protocol, e.g. TCP. Layer 4 LB generates TCP connection table which is similar to TCP connection table on Firewall. External Client Load Balancer Internal server | ------- TCP SYN ------->|-------TCP SYN ----->| |<--------SYN/ACK --------|<------SYN/ACK ------| |---------ACK ------->|-------ACK ----->| |---------HTTP REQ ------->|-------HTTP REQ----->| |<--------HTTP ACK --------|<------HTTP ACK------| |---------DATA ------->|-------DATA ----->| |<--------DATA --------|<------DATA ------| Figure 10: Layer 4 Load Balancing Layer 5 LB identifies and makes load-balancing decesion by Layer 5 protocol, e.g. HTTP, SMTP and FTP, that is found in packet payload. The Layer 5 information could be HTTP URL, HTTP cookie, or HTTP header field user agent. Layer 5 LB generates connection table for HTTP, SMTP, and FTP, etc. External Client Load Balancer Internal server | ------- TCP SYN ----->| |<--------SYN/ACK ------| |---------ACK ----->| |---------HTTP Req ----->| |-------TCP SYN ----->| |<------SYN/ACK ------| |-------ACK ----->| |-------HTTP Req----->| |---------HTTP ACK ----->|-------HTTP ACK----->| |---------DATA ----->|-------DATA ----->| |<--------DATA ------|<------DATA ------| Figure 11: Layer 5 Load Balancing shows the result of lack of Connection table on destination Gu & Fan Expires December 10, 2011 [Page 18] Internet-Draft policy and dynamic information migration June 2011 LB. ------------------------------------------------- | Gateway | ------------------------------------------------- ------------- /* \ / *\ No VM1's Conn Table |Conn Table | /* \ / *\Packets could be dropped ------------- /* \ / *\or mis-redirected ------ <**** ------- -------- ------- ------- ****>------- |LB-A|------| AGG1 |---| AGG2 | | AGG3|---|AGG4 |----- |LB-B| ------ ****> ------- -------- ------- ------- <**** ------ |* | | | | | | |* |* \ / | | \ / |* |* \ / | | \ / |* |* \/ | | \/ |* |* /\ | | /\ |* |* / \ | | / \ |* |* / \ | | / \ |* |* | | | | | | |* ---------- ---------- --------- --------- |Switch1 | |Switch2 | |Switch3| |Switch4| ---------- ---------- --------- --------- |* \ / | | \ / |* |* \/ | | \/ |* |* /\ | | /\ |* |* / \ | | / \ |* ---------- ---------- ---------- ---------- |Server1 | |Server2 | |Server3 | |Server4 | ---------- ---------- ---------- ---------- |* | |* | |\/ | |\/ | VM1 VM2 VM3 new VM1 Figure 12: VM migration without Connection table migration 3.2.1.3.2. Sticky Table Session persistence refers to the capability of the Load alancer to logically group multiple connections from the same client transaction to the service. Session persistence is also known as stickiness or sticky connections because the goal is to stick two or more connections together as part of a single session. This grouping is done to ensure the connections are handled and forwarded to the groups of servers that are aware of and expect to see the remaining connections. This ensures that the client has a successful interaction with the application. LB usually keeps a sticky table for session persistence, which is Gu & Fan Expires December 10, 2011 [Page 19] Internet-Draft policy and dynamic information migration June 2011 used to match incoming requests to existing connections so that they can be grouped together. A sticky table could include the following information. Sticky groups Group name (sticky group identifier) Type (sticky group type, according to stick methods) Sticky server farm Backup server farm Aggregate state (to indicate whether the state of the backup server is tied to the virtual server state) Sticky enabled on backup server farm (to indicate whether the backup server farm is sticky) Replicate (to indicate whether to replicate sticky entries on the standby device) Timeout (a mechanism to age out sticky table entries) Timeout active connections (to specify whether to time out sticky table entries if active connections exists after the sticky timer expires) Sticky methods Sticky connections Real servers In , two connections, represented by * line and $ line, are redirected to VM1, and a sticky table is generated on LB-A. VM1 migrates to new VM1, without Sticky table migrates with VM1. A new connection, represented by & line, arriving at LB-B, which can not know that this connection should be redirected to new VM1, because of lack of sticky table. As a result, LB-B redirect the new connection to other VM than new VM1. This mis-redirection could cause key information lost. For example, in e-business, refer to Gu & Fan Expires December 10, 2011 [Page 20] Internet-Draft policy and dynamic information migration June 2011 External Client-X * $ & * $ & * $ & -------------------------------------------- | Gateway | -------------------------------------------- /*$ \ /\*$& /*$ \ / \*$& No Sticky table for client-X ------------- /*$ \ / \*$& Connections from client-X |Sticky Table| /*$ \ / \*$& may be redirected to ------------- /*$ \ / \*$& different servers ----- <*$*$*------- ------ ------- -------- *$&*> ------ |LB-A|------| AGG1 |----| AGG2| | AGG3|----| AGG4 |----- |LB-B| ----- *$*$> ------- ------ ------- --------<*$&* ------ |*$ | | | | | | |*$& |*$ \ / | | \ / |*$& |*$ \ / | | \ / |*$& |*$ \/ | | \/ |*$& |*$ /\ | | /\ |*$& |*$ / \ | | / \ |*$& |*$ / \ | | / \ |*$& |*$ | | | | | | |*$& --------- -------- --------- ---------- |Switch1| |Switch2| |Switch3| |Switch4 | --------- -------- --------- ---------- |*$ \ / | | \ / |*$& |*$ \/ | | \/ |*$& |*$ /\ | | / \ |*$& |*$ / \ | | / \ |*$& ---------- ---------- --------- ---------- |Server1 | |Server2 | |Server3| |Server4 | ---------- ---------- --------- ---------- |$% | |& |*$ |\/ | |\/ |\/ VM1 VM2 VM3 new VM1 ****** $$$$$$ Three connectons from the same external clients. &&&&&& Figure 13: VM migration without Sticky table migration While server or server farm move to a new location under a new LB, in order to keep the existing connections and sessions, as well as session persistence, both connection table and sticky table need to move with servers. Gu & Fan Expires December 10, 2011 [Page 21] Internet-Draft policy and dynamic information migration June 2011 4. Scenarios In previous section, we introduce two classes of policies, i.e. static policies and dynamic information. The introduction is based on a simplized networking architecture and intentionally omit other factors. The polices in the example network implies <1:1> partner, that is polices need to move from one device to another. However, in real world, network architectures are much complicate than the simplized example networking. We introduce several scenarios in this section. Different architecture raise different requirements and migration partner style. 4.1. Policy migration within a single site A DC site means a geographic location, which could include one subnet, or several subnets. A VM migrates to a new location which is in the same subnet as original location. In this case, there might not be policies on Middleboxes that need to be migrated, since new VM1 is under the same Middleboxes as VM1. But dynamic information on switches need to migrate as well. Gu & Fan Expires December 10, 2011 [Page 22] Internet-Draft policy and dynamic information migration June 2011 ------------------------------------------------- | Gateway | ------------------------------------------------- / \ / \ / \ / \ ------ -------- -------- -------- -------- ------ |FW-A|---| AGG1 |----| AGG2 | | AGG3 |----| AGG4 |----- |FW-B| ------ -------- -------- -------- -------- ------ | | | | | | | | | \ / | | \ / | | \ / | | \ / | | \/ | | \/ | | /\ | | /\ | | / \ | | / \ | | / \ | | / \ | | | | | | | | | ---------- ---------- ---------- ---------- |Switch1 | |Switch2 | |Switch3 | |Switch4 | ---------- ---------- ---------- ---------- | \ / | | \ / | | \/ | | \/ | | /\ | | /\ | | / \ | | / \ | ---------- ---------- ---------- ---------- |Server1 | |Server2 | |Server3 | |Server4 | ---------- ---------- ---------- ---------- | | VM1 new VM1 Figure 14: Policy migration within a single DataCenter site 4.2. Polices migration in asymmetric architecture The network architecture could be asymmetric. Refer to Fig16. In this case, there is possible to encounter <1:n> or partnership. The former partnership implies policies on one orginal device might need to migrate to n destination devices. The latter partnership implies policies on multiple original devices might need to migrate to one destination devices. For example, when VM1 migrate from Server 4 to Server1, the dynamic information on both Switch4 and ToR need to migrate to Switch1, which is 2:1 partnership。 Gu & Fan Expires December 10, 2011 [Page 23] Internet-Draft policy and dynamic information migration June 2011 ------------------------------------------------- | Gateway | ------------------------------------------------- / \ / \ / \ / \ ------ ------- ------- -------- -------- ------ |FW-A|---| AGG1|----| AGG2 | | AGG3 |----| AGG4 |---|FW-B| ------ ------- ------- -------- -------- ------ | | | | | \ / | | \ / | | \/ | | \ / | | /\ | | \/ | ---------- ---------- | /\ | |Switch3 | |Switch4 | | / \ | ---------- ---------- | / \ | | \ / | | | | | | \/ | --------- ---------- | /\ | |Switch1| |Switch2 | ---------- ---------- --------- ---------- | ToR | | ToR | | \ / | ---------- ---------- | \/ | | \/ | | /\ | | /\ | | / \ | | / \ | ---------- --------- ---------- ---------- |Server1 | |Server2| |Server3 | |Server4 | ---------- --------- ---------- ---------- | | new VM1 VM1 Figure 15: Polices migration in asymmetric architecture 4.3. Policies migration between DC sites Currently, most VM hot migration is limited in the same data center site. But in the furture there might be requirements for VM migration between data center sites, as shown in . A use case for this scenario is resource coordination between data center sites. A DC provier has two data center sites in different countries, and both of them serve external clients with similar applications. Some of the applications are high priority applications, others are normal priority ones. At first, requests from external clients are redirected to the data center site which are physically close to the external clients. However, at some time, one data center is approaching its capability limits, which could be servers capability limits (i.e. few server capability are available for new requests) or network capability limits (i.e. network can not support so much packet processing requirement, and packet loss happens) and performance is degraded. In order to guarantee high priority Gu & Fan Expires December 10, 2011 [Page 24] Internet-Draft policy and dynamic information migration June 2011 applications, and try best to make fewest damage to normal priority applications, some VMs, who provide normal priority applications, can be migrated to the other data center sites. In this case, policies on switches, middleboxes and gateways need to migrate to new devices on the other data center site. Policies migration between data centers also need to consider the asymmetric structure. ------------------ --------------------- | Gateway | | Gateway2 | ------------------ --------------------- / \ / \ / \ / \ ------ -------- -------- -------- -------- ------ |FW-A|---| AGG1 |----| AGG2 | | AGG3 |----| AGG4 |-----|FW-B| ------ -------- -------- -------- -------- ------ | | | | | | | | | \ / | | \ / | | \ / | | \ / | | \/ | | \/ | | /\ | | /\ | | / \ | | / \ | | / \ | | / \ | | | | | | | | | ---------- ---------- ---------- ---------- |Switch1 | |Switch2 | |Switch3 | |Switch4 | ---------- ---------- ---------- ---------- | \ / | | \ / | | \/ | | \/ | | /\ | | /\ | | / \ | | / \ | ---------- ---------- ---------- ---------- |Server1 | |Server2 | |Server3 | |Server4 | ---------- ---------- ---------- ---------- | | VM1 new VM1 Figure 16: Polices migration between DC sites 5. General Considerations In this section, we enumerate some general considerations on Policy migration. Only with deep consideration on these factors can we achieve a successful VM migration and policy migration. Gu & Fan Expires December 10, 2011 [Page 25] Internet-Draft policy and dynamic information migration June 2011 5.1. Time-sensitive VM migration needs a period to finish memory and register copy. Fig.3 shows the VMware VMotion process. There are three points we should pay attention to. Pre-copy period: VM begins to prepare for migration. In this period, VM pre-copy memory state to the new VM on destination device. The original VM is still power on and service is still running, which means the memory and register could keep changing. The new VM is power-on. VM not running period: The end phase of memory copy. In this period, original VM stop running service, the memory will not change. Original VM finish copying the rest changed memory and register to new VM. New VM is still power-on. VM power-off point: After original VM receives the OK message from new VM, it turns off the power, and meanwhile the new memory starts to run. We can see that it's unrealistic to make a 'zero delay' VM migration, because there is at least about 1 second period (VM not running period) when neither VM is running. Assuming there is a 3rd-party device can GET and SET dynamic information.(This is only an possible way to migrate polices. Polices could also be migrated distributedly. The author has no preferrence to any approaches in this draft.) The 3rd-party device GET dynamic information at Time A, and finish SET at Time B. At Time A, the Sequence Number of VM1's TCP Connection is 99. After 3rd- party device GET dynamic information, VM1's TCP connection keeps transferring packets and Sequence Number increase to 110, until VM not running period begins. During VM not running period, no TCP packet is acknowledged by VM1, so the Sequence Number is 110. At Time B, the destination Firewall is SET by Sequence Number 99. When new VM1 starts, the packets belonging to VM1's TCP connection comes to Firewall-B with Sequence Number 111. Since the receiving Sequence Number doesn't equal to the Acknowleadge Sequence Number of Firewall-B, this packet will be dropped and the running service is broken down. Another example: After Time A, VM1 may establish new TCP connections, which are not migrated to new Firewall, then following packets belonging to new TCP connections will be dropped and running services are disrupted. Gu & Fan Expires December 10, 2011 [Page 26] Internet-Draft policy and dynamic information migration June 2011 ------------------------------------------------------- ^ || | Power-on destination VM ^ | || | | | ||--->| --------------------------- VMotion VM || | Pre-copy memory state abortable running ||--->| |-------->Time A on source|| | | /\ | ||--->| |Dynamic Information | ||--->| |keep changing V || | | \/ ---------------------------------------------------|----------- ^ |---->| Checkpoint-save src VM and | | |---->| transfer state | | | |-----------------------------------| VM not |---->| transfer remaining changed memory |-------->Time B running | | and checkpoint-restore dst VM V | | |-------------------------------------- ~1sec |<----| send restore OK V | | ------------------------------------------------------ | || Power-off source VM | || VM running on destination Figure 17: VMware VMotion process 5.2. Notification As Section 5.1shows, policy migration is very sensitive to migration time. Either later or earlier may cause service disruption. The best migration time is after the original VM stops running and before the destination VM begins running. Only Hypervisor knows the acurate VM status. So we need a notification from server side, Hypervisor in specific, to tell network side when to migrate polices. In fact, the author has presented the problem to IEEE 802.1 DCB task group, which defines VSI Discovery and Configuration protocols (VDP). DCB has recognized this problem and make effort within its scope. DCB has extended VDP, in 802.1 Qbg, to enable notification from Hypervisor to adjacent bridge. DCB expects corresponding IETF WG to make furthur effort to distribute the notification to devices from Layer 3 to Layer 7. And IEEE 802.1 DCB working group would like to see further progress at IETF on this problem. 5.3. Functionality Data center architecture could be heterogeneous, both in physical structure and functionality. Section 4.2 and shows the asymmetric physical structure. The following figure shows the case Gu & Fan Expires December 10, 2011 [Page 27] Internet-Draft policy and dynamic information migration June 2011 where functionality on devices is heterogenous. While VM migrates from one location to a heterogeneous location, it's necessary to negotiate and make sure that essential functions are supported at destination location. Otherwise, VM migration maybe fail. Even VM successfully migrates, there might be security risk and/or service failure. VM Managers who want to migrate VM between heterogeneous architecutre must be aware of the risk. shows two examples of functionality heterogeneous. FW-A is the source Firewall with packet filtering function, and a dynamic ACL is generated on FW-A for VM1. FW-B is the destination Firewall without packet filtering funtion. In this case, dynamic ACL on FW-A can not be implemented on FW-B. The similar case happens on Switch1 and Switch4. Switch1 has DHCP snooping function enabled, while Switch has not. DHCP snooping table from Switch1 can not be implemented on Switch4. ------------------------------------------------- | Gateway | ------------------------------------------------- / \ / \ / \ / \ ------ -------- -------- ------ ------- ------ |FW-A|----| AGG1 |----| AGG2 | | AGG3|----| AGG4|---|FW-B| ------ -------- -------- ------ ------- ------ with | | | | | | | | without Packet | \ / | | \ / | Packet filtering filtering | \ / | | \ / | | \/ | | \/ | | /\ | | /\ | | / \ | | / \ | | / \ | | / \ | | | | | | | | | ---------- ---------- --------- --------- with |Switch1 | |Switch2 | |Switch3| |Switch4| withou DHCP ---------- ---------- --------- --------- DHCP Snooping | \ / | | \ / | Snooping | \/ | | \/ | | /\ | | /\ | | / \ | | / \ | ---------- ---------- ---------- ---------- |Server1 | |Server2 | |Server3 | |Server4 | ---------- ---------- ---------- ---------- | | VM1 new VM1 Figure 18: Heterogeneous functions on devices Gu & Fan Expires December 10, 2011 [Page 28] Internet-Draft policy and dynamic information migration June 2011 5.4. Resource Capability Even if destination devices can support specific function that is required by dynamic information migration, there might be not enough resources to implement migrated polices. For example, there are 5 connection state items related to VM1 on source LB. However, the destination LB has only 4 table items available. That means the destination LB doesn't have enough resource to support this VM migration, and policies migration on network devices may fail. 5.5. Migration Feedback Sometimes, VM manager has ensure, by some way, that proper functionality and plenty of resources are available on destionation devices, by which we mean all the devices that need to handle VM's packets, but dynamic information migration still fails. In this case, VM manager or Hypervisor needs to know the results of policy migration. The feedback from network can help VM Manager to decide whether to rollback the migration or continue migration with risk. 6. Security Considerations The policies and dynamic information described above are all about security. Besides, we need to be careful to avoid poisoned polices from untrusted source. That means no mather how the polices are migrated, be it distributed or centralized way, authentication and verification are required. A reliable notifation from server side to network side is also necessary, so that the destination and original devices can be sure that a real VM migration happens. 7. Acknowledgments I would like to thank the following people for contributing to this draft: Ning Zong, David harrington, Linda dunbar, Susan Hares, Serge manning, Barry Leiba, Jiang xingfeng, Song Wei, Robert Sultan, and many others. 8. References 8.1. Normative Reference [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", March 1997. Gu & Fan Expires December 10, 2011 [Page 29] Internet-Draft policy and dynamic information migration June 2011 [RFC3303] Srisuresh, P., Kuthan, J., Rosenberg, J., Molitor, A., and A. Rayhan, "Middlebox communication architecture and framework", August 2002. 8.2. Informative Reference [Data_Center_Fundamentals] "Data Center Fundamentals", 2003. Authors' Addresses Gu Yingjie Huawei No. 101 Software Avenue Nanjing, Jiangsu Province 210001 P.R.China Phone: +86-25-56624760 Fax: +86-25-56624702 Email: guyingjie@huawei.com Fan Yongbing China Telecom No. 109 Zhongshan Road West Guangzhou, Guangdong Province P.R.China Phone: 86-20-38639121 Fax: 86-20-38639487 Email: fanyb@gsta.com Gu & Fan Expires December 10, 2011 [Page 30]