Meeting Minutes ARMD IETF 81st Quebec City Address Resolution for Massive numbers of hosts in the Data center (ARMD) Location:205 ABC meeting room, Quebec City Convention Centre, Quebec, Canada Time:29-July-2011, 0900-1130 - Friday, Morning Session I Chairs:Benson Schliesser (bschlies@cisco.com) & Linda Dunbar (linda.dunbar@huawei.com) Jabber:armd@jabber.ietf.org URL:http://tools.ietf.org/wg/armd/ -------------------------------------------------------------------------------- 1. Meeting Administrivia (chairs - 05 min) ?Welcome ?Mailing list and URL ?Minutes Scribe ?Jabber Scribe ?Blue Sheets Slides now being upload to meeting materials, already on webex 2. ARMD Call for Investigation (chairs - 10 min) draft-ietf-armd-call-for-investigation-00 asks a few questions to prompt the thought. Document is only temporary. Will be allowed to expire. Many people raised hand showing that they read the draft. There is no questions There was a NANOG session where operators talked about their problems David Black: are you also interested in DC designer＊s feedback about how address resolution works? 3. Problem Statement for ARMD(Thomas Narten - 15 min) draft-narten-armd-problem-statement-00 deliberately narrowly scoped find to identify representative problems seen in the field Himanshu : - how many DC operatprs are here. About 5? Comment regard to Thomas remarks: history of WG very strange. Scope keeps getting reduced. Are we going to do any work here? Benson: requirements is work, no protocols Then where? Chairs 每 we＊re not doing protocol at this stage of ARMD. Maybe in the future after re-chartering. Ron Bonica, Ops AD 每 no protocol development in Ops area. Narten 每 let＊s not get hung up in process. Let＊s identify and agree problems /requirements and then work out where to do work. If it takes a year to do requirements, then I think we have blown it. Jabber 每 upload the slides!! 4. NANOG 52 Operators Perspective(Susan Hares - 15 min) draft-hares-armd-nanog52-00 The goal for NANOG ARMD session is for operators to please tell us what the pain points are. Speakers at NANOG ARMD track: Adhost, Google, Yahoo, Merit, and AT&T Research Please go to www.nanog.org to see slides that were presented there 每 mid-range DC operator, Google business case etc Why Layer 2 within Data Centers? For many mid-range DC operators, especially the multi-tenant data centers 每 multiple customers don＊t want to change their apps, needs Layer 2 in DC for their business model to work. See ARP problems. Go Grid＊s Application L2 What We Learned: Lots of Middle tier DCs have multi-tenant Need Layer 2 for business model Middle Tier DCs see ARP problems Want to collaborate with other Middle-Tiers Thomas Narten 每 slide 5, I ruled L2 spanning multiple sites as out of scope. Susan 每 when I asked AdHost 每 They have 3 sites, all in single city Rob Shakir 每 main issue is scaling. I build VPLS which can span globe. Answer: VPLS spans globe. However, the number of nodes interconnected by VPLS is not as many as the number hosts in Data Center. Therefore the address scaling in Data Centers has different dimension than VPLS. X (UK): I do VPLSs that are very large. Latency can be a problem but this group is about address resolution. It is scaling, right? We should document real topologies and say what the problems are with that topology. I am willing to help write this. Easier to share privately. My problem with getting people to interact with the IETF is that people think it is irrelevant. Benson: Latency may be a problem not in scope, but what if latency affects address resolution. Ning So: People try to do one size fits all models. When we talk to customers, they have already been running their own data centers around the world. They want to off-load some stuff on us. For an indefinite time they will want to continue to operate their own data centers along with using ours. Warren 每 can you supply pic with your problem? Rob (UK operator) 每 We should have topology pics to identify actual issues. Need to generalize so not commercial info. I will help. Rob 每 I talk with people who think ietf are irrelevant. Ning So 每 we want to run multi-tenant DC. Customers want to offload their internal DCs to us, but will keep running their own DC 每 so will run over both. So the architecture described in slide 5 makes sense to us. Andrew McGregor 每 need switch vendors as well (hands 每 they are here). The scenario is fairly frequent. Cathy Ernson 每 need providers to come to NANOG to give input. Eric Kline: 每 where slides say ARP do they mean address resolution? Answer: Yes Randy Bush: Many operators think they gave considerable input at NANOG. I think we have an ear resolution problem. Benson: Yes, we had a good turnout. Operators can talk to us privately. We also want discussion on the list about this. Need to end up with text. Ning So: Responding to the don't do that comment: Vendor says don't do that. We say don't do that. Customer says that is what I want and what your competitors are providing. So we come back to push vendor to fix it. Such as BGP route problem. Benson: Both views seem reasonable. Different scenarios. Warren: 每 doctor says don＊t do it if it hurts. Big Layer 2 has problems. There is a reason the Internet is not a big Layer 2 network. Maybe a BCP saying just don＊t do that will be the result. VM mobility through overlays, subnet etc as solutions Ning SO: 每 customers can insist we do this. Randy 每 Let＊s get down to technical discussion or else go for coffee. 5. Address Resolution Statistics: Jim Rees, Manish Karir 每 Merit Network draft-karir-armd-statistics-01 500 hosts, configurable interconnect rate. Dino 每 did you study unknown unicast (bridge floods)? We have spikes due to that but didn＊t analyze much. Anoop Ghawani 每 is it spanning tree? Yes. Benson 每 VM does trick so Dino 每 upstream links at top are L3? Yes CPU load spikes to 30% Eric Kline 每 why is cpu load higher for arp than ND? Manish 每 arp broadcasts go everywhere, ND uses multicast. Eric Vynke 每 What is the size? answer /22. Anoop: lower load due to not getting ND messages 每 speculation on the arp vs nd issue. Dino: was packets to switch the same for v4 & v6? 900-1000 v6, 1000-1400 v4 ARP/ND traffic scaled linearly with number of hosts. Dino 每 what＊s the blade? Dell, 2 cores on each blade Thomas Narten 每 what papers are available, the nanog charts are not readable. These slides are a bit better, would like to make data available. Thomas: Interesting work. What papers or docs are available? Chart resolution not good on NANOG documents. Jim: slides better Himanshu 每 why does VM migration have no impact? VMware plays tricks to minimise service disruption during migration. David Black 每 which VMware switch? Manish 每 We bypassed switch William Globus 每 did you look at switch or blades? Monitoring traffic outside cabinet. Wes George 每 spending a long time conjecturing on VMware, can we ask them directly. Warren 每 extrapolating to 20k hosts suggests over 100% load. Actually arp load is in the noise. Manish 每 not number of hosts but traffic pattern. Randy Bush 每 one trailer is on order of 20k hosts. You＊d like trailer to be L2. This is idea of the scale. Thomas Narten 每 would be useful to document different implementations of ARP, they can have very different algorithms. Rob Shakir 每 care where problem is in implementation of protocol or protocol space. Florin balus 每 may be interesting to get data from other environement, eg VPLS. Several 1000 hosts supported on one box, arp load is 3-6% of cpu. Warren 每 my quick rough estimate, 20k hosts in mesh is 660 resolutions per sec 每 should be very easy to do. 6. Address Resolution issues induced by VPN oriented data centre services (Ning So: Verizon) draft-so-armd-vdcs-ar-00 Network controlling DC & mobility of apps etc. An idea at this stage. Dino 每 in terms of scalability shouldn＊t matter, it＊s management. ? Ning 每 edge routers getting hit by complexity Thomas N 每 slide 8 每 is address conflict the pain point? Different issue to address resolution. Anoop [Brocade] 每 solve clashes with isolation technology Andrew McGregor 每 mitigiation techniques can make address resolution worse/. Manish 每 i think your problem belongs to the ARMD wg Manchew Ciena - +1 Rob Shakir UK 每 need to limit failures etc, so won＊t connect to same box. Ning So 每 already there are several boxes. Manish 每 want to maximise sharing. Florent 每 key is to limit number of hosts per broadcast domain; this might be 5 hosts for vpls. David Dyson 每 what＊s the limit? Warren 每 and these 5 things can be routers. Warren 每 can do stuff on Hypervisor. How much of the broadcasts is ARP? 每 broadcast mitigation maybe the key approach. Wes 每 focus on mid size DCs. Write up how did experiments (or code) 7. Vulnerability Issues in Migration每 Yizhou Li draft-liyz-armd-vm-migration-ps-01 Narten 每 when VM1 moves has same MAC address? Yes. Warren 每 arp maps from ip to mac address. IF1 & IF2 are same subnet? Yes. L2 needs to know MAC address has moved, this is not arp but bridge learning Thomas 每 There is a switch learning problem when you move Erik Nordmark 每 just want to reliably update learning tables, probably easiest to do with a broadcast packet, so ARP packet is just convenient. Andrew 每 yes, but gratuitous arp is the right thing to do. David Black 每 this concept has been used successfully in the past. Thomas 每 duplicate address detection was late addition Warren 每 DHCP server does lease. Benson 每 does congestion break arp and nd differently? Dino 每 problem is lots of data from data to control plane, arp can get lost here. Real issue for router & switch vendors. = Meeting closed.