idnits 2.17.1 draft-irtf-p2prg-simulation-survey-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 73 has weird spacing: '...ulation frame...' == Line 128 has weird spacing: '...ulation frame...' == The document doesn't use any RFC 2119 keywords, yet seems to have RFC 2119 boilerplate text. -- The document date (June 28, 2011) is 4686 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- No issues found here. Summary: 0 errors (**), 0 flaws (~~), 4 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 P2P Research Group V. Gurbani, Ed. 3 Internet-Draft Bell Labs, Alcatel-Lucent 4 Intended status: Informational A. Basu 5 Expires: December 30, 2011 Tokai University and University of 6 Sussex 7 T. Schmidt 8 HAW Hamburg 9 S. Fleming 10 University of Sussex 11 M. Kolberg 12 University of Stirling 13 M. Waehlisch 14 link-lab & FU Berlin 15 June 28, 2011 17 Peer-to-peer simulation frameworks: a survey 18 draft-irtf-p2prg-simulation-survey-00 20 Abstract 22 Peer-to-peer (p2p) protocols, like all distributed protocols, are 23 complex, and therefore harder to debug and study in the wild. This 24 is more true of existing p2p protocols, where changing the behaviour 25 of the protocol --- however minor the change may be --- may result in 26 unknown manifestations on the dynamics of the swarm using that 27 protocol. In lieu of the unintended consequences of perturbing a 28 live swarm, researchers have resorted to simulation frameworks. 29 However, simulation results obtained from one simulator are often 30 hard to reproduce when using another simulation framework. This 31 document surveys existing simulator frameworks prevalent in 32 simulating p2p protocols today in order to quantify any assumptions 33 and characteristics inherent in the simulator. This, we hope, will 34 aid future researchers in choosing the right simulation framework for 35 their abstraction. 37 Status of this Memo 39 This Internet-Draft is submitted in full conformance with the 40 provisions of BCP 78 and BCP 79. 42 Internet-Drafts are working documents of the Internet Engineering 43 Task Force (IETF). Note that other groups may also distribute 44 working documents as Internet-Drafts. The list of current Internet- 45 Drafts is at http://datatracker.ietf.org/drafts/current/. 47 Internet-Drafts are draft documents valid for a maximum of six months 48 and may be updated, replaced, or obsoleted by other documents at any 49 time. It is inappropriate to use Internet-Drafts as reference 50 material or to cite them other than as "work in progress." 52 This Internet-Draft will expire on December 30, 2011. 54 Copyright Notice 56 Copyright (c) 2011 IETF Trust and the persons identified as the 57 document authors. All rights reserved. 59 This document is subject to BCP 78 and the IETF Trust's Legal 60 Provisions Relating to IETF Documents 61 (http://trustee.ietf.org/license-info) in effect on the date of 62 publication of this document. Please review these documents 63 carefully, as they describe your rights and restrictions with respect 64 to this document. Code Components extracted from this document must 65 include Simplified BSD License text as described in Section 4.e of 66 the Trust Legal Provisions and are provided without warranty as 67 described in the Simplified BSD License. 69 Table of Contents 71 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4 72 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . 4 73 3. Criteria for evaluating simulation frameworks . . . . . . . . 5 74 4. List of simulation frameworks . . . . . . . . . . . . . . . . . 5 75 5. Security Considerations . . . . . . . . . . . . . . . . . . . . 6 76 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . . 6 77 7. References . . . . . . . . . . . . . . . . . . . . . . . . . . 6 78 7.1. Normative References . . . . . . . . . . . . . . . . . . . 6 79 7.2. Informative References . . . . . . . . . . . . . . . . . . 6 80 Appendix A. Acknowledgments . . . . . . . . . . . . . . . . . . . 7 81 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 7 83 1. Introduction 85 Peer-to-peer (p2p) protocols, like all distributed protocols, are 86 complex, and therefore harder to debug and study in the wild. This 87 is more true of existing p2p protocols, where changing the behaviour 88 of the protocol --- however minor the change may be --- may result in 89 unknown manifestations on the dynamics of the swarm using that 90 protocol. 92 Researchers contemplating on changing the behavior of an existing p2p 93 protocol have to be careful still, least they inadvertently do more 94 harm than good by introducing their changes. Furthermore, any 95 changes to an existing p2p protocol or a newly developed p2p protocol 96 must be tested and evaluated for validity and reproducibility by the 97 research community. While analytical and mathematical modeling 98 (fluid models, optimization and linear programming) is easily 99 validated, it is harder to validate empirical experiments due to the 100 dynamic nature of the networks, hosts, and interconnections between 101 them. Simulation frameworks are attractive since they provide a 102 controlled environment under which new behavior of p2p protocols can 103 be studied and quantified. 105 The good news is that there is a plethora of simulation frameworks 106 for p2p protocols available today, some of them are surveyed in 107 Naicken et al. [naicken]. However, that survey is dated and does not 108 include simulation frameworks like ns-3 [ns-3] and ProtoPeer 109 [protopeer] that have become available since the survey was 110 published. 112 The aim of this document is to update the state-of-art with respect 113 to p2p simulation frameworks available today. We will survey 114 simulator frameworks prevalent --- and actively used --- in 115 simulating p2p protocols today in order to quantify any assumptions 116 and characteristics inherent in the simulator. This, we hope, will 117 aid future researchers in choosing the right simulation framework for 118 their abstraction. This document can also serve as a guidance for 119 those researchers who are interested in developing P2P simulators 120 from scratch. 122 2. Terminology 124 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 125 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 126 document are to be interpreted as described in RFC 2119 [RFC2119]. 128 3. Criteria for evaluating simulation frameworks 130 This is a non-exhaustive list of all criteria that we should evaluate 131 when surveying a simulation framework. 133 o Type of simulator: flow-level, message-level, or packet- level. 134 Advantages and disadvantages of each. 135 o Does the simulator specifically target p2p networks? Some like 136 ns-3 are general purpose simulators, but p2p models can be 137 constructed and evaluated over a general-purpose simulator. 138 o Level of documentation (APIs, wiki, etc.) 139 o Support for building models: script level, compiled language, 140 through a visualization editor, etc. 141 o System limitations imposed by the simulator framework, if any. 142 o Learning curves associated with the simulator framework. 143 o Support for trace-driven simulation (i.e., using live traces to 144 inject events in the simulator queue. 145 o Scalability of the simulator. 146 o Whether or not the simulator framework supports distributed 147 simulations synchronized on a common time source or event queue. 148 o Support for transitioning from a simulation environment to actual 149 system implementation (or, can the code developed for a simulator 150 be used with minimal or no modifications in a real host)? See 151 Galuba et al. [protopeer]. 152 o Support for modeling link-level (delay, latency, loss, data rate) 153 and host-level characteristics (i.e., simulate both low-level 154 events and application PDUs). 155 o Support for interfacing real hosts that inject events into the 156 simulator. 157 o Support for collecting statistics and measurements from the 158 models. 159 o Visualization tools for creating topologies, viewing the 160 simulation in action, etc. 161 o Support for importing existing topologies (GT-ITM) and others. 162 o Support for exporting topologies in a standard graph markup 163 language. 164 o Should we focus on only academic and research simulators or 165 commercial simulators as well? 166 o ... 168 4. List of simulation frameworks 170 A list of simulation frameworks that we can survey appears below 171 (original list is in Naicken et al. [naicken], I have added a couple 172 more simulators). This is a rather exhaustive list, however, going 173 forward, we should focus on those frameworks that are: newer, 174 actively in use today, and those frameworks that are actively used 175 today and have been surveyed before, but could stand to be looked at 176 again in light of hardware and software advances in the last few 177 years (multi-cores, parallel programming, etc.): 179 o ns-3 [ns-3]. 180 o ProtoPeer [protopeer]. 181 o GPS. 182 o PeerSim. 183 o P2PSim. 184 o OverSim. 185 o DHTSim. 186 o PlanetSim. 187 o VPDNS. 188 o Narses. 189 o Neurogrid. 190 o GnutellaSim. 191 o myNS --- we could probably drop this in favor of ns-3. 192 o Overlay Weaver. 193 o Query-cycle Sim. 194 o GTNetS [gtnets] --- seems to be abandoned. 195 o PeerfactSim [peerfactsim]. 196 o ... 198 5. Security Considerations 200 This document does not introduce any new security considerations in 201 p2p protocols. 203 6. IANA Considerations 205 This document does not require any IANA considerations. 207 7. References 209 7.1. Normative References 211 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 212 Requirement Levels", BCP 14, RFC 2119, March 1997. 214 7.2. Informative References 216 [gtnets] "The Georgia Tech Network Simulator (GTNetS)", 217 http://www.ece.gatech.edu/research/labs/MANIACS/GTNetS/. 219 [naicken] Naicken, S., Basu, A., Livingston, B., and S. Rodhetbhai, 220 "A Survey of Peer-to-Peer Network Simulators", 221 Proceedings of the Seventh Annual Postgraduate Symposium, 222 Liverpool, UK, 2006. 224 [ns-3] "The ns-3 network simulator", http://www.nsnam.org. 226 [peerfactsim] 227 "PeerfactSim.KOM - A Large Scale Simulation Framework for 228 Peer-to-Peer System", 229 http://peerfact.kom.e-technik.tu-darmstadt.de/. 231 [protopeer] 232 Galuba, W., Aberer, K., Despotovic, Z., and W. Kellerer, 233 "ProtoPeer: A p2p toolkit bridging the gap between 234 simulation and live deployment", Proceedings of 235 SIMUTools, Rome, Italy, 2009. 237 Appendix A. Acknowledgments 239 Authors' Addresses 241 Vijay K. Gurbani (editor) 242 Bell Labs, Alcatel-Lucent 243 1960 Lucent Lane, Rm 9C-533 244 Naperville, IL 60563 245 USA 247 Email: vkg@bell-labs.com 249 Anirban Basu 250 Tokai University and University of Sussex 252 Email: a.basu@sussex.ac.uk 254 Thomas Schmidt 255 HAW Hamburg 257 Email: schmidt@informatik.haw-hamburg.de 258 Simon Fleming 259 University of Sussex 261 Email: S.Fleming@sussex.ac.uk 263 Mario Kolberg 264 University of Stirling 266 Email: mko@cs.stir.ac.uk 268 Matthias Waehlisch 269 link-lab & FU Berlin 271 Email: mw@link-lab.net