P2P Research Group V. Gurbani, Ed. Internet-Draft Bell Labs, Alcatel-Lucent Intended status: Informational A. Basu Expires: December 30, 2011 Tokai University and University of Sussex T. Schmidt HAW Hamburg S. Fleming University of Sussex M. Kolberg University of Stirling M. Waehlisch link-lab & FU Berlin June 28, 2011 Peer-to-peer simulation frameworks: a survey draft-irtf-p2prg-simulation-survey-00 Abstract Peer-to-peer (p2p) protocols, like all distributed protocols, are complex, and therefore harder to debug and study in the wild. This is more true of existing p2p protocols, where changing the behaviour of the protocol --- however minor the change may be --- may result in unknown manifestations on the dynamics of the swarm using that protocol. In lieu of the unintended consequences of perturbing a live swarm, researchers have resorted to simulation frameworks. However, simulation results obtained from one simulator are often hard to reproduce when using another simulation framework. This document surveys existing simulator frameworks prevalent in simulating p2p protocols today in order to quantify any assumptions and characteristics inherent in the simulator. This, we hope, will aid future researchers in choosing the right simulation framework for their abstraction. Status of this Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at http://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any Gurbani, et al. Expires December 30, 2011 [Page 1] Internet-Draft P2P simulation survey June 2011 time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on December 30, 2011. Copyright Notice Copyright (c) 2011 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Gurbani, et al. Expires December 30, 2011 [Page 2] Internet-Draft P2P simulation survey June 2011 Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . 4 3. Criteria for evaluating simulation frameworks . . . . . . . . 5 4. List of simulation frameworks . . . . . . . . . . . . . . . . . 5 5. Security Considerations . . . . . . . . . . . . . . . . . . . . 6 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . . 6 7. References . . . . . . . . . . . . . . . . . . . . . . . . . . 6 7.1. Normative References . . . . . . . . . . . . . . . . . . . 6 7.2. Informative References . . . . . . . . . . . . . . . . . . 6 Appendix A. Acknowledgments . . . . . . . . . . . . . . . . . . . 7 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 7 Gurbani, et al. Expires December 30, 2011 [Page 3] Internet-Draft P2P simulation survey June 2011 1. Introduction Peer-to-peer (p2p) protocols, like all distributed protocols, are complex, and therefore harder to debug and study in the wild. This is more true of existing p2p protocols, where changing the behaviour of the protocol --- however minor the change may be --- may result in unknown manifestations on the dynamics of the swarm using that protocol. Researchers contemplating on changing the behavior of an existing p2p protocol have to be careful still, least they inadvertently do more harm than good by introducing their changes. Furthermore, any changes to an existing p2p protocol or a newly developed p2p protocol must be tested and evaluated for validity and reproducibility by the research community. While analytical and mathematical modeling (fluid models, optimization and linear programming) is easily validated, it is harder to validate empirical experiments due to the dynamic nature of the networks, hosts, and interconnections between them. Simulation frameworks are attractive since they provide a controlled environment under which new behavior of p2p protocols can be studied and quantified. The good news is that there is a plethora of simulation frameworks for p2p protocols available today, some of them are surveyed in Naicken et al. [naicken]. However, that survey is dated and does not include simulation frameworks like ns-3 [ns-3] and ProtoPeer [protopeer] that have become available since the survey was published. The aim of this document is to update the state-of-art with respect to p2p simulation frameworks available today. We will survey simulator frameworks prevalent --- and actively used --- in simulating p2p protocols today in order to quantify any assumptions and characteristics inherent in the simulator. This, we hope, will aid future researchers in choosing the right simulation framework for their abstraction. This document can also serve as a guidance for those researchers who are interested in developing P2P simulators from scratch. 2. Terminology The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119]. Gurbani, et al. Expires December 30, 2011 [Page 4] Internet-Draft P2P simulation survey June 2011 3. Criteria for evaluating simulation frameworks This is a non-exhaustive list of all criteria that we should evaluate when surveying a simulation framework. o Type of simulator: flow-level, message-level, or packet- level. Advantages and disadvantages of each. o Does the simulator specifically target p2p networks? Some like ns-3 are general purpose simulators, but p2p models can be constructed and evaluated over a general-purpose simulator. o Level of documentation (APIs, wiki, etc.) o Support for building models: script level, compiled language, through a visualization editor, etc. o System limitations imposed by the simulator framework, if any. o Learning curves associated with the simulator framework. o Support for trace-driven simulation (i.e., using live traces to inject events in the simulator queue. o Scalability of the simulator. o Whether or not the simulator framework supports distributed simulations synchronized on a common time source or event queue. o Support for transitioning from a simulation environment to actual system implementation (or, can the code developed for a simulator be used with minimal or no modifications in a real host)? See Galuba et al. [protopeer]. o Support for modeling link-level (delay, latency, loss, data rate) and host-level characteristics (i.e., simulate both low-level events and application PDUs). o Support for interfacing real hosts that inject events into the simulator. o Support for collecting statistics and measurements from the models. o Visualization tools for creating topologies, viewing the simulation in action, etc. o Support for importing existing topologies (GT-ITM) and others. o Support for exporting topologies in a standard graph markup language. o Should we focus on only academic and research simulators or commercial simulators as well? o ... 4. List of simulation frameworks A list of simulation frameworks that we can survey appears below (original list is in Naicken et al. [naicken], I have added a couple more simulators). This is a rather exhaustive list, however, going forward, we should focus on those frameworks that are: newer, actively in use today, and those frameworks that are actively used Gurbani, et al. Expires December 30, 2011 [Page 5] Internet-Draft P2P simulation survey June 2011 today and have been surveyed before, but could stand to be looked at again in light of hardware and software advances in the last few years (multi-cores, parallel programming, etc.): o ns-3 [ns-3]. o ProtoPeer [protopeer]. o GPS. o PeerSim. o P2PSim. o OverSim. o DHTSim. o PlanetSim. o VPDNS. o Narses. o Neurogrid. o GnutellaSim. o myNS --- we could probably drop this in favor of ns-3. o Overlay Weaver. o Query-cycle Sim. o GTNetS [gtnets] --- seems to be abandoned. o PeerfactSim [peerfactsim]. o ... 5. Security Considerations This document does not introduce any new security considerations in p2p protocols. 6. IANA Considerations This document does not require any IANA considerations. 7. References 7.1. Normative References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. 7.2. Informative References [gtnets] "The Georgia Tech Network Simulator (GTNetS)", http://www.ece.gatech.edu/research/labs/MANIACS/GTNetS/. [naicken] Naicken, S., Basu, A., Livingston, B., and S. Rodhetbhai, Gurbani, et al. Expires December 30, 2011 [Page 6] Internet-Draft P2P simulation survey June 2011 "A Survey of Peer-to-Peer Network Simulators", Proceedings of the Seventh Annual Postgraduate Symposium, Liverpool, UK, 2006. [ns-3] "The ns-3 network simulator", http://www.nsnam.org. [peerfactsim] "PeerfactSim.KOM - A Large Scale Simulation Framework for Peer-to-Peer System", http://peerfact.kom.e-technik.tu-darmstadt.de/. [protopeer] Galuba, W., Aberer, K., Despotovic, Z., and W. Kellerer, "ProtoPeer: A p2p toolkit bridging the gap between simulation and live deployment", Proceedings of SIMUTools, Rome, Italy, 2009. Appendix A. Acknowledgments Authors' Addresses Vijay K. Gurbani (editor) Bell Labs, Alcatel-Lucent 1960 Lucent Lane, Rm 9C-533 Naperville, IL 60563 USA Email: vkg@bell-labs.com Anirban Basu Tokai University and University of Sussex Email: a.basu@sussex.ac.uk Thomas Schmidt HAW Hamburg Email: schmidt@informatik.haw-hamburg.de Gurbani, et al. Expires December 30, 2011 [Page 7] Internet-Draft P2P simulation survey June 2011 Simon Fleming University of Sussex Email: S.Fleming@sussex.ac.uk Mario Kolberg University of Stirling Email: mko@cs.stir.ac.uk Matthias Waehlisch link-lab & FU Berlin Email: mw@link-lab.net Gurbani, et al. Expires December 30, 2011 [Page 8]