ICNRG K. Pentikousis, Ed. Internet-Draft EICT Intended Status: Informational B. Ohlman Expires: January 7, 2016 Ericsson E. Davies Trinity College Dublin S. Spirou Intracom Telecom G. Boggia Politecnico di Bari July 6, 2015 Information-centric Networking: Evaluation Methodology draft-irtf-icnrg-evaluation-methodology-02 Abstract This document surveys the evaluation tools currently available to researchers in the information-centric networking (ICN) area and provides suggestions regarding methodology and metrics. Finally, this document sheds some light on the impact of ICN on network security. Status of this Memo This Internet-Draft is submitted to IETF in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/1id-abstracts.html The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html Pentikousis, et al. Expires January 7, 2016 [Page 1] INTERNET DRAFT ICN Evaluation Methodology July 6, 2015 Copyright and License Notice Copyright (c) 2015 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 2 2. Evaluation Methodology . . . . . . . . . . . . . . . . . . . . 4 2.1. ICN Simulators and Testbeds . . . . . . . . . . . . . . . 4 2.1.1. CCN and NDN . . . . . . . . . . . . . . . . . . . . . 4 2.1.2. Publish/Subscribe Internet Architecture . . . . . . . 6 2.1.3. NetInf . . . . . . . . . . . . . . . . . . . . . . . . 6 2.1.4. COMET . . . . . . . . . . . . . . . . . . . . . . . . 7 2.1.5. Experimental Facility Testing . . . . . . . . . . . . 7 2.2. Topology Selection . . . . . . . . . . . . . . . . . . . . 8 2.3. Traffic Load . . . . . . . . . . . . . . . . . . . . . . . 9 2.4. Choosing Relevant Metrics . . . . . . . . . . . . . . . . 13 2.4.1. Traffic Metrics . . . . . . . . . . . . . . . . . . . 16 2.4.2. System Metrics . . . . . . . . . . . . . . . . . . . . 17 2.5. Resource Equivalence and Tradeoffs . . . . . . . . . . . . 18 3. ICN Security Aspects . . . . . . . . . . . . . . . . . . . . . 19 3.1. Authentication . . . . . . . . . . . . . . . . . . . . . . 20 3.2. Authorization, Access Control and Statistics . . . . . . . 21 3.3. Privacy . . . . . . . . . . . . . . . . . . . . . . . . . . 21 3.4. Changes to the Network Security Threat Model . . . . . . . 22 4. Security Considerations . . . . . . . . . . . . . . . . . . . 23 5. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 23 6. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 23 7. Informative References . . . . . . . . . . . . . . . . . . . . 24 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 30 1. Introduction As discussed in [RFC7476], the development phase that information- centric networking (ICN) is going through, and the plethora of approaches to tackle the hardest problems, make this a very active and growing research area but, on the downside, it also makes it more Pentikousis, et al. Expires January 7, 2016 [Page 2] INTERNET DRAFT ICN Evaluation Methodology July 6, 2015 difficult to compare different proposals on an equal footing. To date, different ICN approaches have been evaluated in the peer- reviewed literature using a mixture of theoretical analysis, simulation and emulation techniques, and empirical (testbed) measurements. Typically, researchers follow a specific methodology based on the goal of their experiment, e.g., whether they want to evaluate scalability, quantify resource utilization, or analyze economic incentives. In addition, though, we observe that ease and convenience of setting up and running experiments can sometimes be a factor in published evaluations. It is worth pointing out that for well-established protocols, such as TCP, performance evaluation using actual network deployments has the benefit of realistic workloads and reflects the environment where the service or protocol will be deployed. However, results obtained in this environment are often difficult to replicate independently. Beyond this, the difficulty of deploying future Internet architectures and then engaging sufficient users to make such evaluation realistic is often prohibitive. Moreover, for ICN in particular, it is not yet clear what qualifies as a "realistic workload". As such, trace-based analysis of ICN is in its infancy, and more work is needed towards defining characteristic workloads for ICN evaluation studies. Accordingly, the experimental process itself as well as the evaluation methodology are being actively researched for ICN architectures. Numerous factors affect the experimental results, including the topology selected, the background traffic that an application is being subjected to, network conditions such as available link capacities, link delays, and loss-rate characteristics throughout the selected topology; failure and disruption patterns; node mobility; the diversity of devices used, as we explain in the remainder of this section. Apart from the technical evaluation of the functionality of an ICN architecture, its future success will be largely driven by its deployability and economic viability. Thus ICN evaluations should assess incremental deployability in the existing network environment together with a view of how the technical functions will incentivize deployers to invest in the capabilities that allow the architecture to spread across the network. This document incorporates input from ICNRG participants and their corresponding text contributions, has been reviewed by several ICNRG active participants (see section 6), and represents the consensus of the research group. That said, note that this document does not constitute an IETF standard; see also [RFC5743]. Pentikousis, et al. Expires January 7, 2016 [Page 3] INTERNET DRAFT ICN Evaluation Methodology July 6, 2015 The remainder of this document is organized as follows. Section 2 presents various techniques and considerations for evaluating different ICN architectures. Section 3 discusses the impact of ICN on network security. 2. Evaluation Methodology This document proposes key guidelines alongside suggested data sets and high-level approaches that we expect to be of interest to the ICN community as a whole. Through this, researchers and practitioners alike would be able to compare and contrast different ICN designs against each other, as well as against the state of the art in host- centric solutions, and identify the respective strengths and weaknesses. 2.1. ICN Simulators and Testbeds Since ICN is an emerging area, the community is in the process of developing effective evaluation environments, including simulators, emulators, and testbeds. To date, none of the available evaluation methodologies can be seen as the one and only community reference evaluation tool. Furthermore, no single environment supports all well-known ICN approaches. Simulators and emulators should be able to capture, faithfully, all features and operations of the respective ICN architecture(s); any limitations should be openly documented. It is also essential that these tools and environments come with adequate logging facilities so that one can use them for in-depth analysis as well as debugging. Additional requirements include the ability to support mid- to large-scale experiments, the ability to quickly and correctly set various configurations and parameters, as well as to support the playback of traffic traces captured on a real testbed or network. Obviously, this does not even begin to touch upon the need for strong validation of any evaluated implementations. The rest of this subsection summarizes the ICN simulators and testbeds currently available to the community. 2.1.1. CCN and NDN The CCN project has open-sourced a software reference implementation of the architecture and protocol called CCNx (www.ccnx.org). CCNx is available for deployment on various operating systems and includes C and Java libraries that can be used to build CCN applications. CCN- lite (www.ccn-lite.net) is a lightweight implementation of the CCN protocol, supports most of the key features of CCNx, and is Pentikousis, et al. Expires January 7, 2016 [Page 4] INTERNET DRAFT ICN Evaluation Methodology July 6, 2015 interoperable with CCNx. The core CCNx logic has been implemented in about 1000 lines of code and is ideal for classroom work and course projects as well as for quickly experimenting with CCNx extensions. ndnSIM [ndnSIM] is a module that can be plugged into the ns-3 simulator and supports the core features of CCN. One can use ndnSIM to experiment with various CCN applications and services as well as components developed for CCN such as routing protocols, caching and forwarding strategies. The code for ns-3 and ndnSIM is openly available to the community and can be used as the basis for implementing ICN protocols or applications. For more details see http://www.nsnam.org and http://www.ndnsim.net. ccnSim [ccnSim] is another CCN-specific simulator that was specially designed to handle forwarding of a large number of CCN-chunks. ccnSim is written in C++ for the OMNeT++ simulation framework (www.omnetpp.org). Interested readers could consider also the Content Centric Networking Packet Level Simulator [CCNPL]. Finally, CCN-Joker [CCNj] is an application-layer platform that can be used to build a CCN overlay. CCN-Joker emulates in user-space all basic aspects of a CCN node (e.g., handling of Interest and Data packets, cache sizing, replacement policies), including both flow and congestion control. The code is open source and is suitable for both emulation-based analyses and real experiments. An example of a testbed that supports CCN is the Open Network Lab (see https://onl.wustl.edu/). The ONL testbed currently comprises 18 extensible gigabit routers and over a 100 computers representing clients and is freely available to the public for running CCN experiments. Nodes in ONL are preloaded with CCNx software. ONL provides a graphical user interface for easy configuration and testbed set up as per the experiment requirements, and also serves as a control mechanism, allowing access to various control variables and traffic counters. It is also possible to run and evaluate CCN over popular testbeds such as PlanetLab (www.planet-lab.org), Emulab (www.emulab.net), and Deter (www.isi.deterlab.net) by directly running the CCNx open-source code on PlanetLab and Deter nodes, respectively. NEPI, the Network Experimentation Programming Interface, (http://nepi.inria.fr) is a tool developed for controlling and managing large-scale network experiments. NEPI provides an experiment description language to design network experiments, describing topology, applications, and a controller to automatically deploy those experiments on target experimentation environments, such as PlanetLab. The controller is also capable of collecting result and log files during the experiment execution. NEPI also allows to specify node selection filters while designing the experiment, Pentikousis, et al. Expires January 7, 2016 [Page 5] INTERNET DRAFT ICN Evaluation Methodology July 6, 2015 thereby supporting automatic discovery and provisioning of testbed nodes during experiment deployment, without the user having to hand- pick them. It is simple and efficient to use NEPI to evaluate CCNx on large-scale testbeds such as PlanetLab. 2.1.2. Publish/Subscribe Internet Architecture The PSIRP project has open-sourced its Blackhawk publish-subscribe (Pub/Sub) implementation for FreeBSD; more details are available online at http://www.psirp.org/downloads.html. Despite being limited to one operating system, the code base also provides a virtual image to allow its deployment on other environments through virtualization. The code distribution features a kernel module, a file system and scope daemon, as well as a set of tools, test applications and scripts. This work was extended as part of the PURSUIT project, resulting in the development of the Blackadder prototype for Linux and FreeBSD. It currently runs on a testbed across Europe, America (MIT) and Japan (NICT). All sites are connected via OpenVPN, which exports a virtual Ethernet device to all machines in the testbed. In total, 40 machines in a graph topology containing one Topology Manager and one Rendezvous node that handle all publish/subscribe and topology formation requests are interconnected [IEICE]. Moreover, the ICN simulation environment [ICN-Sim] allows the simulation of new techniques for topology management following the Publish-Subscribe paradigm and the PSIRP approach. The simulator is based on the OMNET++ simulator and the INET/MANET frameworks. It is currently publicly available at http://sourceforge.net/projects/icnsim. A design characteristic of this platform is the separation between the network and topology management policies. An interface is used to provide this functionality and policies can be imported and applied in the network as topology manager applications running on top of this interface. 2.1.3. NetInf The EU FP7 4WARD and SAIL projects have made a set of open-source implementations available; see http://www.netinf.org/open-source for more details. Of note, two software packages are available. The first one is a set of tools for NetInf implementing different aspects of the protocol (e.g., NetInf URI format, HTTP and UDP convergence layer) using different programming languages. The Java implementation provides a local caching proxy and client. The second one, is a OpenNetInf prototype from the 4WARD project. Besides a rich set of NetInf mechanisms implemented, it also provides a browser plug-in and video streaming software. The SAIL project developed a Pentikousis, et al. Expires January 7, 2016 [Page 6] INTERNET DRAFT ICN Evaluation Methodology July 6, 2015 hybrid host-centric and information-centric network architecture called the Global Information Network (GIN). The prototype for this can be downloaded from http://gin.ngnet.it. 2.1.4. COMET The EU FP7 COMET project developed a simulator, called Icarus, which implements ProbCache [PROBCACHE], centrality-based in-network caching [CL4M] and the hash-route-based algorithms detailed in [HASHROUTE]. The simulator is built in Python and makes use of the Fast Network Simulator Setup tool [FNSS] to configure the related parameters of the simulation. The simulator is available from: https://github.com/lorenzosaino/icarus/ 2.1.5. Experimental Facility Testing An important consideration in the evaluation of any kind of future Internet mechanism, lies in the characteristics of that evaluation itself. Often, central to the assessment of the features provided by a novel mechanism, lies the consideration of how it improves over already existing technologies, and by "how much." With the disruptive nature of clean-slate approaches generating new and different technological requirements, it is complex to provide meaningful results for a network layer framework, in comparison with what is deployed in the current Internet. Thus, despite the availability of ICN implementations and simulators, the need for large-scale environments supporting experimental evaluation of novel research is of prime importance to the advancement of ICN deployment. Over the last decade several experimental facilities have become available to the ICN community. Most of them provide a base platform for experimentation using real network links and virtualized computing resources. GENI (www.geni.net), for example, offers experimentation infrastructure as does PlanetLab (www.planet- lab.org), which likely offers the largest testbed available today. Those wishing to perform smaller, more controlled experiments can also consider the Emulab testbed (www.emulab.net), which allows various topologies to be configured. The Asia Future Internet Forum (http://www.asiafi.net) has also developed a testbed used for ICN experiments [AFI]. This testbed consists of multiple servers located in Asia and other locations. Each testbed server (or VM) utilizes a Linux kernel-based container (LXC) for node virtualization. This testbed enables testbed users to run applications and protocols for ICN in two experimentation modes using two different container designs: (1) application-level Pentikousis, et al. Expires January 7, 2016 [Page 7] INTERNET DRAFT ICN Evaluation Methodology July 6, 2015 experimentation using a "common container" and (2) network-level experimentation using a "user container." A common container is shared by all testbed users, and a user container is assigned to one testbed user. A common container possesses a global IP address to connect with each other or outside networks, while each user container uses a private IP address (e.g., 203.0.113.2) and a user space is a closed networking environment. Yet, a user can login to his/her user containers using SSH with his/her certificate, or access them from PCs connected to the Internet using SSH tunnel. This testbed also implements an "on-filesystem cache" to allocate caching data on a UNIX filesystem. The on-filesystem cache system accommodates two kinds of caches: "individual cache" and "shared cache." Individual cache is accessible for one dedicated router for the individual user, while shared cache is accessible for a set of routers in the same group to avoid duplicated caching in the neighborhood for cooperative caching. 2.2. Topology Selection [RFC7476] introduced several topologies that have been used in ICN studies so far but, to date and to the best of our understanding, there is no single topology that can be used to easily evaluate all aspects of the ICN paradigm. There is rough consensus that the classic dumbbell topology cannot serve well future evaluations of ICN approaches. Therefore, one should consider a range of topologies, each of which would stress different aspects. Current Internet traces are also available to assist in this, e.g. see the CAIDA Macroscopic Internet Topology Data Kit (http://www.caida.org/data/active/internet-topology-data-kit) and Rocketfuel (http://www.cs.washington.edu/research/networking/rocketfuel). Depending on what is the focus of the evaluation, intra-domain topologies alone may be appropriate. However, those interested, for example, in quantifying transit costs will require inter-domain traces (note that the above CAIDA traces offer this). Scalability is an important consideration in this choice of this with CAIDA's ITDK traces recording millions of routers across thousands of domains. Beyond these traces there is a wide range of synthetic topologies, such as the Barabasi-Albert model [BA] and the Watts-Strogatz small- world topology [WATTS]. These synthetic traces allow experiments to be performed whilst controlling various key parameters (e.g. degree). Through this, different aspects can be investigated, such as inspecting resilience properties. For some research, this may be more appropriate as, practically speaking, there are no assurances that a future ICN will share the same topology with today's networks. Pentikousis, et al. Expires January 7, 2016 [Page 8] INTERNET DRAFT ICN Evaluation Methodology July 6, 2015 Besides defining the evaluation topology as a graph G = (V,E), where V is the set of vertices (nodes) and E is the set of edges (links), one should also clearly define and list the respective matrices that correspond to the network, storage and computation capacities available at each node as well as the delay characteristics of each link, so that the results obtained can be easily replicated in other studies. Recent work by Hussain and Chen [Montage], although currently addressing host-centric networks, could also be leveraged and be extended by the ICN community. Measurement information can also be taken from existing platforms such as iPlane (http://iplane.cs.washington.edu), which can be used to provide configuration parameters such as access link capacity and delay. Alternatively, synthetic models such as [DELAY] can be used to configure such topologies. Finally, the dynamic aspects of a topology, such as node and content mobility, disruption patterns, packet loss rates as well as link and node failure rates, to name a few, should also be carefully considered. As mentioned in [RFC7476], for example, contact traces from the DTN community could also be used in ICN evaluations. 2.3. Traffic Load In this subsection we provide a set of common guidelines, in the form of what we will refer to as a content catalog for different scenarios. This catalog, which is based on previously published work, could be used to evaluate different ICN proposals, for example, on routing, congestion control, and performance, and can be considered as other kinds of ICN contributions emerge. As we are still lacking ICN-specific traffic workloads we can currently only extrapolate from today's workloads. A significant challenge then relates to the identification of the applications contributing to the observed traffic (e.g., Web, P2P), as well as to the exact amount of traffic they contribute to the overall traffic mixture. Efforts in this direction can take heed from today's traffic mix comprising web, file sharing (BitTorrent-like) and User Generated Content (UGC) platforms (e.g., YouTube), as well as Video on Demand (VoD) services. Publicly available traces for these include those available from web sites such as http://multiprobe.ewi.tudelft.nl/multiprobe.html, http://an.kaist.ac.kr/traces/IMC2007.html, and http://traces.cs.umass.edu/index.php/Network/Network. Taking a more systematic approach, and with the purpose of modeling the traffic load, we can resort to measurement studies that investigate the composition of Internet traffic, such as [1][2]. In [1] a large scale measurement study was performed, with the purpose of studying the traffic crossing inter-domain links. The results Pentikousis, et al. Expires January 7, 2016 [Page 9] INTERNET DRAFT ICN Evaluation Methodology July 6, 2015 indicate the dominance of Web traffic, amounting to 52% over all measured traffic. However, Deep Packet Inspection (DPI) techniques reveal that 25-40% of all HTTP traffic actually carries video traffic. Results from DPI techniques also reveal the difficulty in correctly identifying the application type in the case of P2P traffic: mapping observed port numbers to well known applications shows P2P traffic constituting only 0.85% of overall traffic, while DPI raises this percentage to 18.32% [1]. Relevant studies on a large ISP show the percentage of P2P traffic ranging from 17 to 19% of overall traffic [2]. Table I provides an overview of these figures. The "other" traffic type denotes traffic that cannot be classified in any of the first three application categories, and consists of unclassified traffic and traffic heavily fragmented into several applications (e.g., 0.17% DNS traffic). Table I. Traffic Type Percent of Total Traffic [1, 2] Traffic Type | Ratio ===================== Web | 31-39% --------------------- P2P | 17-19% --------------------- Video | 13-21% --------------------- Other | 29-31% ===================== The content catalog for each type of traffic can be characterized by a specific set of parameters: i. The cardinality of the estimated content catalog. ii. The size of the exchanged contents (either chunks or entire named information objects). iii. The popularity of objects expressed in their request frequency. In most application types, the popularity distribution follows some power law, indicating that a small number of information items triggers a major portion of the entire set of requests. The exact shape of the power law popularity distribution directly impacts the performance of the underlying protocols. For instance, highly skewed popularity distributions (e.g., a Zipf-like distribution with a high slope value) favor the deployment of caching schemes, since caching a very small set of information items can dramatically increase the cache hit ratio. Pentikousis, et al. Expires January 7, 2016 [Page 10] INTERNET DRAFT ICN Evaluation Methodology July 6, 2015 Several studies in the past years have stated that Zipf's law is the discrete distribution that best represents the request frequency in a number of application scenarios, ranging from the Web to video on demand (VoD) services. The key aspect of this distribution is that the frequency of a content request is inversely proportional to the rank of the content itself, i.e., the smaller the rank, the higher the request frequency. If we denote with M the content catalog cardinality and with 1 <= i <= M the rank of the i-th most popular content, we can express the probability of requesting the content with rank "i" as: P(X=i) = ( 1/i^(alpha) ) / C, with C = SUM(1 / j^(alpha)), alpha > 0 where the sum is obtained considering all values of j, 1 <= j <= M. Further, a variation of the Zipf distribution, termed the Mandelbrot- Zipf distribution, has been suggested by [P2PMod] to better model environments where nodes can locally store previously requested content. For example, it was observed that peer-to-peer file sharing applications typically exhibited a 'fetch-at-most-once' style of behavior. This is because peers tend to persistently store the files they download, a behavior that may also be prevalent in ICN. Popularity can also be characterized in terms of: a. The temporal dynamics of popularity, i.e., how requests are distributed in time. The popularity distribution expresses the number of requests submitted for each information item participating into a certain workload. However, they do not describe how these requests are distributed in time. This aspect is of primary importance when considering the performance of caching schemes since the ordering of the requests obviously affects the contents of a cache. For example, with a Least Frequently Used (LFU) cache replacement policy, if all requests for a certain item are submitted close in time, the item is unlikely to be evicted from the cache, even by a (globally) more popular item whose requests are more evenly distributed in time. The temporal ordering of requests gains even more importance when considering workloads consisting of various applications, all competing for the same cache space. b. The spatial locality of popularity i.e., how requests are distributed throughout a network. The importance of spatial locality relates to the ability to avoid redundant traffic in the network. If requests are highly localized in some area of the entire network, then similar requests can be more efficiently served with mechanisms such as caching and/or multicast i.e., the concentration of similar requests in a limited area of the network allows increasing the perceived cache hit ratios at caches in the area and/or the traffic Pentikousis, et al. Expires January 7, 2016 [Page 11] INTERNET DRAFT ICN Evaluation Methodology July 6, 2015 savings from the use of multicast. Table II provides an overview of distributions that can be used to model each of the identified traffic types i.e., Web, Video (based on YouTube measurements) and P2P (based on BitTorrent measurements). These distributions are the outcome of a series of modeling efforts based on measurements of real traffic workloads[3][4][5][6][7][8][9][10][11][12][13]. A tool for the creation of synthetic workloads following these models, and also allowing the generation of different traffic mixes, can be found at [14]. A thorough description of the tool is provided in [15]. Table II. Overview of traffic types models | Object Size | Temporal Locality | Popularity Distribution ===================================================================== Web | Concatenation | Ordering via LRU | Zipf: p(i)=K/i^a | of Lognormal | stack model [5] | i: popularity rank | (body) and | | N: total items | Pareto (tail) | Exact timing via | K: 1/Sum(1/i^a) | [7,8] | exponential | a: distribution slope | | distribution [6] | values 0.64-0.84 [3][4] --------------------------------------------------------------------- VoD | Duration/size: | No analytical models | Weibull: k=0.513, | Concatenated | | lambda=6010 | normal, most | Random distribution | | videos | across total | Gamma: k=0.372, | ~330 kb/s [13] | duration | theta=23910 [12] --------------------------------------------------------------------- P2P | Wide variation | Mean arrival rate of | Mandelbrot-Zipf [9]: | on torrent | 0.9454 torrents/hour | p(i)=K/((i+q)/a) | sizes [9]. | Peers in a swarm | q: plateau factor, | No analytical | arrive as | 5 to 100. | models exist: | l(t)= l0*e^(-t/tau) | Flatter head than in | Sample a real | l0: initial arrival | Zipf-like distribution | BitTorrent [11]| rate (87.74 average) | (where q=0) | or use fixed | tau: object | | value | popularity | | | (1.16 average)* [10] | ===================================================================== * Random ordering of swarm births (first request). For each swarm calculate a different tau. Based on average tau and object popularity. Exponential decay rule for subsequent requests. Table III summarizes the content catalog. With this shared point of reference, the use of the same set of parameters (depending on the Pentikousis, et al. Expires January 7, 2016 [Page 12] INTERNET DRAFT ICN Evaluation Methodology July 6, 2015 scenario of interest) among researchers will be eased, and different proposals could be compared on a common base. Table III. Content Catalog Traffic | Catalog | Mean Object Size | Popularity Distribution Load | Size | [L4][L5][L7][L8] | [L3][L5][L6][L11][L12] | [L1][L2]| [L9][L10] | | [L3][L5]| | ================================================================== Web | 10^12 | Chunk: 1-10 kB | Zipf with | | | 0.64 <= alpha <= 0.83 ------------------------------------------------------------------ File | 5x10^6 | Chunk: 250-4096 kB | Zipf with sharing | | Object: ~800 MB | 0.75 <= alpha<= 0.82 ------------------------------------------------------------------ UGC | 10^8 | Object: ~10 MB | Zipf, alpha >= 2 ------------------------------------------------------------------ VoD | 10^4 | Object: ~100 MB | Zipf, 0.65 <= alpha <= 1 ================================================================== UGC = User Generated Content VoD = Video on Demand 2.4. Choosing Relevant Metrics ICN is a networking concept that spun out of the desire to align the operation model of a network with the model of its typical use. For TCP/IP networks, this means to change the mechanisms of data access and transport from a host-to-host model to a user-to-information model. The premise is that the effort invested in changing models will be offset, or even surpassed, by the potential of a "better" network. However, such a claim can be validated only if it is quantified. Quantification of network performance requires a set of standard metrics. These metrics should be broad enough so they can be applied equally to host-centric and information-centric (or other) networks. This will allow reasoning about a certain ICN approach in relation to an earlier version of the same approach, to another ICN approach or to the incumbent host-centric approach. It will therefore be less difficult to gauge optimization and research direction. On the other hand, the metrics should be targeted to network performance only and should avoid unnecessary expansion into the physical and application layers. Similarly, at this point, it is more important to capture as metrics only the main figures of merit and to leave more esoteric and less frequent cases for the future. Pentikousis, et al. Expires January 7, 2016 [Page 13] INTERNET DRAFT ICN Evaluation Methodology July 6, 2015 To arrive at a set of relevant metrics, it would be beneficial to look at the metrics used in existing ICN approaches, such as CCN [CCN] [VoCCN] [NDNP], NetInf [4WARD6.1] [4WARD6.3] [SAIL-B2] [SAIL- B3], PURSUIT [PRST4.5], COMET [CMT-D5.2] [CMT-D6.2], Connect [SHARE] [RealCCN], and CONVERGENCE [ICN-Web] [ICN-Scal] [ICN-Tran]. The metrics used in these approaches fall into two categories: metrics for the approach as a whole, and metrics for individual components (resolution, routing, etc.). Metrics for the entire approach are further subdivided into traffic and system metrics. It is important to note that the various approaches do not name or define metrics consistently. This is a major problem when trying to find metrics that allow comparison between approaches. For the purposes of exposition, in what follows we have tried to smooth differences by pitting similarly defined metrics under the same name. Also, due to space constraints, we have chosen to report here only the most common metrics between approaches. For more details the reader should consult the references for each approach. Traffic metrics in existing ICN approaches are summarized in Table IV. These are metrics for evaluating an approach mainly from the perspective of the end user, i.e., the consumer, provider, or owner of the content or service. Depending on the level where these metrics are measured, we have made the distinction into user, application and network-level traffic metrics. So for example, network-level metrics are mostly focused on packet characteristics, whereas user-level metrics can cover elements of human perception. The approaches don't make this distinction explicitly, but we can see from the table that CCN and NetInf have used metrics from all levels, PURSUIT and COMET have focused on lower-level metrics, and Connect and CONVERGENCE prefer higher-level metrics. Throughput and download time seem to be the most popular metrics altogether. Pentikousis, et al. Expires January 7, 2016 [Page 14] INTERNET DRAFT ICN Evaluation Methodology July 6, 2015 Table IV. Traffic metrics used in ICN evaluations User | Application | Network ====================================================== Download | Goodput | Startup | Throughput | Packet time | | latency | | delay ================================================================== CCN | x | x | | x | x ------------------------------------------------------------------ NetInf | x | | x | x | x ------------------------------------------------------------------ PURSUIT | | | x | x | x ------------------------------------------------------------------ COMET | | | x | x | ------------------------------------------------------------------ Connect | x | | | | ------------------------------------------------------------------ CONVERGENCE | x | x | | | ================================================================== While traffic metrics are more important for the end user, the owner or operator of the networking infrastructure is normally more interested in system metrics, which can reveal the efficiency of an approach. The various ICN approaches have used system metrics, but unfortunately the situation is not as coherent as with the traffic metrics. The most common system metrics used are: protocol overhead, total traffic, transit traffic, cost savings, router cost, and router energy consumption. Besides the traffic and systems metrics that aim to evaluate an approach as a whole, all of the surveyed approaches also evaluate the performance of individual components. Name resolution, request/data routing, and data caching are the most typical components, as summarized in Table V. FIB size and path length, i.e., the routing component metrics, are almost ubiquitous among approaches, perhaps due to the networking background of the involved researchers. That might be also the reason for the sometimes decreased focus on traffic and system metrics, in favor of component metrics. It can certainly be argued that traffic and system metrics are affected by component metrics, however no approach has made the relationship clear. With this in mind, and also taking into account that traffic and system metrics are readily useful to end users and network operators, we will restrict ourselves to those in the following sections. Pentikousis, et al. Expires January 7, 2016 [Page 15] INTERNET DRAFT ICN Evaluation Methodology July 6, 2015 Table V. Component metrics in existing ICN approaches Resolution | Routing | Cache ====================================================== Resolution | Request | FIB | Path | Size | Hit time | rate | size | length | | ratio ================================================================== CCN | x | | x | x | x | x ------------------------------------------------------------------ NetInf | x | x | | x | | x ------------------------------------------------------------------ PURSUIT | | | x | x | | ------------------------------------------------------------------ COMET | x | x | x | x | | x ------------------------------------------------------------------ CONVERGENCE | | x | x | | x | ================================================================== Before proceeding, we should note that we'd like our metrics to be applicable to host-centric networks as well. Standard metrics already exist for IP networks and it would certainly be beneficial to take them into account. It is encouraging that many of the metrics used by existing ICN approaches can also be used on IP networks and that all of the approaches have tried on occasion to draw the parallels. 2.4.1. Traffic Metrics The IETF has been working for more than a decade on devising metrics and methods for measuring the performance of IP networks. The work has been carried out largely within the IPPM WG, guided by a relevant framework [RFC2330]. IPPM metrics include delay, delay variation, loss, reordering, and duplication. While the IPPM work is certainly based on packet-switched IP networks, it is conceivable that it can be modified and extended to cover ICN networks as well. However, more study is necessary to turn this claim into a certainty. Many experts have toiled for a long time on devising and refining the IPPM metrics and methods, so it would be an advantage to use IPPM on measuring ICN performance. In addition, IPPM works already for host- centric networks, so comparison with information-centric networks would entail only the ICN extension of the IPPM framework. Finally, an important benefit of measuring the transport performance of a network at it's output, using QoS metrics such as IPPM, is that it can be done mostly without any dependence to applications. Another option for measuring transport performance would be to use Pentikousis, et al. Expires January 7, 2016 [Page 16] INTERNET DRAFT ICN Evaluation Methodology July 6, 2015 Quality of Service metrics, not at the output of the network like with IPPM, but at the input to the application. So for an application like live video streaming the relevant metrics would be startup latency, playout lag and playout continuity. The benefit of this approach is that it abstracts away all details of the underlying transport network, so it can be readily applied to compare between networks of different concepts (host-centric, information-centric, or other). As implied earlier, the drawback of the approach is its dependence on the application, so it is likely that different (types of) applications will require different metrics. It might be possible to identify standard metrics for each type of application, but the situation is not as clear as with IPPM metrics and further investigation is necessary. At a higher level of abstraction, we could measure the network's transport performance at the application output. This entails measuring the quality of the transported and reconstructed information as perceived by the user during consumption. In such an instance we would use Quality of Experience (QoE) metrics, which are by definition dependent on the application. For example, the standardized methods for obtaining a Mean Opinion Score (MOS) for VoIP (e.g., ITU-T P.800) is quite different from those for IPTV (e.g., PEVQ). These methods are notoriously hard to implement, as they involve real users in a controlled environment. Such constraints can be relaxed or dropped by using methods that model human perception under certain environments, but these methods are typically intrusive. The most important drawback of measuring network performance at the output of the application is that only one part of each measurement is related to network performance. The rest is related to application performance, e.g., video coding, or even device capabilities, both of which are irrelevant to our purposes here and are generally hard to separate. We therefore see the use of QoE metrics in measuring ICN performance as a poor choice. 2.4.2. System Metrics Overall system metrics that need to be considered include reliability, scalability, energy efficiency, and delay/disconnection tolerance. In deployments where ICN is addressing specific scenarios, relevant system metrics could be derived from current experience. For example, in IoT scenarios, which were discussed earlier in [RFC7476], it is reasonable to consider the current generation of sensor nodes, sources of information, and even measurement gateways (e.g., for smart metering at homes) or smartphones. In this case, ICN operation ought to be evaluated with respect not only to overall scalability and network efficiency, but also the impact on the nodes themselves. Karnouskos et al. Pentikousis, et al. Expires January 7, 2016 [Page 17] INTERNET DRAFT ICN Evaluation Methodology July 6, 2015 [SensReqs] provide a comprehensive set of sensor and IoT-related requirements, for example, which include aspects such as resource utilization, service life-cycle management and device management. Additionally, various specific metrics are also critical in constrained environments, such as CPU processing requirements, signaling overhead, and memory allocation for caching procedures in addition to power consumption and battery lifetime. For gateways, which typically act as a point of service to a large number of nodes and have to satisfy the information requests from remote entities we need to consider scalability-related metrics, such as frequency and processing of successfully satisfied information requests. Finally, given the in-network caching functionality of ICNs, metrics for the efficiency and performance of in-network caching have to be defined. Such metrics will need to guide researchers and operators regarding the performance of in-network caching algorithms. A first step on this direction has been made in [L9]. The paper proposes a formula that approximates the proportion of time that a content stays in a network cache. The model takes as input the rate of requests for a given content (the Content of Interest) and the rate of requests for all other contents that go through the given network element (router) and move the CoI down in the (LRU) cache. The formula takes also into account the size of the cache of this router. The output of the model essentially reflects the probability that the CoI will be found in a given cache. An initial study [L9] is applied to the CCN/NDN framework, where contents get cached at every node they traverse. The formula according to which the probability or proportion is calculated is given by: pi = [mu/(mu+lambda)]^N, where lambda is the request rate for CoI, mu is the request rate for contents that move CoI down the cache and N is the size of the cache (in slots). The formula can be used to assess the caching performance of the system and can also potentially be used to identify the gain of the system due to caching. This can then be used to compare against gains by other factors, e.g., addition of extra bandwidth in the network. 2.5. Resource Equivalence and Tradeoffs As we have seen above, every ICN network is built from a set of resources, which include link capacities, different types of memory structures and repositories used for storing named information Pentikousis, et al. Expires January 7, 2016 [Page 18] INTERNET DRAFT ICN Evaluation Methodology July 6, 2015 objects and chunks temporarily (i.e. caching) or persistently, as well as name resolution and other lookup services. Complexity and processing needs in terms of forwarding decisions, management (e.g. need for manual configuration, explicit garbage collection, and so on), and routing (i.e. amount of state needed, need for manual configuration of routing tables, support for mobility, etc.) set the stage for a range of engineering tradeoffs. In order to be able to compare different ICN approaches it would be beneficial to be able to define equivalence in terms of different resources which today are considered incomparable. For example, would provisioning an additional 5 Mb/s link capacity lead to better performance than adding 100 GB of in-network storage? Within this context one would consider resource equivalence (and the associated tradeoffs) for example for cache hit ratios per GB of cache, forwarding decision times, CPU cycles per forwarding decision, and so on. 3. ICN Security Aspects The introduction of an information-centric networking architecture and the corresponding communication paradigm changes many aspects of network security. These will affect all scenarios described in [RFC7476]. Additional evaluation will be required to ensure relevant security requirements are appropriately met by the implementation of the chosen architecture in the various scenarios. The ICN architectures currently proposed have concentrated on authentication of delivered content to ensure the integrity of the content. However the approaches are primarily applicable to freely accessible content that does not require access authorization, although they will generally support delivery of encrypted content. The introduction of widespread caching mechanisms may also provide additional attack surfaces. The caching architecture to be used also needs to be evaluated to ensure that it meets the requirements of the usage scenarios. In practice, the work on security in the various ICN research projects has been heavily concentrated on authentication of content. Work on authorization, access control, privacy and security threats due to the expanded role of in-network caches has been quite limited. A roadmap for improving the security model in NetInf can be found in [NETINFSC]. In the rest of this section we briefly consider the issues and provide pointers to the work that has been done on the security aspects of the architectures proposed. Pentikousis, et al. Expires January 7, 2016 [Page 19] INTERNET DRAFT ICN Evaluation Methodology July 6, 2015 3.1. Authentication For fully secure content distribution, content access requires that the receiver needs to be able to reliably assess: validity: is it a complete, uncorrupted copy of what was originally published; provenance: can the receiver identify the publisher, and, if so, whether it and the source of any cached version of the document can be adequately trusted; and relevance: is the content an answer to the question that the receiver asked. All ICN architectures considered in this document primarily target the validity requirement using strong cryptographic means to tie the content request name to the content. Provenance and relevance are directly targeted to varying extents: There is a tussle or trade-off between simplicity and efficiency of access and level of assurance of all these traits. For example, maintaining provenance information can become extremely costly, particularly when considering (historic) relationships between multiple objects. Architectural decisions have therefore been taken in each case as to whether the assessment is carried out by the ICN or left to the application. An additional consideration for authentication is whether a name should be irrevocably and immutably tied to a static piece of preexisting content or whether the name can be used to refer to dynamically or subsequently generated content. Schemes that only target immutable content can be less resource hungry as they can use digest functions rather than public key cryptography for generating and checking signatures. However, this can increase the load on applications because they are required to manage many names, rather than using a single name for an item of evolving content that changes over time (e.g. a piece of data containing an age reference). NetInf uses the Named Information (ni) URI scheme [RFC6920] to identify content. This allows NetInf to assure validity without any additional information but gives no assurance on provenance or relevance. A "search" request allows an application to identify relevant content and applications may choose to structure content to allow provenance assurance but this will typically require additional network access. NetInf validity authentication is consequently efficient in a network environment with intermittent connectivity as it does not force additional network accesses and allows the application to decide on provenance validation if required. NetInf primarily targets static content, but an extension would allow Pentikousis, et al. Expires January 7, 2016 [Page 20] INTERNET DRAFT ICN Evaluation Methodology July 6, 2015 dynamic content to be handled. The immutable case only uses digest functions. DONA [DONA] and CCN [CCN], [SECCONT] integrate most of the data needed to verify provenance into all content retrievals but need to be able to retrieve additional information (typically a security certificate) in order to complete the provenance authentication. Whether the application has any control of this extra retrieval will depend on the implementation. CCN is explicitly designed to handle dynamic content allowing names to be pre-allocated and attached to subsequently generated content. DONA offers variants for dynamic and immutable content. PURSUIT [PSTSEC] appears to allow implementers to choose the authentication mechanism so that it can, in theory, emulate the authentication strategy of any of the other architectures. It is not clear whether different choices would lead to lack of interoperability. 3.2. Authorization, Access Control and Statistics A potentially major concern for all ICN architectures considered here is that they do not provide any inbuilt support for an authorization framework or for statistics monitoring. Once content has been published and cached in servers, routers or end points not controlled by the publisher, the publisher has no way to enforce access control, determine which users have accessed the content or revoke its publication. In fact, in some cases, it is even difficult for the publishers themselves to perform access control, where requests do not necessarily contain host/user identifier information. Access could be limited by encrypting the content but the necessity of distributing keys out-of-band appears to negate the advantages of in-network caching. This also creates significant challenges when attempting to manage and restrict key access. An authorization delegation scheme has been proposed [ACDICN] but this requires access to a server controlled by the publisher to obtain an access token making it essentially just an out-of-band key distribution system. Evaluating the impact of the absence of these features will be essential for any scenario where an ICN architecture might be deployed. It may have a seriously negative impact on the applicability of ICN in commercial environments unless a solution can be found. 3.3. Privacy Pentikousis, et al. Expires January 7, 2016 [Page 21] INTERNET DRAFT ICN Evaluation Methodology July 6, 2015 Another area where the architectures have not been significantly analyzed is privacy. Caching implies a trade-off between network efficiency and privacy. The activity of users is significantly more exposed to the scrutiny of cache owners with whom they may not have any relationship. Although in many ICN architectures, the source of a request is not explicitly identified, an attacker may be able to obtain considerable information if s/he can monitor transactions on the cache and obtain details of the objects accessed, the topological direction of requests and information about the timing of transactions. The persistence of data in the cache can make life easier for an attacker by giving a longer timescale for analysis. The impact of CCN on privacy has been investigated in [CCNSEC] and the analysis is applicable to all ICN architectures because it is mostly focused on the common caching aspect. The privacy risks of named data networking are also highlighted in [CCNPRIV]. Further work on privacy in ICNs can be found in [CONPRV]. 3.4. Changes to the Network Security Threat Model The architectural differences of the various ICN models as compared to TCP/IP have consequences for network security. There is limited consideration of the threat models and potential mitigation in the various documents describing the architectures.[CCNSEC] and [CONPRV] also consider the changed threat model. Some of the key aspects are: o Caching implies a tradeoff between network efficiency and user privacy as discussed in Section 3.3. o More powerful routers upgraded to handle persistent caching increase the network's attack surface. This is particularly the case in systems (e.g., CCN) that may need to perform cryptographic checks on content that is being cached. For example, not doing this could lead routers to disseminate invalid content. o ICNs makes it difficult to identify the origin of a request as mentioned in Section 4.3 slowing down the process of blocking requests and requiring alternative mechanisms to differentiate legitimate requests from inappropriate ones as access control lists (ACLs) will probably be of little value for ICN requests. o Denial-of-service (DoS) attacks may require more effort on ICN than on TCP/IP but they are still feasible. One reason for this is that it is difficult for the attacker to force repeated requests for the same content onto a single node; ICNs naturally spread content so that after the initial few requests, subsequent Pentikousis, et al. Expires January 7, 2016 [Page 22] INTERNET DRAFT ICN Evaluation Methodology July 6, 2015 requests will generally be satisfied by alternative sources, blunting the impact of a DoS attack. That said, there are many ways around this, e.g., generating random suffix identifiers that always result in cache misses. o Per-request state in routers can be abused for DoS attacks. o Caches can be misused in the following ways: + Attackers can use caches as storage to make their own content available. + The efficiency of caches can be decreased by attackers with the goal of DoS attacks. + Content can be extracted by any attacker connected to the cache, putting users' privacy at risk. Appropriate mitigation of these threats will need to be considered in each scenario. 4. Security Considerations This document does not impact the security of the Internet. 5. IANA Considerations This document presents no IANA considerations. 6. Acknowledgments Konstantinos Katsaros contributed the updated text of Section 2.3 along with an extensive set of references. Priya Mahadevan, Daniel Corujo and Gareth Tyson contributed to an earlier version of this document. This document has benefited from reviews, pointers to the growing ICN literature, suggestions, comments and proposed text provided by the following members of the IRTF Information-Centric Networking Research Group (ICNRG), listed in alphabetical order: Marica Amadeo, Hitoshi Asaeda, Claudia Campolo, Suyong Eum, Dorothy Gellert, Luigi Alfredo Grieco, Myeong-Wuk Jang, Ren Jing, Will Liu, Antonella Molinaro, Ioannis Psaras, Dirk Trossen, Jianping Wang, Yuanzhe Xuan, and Xinwen Zhang. Pentikousis, et al. Expires January 7, 2016 [Page 23] INTERNET DRAFT ICN Evaluation Methodology July 6, 2015 7. Informative References [RFC6920] Farrell, S., Kutscher, D., Dannewitz, C., Ohlman, B., Keranen, A., and P. Hallam-Baker, "Naming Things with Hashes", RFC 6920, April 2013. [RFC7476] Pentikousis, K., Ohlman, B., Corujo, D., Boggia, G., Tyson, G., Davies, E., Molinaro, A., and S. Eum, "Information-Centric Networking: Baseline Scenarios ", RFC 7476, March 2015. [ndnSIM] Afanasyev, A. et al., ndnSIM: NDN simulator for NS-3 NDN Technical Report NDN-0005, Revision 2, October 2012. [ccnSim] Rossini, G. and D. Rossi, "Large scale simulation of CCN networks", Proc. Algotel 2012 , La Grande Motte, France, May 2012. [CCNPL] Muscariello, L., "Content centric networking packet level simulator", available online at http://perso.rd.francetelecom.fr/muscariello/sim.html [CCNj] Cianci, I. et al. "CCN - Java Opensource Kit EmulatoR for Wireless Ad Hoc Networks", Proc. 7th ACM Int. Conf. on Future Internet Technologies, Seoul, Korea, Sept., 2012. [IEICE] G. Parisis, D. Trossen, and H. Asaeda, "A Node Design and a Framework for Development and Experimentation for an Information-Centric Network", IEICE Trans. Commun., vol. E96-B, no. 7, pp.1650-1660, July 2013. [ICN-Sim] N. Vastardis et al., "Simulation Tools Enabling Research on Information-centric Networks", Proc. ICC FutureNet Workshop. IEEE, 2012. [PROBCACHE] I. Psaras, W. Chai, G. Pavlou, "Probabilistic In-Network Caching for Information-Centric Networks", Proc. SIGCOMM ICN Workshop. ACM, 2012. [CL4M] Chai, W. K. et al., "Cache 'Less for More' in Information- centric Networks", Proc. Networking. IFIP, 2012. [HASHROUTE] L. Saino, I. Psaras, G. Pavlou, "Hash-routing Schemes for Information-Centric Networking", Proc. SIGCOMM ICN Workshop. ACM, 2013. [FNSS] L. Saino, C. Cocora and G. Pavlou, "A Toolchain for Simplifying Network Simulation Setup", Proc. SIMUTOOLS. Pentikousis, et al. Expires January 7, 2016 [Page 24] INTERNET DRAFT ICN Evaluation Methodology July 6, 2015 ACM, 2013. [BA] Barabasi, A. and R. Albert, "Emergence of scaling in random networks", Science, vol. 286, no. 5439, pp. 509- 512, 1999. [WATTS] Watts, D. J. and S. H. Strogatz, "Collective dynamics of small-world networks", Nature, vol. 393, no. 6684, pp. 40- "10, 1998. [Montage] Hussain, A. and J. Chen, "Montage Topology Manager: Tools for Constructing and Sharing Representative Internet Topologies", DETER Technical Report, ISI-TR-684, Aug. 2012. [DELAY] Kaune, S. et al., "Modelling the Internet Delay Space Based on Geographical Locations", Proc. Euromicro, Weimar, Germany, 2009. [L1] http://googleblog.blogspot.it/2008/07/we-knew-web-was- big.html [L2] Zhang, C., Dhungel, P., and K. Di Wu., "Unraveling the BitTorrent ecosystem", IEEE Transactions on Parallel and Distributed Systems, pp. 1164-1177, 2010. [L3] Cha, M., Kwak, H., Rodriguez, P., Ahn, Y.-Y., and S. Moon, "I tube, you tube, everybody tubes: analyzing the world's largest user generated content video system", Proc. ACM SIGCOMM conference on Internet measurement (IMC), San Diego (CA), USA, Oct. 2007. [L4] Zhou, J., Li, Y., Adhikari, K., and Z.-L. Zhang, "Counting YouTube videos via random prefix sampling", In Proc. of IMC'11, Berlin, Germany, Nov. 2011. [L5] Fricker, C., Robert, P., Roberts, J. and N. Sbihim, "Impact of traffic mix on caching performance in a content-centric network", In Proc. of IEEE NOMEN 2012, Workshop on Emerging Design Choices in Name-Oriented Networking, Orlando, USA, Mar. 2012. [L6] Yu, H., Zheng, D., Zhao, B. Y. and W. Zheng, "Understanding user behavior in large-scale video-on- demand systems", In SIGOPS Oper. Syst. Rev., Vol. 40, pp. 333-344, April 2006. [L7] Marciniak, P., Liogkas, N., Legout, A. and E. Kohler, Pentikousis, et al. Expires January 7, 2016 [Page 25] INTERNET DRAFT ICN Evaluation Methodology July 6, 2015 "Small is not always beautiful", In Proc. of IPTPS, International Workshop of Peer-to-Peer Systems, Tampa Bay, Florida (FL), USA, Feb. 2008. [L8] Bellissimo, A., Levine, B. and P. Shenoy, "Exploring the use of BitTorrent as the basis for a large trace repository", University of Massachusetts, Tech. Rep., 2004. [L9] Psaras, I. et al., "Modelling and Evaluation of CCN- Caching Trees", In Proc. of the 10th international IFIP conference on Networking, Valencia, Spain, May 2011. [L10] Carofiglio, G., Gallo, M., Muscariello, L., and D. Perino, "Modeling Data Transfer in Content-Centric Networking", In Proc. of ITC, San Francisco, USA, Sep. 2011. [L11] Breslau, L., Cao, P., Fan, L., Phillips, G. and S. Shenker, "Web caching and zipf-like distributions: evidence and implications", In Proc. of INFOCOM '99, New York (NY), USA, Mar. 1999. [L12] Mahanti, A., Williamson, C., and D. Eager., "Traffic analysis of a web proxy caching hierarchy", IEEE Network, Vol.14, No.3, pp.16-23, May/June 2000. [P2PMod] Saleh, O., and M. Hefeeda, "Modeling and caching of peer- to-peer traffic", Proc. ICNP. IEEE, 2006. [CCN] Jacobson, V. et al., "Networking Named Content", Proc. CoNEXT. ACM, 2009. [VoCCN] Jacobson, V. et al., "VoCCN: Voice-over Content-Centric Networks", Proc. CoNEXT Re-Arch Workshop. ACM, 2009. [NDNP] Zhang, L. et al., "Named Data Networking (NDN) Project", NDN Technical Report NDN-0001, Oct. 2010. Available: http://named-data.net/publications/techreports/ [4WARD6.1] Ohlman, B. et al., "First NetInf Architecture Description", 4WARD Project Deliverable D-6.1, Apr. 2009. [4WARD6.3] Ahlgren, B. et al., "NetInf Evaluation", 4WARD Project Deliverable D-6.3, June 2010. [SAIL-B2] SAIL, "NetInf Content Delivery and Operations", SAIL Project Deliverable D-B.2, May. 2012. Pentikousis, et al. Expires January 7, 2016 [Page 26] INTERNET DRAFT ICN Evaluation Methodology July 6, 2015 [SAIL-B3] Kutscher, D. (ed.) et al., "Final NetInf Architecture", SAIL Project Deliverable D-B.3 , Jan. 2013. Available: http://www.sail-project.eu/deliverables/ [PRST4.5] Riihijarvi, J. et al., "Final Architecture Validation and Performance Evaluation Report", PURSUIT Project Deliverable D4.5, Jan. 2013. [CMT-D5.2] B ben, A. et al, "Scalability of COMET System", COMET Project Deliverable D5.2, Feb. 2013. [CMT-D6.2] Georgiades, M. et al., "Prototype Experimentation and Demonstration", COMET Project Deliverable D6.2, Feb. 2013. [SHARE] Muscariello, L. et al., "Bandwidth and storage sharing performance in information centric networking", Proc. SIGCOMM ICN Workshop. ACM, 2011. [RealCCN] Perino, D. et al., "A Reality Check for Content Centric Networking", Proc. SIGCOMM ICN Workshop. ACM, 2011. [ICN-Web] Detti, A. et al., "Supporting the Web with an Information Centric Network that Routes by Name", Elsevier Computer Networks, vol. 56, no. 17, Nov. 2012. [ICN-Scal] Blefari Melazzi, N. et al., "Scalability Measurements in an Information-Centric Network", Springer Lecture Notes in Computer Science (LNCS), vol. 7586, 2012. [ICN-Tran] Salsano, S. et al., "Transport-Layer Issues in Information Centric Networks ", Proc. SIGCOMM ICN Workshop. ACM, 2012. [SensReqs] Karnouskos, S. et al., "Requirement considerations for ubiquitous integration of cooperating objects", Proc. NTMS. IFIP, 2011. [NETINFSC] Renault, E, Ahmad, A., and M. Abid, "Towards a Security Model for the Future Network of Information", Proc. Conf. Ubiquitous Information Technologies and Applications, IEEE, 2009. [DONA] Koponen, T. et al., "A Data-Oriented (and Beyond) Network Architecture", Proc. SIGCOMM. ACM, 2007. [SECCONT] Smetters, D., and V. Jacobson, "Securing network content", Technical Report TR-2009-01, PARC, 2009. [PSTSEC] Tagger, B., et al, "Update on the Architecture and Report Pentikousis, et al. Expires January 7, 2016 [Page 27] INTERNET DRAFT ICN Evaluation Methodology July 6, 2015 on Security Analysis", Deliverable 2.4, PURSUIT EU FP7 project, April 2012. [ACDICN] Fotiou, N. et al., "Access control enforcement delegation for information-centric networking architectures", Proc. SIGCOMM ICN Workshop. ACM, 2012. [CCNSEC] Lauinger, T., "Security and Scalability of Content-Centric Networking", Masters Thesis, Technische Universitaet Darmstadt and Eurecom, Sep. 2010. [CCNPRIV] Lauinger, Y., et al, "Privacy Risks in Named Data Networking: What is the Cost of Performance?", ACM SIGCOMM Computer Communication Review Editorial Note, vol. 42, iss. 5, 2012 [CONPRV] Chaabane, A et al, "Privacy in Content-Oriented Networking: Threats and Countermeasures", arXiv:1211.5183, 2012. [1] C. Labovitz, S. Iekel-Johnson, D. McPherson, J. Oberheide, and F. Jahanian. 2010. Internet inter-domain traffic. In Proceedings of the ACM SIGCOMM 2010 conference (SIGCOMM '10). ACM, New York, NY, USA, 75-86. DOI=10.1145/1851182.1851194, http://doi.acm.org/10.1145/1851182.1851194 [2] G. Maier, A. Feldmann, V. Paxson, and M. Allman. 2009. On dominant characteristics of residential broadband internet traffic. In Proceedings of the 9th ACM SIGCOMM conference on Internet measurement conference (IMC '09). ACM, New York, NY, USA, 90-102. DOI=10.1145/1644893.1644904 http://doi.acm.org/10.1145/1644893.1644904 [3] L. Breslau, P. Cao, L. Fan, G. Phillips, and S. Shenker, "Web Caching and Zipf-like Distributions: Evidence and Implications," in IEEE INFOCOM, 1999, pp. 126-134. [4] A. Mahanti, C. Williamson, and D. Eager, "Traffic analysis of a web proxy caching hierarchy," IEEE Network, vol. 14, no. 3, pp. 16 -23, 2000. [5] M. Busari and C. Williamson, "ProWGen: a synthetic workload generation tool for simulation evaluation of web proxy caches," Computer Networks, vol. 38, no. 6, pp. 779- 794, 2002. [6] M. F. Arlitt and C. L. Williamson, "Internet web servers: Pentikousis, et al. Expires January 7, 2016 [Page 28] INTERNET DRAFT ICN Evaluation Methodology July 6, 2015 workload char-acterization and performance implications," IEEE/ACM Transactions on Networking, vol. 5, pp. 631-645, 1997. [7] P. Barford and M. Crovella, "Generating representative web workloads for network and server performance evaluation," in ACM SIGMET- RICS/PERFORMANCE, 1998, pp. 151-160. [8] P. Barford, A. Bestavros, A. Bradley, and M. Crovella, "Changes in web client access patterns: Characteristics and caching implications," World Wide Web, vol. 2, pp. 15- 28, 1999. [9] M. Hefeeda and O. Saleh, "Traffic Modeling and Proportional Partial Caching for Peer-to-Peer Systems," IEEE/ACM Transactions on Net- working, vol. 16, no. 6, pp. 1447-1460, 2008. [10] L. Guo, S. Chen, Z. Xiao, E. Tan, X. Ding, and X. Zhang, "A perfor-mance study of BitTorrent-like peer-to-peer systems," IEEE Journal on Selected Areas in Communication, vol. 25, no. 1, pp. 155-169, 2007. [11] A. Bellissimo, B. N. Levine, and P. Shenoy, "Exploring the use of BitTorrent as the basis for a large trace repository," University of Massachusetts Amherst, Tech. Rep., 2004. [12] X. Cheng, C. Dale, and J. Liu, "Statistics and social network of YouTube videos," in IEEE IWQoS. IEEE, 2008, pp. 229-238. [13] X. Cheng, C. Dale, and J. Liu, "Understanding the Characteristics of Internet Short Video Shar-ing: YouTube as a Case Study," CoRR, vol. abs/0707.3670, 2007. [14] GlobeTraff Tool, available at: http://pages.cs.aueb.gr/~katsaros/GlobeTraff.zip [15] K. V. Katsaros, G. Xylomenos, and G. C. Polyzos, "GlobeTraff: a traffic workload generator for the performance evaluation of future Internet architectures," IEEE/IFIP International Conference on New Technologies, Mobility and Security (NTMS), 2012. [AFI] H. Asaeda, R. Li, and N. Choi, "Container-Based Unified Testbed for Information-Centric Networking," IEEE Network, vol. 28, no. 6, pp. 60-66, 2014. Pentikousis, et al. Expires January 7, 2016 [Page 29] INTERNET DRAFT ICN Evaluation Methodology July 6, 2015 Authors' Addresses Kostas Pentikousis (editor) EICT GmbH Torgauer Strasse 12-15 10829 Berlin Germany Email: k.pentikousis@eict.de Borje Ohlman Ericsson Research S-16480 Stockholm Sweden Email: Borje.Ohlman@ericsson.com Elwyn Davies Trinity College Dublin/Folly Consulting Ltd Dublin, 2 Ireland Email: davieseb@scss.tcd.ie Spiros Spirou Intracom Telecom 19.7 km Markopoulou Avenue 19002 Peania, Athens Greece Email: spis@intracom.com Gennaro Boggia Dep. of Electrical and Information Engineering Politecnico di Bari Via Orabona 4 70125 Bari Italy Email: g.boggia@poliba.it Pentikousis, et al. Expires January 7, 2016 [Page 30]