ICNRG K. Pentikousis, Ed. Internet-Draft EICT Intended Status: Informational B. Ohlman Expires: April 24, 2014 Ericsson E. Davies Trinity College Dublin S. Spirou Intracom Telecom G. Boggia Politecnico di Bari P. Mahadevan PARC October 21, 2013 Information-centric Networking: Evaluation Methodology draft-irtf-icnrg-evaluation-methodology-00 Abstract This document surveys the evaluation tools currently available to researchers in the information-centric networking (ICN) area and provides suggestions regarding methodology and metrics. Finally, this document sheds some light on the impact of ICN on network security. Status of this Memo This Internet-Draft is submitted to IETF in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/1id-abstracts.html The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html Pentikousis, et al. Expires April 24, 2014 [Page 1] INTERNET DRAFT ICN Evaluation Methodology October 21, 2013 Copyright and License Notice Copyright (c) 2013 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 2 2. Evaluation Methodology . . . . . . . . . . . . . . . . . . . . 4 2.1. ICN Simulators and Testbeds . . . . . . . . . . . . . . . 4 2.1.1. CCN and NDN . . . . . . . . . . . . . . . . . . . . . 4 2.1.2. Publish/Subscribe Internet Architecture . . . . . . . 6 2.1.3. NetInf . . . . . . . . . . . . . . . . . . . . . . . . 6 2.1.4. COMET . . . . . . . . . . . . . . . . . . . . . . . . 7 2.1.5. Large-scale Testing . . . . . . . . . . . . . . . . . 7 2.2. Topology Selection . . . . . . . . . . . . . . . . . . . . 8 2.3. Traffic Load . . . . . . . . . . . . . . . . . . . . . . . 9 2.4. Choosing Relevant Metrics . . . . . . . . . . . . . . . . 11 2.4.1. Traffic Metrics . . . . . . . . . . . . . . . . . . . 13 2.4.2. System Metrics . . . . . . . . . . . . . . . . . . . . 15 2.5. Resource Equivalence and Tradeoffs . . . . . . . . . . . . 16 3. ICN Security Aspects . . . . . . . . . . . . . . . . . . . . . 16 3.1. Authentication . . . . . . . . . . . . . . . . . . . . . . 17 3.2. Authorization, Access Control and Statistics . . . . . . . 19 3.3. Privacy . . . . . . . . . . . . . . . . . . . . . . . . . . 19 3.4. Changes to the Network Security Threat Model . . . . . . . 20 4. Security Considerations . . . . . . . . . . . . . . . . . . . 21 5. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 21 6. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 21 7. Informative References . . . . . . . . . . . . . . . . . . . . 21 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 26 1. Introduction Information-centric networking (ICN) marks a fundamental shift in communications and networking. As discussed in [draft-irtf-icnrg- scenarios], the development phase that ICN is going through, and the plethora of approaches to tackle the hardest problems, make this a Pentikousis, et al. Expires April 24, 2014 [Page 2] INTERNET DRAFT ICN Evaluation Methodology October 21, 2013 very active and growing research area but, on the downside, it also makes it more difficult to compare different proposals on an equal footing. Different ICN approaches have been evaluated in the peer- reviewed literature using a mixture of theoretical analysis, simulation and emulation techniques, and empirical (testbed) measurements. These are all popular methods for evaluating network protocols, architectures, and services in the networking community. Typically, researchers follow a specific methodology based on the goal of their experiment, e.g., whether they want to evaluate scalability, quantify resource utilization, analyze economic incentives, and so on, as we have discussed earlier. In addition, though, we observe that ease and convenience of setting up and running experiments can sometimes be a factor in published evaluations. It is worth pointing out that for well-established protocols, such as TCP, performance evaluation using actual network deployments has the benefit of realistic workloads and reflects the environment where the service or protocol will be deployed. However, results obtained in this environment are often difficult to replicate independently. Beyond this, the difficulty of deploying future Internet architectures and then engaging sufficient users to make such evaluation realistic is often prohibitive. Moreover, for ICN in particular, it is not yet clear what qualifies as a "realistic workload". As such, trace-based analysis of ICN is in its infancy, and more work is needed towards defining characteristic workloads for ICN evaluation studies. Accordingly, the experimental process itself as well as the evaluation methodology are being actively researched for ICN architectures. Numerous factors affect the experimental results, including the topology selected, the background traffic that an application is being subjected to, network conditions such as available link capacities, link delays, and loss-rate characteristics throughout the selected topology; failure and disruption patterns; node mobility; as well as other aspects such as the diversity of devices used, and so on, as we explain in the remainder of this section. Apart from the technical evaluation of the functionality of an ICN architecture, its future success will be largely driven by its deployability and economic viability. Thus any evaluation will also have to include an assessment of its incremental deployability in the existing network environment together with a view of how the technical functions will incentivize deployers to invest in the capabilities that allow the architecture to spread across the network. This document incorporates input from ICNRG participants and their Pentikousis, et al. Expires April 24, 2014 [Page 3] INTERNET DRAFT ICN Evaluation Methodology October 21, 2013 corresponding text contributions, has been reviewed by several ICNRG active participants (see section 6), and represents the consensus of the research group. That said, note that this document does not constitute an IETF standard; see also [RFC5743]. The remainder of this document is organized as follows. Section 2 presents various techniques and considerations for evaluating different ICN architectures. Then, Section 3 discusses the impact of ICN on network security. 2. Evaluation Methodology At this stage, we do not intend to develop a complete methodology or a benchmarking tool. Instead, this document proposes key guidelines alongside suggested data sets and high-level approaches that we expect to be of interest to the ICN community as a whole. Through this, researchers and practitioners alike would be able to compare and contrast different ICN designs against each other, as well as against the state of the art in host-centric solutions, and identify the respective strengths and weaknesses. 2.1. ICN Simulators and Testbeds Since ICN is still an emerging area, the community is still in the process of developing effective evaluation environments, including simulators, emulators, and testbeds. To date, none of the available evaluation methodologies can be seen as the one and only community reference evaluation tool. Furthermore, no single environment supports all well-known ICN approaches. Simulators and emulators should be able to capture, faithfully, all features and operations of the respective ICN architecture(s). It is also essential that these tools and environments come with adequate logging facilities so that one can use them for in-depth analysis as well as debugging. Additional requirements include the ability to support mid- to large- scale experiments, the ability to quickly and correctly set various configurations and parameters, as well as to support the playback of traffic traces captured on a real testbed or network. Obviously, this does not even begin to touch upon the need for strong validation of any evaluated implementations. The rest of this subsection summarizes the ICN simulators and testbeds currently available to the community. 2.1.1. CCN and NDN Pentikousis, et al. Expires April 24, 2014 [Page 4] INTERNET DRAFT ICN Evaluation Methodology October 21, 2013 The CCN project has open-sourced a software reference implementation of the architecture and protocol called CCNx (www.ccnx.org). CCNx is available for deployment on various operating systems and includes C and Java libraries that can be used to build CCN applications. CCN- lite (www.ccn-lite.net) is a lightweight implementation of the CCN protocol, supports most of the key features of CCNx, and is interoperable with CCNx. The core CCNx logic has been implemented in about 1000 lines of code and is ideal for classroom work and course projects as well as for quickly experimenting with CCNx extensions. ndnSIM [ndnSIM] is a module that can be plugged into the ns-3 simulator and supports the core features of CCN. One can use ndnSIM to experiment with various CCN applications and services as well as components developed for CCN such as routing protocols, caching and forwarding strategies. The code for ns-3 and ndnSIM is openly available to the community and can be used as the basis for implementing ICN protocols or applications. For more details see http://www.nsnam.org and http://www.ndnsim.net. ccnSim [ccnSim] is another CCN-specific simulator that was specially designed to handle forwarding of a large number of CCN-chunks. ccnSim is written in C++ for the OMNeT++ simulation framework (www.omnetpp.org). Interested readers could consider also the Content Centric Networking Packet Level Simulator [CCNPL]. Finally, CCN-Joker [CCNj] is an application-layer platform that can be used to build a CCN overlay. CCN-Joker emulates in user-space all basic aspects of a CCN node (e.g., handling of Interest and Data packets, cache sizing, replacement policies), including both flow and congestion control. The code is open source and is suitable for both emulation-based analyses and real experiments. An example of a testbed that supports CCN is the Open Network Lab (see https://onl.wustl.edu/). The ONL testbed currently comprises 18 extensible gigabit routers and over a 100 computers representing clients and is freely available to the public for running CCN experiments. Nodes in ONL are preloaded with CCNx software. ONL provides a graphical user interface for easy configuration and testbed set up as per the experiment requirements, and also serves as a control mechanism, allowing access to various control variables and traffic counters. It is also possible to run and evaluate CCN over popular testbeds such as PlanetLab (www.planet-lab.org), Emulab (www.emulab.net), and Deter (www.isi.deterlab.net) by directly running the CCNx open-source code on PlanetLab and Deter nodes, respectively. NEPI, the Network Experimentation Programming Interface, (http://nepi.inria.fr) is a tool developed for controlling and managing large-scale network experiments. NEPI provides an experiment Pentikousis, et al. Expires April 24, 2014 [Page 5] INTERNET DRAFT ICN Evaluation Methodology October 21, 2013 description language to design network experiments, describing topology, applications, and a controller to automatically deploy those experiments on target experimentation environments, such as PlanetLab. The controller is also capable of collecting result and log files during the experiment execution. NEPI also allows to specify node selection filters while designing the experiment, thereby supporting automatic discovery and provisioning of testbed nodes during experiment deployment, without the user having to hand- pick them. It is simple and efficient to use NEPI to evaluate CCNx on large-scale testbeds such as PlanetLab. 2.1.2. Publish/Subscribe Internet Architecture The PSIRP project has open-sourced its Blackhawk publish-subscribe (Pub/Sub) implementation for FreeBSD; more details are available online at http://www.psirp.org/downloads.html. Despite being limited to one operating system, the code base also provides a virtual image to allow its deployment on other environments through virtualization. The code distribution features a kernel module, a file system and scope daemon, as well as a set of tools, test applications and scripts. This work was extended as part of the PURSUIT project, resulting in the development of the Blackadder prototype for Linux and FreeBSD. It currently runs on a testbed across Europe, America (MIT) and Japan (NICT). All sites are connected via OpenVPN, which exports a virtual Ethernet device to all machines in the testbed. In total, 40 machines in a graph topology containing one Topology Manager and one Rendezvous node that handle all publish/subscribe and topology formation requests are interconnected [IEICE]. Moreover, the ICN simulation environment [ICN-Sim] allows the simulation of new techniques for topology management following the Publish-Subscribe paradigm and the PSIRP approach. The simulator is based on the OMNET++ simulator and the INET/MANET frameworks. It is currently publicly available at http://sourceforge.net/projects/icnsim. A design characteristic of this platform is the separation between the network and topology management policies. An interface is used to provide this functionality and policies can be imported and applied in the network as topology manager applications running on top of this interface. 2.1.3. NetInf The EU FP7 4WARD and SAIL projects have made a set of open-source implementations available; see http://www.netinf.org/open-source for more details. Of note, two software packages are available. The first one is a set of tools for NetInf implementing different aspects Pentikousis, et al. Expires April 24, 2014 [Page 6] INTERNET DRAFT ICN Evaluation Methodology October 21, 2013 of the protocol (e.g., NetInf URI format, HTTP and UDP convergence layer) using different programming languages. The Java implementation provides a local caching proxy and client. The second one, is a OpenNetInf prototype from the 4WARD project. Besides a rich set of NetInf mechanisms implemented, it also provides a browser plug-in and video streaming software. The SAIL project developed a hybrid host-centric and information-centric network architecture called the Global Information Network (GIN). The prototype for this can be downloaded from http://gin.ngnet.it. 2.1.4. COMET The EU FP7 COMET project developed a simulator, called Icarus, which implements ProbCache [PROBCACHE], centrality-based in-network caching [CL4M] and the hash-route-based algorithms detailed in [HASHROUTE]. The simulator is built in Python and makes use of the Fast Network Simulator Setup tool [FNSS] to configure the related parameters of the simulation. The simulator is available from: https://github.com/lorenzosaino/icarus/ 2.1.5. Large-scale Testing An important consideration in the evaluation of any kind of future Internet mechanism, lies in the characteristics of that evaluation itself. Often, central to the assessment of the features provided by a novel mechanism, lies the consideration of how it improves over already existing technologies, and by "how much." With the disruptive nature of clean-slate approaches generating new and different technological requirements, it is complex to provide meaningful results for a network layer framework, in comparison with what is deployed in the current Internet. Thus, despite the availability of ICN implementations and simulators, the need for large-scale environments supporting experimental evaluation of novel research is of prime importance to the advancement of ICN deployment. In this regard, initiatives such as the Future Internet Research and Experimentation Initiative (www.ict-fire.eu), enable researchers to test new protocols and architectures in real conditions over production networks (e.g., through virtualization and software- defined networking mechanisms), simplifying the validation of future evolutions and reducing the gap between research and deployment. Similarly, Future Internet Design (www.nets-find.net) is a long-term initiative along the same direction in the US. GENI (www.geni.net) also offers experimentation infrastructure as does PlanetLab (www.planet-lab.org), which likely offers the largest testbed available today. Those wishing to perform smaller, more controlled Pentikousis, et al. Expires April 24, 2014 [Page 7] INTERNET DRAFT ICN Evaluation Methodology October 21, 2013 experiments can also consider the Emulab testbed (www.emulab.net), which allows various topologies to be configured. The Asia Future Internet Forum (www.asiafi.net) has also designed a testbed mainly used for ICN experiments. This testbed consists of multiple servers located in Asia and will be (presumably) expanded with servers in other locations. Each testbed server includes multiple Linux kernel-based LXCs. One container is called "bridge container" and the other container is called "user container". A bridge container has a global IP address used to connect to the physical network through bridge mode, and a local (private) IP address. A user container, which is assigned to each researcher, connects to the bridge container in the same server using their local IP addresses. A user container connects to the researcher's remote containers located in different servers via tunnels established between its local bridge container and the remote bridge containers. Finally, the National Institute of Information and Communications Technology (NICT) builds and operates the high-performance testbed JGN-X (see http://www.jgn.nict.go.jp/english/index.html), which has cutting-edge network functions and technologies including those currently in development. JGN-X aims to establish new-generation network technology and accelerate the R&D in areas such as network virtualization and advanced operations of virtualized layers. JGN-X is used for collaboration among developers in order to foster the establishment and expansion of new-generation network technology. 2.2. Topology Selection [draft-irtf-icnrg-scenarios] introduced several topologies that have been used in ICN studies so far but, to date and to the best of our understanding, there is no single topology that can be used to easily evaluate all aspects of the ICN paradigm. There is rough consensus that the classic dumbbell topology cannot serve well future evaluations of ICN approaches. Therefore, one should consider a range of topologies, each of which would stress different aspects, as outlined earlier in this document. Current Internet traces are also available to assist in this, e.g. see http://www.caida.org/data/active/internet-topology-data-kit and http://www.cs.washington.edu/research/networking/rocketfuel. Depending on what is the focus of the evaluation, intra-domain topologies alone may be appropriate. However, those interested, for example, in quantifying transit costs will require inter-domain traces (note that the above CAIDA traces offer this). Scalability is an important consideration in this choice of this with CAIDA's ITDK traces recording millions of routers across thousands of domains. Pentikousis, et al. Expires April 24, 2014 [Page 8] INTERNET DRAFT ICN Evaluation Methodology October 21, 2013 Beyond these traces there is a wide range of synthetic topologies, such as the Barabasi-Albert model [BA] and the Watts-Strogatz small- world topology [WATTS]. These synthetic traces allow experiments to be performed whilst controlling various key parameters (e.g. degree). Through this, different aspects can be investigated, such as inspecting resilience properties. For some research, this may be more appropriate as, practically speaking, there are no assurances that a future ICN will share the same topology with today's networks. Besides defining the evaluation topology as a graph G = (V,E), where V is the set of vertices (nodes) and E is the set of edges (links), one should also clearly define and list the respective matrices that correspond to the network, storage and computation capacities available at each node as well as the delay characteristics of each link, so that the results obtained can be easily replicated in other studies. Recent work by Hussain and Chen [Montage], although currently addressing host-centric networks, could also be leveraged and be extended by the ICN community. Measurement information can also be taken from existing platforms such as iPlane (http://iplane.cs.washington.edu), which can be used to provide configuration parameters such as access link capacity and delay. Alternatively, synthetic models such as [DELAY] can be used to configure such topologies. Finally, the dynamic aspects of a topology, such as node and content mobility, disruption patterns, packet loss rates as well as link and node failure rates, to name a few, should also be carefully considered. As mentioned in [draft-irtf-icnrg-scenarios], for example, contact traces from the DTN community could also be used in ICN evaluations. 2.3. Traffic Load As we are still lacking ICN-specific traffic workloads we can currently only extrapolate from today's workloads. In this subsection we provide a first draft of a set of common guidelines, in the form of what we will refer to as a content catalog for different scenarios. This catalog, which is based on previously published work, could be used to evaluate different ICN proposals, for example, on routing, congestion control, and performance, and can be considered as other kinds of ICN contributions emerge. We take scenarios from today's Web, file sharing (BitTorrent-like) and User Generated Content (UGC) platforms (e.g., YouTube), as well as Video on Demand (VoD) services. Publicly available traces for these include those available from web sites such as http://mikel.tlm.unavarra.es/~mikel/bt_pam2004, Pentikousis, et al. Expires April 24, 2014 [Page 9] INTERNET DRAFT ICN Evaluation Methodology October 21, 2013 http://multiprobe.ewi.tudelft.nl/multiprobe.html, http://an.kaist.ac.kr/traces/IMC2007.html, and http://traces.cs.umass.edu/index.php/Network/Network. The content catalog for each type of traffic can be characterized by a specific set of parameters: the cardinality of the estimated content catalog, the average size of the exchanged contents (either chunks or entire named information objects), and the statistical distribution that best reflect the popularity of objects and their request frequency. Table I summarizes the content catalog. With this shared point of reference, the use of the same set of parameters (depending on the scenario of interest) among researchers will be eased, and different proposals could be compared on a common base. Table I. Content Catalog Traffic | Catalog | Mean Object Size | Popularity Distribution Load | Size | [L4][L5][L7][L8] | [L3][L5][L6][L11][L12] | [L1][L2]| [L9][L10] | | [L3][L5]| | ================================================================== Web | 10^12 | Chunk: 1-10 kB | Zipf with | | | 0.64 <= alpha <= 0.83 ------------------------------------------------------------------ File | 5x10^6 | Chunk: 250-4096 kB | Zipf with sharing | | Object: ~800 MB | 0.75 <= alpha<= 0.82 ------------------------------------------------------------------ UGC | 10^8 | Object: ~10 MB | Zipf, alpha >= 2 ------------------------------------------------------------------ VoD | 10^4 | Object: ~100 MB | Zipf, 0.65 <= alpha <= 1 ================================================================== * UGC = User Generated Content ** VoD = Video on Demand Several studies in the past years have stated that Zipf's law is the discrete distribution that best represents the request frequency in a number of application scenarios, ranging from the Web to VoD services. The key aspect of this distribution is that the frequency of a content request is inversely proportional to the rank of the content itself, i.e., the smaller the rank, the higher the request frequency. If we denote with M the content catalog cardinality and with 1 <= i <= M the rank of the i-th most popular content, we can express the probability of requesting the content with rank "i" as: P(X=i) = ( 1/i^(alpha) ) / C, with C = SUM(1 / j^(alpha)), alpha > 0 where the sum is obtained considering all values of j, 1 <= j <= M. Pentikousis, et al. Expires April 24, 2014 [Page 10] INTERNET DRAFT ICN Evaluation Methodology October 21, 2013 Further, a variation of the Zipf distribution, termed the Mandelbrot- Zipf distribution, has been suggested by [P2PMod] to better model environments where nodes can locally store previously requested content. For example, it was observed that peer-to-peer file sharing applications typically exhibited a 'fetch-at-most-once' style of behavior. This is because peers tend to persistently store the files they download, a behavior that may also be prevalent in ICN. 2.4. Choosing Relevant Metrics ICN is a networking concept that spun out of the desire to align the operation model of a network with the model of its typical use. For TCP/IP networks, this means to change the mechanisms of data access and transport from a host-to-host model to a user-to-information model. The premise is that the effort invested in changing models will be offset, or even surpassed, by the potential of a "better" network. However, such a claim can be validated only if it is quantified. Quantification of network performance requires a set of standard metrics. These metrics should be broad enough so they can be applied equally to host-centric and information-centric (or other) networks. This will allow reasoning about a certain ICN approach in relation to an earlier version of the same approach, to another ICN approach or to the incumbent host-centric approach. It will therefore be less difficult to gauge optimization and research direction. On the other hand, the metrics should be targeted to network performance only and should avoid unnecessary expansion into the physical and application layers. Similarly, at this point, it is more important to capture as metrics only the main figures of merit and to leave more esoteric and less frequent cases for the future. To arrive at a set of relevant metrics, it would be beneficial to look at the metrics used in existing ICN approaches, such as CCN [CCN] [VoCCN] [NDNP], NetInf [4WARD6.1] [4WARD6.3] [SAIL-B2] [SAIL- B3], PURSUIT [PRST4.5], COMET [CMT-D5.2] [CMT-D6.2], Connect [SHARE] [RealCCN], and CONVERGENCE [ICN-Web] [ICN-Scal] [ICN-Tran]. The metrics used in these approaches fall into two categories: metrics for the approach as a whole, and metrics for individual components (resolution, routing, etc.). Metrics for the entire approach are further subdivided into traffic and system metrics. It is important to note that the various approaches do not name or define metrics consistently. This is a major problem when trying to find metrics that allow comparison between approaches. For the purposes of exposition, in what follows we have tried to smooth differences by pitting similarly defined metrics under the same name. Also, due to space constraints, we have chosen to report here only the most common Pentikousis, et al. Expires April 24, 2014 [Page 11] INTERNET DRAFT ICN Evaluation Methodology October 21, 2013 metrics between approaches. For more details the reader should consult the references for each approach. Traffic metrics in existing ICN approaches are summarized in Table II. These are metrics for evaluating an approach mainly from the perspective of the end user, i.e., the consumer, provider, or owner of the content or service. Depending on the level where these metrics are measured, we have made the distinction into user, application and network-level traffic metrics. So for example, network-level metrics are mostly focused on packet characteristics, whereas user-level metrics can cover elements of human perception. The approaches don't make this distinction explicitly, but we can see from the table that CCN and NetInf have used metrics from all levels, PURSUIT and COMET have focused on lower-level metrics, and Connect and CONVERGENCE prefer higher-level metrics. Throughput and download time seem to be the most popular metrics altogether. Table II. Traffic metrics used in ICN evaluations User | Application | Network ====================================================== Download | Goodput | Startup | Throughput | Packet time | | latency | | delay ================================================================== CCN | x | x | | x | x ------------------------------------------------------------------ NetInf | x | | x | x | x ------------------------------------------------------------------ PURSUIT | | | x | x | x ------------------------------------------------------------------ COMET | | | x | x | ------------------------------------------------------------------ Connect | x | | | | ------------------------------------------------------------------ CONVERGENCE | x | x | | | ================================================================== While traffic metrics are more important for the end user, the owner or operator of the networking infrastructure is normally more interested in system metrics, which can reveal the efficiency of an approach. The various ICN approaches have used system metrics, but unfortunately the situation is not as coherent as with the traffic metrics. The most common system metrics used are: protocol overhead, total traffic, transit traffic, cost savings, router cost, and router energy consumption. Besides the traffic and systems metrics that aim to evaluate an Pentikousis, et al. Expires April 24, 2014 [Page 12] INTERNET DRAFT ICN Evaluation Methodology October 21, 2013 approach as a whole, all of the surveyed approaches also evaluate the performance of individual components. The name resolution, request/data routing, and data caching are the most typical components, so Table III presents the popular metrics for each of those components. FIB size and path length, i.e., the routing component metrics, are almost ubiquitous among approaches, perhaps due to the networking background of the involved researchers. That might be also the reason for the sometimes decreased focus on traffic and system metrics, in favor of component metrics. It can certainly be argued that traffic and system metrics are affected by component metrics, however no approach has made the relationship clear. With this in mind, and also taking into account that traffic and system metrics are readily useful to end users and network operators, we will restrict ourselves to those in the following sections. Table III. Component metrics in existing ICN approaches Resolution | Routing | Cache ====================================================== Resolution | Request | FIB | Path | Size | Hit time | rate | size | length | | ratio ================================================================== CCN | x | | x | x | x | x ------------------------------------------------------------------ NetInf | x | x | | x | | x ------------------------------------------------------------------ PURSUIT | | | x | x | | ------------------------------------------------------------------ COMET | x | x | x | x | | x ------------------------------------------------------------------ CONVERGENCE | | x | x | | x | ================================================================== Before proceeding, we should note that we'd like our metrics to be applicable to host-centric networks as well. Standard metrics already exist for IP networks and it would certainly be beneficial to take them into account. It is encouraging that many of the metrics used by existing ICN approaches can also be used on IP networks and that all of the approaches have tried on occasion to draw the parallels. 2.4.1. Traffic Metrics At their core, host-centric and information-centric networking function as data transport services. Information of interest to a user resides in one or more storage points connected to the network Pentikousis, et al. Expires April 24, 2014 [Page 13] INTERNET DRAFT ICN Evaluation Methodology October 21, 2013 and, on the user's request, the network transports this information to the user for consumption. We could therefore do worse than to quantify the data transport performance of the network in terms of Quality of Service (QoS) metrics. The IETF has been working for more than a decade on devising metrics and methods for measuring the performance of IP networks. The work has been carried out largely within the IPPM WG, guided by a relevant framework [RFC2330]. IPPM metrics include delay, delay variation, loss, reordering, and duplication. While the IPPM work is certainly based on packet-switched IP networks, it is conceivable that it can be modified and extended to cover ICN networks as well. However, more study is necessary to turn this claim into a certainty. Many experts have toiled for a long time on devising and refining the IPPM metrics and methods, so it would be an advantage to use IPPM on measuring ICN performance. In addition, IPPM works already for host-centric networks, so comparison with information-centric networks would entail only the ICN extension of the IPPM framework. Finally, an important benefit of measuring the transport performance of a network at it's output, using QoS metrics such as IPPM, is that it can be done mostly without any dependence to applications. Another option for measuring transport performance would be to use Quality of Service metrics, not at the output of the network like with IPPM, but at the input to the application. So for an application like live video streaming the relevant metrics would be startup latency, playout lag and playout continuity. The benefit of this approach is that it abstracts away all details of the underlying transport network, so it can be readily applied to compare between networks of different concepts (host-centric, information-centric, or other). As implied earlier, the drawback of the approach is its dependence on the application, so it is likely that different (types of) applications will require different metrics. It might be possible to identify standard metrics for each type of application, but the situation is not as clear as with IPPM metrics and further investigation is necessary. At a higher level of abstraction, we could measure the network's transport performance at the application output. This entails measuring the quality of the transported and reconstructed information as perceived by the user during consumption. In such an instance we would use Quality of Experience (QoE) metrics, which are by definition dependent on the application. For example, the standardized methods for obtaining a Mean Opinion Score (MOS) for VoIP (e.g., ITU-T P.800) is quite different from those for IPTV (e.g., PEVQ). These methods are notoriously hard to implement, as they involve real users in a controlled environment. Such constraints can be relaxed or dropped by using methods that model Pentikousis, et al. Expires April 24, 2014 [Page 14] INTERNET DRAFT ICN Evaluation Methodology October 21, 2013 human perception under certain environments, but these methods are typically intrusive. The most important drawback of measuring network performance at the output of the application is that only one part of each measurement is related to network performance. The rest is related to application performance, e.g., video coding, or even device capabilities, both of which are irrelevant to our purposes here and are generally hard to separate. We therefore see the use of QoE metrics in measuring ICN performance as a poor choice. 2.4.2. System Metrics Overall system metrics that need to be considered include reliability, scalability, energy efficiency, and delay/disconnection tolerance. In deployments where ICN is addressing specific scenarios, relevant system metrics could be derived from current experience. For example, in IoT scenarios, which were discussed earlier in [draft-irtf-icnrg-scenarios], it is reasonable to consider the current generation of sensor nodes, sources of information, and even measurement gateways (e.g., for smart metering at homes) or smartphones. In this case, ICN operation ought to be evaluated with respect not only to overall scalability and network efficiency, but also the impact on the nodes themselves. Karnouskos et al. [SensReqs] provide a comprehensive set of sensor and IoT-related requirements, for example, which include aspects such as resource utilization, service life-cycle management and device management. Additionally, various specific metrics are also critical in constrained environments, such as CPU processing requirements, signaling overhead, and memory allocation for caching procedures in addition to power consumption and battery lifetime. Also, in nodes acting as gateways, which typically not only act as a point of service to a large number of nodes, but also have to satisfy the information requests from remote entities; they need to consider scalability-related metrics, such as frequency and processing of successfully satisfied information requests. Finally, given the in-network caching functionality of Information- Centric Networks, metrics for the efficiency and performance of in- network caching have to be defined. Such metrics will need to guide researchers and operators regarding the performance of in-network caching algorithms. A first step on this direction has been made in [L9]. The paper proposes a formula that approximates the proportion of time that a content stays in a network cache. The model takes as input the rate of requests for a given content (the Content of Interest) and the rate of requests for all other contents that go through the given network element (router) and move the CoI down in the (LRU) cache. The formula takes also into account the size of the Pentikousis, et al. Expires April 24, 2014 [Page 15] INTERNET DRAFT ICN Evaluation Methodology October 21, 2013 cache of this router. The output of the model essentially reflects the probability that the CoI will be found in a given cache. The initial study [L9] is applied to the CCN/NDN framework, where contents get cached at every node they traverse, while efforts are underway to assess the accuracy of the model for other caching strategies. The formula according to which the probability or proportion is calculated is given by: pi = [mu/(mu+lambda)]^N, where lambda is the request rate for CoI, mu is the request rate for contents that move CoI down the cache and N is the size of the cache (in slots). The formula can be used to assess the caching performance of the system and can also potentially be used to identify the gain of the system due to caching. This can then be used to compare against gains by other factors, e.g., addition of extra bandwidth in the network. 2.5. Resource Equivalence and Tradeoffs As we have seen above, every ICN network is built from a set of resources, which include link capacities, different types of memory structures and repositories used for storing named information objects and chunks temporarily (i.e. caching) or persistently, as well as name resolution and other lookup services. Complexity and processing needs in terms of forwarding decisions, management (e.g. need for manual configuration, explicit garbage collection, and so on), and routing (i.e. amount of state needed, need for manual configuration of routing tables, support for mobility, etc.) set the stage for a range of engineering tradeoffs. In order to be able to compare different ICN approaches it would be beneficial to be able to define equivalence in terms of different resources which today are considered incomparable. For example, would provisioning an additional 5 Mb/s link capacity lead to better performance than adding 100 GB of in-network storage? Within this context one would consider resource equivalence (and the associated tradeoffs) for example for cache hit ratios per GB of cache, forwarding decision times, CPU cycles per forwarding decision, and so on. 3. ICN Security Aspects The introduction of an information-centric networking architecture Pentikousis, et al. Expires April 24, 2014 [Page 16] INTERNET DRAFT ICN Evaluation Methodology October 21, 2013 and the corresponding communication paradigm changes many aspects of network security. These will affect all the scenarios described in [draft-irtf-icnrg-scenarios]. Additional evaluation will be required to ensure relevant security requirements are appropriately met by the implementation of the chosen architecture in the various scenarios. The various ICN architectures that are currently proposed have concentrated on authentication of delivered content to ensure the integrity of the content. However the approaches are primarily applicable to freely accessible content that does not require access authorization, although they will generally support delivery of encrypted content. The introduction of widespread caching mechanisms may also provide additional attack surfaces. The caching architecture to be used also needs to be evaluated to ensure that it meets the requirements of the usage scenarios. In practice, the work on security in the various ICN research projects has been heavily concentrated on authentication of content. Work on authorization, access control, privacy and security threats due to the expanded role of in-network caches has been quite limited. A roadmap for improving the security model in NetInf can be found in [NETINFSC]. In the rest of this section we briefly consider the issues and provide pointers to the work that has been done on the security aspects of the architectures proposed. 3.1. Authentication For fully secure content distribution, content access requires that the receiver needs to be able to reliably assess: validity: is it a complete, uncorrupted copy of what was originally published; provenance: can the receiver identify the publisher, and, if so, whether it and the source of any cached version of the document can be adequately trusted; and relevance: is the content an answer to the question that the receiver asked. All the ICN architectures considered in this document primarily target the validity requirement using strong cryptographic means to tie the content request name to the content. Provenance and relevance are directly targeted to varying extents: There is a tussle or trade-off between simplicity and efficiency of access and Pentikousis, et al. Expires April 24, 2014 [Page 17] INTERNET DRAFT ICN Evaluation Methodology October 21, 2013 level of assurance of all these traits. For example, maintaining provenance information can become extremely costly, particularly when considering (historic) relationships between multiple objects. Architectural decisions have therefore been taken in each case as to whether the assessment is carried out by the ICN or left to the application. An additional consideration for authentication is whether a name should be irrevocably and immutably tied to a static piece of preexisting content or whether the name can be used to refer to dynamically or subsequently generated content. Schemes that only target immutable content can be less resource hungry as they can use digest functions rather than public key cryptography for generating and checking signatures. However, this can increase the load on applications. This is because they are required to manage many names, rather than using a single name for an item of evolving content that changes over time (e.g. a piece of data containing an age reference). NetInf uses the Named Information (ni) URI scheme [RFC6920] to identify content. This allows NetInf to assure validity without any additional information but gives no assurance on provenance or relevance. A "search" request allows an application to identify relevant content and applications may choose to structure content to allow provenance assurance but this will typically require additional network access. NetInf validity authentication is consequently efficient in a network environment with intermittent connectivity as it does not force additional network accesses and allows the application to decide on provenance validation if required. NetInf primarily targets static content, but an extension would allow dynamic content to be handled. The immutable case only uses digest functions. DONA [DONA] and CCN [CCN], [SECCONT] integrate most of the data needed to verify provenance into all content retrievals but need to be able to retrieve additional information (typically a security certificate) in order to complete the provenance authentication. Whether the application has any control of this extra retrieval will depend on the implementation. CCN is explicitly designed to handle dynamic content allowing names to be pre-allocated and attached to subsequently generated content. DONA offers variants for dynamic and immutable content. PURSUIT [PSTSEC] appears to allow implementers to choose the authentication mechanism so that it can, in theory, emulate the authentication strategy of any of the other architectures. It is not clear whether different choices would lead to lack of interoperability. Pentikousis, et al. Expires April 24, 2014 [Page 18] INTERNET DRAFT ICN Evaluation Methodology October 21, 2013 3.2. Authorization, Access Control and Statistics A potentially major concern for all the ICN architectures considered here is that they do not provide any inbuilt support for an authorization framework or for statistics monitoring. Once content has been published and cached in servers, routers or end points not controlled by the publisher, the publisher has no way to enforce access control, determine which users have accessed the content or revoke its publication. In fact, in some cases, it is even difficult for the publishers themselves to perform access control, where requests do not necessarily contain host/user identifier information. Access could be limited by encrypting the content but the necessity of distributing keys out-of-band appears to negate the advantages of in-network caching. This also creates significant challenges when attempting to manage and restrict key access. An authorization delegation scheme has been proposed [ACDICN] but this requires access to a server controlled by the publisher to obtain an access token making it essentially just an out-of-band key distribution system. Evaluating the impact of the absence of these features will be essential for any scenario where an ICN architecture might be deployed. It may have a seriously negative impact on the applicability of ICN in commercial environments unless a solution can be found. 3.3. Privacy Another area where the architectures have not been significantly analyzed is privacy. Caching implies a trade-off between network efficiency and privacy. The activity of users is significantly more exposed to the scrutiny of cache owners with whom they may not have any relationship. Although in many ICN architectures, the source of a request is not explicitly identified, an attacker may be able to obtain considerable information if s/he can monitor transactions on the cache and obtain details of the objects accessed, the topological direction of requests and information about the timing of transactions. The persistence of data in the cache can make life easier for an attacker by giving a longer timescale for analysis. The impact of CCN on privacy has been investigated in a useful master's thesis [CCNSEC]. The analysis in this thesis is mostly applicable to all of the ICN architectures because it is mostly focused on the common caching aspect. The privacy risks of named data networking are also highlighted in [CCNPRIV]. Further work on Pentikousis, et al. Expires April 24, 2014 [Page 19] INTERNET DRAFT ICN Evaluation Methodology October 21, 2013 privacy in ICNs can be found in [CONPRV]. 3.4. Changes to the Network Security Threat Model The architectural differences of the various ICN models as compared to TCP/IP have consequences for network security. There is limited consideration of the threat models and potential mitigation in the various documents describing the architectures. The references [CCNSEC] and [CONPRV] also consider the changed threat model. Some of the key aspects are: o Caching implies a tradeoff between network efficiency and user privacy as discussed in Section 3.3. o More powerful routers upgraded to handle persistent caching increase the network's attack surface. This is particularly the case in systems (e.g., CCN) that may need to perform cryptographic checks on content that is being cached. For example, not doing this could lead routers to disseminate invalid content. o ICNs makes it difficult to identify the origin of a request as mentioned in Section 4.3 slowing down the process of blocking requests and requiring alternative mechanisms to differentiate legitimate requests from inappropriate ones as access control lists (ACLs) will probably be of little value for ICN requests. o Denial-of-service (DoS) attacks may require more effort on ICN than on TCP/IP but they are still feasible. One reason for this is that it is difficult for the attacker to force repeated requests for the same content onto a single node; ICNs naturally spread content so that after the initial few requests, subsequent requests will generally be satisfied by alternative sources, blunting the impact of a DoS attack. That said, there are many ways around this, e.g., generating random suffix identifiers that always result in cache misses. o Per-request state in routers can be abused for DoS attacks. o Caches can be misused in the following ways: + Attackers can use caches as storage to make their own content available. + The efficiency of caches can be decreased by attackers with the goal of DoS attacks. + Content can be extracted by any attacker connected to the cache, putting users' privacy at risk. Pentikousis, et al. Expires April 24, 2014 [Page 20] INTERNET DRAFT ICN Evaluation Methodology October 21, 2013 Appropriate mitigation of these threats will need to be considered in each scenario. 4. Security Considerations This document does not impact the security of the Internet. 5. IANA Considerations This document presents no IANA considerations. 6. Acknowledgments Daniel Corujo and Gareth Tyson contributed to an earlier version of this document. This document has benefited from reviews, pointers to the growing ICN literature, suggestions, comments and proposed text provided by the following members of the IRTF Information-Centric Networking Research Group (ICNRG), listed in alphabetical order: Marica Amadeo, Hitoshi Asaeda, Claudia Campolo, Suyong Eum, Dorothy Gellert, Luigi Alfredo Grieco, Myeong-Wuk Jang, Ren Jing, Will Liu, Antonella Molinaro, Ioannis Psaras, Dirk Trossen, Jianping Wang, Yuanzhe Xuan, and Xinwen Zhang. 7. Informative References [RFC5743] Falk, A., "Definition of an Internet Research Task Force (IRTF) Document Stream", RFC 5743, December 2009. [RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis, "Framework for IP Performance Metrics", RFC 2330, May 1998. [RFC6920] Farrell, S., Kutscher, D., Dannewitz, C., Ohlman, B., Keranen, A., and P. Hallam-Baker, "Naming Things with Hashes", RFC 6920, April 2013. [ndnSIM] Afanasyev, A. et al., ndnSIM: NDN simulator for NS-3 NDN Technical Report NDN-0005, Revision 2, October 2012. [ccnSim] Rossini, G. and D. Rossi, "Large scale simulation of CCN networks", Proc. Algotel 2012 , La Grande Motte, France, May 2012. Pentikousis, et al. Expires April 24, 2014 [Page 21] INTERNET DRAFT ICN Evaluation Methodology October 21, 2013 [CCNPL] Muscariello, L., "Content centric networking packet level simulator", available online at http://perso.rd.francetelecom.fr/muscariello/sim.html [CCNj] Cianci, I. et al. "CCN - Java Opensource Kit EmulatoR for Wireless Ad Hoc Networks", Proc. 7th ACM Int. Conf. on Future Internet Technologies, Seoul, Korea, Sept., 2012. [IEICE] G. Parisis, D. Trossen, and H. Asaeda, "A Node Design and a Framework for Development and Experimentation for an Information-Centric Network", IEICE Trans. Commun., vol. E96-B, no. 7, pp.1650-1660, July 2013. [ICN-Sim] N. Vastardis et al., "Simulation Tools Enabling Research on Information-centric Networks", Proc. ICC FutureNet Workshop. IEEE, 2012. [PROBCACHE] I. Psaras, W. Chai, G. Pavlou, "Probabilistic In-Network Caching for Information-Centric Networks", Proc. SIGCOMM ICN Workshop. ACM, 2012. [CL4M] Chai, W. K. et al., "Cache 'Less for More' in Information- centric Networks", Proc. Networking. IFIP, 2012. [HASHROUTE] L. Saino, I. Psaras, G. Pavlou, "Hash-routing Schemes for Information-Centric Networking", Proc. SIGCOMM ICN Workshop. ACM, 2013. [FNSS] L. Saino, C. Cocora and G. Pavlou, "A Toolchain for Simplifying Network Simulation Setup", Proc. SIMUTOOLS. ACM, 2013. [BA] Barabasi, A. and R. Albert, "Emergence of scaling in random networks", Science, vol. 286, no. 5439, pp. 509- 512, 1999. [WATTS] Watts, D. J. and S. H. Strogatz, "Collective dynamics of small-world networks", Nature, vol. 393, no. 6684, pp. 40- "10, 1998. [Montage] Hussain, A. and J. Chen, "Montage Topology Manager: Tools for Constructing and Sharing Representative Internet Topologies", DETER Technical Report, ISI-TR-684, Aug. 2012. [DELAY] Kaune, S. et al., "Modelling the Internet Delay Space Based on Geographical Locations", Proc. Euromicro, Weimar, Germany, 2009. Pentikousis, et al. Expires April 24, 2014 [Page 22] INTERNET DRAFT ICN Evaluation Methodology October 21, 2013 [L1] http://googleblog.blogspot.it/2008/07/we-knew-web-was- big.html [L2] Zhang, C., Dhungel, P., and K. Di Wu., "Unraveling the BitTorrent ecosystem", IEEE Transactions on Parallel and Distributed Systems, pp. 1164-1177, 2010. [L3] Cha, M., Kwak, H., Rodriguez, P., Ahn, Y.-Y., and S. Moon, "I tube, you tube, everybody tubes: analyzing the world's largest user generated content video system", Proc. ACM SIGCOMM conference on Internet measurement (IMC), San Diego (CA), USA, Oct. 2007. [L4] Zhou, J., Li, Y., Adhikari, K., and Z.-L. Zhang, "Counting YouTube videos via random prefix sampling", In Proc. of IMC'11, Berlin, Germany, Nov. 2011. [L5] Fricker, C., Robert, P., Roberts, J. and N. Sbihim, "Impact of traffic mix on caching performance in a content-centric network", In Proc. of IEEE NOMEN 2012, Workshop on Emerging Design Choices in Name-Oriented Networking, Orlando, USA, Mar. 2012. [L6] Yu, H., Zheng, D., Zhao, B. Y. and W. Zheng, "Understanding user behavior in large-scale video-on- demand systems", In SIGOPS Oper. Syst. Rev., Vol. 40, pp. 333-344, April 2006. [L7] Marciniak, P., Liogkas, N., Legout, A. and E. Kohler, "Small is not always beautiful", In Proc. of IPTPS, International Workshop of Peer-to-Peer Systems, Tampa Bay, Florida (FL), USA, Feb. 2008. [L8] Bellissimo, A., Levine, B. and P. Shenoy, "Exploring the use of BitTorrent as the basis for a large trace repository", University of Massachusetts, Tech. Rep., 2004. [L9] Psaras, I. et al., "Modelling and Evaluation of CCN- Caching Trees", In Proc. of the 10th international IFIP conference on Networking, Valencia, Spain, May 2011. [L10] Carofiglio, G., Gallo, M., Muscariello, L., and D. Perino, "Modeling Data Transfer in Content-Centric Networking", In Proc. of ITC, San Francisco, USA, Sep. 2011. [L11] Breslau, L., Cao, P., Fan, L., Phillips, G. and S. Shenker, "Web caching and zipf-like distributions: Pentikousis, et al. Expires April 24, 2014 [Page 23] INTERNET DRAFT ICN Evaluation Methodology October 21, 2013 evidence and implications", In Proc. of INFOCOM '99, New York (NY), USA, Mar. 1999. [L12] Mahanti, A., Williamson, C., and D. Eager., "Traffic analysis of a web proxy caching hierarchy", IEEE Network, Vol.14, No.3, pp.16-23, May/June 2000. [P2PMod] Saleh, O., and M. Hefeeda, "Modeling and caching of peer- to-peer traffic", Proc. ICNP. IEEE, 2006. [CCN] Jacobson, V. et al., "Networking Named Content", Proc. CoNEXT. ACM, 2009. [VoCCN] Jacobson, V. et al., "VoCCN: Voice-over Content-Centric Networks", Proc. CoNEXT Re-Arch Workshop. ACM, 2009. [NDNP] Zhang, L. et al., "Named Data Networking (NDN) Project", NDN Technical Report NDN-0001, Oct. 2010. Available: http://named-data.net/publications/techreports/ [4WARD6.1] Ohlman, B. et al., "First NetInf Architecture Description", 4WARD Project Deliverable D-6.1, Apr. 2009. [4WARD6.3] Ahlgren, B. et al., "NetInf Evaluation", 4WARD Project Deliverable D-6.3, June 2010. [SAIL-B2] SAIL, "NetInf Content Delivery and Operations", SAIL Project Deliverable D-B.2, May. 2012. [SAIL-B3] Kutscher, D. (ed.) et al., "Final NetInf Architecture", SAIL Project Deliverable D-B.3 , Jan. 2013. Available: http://www.sail-project.eu/deliverables/ [PRST4.5] Riihijarvi, J. et al., "Final Architecture Validation and Performance Evaluation Report", PURSUIT Project Deliverable D4.5, Jan. 2013. [CMT-D5.2] B ben, A. et al, "Scalability of COMET System", COMET Project Deliverable D5.2, Feb. 2013. [CMT-D6.2] Georgiades, M. et al., "Prototype Experimentation and Demonstration", COMET Project Deliverable D6.2, Feb. 2013. [SHARE] Muscariello, L. et al., "Bandwidth and storage sharing performance in information centric networking", Proc. SIGCOMM ICN Workshop. ACM, 2011. [RealCCN] Perino, D. et al., "A Reality Check for Content Centric Pentikousis, et al. Expires April 24, 2014 [Page 24] INTERNET DRAFT ICN Evaluation Methodology October 21, 2013 Networking", Proc. SIGCOMM ICN Workshop. ACM, 2011. [ICN-Web] Detti, A. et al., "Supporting the Web with an Information Centric Network that Routes by Name", Elsevier Computer Networks, vol. 56, no. 17, Nov. 2012. [ICN-Scal] Blefari Melazzi, N. et al., "Scalability Measurements in an Information-Centric Network", Springer Lecture Notes in Computer Science (LNCS), vol. 7586, 2012. [ICN-Tran] Salsano, S. et al., "Transport-Layer Issues in Information Centric Networks ", Proc. SIGCOMM ICN Workshop. ACM, 2012. [SensReqs] Karnouskos, S. et al., "Requirement considerations for ubiquitous integration of cooperating objects", Proc. NTMS. IFIP, 2011. [NETINFSC] Renault, E, Ahmad, A., and M. Abid, "Towards a Security Model for the Future Network of Information", Proc. Conf. Ubiquitous Information Technologies and Applications, IEEE, 2009. [DONA] Koponen, T. et al., "A Data-Oriented (and Beyond) Network Architecture", Proc. SIGCOMM. ACM, 2007. [SECCONT] Smetters, D., and V. Jacobson, "Securing network content", Technical Report TR-2009-01, PARC, 2009. [PSTSEC] Tagger, B., et al, "Update on the Architecture and Report on Security Analysis", Deliverable 2.4, PURSUIT EU FP7 project, April 2012. [ACDICN] Fotiou, N. et al., "Access control enforcement delegation for information-centric networking architectures", Proc. SIGCOMM ICN Workshop. ACM, 2012. [CCNSEC] Lauinger, T., "Security and Scalability of Content-Centric Networking", Masters Thesis, Technische Universitaet Darmstadt and Eurecom, Sep. 2010. [CCNPRIV] Lauinger, Y., et al, "Privacy Risks in Named Data Networking: What is the Cost of Performance?", ACM SIGCOMM Computer Communication Review Editorial Note, vol. 42, iss. 5, 2012 [CONPRV] Chaabane, A et al, "Privacy in Content-Oriented Networking: Threats and Countermeasures", arXiv:1211.5183, 2012. Pentikousis, et al. Expires April 24, 2014 [Page 25] INTERNET DRAFT ICN Evaluation Methodology October 21, 2013 Authors' Addresses Kostas Pentikousis (editor) EICT GmbH Torgauer Strasse 12-15 10829 Berlin Germany Email: k.pentikousis@eict.de Borje Ohlman Ericsson Research S-16480 Stockholm Sweden Email: Borje.Ohlman@ericsson.com Elwyn Davies Trinity College Dublin/Folly Consulting Ltd Dublin, 2 Ireland Email: davieseb@scss.tcd.ie Spiros Spirou Intracom Telecom 19.7 km Markopoulou Avenue 19002 Peania, Athens Greece Email: spis@intracom.com Gennaro Boggia Dep. of Electrical and Information Engineering Politecnico di Bari Via Orabona 4 70125 Bari Italy Email: g.boggia@poliba.it Priya Mahadevan Pentikousis, et al. Expires April 24, 2014 [Page 26] INTERNET DRAFT ICN Evaluation Methodology October 21, 2013 Palo Alto Research Center 3333 Coyote Hill Rd Palo Alto, CA 94304 USA Email: Priya.Mahadevan@parc.com Pentikousis, et al. Expires April 24, 2014 [Page 27]