IPN Research Group V. Cerf INTERNET-DRAFT Worldcom/Jet Propulsion Laboratory S. Burleigh August 2002 A. Hooke Expires February 2003 L. Torgerson NASA/Jet Propulsion Laboratory R. Durst K. Scott The MITRE Corporation K. Fall Intel Corporation H. Weiss SPARTA, Inc. Delay-Tolerant Network Architecture: The Evolving Interplanetary Internet Status of this Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. Abstract This document describes an architecture for delay-tolerant networks, and is a generalization of the architecture designed for the Interplanetary Internet: a communication system to provide Internet- like services across interplanetary distances in support of deep space exploration. This generalization addresses networks whose operational characteristics make conventional networking approaches become either unworkable or impractical. We define a message-based overlay that exists above the transport layer of the networks on which it is hosted. The draft presents an architectural overview followed by discussions of services, topology, routing, security, reliability and state management. It concludes with a discussion of end-to-end information exchange, including an example. Cerf, et al. Expires February 2003 [Page 1] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 Table of Contents Status of this Memo................................................1 Abstract...........................................................1 Table of Contents..................................................2 Copyright Notice...................................................3 Foreword...........................................................5 1 Introduction................................................6 2 Why an Architecture for Delay-Tolerant Networking?..........6 2.1 Extreme Environments...................................7 2.2 Problems with Internet Protocols and Applications.....11 2.3 DTN Principles of Design..............................14 3 DTN Architectural Overview.................................16 3.1 The DTN Architecture is Based on Message Switching....16 3.2 DTN Classes of Service Mimic Postal Operation.........17 3.3 Regions and Region Identifiers........................17 3.4 Tuples................................................18 3.5 Late Binding..........................................18 3.6 The "Bundle Layer" Terminates Local Transport Protocols and Operates End-to-End...............................19 3.7 Routing...............................................19 3.8 Bundle Layer Reliability and Custodianship............19 3.9 Security..............................................20 3.10 Time Synchronization..................................21 4 Service Considerations: Application Instances and Bundles.21 4.1 Networking Style Issues...............................21 4.2 Bundle Lifetime.......................................22 4.3 Classes of Service....................................23 4.4 Delivery Options......................................23 4.5 Naming and Addressing.................................24 5 Topological Considerations: Nodes, Regions, and Gateways..24 5.1 Nodes.................................................25 5.2 Regions...............................................25 5.3 Gateways..............................................26 5.4 Discussion............................................26 6 Routing considerations.....................................26 6.1 Types of Routes.......................................27 6.2 Store-and-forward operation...........................28 6.3 Contact-oriented routing..............................29 6.4 Routing protocols.....................................29 7 Security considerations....................................30 7.1 User-oriented security services.......................30 7.2 Infrastructure security services......................32 8 Reliability Considerations.................................35 8.1 Custodial Operation...................................35 8.2 End-to-end Retransmission.............................36 8.3 Congestion and Flow Control at the Bundle Layer.......37 9 State Management Considerations............................38 9.1 Application Interface State...........................39 9.2 Bundle retransmission state...........................39 Cerf, et al. Expires February 2003 [Page 2] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 9.3 Bundle routing state..................................40 9.4 Transmission queue state..............................41 9.5 Receive queue state...................................41 9.6 Network management state..............................41 10 Convergence Layer Considerations for Use of Underlying Protocols..................................................42 11 Bundle Header Information..................................42 12 An Example Bundle Transfer.................................44 12.1 Rules for forming tuples in the example network.......44 12.2 Example Network Topology at the Region Level..........45 12.3 DTN Gateway routing...................................47 12.4 Systems participating in example bundle data transfer.48 12.5 End-to-end Transfer...................................50 12.6 Error Conditions at the Bundle Layer..................53 13 Summary....................................................55 14 References.................................................56 15 Security Considerations....................................57 16 Authors' Addresses.........................................58 Copyright Notice Copyright (C) The Internet Society (2002). All Rights Reserved. Cerf, et al. Expires February 2003 [Page 3] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 Acknowledgments John Wroclawski, David Mills, Greg Miller, James P. G. Sterbenz, Joe Touch, Steven Low, Lloyd Wood, Robert Braden, Deborah Estrin, Stephen Farrell and Craig Partridge all contributed useful thoughts and criticisms to this document. We are grateful for their time and participation. This work was performed in part under DOD Contract DAA-B07-00-CC201, DARPA AO H912; JPL Task Plan No. 80-5045, DARPA AO H870; and NASA Contract NAS7-1407. Cerf, et al. Expires February 2003 [Page 4] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 Foreword "The term 'sonata' can be applied to a large work in three movements or to the three sections of a single movement: exposition, development, recapitulation. They resonate with the three acts of a movie or play, the three elements of a primal myth or legend (life, death, rebirth), the cycle of days (daylight, night, then daylight again) and of the seasons in temperate climates (nice weather, then winter, then it gets nice again), the Star Wars trilogy, etc. Basically, in any human experience we start off brightly, full of energy and hope; then we experience reality and become thoughtful and reflective, darker and slower; then Something Happens and hope is reborn, our energy returns, and we reach that happy synthesis of Innocence and Experience that is maturity. Allegro, andante, rondo..." -- Scott Burleigh Release Notes draft-irtf-ipnrg-arch-00.txt, May 2001: Allegro. Original Issue. draft-irtf-ipnrg-arch-01.txt, August 2002: Andante. Restructured document to distinguish between architecture for delay- tolerant networking and the *application* of that architecture to a number of different environments, potentially including interplanetary internetworking, military tactical networking, and sensor internetworking. Refined DTN classes of service and delivery options. Added a "reply- to" address to have response information such as error messages or end-to-end acks directed to a source-specified third party. Further defined the topological elements of delay tolerant networks. Elaborated routing and reliability considerations. Presented a model for securing the delay tolerant network infrastructure. Expanded discussion of flow and congestion control. Added section discussing state information at the bundle layer. Updated bundle header information and end-to-end data transfer. Cerf, et al. Expires February 2003 [Page 5] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 1 Introduction This document describes an architecture for delay-tolerant networks. We present this as a generalization and extension of our ongoing work on the Interplanetary Internet (http://www.ipnsig.org). We have come to the realization that there are a number of different environments that share essential characteristics for which the architecture presented herein is appropriate. For example, sensor-based networks may have scheduled intermittent connectivity with a "conventional" internet with long delays between contacts. These delays may result from limited availability of communication assets (e.g. Low Earth Orbiting satellite contacts) or from limited availability of other communication assets (e.g., power to run a transmitter). Applications using the network may perceive long, highly variable end-to-end delays, particularly if multiple instances of intermittent connectivity affect a path from source to destination. This fully terrestrial example may appear no different to an application than an interplanetary communication, even though each individual link may be low-delay. The particular approach we employ is that of a message-based overlay that exists above the transport layers of the networks on which it is hosted. 2 Why an Architecture for Delay-Tolerant Networking? The existing TCP/IP-based internet, while fabulously successful in many environments, does not suit all environments. The ability of the "TCP/IP suite" to provide service depends on a number of important assumptions: . that an end-to-end path between source and destination exists for the duration of the communication session; . (for reliable communication) that the maximum round-trip time over that path is not excessive and not highly variable from packet to packet; and . that the end-to-end loss is relatively small. A class of "challenged networks," which may violate one or more of these assumptions, is becoming important and may not be well served by the current end-to-end TCP/IP model. These networks typically serve environments in which it is either impractical or impossible to configure a communication environment that supports the assumptions on which the TCP/IP suite depends. This section examines some of the constraints posed by extreme environments that may result in "challenged networks," and considers the problems that these environments might cause for common Internet protocols and applications. Finally, we derive some "principles of design" for the DTN. Cerf, et al. Expires February 2003 [Page 6] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 2.1 Extreme Environments 2.1.1 Constraints Posed by Extreme Environments The kinds of extreme environments that this DTN architecture addresses constrain both the communication system and the applications using that communication system. This section describes some of the characteristics that make environments "extreme." Clearly, not all environments that would be considered "extreme" exhibit all of these characteristics. Long and/or Variable Delays Delay affects networks in at least two different ways. First, interactivity is directly affected by delay. For example, telephone conversations over a terrestrial telephone line are markedly different than telephone conversations that are routed over a geostationary satellite hop. Second, variability in delay can affect applications and protocols. Consider a stream of packets carrying voice data over a network. If the delay in the network varies greatly from one packet to the next, a buffer is required to avoid the perception of pauses in the reconstructed voice track. If the application is hosted on a network with greater delay variability than the application was designed for, the stream will again have pauses, possibly of sufficient magnitude to make the data unintelligible. Delays are comprised primarily of three components: propagation delay through the medium; queuing delay within relay points, source, and destination; and clocking delays associated with transmitting an atomic unit of data onto the medium. Propagation delays can be long due to speed-of-light delays to cross long distances (e.g., deep space). Alternatively, propagation delays could be long due to the propagation medium (e.g., acoustic/underwater). How long is long? Light time (round trip) between Earth and Mars ranges from ~8 minutes to ~40 minutes. (Mars's average distance from the sun is 1.5AU, and light in a vacuum travels at roughly 8 minutes per AU.) Queuing delays are affected by traffic and service rate. In extreme environments, the service rate of a node may be low due to power limitations requiring a slow processor or a long wait while the unit is quiescent. Arrival distributions may cause significant variability in delays, particularly for heavy-tailed distributions [KP97]. (Heavy-tailed distributions are those that asymptotically follow a power law. That is, P[X > x] ~ x^-alpha as x approaches infinity, for 0 < alpha < 2. For alpha < 2, the variance of a heavy- Cerf, et al. Expires February 2003 [Page 7] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 tailed distribution is infinite. For alpha < 1, the mean is also infinite. "Thus, as alpha decreases, a large portion of the probability mass resides in the tail of the distribution." [KP97]) Clocking delays accrue each time a packet is transmitted if the packet was fully received prior to transmission. "Cut-through" routing, in which the first bits/bytes of a data unit are operated upon before the last bits/bytes have been received, is a fairly rare operation. For many types of data, the latency advantages of cut- through routing are more than offset by the reluctance of network designers to forward corrupt data. (Although some types of data are useful even if partially corrupted.) For data that are not forwarded before being fully received (and error checked), clocking delays accumulate at each hop in the network. For relatively slow, multi- hop networks, this can result in high per-packet delays, particularly if packet sizes are large. (Which can, of course, increase queuing delays for other packets, increasing both the round trip times and the variability of delay.) We assume that processing delay, another contributor to overall delay, is comparatively low. However for some simple processors, cryptographic processing, and in particular key generation, can contribute sufficiently to processing delay and in some circumstances make it become a noticeable contributor to overall delay. Variability of delay can have a significant effect on communication in addition to absolute delay. Many protocols attempt to provide reliability through retransmission (ARQ) mechanisms. Timers are a critical element of most retransmission mechanisms, and establishing a retransmission time out value is an important aspect of the mechanism. Some protocols, particularly those at the link layer, have the ability to eliminate all components of round trip time other than propagation and clocking delays from their timings, and are able to produce retransmission time out estimates that are quite close to the actual round trip time. When queuing delay must be considered and the timing does not have direct knowledge of the delays at all of the queues, estimating a reasonable retransmission time out value becomes more complex. A timeout value that is too low will cause unnecessary retransmission, while one that is too high will cause excessive delay in data delivery. Frequent Partitioning (Intermittent Reachability) Network partitioning typically adds to data loss, because data that cannot be forwarded essentially immediately is generally discarded. If losses become too severe, applications typically fail. In addition to contributing to loss, network partitioning adds significantly to overall delay if nodes are configured to store data until the outbound link is restored. This behavior can also contribute to Cerf, et al. Expires February 2003 [Page 8] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 misordered data arriving at the destination, which can cause poor performance in some protocols. Data Rate Asymmetry Data rate asymmetry measures the ratio of forward-path minimum capacity versus reverse-path minimum capacity. Data rate asymmetry adversely affects protocols using ARQ for reliable delivery by altering the ACK or NACK return path. This effect has been studied extensively with TCP, and is discussed below. At a higher layer, request/response applications that have not been appropriately tuned to balance the amount of request versus response traffic can time out waiting for data to travel across the lower-capacity link. In the most extreme case, the asymmetry is infinite (as is the case with communications to radio-silent submarines, for example) ARQ is simply not possible. In these cases, forward error correction and periodic retransmission are typically used to improve communication reliability. See [BLMR98] as an example of how such techniques are used in the Internet. Data rate asymmetry can arise due to routing path asymmetry, different forward/reverse path link technologies, or intentional engineering tradeoffs. Moderate asymmetries in the wired Internet are found most commonly in asymmetric access technologies such as in CATV or ADSL-based subscriber lines. The largest asymmetries for wireless Internet access appear to be prevalent in in-flight Internet access systems. The AirTV system, for example, being proposed for 2004 and beyond, could provide a ground-to-air speed of up to 45Mb/s using DVB satellite technology with a comparatively anemic air-to- ground bandwidth of a 128kb/s or less using INMARSAT [ARINC]. In the case of deep space communications, downlink data rate is intentionally engineered to be much greater than uplink data rate. This tradeoff is not surprising given the goal of most science missions: the capacity demands of downlinking high-resolution images and telemetry from a spacecraft dwarf the capacity required to uplink short commands to that spacecraft. Packet Loss and Errors Consider the following probability of success metric: Pr = (1-pe)^(i*(8*m)) This metric is the probability of successful end-to-end delivery of a particular message of size m bytes across i links assuming an identical, independently distributed (IID) stationary loss process with constant bit error rate pe across all intervening forwarders. (In this equation, the carat -- ^ -- indicates exponentiation.) This metric, as expressed, does not account for congestive loss but does Cerf, et al. Expires February 2003 [Page 9] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 account for message size and number of cascaded links given a fixed BER. For packet networks, most links employ some form of error detection, implying that any bit loss creates an end-to-end packet loss. As can be clearly seen from the formula, the end-to-end probability of successful delivery decays exponentially with path hop count. Any congestive loss would only worsen the performance. For reliable transfer, excessive errors require repair using either error correction or retransmission. If the path is sufficiently lossy, end-to-end retransmission can become essentially useless. To understand why this is so, consider a probability of packet loss pp = 1 ¡ Pr for some fixed m. Given the assumptions listed above, the expected number of retransmissions required before successful delivery is given by: (1-(1-pp)^i)/(1-pp)^i. Considering a packet error probability of 0.3. In this case, 4 hops requires 3 retransmissions, 10 hops requires 34, and 20 hops requires about 1200 retransmissions. If a hop-by-hop retransmission scheme were used instead, the total (network wide) number of retransmissions is given by i*pp/(1-pp). For the same error probability of 0.3, the number of retransmissions for 4, 10, and 20 hops would be 2, 5, and 9, respectively. Thus, for very lossy environments, hop-by-hop retransmission is preferable to end to end retransmission. 2.1.2 Interoperating Among Differently-Challenged Networks Two complicating factors in building internetworks comprised of networks operating in multiple different extreme environments are 1. Network adaptations necessary to achieve required operational characteristics in one extreme environment may be mutually exclusive with those required in another environment, and 2. Evolution to achieve interoperability with networks not anticipated at design time may be technically or politically infeasible. There is no guarantee of support for any common communication technology across networks designed for operation in extreme environments. In many instances, at the time of deployment only the barest minimum of capabilities can be fielded to support the network's mission. If these networks are successful, they evolve. In many cases, this evolution leads to an expansion of scope and span for the network. Often this can result in a requirement to interoperate with another network in an environment that presents different challenges that have been addressed by dramatically different solutions. The ability to operate over many different types of networks that have been adapted to support a range of benign and extreme Cerf, et al. Expires February 2003 [Page 10] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 environments is both a challenge and a requirement for this Delay Tolerant Networking Architecture. 2.1.3 Examples of some extreme environments Several types of extreme environments currently exist, with more appearing almost daily. If one considers as the norm the wired Internet, with multi-gigabit backbones, almost no corruption-related data losses and relatively short round trip times, then many different types of environments seem extreme. This architecture was originally conceived to support the Interplanetary Internet, which exhibits round trip delays on the order of tens of minutes and intermittent connectivity that can result in disconnection for weeks. Military tactical networks, in which data rates are low, error rates are high, and link interruptions are frequent, can likely derive great benefit from application of this architecture. Similarly, sensor networks deployed in oceanic environments exhibit long delays, significant data loss, and the potential for long link outages. An individual seeking a networking solution for a particularly intriguing and challenging environment has recently approached us to determine if the DTN architecture was right for the following environment. The Sami people in Lapland live in widely dispersed communities in remote areas and are not well served by either wired, fixed wireless, or satellite internet service. They do, however, frequently travel on snowmobiles from community to community, and congregate at work sites and larger towns. To effectively serve this environment, an architecture is required that can embrace intermittent, probabilistic connectivity, store and forward operation, and high and variable delays. (The architecture described in this document meets these needs.) 2.2 Problems with Internet Protocols and Applications This section discusses some of the issues associated with attempting to serve challenged networks with the Internet suite of protocols. 2.2.1 Internet "Core" Protocols The performance characteristics of challenged networks confound the efficient operation of the core Internet protocols. By the 'core' Internet protocols we mean IP, TCP, UDP, BGP, common IGPs (RIP, OSPF, or EIGRP) and DNS. These protocols span the services of end-to-end datagram delivery, reliable two-party stream delivery, regional Cerf, et al. Expires February 2003 [Page 11] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 (aggregated) routing path discovery with policy, intra-domain path selection and distributed support for name resolution. Although some of these protocols are technically "application" protocols from a layering point of view, we treat them here together as core protocols for the purposes of discussion. Excessive latency affects TCP directly by severely limiting its throughput performance or interfering with connection establishment [DMT96]. Any application-layer protocol using TCP as its underlying transport is therefore affected (for example, BGP and DNS zone transfers). Excessive latency also adversely affects the proper operation of conventional routing protocols as links will be incorrectly discovered to be non-operating when soft-state refreshes are delayed too long (RIP), or a lack of response to link state "hello" messages (OSPF, EIGRP) is observed. UDP is not sensitive to excessive latency because it does not contain timers that affect its operation. However, core application protocols that require it (for example, DNS queries/responses; also SNMP, if used) will take too long to complete when faced with excessive delay, triggering early application failure (or the lack of ability to use DNS name/address mappings in the case of address-to-name queries). Data rate asymmetry adversely affects TCP by altering the smooth flow of acknowledgments. Extensive studies have been conducted on TCP using the normalized asymmetry ratio, which takes into account the asymmetric message sizes between data packets and returning ACKS [MLS00]. Results indicate that bandwidth asymmetry can lead to poor performance of unidirectional transfers due to alterations in the time series of the ACK channel. In particular, if ACKs are queued, their spacing in time causes the TCP sender to clock out subsequent packets less frequently. If ACKs are lost, burstiness, slow congestion window growth, or defeat of the fast retransmit/recovery algorithms can occur. (See [PILC-ASYM] characterization of this problem and some possible approaches to ameliorate it). Any application layer protocols attempting to use TCP over asymmetric links are therefore affected as well. Significant path loss affects the TCP transport most strongly, causing it a number of problems. After multiple loss events it will continue retrying with a backed-off retransmission timer until it gives up on retransmitting altogether and closes the connection. Somewhat more moderate losses will contribute to problems invoking the fast retransmit and recovery algorithms, even in the presence of SACK. IP performance can also be affected by path loss if fragmenta- tion is required. IP provides no mechanism for fragment retransmission, thereby causing the overall probability of successful datagram delivery to be further reduced if datagrams are fragmented [M95]. Cerf, et al. Expires February 2003 [Page 12] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 Node longevity affects the probability of end-to-end delivery, as round-trip or even one-way delivery time of a particular message may exceed the sender's lifetime. Clearly, in such cases it is useless to arrange for immediate acknowledgements to be returned. In such cases, any notification of successful or unsuccessful delivery needs to be directed to some alternative node that remains functional. 2.2.2 Common Internet Applications and Supporting Protocols The most common user services within the Internet today are the web, e-mail, and perhaps FTP. These services employ a number of different application protocols, including HTTP, SMTP, POP, IMAP, Microsoft's Exchange Protocol and FTP. These protocols, or in some cases the applications that implement them, are either severely degraded or fail to operate entirely under challenging conditions. Two primary reasons are application-level timeouts mismatched to the actual network latency and excessive numbers of round-trip request/response message exchanges required in order to accomplish a protocol transaction. For example, web browsers and mail clients will often time out during connection establishment after a minute or two. The SMTP protocol requires peers to mutually identify themselves prior to actually exchanging the mail payload, even though this early name exchange is not fundamental to the process of delivering e-mail. Such protocols have been called "chatty" due to their excessive number of round-trip times required to accomplish simple tasks. FTP has a similarly chatty interaction during its initial association as user login and authentication information is exchanged between client and server. 2.2.3 Discussion When considering the use of existing Internet protocols as a basis for interconnecting challenged networks, certain properties of the operational environment seem insurmountable. The possibility of intermittently infinite data rate asymmetry appears to preclude the use of conventional TCP-like protocols based on ARQ for reliability because an ACK channel may not be available. With exponentially decaying probability of successful end-to-end delivery as path length grows, end-to-end reliability should be avoided in favor of hop-by- hop retransmission in lossy environments. Given the possibility of very high round-trip times, any expectation of timely performance in request/response protocols must also be avoided. Finally, the standard Internet routing protocols assume the immediate existence of an end-to-end path and do not accommodate the notion of future (scheduled) routing opportunities that are needed for operation in frequently partitioned networks. Cerf, et al. Expires February 2003 [Page 13] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 2.3 DTN Principles of Design After examining the environmental conditions cited above and seeing the problems that the Internet protocols have with these conditions, we derived the following principles of design to guide the definition of a Delay Tolerant Networking Architecture. 2.3.1 Use transport and network layer technologies appropriate to the local environment Where Internet protocols work well, there is a distinct advantage to using them. However, in other environments, such as sensor networks or certain radio networks, the internet suite may not be the best fit for the task at hand. Some sensor networks, for example, use very different routing and node naming concepts (e.g., directed diffusion [IGE00]). The DTN architecture should not inhibit designers from building networks using transport, network, link, and physical layer technologies that are best suited for their environments. Rather, the DTN architecture should provide the means for dissimilar networks to interoperate. The dissimilarity of the extreme environments discussed precludes the use of a single end-to-end transport protocol. Instead, end-to-end operations require us to provide a common substrate above the *transport* layers of the constituent networks. Thus, whereas IP provides the common substrate for the Internet above dissimilar link layers, the DTN architecture creates a "network of Internets" by providing an end-to-end layer above the transport layer. We call this the "bundle layer." 2.3.2 Employ a "non-chatty" communication model Because delays may be long and network partitioning may be frequent, we suggest that highly interactive styles of communication be avoided in the kinds of extreme environments described above. Rather, entire application-layer messages, which we call "bundles," move through the network as units. These bundles are intended to have all of the primary data and meta data necessary for their use at the destination. Although this *may* mean that bundles are large (relative to packets in a TCP/IP network), it does not require it. The intent simply is to not have multiple exchanges between source and destination (e.g. "user:", response, "password:", response, etc.) when such exchanges can be "bundled" together into a single communication - a single bundle or a unidirectional sequence of bundles. 2.3.3 Base transfers between nodes on store-and-forward techniques The Internet Protocol (IP) is based on store and forward transmission. However, in IP routers, the "store" portion of the Cerf, et al. Expires February 2003 [Page 14] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 operation typically lasts only long enough to determine the next hop destination and to wait for any packets to be transmitted that are ahead of the current one. If no route to the destination exists at the time that the routing engine examines the packet, that packet is typically discarded. In a Delay-Tolerant Network, it is considered *normal* for there to be no path to the destination at the time a bundle is received. Bundles are routed based not on current connectivity but on the expectation of connectivity, drawn either from a firm schedule or from previous experience. If an episode of connectivity doesn't occur as expected, bundles are rerouted accordingly. This approach assumes that there is a considerable amount of storage available within the network, which is a fundamental assumption for these types of delay-tolerant networks. The result of this interpretation of store-and-forward operation is that bundles of data can still make forward progress toward their destination even if an end to end path never exists at any single moment in time. 2.3.4 Advance the point of retransmission toward the destination The retransmission of lost data in the Internet is (nominally) always end-to-end: the source must commit resources to the retention of data for possible retransmission by TCP until reception by the destination is acknowledged, and retransmitted data must always traverse the entire route from source to destination. In a Delay-Tolerant Network such behavior might impose high performance costs due to the length of time needed to traverse that route, so the point of data retransmission is advanced toward the destination whenever possible. This allows earlier release of retransmission resources at any given node and enables retransmitted data to arrive sooner. There is an obvious benefit to this approach if the point of retransmission progresses across a *very* long delay hop (such as one that spans interplanetary distance). However, in a network composed of a series of lossy links (such as a congested multi-hop wireless network), there is similar benefit, as discussed in section 2.1.1, "Packet Loss and Errors." Retransmission point advance occurs on two of the three levels at which the DTN Architecture supports retransmission (the third being the optional end-to-end retransmission discussed in section 8.2). First, wherever transmission between pairs of DTN nodes is accomplished by underlying reliable transport-layer protocols such as TCP, the probability of correct end-to-end delivery at the bundle Cerf, et al. Expires February 2003 [Page 15] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 layer is enhanced simply by the concatenation of episodes of point- to-point reliability provided by those protocols. (The "points" of transmission at the bundle layer look like "ends" at the transport layer.) Data arrival at a transport-layer destination informally advances the point of retransmission within the DTN: the destination of one transmission that is reliable at the transport layer becomes the source of the next one. Second, upon receipt of a bundle, a node may *take custody* of it. This is a commitment made by the node that it will shoulder the burden of retransmission *at the bundle layer* to insure that the bundle reaches its destination, and is accomplished by issuing a bundle-layer acknowledgment to the previous custodian (which might be the source). This allows the previous custodian to release the resources necessary to ensure delivery and formally moves the point of bundle-layer retransmission closer to the destination. 2.3.5 Provide security services to secure both the infrastructure and the information Networks in extreme environments may range from being completely disposable to irreplaceably precious. Those that are disposable are typically so resource starved that difficult tradeoffs must be made regarding provision of service versus protection of infrastructure. These tradeoffs must be made by those who instantiate the architecture, and the architecture must provide for a range of decisions. Minimally, we have concluded that the architecture must provide for infrastructure protection, up to and including mutual suspicion among DTN routers. However, this must be optional at the discretion of those who actually instantiate a particular delay tolerant network. Security services offered to users should include key management, authentication, integrity, and confidentiality, but availability and use of those services may vary from DTN to DTN. 3 DTN Architectural Overview The previous section presented the design principles that guide the definition of the DTN architecture. This section presents an overview of the decisions that result from those design principles. 3.1 The DTN Architecture is Based on Message Switching In order to avoid unnecessary interaction between the source and destination, a DTN transmits "bundles" that contain both user data and relevant metadata. These bundles are the "messages" on which the DTN architecture is based. Cerf, et al. Expires February 2003 [Page 16] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 A message-switched architecture provides the network with a priori knowledge of the size and performance requirements of requested data transfers. When there is a significant amount of queuing that can occur prior to transmission over an outbound route (as is the case in the DTN version of store-and-forward) the advantage provided by this information provided is significant. In an environment that frequently experiences network partitioning, messages are efficient units for queuing while awaiting network reconnection. 3.2 DTN Classes of Service Mimic Postal Operation The (U.S. or similar) Postal Service provides a strong metaphor for the services that a Delay-Tolerant Network offers. Traffic is generally not interactive. It may be one-way in nature. There are generally no strong guarantees of timely delivery. The DTN Architecture, like the Postal Service, offers *relative* measures of priority (low, medium, high). It may offer basic notifications, for example: "the intended recipient has accepted delivery of the message," "the route taken by this message is as follows..." and "the message has been transmitted." An essential element of Postal Service operation is that mail has a place to wait in queue until an outbound hop is available. This highlights the previously stated assumptions: 1. That there is storage available in the network, 2. that storage is well-distributed throughout the network, 3. that it sufficiently persistent to store bundles until custody transfer can occur, and 4. (implicitly) that this 'store-and-forward' model is a better trade-off than using resources to attempt to effect continuous connectivity or other alternatives. For a network to effectively support the DTN architecture, these assumptions must be considered and must be found to hold. 3.3 Regions and Region Identifiers The DTN architecture defines a "network of Internets." Each of these Internets is called a "DTN region." Regions may be delimited by differences in network protocol, transport protocol, or addressing mechanism. Regions may also be delimited by other criteria, such as trust boundaries [NEWARCH]. Each DTN region has a unique identifier that is known (or knowable) among all regions of the DTN. Cerf, et al. Expires February 2003 [Page 17] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 All inter-region communication takes place via DTN *gateways*, which provide the interconnection points between regions. These correspond to "waypoints" in [META], and to the gateways described in the original ARPANET design [CK74]. DTN gateways differ from ARPANET gateways because they operate above the transport layer and are focused on message switching rather than packet switching. However, both DTN gateways and ARPANET gateways provide conversions between the protocols specific to one region and the protocols specific to another. Region definition and region identification are key concepts in the DTN architecture. DTN bundles that originate outside the destination region are first transmitted to one of the DTN gateways that connect the source region with one or more other regions. Routing outside the destination region is based on the destination region's identification, meaning that there is a single region identifier space that is common across the end-to-end DTN. 3.4 Tuples The region identifier described above is sufficient to route a bundle of data to its destination region, but not to deliver it to the specific entity for which it is intended. The DTN architecture uses a tuple of the region identifier and a region-specific entity identifier to identify uniquely a particular entity in the DTN. The tuple notion allows us to distinguish data that is necessary for routing *across* regions from that which is necessary for delivery within a region. As a result of this, the entity ID is *opaque* outside the region of definition. The referenced entity might be a host, a protocol, an application, or something else. Further, the entity identifier might be a name, an address, or a descriptor such as a URL. This is a local issue within the entity's region. (If the entity ID is a name, there is an assumption that a name-to-address binding function exists within the region to support eventual routing of the bundle to the destination entity.) 3.5 Late Binding The opacity of entity IDs outside their local region enforces another key concept in the DTN Architecture: that of late binding. Late binding refers to the fact that the entity ID of a tuple is not interpreted (e.g., bound to an address) outside its local region. This avoids having a universal name-to-address binding space (and its associated database and synchronization issues). This approach preserves a significant amount of autonomy within each region. The entity IDs used in bundles might be built on DNS names, Cerf, et al. Expires February 2003 [Page 18] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 or URLs, but they might also be "expressions of interest" as in a directed diffusion-routed network [IGE00]. Additionally, late binding avoids the delay-related issue that the binding information might be highly ephemeral relative to the transit time of a bundle. We assume that the internal details of a destination network will be sufficiently stable for the duration of a transmission of a message within that region, or that delay-tolerant mechanisms will be employed *within* the region to compensate. (This is, by definition, a local issue within the region and may be accommodated in whatever manner is most practical for that region.) 3.6 The "Bundle Layer" Terminates Local Transport Protocols and Operates End-to-End The bundle layer provides an end-to-end protocol that employs custody in-network retransmission, both point-to-point retransmission between transport-layer endpoints and also custody transfer between (possibly non-adjacent) DTN nodes. The bundle layer makes use of the reliability mechanisms available from the underlying transport layers of each region, and a single bundle-layer hop *may* span an entire region. 3.7 Routing The bundle layer provides routing among DTN-capable nodes, including DTN gateways. There may be many DTN gateways that interconnect adjacent regions, and there may be multiple bundle routing "hops" within a region (particularly if intra-regional connectivity is intermittent). Routing information exchange mechanisms may differ from one instantiation of the Delay Tolerant Network Architecture to another, reflecting different needs and constraints of the extreme environments. Different routing protocols may be employed for intra- region routing and for inter-region routing. We do not require that the *same* inter-region routing protocol run among all (inter-region) DTN gateways in a Delay Tolerant Network. (It is required that necessary routing information be able to be propagated throughout the DTN, but the mechanisms for doing that are neither defined nor constrained by the DTN Architecture.) 3.8 Bundle Layer Reliability and Custodianship The bundle layer provides three types of reliability services: end- to-end acknowledgment of a bundle (and accompanying retransmission), custodial transmission (with in-network retransmission if required), and unacknowledged bundle transmission. End-to-end acknowledgment and custodial transfers may be combined for additional reliability. Cerf, et al. Expires February 2003 [Page 19] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 Reliability services at the bundle layer are relatively simple because the bundle layer depends upon the reliability mechanisms in each of the underlying internets. Bundles are delivered atomically and are, generally, independent of other bundles. As a result, positive acknowledgment is possible, but partial or negative acknowledgment is not. Since delays may be long and/or highly variable, retransmission timers must, in general, be coarse and may be strongly affected by local policy. Custody transfer, introduced in section 2.3.4, above, allows the source to delegate retransmission responsibility and recover its retransmission-related resources relatively soon after sending the bundle. While not every node in a DTN must be capable of providing custodial services, all DTN gateways (that span two or more DTN regions) are expected to be able to be custodians. 3.9 Security Resource scarcity in delay-tolerant networks dictates that some form of access control is required. It is not acceptable for an unauthorized user to be able to flood the network with high-priority traffic, possibly preventing the network's mission from being accomplished. Implicit in this statement is a requirement for some form of admission control and/or in-network authentication that is sensitive to the class of service that a user has requested, and the means to verify that the user is authorized to make that request. In a low delay environment, this would simply be tedious. In a high/variable delay, low data rate environment, it is potentially much worse: access control lists are difficult to update, re-keying presents hard problems, and routers or end nodes might be compromised. Key management presents a number of dilemmas. Keys will almost certainly require replacement in all but the most disposable of networks. Re-keying and related operations on highly remote nodes is challenging, and proper approaches to key distribution in this environment remain active areas for research. The DTN must be able to support end-to-end user services that include verification of a message's integrity and the ability to provide confidentiality of a user's message. It must do this in a way that does not require a great deal of interaction across long-delay (or resource-constrained) data links. "Support" for these services may involve provision of a key management service to support user-level confidentiality and integrity, or it may involve providing these services as part of the bundle layer. This is an area still under consideration. Cerf, et al. Expires February 2003 [Page 20] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 3.10 Time Synchronization The DTN architecture depends on time synchronization for two primary purposes: routing, and "time to live" computations. Routing that is based on schedules and dependent upon coordination of shared assets (such as directional antennas) creates a requirement for time synchronization. As with many requirements, just how *much* time synchronization is required is an economic trade-off. The cost to achieve a particular degree of time synchronization must be weighed against the cost of not having it. The costs of not having time synchronization include reduced overall efficiency of a communication asset, potential loss of user data, power inefficiency, and many others. For some communication assets, such as NASA's Deep Space Network, the operational cost is many thousands of dollars per hour, and the cost to achieve a modest level of time synchronization between peers is quite justifiable. There is a study currently underway [DM02] to examine the time synchronization issues across NASA's Deep Space Network and to determine a particular time synchronization scheme suited to that network. Given that DTN routing depends to some degree on time synchronization, the DTN architects have made the decision to exploit its availability. In particular, time to live computations may be done by including a source time stamp and an explicit time to live field (in time units after the time in the source time stamp). This requires some level of time synchronization, but since its sole use is for purging data from the network, the synchronization requirements are not strict. This approach allows a source time stamp to be used for a number of purposes, such as to provide replay protection and to identify individual messages. 4 Service Considerations: Application Instances and Bundles This section describes the basic services available to applications using the DTN Architecture's Bundle Service. The Bundle Service is the message-oriented communication service provided by the "Bundle Layer" described in section 3.6. 4.1 Networking Style Issues Before discussing specific Bundle Services, a note about application interaction style is in order. This service is intended to operate in *delay-tolerant* networks. That does not mean that there *must* be delays in the network, or that there *will* be delays, but that there *may* be delays. The protocols providing the services exposed by the bundle layer are delay tolerant. Applications using those services should be, too. Cerf, et al. Expires February 2003 [Page 21] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 Message-oriented communication differs from conversational communication. In general, applications should attempt to include enough information in a bundle so that it may be treated as an independent unit of work by the remote entity (a "transaction", although transactions carry connotations of multi-phase commitment that we do not intend here). The goal is to minimize conversation between application entities that are separated by a network that presents long and possibly highly variable delays, and to constrain any conversation that does occur to be asynchronous in nature. A file transfer application, for example, might incorporate account information, file location information, file operation information, and file content within a bundle, allowing atomic operation on that bundle. Comparing this style of operation to a classic FTP transfer, one sees that the bundled model can complete in one round trip time, whereas an FTP file "put" operation can take as many as eight round trips to get to a point where file data can flow [WhynotTCP]. Delay-tolerant applications must consider additional factors beyond the conversational implications of long delay paths. Application instances may terminate (voluntarily or not) between the time that a bundle is sent and the time its response is received. If an application has anticipated this possibility, it is possible to reinstantiate the application instance based on state information saved in persistent storage. This is an implementation issue, but also an application design consideration. Similarly, either host or application mobility may result in the application's address having changed by the time a response is received. If applications embed physical addresses into their data, they will defeat the potential benefits that late binding provides. Some consideration of delay-tolerant application design can result in applications that work well in low-delay environments, and that do not suffer extraordinarily in high or highly-variable delay environments. 4.2 Bundle Lifetime Applications are requested to provide an expiration time (actually, a time to live in seconds) for a bundle if the value of the data has a finite duration. If not supplied, or if the user-supplied value is larger than local policy permits, the bundle layer will provide a value. Note that this value is treated as an actual "time to live" - - it is added to the time that a bundle was submitted by the application to determine the time at which the bundle will be purged from the network. Appropriate values depend on the network and the data, and may range from seconds to weeks. Cerf, et al. Expires February 2003 [Page 22] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 4.3 Classes of Service The DTN architecture provides for a range of classes of service. These classes of service are requests to the bundle layer to provide *relative* distinctions in the immediacy with which bundles are transmitted. We have currently defined three classes of service in the DTN architecture: . Bulk - In bulk class, bundles are shipped on a "best effort" basis. No bulk class bundle will be shipped until all bundles of other classes bound for the same next hop destination have been shipped. . Normal - Normal class bundles that are in queue and bound for the same next hop destination are shipped prior to any bulk class that are in queue. . Expedited - Expedited bundles, in general, are shipped prior to bundles of other classes. However, the bundle layer *may* implement a queue service discipline that prevents starvation of the normal class, which may result in some normal bundles being shipped before expedited bundles bound for the same next hop destination as the normal class bundles. 4.4 Delivery Options The bundle layer offers a number of delivery options to users. First and foremost is the option of custodial delivery. Custodial delivery means that a bundle will be reliably transmitted by the bundle layer, and that the point of retransmission at the bundle layer will advance through the network from one custodian to another until the bundle reaches its destination. The bundle layer depends on the underlying transport protocols of the networks that it operates over to provide the primary means of reliable transfer from one bundle layer entity to the next. However, when custodial delivery is requested, the bundle layer additionally provides a coarse-grained timeout and retransmission mechanism and an accompanying acknowledgment mechanism. When a bundle application does *not* request custodial delivery, this bundle layer timeout and retransmission mechanism is not employed, and the bundle depends solely on the reliability mechanisms of the underlying transport layers. A second delivery option is the option of a return receipt. An application may request a return receipt for a bundle that is being transmitted. When the subject bundle is received *by the destination application instance* (not simply the destination bundle layer entity), a return receipt is generated by the bundle layer entity and returned to the source or the source's designated alternate (reply- to) application instance, which would typically be located on a Cerf, et al. Expires February 2003 [Page 23] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 different host. The return receipt uses the same custodial delivery option used in the bundle that caused the return receipt to be sent. (Return receipts are never issued with the return receipt delivery option requested.) The third and fourth delivery options are used for diagnostic purposes, and may not be available to all users, subject to local policy. These two delivery options result in notifications being sent to the reply-to application. The first causes a notification to be sent every time a bundle is forwarded. The second option causes a notification to be sent every time a custody transfer occurs. 4.5 Naming and Addressing As described in section 3.4, bundle application instances refer to one another using tuples. The tuple consists of two parts: a region identifier and an entity identifier. DTN regions will be discussed in more detail in section 5.2, but they are named by applications using syntax identical to that used in the domain name system (DNS). (That is, hierarchical tree structure, dot-separated text node names, left to right progresses from leaf to root, sibling nodes must have different names.) When a region is served by the Internet's Domain Name System distributed database, DTN region names may or may not be stored in that database. The bundle layer may translate a region name to a bundle-layer-specific region address for transmission. The scope of the region name space and region address space spans an entire DTN, and name-to-address translations of DTN regions must be consistent and reversible throughout the DTN. The other portion of a tuple (the entity identifier) is opaque outside the region specified by the tuple's region identifier (the "home region" for the entity). In internet-served regions, an entity ID may be a hostname, protocol identifier, port (used to find the bundle service on the destination host) and potentially a token used for demultiplexing to a particular application using the bundle service. These could potentially be combined into a URL (depending on the region's internal rules). In diffusion-routed regions, they may be an expression of interest [IGE00]. In any case, no attempt shall be made by the bundle layer to bind the entity ID to an address from any location outside the entity's home region. Discovery of valid DTN region names and entity identifiers is out of the scope of this document. 5 Topological Considerations: Nodes, Regions, and Gateways This section describes the three key topological elements in the DTN architecture and how they are related: DTN nodes, DTN regions, and DTN gateways. Cerf, et al. Expires February 2003 [Page 24] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 5.1 Nodes A DTN node (or "node" in this document) is an engine for sending and receiving DTN messages (bundles). DTN nodes may act as sources, destinations, or forwarders of bundles. There is a one-to-one mapping between DTN nodes and DTN entities. Each node is uniquely identified by at least one tuple containing region identifier and entity identifier; a node that is reachable within multiple regions will have at least one identifying tuple for each region within which it is reachable. The identifier for a DTN node, meaning the bundle sending and receiving engine itself as opposed to an application *using* that engine, is specified in a region-specific manner using all of part of the entity identifier. 5.2 Regions The following list identifies the requirements for DTN regions: . Each DTN region shall have an identifier space that is shared by all DTN nodes within the region. A region must specify the naming conventions to be employed within that region for entity identification. . Each node that is a member of the region shall have a unique identifier drawn from that identifier space. (Note that for some types of regions, a "node" may be made up of a collection of computational elements, possibly geographically distributed. A single unique identifier may collectively refer to them. Further, the unique identifier requirement only applies to nodes that are intended to receive data from other DTN nodes.) . To be considered a member of a region, each prospective member of the region shall have the ability to reach every other member of the region without depending on any DTN nodes outside the region. (Although a DTN node may not necessarily be *directly* reachable. This may require forwarding and/or store-and-forward operation by other DTN nodes inside the same region.) A DTN region is a generalization of the notion of a "network" that relaxes the assumption of real-time end-to-end connectivity. DTN regions may be created for a number of reasons. Accommodation of tailored support for a particularly challenging networking environment is one reason for defining a DTN region. Delimiting a particular security boundary may be another reason for defining a DTN region boundary. Cerf, et al. Expires February 2003 [Page 25] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 5.3 Gateways A gateway is an entity that has membership in two or more DTN regions and relays messages between regions. Entities that reside (only) in one region can only send messages to entities in other regions with the assistance of one or more DTN gateways. 5.4 Discussion DTN nodes may act as "hosts," meaning entities that send and receive bundles, but do not forward them. DTN nodes may also act as "routers", meaning they perform the functions of a host, but also forward bundles within a single region. Finally, DTN nodes may act as gateways, forwarding bundles across region boundaries. A single DTN node may act in any subset of these roles simultaneously. As members in a store-and-forward chain, DTN nodes have the responsibility for resource allocation to support bundle transfers. These resources include, among other things, buffer space and transmission capacity. Additionally, DTN nodes have the responsibility of actually executing the bundle transfer. Reliability requirements for bundle transfers are specified by the using application, and include both reliable and unreliable transfers. The DTN nodes are responsible for using whatever reliability mechanisms exist in the underlying (transport- and-below) layers, and augmenting those mechanisms with bundle-layer mechanisms to effect the required reliability. We require that each DTN region have a unique identifier that is known among all regions in the DTN or that can be discovered as required. The entity identification rules for a particular DTN region have scope only within that region, and are opaque outside that region. However, all DTN nodes within a region must know and conform to the entity identification rules in order to successfully communicate. 6 Routing considerations Communication within a DTN can be either intra-regional or inter- regional. While this architecture does not prescribe specific methods for the exchange of routing data, it acknowledges the fact that mechanisms for the exchange of intra-regional bundle routing data may well be quite different from the mechanisms to exchange inter-regional routing information. Further, the architecture Cerf, et al. Expires February 2003 [Page 26] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 recognizes that mechanisms for inter-regional transfer in one portion of a DTN may be inappropriate for use in other portions of that DTN, and does not require homogeneity of inter-region routing protocol throughout the DTN. 6.1 Types of Routes DTNs may be required to function in the presence of any or all of the following types of routes. These are presented as information rather than as requirements for all DTNs, although any of these may be required for a *particular* DTN. 6.1.1 Persistent Contacts Persistent contacts are sessions with a neighboring DTN node that are always available or that can be made available on demand. In the IP world, Digital Subscriber Line (DSL) connections are an example of the former, while dial-up connections are an example of the latter. 6.1.2 Intermittent - Scheduled Contacts Scheduled routes are those where there is an agreement to establish a link between two points at a particular time, for a particular duration. Attributes of that link, such as its data rate, probability of loss, and the routing metrics to destinations via that link, are routing protocol-specific. An example of a scheduled route is a link that uses a low-earth orbiting satellite. A schedule of view times and durations can be constructed when next-hop neighbors are accessible via the satellite. 6.1.3 Intermittent - Opportunistic Contacts Opportunistic contacts are those that are not scheduled, but rather present themselves unexpectedly. Such contacts could be with an aircraft flying overhead and beaconing, advertising its availability for communication. Another type of opportunistic contacts might be via infrared communication link between a personal digital assistant (PDA) and a kiosk in an airport concourse offering bundle service as the PDA's owner walks by. If the PDA's owner authorizes it, the PDA could communicate the owner's planned path and a desire for contacts with subsequent kiosks in the concourse, resulting in a set of probable contacts for which bundle transfers can be scheduled. 6.1.4 Intermittent - Predicted Contacts Cerf, et al. Expires February 2003 [Page 27] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 Predicted contacts are those that are based on no fixed schedule, but a history of opportunistic contacts that suggests that contact with an intermittently-connected neighbor will probably occur within a certain period of time and will probably last for a certain duration. Given a great enough certainty that the contact will occur, a DTN node may allocate bundles to that predicted contact period that would be allocated to different contacts otherwise. In the previous example, the probable contacts in the airport concourse would result in the establishment of a set of predicted contacts of a given duration (perhaps based on the PDA owner's walking speed and the kiosk's coverage area). The PDA could decide how to use those contacts for transmitting waiting bundles, as well as perhaps to request bundles that were awaiting delivery at any of a number of store-and-forward points to which the user had access. The algorithms for establishing the predicted time and duration of a contact, the degree of uncertainty in those estimates, the time at which to abandon the wait for a predicted contact, and the guidelines for allocating bundles to such contacts are all open research areas. 6.2 Store-and-forward operation While IP networks are based on "store-and-forward" operation, there is an assumption that the "storing" will not persist for more than a modest amount of queuing and transmission delay. In contrast, the DTN architecture does not expect that an outbound link will be available when a bundle is received, and expects to store that bundle for some time until a link does become available. We anticipate that most DTN nodes will use some form of persistent storage for this -- disk, flash memory, etc. All DTN forwarding nodes ("DTN routers"), even those that do not provide custodial operations as described in section 3.8, must be able to queue bundles until outbound contacts are available. Each DTN forwarding node and DTN gateway that also provides custody transfer operations must provide for longer-term storage of bundles, committing to store bundles for which it takes custody for the entire remainder of their lifetime, if necessary. It is entirely possible that a custodian will need to "take a break" from further custodianship while it completes pending custodial operations and recovers persistent storage. Two mechanisms support this: First, the node can simply forward incoming bundles without taking custody. The fact that a node is a potential custodian is no guarantee that it will take custody of any given bundle that is routed to it. Second, the node can revise its advertisement of custodial capability in routing updates (see section 6.4). This is a longer-term solution, but has the attractive property that DTN nodes searching for a custodian do not route a bundle out of its way vainly in search of custodianship at the node in question. Cerf, et al. Expires February 2003 [Page 28] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 6.3 Contact-oriented routing Making routing decisions to allocate bundles to future contacts requires a DTN forwarding node to probabilistically estimate what the connectivity to other points in the network will be via that contact. We do not yet have a significant body of experience with this, but we suspect that maintaining a weighted average of previous routing metrics and developing an uncertainty metric will be useful. Based on the expected connectivity to the remainder of the network from each of these intermittent contacts, and similar characterizations of connectivity for persistently connected neighbors, a DTN forwarding node can make decisions about how to allocate bundles to contacts. One could think of this allocation of bundles to contacts as being made by creating an ordered list of references to the bundles in persistent storage and providing that list to the lower layer protocol (or to the convergence layer) for transmission at the start of a contact. (We refer to such a list of bundles as a "contact list.") Bundles not transmitted during the contact could be returned on the list for allocation to subsequent contacts. With such an approach, the bundle layer is certainly free to adjust the allocation of bundles to future contacts, based on the receipt of new, possibly high priority bundles. It is also possible that the bundle layer could make changes to a contact list during the course of a contact, if the implementation permitted. 6.4 Routing protocols We have not yet developed routing protocols for this architecture, but envision the need for at least two: one to support inter-region routing, and at least one to support intra-region routing. Realistically, there will probably be more than two. It is likely that an inter-region routing protocol for a network having one or more links with 40-minute one way delays and two-week link outages (such as might be encountered in a deep-space mission) would be wholly different than an inter-region routing protocol to support a sensor network connected to the Internet via a low-earth orbiting satellite. Our current thinking in routing protocols to support intermittent connectivity is to maintain one-hop link state information and a distance-vector type of aggregation beyond one hop. We feel that attempting to maintain an "accurate" picture of network connectivity beyond one hop when routes are composed of scheduled or probabilistic connections of uncertain capacity will be futile. However, we Cerf, et al. Expires February 2003 [Page 29] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 believe that a probabilistic view of connectivity can be built and advertised in a scalable way that permits effective use of these intermittent contacts. The fact that there is no expectation of a contemporaneous end-to-end path makes the overall job seem more feasible. As previously mentioned, this area is teeming with interesting research topics. We intend to address some of these in coming months, but we certainly solicit the participation of interested researchers. 7 Security considerations While the particular security problems presented by delay-tolerant networks do not necessarily differ from those presented by other networks, the environment has a significant effect on the available solution space. In addition to typical end-user oriented security, providing writer- to-reader services such as message confidentiality or end-to-end user authentication, protection of the network infrastructure and controlling access to that infrastructure tend to be important in delay-tolerant networks, which are typically resource-challenged and are critical components to mission success. Along with high round- trip times and frequent partitioning, delay-tolerant networks often have low data rate links, making efficiency a necessary consideration in any solution. It is not unusual for delay-tolerant networks to be deployed in places where a portion of the network might become compromised. Such compromise, should it occur, must be limited in scope and the effect finite in duration. 7.1 User-oriented security services We have not received explicit user requirements for the following security services, but anticipate their need. We are as yet undecided whether to provide confidentiality and user authentication as explicit bundle-layer services (a la IPSEC) or to simply make the key management and distribution mechanisms we require available to user applications for them to provide their own confidentiality and authentication (a la PGP, SSL, etc.). Current discussions have focused on using a combination of public key and symmetric key cryptographic methods for the security services. 7.1.1 Confidentiality The confidentiality service attempts to insure that the information content of a bundle is not disclosed to anyone except the intended recipient. If the bundle layer provides the confidentiality service (a la IPSec Encapsulating Security Payload) the intended recipient Cerf, et al. Expires February 2003 [Page 30] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 will be the bundle layer entity at the destination (rather than any specific application). In a public key based system, this requires a high confidence that the sender has the public key of the intended recipient and that said key is unique. The confidentiality service will render all affected portions of the bundle unreadable, so it cannot be applied to fields in the bundle header that are needed for delivery. In general, the bundle user data is covered by the confidentiality service but none of the rest of the bundle header is covered. 7.1.2 User Authentication User authentication is a capability to prove that the bundle has been sent by the entity that claims to have sent it. Typically, an authentication service ensures that the content of the bundle is unaltered as a means to prevent replay attacks. Authentication is accomplished in a public key system by the use of a digital signature. A digital signature is created by computing a hash of the data to be covered by the authentication, and then encrypting that information with the private key of the sender. If the sender's public/private key pair is uncompromised and the cryptographic algorithm is sufficiently strong, the recipient can apply the purported sender's public key to the encrypted data, recover the hash, and compare it to a hash of the corresponding information in the bundle. This allows the recipient to determine if the sender's key is correct (authenticating the sender's identity), and if the data received is identical to the data that was sent (verifying the integrity of the data). Note that this service is required to support the infrastructure security model described in section 7.2, and hence could be made available to end-users to authenticate them to the remote bundle entity at relatively low cost. However, the remote bundle entity will have to request a trusted copy of the purported sender's public key, potentially adding considerable delay to the receipt of the first bundles until a signed copy of the sources public key is received, verified, and cached. 7.1.3 Key Management and Distribution The DTN Architecture requires that a key management service be available that can produce, upon request, a set of bytes that are useful to the recipient in verifying the cryptographic correctness of an information object. An additional service is required that allows the verifier to determine the status of the above-mentioned cryptographic verification information. Cerf, et al. Expires February 2003 [Page 31] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 This description is intentionally abstract, as these services may vary with the specific cryptographic mechanisms that are selected (e.g., Kerberos tickets, CRLs with X.509, etc.). 7.2 Infrastructure security services The following model describes an approach to provide infrastructure and data security for a delay tolerant network. It is based on the sensitivity to the effects of long delays, constrained bandwidth, constrained data rates, and intermittent connectivity. Like the rest of this architecture, this security model is preliminary and is still a work in progress. The security architecture is loosely based on the model of secure electronic mail. In the secure electronic mail environment, a source sends secured electronic mail to a destination only knowing the destination's email address. The security attributes of the electronic mail may be that the contents have been encrypted, digitally signed, or both encrypted and signed. Encryption provides "writer-to-reader" confidentiality. That is, only the source and the destination will be able to read the contents of the message. Digital signatures provide authentication of the source of the message as well as message content integrity. In this way, the receiver of the message is assured that the message came from the claimed source and that the message contents have not been tampered with in transit ¡ what is received is exactly what was sent. In the secure electronic mail environment, few assumptions are made about the receiver of the mail. When a message is digitally signed, the sender's private key is utilized. In order to assure that the receiver will be able to authenticate the signature, the sender's authenticated public key is sent along with the message to the receiver. The receiver is able to extract the sender's public key from the message, verify the public key's authenticity based on its being signed by a trusted-third party (e.g., a Certificate Authority (CA)), and then use the verified public key to verify the message digital signature. The receiver needs to know nothing, security- wise, about the sender other than having a copy of the public key of the certificate authority (or equivalent). In order to encrypt the message contents, the sender must have the receiver's public key. By virtue of public key cryptography, only the receiver, using its private key, will be able to decrypt what has been encrypted in the receiver's public key. Typically, the message contents are encrypted in a randomly selected symmetric key. The symmetric key is then encrypted using the public key of the receiver. The receiver is then able to use its private key to decrypt the symmetric key and then use the symmetric key to decrypt the message Cerf, et al. Expires February 2003 [Page 32] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 payload. However, in this case, unlike the digital signature-only case, the sender must have some information about the receiver other than a destination address. The sender must have previously obtained the receiver's public key via a previous exchange, or must have retrieved it from a key server. In a DTN, key servers may or may not exist. In addition, bandwidth to communicate with key servers may be extremely limited or the delay to accomplish a round-trip may be intolerable. Nevertheless, there still needs to be a means by which public keys may be obtained or distributed. For this architecture, we make the assumption that DTN systems that require public keys will have the means to acquire them. They may be able to generate their own public/private key pairs (as in the PGP model) and then have the public key certified as belonging to the owner by a trusted third-party. In this way, a DTN system may only need to expose its public key to the trusted third-party and does not have to trust the third party with its private key at all. The downside to this is that the pedigree of the key generation software may be suspect. That is, the key generators may exist on the specific DTN system but may have been compromised or corrupted. As a result, the keys generated on directly on a DTN system might be weak or otherwise bad keys. Alternatively, a DTN system's keys may be generated and certified by a trusted third-party agent (e.g., Verisign, Thawte, Baltimore). Because the trusted third parties base their livelihood on the goodness of the keys they generate and the trust they provide through their certification of keys, the danger of a corrupted key generator producing weak keys is minimized. Key pairs are required by both end-systems (e.g., principal investigator workstations, space-based instruments, terrestrial-based instruments in a sensor web) as well as intermediate DTN systems (e.g., DTN routers). The end-systems will use their key pairs for both admission control to a DTN as well as for writer-to-reader confidentiality, authentication, and integrity when required. DTN routers will make use of their keys in order to provide assurance that only certified DTN systems have the ability to route data through a DTN. In this manner, DTN systems implement "mutual suspicion." That is, they do not trust each other without each providing proof of identity. It is desirable that all DTN router interactions are authenticated, however that choice will have to be left up to the DTN implementer/operator. Four types of DTN elements have security mechanisms and services associated with them: an end-user system, a DTN router, a DTN Cerf, et al. Expires February 2003 [Page 33] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 interregional gateway (DTN gateway for short), and a DTN certificate authority (CA). As was already stated, each of the DTN elements will have a public/private key pair in place either self-generated and then certified by a DTN CA, or generated by a DTN CA and then populated as necessary in the DTN elements. The final decisions about key management, key/certificate distribution, and key/certificate revocation have not been decided yet and are therefore left for future debates. The key pairs may be pre-placed in the DTN elements. Alternatively, they may be exchanged during a "neighbor discovery" process. Or finally, they may be exchanged in the same manner as with secure electronic mail. That is, they may be sent in a certificate along with the payload. The certificates should be cached by the recipient systems. The "safe" method to ensure that recipients always have the most up-to-date keys would be to send the certificate with every payload. However, because of bandwidth constraints, this may prove to be impractical and therefore an intelligent mode of operation should be employed where state information is maintained regarding communicating end-points to determine if the end-points already have the necessary certificates from previous communications. When an end-system requires admission to a DTN, that is, when it needs to inject data into a DTN, it first presents its credentials to a DTN "gatekeeper" which is analogous to a present day Radius (or other flavor) access control server. The end-system's credentials, based on a certified public key are checked for authenticity and then an access control list is consulted to determine if the end-system should be allowed access. If the end-system is authenticated and access is allowed to a DTN, the end-system must decide whether or not end-to-end security services are required. That is, the end-system must decide whether "write-to-reader" confidentiality, authentication, or integrity is required. If end-to-end services are required, then the end-system may encrypt the payload with a randomly chosen key and then encrypt the key using the public key of the intended recipient of the payload. This would be the equivalent of using Secure Sockets Layer (SSL) or its IETF-version, Transport Layer Security (TLS) at the application layer. Alternatively, the end-system could request and end-to-end service from the bundle layer, if it is chosen to implement such a service at the bundle layer. If this is done, the ends of the end-to-end services would be the sender and receiver bundle agents rather than the actual end-systems. This is because the bundle agents have acted on behalf of the end-system to provide end-to-end security services. Cerf, et al. Expires February 2003 [Page 34] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 Regardless of whether end-to-end security services are required or not, the initial DTN router or gateway that receives the payload from an end-system applies its own digital signature to the bundle so that subsequent DTN routers and gateway can be assured of hop-by-hop authenticity. Likewise, when DTN routers and gateways perform housekeeping chores (e.g., routing updates, etc.) those bundles are also digitally signed in order to assure that only secured transactions between the DTN entities occurs. When a DTN router or DTN gateway receives a bundle, and before it forwards it onward to a neighbor DTN router or gateway, it first validates the bundle by authenticating digital signature from its precursor hop. As the bundle progresses through a DTN, each subsequent DTN device performs an authentication function and then forwards the bundle onwards with its own digital signature to its next hop. 8 Reliability Considerations The bundle layer depends on underlying transport services to provide hop-by-hop reliability. In this context, "hop-by-hop" means from one DTN node to the next. Those nodes may be separated by *many* hops in an underlying network, and the DTN architecture intends to exploit the reliability, flow control, and congestion control capabilities of those underlying networks. However, some significant reliability issues remain at the bundle layer. This section discusses the issues that we have currently identified. 8.1 Custodial Operation One of the fundamental tenets of the DTN architecture is that we must be able to provide reliable end-to-end message transfer via an infrastructure that may *never* be connected end-to-end contemporaneously. Because no single transport-layer protocol operates end-to-end across the IPN, end-to-end reliability can only be assured at the bundle layer. At each node along an end-to-end route, the bundle-layer protocol entity passes bundle data to the Transport layer for transmission. Each bundle layer entity is highly confident that the transport layer will successfully convey the data entrusted to it to the next bundle-layer protocol entity (which may "take custody" of the data or merely relay it; a single hop). But failures are possible (e.g., a host computer does an unplanned reboot). A bundle custodian makes a commitment to store the subject bundle until one of two events occurs: another DTN node accepts custody of the bundle, or the bundle's time to live expires. The bundle layer attempts to deal with bundles atomically and independently of other Cerf, et al. Expires February 2003 [Page 35] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 bundles. In this model, positive acknowledgement of a bundle is possible, but negative acknowledgment is not. We consider the opportunity to *not* deal with partial bundles a significant plus. Since the underlying transport protocols are working on our behalf, we do not consider the lack of a negative acknowledgment capability a significant minus. (Note that there actually *is* a circumstance in which a negative acknowledgment can be sent -- when the time to live for a bundle expires. However, at that point, *all* copies of the bundle, including those stored at a custodian, are purged from the network. A "timed-out" error can be sent to the source, which *might* be able to cause either an application layer retransmission or a bundle-layer retransmission (see section 8.2, below).) Without a negative acknowledgment mechanism, retransmission must be accomplished solely via timer expiration. The bundle layer's confidence in the effectiveness of the underlying transport-layer protocols is reflected in the design of the timers for bundle-layer reliability. These timers are highly optimistic ¡ that is, they expire as late as possible ¡ in order to give the transport protocols every opportunity to complete reliable transmission. The effect of this optimism is to minimize the chance of unnecessary bundle-layer retransmission, which could seriously degrade DTN performance by consuming valuable bandwidth. 8.2 End-to-end Retransmission Even custody transfer operating on a bundle-hop by bundle-hop basis does not provide guaranteed end-to-end reliability. This can only be achieved with an end-to-end reliability mechanism operating between source and destination. In most cases, custodial reliability should suffice. However, the architecture also provides for end-to-end reliability using retransmission mechanisms that are in addition to those used for custody transfers. The availability of timer-based retransmission and acknowledgment mechanisms to support custody transfers means that the mechanisms to support end-to-end retransmission at the bundle layer are essentially available for free. If a DTN user has requested both custodial transfer and the return receipt delivery option, it is reasonable to assume that end-to-end retransmission would be appropriate if, for some reason, the custody transfer operation failed. However, for timer-based retransmission, we must set the retransmission timer to some reasonable value to account for the diligent efforts and ultimate failure of the underlying transport protocols to succeed in either the end-to-end transfer or the end-to-end return receipt. For custody transfers, appropriate retransmission timer setting is less of an issue, since presumably the transfer between the current custodian and the next-hop custodian traverses fewer hops carried by Cerf, et al. Expires February 2003 [Page 36] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 underlying transport protocols. One could reasonably expect better predictions of this round trip time than one could for an end-to-end transfer. The result is that a conservative (optimistic) retransmission time out value for an end-to-end transfer will likely be VERY large. And that value plus the anticipated end-to-end delay for a retransmitted bundle must be less than the time to live for the user's data or else the retransmitted data will time out in transit. If a bundle timed out in transit and the source received the indication of that error, the bundle layer could respond with an end-to-end retransmission, assuming that the remainder of the application's statement of the time-to-live for that bundle to have a reasonable chance of end-to- end transfer success. So, while we believe that the mechanisms are available to effect end- to-end retransmission, we are unsure whether it will be of value in all DTN environments. It is possible that it may provide value in certain environments, and we will continue to consider whether it should be provided as an implicitly-requested capability when users request both custody transfer and return receipt. 8.3 Congestion and Flow Control at the Bundle Layer The subject of congestion control and flow control at the bundle layer is one on which the authors of this draft have not yet reached consensus. We have unresolved concerns about the efficiency and efficacy of congestion and flow control schemes implemented across long and/or highly variable delay environments. One view of congestion control is as follows: "Congestion control is concerned with allocating the resources in a network such that the network can operate at an acceptable performance level when the demand exceeds or is near the capacity of the network resources. These resources include bandwidths of links, buffer space (memory), and processing capacity at intermediate nodes. Congestion occurs when the demand is greater than the available resources." [RJ90] For the purposes of this document, we define "flow control" as a means of assuring that the rate at which a sending node transmits data to a receiving node does not exceed the maximum rate at which the receiving node is prepared to receive data from that sender. (Note that this is a generalized notion of flow control, rather than one that applies only to end-to-end communication. In particular, the concept of flow control between the two ends of a single link may be indispensable in such DTN regions as the "interplanetary backbone.") We define "congestion control" as a means of assuring that the aggregate rate at which all traffic sources inject data into a network does not exceed the maximum aggregate rate at which the Cerf, et al. Expires February 2003 [Page 37] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 network can deliver data to destination nodes. If flow control is propagated backward from destination nodes to routers and eventually back to traffic sources, then flow control can be at least a partial solution to the problem of congestion as well. DTN flow control decisions must be made within the bundling layer itself based on information about resources available within the bundle node. However, the bundle layer *might* be able to delegate the implementation of those decisions to the underlying Transport protocol, as follows. We have not yet considered multipoint communication at or below the bundle layer, so each individual flow control problem involves just two nodes. Since inter-region traffic must pass through inter-region gateways, any two nodes between which flow control is an issue must inhabit a common DTN region and be using a common Transport protocol below the bundle layer (otherwise they could not be communicating and there would be no flow to control). Therefore, it may be possible to accomplish DTN flow control by invoking whatever flow control mechanisms are already provided within the region. Alternatively, a new, supplementary flow control protocol could be developed at the bundling layer; this would have the advantage of reducing DTN's reliance on capabilities provided by the underlying protocols. At this time it's still unclear which approach is preferable. In either case, the problem of flow control between nodes separated by very large signal propagation times remains to be solved: TCP's flow control and congestion control facilities could be leveraged within regions characterized by small round-trip times, but at this time no comparable protocol exists for very long delay paths, such as the "interplanetary backbone". [The Long-haul Transmission Protocol referenced below has yet to be developed.] It remains as an exercise for us to demonstrate that a hop-by-hop flow control scheme in long and/or highly variable delay environments can effect end-to-end congestion control in a fair and efficient manner. 9 State Management Considerations An important aspect of any networking architecture is its management of state information. This section describes the state information that is managed at the bundle layer and discusses the conditions under which that state information is established and how it is removed. Cerf, et al. Expires February 2003 [Page 38] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 9.1 Application Interface State Although it is implementation-specific, it is very likely that application-programming interfaces (APIs) to the bundle layer will be stateful. In long/variable delay environments, an asynchronous interface seems to be the only sensible approach, and such interfaces involve registration actions that create state information. This information is typically created by explicit request of the application, and is removed by a separate explicit request (or, possibly, upon termination of the application). Since operation in a DTN environment means that delays might be quite long, it is reasonable to expect that an application instance might terminate (voluntarily or not) between the time that a bundle is sent and the time its response is received. The application interface to the bundle layer will, in most cases, provide a mechanism to reinstantiate applications that have terminated, assuming the application has been written to take advantage of such a service. When a DTN node that is the destination of a bundle receives it, the bundle layer attempts to deliver it to the destination application instance indicated in the destination tuple. If the application instance is unavailable for immediate delivery (for example, it runs as a result of a "cron" job), then state is created pending delivery of the bundle. That state is removed on successful delivery of the bundle to the application instance or on expiration of the time to live in the bundle. Note that if the return receipt delivery option is enabled for a bundle, that return receipt is not generated until the bundle has been successfully delivered to the destination application. Note that in some DTN environments, delays might REALLY be long, and bundle layer operations may be required to operate across system reboots, host changes, or possibly even operating system upgrades. To facilitate this, some bundle layer implementations will employ various forms of persistent storage for their state information, and will avoid operating system attempts to "clean up" state information. 9.2 Bundle retransmission state State information to support bundle retransmission is created at a DTN node as a result of either of two events: First, state is created when a bundle is received from a DTN user application requesting custodial transmission (assuming that DTN node is capable of supporting custodial operation). Second, state is created when a bundle is received from a peer DTN node and the receiving node intends to assume custody of the bundle. Cerf, et al. Expires February 2003 [Page 39] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 In the first case, recall that applications may indicate a bundle time to live that is greater than that which is allowed in the network. (For example, a DTN application may legitimately feel that its data has value for a century or more, while the DTN that is carrying it doesn't want a particular bundle to potentially be in transit for that long.) The bundle layer may reduce the time to live value in the bundle itself to a value that is consistent with network operation rules. The bundle layer at the source *should* retain the information about the application's view of the bundle's valuable lifetime. In the event that a bundle *transmission* times out before custody is assumed, the bundle layer *may* restart the bundle transmission procedure. This highlights the need to be careful to ensure that bundle acknowledgments have a high probability of receipt, to avoid spurious retransmissions of bundles. At the source, the state associated with a custodially transmitted bundle is removed when a custody transfer acknowledgment is received, or when a period of time subject to local policy has elapsed. For the second case (that of an in-network custodian), state information is removed when a custodial acknowledgment for the bundle is received or when the bundle's time to live has been exceeded. 9.3 Bundle routing state Routing tables, whether statically configured or maintained by routing protocols, introduce state information that must be managed. If the general approach to maintaining routing information is as described in section 6.4 (that is, maintaining link state information for next-hop neighbors, and distance vector information for points beyond), then we can make some basic statements about the amount of state information that must be maintained. For persistent links, in the worst case the volume of routing information for inter-region routing scales by (n*R), where n is the number of next hop neighbors and R is the number of regions in the DTN. For intermittent links, the volume of information scales by (c*n*R), where c is the number of contact periods maintained for planning purposes. These values are worst-case. Pruning and aggregation mechanisms can be used reduce the volume of information if necessary. In particular, it is likely that the intermittent links may be reduced to order (n*R) by simply assuming that the distance vector information is constant for all contacts. (There probably will not be enough useful information to do otherwise.) We have not yet seriously considered the routing metrics that we will maintain, so we do not have an estimate for the size of each routing entry. This remains work to be done. Additionally, we have not given significant consideration to intra- region routing, which will likely have different scaling properties. Intra-region routing information will be affected by the number of DTN gateways that are members of the region and by the routing Cerf, et al. Expires February 2003 [Page 40] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 approach used within the region. (By routing approach we mean broadly either proactive, meaning that routes to all possible destinations are maintained, or reactive, meaning that routes are discovered only when necessary and discarded after some period of time.) This also directly affects how and when routing state information is removed from a DTN forwarding node. 9.4 Transmission queue state Transmission queues are maintained within DTN nodes to manage bundles awaiting transmission. Although implementation dependent, this state information will likely include a list of next hop destinations, and for intermittently connected next-hop destinations, a sublist of upcoming contacts. These lists will likely contain lists of bundles for transmission on those contacts. When a new contact is scheduled or predicted with an intermittently-connected next-hop neighbor, a new list is created and made available for population with bundles. Bundles are removed from these lists upon successful transmission. If a contact ends with bundles remaining on the list to be transmitted, those bundles are allocated to subsequent contact lists and the list for the completed contact is removed. 9.5 Receive queue state Incoming bundles are received from lower layer entities and either placed on a transmission queue (for bundles to be forwarded) or placed in a receive queue (for local delivery). Receive queues are maintained to support demultiplexing of incoming bundles. Again, although implementation-specific, receive queues are likely associated with particular local application entities. A bundle is removed from its receive queue when it has been successfully delivered to its destination application entity. The receive queue itself is removed when an application explicitly requests that its information be purged from the bundle layer (as part of an un- registration action), or when the application entity has terminated without leaving instructions for the disposition of remaining bundles. 9.6 Network management state We acknowledge that network management information is likely to be a necessary part of the operation of delay tolerant networks. We have not, at this point, done a meaningful amount of work in identifying the network management requirements for DTNs or in defining the state to support such management. This remains an item for future work. Cerf, et al. Expires February 2003 [Page 41] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 10 Convergence Layer Considerations for Use of Underlying Protocols The DTN Architecture provides for the existence of a convergence layer between the bundle layer and underlying transport protocols. The convergence layer manages the protocol-specific details of interfacing with a variety of underlying transport services and presents a consistent interface to the bundle layer. The convergence layer also allows additional capability to be inserted as necessary to augment the services of the underlying transport protocols. The convergence layer provides an abstraction to the bundle layer in which bundles are delivered atomically. Partial bundle delivery, especially at the end of a contact, is accommodated within the convergence layer. Additionally, the convergence layer may provide performance enhancement services such as the ability to resume a partially-received bundle on a subsequent contact with the same next- hop neighbor (versus starting the bundle transmission over). 11 Bundle Header Information The bundle layer must carry some information end-to-end. This section documents our current thinking on the information that must be carried end-to-end, and notes which of those data elements may be supplied by the application using the bundle service. * Version Identifier: this is small integer that indicates which version of the bundle protocol is being invoked. * Destination entity ID: this is the identifier of the destination bundle application instance, and is a tuple, as described above. It is supplied by the local application using the bundle service. * Source entity ID: this is the identifier of the source bundle application instance, and is a tuple. It is supplied by the local bundle service, since a particular host may have multiple names and one may be chosen based on routing decisions or other criteria opaque to the application (as in multihomed hosts). The source entity ID may be returned to the application to support return receipt processing. * Reply-to entity ID (optional): a source may anticipate not being able to accept replies, and may use the reply-to entity ID to specify the destination for return receipts and delivery records. * Current custodian ID (optional): the bundle protocol supports store-and-forward operation in which the custody of a bundle (that is, the responsibility for ensuring reliable delivery) may transfer from one DTN node to another as the bundle progresses through the DTN. There is not a requirement for *each* DTN node encountered to assume custody of a bundle. As a result, it is necessary to identify the upstream node that currently has Cerf, et al. Expires February 2003 [Page 42] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 custody of the bundle, in order to acknowledge correct receipt of the bundle and to accept custody of the bundle. * Class of service flags: . an indication that custody transfer is requested, . an indication that a return receipt is requested, . an indication that a delivery record for custody transfers is requested, . an indication that a delivery record for all forwarding operations is requested, . an indication of the priority of the bundle (bulk, normal, or expedited), and . an indication of whether authentication information is present and its type. * Send timestamp: the time that a bundle was presented by the sending application to the bundle layer for transmission. * Time to live: an offset, in seconds, from the send timestamp that indicates when the bundle shall be purged from the DTN. * Authentication information (optional): access control information to prove that this bundle should be forwarded in the network. * User data: this is intended to be all of the data that the remote entity requires to perform whatever operation is requested. Since the environmental characteristics of a DTN tend to make interactivity difficult, the notion is that all of the information that is required to perform a particular "transaction" would be provided in a single bundle or unidirectional sequence of bundles. Some bundles cause a response to be generated by the bundle layer. Bundle layer responses are sent as bundles with the user data portion replaced by a Status Report, consisting of the following information: * Source entity ID of the subject bundle: this field partially identifies the bundle that is the subject of the Status Report. * Send timestamp of the subject bundle: this field completes the identification of the bundle that is the subject of the Status Report. * Status flags, indicating whether or not: . The subject bundle was received correctly by the source of the Status Report . The source of the Status Report has taken custody of the subject bundle . The source of the Status Report has forwarded the subject bundle . The Status Report contains the time at which the source of the Status Report received the subject bundle . The Status Report contains the time at which the source of the Status Report forwarded the subject bundle . Cerf, et al. Expires February 2003 [Page 43] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 * Time of Receipt (optional): the time at which the sender of the Status Report received the bundle. * Time of Forward (optional): the time at which the sender of the Status Report forwarded the bundle. 12 An Example Bundle Transfer We provide the following example of an end-to-end bundle transfer across a Delay-Tolerant Network. This particular example is based on the Interplanetary Internet: A host on Earth sends a bundle to a destination on Mars. Figure 2 illustrates the network that is the subject of this example ¡ the Interplanetary Internet in this example has five distinct regions interconnected by four DTN Gateways. This example highlights the actions taken by the bundle layer and the interactions of the bundle layer with applications and with underlying transport protocols. 12.1 Rules for forming tuples in the example network As noted in section 4.5, application instance ID tuples consist of two parts: a region ID, an entity ID. Section 5.2 requires that each DTN region have an entity identifier space shared by all DTN nodes within that region. Section 5.4 requires that each DTN region have a unique identifier that is known (or discernable) by all regions in the DTN. This particular delay-tolerant network is the Interplanetary Internet. In this section we will establish the naming rules that permit interoperation within this network. Note that this is only an example, and that not all DTNs must use this particular region identifier space (subject to the requirements of section 4.5). 12.1.1 Region identification All regions in *this* DTN (the example Interplanetary Internet) must share a region identifier space. Per section 4.5, the DTN region *name* space is hierarchical, dot-separated text, structured such that left to right is leaf-to-root in the tree. The *identifier* space may be exactly this name space, or a DTN may define a translation from the name to a particular identifier, in the same way that names in the DNS may be translated to Internet addresses. For this example, we will use the *names* as the *identifiers*. The DTN region name space is shown in Figure 1: Cerf, et al. Expires February 2003 [Page 44] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 Root . | The Interplanetary Internet int | The Solar System sol +-----+-----+-+---+-----+ | | | | | Region jupiter mars earth venus ipn Figure 1. An Example Interplanetary Internet Region Name Space 12.1.2 Intra-region identification For simplicity in this example, we will assume that all regions use a well-known TCP port in combination with DNS host names as the portion of their entity identifier that locates the application providing the bundle service. The bundle layer application resides above this well-known port (which we arbitrarily choose to be TCP port 6769, a currently unassigned port in the registered port space). The interplanetary backbone region, labeled "ipn" in Figure 1, will certainly NOT use TCP as its underlying transport protocol. For the sake of simplicity in this example, let us assume that the protocol used within the interplanetary backbone region uses the same entity identifier space (IP addresses and port numbers) that the remainder of the network uses. The mechanism used to demultiplex bundles received by a bundle node is entity-specific, and for the sake of simplicity, we will assume that this is a process ID that has been incorporated into the entity ID. [Note: this is a simple, but quite limiting decision, as it has implications on process reinstantiation and process portability/migration from one entity to the next. We choose it only for simplicity.] 12.2 Example Network Topology at the Region Level It is important to have some understanding of the routing that is required at the DTN Gateways, so it is important to understand the topology of the network at the region level. Unlike typical terrestrial communications, the Interplanetary Internet's (IPN's) *long-haul* communication links are directional, mobile, and highly scheduled. This is important, because directionality combined with mobility means that a transmitter and receiver must track each other in order to establish and maintain a communication link. In the IPN, much of the mobility is due to orbital mechanics and is therefore relatively predictable. However, this means that nodes that we would normally consider to be fixed, such as antennas on the surface of the Earth, are actually highly mobile as a result of the Earth's rotation and its revolution around the Sun. (In this example, we confine Cerf, et al. Expires February 2003 [Page 45] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 ourselves to our local solar system, and do not consider the motion of our sun relative to celestial bodies outside our solar system.) We can describe the predictable aspects of node mobility with an ephemeris, a table of the positions of celestial bodies at specified intervals of time. Both a directional sender and receiver must each know the other's position in order to establish a link between the pair. +-------------------------+ | Earth's Internet | | DTN Region: earth.sol | | +---+ | +-----------/ /|--------+ +---+ | |G/W| + +----------| 1 |/---------+ | +---+ | | The "Backbone" | | DTN Region: ipn.sol | +---+ +---+ +---+ / /|------/ /|-------/ /| +---+ | +---+ | +---+ | |G/W| + |G/W| + |G/W| + +----------------| 3 |/ +---| 4 |/-----+ | 2 |/-------------------+ | +---+ | +---+ | +---+ | | Venus's Internet | | Jupiter's | | Mars's Internet | | DTN Region | | Internet | | DTN region: | | venus.sol | | DTN Region | | mars.sol | +------------------+ | jupiter.sol | +----------------------+ +--------------+ Figure 2. An Interplanetary Internet of Five IPN Regions In addition, these communication resources are highly scheduled. It is not sufficient for a receiver to point at a prospective target and just wait ¡ for example, a terrestrial node will typically have to point at several targets sequentially, and an interplanetary node will typically not have enough power to just wait for incoming messages. Rather, a schedule of communication *opportunities* must be calculated and then refined with planned communication *contacts*. A communication opportunity establishes that the endpoints could establish a link if they were pointing at each other at the proper times. We refer to a planned communication contact as an agreement (although not irrevocable) between the two parties to establish contact and communicate for a defined period of time. Cerf, et al. Expires February 2003 [Page 46] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 The protocols for establishing the schedule of communication contacts between all pairs of possible communicants will evolve -- from something primarily done manually to something more automated as the real Interplanetary Internet grows. The scheduled nature of connectivity in the Interplanetary Internet, particularly across the deep-space links, means that at the time of a bundle's arrival at a DTN Gateway, some or all of the possible outbound routes may be "down." The gateway must store the bundle until the appropriate link is available and then transmit the bundle over that link. One of the fundamental differences between the Interplanetary Internet and the terrestrial Internet is this inherent use of store-and-forward mechanisms in routing bundles. With that in mind, let us consider the routing decisions made at some of the DTN Gateways in Figure 3. 12.3 DTN Gateway routing Bundle routing at a DTN gateway will typically have to deal with a mix of continuously available links and intermittently available links. Routing across a continuously available link is a relatively straightforward activity. Routing in the presence of intermittently available links can be significantly more complex, as the gateway will need to decide not only the next hop destination but also the time at which to send the bundle. Conditions elsewhere in the network may make it more desirable to send a bundle to a next-hop destination that is not yet accessible, even when an alternative route is currently available. This decision process is discussed in section 6.3. The intermittent links in this example provide connectivity among the DTN Gateways that are part of the ipn.sol.int region. The contacts among gateways in this region are scheduled. As is currently the case, Gateway 1 (the Earth gateway) has scheduled contacts with each of the other DTN gateways in the ipn.sol.int region (the "backbone"), but there is no direct contact among any of the DTN gateways on Venus, Jupiter, or Mars. Recall that this communication uses directional antennas, so it is generally not possible to communicate with two different entities at once unless they are in the same field of view. Power availability on the remote planets is an issue, so transmitters and receivers are turned on only during a contact. Further, the speed of light delays are nontrivial, skewing transmit and receive times between remote gateways. In Table 1, a schedule of contacts is presented that will be used in the example. All times are referenced to Gateway 1. The information in this table is completely hypothetical, and the speed of light delays are plausible, but completely back-of-the-envelope. One-way light times between Gateway 1 and Gateways 3, 4, and 2 are 4 minutes, 32 minutes, and 10 Cerf, et al. Expires February 2003 [Page 47] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 minutes, respectively. Those details are perhaps interesting but not the point of the example. Note that any bundles sent by GW3 after 1156 GW1 time cannot be acknowledged before the next contact, because the bundle will arrive at GW1 after the end of GW1's transmission time (1200). Since the next contact between GW1 and GW3 might be the subsequent Monday, the acknowledgement delay might be VERY long. Bundles sent by GW4 after 1358 cannot be acknowledged during the current contact, because they will not be received before GW1's transmission time ends at 1430. Similarly, bundles sent by GW2 after 1150 cannot be acknowledged during the contact. Table 1. Contact schedule for Gateway 1. +---------+------------+-----------+-------------+------------------+ | Day |Destination | GW1 Send | GW1 Receive | Destination Send | | | | Time | Time | /Receive Time | +---------+------------+-----------+-------------+------------------+ | Monday | GW3 | 0900-1200 | 0908-1208 | 0904-1204 | +---------+------------+-----------+-------------+------------------+ | Tuesday | GW4 | 1300-1430 | 1404-1534 | 1332-1502 | +---------+------------+-----------+-------------+------------------+ |Wednesday| GW2 | 1000-1200 | 1020-1220 | 1010-1210 | +---------+------------+-----------+-------------+------------------+ DTN Gateways 2, 3, and 4 have entries in their contact schedules for the contacts with Gateway 1. 12.4 Systems participating in example bundle data transfer Figure 3 is a revision of Figure 2 that highlights those portions of the Interplanetary Internet that participate directly in the bundle transfer example. Also shown in Figure 3 are the source and destination for the transfer and the Domain Name System equivalents in the terrestrial Internet (DNS 1), in the IPN "Backbone" (DNS 2), and in the Mars Internet (DNS 3). This figure will serve as the basis for the bundle data transfer example. Table 2 provides the full host names of each of the primary elements in Figure 4. Recall that in this example all bundle layer Cerf, et al. Expires February 2003 [Page 48] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 +---+ / /| +------------------------+---+ | +---+ Earth's Internet |DNS| + / /|IPN Region: earth.sol| 1 |/ +---+ | +---+ +---+ |SRC| +---------/ /|--------+ | |/ +---+ | +---+ |G/W| + +----------| 1 |/---------+ | +---+ | | The "Backbone" | | IPN Region: ipn.sol | +---+ +---+ +---+ / /|-------------------/ /| / /| +---+ | +---+ | +---+ | |DNS| + |G/W| + |DST| + | 2 |/ | 2 |/-------------| |/+ +---+ +---+ +---+ | | Mars's Internet | +---+ IPN region: | / /| mars.sol | +---+ |---------------------+ |DNS| + | 3 |/ +---+ Figure 1. Interplanetary Internet showing principal systems. Table 1. Host name tuples (entity ID demultiplexing info omitted). +------------+------------------+---------------------------+ | Host | IPN Regions | Host name tuples | +------------+------------------+---------------------------+ | SRC | earth.sol |{src.jpl.nasa.gov:6769, | | | | earth.sol} | +------------+------------------+---------------------------+ | IPN G/W1 | earth.sol |{ipngw1.jpl.nasa.gov:6769, | | | | earth.sol} | | | ipn.sol |{ipngw1.jpl.nasa.gov:6769, | | | | ipn.sol} | +------------+------------------+---------------------------+ | IPN G/W2 | ipn.sol |{ipngw2.nasa.mars.org:6769,| | | | ipn.sol} | | | mars.sol |{ipngw2.nasa.mars.org:6769,| | | | mars.sol} | +------------+------------------+---------------------------+ | DST | mars.sol |{dst.jpl.nasa.gov:6769, | | | | mars.sol} | +------------+------------------+---------------------------+ Cerf, et al. Expires February 2003 [Page 49] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 applications reside on TCP port 6769. The portion of the entity identifier used to demultiplex to the application using the bundle service has been omitted until the applications are discussed in section 12.5.1. 12.5 End-to-end Transfer 12.5.1 Step 0: Application instance registration Before the beginning of this example, all of the bundle nodes in the network exchange their signed key material and credentials with next hop neighbors. These materials are cached for subsequent use. The applications at the source and destination register with their local bundle layers, providing similar key material and credentials to the bundle layer, and receiving in return their application instance identifiers. The destination has registered its IPN-standardized well-known demultiplexing id of "8," which becomes part of the entity ID used to refer to this particular application. The source has registered a demultiplexing identifier "0x1763421A" (a hexadecimal number) that (arbitrarily) corresponds to its process ID. 12.5.2 Step 1: Bundle creation and first-hop transmission The application on the source host in Figure 3 has data that it wishes to send to the destination on Mars. The exact content of this data is opaque to the bundle transfer, but assume that it contains all of the information necessary to accomplish some desired function. That is, assume that application-specific instruction for storage, handling, error processing, and disposal accompanies whatever data object is to be operated upon. The application invokes the bundle layer, supplying it the information shown in Table 2. The bundle agent verifies the signature, then creates a bundle and stores it in persistent storage (on disk or other non-volatile memory). There are some fields of the bundle header that the bundle agent must supply: the Sending Timestamp, the Source Host name tuple, the Custodian name tuple, and the time to live. (The application may state a time after which the data are obsolete, but the actual time- to-live field in the bundle header uses the application's data in combination with network restrictions on time-to-live to initialize this field properly.) The bundle layer consults routing tables notes that its next-hop destination to reach the mars.ipn.sol region within the earth.ipn.sol region is ipngw1.jpl.nasa.gov:6769. (Since a host may reside in multiple IPN Regions, the source host name tuple is a function of the outbound route selected. The bundle layer uses this information to complete the Source Host and Custodian name tuples prior to transmission.) Having verified the application's signature, it incorporates this into the authentication information of the Cerf, et al. Expires February 2003 [Page 50] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 bundle header and appends its own credentials, as described in section 7.2. It further notes that it has a continuous connection to that gateway, and transmits the bundle to it, retaining a copy for custodial storage. The bundle layer starts a retransmission timer for the bundle and awaits a custodial transfer acknowledgment. Table 2. Information supplied by source application. +-------------+---------------------+------------------------------+ | Item | Value | Description | +-------------+---------------------+------------------------------+ | Destination | {mars.sol, | IPN Name tuple of the | | DTN | dst.jpl.nasa.gov, | destination. Note that appl.| | tuple | 8} | demuxing ID 8 is supplied. | +-------------+---------------------+------------------------------+ | Source | 0x1763421A | Value used to identify the | | application | | appropriate instance of the | | demultiplex | | source application for | | identifier | | response processing (becomes | | | | part of the source entity ID)| +-------------+---------------------+------------------------------+ | Class of | Custodial delivery, | The services requested from | | service | normal priority, | the bundle layer. | | | data obsolete in 36 | | | | hours. | | +-------------+---------------------+------------------------------+ | Signature | Opaque | Digital signature of the | | | | app.-supplied information | +-------------+---------------------+------------------------------+ | User data | N/A | | +-------------+---------------------+------------------------------+ 12.5.3 Step 2: Bundle processing at first-hop destination When the IPN Gateway {ipngw1.jpl.nasa.gov, earth.sol} receives the bundle via TCP, it checks the signature of the previous router and determines that the bundle has been forwarded by a legitimate source. Further, since this is the security boundary for the Interplanetary Internet, it verifies the signature of the sending application, requesting a copy of the application's public key from a secure distribution service if it does not already have one cached. Having verified that the application is authorized to access the deep-space portion of the Interplanetary Internet, the bundle layer replaces the signature block of the bundle layer entity with its own, leaving the application's signature block intact. It stores the received bundle on persistent storage (disk). The bundle layer consults the contact schedule and determines that the appropriate next-hop destination is in the "ipn.sol" IPN Region: {ipngw2.nasa.mars.org, ipn.sol}, and Cerf, et al. Expires February 2003 [Page 51] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 that the next contact is the following Wednesday. The bundle layer manifests the bundle on the contact for that Wednesday. In considering alternative contacts for the bundle, the dispatcher checks the time-to-live in the bundle, which was 36 hours from the time of initial submission to the bundle agent at the source, to ensure that the route selected is consistent with the time-to-live requirements of the bundle. The bundle transport functionality of the bundle agent in ipngw1 accepts custody of the bundle, updates this information in the bundle header, and informs the source that has done so. The sources bundle agent deletes its custodial copy of the bundle. When the time arrives for contact with the ipngw2.nasa.mars.org DTN gateway to be established, the convergence layer establishes that contact via the Long Haul Transport Protocol (LTP). When informed that the contact is available, the bundle layer posts its bundles to the convergence layer for transmission via LTP. 12.5.4 Step 3: Bundle processing at gateway to destination IPN Region The Mars gateway, {ipngw2.nasa.mars.org, mars.sol}, receives the bundle from the Earth gateway via LTP. It verifies the signature block of the Earth gateway, then replaces that signature block with its own. It stores the bundle in persistent storage and accepts custody of the bundle, signaling back to the Earth gateway that it has done so. When the notification of acceptance reaches the Earth gateway, ipngw1 deletes its custodial copy. The Mars gateway consults its routing tables to find an outbound contact to forward the bundle over. It determines that the appropriate next hop is the destination itself, that the proper transport protocol is TCP, and that the destination is accessible immediately. The gateway verifies that the time-to-live has not expired, and forwards the bundle to the destination. 12.5.5 Step 4: Bundle processing at destination The bundle layer at the destination entity receives the bundle via TCP, verifies the signature block of the IPN G/W2, stores it on its own persistent storage, and accepts custody of the bundle from IPN G/W2. The bundle agent "awakens" the destination application process identified by the Destination Application demultiplexing portion of the entity ID. This may involve creating a new instance of a server from a daemon process, signaling an idle running process, or reinstantiating a process that has been suspended with its state stored on persistent storage. The bundle agent deletes the copy of the bundle from persistent storage when the application has received it. The destination application may generate an application-layer acknowledgment in a new bundle and send it to the source. Cerf, et al. Expires February 2003 [Page 52] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 12.6 Error Conditions at the Bundle Layer This section describes the error conditions that may arise at the bundle layer during bundle creation and transport. When these errors occur within the sender's DTN region, it may be possible to conduct a near-real-time dialog to correct them before the bundle is forwarded. We say 'may be possible' because even if two nodes are in the same IPN domain, they may not have real-time connectivity. An example of such a situation would be if a lander were on the opposite side of the planet from its DTN gateway, and used bundles to communicate with the gateway through a low altitude orbiter, with the orbiter itself serving as a bundle agent. Table 3: Error conditions at the bundle layer. +-------+---------------------------+------------------------------+ | Error | Description | Places where Error can Occur | +-------+---------------------------+------------------------------+ | 1* | Unknown destination region| Source Bundle Node | +-------+---------------------------+------------------------------+ | 2* | Invalid Source App. | Source Bundle Node | +-------+---------------------------+------------------------------+ | 3* | Bundle Parameter Syntax | Source Bundle Node | | | Error | | +-------+---------------------------+------------------------------+ | 4* | Bundle Parameter Semantic | Source Bundle Node | | | Error | | +-------+---------------------------+------------------------------+ | 5 | Insufficient buffer space | Any bundle node | +-------+---------------------------+------------------------------+ | 6 | DNS unreachable | Any bundle node | +-------+---------------------------+------------------------------+ | 7* | Time exceeded | Any bundle node other than | | | | the source | +-------+---------------------------+------------------------------+ | 8* | Source Entity Access | Any bundle node other than | | | Denied | the source | +-------+---------------------------+------------------------------+ | 9 * | Invalid Entity ID in | IPN gateway serving | | | Destination Name | destination DTN region | +-------+---------------------------+------------------------------+ | 11* | Invalid Destination App. | Destination | +-------+---------------------------+------------------------------+ | 12* | End-to-end Access Denied | Destination | +-------+---------------------------+------------------------------+ The errors that can occur at the bundle layer are shown in Table 3. Error numbers marked with an asterisk (*) are reported back to the sending application. Cerf, et al. Expires February 2003 [Page 53] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 * Unknown Destination Region: This error occurs when the source bundle node is directed to create a bundle destined for an DTN Region that is not recognized. Note that only the DTN Region part of the destination name has to be interpretable outside the destination's DTN Region. In particular, the entity identifier of the destination name need not be interpretable to the source (assuming the source and destination are in different DTN Regions), so it cannot necessarily be checked when the bundle is created. * Invalid Source Application: If the source authentication information fails, the source bundle layer responds with an Invalid Source Application error. * Bundle Parameter Syntax Error: The source bundle layer may check the syntax of some of the bundle-creation parameters (i.e. it may ensure that the end-to-end and DTN access security certificates are well-formed, etc.) If a parameter is found to be syntactically incorrect or obviously and definitely erroneous, the bundle layer will report a Bundle Parameter Syntax Error back to the source that includes, at a minimum, the parameter that caused the error. * Bundle Parameter Semantic Error: If the source bundle layer can identify a particular bundle creation parameter as being well- formed but unserviceable, it will report a Bundle Parameter Semantic Error to the source application that includes, at a minimum, the parameter that caused the error. * Insufficient Buffer Space: If a bundle node does not have sufficient buffer space to accept a bundle, it drops the bundle and generates an Insufficient Buffer Space error. Note that a bundle node may choose to drop lower priority bundles in order to make room for higher priority ones. This error is not propagated back to the source. * DNS Unreachable: If a bundle node needs access to its DNS (or DNS- equivalent) and cannot obtain information from it, it generates a DNS Unreachable error. This information is not propagated back to the source application. * Time Exceeded: If the time-to-live field (either the source- provided TTL or the internal bundle TTL) expires, the source is notified with a Time Exceeded message. These errors are propagated to the source application. * Source Entity Access Denied: This error indicates that the source entity does not have access to a needed resource at a DTN node. The source might not be authorized to use the node at all, or it might not have authorization to use a particular interface required by the bundle. Source Entity Access Denied errors indicate that the source is not *authorized* to use a particular resource; other errors (e.g. Insufficient Buffer Space) indicate that a particular resource is *unavailable* to a bundle. For example, an entity on the surface of a planet might be authorized to communicate, using the bundle protocol, with another entity on Cerf, et al. Expires February 2003 [Page 54] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 the other side of the planet via a low-altitude orbiter that is also an IPN gateway. The sender might not, however, be authorized to send bundles across interplanetary space. In this case bundles sent to the orbiter destined for the other side of the planet would not cause errors, while any bundles with off-planet destination addresses would. Source Entity Access Denied errors are propagated back to the source application. * Invalid Entity ID in Destination Name: Once a bundle has reached its destination DTN Region, the Entity ID part of the destination name can be parsed. If the Entity ID of the destination name is not valid, the source is notified with an Invalid Administrative Destination Name error message. * Invalid Destination Application: If the destination bundle layer cannot instantiate the destination application (based on the demultiplexing information the region-specific entity ID of the bundle), it notifies the source application with an Invalid Destination Application error message. * End-to-End Access Denied: If the bundle destination does not accept the bundle due to an authentication or access-control error, the source is notified with an End-to-End Access Denied Message. 13 Summary This architecture addresses many of the problems of networks that must operate in extreme environments. It is message based, and uses as a model of service classes those offered by the postal mail system. It accommodates many different forms of connectivity, including scheduled, predicted, and opportunistically connected links. It introduces a novel approach to end-to-end reliability across frequently partitioned and unreliable networks. It also proposes a scheme for securing the network infrastructure against unauthorized access. It is our belief that this architecture is applicable to many different types of extreme environments. In subsequent documents, we intend to specify profiles of this architecture that address specific environments in detail. We plan to develop profiles of this architecture for Interplanetary Internetworking (the genesis of the architecture), for Sensor Internetworking, for Military Tactical Internetworking, and for "Snowmobile" Internetworking. In addition to these profiles, our upcoming tasks include precise specification of the constituent protocols in concert with their prototyping and testing. These activities are currently under way. Cerf, et al. Expires February 2003 [Page 55] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 14 References [KP97] K. Park, G. Kim, and M. Crovella, "On the Effect of Traffic Self-Similarity on Network Performance," Proceedings of the 1997 SPIE International Conference on Performance and Control of Network Systems. [BLMR98] J. Byers, M. Luby, M. Mitzenmacher, A. Rege, "A Digital Fountain Approach to Reliable Distribution of Bulk Data", SIGCOMM 1998 [ARINC] Presentation by ARINC, see page at http://www.arinc.com [DMT96] R. Durst, G. Miller, E. Travis, "TCP Extensions for Space Communications", Wireless Networks 3(5):389-403, 1997. [MLS00] U. Madhow, T. V. Lakshman, B. Suter, "TCP/IP Performance with Random Loss and Bidirectional Congestion", IEEE/ACM Transactions on Networking, 8(5), Oct 2000. [PILC-ASYM] H. Balakrishnan, V. Padmanabhan, G. Fairhurst, M. Sooriyabandara, "TCP Performance Implications of Network Path Asymmetry", IETF draft-ietf-pilc-asym-07, (Work in Progress), Nov 2001. [M95] J. Mogul, "Fragmentation Considered Harmful", SIGCOMM 1995 [IGE00] C. Intanagonwiwat, R. Govindan, D. Estrin, "Directed Diffusion: A scalable and robust communication paradigm for sensor networks", MobiCOM 2000, Aug 2000 [NEWARCH] NewArch Project: Future-Generation Internet Architecture. http://www.isi.edu/newarch [META] Wroclawski, John T., "The Metanet", Workshop on Research Directions for the Next Generation Internet, May 12-14, 1997, Vienna, VA. http://www.cra.org/Policy/NGI/papers/wroklawWP. [CK74] V. Cerf, R. Kahn, "A Protocol for Packet Network Intercommunication", IEEE Trans. on Comm., COM-22(5), May 1974 [DM02] D. Mills, "Timekeeping in the Interplanetary Internet", July 2002, http://www.eecis.udel.edu/~mills/database/brief/ipin/ipin.pdf [WhynotTCP] "Why not use the Standard Internet Suite for the Interplanetary Internet?" http://www.ipnsig.org/reports/TCP_IP.pdf [RJ90] R. Jain, "Congestion Control in Computer Networks: Issues and Trends," IEEE Network Magazine, May 1990. Cerf, et al. Expires February 2003 [Page 56] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 15 Security Considerations Security is an integral concern of the Delay Tolerant Network Architecture. Section 7 of this document, also titled "Security considerations" is devoted to examining the issues involved in securing the DTN architecture and providing basic security services to DTN applications. Cerf, et al. Expires February 2003 [Page 57] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 16 Authors' Addresses Dr. Vinton G. Cerf MCI WorldCom 22001 Loudoun County Parkway Building F2, Room 4115, ATTN: Vint Cerf Ashburn, VA 20147 Telephone +1 (703) 886-1690 FAX +1 (703) 886-0047 Email vcerf@mci.net Scott C. Burleigh Jet Propulsion Laboratory 4800 Oak Grove Drive M/S: 179-206 Pasadena, CA 91109-8099 Telephone +1 (818) 393-3353 FAX +1 (818) 354-1075 Email Scott.Burleigh@jpl.nasa.gov Robert C. Durst The MITRE Corporation 1820 Dolley Madison Blvd. M/S W650 McLean, VA 22102 Telephone +1 (703) 883-7535 FAX +1 (703) 883-7142 Email durst@mitre.org Dr. Kevin Fall Intel Research, Berkeley 2150 Shattuck Ave., #1300 Berkeley, CA 94704 Telephone +1 (510) 495-3014 FAX +1 (510) 495-3049 Email kfall@intel-research.net Adrian J. Hooke Jet Propulsion Laboratory 4800 Oak Grove Drive M/S: 303-400 Pasadena, CA 91109-8099 Telephone +1 (818) 354-3063 FAX +1 (818) 393-3575 Email Adrian.Hooke@jpl.nasa.gov Dr. Keith L. Scott The MITRE Corporation 1820 Dolley Madison Blvd. Cerf, et al. Expires February 2003 [Page 58] Internet Draft draft-irtf-ipnrg-arch-01.txt August 2002 M/S W650 McLean, VA 22102 Telephone +1 (703) 883-6547 FAX +1 (703) 883-7142 Email kscott@mitre.org Leigh Torgerson Jet Propulsion Laboratory 4800 Oak Grove Drive M/S: T1710- Pasadena, CA 91109-8099 Telephone +1 (818) 393-0695 FAX +1 (818) 354-9068 Email Leigh.Torgerson@jpl.nasa.gov Howard S. Weiss SPARTA, Inc. 9861 Broken Land Parkway Columbia, MD 21046 Telephone +1 (410) 381-9400 x201 FAX +1 (410) 381-5559 Email hsw@sparta.com Please refer comments to ipn-team@mailman.ipnrg.org