Sorry for the long email.  I hope it is worth your time.

Disclaimer:  The author does not represent the NASA Space Operations Mission Directorate (SOMD) nor does the author claim to be the NASA DTN representative to DTNRG.


NASA Glenn has been working advance Space-Based Sensoweb technology with a goal of complete automation of sensor trigger to data deliver.  The concepts are available at the following URL (note, 3 Mbyte file, play with animation):
http://roland.grc.nasa.gov/~ivancic/papers_presentations/2007/IEEE2007T6P1104.ppt

We use DTN to break control loops from space-ground and ground-ground as well as to enable large files to proactively fragmented and received at multiple ground stations.  For example, I may need 15 minutes to downlink a file and only have 8 minute contact times at each ground station.

On August 27 and 28 of 2008 we performed proactive fragmentation using a single ground station (two passes).
http://maillists.intel-research.net/pipermail/dtn-interest/2008-September/003272.html
Since that time we have been working to establish infrastructure to perform the same tests over multiple ground stations.  We recently completed the infrastructure and now have ground stations at Guildford England (Surrey Satellite Technology Limited) and Alaska, Hawaii and Australia (Universal Space Networks).  Bundle Agents are at each of these ground stations.

EXECUTIVE SUMMARY

On September 30 and October 1 we successfully demonstrated DTN proactive fragmentation and, by accident, reactive fragmentation using two different ground stations.  The packets were reassembled at a bundle agent at Glenn Research Center in Cleveland Ohio.  The following details the experiment.



  +---------------------------------------------------------+
  |                                                         |
  |                 CREATED ONBOARD UK-DMC                  |
  |                                                         |
  |                    +---------------+                    |
  |                    |               |                    |
  |                    |  Full Bundle  |                    |
  |                    |     157 MB    |                    |
  |                    |               |                    |
  |                    +-----.-'-.-----+                    |
  |                       .-'     `-.                       |
  |    +----------------.'-----------`-.---------------+    |
  |    |             .-'                `-.            |    |
  |    |  +-------.-'-----+         +------`-.------+  |    |
  |    |  |  Proactive    |         |  Proactive    |  |    |
  |    |  |  Fragment #1  |         |  Fragment #2  |  |    |
  |    |  |     80 MB     |         |     77 MB     |  |    |
  |    |  |               |         |               |  |    |
  |    |  +---------------+         +---------------+  |    |
  |    |  Received at Alaska        Received At Hawaii |    |
  |    |          /                       ,`           |    |
  |    +---------+----------------------,'.'+----------+    |
  |             /                     ,'  |  \              |
  |            /                    ,'   .'   \             |
  +-----------+-------------------,'-----+-----`.-----------+
             /                  ,'      .'      `.
            /                 ,'        |        '.
  +--------,'---------------,'----------+----------+---------+
  |       ,'              ,'           |            \        |
  | +-----'----+   +-----'----+  +-----'----+  +----------+  |
  | | Proactive|   | Reactive |  | Reactive |  | Reactive |  |
  | | Fragment |   | Fragment |  | Fragment |  | Fragment |  |
  | |   #1     |   |   #1     |  |   #2     |  |   #3     |  |
  | |  80 MB   |   |  33 MB   |  |  43 MB   |  |   1 MB   |  |
  | +-----.=---+   +-----+----+  +-----+----+  +--=.------+  |
  |         `--._         \          .'       _.-'           |
  |              `-.._     \        /     _.-'               |
  |                   `-._  `.    .'  _.-'                   |
  |                       ``-.\  /_.-'                       |
  |                     +------.-'------+                    |
  |                     |               |                    |
  |                     |  Full Bundle  |                    |
  |                     |     157 MB    |                    |
  |                     |               |                    |
  |                     +---------------+                    |
  |                                                          |
  |                   RECEIVED AT NASA GRC                   |
  |                                                          |
  +----------------------------------------------------------+


END EXECUTIVE SUMMARY

TEST DETAILS

These tests were performed using the United Kingdom Disaster Monitoring Constellation (UK-DMC) satellite and USN ground stations in Alaska and Hawaii.  The first pass was over Alaska and the second pass about 73 minutes later.  The Solid State Data Recorder (SSDR) had to remain powered on during eclipse (dark) but all transmission were during light to put as little train on the satellite power system as possible.  The SSDR hold the operating system.
These experiments were done with standard commercial off the shelf (COTS) hardware and protocols. The space/ground link is a serial link and uses HDLC and Frame Relay.  Uplink is 9.6 kbps.  Downlink is 8 Mbps. The transport protocol is Saratoga version 0(SSTL's negative acknowledgement protocol) over UDP/IP. Note, we are working to standardize Saratoga (draft-wood-tsvwg-saratoga-03, draft-wood-dtnrg-saratoga-05). The ground-to-ground convergence layer was TCP.

We used images of 150 Mbytes with proactive fragmentation of bundles at 80 Mbyte.  150 Mbyte files enables us to prove the proactive fragmentation but to provide time during a pass to also download the entire file using non-DTN and to allow us some time for recovering from human error.  SSTL image files are often  a Gbyte on the UKDMC and larger on their newer satellites.  UKDMC is over 5 years old and near end-of-life.

30th September
150 Mbyte Image capture at 17:00:21 UTC

MD5 Command at 17:02:02 UTC
(We put in an optional MD5 command as it can take some time to run and MD5 check on a Gigabyte file and SSTL does not require CRC checks in their normal operations.)

Downlink 1 - 17:11:00 to 17:24:00 (Full downlink duration scheduled, eclipse starts at 17:30 UTC)
Downlink 2 - 18:37:30 to 18:50:30

Test Plan for 30 Sept 2009
==========================

Order of tests Pass 1 over Alaska:
===================================
1) Download Syslog (Check MD5)
2) Download Frag 1 (DTN proactive Fragmentation)
3) Download File using GRC Saratoga
4) Download Syslog again


Check that frag 1 was transferred to Bundle Master Check that frag 1 was received by Bundle Master

Order of tests Pass 2 over Hawaii:
===================================
1) Download Syslog
2) Download Frag 2 (DTN proactive Fragmentation)
3) Download File using GRC Saratoga
4) Download Syslog again

Check that frag 2 was transferred to Bundle Master Check that frag 2 was received by Bundle Master Check MD5 of combined file to confirm accurate combining.

Note, the Syslog file gets appended to, so the last download of the Syslog file, step 4, in Hawaii contains all the information in the Syslog of the previous downloads and more.


We successfully downloaded the Syslog, 80 Mbyte fragment (fragment#1) and the entire non-bundle 150 Mbyte file over Alaska. We also successfully downloaded the Syslog file, the 70 Mbyte fragment (fragment#2) and the entire 150 Mbyte non-bundle file over Hawaii.

Now comes the interesting part.

Last year we had a default route in the DTN2 implementation at the ground station in Guildford, England.  We had removed that and the route we and in for both Alaska and Hawaii was:

link add link_grc1 bundling1:4556 ONDEMAND tcp
route add dtn://bundling-grc1/* link_grc1

But, in the Bundle creation onboard the UK-DMC we create and EID of:
dtn:bundling-grc1.  Note, no back slashes. dtn://bundling-grc1!

With the default route last year, this got forwarded.  So, we now have bundle fragments at Alaska and Hawaii, but no routes.  We are stored, but no forward yet as we have no route to destination, but the bundles have not expired.

In Alaska:
Currently Pending Bundles (1):
        11503: dtn:uk-dmc/i -> dtn:bundling-grc1/i length 80000000

In Hawaii:
Currently Pending Bundles (1):
        26558: dtn:uk-dmc/i -> dtn:bundling-grc1/i length 77260545


So, we added this route at both sites:

route add dtn:bundling-grc1/* link_grc1

We add the route in Alaska first and forwarding of proactive bundle fragment #1 begins. We see the fragment #1 at GRC "bundle master".

We receive the first bundle fragment from Alaska

bundling-grc dtn% bundle list

bundling-grc dtn% bundle list
bundling-grc dtn% bundle list
Currently Pending Bundles (4):

154398: dtn://bundling-ak/ -> dtn://bundling-grc1/ length 13601
154399: dtn://bundling-ak/ -> dtn://bundling-grc1/ length 13601
154400: dtn:uk-dmc/i -> dtn:bundling-grc1/i length 80000000
154401: dtn:bundling-AK/ -> dtn:bundling-grc1/ length 13601

Examining the information from source uk-dmc show full proactive bundle fragment was received.

dtn% bundle info 154402
bundle id 154402:

source: dtn:uk-dmc/i
dest: dtn:bundling-grc1/i
custodian: dtn:none
replyto: dtn:none
prevhop:
payload_length: 33128366
priority: 0
custody_requested: false
local_custody: false
singleton_dest: false
receive_rcpt: false
custody_rcpt: false
forward_rcpt: false
delivery_rcpt: false
deletion_rcpt: false
app_acked_rcpt: false
creation_ts: 307602048.0
expiration: 604800
is_fragment: true
is_admin: false
do_not_fragment: true
orig_length: 157260545
frag_offset: 80000000
transmission_count: 0


Now we get back into Hawaii Bundle agent.


localhost dtn% route dump bundle list
Currently Pending Bundles (1):

        26558: dtn:uk-dmc/i -> dtn:bundling-grc1/i length 77260545


We then added the route to the Hawaii bundling agent.

localhost dtn%  route add dtn:bundling-grc1/* link_grc1

And forwarding begins.

Another interesting turn of events.

While watching tcpdumps at the GRC bundle agent, bundlemaster, we notice the tcp connection slow, stop and restart.  This happens three times and we get three REACTIVE fragments of our 70 Mbyte proactive fragment!

>From the Hawaii putty log file:

localhost dtn% bundle list
Currently Pending Bundles (1):
        26558: dtn:uk-dmc/i -> dtn:bundling-grc1/i length 77260545

localhost dtn% [1254339449.785320 /dtn/bundle/daemon warning] event BUNDLE_RECEIVED took 2338 ms to process

localhost dtn% bundle list
Currently Pending Bundles (1):
        26559: dtn:uk-dmc/i -> dtn:bundling-grc1/i length 44136275

localhost dtn% bundle info
wrong number of arguments to 'bundle info' expected 3, got 2

while evaluating {bundle info}

localhost dtn% bundle info list
Currently Pending Bundles (1):

        26559: dtn:uk-dmc/i -> dtn:bundling-grc1/i length 44136275

localhost dtn% bundle list info 26559

bundle id 26559:
source: dtn:uk-dmc/i
dest: dtn:bundling-grc1/i
custodian: dtn:none
replyto: dtn:none
prevhop:
payload_length: 44136275
priority: 0
custody_requested: false
local_custody: false
singleton_dest: false
receive_rcpt: false
custody_rcpt: false
forward_rcpt: false
delivery_rcpt: false
deletion_rcpt: false
app_acked_rcpt: false
creation_ts: 307602048.0
expiration: 604800
is_fragment: true
is_admin: false
do_not_fragment: false
orig_length: 157260545
frag_offset: 113124270
transmission_count: 0



forwarding log:

        IN_FLIGHT -> tcp 1254339452.607140 [FORWARD cl:bundling1:4556] [custody min 1800 pct 25 max 0]


localhost dtn%
localhost dtn% bundle info 26559

bundle id 26559:
source: dtn:uk-dmc/i
dest: dtn:bundling-grc1/i
custodian: dtn:none
replyto: dtn:none
prevhop:
payload_length: 44136275
priority: 0
custody_requested: false
local_custody: false
singleton_dest: false
receive_rcpt: false
custody_rcpt: false
forward_rcpt: false
delivery_rcpt: false
deletion_rcpt: false
app_acked_rcpt: false
creation_ts: 307602048.0
expiration: 604800
is_fragment: true
is_admin: false
do_not_fragment: false
orig_length: 157260545
frag_offset: 113124270
transmission_count: 0
forwarding log:

        IN_FLIGHT -> tcp 1254339452.607140 [FORWARD cl:bundling1:4556] [custody min 1800 pct 25 max 0]

localhost dtn% bundle info 26559

bundle id 26559:
source: dtn:uk-dmc/i
dest: dtn:bundling-grc1/i
custodian: dtn:none
replyto: dtn:none
prevhop:
payload_length: 44136275
priority: 0
custody_requested: false
local_custody: false
singleton_dest: false
receive_rcpt: false
custody_rcpt: false
forward_rcpt: false
delivery_rcpt: false
deletion_rcpt: false
app_acked_rcpt: false
creation_ts: 307602048.0
expiration: 604800
is_fragment: true
is_admin: false
do_not_fragment: false
orig_length: 157260545
frag_offset: 113124270
transmission_count: 0



forwarding log:

        IN_FLIGHT -> tcp 1254339452.607140 [FORWARD cl:bundling1:4556] [custody min 1800 pct 25 max 0]

Back on the GRC BundleMaster"

Currently Pending Bundles (7):

        154398: dtn://bundling-ak/ -> dtn://bundling-grc1/ length 13601
        154399: dtn://bundling-ak/ -> dtn://bundling-grc1/ length 13601
        154400: dtn:uk-dmc/i -> dtn:bundling-grc1/i length 80000000
        154401: dtn:bundling-AK/ -> dtn:bundling-grc1/ length 13601
        154402: dtn:uk-dmc/i -> dtn:bundling-grc1/i length 33128366
        154403: dtn:uk-dmc/i -> dtn:bundling-grc1/i length 42758078
        154404: dtn:uk-dmc/i -> dtn:bundling-grc1/i length 1382309


Se we now have 80 Mbyte proactive bundle fragment #1 from Alaska and 70 Mbyte proactive fragment #2 from Hawaii, but 70 Mbyte proactive fragment #2 is in three three reactive fragments!

We now combine the files using dtnrec
dtnrecv -n 1 -O $1 dtn:bundling-grc1/i


And check that the received file matches what was onboard the spacecraft.

>From Spacecraft Syslog file:
<14>syslog[17:02:02+052 30/09/2009]: looking for files in /home to run MD5 on
<14>syslog[17:02:02+054 30/09/2009]:   syslog.txt
<14>syslog[17:02:02+055 30/09/2009]:   tmp
<14>syslog[17:02:02+056 30/09/2009]:   DU000999pm
<14>syslog[17:02:02+058 30/09/2009]: running MD5 on /home/DU000999pm
<14>syslog[17:07:27+028 30/09/2009]: MD5 of DU000999pm : 0x6964a515 , 0xf7672d18, 0x7a89ee21, 0xce7aeab7

At GRC BundleMaster:
[weddy@Bundle-Master Sep302009_multiterminal_AK-HI_pass]$ md5sum DU000999pm_bak
6964a515f7672d187a89ee21ce7aeab7  DU000999pm_bak

SUCCESS !!!

See the executive summary for the ASCII art.




                    OCTOBER 1


On Thursday, October 1, 2009 we ran nearly the identical test.  However, since we were successful on September 30, we decided to change things.  The routes where now in place so bundles would forward as soon as they hit the ground station so during the first pass in
Alaska we decided to download bundle fragment #2 first, th 70 Mbyte proactive fragment.  The Second pass over Hawaii we downloaded the first bundle fragment, the 80 Mbyte bundle fragment.

1st October 2009

150 Mbyte Image capture at 17:30:21 UTC

MD5 Command at 17:32:02 UTC

Downlink 1 - 17:49:00 to 18:02:00 (Full downlink duration scheduled, eclipse starts at 18:08 UTC)
Downlink 2 - 19:14:00 to 19:27:00


Test Plan for 01 October 2009
==============================

Order of tests Pass 1 over Alaska:
===================================
1) Download Syslog (Check MD5)
2) Download Frag 2 (DTN proactive Fragmentation)
3) Download File using GRC Saratoga
4) Download Syslog again


Check that frag 1 was transferred to Bundle Master Check that frag 1 was received by Bundle Master

Order of tests Pass 2 over Hawaii:
===================================
1) Download Syslog
2) Download Frag 1 (DTN proactive Fragmentation)
3) Download File using GRC Saratoga
4) Download Syslog again


As with the 30 September tests, we were able to download the Syslog file multiple times at each ground station and the proactive fragments.  We also downloaded the entire file without bundling.

As with the  30 September tests, the TCP link between Hawaii and GRC timed out.  This time only once so there were only two reactive fragments.  The proactive and reactive fragments were reassembled and the file MD5 match that onboard the spacecraft.

bundling-grc dtn% bundle list
Currently Pending Bundles (3):

        156002: dtn:uk-dmc/i -> dtn:bundling-grc1/i length 77260545
        156003: dtn:uk-dmc/i -> dtn:bundling-grc1/i length 12320689
        156004: dtn:uk-dmc/i -> dtn:bundling-grc1/i length 67683407



bundling-grc dtn% bundle list info 156004

bundle id 156003:
source: dtn:uk-dmc/i
dest: dtn:bundling-grc1/i
custodian: dtn:none
replyto: dtn:none
prevhop:
payload_length: 12320689
priority: 0
custody_requested: false
local_custody: false
singleton_dest: false
receive_rcpt: false
custody_rcpt: false
forward_rcpt: false
delivery_rcpt: false
deletion_rcpt: false
app_acked_rcpt: false
creation_ts: 307690248.0
expiration: 604800
is_fragment: true
is_admin: false
do_not_fragment: true
orig_length: 157260545
frag_offset: 0
transmission_count: 0


>From the Spacecraft Syslog File:
<14>syslog[17:32:02+075 01/10/2009]:   DU000998pm
<14>syslog[17:32:02+076 01/10/2009]: running MD5 on /home/DU000998pm
<14>syslog[17:37:27+054 01/10/2009]: MD5 of DU000998pm : 0x55e1a3d4 , 0xb0906b00, 0xd95084bf, 0x1353ec50

At GRC BundleMaster:
[weddy@Bundle-Master Oct012009_multiterminal_AK-HI_pass]$ md5sum Oct012009_multiterminal_ak-hi_img
55e1a3d4b0906b00d95084bf1353ec50  Oct012009_multiterminal_ak-hi_img

SUCCESS AGAIN !!!



Regarding why we are losing TCP connection from USN Hawaii to GRC Cleveland:
=============================================
We do not know at this time but we will investigate.  This apparently was a feature as it allowed us to exercise reactive fragmentation - albeit by accident.




Regarding GRC's Implementation of Saratoga:
==========================================
On 27 August of 2008 during our tests, we received bursts of errors on initial download.  This caused a fail of transfer as "inactivity timeout". GRC's implementation Saratoga timed out and had to be restarted.  As GRC had not changed our implementation of Saratoga, we experience a similar situation on 01 October 2009 over Hawaii.

>From conversations with SSTL, we believe that SSTL has their timeout settings longer GRC's for Saratoga.  Also, we understand that SSTL's Saratoga can restart from where it left off whereas GRC's restarts at the beginning.  Finally, in order to minimize transmissions on the 9600 bps uplink, SSTL does some hole combining requests prior to asking for retransmissions - basically just requesting retransmission of a large segment rather than small gaps of holes.  This is done to minimize the back-channel transmission otherwise one creates self congestion in the 9600 uplink.


--Will