[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Rmt] Should FLUTE File Aggregation I-D be a working group item?



Hi Rod,

I think fioe aggregation is basically a useful thing to consider as
the motivation clearly points out.  In Vancouver, I raised the point
about re-inventing gzip.  The following statement in the motivation
section makes me a bit nervous:

   Another motivation for having an object aggregation scheme compared
   to a basic archive based solution (e.g. tarball), is that no extra
   transformation (i.e. archive creation or extraction) is required at
   either the FLUTE sender or receiver.  Everything is managed
   automatically by the transport mechanism according to transport-
   specific optimizations and can be transparent to upper applications
   (i.e. built on top of FLUTE), or enhanced by application hints on
   file relationships, without breaking the basic semantics of FLUTE
   sessions.

Yes, you can always push functionality down a layer (which still means
that you have to perform the function but you just abstract from its
details for the application and limit its amount of work and the
flexibility in applying this function).  But you should carefully
ask yourself how far you want to take this.  Will we see
semantically more complex relationships tomorrow?  Compression
of data?  ...?  How do you deal with partially received objects?
How complex does the API to your transport get?

There is a certain number of things that only the application can do
right by itself.  I would like to see a clear borderline drawn
of what shall go into the FLUTE transport and what should stay up
at the application layer and then we should make sure that the
extensions we define now are going to be orthogonal to other
features we may want to add to the transport later and that they
are easily usable by the application for what it may want to do.

(Remember that we are not building "systems" in the Transport Area
and that we do not necessarily have to build application libraries
either.  So, how much is needed for interoperability in FLUTE?)

Cheers,
Joerg

Hi Mark et al.

Thanks for the support

<I'll take time to consider the AO semantic suggestions - an email for
another day>



Some other minor comments:


- I don't think we need to worry about backwards compatibility with FLUTE receivers compliant to the experimental version which did not have element extensibility in the FDT XML - this was a mistake and the 'experimental' phase is exactly supposed to allow us to find and correct such mistakes.


Now the update FLUTE is practically ready (anticipate one last I-D
update), I agree.

The AO I-D previously the use of non-TOI=0 FDTs to help  with Exp
backwards compatibility. Non-TOI=0 FDTs are allowed anyway, but we need
not explain that this is useful in handling the experimental FDT
extensibility issue.


- the draft includes some text attaching semantics to TOI ordering - something we removed from FLUTE in favour of FDT Instance IDs. We should stick with the FLUTE principle that later version of a file is signalled by using a different TOI in an FDT with a larger FDT Instance ID - it doesn't matter whether the TOI value is greater or less than the previous one.


Note, for FLUTE there is no such semantic. The client has no gauntees
about FDT Instance numbers, no garuntees about TOI numbers.

Generally, using TOI and 'FDT Instance number' values for this has
similar issues (assuming wrap-around is handled smoothly, higher/later
values represent later versions, gaps in the range of number are
feasible, FDT Instances may never be received). TOI keeps semantics with
the file declaration; FDT requires inheritence from FDT Instance to
files described for that Instance. Implemantation-wise there's little
difference between the two: probably the server will be incrementing FDT
Instance numbers and not describing out-of-date TOIs on new Instances;
probably the server will use higher TOI values for later versions; the
client must store TOI with fileURI. There is no requierment that the
client stores FDT Instance numbers.

Quick FLUTE quote:

   A receiver of the file delivery session keeps an FDT database for
   received file description entries.  The receiver maintains the
   database, for example, upon reception of FDT Instances.  Thus, at any
   given time the contents of the FDT database represent the receiver's
   current view of the FDT of the file delivery session.  Since each
   receiver behaves independently of other receivers, it SHOULD NOT be
   assumed that the contents of the FDT database are the same for all
   the receivers of a given file delivery session.

   Since FDT database is an abstract concept, the structure and the
   maintaining of the FDT database are left to individual
   implementations and are thus out of scope of this specification.
...
   *  The receiver SHOULD NOT use a received FDT Instance to interpret
      packets received beyond the expiration time of the FDT Instance.

i.e. the expiry time of file description should be kept with other file
data, but FDT Instance ID is up to the implementation.

Forcing clients to additional store this value is not much trouble - but
it is _not_ "the FLUTE principle that later version of a file is
signaled by using a different TOI in an FDT with a larger FDT Instance
ID". Which is exactly why the AO draft offers the simplest solution to
this issue. If we make the assumptions of "well-behaved simpple FLUTE
servers" above, there really is nothing between FDT Instance ID and TOI
to be used for this purpose. The reason I give all this background is
that this doesn't hold for some obvious AO usage...

For a physically aggregated object (e.g. very large) with some small
updates (e.g. 2 file versions are new from 10,000 files). The server
will use TOI (i) for the initial full object files descriptions. It is
highly likely that the big aggregated object will continue to be
transmitted for some time and with only minimal updates it is probably
more bandwidth efficient to send the symbols for the original object and
the few (e.g. 2) new file versions seperates (more efficient due to,
e.g. fountain, symbols of earlier reception still being useful to all
receivers regarless of complete, partial or not-started receiption
status). So the server sends FDT (ii) with the new TOIs for the files
which have been updated. <At this point using either TOI or FDT Instance
ID would work>. However, some objects of FDT Insatnce (i) are now
changed/removed so that the server need to describe the slighlty old
large aggregated object on TOI (iii). All recievers throw away the new
files (even though the TOI is higher) as the old files are described in
a later FDT Instance.

FLUTE is specifically designed to allow this ebhaviour and make no
assumption the receivers FDT/file database state and no assumption on
reception of individual FDT Instances.

Using TOI for this is simple and works. FDT Instance ID fails. Hence,
the mention of TOI for this purpose in the I-D. (Originally it was start
with TOI=1 and increment by 1, but that's overkill as just 'higher
value' is needed for subseqent versions of a file (fileURI) instance)

(I've been trying to explain this for a while - but it's not important
in 3GPP rel6, and I'm not up-to-date with DVB or OMA to see if
something's fractured there - so maybe this is the first time I've
explained by mail.)


- the draft mentions that one might only consider file aggregation when the aggregate is less than a single source block - I think it is simpler and has no disadvantages if you decouple the source blocking (which is a feature of the FEC scheme chosen) from the file aggregation. If files belong together in an aggregate then they should be put together and then if the FEC code needs to split that data into blocks this is ok and does not result in any worse performance so long as it correctly interleaves the packets from the different blocks.


Exactly.

The I-D does not mean to bind AO and FEC blocking. The discussion is for
implementation guidence: for files much greater than the block size
there's no FEC benefit in physical aggregation; for the composite files
being less than the block size it always makes sense to aggregate for
robustness; inbetween it's a compromise; the I-D absolutely does not
impose any FEC instance retraints and so it's implementation specific to
optomise complexity and performance in this aspect. The guidence is
valuable. If we inadvertantly made normative specification on this, it
was an error to be corrected.

Also, the logical grouping between objects can be used to get the
"belongs together" semantics for multiple AOs - e.g. where partial
recovery is undesirable, or different levels of robustness are
desirable).

Cheers, Rod.










_______________________________________________ Rmt mailing list Rmt at ietf.org https://www1.ietf.org/mailman/listinfo/rmt



_______________________________________________
Rmt mailing list
Rmt at ietf.org
https://www1.ietf.org/mailman/listinfo/rmt