Short version:

  Responding to Bill's suggested changes to the text.

  Detailed discussion of prospects for DFZ routers having new
  functions to suit the scalable routing solution.

  A new version of 6 - I will update the draft in the next day
  or so:

     http://www.firstpr.com.au/ip/ivip/RRG-2009/constraints/


Hi Bill,

Thanks for this:

> I like where you're going with this list.

and for your suggestions.


> I'd suggest a change to point 6. You have:
> 
> 6 - The solution must be compatible with all
>       existing DFZ routers. (Unless it can be shown
>       that all DFZ routers can be upgraded - such as
>       via a software update - by the time of
>       introduction.)
> 
> First, I'd drop the "unless." There really is no "unless" here. We've
> demonstrated rather conclusively that in today's Internet it is highly
> impractical to try to get the 31,000 individual organizations who
> compose the DFZ to deploy configuration changes ahead of that
> individual organization's direct use for the change.

OK - this prompted me to think in more detail about altering DFZ
router functionality.  Below is updated text which is more narrow
than "all DFZ routers", but which continues to allow for the
possibility of new DFZ router functionality in time for actual
deployment.


Core-edge separation schemes rely on the DFZ for transporting packets
in a manner different to that which occurs according to their
destination address.  One approach is for an ITR to encapsulate the
packet - give it a new outer destination address - and then use the
DFZ and all other as they function today to forward the encapsulated
packet to ETR.  Another approach (Six/One Router) involves changing
the destination address so the packet goes from the ITR-like router
to the ETR-like router, and than changing it back to the original
address at the ETR-like router.

Encapsulation is inefficient and has PMTUD problems which requires
the addition of considerable complexity, state, processing effort
etc. in ITRs and ETRs.  The only approaches to address rewriting
(Six/One Router and SHIM6) are for IPv6.

I made two proposals for tweaking DFZ router behaviour (and also some
internal routers of ISP networks adopting the scalable routing
solution) which would enable ITR -> ETR tunneling of packets without
altering their destination address and without encapsulation or any
extra bytes being added:

  EAF - http://tools.ietf.org/html/draft-whittle-ivip4-etr-addr-forw
  PFL - http://www.firstpr.com.au/ip/ivip/ivip6/

These two "Modified Header Forwarding" (MHF) approaches are the only
ones I know of - but perhaps more will be developed.  Neither involve
changes to BGP.  Both involve changes to the FIB and changes to the
code which creates Packet Too Big messages. For IPv6 PFL, there is
also a change to the way the RIB drives the FIB.   There would be a
single new configuration item for each one - whether to enable it or
not.  None of these changes are particularly complex considering the
enormous number of things the FIB and RIB etc. already do.


Considering just IPv6, let's say that in 2012 the IETF develops PFL
and recommends that all new routers have this function.  There's no
tearing hurry deploying the solution in IPv6, but we need to plan for
it.   Let's imagine that by 2017, 80% of DFZ routers have been
manufactured since 2012 and have PFL built-in.

IF the remaining 20% of routers have had firmware updates to
implement PFL and IF the operators have either as a matter of course
installed suitable updated firmware or could be cajoled into doing so
by 2017, then we have a basis for scalable routing ITR -> ETR
forwarding in 2017 which is much more simple and efficient than
encapsulation and its PMTUD complexities.  IPv6 encapsulation
overhead is highly significant, especially for VoIP packets.

I doubt if any router now, less still in 2017, implements its IPv6
FIB in raw silicon.  If none of them do, then they can all be made to
support PFL with a firmware update.

I am keen to craft some words which allow for this sort of development.

Doing the same for IPv4 could be a lot trickier.  Maybe by the time
we need to deploy the solution, there will still be too many DFZ
routers with FIBs implemented in hardware.  On the other hand, at the
current rate of progress, it may be 2017 before we implement scalable
routing for IPv4 - by which time I expect all routers would have
firmware-defined FIB functions.

You mention "31,000 individual organizations who compose the DFZ".  I
guess you are referring to the number depicted in Fig 7 of:

   http://www.potaroo.net/tools/asn32/

(Salute Geoff Huston for maintaining this and related pages!)

Now I will discuss today's IPv4 figures as a basis for estimating how
many ASes would need to upgrade their "DFZ router".

Any AS end-user network such as NET-A which has its PI prefixes
advertised in the DFZ, but which only connects to the DFZ via its
upstream ISPs is not a concern for us.  These ASes have no DFZ routers.

NET-B is an AS end-user network with a PI prefix advertised in BGP,
and with a BGP router single-homed to an ISP.  That router is not
part of the DFZ.  As long as NET-B doesn't adopt the scalable routing
solution (by installing ITRs and/or ETRs), it need not upgrade its
router.

ISP-X has one or more BGP routers multihomed to other ISPs, but it
does not allow transit traffic.  These are DFZ routers, but as long
as ISP-X does not adopt the scalable routing solution (that is, it
does not have ITRs or ETRs in its own network or in the network of
any of its PI or PA customers) then it doesn't need to upgrade its
routers either.

NET-C is an end-user network which resembles ISP-X in that it has one
or more DFZ routers, multihomed to upstream ISPs.  By definition,
end-user networks don't allow transit through their routers.  So as
long as NET-C does not adopt the scalable routing solution, it need
not upgrade its routers.

So the routers which need to be upgraded belong to only a subset of
the 31,000 ASes:

 1 - The DFZ routers and some, many or all internal routers of any
     end-user network or ISP which adopts the scalable routing
     solution.

 2 - All transit routers.

Depending on when the scalable routing solution is needed, this could
be too much of an ask for IPv4, but I don't rule it out.  I think it
is a perfectly practical idea for IPv6, unless IPv6's adoption rate
skyrockets soon and requires a solution to be implemented much faster
than I anticipate.

I figure transit routers are typically larger, recent production,
routers and so would be likely to be amenable to firmware upgrade if
they were not already manufactured with the EAF or PFL features built-in.


I can't prove any of this is possible, but considering the benefits
in avoiding encapsulation altogether, and my guess that it will be
later in the coming decade when a scalable routing solution is
actually deployed, I want to write the constraints text in a way
which does not rule out such new functionality in DFZ routers.


> Then I'd change it to something like:
> 
> 6 - The solution must function compatibly with existing DFZ router deployments.

This means just the same as the first sentence.  I think this on its
own, without the original second sentence, unnecessarily assumes that
transit router functionality can't be upgraded in time for deployment.

Here is another attempt, but it specifically allows for the
possibility of upgraded functionality which you just argued could
never be possible.  I hope I have convinced you that it is not out of
the question, especially with a longer timeframe for deploying the
solution.  IT projects typically take longer than anyone cares to
admit.

This is a major upgrade to the world's most entrenched and important
IT system.  If I had to pick a year for 50/50 odds of initial actual
deployment, I would say 2017 for IPv4.  At current growth rates
(which may increase, I admit) the DFZ would have grown to about 650k
by then.  So maybe it will be 1M after allowing for lots more
chopping of address space once the fresh pastures are all used up.


The new text involves a more narrow definition of than the previous
"DFZ routers":

   6 - If the solution requires extra functionality in
       interdomain routers, then it must be feasible from
       technical and business perspectives for the relevant
       routers to have this functionality in time for
       deployment.  For instance, a core-edge separation
       scheme which relied on new router functions for
       ITR to ETR tunneling would require the functionality
       to be present in all transit routers and in the
       routers of all ISPs with customers which adopt the
       solution.


> Asking them to forklift their hardware is bad but they're going to do
> that on a 3 to 5 year cycle anyway. Where the effort falls apart is
> asking them to reprogram their systems before they're in a position to
> derive a benefit from doing so.

ISPs adopting the solution have a clear economic incentive to upgrade
their routers.  I figure the cost of firmware upgrades would be
minimal.  Maybe they upgrade their firmware regularly anyway - so
there would be no extra cost or inconvenience, assuming the router
vendors made the one-off investment of adding the new feature.

I figure router vendors would be highly motivated to do so, in part
to ensure their routers continue to implement every protocol in
creation - and also to show how well engineered and open-ended their
products are.

Assuming modern routers can be upgraded with firmware, I don't think
it is too much to ask all transit ISPs to ensure their routers are
ready for the new arrangements, with 2 to 4 years notice, or whatever
is required.  The alternative is launching with complex, inefficient
encapsulation and then in the longer term transitioning to something
like EAF and PFL once all routers have the new capabilities.


One gotcha with this idea of avoiding encapsulation from the start is
the difficulty of testing the system on a global basis before the
transit routers have the updated functionality.   Maybe we could
prototype with encapsulation, without bothering to handle the PMTUD
stuff.


> I'd also suggest a change in point 1. You have:
> 
>   1 - The solution must provide immediate and
>       compelling benefits to any new or existing
>       end-user networks which adopt it, irrespective
>       of how many other networks have so far adopted
>       it.
> 
> I'd change "end-user networks which" to "end users who," and then
> "other networks" to "others."
> 
> I know you're not a fan of the strategy B approaches, but some of them
> have the prospect of being functional as soon as two participating
> hosts have been upgraded regardless of what happens with the rest of
> their local networks.

Initially I created new text based on your suggestion.  Then I had
second thoughts and for now will retain the current text.

I don't want the text to assume that all scalable routing solutions
are all core-edge separation approaches, meaning network-based
systems such as LISP, APT, Ivip and TRRP (Six/One Route too?) with
ITRs and ETRs etc.

(The next bit is for readers learning their way in scalable routing.)

   In your taxonomy:

     http://bill.herrin.us/network/rrgarchitectures.html

   as used in:

     http://tools.ietf.org/html/draft-irtf-rrg-recommendation-02

   and as I attempted to match to actual proposals:

     http://www.firstpr.com.au/ip/ivip/rrgarch/

   Strategy B covers core-edge-elimination architectures.  This is
   very different from core-edge separation.  See:

     Towards a Future Internet Architecture: Arguments for Separating
     Edges from Transit Core
     Dan Jen, Lixia Zhang, Lan Wang, Beichuan Zhang
     http://conferences.sigcomm.org/hotnets/2008/papers/18.pdf

   Core-edge elimination involves major host changes - to stacks and
   I think applications.  There are no changes to DFZ or other
   routers.  Hosts do all the work.  AFAIK, such schemes are only
   potentially practical for IPv6, because every host needs two
   addresses - at least one ("locator" I think) that it uses to
   connect to the network and one ("identifier" I think) for its
   applications to use, which is still used if the "locator" changes.
   There's not enough space in IPv4 to do his on a large scale.

I think your suggested change was to allow for a solution which
involved individual hosts, rather than whole networks.  Can these
Strategy B approaches be adopted by individual hosts, without any
change to addressing in the networks they are operating within?

I will assume they can be - and that they do not involve any network
making a change to its routers or connection to ISPs.

Ideally, the wording wouldn't assume the schemes are either
host-based or router-based.  I could update point 1 like:

   1 - The solution must provide immediate and
       compelling benefits to any new or existing
       end-user networks or end-users who adopt it,
       irrespective of how many other networks
       or end-users have so far adopted it.

But the rest of the text assumes that networks adopt the solution,
not individual hosts.

Whatever the type of solution, it will in practice be adopted by
entire end-user networks.  (Though in principle there might be only
one host in such a network.)

Our goal is to provide multihoming etc. on a scalable basis for
end-user networks, so I think it is best to retain consistent
terminology about networks adopting the solution, even if the
solution involves changing only the hosts in the network, rather than
the network itself, by way of altering its routers or method of
connecting to one or more ISPs.


Thank again.


  - Robin