Re: [RAM] Ivip (was ViP ...) placement

[Robin Whittle <rw at firstpr.com.au>]

This is another long "thinking-aloud" brainstorming message,
concerning where ITRs would be, who would run them and pay for
them, how Ivip might be introduced as a single system, whether
RIRs should allow ASNs to use their assigned space for Ivip
systems which are separate from the main one etc.

Hi Joel,

You wrote:

> The note on the list comparing lvip with Mobile IP suggested a question
> in my mind.
> 
> In LISP. the TR (the device provides I and E functionality) is providing
> service directly to a specific set of customers.  Its capacity is tied
> to those customers, and it is operated either by those customers or by
> the ISP associated with those customers.  If we assume that there is
> value in having this, one can envision how the capability gets paid for.


By "TR" I think you mean "Tunnel Router" - I guess a router
which also provides LISP ITR and ETR functions.  These functions
don't have to be on the one router.

These LISP ITR/ETR routers could be internal routers and/or
border routers.  Theoretically the whole edge network could be
served by the ITR/ETR functions being implemented solely a the
border routers.  For scaling purposes, it might be better to
have ITR functions in routers carrying packets towards the
border routers, and to do the decapsulation ETR work in those or
in other internal routers.

In all cases, these functions can only be used by hosts which
are part of, or which connect to the net through, that edge
network, so this generally solves the question of who pays for
it: the users of the edge network.


> If there is a rendezvous point somewhere off in the net, advertising a
> short prefix, and then disagregating that prefix into tunnels, that
> device is operated by someone.  But that someone is not the customer
> whose blocks are being advertised, nor is it the ISP serving those
> customers.  This leads to both authentication questions (how do you know
> this rendezvous box is allowed to issue that advertisement) and business
> model questions (how does it get paid for?
> 
> While conceptually decomposing the devices and separating the deployment
> is attractive, I fear that there are issues raised by such a separation
> that require attention.

Regarding authorisation, I will first try to correct my earlier
confusion about anycasting.  I am now sure that what I am
suggesting for Ivip ITRs really is "anycasting", although a
little different from the usual approach.

I understand that ordinary anycasting in the BGP system involves
multiple BGP routers all over the world advertising a particular
subnet, let's say 11.0.0.0/24.  Then, each such router sends the
packets it receives which are addressed to this subnet to local
hosts which have these IP addresses.  Those hosts accept each
packet and generate a response packet, which the BGP router
handles normally and forwards via the global BGP system to the
correspondent host which sent the initial packet.

This works fine for UDP single query-response systems, such as
DNS queries.  It is not normally used for TCP sessions, since
the correspondent host might at various times have its packets
forwarded to multiple such anycast routers.

Ivip uses the same anycast BGP advertisements.  So whatever
current authentication or policing arrangements work for BGP
anycast - I don't know if there are any - might work for Ivip's
ITRs.

The difference between DNS anycast and Ivip's anycast approach
is that packets received by ITRs all over the world are always
tunneled to the same ETR and so arrive at the one physical host
which has that IP address.  This means it will work fine for
TCP, as far as I know.  Maybe there could be flaky situations in
which the path taken by packets from the correspondent host
fluctuates to direct them to two or more ITRs, with one ITR
having a much longer path to the ETR.  Then, there could be some
out-of-order packets.  But the Internet doesn't guarantee
in-order packet delivery, and in most situations I guess that
the packets would flow to a single ITR in any one period of time
such as 0.2 to 2 seconds.

The business question of who will pay for public-access ITRs
outside edge networks only applies to Ivip's ITRs.  As with
LISP, ordinary (Ivip-basic) ETRs are only located at the border
routers or internal routers of edge networks and can only
successfully decapsulate packets and have them forwarded to the
hosts with Ivip-mapped IP addresses as long as the host is in
that network (and so pays the owner of the network), as long as
the edge network routing system knows how to route the packets
to the host, and as long as the assignee of that Ivip-mapped IP
address or subnet has configured the central database to map
packets to this ETR.

I will leave the question of TTRs which act as EARs for
Ivip-mobile out of this discussion, except to say that I
envisage there being multiple companies operating such systems
of TTRs, with mobile hosts needing to authenticate themselves
before being able to use their services, which the operators of
the mobile hosts, who are the assignees of the Ivip-mapped
addresses, would need to pay for.  ETRs probably don't need to
be coordinated in any way - but maybe there could be some
protocol for asking them to report on connectivity with the host
for a particular IP address.   At a minimum, a TTR must act as
an ETR and as an ordinary BGP router for egress of packets.  As
such, it needs one or a few normal IP addresses.  It doesn't
take up large amounts of address space, as the Ivip ITR system
does.  A TTR may well integrate an ITR function, as I mentioned
in my previous message.  I figure a single global Ivip system
would allow anyone to run an ITR function, provided they
complied with a set of rules, such as always correctly
tunnelling the packets it receives, using the most up-to-date
mapping information etc.


Before discussing the question of who pays to run ITRs, I want
to think more about terminology and situations with the end-user
whose IP addresses are mapped by the Ivip network.

I will try using the term "assignee of an Ivip-mapped address"
to refer to whoever has the effective authority to control the
mapping of a single IP address or a subnet (prefix) within the
Ivip system.

Here are some examples:

1 - One of the many "master-subnets" handled by the Ivip system
    is 11.11.0.0/16.  This was assigned by an RIR as PI address
    to company Z.

    Company Z is the assignee of this subnet (AKA prefix) in
    RIR terms.

    Company Z has registered with the central "Ivip Authority"
    - whoever controls the database which controls the ITRs -
    for simplicity I am assuming a single Ivip system.  There
    could be multiple systems, in which case company Z may have
    registered with the VVV Ivip network.

    In either case, company Z has told the Ivip system that all
    that system's ITRs should advertise 11.11.0.0/16 on all its
    with those ITRs tunnelling the packets however company Z
    chooses.

    Company Z has authenticated access to the control interface
    for the database, so one or more people or hosts under
    company Z's control are physically able to alter the
    contents of the database, and so have complete control
    over how the ITRs tunnel packets sent to 11.11.0.0/16.

    Company Z could have these 65,536 IP addresses mapped in
    various ways:

    a - The whole subnet could be tunneled to a single ETR.

    b - The subnet could be split and tunneled to multiple
        ETRs as somewhat smaller subnets.

    c - The subnet could be split, potentially into individual
        IP addresses, each to be tunneled to an independently
        chosen ITR.

    In all cases, company Z could do this for its own internal
    purposes and/or it could use this address space partially
    or wholly for its customers.

    If some or all of the subnet is assigned to customers, then
    presumably those customers are paying for this service.
    If the Ivip system involves fees, then Company Z needs to
    pay those fees and presumably collect fees from any
    customers.

    In order for this to be useful for the customers each
    customer needs some way of controlling which ETR their
    IP address(es) or subnet(s) are mapped to.

    This could be by company Z having its own user interface
    system which its customers use, via various interfaces
    including a web page for manual configuration, or some
    protocol interface for automated operation.  Company Z's
    system would convey all changes immediately to the central
    Ivip database.

    Company Z may also run a system which checks connectivity of
    its customers who want multihoming, and switches the
    mapping of their address(es) to another ETR if their
    host or router can't be reached via the ETR which
    packets are currently being tunnelled to.

    An alternative would be that company Z tells the
    "Ivip-authority" that until further notice, its customer AA
    should be authorised to directly control the mapping of one
    or more IP addresses, subnets etc. within 11.11.0.0/16.
    This enables customer AA to have faster, more direct,
    control of the mapping of the IP addresses they have been
    "assigned" by company Z.

    In this case customer AA would be the "assignee".

    Customer AA probably doesn't have an ASN or know what BGP
    is.  Customer AA has no relationship with an RIR.  It
    depends entirely on the stability and continued good
    relations with company Z for its continued use of this
    IP address space.

    If company Z has physical control of the mapping of
    customer's AA addresses, but is doing so entirely as
    customer AA desires, then I would still call customer AA
    the "assignee".  This is all sounding rather legalistic...

2 - The Ivip system itself may have been assigned PI space by
    an RIR.  Then, it can assign control (presumably for some
    kind of fee) to end-users for any one, two or up to millions
    of IP addresses.  These do not necessarily need to be
    ordinary prefixes or on binary boundaries.  There is no
    concept of route-aggregation in this Ivip-mapped space.

    The Ivip system could advertise hundreds or even thousands
    of "master-prefixes".  Some could be directly assigned by
    RIRs or others could be assigned by AS-end-users or ISPs
    which had this space assigned to them by RIRs.

    If the Ivip-authority assigns 33.33.27.64 - 33.33.29.16 to
    company N, company N is the "assignee" of these addresses.
    If company N assigns 33.33.28.80 to 33.33.28.87 to company
    M, then company M is the assignee of these 8 addresses.


So there are various ways by which the end-user assignee could
have gained their addresses.  They may be assigned directly from
an RIR and then the end-user authorises the Ivip system to
advertise and map them.  The Ivip system may be assigned PI
space and then rents it out, assigns it or whatever to other
companies or end-users.  An end-user may get their space via a
chain of RIR > Ivip-authority > wholesaler 1 > retailer 2 >
final end-user.


Let's say the RIRs collectively create, or act as a single
Ivip-authority.  While there may also be commercial or
independent Ivip systems, those systems are only going to be of
value if they have lots of well-located ITRs.  For the moment,
let's assume that the single global Ivip-authority system is so
successful that no-one tries to set up their own.


I will try to imagine a stable state of widespread Ivip usage,
then try to imagine how a single system could develop into that
state incrementally.

Let's say the Ivip-authority had enough funding to run a
centralised database server which was capable of sending updates
to ITRs, either by push or by pull methods, and a basic user
interface of manual web-based configuration and direct protocol
interfaces for automated systems to control the mapping of one
or more IP addresses or subnets.   I know virtually nothing
about the finances and business arrangements of RIRs, domain
name authorities, commercial domain name registers and the root,
TLD and major second-level name servers.  I figure the whole
system is paid for largely or wholly by the fees we pay to
register domain names.

Assuming a single Ivip system without any competitors, I guess
there would need to be some similar commercial reseller system
for IP addresses, similar in some ways to that for domain name
registrations, and probably involving the same companies.
However, I hope that the demand for Ivip-mapped addresses would
be for a smaller subset than the mass-market, whole of business
and organisational and personal-domain-name humanity, which the
domain-name business serves.

Hopefully, most people wouldn't need to worry about the nature
of the IP addresses they use for home and small office access.

Hopefully, Ivip-mapped addresses will be managed by end-users
(or their network managers or ISPs) with genuine needs for
portability, multihoming and TE etc.  Also, these are just
numbers, not names, so hopefully the whole thing would be more
technical, less marketing driven etc. than the domain-name business.

I feel sure there will soon emerge some kind of market-based
system for deciding who has IP addresses, because when fresh
supplies run out, the RIRs won't want to handle the endless
disputes and requests about who should have space, when some
ISPs or AS-end-users have large slabs of space which they appear
not to be using efficiently, or perhaps at all.  With a market,
that space would either be returned to the RIRs or sold or
rented out or whatever.

There may well be a commercial market for addresses within the
growing proportion of addresses which are mapped via Ivip or
whatever.  This may be quite separate for whatever commercial
arrangements develop for raw directly BGP-routed space, in 256
IP address increments, as currently assigned by RIRs to ISPs and
AS-end-users.

Competing systems of ITRs, each operated by a company and
renting out the mapping of its IP addresses, could only persist
as long as at least one RIR considered this to be a legitimate
use of the space it had assigned.  Normally, I understand, RIRs
assign space according to projected need of an ISP or
AS-end-users - need for space in their own physical networks.

A company or organisation running an Ivip system is not running
a physical network.  They would be engaging in the currently
unprecedented practice (I think) of having a bunch of anycast
routers all over the Net, for the purpose of tunnelling packets
one-way to one or probably numerous edge networks, usually not
their own.

So I think it is probably a matter of IANA/RIR policy whether
there will ever be multiple Ivip systems.

I imagine this stable, very widely used Ivip-authority single
Ivip system of ITRs being made up of ITRs and various other
things which are not operated by the Ivip-authority itself.

The Ivip-authority would be running something like the first
levels of the DNS.  There would be millions or billions of IP
addresses under its control.  Many of them would have stable
mapping which would only change every few years.  Some might
have mappings which change every few minutes, as a mobile device
takes its packets through different TTRs.

Maybe the Ivip-authority charges for its services partly by:

1 - Total number of IP addresses.

2 - Total number of changes, where a single change can apply to
    a large number of IP addresses, according to whatever
    physical protocol is used to send the changes to the ITRs.
    So changing the mapping of 32 randomly distributed IP
    addresses would be charged as 32 changes, but only as
    one change if it was a contiguous set of addresses which
    could be specified in a single change message.

3 - Also, perhaps, by the number of queries it receives for
    these addresses via any "pull" system it operates - but
    this is open to abuse by hackers who want to bankrupt
    an assignee of an Ivip-mapped address.


There are some significant scaling problems and questions of
commercial cost in any such system.

Let's say that most edge-networks perform their own ITR
functions.  They do this to remain competitive, since the
alternative is relying on ITRs out in the DFZ.  These may
involve longer paths or dropped packets due to being overloaded.

This would be ironic:

   LISP ITR's must be in internal routers or border routers.

   LISP is never attempted because no-one wants to be the first
   to use LISP-mapped addresses while most edge-networks have
   no ITRs.

   Ivip is launched instead, with an initial seeding of ITRs in
   the DFZ.

   After a significant amount, say 20% of traffic, flows to
   Ivip-mapped addresses, pretty much every edge network has
   its own ITRs.

So Ivip winds up resembling LISP, in many respects!

Ivip's only two real differences from LISP are:

1 - Mapped addresses are advertised in BGP.

2 - ITRs can be in the DFZ, using anycast.

(Someone would have thought of these sooner or later.  My idea
of using Ivip ITRs to beam packets to freely chosen TTRs for
Ivip-mobile is equally applicable to LISP and would have been
seized upon by anyone thinking about LISP and mobile IP at the
same time.  My idea of how LISP or Ivip should be used may
differ from how others think about it, and I have written more
expansively/ramblingly and given some diagrams, maybe more
concrete examples etc., without a large number of variants
numbered to two decimal places.)

With ubiquitous adoption, there wouldn't be a lot of difference
between LISP and Ivip.  But the fact (I think) that Ivip could
be adopted, and that it could continue to work with a handful of
non-upgraded edge networks, is a vital distinction.


Very roughly, here is how a single global Ivip system might be
developed.  Please excuse my probable ignorance of how the
various organisations work together.

1 - IETF develops protocols, standards etc.

2 - IANA and RIRs decide that a single Ivip system needs to be
    established, and that anyone using address space for their
    own Ivip system is not using it in accordance with
    prevailing policies.

3 - ICANN, IANA, RIRs and maybe peak organisations of domain
    name resellers set up some new authority to run the
    database.

4 - A handful of transit providers promise to run ITRs around
    the world to get the system going and to support, in the
    long term, edge networks which don't provide their own
    ITRs, although that service would probably be slow.

    (Maybe there is a peak body of transit providers and major
    telcos/ISPs who could commit their members to providing some
    level of ITR support.  This would cost money, but by this
    time, Ivip or whatever would be seen as the best or only
    hope of containing the growth in the BGP routing table
    which is threatening to make all their routers obsolete.)

5 - In the light of the Ivip alternative, maybe the RIRs
    develop some policies about the addresses they assign
    for normal BGP use, how assignees are expected to use
    that space without excessive numbers of changes etc.

    Also, there may need to be some new policies about RIRs
    assigning large slabs of address space to the Ivip system.
    The more space, and the fewer the divisions between its
    contiguous divisions, the fewer BGP advertisements will be
    needed.


The ISPs and AS-end-users could install ETRs and organise their
internal routing systems to cope with the various Ivip-mapped IP
addresses or subnets in their edge networks.  This shouldn't
cost much, and only needs to be done by edge networks where
customers or internal users will want to locate hosts with
Ivip-mapped IP addresses.

ITR capabilities in routers could be developed.  As a low-cost
alternative, it might be possible to develop open-source
software for a dedicated ITR box, such as a 64 bit Linux box
with a 1G or 10G interface.  This can do nothing but be a BGP
advertiser for the Ivip "master-prefixes" and send the
encapsulated packets back on the same Ethernet cable.  This
"stub-ITR" could be built for a few thousand dollars and plug
into an interface of any router, to make that router effectively
an ITR.  Load balancing could be done by having multiple boxes,
each advertising a subset of the Ivip system's "master-prefixes".

If existing routers couldn't do the ETR function, there could be
a similar Linux box with open-source software for that too.
This would greatly ease the adoption costs for many ISPs and
edge networks.

After a few years, router vendors would implement ITR and ETR
functions as standard in routers - so the ITR function would
migrate into real routers as part of the standard upgrade cycle.



Ivip's ETR functions (not counting TTRs, which may be outside or
inside edge networks) must be implemented either at border
routers and/or internal routers.  In either case, as with LISP,
it is clear that the users of those ETRs will pay their ISP in
some way for traffic which flows through those ETRs.  I think
the ETR cost could be low, since they don't require a database,
don't require authenticated access or maybe any management at
all.  It may be as simple as having a bunch of routers in the
network, including the border routers, all providing ETR
functions, and giving advice to users inside the edge network
about which ETR they should best map their Ivip-mapped IP
address or subnet to.

If Ivip became ubiquitous, then as for LISP, I think most of the
edge networks would implement their own ITR functions, at
internal and/or border routers.  This would provide their users
with the best performance, because the ISP could guarantee that
all packets generated in the edge network for Ivip-mapped IP
addresses would be locally encapsulated, removing any dependence
on external ITRs.


Lets say that Ivip ITRs are widely deployed in most edge
networks and many users depend on Ivip-mapped IP addresses.

Any hosting company with a proportion of their servers using
Ivip-mapped addresses (I am not sure that this is a justifiable
use of Ivip - there's nothing wrong with web servers using PA
addresses from whatever hosting company they are sitting is and
having the customer's nameserver point to that address) will
want to ensure those servers are reachable via any edge network,
including those with no internal ITRs.

The hosting company could do this by making their border routers
act as ITRs.  They could also plant an ITR just outside their
network.

The ITR, in both cases, I assume would have to advertise the
complete set of "master-subnets" which the single global Ivip
scheme advertises.  (If there were multiple such schemes, the
hosting company's ITR would need to implement ITR functions for
the "master-subnets" of every such scheme for which some edge
networks did not themselves implement ITRs.)

An ITR should, I think, not just accept all incoming packets for
the "master-subnets" it advertises, but properly handle them by
tunneling them to whichever IP address of an ETR which they are
currently mapped to.

It would be unfriendly and disruptive to have an ETR advertising
the whole set of "master-prefixes" but dropping all packets
which were not to be tunneled to the hosting company's ETRs.  I
wonder how disruptive this would actually be, however.  The
hosting company could say: "If these packets found their way to
our border router, or our nearby outrigger ITR, then firstly
they came from edge networks which were not equipped with ITRs
and secondly those edge networks were deficient in these nearly
ubiquitous Ivip times by not ensuring there was an upstream ITR."

I am trying to find a long-term reason why an ISP, hosting
company, transit provider or AS-end-user would want to run an
ITR in the DFZ in the long term.

Someone needs to do this, but to provide a "second-best" service
so as to encourage edge networks to install their own ITRs.

Probably the transit providers are the best placed to run these
ITRs.  But why should they do so?

Initially, they might volunteer to do so in order to get Ivip
off the ground, since the rising tide of the BGP routing table
will be making them fearful.

Initially, the Ivip-authority could run a few ITRs to get the
system going.

A commercially run Ivip system would either run its own ITRs
and/or pay others, such as transit providers, and ISPs, to run
the ITR functions on their own routers.   If there were multiple
such commercial systems operating with compatible ITR <->
database pull or push protocols, then ISPs might have a single
router which runs the ITR functions of multiple commercial Ivip
systems on, charging each such system a fee for the traffic volume.

The commercial Ivip systems would generate revenue by charging
their customers according to IP addresses assigned, the number
of changes made to the mapping, and the volume of packets forwarded.

I am not sure if independent Ivip systems are a good idea.  My
sense is that they are not, as long as a single centrally run
Ivip system is well run.

Transition to a major new architecture requires a lot of thought.


  - Robin         http://www.firstpr.com.au/ip/ivip/












_______________________________________________
RAM mailing list
RAM at iab.org
https://www1.ietf.org/mailman/listinfo/ram