Re: [IMRG] ipmp review team report

[Tony McGregor <tonym at cs.waikato.ac.nz>]

Firstly, we want to thank Mark and the team for taking the time to do
the review. Although we don't agree with the conclusion, many of the
points made are closely aligned with our own views.  In particular, the
need for a simple light weight protocol (see the points we have
labelled as [1] and [2] in the report) and the need for ISPs to want
the protocol before it will be widely implemented [3,10].

Unfortunately, we think that the group has not understood the
motivation for the protocol and has misunderstood what we propose at a
technical level.  The reasons are probably because the group was
presented with two competing and inconsistent _drafts_ of the
protocol, not the thinking behind them, and also that we were unable
to contribute to the discussion.


Motivation
~~~~~~~~~~

We want to focus this reply on two aspects.  Firstly, the motivation
for the protocol.  From the report, it seems that the group did not
get very far beyond this fundamental point [4,5].  Perhaps that is
because the two Internet drafts that formed the input to the group
are primarily technical documents.

The report begins by narrowing the focus of what is useful.  We do not
agree with some points in the reduction in focus.  These include the
limitation of what is useful to software only proposals [6] that can
be implemented on existing equipment [7] and that will be adopted
universally by ISPs [9].  This is unduly restrictive.  Many useful new
technologies would never be implemented under these restrictions.  We
believe this is a fundamental error in the thinking of the review
group and has influenced the rest of the report heavily.

It is a fundamental tenet of IPMP that it can be usefully deployed
in a piecemeal fashion by only those ISPs and other network operators
who wish to do so. It is, for example, not required that "at minimum
[ISPs] not turn [the measurement protocol] off." [8]

The group makes an important point when they say that router vendors
will not implement a protocol unless they believe their customers want
it [10].  However, asking whether ISPs are currently asking for IPMP
[3] is not, in our opinion, the correct question.  There is a Catch-22
here. There is no standard, so how can ISPs ask for it - they don't
even know what it would do.  The inference is that because ISPs aren't
asking for IPMP [11], we don't need a standard.  A better question
would be more abstract. E.G.

 "Would you have a use for a protocol that would allow you to
  demonstrate to your customers that your network is performing well
  and would allow you to collect more information about the internal
  performance of your network [20]?"

The answer, from at least some large ISPs, is almost certainly yes, but
probably with provisos.

We would like to note that it was listening to people from a group of
US ISPs (at a CAIDA ISMA workshop) that originally motivated IPMP.
Those ISPs raised a fundamental problem.  They said something like:
"The question for us is not whether we will allow our network to be
measured.  We are measured all the time.  Our problem is that we are
measured badly and blamed for problems that aren't ours. This is done
by customers who then call us when the fault is local to their network
or at an upstream ISP, and by the press when they do a bunch of
measurements and make erroneous claims about how we compare to other
IPSs.  What some of us want is the ability to be measured well."  IPMP
would allow those ISPs to justify their claim that they offer good
service, either to the general public or to select users.

Although the report does not mention them, there are also many non-ISP
router vendor customers.  The usefulness of IPMP is not limited to
ISPs.  A campus too, could find IPMP useful in a similar way; even if
the only IPMP devices in existence were those on its borders.  Of
course, deeper deployment will bring more benefits.


Technology
~~~~~~~~~~

The group notes repeatedly that it is important that a measurement
protocol is simple and lightweight [1,2,12,13].  We strongly support
this view.  In discussion on IPMP we have constantly maintained the
need for simplicity.  Doing so has been the cause of considerable
tension, and a major reason for there being two competing draft
series.  In recent mcgregor drafts [ e.g.
http://www.wand.net.nz/~mjl12/draft-mcgregor-ipmp-04.txt ] we have
compromised this principle a little, in an attempt to bring some unity
and make progress.  However, a careful reading of the draft will show
that much of the complexity is optional.

The review group claims that IPMP is complex [12,13].  The core of
IPMP is not complex.  As an alternative, the group suggests a
protocol which is essentially the same as the heart of the mcgregor
draft [16].  (The difference is just replacing an IP address
with an opaque ID, which can trivially be converted back to an IP
address.)  That the proposal is very similar to IPMP, and no simpler
than IPMP, indicates confusion about the technology of IPMP.

We have several implementations of the protocol both in software and
hardware.  The forwarding path modifications require 12 bytes (IPv4)
or 24 bytes (IPv6) to be written into the packet, and the last 4 bytes
to be modified.  The additional burden on the forwarding path is not
significant.  In hardware, the required operations can be implemented
in parallel with other packet processing.  The protocol has also been
designed to allow it to be processed in "serial forward mode only".
That is, the packet can be processed a byte or word at a time as it
passes through the packet processor. It is never necessary to modify
data early in the packet based on what comes later. This supports
implementation in a simple separate processing unit if required.

The review group also suggests that "clever inference techniques"
might gain some of the same information [17,24,26]. Most measurement
research takes this path.  However, direct measurement has many
benefits.  Frequently, inference techniques require the use of
approximations that are then applied to other approximations.  An
unstable tower of successive estimates is built.  Direct measurement
may reduce some of the uncertainty.  (See our paper on IPMP and
bandwidth estimation, for example). As a matter of principal, it is
unreasonable to cut off one avenue of research just because another
might produce similar information.  This is particularly true where
it is direct measurement that is discarded in favour of inference.
It is the difference between "wondering how much could be
learned" [18] and knowing what is measured.


More Minor Points
~~~~ ~~~~~ ~~~~~~

Ping and traceroute are used extensively by researchers, end users
and ISPs for a range of purposes including network troubleshooting.
They are the industry de facto tools for such things.  Making a
better ping and traceroute is a worthy goal and is not either
"narrow" or "focused on what researches find compelling rather than
what operators would"[18].

The goal of IPMP is performance measurement not to "reverse engineer
a network"[19].  IPMP measures the path and latency of actual traffic
and ISPs _are_ interested in such things.

We suspect ISPs trust one another[23] when they can't do anything
else. IPMP is ideally suited to measure inter-ISP SLA.[21]

We suspect the review group did not fully grasp the significance of
the separation of the protocol into a forwarding path component and
an offline, information request component.  In addition to making
the forwarding protocol very simple it is this separation that could,
for example, support disclosure of details to just some parties.

"The indirect cost of the leaking [of] proprietary topology
information" (via tracing the path that a packet takes) is not likely
to be a "barrier to deployment" of a new protocol.  This information
is already available through traceroute.[25]


Summary
~~~~~~~

In summary, we support many of the comments of the review group, but
believe that they came to the wrong conclusion, largely because of
lack of input and the opportunity to discuss motivations and
technology in detail with the authors of the protocol.

Thanks to those of you who made it this far.


Tony and Matthew




On Tue, 7 Sep 2004, Mark Allman wrote:


>  
> Folks-
> 
> A while back you might remember that we had some discussion on the IP
> Measurement Protocol.  In order to try to gain some traction I asked a
> few folks to act as a review team to look over the two proposals on the
> table (and the ancillary information).  The team has completed their
> work and did a very nice job of debating the issues involved in IPMP and
> coming up with a summary of their feelings.
> 
> The team members are listed at the bottom of the report and I wish to
> thank them for their diligence in reviewing these documents.
> 
> The report from the group is below.  Please feel free to discuss the
> ideas enumerated in the report on this mailing list all you want.  The
> team is not the final word.  I convened the team to get some focued
> energy thinking about these issues.  The report is in no way binding nor
> the community's final judgement.  So, please feel very free to continue
> the discussion.
> 
> allman
> 
> 
> 
> 
> 
> IMRG IPMP Review Team Report
> ----------------------------
> 
> The Internet Measurement Research Group (within the IRTF) convened a
> small team to review the materials related to the IP Measurement
> Protocol (IPMP).  The members of the group (listed at the end of
> this report) discussed IPMP and several larger issues.  In
> particular, the team reviewed the following two Internet-Drafts:
> 
>     draft-mcgregor-ipmp-03.txt
>     draft-bennett-ippm-ipmp-01.txt
> 
> The goal of this effort was to chart a strawman course for moving
> forward with some sort of measurement protocol (if possible).
> 
> Note: This message represents the group's consensus.  However, that
> does not mean that each member of the team agrees with each point in
> this note.  The group reached rough agreement, not unaminity.
> 
> The following are the high-order bits from the discussion.
> 
> The fundamental challenge that measurement protocols attempt to
> address is to provide a means to measure the network characteristics
> researchers and operators want to understand in a way that provides
> fine grained information about the network in a lightweight fashion.[1]
> To this end, we would suggest that IPMP wants to develop tools that
> are:[6]
> 
>   - implementable in reasonable timeframes on existing equipment,
>     which means that they should not depend on ASIC development [6] or
>     new equipment purchase [7]
> 
>   - deployable; ISPs would ideally want them, and at minimum not
>     turn them off [8]
> 
>   - useful to the ISPs in terms of their business rules and the
>     questions they ask about their own networks [9]
> 
> If the procedures or protocols are useful to the ISPs, one can
> expect that they will be willing to collect the data, and may under
> some appropriate rules also allow researchers to collect data or
> share collected data with researchers.
> 
> In the above context, the team found the motivation for IPMP given
> in both documents to be lacking --- to the point where the team did
> not feel the current proposals are viable.[4]  Several
> related/supporting points were discussed:
> 
>   * From the perspective of a vendor developing equipment and
>     protocols or an ISP deploying them, the IPMP proposals on the
>     table do not look viable.  The fundamental goal of IPMP is to
>     display the structure of a network and many of its fine-scale
>     characteristics.  This is information that a service provider
>     does not share with anyone else except - maybe - under
>     NDA. Given that the protocols to obtain the information are
>     fairly complex[14] and involve a fair level of memory writes[15], the
>     vendor will do this if and only if its ISP customers ask for it, [3]
>     and they are not asking for this. [11]
> 
>   * Making a better ping or traceroute is, on the one hand, too
>     narrow and mechanistic a focus and yet also too focused on what
>     researchers might find compelling rather than what operators
>     would. [18]
> 
>   * A tool to reverse engineer a network isn't needed by the ISPs.[19]
>     They already know the structure of their own networks. 
> 
> That said, the team **strongly** believes that there is much room
> for improvement in the state of network troubleshooting and
> debugging.  In particular:
> 
>   * Some service providers are asking for a solution to a problem
>     that may yield data that researchers may find valuable.  Within
>     its own network, a service provider is generally interested in
>     locating the links that introduce variability into their
>     network.  It may view them as under-provisioned for offered
>     load, as inappropriately routed, or whatever, but they are in
>     fact interested in locating links that require upgrading in some
>     form.[20] 
> 
>   * Some service providers are asking (in TIA and related fora) how
>     they can deploy SLAs that cross ISP boundaries. These may be
>     among ISPs that form business coalitions, such as Teleglobe has
>     tried to set up with its transit network customers, or among
>     regional networks such as US RBOCs that view transitive SLAs as
>     a rational approach.  The watchword in such consortia is "trust[22]
>     but verify"; it is in their interest to have a procedure or
>     protocol that will allow them to isolate issues that may prevent
>     them from meeting SLA guarantees in something resembling real
>     time.  Since those SLAs are one-way, this means accurate one-way
>     delay and jitter measurements host to host, POP to POP, or CPE
>     to CPE.[21]
> 
> In addition, in looking at the protocols themselves, we found
> ourselves wondering how much could be learned[18] by clever inference from
> fairly simple data collection and black box measurement, as opposed
> to explicit reporting of values.[17]
> 
> As another example, we note that the intention of such procedures as
> CalTech's FAST and MIT's XCP protocols is to detect and measure
> variable delays in the network and cause traffic to be sent in such
> a way as to maximize throughput while minimizing such delays. This
> fundamental question is a direct corollary to that raised in
> http://www.nwfusion.com/research/2002/1216isptestside1.html, and
> that raised in the context of transitive Tier 2 network SLAs. These
> would like to be able to identify the existence of an SLA failure or
> other disturbance in the Force on a route, report its magnitude, and
> isolate the disturbing device. To that end, we wonder can be done
> with the numbers measured by Dina Katabi's XCP protocol.[24]
> 
> Finally, the team wondered if a protocol that carries less global
> information but more precision would be more deployable.  For
> example if the stamps just consisted of an opaque ID, TTL and simple
> 32 bit counter running on "the most stable local frequency source",[16]
> then the ISP (w/ the engineering documentation for their own gear)
> can use database techniques to compute everything carried by the
> current protocol.  The stamps are simple enough where we can, with a
> straight face, ask for them in multiple places within one box: input
> and output framers, bus DMA engines, etc.  We can envision that this
> would be an extremely valuable tool for an ISP to understand (and
> diagnose) certain QoS properties of their own network.  Note that,
> globally parsable metadata in the stamps probably has negative value
> to most ISPs because it reduces an ISP's ability to keep it's assets
> private.  The barrier to deployment in not so much the cost of the
> implementation, but the indirect cost of the leaking proprietary
> topology information.[25]
> 
> At the same time, external researchers could use inference
> techniques to get some of the same information, including most
> dynamic properties such as queue depths etc.  The external users get
> much less topology information, unless they make an explicit
> arrangement with the ISP to get the annotations associated with the
> opaque IDs.[26]
> 
> In summary, the team came to two points of consensus: 1) that the
> protocol is inadequately motivated by the proposals,[5] even though
> ISPs would like to be able to measure their and their neighbors'
> networks; 2) that the protocol's complexity[13] and intrusiveness are
> inadequately justified with respect to other, potentially more
> lightweight approaches that may be easier to deploy.[2]  The main point
> is that to get a protocol deployed, ISPs need to ask for it loudly
> enough and router vendors need to be able to implement it[10] easily
> enough[12], and neither is argued by these proposals.



----------------------------------------------------------------------------
Tony McGregor                   Mail:   T.McGregor at cs.waikato.ac.nz 
Department of Computer Science  Phone:  +64 7 838 4651 
Waikato University              Fax:    +64 7 858 5095(w) +64 7 825 5047(h)
Private Bag 3105                Home:   +64 7 825 5040 mobile: (021)313004
Hamilton, New Zealand           www:    http://www.cs.waikato.ac.nz/~tonym
----------------------------------------------------------------------------


_______________________________________________
IMRG mailing list
IMRG at ietf.org
https://www1.ietf.org/mailman/listinfo/imrg