[PMOL] Several questions/suggestions from my review of draft-ietf-pmol-sip-perf-metrics-04

[Robert Sparks <rjsparks at nostrum.com>]

Hi All -

I have several concerns about draft-ietf-pmol-sip-perf-metrics that I  
would like to discuss.
I've asked for a dedicated RAI-review of this document, so there may  
be additional comments
later, but I wanted to get these to you now so we can start working  
through them.

These comments are more-or-less in document order, with a couple of  
nits moved to the end.
I've numbered them to help split responses into threads later. When  
replying to just one of the
items below, please change the subject line to indicate what you're  
replying to.

Thanks,

RjS

--------------------------------------------------------------------------------------------------

1 The document should more carefully describe its scope (and consider
  changing its title). This document focuses on the use of SIP for
  simple telephony and relies on measurements in earlier telephony
  networks for guidance.  But telephony is only one use of SIP. These
  aren't the same metrics that would be most useful for observing a
  network that was involved primarily in setting up MSRP sessions for
  file transfer, for instance. A eventual set of generic SIP
  performance metrics will need to focus on the primitives rather than
  artifacts from any particular application.

2 That said, I'm skeptical of the utility of many of these metrics even
  for monitoring systems that are focusing only on delivering basic
  telephony. Has the group surveyed operators to see what they're
  measuring, what they're finding useful, and what they're just
  throwing away? Some additional text motivating why this particular
  set of metrics were chosen should be provided to help
  operators/implementers choose which ones they are going to try to  
use.

3 "Each session is identified by a unique Call-ID" is incorrect. You
  need at least Call-ID, to-tag, and from-tag here. And to be pedantic,
  you're describing the SIP dialog, not one of the sessions it manages.
  The session is what is described by the Session Description Protocol.
  The metrics in this draft are derived from signaling events, not
  session events, and is making assumptions about how those correlate
  for a simple voice call that may not be true for more advanced uses.

4 The document is inconsistent about whether the metrics will describe
  any part of an early-dialog/early session. The introduction indicates
  it won't and focuses on the delivery of a 200 OK, but there are
  metrics that measure the arrival time of 180s. This should be
  reconciled. Do take note that early sessions are pervasive in real
  deployments at this point in time.

5 These metrics are intentionally designed to not measure (or be
  perturbed by) the hop-hop retransmission mechanisms. This should be
  made explicit. There should also be some discussion of the effect of
  the end-to-end retransmission of 200OK/ACK on the metrics based on
  those messages.

6 The document should consider the effects of the presence or absence
  of the reliable-provisional extension on its metrics (some of the
  metrics will be perturbed by a lost 18x that isn't sent reliably).

7 Using T1 and T4 as the timing interval measurement tokens is
  unfortunate. SIP uses those symbols already to mean something
  completely different. Is there a reason not to change these and avoid
  the confusion that the collision will cause?

8 The document uses the terms UAC and UAS incorrectly. It is trying to
  use them to mean the initiator and recipient of a simple phone call.
  But the terms are roles scoped to a particular transaction, not to a
  dialog. When an endpoint sends a BYE request, it is by definition
  acting as a UAC.

9 The document uses the word "dialog" in a way that's not the same as
  the formal term with the same name defined in RFC3261 and that will
  lead to confusion. (A sequence of register requests and responses,
  for example, are never part of any dialog. The INVITE/302/ACK
  messages shown in the call setup flows are not part of any dialog.)
  Please choose another word or phrase for this draft. I suggest
  "message exchange".

10 The 3rd to last paragraph of section 4 should be expanded. I think
  it's unlikely that implementers, especially those with other language
  backgrounds,  will understand the subtlety of the quotes around
  "final".  Enumerating the cases where you want the measurement to
  span from the request of one transaction to the final response of
  some other transaction will help. (I'm guessing you were primarily
  considering redirection, but I suspect you also wanted to capture the
  additional delay due to Requires-based negotiation or 488
  not-acceptable-here style re-attempts?). You may also want to
  consider the effect of the negotiation phase of extensions like
  session-timer on these metrics.

11 The document assumes that a registration will be DIGEST challenged.
  That's a common deployment model, but it is not required. If other
  authentication mechanics are used (such as SIP Identity), the RRD
  metric, for example, becomes muddied.

12 In section 4.2, "Subsequent REGISTER retries are identified by the
  same Call-ID" should say "identified by the same transaction
  identifier (same topmost Via header field branch parameter value".
  Completely different REGISTER transactions from a given registrant
  are likely to have the same Call-ID.

13 The SRD metric definition in 4.3.1 ignores the effect of forking.
  Unlike 200 OKs, where receiving multiple 200s in response to a single
  INVITE only happens if a race is won, it is the _normal_ state of
  affairs for a UAC to receive provisional responses from multiple
  branches when a request forks. Deployed systems are increasingly
  sending 18x responses reliably with an answer, establishing early
  sessions, so when forking is present it is _highly_ likely that there
  will be multiple 18x's from different branches arriving at the UA.
  This section should provide guidance on what to report when this
  happens.

14 The Failed Session Setup SRD claims to be useful in detecting
  problems in downstream signaling functions. Please provide some text
  or a reference supporting that claim. As written, this metric could
  be dominated by how long the called user lets his phone ring. Is that
  what was intended? You might consider separate treatment for 408s and
  for explicit decline response codes.

15 What was the motivation for making MESSAGE special in section 4.3.3.
  Why didn't the group instead extend the concept to measuring _any_
  non-INVITE transaction (with the possible exception of CANCEL)?

16 In section 4.4, what does it mean to measure the delay in the
  disconnect of a failed session completion? Without a successful
  session completion, there can be no BYE. This section also begs the
  very hard to answer question about what to do when BYEs receive
  failure responses. It would be better to note that edge-case exists
  and what, if anything, the metric is going to say about it if it
  happens.

17 Section 4.5 is a particularly strong example of these metrics
  focusing on the simple telephony application. It may even be falling
  into the same traps that lead to trying to build fraud-resistant
  billing based on the time difference between an INVITE and a BYE.
  Some additional discussion noting that the metric doesn't capture
  early media and recommendation on when to give up on seeing a BYE
  would be useful. (Sometimes BYEs don't happen even when there is no
  malicious intent.)

18 Trying to use Max-Forwards to determine how many hops a request took
  is going to produce incorrect results in any but the most simple of
  network deployments (I would have expected this to be based on
  counting Vias with a note pointing to the discussion on the problems
  B2BUAs introduce). Proxies  can reduce Max-Forwards by more than one.
  There are many implementations in the wild that cap Max-Forwards. If
  this metric remains as defined, you should also point out that
  neither endpoint can calculate it. Some third entity will have to
  collect information from each end to make this calculation.

19 The ratio metrics don't define (or convey) the interval that totals
  are taken over. Are these supposed to be "# requests received since
  this instance was manufactured' or "since last reboot" or "since last
  reset of statistics" or something else? What is the implementation
  supposed to report when the denominator of a ratio is 0?

20 Please add some discussion motivating why all 300s, 401, 402, and 407
  are treated specially (vrs several other candidate 4xx and 6xx
  responses) in sections like section 4.8. Were other codes considered?
  If so, why were they rejected?

21 Section 4.9 seems to be implying that you can't receive a 500 class
  response to a reINVITE which is not true. If you want this metric to
  only reflect the results of initial INVITEs, more definition will be
  needed.

22 ISA in section 4.10 claims that 408s indicate an overloaded state in
  a downstream element. Overload may induce 408s, but 408s do _not_
  indicate overload. Its possible to receive them just because someone
  is not answering a phone.

23 In section 5, why where these correlation dimensions chosen. Was the
  Request-URI considered? If so, why was it rejected?

24 The treatment of forking in section 6.3 is insufficient. As noted
  earlier, provisional messages establishing early sessions is becoming
  common, and there will be multiple early sessions for a given INVITE
  when there is forking. The recommendation to latch onto the "first"
  200 (or 18x) and ignore the others only marginally works for playing
  media for simple telephony applications - we're seeing phones that
  mix or present multiple lines, and applications that go beyond basic
  phone calls (like file transfer) that make use of all the responses.
  Trying to dodge the complexity as the current section does will lead
  to metrics that don't reflect what the network is doing.

25 I'm a little surprised there is no discussion on privacy,
  particularly on profiling the usage patterns of individuals or
  organizations, in the security considerations section.

26 Nits:
    26.1 What does it mean in section 4.3.1 for the "user" to send the
      first bit of a message? Suggest deleting "or user" from the
      sentence.
    26.2 Section 4.11 has a stale internal pointer to a non-existant
      section 3.5 I suspect it's trying to point back into 4 somewhere.