[18:15:24] --- bmwgkd has joined
[18:15:31] --- bmwgkd has left
[18:33:50] --- mattz has joined
[18:33:58] * mattz has changed the subject to: IPPM Working Group
[18:35:14] --- pfc has joined
[18:35:28] <mattz> Wanna scribe? Or take Jabber notes?
[18:35:31] <mattz> :)
[18:35:41] <pfc> I can try to scribe
[18:36:04] <pfc> I usually take notes anyway. People have to speak clearly though.
[18:36:07] <mattz> every little bit helps, thanks
[18:36:08] <pfc> Can'
[18:36:54] --- lars has joined
[18:40:59] --- lars has left
[18:43:58] --- lars has joined
[18:43:59] --- jishac has joined
[18:44:34] <mattz> we're starting
[18:47:11] <mattz> agenda bashing
[18:47:13] <mattz> al to mike
[18:47:45] <mattz> new standards body, IPTV interoperability forum, very interested in metrics. Al gave a talk, they are still interested, can say a few words at the end
[18:47:53] <mattz> "ok"
[18:48:18] <mattz> status of drafts
[18:49:13] --- simon.leinen has joined
[18:49:52] --- acmacm has joined
[18:51:35] <mattz> capacity draft: mail from joe yesterday afternoon; comments incorporated but didn't make it to the ietf cutoff; version will go out just after IETF
[18:51:53] --- wouter has joined
[18:51:54] <mattz> henk: read draft, think it needs more review
[18:52:19] <mattz> think things are consistent, but do we have the right set -- is everything that should be there is in there
[18:52:22] <mattz> pfc to mike
[18:52:27] <jishac> yes, please read the draft
[18:52:30] <mattz> think joe and I have answered all comments so far
[18:52:48] <mattz> so if people have comments, or think needs more stuff, please say so
[18:53:30] <mattz> please comment, even if you think it's fine, so we can see that it has been reviewed
[18:53:38] <mattz> milestone status
[18:54:20] <mattz> end of status. next is al on composition framework
[18:54:25] <mattz> joe, can you hear OK?
[18:54:44] <jishac> yes
[18:54:53] <mattz> great
[18:55:16] <mattz> slide fumbling...
[18:55:20] <mattz> (our fault)
[18:56:26] <mattz> ok, off we go
[18:56:59] <mattz> al morton, framework for metric composition
[18:58:17] --- wouter has left: Logged out
[19:03:01] --- dahmna has joined
[19:03:59] --- dahmna has left
[19:08:01] <mattz> emile is going to comment
[19:08:35] <mattz> discuss point related to slide #6
[19:08:49] <mattz> if look at last line, have spatial metric
[19:08:58] <mattz> with S2 have measurment for a segment of a path
[19:09:11] <mattz> ocnsider that S2 might be a subpath measurement of the top line
[19:09:26] <mattz> al: think s2 is from source to end of segment 2
[19:09:38] <mattz> emile: could be either. start to end of seg 2, or just over segment 2
[19:09:45] <mattz> S2 may be measured using passive techniques
[19:10:06] <mattz> decide if we make a definition for passive metric, to have sub-path metrics be computed
[19:10:27] <mattz> [mjz: I believe there has been no passive measurements produced so far]
[19:10:50] <mattz> question is: is it acceptable to have sub-path measurement made by passive means
[19:11:13] <mattz> passive on real traffic
[19:11:34] <mattz> henk: m1,m2,m3 - one with active, one with passive, one with active
[19:11:49] <mattz> issue is what you are measuring with passive ones. can do one-way delays with passive methods
[19:12:06] <mattz> but passive iwth active, can't add them up, apples and oranges?
[19:12:19] <mattz> emile: issue is that for definition of spatial metrics had this case. not active
[19:12:34] <mattz> even if test packet is intrusive
[19:13:08] <mattz> al: comes to general issue of passive measurments. don't control sending discipline, at mercy of whatever the stream looks like in terms of sampling, performance
[19:13:24] <mattz> it's probably a reasonable sized effort to talk about comparing active and passive.
[19:13:36] <mattz> emile: but case thinking about... source is active, and we control.
[19:13:46] <mattz> and observe the packet as it goes along the way
[19:14:02] <mattz> al: so if passively measuring controlled source, it's really active measurement
[19:14:24] <mattz> mathis: comparing active and passive... both have a problem, active measurments have heisenberg probs
[19:14:30] <mattz> passive - don't have control over sampling
[19:14:41] <mattz> so if particular metric, loss rate, same no matter how metric.
[19:14:51] <mattz> if it's not, then idea of composition falls apart
[19:14:57] <mattz> but to call it loss rate, drives toward that ida
[19:15:11] <mattz> we wrote framework draft... this is one of the thigns we envisioned but had no idea how to do
[19:15:15] <mattz> cool that it is happening
[19:15:31] <mattz> but deep traps in here for some metrics... bulk capacity... jitter in M1 could change bulk capacity on M3
[19:15:39] <mattz> loss rate, simple jitter, simple delay, right way to start.
[19:15:49] <mattz> but don't be suprised if there are some things we don't understand
[19:16:04] <mattz> al: when talked in paris, talked about deliberately excluding reordrering from this
[19:16:22] <mattz> mathis: better strategy to observe that some have special challenges, reorderin gis one, BTC another
[19:16:36] <mattz> end of questions... moving to next talk
[19:17:12] <mattz> al morton: spatial composition draft
[19:17:22] <mattz> "one of the blue drafts in the heirarchy from the last talk"
[19:21:59] <mattz> I believe Al said that he was working on some metrics for composing jitter or delay variation, but found out that one was potentially IPR encumbered, so he didn't include any of them yet.
[19:22:39] <mattz> open issues section
[19:22:42] <mattz> emile to mike
[19:23:04] <mattz> shocked when first read composition draft. if receive stream of owd, 80% with infinite delay
[19:23:12] <mattz> then will receive 20% of packets.
[19:23:30] <mattz> if make some sort of aggregation, result could be excellent, even if only receive very few packets
[19:24:02] <mattz> idea of combination metric is that you need to know that most packets are not arriving
[19:24:07] <mattz> could lead to lots of misunderstanding.
[19:24:22] <mattz> roman: didn't read draft, but listening to emile
[19:24:45] <mattz> if combine loss and delay get some kind of index. but use of index itself, would lead to misinterpretation.
[19:24:55] <mattz> what is reason for this.
[19:25:09] <mattz> very sensitive to have index that combines basic metrics into one composit
[19:25:16] <mattz> didn't see metric, no experience, just word of warning
[19:25:20] <mattz> stanislav to mike
[19:25:24] <mattz> have not read definition
[19:25:35] <mattz> in what units do you measure the combination of loss and delay
[19:25:43] <mattz> al: measured as couple, loss and delay reported together
[19:25:59] <mattz> some people could argue that if indicate not available, or something in delay posn, that could be inferred as loss
[19:26:23] <mattz> stanislav: loss is specal case of delay. a large one greater than timeout
[19:26:33] <mattz> if number is there, what is point of extra bit, can compute it
[19:26:50] <mattz> emile: yeah, but in the way define compoistion you discard it. descard infinitie delays
[19:26:58] <mattz> al: with finite delay metric that is true
[19:27:21] <mattz> emile: tyring to compose loss & delay. m1,m2,m3 for delay. l1,l2,l3 for losses
[19:28:09] <mattz> if compose, good delay with 80% loss.... if do with separate measure, and don't export # loss, would lead to a lot of confusion
[19:28:29] <mattz> al: set of valid delays plus total packets in sample is sufficient to determine number lost
[19:29:00] --- raviraj has joined
[19:29:04] <mattz> with some add'l info, may not need combo metric
[19:29:24] <mattz> ? with reporting section, can remove combination
[19:29:28] <mattz> phil c to mike
[19:29:39] <mattz> when computing avg delay on all packets that arrive, reducing sample space
[19:29:57] <mattz> conditioning on arrival. not avg on whole space, is conditional avg. when understand that, then understand only getting part of the picture.
[19:30:14] <mattz> think determination is made in original defn to make infinite, because can't pick finite number and get sensible avg.
[19:30:17] <mattz> still think that makes sense to me
[19:30:21] --- raviraj has left
[19:30:27] <mattz> only compute delay on packets that actually do arrive
[19:30:51] <mattz> if report metric, report all the circumstances of the metric, which should do anyway as scientist, then get proper understanding. if just put single number on web page are misleading
[19:30:56] <mattz> stanislav
[19:31:29] <mattz> phil almost said what I want to , but when you cut off tail of distribution, the property of commutivity; addition of independant variables to take expectation no longer holds
[19:31:46] <mattz> al: could argue about if infinitely delayed or not
[19:31:57] <mattz> stanislav: not relevant; if cut off tail, no longer possible to add averages
[19:32:27] <mattz> it's a nice property to be able to add averages. classical stats is nice in that way. but are some disadvantages. if drop one item from sample, the prpoerty no longer holds, and can be off very significantly
[19:32:34] <mattz> because classical statistics are not robust
[19:32:40] <mattz> should use robust statistics, or drop nothing.
[19:32:49] <mattz> an unpleasant reality, but mathematical fact.
[19:32:56] <mattz> al: would like to take this particular one off line.
[19:33:13] <mattz> would argue that loss is not necessarily infinite delay
[19:33:17] --- bmwgkd has joined
[19:33:23] <mattz> stanislav: loss does not mean that pakcet has not arrived?
[19:33:26] <mattz> al: yes
[19:33:40] <mattz> stanislav: then delay is at least greater than timeout. that does not mean that it is necessarily infinite.
[19:33:55] <mattz> remains a degree of indeterminacy. don't know ... maybe it'll show up in 50 yrs
[19:34:17] <mattz> do know that delay is at least timeout, and maybe infinity
[19:34:42] <mattz> al: to me, that means that when defined timout, defined a portion of the tail, the distribution, that we are not really interested in
[19:35:06] <mattz> stanislav: that's fine. but once done it, no longer possible to claim that you can add two independent variables and expect that the mean of hte sum is the sum of the means
[19:35:37] <mattz> (paraphrasing): risk of huge error.
[19:35:49] <mattz> al: good point, let's take offline and not take up WG time
[19:35:57] <mattz> henk: take to list.
[19:36:02] <mattz> would you post some pointers
[19:36:20] <mattz> emile: similar for me, but enhance the form of user defined (?) metrics
[19:36:38] <mattz> [mjz: how did I get that character ? ? ? ]
[19:37:07] <mattz> ...formalize more output. result+context, then can make application of context to avoid decision
[19:37:22] <mattz> al: think power in what suggest, but have more to do here to satisfy stanislav's concern
[19:37:28] <mattz> would like to do some f2f this week
[19:37:55] <mattz> back to open issues... multicast metrics (would like to defer until unicast is done) and "decomposition"
[19:38:58] <mattz> no other comments in room
[19:39:10] <mattz> emile is next, mulitmetrics one
[19:50:55] <mattz> al has comment on slides
[19:51:09] <mattz> either add whole section into framework on reporting, or try to put together good solid framework for reporting out metrics
[19:51:46] <mattz> emile: two points. 1. clear what reporting ; which metrics definition. not just delay but in detail about what's there.
[19:51:53] <mattz> ? something about inverse percentile
[19:52:53] <mattz> other point: how to get this information from another provider
[19:52:58] <mattz> al: so that's data format
[19:53:09] <mattz> emile: have registry, have to compute, complete
[19:53:50] <mattz> if want to exchange information, can use pointers to fields in ipfix data model...
[19:54:14] <mattz> al: sounds to me points like to reflect into the individual drafts, like spatial comp draft, call out metric ids
[19:54:38] <mattz> emile: either point to section of rfc or
[19:55:06] <mattz> ...
[19:55:20] <mattz> draft should have kind of table. if want composition of minimum delay, here are combinations you can use
[19:55:26] <mattz> otherwise strange results
[19:55:53] <mattz> easy to add new id in registry
[19:57:21] <mattz> jurgen up next on traceroute stuff
[20:05:23] --- pfc has left: Replaced by new connection
[20:05:26] <mattz> henk: one comment, speaking as a user, should include as #s.
[20:05:37] <mattz> that's often more important than IP addrs
[20:05:44] <mattz> carter: what is soln for millisecond thing
[20:05:49] <mattz> don't knw yet
[20:05:58] <mattz> carter: could make time resolution better.
[20:06:06] <mattz> j: one option. could be special value
[20:06:20] <mattz> carter: any issues making this a hard issue?
[20:06:29] <mattz> j: no problem, just want to make sure discuss first.
[20:06:39] <mattz> carter: so no roadblock, just something not finished yet.
[20:07:16] <mattz> henk: anyone that does not think pick up as a WG doc
[20:07:19] <mattz> nope
[20:07:31] <mattz> next version, WG doc. will try to finish sooner rather than later
[20:08:16] <mattz> next up: al on jitter metric comparison
[20:08:19] --- pfc has joined
[20:09:32] --- simon.leinen has left
[20:10:19] <mattz> first pic: basically inter-packet dleay, difference from last packet
[20:10:43] <mattz> on bottom, showing packet 4, using packet 3 instead
[20:11:04] <mattz> 1,3,5 have same delay
[20:11:07] <mattz> 2 is somewhat short
[20:11:11] <mattz> 4 is longer
[20:12:25] <mattz> second one, packet delay distn. emphasize OWD.
[20:12:30] <mattz> everything relative to packet with min delay
[20:12:54] <mattz> normalizing delay distn to minimum
[20:13:21] <mattz> two definitions roman described as in common use
[20:13:45] <mattz> (back to text slides)
[20:16:53] --- pfc has left: Replaced by new connection
[20:19:41] --- pfc has joined
[20:19:57] <mattz> alan clark at mic
[20:20:07] <mattz> other jitter metrics, for example MAPDV (?)
[20:20:12] <mattz> ah, ( ? )
[20:20:18] <mattz> short term avg of delay
[20:20:23] <mattz> keep deviations from that short term avg
[20:20:29] <mattz> envelope of jitter around short-term moving average
[20:20:44] <mattz> nice that keeps track of own reference point; also emulating (simplistically) how a jitter buffer would work
[20:20:58] <mattz> found in practice that MAPDV correlates with discard rates in jitter buffers
[20:21:24] <mattz> second thing: observation PDV metric. there is a philosophical point. when measuring, what are trying to measure
[20:21:39] <mattz> if one on time, +50ms then on time the +50ms
[20:21:52] <mattz> but what if 0 0 +50ms +50ms is that 50ms of jitter or not
[20:22:04] <mattz> first metric, PDV, report half delay measurent. 25ms
[20:22:10] <mattz> if 3 on time, then 3 late, report a third
[20:22:20] --- gdweber has joined
[20:22:25] <mattz> think that ones looking at absolute PDV would always report peak-to-peak
[20:22:55] <mattz> in case of jitter buffers want max.
[20:23:15] <mattz> people also do tend to smooth a bit
[20:23:35] <mattz> al: shoing pitfall in measuring these... is how summarize. if summarize in right way, can tell jitter buffer
[20:23:47] <mattz> alan: not sure if there is a clear definition of what it is.
[20:23:55] <mattz> 0 0 50 50 0 0 50 50 what should you report.
[20:24:01] <mattz> don't think that's been defined anywhere.
[20:24:20] <mattz> al: right, that's why I am raising the q. how are you using this. if buffer, range, or approx of range is most helpful
[20:24:43] <mattz> stanislav: important to note that both of these transforms remove one degree of freedom. sample of N packets, end up with N-1 numbers
[20:25:22] <mattz> not much; could reconstruct with missing add'l info
[20:25:37] <mattz> the value would like in further reduction of degrees of freedom... typically to one.
[20:25:50] <mattz> more specific point:
[20:26:08] <mattz> regardless of which purposes of the measurement; the metrics can have certain properties
[20:26:20] <mattz> for example, one might find desirable to have a metric invariant wrt sampling frequency
[20:26:44] <mattz> sample 1/sec, then to 10/sec, and if relevant phenomena are on scales of 100sec, then want to produce same numbers, or preferably same number
[20:27:12] <mattz> if look at first form of delay variation, if increase freq by facdtor of 10, decrease #'s by factor of 10. network behaving better?
[20:27:24] <mattz> believe that is clearly undesirable
[20:27:35] <mattz> al: the negative excursions are limited to packet spacing unless you have reordering
[20:28:01] <mattz> what get in distn is related to original spacing
[20:28:17] <mattz> stanislav: have other examples of perverse behavior of first one.
[20:28:45] <mattz> this is not the first group to encounter problem of estimating variances of different quantities
[20:29:04] <mattz> problem approached before, solutions to problem, generally in wide use. might make sense to look at those as well.
[20:29:07] --- lars has left: Logged out
[20:29:09] <mattz> henk: make suggestions on list
[20:29:24] <mattz> rod pashby, atm has Cell Delay Variaion
[20:29:33] <mattz> carter: but not measured in the same way
[20:29:55] <mattz> carter: unhappy about use of word jitter. variance on any expected observation
[20:30:03] <mattz> and at least 40 or 50 potential metrics where very important
[20:30:16] <mattz> don't use jitter consistently in here
[20:30:28] <mattz> any opportunity to eliminate term "jitter" is a good one.
[20:30:49] <mattz> a lot of other variation important, not just this one
[20:31:00] <mattz> in atm, not way to measure src and receive time, so CDV is a singl epoint meas
[20:31:06] <mattz> cell arrival rate variation, not delay variation.
[20:31:15] <mattz> even in the industry, lots of problems trying to define.
[20:31:26] <mattz> be a bit more elaborate in definitions and less use of jitter is important
[20:32:05] <mattz> [back to slides]
[20:32:38] <mattz> separate talk, sorta composition spatially
[20:33:10] <mattz> showing some actual lab experiments
[20:33:22] <mattz> red boxes are congesting T1s
[20:33:29] <mattz> also some BB stuff that wasn't shown
[20:33:36] <mattz> next slide shows measurements
[20:33:49] <mattz> cdf is superimposed
[20:34:10] <mattz> core wasn't loaded
[20:34:18] <mattz> edges look similar, uniform
[20:34:35] <mattz> see classic S-curve CDF and triangle, when convolving two uniform distns
[20:34:52] <mattz> compared methods to take small amt of info from distns
[20:34:59] <mattz> and compute 99th percentile of composed one
[20:35:26] <mattz> see table of results
[20:35:33] --- jishac has left
[20:35:44] --- gdweber has left
[20:35:48] <mattz> heuristic composition of percentiles... worked well here, but doesn't work well for multiple nets
[20:35:58] <mattz> not looking at that any more
[20:36:11] <mattz> next time draft, have details on approximate convolution, and est from 3rd column
[20:36:36] <mattz> did 6 experiments like this with different delays
[20:36:42] <mattz> some got correlated, and then things fell apart
[20:37:11] <mattz> with this in mind, packet delay variation is the only one that works. inter-packet delay variation doesn't work
[20:37:20] <mattz> PDV seems to serve best
[20:37:35] <mattz> next up, Roman on "initial considerations for IPPM burst metric"
[20:37:39] <mattz> willb e brief.
[20:38:04] <mattz> from discussion with operation team, video and voice
[20:38:19] <mattz> not so easy. talked to vendors, everyone had different concept of what it is
[20:38:27] <mattz> thought this might be right place to discuss
[20:38:46] <mattz> first thing is to define what burst is. diagram at bottom is what operations presented
[20:39:48] <mattz> [lots of text on first slide]
[20:40:33] <mattz> looked around, find loss patterns, and stuff, but no burst patterns
[20:40:40] <mattz> looked around in literature, listed on second slide
[20:41:04] <mattz> found in literature, but no comparision, justification
[20:41:20] <mattz> if decide to work on it, must define phenomena.
[20:41:25] <mattz> no agreement on what burst is
[20:41:32] <mattz> [mjz: similar to reordering...]
[20:41:45] <mattz> need a metric, and way to operationalize
[20:42:00] <mattz> did some legwork before meeting. colleagues, got positive feedback. open to discussion.
[20:42:09] <mattz> do we want to work on it, does it make sense, what is opinion.
[20:42:19] <mattz> phil c to mike:
[20:42:27] <mattz> correctly went back and looked at literature
[20:42:43] <mattz> really a property of the interrival time distn, rather than phenomena on its own
[20:43:00] <mattz> not sure that it makes sense to measure as a phenomenon apart from measuring characteristics of interarrival time disn
[20:43:05] <mattz> distn
[20:43:31] <mattz> when will leland and his colleagues did thier work, since then, various characteristics related to hurst parameter, and variations, have been what was used for burstiness
[20:43:47] <mattz> characteristics on all time scales of interarrival distn
[20:43:54] <mattz> not sure it makes sense to measure independently.
[20:44:02] <mattz> roman: way of reporting rather than measurement
[20:44:24] <mattz> pfc: in order to characterize burstiness, have to est. parameters of interarrival distn. that's measure of burstiness
[20:44:26] <mattz> carter to mic
[20:44:35] <mattz> burst is an interesting sideline to a more fundimentla metric
[20:44:47] <mattz> inverse of burst might be more interesting. lack of burstiness ...
[20:44:59] <mattz> rate limiting link, or other phenomena that controls traffic
[20:45:06] <mattz> burstiness may be baseline behavior of interest
[20:45:12] <mattz> deviations from bursty norm might be more interesting.
[20:45:29] <mattz> must continue on list. got a feeling intersted to work on it.
[20:45:42] <mattz> roman: if interested, exchange emails and see how far we can go if anywhere.
[20:45:53] <mattz> al, briefly on new group.
[20:46:08] <mattz> new forum: IPTV interoperability forum. ATIS committee meeting this week in LosVegas
[20:46:12] <mattz> doing multicast performance
[20:46:17] <mattz> what a lot of IPTV is based on
[20:46:21] <mattz> at least live broadcasts
[20:46:31] <mattz> our multimetrics draft, with some one-to-group metrics would be very interesting to them.
[20:46:41] <mattz> they are also looking at comparing value of active meas, pasv meas
[20:46:48] <mattz> in large streams, continuous traffic
[20:46:58] <mattz> no real need for add'l test traffic. lots of data
[20:47:09] <mattz> if definitions applicable in passive world, re-use that way
[20:47:26] <mattz> group of folks that are willing to make use of whatever we can provide, if developed on thier time scales
[20:47:30] <mattz> will also work on specific things
[20:47:33] <mattz> igmp join time
[20:47:40] <mattz> channel change time, channel zapping time...
[20:47:55] <mattz> willing to reuse what makes sense. good to know plenty of users out there for multicast things
[20:48:05] <mattz> henk: post pointer to ML, please
[20:48:29] <mattz> we be done
[20:49:18] --- pfc has left
[20:49:56] --- mattz has left
[20:58:08] --- acmacm has left