Jan and Jeff,[I started with wanting to respond to both Jeff Byzek's and Jan Novak's e-mails separately, but didn't want to say "look at my other response" :-) So this is a combined response to both Jan and Jeff]
I concur that there can be a bit of complexity in verifying the flows and I'm interested in how the draft evolves. I understand that the draft is intended to measure performance and not necessarily standards compliance.
I also admit that I'm stuck on wanting to confirm the quality of the flow data in some manner before stating that a specific benchmark value was achieved (summarizing a paragraph I deleted in this e-mail where I essentially stated a subtle variation of what I've said in earlier e-mails :-) ).
As a compromise, perhaps the draft could state a mechanism whereas measurements values for which the accuracy of the flow data is somehow confirmed (such as in the DUT CPU) should be suffixed "with verified flow data" (or something like that). Measurement values for which flow data was not confirmed should be suffixed "without verified flow data"
I won't be in Dublin, but I am planning on attending the meeting in Minneapolis. I will attempt to attend the Dublin BMWG meeting virtually, I just won't be in the hallways (acknowledging that the jabber "hallway" isn't quite the same thing :-) ).
Bill On Jul 17, 2008, at 6:16 AM, Jan Novak (janovak) wrote:
Hi,I just think that making statements about a router's flow handling and reporting capabilities aren't complete if it can't be verified that the flow data reported is accurate.The draft covers it to certain extent - checking the flow monitoring counters to make sure that the expected flow conditions are met - e.g. the network device reacts to and registers all the flows. Something in terms of what you suggest is certainly possible, but the biggest concern we always met was "what is flow monitoring performance and where (which device in the network) can I afford to enable it". The accuracy of the data comes always as second (when you know you can actually do it :-) at all). It would be possible to check the individual flows counters on the network device in most of the cases described in the draft (but not with cache overflow) but not as part of the performance measurement since such a check would heavily load the CPU when collecting the necessary outputs. May be we can discuss this in Dublin ?? I am not in the position to suggest anything regarding external collectors since I am not familiar with them and never tested them and would like to keep this draft scope to just the network devices. Jan The climate of Edinburgh is such that the weak succumb young .... and the strong envy them. Dr. Johnson-----Original Message----- From: Bill Cerveny [mailto:bill at wjcerveny.com] Sent: 17 July 2008 03:53 To: Jan Novak (janovak) Cc: bmwg at ietf.org Subject: Re: [bmwg] [answer to Bill Cerveny] RE: New Version Notification for draft-janovak-bmwg-ipflow-meth-00 [answer to Bill Cerveny] Comments below ... Bill On Jun 11, 2008, at 9:43 AM, Jan Novak (janovak) wrote:Hi Bill, I am sorry, was on holidays and missed your answer ....devices, the throughput measurement wouldn't beappropriate. Perhapsthe scope of this document should be specifically onnetwork devicesthat transit network traffic?Yes, the intention was to make it specific for only network devices. Will review the text to make sure it is clear.OKWouldn't a big consideration be the quality and accuracyof the flowdata? How would the accuracy of the generated flow beevaluated?Theflow generating capabilities of a network device is certainly less useful if the device's flow accuracy degrades when it is transiting more traffic.I would consider this out of the scope since to check thatthe testingwould have to involve some external collector and the way thosehandle datais even more complex than what happens on the network devices when just registering the traffic. I will think about it yet, but based on the testing we have done I don't quite see how to do it.Couldn't you do something like send a known set of network traffic (defined by number of packets, types of packets, packet sizes, etc) across the router? If the collector's interpretation of of the flow data matches what was sent across the router, a statement could be made about flow accuracy. If the collector's interpretation of the flow data does not match what was sent across the router, then a statement of accuracy cannot be made. If the collector is unable to keep up with the flow data, perhaps another way of verifying the accuracy of the flow may be recording the flow data and playing it back at a rate that a collector could understand. I just think that making statements about a router's flow handling and reporting capabilities aren't complete if it can't be verified that the flow data reported is accurate.
_______________________________________________ bmwg mailing list bmwg at ietf.org https://www.ietf.org/mailman/listinfo/bmwg