CURRENT MEETING REPORT

Reported by Jim McQuaid, Bay Networks

Minutes of the Benchmarking Methodology Working Group (bmwg)

The meeting began with a brief discussion of the agenda. There were four principal sections to the meeting, as detailed below.

1. Organizational

Jim McQuaid is resigning as co-chair for this half of the BMWG before the next IETF meeting. Kevin Dubray will be taking over as the co-chair.

2. Status of the Network Element Draft (router test document)

The small changes discussed at the last meeting and raised on the list have been implemented and the 03 version of the draft has been posted as an Internet Draft. A last call was issued on the BMWG list.

The chair summarized the three changes and asked for discussion. There was no additional comment or discussion and the consensus is to forward the document next week.

3. Ethernet switch testing methodology draft

A large portion of the meeting was devoted to discussing the current version of the Ethernet switch testing methodology draft. Both the authors were present. The following is a synopsis of the discussion. In general, matters of merely editorial correction were not discussed because the draft is still in a very early stage.

Note: the 00 version of the draft is currently posted in IETF directories. The 01 version was mailed to the BMWG list and contains some additional material. In addition, two sections were accidentally left out of the 01 version (dealing with latency and behavior in abnormal conditions). The next version, reflecting the changes from the meeting discussion, version 02, will be posted in the near future.

• 3.1 Specific comments
  

In section 1, Introduction, it was noted that there are interesting editorial comments on the state of the industry which should probably be deleted in the future.

In section 6, Device set up, it was agreed that the first requirement (The device MUST be in a stable state) cannot be modified by a parenthetic SHOULD and that the parenthetic statement ought to read MUST.

Section 13, Bursty Traffic fostered a wide-ranging general discussion.

A number of comments were made about Appendix A, Testing Considerations. Several practical recommendations made at various points in the draft should be collected under Appendix A, for example, the observation at the very end of section 16.1.2, The load of the test traffic, which starts out, Experience shows that . . .

• 3.2 General discussion: Burstiness
  

The WG had failed several times in the past to agree on any particular traffic characteristics as normal for test purposes. However, there was agreement about burstiness as follows.

a) Data traffic is bursty.

b) Test loads should be bursty as one possible scenario, but . . .

c) there is no consensus yet on what kind of (artificial) test load is an adequate simulation of bursty traffic.

Bob Mandeville asserted that bursty loads will reveal frame loss at a lower overall load than steady-state traffic for some devices and that this knowledge represents useful information about the DUT, given the bursty nature of real traffic.

McQuaid pointed out that this was very similar to the back-to-back test defined already, although this draft added several additional parameters. It became clear that we needed a definition for burst. In fact (noted after the meeting adjourned) there is an implicit definition in RFC 1242, under the discussion of back to back. The Ethernet switch draft, in effect, specifies multiple back to back tests, separated by specific intervals. It may be helpful to use the language of RFC 1242 in this definition. At least three aspects of bursty loads were itemized in the meeting.

1. Number of packets with minimum inter-packet gap

2. Size of packets in a burst

3. Recovery time after the first observed packet loss

• 3.3 MAC vs. Port vs. Address
  

Another major area of discussion concerned the separation and characterization of test issues pertaining to characteristics of the media access behavior, the port distribution of traffic across the switch fabric and the handling of multiple addresses per switch port.

The new draft includes a mini-procedure for determining the backoff algorithm of the DUT, based on experience that some switches do not adhere to the IEEE standard in order to achieve better performance.

In addition, there are still some questions to be resolved regarding the simultaneous sending and receiving of traffic on Ethernet at high loads.

The port issues are fairly clear and a diagram offered by Bob Mandeville (not reproduced here) illustrates what is explained in the draft.

The address handling issues, likewise illustrated by a complex diagram (not included here) seem reasonably clearly defined.

It became clear that the definition of bi-directional (and uni-directional) and the use of the term multidirectional were a source of confusion. There is, at a minimum, a need to clarify the direction of frames on a given Ethernet with a tester and DUT talking and the destination / direction of frames within a given stream which may have a mixture of MAC addresses.

MAC layer issues should be cleanly isolated from the balance of methodology since testing of Token Ring switches and even ATM switches could build on the non-MAC-specific parts of the document if it is modular.

• 3.4 Other Issues mentioned
  

Since latency was removed (accidentally) from this draft, there was mention that the latency of broadcast frames was the most telling metric.

There was a question raised about testing for switch fairness, i.e. a switch architecture which tended to favor one port over others. Reporting the results on a per-port as well as an aggregate basis should provide the information to determine this.

• 3.5 Future Actions
  

The authors will make a major revision of the draft with the fundamental goal of separating the various issues identified and defining each and the utility of the metrics associated with each. After each parameter has been clearly identified, the synthesis of complete testing procedures should be much easier to understand.

4. Call Setup / ATM SVC Interest Area

A smaller group met to discuss plans and possibilities in this area. After a round of general discussion in an attempt to understand the concerns and issues offered by the group and under discussion in other industry groups, it was agreed that Kevin Dubray will lead the effort to create a charter document for this particular work.

The charter will be circulated to the BMWG list soon and after some discussion, shared with ATM Forum groups working in related areas.

The rough shape of this charter is as follows:

1) We are interested in some scope of ATM switch testing. This scope must be defined but has to do with determining the overhead of setting up connections in a WAN and in a LANE environment. There are issues at the cell level, at the message, semantic or signaling level and at the frame level which must be clarified.

2) The metrics should describe the performance / overhead of:

• connection setup,
• data transfer after setup and
• connection teardown
  

The group will not attempt to suggest or define acceptable levels of performance, as is done in the telephony world, but rather methods for deriving meaningful performance numbers.