idnits 2.17.1 draft-ietf-bmwg-benchres-term-08.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** It looks like you're using RFC 3978 boilerplate. You should update this to the boilerplate described in the IETF Trust License Policy document (see https://trustee.ietf.org/license-info), which is required now. -- Found old boilerplate from RFC 3978, Section 5.1 on line 17. -- Found old boilerplate from RFC 3978, Section 5.5, updated by RFC 4748 on line 1115. -- Found old boilerplate from RFC 3979, Section 5, paragraph 1 on line 1083. -- Found old boilerplate from RFC 3979, Section 5, paragraph 2 on line 1090. -- Found old boilerplate from RFC 3979, Section 5, paragraph 3 on line 1096. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- == No 'Intended status' indicated for this document; assuming Proposed Standard Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust Copyright Line does not match the current year -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (February 2007) is 6274 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) ** Downref: Normative reference to an Informational RFC: RFC 1242 (ref. '1') ** Downref: Normative reference to an Informational RFC: RFC 2285 (ref. '2') == Outdated reference: A later version (-20) exists of draft-ietf-nsis-ntlp-11 == Outdated reference: A later version (-18) exists of draft-ietf-nsis-qos-nslp-12 == Outdated reference: A later version (-24) exists of draft-ietf-nsis-qspec-14 Summary: 3 errors (**), 0 flaws (~~), 5 warnings (==), 7 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Benchmarking Working Group Gabor Feher, BUTE 3 INTERNET-DRAFT Krisztian Nemeth, BUTE 4 Expiration Date: August 2007 Andras Korn, BUTE 5 Istvan Cselenyi, TeliaSonera 7 February 2007 9 Benchmarking Terminology for Resource Reservation Capable Routers 10 12 Status of this Memo 14 By submitting this Internet-Draft, each author represents that any 15 applicable patent or other IPR claims of which he or she is aware 16 have been or will be disclosed, and any of which he or she becomes 17 aware will be disclosed, in accordance with Section 6 of BCP 79. 19 Internet-Drafts are working documents of the Internet Engineering 20 Task Force (IETF), its areas, and its working groups. Note that 21 other groups may also distribute working documents as Internet- 22 Drafts. 24 Internet-Drafts are draft documents valid for a maximum of six 25 months and may be updated, replaced, or obsoleted by other documents 26 at any time. It is inappropriate to use Internet-Drafts as 27 reference material or to cite them other than as "work in progress." 29 The list of current Internet-Drafts can be accessed at 30 http://www.ietf.org/1id-abstracts.html 32 The list of Internet-Draft Shadow Directories can be accessed at 33 http://www.ietf.org/shadow.html 35 Copyright Notice 37 Copyright (C) The IETF Trust (2007). 39 Table of contents 41 Abstract...........................................................2 42 1. Introduction....................................................2 43 2. Existing definitions............................................3 44 3. Definition of Terms.............................................3 45 3.1 Traffic Flow Types..........................................3 46 3.1.1 Data Flow..............................................4 47 3.1.2 Distinguished Data Flow................................4 48 3.1.3 Best-Effort Data Flow..................................4 49 3.2 Resource Reservation Protocol Basics........................5 50 3.2.1 QoS Session............................................5 51 3.2.2 Resource Reservation Protocol..........................6 52 3.2.3 Resource Reservation Capable Router....................6 53 3.2.4 Reservation State......................................7 54 3.2.5 Resource Reservation Protocol Orientation..............8 55 3.3 Router Load Factors.........................................8 56 3.3.1 Best-Effort Traffic Load Factor........................9 57 3.3.2 Distinguished Traffic Load Factor......................9 58 3.3.3 Session Load Factor...................................10 59 3.3.4 Signaling Intensity Load Factor.......................11 60 3.3.5 Signaling Burst Load Factor...........................11 61 3.4 Performance Metrics........................................12 62 3.4.1 Signaling Message Handling Time.......................12 63 3.4.2 Distinguished Traffic Delay...........................13 64 3.4.3 Best-effort Traffic Delay.............................14 65 3.4.4 Signaling Message Deficit.............................14 66 3.4.5 Session Maintenance Capacity..........................15 67 3.5 Router Load Conditions and Scalability Limit...............16 68 3.5.1 Loss-Free Condition...................................16 69 3.5.2 Lossy Condition.......................................17 70 3.5.3 QoS Compliant Condition...............................18 71 3.5.4 Not QoS Compliant Condition...........................18 72 3.5.5 Scalability Limit.....................................18 73 4. Security Considerations........................................19 74 5. IANA Considerations............................................19 75 6. Acknowledgements...............................................20 76 7. References.....................................................20 77 7.1 Normative References.......................................20 78 7.2 Informative References.....................................20 79 Authors' Addresses................................................21 80 Disclaimer of Validity............................................22 81 Copyright Notice..................................................22 82 Disclaimer........................................................22 84 Abstract 86 The primary purpose of this document is to define terminology 87 specific to the benchmarking of resource reservation signaling of 88 Integrated Services IP routers. These terms can be used in 89 additional documents that define benchmarking methodologies for 90 routers that support resource reservation or reporting formats for 91 the benchmarking measurements. 93 1. Introduction 95 Signaling based resource reservation using the IntServ paradigm [3] 96 is an important part of the different QoS provisioning approaches. 97 Therefore network operators who are planning to deploy signaling 98 based resource reservation may want to examine the scalability 99 limitations of reservation capable routers and the impact of 100 signaling on their data forwarding performance. 102 An objective way of quantifying the scalability constraints of QoS 103 signaling is to perform measurements on routers that are capable of 104 IntServ based resource reservation. This document defines 105 terminology for a specific set of tests that vendors or network 106 operators can carry out to measure and report the signaling 107 performance characteristics of router devices that support resource 108 reservation protocols. The results of these tests provide comparable 109 data for different products, and thus support the decision-making 110 process before purchase. Moreover, these measurements provide input 111 characteristics for the dimensioning of a network in which resources 112 are provisioned dynamically by signaling. Finally, the tests are 113 applicable for characterizing the impact of the resource reservation 114 signaling on the forwarding performance of the routers. 116 This benchmarking terminology document is based on the knowledge 117 gained by examination of (and experimentation with) different 118 resource reservation protocols: the IETF standard RSVP [4], NSIS 119 [5][6][7][8] and several experimental ones, such as YESSIR [9], ST2+ 120 [10], SDP [11], Boomerang [12] and Ticket [13]. Some of these 121 protocols were also analyzed by the IETF NSIS working group [14]. 122 Although at the moment the authors are only aware of resource 123 reservation capable router products that interpret RSVP, this 124 document defines terms that are valid in general and not restricted 125 to any of the above listed protocols. 127 In order to avoid any confusion we would like to emphasize that this 128 terminology considers only signaling protocols that provide IntServ 129 resource reservation; for example, techniques in the DiffServ 130 toolbox are predominantly beyond our scope. 132 2. Existing definitions 134 RFC 1242 "Benchmarking Terminology for Network Interconnect 135 Devices" [1] and RFC 2285 "Benchmarking Terminology for LAN 136 Switching Devices" [2] contain discussions and definitions for a 137 number of terms relevant to the benchmarking of signaling 138 performance of reservation capable routers and should be consulted 139 before attempting to make use of this document. 141 Additionally, this document defines terminology in a way that is 142 consistent with the terms used by Next Steps in Signaling working 143 group laid out in [5][6][7]. 145 For the sake of clarity and continuity this document adopts the 146 template for definitions set out in Section 2 of RFC 1242. 147 Definitions are indexed and grouped together into different sections 148 for ease of reference. 150 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 151 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in 152 this document are to be interpreted as described in RFC 2119. 154 3. Definition of Terms 156 3.1 Traffic Flow Types 157 This group of definitions describes traffic flow types forwarded by 158 resource reservation capable routers. 160 3.1.1 Data Flow 162 Definition: 163 A data flow is a stream of data packets from one sender to one or 164 more receivers, where each packet has a flow identifier unique to 165 the flow. 167 Discussion: 168 The flow identifier can be an arbitrary subset of the packet 169 header fields that uniquely distinguishes the flow from others. 170 For example, the 5-tuple "source address; source port; destination 171 address; destination port; protocol number" is commonly used for 172 this purpose (where port numbers are applicable). It is also 173 possible to take advantage of the Flow Label field of IPv6 174 packets. For more comment on flow identification refer to [5]. 176 3.1.2 Distinguished Data Flow 178 Definition: 179 Distinguished data flows are flows that resource reservation 180 capable routers intentionally treat better or worse than best- 181 effort data flows, according to a QoS agreement defined for the 182 distinguished flow. 184 Discussion: 185 Routers classify the packets of distinguished data flows and 186 identify the data flow they belong to. 188 The most common usage of the distinguished data flow is to get 189 higher priority treatment than that of best-effort data flows (see 190 the next definition). In these cases, a distinguished data flow is 191 sometimes referred to as a "premium data flow". Nevertheless 192 theoretically it is possible to require worse treatment than that 193 of best-effort flows. 195 3.1.3 Best-Effort Data Flow 197 Definition: 198 Best-effort data flows are flows that are not treated in any 199 special manner by resource reservation capable routers; thus, 200 their packets are served (forwarded) in some default way. 202 Discussion: 203 "Best-effort" means that the router makes its best effort to 204 forward the data packet quickly and safely, but does not guarantee 205 anything (e.g. delay or loss probability). This type of traffic is 206 the most common in today's Internet. 208 Packets that belong to best-effort data flows need not be 209 classified by the routers; that is, the routers don't need to find 210 a related reservation session in order to find out what treatment 211 the packet is entitled to. 213 3.2 Resource Reservation Protocol Basics 215 This group of definitions applies to signaling based resource 216 reservation protocols implemented by IP router devices. 218 3.2.1 QoS Session 220 Definition: 221 A QoS session is an application layer concept, shared between a 222 set of network nodes, that pertains to a specific set of data 223 flows. The information associated with the session includes the 224 data required to identify the set of data flows in addition to a 225 specification of the QoS treatment they require. 227 Discussion: 228 A QoS session is an end-to-end relationship. Whenever end-nodes 229 decide to obtain special QoS treatment for their data 230 communication, they set up a QoS session; as part of the process, 231 they or their proxies make a QoS agreement with the network, 232 specifying their data flows and the QoS treatment that the flows 233 require. 235 It is possible for the same QoS session to span multiple network 236 domains that have different resource provisioning architectures. 237 In this document, however, we only deal with the case where the 238 QoS session is realized over an IntServ architecture. It is 239 assumed that sessions will be established using signaling messages 240 of a resource reservation protocol. 242 QoS sessions must have unique identifiers; it must be possible to 243 determine which QoS session a given signaling message pertains to. 244 Therefore, each signaling message should include the identifier of 245 its corresponding session. As an example, in the case of RSVP, the 246 "session specification" identifies the QoS session plus refers to 247 the data flow; the "flowspec" specifies the desired QoS treatment 248 and the "filter spec" defines the subset of data packets in the 249 data flow that receive the QoS defined by the flowspec. 251 QoS sessions can be unicast or multicast depending on the number 252 of participants. In a multicast group there can be several data 253 traffic sources and destinations. Here the QoS agreement does not 254 have to be the same for each branch of the multicast tree 255 forwarding the data flow of the group. Instead, a dedicated 256 network resource in a router can be shared among many traffic 257 sources from the same multicast group (c.f. multicast reservation 258 styles in the case of RSVP). 260 Issues: 261 Even though QoS sessions are considered to be unique, resource 262 reservation capable routers might aggregate them and allocate 263 network resources to these aggregated sessions at once. The 264 aggregation can be based on similar data flow attributes (e.g. 265 similar destination addresses) or it can combine arbitrary 266 sessions as well. While reservation aggregation significantly 267 lightens the signaling processing task of a resource reservation 268 capable router, it also requires the administration of the 269 aggregated QoS sessions and might also lead to the violation of 270 the quality guaranties referring to individual data flows within 271 an aggregation [15]. 273 3.2.2 Resource Reservation Protocol 275 Definition: 276 Resource reservation protocols define signaling messages and 277 message processing rules used to control resource allocation in 278 IntServ architectures. 280 Discussion: 281 It is the signaling messages of a resource reservation protocol 282 that carry the information related to QoS sessions. This 283 information includes a session identifier, the actual QoS 284 parameters, and possibly flow descriptors. 286 The message processing rules of the signaling protocols ensure 287 that signaling messages reach all network nodes concerned. Some 288 resource reservation protocols (e.g. RSVP, NSIS QoS NSLP[7]) are 289 only concerned with this, i.e. carrying the QoS-related 290 information to all the appropriate network nodes, without being 291 aware of its content. This latter approach allows changing the way 292 the QoS parameters are described, and different kinds of 293 provisioning can be realized without the need to change the 294 protocol itself. 296 3.2.3 Resource Reservation Capable Router 298 Definition: 299 A router is resource reservation capable (it supports resource 300 reservation) if it is able to interpret signaling messages of a 301 resource reservation protocol, and based on these messages is able 302 to adjust the management of its flow classifiers and network 303 resources so as to conform to the content of the signaling 304 messages. 306 Discussion: 307 Routers capture signaling messages and manipulate reservation 308 states and/or reserved network resources according to the content 309 of the messages. This ensures that the flows are treated as their 310 specified QoS requirements indicate. 312 3.2.4 Reservation State 314 Definition: 315 A reservation state is the set of entries in the router's memory 316 that contain all relevant information about a given QoS session 317 registered with the router. 319 Discussion: 320 States are needed because IntServ related resource reservation 321 protocols require the routers to keep track of QoS session and 322 data-flow-related metadata. The reservation state includes the 323 parameters of the QoS treatment; the description of how and where 324 to forward the incoming signaling messages; refresh timing 325 information; etc. 327 Based on how reservation states are stored in a reservation 328 capable router, the routers can be categorized into two classes: 330 Hard-state resource reservation protocols (e.g. ST2 [10]) require 331 routers to store the reservation states permanently, established 332 by a set-up signaling primitive, until the router is explicitly 333 informed that the QoS session is canceled. 335 There are also soft-state resource reservation capable routers, 336 where there are no permanent reservation states, and each state 337 has to be regularly refreshed by appropriate refresh signaling 338 messages. If no refresh signaling message arrives during a certain 339 period then the router stops the maintenance of the QoS session 340 assuming that the end-points do not intend to keep the session up 341 any longer or the communication lines are broken somewhere along 342 the data path. This feature makes soft-state resource reservation 343 capable routers more robust than hard-state routers, since no 344 failures can cause resources to stay permanently stuck in the 345 routers. (Note that it is still possible to have an explicit 346 teardown message in soft-state protocols for quicker resource 347 release.) 349 Issues: 350 Based on the initiating point of the refresh messages, soft-state 351 resource reservation protocols can be divided into two groups. 352 First, there are protocols where it is the responsibility of the 353 end-points or their proxies to initiate refresh messages. These 354 messages are forwarded along the path of the data flow refreshing 355 the corresponding reservation states in each router affected by 356 the flow. Secondly, there are other protocols, where routers and 357 end-points have their own schedule for the reservation state 358 refreshes and they signal these refreshes to the neighboring 359 routers. 361 3.2.5 Resource Reservation Protocol Orientation 363 Definition: 364 The orientation of a resource reservation protocol tells which end 365 of the protocol communication initiates the allocation of the 366 network resources. Thus, the protocol can be sender or receiver 367 initiated, depending on the location of the data flow source 368 (sender) and destination (receiver) compared to the reservation 369 initiator. 371 Discussion: 372 In the case of sender-initiated protocols the resource reservation 373 propagates the same directions as of the data flow. Consequently, 374 in the case of receiver-initiated protocols the signaling messages 375 reserving resources are forwarded backward on the path of the data 376 flow. Due to the asymmetric routing nature of the Internet, in 377 this latter case, the path of the desired data flow should be 378 known before the reservation initiator would be able to send the 379 resource allocation messages. For example in the case of RSVP, the 380 RSVP PATH message, traveling from the data flow sources towards 381 the destinations, first marks the path of the data flow on which 382 the resource allocation messages will travel backward. 384 This definition considers only protocols that reserve resources 385 for just one data flow between the end-nodes. The reservation 386 orientation of protocols that reserve more than one data flow is 387 not defined here. 389 Issues: 390 The location of the reservation initiator affects the basics of 391 the resource reservation protocols and therefore is an important 392 aspect of characterization. Most importantly, in the case of 393 multicast QoS sessions, the sender-oriented protocols require the 394 traffic sources to maintain a list of receivers and send their 395 allocation messages considering the different requirements of the 396 receivers. Using multicast QoS sessions, the receiver-oriented 397 protocols enable the receivers to manage their own resource 398 allocation requests and thus ease the task of the sources. 400 3.3 Router Load Factors 402 When a router is under "load", it means that there are tasks its 403 CPU(s) must attend to; and/or that its memory contains data it must 404 keep track of; and/or that its interface buffers are utilized to 405 some extent; etc. Unfortunately, we cannot assume that the full 406 internal state of a router can be monitored during a benchmark; 407 rather, we must consider the router to be a black box. 409 We need to look at router "load" in a way that makes this "load" 410 measurable and controllable. Instead of focusing on the internal 411 processes of a router, we will consider the external, and therefore 412 observable, measurable and controllable processes that result in 413 "load". 415 In this chapter we introduce several ways of creating "load" on a 416 router; we will refer to these as "load factors" henceforth. These 417 load factors are defined so that they each impact the performance of 418 the router in a different way (or by different means), by utilizing 419 different components of a resource reservation capable router as 420 separately as possible. 422 During a benchmark, the performance of the device under test will 423 have to be measured under different controlled load conditions, that 424 is, with different values of these load factors. 426 3.3.1 Best-Effort Traffic Load Factor 428 Definition: 429 The best-effort traffic load factor is defined as the number and 430 length of equal-sized best-effort data packets that traverses the 431 router in a second. 433 Discussion: 434 Forwarding the best-effort data packets, which requires obtaining 435 the routing information and transferring the data packet between 436 network interfaces, requires processing power. This load factor 437 creates load on the CPU(s) and buffers of the router. 439 For the purpose of benchmarking we define a traffic flow as a 440 stream of equal-sized packets with even interpacket delay. It is 441 possible to specify traffic with varying packet sizes as a 442 superposition of multiple best-effort traffic flows as they are 443 defined here. 445 Issues: 446 The same amount of data segmented into differently sized packets 447 causes different amounts of load on the router, which has to be 448 considered during benchmarking measurements. The measurement unit 449 of this load factor reflects this as well. 451 Measurement unit: 452 This load factor has a composite unit of [packets per second 453 (pps); bytes]. For example, [5 pps; 100 bytes] means five pieces 454 of one-hundred-byte packets per second. 456 3.3.2 Distinguished Traffic Load Factor 458 Definition: 459 The distinguished traffic load factor is defined as the number and 460 length of the distinguished data packets that traverses the router 461 in a second. 463 Discussion: 464 Similarly to the best-effort data, forwarding the distinguished 465 data packets requires obtaining the routing information and 466 transferring the data packet between network interfaces. However, 467 in this case packets have to be classified as well, which requires 468 additional processing capacity. 470 For the purpose of benchmarking we define a traffic flow as a 471 stream of equal-sized packets with even interpacket delay. It is 472 possible to specify traffic with varying packet sizes as a 473 superposition of multiple distinguished traffic flows as they are 474 defined here. 476 Issues: 477 Just as in the best-effort case, the same amount of data segmented 478 into differently sized packets causes different amounts of load on 479 the router, which has to be considered during the benchmarking 480 measurements. The measurement unit of this load factor reflects 481 this as well. 483 Measurement unit: 484 This load factor has a composite unit of [packets per second 485 (pps); bytes]. For example, [5 pps; 100 bytes] means five pieces 486 of one-hundred-byte packets per second. 488 3.3.3 Session Load Factor 490 Definition: 491 The session load factor is the number of QoS sessions the router 492 is keeping track of. 494 Discussion: 495 Resource reservation capable routers maintain reservation states 496 to keep track of QoS sessions. Obviously, the more reservation 497 states are registered with the router, the more complex the 498 traffic classification becomes, and the more time it takes to look 499 up the corresponding resource reservation state. Moreover, not 500 only the traffic flows, but also the signaling messages that 501 control the reservation states have to be identified first, before 502 taking any other action, and this kind of classification also 503 means extra work for the router. 505 In the case of soft-state resource reservation protocols, the 506 session load also affects reservation state maintenance. For 507 example, the supervision of timers that watchdog the reservation 508 state refreshes may cause further load on the router. 510 This load factor utilizes the CPU(s), the main memory and the 511 session management logic (e.g. content addressable memory), if 512 any, of the resource reservation capable router. 514 Measurement unit: 515 This load component is measured by the number of QoS sessions that 516 impact the router. 518 3.3.4 Signaling Intensity Load Factor 520 Definition: 521 The signaling intensity load factor is the number of signaling 522 messages that are presented at the input interfaces of the router 523 during one second. 525 Discussion: 526 The processing of signaling messages requires processor power that 527 raises the load on the control plane of the router. 529 In routers where the control plane and the data plane are not 530 totally independent (e.g. certain parts of the tasks are served by 531 the same processor; or the architecture has common memory buffers, 532 transfer buses or any other resources) the signaling load can have 533 an impact on the router's packet forwarding performance as well. 535 Naturally, just as everywhere else in this document, the term 536 "signaling messages" refer only to the resource reservation 537 protocol related primitives. 539 Issues: 540 Most resource reservation protocols have several protocol 541 primitives realized by different signaling message types. Each of 542 these message types may require a different amount of processing 543 power from the router. This fact has to be considered during the 544 benchmarking measurements. 546 Measurement unit: 547 The unit of this factor is signaling messages/second. 549 3.3.5 Signaling Burst Load Factor 551 Definition: 552 The signaling burst load factor is defined as the number of 553 signaling messages that arrive to one input port of the router 554 back-to-back ([1]), causing persistent load on the signaling 555 message handler. 557 Discussion: 558 The definition focuses on one input port only and does not 559 consider the traffic arriving at the other input ports. 560 As a consequence, a set of messages arriving at different ports, 561 but with such a timing that would be a burst if the messages 562 arrived at the same port, is not considered to be a burst. The 563 reason for this is that it is not guaranteed in a black-box test 564 that this would have the same effect on the router as a burst 565 (incoming at the same interface) has. 567 This definition conforms to the burst definition given in [2]. 569 Issues: 570 Most of the resource reservation protocols have several protocol 571 primitives realized by different signaling message types. Bursts 572 built up of different messages may have a different effect on the 573 router. Consequently, during measurements the content of the burst 574 has to be considered as well. 576 Likewise, the first one of multiple idempotent signaling messages 577 that each accomplish exactly the same end will probably not take 578 the same amount of time to be processed as subsequent ones. 579 Benchmarking methodology will have to consider the intended effect 580 of the signaling messages, as well as the state of the router at 581 the time of their arrival. 583 Measurement unit: 584 This load factor is characterized by the number of messages in the 585 burst. 587 3.4 Performance Metrics 589 This group of definitions is a collection of measurable quantities 590 that describe the performance impact the different load components 591 have on the router. 593 During a benchmark, the values of these metrics will have to be 594 measured under different load conditions. 596 3.4.1 Signaling Message Handling Time 598 Definition: 599 The signaling message handling time (or, in short, signal handling 600 time) is the latency ([1], for store-and-forward devices) of a 601 signaling message passing through the router. 603 Discussion: 604 The router interprets the signaling messages, acts based on their 605 content and usually forwards them in an unmodified or modified 606 form. Thus the message handling time is usually longer than the 607 forwarding time of data packets of the same size. 609 There might be signaling message primitives, however, that are 610 drained or generated by the router, like certain refresh messages. 611 In this case the signal handling time is not necessarily 612 measureable, therefore it is not defined for such messages. 614 In the case of signaling messages that carry information 615 pertaining to multicast flows, the router might issue multiple 616 signaling messages after processing them. In this case, by 617 definition, the signal handling time is the latency between the 618 incoming signaling message and the last outgoing signaling message 619 related to the received one. 621 The signal handling time is an important characteristic as it 622 directly affects the setup time of a QoS session. 624 Issues: 625 The signal handling time may be dependent on the type of the 626 signaling message. For example, it usually takes a shorter time 627 for the router to remove a reservation state than to set it up. 628 This fact has to be considered during the benchmarking process. 630 As noted above, the first one of multiple idempotent signaling 631 messages that each accomplish exactly the same end will probably 632 not take the same amount of time to be processed as subsequent 633 ones. Benchmarking methodology will have to consider the intended 634 effect of the signaling messages, as well as the state of the 635 router at the time of their arrival. 637 Measurement unit: 638 The dimension of the signaling message handling time is the 639 second, reported with a resolution sufficient to distinguish 640 between different events/DUTs (e.g., milliseconds). Reported 641 results MUST clearly indicate the time unit used. 643 3.4.2 Distinguished Traffic Delay 645 Definition: 646 Distinguished traffic delay is the latency ([1], for store-and- 647 forward devices) of a distinguished data packet passing through 648 the tested router device. 650 Discussion: 651 Distinguished traffic packets must be classified first in order to 652 assign the network resources dedicated to the flow. The time of 653 the classification is added to the usual forwarding time 654 (including the queuing) that a router would spend on the packet 655 without any resource reservation capability. This classification 656 procedure might be quite time consuming in routers with vast 657 amounts of reservation states. 659 There are routers where the processing power is shared between the 660 control plane and the data plane. This means that the processing 661 of signaling messages may have an impact on the data forwarding 662 performance of the router. In this case the distinguished traffic 663 delay metric also indicates the influence the two planes have on 664 each other. 666 Issues: 667 Queuing of the incoming data packets in routers can bias this 668 metric, so the measurement procedures have to consider this 669 effect. 671 Measurement unit: 672 The dimension of the signaling message handling time is the 673 second, reported with resolution sufficient to distinguish between 674 different events/DUTs (e.g., millisecond units). Reported results 675 MUST clearly indicate the time unit used. 677 3.4.3 Best-effort Traffic Delay 679 Definition: 680 Best-effort traffic delay is the latency of a best-effort data 681 packet traversing the tested router device. 683 Discussion: 684 If the processing power of the router is shared between the 685 control and data plane, then the processing of signaling messages 686 may have an impact on the data forwarding performance of the 687 router. In this case the best-effort traffic delay metric is an 688 indicator of the influence the two planes have on each other. 690 Issues: 691 Queuing of the incoming data packets in routers can bias this 692 metric as well, so measurement procedures have to consider this 693 effect. 695 Measurement unit: 696 The dimension of the signaling message handling time is the 697 second, reported with resolution sufficient to distinguish between 698 different events/DUTs (e.g., millisecond units). Reported results 699 MUST clearly indicate the time unit used. 701 3.4.4 Signaling Message Deficit 703 Definition: 704 Signaling message deficit is one minus the ratio of the actual and 705 the expected number of signaling messages leaving a resource 706 reservation capable router. 708 Discussion: 709 This definition gives the same value as the ratio of the lost 710 (that is, not forwarded or not generated) and the expected 711 messages. The above calculation must be used because the number of 712 lost messages cannot be measured directly. 714 There are certain types of signaling messages that reservation 715 capable routers are required to forward as soon as their 716 processing is finished. However, due to lack of resources or other 717 reasons, the forwarding or even the processing of these signaling 718 messages might not take place. 720 Certain other kinds of signaling messages must be generated by the 721 router in the absence of any corresponding incoming message. It is 722 possible that an overloaded router does not have the resources 723 necessary to generate such a message. 725 To characterize these situations we introduce the signaling 726 message deficit metric that expresses the ratio of the signaling 727 messages that have actually left the router and those ones that 728 were expected to leave the router. We subtract this ratio from one 729 in order to obtain a loss-type metric instead of a "message 730 survival metric". 732 Since the most frequent reason for signaling message deficit is 733 high router load, this metric is suitable for sounding out the 734 scalability limits of resource reservation capable routers. 736 During the measurements one must be able to determine whether a 737 signaling message is still in the queues of the router or if it 738 has already been dropped. For this reason we define a signaling 739 message as lost if no forwarded signaling message is emitted 740 within a reasonably long time period. This period is defined along 741 with the benchmarking methodology. 743 Measurement unit: 744 This measure has no unit; it is expressed as a real number, which 745 is between zero and one, including the limits. 747 3.4.5 Session Maintenance Capacity 749 Definition: 750 The session maintenance capacity metric is used in the case of 751 soft-state resource reservation protocols only. It is defined as 752 the ratio of the number of QoS sessions actually being maintained 753 and the number of QoS sessions that should have been maintained 754 during one refresh period. 756 Discussion: 757 For soft-state protocols maintaining a QoS session means 758 refreshing the reservation states associated with it. 760 When a soft-state resource reservation capable router is 761 overloaded, it may happen that the router is not able to refresh 762 all the registered reservation states, because it does not have 763 the time to run the state refresh task. In this case sooner or 764 later some QoS sessions will be lost even if the endpoints still 765 require their maintenance. 767 The session maintenance capacity sounds out the maximal number of 768 QoS sessions that the router is capable of maintaining. 770 Issues: 771 The actual process of session maintenance is protocol and 772 implementation dependent, thus so is the method to examine whether 773 a session is maintained or not. 775 In the case of soft-state resource reservation protocols a router 776 that fails to maintain a QoS session may not emit refresh 777 signaling messages either. This has direct consequences on the 778 signaling message deficit metric. 780 Measurement unit: 781 This measure has no unit; it is expressed as a real number, which 782 is between zero and one (including the limits). 784 3.5 Router Load Conditions and Scalability Limit 786 Depending mainly, but not exclusively, on the overall load of a 787 router, it can be in exactly one of the following four conditions at 788 a time: loss-free and QoS compliant; lossy and QoS compliant; loss- 789 free but not QoS compliant; and neither loss-free nor QoS compliant. 790 These conditions are defined below, along with the scalability 791 limit. 793 3.5.1 Loss-Free Condition 795 Definition: 796 A router is in loss-free condition, or loss-free state, if and 797 only if it is able to perform its tasks correctly and in a timely 798 fashion. 800 Discussion: 801 All existing routers have finite buffer memory and finite 802 processing power. If a router is in loss-free state, the buffers 803 of the router still contain enough free space to accommodate the 804 next incoming packet when it arrives. Also, the router has enough 805 processing power to cope with all its tasks, thus all required 806 operations are carried out within the time the protocol 807 specification allows; or, if this time is not specified by the 808 protocol, then in "reasonable time" (which is then defined in the 809 benchmarks). Similar considerations can be applied to other 810 resources a router may have, if any; in loss-free states, the 811 utilization of these resources still allows the router to carry 812 out its tasks in accordance with applicable protocol 813 specifications and in "reasonable time". 815 Note that loss-free states as defined above are not related to the 816 reservation states of resource reservation protocols. The word 817 "state" is used to mean "condition". 819 Also note that it is irrelevant what internal reason causes a 820 router to fail to perform in accordance with protocol 821 specifications or in "reasonable time"; if it is not high load but 822 -- for example -- an implementation error that causes the device 823 to perform inadequately, it still cannot be said to be in a loss- 824 free state. The same applies to the random early dropping of 825 packets in order to prevent congestion. In a black-box measurement 826 it is impossible to determine whether a packet was dropped as part 827 of a congestion control mechanism or because the router was unable 828 to forward it; therefore, if packet loss is observed except as 829 noted below, the router is by definition in lossy state (lossy 830 condition). 832 If a distinguished data flow exceeds its allotted bandwidth, it is 833 acceptable for routers to drop excess packets. Thus, a router that 834 is QoS Compliant (see below) is also loss-free provided that it 835 only drops packets from distinguished data flows. 837 If a device is not in a loss-free state, it is in a lossy 838 condition/state. 840 Related definitions: 841 Lossy Condition 842 QoS Compliant Condition 843 Not QoS Compliant Condition 844 Scalability Limit 846 3.5.2 Lossy Condition 848 Definition: 849 A router is in a lossy condition, or lossy state, if it cannot 850 perform its duties adequately for some reason; that is, if it does 851 not meet protocol specifications (except QoS guarantees, which are 852 treated separately), or -- if time-related specifications are 853 missing -- doesn't complete some operations in "reasonable time" 854 (which is then defined in the benchmarks). 856 Discussion: 857 A router may be in a lossy state for several reasons, including 858 but not necessarily limited to the following: 860 a) Buffer memory has run out, so either an incoming or a buffered 861 packet has to be dropped. 862 b) The router doesn't have enough processing power to cope with 863 all its duties. Some required operations are skipped, aborted 864 or suffer unacceptable delays. 865 c) Some other finite internal resource is exhausted. 866 d) The router runs a defective (non-conforming) protocol 867 implementation. 868 e) Hardware malfunction. 869 f) A congestion control mechanism is active. 871 Loss can mean the loss of data packets as well as signaling 872 message deficit. 874 A router that does not lose data packets and does not experience 875 signaling message deficit but fails to meet required QoS 876 parameters is in the loss-free, but not in the QoS compliant 877 state. 879 If a device is not in a lossy state, it is in a loss-free 880 condition/state. 882 Related definitions: 883 Loss-Free Condition (especially the discussion of congestion 884 control mechanisms that cause packet loss) 885 Scalability Limit 886 Signaling Message Deficit 887 QoS Compliant Condition 888 Not QoS Compliant Condition 890 3.5.3 QoS Compliant Condition 892 Definition: 893 A router is in the QoS compliant state if and only if all 894 distinguished data flows receive the QoS treatment they are 895 entitled to. 897 Discussion: 898 Defining what specific QoS guarantees must be upheld is beyond the 899 scope of this document because every reservation model may specify 900 a different set of such parameters. 902 Loss, delay, jitter etc. of best-effort data flows are irrelevant 903 when considering whether a router is in the QoS compliant state. 905 Related definitions: 906 Loss-Free Condition 907 Lossy Condition 908 Not QoS Compliant Condition 909 Scalability Limit 911 3.5.4 Not QoS Compliant Condition 913 Definition: 914 A router is in the not QoS compliant state if and only if it is 915 not in the QoS compliant condition. 917 Related definitions: 918 Loss-Free Condition 919 Lossy Condition 920 QoS Compliant Condition 921 Scalability Limit 923 3.5.5 Scalability Limit 925 Definition: 926 The scalability limits of a router are the boundary load 927 conditions where the router is still in the loss-free and QoS 928 compliant ("good") state but the smallest amount of additional 929 load would drive it to a state that is "bad": either not loss- 930 free; or not QoS compliant; or neither loss-free nor QoS 931 compliant. 933 Discussion: 934 An unloaded router that operates correctly is in a "good" state 935 because it is both loss-free and QoS compliant. As load increases, 936 the resources of the router are becoming more and more utilized. 937 There is a certain point where the router leaves the "good" state 938 and enters a "bad" state as defined above. Note that such a point 939 may be impossible to reach in some cases (for example if the 940 bandwidth of the physical medium prevents increasing the traffic 941 load any further). 943 A particular load condition can be identified by the corresponding 944 values of the load factors (as defined in 3.3 Router Load Factors) 945 impacting the router. These values can be represented as a 7-tuple 946 of numbers (there are only five load factors, but the traffic load 947 factors have composite units and thus require two numbers each to 948 express). We can think of these tuples as vectors that correspond 949 either to a "good" state or to a "bad" state. The scalability 950 limit of the router is, then, the boundary between the sets of 951 vectors corresponding to "good" and "bad" states. Finding these 952 boundary points if one of the objectives of benchmarking. 954 Benchmarks MAY try to separately identify the boundaries of the 955 loss-free and of the QoS compliant conditions in the (seven- 956 dimensional) space defined by the load-vectors. 958 Related definitions: 959 Lossy Condition 960 Loss-Free Condition 961 QoS Compliant Condition 962 Non QoS Compliant Condition 964 4. Security Considerations 966 As this document only provides terminology and describes neither a 967 protocol nor an implementation or a procedure, there are no security 968 considerations associated with it. 970 5. IANA Considerations 972 This document requires no IANA actions. 974 6. Acknowledgements 976 We would like to thank the following individuals for their help in 977 the research and development work which contributed to the creation 978 of this document: Joakim Bergkvist and Norbert Vegh from Telia 979 Research AB, Sweden; Balazs Szabo, Gabor Kovacs and Peter Vary from 980 the High Speed Networks Laboratory, Department of Telecommunication 981 and Mediainformatics, Budapest University of Technology and 982 Economics, Hungary. 984 7. References 986 7.1 Normative References 988 [1] S. Bradner, "Benchmarking Terminology for Network 989 Interconnection Devices", RFC 1242, July 1991 991 [2] R. Mandeville, "Benchmarking Terminology for LAN Switching 992 Devices", RFC 2285, February 1998 994 7.2 Informative References 996 [3] Braden R., Clark D., Shenker S., "Integrated Services in the 997 Internet Architecture: an Overview", RFC 1633, June 1994 999 [4] B. Braden, Ed., et. al., "Resource Reservation Protocol (RSVP) 1000 - Version 1 Functional Specification", RFC 2205, September 1997 1002 [5] R. Hancock, et al., "Next Steps in Signaling (NSIS): 1003 Framework", RFC4080, June 2005 1005 [6] H. Schulzrinne, R. Hancock, "GIST: General Internet Signaling 1006 Transport", Internet Draft (draft-ietf-nsis-ntlp-11), August 1007 2006 (work in progress) 1009 [7] J. Manner (ed.), G. Karagiannis, A. McDonald, "NSLP for 1010 Quality-of-Service Signaling", Internet Draft (draft-ietf-nsis- 1011 qos-nslp-12), October 2006 (work in progress) 1013 [8] J. Ash, A. Bader, C. Kappler, D. Oran, "QoS NSLP QSPEC 1014 Template", Internet Draft (draft-ietf-nsis-qspec-14), January 1015 2007 (work in progress) 1017 [9] P. Pan, H. Schulzrinne, "YESSIR: A Simple Reservation Mechanism 1018 for the Internet", Computer Communication Review, on-line 1019 version, volume 29, number 2, April 1999 1021 [10] L. Delgrossi, L. Berger, "Internet Stream Protocol Version 2 1022 (ST2) Protocol Specification - Version ST2+", RFC 1819, August 1023 1995 1025 [11] P. White, J. Crowcroft, "A Case for Dynamic Sender-Initiated 1026 Reservation in the Internet", Journal on High Speed Networks, 1027 Special Issue on QoS Routing and Signaling, Vol. 7 No. 2, 1998 1029 [12] J. Bergkvist, D. Ahlard, T. Engborg, K. Nemeth, G. Feher, I. 1030 Cselenyi, M. Maliosz, "Boomerang : A Simple Protocol for 1031 Resource Reservation in IP Networks", Vancouver, IEEE Real-Time 1032 Technology and Applications Symposium, June 1999 1034 [13] A. Eriksson, C. Gehrmann, "Robust and Secure Light-weight 1035 Resource Reservation for Unicast IP Traffic", International WS 1036 on QoS'98, IWQoS'98, May 18-20, 1998 1038 [14] J. Manner, X. Fu, "Analysis of Existing Quality of Service 1039 Signaling Protocols", RFC4094, May 2005 1041 [15] F. Baker, C. Iturralde, F. Le Faucheur, B. Davie, "Aggregation 1042 of RSVP for IPv4 and IPv6 Reservations", RFC 3175, September 1043 2001 1045 Authors' Addresses 1047 Gabor Feher 1048 Budapest University of Technology and Economics 1049 Department of Telecommunications and Mediainformatics 1050 Magyar Tudosok krt. 2, H-1117, Budapest, Hungary 1051 Phone: +36 1 463-1538 1052 Email: Gabor.Feher@tmit.bme.hu 1054 Krisztian Nemeth 1055 Budapest University of Technology and Economics 1056 Department of Telecommunications and Mediainformatics 1057 Magyar Tudosok krt. 2, H-1117, Budapest, Hungary 1058 Phone: +36 1 463-1565 1059 Email: Krisztian.Nemeth@tmit.bme.hu 1061 Andras Korn 1062 Budapest University of Technology and Economics 1063 Department of Telecommunication and Mediainformatics 1064 Magyar Tudosok krt. 2, H-1117, Budapest, Hungary 1065 Phone: +36 1 463-2664 1066 Email: Andras.Korn@tmit.bme.hu 1068 Istvan Cselenyi 1069 TeliaSonera International Carrier 1070 Vaci ut 22-24, H-1132 Budapest, Hungary 1071 Phone: +36 1 412-2705 1072 Email: Istvan.Cselenyi@teliasonera.com 1074 Disclaimer of Validity 1076 The IETF takes no position regarding the validity or scope of any 1077 Intellectual Property Rights or other rights that might be claimed 1078 to pertain to the implementation or use of the technology 1079 described in this document or the extent to which any license 1080 under such rights might or might not be available; nor does it 1081 represent that it has made any independent effort to identify any 1082 such rights. Information on the procedures with respect to 1083 rights in RFC documents can be found in BCP 78 and BCP 79. 1085 Copies of IPR disclosures made to the IETF Secretariat and any 1086 assurances of licenses to be made available, or the result of an 1087 attempt made to obtain a general license or permission for the use 1088 of such proprietary rights by implementers or users of this 1089 specification can be obtained from the IETF on-line IPR repository 1090 at http://www.ietf.org/ipr. 1092 The IETF invites any interested party to bring to its attention 1093 any copyrights, patents or patent applications, or other 1094 proprietary rights that may cover technology that may be required 1095 to implement this standard. Please address the information to the 1096 IETF at ietf-ipr@ietf.org. 1098 Copyright Notice 1100 Copyright (C) The IETF Trust (2007). 1102 This document is subject to the rights, licenses and restrictions 1103 contained in BCP 78, and except as set forth therein, the authors 1104 retain all their rights. 1106 Disclaimer 1108 This document and the information contained herein are provided 1109 on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE 1110 REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE 1111 IETF TRUST AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL 1112 WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY 1113 WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE 1114 ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS 1115 FOR A PARTICULAR PURPOSE.