idnits 2.17.1 draft-ietf-bmwg-firewall-01.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** Looks like you're using RFC 2026 boilerplate. This must be updated to follow RFC 3978/3979, as updated by RFC 4748. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- ** Missing expiration date. The document expiration date should appear on the first and last page. == No 'Intended status' indicated for this document; assuming Proposed Standard == It seems as if not all pages are separated by form feeds - found 0 form feeds but 23 pages Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack an Abstract section. ** The document seems to lack a Security Considerations section. ** The document seems to lack an IANA Considerations section. (See Section 2.2 of https://www.ietf.org/id-info/checklist for how to handle the case when there are no actions for IANA.) ** The document seems to lack an Authors' Addresses Section. ** There are 78 instances of too long lines in the document, the longest one being 4 characters in excess of 72. ** There are 43 instances of lines with control characters in the document. Miscellaneous warnings: ---------------------------------------------------------------------------- == Line 786 has weird spacing: '... mapped on th...' == Using lowercase 'not' together with uppercase 'MUST', 'SHALL', 'SHOULD', or 'RECOMMENDED' is not an accepted usage according to RFC 2119. Please use uppercase 'NOT' together with RFC 2119 keywords (if that is what you mean). Found 'MUST not' in this paragraph: The tester originating the TCP SYN attack MUST be attached to the Unprotected network. In addition, the tester MUST not respond to the SYN ACK packets sent by target server in response to the SYN packet. -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (November 2000) is 8562 days in the past. Is this intentional? -- Found something which looks like a code comment -- if you have code sections in the document, please surround them with '' and '' lines. Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) -- Missing reference section? '1' on line 1115 looks like a reference -- Missing reference section? '2' on line 1118 looks like a reference -- Missing reference section? '3' on line 1120 looks like a reference -- Missing reference section? '4' on line 1123 looks like a reference Summary: 8 errors (**), 0 flaws (~~), 4 warnings (==), 7 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 Network Working Group Terry Martin 2 Internet-Draft M2networx INC 3 Expiration Date: B. Hickman 4 Netcom Systems 5 November 2000 7 Benchmarking Methodology for Firewalls 8 10 Status of this Memo 12 This document is an Internet-Draft and is in full conformance with 13 all provisions of Section 10 of RFC2026. 15 Internet-Drafts are working documents of the Internet Engineering 16 Task Force (IETF), its areas, and its working groups. Note that 17 other groups may also distribute working documents as Internet- 18 Drafts. 20 Internet-Drafts are draft documents valid for a maximum of six months 21 and may be updated, replaced, or obsoleted by other documents at any 22 time. It is inappropriate to use Internet-Drafts as reference 23 material or to cite them other than as "work in progress." 25 The list of current Internet-Drafts can be accessed at 26 http://www.ietf.org/ietf/1id-abstracts.txt 28 The list of Internet-Draft Shadow Directories can be accessed at 29 http://www.ietf.org/shadow.html. 31 Table of Contents 33 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 34 2. Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 35 3. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 36 4. Test setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 37 4.1 Test Considerations . . . . . . . . . . . . . . . . . . . . . 4 38 4.1.1 Virtual Client/Servers . . . . . . . . . . . . . . . . . . 4 39 4.1.2 Test Traffic Requirements . . . . . . . . . . . . . . . . 4 40 4.1.3 DUT/SUT Traffic Flows . . . . . . . . . . . . . . . . . . 5 41 4.1.4 Multiple Client/Server Testing . . . . . . . . . . . . . . 5 42 4.1.5 NAT(Network Address Translation) . . . . . . . . . . . . . 6 43 4.1.6 Rule Sets . . . . . . . . . . . . . . . . . . . . . . . . 6 44 4.1.7 Web Caching . . . . . . . . . . . . . . . . . . . . . . . 6 45 4.1.8 Authentication . . . . . . . . . . . . . . . . . . . . . . 6 46 5. Benchmarking Tests . . . . . . . . . . . . . . . . . . . . . . . . 7 47 5.1 Concurrent Connection Capacity . . . . . . . . . . . . . . . . 7 48 5.2 Maximum Connection Rate . . . . . . . . . . . . . . . . . . . 8 49 5.3 Connection Establishment Time . . . . . . . . . . . . . . . . 10 50 5.4 Denial Of Service Handling . . . . . . . . . . . . . . . . . . 11 51 5.5 Single Application Goodput . . . . . . . . . . . . . . . . . . 12 52 5.5.1 FTP Goodput . . . . . . . . . . . . . . . . . . . . . . . 12 53 5.5.2 SMTP Goodput . . . . . . . . . . . . . . . . . . . . . . . 14 54 5.5.3 HTTP Goodput . . . . . . . . . . . . . . . . . . . . . . . 15 55 5.6 Concurrent Application Goodput . . . . . . . . . . . . . . . . 17 56 5.7 Illegal Traffic Handling . . . . . . . . . . . . . . . . . . . 19 57 5.8 Latency . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 58 Appendices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 59 A. File Transfer Protocol(FTP) . . . . . . . . . . . . . . . . . . . 19 60 A.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 19 61 A.2 Connection Establishment/Teardown . . . . . . . . . . . . . . 20 62 A.3 Object Format . . . . . . . . . . . . . . . . . . . . . . . . 20 63 B. Simple Mail Transfer Protocol(SMTP) . . . . . . . . . . . . . . . 21 64 B.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . 21 65 B.2 Connection Establishment/Teardown . . . . . . . . . . . . . . 21 66 B.3 Object Format . . . . . . . . . . . . . . . . . . . . . . . . 22 67 C. HyperText Transfer Protocol(HTTP) . . . . . . . . . . . . . . . . 22 68 C.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 22 69 C.2 Version Considerations . . . . . . . . . . . . . . . . . . . . 23 70 C.3 Object Format . . . . . . . . . . . . . . . . . . . . . . . . 23 71 E. TCP establishment/teardown . . . . . . . . . . . . . . . . . . . . 23 72 D. GoodPut Measurements . . . . . . . . . . . . . . . . . . . . . . . 23 73 F. References . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 75 1. Introduction 77 This document is intended to provide methodology for the benchmarking 78 of firewalls. It provides methodologies for benchmarking forwarding 79 performance, connection performance, latency and filtering. In 80 addition to defining the tests, this document also describes specific 81 formats for reporting the results of the tests. 83 A previous document, "Benchmarking Terminology for Firewall 84 Performance" [1], defines many of the terms that are used in this 85 document. The terminology document SHOULD be consulted before 86 attempting to make use of this document. 88 2. Requirements 90 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 91 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 92 document are to be interpreted as described in RFC 2119. 94 3. Scope 96 Firewalls can provide a single point of defense between two 97 networks--it protects one network from the other. Usually, a firewall 98 protects the company's private network from the public or shared 99 networks to which it is connected. A firewall can be as simple as a 100 device that filters different packets or as complex as a group of 101 devices providing solutions that offers combined packet filtering 102 and application-level proxy or network translation services. This RFC 103 will focus on developing benchmark testing of systems from an 104 application perspective and will be developed independent of any 105 firewall implementation. These tests will evaluate the ability of 106 firewall devices to control and manage applications services used 107 by today's businesses such as applications like the World Wide Web, 108 File transfer procedures and e-mail. 110 Even through there are many different control methods of managing 111 application level being implemented, this RFC does not condone or 112 promote any aforementioned process or procedure. It's goal is to 113 present a procedure that will evaluate firewall performance 114 independent of their implementation. 116 4. Test Setup 118 Test configurations defined in this document will be confined to 119 dual-homed and tri-homed as shown in figure 1 and figure 2 120 respectively. 122 Firewalls employing Dual-Homed configurations connect two networks. 123 One interface of the firewall is attached to the unprotected network, 124 typically the public network(i.e. - Internet). The other interface is 125 connected to the protected network, typically the internal LAN. In 126 the case of Dual-Homed configurations, servers which are made 127 accessible to the public(Unprotected) network are attached to the 128 private(Protected) network. 130 +----------+ +----------+ 131 | | | +----------+ | | | 132 | Servers/ |----| | | |------| Servers/ | 133 | Clients | | | | | | Clients | 134 | | |-------| DUT/SUT |--------| | | 135 +----------+ | | | | +----------+ 136 | +----------+ | 137 Protected | | Unprotected 138 Network Network 139 Figure 1(Dual-Homed) 141 Tri-homed[1] configurations employ a third segment called a DMZ. With 142 tri-homed configurations, servers accessible to the public network 143 are attached to the DMZ. Tri-Homed configurations offer additional 144 security by separating server accessible to the public network from 145 internal hosts. 147 +----------+ +----------+ 148 | | | +----------+ | | | 149 | Clients |----| | | |------| Servers/ | 150 | | | | | | | Clients | 151 +----------+ |-------| DUT/SUT |--------| | | 152 | | | | +----------+ 153 | +----------+ | 154 Protected | | | Unprotected 155 Network | Network 156 | 157 | 158 ----------------- 159 | DMZ 160 | 161 | 162 +-----------+ 163 | | 164 | Servers | 165 | | 166 +-----------+ 168 Figure 2(Tri-Homed) 170 4.1 Test Considerations 172 4.1.1 Virtual Clients/Servers 174 Since firewall testing may involve data sources which emulate multiple 175 users or hosts, the methodology uses the terms virtual clients/servers. 176 For these firewall tests, virtual clients/servers specify application 177 layer entities which may not be associated with a unique physical 178 interface. For example, four virtual clients may originate from the 179 same data source[1]. The test report SHOULD indicate the number of 180 virtual clients and virtual servers participating in the test on a per 181 interface(See 4.1.3) basis. 183 Need to include paragraph for synchronize start of test. Data sources 184 MUST be synchronized to start initiating connections within a specified 185 time of each other. 187 4.1.2 Test Traffic Requirements 189 While the function of a firewall is to enforce access control 190 policies, the criteria by which those policies are defined vary 191 depending on the implementation. Firewalls may use network layer 192 and/or, in many cases, application-layer criteria to make 193 access-control decisions. Therefore, the test equipment used to 194 benchmark the DUT/SUT performance MUST consist of real clients and 195 servers generating legitimate layer 7 conversations. 197 The tests defined in this document use HTTP, FTP, and SMTP sessions 198 for benchmarking the performance of the DUT/SUT. Other layer 7 199 conversations are outside the scope of this document. See appendices 200 for specific information regarding the transactions involved in 201 establishing/tearing down connections as well as object formats 202 for each of the aforementioned protocols. 204 4.1.3 DUT/SUT Traffic Flows 206 Since the number of interfaces are not fixed, the traffic flows will 207 be dependent upon the configuration used in benchmarking the DUT/SUT. 208 Note that the term "traffic flows" is associated with client-to- 209 server requests. 211 For Dual-Homed configurations, there are two unique traffic flows: 213 Client Server 214 ------ ------ 215 Protected -> Unprotected 216 Unprotected -> Protected 218 For Tri-Homed configurations, there are three unique traffic flows: 220 Client Server 221 ------ ------ 222 Protected -> Unprotected 223 Protected -> DMZ 224 Unprotected -> DMZ 226 4.1.4 Multiple Client/Server Testing 228 One or more clients may target multiple servers for a given 229 application. Each virtual client MUST initiate requests(Connection, 230 file transfers, etc.) in a round-robin fashion. For example, if 231 the test consisted of six virtual clients targeting three servers, 232 the pattern would be as follows: 234 Client Target Server(In order of request) 236 #1 1 2 3 1... 237 #2 2 3 1 2... 238 #3 3 1 2 3... 239 #4 1 2 3 1... 240 #5 2 3 1 2... 241 #6 3 1 2 3... 243 4.1.5 NAT(Network Address Translation) 245 Most firewalls come with Network Address Translation(NAT)networks 246 built in which translates internal host IP addresses attached to the 247 protected network to a virtual IP address for communicating across the 248 unprotected network(Internet). This involves additional processing 249 on the part of the DUT/SUT and may impact on performance. Therefore, 250 tests SHOULD be ran with NAT disabled and NAT enabled to determine the 251 performance differentials. The test report SHOULD indicate whether NAT 252 was enabled or disabled. 254 4.1.6 Rule Sets 256 Rule sets[1] are a collection of access control policies that 257 determines which packets the DUT/SUT will forward and which it will 258 reject. The criteria by which these access control policies may be 259 defined will vary depending on the capabilities of the DUT/SUT. The 260 scope of this document is limited to how the rule sets should be 261 applied when testing the DUT/SUT. 263 The firewall monitors the incoming traffic and checks to make sure 264 that the traffic meets one of the defined rules before allowing it to 265 be forwarded. It is RECOMMENDED that a rule be entered for each 266 host(i.e. - Virtual client). Although many firewalls permit groups 267 of IP addresses to be defined for a given rule, tests SHOULD be 268 performed with large rule sets, which are more stressful to the 269 DUT/SUT. 271 The DUT/SUT SHOULD be configured to denies access to all traffic which 272 was not previously defined in the rule set. 274 4.8 Web Caching 276 Some firewalls include caching agents in order to reduce network 277 load. When making a request through a caching agent, the caching 278 agent attempts to service the response from its internal resources. 279 The cache itself saves responses it receives, such as responses for 280 HTTP GET requests. The report SHOULD indicate whether web caching 281 was enabled or disabled on the DUT/SUT. The test report SHOULD 282 indicate whether NAT was enabled or disabled. 284 4.9 Authentication 286 Access control may involve authentication processes such as user, 287 client or session authentication. Authentication is usually performed 288 by devices external to the firewall itself, such as an authentication 289 servers and may add to the latency of the system. Any authentication 290 processes MUST be included as part of connection setup process. 292 5. Benchmarking Tests 294 5.1 Concurrent Connection Capacity 296 5.1.1 Objective 298 To determine the maximum number of concurrent connections supported 299 by the DUT/SUT, as defined in RFC2647[1]. This test will employ a 300 step search algorithm to obtain the maximum number of concurrent 301 FTP,HTTP or SMTP connections the DUT/SUT can maintain. 303 5.1.2 Setup Parameters 305 The following parameters MUST be defined. Each parameters is 306 configured with the following considerations. 308 Connection Attempt Rate - The rate at which new connection requests 309 are attempted. The rate SHOULD be set lower than maximum rate at 310 which the DUT/SUT can accept new connection requests. 312 Connection Step count - Defines the number of additional connections 313 attempted for each iteration of the step search algorithm. 315 Object/Message - Defines the number of bytes to be transferred across 316 each connection. 318 5.1.3 Procedure 320 Each virtual client will attempt to establish connections to their 321 target server(s) at a fixed rate in a round robin fashion. Each 322 iteration will involve the virtual clients attempting to establish a 323 fixed number of additional connections. This search algorithm will 324 be repeated until either: 326 - One or more of the additional connection attempts fail to 327 complete 328 - One or more of the previously established connections failed. 330 Data transfers SHOULD be performed on each connection after the 331 given connection is established. Data transfers MUST be performed on 332 all connections after all of the addition connection have been 333 established. 335 When benchmarking with FTP, virtual clients will issue NOOP command's 336 to validate that work can be performed across each connection. The 337 virtual clients must receive a Command Successful reply from the 338 target server in order to be considered a valid connection. 340 When benchmarking with other applications such as HTTP or SMTP, 341 validation of the connection will be performed by initiating 342 object/message transfers. All bytes associated with the object/message 343 transfers MUST be received by the requesting virtual client in order 344 to be considered a valid connection. 346 5.1.5 Measurements 348 Total number of connections that were successfully completed in a 349 step. Test equipment MUST be able to track each connection to verify 350 all required transaction between the virtual client and server 351 completed successfully. This includes successful completion of both 352 the command sequences and exchanging of any data across each of those 353 connections. 355 5.1.6 Reporting Format 357 Maximum concurrent connections reported MUST be the aggregate number 358 of connections completed for the last successful iteration. Report 359 SHOULD also include: 361 - Connection Attempt Rate. 362 - Connection Step Count. 364 A log file MAY be generated which includes for each step iteration: 366 - Pass/Fail Status. 367 - Total connections established. 368 - Number of previously established connections dropped. 369 - Number of the additional connections that failed to complete. 371 5.2 Maximum Connection Setup Rate 373 5.2.1 Objective 375 To determine the maximum connection rate which can be supported 376 through the DUT/SUT. As with the Concurrent Connection Capacity test, 377 FTP,HTTP and SMTP sessions will be used to determine this metric. 379 5.2.2 Setup Parameters 381 The following parameters MUST be defined. Each test parameter is 382 configured with the following considerations. 384 Initial Attempt Rate - The rate at which the initial connection 385 requests are attempted. 387 Number of Connections - Defines the number of connections that must 388 be established. The number MUST be between the number of 389 participating virtual clients and the maximum number supported by the 390 DUT/SUT. It is RECOMMENDED not to exceed the concurrent connection 391 capacity found in section 5.1. The connection rate may vary depending 392 on the number of connections attempted. 394 Object/Message - Defines the number of bytes to be transferred across 395 each connection. 397 5.2.3 Procedure 399 An iterative search algorithm will be used to determine the maximum 400 connection rate. This test iterates through different connection rates 401 with a fixed number of connections attempted by the virtual clients to 402 their associated server(s). 404 Each iteration will use the same connection establishment and 405 connection validation algorithms defined in the concurrent capacity 406 test(See 5.1). After each iteration, the tester MUST close all 407 connections prior to continuing to the next iteration. 409 5.2.4 Measurements 411 The highest connection rate, in connections per second, for which 412 all connections completed successfully. Test equipment MUST be able 413 to track each connection to verify all required transaction between 414 the virtual client and server completed successfully. This includes 415 successful completion of both the command sequences and exchanging 416 of any data across each of those connections. 418 5.2.5 Reporting Format 420 The maximum connection rate reported MUST be the maximum rate for 421 which all connections successfully completed. 423 A log file MAY be generated which includes for each step iteration: 425 - Pass/Fail Status. 426 - Connection attempt rate. 427 - Number of the connections that failed to complete. 428 - Total connections established. 430 5.3 Connection Establishment Time 432 5.3.1 Objective 434 To characterize the connection establishment time[1] through or with 435 the DUT/SUT as a function of the number of open connections. 437 5.3.2 Setup Parameters 439 The following parameters MUST be defined. Each parameters is 440 configured with the following considerations. 442 Connection Attempt Rate - The rate at which new connection requests 443 are attempted. The rate SHOULD be set lower than maximum rate at 444 which the DUT/SUT can accept new connection requests. 446 Connection Step count - Defines the number of additional connections 447 attempted for each iteration of the step algorithm. 449 Object/Message - Defines the number of bytes to be transferred across 450 each connection. 452 5.3.3 Procedure 454 The test will use the same algorithm as defined in the Concurrent 455 Capacity Test. This includes both the connection establishment and 456 validation of each connection by transferring data across each 457 connection. 459 5.3.4 Measurement 461 For each iteration, the tester MUST measure the Min/Avg/Max connection 462 times for the additional connections. It is RECOMMENDED that in addition 463 to the application layer, the tester include measurements at the lower 464 layer protocols(i.e. - TCP, ATM) when applicable. For each of the 465 protocols which the tester is measuring, the connection establishment 466 time shall consist of all transactions required to enable data to be 467 transferred across the given connection. 469 For example, FTP requires the user to login prior to being able to get 470 files, view directories and so forth. Connection establishment times 471 MUST include all of these transactions. In the case of TCP, the 472 connection establishment time would consist of the three-way handshake 473 between the two hosts(See Appendix D). 475 5.3.5 Reporting Format 477 Graph of the min/avg/maximum connection establishment times versus the 478 number of open connections. The report MUST identify the layer for which 479 the measurement was taken(i.e. - Application, transport, etc). 481 5.4 Denial Of Service Handling 483 5.4.1 Objective 485 To determine the effect of a denial of service attack on connection 486 establishment rates through the DUT/SUT. The Denial Of Service Handling 487 test should be ran after obtaining baseline measurements from section 488 5.2. 490 When a normal TCP connection starts, a destination host receives a SYN 491 (synchronize/start)packet from a source host and sends back a SYN ACK 492 (synchronize acknowledge). The destination host must then hear an ACK 493 (acknowledge) of the SYN ACK before the connection is established. The 494 TCP SYN attack exploits this design by having an attacking source host 495 generate TCP SYN packets with random source addresses towards a victim 496 host, thereby consuming that hosts resources. 498 Some firewalls employ one or more mechanisms to guard against SYN 499 attacks. If such mechanisms exist on the DUT/SUT, tests SHOULD be ran 500 with these mechanisms enabled to determine how well the DUT/SUT can 501 maintain the baseline connection rates determined in section 5.2 502 under such attacks. 504 5.4.2 Setup Parameters 506 The following parameters MUST be defined. Each parameter is configured 507 with the following considerations. 509 Initial Attempt Rate - The rate at which the initial connection 510 requests are attempted. 512 Number of Connections - Defines the number of connections that must 513 be established. The number MUST be between the number of 514 participating clients and the maximum number supported by the 515 DUT/SUT. It is RECOMMENDED not to exceed the concurrent connection 516 capacity found in section 5.1. 518 SYN Attack Rate - Defines the rate at which the server(s) are targeted 519 with TCP SYN packets. 521 5.4.3 Procedure 523 This test uses the same procedure as defined in the maximum connection 524 setup rate, with the addition of TCP SYN packets targeting the 525 server(s) IP address or NAT proxy address. 527 The tester originating the TCP SYN attack MUST be attached to the 528 Unprotected network. In addition, the tester MUST not respond to the 529 SYN ACK packets sent by target server in response to the SYN packet. 531 5.4.4 Measurements 533 The highest connection rate, in connections per second, for which 534 all legitimate connections completed successfully. Test equipment 535 MUST be able to track each connection to verify all required 536 transaction between the virtual client and server completed 537 successfully. This includes successful completion of both the command 538 sequences and exchanging of any data across each of those connections. 540 In addition, the tester SHOULD track SYN packets associated with the 541 SYN attack which the DUT/SUT forwards on the protected or DMZ 542 interface(s). 544 5.4.5 Reporting Format 546 The maximum connection rate reported MUST be the maximum rate for 547 which all connections successfully completed. The report SHOULD 548 include what percentage of TCP SYN packets were forwarded by the 549 DUT/SUT. 551 A log file MAY be generated which includes for each step iteration: 553 - Pass/Fail Status. 554 - Connection attempt rate. 555 - Number of the connections that failed to complete. 556 - Total connections established. 558 5.5 Single Application Goodput 560 This section defined the procedures for base lining the Goodput[1] of the 561 DUT/SUT for FTP, HTTP and SMTP traffic. 563 5.5.1 FTP Goodput 565 5.5.1.1 Objective 567 The File Transfer Protocol is a common application used by companies 568 to transfer files from one device to another. Evaluating FTP Goodput 569 will allow individuals to measure how much successful traffic has 570 been forwarded by the DUT/SUT. 572 5.5.1.2 Setup Parameters 574 The following parameters MUST be defined. Each parameter is configured 575 with the following considerations. 577 Number of Connections - Defines the number of connections to be 578 opened for transferring data objects. Number MUST be equal or 579 greater than the number of virtual clients participating in the 580 test. The number SHOULD be a multiple of the virtual client 581 participating in the test. 583 Connection Rate - Defines the rate at which connections are established. 585 Object Size - Defines the number of bytes to be transferred across each 586 connection. 588 5.5.1.3 Procedure 590 Each virtual client will establish a FTP connection to its respective 591 server(s) in a round robin fashion at the connection rate. The 592 transaction involved in establishing the FTP connection should follow 593 the procedure defined in Appendix A. 595 After the login process has been completed, the virtual client will 596 initiate a file transfer by issuing a "Get" command. The "Get" 597 command will target a predefined file of Object Size bytes. 599 Once the file transfer has completed, the virtual client will close 600 the FTP connection by issuing the QUIT command. 602 5.5.1.4 Measurement 604 The Goodput for each interface of the DUT/SUT MUST be measured. See 605 appendix D for details in regards to measuring the Goodput of the 606 DUT/SUT. Only file transfers which have been completed are to be 607 included in the Goodput measurements. 609 The average transaction time each object successfully transferred MAY 610 be measured. The start time will begin when the time the "Get" 611 commands is initiated and end at the time when the client receives 612 an acknowledgment from the server that file transfer has completed. 614 5.5.1.5 Reporting Format 616 The Goodput for each interface of the DUT/SUT MUST be reported in 617 bits per second. This will be the aggregate of session Goodput's 618 measured for a given interface. 620 Failure analysis: 622 The report SHOULD include the percentage of connections that 623 failed. This includes: 625 - Connections which failed to establish 626 - Connections which failed to complete the object transfer 628 Transaction Processing analysis: 630 The report SHOULD include average transaction time in transactions 631 per second. 633 The report SHOULD also include the object size(Bytes) being transferred. 635 5.5.2 SMTP Goodput 637 5.5.2.1 Objective 639 Another application commonly in use today is the mail transfer. One 640 the common transport mechanisms for mail messages is the Simple Mail 641 Transfer Protocol(SMTP). The SMTP Goodput will allow individuals to 642 measure how much successful SMTP traffic has been forwarded by the 643 DUT/SUT. 645 5.5.2.2 Setup Parameters 647 The following parameters MUST be defined. Each parameter is 648 configured with the following considerations. 650 Number of Connections - Defines the number of connections to be 651 opened for transferring data objects. Number MUST be equal or 652 greater than the number of virtual clients participating in the 653 test. The number SHOULD be a multiple of the virtual client 654 participating in the test. 656 Connection Rate - Defines the rate at which connections are 657 attempted. 659 Message Size - Defines the number of bytes to be transferred across 660 each connection. 662 5.5.2.3 Procedure 664 Each virtual client will establish a SMTP connection to its 665 respective server(s) in a round robin fashion at the connection rate. 666 The transaction involved in establishing the SMTP connection should 667 follow the procedure defined in Appendix B. 669 After the greeting exchanges have been completed, the client will 670 initiate the transfer of the message by initiating the MAIL command. 671 The client will then send the predefined message of Object Size. 673 Once the message has been acknowledged as being received by the 674 server, the virtual client will then close the connection. 676 5.5.2.4 Measurement 678 The Goodput for each interface of the DUT/SUT MUST be measured. See 679 appendix D for details in regards to measuring the Goodput of the 680 DUT/SUT. Only message transfers which have been completed are to be 681 included in the Goodput measurements. 683 The average transaction time for each message transferred MAY be 684 measured. The start time will begin when the time the "MAIL" command 685 is initiated and end at the time when the client receives an 686 acknowledgment from the server that the message has been received. 688 5.5.2.5 Reporting Format 690 Goodput analysis: 692 The Goodput for each interface of the DUT/SUT MUST be reported in 693 bits per second. This will be the aggregate of session Goodput's 694 measured for a given interface. 696 Failure analysis: 698 The report SHOULD include the percentage of connections that 699 failed. This includes: 701 - Connections which failed to establish 702 - Connections which failed to complete the object transfer 704 Transaction Processing analysis: 706 The report SHOULD include average transaction time in transactions 707 per second. 709 The report SHOULD also include the object size(Bytes) being transferred. 711 5.5.3 HTTP Goodput Goodput 713 5.5.3.1 Objective 715 Another common application is the World Wide Web (WWW) application 716 that can access documents over the Internet. This application uses 717 the Hypertext Transfer Control Protocol (HTTP) to copy information 718 from one system to another. 720 HTTP Goodput measurement is actually determined by evaluating the 721 Forwarding rate of packets. This is based on measuring only data 722 that has successfully been forwarded to the destination interface. 724 When benchmarking the performance of the DUT/SUT, consideration of 725 the HTTP version being used must be taken into account. Appendix C 726 of this document discusses enhancements to the HTTP protocol which 727 may impact performance results. 729 5.5.3.2 Setup Parameters 731 The following parameters MUST be defined. Each variable is 732 configured with the following considerations. 734 Number of Connections - Defines the number of HTTP connections 735 to be opened for transferring data objects. Number MUST be equal or 736 greater than the number of virtual clients participating in the 737 test. The number SHOULD be a multiple of the virtual client 738 participating in the test. 740 Connection Rate - Defines the rate at which connections are 741 attempted. 743 Object Size - Defines the number of bytes to be transferred 744 across each connection. 746 5.5.3.3 HTTP Procedure 748 For the HTTP Goodput tests, it is RECOMMENDED to determine which 749 version of HTTP the DUT/SUT has implemented and use the same 750 version for the test. To determine the version of HTTP, the user 751 documentation of the DUT/SUT SHOULD be consulted. 753 Each client will attempt to establish HTTP connection's to their 754 respective servers a user defined rate. The clients will attach to 755 the servers using either the servers IP address or NAT proxy 756 address. 758 After the client has established the connection with the server, 759 the client will initiate GET command(s) to retrieve predefined 760 web page(s). 762 When employing HTTP/1.0 in benchmarking the performance of the 763 DUT/SUT, only one object will be retrieved for each of the defined 764 object sizes. After the object has been transferred, the connection 765 should then be torn down. When defining multiple objects, object 766 transfers must be completed and the connections closed for all 767 of the participating clients prior testing the next object size. 768 This process is repeated until all of the defined objects are 769 tested. 771 When employing HTTP/1.1, all objects defined by the user will 772 be requested with a GET request over the same connection. The 773 connection should then be torn down after all of the objects 774 have been transferred. 776 5.5.3.4 Measurement 778 The Goodput for each of the objects sizes transferred MUST be 779 measured. See appendix D for details in regards to measuring the 780 Goodput of the DUT/SUT. Only objects which have been successfully 781 acknowledged by the server are to be included in the Goodput 782 measurements. 784 The transaction times for each object transferred MUST measured. 785 The transaction connection time starts when the connection is 786 made and will end when the web pages is completely mapped on the 787 virtual client (when the client sends an acknowledgment packet is 788 sent from the client). 790 5.5.3.5 Reporting Format 792 Goodput analysis: 794 The Goodput for each interface of the DUT/SUT MUST be reported in 795 bits per second. This will be the aggregate of session Goodput's 796 measured for a given interface. 798 Failure analysis: 800 The report SHOULD include the percentage of connections that 801 failed. This includes: 803 - Connections which failed to establish 804 - Connections which failed to complete the object transfer 806 Transaction Processing analysis: 808 The report SHOULD include average transaction time in transactions 809 per second. 811 The report SHOULD also include the object size(Bytes) being transferred. 813 Version Information 815 Report MUST include the version of HTTP used for the test. In 816 addition, if the HTTP/1.1 is used, the number of concurrent GET's 817 allowable(Pipelining) SHOULD be reported. 819 5.6 Concurrent Application Goodput 821 5.6.1 Objective 823 To determine the Goodput of the DUT/SUT when offering a mix of FTP, 824 SMTP and HTTP traffic flows. Real world traffic does not consist 825 of a single protocol, but a mix of different applications. This 826 test will allow an individual to determine how well the DUT/SUT 827 handles a mix of applications by comparing the results to the 828 individual baseline measurements. 830 5.6.2 Setup Parameters 832 The following parameters MUST be defined. Each variable is 833 configured with the following considerations. 835 Number of Connections - Defines the aggregate number of connections 836 to be opened for transferring data/message objects. Number MUST be 837 equal to or greater than the number of virtual clients participating 838 in the test. The number SHOULD be a multiple of the virtual client 839 participating in the test. 841 Connection Rate - Defines the rate at which connections attempts are 842 opened. Number MUST be evenly divided among all of the virtual 843 clients participating in the test. 845 Object/Message Size - Defines the number of bytes to be transferred 846 across each connection. RECOMMENDED message sizes still needs to be 847 determined. 849 At a minimum, at least one of the following parameters MUST be 850 defined. In addition, the cumulative percentage all the defined 851 percentages MUST equal 100%. 853 FTP Percentage - Defines the percentage of traffic connections 854 which are to consist of FTP file transfers. 856 SMTP Percentage - Defines the percentage of traffic connections 857 which are to consist of SMTP Message transfers. 859 HTTP Percentage - Defines the percentage of traffic connections 860 which are to consist of HTTP GET requests. 862 5.6.3 Procedure 864 This test will run each of the single application Goodput tests, 865 for which there is a defined percentage, concurrently. For each 866 of the defined traffic types, the connection establishment, 867 data/message transfer and teardown procedures will be the same 868 as defined in the individual tests. 870 5.6.4 Measurements 872 As with the individual tests, the Goodput for each of the defined 873 traffic types MUST be measured. See appendix D for details in 874 regards to measuring the Goodput of the DUT/SUT. Only messages/data 875 which have been successfully acknowledged as being transferred are 876 to be included in the Goodput measurements. 878 The transaction times for each of the defined applications MUST be 879 measured. See the appropriate single application Goodput test for 880 the specifics of measuring the transaction times for each of the 881 defined traffic types. 883 5.6.5 Reporting Format 885 Goodput analysis: 886 Reporting SHOULD include a graph of the number of connections 887 versus the measured Goodput in Mbps for each of the defined 888 traffic types(FTP, SMTP, HTTP). 890 Failure analysis: 891 Reporting should include a graph of number of connections versus 892 percent success for each of the defined traffic types. 894 Transaction Processing analysis: 895 Reporting should include a graph of number of virtual connections 896 versus average transaction for each of the defined traffic types. 898 5.7 Illegal Traffic Handling 900 To determine the behavior of the DUT/SUT when presented with a 901 combination of both legal and Illegal traffic. 903 5.7.1 Procedure 905 Still Needs to be determined 907 5.7.2 Measurements 909 Still Needs to be determined 911 5.7.3 Reporting Format 913 Still Needs to be determined 915 5.8 Latency 917 Determine the latency of application layer data through the DUT/SUT. 919 5.8.1 Procedure 921 Still Needs to be determined 923 5.8.2 Measurements 925 Still needs to be determined. 927 5.8.3 Reporting format 929 Still needs to be determined. 931 APPENDICES 933 APPENDIX A: FTP(File Transfer Protocol) 935 A.1 Introduction 937 The FTP protocol was designed to be operated by interactive end users 938 or application programs. The communication protocol to transport this 939 service is TCP. The core functions of this application enable users 940 to copy files between systems, view directory listings and perform 941 house keeping chores - such as renaming, deleting and copying files. 942 Unlike other protocols, FTP uses two connections. One connection, 943 called the control connection, is used to pass commands between 944 the client and the server. The other, called the data connection, is 945 used to transfer the actual data(Files, directory lists, etc.). 947 A.2 Connection Establishment/Teardown(Control) 949 FTP control connections are established by issuing OPEN command 950 targeting either the URL or a specific IP address. Since the 951 methodology does not include DNS servers, OPEN commands should 952 target specific IP address of target server. A TCP connection 953 will be established between the client and target server. 955 The client will then begin the login process. When logging in, 956 it is RECOMMENDED to perform the test using Anonymous FTP Login 957 and should use the following syntax: 959 User ID: Anonymous 960 Password: will correspond to the System ID 961 (client1_1@test.net through client 1_8@test.net) 963 Once a successful login acknowledgment is received from the server, 964 the client may then initiate a file transfer. After all transfer 965 operations have been completed, the FTP connection may be closed by 966 issuing a QUIT command. 968 A.3 Data Connection 970 The data connection is established each time the user requests a file 971 transfer and torn down when the transfer is completed. FTP supports 972 two modes of operation, namely normal mode and passive mode, which 973 determine who initiates the data connection. In normal mode 974 operation, the server initiates the data connection, targeting a 975 predefined PortID specified in the PORT command. In passive mode, the 976 client initiates the data connection, targeting the PortID returned 977 in response to the PASV Command. It is RECOMMENDED to perform the 978 tests in normal mode operation. 980 File transfers are initiated by using the "Get" or "Put" command and 981 specifying the desired filename. The tests defined in this document 982 will use the "Get" command to initiate file transfers from the target 983 server to the client. 985 A.4 Object Format 987 Need to define the object format. 989 APPENDIX B: SMTP (Simple Mail Transfer Protocol) 991 B.1 Introduction 993 The SMTP defines a simple straight forward way to move messages 994 between hosts. There are two roles in the SMTP protocol, one is the 995 sender and one is the receiver. The sender acts like a client and 996 establishes a TCP connection with the receiver which acts like a 997 server. The transactions defined in this section will use the terms 998 client and server in place of sender and receiver. 1000 B.2 Connection Establishment/Teardown 1002 Each connection involves a connection greeting between the 1003 sender(Client) and receiver(Server). The syntax used to identify each 1004 other's hostnames during this greeting exchange SHOULD be: 1006 "SMTPRcv_.com" 1007 "SMTPSender_.com" 1009 where and represent a 1010 unique virtual number for the client and server 1011 respectively. 1013 The basic transactions in moving mail between two hosts involve three 1014 basic steps which are outlined below. These are: 1016 1) Client issuing a MAIL command identifying the message originator 1017 for that session. Syntax used to identify the originator SHOULD 1018 be as follows: 1020 connection1,2,3...@hostname 1022 2) Client issues an RCPT command identifying the recipient of the 1023 message for that session. Syntax used to identify the recipient 1024 of the message SHOULD be as follows: 1026 reciever1,2,3...@hostname 1028 3) Client issues a DATA command. After receiving the 1029 acknowledgment from the server, the client will then transfer 1030 the message which MUST include a line with a period to 1031 indicate to the server the end of the message. Once the end of 1032 message is received by the server, it will acknowledge the end 1033 of message. 1035 The client may initiate another message transfer or close the session 1036 by initiating the QUIT command. 1038 B.3 Message Format 1040 As Internet e-mail has evolved, SMTP extensions have been added to 1041 support both audio and video message transfers. For these firewall 1042 tests, messages SHOULD consist of plain text ASCII. 1044 APPENDIX C: HTTP(HyperText Transfer Protocol) 1046 C.1 Introduction 1048 As HTTP has evolved over the years, changes to the protocol have 1049 occurred to both fix problems of previous versions as well as improve 1050 performance. The most common versions in use today are HTTP/1.0 and 1051 HTTP/1.1 and are and are discussed below. 1053 C.2 Version Considerations 1055 HTTP/1.1 was approved by the WWW Consortium in July 1999 as an IETF 1056 Draft Standard. This is a formal recognition of the fact that all 1057 known technical issues have been resolved in the specification which 1058 was brought out in June 1997. HTTP/1.1 is also downward compatible 1059 with HTTP/1.0. Both protocols on the popular browsers in use today. 1060 The following is a list of features that are offered in HTTP 1.1 that 1061 are not in HTTP 1.0. 1063 - Persistent connections and pipelining 1065 Though both use TCP for data transfer, but differ in the way it is 1066 used, with the later version being more efficient. Once a connection 1067 is opened, it is not closed until the HTML document and all objects 1068 referred by it are downloaded. This technique is called persistent 1069 connection. By serving multiple requests on the same TCP segment, 1070 many control packets (which are not part of actual data transfer) 1071 are avoided. The technique of containing multiple requests and 1072 responses within the same TCP segment over a persistent connection 1073 is called pipelining. 1075 - Data compression 1077 HTTP/1.1 provides for compression of documents before file transfer. 1078 Since most other objects like images and binaries are already 1079 compressed, this feature applies only to HTML and plain text 1080 documents. 1082 - Range request and validation 1084 Bandwidth saving measure is the introduction of two new fields in 1085 an HTTP request header, viz. If-Modified-Since: and If-Unmodified- 1086 Since:. The significance of this feature is that if a browser 1087 identifies a file in its cache, it needn't reload it unless it has 1088 changed since the last time it was used. 1090 - Support for multiple hosts 1092 It is common for an ISP to host more than one Web site on a single 1093 server. In such a case, each domain requires its own IP address. 1095 C.3 Object Format 1097 Object SHOULD be an HTML formatted object. 1099 Append D. GOODPUT Measurements. 1101 The Goodput will measure the number of bits per second forwarded by 1102 the DUT/SUT and will be referenced to the application level data. The 1103 formula for determining Goodput of the DUT/SUT is as follows: 1105 ObjectSize(Bytes) * 8 1106 Goodput(Bits/Sec) = Transfer Time(Seconds) 1108 Transfer Time starts when the first bit of the object/message is 1109 received at the destination port of the tester. The transfer time ends 1110 when the last bit of the object/message is received at the destination 1111 port of the tester. 1113 Appendix E. References 1115 [1] Newman, "Benchmarking Terminology for Firewall Devices", RFC 2647, 1116 February 1998. 1118 [2] J. Postel, "Simple Mail Transfer Protocol", RFC 821, August 1982. 1120 [3] R. Fielding, J. Gettys, J. Mogul, H Frystyk, T. Berners, "Hypertext 1121 Transfer Protocol -- HTTP/1.1", January 1997 1123 [4] J. Postel, J. Reynolds, "File Transfer Protocol(FTP)", October 1985