idnits 2.17.1 draft-ietf-bmwg-firewall-03.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** Looks like you're using RFC 2026 boilerplate. This must be updated to follow RFC 3978/3979, as updated by RFC 4748. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- == No 'Intended status' indicated for this document; assuming Proposed Standard == The page length should not exceed 58 lines per page, but there was 1 longer page, the longest (page 10) being 114 lines == It seems as if not all pages are separated by form feeds - found 0 form feeds but 23 pages Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack an Abstract section. ** The document seems to lack a Security Considerations section. ** The document seems to lack an IANA Considerations section. (See Section 2.2 of https://www.ietf.org/id-info/checklist for how to handle the case when there are no actions for IANA.) ** The document seems to lack an Authors' Addresses Section. ** There are 50 instances of too long lines in the document, the longest one being 7 characters in excess of 72. ** There are 8 instances of lines with control characters in the document. == There are 12 instances of lines with non-RFC6890-compliant IPv4 addresses in the document. If these are example addresses, they should be changed. Miscellaneous warnings: ---------------------------------------------------------------------------- == Line 803 has weird spacing: '... a row for e...' == Using lowercase 'not' together with uppercase 'MUST', 'SHALL', 'SHOULD', or 'RECOMMENDED' is not an accepted usage according to RFC 2119. Please use uppercase 'NOT' together with RFC 2119 keywords (if that is what you mean). Found 'MUST not' in this paragraph: The tester originating the TCP SYN attack MUST be attached to the unprotected network. In addition, the tester MUST not respond to the SYN/ACK packets sent by target server in response to the SYN packet. -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (October 2001) is 8230 days in the past. Is this intentional? -- Found something which looks like a code comment -- if you have code sections in the document, please surround them with '' and '' lines. Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) -- Missing reference section? '1' on line 1130 looks like a reference -- Missing reference section? '5' on line 1143 looks like a reference -- Missing reference section? '3' on line 1137 looks like a reference -- Missing reference section? '4' on line 1140 looks like a reference -- Missing reference section? '2' on line 1133 looks like a reference Summary: 7 errors (**), 0 flaws (~~), 6 warnings (==), 8 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 Network Working Group Brooks Hickman 2 Internet-Draft Spirent Communications 3 Expiration Date: April 2002 David Newman 4 Network Test 5 Saldju Tadjudin 6 Spirent Communications 7 Terry Martin 8 M2networx INC 9 October 2001 11 Benchmarking Methodology for Firewall Performance 12 14 Status of this Memo 16 This document is an Internet-Draft and is in full conformance with 17 all provisions of Section 10 of RFC2026. 19 Internet-Drafts are working documents of the Internet Engineering 20 Task Force (IETF), its areas, and its working groups. Note that 21 other groups may also distribute working documents as Internet- 22 Drafts. 24 Internet-Drafts are draft documents valid for a maximum of six 25 months and may be updated, replaced, or obsoleted by other documents 26 at any time. It is inappropriate to use Internet-Drafts as 27 reference material or to cite them other than as "work in progress." 29 The list of current Internet-Drafts can be accessed at 30 http://www.ietf.org/ietf/1id-abstracts.txt 32 The list of Internet-Draft Shadow Directories can be accessed at 33 http://www.ietf.org/shadow.html. 35 Table of Contents 37 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 38 2. Requirements . . . . . . . . . . . . . . . . . . . . . . . . 2 39 3. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 40 4. Test setup . . . . . . . . . . . . . . . . . . . . . . . . . 2 41 4.1 Test Considerations . . . . . . . . . . . . . . . . . . 3 42 4.2 Virtual Client/Servers . . . . . . . . . . . . . . . . . 3 43 4.3 Test Traffic Requirements . . . . . . . . . . . . . . . . 4 44 4.4 DUT/SUT Traffic Flows . . . . . . . . . . . . . . . . . . 4 45 4.5 Multiple Client/Server Testing . . . . . . . . . . . . . 5 46 4.6 NAT(Network Address Translation) . . . . . . . . . . . . 5 47 4.7 Rule Sets . . . . . . . . . . . . . . . . . . . . . . . . 5 48 4.8 Web Caching . . . . . . . . . . . . . . . . . . . . . . . 5 49 4.9 Authentication . . . . . . . . . . . . . . . . . . . . . 6 50 5. Benchmarking Tests . . . . . . . . . . . . . . . . . . . . . 6 51 5.1 Concurrent Connection Capacity . . . . . . . . . . . . . 6 52 5.2 Maximum Connection Setup Rate . . . . . . . . . . . . . . 7 53 5.3 Connection Establishment Time . . . . . . . . . . . . . . 9 54 5.4 Connection Teardown Time . . . . . . . . . . . . . . . . 11 55 5.5 Denial Of Service Handling . . . . . . . . . . . . . . . 13 56 5.6 HTTP . . . . . . . . . . . . . . . . . . . . . . . . . . 14 57 5.7 IP Fragmentation Handling . . . . . . . . . . . . . . . . 16 58 5.8 Illegal Traffic Handling . . . . . . . . . . . . . . . . 18 59 5.9 Latency . . . . . . . . . . . . . . . . . . . . . . . . . 19 60 Appendices . . . . . . . . . . . . . . . . . . . . . . . . . . 22 61 A. HyperText Transfer Protocol(HTTP) . . . . . . . . . . . . 22 62 B. References . . . . . . . . . . . . . . . . . . . . . . . . 23 64 1. Introduction 66 This document provides methodologies for the performance 67 benchmarking of firewalls. It provides methodologies in four areas: 68 forwarding, connection, latency and filtering. In addition to 69 defining the tests, this document also describes specific formats 70 for reporting the results of the tests. 72 A previous document, "Benchmarking Terminology for Firewall 73 Performance" [1], defines many of the terms that are used in this 74 document. The terminology document SHOULD be consulted before 75 attempting to make use of this document. 77 2. Requirements 79 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 80 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in 81 this document are to be interpreted as described in RFC 2119. 83 3. Scope 85 Firewalls can provide a single point of defense between networks. 86 Usually, a firewall protects private networks from the public or 87 shared networks to which it is connected. A firewall can be as 88 simple as a device that filters different packets or as complex 89 as a group of devices that combine packet filtering and 90 application-level proxy or network translation services. This RFC 91 will focus on developing benchmark testing of DUT/SUTs, wherever 92 possible, independent of their implementation. 94 4. Test Setup 96 Test configurations defined in this document will be confined to 97 dual-homed and tri-homed as shown in figure 1 and figure 2 98 respectively. 100 Firewalls employing dual-homed configurations connect two networks. 101 One interface of the firewall is attached to the unprotected 102 network, typically the public network(Internet). The other interface 103 is connected to the protected network, typically the internal LAN. 105 In the case of dual-homed configurations, servers which are made 106 accessible to the public(Unprotected) network are attached to the 107 private(Protected) network. 109 +----------+ +----------+ 110 | | | +----------+ | | | 111 | Servers/ |----| | | |------| Servers/ | 112 | Clients | | | | | | Clients | 113 | | |-------| DUT/SUT |--------| | | 114 +----------+ | | | | +----------+ 115 | +----------+ | 116 Protected | | Unprotected 117 Network Network 118 Figure 1(Dual-Homed) 120 Tri-homed[1] configurations employ a third segment called a DMZ. 121 With tri-homed configurations, servers accessible to the public 122 network are attached to the DMZ. Tri-Homed configurations offer 123 additional security by separating server accessible to the public 124 network from internal hosts. 126 +----------+ +----------+ 127 | | | +----------+ | | | 128 | Clients |----| | | |------| Servers/ | 129 | | | | | | | Clients | 130 +----------+ |-------| DUT/SUT |--------| | | 131 | | | | +----------+ 132 | +----------+ | 133 Protected | | | Unprotected 134 Network | Network 135 | 136 | 137 ----------------- 138 | DMZ 139 | 140 | 141 +-----------+ 142 | | 143 | Servers | 144 | | 145 +-----------+ 147 Figure 2(Tri-Homed) 149 4.1 Test Considerations 151 4.2 Virtual Clients/Servers 153 Since firewall testing may involve data sources which emulate 154 multiple users or hosts, the methodology uses the terms virtual 155 clients/servers. For these firewall tests, virtual clients/servers 156 specify application layer entities which may not be associated with 157 a unique physical interface. For example, four virtual clients may 158 originate from the same data source[1]. The test report SHOULD 159 indicate the number of virtual clients and virtual servers 160 participating in the test on a per interface(See 4.0) basis. 162 Testers MUST synchronize all data sources participating in a test. 164 4.3 Test Traffic Requirements 166 While the function of a firewall is to enforce access control 167 policies, the criteria by which those policies are defined vary 168 depending on the implementation. Firewalls may use network layer, 169 transport layer or, in many cases, application-layer criteria to 170 make access-control decisions. Therefore, the test equipment used to 171 benchmark the DUT/SUT performance MUST consist of real clients and 172 servers generating legitimate layer seven conversations. 174 For the purposes of benchmarking firewall performance, HTTP 1.1 175 will be referenced in this document, although the methodologies 176 may be used as a template for benchmarking with other applications. 177 Since testing may involve proxy based DUT/SUTs, HTTP version 178 considerations are discussed in appendix A. 180 4.4 DUT/SUT Traffic Flows 182 Since the number of interfaces are not fixed, the traffic flows will 183 be dependent upon the configuration used in benchmarking the 184 DUT/SUT. Note that the term "traffic flows" is associated with 185 client-to- server requests. 187 For Dual-Homed configurations, there are two unique traffic flows: 189 Client Server 190 ------ ------ 191 Protected -> Unprotected 192 Unprotected -> Protected 194 For Tri-Homed configurations, there are three unique traffic flows: 196 Client Server 197 ------ ------ 198 Protected -> Unprotected 199 Protected -> DMZ 200 Unprotected -> DMZ 202 4.5 Multiple Client/Server Testing 204 One or more clients may target multiple servers for a given 205 application. Each virtual client MUST initiate requests(Connection, 206 object transfers, etc.) in a round-robin fashion. For example, if 207 the test consisted of six virtual clients targeting three servers, 208 the pattern would be as follows: 210 Client Target Server(In order of request) 211 #1 1 2 3 1... 212 #2 2 3 1 2... 213 #3 3 1 2 3... 214 #4 1 2 3 1... 215 #5 2 3 1 2... 216 #6 3 1 2 3... 218 4.6 NAT(Network Address Translation) 220 Many firewalls implement network address translation(NAT), a 221 function which translates internal host IP addresses attached to 222 the protected network to a virtual IP address for communicating 223 across the unprotected network(Internet). This involves additional 224 processing on the part of the DUT/SUT and may impact performance. 225 Therefore, tests SHOULD be ran with NAT disabled and NAT enabled 226 to determine the performance differentials. The test report SHOULD 227 indicate whether NAT was enabled or disabled. 229 4.7 Rule Sets 231 Rule sets[1] are a collection of access control policies that 232 determines which packets the DUT/SUT will forward and which it will 233 reject[1]. The criteria by which these access control policies may 234 be defined will vary depending on the capabilities of the DUT/SUT. 235 The scope of this document is limited to how the rule sets should 236 be applied when testing the DUT/SUT. 238 The firewall monitors the incoming traffic and checks to make sure 239 that the traffic meets one of the defined rules before allowing it 240 to be forwarded. It is RECOMMENDED that a rule be entered for each 241 host(Virtual client). Although many firewalls permit groups of IP 242 addresses to be defined for a given rule, tests SHOULD be performed 243 with large rule sets, which are more stressful to the DUT/SUT. 245 The DUT/SUT SHOULD be configured to denies access to all traffic 246 which was not previously defined in the rule set. 248 4.7 Web Caching 250 Some firewalls include caching agents in order to reduce network 251 load. When making a request through a caching agent, the caching 252 agent attempts to service the response from its internal memory. 254 The cache itself saves responses it receives, such as responses 255 for HTTP GET requests. The report SHOULD indicate whether caching 256 was enabled or disabled on the DUT/SUT. 258 4.8 Authentication 260 Access control may involve authentication processes such as user, 261 client or session authentication. Authentication is usually 262 performed by devices external to the firewall itself, such as an 263 authentication servers and may add to the latency of the system. 264 Any authentication processes MUST be included as part of connection 265 setup process. 267 5. Benchmarking Tests 269 5.1 Concurrent Connection Capacity 271 5.1.1 Objective 273 To determine the maximum number of TCP concurrent connections through 274 or with the DUT/SUT, as defined in RFC2647[1]. This test will employ 275 a step algorithm to obtain the maximum number of concurrent TCP 276 connections that the DUT/SUT can maintain. 278 5.1.2 Setup Parameters 280 The following parameters MUST be defined for all tests. 282 Connection Attempt Rate - The rate, expressed in connections per 283 second, at which new TCP connection requests are attempted. The 284 rate SHOULD be set lower than maximum rate at which the DUT/SUT can 285 accept connection requests. 287 Connection Step Count - Defines the number of additional TCP 288 connections attempted for each iteration of the step search 289 algorithm. 291 Object Size - Defines the number of bytes to be transferred in 292 response to a HTTP 1.1 GET request . It is RECOMMENDED to use 293 1-byte object sizes for this test. 295 5.1.3 Procedure 297 Each virtual client will attempt to establish TCP connections to its 298 target server(s), using either the target server's IP address or NAT 299 proxy address, at a fixed rate in a round robin fashion. Each 300 iteration will involve the virtual clients attempting to establish a 301 fixed number of additional TCP connections. This search algorithm 302 will be repeated until either: 304 - One or more of the additional connection attempts fail to 305 complete. 306 - One or more of the previously established connections fail. 308 The test MUST include application layer data transfers in order to 309 validate the TCP connections since, in the case of proxy based 310 DUT/SUTs, the tester does not own both sides of the connection. 311 After all the addition connections have been attempted for each 312 iteration of the test, the virtual client(s) will request an 313 object from its target server(s) using an HTTP 1.1 GET request on 314 the additional connections as well as all previously established 315 connections. Both the client request and server response MUST exclude 316 the connection-token close in the connection header(See Appendix A). 318 5.1.4 Measurements 320 Maximum concurrent connections - Total number of TCP connections 321 open for the last successful iteration performed in the search 322 algorithm. 324 5.1.5 Reporting Format 326 5.1.5.1 Application-Layer Reporting: 328 The test report MUST note the use of HTTP 1.1 client and server 329 and the object size. 331 5.1.5.2 Transport-Layer Reporting: 333 The test report MUST note the connection attempt rate, connection 334 step count and maximum concurrent connections measured. 336 5.1.5.3 Log Files 338 A log file MAY be generated which includes the TCP connection 339 attempt rate, connection step count, object size and for each 340 iteration: 342 - Step Iteration 343 - Pass/Fail Status. 344 - Total TCP connections established. 345 - Number of previously established TCP connections that failed. 346 - Number of the additional TCP connections that failed to 347 complete. 349 5.2 Maximum Connection Setup Rate 351 5.2.1 Objective 353 To determine the maximum TCP connection setup rate through or with 354 the DUT/SUT, as defined by RFC2647[1]. This test will employ a 355 search algorithm to obtain the maximum rate at which TCP connections 356 can be established through or with the DUT/SUT. 358 5.2.2 Setup Parameters 360 The following parameters MUST be defined. 362 Initial Attempt Rate - The rate, expressed in connections per 363 second, at which the initial TCP connection requests are attempted. 365 Number of Connections - Defines the number of TCP connections that 366 must be established. The number MUST be between the number of 367 participating virtual clients and the maximum number supported by 368 the DUT/SUT. 370 Object Size - Defines the number of bytes to be transferred in 371 response to a HTTP 1.1 GET request . It is RECOMMENDED to use 372 1-byte object sizes for this test. 374 Age Time - The time, expressed in seconds, the DUT/SUT will keep a 375 connection in it's state table after receiving a TCP FIN or RST 376 packet. 378 5.2.3 Procedure 380 An iterative search algorithm will be used to determine the maximum 381 connection rate. This test iterates through different connection rates 382 with a fixed number of connections attempted by the virtual clients to 383 their associated server(s). 385 Each iteration will use the same connection establishment and 386 connection validation algorithms defined in the concurrent capacity 387 test(See section 5.1). 389 Between each iteration of the test, the tester must close all 390 connections completed for the previous iteration. In addition, 391 it is RECOMMENDED to abort all unsuccessful connections attempted. 392 The tester will wait for the period of time, specified by age time, 393 before continuing to the next iteration. 395 5.2.4 Measurements 397 Highest connection rate - Highest rate, in connections per second, 398 for which all TCP connections completed successfully. 400 5.2.5 Reporting Format 402 5.1.5.1 Application-Layer Reporting: 404 The test report MUST note the use of HTTP 1.1 client and server 405 and the object size. 407 5.2.5.2 Transport-Layer Reporting: 409 The test report MUST note the number of connections, age time 410 and highest connection rate measured. 412 5.2.5.3 Log Files 414 A log file MAY be generated which includes the number of TCP 415 connections attempt, age time, object size and 416 for each iteration: 418 - Step Iteration 419 - Pass/Fail Status. 420 - Attempted Connection Establishment Rate 421 - Total TCP connections established. 422 - Number of TCP connections that failed to complete. 424 5.3 Connection Establishment Time 426 5.3.1 Objective 428 To determine the connection establishment times[1] through or with 429 the DUT/SUT. 431 A connection for a client/server application is not atomic, in that 432 it not only involves transactions at the application layer, but 433 involves first establishing a connection using one or more underlying 434 connection oriented protocols(TCP, ATM, etc). Therefore, it is 435 encouraged to make separate measurements for each connection oriented 436 protocol required in order to perform the application layer 437 transaction. 439 5.3.2 Setup Parameters 441 The following parameters MUST be defined. 443 Connection Attempt Rate - The rate, expressed in connections per 444 second, at which new TCP connection requests are attempted. It is 445 RECOMMENDED not to exceed the maximum connection rate found in 446 section 5.2. 448 Connection Attempt Step count - Defines the number of additional 449 TCP connections attempted for each iteration of the step algorithm. 451 Maximum Attempt Connection Count - Defines the maximum number of 452 TCP connections attempted in the test. 454 5.3.3 Procedure 456 Each virtual client will attempt to establish TCP connections to its 457 target server(s) at a fixed rate in a round robin fashion. Each 458 iteration will involve the virtual clients attempting to establish 459 a fixed number of additional connections until the maximum attempt 460 connection count is reached. 462 After each connection has been completed, the virtual client(s) MUST 463 request a 1-byte object from its target server(s) using an HTTP 1.1 464 GET request. Both the client request and server response MUST exclude 466 [Page 9] 467 the connection-token close in the connection header(See Appendix A). 469 Since testing may involve proxy based DUT/SUTs, which terminates the 470 TCP connection, making a direct measurement of the TCP connection 471 establishment time is not possible since the protocol involves an 472 odd number of messages in establishing a connection. Therefore, when 473 testing with proxy based firewalls, the datagram following the final 474 ACK on the three-way handshake will be used in determining the 475 connection setup time. 477 The following shows the timeline for the TCP connection setup 478 involving a proxy DUT/SUT and is referenced in the measurements 479 section(5.3.4). Note that this methodology may be applied when 480 measuring other connection oriented protocols involving an odd number 481 of messages in establishing a connection. 483 t0: Client sends a SYN. 484 t1: Proxy sends a SYN/ACK. 485 t2: Client sends the final ACK. 486 t3: Proxy establishes separate connection with server. 487 t4: Client sends TCP datagram to server. 488 *t5: Proxy sends ACK of the datagram to client. 490 * While t5 is not considered part of the TCP connection establishment, 491 acknowledgement of t4 must be received for the connection to be 492 considered successful. 494 When comparing firewalls with different architectures, such as proxy 495 based and stateful packet filtering, the same method SHOULD be used 496 when measuring establishment times. 498 5.3.4 Measurements 500 For each iteration of the test, the tester MUST measure the minimum, 501 maximum and average TCP connection establishment times. If the DUT/SUT 502 is proxy based, the connection establishment time is considered to be 503 from the time the first bit of the first SYN packet is transmitted by 504 the client to the time the client transmits the first bit of the first 505 acknowledged TCP datagram(t4-t0 in the above timeline). For non-proxy 506 based DUT/SUTs , the establishment time shall be directly measured and 507 is considered to be from the time the first bit of the first SYN packet 508 is transmitted by the client to the time the last bit of the final ACK 509 in the three-way handshake is received by the target server. 511 In addition, the tester SHOULD measure the minimum, maximum and 512 average connection establishment times for all other underlying 513 connection oriented protocols. For purposes of benchmarking 514 firewall performance, the connection establishment time will be 515 considered the interval between the transmission of the first bit 516 of the first octet of the packet carrying the connection request 517 to the DUT/SUT interface to receipt of the last bit of the last 518 octet of the last packet of the connection setup traffic received 519 on the client or server, depending on whether a given connection 520 requires an even or odd number of messages, respectfully. 522 Tester SHOULD measure the aggregate connection time and the total 523 number of connections completed for all measured protocols for each 524 iteration of the test. 526 5.3.5 Reporting Format 528 5.3.5.1 Application-Layer Reporting: 530 The test report MUST note the use of HTTP 1.1 client and server. 532 5.3.5.2 Transport-Layer and Below Reporting: 534 The test report MUST note the TCP connection attempt rate, TCP 535 connection attempt step count and maximum TCP connections attempted. 537 For each connection oriented protocol the tester measured, the 538 connection establishment time results SHOULD be in tabular form 539 with a row for each iteration of the test. There SHOULD be a column 540 for the iteration count, minimum connection establishment time, 541 average connection establishment time, maximum connection 542 establishment time, attempted connections completed and aggregate 543 connection time. 545 The report MUST also identify the layer/protocol for which the 546 measurements were made. 548 5.4 Connection Teardown Time 550 5.4.1 Objective 552 To determine the connection teardown time[1] through or with the 553 DUT/SUT. As with the connection establishment time, separate 554 measurements SHOULD be taken for each connection oriented protocol 555 involved in closing a connection. 557 5.4.2 Setup Parameters 559 The following parameters MUST be defined. Each parameters is 560 configured with the following considerations. 562 Initial connections - Defines the number of TCP connections to 563 initialize the test with. 565 Teardown attempt rate - The rate at which the tester will attempt 566 to tear down TCP connections. 568 5.4.3 Procedure 570 The virtual clients will initialize the test by establishing TCP 571 connections defined by initial connections. The test will use the 572 same algorithm for establishing the TCP connections as described in 573 the connection capacity test(Section 5.1) with the exception that 574 no object transfers are required. 576 The virtual clients will then attempt to tear down all of TCP 577 connections at a rate defined by tear down attempt rate. The 578 tester(Virtual Clients) MUST exclude any connections which do not 579 properly close in its measurements. For example, connections in 580 which the DUT/SUT transmits a TCP RST in response to a TCP FIN 581 packet or connections which do not acknowledge the FIN packet 582 requesting the connection be closed. 584 In the case of proxy based DUT/SUTs, the DUT/SUT will itself receive 585 the final ACK when closing out it's side of the TCP connection. For 586 validation purposes, the virtual client(s) SHOULD verify that the 587 DUT/SUT received the final ACK in the connection tear down exchange 588 for all connections by transmitting a TCP datagram(s) referencing 589 the previously town down connection(s). A TCP RST should be received 590 in response to the TCP datagram(s), if the ACK was received by the 591 DUT/SUT. 593 5.4.4 Measurements 595 The tester MUST measure the minimum, average and maximum TCP 596 connection tear down times. The TCP connection tear down time will 597 be considered the interval between the transmission of the first TCP 598 FIN packet transmitted by the tester requesting a connection tear 599 down to receipt of the ACK packet on the same tester interface. 601 The tester SHOULD measure the minimum, maximum and average tear down 602 times for all other underlying connection oriented protocols. For 603 purposes of benchmarking firewall performance, the connection tear down 604 time will be considered the interval between the transmission of the 605 first bit of the first octet of the packet carrying the tear down 606 request to the DUT/SUT interface to receipt of the last bit of the 607 last octet of the last packet of the connection tear down traffic 608 headed in the opposite direction. 610 The tester SHOULD measure the aggregate connection tear down time and 611 the total number of connections torn down for each protocol measured. 613 5.4.5 Reporting Format 615 The test report MUST note the initial connections , tear down attempt 616 rate and tear down step count. 618 For each connection oriented protocol the tester measured, the 619 report MUST note the minimum, average and maximum connection tear 620 down. In addition, it SHOULD include the aggregate connection tear 621 down time and attempted tear downs completed. The report MUST 622 identify the layer/protocol for which the measurements were made. 624 Failure analysis: 626 The test report SHOULD indicate the number of connections which failed 627 the validation step. 629 5.5 Denial Of Service Handling 631 5.5.1 Objective 633 To determine the effect of a denial of service attack on a DUT/SUTs 634 connection establishment and/or forwarding rate. The Denial Of 635 Service Handling test MUST be run after obtaining baseline 636 Measurements from sections 5.2 and/or 5.6. 638 The TCP SYN flood attack exploits TCP's three-way handshake mechanism 639 by having an attacking source host generate TCP SYN packets with 640 random source addresses towards a victim host, thereby consuming that 641 host's resources. 643 Some firewalls employ mechanisms to guard against SYN attacks. If such 644 mechanisms exist on the DUT/SUT, tests SHOULD be run with these 645 mechanisms enabled to determine how well the DUT/SUT can maintain, 646 under such attacks, the baseline connection rates and HTTP forwarding 647 rates determined in section 5.2 and section 5.6, respectively. 649 5.5.2 Setup Parameters 651 Use the same setup parameters as defined in section 5.2.2 or 5.6.2, 652 depending on whether testing against the baseline connection setup 653 rate test or HTTP test, respectfully. 655 In addition, the following setup parameters MUST be defined. 657 SYN Attack Rate - Defines the rate, in packets per second at which 658 the server(s) are targeted with TCP SYN packets. 660 5.5.3 Procedure 662 Use the same procedure as defined in section 5.2.3 or 5.6.3, 663 Depending on whether testing against the baseline connection setup 664 rate test or HTTP test, respectfully. In addition, the tester will 665 generate TCP SYN packets targeting the server(s) IP address or NAT 666 proxy address at a rate defined by SYN attack rate. 668 The tester originating the TCP SYN attack MUST be attached to the 669 unprotected network. In addition, the tester MUST not respond to the 670 SYN/ACK packets sent by target server in response to the SYN packet. 672 5.5.4 Measurements 674 Perform the same measurements as defined in section 5.2.4 or 5.6.4, 675 depending on whether testing against the baseline connection setup 676 rate test or HTTP test, respectfully. 678 In addition, the tester SHOULD track SYN packets associated with the 679 SYN attack which the DUT/SUT forwards on the protected or DMZ 680 interface(s). 682 5.5.5 Reporting Format 684 The test SHOULD use the same reporting format as described in 685 section 5.2.5 or 5.6.5, depending on whether testing against 686 baseline throughput rates or HTTP test, respectively. 688 In addition, the report MUST indicate a denial of service handling 689 test, SYN attack rate, number SYN attack packets transmitted and 690 number of SYN attack packets received. The report SHOULD indicate 691 whether or not the DUT has any SYN attack mechanisms enabled. 693 5.6 HTTP 695 5.6.1 Objective 697 To determine the bit forwarding rate, as defined by RFC2647, of the 698 DUT/SUT when presented with HTTP traffic flows. 700 5.6.2 Setup Parameters 702 Connection type - The tester MUST use HTTP 1.1 for HTTP measurements. 704 Number of GET Requests - Defines the number of HTTP 1.1 GET 705 requests attempted per connection. 707 GET Request Rate - Defines the rate, in GET requests per second, at 708 which HTTP GET requests are attempted on any given connection. 710 Object Size - Defines the number of bytes to be transferred in 711 response to an HTTP GET request. 713 5.6.3 HTTP Procedure 715 Each HTTP 1.1 virtual client will attempt to establish each 716 connection to its HTTP 1.1 target server(s), using either the target 717 server's IP address or NAT proxy address, in a round robin fashion. 718 The tester will initiate GET requests for each connection at a 719 constant rate defined by GET request rate, regardless of the state 720 of the DUT/SUT. 722 Baseline measurements SHOULD be performed using a single GET request 723 per connection with a 1-byte object size. If the tester makes multiple 724 HTTP GET requests per connection, it MUST request the same object size 725 for each GET request. Testers SHOULD run multiple iterations of this 726 test using other object sizes and/or multiple requests per connection. 727 See appendix A when testing proxy based DUT/SUT regarding HTTP version 728 considerations. 730 5.6.4 Measurements 732 Version information: 734 The test report MUST note the use of an HTTP 1.1 client and server. 736 Application Layer 738 Bit Forwarding Rate - The bit forwarding rate of the DUT/SUT MUST, 739 at a minimum, be measured at the application layer and shall be 740 referenced to the requested object(s). The measurement will 741 start on transmission of the first bit of the first requested object 742 and end on transmission of the last bit of the last requested object. 743 The aggregate bit forwarding rate, in bits per second, will be 744 calculated using the following formula: 746 OBJECTS * OBJECTSIZE * 8 747 FORWARDING RATE(bit/s) = -------------------------- 748 DURATION 750 OBJECTS - Objects successfully transferred 752 OBJECTSIZE - Object size in bytes 754 DURATION - Aggregate transfer time based on aforementioned time 755 references. 757 Transport-Layer and Below 759 Bit forwarding rate for layers at or below the transport layer SHOULD 760 also be performed. Bit forwarding rate for these underlying 761 layers/protocols MAY be measured in either bits per seconds or UOTs 762 per second. In both cases, the measurement will start on transmission 763 of the first packet containing the first HTTP GET request and end on 764 transmission of the last packet containing the last octet of the last 765 requested object. The aggregate bit forwarding rate, in bits per second, 766 will be calculated using the following formula: 768 TX - RETX 769 FORWARDING RATE(bit/s or UOT/s) = ----------- 770 DURATION 772 TX - If measuring in units of bits per seconds, TOTAL is the total 773 bits transmitted including header and optional data for a 774 given protocol. If measuring in UOTs per seconds, total is the 775 total number of UOTs transmitted. This excludes any bits or UOT 776 that are associated with connection maintenance[1], such as TCP 777 keep-alives. 779 RETX - If measuring in units of bits per seconds, RETX is the total 780 number of bits, including header and optional data for a given 781 protocol, that was retransmitted. If measuring in units of UOTs 782 per second, RETX is the number of UOTs retransmitted. This 783 excludes any bits or UOTs that are associated with connection 784 maintenance, such as TCP keep-alives. 786 DURATION - Test duration based on aforementioned time references. 788 5.6.5 Reporting Format 790 The test report MUST note the number of GET requests, GET request 791 rate and object size. 793 Application layer bit forwarding rate results SHOULD be reported in 794 tabular form with a row for each of the object sizes. There SHOULD 795 be columns for the object size, the number of completed requests, 796 the number of completed responses, and the bit forwarding rate 797 results for each test. 799 When reporting bit forwarding measurements for layers below the 800 application layer, such as TCP or IP, the report MUST note whether 801 the measurements are in bit per second or UOTs per second and the 802 object size transferred. The report SHOULD be in tabular form with 803 a row for each layer/protocol. There should be columns for 804 transmitted bits/UOTs, retransmitted bits/UOTs and the measured 805 forwarding rate. 807 Failure analysis: 809 The test report SHOULD indicate the number and percentage of HTTP GET 810 request or responses that failed to complete. 812 5.7 IP Fragmentation 814 5.7.1 Objective 816 To determine the performance impact when the DUT/SUT is presented 817 with IP fragmented[5] traffic. IP datagrams which have been 818 fragmented, due to crossing a network that supports a smaller 819 MTU(Maximum Transmission Unit) than the actual datagram, may 820 require the firewall to perform re-assembly prior to the datagram 821 being applied to the rule set. 823 While IP fragmentation is a common form of attack, either on the 824 firewall itself or on internal hosts, this test will focus on 825 determining how the additional processing associated with the 826 re-assembly of the datagrams has on the goodput of the DUT/SUT. 828 5.7.2 Setup Parameters 830 The following parameters MUST be defined. 832 Trial duration - Trial duration SHOULD be set for 30 seconds. 834 5.7.2.1 Non-Fragmented Traffic Parameters 836 Session rate - Defines the rate, in sessions per second, that the 837 HTTP sessions are attempted. 839 Requests per session - Defines the number of HTTP GET requests per 840 session. 842 Object Size - Defines the number of bytes to be transferred in 843 response to an HTTP GET request. 845 5.7.2.1 Fragmented Traffic Parameters 847 Packet size, expressed as the number of bytes in the IP/UDP packet, 848 exclusive of link-layer headers and checksums. 850 Fragmentation Length - Defines the length of the data portion of the 851 IP datagram and MUST be multiple of 8. Testers SHOULD use the minimum 852 value, but MAY use other sizes as well. 854 Intended Load - Intended load, expressed as percentage of media 855 utilization. 857 5.7.3 Procedure 859 Each HTTP 1.1 virtual client will attempt to establish sessions 860 to its HTTP 1.1 target server(s), using either the target server's 861 IP address or NAT proxy address, at a fixed rate in a round robin 862 fashion. At the same time, a client attached to the unprotected side 863 of the network will offer a unidirectional stream of unicast UDP/IP 864 packets to a server connected to the protected side of the network. 865 The tester MUST offer IP/UDP packets in a steady state. 867 Baseline measurements SHOULD be performed with a deny rule(s) that 868 filters the fragmented traffic. If the DUT/SUT has logging 869 capability, the log SHOULD be checked to determine if it contains 870 the correct information regarding the fragmented traffic. 872 The test SHOULD be repeated with the DUT/SUT rule set changed to 873 allow the fragmented traffic through. When running multiple 874 iterations of the test, it is RECOMMENDED to vary the fragment 875 length while keeping all other parameters constant. 877 5.7.4 Measurements 879 Aggregate Goodput - The aggregate bit forwarding rate of the 880 requested HTTP objects.(See section 5.6). Only objects which have 881 successfully completed transferring within the trial duration are 882 to be included in the goodput measurement. 884 Transmitted UDP/IP Packets - Number of UDP packets transmitted by 885 client. 887 Received UDP/IP Packets - Number of UDP/IP Packets received by 888 server. 890 5.7.5 Reporting Format 892 The test report MUST note the test duration. 894 The test report MUST note the packet size(s), offered load(s) and 895 IP fragmentation length of the UDP/IP traffic. It SHOULD also note 896 whether the DUT/SUT egresses the offered UDP/IP traffic fragmented 897 or not. 899 The test report MUST note the object size(s), session rate and 900 requests per session. 902 The results SHOULD be reported in the format of a table with a 903 row for each of the fragmentation lengths. There SHOULD be columns 904 for the fragmentation length, IP/UDP packets transmitted by client, 905 IP/UDP packets received by server, HTTP object size, and measured 906 goodput. 908 5.8 Illegal Traffic Handling 910 5.8.1 Objective 912 To determine the behavior of the DUT/SUT when presented with a 913 combination of both legal and Illegal traffic flows. Note that 914 Illegal traffic does not refer to an attack, but to traffic which 915 has been explicitly defined by a rule(s) to drop. 917 5.8.2 Setup Parameters 919 Connection type - The tester MUST use HTTP 1.1 for HTTP measurements. 921 Number of GET Requests - Defines the number of HTTP 1.1 GET 922 requests attempted per connection. 924 GET Request Rate - Defines the rate, in GET requests per second, at 925 which HTTP GET requests are attempted on any given connection. 927 Object Size - Defines the number of bytes to be transferred in 928 response to an HTTP GET request. 930 Illegal traffic percentage - Percentage of HTTP connections which 931 have been explicitly defined in a rule(s) to drop. 933 5.8.3 Procedure 935 Each HTTP 1.1 virtual client will attempt to establish sessions 936 to its HTTP 1.1 target server(s), using either the target server's 937 IP address or NAT proxy address, at a fixed rate in a round robin 938 fashion. 940 The tester MUST present the connection requests, both legal and 941 illegal, in an evenly distributed manner. Many firewalls have 942 the capability to filter on different traffic criteria( IP 943 addresses, Port numbers, etc). Testers may run multiple 944 iterations of this test with the DUT/SUT configured to filter 945 on different traffic criteria. 947 5.8.4 Measurements 949 Tester SHOULD perform the same bit forwarding measurements as defined 950 in HTTP test(Section 5.6.4). 952 5.8.5 Reporting Format 954 Tester SHOULD report SHOULD be the same as specified in the HTTP 955 test(Section 5.6.5). 957 In addition, the report MUST note the percentage of illegal HTTP 958 connections. 960 Failure analysis 962 Test report MUST note the number and percentage of illegal connections 963 allowed by the DUT/SUT. 965 5.9 Latency 967 5.9.1 Objective 969 To determine the latency of network-layer or application-layer data 970 traversing the DUT/SUT. RFC 1242 [3] defines latency. 972 5.9.2 Setup Parameters 974 The following parameters MUST be defined: 976 5.9.2.1 Network-layer Measurements 978 Packet size, expressed as the number of bytes in the IP packet, 979 exclusive of link-layer headers and checksums. 981 Intended load, expressed as percentage of media utilization. 983 Offered load, expressed as percentage of media utilization. 985 Test duration, expressed in seconds. 987 Test instruments MUST generate packets with unique timestamp signatures. 989 5.9.2.2 Application-layer Measurements 991 Object size, expressed as the number of bytes to be transferred across a 992 connection in response to an HTTP GET request. Testers SHOULD use the 993 minimum object size supported by the media, but MAY use other object 994 sizes as well. 996 Connection type. The tester MUST use one HTTP 1.1 connection for latency 997 measurements. 999 Number of objects requested. 1001 Number of objects transferred. 1003 Test duration, expressed in seconds. 1005 Test instruments MUST generate packets with unique timestamp signatures. 1007 5.9.3 Network-layer procedure 1009 A client will offer a unidirectional stream of unicast packets to a server. 1010 The packets MUST use a connectionless protocol like IP or UDP/IP. 1012 The tester MUST offer packets in a steady state. As noted in the latency 1013 discussion in RFC 2544 [4], latency measurements MUST be taken at the 1014 throughput level -- that is, at the highest offered load with zero packet 1015 loss. Measurements taken at the throughput level are the only ones that can 1016 legitimately be termed latency. 1018 It is RECOMMENDED that implementers use offered loads not only at the 1019 throughput level, but also at load levels that are less than or greater 1020 than the throughput level. To avoid confusion with existing terminology, 1021 measurements from such tests MUST be labeled as delay rather than latency. 1022 If desired, the tester MAY use a step test in which offered loads increment 1023 or decrement through a range of load levels. 1025 The duration of the test portion of each trial MUST be at least 30 seconds. 1027 5.9.4 Application layer procedure 1029 An HTTP 1.1 client will request one or more objects from an HTTP 1.1 server 1030 using one or more HTTP GET requests. If the tester makes multiple HTTP GET 1031 requests, it MUST request the same-sized object each time. Testers may run 1032 multiple iterations of this test with objects of different sizes. 1034 Implementers MAY configure the tester to run for a fixed duration. In this 1035 case, the tester MUST report the number of objects requested and returned 1036 for the duration of the test. For fixed-duration tests it is RECOMMENDED 1037 that the duration be at least 30 seconds. 1039 5.9.5 Measurements 1041 Minimum delay - The smallest delay incurred by data traversing the DUT/SUT 1042 at the network layer or application layer, as appropriate. 1044 Maximum delay - The largest delay incurred by data traversing the DUT/SUT 1045 at the network layer or application layer, as appropriate. 1047 Average delay - The mean of all measurements of delay incurred by data 1048 traversing the DUT/SUT at the network layer or application layer, as 1049 appropriate. 1051 Delay distribution - A set of histograms of all delay measurements observed 1052 for data traversing the DUT/SUT at the network layer or application layer, 1053 as appropriate. 1055 5.9.6 Network-layer reporting format 1057 The test report MUST note the packet size(s), offered load(s) and test 1058 duration used. 1060 The latency results SHOULD be reported in the format of a table with a row 1061 for each of the tested packet sizes. There SHOULD be columns for the 1062 packet size, the intended rate, the offered rate, and the resultant latency 1063 or delay values for each test. 1065 5.9.7 Application-layer reporting format 1067 The test report MUST note the object size(s) and number of requests and 1068 responses completed. If applicable, the report MUST note the test duration 1069 if a fixed duration was used. 1071 The latency results SHOULD be reported in the format of a table with a row 1072 for each of the object sizes. There SHOULD be columns for the object size, 1073 the number of completed requests, the number of completed responses, and the 1074 resultant latency or delay values for each test. 1076 Failure analysis: 1078 The test report SHOULD indicate the number and percentage of HTTP GET 1079 request or responses that failed to complete within the test duration. 1081 Version information: 1083 The test report MUST note the use of an HTTP 1.1 client and server. 1085 APPENDICES 1087 APPENDIX A: HTTP(HyperText Transfer Protocol) 1089 The most common versions of HTTP in use today are HTTP/1.0 and 1090 HTTP/1.1 with the main difference being in regard to persistent 1091 connections. HTTP 1.0, by default, does not support persistent 1092 connections. A separate TCP connection is opened up for each 1093 GET request the client wants to initiate and closed after the 1094 requested object transfer is completed. Some implementations of 1095 HTTP/1.0 supports persistence by adding an additional header 1096 to the request/response: 1098 Connection: Keep-Alive 1100 However, under HTTP 1.0, there is no official specification for 1101 how the keep-alive operates. In addition, HTTP 1.0 proxies do 1102 support persistent connection as they do not recognize the 1103 connection header. 1105 HTTP/1.1, by default, does support persistent connection and 1106 is therefore the version that is referenced in this methodology. 1107 When HTTP/1.1 entities want the underlying transport layer 1108 connection closed after a transaction has completed, the 1109 request/response will include a connection-token close in the 1110 connection header: 1112 Connection: close 1114 If no such connection-token is present, the connection remains 1115 open after the transaction is completed. In addition, proxy 1116 based DUT/SUTs may monitor the TCP connection and after a 1117 timeout, close the connection if no activity is detected. The 1118 duration of this timeout is not defined in the HTTP/1.1 1119 specification and will vary between DUT/SUTs. If the DUT/SUT 1120 closes inactive connections, the aging timer on the DUT SHOULD 1121 be configured for a duration that exceeds the test time. 1123 While this document cannot foresee future changes to HTTP 1124 and it impact on the methodologies defined herein, such 1125 changes should be accommodated for so that newer versions of 1126 HTTP may be used in benchmarking firewall performance. 1128 Appendix B. References 1130 [1] D. Newman, "Benchmarking Terminology for Firewall Devices", RFC 2647, 1131 August 1999. 1133 [2] R. Fielding, J. Gettys, J. Mogul, H Frystyk, L.Masinter, P. Leach, 1134 T. Berners-Lee , "Hypertext Transfer Protocol -- HTTP/1.1", 1135 RFC 2616 June 1999 1137 [3] S. Bradner, editor. "Benchmarking Terminology for Network 1138 Interconnection Devices," RFC 1242, July 1991. 1140 [4] S. Bradner, J. McQuaid, "Benchmarking Methodology for Network 1141 Interconnect Devices," RFC 2544, March 1999. 1143 [5] David C. Clark, "IP Datagram Reassembly Algorithm", RFC 815 , 1144 July 1982.