idnits 2.17.1 draft-ietf-bmwg-firewall-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** Looks like you're using RFC 2026 boilerplate. This must be updated to follow RFC 3978/3979, as updated by RFC 4748. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- ** Missing expiration date. The document expiration date should appear on the first and last page. == No 'Intended status' indicated for this document; assuming Proposed Standard == It seems as if not all pages are separated by form feeds - found 0 form feeds but 23 pages Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack an Abstract section. ** The document seems to lack a Security Considerations section. ** The document seems to lack an IANA Considerations section. (See Section 2.2 of https://www.ietf.org/id-info/checklist for how to handle the case when there are no actions for IANA.) ** The document seems to lack an Authors' Addresses Section. ** There are 45 instances of too long lines in the document, the longest one being 4 characters in excess of 72. ** There are 22 instances of lines with control characters in the document. Miscellaneous warnings: ---------------------------------------------------------------------------- == Line 353 has weird spacing: '...command over ...' == Line 758 has weird spacing: '... mapped on th...' == Using lowercase 'not' together with uppercase 'MUST', 'SHALL', 'SHOULD', or 'RECOMMENDED' is not an accepted usage according to RFC 2119. Please use uppercase 'NOT' together with RFC 2119 keywords (if that is what you mean). Found 'MUST not' in this paragraph: Min/Avg/Max connection times MUST be measured. The connection time SHALL begin when the client initiates the OPEN command and end when the client received a successful login acknowledgment from the server. Note that although the NOOP command is included as part of the procedure for establishing the connection, it MUST not be included as part of the connection establishment time. == Using lowercase 'not' together with uppercase 'MUST', 'SHALL', 'SHOULD', or 'RECOMMENDED' is not an accepted usage according to RFC 2119. Please use uppercase 'NOT' together with RFC 2119 keywords (if that is what you mean). Found 'MUST not' in this paragraph: The tester originating the TCP SYN attack MUST be attached to the Unprotected network. In addition, the tester MUST not respond to the SYN ACK packets sent by target server in response to the SYN packet. == Using lowercase 'not' together with uppercase 'MUST', 'SHALL', 'SHOULD', or 'RECOMMENDED' is not an accepted usage according to RFC 2119. Please use uppercase 'NOT' together with RFC 2119 keywords (if that is what you mean). Found 'MUST not' in this paragraph: Min/Avg/Max connection times MUST be measured. The connection time SHALL begin when the client initiates the OPEN command and end when the client received a successful login acknowledgment from the server. Note that although the NOOP command is included as part of the procedure for establishing the connection, it MUST not be included as part of the connection establishment time. -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (June 2000) is 8709 days in the past. Is this intentional? -- Found something which looks like a code comment -- if you have code sections in the document, please surround them with '' and '' lines. Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) -- Missing reference section? '1' on line 288 looks like a reference Summary: 8 errors (**), 0 flaws (~~), 7 warnings (==), 4 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 Network Working Group Terry Martin 2 Internet-Draft M2networx INC 3 Expiration Date: B. Hickman 4 Netcom Systems 5 June 2000 7 Benchmarking Methodology for Firewalls 8 10 Status of this Memo 12 This document is an Internet-Draft and is in full conformance with 13 all provisions of Section 10 of RFC2026. 15 Internet-Drafts are working documents of the Internet Engineering 16 Task Force (IETF), its areas, and its working groups. Note that 17 other groups may also distribute working documents as Internet- 18 Drafts. 20 Internet-Drafts are draft documents valid for a maximum of six months 21 and may be updated, replaced, or obsoleted by other documents at any 22 time. It is inappropriate to use Internet-Drafts as reference 23 material or to cite them other than as "work in progress." 25 The list of current Internet-Drafts can be accessed at 26 http://www.ietf.org/ietf/1id-abstracts.txt 28 The list of Internet-Draft Shadow Directories can be accessed at 29 http://www.ietf.org/shadow.html. 31 Table of Contents 33 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 34 2. Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 35 3. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 36 4. Test setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 37 4.1 Test Considerations . . . . . . . . . . . . . . . . . . . . . 4 38 4.2 Virtual Client/Servers . . . . . . . . . . . . . . . . . . . 4 39 4.3 Test Traffic Requirements . . . . . . . . . . . . . . . . . . 4 40 4.4 DUT/SUT Traffic Flows . . . . . . . . . . . . . . . . . . . . 5 41 4.5 Multiple Client/Server Testing . . . . . . . . . . . . . . . 5 42 4.6 NAT(Network Address Translation) . . . . . . . . . . . . . . 6 43 4.7 Rule Sets . . . . . . . . . . . . . . . . . . . . . . . . . . 6 44 4.8 Web Caching . . . . . . . . . . . . . . . . . . . . . . . . . 6 45 5. Benchmarking Tests . . . . . . . . . . . . . . . . . . . . . . . . 7 46 5.1 Concurrent Connection Capacity . . . . . . . . . . . . . . . 7 47 5.2 Maximum Connection Rate . . . . . . . . . . . . . . . . . . . 9 48 5.3 Denial Of Service Handling . . . . . . . . . . . . . . . . . 10 49 5.3 Single Application Goodput . . . . . . . . . . . . . . . . . 12 50 5.4 FTP Goodput . . . . . . . . . . . . . . . . . . . . . . . . . 12 51 5.5 SMTP Goodput . . . . . . . . . . . . . . . . . . . . . . . . 13 52 5.6 HTTP Goodput . . . . . . . . . . . . . . . . . . . . . . . . 15 53 5.7 Mixed Application Goodput . . . . . . . . . . . . . . . . . . 17 54 5.8 Illegal Traffic Handling . . . . . . . . . . . . . . . . . . 18 55 5.9 Latency . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 56 Appendices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 57 A. File Transfer Protocol(FTP) . . . . . . . . . . . . . . . . . . . 19 58 A.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 19 59 A.2 Connection Establishment/Teardown . . . . . . . . . . . . . . 19 60 A.3 Object Format . . . . . . . . . . . . . . . . . . . . . . . . 20 61 B. Simple Mail Transfer Protocol(SMTP) . . . . . . . . . . . . . . . 20 62 B.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . 20 63 B.2 Connection Establishment/Teardown . . . . . . . . . . . . . . 20 64 B.3 Object Format . . . . . . . . . . . . . . . . . . . . . . . . 21 65 C. HyperText Transfer Protocol(HTTP) . . . . . . . . . . . . . . . . 21 66 C.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 21 67 C.2 Version Considerations . . . . . . . . . . . . . . . . . . . . 21 68 C.3 Object Format . . . . . . . . . . . . . . . . . . . . . . . . 21 69 D. GoodPut Measurements . . . . . . . . . . . . . . . . . . . . . . . 22 70 E. References . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 72 1. Introduction 74 This document is intended to provide methodology for the benchmarking 75 of firewalls. It provides methodologies for benchmarking forwarding 76 performance, connection performance, latency and filtering. In 77 addition to defining the tests, this document also describes specific 78 formats for reporting the results of the tests. 80 A previous document, "Benchmarking Terminology for Firewall 81 Performance" [1], defines many of the terms that are used in this 82 document. The terminology document SHOULD be consulted before 83 attempting to make use of this document. 85 2. Requirements 87 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 88 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 89 document are to be interpreted as described in RFC 2119. 91 3. Scope 93 Firewalls can provide a single point of defense between two 94 networks--it protects one network from the other. Usually, a firewall 95 protects the company's private network from the public or shared 96 networks to which it is connected. A firewall can be as simple as a 97 device that filters different packets or as complex as a group of 98 devices providing solutions that offers combined packet filtering 99 and application-level proxy or network translation services. This RFC 100 will focus on developing bench mark testing of systems from an 101 application perspective and will be developed independent of any 102 firewall implementation. These tests will evaluate the ability of 103 firewall devices to control and manage applications services used 104 by today's businesses such as applications like the World Wide Web, 105 File transfer procedures and e-mail. 107 Even through there are many different control methods of managing 108 application level being implemented, this RFC does not condone or 109 promote any aforementioned process or procedure. It's goal is to 110 present a procedure that will evaluate firewall performance 111 independent of their implementation. 113 4. Test Setup 115 Test configurations defined in this document will be confined to 116 dual-homed and tri-homed as shown in figure 1 and figure 2 117 respectively. 119 Firewalls employing Dual-Homed configurations connect two networks. 120 One interface of the firewall is attached to the unprotected network, 121 typically the public network(i.e. - Internet). The other interface is 122 connected to the protected network, typically the internal LAN. In 123 the case of Dual-Homed configurations, servers which are made 124 accessible to the public(Unprotected) network are attached to the 125 private(Protected) network. 127 +----------+ +----------+ 128 | | | +----------+ | | | 129 | Servers/ |----| | | |------| Servers/ | 130 | Clients | | | | | | Clients | 131 | | |-------| DUT/SUT |--------| | | 132 +----------+ | | | | +----------+ 133 | +----------+ | 134 Protected | | Unprotected 135 Network Network 136 Figure 1(Dual-Homed) 138 Tri-homed configurations employ a third segment called a DMZ. With 139 tri-homed configurations, servers accessible to the public network 140 are attached to the DMZ. Tri-Homed configurations offer additional 141 security by separating server accessible to the public network from 142 internal hosts. 144 +----------+ +----------+ 145 | | | +----------+ | | | 146 | Clients |----| | | |------| Servers/ | 147 | | | | | | | Clients | 148 +----------+ |-------| DUT/SUT |--------| | | 149 | | | | +----------+ 150 | +----------+ | 151 Protected | | | Unprotected 152 Network | Network 153 | 154 | 155 ----------------- 156 | DMZ 157 | 158 | 159 +-----------+ 160 | | 161 | Servers | 162 | | 163 +-----------+ 165 Figure 2(Tri-Homed) 167 4.1 Test Considerations 169 4.2 Virtual Clients/Servers 171 Since firewall testing may involve data sources which emulate multiple 172 users or hosts, the methodology uses the terms virtual clients/servers. 173 For these firewall tests, virtual clients/servers specify application 174 layer entities which may not be associated with a unique physical 175 interface. For example, four virtual clients may originate from the 176 same data source. 178 4.3 Test Traffic Requirements 180 While the function of a firewall is to enforce access control 181 policies, the criteria by which those policies are defined vary 182 depending on the implementation. Firewalls may use network layer 183 and/or, in many cases, application-layer criteria to make 184 access-control decisions. Therefore, the test equipment used to 185 benchmark the DUT/SUT performance MUST consist of real clients and 186 servers generating legitimate layer 7 conversations. Specifically, 187 the tests defined in this document use HTTP, FTP, and SMTP sessions 188 for benchmarking the performance of the DUT/SUT. See appendices for 189 specific information regarding the transactions involved in 190 establishing/tearing down connections as well as object formats 191 for each of the aforementioned protocols. 193 4.4 DUT/SUT Traffic Flows 195 This section discusses the client -> server traffic flow 196 requirements associated with the source -> destination interfaces 197 of the firewall. Since the number of interfaces are not fixed, the 198 traffic flows will be dependent upon the configuration used in 199 benchmarking the DUT/SUT. Note that the term "traffic flows" is in 200 the context of client requests since traffic between clients and 201 servers are, by nature, bi-directional with transactions occurring 202 between both entities. 204 In the case of Dual-Homed configurations, traffic flows presented 205 to the DUT/SUT SHOULD consist of client -> server requests associated 206 with the following interfaces: 208 Protected -> Unprotected 209 Unprotected -> Protected 211 In the case of Tri-Homed configurations, traffic flows presented 212 to the DUT/SUT SHOULD consist of client -> server requests associated 213 with the following interfaces: 215 Protected -> Unprotected 216 Protected -> DMZ 217 Unprotected -> DMZ 219 4.5 Multiple Client/Server Testing 221 The methodologies defined in this document MAY be performed with one 222 or more clients targeting multiple servers for a given application. 223 When performing tests with one or more virtual clients targeting 224 multiple servers, each virtual client MUST initiate requests 225 (Connection, file transfers, etc.) in a round-robin fashion. For 226 example, if the test consisted of six virtual clients targeting 227 three servers, the pattern would be as follows: 229 Client Target Server(In order of request) 231 #1 1 2 3 1... 232 #2 2 3 1 2... 233 #3 3 1 2 3... 234 #4 1 2 3 1... 235 #5 2 3 1 2... 236 #6 3 1 2 3... 238 4.6 NAT(Network Address Translation) 240 Most firewalls come with Network Address Translation(NAT)networks 241 built in which translates internal host IP addresses attached to the 242 protected network to a virtual IP address for communicating across the 243 unprotected network(Internet). Since this involves additional processing 244 on the part of the DUT/SUT, its impact on performance should be measured. 245 Therefore, tests defined in this document SHOULD first be ran with NAT 246 disabled and then repeated with NAT enabled to determine the performance 247 differentials. 249 4.7 Rule Sets 251 Rule sets are a collection of access control policies that determines 252 which packets the DUT/SUT will forward and which it will reject. The 253 criteria by which these access control policies may be defined will 254 vary depending on the capabilities of the product. Therefore, this 255 document will not attempt to define the specifics of the rule sets, 256 but instead define how the rule sets should be applied when testing 257 the DUT/SUT. 259 The firewall monitors the incoming traffic and checks to make sure 260 that the traffic meets one of the defined rules before allowing it to 261 be forwarded. It is RECOMMENDED that a rule be entered for each 262 host(i.e. - Virtual client). Although many firewalls permit groups 263 of IP addresses to be defined for a given rule, tests SHOULD be 264 performed with as large a rule set as possible to better stress test 265 the DUT/SUT. In addition, a rule MUST be defined which denies access 266 to all traffic which was not previously defined in the rule set, 267 provided such a rule can be defined in the DUT/SUT. 269 4.8 Web Caching 271 Some firewalls include caching agents in order to reduce network 272 load. When making a request through a caching agent, the caching 273 agent attempts to service the response from its internal resources. 274 The cache itself saves responses it receives, such as responses for 275 HTTP GET requests. Therefore, caching SHOULD be disabled on the 276 DUT/SUT prior to performing the tests. 278 5. Benchmarking Tests 280 The following tests offer objectives, procedures, and reporting 281 formats for benchmarking firewalls. 283 5.1 Concurrent Connection Capacity 285 5.1.1 Objective 287 To determine the maximum number of concurrent connections supported 288 by the DUT/SUT, as defined in RFC2647[1]. This test will employ a 289 step search algorithm to obtain the maximum number of concurrent FTP 290 connections the DUT/SUT can maintain. 292 5.1.2 Setup Parameters 294 The following parameters MUST be defined. Each variable is configured 295 with the following considerations. 297 Connection Attempt Rate - The rate at which new connection requests 298 are attempted. The rate SHOULD be set lower than maximum rate at 299 which the DUT/SUT can accept new connection requests. The connection 300 attempt rate MUST be evenly divided among all of the participating 301 virtual clients. 303 Connection Step count - Defines the number of additional connections 304 attempted for each iteration of the search algorithm. The step count 305 SHOULD be a multiple of the number of virtual clients participating 306 in the test. In addition, the number MUST be greater than or equal to 307 the number of virtual clients participating in the test. 309 5.1.3 Procedure 311 Each virtual client will attempt to establish FTP control connections 312 by issuing an OPEN command to their target server(s) at a fixed rate. 313 Each iteration will involve the virtual clients attempting to 314 establish a fixed number of additional connections. This search 315 algorithm will be repeated until either: 317 - One or more of the additional connection attempts fail to 318 complete 319 - One or more of the previously established connections drop. 321 The transactions between the client and server associated with 322 establishing FTP connections are defined in Appendix A. In order to 323 verify that work can be performed across that connection, the virtual 324 client MUST issue a NOOP command. This command is in addition to the 325 transactions define in Appendix A, and must be accompanied by the 326 Command Successful reply from the target server in order to be 327 considered a successful connection. 329 After each iteration, all virtual clients MUST issue a NOP command 330 across all of their open connections to verify that none of the 331 previously opened connections were dropped by the DUT/SUT. 333 The algorithm for the test is as follows: 335 CONSTANT 337 STEP = Connection Step Count; {# of additional connection attempts 338 per iteration} 339 MAX = ...; {maximum connections supported by test tool 340 implementation} 341 VARIABLE 343 CURRENT := 0; {Highest passed value} 344 STATUS = PASS; 346 BEGIN 347 RESET; {RESET all connections} 348 DO 349 BEGIN 350 EstablishStepConnections; { See Appendix A.3} 351 IF(AttemptsSucessful) THEN 352 BEGIN 353 VerifyAllConnections { Send NOP command over all open 354 connections } 355 IF(Received replies for all NO0P commands) THEN 356 BEGIN { Maximum number of connections NOT found } 357 CURRENT := CURRENT + Step Count 358 END 359 ELSE 360 BEGIN { Maximum number of connections < Step Count } 361 STATUS := FAIL 362 END 363 END 364 ELSE 365 BEGIN { Maximum number of connections < Step Count } 366 STATUS := FAIL 367 END 368 END 369 END WHILE(STATUS == PASS && [[CURRENT + Step Count] < MAX]) 370 END 371 END {Value of CURRENT equals number connections supported by DUT/SUT} 373 5.1.4 Measurements 375 Whether all of the connection attempts were successfully completed. 376 Test equipment MUST be able to track each connection to verify all 377 required transaction between the virtual client and server completed 378 successfully. This includes successful completion of both the 379 command sequences and exchanging of any data across each of those 380 connections. 382 5.1.5 Reporting Format 384 Maximum concurrent connections reported MUST be the aggregate number 385 of connections completed for the last successful iteration. Report 386 SHOULD also include: 388 - Number of virtual clients and servers participating in the test 389 - Whether NAT was enabled or disabled. 391 5.2 Maximum Connection Setup Rate 393 5.2.1 Objective 395 To determine the maximum connection rate which the DUT/SUT can 396 successfully establish connections. As with the Concurrent Connection 397 Capacity test, FTP session will be used to determine this metric. 399 5.2.2 Setup Parameters 401 The following parameters MUST be defined. Each variable is configured 402 with the following considerations. 404 Initial Attempt Rate - The rate at which the initial connection 405 requests are attempted. The connection attempt rate MUST be evenly 406 divided among all of the participating virtual clients. 408 Rate Step Count - The rate at which connection attempts are either 409 increased or decreased during the search algorithm. The rate step 410 count MUST be evenly divided among all of the participating virtual 411 clients. 413 Number of Connections - Defines the number of connections that must 414 be established. The number MUST be between the number of 415 participating virtual clients and the maximum number supported by the 416 implementation. It is RECOMMENDED not to exceed the concurrent 417 connection capacity found in section 5.1. 419 5.2.3 Procedure 421 An algorithm similar to the one used to determine the concurrent 422 connection capacity will be used to determine the maximum connection 423 rate. This test iterates the rate at which connections are attempted 424 by the virtual clients to their associated server(s). 426 The connection establishment rate might be determined for different 427 numbers of connections but in each test run, the number MUST remain 428 constant and SHOULD be equal to or less than the maximum connection 429 capacity. 431 As with the connection capacity test, NOOP commands will be 432 generated when both establishing the connection and after all 433 connections have been established to verify none of the previously 434 established connections have dropped. 436 5.2.4 Measurements 438 Min/Avg/Max connection times MUST be measured. The connection time 439 SHALL begin when the client initiates the OPEN command and end when 440 the client received a successful login acknowledgment from the 441 server. Note that although the NOOP command is included as part of 442 the procedure for establishing the connection, it MUST not be 443 included as part of the connection establishment time. 445 5.2.5 Reporting Format 447 The results for these tests SHOULD be reported in the form of a 448 graph. The x coordinate SHOULD be the connection count. The y 449 coordinate should be the connection establishment time. The graph 450 SHOULD include the Minimum, Average and Maximum connection times. 452 Report SHOULD also include: 454 - Number of virtual clients and servers participating in the test 455 - Whether NAT was enabled or disabled. 457 5.3 Denial Of Service Handling 459 5.3.1 Objective 461 To determine the effect of a denial of service attack in so far as 462 it's impacts on connection establishment rates through the DUT/SUT. 463 The Denial Of Service Handling test should be ran after obtaining 464 baseline measurements from section 5.2. The results can then be 465 compared with the baseline measurements to obtain the performance 466 differentials. 468 5.3.2 Setup Parameters 470 The following parameters MUST be defined. Each variable is 471 configured with the following considerations. 473 Initial Attempt Rate - The rate at which the initial connection 474 requests are attempted. The connection attempt rate MUST be evenly 475 divided among all of the participating virtual clients. 477 Rate Step Count - The rate at which connection attempts are either 478 increased or decreased during the search algorithm. The rate step 479 count MUST be evenly divided among all of the virtual clients 480 participating in the test. 482 Number of Connections - Defines the number of connections that must 483 be established. The number MUST be between the number of 484 participating clients and the maximum number supported by the 485 implementation. It is RECOMMENDED not to exceed the concurrent 486 connection capacity found in section 5.1. 488 SYN Attack Rate - Defines the rate at which the server(s) are 489 targeted with TCP SYN packets. 491 5.3.3 Procedure 493 This test will use the same procedure as defined in the maximum 494 connection setup rate, except that the test traffic will include 495 TCP SYN packets targeting the server(s) IP address or NAT proxy 496 address. 498 TCP employs a "three-way handshake" when establishing a connection 499 between two hosts. When a normal TCP connection starts, a destination 500 host receives a SYN (synchronize/start)packet from a source host and 501 sends back a SYN ACK (synchronize acknowledge). The destination host 502 must then hear an ACK (acknowledge) of the SYN ACK before the 503 connection is established. The TCP SYN attack exploits this design 504 by having an attacking source host generate TCP SYN packets with 505 random source addresses towards a victim host, thereby consuming 506 that hosts resources. 508 The tester originating the TCP SYN attack MUST be attached to the 509 Unprotected network. In addition, the tester MUST not respond to the 510 SYN ACK packets sent by target server in response to the SYN packet. 512 5.3.4 Measurements 514 Min/Avg/Max connection times MUST be measured. The connection time 515 SHALL begin when the client initiates the OPEN command and end when 516 the client received a successful login acknowledgment from the 517 server. Note that although the NOOP command is included as part of 518 the procedure for establishing the connection, it MUST not be 519 included as part of the connection establishment time. 521 5.2.5 Reporting Format 523 The results for these tests SHOULD be reported in the form of a 524 graph. The x coordinate SHOULD be the connection count. The y 525 coordinate should be the connection establishment time. The graph 526 SHOULD include the Minimum, Average and Maximum connection times. 528 Report SHOULD also include: 530 - Number of virtual clients and servers participating in the test 531 - Whether NAT was enabled or disabled. 532 - TCP SYN Attack Rate 534 5.4 Single Application Goodput 536 This section defined the procedures for baselining the Goodput of the 537 DUT/SUT for FTP, HTTP and SMTP traffic. 539 5.4.1.1 FTP Goodput 541 5.4.1.2 Objective 543 The File Transfer Protocol is a common application used by companies 544 to transfer files from one device to another. Evaluating FTP Goodput 545 will allow individuals to measure how much successful traffic has 546 been forwarded by the DUT/SUT. 548 5.4.1.3 Setup Parameters 550 The following parameters MUST be defined. Each variable is 551 configured with the following considerations. 553 Number of Connections - Defines the number of connections to be 554 opened for transferring data objects. Number MUST be equal or 555 greater than the number of virtual clients participating in the 556 test. The number SHOULD be a multiple of the virtual client 557 participating in the test. 559 Connection Rate - Defines the rate at which connections are 560 established. Number MUST be evenly divided among all of the 561 virtual clients participating in the test. 563 Object Size - Defines the number of bytes to be transferred across 564 each connection. RECOMMENDED object sizes still need to be 565 determined. 567 5.4.1.4 Procedure 569 Each virtual client will establish a FTP connection to its respective 570 server(s) at a user specified rate. The transaction involved in 571 establishing the FTP connection should follow the procedure defined 572 in Appendix A. 574 After the login process has been completed, the virtual client will 575 initiate a file transfer by issuing a "Get" command. The "Get" 576 command will target a predefined file of object size bytes. 578 Once the file transfer has completed, the virtual client will close 579 the FTP connection by issuing the QUIT command. When testing multiple 580 object sizes, all connections established for the previous file 581 transfer MUST have been torn down before testing the next object 582 size. 584 5.4.2.2 Measurement 586 The Goodput for each of the object sizes transferred MUST be 587 measured. See appendix B for details in regards to measuring the 588 Goodput of the DUT/SUT. Only file transfers which have been 589 successfully acknowledged by the server are to be included in the 590 Goodput measurements. 592 The transaction time for each object transferred MUST be measured. 593 The start time will begin when the time the "Get" commands is 594 initiated and end at the time when the client receives an 595 acknowledgment from the server that file transfer has completed. 597 5.4.2.3 Reporting Format 599 Goodput analysis: 600 Reporting SHOULD include a graph of the number of connections 601 versus the measured Goodput in Mbps. 603 Failure analysis: 604 Reporting should include a graph of number of connections versus 605 percent success. 607 Transaction Processing analysis: 608 Reporting should include a graph of number of virtual connections 609 versus average transaction time. 611 6.4.3 SMTP Goodput 613 Another application commonly in use today is the mail transfer. One 614 the common transport mechanisms for mail messages is the Simple Mail 615 Transfer Protocol(SMTP). As with the FTP Goodput, the SMTP Goodput 616 will allow individuals to measure how much successful SMTP traffic 617 has been forwarded by the DUT/SUT. 619 5.4.1.3 Setup Parameters 621 The following parameters MUST be defined. Each variable is 622 configured with the following considerations. 624 Number of Connections - Defines the number of connections to be 625 opened for transferring data objects. Number MUST be equal or 626 greater than the number of virtual clients participating in the 627 test. The number SHOULD be a multiple of the virtual client 628 participating in the test. 630 Connection Rate - Defines the rate at which connections are 631 attempted. Number MUST be evenly divided among all of the 632 virtual clients participating in the test. 634 Message Size - Defines the number of bytes to be transferred across 635 each connection. RECOMMENDED message sizes still need to be 636 determined. 638 5.4.1.4 Procedure 640 Each virtual client will establish a SMTP connection to its 641 respective server(s) at a user specified rate. The transaction 642 involved in establishing the SMTP connection should follow the 643 procedure defined in Appendix B. 645 After the greeting exchanges have been completed, the client will 646 initiate the transfer of the message by initiating the MAIL command. 647 The client will then send the predefined message. 649 Once the message has been acknowledged as being received by the 650 server, the virtual client will then close the connection. When 651 testing multiple message sizes, all connections established for the 652 previous test MUST be torn down before testing the next message 653 size. This process is to be repeated for each of the defined message 654 sizes. 656 5.4.2.2 Measurement 658 The Goodput for each of the message sizes transferred MUST be 659 measured. See appendix D for details in regards to measuring the 660 Goodput of the DUT/SUT. Only messages which have been successfully 661 acknowledged by the server are to be included in the Goodput 662 measurements. 664 The transaction time for each message transferred MUST be measured. 665 The start time will begin when the time the "MAIL" command is 666 initiated and end at the time when the client receives an 667 acknowledgment from the server that the message has been received. 669 6.4.2.3 Reporting Format 671 Goodput analysis: 672 Reporting SHOULD include a graph of the number of connections 673 versus the measured Goodput in Mbps. 675 Failure analysis: 676 Reporting should include a graph of number of connections versus 677 percent success. 679 Transaction Processing analysis: 680 Reporting should include a graph of number of virtual connections 681 versus average transaction time. 683 6.4.3 HTTP Goodput 685 Another common application is the World Wide Web (WWW) application 686 that can access documents over the Internet. This application uses 687 the Hypertext Transfer Control Protocol (HTTP) to copy information 688 from one system to another. 690 HTTP Goodput measurement is actually determined by evaluating the 691 Forwarding rate of packets. This is based on measuring only data 692 that has successfully been forwarded to the destination interface. 694 When benchmarking the performance of the DUT/SUT, consideration of 695 the HTTP version being used must be taken into account. Appendix C 696 of this document discusses enhancements to the HTTP protocol which 697 may impact performance results. 699 6.4.1.3 Setup Parameters 701 The following parameters MUST be defined. Each variable is 702 configured with the following considerations. 704 Number of Connections - Defines the number of HTTP connections 705 to be opened for transferring data objects. Number MUST be equal or 706 greater than the number of virtual clients participating in the 707 test. The number SHOULD be a multiple of the virtual client 708 participating in the test. 710 Connection Rate - Defines the rate at which connections are 711 attempted. Number MUST be evenly divided among all of the 712 virtual clients participating in the test. 714 Object Size - Defines the number of bytes to be transferred 715 across each connection. Need to determine the RECOMMENDED 716 object sizes. 718 6.4.2.1 HTTP Procedure 720 For the HTTP Goodput tests, it is RECOMMENDED to determine which 721 version of HTTP the DUT/SUT has implemented and use the same 722 version for the test. To determine the version of HTTP, the user 723 documentation of the DUT/SUT SHOULD be consulted. 725 Each client will attempt to establish HTTP connection's to their 726 respective servers a user defined rate. The clients will attach to 727 the servers using either the servers IP address or NAT proxy 728 address. 730 After the client has established the connection with the server, 731 the client will initiate GET command(s) to retrieve predefined 732 web page(s). 734 When employing HTTP/1.0 in benchmarking the performance of the 735 DUT/SUT, only one object will be retrieved for each of the defined 736 object sizes. After the object has been transferred, the connection 737 should then be torn down. When defining multiple objects, object 738 transfers must be completed and the connections closed for all 739 of the participating clients prior testing the next object size. 740 This process is repeated until all of the defined objects are 741 tested. 743 When employing HTTP/1.1, all objects defined by the user will 744 be requested with a GET request over the same connection. The 745 connection should then be torn down after all of the objects 746 have been transferred. 748 6.4.2.2 Measurement 750 The Goodput for each of the objects sizes transferred MUST be 751 measured. See appendix D for details in regards to measuring the 752 Goodput of the DUT/SUT. Only objects which have been successfully 753 acknowledged by the server are to be included in the Goodput 754 measurements. 756 The transaction times for each object transferred MUST measured. 757 The transaction connection time starts when the connection is 758 made and will end when the web pages is completely mapped on the 759 virtual client (when the client sends an acknowledgment packet is 760 sent from the client). 762 6.4.2.3 Reporting Format 764 Goodput analysis: 765 Reporting SHOULD include a graph of the number of connections 766 versus the measured Goodput in Mbps. 768 Failure analysis: 769 Reporting should include a graph of number of connections versus 770 percent success. 772 Transaction Processing analysis: 773 Reporting should include a graph of number of virtual connections 774 versus average transaction time. 776 Version Information 778 Report MUST include the version of HTTP used for the test. In 779 addition, if the HTTP/1.1 is used, the number of concurrent GET's 780 allowable(Pipelining) SHOULD be reported. 782 6.5 Concurrent Application Goodput 784 To determine the Goodput of the DUT/SUT when offering a mix of FTP, 785 SMTP and HTTP traffic flows. Real world traffic does not consist 786 of a single protocol, but a mix of different applications. This 787 test will allow an individual to determine how well the DUT/SUT 788 handles a mix of applications by comparing the results to the 789 individual baseline measurements. 791 6.5.1 Setup Parameters 793 The following parameters MUST be defined. Each variable is 794 configured with the following considerations. 796 Number of Connections - Defines the aggregate number of connections 797 to be opened for transferring data/message objects. Number MUST be 798 equal to or greater than the number of virtual clients participating 799 in the test. The number SHOULD be a multiple of the virtual client 800 participating in the test. 802 Connection Rate - Defines the rate at which connections attempts are 803 opened. Number MUST be evenly divided among all of the virtual 804 clients participating in the test. 806 Object/Message Size - Defines the number of bytes to be transferred 807 across each connection. RECOMMENDED message sizes still needs to be 808 determined. 810 At a minimum, at least one of the following parameters MUST be 811 defined. In addition, the cumulative percentage all the defined 812 percentages MUST equal 100%. 814 FTP Percentage - Defines the percentage of traffic connections 815 which are to consist of FTP file transfers. 817 SMTP Percentage - Defines the percentage of traffic connections 818 which are to consist of SMTP Message transfers. 820 HTTP Percentage - Defines the percentage of traffic connections 821 which are to consist of HTTP GET requests. 823 6.5.1 Procedure 825 This test will run each of the single application Goodput tests, 826 for which there is a defined percentage, concurrently. For each 827 of the defined traffic types, the connection establishment, 828 data/message transfer and teardown procedures will be the same 829 as defined in the individual tests. 831 6.5.2 Measurements 833 As with the individual tests, the Goodput for each of the defined 834 traffic types MUST be measured. See appendix D for details in 835 regards to measuring the Goodput of the DUT/SUT. Only messages/data 836 which have been successfully acknowledged as being transferred are 837 to be included in the Goodput measurements. 839 The transaction times for each of the defined applications MUST be 840 measured. See the appropriate single application Goodput test for 841 the specifics of measuring the transaction times for each of the 842 defined traffic types. 844 6.5.3 Reporting Format 846 Goodput analysis: 847 Reporting SHOULD include a graph of the number of connections 848 versus the measured Goodput in Mbps for each of the defined 849 traffic types(FTP, SMTP, HTTP). 851 Failure analysis: 852 Reporting should include a graph of number of connections versus 853 percent success for each of the defined traffic types. 855 Transaction Processing analysis: 856 Reporting should include a graph of number of virtual connections 857 versus average transaction for each of the defined traffic types. 859 6.6 Illegal Traffic Handling 861 To determine the behavior of the DUT/SUT when presented with a 862 combination of both legal and Illegal traffic. 864 6.6.1 Procedure 866 Still Needs to be determined 868 6.6.2 Measurements 870 Still Needs to be determined 872 6.6.3 Reporting Format 874 Still Needs to be determined 876 6.7 Latency 878 Determine the latency of application layer data through the DUT/SUT. 880 6.7.1 Procedure 882 Still Needs to be determined 884 6.7.2 Measurements 886 Still needs to be determined. 888 6.7.3 Reporting format 890 Still needs to be determined. 892 APPENDICES 894 APPENDIX A: FTP(File Transfer Protocol) 896 A.1 Introduction 898 The FTP protocol was designed to be operated by interactive end users 899 or application programs. The communication protocol to transport this 900 service is TCP. The core functions of this application enable users 901 to copy files between systems, view directory listings and perform 902 house keeping chores - such as renaming, deleting and copying files. 903 Unlike other protocols, FTP uses two connections. One connection, 904 called the control connection, is used to pass commands between 905 the client and the server. The other, called the data connection, is 906 used to transfer the actual data(Files, directory lists, etc.). 908 A.2 Connection Establishment/Teardown(Control) 910 FTP control connections are established by issuing OPEN command 911 targeting either the URL or a specific IP address. Since the 912 methodology does not include DNS servers, OPEN commands should 913 target specific IP address of target server. 915 It is RECOMMENDED to perform the test using Anonymous FTP Login and 916 should use the following syntax: 918 User ID: Anonymous 919 Password: will correspond to the System ID 920 (client1_1@test.net through client 1_8@test.net) 922 Once a successful login acknowledgment is received from the server, 923 the client may then initiate a file transfer. After all transfer 924 operations have been completed, the FTP connection may be closed by 925 issuing a QUIT command. 927 A.3 Data Connection 929 The data connection is established each time the user requests a file 930 transfer and torn down when the transfer is completed. FTP supports 931 two modes of operation, namely normal mode and passive mode, which 932 determine who initiates the data connection. In normal mode 933 operation, the server initiates the data connection, targeting a 934 predefined PortID specified in the PORT command. In passive mode, the 935 client initiates the data connection, targeting the PortID returned 936 in response to the PASV Command. It is RECOMMENDED to perform the 937 tests in normal mode operation. 939 File transfers are initiated by using the "Get" or "Put" command and 940 specifying the desired filename. The tests defined in this document 941 will use the "Get" command to initiate file transfers from the target 942 server to the client. 944 A.4 Object Format 946 Need to define the object format. 948 APPENDIX B: SMTP (Simple Mail Transfer Protocol) 950 B.1 Introduction 952 The SMTP defines a simple straight forward way to move messages 953 between hosts. There are two roles in the SMTP protocol, one is the 954 sender and one is the receiver. The sender acts like a client and 955 establishes a TCP connection with the receiver which acts like a 956 server. The transactions defined in this section will use the terms 957 client and server in place of sender and receiver. 959 B.2 Connection Establishment/Teardown 961 Each connection involves a connection greeting between the 962 sender(Client) and receiver(Server). After a connection is first 963 established, both the server and the client will exchange greetings 964 identifying themselves. The syntax used to identify each other's 965 hostnames SHOULD be: 967 "SMTPRcv_.com" 968 "SMTPSender_.com" 970 where and represent a 971 unique virtual number for the client and server 972 respectively. 974 The basic transactions in moving mail between two hosts involve three 975 basic steps which are outlined below. These are: 977 1) Client issuing a MAIL command identifying the message originator 978 for that session. Syntax used to identify the originator SHOULD 979 be as follows: 981 connection1,2,3...@hostname 983 2) Client issues an RCPT command identifying the recipient of the 984 message for that session. Syntax used to identify the recipient 985 of the message SHOULD be as follows: 987 reciever1,2,3...@hostname 989 3) Client issues a DATA command. After receiving the 990 acknowledgment from the server, the client will then transfer 991 the message which MUST include a line with a period to 992 indicate to the server the end of the message. Once the end of 993 message is received by the server, it will acknowledge the end 994 of message. 996 The client may initiate another message transfer or close the session 997 by initiating the QUIT command. 999 B.3 Message Format 1001 As Internet e-mail has evolved, SMTP extensions have been added to 1002 support both audio and video message transfers. For these firewall 1003 tests, messages SHOULD consist of plain text ASCII. 1005 APPENDIX C: HTTP(HyperText Transfer Protocol) 1007 C.1 Introduction 1009 As HTTP has evolved over the years, changes to the protocol have 1010 occurred to both fix problems of previous versions as well as improve 1011 performance. The most common versions in use today are HTTP/1.0 and 1012 HTTP/1.1 and are and are discussed below. 1014 C.2 Version Considerations 1016 HTTP/1.1 was approved by the WWW Consortium in July 1999 as an IETF 1017 Draft Standard. This is a formal recognition of the fact that all 1018 known technical issues have been resolved in the specification which 1019 was brought out in June 1997. HTTP/1.1 is also downward compatible 1020 with HTTP/1.0. Both protocols on the popular browsers in use today. 1021 The following is a list of features that are offered in HTTP 1.1 that 1022 are not in HTTP 1.0. 1024 - Persistent connections and pipelining 1026 Though both use TCP for data transfer, but differ in the way it is 1027 used, with the later version being more efficient. Once a connection 1028 is opened, it is not closed until the HTML document and all objects 1029 referred by it are downloaded. This technique is called persistent 1030 connection. By serving multiple requests on the same TCP segment, 1031 many control packets (which are not part of actual data transfer) 1032 are avoided. The technique of containing multiple requests and 1033 responses within the same TCP segment over a persistent connection 1034 is called pipelining. 1036 - Data compression 1038 HTTP/1.1 provides for compression of documents before file transfer. 1039 Since most other objects like images and binaries are already 1040 compressed, this feature applies only to HTML and plain text 1041 documents. 1043 - Range request and validation 1045 Bandwidth saving measure is the introduction of two new fields in 1046 an HTTP request header, viz. If-Modified-Since: and If-Unmodified- 1047 Since:. The significance of this feature is that if a browser 1048 identifies a file in its cache, it needn't reload it unless it has 1049 changed since the last time it was used. 1051 - Support for multiple hosts 1053 It is common for an ISP to host more than one Web site on a single 1054 server. In such a case, each domain requires its own IP address. 1056 C.3 Object Format 1058 Object SHOULD be an HTLM formatted object. 1060 Append D GOODPUT Measurements 1062 Methodology for determining the "Goodput" of the DUT still needs 1063 to be determined. Note that the methodology should be independent 1064 of the application which is initiating the transfer of the object 1065 (File, message, Web Page, etc.). 1067 Appendix E. References 1069 Newman, "Benchmarking Terminology for Firewall Devices", RFC 2647, 1070 February 1998. 1072 J. Postel, "Simple Mail Transfer Protocol", RFC 821, August 1982. 1074 R. Fielding, J. Gettys, J. Mogul, H Frystyk, T. Berners, "Hypertext 1075 Transfer Protocol -- HTTP/1.1", January 1997 1077 J. Postel, J. Reynolds, "File Transfer Protocol(FTP)", October 1985