File Transfer Protocol HASH Command for Cryptographic HashesPompano BeachFLUSAanthonybryan@gmail.comhttp://www.metalinker.orgtim.kosse@filezilla-project.orghttp://filezilla-project.org/daniel@haxx.sehttp://www.haxx.se/The File Transfer Protocol does not offer any method to verify the integrity of a transferred file, nor can two files be compared against each other without actually transferring them first. Cryptographic hashes are a possible solution to this problem. In the past, several attempts have been made to add commands to obtain checksums and hashes,
however none have been formally specified, leading to non-interoperability and confusion. To solve these issues, this document specifies a new FTP command to be used by clients to request cryptographic hashes of files.
The File Transfer Protocol does not offer any method to verify the integrity of a transferred file, nor can two files be compared against each other without actually transferring them first. Cryptographic hashes are a possible solution to this problem. In the past, several attempts have been made to add commands to obtain checksums and hashes,
however none have been formally specified, leading to non-interoperability and confusion. (See Appendix B for more information). To solve these issues, this document specifies a new FTP command to be used by clients to request cryptographic hashes of files.
HTTP has a similar feature named Instance Digests which allows a client to request the cryptographic hash of a file.[[ Discussion of this draft should take place on ftpext@ietf.org (or apps-discuss@ietf.org if necessary). ]]This specification describes conformance of File Transfer Protocol Extension for cryptographic hashes.The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in BCP 14, , as scoped to those conformance targets.This document also uses notation defined in STD 9, .
In particular, the terms or commands "reply", "user", "file", "pathname", "FTP commands", "user-PI" (user protocol interpreter), "server-FTP process",
"server-PI", "mode", "type", "STOR", "RETR", and "ASCII", are all used here as defined there.In the examples of FTP dialogs presented in this document, lines that
begin "C> " were sent over the control connection from the user-PI to
the server-PI, and lines that begin "S> " were sent over the control
connection from the server-PI to the user-PI. In all cases, the
prefixes shown above, including the one space, have been added for
the purposes of this document, and are not a part of the data
exchanged between client and server.Syntax required is defined using the Augmented BNF defined in .This document imports the core definitions given in Appendix B of
. There definitions will be found for basic ABNF elements
like ALPHA, DIGIT, SP, etc. To that, the following term is added
for use in this document.The VCHAR (from ) and TCHAR rules give basic character
types from varying sub-sets of the ASCII character set for use in
various commands and responses.Note that in ABNF, string literals are case insensitive. That
convention is preserved in this document, and implies that FTP
commands and parameters that are added by this specification have
values that can be represented in any case. That is, "HASH" is the
same as "hash", "Hash", "HaSh", etc., and "ftp.example.com" is the
same as "Ftp.Example.Com", "fTp.eXample.cOm", etc.Section 4.2 of defines the format and meaning of replies
by the server-PI to FTP commands from the user-PI. Those reply
conventions are used here without change.Implementers should note that the ABNF syntax (which was not used in
) used in this document, and other FTP related documents,
sometimes shows replies using the one line format. Unless otherwise
explicitly stated, that is not intended to imply that multi-line
responses are not permitted. Implementers should assume that, unless
stated to the contrary, any reply to any FTP command (including QUIT)
can be of the multi-line format described in .Throughout this document, replies will be identified by the three
digit code that is their first element. Thus the term "500 reply"
means a reply from the server-PI using the three digit code "500".A new command "HASH" is added to the FTP command set to allow the client to request the cryptographic hash of a file from a server-FTP process.The syntax for the HASH command is:
As with all FTP commands, the "HASH" command word is case independent, and MAY be specified in any character case desired.The HASH command keyword MUST be followed by a single space (ASCII 32) followed by the pathname.The pathname argument should reference the same file as other file based commands such as STOR or RETR which the same argument would reference. The pathname argument MUST represent a file path, not a directory path.
The text returned in response to the HASH command MUST be:All hash values MUST be encoded in lowercase hexadecimal format.The HASH command uses the currently selected hash algorithm. The currently selected hash algorithm can be determined with FEAT or OPTS HASH, and changed with OPTS HASH.The HASH command is meant to be used for files transmitted in Image
type mode (TYPE I) and Stream transfer mode (MODE S). The returned hash
MUST be calculated over the raw octet data of the file irrespective of
the selected data type, transfer mode or any other state affecting the
transfer. In other words, if a client were to download a full file using
TYPE I and MODE S and were to calculate the hash on the received octet
data, it would be identical to the hash returned by HASH.When replying to the FEAT command , a server-FTP process that supports the HASH command MUST include a feature line indicating that the HASH command is supported, along with a list of all supported hash algorithms in a semicolon separated list. The hash algorithm that is currently selected MUST be marked with an asterisk. The order of hash algorithms is insignificant.
This command word is case insensitive, and MAY be sent in any mixture of upper or lower case, however it SHOULD be sent in upper case. That is, the response SHOULD be:The ellipses indicate place holders where other features may be included, and are not required. The one-space indentation of the feature lines is mandatory .The IANA registry named "Hash Function Textual Names" defines values for hash algorithms. Hash names should be presented in uppercase, but comparisons should be case-insensitive, e.g. MD5, md5, Md5 are all the same.To query the current hash algorithm and to change it, the OPTS command as defined in is used with HASH as the first argument.If no second argument is passed, OPTS HASH simply returns the currently selected hash algorithm.To change the algorithm, a valid hash algorithm MUST be given as second argument. A list of valid hash algorithms is available via the FEAT command. If the command is successful, all future calls to HASH until the next successful OPTS HASH command or until the session is reinitialized (REIN) will use the selected hash algorithm.Requesting unknown or unsupported algorithms produces an error response.The syntax for OPTS HASH:The user-PI issues the FEAT command to query the server-PI about which algorithm is currently selected. This also reveals the other algorithms that the server supports. In this example, the SHA-1 algorithm is currently selected.OPTS HASH is an alternative method for the user-PI to query the server-PI about which algorithm is currently selected.
In this example, we wish to select SHA-256, a different algorithm.
The user-PI requests the cryptographic hash of a file with HASH command. Server-PI replies with cryptographic hash of file.
Client downloads file. Client hashes the downloaded file and compares its hash to the hash obtained from the server. The HASH command could also be used to verify that an uploaded file has the same hash as the local file.The server-PI SHOULD reply with a 500 reply if the HASH command is unrecognized or unimplemented.The server-PI SHOULD reply with a 501 reply to the OPTS HASH command if the user-PI has requested an unknown or unsupported algorithm.The server-PI SHOULD reply with a 550 reply if the HASH command is used on a file that can not be found.The server-PI SHOULD reply with a 552 reply if the user is not allowed to use the HASH command.The server-PI SHOULD reply with a 450 reply if the server is busy, e.g. already hashing other files yet inviting the client to retry in the future.This new command is added to the "FTP Commands and Extensions" registry created by .Command Name: HASHDescription: Cryptographic Hash of a fileFEAT String: HASHCommand Type: Service executionConformance Requirements: OptionalReference: This specificationAll conforming implementations MUST at least support the SHA-1 algorithm . Implementations SHOULD NOT make any algorithm the default that is known to be weaker than SHA-1. Support for any additional algorithms is OPTIONAL.Implementing the HASH command may impose a considerable load on the server, which could lead to denial-of-service attacks. Servers have, however,
implemented this for many years, without significant reported difficulties.
On an affected server a malicious user could, for example,
continuously send HASH commands over multiple connections and thus
consume most of the FTP server's resources, leaving little room for other
operations.
To mitigate this risk, a server SHOULD cache the calculated hashes so
that the hash of a file is only calculated once even if multiple hash
requests are sent for that file.For performance reasons, a server SHOULD a avoid hashing multiple files at the
same time which are located on the same physical media and SHOULD
instead hash them sequentially. The FTP server's right to refuse to calculate the hash is of course important to help against denial-of-service risks. A possible solution is to use the 450
reply code of HASH to indicate that the server is already busy with
another HASH operation.In addition, the HASH command can be used to draw conclusions about the
contents of a file. If the hash of a file on some server matches the
hash of some known file, then both files are likely identical. To
prevent this scenario it suffices to limit use of the HASH command to
users who would already be able to download the file.Currently, some of the hash algorithms defined in the IANA registry named
"Hash Function Textual Names" are considered insecure. These include
the whole Message Digest family of algorithms that are not suitable
for cryptographically strong verification. Malicious people could
provide files that appear to be identical to another file because of
a collision, i.e., the weak cryptographic hashes of the intended file
and a substituted malicious file could match.File Transfer ProtocolKey words for use in RFCs to Indicate Requirement LevelsHarvard University1350 Mass. Ave.CambridgeMA 02138- +1 617 495 3864sob@harvard.edu
General
keyword
In many standards track documents several words are used to signify
the requirements in the specification. These words are often
capitalized. This document defines these words as they should be
interpreted in IETF documents. Authors who follow these guidelines
should incorporate this phrase near the beginning of their document:
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL
NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described in
RFC 2119.
Note that the force of these words is modified by the requirement
level of the document in which they are used.
Feature negotiation mechanism for the File Transfer ProtocolAugmented BNF for Syntax Specifications: ABNFSecure Hash Standard (SHS)National Institute of Standards and Technology (NIST)Instance Digests in HTTPExtensions to FTPFTP Command and Extension RegistryThe MD5 and MMD5 FTP Command Extensionsjtwine@jrtwine.comThanks to John C. Klensin, Alfred Hoenes, James Twine, Robert McMurray, Mathias Berchtold, and Tatsuhiro Tsujikawa.Portions of were wholly reused in this document.[[ to be removed by the RFC editor before publication as an RFC. ]]A number of similar checksum or hash commands exist, but are not formally specified, leading to non-interoperability and confusion. The commands, any specifications, and relevant details:CKSM: GridFTP v2 Protocol Description http://www.ogf.org/documents/GFD.47.pdf Usage: OPTS CKSM <algorithm> CRLF. Supports ADLER32, MD5, CRC32.MD5/MMD5: Expired Internet Draft from 2002. Usage: MD5 <filepath> Algorithm specific command. Response codes: 251 positive completion, 500 Command Not Recognized, 502 Command Not Implemented, 504 Command Not Implemented for the Specified Argument.SITE CHECKSUM: Usage: SITE check_login SP CHECKSUM SP pathname CRLF. Supports CRC32 and MD5.SITE SHOHASH: Usage: site shohash [filename]. Supports MD5. Response codes: 200 positive completion.XCRC: By GlobalSCAPE in 2001. http://help.globalscape.com/help/secureserver2/File_Integrity_Checking.htm Usage: XCRC <filename> SP EP. SP is starting point and EP is ending point in bytes and are optional parameters. Algorithm specific command. Response codes: 250 positive completion, 450 Requested file action not taken. (File is busy), 550 Requested action not taken. (File not found, no read permission, SP or EP not correct).XMD5: XMD5 <filename> SP EP. Similar to XCRC. Algorithm specific command.XSHA, XSHA1, XSHA256, XSHA512: Usage similar to XCRC, although SP/EP usage unknown. Algorithm specific commands.An incomplete list of FTP clients and servers that have implemented these commands:Akamai NetStorage (supports SITE CHKHSH/SITE SHOHASH) p17-18 http://pigdogslow.dyndns.org/NetStorage_UserGuide.pdfApache Ftp Server (supports MD5/MMD5 from draft-twine-ftpmd5) http://cwiki.apache.org/FTPSERVER/documentation.htmlBackup4all Pro (supports XCRC)Backup to FTP (supports XCRC)BlackMoon FTP Server (supports XCRC) http://www.blackmoonftpserver.com/portal/readmore/features.htmlC.P.A. Secure (supports XCRC) http://www.cpasecure.com/CPASecureVsSecureFTP.htmlCerberus FTP server (supports XCRC, XMD5, XSHA1, XSHA256, XSHA512) http://www.softpedia.com/progChangelog/Cerberus-FTP-Server-Changelog-1904.htmlCore FTP Pro (supports XCRC)Cross FTP Server (supports MD5/MMD5)FileCOPA FTP Server (supports XCRC, XMD5, XSHA1) http://www.filecopa-ftpserver.com/features.htmlFile Watchdogs FTP Server (supports XCRC, XMD5, XSHA1, XSHA256, XSHA512) http://www.filewatchdogs.com/ftpsitehosting/help/15559.htmFireFTP (supports XMD5, XSHA1) http://fireftp.mozdev.org/features.htmlFTP Daemon (supports SITE CHECKMETHOD/SITE CHECKSUM) http://www.pro-bono-publico.de/projects/ftpd.htmlFTP Voyager (supports XCRC) http://www.ftpvoyager.com/XCRC.aspGene6 FTP Server http://www.g6ftpserver.com/en/information#featuresGlobalSCAPE's Secure FTP Server / EFT Server / CuteFTP clients (supports XCRC)Globus FTP client / Globus Toolkit(supports CKSM) http://www.globus.org/toolkit/releasenotes/3.2.0/gridftp_notes.htmlGoldenGate FTP (Ftp Full Java Server) (supports XCRC, XMD5, XSHA1)IceWarp FTP Server http://www.icewarp.com/products/ftp_server/ICS FTP client (supports XCRC, XMD5) http://www.magsys.co.uk/delphi/magics.aspioFTPD (supports XCRC)JAFS (supports XCRC and MD5) http://www.sbbi.net/site/jafs/features.htmlKellerman FTP (supports XCRC) http://sharptoolbox.com/tools/kellerman-ftpLimagito FTP server (supports XCRC, XMD5, XSHA1) http://www.limagito.com/file-mover-features.htmlMOVEit DMZ (supports XSHA1)Nofeel FTP server (supports XCRC, XMD5, XSHA1) http://www.nftpserver.com/history.phpNull FTP (supports XCRC, XMD5, XSHA) http://www.sharewareconnection.com/null-ftp-client-pro.htmOrenosv FTP Client (supports XCRC, XMD5) http://www.orenosv.com/orenosv/ftpcli_en.htmlProFTPD module mod_digest (supports XCRC, XMD5, XSHA1, SHA256) http://www.smartftp.com/oss/proftpd/mod_digest.htmlPSFTPd Secure FTP Server (supports XCRC, XMD5, XSHA) http://www.psftp.de/psftpd_fo.phpQuick 'n Easy FTP Server (supports XCRC) http://www.pablosoftwaresolutions.com/html/quick__n_easy_ftp_server_pro.htmlRaidenFTPD32 FTP server (supports XCRC, XMD5)Robo-FTP Server (supports XCRC, XMD5, XSHA1) http://kb.robo-ftp.com/change_log/show/61SyncBackPro and SyncBackSE (supports XCRC) http://www.2brightsparks.com/syncback/sbpro-changes.htmlSecure FTP Factory (supports XCRC)Serv-U FTP Server (supports XCRC) http://www.serv-u.com/help/serv_u_help/additional_ftp_commands_supported_by_serv_u.htmSmartFTP client (supports XCRC, XMD5, XSHA, CKSM) http://www.smartftp.com/features/Starksoft Ftp Component for .NET / Mono (supports XCRC, XMD5, XSHA1) http://www.starksoft.com/prod_ftp.htmlTitan FTP Server (supports XCRC)Turbo FTP (supports XCRC)WISE-FTP (supports XCRC) http://www.wise-ftp.com/news/WS_FTP client / server (supports XSHA1, server also XMD5, XSHA1, XSHA256, XSHA512) http://ipswitchft.custhelp.com/app/answers/detail/a_id/671/kw/xmd5/r_id/166/sno/1wuftpd (supports SITE CHECKMETHOD/SITE CHECKSUM)wzdFTPd (supports XCRC, XMD5) http://www.wzdftpd.net/wiki/index.php/CommandsZalman FTP Client (supports XCRC) http://www.zalmansoftware.com/download.htmlzFTPServer[[ to be removed by the RFC editor before publication as an RFC. ]]Known issues concerning this draft:
Should HASH support partial file hashes, similar to the Content-MD5 HTTP Header.Underspecification of the representation of the file that shall undergo the hash calculation.Should the server response include the algorithm? i.e. "S> 213 SHA-256 xxxxxxxxxxxxxxx"-05 : June 29, 2010.
Add Basic Tokens and Server Replies subsections from RFC 3659.-04 : June 11, 2010.
User-PI usage and command errors sections updated.-03 : May 21, 2010.
List of non-standard checksum and hash commands and their implementations.-02 : April 16, 2010.
Error codes section.-01 : April 7, 2010.
Changing HASH algorithm with OPTS.Reference RFC 5797 and add IANA Considerations section.Informative Reference to expired Internet Draft (draft-twine-ftpmd5) which attempted to address this issue (it only supported one hash, MD5).-00 : October 19, 2009.
Initial draft.