Pompano Beach FL USA anthonybryan@gmail.com http://www.metalinker.org

Shiga Japan tatsuhiro.t@gmail.com http://aria2.sourceforge.net

neil@nabber.org http://www.nabber.org

MirrorBrain

Venloer Str. 317 Koeln 50823 DE +49 221 6778 333 8 peter@poeml.de http://mirrorbrain.org/~poeml/

This document specifies behavior for Metalink/XML clients, publishers, and proxy caches. Metalink XML files contain multiple download locations (mirrors and Peer-to-Peer), cryptographic hashes, digital signatures, and other information. Metalink clients can use this information to make file transfers more robust and reliable. Normative requirements for Metalink/XML clients, publishers, and proxy caches are described here. Discussion of this draft should take place on the apps-discuss mailing list (apps-discuss@ietf.org), although this draft is not a WG item. The changes in this draft are summarized in .

Metalink is a document format based on Extensible Markup Language (XML) that describes a file or list of files to be downloaded from a server. Metalinks can list a number of files, each with an extensible set of attached metadata. Each listed file can have a description, multiple cryptographic hashes, and a list of Uniform Resource Identifiers (URIs) from which it is available. Identical copies of a file are frequently accessible in multiple locations on the Internet over a variety of protocols (such as FTP, HTTP, and Peer-to-Peer). In some cases, users are shown a list of these multiple download locations (mirrors) and must manually select a single one on the basis of geographical location, priority, or bandwidth. This distributes the load across multiple servers, and should also increase throughput and resilience. At times, however, individual servers can be slow, outdated, or unreachable, but this can not be determined until the download has been initiated. Users will rarely have sufficient information to choose the most appropriate server, and will often choose the first in a list which might not be optimal for their needs, and will lead to a particular server getting a disproportionate share of load. The use of suboptimal mirrors can lead to the user canceling and restarting the download to try to manually find a better source. During downloads, errors in transmission can corrupt the file. There are no easy ways to repair these files. For large downloads this can be extremely troublesome. Any of the number of problems that can occur during a download lead to frustration on the part of users. Knowledge about availability of a download on mirror servers can be acquired and maintained by the operators of the origin server or by a third party. This knowledge, together with cryptographic hashes, digital signatures, and more, can be stored in a machine-readable Metalink file. The Metalink file can transfer this knowledge to the user agent, which can peruse it in automatic ways or present the information to a human user. User agents can fall back to alternate mirrors if the current one has an issue. Thereby, clients are enabled to work their way to a successful download under adverse circumstances. All this can be done transparently to the human user and the download is much more reliable and efficient. In contrast, a traditional HTTP redirect to one mirror conveys only comparatively minimal information -- a referral to a single server, and there is no provision in the HTTP protocol to handle failures. Other features that some clients provide include multi-source downloads, where chunks of a file are downloaded from multiple mirrors (and optionally, Peer-to-Peer) simultaneously, which frequently results in a faster download. Metalinks can leverage HTTP, FTP, and Peer-to-Peer protocols together, because regardless of the protocol over which the Metalink was obtained, it can make a resource accessible through other protocols. If the Metalink was obtained from a trusted source, included verification metadata can solve trust issues when downloading files from replica servers operated by third parties. Metalinks also provide structured information about downloads that can be indexed by search engines. Metalink/HTTP is an alternative and complementary representation of Metalink information, using HTTP header fields instead of the XML-based document format . Metalink/HTTP is used to list information about a file to be downloaded. This can include lists of multiple URIs (mirrors and Peer-to-Peer information), cryptographic hashes, and digital signatures. Some popular sites automate the process of selecting mirrors using DNS load balancing, both to approximately balance load between servers, and to direct clients to nearby servers with the hope that this improves throughput. Indeed, DNS load balancing can balance long-term server load fairly effectively, but it is less effective at delivering the best throughput to users when the bottleneck is not the server but the network. This document describes a mechanism by which the benefit of mirrors can be automatically and more effectively realized. All the information about a download, including mirrors, cryptographic hashes, digital signatures, and more can be transferred in a Metalink. This Metalink transfers the knowledge of the download server (and mirror database) to the client. Clients can fallback to other mirrors if the current one has an issue. With this knowledge, the client is enabled to work its way to a successful download even under adverse circumstances. All this can be done without complicated user interaction and the download can be much more reliable and efficient. Furthermore, in order to provide better load distribution across servers and potentially faster downloads to users, Metalinks facilitates multi-source downloads, where portions of a file are downloaded from multiple mirrors (and optionally, Peer-to-Peer) simultaneously. The client will then be able to request chunks of the file from the various sources, scheduling appropriately in order to maximize the download rate.

This specification describes conformance of Metalink/HTTP. The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14, , as scoped to those conformance targets.

In this context, "Metalink" refers to "Metalink/XML" refers to the XML format described in . The following terms as used in this document are defined here: Metalink Generator : Application that creates Metalink/XML files, and includes information about the files described in the Metalink such as locations (on Mirror servers or other methods like P2P), file sizes, and cryptographic hashes. Metalink Publisher : One who uses a Metalink Generator to create Metalink/XML files that are then offered to people to improve their download experience. Mirror server : Typically FTP or HTTP servers that "mirror" the Metalink server, as in they provide identical copies of (at least some) files that are also on the mirrored server. Metalink/XML : An XML file which describes a file that can contain information such as mirrors, cryptographic hashes, digital signatures, and more. Metalink Processors or Clients : Applications that process Metalink/XML and use them provide an improved download experience. They support HTTP and could also support other download protocols like FTP or various Peer-to-Peer methods. Metalink/HTTP : An HTTP header that contains information such as mirrors, cryptographic hashes, and digital signatures, which could also be in a Metalink/XML file.

Metalink/XML clients use the mirrors provided by a Metalink/XML file. Metalink clients SHOULD support HTTP and SHOULD support FTP . Metalink clients MAY support BitTorrent , or other download methods. Metalink clients SHOULD switch downloads from one mirror to another if a file is not available or if a mirror becomes unreachable. Metalink clients MAY support multi-source, or parallel, downloads, where portions of a file can be downloaded from multiple mirrors simultaneously (and optionally, from Peer-to-Peer sources). Metalink clients SHOULD support error recovery by using the cryptographic hashes of parts of the file listed in Metalink/XML files, as described in , Section 4.1.3. Metalink clients SHOULD support checking digital signatures. Digital signatures should be given preference over file hashes, if both are included in a Metalink/XML file. Metalink/XML clients MUST process metalinks available by URI. They MAY process local Metalinks. Metalink/XML clients SHOULD recognize Metalink/XML files by MIME type. [[What about misconfigured/unupdated servers that do not have correct MIME type? SHOULD(?) clients recognize Metalinks by file extension as well?]] If Metalink/XML clients support HTTP, they SHOULD support "transparent metalink" usage to upgrade from a regular download to an enhanced download, using the information in the Metalink/XML file.(see ). If a file with same name already exists locally, Metalink/XML clients SHOULD verify full file hash and if hash is correct, do not re-download the file. If a file exists and full file hash is incorrect, Metalink/XML clients MAY repair file if partial file hashes exist. Otherwise, Metalink/XML clients MAY write to a different output file name (file_2 or file(2) like some apps already do). Metalink/XML clients SHOULD (or MUST?) verify full file hash after download completes. If there is an error, MUST describe as corrupted and MAY re-download or keep download? SHOULD verify chunk hash if available and re-get error parts. SHOULD (or MAY?) be done during initial download process, MAY be done after download completed or to repair file downloaded another way? Metalink/XML clients SHOULD(?) use BitTorrent chunk hashes with HTTP/FTP downloads to repair file errors if the client supports torrents downloads. (What if chunk hashes are present in torrent and metalink, should one be preferred?) If client supports Metalink/XML AND Metalink/HTTP, which should be preferred (in case mirrors/hashes differ)? Metalink/XML clients SHOULD make use of Metalink/XML origin element if dynamic="true" to check for updated Metalink. Metalink/XML clients MAY make use of the alpha-2 two-letter country code for the geographical location of the physical server the URI is used to access, in an attempt to improve the download experience. Metalink/XML clients could support multiple versions of Metalink/XML, and if they do they SHOULD prefer Metalink/XML files. If a client supports Metalink/HTTP and Metalink/XML, it MAY prefer Metalink/HTTP but still use partial file hashes in the Metalink/XML files. Metalink/XML clients MAY create the directory structure described in the Metalink/XML at a relative download location, but see . Metalink clients could use the location of the original Metalink in the "Referer" header field for these ranged requests. Metalink clients MAY support the use of metainfo files (such as BitTorrent) for downloading files. Metalink clients SHOULD support the use of OpenPGP signatures. Metalink clients SHOULD support the use of S/MIME signatures. Previously, Metalink/XML clients that support HTTP would do Accept header transparent content negotiation, but this has been deprecated. [[ NOTE: A number of requirements of Metalink clients are also in . Should these be repeated or referenced?]]

Metalink/XML publishers MUST use correct MIME type for metalink files Metalink/XML publishers SHOULD advertise Metalink/XML file with Link HTTP header field from regular download for "transparent metalink" usage (see ). Metalink/XML publishers SHOULD publish with chunk hashes if error recovery ability is desired (and files meet certain criteria like "large enough" - no point for 10k size file). Metalink Generators SHOULD offer Metalink/XML documents that contain cryptographic hashes of parts of the file (and other information) if error recovery is desirable. Metalink/XML publishers SHOULD publish with size element if it refers to a specific file. Metalink/XML publishers MAY do Accept header transparent content negotiation (deprecated?) Metalink/XML publishers SHOULD include Metalink/XML origin element and dynamic="true" if updated metalinks will be offered. Metalink publishers SHOULD include digital signatures, as described in Section 4.2.13.

Metalink/XML files for a given resource MAY be provided in a Link header field as shown in this example:

This example shows a brief HTTP response header with .meta4: Link: <http://example.com/example.ext.meta4>; rel=describedby; type="application/metalink4+xml"

Metalink/XML files are specified in , and they are particularly useful for providing metadata such as cryptographic hashes of parts of a file (see Section 4.1.3), allowing a client to recover from errors (see ). Metalink servers SHOULD provide Metalink/XML files with partial file hashes in Link header fields, and Metalink clients SHOULD use them for error recovery.

Mirror servers are typically FTP or HTTP servers that "mirror" another server. That is, they provide identical copies of (at least some) files that are also on the mirrored server. Mirror servers SHOULD support serving partial content.

Metalink/XML proxy cache could detect and log Metalink usage. Metalink/XML proxy cache MUST? use a whitelist for trusted sources by domain name (ie kde.org, ubuntu.com, fedoraproject.org) to prevent cache poisoning. Metalink/XML proxy cache SHOULD use preferred mirrors (those that are most cost efficient/better/local) Metalink/XML proxy cache MAY? repair errors or use hashes? I guess so, but the client will also be verifying hashes.

Metalink clients begin with a Metalink/XML document. They parse the XML and obtain a list of ways to retrieve a file or files from FTP or HTTP mirrors or P2P. After that, the client follows with a GET request to the desired mirrors. From the Metalink/XML file, the client learns some or all of the following metadata about the requested object: Mirror list, which can describe the mirror's priority and geographical location. Whole and partial file cryptographic hash. Object size. Peer-to-peer information. Digital signature. Next, the Metalink client requests a Range of the object from a mirror server:

GET /example.ext HTTP/1.1 Host: www2.example.com Range: bytes=7433802- Referer: http://www.example.com/distribution/example.ext

Metalink clients SHOULD use partial file cryptographic hashes as described in , if available, to detect if the mirror server returned the correct data. Errors in transmission and substitutions of incorrect data on mirrors, whether deliberate or accidental, can be detected with error correction as described in . Here, the mirror server has the correct file and responds with a 206 Partial Content HTTP status code and appropriate "Content-Length" and "Content Range" header fields. In this example, the mirror server responds, with data, to the above request:

HTTP/1.1 206 Partial Content Accept-Ranges: bytes Content-Length: 7433801 Content-Range: bytes 7433802-14867602/14867603

Metalink clients MAY start a number of parallel ranged downloads (one per selected mirror server other than the first) using mirrors provided by the Metalink/XML. Metalink clients MUST limit the number of parallel connections to mirror servers, ideally based on observing how the aggregate throughput changes as connections are opened. It would be pointless to blindly open connections once the path bottleneck is filled. After establishing a new connection, a Metalink client SHOULD monitor whether the aggregate throughput increases over all connections that are part of the download. The client SHOULD NOT open additional connections during this period. If the aggregate throughput has increased, the client MAY open an additional connection and repeat these steps. Otherwise, the client SHOULD NOT open a new connection until an established one closes. The Metalink client can determine the size and number of ranges requested from each server, based upon the type and number of mirrors and performance observed from each mirror. Note that Range requests impose an overhead on servers and clients need to be aware of that and not abuse them. When dowloading a particular file, metalink clients MUST NOT make more than one concurrent request to each mirror server that it downloads from. Metalink clients SHOULD close all but the fastest connection if any Ranged requests generated after the first request end up with a complete response, instead of a partial response (as some mirrors might not support HTTP ranges), if the goal is the fastest transfer. Metalink clients MAY monitor mirror conditions and dynamically switch between mirrors to achieve the fastest download possible. Similarly, Metalink clients SHOULD abort extremely slow or stalled range requests and finish the request on other mirrors. If all ranges have finished except for the final one, the Metalink client can split the final range into multiple range requests to other mirrors so the transfer finishes faster. Metalink clients MUST reject individual downloads from mirrors where the file size does not match the file size as reported by the Metalink server. If a Metalink client does not support certain download methods (such as FTP or BitTorrent) that a file is available from, and there are no available download methods that the client supports, then the download will have no way to complete. Metalink clients MUST verify the cryptographic hash of the file once the download has completed. If the cryptographic hash offered in the Metalink/XML does not match the cryptographic hash of the downloaded file, see for a possible way to repair errors. If the download can not be repaired, it is considered corrupt. The client can attempt to re-download the file. Metalink clients that support verifying digital signatures MUST verify digital signatures of requested files if they are included. Digital signatures MUST validate back to a trust anchor as described in the validation rules in and .

Error prevention, or early file mismatch detection, is possible before file transfers with the use of file sizes provided in Metalink/XML. Error detection requires full file cryptographic hashes in the Metalink/XML to detect errors in transfer after the transfers have completed. Error correction, or download repair, is possible with partial file cryptographic hashes.

To verify the individual ranges of files, which might have been requested from different sources, see .

Partial file cryptographic hashes can be used to detect errors during the download. Metalink servers SHOULD provide Metalink/XML files with partial file hashes in Link header fields as specified in , and Metalink clients SHOULD use them for error correction. An error in transfer or a substitution attack will be detected by a cryptographic hash of the object not matching the full file checksum from the Metalink/XML. If the cryptographic hash of the object does not match the full file checksum from the Metalink/XML, then the client SHOULD use the partial file cryptographic hashes (if available). This may contain partial file cryptographic hashes which will allow detection of which mirror server returned incorrect data. Metalink clients SHOULD use the Metalink/XML data to figure out what ranges of the downloaded data can be recovered and what needs to be fetched again. Other methods can be used for error correction. For example, some other metainfo files also include partial file hashes that can be used to check for errors.

None.

Metalink/XML clients MUST sanitize directory traversal information as specified in Section 4.1.2.1. Also see Section 5 and Section 4.3 for similar considerations when creating file names supplied by Metalink/XML files which could be created without user input or awareness.

Metalink clients handle URIs and IRIs. See Section 7 of and Section 8 of for security considerations related to their handling and use.

There is potential for spoofing attacks where the attacker publishes Metalinks with false information. In that case, this could deceive unaware downloaders into downloading a malicious or worthless file. As with all downloads, users should only download from trusted sources. Also, malicious publishers could attempt a distributed denial of service attack by inserting unrelated URIs into Metalinks. contains information on amplification attacks and denial of service attacks.

Proxy caches MUST prevent cache poisoning.

Currently, some of the hash types defined in the IANA registry named "Hash Function Textual Names" are considered insecure. These include the whole Message Digest family of algorithms that are not suitable for cryptographically strong verification. Malicious parties could provide files that appear to be identical to another file because of a collision, i.e., the weak cryptographic hashes of the intended file and a substituted malicious file could match. Metalink Generators and Processors MUST support "sha-256", which is SHA-256, as specified in , and MAY support stronger hashes. If a Metalink Document contains hashes, it SHOULD include "sha-256", which is SHA-256, or stronger. It MAY also include other hashes from the IANA registry named "Hash Function Textual Names".

Metalinks SHOULD include digital signatures, as described in Section 4.2.13. Digital signatures provide authentication, message integrity, and enable non-repudiation with proof of origin.

Harvard University

1350 Mass. Ave. Cambridge MA 02138 - +1 617 495 3864 sob@harvard.edu

General keyword In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. Authors who follow these guidelines should incorporate this phrase near the beginning of their document: The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119. Note that the force of these words is modified by the requirement level of the document in which they are used. This document describes how the OpenPGP Message Format can be used to provide privacy and authentication using the Multipurpose Internet Mail Extensions (MIME) security content types described in RFC 1847. [STANDARDS-TRACK] This memo profiles the X.509 v3 certificate and X.509 v2 certificate revocation list (CRL) for use in the Internet. An overview of this approach and model is provided as an introduction. The X.509 v3 certificate format is described in detail, with additional information regarding the format and semantics of Internet name forms. Standard certificate extensions are described and two Internet-specific extensions are defined. A set of required certificate extensions is specified. The X.509 v2 CRL format is described in detail along with standard and Internet-specific extensions. An algorithm for X.509 certification path validation is described. An ASN.1 module and examples are provided in the appendices. [STANDARDS-TRACK]

mnot@mnot.net

National Institute of Standards and Technology (NIST) This document defines Secure/Multipurpose Internet Mail Extensions (S/MIME) version 3.2. S/MIME provides a consistent way to send and receive secure MIME data. Digital signatures provide authentication, message integrity, and non-repudiation with proof of origin. Encryption provides data confidentiality. Compression can be used to reduce data size. This document obsoletes RFC 3851. [STANDARDS-TRACK] Metalinker Project

anthonybryan@gmail.com http://www.metalinker.org

Metalinker Project

tatsuhiro.t@gmail.com http://aria2.sourceforge.net

Metalinker Project

neil@nabber.org http://www.nabber.org

MirrorBrain

peter@poeml.de http://mirrorbrain.org/~poeml/

International Organization for Standardization New Century Systems

324 East 41st Street #804 New York NY 10017 USA +1 (212) 557-2050 +1 (212) 557-2049 rens@century.com

QUALCOMM Incorporated

6455 Lusk Boulevard San Diego CA 92121 USA sdorner@qualcomm.com

Department of Computer Science

University of Tennessee Knoxville 107 Ayres Hall Knoxville TN 37996-1301 USA +1 (423) 974-5067 +1 (423) 974-8296 moore@cs.utk.edu

Applications MIME internet message multipurpose internet mail extensions This memo provides a mechanism whereby messages conforming to the MIME specifications [RFC 2045, RFC 2046, RFC 2047, RFC 2048, RFC 2049] can convey presentational information. It specifies the "Content-Disposition" header field, which is optional and valid for any MIME entity ("message" or "body part"). Two values for this header field are described in this memo; one for the ordinary linear presentation of the body part, and another to facilitate the use of mail to transfer files. It is expected that more values will be defined in the future, and procedures are defined for extending this set of values. This document is intended as an extension to MIME. As such, the reader is assumed to be familiar with the MIME specifications, and [RFC 822]. The information presented herein supplements but does not replace that found in those documents. This document is a revision to the Experimental protocol defined in RFC 1806. As compared to RFC 1806, this document contains minor editorial updates, adds new parameters needed to support the File Transfer Body Part, and references a separate specification for the handling of non-ASCII and/or very long parameter values. IAB This document provides an overview of possible avenues for denial-of-service (DoS) attack on Internet systems. The aim is to encourage protocol designers and network engineers towards designs that are more robust. We discuss partial solutions that reduce the effectiveness of attacks, and how some solutions might inadvertently open up alternative vulnerabilities. This memo provides information for the Internet community. This document specifies Metalink/HTTP: Mirrors and Cryptographic Hashes in HTTP header fields, a different way to get information that is usually contained in the Metalink XML-based download description format. Metalink/HTTP describes multiple download locations (mirrors), Peer-to-Peer, cryptographic hashes, digital signatures, and other information using existing standards for HTTP header fields. Metalink clients can use this information to make file transfers more robust and reliable. Normative requirements for Metalink/HTTP clients and servers are described here. [STANDARDS-TRACK] RFC 2616 defines the Content-Disposition response header field, but points out that it is not part of the HTTP/1.1 Standard. This specification takes over the definition and registration of Content-Disposition, as used in HTTP, and clarifies internationalization aspects. [STANDARDS-TRACK]

Some text borrowed from our previous RFCs: and . Thanks to the Metalink community, Daniel Stenberg, Micah Cowan, Guiseppe Scrivano, Ilim Ugur, Jack Bates, and Mark Nottingham. This document is dedicated to Zimmy Bryan and Juanita Anthony.

[[ to be removed by the RFC editor before publication as an RFC. ]] Known issues concerning this draft: Intro, term, downloads. Repeat requirements from RFC5854 or add pointer to them? -02 : January 17, 2013. Minor tweaks. -01 : July 16, 2012. Minor tweaks. -00 : July 4, 2012. Initial draft.