Network Working Group F. Le Faucheur Internet-Draft M. Viveganandhan Intended status: Informational K. Leung Expires: January 14, 2013 Cisco July 13, 2012 CDNI Logging Formats for HTTP and HTTP Adaptive Streaming Deliveries draft-lefaucheur-cdni-logging-delivery-01 Abstract The interconnection of Content Distribution Networks (CDNs) is motivated by several use cases. CDN Interconnection can be achieved through four CDNI interfaces, one of which is the CDNI Logging interface. This document discusses CDNI logging formats for content deliveries performed using HTTP or HTTP adaptive streaming. Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119]. Status of this Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at http://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on January 14, 2013. Copyright Notice Copyright (c) 2012 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents Le Faucheur, et al. Expires January 14, 2013 [Page 1] Internet-Draft CDNI Logging for HTTP Delivery July 2012 (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.1. Terminology . . . . . . . . . . . . . . . . . . . . . . . 4 2. CDNI Logging for regular HTTP Delivery . . . . . . . . . . . . 4 2.1. Regular HTTP Log Triggers . . . . . . . . . . . . . . . . 4 2.2. Regular HTTP Log Fields . . . . . . . . . . . . . . . . . 4 3. CDNI Logging for HTTP Adaptive Streaming . . . . . . . . . . . 6 3.1. HAS Chunk-Based Log Triggers . . . . . . . . . . . . . . . 6 3.2. HAS Chunk-Based Log fields . . . . . . . . . . . . . . . . 6 4. Performance Monitoring . . . . . . . . . . . . . . . . . . . . 7 5. CDNI Log Encoding and Transport . . . . . . . . . . . . . . . 7 5.1. CDNI Log Header Information . . . . . . . . . . . . . . . 8 5.2. CDNI per-Log-Record Information . . . . . . . . . . . . . 8 5.3. CDNI Log Footer Information . . . . . . . . . . . . . . . 9 5.4. CDNI Customized Log Format . . . . . . . . . . . . . . . . 9 6. Impact on CDNI Metadata Requirements . . . . . . . . . . . . . 9 7. Impact on CDNI Footprint and Capabilities Advertisement . . . 10 8. Generation of CDNI Logs . . . . . . . . . . . . . . . . . . . 10 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 10 10. Security Considerations . . . . . . . . . . . . . . . . . . . 11 11. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 11 12. References . . . . . . . . . . . . . . . . . . . . . . . . . . 11 12.1. Normative References . . . . . . . . . . . . . . . . . . . 11 12.2. Informative References . . . . . . . . . . . . . . . . . . 11 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 12 Le Faucheur, et al. Expires January 14, 2013 [Page 2] Internet-Draft CDNI Logging for HTTP Delivery July 2012 1. Introduction The interconnection of Content Distribution Networks (CDNs) is motivated by several use cases, such as those described in [I-D.ietf-cdni-use-cases]. The overall problem space for CDN Interconnection (CDNI) is described in [I-D.ietf-cdni-problem-statement]. The CDN Interconnection framework is defined in [I-D.ietf-cdni-framework] and the requirements for the CDN Interconnection solution are specified in [I-D.ietf-cdni-requirements]. One of the CDNI interfaces defined in these documents is the CDNI Logging interface whose description is quoted here: "o CDNI Logging interface: This interface allows the Logging system in interconnected CDNs to communicate the relevant activity logs in order to allow log consuming applications to operate in a multi-CDN environments. For example, an upstream CDN may collect delivery logs from a downstream CDN in order to perform consolidated charging of the CSP or for settlement purposes across CDNs. Similarly, an upstream CDN may collect delivery logs from a downstream CDN in order to provide consolidated reporting and monitoring to the CSP." The logging interface is discussed in details in [I-D.bertrand-cdni-logging]. The present document identifies a number of additional considerations regarding logging for content delivery performed using HTTP. [I-D.brandenburg-cdni-has] discusses the interactions of HTTP Adaptive Streaming with CDNI Logging and provides recommendations on how to specifically perform CDNI Logging for delievry performed using HTTP Adaptive Streaming. The present document discusses in more details how these recommendations woudl impact the CDNI Logging interface. Section 2 discusses the aspect of CDNI Logging that are specific to content delivery performed via regular HTTP. Section 3 discusses the aspect of CDNI Logging that are specific to content delivery performed via HTTP adaptive streaming. Section 5 discusses the aspect of CDNI Logging that are generic to all the CDNI Logs for delivery. CDNI Logging for other events than content delivery (e.g. failures, request routing, service monitoring,...) are not discussed in the present document. Le Faucheur, et al. Expires January 14, 2013 [Page 3] Internet-Draft CDNI Logging for HTTP Delivery July 2012 1.1. Terminology This document uses all terminology defined in [I-D.brandenburg-cdni-has]. 2. CDNI Logging for regular HTTP Delivery This section discusses the triggers on which a CDNI log is generated for delivery of content using regular HTTP delivery, as well as the fields contained in the corresponding log. 2.1. Regular HTTP Log Triggers A CDNI Log is generated for a regular HTTP delivery on the following triggers: +-----------------+-------------------------------------------------+ | Event | Description | +-----------------+-------------------------------------------------+ | content Request | Reception and processing of a request for a | | | content | +-----------------+-------------------------------------------------+ 2.2. Regular HTTP Log Fields A CDNI Log for regular HTTP delivery contains the following fields (or a subset therefof as discused in Section 5.4): +------------------+------------+-----------------------------------+ | Field | Descriptio | Examples | | | n | | +------------------+------------+-----------------------------------+ | Current-Time | Time, in | [20/Feb/2012:00:29.510+0200] | | | millisecon | | | | ds, at | | | | which the | | | | request | | | | was | | | | received. | | | Time-to-Serve | Time, in | 952195 | | | microsecon | | | | ds, taken | | | | to | | | | complete | | | | the | | | | request. | | Le Faucheur, et al. Expires January 14, 2013 [Page 4] Internet-Draft CDNI Logging for HTTP Delivery July 2012 | Client-IP | IP address | 203.0.113.2 | | | of the | | | | requesting | | | | client. | | | Action | Squid | TCP_HIT, TCP_MISS, ... | | | action | | | | describing | | | | how the | | | | request | | | | was | | | | treated | | | | locally | | | | descriptio | | | | n(e.g., | | | | cache | | | | hit/miss) | | | | . | | | Status-Returned | HTTP | 200, 404, ... | | | response | | | | code. | | | Bytes-Transferre | Bytes sent | 23567992 | | d | to the | | | | client, | | | | including | | | | the | | | | headers. | | | Method | HTTP | GET | | | request | | | | method. | | | URI | URI of | http://cache3.cdn1.com/movie/ice/ | | | requested | icemovie.mpg | | | content. | | | Content-Type | MIME-Type | video/mpeg | | | from the | | | | reply | | | | header. | | | User-Agent (and | content of | Mozilla/5.0 (X11; Linux x86_64; | | possibly some | the | rv:12.0) Gecko/20100101 | | other HTTP | User-Agent | Firefox/12.0 | | headers) | HTTP | | | | Header | | Le Faucheur, et al. Expires January 14, 2013 [Page 5] Internet-Draft CDNI Logging for HTTP Delivery July 2012 | URI-Signing-Vali | Flag | 0/1 | | dation | indicating | | | | whether | | | | URI | | | | signature | | | | validation | | | | was | | | | perfomed | | +------------------+------------+-----------------------------------+ 3. CDNI Logging for HTTP Adaptive Streaming This section discusses the CDNI logs for delivery of content using HTTP adaptive streaming. In line with the recommendations of [I-D.brandenburg-cdni-has], this document proposes that a log record be generated for delivery of each chunk or manifest file. 3.1. HAS Chunk-Based Log Triggers A chunk-based Log record is generated for HAS delivery on the following triggers: +--------------------+----------------------------------------------+ | Event | Description | +--------------------+----------------------------------------------+ | Chunk/Manifest | Reception of a request for a Segment (or | | Request | Manifest File) | +--------------------+----------------------------------------------+ 3.2. HAS Chunk-Based Log fields A chunk-based Log record for HTTP adaptive streaming may contain all the same fields as a CDNI Log for a regular HTTP request, as well as the following additional fields: +------------+-------------------+----------------------------------+ | Field | Description | Examples | +------------+-------------------+----------------------------------+ | Content | Identifier for | format and scope of unicity are | | Collection | Content | TBD | | ID | Collection | | Le Faucheur, et al. Expires January 14, 2013 [Page 6] Internet-Draft CDNI Logging for HTTP Delivery July 2012 | Session-id | a string | 6141F5795BE774691D234A0465B9667A | | | generated by the | | | | delivering CDN | | | | and unique (to | | | | the delivering | | | | CDN) to identify | | | | the Session. (*) | | +------------+-------------------+----------------------------------+ (*) The Session-ID value to be included in a log record by the delivering CDN is such that: o different per-chunk log records with the same Session-ID value must correspond to the same user session (.i.e delivery of same content to same enduser at a given point in time) o log records for different chunks of the same user session (.i.e delivery of same content to same enduser at a given point in time) should be provided with the same session-ID value. While undesirable, there may be situations where the delivering CDN uses more than one session-ID value for different per-chunk log records of a given session, for example in scenarios of fail-over or load- balancing across multiple Surrogates and where the delivering CDN does not implement mechanism to synchronize session-IDs across Surrogates. 4. Performance Monitoring The CDNI Log fields listed in the previous sections allow monitoring of essential delivery performance indicators across the CDN Mesh. For example, for a regular HTTP delivery, these fields allow tracking of the time taken to serve the requested content, tracking of content delivery failures, tracking of partial deliveries and tracking of cache hit ratios. As another example, for HTTP adaptive streaming, these fields allow tracking of the presentation (and its fluctuation over time) served to the End-User as well as the End-User random content access (e.g. Play/Stop/Seek). Subsequent versions of this document may discuss potential additional log fields for enhanced performance monitoring. 5. CDNI Log Encoding and Transport Details for CDNI Log encoding and transport will be specified in subsequent versions. WE observe that this is expected to allow optional use of common compression techniques (e.g. gzip). However, Le Faucheur, et al. Expires January 14, 2013 [Page 7] Internet-Draft CDNI Logging for HTTP Delivery July 2012 Section 5.1 identifies the information that is to be included in the header of CDNI Logs, Section 5.2 identifies information that is to be attached to every CDNI Log record and Section 5.3 identifies the information that is to be included in the footer of CDNI Logs. Finally, Section 5.4 discusses the notion of customized Logging. 5.1. CDNI Log Header Information The header of CDNI Logs contains the following fields: +----------------+-------------------+------------------------------+ | Field | Description | Examples | +----------------+-------------------+------------------------------+ | Format-Version | Version of the | v1.0 | | | CDNI Log format. | | | Log-Field-List | The list of the | time cs-method cs-uri | | | fields provided | | | | in the log | | | | records | | | Log-ID | Unique identifier | | | | for the CDNI Log | | | | (facilitates | | | | detection of | | | | duplicate Logs | | | | and tracking in | | | | case of | | | | aggregation). | | | Log-Timestamp | Time, in | [20/Feb/2012:00:29.510+0200] | | | milliseconds, the | | | | CDNI Log was | | | | generated. | | +----------------+-------------------+------------------------------+ 5.2. CDNI per-Log-Record Information In addition to the log fields discussed in previous sections, each CDNI log record contains the following fields: +----------+---------------------------------------------+----------+ | Field | Description | Examples | +----------+---------------------------------------------+----------+ | Log | Digest of the Log records (facilitates | | | Record | recovery of uncorrupted Log records inside | | | Digest | a corrupted CDNI Log) | | +----------+---------------------------------------------+----------+ Le Faucheur, et al. Expires January 14, 2013 [Page 8] Internet-Draft CDNI Logging for HTTP Delivery July 2012 5.3. CDNI Log Footer Information The footer of CDNI Logs contains the following fields: +---------+----------------------------------------------+----------+ | Field | Description | Examples | +---------+----------------------------------------------+----------+ | Log | Digest of the complete Log (facilitates | | | Digest | detection of Log corruption) | | +---------+----------------------------------------------+----------+ 5.4. CDNI Customized Log Format This document proposes that customized logs be supported by CDNI in the following manner: o the uCDN uses the CDNI Metadata interface to indicate to the dCDN which subset of the CDNI logging fields are to be provided in a log record for corresponding to a request for a given content o the dCDN provides, via the CDNI Logging interface, log records containing the subset of CDNI logging fields requested by the uCDN. o The dCDN explicitly lists in the CDNI Log Header the fields actually provided (as discussed in Section 5.1). 6. Impact on CDNI Metadata Requirements We request that the following requirements be added in section 6 of [I-D.ietf-cdni-requirements]: " META-X [HIGH] The CDNI Metadata Distribution interface shall support an OPTIONAL mechanism allowing the Upstream CDN to indicate to the Downstream CDN which CDNI Log fields are to be provided for all, for specific sets of, or for specific content items delivered using HTTP. A CDNI implementation that does not support this optional CDNI Metadata Distribution Interface mechanism MUST ignore this log format indication and generate CDNI logging format for adaptive streaming using the default set of CDNI Logging fields. [Editor's note: A default set of logging fields need to be defined] META-X [MID] The CDNI Metadata Distribution interface shall allow the uCDN to signal to the dCDN the Content Collection ID value for all, for specific sets of, or for specific content items delivered using Le Faucheur, et al. Expires January 14, 2013 [Page 9] Internet-Draft CDNI Logging for HTTP Delivery July 2012 HTTP. Whenever the dCDN is instructed by the uCDN (using the customized logging mechanism described in Section 5.4) to report the Content Collection ID field in the log records, the dCDN is to use the value provided through the CDNI Metadata interface for the corresponding content. " 7. Impact on CDNI Footprint and Capabilities Advertisement We request that the following requirement be added in section 5 of [I-D.ietf-cdni-requirements]: " REQ-X [MID] The CDNI Request Routing/Footprint and Advertisement Interface shall support advertisement of the following capabilities: o support for customized CDNI Logging o support of Content Collection ID logging o support for Session-ID logging " 8. Generation of CDNI Logs Like other CDNI interfaces, the CDNI Logging interface specifies operations across CDNs and not inside a CDN. Therefore, the log formats specified in this document apply to CDNI logging information exchanged across CDNs and does not constrain the process for generating such inter-CDN logs within a given CDN. The format of the logs generated by a given CDN Surrogate is beyond the scope of the present document. We observe that a given CDN could elect to have its Surrogates natively generate logs in the same format as the one to be used for exchange with another CDN, or that the CDN could elect to have its Surrogates generate logs in any other format (as long as they include the necessary information) and have these logs then reformatted prior to exchange with another CDN. 9. IANA Considerations [This will be specified in subsequent versions]. Le Faucheur, et al. Expires January 14, 2013 [Page 10] Internet-Draft CDNI Logging for HTTP Delivery July 2012 10. Security Considerations CDNI Logs exchanged over the CDNI Logging interface can be consumed by very sensitive applications including inter-CDN accounting and billing. The associated security concerns are discussed in [I-D.ietf-cdni-framework]. At this stage, we observe that the CDNI Logging interface can leverage the existing security mechanisms supported by the underlying transport protocol that will be selected for transport of CDNI Logs (e.g. to support authentication of the entities exchanging CDNI Logs through the CDNI Logging interface, to support privacy and protection against spoofing). This will be further discussed in subsequent versions of this document. 11. Acknowledgements The authors want to thank Gene Halbrooks for his input into this document. 12. References 12.1. Normative References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. 12.2. Informative References [I-D.bertrand-cdni-logging] Bertrand, G. and S. Emile, "CDNI Logging Interface", draft-bertrand-cdni-logging-00 (work in progress), February 2012. [I-D.brandenburg-cdni-has] Brandenburg, R., Deventer, O., Faucheur, F., and K. Leung, "Models for adaptive-streaming-aware CDN Interconnection", draft-brandenburg-cdni-has-03 (work in progress), July 2012. [I-D.ietf-cdni-framework] Peterson, L. and B. Davie, "Framework for CDN Interconnection", draft-ietf-cdni-framework-00 (work in progress), April 2012. [I-D.ietf-cdni-problem-statement] Niven-Jenkins, B., Faucheur, F., and N. Bitar, "Content Distribution Network Interconnection (CDNI) Problem Le Faucheur, et al. Expires January 14, 2013 [Page 11] Internet-Draft CDNI Logging for HTTP Delivery July 2012 Statement", draft-ietf-cdni-problem-statement-08 (work in progress), June 2012. [I-D.ietf-cdni-requirements] Leung, K. and Y. Lee, "Content Distribution Network Interconnection (CDNI) Requirements", draft-ietf-cdni-requirements-03 (work in progress), June 2012. [I-D.ietf-cdni-use-cases] Bertrand, G., Emile, S., Burbridge, T., Eardley, P., Ma, K., and G. Watson, "Use Cases for Content Delivery Network Interconnection", draft-ietf-cdni-use-cases-09 (work in progress), July 2012. Authors' Addresses Francois Le Faucheur Cisco Systems Greenside, 400 Avenue de Roumanille Sophia Antipolis 06410 France Phone: +33 4 97 23 26 19 Email: flefauch@cisco.com Mahesh Viveganandhan Cisco Systems 375 East Tasman Drive San Jose 95134 USA Email: mvittal@cisco.com Kent Leung Cisco Systems 3625 Cisco Way San Jose 95134 USA Phone: +1 408 526 5030 Email: kleung@cisco.com Le Faucheur, et al. Expires January 14, 2013 [Page 12]