idnits 2.17.1 draft-ietf-cdni-logging-16.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (March 9, 2015) is 3335 days in the past. Is this intentional? -- Found something which looks like a code comment -- if you have code sections in the document, please surround them with '' and '' lines. Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Unused Reference: 'RFC5288' is defined on line 2261, but no explicit reference was found in the text ** Obsolete normative reference: RFC 5226 (Obsoleted by RFC 8126) ** Obsolete normative reference: RFC 7230 (Obsoleted by RFC 9110, RFC 9112) ** Obsolete normative reference: RFC 7231 (Obsoleted by RFC 9110) ** Obsolete normative reference: RFC 7232 (Obsoleted by RFC 9110) ** Obsolete normative reference: RFC 7233 (Obsoleted by RFC 9110) ** Obsolete normative reference: RFC 7234 (Obsoleted by RFC 9111) ** Obsolete normative reference: RFC 7235 (Obsoleted by RFC 9110) == Outdated reference: A later version (-21) exists of draft-ietf-cdni-metadata-09 -- Obsolete informational reference (is this intentional?): RFC 2818 (Obsoleted by RFC 9110) Summary: 7 errors (**), 0 flaws (~~), 3 warnings (==), 3 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Internet Engineering Task Force F. Le Faucheur, Ed. 3 Internet-Draft Cisco Systems 4 Intended status: Standards Track G. Bertrand, Ed. 5 Expires: September 10, 2015 I. Oprescu, Ed. 6 Orange 7 R. Peterkofsky 8 Skytide, Inc. 9 March 9, 2015 11 CDNI Logging Interface 12 draft-ietf-cdni-logging-16 14 Abstract 16 This memo specifies the Logging interface between a downstream CDN 17 (dCDN) and an upstream CDN (uCDN) that are interconnected as per the 18 CDN Interconnection (CDNI) framework. First, it describes a 19 reference model for CDNI logging. Then, it specifies the CDNI 20 Logging File format and the actual protocol for exchange of CDNI 21 Logging Files. 23 Status of This Memo 25 This Internet-Draft is submitted in full conformance with the 26 provisions of BCP 78 and BCP 79. 28 Internet-Drafts are working documents of the Internet Engineering 29 Task Force (IETF). Note that other groups may also distribute 30 working documents as Internet-Drafts. The list of current Internet- 31 Drafts is at http://datatracker.ietf.org/drafts/current/. 33 Internet-Drafts are draft documents valid for a maximum of six months 34 and may be updated, replaced, or obsoleted by other documents at any 35 time. It is inappropriate to use Internet-Drafts as reference 36 material or to cite them other than as "work in progress." 38 This Internet-Draft will expire on September 10, 2015. 40 Copyright Notice 42 Copyright (c) 2015 IETF Trust and the persons identified as the 43 document authors. All rights reserved. 45 This document is subject to BCP 78 and the IETF Trust's Legal 46 Provisions Relating to IETF Documents 47 (http://trustee.ietf.org/license-info) in effect on the date of 48 publication of this document. Please review these documents 49 carefully, as they describe your rights and restrictions with respect 50 to this document. Code Components extracted from this document must 51 include Simplified BSD License text as described in Section 4.e of 52 the Trust Legal Provisions and are provided without warranty as 53 described in the Simplified BSD License. 55 Table of Contents 57 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 58 1.1. Terminology . . . . . . . . . . . . . . . . . . . . . . . 3 59 1.2. Requirements Language . . . . . . . . . . . . . . . . . . 4 60 2. CDNI Logging Reference Model . . . . . . . . . . . . . . . . 5 61 2.1. CDNI Logging interactions . . . . . . . . . . . . . . . . 5 62 2.2. Overall Logging Chain . . . . . . . . . . . . . . . . . . 8 63 2.2.1. Logging Generation and During-Generation Aggregation 9 64 2.2.2. Logging Collection . . . . . . . . . . . . . . . . . 10 65 2.2.3. Logging Filtering . . . . . . . . . . . . . . . . . . 10 66 2.2.4. Logging Rectification and Post-Generation Aggregation 11 67 2.2.5. Log-Consuming Applications . . . . . . . . . . . . . 12 68 2.2.5.1. Maintenance/Debugging . . . . . . . . . . . . . . 12 69 2.2.5.2. Accounting . . . . . . . . . . . . . . . . . . . 12 70 2.2.5.3. Analytics and Reporting . . . . . . . . . . . . . 12 71 2.2.5.4. Security . . . . . . . . . . . . . . . . . . . . 13 72 2.2.5.5. Legal Logging Duties . . . . . . . . . . . . . . 13 73 2.2.5.6. Notions common to multiple Log Consuming 74 Applications . . . . . . . . . . . . . . . . . . 13 75 3. CDNI Logging File . . . . . . . . . . . . . . . . . . . . . . 15 76 3.1. Rules . . . . . . . . . . . . . . . . . . . . . . . . . . 15 77 3.2. CDNI Logging File Structure . . . . . . . . . . . . . . . 16 78 3.3. CDNI Logging Directives . . . . . . . . . . . . . . . . . 19 79 3.4. CDNI Logging Records . . . . . . . . . . . . . . . . . . 23 80 3.4.1. HTTP Request Logging Record . . . . . . . . . . . . . 24 81 3.5. CDNI Logging File Example . . . . . . . . . . . . . . . . 33 82 3.6. Cascaded CDNI Logging Files Example . . . . . . . . . . . 34 83 4. Protocol for Exchange of CDNI Logging File After Full 84 Collection . . . . . . . . . . . . . . . . . . . . . . . . . 37 85 4.1. CDNI Logging Feed . . . . . . . . . . . . . . . . . . . . 38 86 4.1.1. Atom Formatting . . . . . . . . . . . . . . . . . . . 38 87 4.1.2. Updates to Log Files and the Feed . . . . . . . . . . 38 88 4.1.3. Redundant Feeds . . . . . . . . . . . . . . . . . . . 39 89 4.1.4. Example CDNI Logging Feed . . . . . . . . . . . . . . 39 90 4.2. CDNI Logging File Pull . . . . . . . . . . . . . . . . . 41 91 5. Protocol for Exchange of CDNI Logging File During Collection 42 92 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 43 93 6.1. CDNI Logging Directive Names Registry . . . . . . . . . . 43 94 6.2. CDNI Logging File Version Registry . . . . . . . . . . . 43 95 6.3. CDNI Logging Record-Types Registry . . . . . . . . . . . 44 96 6.4. CDNI Logging Field Names Registry . . . . . . . . . . . . 45 97 6.5. CDNI Logging MIME Media Type . . . . . . . . . . . . . . 46 98 7. Security Considerations . . . . . . . . . . . . . . . . . . . 46 99 7.1. Authentication, Authorization, Confidentiality, Integrity 100 Protection . . . . . . . . . . . . . . . . . . . . . . . 47 101 7.2. Denial of Service . . . . . . . . . . . . . . . . . . . . 48 102 7.3. Privacy . . . . . . . . . . . . . . . . . . . . . . . . . 48 103 8. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 49 104 9. References . . . . . . . . . . . . . . . . . . . . . . . . . 49 105 9.1. Normative References . . . . . . . . . . . . . . . . . . 49 106 9.2. Informative References . . . . . . . . . . . . . . . . . 50 107 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 52 109 1. Introduction 111 This memo specifies the CDNI Logging interface between a downstream 112 CDN (dCDN) and an upstream CDN (uCDN). First, it describes a 113 reference model for CDNI logging. Then, it specifies the CDNI 114 Logging File format and the actual protocol for exchange of CDNI 115 Logging Files. 117 The reader should be familiar with the following documents: 119 o CDNI problem statement [RFC6707] and framework [RFC7336] identify 120 a Logging interface, 122 o Section 8 of [RFC7337] specifies a set of requirements for 123 Logging, 125 o [RFC6770] outlines real world use-cases for interconnecting CDNs. 126 These use cases require the exchange of Logging information 127 between the dCDN and the uCDN. 129 As stated in [RFC6707], "the CDNI Logging interface enables details 130 of logs or events to be exchanged between interconnected CDNs". 132 The present document describes: 134 o The CDNI Logging reference model (Section 2), 136 o The CDNI Logging File format (Section 3), 138 o The CDNI Logging File Exchange protocol (Section 4). 140 1.1. Terminology 142 In this document, the first letter of each CDNI-specific term is 143 capitalized. We adopt the terminology described in [RFC6707] and 144 [RFC7336], and extend it with the additional terms defined below. 146 Intra-CDN Logging information: logging information generated and 147 collected within a CDN. The format of the Intra-CDN Logging 148 information may be different to the format of the CDNI Logging 149 information. 151 CDNI Logging information: logging information exchanged across CDNs 152 using the CDNI Logging Interface. 154 Logging information: logging information generated and collected 155 within a CDN or obtained from another CDN using the CDNI Logging 156 Interface. 158 CDNI Logging Field: an atomic element of information that can be 159 included in a CDNI Logging Record. The time an event/task started, 160 the IP address of an End User to whom content was delivered, and the 161 Uniform Resource Identifier (URI) of the content delivered, are 162 examples of CDNI Logging Fields. 164 CDNI Logging Record: an information record providing information 165 about a specific event. This comprises a collection of CDNI Logging 166 Fields. 168 CDNI Logging File: a file containing CDNI Logging Records, as well as 169 additional information facilitating the processing of the CDNI 170 Logging Records. 172 CDN Reporting: the process of providing the relevant information that 173 will be used to create a formatted content delivery report provided 174 to the CSP in deferred time. Such information typically includes 175 aggregated data that can cover a large period of time (e.g., from 176 hours to several months). Uses of Reporting include the collection 177 of charging data related to CDN services and the computation of Key 178 Performance Indicators (KPIs). 180 CDN Monitoring: the process of providing or displaying content 181 delivery information in a timely fashion with respect to the 182 corresponding deliveries. Monitoring typically includes visibility 183 of the deliveries in progress for service operation purposes. It 184 presents a view of the global health of the services as well as 185 information on usage and performance, for network services 186 supervision and operation management. In particular, monitoring data 187 can be used to generate alarms. 189 1.2. Requirements Language 191 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 192 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 193 document are to be interpreted as described in RFC 2119 [RFC2119]. 195 2. CDNI Logging Reference Model 197 2.1. CDNI Logging interactions 199 The CDNI logging reference model between a given uCDN and a given 200 dCDN involves the following interactions: 202 o customization by the uCDN of the CDNI Logging information to be 203 provided by the dCDN to the uCDN (e.g., control of which CDNI 204 Logging Fields are to be communicated to the uCDN for a given task 205 performed by the dCDN or control of which types of events are to 206 be logged). The dCDN takes into account this CDNI Logging 207 customization information to determine what Logging information to 208 provide to the uCDN, but it may, or may not, take into account 209 this CDNI Logging customization information to influence what CDN 210 logging information is to be generated and collected within the 211 dCDN (e.g., even if the uCDN requests a restricted subset of the 212 logging information, the dCDN may elect to generate a broader set 213 of logging information). The mechanism to support the 214 customization by the uCDN of CDNI Logging information is outside 215 the scope of this document and left for further study. Until such 216 a mechanism is available, the uCDN and dCDN are expected to agree 217 off-line on what exact set of CDNI Logging information is to be 218 provided by the dCDN to the uCDN, and to rely on management plane 219 actions to configure the CDNI Logging functions in the dCDN to 220 generate this information set and in the uCDN to expect this 221 information set. 223 o generation and collection by the dCDN of the intra-CDN Logging 224 information related to the completion of any task performed by the 225 dCDN on behalf of the uCDN (e.g., delivery of the content to an 226 End User) or related to events happening in the dCDN that are 227 relevant to the uCDN (e.g., failures or unavailability in dCDN). 228 This takes place within the dCDN and does not directly involve 229 CDNI interfaces. 231 o communication by the dCDN to the uCDN of the Logging information 232 collected by the dCDN relevant to the uCDN. This is supported by 233 the CDNI Logging interface and in the scope of the present 234 document. For example, the uCDN may use this Logging information 235 to charge the CSP, to perform analytics and monitoring for 236 operational reasons, to provide analytics and monitoring views on 237 its content delivery to the CSP or to perform trouble-shooting. 238 This document exclusively specifies non-real-time exchange of 239 Logging information. Closer to real-time exchange of Logging 240 information (say sub-minute or sub-second) is outside the scope of 241 the present document and left for further study. This document 242 exclusively specifies exchange of Logging information related to 243 content delivery. Exchange of Logging information related to 244 operational events (e.g., dCDN request routing function 245 unavailable, content acquisition failure by dCDN) for audit or 246 operational reactive adjustments by uCDN is outside the scope of 247 the present document and left for further study. 249 o customization by the dCDN of the CDNI Logging information to be 250 provided by the uCDN on behalf of the dCDN. The mechanism to 251 support the customization by the dCDN of CDNI Logging information 252 is outside the scope of this document and left for further study. 254 o generation and collection by the uCDN of Intra-CDN Logging 255 information related to the completion of any task performed by the 256 uCDN on behalf of the dCDN (e.g., serving of content by uCDN to 257 dCDN for acquisition purposes by dCDN) or related to events 258 happening in the uCDN that are relevant to the dCDN. This takes 259 place within the uCDN and does not directly involve CDNI 260 interfaces. 262 o communication by the uCDN to the dCDN of the Logging information 263 collected by the uCDN relevant to the dCDN. For example, the dCDN 264 might potentially benefit from this information for security 265 auditing or content acquisition troubleshooting. This is outside 266 the scope of this document and left for further study. 268 Figure 1 provides an example of CDNI Logging interactions (focusing 269 only on the interactions that are in the scope of this document) in a 270 particular scenario where four CDNs are involved in the delivery of 271 content from a given CSP: the uCDN has a CDNI interconnection with 272 dCDN-1 and dCDN-2. In turn, dCDN-2 has a CDNI interconnection with 273 dCDN-3, where dCDN-2 is acting as an upstream CDN relative to dCDN-3. 274 In this example, uCDN, dCDN-1, dCDN-2 and dCDN-3 all participate in 275 the delivery of content for the CSP. In this example, the CDNI 276 Logging interface enables the uCDN to obtain Logging information from 277 all the dCDNs involved in the delivery. In the example, the uCDN 278 uses the Logging information: 280 o to analyze the performance of the delivery performed by the dCDNs 281 and to adjust its operations after the fact (e.g., request 282 routing) as appropriate, 284 o to provide (non-real-time) reporting and monitoring information to 285 the CSP. 287 For instance, the uCDN merges Logging information, extracts relevant 288 KPIs, and presents a formatted report to the CSP, in addition to a 289 bill for the content delivered by uCDN itself or by its dCDNs on the 290 CSP's behalf. The uCDN may also provide Logging information as raw 291 log files to the CSP, so that the CSP can use its own logging 292 analysis tools. 294 +-----+ 295 | CSP | 296 +-----+ 297 ^ Reporting and monitoring data 298 * Billing 299 ,--*--. 300 Logging ,-' `-. 301 Data =>( uCDN )<= Logging 302 // `-. _,-' \\ Data 303 || `-'-'-' || 304 ,-----. ,-----. 305 ,-' `-. ,-' `-. 306 ( dCDN-1 ) ( dCDN-2 )<== Logging 307 `-. ,-' `-. _,-' \\ Data 308 `--'--' `--'-' || 309 ,-----. 310 ,' `-. 311 ( dCDN-3 ) 312 `. ,-' 313 `--'--' 315 ===> CDNI Logging Interface 316 ***> outside the scope of CDNI 318 Figure 1: Interactions in CDNI Logging Reference Model 320 A downstream CDN relative to uCDN (e.g., dCDN-2) integrates the 321 relevant Logging information obtained from its own downstream CDNs 322 (i.e., dCDN-3) in the Logging information that it provides to the 323 uCDN, so that the uCDN ultimately obtains all Logging information 324 relevant to a CSP for which it acts as the authoritative CDN. Such 325 aggregation is further discussed in Section 3.6. 327 Note that the format of Logging information that a CDN provides over 328 the CDNI interface might be different from the one that the CDN uses 329 internally. In this case, the CDN needs to reformat the Logging 330 information before it provides this information to the other CDN over 331 the CDNI Logging interface. Similarly, a CDN might reformat the 332 Logging information that it receives over the CDNI Logging interface 333 before injecting it into its log-consuming applications or before 334 providing some of this Logging information to the CSP. Such 335 reformatting operations introduce latency in the logging distribution 336 chain and introduce a processing burden. Therefore, there are 337 benefits in specifying CDNI Logging formats that are suitable for use 338 inside CDNs and also are close to the intra-CDN Logging formats 339 commonly used in CDNs today. 341 2.2. Overall Logging Chain 343 This section discusses the overall logging chain within and across 344 CDNs to clarify how CDN Logging information is expected to fit in 345 this overall chain. Figure 2 illustrates the overall logging chain 346 within the dCDN, across CDNs using the CDNI Logging interface and 347 within the uCDN. Note that the logging chain illustrated in the 348 Figure is obviously only an example and varies depending on the 349 specific environments. For example, there may be more or fewer 350 instantiations of each entity (e.g., there may be 4 Log consuming 351 applications in a given CDN). As another example, there may be one 352 instance of Rectification process per Log Consuming Application 353 instead of a shared one. 355 Log Consuming Log Consuming 356 App App 357 ^ ^ 358 | | 359 Rectification---------- 360 ^ 361 | 362 Filtering 363 ^ 364 | 365 Collection 366 ^ ^ 367 | | 368 | Generation 369 | 370 | uCDN 371 CDNI Logging --------------------------------------------------- 372 exchange dCDN 373 ^ 374 | Log Consuming Log Consuming 375 | App App 376 | ^ ^ 377 | | | 378 Rectification Rectification--------- 379 ^ ^ 380 | | 381 Filtering 382 ^ 383 | 384 Collection 385 ^ ^ 386 | | 387 Generation Generation 389 Figure 2: CDNI Logging in the overall Logging Chain 391 The following subsections describe each of the processes potentially 392 involved in the logging chain of Figure 2. 394 2.2.1. Logging Generation and During-Generation Aggregation 396 CDNs typically generate Logging information for all significant task 397 completions, events, and failures. Logging information is typically 398 generated by many devices in the CDN including the surrogates, the 399 request routing system, and the control system. 401 The amount of Logging information generated can be huge. Therefore, 402 during contract negotiations, interconnected CDNs often agree on a 403 retention duration for Logging information, and/or potentially on a 404 maximum volume of Logging information that the dCDN ought to keep. 405 If this volume is exceeded, the dCDN is expected to alert the uCDN 406 but may not keep more Logging information for the considered time 407 period. In addition, CDNs may aggregate Logging information and 408 transmit only summaries for some categories of operations instead of 409 the full Logging information. Note that such aggregation leads to an 410 information loss, which may be problematic for some usages of the 411 Logging information (e.g., debugging). 413 [RFC6983] discusses logging for HTTP Adaptive Streaming (HAS). In 414 accordance with the recommendations articulated there, it is expected 415 that a surrogate will generate separate Logging information for 416 delivery of each chunk of HAS content. This ensures that separate 417 Logging information can then be provided to interconnected CDNs over 418 the CDNI Logging interface. Still in line with the recommendations 419 of [RFC6983], the Logging information for per-chunck delivery may 420 include some information (a Content Collection IDentifier and a 421 Session IDentifier) intended to facilitate subsequent post-generation 422 aggregation of per-chunk logs into per-session logs. Note that a CDN 423 may also elect to generate aggregate per-session logs when performing 424 HAS delivery, but this needs to be in addition to, and not instead 425 of, the per-chunk delivery logs. We note that aggregate per-session 426 logs for HAS delivery are for further study and outside the scope of 427 this document. 429 2.2.2. Logging Collection 431 This is the process that continuously collects Logging information 432 generated by the log-generating entities within a CDN. 434 In a CDNI environment, in addition to collecting Logging information 435 from log-generating entities within the local CDN, the Collection 436 process also collects Logging information provided by another CDN, or 437 other CDNs, through the CDNI Logging interface. This is illustrated 438 in Figure 2 where we see that the Collection process of the uCDN 439 collects Logging information from log-generating entities within the 440 uCDN as well as Logging information coming from the dCDNs through the 441 CDNI Logging interface. 443 2.2.3. Logging Filtering 445 A CDN may be required to only present different subsets of the whole 446 Logging information collected to various log-consuming applications. 447 This is achieved by the Filtering process. 449 In particular, the Filtering process can also filter the right subset 450 of Logging information that needs to be provided to a given 451 interconnected CDN. For example, the filtering process in the dCDN 452 can be used to ensure that only the Logging information related to 453 tasks performed on behalf of a given uCDN are made available to that 454 uCDN (thereby filtering out all the Logging information related to 455 deliveries by the dCDN of content for its own CSPs). Similarly, the 456 Filtering process may filter or partially mask some fields, for 457 example, to protect End Users' privacy when communicating CDNI 458 Logging information to another CDN. Filtering of Logging information 459 prior to communication of this information to other CDNs via the CDNI 460 Logging interface requires that the downstream CDN can recognize the 461 subset of Logging information that relate to each interconnected CDN. 463 The CDN will also filter some internal scope information such as 464 information related to its internal alarms (security, failures, load, 465 etc). 467 In some use cases described in [RFC6770], the interconnected CDNs do 468 not want to disclose details on their internal topology. The 469 filtering process can then also filter confidential data on the 470 dCDNs' topology (number of servers, location, etc.). In particular, 471 information about the requests served by each Surrogate may be 472 confidential. Therefore, the Logging information needs to be 473 protected so that data such as Surrogates' hostnames are not 474 disclosed to the uCDN. In the "Inter-Affiliates Interconnection" use 475 case, this information may be disclosed to the uCDN because both the 476 dCDN and the uCDN are operated by entities of the same group. 478 2.2.4. Logging Rectification and Post-Generation Aggregation 480 If Logging information is generated periodically, it is important 481 that the sessions that start in one Logging period and end in another 482 are correctly reported. If they are reported in the starting period, 483 then the Logging information of this period will be available only 484 after the end of the session, which delays the Logging information 485 generation. A simple approach is to provide the complete Logging 486 Record for a session in the Logging Period of the session end. 488 A Logging rectification/update mechanism could be useful to reach a 489 good trade-off between the Logging information generation delay and 490 the Logging information accuracy. 492 In the presence of HAS, some log-consuming applications can benefit 493 from aggregate per-session logs. For example, for analytics, per- 494 session logs allow display of session-related trends which are much 495 more meaningful for some types of analysis than chunk-related trends. 496 In the case where aggregate logs have been generated directly by the 497 log-generating entities, those can be used by the applications. In 498 the case where aggregate logs have not been generated, the 499 Rectification process can be extended with a Post-Generation 500 Aggregation process that generates per-session logs from the per- 501 chunk logs, possibly leveraging the information included in the per- 502 chunk logs for that purpose (Content Collection IDentifier and a 503 Session IDentifier). However, in accordance with [RFC6983], this 504 document does not define exchange of such aggregate logs on the CDNI 505 Logging interface. We note that this is for further study and 506 outside the scope of this document. 508 2.2.5. Log-Consuming Applications 510 2.2.5.1. Maintenance/Debugging 512 Logging information is useful to permit the detection (and limit the 513 risk) of content delivery failures. In particular, Logging 514 information facilitates the detection of configuration issues. 516 To detect faults, Logging information needs to report success and 517 failure of CDN delivery operations. The uCDN can summarize such 518 information into KPIs. For instance, Logging information needs to 519 allow the computation of the number of times, during a given time 520 period, that content delivery related to a specific service succeeds/ 521 fails. 523 Logging information enables the CDN providers to identify and 524 troubleshoot performance degradations. In particular, Logging 525 information enables tracking of traffic data (e.g., the amount of 526 traffic that has been forwarded by a dCDN on behalf of an uCDN over a 527 given period of time), which is particularly useful for CDN and 528 network planning operations. 530 2.2.5.2. Accounting 532 Logging information is essential for accounting, to permit inter-CDN 533 billing and CSP billing by uCDNs. For instance, Logging information 534 provided by dCDNs enables the uCDN to compute the total amount of 535 traffic delivered by every dCDN for a particular Content Provider, as 536 well as, the associated bandwidth usage (e.g., peak, 95th 537 percentile), and the maximum number of simultaneous sessions over a 538 given period of time. 540 2.2.5.3. Analytics and Reporting 542 The goal of analytics is to gather any relevant information to track 543 audience, analyze user behavior, and monitor the performance and 544 quality of content delivery. For instance, Logging information 545 enables the CDN providers to report on content consumption (e.g., 546 delivered sessions per content) in a specific geographic area. 548 The goal of reporting is to gather any relevant information to 549 monitor the performance and quality of content delivery and allow 550 detection of delivery issues. For instance, reporting could track 551 the average delivery throughput experienced by End Users in a given 552 region for a specific CSP or content set over a period of time. 554 2.2.5.4. Security 556 The goal of security is to prevent and monitor unauthorized access, 557 misuse, modification, and denial of access of a service. A set of 558 information is logged for security purposes. In particular, a record 559 of access to content is usually collected to permit the CSP to detect 560 infringements of content delivery policies and other abnormal End 561 User behaviors. 563 2.2.5.5. Legal Logging Duties 565 Depending on the country considered, the CDNs may have to retain 566 specific Logging information during a legal retention period, to 567 comply with judicial requisitions. 569 2.2.5.6. Notions common to multiple Log Consuming Applications 571 2.2.5.6.1. Logging Information Views 573 Within a given log-consuming application, different views may be 574 provided to different users depending on privacy, business, and 575 scalability constraints. 577 For example, an analytics tool run by the uCDN can provide one view 578 to an uCDN operator that exploits all the Logging information 579 available to the uCDN, while the tool may provide a different view to 580 each CSP exploiting only the Logging information related to the 581 content of the given CSP. 583 As another example, maintenance and debugging tools may provide 584 different views to different CDN operators, based on their 585 operational role. 587 2.2.5.6.2. Key Performance Indicators (KPIs) 589 This section presents, for explanatory purposes, a non-exhaustive 590 list of Key Performance Indicators (KPIs) that can be extracted/ 591 produced from logs. 593 Multiple log-consuming applications, such as analytics, monitoring, 594 and maintenance applications, often compute and track such KPIs. 596 In a CDNI environment, depending on the situation, these KPIs may be 597 computed by the uCDN or by the dCDN. But it is usually the uCDN that 598 computes KPIs, because the uCDN and dCDN may have different 599 definitions of the KPIs and the computation of some KPIs requires a 600 vision of all the deliveries performed by the uCDN and all its dCDNs. 602 Here is a list of important examples of KPIs: 604 o Number of delivery requests received from End Users in a given 605 region for each piece of content, during a given period of time 606 (e.g., hour/day/week/month) 608 o Percentage of delivery successes/failures among the aforementioned 609 requests 611 o Number of failures listed by failure type (e.g., HTTP error code) 612 for requests received from End Users in a given region and for 613 each piece of content, during a given period of time (e.g., 614 hour/day/week/month) 616 o Number and cause of premature delivery termination for End Users 617 in a given region and for each piece of content, during a given 618 period of time (e.g., hour/day/week/month) 620 o Maximum and mean number of simultaneous sessions established by 621 End Users in a given region, for a given Content Provider, and 622 during a given period of time (e.g., hour/day/week/month) 624 o Volume of traffic delivered for sessions established by End Users 625 in a given region, for a given Content Provider, and during a 626 given period of time (e.g., hour/day/week/month) 628 o Maximum, mean, and minimum delivery throughput for sessions 629 established by End Users in a given region, for a given Content 630 Provider, and during a given period of time (e.g., hour/day/week/ 631 month) 633 o Cache-hit and byte-hit ratios for requests received from End Users 634 in a given region for each piece of content, during a given period 635 of time (e.g., hour/day/week/month) 637 o Top 10 most popularly requested contents (during a given day/week/ 638 month) 640 o Terminal type (mobile, PC, STB, if this information can be 641 acquired from the browser type header, for example). 643 Additional KPIs can be computed from other sources of information 644 than the Logging information, for instance, data collected by a 645 content portal or by specific client-side application programming 646 interfaces. Such KPIs are out of scope for the present document. 648 The KPIs used depend strongly on the considered log-consuming 649 application -- the CDN operator may be interested in different 650 metrics than the CSP is. In particular, CDN operators are often 651 interested in delivery and acquisition performance KPIs, information 652 related to Surrogates' performance, caching information to evaluate 653 the cache-hit ratio, information about the delivered file size to 654 compute the volume of content delivered during peak hour, etc. 656 Some of the KPIs, for instance those providing an instantaneous 657 vision of the active sessions for a given CSP's content, are useful 658 essentially if they are provided in a timely manner. By contrast, 659 some other KPIs, such as those averaged on a long period of time, can 660 be provided in non-real-time. 662 3. CDNI Logging File 664 3.1. Rules 666 This specification uses the Augmented Backus-Naur Form (ABNF) 667 notation and core rules of [RFC5234]. In particular, the present 668 document uses the following rules from [RFC5234]: 670 CR = %x0D ; carriage return 672 ALPHA = %x41-5A / %x61-7A ; A-Z / a-z 674 DIGIT = %x30-39 ; 0-9 676 DQUOTE = %x22 ; " (Double Quote) 678 CRLF = CR LF ; Internet standard newline 680 HEXDIG = DIGIT / "A" / "B" / "C" / "D" / "E" / "F" 682 HTAB = %x09 ; horizontal tab 684 LF = %x0A ; linefeed 686 OCTET = %x00-FF ; 8 bits of data 688 The present document also uses the following rules from [RFC3986]: 690 host = as specified in section 3.2.2 of [RFC3986]. 692 IPv4address = as specified in section 3.2.2 of [RFC3986]. 694 IPv6address = as specified in section 3.2.2 of [RFC3986]. 696 The present document also defines the following additional rules: 698 ADDRESS = IPv4address / IPv6address 700 ALPHANUM = ALPHA / DIGIT 702 DATE = 4DIGIT "-" 2DIGIT "-" 2DIGIT 704 Dates are recorded in the format YYYY-MM-DD where YYYY, MM and 705 DD stand for the numeric year, month and day respectively. All 706 dates are specified in Universal Time Coordinated (UTC). 708 DEC = 1*DIGIT ["." *DIGIT] 710 NAMEFORMAT = ALPHANUM *(ALPHANUM / "_" / "-") 712 QSTRING = DQUOTE *NDQUOTE DQUOTE ; where 714 NDQUOTE = / 2DQUOTE ; whereby a 715 DQUOTE is conveyed inside a QSTRING unambiguously by repeating 716 it. 718 NHTABSTRING = *NHTAB ; where 720 NHTAB = 722 TIME = 2DIGIT ":" 2DIGIT ":" 2DIGIT ["." *DIGIT] 724 Times are recorded in the form HH:MM:SS or HH:MM:SS.S where HH 725 is the hour in 24 hour format, MM is minutes and SS is seconds. 726 All times are specified in Universal Time Coordinated (UTC). 728 3.2. CDNI Logging File Structure 730 As defined in Section 1.1: a CDNI Logging Field is as an atomic 731 logging information element, a CDNI Logging Record is a collection of 732 CDNI Logging Fields containing all logging information corresponding 733 to a single logging event, and a CDNI Logging File contains a 734 collection of CDNI Logging Records. This structure is illustrated in 735 Figure 3. The use of a file structure for transfer of CDNI Logging 736 information is selected since this is the most common practise today 737 for exchange of logging information within and across CDNs. 739 +----------------------------------------------------------+ 740 |CDNI Logging File | 741 | | 742 | #Directive 1 | 743 | #Directive 2 | 744 | ... | 745 | #Directive P | 746 | | 747 | +------------------------------------------------------+ | 748 | |CDNI Logging Record 1 | | 749 | | +-------------+ +-------------+ +-------------+ | | 750 | | |CDNI Logging | |CDNI Logging | ... |CDNI Logging | | | 751 | | | Field 1 | | Field 2 | | Field N | | | 752 | | +-------------+ +-------------+ +-------------+ | | 753 | +------------------------------------------------------+ | 754 | | 755 | +------------------------------------------------------+ | 756 | |CDNI Logging Record 2 | | 757 | | +-------------+ +-------------+ +-------------+ | | 758 | | |CDNI Logging | |CDNI Logging | ... |CDNI Logging | | | 759 | | | Field 1 | | Field 2 | | Field N | | | 760 | | +-------------+ +-------------+ +-------------+ | | 761 | +------------------------------------------------------+ | 762 | | 763 | ... | 764 | | 765 | #Directive P+1 | 766 | | 767 | ... | 768 | | 769 | +------------------------------------------------------+ | 770 | |CDNI Logging Record M | | 771 | | +-------------+ +-------------+ +-------------+ | | 772 | | |CDNI Logging | |CDNI Logging | ... |CDNI Logging | | | 773 | | | Field 1 | | Field 2 | | Field N | | | 774 | | +-------------+ +-------------+ +-------------+ | | 775 | +------------------------------------------------------+ | 776 | | 777 | | 778 | #Directive P+Q | 779 +----------------------------------------------------------+ 781 Figure 3: Structure of Logging Files 783 The CDNI Logging File format is inspired from the W3C Extended Log 784 File Format [ELF]. However, it is fully specified by the present 785 document. Where the present document differs from the W3C Extended 786 Log File Format, an implementation of the CDNI Logging interface MUST 787 comply with the present document. The W3C Extended Log File Format 788 was used as a starting point, reused where possible and expanded when 789 necessary. 791 Using a format that resembles the W3C Extended Log File Format is 792 intended to keep CDNI logging format close to the intra-CDN Logging 793 information format commonly used in CDNs today, thereby minimizing 794 systematic translation at CDN/CDNI boundary. 796 A CDNI Logging File MUST contain a sequence of lines containing US- 797 ASCII characters [CHAR_SET] terminated by CRLF. 799 Each line of a CDNI Logging File MUST contain either a directive or a 800 CDNI Logging Record. 802 Directives record information about the CDNI Logging process itself. 803 Lines containing directives MUST begin with the "#" character. 804 Directives are specified in Section 3.3. 806 Logging Records provide actual details of the logged event. Logging 807 Records are specified in Section 3.4. 809 The CDNI File structure is defined by the following rules: 811 DIRLINE = "#" directive CRLF 813 DIRGROUP = 1*DIRLINE 815 RECLINE = CRLF 817 RECGROUP = *RECLINE 819 = 1* 821 3.3. CDNI Logging Directives 823 The CDNI Logging directives are defined by the following rules: 825 directive = DIRNAME ":" HTAB DIRVAL 827 DIRNAME = 830 DIRVAL = 834 An implementation of the CDNI Logging interface MUST support all of 835 the following directives, listed below by their directive name: 837 o Version: 839 * format: "CDNI" "/" 1*DIGIT "." 1*DIGIT 841 * directive value: indicates the version of the CDNI Logging File 842 format. The entity transmitting a CDNI Logging File as per the 843 present document MUST set the value to "CDNI/1.0". In the 844 future, other versions of CDNI Logging File might be specified; 845 those would use a value different to "CDNI/1.0" allowing the 846 entity receiving the CDNI Logging File to identify the 847 corresponding version. 849 * occurrence: there MUST be one and only one instance of this 850 directive per CDNI Logging File. It MUST be the first line of 851 the CDNI Logging File. 853 o UUID: 855 * format: NHTABSTRING 857 * directive value: this a Universally Unique IDentifier (UUID) 858 from the UUID Uniform Resource Name (URN) namespace specified 859 in [RFC4122]) for the CDNI Logging File. 861 * occurrence: there MUST be one and only one instance of this 862 directive per CDNI Logging File. 864 o Claimed-Origin: 866 * format: host 868 * directive value: this contains the claimed identification of 869 the entity transmitting the CDNI Logging File (e.g., the host 870 in a dCDN supporting the CDNI Logging interface) or the entity 871 responsible for transmitting the CDNI Logging File (e.g., the 872 dCDN). 874 * occurrence: there MUST be zero or exactly one instance of this 875 directive per CDNI Logging File. This directive MAY be 876 included by the dCDN. It MUST NOT be included or modified by 877 the uCDN. 879 o Established-Origin: 881 * format: host 882 * directive value: this contains the identification, as 883 established by the entity receiving the CDNI Logging File, of 884 the entity transmitting the CDNI Logging File (e.g., the host 885 in a dCDN supporting the CDNI Logging interface) or the entity 886 responsible for transmitting the CDNI Logging File (e.g., the 887 dCDN). 889 * occurrence: there MUST be zero or exactly one instance of this 890 directive per CDNI Logging File. This directive MAY be added 891 by the uCDN (e.g., before storing the CDNI Logging File). It 892 MUST NOT be included by the dCDN. The mechanisms used by the 893 uCDN to establish and validate the entity responsible for the 894 CDNI Logging File is outside the scope of the present document. 895 We observe that, in particular, this may be achieved through 896 authentication mechanisms that are part of the transport layer 897 of the CDNI Logging File pull mechanism (Section 4.2). 899 o Record-Type: 901 * format: NAMEFORMAT 903 * directive value: indicates the type of the CDNI Logging Records 904 that follow this directive, until another Record-Type directive 905 (or the end of the CDNI Logging File). This can be any CDNI 906 Logging Record type registered in the CDNI Logging Record-types 907 registry (Section 6.3). For example this may be 908 "cdni_http_request_v1" as specified in Section 3.4.1. 910 * occurrence: there MUST be at least one instance of this 911 directive per CDNI Logging File. The first instance of this 912 directive MUST precede a Fields directive and MUST precede all 913 CDNI Logging Records. 915 o Fields: 917 * format: FIENAME * ; where FIENAME can take any 918 CDNI Logging field name registered in the CDNI Logging Field 919 Names registry (Section 6.4). 921 * directive value: this lists the names of all the fields for 922 which a value is to appear in the CDNI Logging Records that 923 follow the instance of this directive (until another instance 924 of this directive). The names of the fields, as well as their 925 occurrences, MUST comply with the corresponding rules specified 926 in the document referenced in the CDNI Logging Record-types 927 registry (Section 6.3) for the corresponding CDNI Logging 928 Record-Type. 930 * occurrence: there MUST be at least one instance of this 931 directive per Record-Type directive. The first instance of 932 this directive for a given Record-Type MUST appear before any 933 CDNI Logging Record for this Record-Type. One situation where 934 more than one instance of the Fields directive can appear 935 within a given CDNI Logging File, is when there is a change, in 936 the middle of a fairly large logging period, in the agreement 937 between the uCDN and the dCDN about the set of Fields that are 938 to be exchanged. The multiple occurences allow records with 939 the old set of fields and records with the new set of fields to 940 be carried inside the same Logging File. 942 o Integrity-Hash: 944 * format: 32HEXDIG 946 * directive value: This directive permits the detection of a 947 corrupted CDNI Logging File. This can be useful, for instance, 948 if a problem occurs on the filesystem of the dCDN Logging 949 system and leads to a truncation of a logging file. The valid 950 Integrity-Hash value is included in this directive by the 951 entity that transmits the CDNI Logging File. It is computed by 952 applying the MD5 ([RFC1321]) cryptographic hash function on the 953 CDNI Logging File, including all the directives and logging 954 records, up to the Integrity-Hash directive itself, excluding 955 the Integrity-Hash directive itself. The Integrity-Hash value 956 is represented as a US-ASCII encoded hexadecimal number, 32 957 digits long (representing a 128 bit hash value). The entity 958 receiving the CDNI Logging File also computes in a similar way 959 the MD5 hash on the received CDNI Logging File and compares 960 this hash to the value of the Integrity-Hash directive. If the 961 two values are equal, then the received CDNI Logging File MUST 962 be considered non-corrupted. If the two values are different, 963 the received CDNI Logging File MUST be considered corrupted. 964 The behavior of the entity that received a corrupted CDNI 965 Logging File is outside the scope of this specification; we 966 note that the entity MAY attempt to pull again the same CDNI 967 Logging File from the transmitting entity. If the entity 968 receiving a non-corrupted CDNI Logging File adds an 969 Established-Origin directive, it MUST then recompute and update 970 the Integrity-Hash directive so it also protects the added 971 Established-Origin directive. 973 * occurrence: there MUST be zero or exactly one instance of this 974 directive. There SHOULD be exactly one instance of this 975 directive. One situation where that directive could be omitted 976 is where integrity protection is already provided via another 977 mechanism (for example if an integrity hash is associated to 978 the CDNI Logging File out of band through the CDNI Logging Feed 979 ( Section 4.1) leveraging ATOM extensions such as those 980 proposed in [I-D.snell-atompub-link-extensions]. When present, 981 the Integrity-Hash field MUST be the last line of the CDNI 982 Logging File. 984 An uCDN-side implementation of the CDNI Logging interface MUST reject 985 a CDNI Logging File that does not comply with the occurences 986 specified above for each and every directive. For example, an uCDN- 987 side implementation of the CDNI Logging interface receiving a CDNI 988 Logging file with zero occurence of the Version directive, or with 989 two occurences of the Integrity-hash, MUST reject this CDNI Logging 990 File. 992 An entity receiving a CDNI Logging File with a value set to 993 "CDNI/1.0" MUST process the CDNI Logging File as per the present 994 document. An entity receiving a CDNI Logging File with a value set 995 to a different value MUST process the CDNI Logging File as per the 996 specification referenced in the CDNI Logging File Version registry 997 (see Section 6.1) if the implementation supports this specification 998 and MUST reject the CDNI Logging File otherwise. 1000 3.4. CDNI Logging Records 1002 A CDNI Logging Record consists of a sequence of CDNI Logging Fields 1003 relating to that single CDNI Logging Record. 1005 CDNI Logging Fields MUST be separated by the "horizontal tabulation 1006 (HTAB)" character. 1008 To facilitate readability, a prefix scheme is used for CDNI Logging 1009 field names in a similar way to the one used in W3C Extended Log File 1010 Format [ELF]. The semantics of the prefix in the present document 1011 is: 1013 o c: refers to the User Agent that issues the request (corresponds 1014 to the "client" of W3C Extended Log Format) 1016 o d: refers to the dCDN (relative to a given CDN acting as a uCDN) 1018 o s: refers to the dCDN Surrogate that serves the request 1019 (corresponds to the "server" of W3C Extended Log Format) 1021 o u: refers to the uCDN (relative to a given CDN acting as a dCDN) 1023 o cs: refers to communication from the User Agent towards the dCDN 1024 Surrogate 1026 o sc: refers to communication from the dCDN Surrogate towards the 1027 User Agent 1029 An implementation of the CDNI Logging interface as per the present 1030 specification MUST support the CDNI HTTP Request Logging Record as 1031 specified in Section 3.4.1. 1033 A CDNI Logging Record is defined by the following rules: 1035 FIEVAL = 1037 = FIEVAL * ; where FIEVAL 1038 contains the CDNI Logging field value corresponding to the CDNI 1039 Logging field names (FIENAME) listed is the last Fields directive 1040 preceding the present CDNI Logging Record. 1042 3.4.1. HTTP Request Logging Record 1044 This section defines the CDNI Logging Record of Record-Type 1045 "cdni_http_request_v1". It is applicable to content delivery 1046 performed by the dCDN using HTTP/1.0([RFC1945]), 1047 HTTP/1.1([RFC7230],[RFC7231], [RFC7232], [RFC7233], [RFC7234], 1048 [RFC7235]) or HTTPS ([RFC2818], [RFC7230]). We observe that, in the 1049 case of HTTPS delivery, there may be value in logging additional 1050 information specific to the operation of HTTP over TLS and we note 1051 that this is outside the scope of the present document and may be 1052 addressed in a future document defining another CDNI Logging Record 1053 or another version of the HTTP Request Logging Record. 1055 The "cdni_http_request_v1" Record-Type is also expected to be 1056 applicable to HTTP/2 [I-D.ietf-httpbis-http2] (which is still under 1057 development at the time of writing the present document) since a 1058 fundamental design tenet of HTTP/2 is to preserve the HTTP/1.1 1059 semantics. We observe that, in the case of HTTP/2 delivery, there 1060 may be value in logging additional information specific to the 1061 additional functionality of HTTP/2 (e.g. information related to 1062 connection identification, to stream identification, to stream 1063 priority and to flow control). We note that such additional 1064 information is outside the scope of the present document and may be 1065 addressed in a future document defining another CDNI Logging Record 1066 or another version of the HTTP Request Logging Record. 1068 The "cdni_http_request_v1" Record-Type contains the following CDNI 1069 Logging Fields, listed by their field name: 1071 o date: 1073 * format: DATE 1074 * field value: the date at which the processing of request 1075 completed on the Surrogate. 1077 * occurrence: there MUST be one and only one instance of this 1078 field. 1080 o time: 1082 * format: TIME 1084 * field value: the time at which the processing of request 1085 completed on the Surrogate. 1087 * occurrence: there MUST be one and only one instance of this 1088 field. 1090 o time-taken: 1092 * format: DEC 1094 * field value: decimal value of the duration, in seconds, between 1095 the start of the processing of the request and the completion 1096 of the request processing (e.g., completion of delivery) by the 1097 Surrogate. 1099 * occurrence: there MUST be one and only one instance of this 1100 field. 1102 o c-ip: 1104 * format: ADDRESS 1106 * field value: the source IPv4 or IPv6 address (i.e., the 1107 "client" address) in the request received by the Surrogate. 1109 * occurrence: there MUST be one and only one instance of this 1110 field. 1112 o c-ip-anonymizing: 1114 * format: 1*DIGIT 1116 * field value: the number of rightmost bits of the address in the 1117 c-ip field that are zeroed-out in order to anonymize the 1118 logging record. The mechanism by which the two ends of the 1119 CDNI Logging interface agree on whether anonymization is to be 1120 supported and the number of bits that need to be zeroed-out for 1121 this purpose are outside the scope of the present document. 1123 * occurrence: there MUST be zero or exactly one instance of this 1124 field. 1126 o c-port: 1128 * format: 1*DIGIT 1130 * field value: the source TCP port (i.e., the "client" port) in 1131 the request received by the Surrogate. 1133 * occurrence: there MUST be zero or exactly one instance of this 1134 field. 1136 o s-ip: 1138 * format: ADDRESS 1140 * field value: the IPv4 or IPv6 address of the Surrogate that 1141 served the request (i.e., the "server" address). 1143 * occurrence: there MUST be zero or exactly one instance of this 1144 field. 1146 o s-hostname: 1148 * format: host 1150 * field value: the hostname of the Surrogate that served the 1151 request (i.e., the "server" hostname). 1153 * occurrence: there MUST be zero or exactly one instance of this 1154 field. 1156 o s-port: 1158 * format: 1*DIGIT 1160 * field value: the destination TCP port (i.e., the "server" port) 1161 in the request received by the Surrogate. 1163 * occurrence: there MUST be zero or exactly one instance of this 1164 field. 1166 o cs-method: 1168 * format: NHTABSTRING 1169 * field value: this is the method of the request received by the 1170 Surrogate. In the case of HTTP delivery, this is the HTTP 1171 method in the request. 1173 * occurrence: There MUST be one and only one instance of this 1174 field. 1176 o cs-uri: 1178 * format: NHTABSTRING 1180 * field value: this is the "effective request URI" of the request 1181 received by the Surrogate as specified in [RFC7230]. It 1182 complies with the "http" URI scheme or the "https" URI scheme 1183 as specified in [RFC7230]). 1185 * occurrence: there MUST be zero or exactly one instance of this 1186 field. 1188 o u-uri: 1190 * format: NHTABSTRING 1192 * field value: this is a complete URI, derived from the 1193 "effective request URI" ([RFC7230]) of the request received by 1194 the Surrogate (i.e., the cs-uri) but transformed by the entity 1195 generating or transmitting the CDNI Logging Record, in a way 1196 that is agreed upon between the two ends of the CDNI Logging 1197 interface, so the transformed URI is meaningful to the uCDN. 1198 For example, the two ends of the CDNI Logging interface could 1199 agree that the u-uri is constructed from the cs-uri by removing 1200 the part of the hostname that exposes which individual 1201 Surrogate actually performed the delivery. The details of 1202 modification performed to generate the u-uri, as well as the 1203 mechanism to agree on these modifications between the two sides 1204 of the CDNI Logging interface are outside the scope of the 1205 present document. 1207 * occurrence: there MUST be one and only one instance of this 1208 field. 1210 o protocol: 1212 * format: NHTABSTRING 1214 * field value: this is value of the HTTP-Version field as 1215 specified in [RFC7230] of the Request-Line of the request 1216 received by the Surrogate (e.g., "HTTP/1.1"). 1218 * occurrence: there MUST be one and only one instance of this 1219 field. 1221 o sc-status: 1223 * format: 3DIGIT 1225 * field value: this is the Status-Code in the response from the 1226 Surrogate. In the case of HTTP delivery, this is the HTTP 1227 Status-Code in the HTTP response. 1229 * occurrence: There MUST be one and only one instance of this 1230 field. 1232 o sc-total-bytes: 1234 * format: 1*DIGIT 1236 * field value: this is the total number of bytes of the response 1237 sent by the Surrogate in response to the request. In the case 1238 of HTTP delivery, this includes the bytes of the Status-Line, 1239 the bytes of the HTTP headers and the bytes of the message- 1240 body. 1242 * occurrence: There MUST be one and only one instance of this 1243 field. 1245 o sc-entity-bytes: 1247 * format: 1*DIGIT 1249 * field value: this is the number of bytes of the message-body in 1250 the HTTP response sent by the Surrogate in response to the 1251 request. This does not include the bytes of the Status-Line or 1252 the bytes of the HTTP headers. 1254 * occurrence: there MUST be zero or exactly one instance of this 1255 field. 1257 o cs(): 1259 * format: QSTRING 1261 * field value: the value of the HTTP header (identified by the 1262 in the CDNI Logging field name) as it 1263 appears in the request processed by the Surrogate, but 1264 prepended by a DQUOTE and appended by a DQUOTE. For example, 1265 when the CDNI Logging field name (FIENAME) listed in the 1266 preceding Fields directive is cs(User-Agent), this CDNI Logging 1267 field value contains the value of the User-Agent HTTP header as 1268 received by the Surrogate in the request it processed, but 1269 prepended by a DQUOTE and appended by a DQUOTE. If the HTTP 1270 header as it appeared in the request processed by the Surrogate 1271 contains one or more DQUOTE, each DQUOTE MUST be escaped by an 1272 additional DQUOTE. For example, if the HTTP header contains 1273 My_Header"value", then the field value of the cs() is "My_Header""value""". The entity transmitting the 1275 CDNI Logging File MUST ensure that the of 1276 the cs(): 1286 * format: QSTRING 1288 * field value: the value of the HTTP header (identified by the 1289 in the CDNI Logging field name) as it 1290 appears in the response issued by the Surrogate to serve the 1291 request, but prepended by a DQUOTE and appended by a DQUOTE. 1292 If the HTTP header as it appeared in the request processed by 1293 the Surrogate contains one or more DQUOTE, each DQUOTE MUST be 1294 escaped by an additional DQUOTE. For example, if the HTTP 1295 header contains My_Header"value", then the field value of the 1296 cs() is "My_Header""value""". The entity 1297 transmitting the CDNI Logging File MUST ensure that the of the cs(, there MUST be 1306 zero or exactly one instance of this field. 1308 o s-ccid: 1310 * format: QSTRING 1312 * field value: this contains the value of the Content Collection 1313 IDentifier (CCID) associated by the uCDN to the content served 1314 by the Surrogate via the CDNI Metadata interface 1315 ([I-D.ietf-cdni-metadata]), prepended by a DQUOTE and appended 1316 by a DQUOTE. If the CCID conveyed in the CDNI Metadata 1317 interface contains one or more DQUOTE, each DQUOTE MUST be 1318 escaped by an additional DQUOTE. For example, if the CCID 1319 conveyed in the CDNI Metadata interface is My_CCIDD"value", 1320 then the field value of the s-ccid is "My_CCID""value""". 1322 * occurrence: there MUST be zero or exactly one instance of this 1323 field. For a given , there MUST be zero or 1324 exactly one instance of this field. 1326 o s-sid: 1328 * format: QSTRING 1330 * field value: this contains the value of a Session IDentifier 1331 (SID) generated by the dCDN for a specific HTTP session, 1332 prepended by a DQUOTE and appended by a DQUOTE. In particular, 1333 for HTTP Adaptive Streaming (HAS) session, the Session 1334 IDentifier value is included in the Logging record for every 1335 content chunk delivery of that session in view of facilitating 1336 the later correlation of all the per content chunk log records 1337 of a given HAS session. See section 3.4.2.2. of [RFC6983] for 1338 more discussion on the concept of Session IDentifier in the 1339 context of HAS. If the SID conveyed contains one or more 1340 DQUOTE, each DQUOTE MUST be escaped by an additional DQUOTE. 1341 For example, if the SID is My_SID"value", then the field value 1342 of the s-sid is "My_SID""value""". 1344 * occurrence: there MUST be zero or exactly one instance of this 1345 field. 1347 o s-cached: 1349 * format: 1DIGIT 1351 * field value: this characterises whether the Surrogate served 1352 the request using content already stored on its local cache or 1353 not. The allowed values are "0" (for miss) and "1" (for hit). 1354 "1" MUST be used when the Surrogate did serve the request using 1355 exclusively content already stored on its local cache. "0" MUST 1356 be used otherwise (including cases where the Surrogate served 1357 the request using some, but not all, content already stored on 1358 its local cache). Note that a "0" only means a cache miss in 1359 the Surrogate and does not provide any information on whether 1360 the content was already stored, or not, in another device of 1361 the dCDN, i.e., whether this was a "dCDN hit" or "dCDN miss". 1363 * occurrence: there MUST be zero or exactly one instance of this 1364 field. 1366 The "Fields" directive corresponding to a HTTP Request Logging Record 1367 MUST contain all the fields names whose occurrence is specified above 1368 as "There MUST be one and only one instance of this field". The 1369 corresponding fields value MUST be present in every HTTP Request 1370 Logging Record. 1372 The "Fields" directive corresponding to a HTTP Request Logging Record 1373 MAY list all the fields value whose occurrence is specified above as 1374 "there MUST be zero or exactly one instance of this field" or "there 1375 MAY be zero, one or any number of instances of this field". The set 1376 of such field names actually listed in the "Fields" directive is 1377 selected by the CDN generating the CDNI Logging File based on 1378 agreements between the interconnected CDNs established through 1379 mechanisms outside the scope of this specification (e.g., contractual 1380 agreements). When such a field name is not listed in the "Fields" 1381 directive, the corresponding field value MUST NOT be included in the 1382 Logging Record. When such a field name is listed in the "Fields" 1383 directive, the corresponding field value MUST be included in the 1384 Logging Record; if the value for the field is not available, this 1385 MUST be conveyed via a dash character ("-"). 1387 The fields names listed in the "Fields" directive MAY be listed in 1388 the order in which they are listed in Section 3.4.1 or MAY be listed 1389 in any other order. 1391 A dCDN-side implementation of the CDNI Logging interface MUST 1392 implement all the following Logging Fields in a CDNI Logging Record 1393 of Record-Type "cdni_http_request_v1", and MUST support the ability 1394 to include valid values for each of them: 1396 o date 1398 o time 1400 o time-taken 1402 o c-ip 1404 o c-port 1406 o s-ip 1408 o s-hostname 1410 o s-port 1411 o cs-method 1413 o cs-uri 1415 o u-uri 1417 o protocol 1419 o sc-status 1421 o sc-total-bytes 1423 o sc-entity-bytes 1425 o cs() 1427 o sc() 1429 o s-cached 1431 A dCDN-side implementation of the CDNI Logging interface MAY support 1432 the following Logging Fields in a CDNI Logging Record of Record-Type 1433 "cdni_http_request_v1": 1435 o c-ip-anonymizing 1437 o s-ccid 1439 o s-sid 1441 If a dCDN-side implementation of the CDNI Logging interface supports 1442 these Fields, it MUST support the ability to include valid values for 1443 them. 1445 An uCDN-side implementation of the CDNI Logging interface MUST be 1446 able to accept CDNI Logging Files with CDNI Logging Records of 1447 Record-Type "cdni_http_request_v1" containing any CDNI Logging Field 1448 defined in Section 3.4.1 as long as the CDNI Logging Record and the 1449 CDNI Logging File are compliant with the present document. 1451 In case, an uCDN-side implementation of the CDNI Logging interface 1452 receives a CDNI Logging File with HTTP Request Logging Records that 1453 do not contain field values for exactly the set of field names 1454 actually listed in the preceding "Fields" directive, the 1455 implementation MUST reject those HTTP Request Logging Records, and 1456 MUST accept the other HTTP Request Logging Records . 1458 3.5. CDNI Logging File Example 1460 Let us consider the upstream CDN and the downstream CDN labelled uCDN 1461 and dCDN-1 in Figure 1. When dCDN-1 acts as a downstream CDN for 1462 uCDN and performs content delivery on behalf of uCDN, dCDN-1 will 1463 include the CDNI Logging Records corresponding to the content 1464 deliveries performed on behalf of uCDN in the CDNI Logging Files for 1465 uCDN. An example CDNI Logging File communicated by dCDN-1 to uCDN is 1466 shown below in Figure 4. 1468 #Version:CDNI/1.0 1470 #UUID:urn:uuid:f81d4fae-7dec-11d0-a765-00a0c91e6bf6 1472 #Claimed-Origin:cdni-logging-entity.dcdn-1.example.com 1474 #Record-Type:cdni_http_request_v1 1476 #Fields:datetimetime-takenc-ip 1477 cs-methodu-uriprotocolsc-status 1478 sc-total-bytescs(User-Agent)cs(Referer) 1479 s-cached 1481 2013-05-1700:38:06.8259.05810.5.7.1GET 1482 http://cdni-ucdn.dcdn-1.example.com/video/movie100.mp4 1483 HTTP/1.12006729891"Mozilla/5.0 1484 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/533.4 (KHTML, like 1485 Gecko) Chrome/5.0.375.127 Safari/533.4" 1486 "host1.example.com"1 1488 2013-05-1700:39:09.14515.3210.5.10.5GET 1489 http://cdni-ucdn.dcdn-1.example.com/video/movie118.mp4 1490 HTTP/1.120015799210"Mozilla/5.0 1491 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/533.4 (KHTML, like 1492 Gecko) Chrome/5.0.375.127 Safari/533.4" 1493 "host1.example.com"1 1495 2013-05-1700:42:53.43752.87910.5.10.5GET 1496 http://cdni-ucdn.dcdn-1.example.com/video/picture11.mp4 1497 HTTP/1.020097234724"Mozilla/5.0 1498 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/533.4 (KHTML, like 1499 Gecko) Chrome/5.0.375.127 Safari/533.4" 1500 "host5.example.com"0 1502 #Integrity-Hash:...32-hexadecimal-digit hash value... 1504 Figure 4: CDNI Logging File Example 1506 If uCDN establishes by some means (e.g. via TLS authentication when 1507 pulling the CDNI Logging File) the identity of the entity from which 1508 it pulled the CDNI Logging File, uCDN can add to the CDNI Logging an 1509 Established-Origin directive as illustrated below: 1511 #Established-Origin:cdni-logging-entity.dcdn- 1512 1.example.com 1514 As illustrated in Figure 2, uCDN will then ingest the corresponding 1515 CDNI Logging Records into its Collection process, alongside the 1516 Logging Records generated locally by the uCDN itself. This allows 1517 uCDN to aggregate Logging Records for deliveries performed by itself 1518 (through Records generated locally) as well as for deliveries 1519 performed by its downstream CDN(s). This aggregate information can 1520 then be used (after Filtering and Rectification, as illustrated in 1521 Figure 2) by Log Consuming Applications that take into account 1522 deliveries performed by uCDN as well as by all of its downstream 1523 CDNs. 1525 We observe that the time between 1527 1. when a delivery is completed in dCDN and 1529 2. when the corresponding Logging Record is ingested by the 1530 Collection process in uCDN 1532 depends on a number of parameters such as the Logging Period agreed 1533 to by uCDN and dCDN, how much time uCDN waits before pulling the CDNI 1534 Logging File once it is advertised in the CDNI Logging Feed, and the 1535 time to complete the pull of the CDNI Logging File. Therefore, if we 1536 consider the set of Logging Records aggregated by the Collection 1537 process in uCDN in a given time interval, there could be a permanent 1538 significant timing difference between the CDNI Logging Records 1539 received from the dCDN and the Logging Records generated locally. 1540 For example, in a given time interval, the Collection process in uCDN 1541 may be aggregating Logging Records generated locally by uCDN for 1542 deliveries performed in the last hour and CDNI Logging Records 1543 generated in the dCDN for deliveries in the hour before last. 1545 3.6. Cascaded CDNI Logging Files Example 1547 Let us consider the cascaded CDN scenario of uCDN, dCDN-2 and dCDN-3 1548 as depicted in Figure 1. After completion of a delivery by dCDN-3 on 1549 behalf of dCDN-2, dCDN-3 will include a corresponding Logging Record 1550 in a CDNI Logging File that will be pulled by dCDN-2 and that is 1551 illustrated below in Figure 5. In practice, a CDNI Logging File is 1552 likely to contain a very high number of CDNI Logging Records. 1554 However, for readability, the example in Figure 5 contains a single 1555 CDNI Logging Record. 1557 #Version:CDNI/1.0 1559 #UUID:urn:uuid:65718ef-0123-9876-adce4321bcde 1561 #Claimed-Origin:cdni-logging-entity.dcdn-3.example.com 1563 #Record-Type:cdni_http_request_v1 1565 #Fields:datetimetime-takenc-ip 1566 cs-methodu-uriprotocolsc-status 1567 sc-total-bytescs(User-Agent)cs(Referer) 1568 s-cached 1570 2013-05-1700:39:09.11914.0710.5.10.9GET 1571 http://cdni-dcdn-2.dcdn-3.example.com/video/movie118.mp4 1572 HTTP/1.120015799210"Mozilla/5.0 1573 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/533.4 (KHTML, like 1574 Gecko) Chrome/5.0.375.127 Safari /533.4" 1575 "host1.example.com"1 1577 #Integrity-Hash:...32-hexadecimal-digit hash value... 1579 Figure 5: Cascaded CDNI Logging File Example (dCDN-3 to dCDN-2) 1581 If dCDN-2 establishes by some means (e.g. via TLS authentication when 1582 pulling the CDNI Logging File) the identity of the entity from which 1583 it pulled the CDNI Logging File, dCDN-2 can add to the CDNI Logging 1584 an Established-Origin directive as illustrated below: 1586 #Established-Origin:cdni-logging-entity.dcdn- 1587 3.example.com 1589 dCDN-2 (behaving as an upstream CDN from the viewpoint of dCDN-3) 1590 will then ingest the CDNI Logging Record for the considered dCDN-3 1591 delivery into its Collection process (as illustrated in Figure 2). 1592 This Logging Record may be aggregated with Logging Records generated 1593 locally by dCDN-2 for deliveries performed by dCDN-2 itself. Say, 1594 for illustration, that the content delivery performed by dCDN-3 on 1595 behalf of dCDN-2 had actually been redirected to dCDN-2 by uCDN, and 1596 say that another content delivery has just been redirected by uCDN to 1597 dCDN-2 and that dCDN-2 elected to perform the corresponding delivery 1598 itself. Then after Filtering and Rectification (as illustrated in 1599 Figure 2), dCDN-2 will include the two Logging Records corresponding 1600 respectively to the delivery performed by dCDN-3 and the delivery 1601 performed by dCDN-2, in the next CDNI Logging File that will be 1602 communicated to uCDN. An example of such CDNI Logging File is 1603 illustrated below in Figure 6. 1605 #Version:CDNI/1.0 1607 #UUID:urn:uuid:1234567-8fedc-abab-0987654321ff 1609 #Claimed-Origin:cdni-logging-entity.dcdn-2.example.com 1611 #Record-Type:cdni_http_request_v1 1613 #Fields:datetimetime-takenc-ip 1614 cs-methodu-uriprotocolsc-status 1615 sc-total-bytescs(User-Agent)cs(Referer) 1616 s-cached 1618 2013-05-1700:39:09.11914.0710.5.10.9GET 1619 http://cdni-ucdn.dcdn-2.example.com/video/movie118.mp4 1620 HTTP/1.120015799210"Mozilla/5.0 1621 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/533.4 (KHTML, like 1622 Gecko) Chrome/5.0.375.127 Safari /533.4" 1623 "host1.example.com"1 1625 2013-05-1701:42:53.43752.87910.5.10.12GET 1626 http://cdni-ucdn.dcdn-2.example.com/video/picture11.mp4 1627 HTTP/1.020097234724"Mozilla/5.0 1628 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/533.4 (KHTML, like 1629 Gecko) Chrome/5.0.375.127 Safari /533.4" 1630 "host5.example.com"0 1632 #Integrity-Hash:...32-hexadecimal-digit hash value... 1634 Figure 6: Cascaded CDNI Logging File Example (dCDN-2 to uCDN) 1636 If uCDN establishes by some means (e.g. via TLS authentication when 1637 pulling the CDNI Logging File) the identity of the entity from which 1638 it pulled the CDNI Logging File, uCDN can add to the CDNI Logging an 1639 Established-Origin directive as illustrated below: 1641 #Established-Origin:cdni-logging-entity.dcdn- 1642 2.example.com 1644 In the example of Figure 6, we observe that: 1646 o the first Logging Record corresponds to the Logging Record 1647 communicated earlier to dCDN-2 by dCDN-3, which corresponds to a 1648 delivery redirected by uCDN to dCDN-2 and then redirected by 1649 dCDN-2 to dCDN-3. The fields values in this Logging Record are 1650 copied from the corresponding CDNI Logging REcord communicated to 1651 dCDN2 by dCDN-3, with the exception of the u-uri that now reflects 1652 the URI convention between uCDN and dCDN-2 and that presents the 1653 delivery to uCDN as if it was performed by dCDN-2 itself. This 1654 reflects the fact that dCDN-2 had taken the full responsibility of 1655 the corresponding delivery (even if in this case, dCDN-2 elected 1656 to redirect the delivery to dCDN-3 so it is actually performed by 1657 dCDN-3 on behalf of dCDN-2). 1659 o the second Logging Record corresponds to a delivery redirected by 1660 uCDN to dCDN-2 and performed by dCDN-2 itself. The time of the 1661 delivery in this Logging Record may be significantly more recent 1662 than the first Logging Record since it was generated locally while 1663 the first Logging Record was generated by dCDN-3 and had to be 1664 advertised , and then pulled and then ingested into the dCDN-2 1665 Collection process, before being aggregated with the second 1666 Logging Record. 1668 4. Protocol for Exchange of CDNI Logging File After Full Collection 1670 This section specifies a protocol for the exchange of CDNI Logging 1671 Files as specified in Section 3 after the CDNI Logging File is fully 1672 collected by the dCDN. 1674 This protocol comprises: 1676 o a CDNI Logging feed, allowing the dCDN to notify the uCDN about 1677 the CDNI Logging Files that can be retrieved by that uCDN from the 1678 dCDN, as well as all the information necessary for retrieving each 1679 of these CDNI Logging Files. The CDNI Logging feed is specified 1680 in Section 4.1. 1682 o a CDNI Logging File pull mechanism, allowing the uCDN to obtain 1683 from the dCDN a given CDNI Logging File at the uCDN's convenience. 1684 The CDNI Logging File pull mechanisms is specified in Section 4.2. 1686 An implementation of the CDNI Logging interface on the dCDN side (the 1687 entity generating the CDNI Logging file) MUST support the server side 1688 of the CDNI Logging feed (as specified in Section 4.1) and the server 1689 side of the CDNI Logging pull mechanism (as specified in 1690 Section 4.2). 1692 An implementation of the CDNI Logging interface on the uCDN side (the 1693 entity consuming the CDNI Logging file) MUST support the client side 1694 of the CDNI Logging feed (as specified in Section 4.1) and the client 1695 side of the CDNI Logging pull mechanism (as specified in 1696 Section 4.2). 1698 4.1. CDNI Logging Feed 1700 The server-side implementation of the CDNI Logging feed MUST produce 1701 an Atom feed [RFC4287]. This feed is used to advertise log files 1702 that are available for the client-side to retrieve using the CDNI 1703 Logging pull mechanism. 1705 4.1.1. Atom Formatting 1707 A CDNI Logging feed MUST be structured as an Archived feed, as 1708 defined in [RFC5005], and MUST be formatted in Atom [RFC4287]. This 1709 means it consists of a subscription document that is regularly 1710 updated as new CDNI Logging Files become available, and information 1711 about older CDNI Logging files is moved into archive documents. Once 1712 created, archive documents are never modified. 1714 Each CDNI Logging File listed in an Atom feed MUST be described in an 1715 atom:entry container element. 1717 The atom:entry MUST contain an atom:content element whose "src" 1718 attribute is a link to the CDNI Logging File and whose "type" 1719 attribute is the MIME Media Type indicating that the entry is a CDNI 1720 Logging File. We define this MIME Media Type as "application/ 1721 cdni.LoggingFile" (See Section 6.5). 1723 For compatibility with some Atom feed readers the atom:entry MAY also 1724 contain an atom:link entry whose "href" attribute is a link to the 1725 CDNI Logging File and whose "type" attribute is the MIME Media Type 1726 indicating that the entry is a CDNI Logging File using the 1727 "application/cdni.LoggingFile" MIME Media Type (See Section 6.5). 1729 The URI used in the atom:id of the atom:entry MUST contain the UUID 1730 of the CDNI Logging File. 1732 The atom:updated in the atom:entry MUST indicate the time at which 1733 the CDNI Logging File was last updated. 1735 4.1.2. Updates to Log Files and the Feed 1737 CDNI Logging Files MUST NOT be modified by the dCDN once published in 1738 the CDNI Logging feed. 1740 The frequency with which the subscription feed is updated, the period 1741 of time covered by each CDNI Logging File or each archive document, 1742 and timeliness of publishing of CDNI Logging Files are outside the 1743 scope of the present document and are expected to be agreed upon by 1744 uCDN and dCDN via other means (e.g., human agreement). 1746 The server-side implementation MUST be able to set, and SHOULD set, 1747 HTTP cache control headers on the subscription feed to indicate the 1748 frequency at which the client-side is to poll for updates. 1750 The client-side MAY use HTTP cache control headers (set by the 1751 server-side) on the subscription feed to determine the frequency at 1752 which to poll for updates. The client-side MAY instead, or in 1753 addition, use other information to determine when to poll for updates 1754 (e.g., a polling frequency that may have been negotiated between the 1755 uCDN and dCDN by mechanisms outside the scope of the present document 1756 and that is to override the indications provided in the HTTP cache 1757 control headers). 1759 The potential retention limits (e.g., sliding time window) within 1760 which the dCDN is to retain and be ready to serve an archive document 1761 is outside the scope of the present document and is expected to be 1762 agreed upon by uCDN and dCDN via other means (e.g., human agreement). 1763 The server-side implementation MUST retain, and be ready to serve, 1764 any archive document within the agreed retention limits. Outside 1765 these agreed limits, the server-side implementation MAY indicate its 1766 inability to serve (e.g., with HTTP status code 404) an archive 1767 document or MAY refuse to serve it (e.g., with HTTP status code 403 1768 or 410). 1770 4.1.3. Redundant Feeds 1772 The server-side implementation MAY present more than one CDNI Logging 1773 feed for redundancy. Each CDNI Logging File MAY be published in more 1774 than one feed. 1776 A client-side implementation MAY support such redundant CDNI Logging 1777 feeds. If it supports redundant CDNI Logging feed, the client-side 1778 can use the UUID of the CDNI Logging File, presented in the atom:id 1779 element of the Atom feed, to avoid unnecessarily pulling and storing 1780 a given CDNI Logging File more than once. 1782 4.1.4. Example CDNI Logging Feed 1784 Figure 7 illustrates an example of the subscription document of a 1785 CDNI Logging feed. 1787 1788 1789 CDNI Logging Feed 1790 2013-03-23T14:46:11Z 1791 urn:uuid:663ae677-40fb-e99a-049d-c5642916b8ce 1792 1794 1796 1798 CDNI Log Feed 1799 Generator 1800 dcdn.example 1801 1802 CDNI Logging File for uCDN at 1803 2013-03-23 14:15:00 1804 urn:uuid:12345678-1234-abcd-00aa-01234567abcd 1805 2013-03-23T14:15:00Z 1806 1809 CDNI Logging File for uCDN at 1810 2013-03-23 14:15:00 1811 1812 1813 CDNI Logging File for uCDN at 1814 2013-03-23 14:30:00 1815 urn:uuid:87654321-4321-dcba-aa00-dcba7654321 1816 2013-03-23T14:30:00Z 1817 1820 CDNI Logging File for uCDN at 1821 2013-03-23 14:30:00 1822 1823 ... 1824 1825 ... 1826 1827 1829 Figure 7: Example subscription document of a CDNI Logging Feed 1831 4.2. CDNI Logging File Pull 1833 A client-side implementation of the CDNI Logging interface MAY pull, 1834 at its convenience, a CDNI Logging File that is published by the 1835 server-side in the CDNI Logging Feed (in the subscription document or 1836 an archive document). To do so, the client-side: 1838 o MUST implement HTTP/1.1 ([RFC7230],[RFC7231], [RFC7232], 1839 [RFC7233], [RFC7234], [RFC7235]), MAY also support other HTTP 1840 versions (e.g., HTTP/2 [I-D.ietf-httpbis-http2]) and MAY negotiate 1841 which HTTP version is actually used. This allows operators and 1842 implementers to choose to use later versions of HTTP to take 1843 advantage of new features, while still ensuring interoperability 1844 with systems that only support HTTP/1.1. 1846 o MUST use the URI that was associated to the CDNI Logging File 1847 (within the "src" attribute of the corresponding atom:content 1848 element) in the CDNI Logging Feed; 1850 o MUST support exchange of CDNI Logging Files with no content 1851 encoding applied to the representation; 1853 o MUST support exchange of CDNI Logging Files with "gzip" content 1854 encoding (as defined in [RFC7230]) applied to the representation. 1856 Note that a client-side implementation of the CDNI Logging interface 1857 MAY pull a CDNI Logging File that it has already pulled. 1859 The server-side implementation MUST respond to valid pull request by 1860 a client-side implementation for a CDNI Logging File published by the 1861 server-side in the CDNI Logging Feed (in the subscription document or 1862 an archive document). The server-side implementation: 1864 o MUST implement HTTP/1.1 to handle the client-side request and MAY 1865 also support other HTTP versions (e.g., HTTP/2); 1867 o MUST include the CDNI Logging File identified by the request URI 1868 inside the body of the HTTP response; 1870 o MUST support exchange of CDNI Logging Files with no content 1871 encoding applied to the representation; 1873 o MUST support exchange of CDNI Logging Files with "gzip" content 1874 encoding (as defined in [RFC7231]) applied to the representation. 1876 Content negotiation approaches defined in [RFC7231] (e.g., using 1877 Accept-Encoding request-header field or Content-Encoding entity- 1878 header field) MAY be used by the client-side and server-side 1879 implementations to establish the content-coding to be used for a 1880 particular exchange of a CDNI Logging File. 1882 Applying compression content encoding (such as "gzip") is expected to 1883 mitigate the impact of exchanging the large volumes of logging 1884 information expected across CDNs. This is expected to be 1885 particularly useful in the presence of HTTP Adaptive Streaming (HAS) 1886 which, as per the present version of the document, will result in a 1887 separate CDNI Log Record for each HAS segment delivery in the CDNI 1888 Logging File. 1890 The potential retention limits (e.g., sliding time window, maximum 1891 aggregate file storage quotas) within which the dCDN is to retain and 1892 be ready to serve a CDNI Logging File previously advertised in the 1893 CDNI Logging Feed is outside the scope of the present document and is 1894 expected to be agreed upon by uCDN and dCDN via other means (e.g., 1895 human agreement). The server-side implementation MUST retain, and be 1896 ready to serve, any CDNI Logging File within the agreed retention 1897 limits. Outside these agreed limits, the server-side implementation 1898 MAY indicate its inability to serve (e.g., with HTTP status code 404) 1899 a CDNI Logging File or MAY refuse to serve it (e.g., with HTTP status 1900 code 403 or 410). 1902 5. Protocol for Exchange of CDNI Logging File During Collection 1904 We note that, in addition to the CDNI Logging File exchange protocol 1905 specified in Section 4, implementations of the CDNI Logging interface 1906 may also support other mechanisms to exchange CDNI Logging Files. In 1907 particular, such mechanisms might allow the exchange of the CDNI 1908 Logging File to start before the file is fully collected. This can 1909 allow CDNI Logging Records to be communicated by the dCDN to the uCDN 1910 as they are gathered by the dCDN without having to wait until all the 1911 CDNI Logging Records of the same logging period are collected in the 1912 corresponding CDNI Logging File. This approach is commonly referred 1913 to as "tailing" of the file. 1915 Such an approach could be used, for example, to exchange logging 1916 information with a significantly reduced time-lag (e.g., sub-minute 1917 or sub-second) between when the event occurred in the dCDN and when 1918 the corresponding CDNI Logging Record is made available to the uCDN. 1919 This can satisfy log-consuming applications requiring extremely fresh 1920 logging information such as near-real-time content delivery 1921 monitoring. Such mechanisms are for further study and outside the 1922 scope of this document. 1924 6. IANA Considerations 1926 6.1. CDNI Logging Directive Names Registry 1928 The IANA is requested to create a new registry, CDNI Logging 1929 Directive Names. 1931 The initial contents of the CDNI Logging Directives registry comprise 1932 the names of the directives specified in Section 3.3 of the present 1933 document, and are as follows: 1935 +------------------------------+-----------+ 1936 | Directive Name | Reference | 1937 +------------------------------+-----------+ 1938 | Version | RFC xxxx | 1939 | UUID | RFC xxxx | 1940 | Claimed-Origin | RFC xxxx | 1941 | Established-Origin | RFC xxxx | 1942 | Record-Type | RFC xxxx | 1943 | Fields | RFC xxxx | 1944 | Integrity-Hash | RFC xxxx | 1945 +------------------------------+-----------+ 1947 Figure 8 1949 [Instructions to IANA: Replace "RFC xxxx" above by the RFC number of 1950 the present document] 1952 Within the registry, names are to be allocated by IANA according to 1953 the "Specification Required" policy specified in [RFC5226]. 1954 Directive names are to be allocated by IANA with a format of 1955 NAMEFORMAT (see Section 3.1). 1957 Each specification that defines a new CDNI Logging directive needs to 1958 contain a description for the new directive with the same set of 1959 information as provided in Section 3.3 (i.e., format, directive value 1960 and occurrence). 1962 6.2. CDNI Logging File Version Registry 1964 The IANA is requested to create a new registry, CDNI Logging File 1965 Version. 1967 The initial contents of the CDNI Logging Logging File Version 1968 registry comprise the value "CDNI/1.0" specified in Section 3.3 of 1969 the present document, and are as follows: 1971 +-----------------+-----------+----------------------------------+ 1972 | Version | Reference | Description | 1973 +-----------------+-----------+----------------------------------+ 1974 | CDNI/1.0 | RFC xxxx | CDNI Logging File version 1.0 | 1975 | | | as specified in RFC xxxx | 1976 +-----------------+-----------+----------------------------------+ 1978 Figure 9 1980 [Instructions to IANA: Replace "RFC xxxx" above by the RFC number of 1981 the present document] 1983 Within the registry, Version values are to be allocated by IANA 1984 according to the "Specification Required" policy specified in 1985 [RFC5226]. Version values are to be allocated by IANA with a format 1986 as specified for the Version directive in Section 3.3. 1988 6.3. CDNI Logging Record-Types Registry 1990 The IANA is requested to create a new registry, CDNI Logging Record- 1991 Types. 1993 The initial contents of the CDNI Logging Record-Types registry 1994 comprise the names of the CDNI Logging Record types specified in 1995 Section 3.4 of the present document, and are as follows: 1997 +----------------------+-----------+----------------------------------+ 1998 | Record-Types | Reference | Description | 1999 +----------------------+-----------+----------------------------------+ 2000 | cdni_http_request_v1 | RFC xxxx | CDNI Logging Record version 1 | 2001 | | | for content delivery using HTTP | 2002 +----------------------+-----------+----------------------------------+ 2004 Figure 10 2006 [Instructions to IANA: Replace "RFC xxxx" above by the RFC number of 2007 the present document] 2009 Within the registry, Record-Types are to be allocated by IANA 2010 according to the "Specification Required" policy specified in 2011 [RFC5226]. Record-Types are to be allocated by IANA with a format of 2012 NAMEFORMAT (see Section 3.1). Record-Types corresponding to 2013 specifications produced by the IETF CDNI Working Group are to be 2014 allocated a name starting with "cdni_". All other Record-Types are 2015 to be allocated a name that does not start with "cdni". 2017 Each specification that defines a new Record-Type needs to contain a 2018 description for the new Record-Type with the same set of information 2019 as provided in Section 3.4.1. This includes: 2021 o a list of all the CDNI Logging Fields that can appear in a CDNI 2022 Logging Record of the new Record-Type 2024 o for all these Fields: a specification of the occurrence for each 2025 Field in the new Record-Type 2027 o for every newly defined Field, i.e., for every Field that results 2028 in a registration in the CDNI Logging Field Names Registry 2029 (Section 6.4): a specification of the field name, format and field 2030 value. 2032 6.4. CDNI Logging Field Names Registry 2034 The IANA is requested to create a new registry, CDNI Logging Field 2035 Names. 2037 This registry is intended to be shared across the currently defined 2038 Record-Type (i.e., cdni_http_request_v1) as well as potential other 2039 CDNI Logging Record-Types that may be defined in separate 2040 specifications. When a Field from this registry is used by another 2041 CDNI Logging Record-Type, it is to be used with the exact semantics 2042 and format specified in the document that registered this field and 2043 that is identified in the Reference column of the registry. If 2044 another CDNI Logging Record-Type requires a Field with semantics that 2045 are not strictly identical, or a format that is not strictly 2046 identical then this new Field is to be registered in the registry 2047 with a different Field name. When a Field from this registry is used 2048 by another CDNI Logging Record-Type, it can be used with different 2049 occurence rules. 2051 The initial contents of the CDNI Logging Fields Names registry 2052 comprise the names of the CDNI Logging fields specified in 2053 Section 3.4 of the present document, and are as follows: 2055 +------------------------------------------+-----------+ 2056 | Field Name | Reference | 2057 +------------------------------------------+-----------+ 2058 | date | RFC xxxx | 2059 | time | RFC xxxx | 2060 | time-taken | RFC xxxx | 2061 | c-ip | RFC xxxx | 2062 | c-ip-anonymizing | RFC xxxx | 2063 | c-port | RFC xxxx | 2064 | s-ip | RFC xxxx | 2065 | s-hostname | RFC xxxx | 2066 | s-port | RFC xxxx | 2067 | cs-method | RFC xxxx | 2068 | cs-uri | RFC xxxx | 2069 | u-uri | RFC xxxx | 2070 | protocol | RFC xxxx | 2071 | sc-status | RFC xxxx | 2072 | sc-total-bytes | RFC xxxx | 2073 | sc-entity-bytes | RFC xxxx | 2074 | cs() | RFC xxxx | 2075 | sc() | RFC xxxx | 2076 | s-ccid | RFC xxxx | 2077 | s-sid | RFC xxxx | 2078 | s-cached | RFC xxxx | 2079 +------------------------------------------+-----------+ 2081 Figure 11 2083 [Instructions to IANA: Replace "RFC xxxx" above by the RFC number of 2084 the present document] 2086 Within the registry, names are to be allocated by IANA according to 2087 the "Specification Required" policy specified in [RFC5226]. Field 2088 names are to be allocated by IANA with a format of NHTABSTRING (see 2089 Section 3.1). 2091 6.5. CDNI Logging MIME Media Type 2093 The IANA is requested to allocate the "application/cdni.LoggingFile" 2094 MIME Media Type (whose use is specified in Section 4.1.1 of the 2095 present document) in the MIME Media Types registry. 2097 7. Security Considerations 2098 7.1. Authentication, Authorization, Confidentiality, Integrity 2099 Protection 2101 An implementation of the CDNI Logging interface MUST support TLS 2102 transport of the CDNI Logging feed (Section 4.1) and of the CDNI 2103 Logging File pull (Section 4.2) as per [RFC2818] and [RFC7230]. 2105 The use of TLS for transport of the CDNI Logging feed and CDNI 2106 Logging File pull allows: 2108 o the dCDN and uCDN to authenticate each other 2110 and, once they have mutually authenticated each other, it allows:: 2112 o the dCDN and uCDN to authorize each other (to ensure they are 2113 transmitting/receiving CDNI Logging File to/from an authorized 2114 CDN) 2116 o the CDNI Logging information to be transmitted with 2117 confidentiality 2119 o the integrity of the CDNI Logging information to be protected 2120 during the exchange. 2122 In an environment where any such protection is required, the use of a 2123 mutually authenticated encrypted transport MUST be used to ensure 2124 confidentiality of the logging information. TLS SHOULD be used 2125 (including authentication of the remote end) by the server- side and 2126 the client-side of the CDNI Logging feed, as well as the server- side 2127 and the client-side of the CDNI Logging File pull mechanism, unless 2128 alternate methods are used. Alternate methods could include 2129 establishing an IPsec tunnel between the two CDNs or using a 2130 physically secured internal network between two CDNs that are owned 2131 by the same corporate entity). 2133 The general TLS usage guidance in [I-D.ietf-uta-tls-bcp] SHOULD be 2134 followed. 2136 The Integrity-Hash directive inside the CDNI Logging File provides 2137 additional integrity protection, this time targeting potential 2138 corruption of the CDNI logging information during the CDNI Logging 2139 File generation, storage or exchange. This mechanism does not itself 2140 allow restoration of the corrupted CDNI Logging information, but it 2141 allows detection of such corruption and therefore triggering of 2142 appropriate corrective actions (e.g., discard of corrupted 2143 information, attempt to re-obtain the CDNI Logging information). 2144 Note that the Integrity-Hash does not protect against tampering by a 2145 third party, since such a third party could have recomputed and 2146 updated the Integrity-Hash after tampering. Protection against third 2147 party tampering can be achieved as discussed above through the use of 2148 TLS. 2150 7.2. Denial of Service 2152 This document does not define specific mechanism to protect against 2153 Denial of Service (DoS) attacks on the Logging Interface. However, 2154 the CDNI Logging feed and CDNI Logging pull endpoints are typically 2155 to be accessed only by a very small number of valid remote endpoints 2156 and therefore can be easily protected against DoS attacks through the 2157 usual conventional DOS protection mechanisms such as firewalling or 2158 use of Virtual Private Networks (VPNs). 2160 Protection of dCDN Surrogates against spoofed delivery requests is 2161 outside the scope of the CDNI Logging interface. 2163 7.3. Privacy 2165 CDNs have the opportunity to collect detailed information about the 2166 downloads performed by End Users. The provision of this information 2167 to another CDN introduces potential End Users privacy protection 2168 concerns. 2170 The use of mutually authenticated TLS to establish a secure session 2171 for the transport of the CDNI Logging feed and CDNI Logging pull as 2172 discussed in Section 7.1 access to the logging information. This 2173 provides confidentiality while the logging information is in transit 2174 and prevents any other party than the authorised uCDN to gain access 2175 to the logging information. 2177 We observe that when CDNI interconnection is realised as per 2178 [RFC7336], the uCDN handles the initial End User requests (before it 2179 is redirected to the dCDN) so, regardless of which information is, or 2180 is not, communicated to the uCDN through the CDNI Logging interface, 2181 the uCDN has visibility on significant information such as the IP 2182 address of the End User request and the URL of the request. 2184 Nonetheless, if the dCDN and uCDN agree that anonymization is 2185 required to avoid making some detailed information available to the 2186 uCDN (such as how many bytes of the content have been watched by an 2187 End User and/or at what time) or is required to meet some legal 2188 obligations, then the uCDN and dCDN can agree to exchange anonymized 2189 End Users IP address in CDNI Logging Files and the c-ip-anonymization 2190 field can be used to convey the number of bits that have been 2191 anonymized so that the meaningful information can still be easily 2192 extracted from the anonymized addressses (e.g., for geolocation aware 2193 analytics). 2195 We note that anonymization of End Users IP address does not fully 2196 protect against deriving potentially sensitive information about 2197 traffic patterns; in general, increasing the number of bits that are 2198 anonymized can mitigate the risks of deriving such sensitive traffic 2199 pattern information. 2201 We also note that independently of IP addresses, the query string 2202 portion of the URL that may be conveyed inside the cs-uri and u-uri 2203 fields of CDNI Logging Files, or the HTTP cookies( [RFC6265]) that 2204 may be conveyed inside the cs() field of CDNI 2205 Logging Fields, may contain personnal information or information that 2206 can be exploited to derive personal information. Where this is a 2207 concern, the CDNI Logging interface specification allows the dCDN to 2208 not include the cs-uri and to include a u-uri that removes (or hides) 2209 the sensitive part of the query string and allows the dCDN to not 2210 include the cs() fields corresponding to HTTP 2211 headers associated with cookies. 2213 8. Acknowledgments 2215 This document borrows from the W3C Extended Log Format [ELF]. 2217 Rob Murray significantly contributed into the text of Section 4.1. 2219 The authors thank Ben Niven-Jenkins, Kevin Ma, David Mandelberg and 2220 Ray van Brandenburg for their ongoing input. 2222 Finally, we also thank Sebastien Cubaud, Pawel Grochocki, Christian 2223 Jacquenet, Yannick Le Louedec, Anne Marrec , Emile Stephan, Fabio 2224 Costa, Sara Oueslati, Yvan Massot, Renaud Edel, Joel Favier and the 2225 contributors of the EU FP7 OCEAN project for their input in the early 2226 versions of this document. 2228 9. References 2230 9.1. Normative References 2232 [I-D.ietf-uta-tls-bcp] 2233 Sheffer, Y., Holz, R., and P. Saint-Andre, 2234 "Recommendations for Secure Use of TLS and DTLS", draft- 2235 ietf-uta-tls-bcp-11 (work in progress), February 2015. 2237 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 2238 Requirement Levels", BCP 14, RFC 2119, March 1997. 2240 [RFC3986] Berners-Lee, T., Fielding, R., and L. Masinter, "Uniform 2241 Resource Identifier (URI): Generic Syntax", STD 66, RFC 2242 3986, January 2005. 2244 [RFC4122] Leach, P., Mealling, M., and R. Salz, "A Universally 2245 Unique IDentifier (UUID) URN Namespace", RFC 4122, July 2246 2005. 2248 [RFC4287] Nottingham, M., Ed. and R. Sayre, Ed., "The Atom 2249 Syndication Format", RFC 4287, December 2005. 2251 [RFC5005] Nottingham, M., "Feed Paging and Archiving", RFC 5005, 2252 September 2007. 2254 [RFC5226] Narten, T. and H. Alvestrand, "Guidelines for Writing an 2255 IANA Considerations Section in RFCs", BCP 26, RFC 5226, 2256 May 2008. 2258 [RFC5234] Crocker, D. and P. Overell, "Augmented BNF for Syntax 2259 Specifications: ABNF", STD 68, RFC 5234, January 2008. 2261 [RFC5288] Salowey, J., Choudhury, A., and D. McGrew, "AES Galois 2262 Counter Mode (GCM) Cipher Suites for TLS", RFC 5288, 2263 August 2008. 2265 [RFC7230] Fielding, R. and J. Reschke, "Hypertext Transfer Protocol 2266 (HTTP/1.1): Message Syntax and Routing", RFC 7230, June 2267 2014. 2269 [RFC7231] Fielding, R. and J. Reschke, "Hypertext Transfer Protocol 2270 (HTTP/1.1): Semantics and Content", RFC 7231, June 2014. 2272 [RFC7232] Fielding, R. and J. Reschke, "Hypertext Transfer Protocol 2273 (HTTP/1.1): Conditional Requests", RFC 7232, June 2014. 2275 [RFC7233] Fielding, R., Lafon, Y., and J. Reschke, "Hypertext 2276 Transfer Protocol (HTTP/1.1): Range Requests", RFC 7233, 2277 June 2014. 2279 [RFC7234] Fielding, R., Nottingham, M., and J. Reschke, "Hypertext 2280 Transfer Protocol (HTTP/1.1): Caching", RFC 7234, June 2281 2014. 2283 [RFC7235] Fielding, R. and J. Reschke, "Hypertext Transfer Protocol 2284 (HTTP/1.1): Authentication", RFC 7235, June 2014. 2286 9.2. Informative References 2288 [CHAR_SET] 2289 "IANA Character Sets registry", 2290 . 2293 [ELF] Phillip M. Hallam-Baker, and Brian Behlendorf, "Extended 2294 Log File Format, W3C (work in progress), WD-logfile- 2295 960323", . 2297 [I-D.ietf-cdni-metadata] 2298 Niven-Jenkins, B., Murray, R., Caulfield, M., and K. Ma, 2299 "CDN Interconnection Metadata", draft-ietf-cdni- 2300 metadata-09 (work in progress), March 2015. 2302 [I-D.ietf-httpbis-http2] 2303 Belshe, M., Peon, R., and M. Thomson, "Hypertext Transfer 2304 Protocol version 2", draft-ietf-httpbis-http2-17 (work in 2305 progress), February 2015. 2307 [I-D.snell-atompub-link-extensions] 2308 Snell, J., "Atom Link Extensions", draft-snell-atompub- 2309 link-extensions-09 (work in progress), June 2012. 2311 [RFC1321] Rivest, R., "The MD5 Message-Digest Algorithm", RFC 1321, 2312 April 1992. 2314 [RFC1945] Berners-Lee, T., Fielding, R., and H. Nielsen, "Hypertext 2315 Transfer Protocol -- HTTP/1.0", RFC 1945, May 1996. 2317 [RFC2818] Rescorla, E., "HTTP Over TLS", RFC 2818, May 2000. 2319 [RFC6265] Barth, A., "HTTP State Management Mechanism", RFC 6265, 2320 April 2011. 2322 [RFC6707] Niven-Jenkins, B., Le Faucheur, F., and N. Bitar, "Content 2323 Distribution Network Interconnection (CDNI) Problem 2324 Statement", RFC 6707, September 2012. 2326 [RFC6770] Bertrand, G., Stephan, E., Burbridge, T., Eardley, P., Ma, 2327 K., and G. Watson, "Use Cases for Content Delivery Network 2328 Interconnection", RFC 6770, November 2012. 2330 [RFC6983] van Brandenburg, R., van Deventer, O., Le Faucheur, F., 2331 and K. Leung, "Models for HTTP-Adaptive-Streaming-Aware 2332 Content Distribution Network Interconnection (CDNI)", RFC 2333 6983, July 2013. 2335 [RFC7336] Peterson, L., Davie, B., and R. van Brandenburg, 2336 "Framework for Content Distribution Network 2337 Interconnection (CDNI)", RFC 7336, August 2014. 2339 [RFC7337] Leung, K. and Y. Lee, "Content Distribution Network 2340 Interconnection (CDNI) Requirements", RFC 7337, August 2341 2014. 2343 Authors' Addresses 2345 Francois Le Faucheur (editor) 2346 Cisco Systems 2347 E.Space Park - Batiment D 2348 6254 Allee des Ormes - BP 1200 2349 Mougins cedex 06254 2350 FR 2352 Phone: +33 4 97 23 26 19 2353 Email: flefauch@cisco.com 2355 Gilles Bertrand (editor) 2356 Orange 2357 38-40 rue du General Leclerc 2358 Issy les Moulineaux 92130 2359 FR 2361 Phone: +33 1 45 29 89 46 2362 Email: gilles.bertrand@orange.com 2364 Iuniana Oprescu (editor) 2365 Orange 2366 38-40 rue du General Leclerc 2367 Issy les Moulineaux 92130 2368 FR 2370 Phone: +33 6 89 06 92 72 2371 Email: iuniana.oprescu@orange.com 2373 Roy Peterkofsky 2374 Skytide, Inc. 2375 One Kaiser Plaza, Suite 785 2376 Oakland CA 94612 2377 USA 2379 Phone: +01 510 250 4284 2380 Email: roy@skytide.com