idnits 2.17.1 draft-ietf-cdni-logging-02.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- == There are 3 instances of lines with private range IPv4 addresses in the document. If these are generic example addresses, they should be changed to use any of the ranges defined in RFC 6890 (or successor): 192.0.2.x, 198.51.100.x or 203.0.113.x. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document seems to lack the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords. (The document does seem to have the reference to RFC 2119 which the ID-Checklist requires). == The document seems to contain a disclaimer for pre-RFC5378 work, but was first submitted on or after 10 November 2008. The disclaimer is usually necessary only for documents that revise or obsolete older RFCs, and that take significant amounts of text from those RFCs. If you can contact all authors of the source material and they are willing to grant the BCP78 rights to the IETF Trust, you can and should remove the disclaimer. Otherwise, the disclaimer is needed and you can ignore this comment. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (May 27, 2013) is 3979 days in the past. Is this intentional? -- Found something which looks like a code comment -- if you have code sections in the document, please surround them with '' and '' lines. Checking references for intended status: Informational ---------------------------------------------------------------------------- == Missing Reference: 'MED' is mentioned on line 1342, but not defined == Unused Reference: 'RFC2119' is defined on line 1494, but no explicit reference was found in the text == Unused Reference: 'RFC5424' is defined on line 1509, but no explicit reference was found in the text == Outdated reference: A later version (-21) exists of draft-ietf-cdni-metadata-01 ** Obsolete normative reference: RFC 2616 (Obsoleted by RFC 7230, RFC 7231, RFC 7232, RFC 7233, RFC 7234, RFC 7235) == Outdated reference: A later version (-14) exists of draft-ietf-cdni-framework-03 == Outdated reference: A later version (-17) exists of draft-ietf-cdni-requirements-06 Summary: 1 error (**), 0 flaws (~~), 10 warnings (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Internet Engineering Task Force G. Bertrand, Ed. 3 Internet-Draft I. Oprescu, Ed. 4 Intended status: Informational France Telecom - Orange 5 Expires: November 28, 2013 F. Le Faucheur, Ed. 6 Cisco Systems 7 R. Peterkofsky 8 Skytide, Inc. 9 May 27, 2013 11 CDNI Logging Interface 12 draft-ietf-cdni-logging-02 14 Abstract 16 This memo specifies the Logging interface between a downstream CDN 17 (dCDN) and an upstream CDN (uCDN) that are interconnected as per the 18 CDN Interconnection (CDNI) framework. First, it describes a 19 reference model for CDNI logging. Then, it specifies the CDNI 20 Logging File format and the actual protocol for exchange of CDNI 21 Logging Files. 23 Status of This Memo 25 This Internet-Draft is submitted in full conformance with the 26 provisions of BCP 78 and BCP 79. 28 Internet-Drafts are working documents of the Internet Engineering 29 Task Force (IETF). Note that other groups may also distribute 30 working documents as Internet-Drafts. The list of current Internet- 31 Drafts is at http://datatracker.ietf.org/drafts/current/. 33 Internet-Drafts are draft documents valid for a maximum of six months 34 and may be updated, replaced, or obsoleted by other documents at any 35 time. It is inappropriate to use Internet-Drafts as reference 36 material or to cite them other than as "work in progress." 38 This Internet-Draft will expire on November 28, 2013. 40 Copyright Notice 42 Copyright (c) 2013 IETF Trust and the persons identified as the 43 document authors. All rights reserved. 45 This document is subject to BCP 78 and the IETF Trust's Legal 46 Provisions Relating to IETF Documents 47 (http://trustee.ietf.org/license-info) in effect on the date of 48 publication of this document. Please review these documents 49 carefully, as they describe your rights and restrictions with respect 50 to this document. Code Components extracted from this document must 51 include Simplified BSD License text as described in Section 4.e of 52 the Trust Legal Provisions and are provided without warranty as 53 described in the Simplified BSD License. 55 This document may contain material from IETF Documents or IETF 56 Contributions published or made publicly available before November 57 10, 2008. The person(s) controlling the copyright in some of this 58 material may not have granted the IETF Trust the right to allow 59 modifications of such material outside the IETF Standards Process. 60 Without obtaining an adequate license from the person(s) controlling 61 the copyright in such materials, this document may not be modified 62 outside the IETF Standards Process, and derivative works of it may 63 not be created outside the IETF Standards Process, except to format 64 it for publication as an RFC or to translate it into languages other 65 than English. 67 Table of Contents 69 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 70 1.1. Terminology . . . . . . . . . . . . . . . . . . . . . . . 4 71 2. CDNI Logging Reference Model . . . . . . . . . . . . . . . . 5 72 2.1. CDNI Logging interactions . . . . . . . . . . . . . . . . 5 73 2.2. Overall Logging Chain . . . . . . . . . . . . . . . . . . 8 74 2.2.1. Logging Generation and During-Generation Aggregation 9 75 2.2.2. Logging Collection . . . . . . . . . . . . . . . . . 10 76 2.2.3. Logging Filtering . . . . . . . . . . . . . . . . . . 10 77 2.2.4. Logging Rectification and Post-Generation Aggregation 11 78 2.2.5. Log-Consuming Applications . . . . . . . . . . . . . 12 79 2.2.5.1. Maintenance/Debugging . . . . . . . . . . . . . . 12 80 2.2.5.2. Accounting . . . . . . . . . . . . . . . . . . . 12 81 2.2.5.3. Analytics and Reporting . . . . . . . . . . . . . 13 82 2.2.5.4. Security . . . . . . . . . . . . . . . . . . . . 13 83 2.2.5.5. Legal Logging Duties . . . . . . . . . . . . . . 13 84 2.2.5.6. Notions common to multiple Log Consuming 85 Applications . . . . . . . . . . . . . . . . . . 13 86 3. CDNI Logging File Format . . . . . . . . . . . . . . . . . . 15 87 3.1. CDNI Logging File Directives . . . . . . . . . . . . . . 16 88 3.2. Logging Records . . . . . . . . . . . . . . . . . . . . . 19 89 3.2.1. HTTP Request Logging Record . . . . . . . . . . . . . 20 90 3.2.2. CDNI Logging File Example . . . . . . . . . . . . . . 26 91 3.3. Fields and Directives Formats . . . . . . . . . . . . . . 27 92 4. CDNI Logging File Exchange Protocol . . . . . . . . . . . . . 27 93 4.1. CDNI Logging Feed . . . . . . . . . . . . . . . . . . . . 28 94 4.2. CDNI Logging File Pull . . . . . . . . . . . . . . . . . 28 95 5. Open Issues . . . . . . . . . . . . . . . . . . . . . . . . . 29 96 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 31 97 7. Security Considerations . . . . . . . . . . . . . . . . . . . 31 98 7.1. Authentication, Confidentiality, Integrity Protection . . 31 99 7.2. Non Repudiation . . . . . . . . . . . . . . . . . . . . . 32 100 7.3. Privacy . . . . . . . . . . . . . . . . . . . . . . . . . 32 101 8. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 32 102 9. References . . . . . . . . . . . . . . . . . . . . . . . . . 33 103 9.1. Normative References . . . . . . . . . . . . . . . . . . 33 104 9.2. Informative References . . . . . . . . . . . . . . . . . 33 105 Appendix A. Requirements . . . . . . . . . . . . . . . . . . . . 34 106 A.1. Compliance with cdni-requirements . . . . . . . . . . . . 34 107 A.2. Additional Requirements . . . . . . . . . . . . . . . . . 34 108 A.2.1. Timeliness . . . . . . . . . . . . . . . . . . . . . 34 109 A.2.2. Reliability . . . . . . . . . . . . . . . . . . . . . 35 110 A.2.3. Security . . . . . . . . . . . . . . . . . . . . . . 35 111 A.2.4. Scalability . . . . . . . . . . . . . . . . . . . . . 35 112 A.2.5. Consistency between CDNI Logging and CDN Logging . . 35 113 A.2.6. Dispatching/Filtering . . . . . . . . . . . . . . . . 35 114 Appendix B. Analysis of candidate protocols for Logging 115 Transport . . . . . . . . . . . . . . . . . . . . . 36 116 B.1. Syslog . . . . . . . . . . . . . . . . . . . . . . . . . 36 117 B.2. XMPP . . . . . . . . . . . . . . . . . . . . . . . . . . 36 118 B.3. SNMP . . . . . . . . . . . . . . . . . . . . . . . . . . 36 119 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 36 121 1. Introduction 123 This memo specifies the Logging interface between a downstream CDN 124 (dCDN) and an upstream CDN (uCDN). First, it describes a reference 125 model for CDNI logging. Then, it specifies the CDNI Logging File 126 format and the actual protocol for exchange of CDNI Logging Files. 128 The reader should be familiar with the following documents: 130 o CDNI problem statement [RFC6707] and framework 131 [I-D.ietf-cdni-framework] identify a Logging interface, 133 o Section 8 of [I-D.ietf-cdni-requirements] specifies a set of 134 requirements for Logging, 136 o [RFC6770] outlines real world use-cases for interconnecting CDNs. 137 These use cases require the exchange of Logging information 138 between the dCDN and the uCDN. 140 As stated in [RFC6707], "the CDNI Logging interface enables details 141 of logs or events to be exchanged between interconnected CDNs". 143 The present document describes: 145 o The CDNI Logging reference model (Section 2), 147 o The CDNI Logging File format (Section 3), 149 o The CDNI Logging File Exchange protocol (Section 4). 151 1.1. Terminology 153 In this document, the first letter of each CDNI-specific term is 154 capitalized. We adopt the terminology described in [RFC6707] and 155 [I-D.ietf-cdni-framework], and extend it with the additional terms 156 defined below. 158 For clarity, we use the word "Log" only for referring to internal CDN 159 logs and we use the word "Logging" for any inter-CDN information 160 exchange and processing operations related to CDNI Logging interface. 161 Log and Logging formats may be different. 163 CDN Logging information: logging information generated and collected 164 within a CDN 166 CDNI Logging information: logging information exchanged across CDNs 167 using the CDNI Logging Interface 169 Logging information: logging information generated and collected 170 within a CDN or obtained from another CDN using the CDNI Logging 171 Interface 173 CDNI Logging Field: an atomic element of information that can be 174 included in a CDNI Logging Record. The time an event/task started, 175 the IP address of an End user to whom content was delivered, and the 176 URI of the content delivered are examples of CDNI Logging Fields. 178 CDNI Logging Record: an information record providing information 179 about a specific event. This comprises a collection of CDNI Logging 180 Fields. 182 CDNI Logging File: a file containing CDNI Logging Records, as well as 183 additional information facilitating the processing of the CDNI 184 Logging Records. 186 CDN Reporting: the process of providing the relevant information that 187 will be used to create a formatted content delivery report provided 188 to the CSP in deferred time. Such information typically includes 189 aggregated data that can cover a large period of time (e.g., from 190 hours to several months). Uses of Reporting include the collection 191 of charging data related to CDN services and the computation of Key 192 Performance Indicators (KPIs). 194 CDN Monitoring: the process of providing content delivery information 195 in real-time. Monitoring typically includes data in real time to 196 provide visibility of the deliveries in progress, for service 197 operation purposes. It presents a view of the global health of the 198 services as well as information on usage and performance, for network 199 services supervision and operation management. In particular, 200 monitoring data can be used to generate alarms. 202 2. CDNI Logging Reference Model 204 2.1. CDNI Logging interactions 206 The CDNI logging reference model between a given uCDN and a given 207 dCDN involves the following interactions: 209 o customization by the uCDN of the CDNI logging information to be 210 provided by the dCDN to the uCDN (e.g. control of which logging 211 fields are to be communicated to the uCDN for a given task 212 performed by the dCDN, control of which types of events are to be 213 logged). The dCDN takes into account this CDNI logging 214 customization information to determine what logging information to 215 provide to the uCDN, but it may, or may not, take into account 216 this CDNI logging customization information to influence what CDN 217 logging information is to be generated and collected within the 218 dCDN (e.g. even if the uCDN requests a restricted subset of the 219 logging information, the dCDN may elect to generate a broader set 220 of logging information). The mechanism to support the 221 customisation by the uCDN of CDNI Logging information is outside 222 the scope of this document and left for further study. We note 223 that the CDNI Control interface or the CDNI Metadata interface 224 appear as candidate interfaces on which to potentially build such 225 a customisation mechanism in the future. Before such a mechanism 226 is available, the uCDN and dCDN are expected to agree off-line on 227 what CDNI logging information is to be provide by dCDN to UCDN and 228 rely on management plane actions to configure the CDNI Logging 229 functions to generate (respectively, expect) in dCDN 230 (respectively, in uCDN). 232 o generation and collection by the dCDN of logging information 233 related to the completion of any task performed by the dCDN on 234 behalf of the uCDN (e.g., delivery of the content to an end user) 235 or related to events happening in the dCDN that are relevant to 236 the uCDN (e.g., failures or unavailability in dCDN). This takes 237 place within the dCDN and does not directly involve CDNI 238 interfaces. 240 o communication by the dCDN to the uCDN of the logging information 241 collected by the dCDN relevant to the uCDN. This is supported by 242 the CDNI Logging interface and in the scope of the present 243 document. For example, the uCDN may use this logging information 244 to charge the CSP, to perform analytics and monitoring for 245 operational reasons, to provide analytics and monitoring views on 246 its content delivery to the CSP or to perform trouble-shooting. 248 o customization by the dCDN of the logging to be performed by the 249 uCDN on behalf of the dCDN. The mechanism to support the 250 customisation by the dCDN of CDNI Logging information is outside 251 the scope of this document and left for further study. 253 o generation and collection by the uCDN of logging information 254 related to the completion of any task performed by the uCDN on 255 behalf of the dCDN (e.g., serving of content by uCDN to dCDN for 256 acquisition purposes by dCDN) or related to events happening in 257 the uCDN that are relevant to the dCDN. This takes place within 258 the uCDN and does not directly involve CDNI interfaces. 260 o communication by the uCDN to the dCDN of the logging information 261 collected by the uCDN relevant to the dCDN. For example, the dCDN 262 might potentially benefit form this information for security 263 auditing or content acquisition troubleshooting. This is outside 264 the scope of this document and left for further study. 266 Figure 1 provides an example of CDNI Logging interactions (focusing 267 only on the interactions that are in the scope of this document) in a 268 particular scenario where 4 CDNs are involved in the delivery of 269 content from a given CSP: the uCDN has a CDNI interconnection with 270 dCDN-1 and dCDN-2. In turn, dCDN2 has a CDNI interconnection with 271 dCDN3. In this example, uCDN, dCDN-1, dCDN-2 and dCDN-3 all 272 participate in the delivery of content for the CSP. In this example, 273 the CDNI Logging interface enables the uCDN to obtain logging 274 information from all the dCDNs involved in the delivery. In the 275 example, uCDN uses the Logging data: 277 o to analyze the performance of the delivery operated by the dCDNs 278 and to adjust its operations (e.g., request routing) as 279 appropriate, 281 o to provide reporting (non real-time) and monitoring (real-time) 282 information to CSP. 284 For instance, uCDN merges Logging data, extracts relevant KPIs, and 285 presents a formatted report to the CSP, in addition to a bill for the 286 content delivered by uCDN itself or by its dCDNs on his behalf. uCDN 287 may also provide Logging data as raw log files to the CSP, so that 288 the CSP can use its own logging analysis tools. 290 +-----+ 291 | CSP | 292 +-----+ 293 ^ Reporting and monitoring data 294 * Billing 295 ,--*--. 296 Logging ,-' `-. 297 Data =>( uCDN )<= Logging 298 // `-. _,-' \\ Data 299 || `-'-'-' || 300 ,-----. ,-----. 301 ,-' `-. ,-' `-. 302 ( dCDN-1 ) ( dCDN-2 )<== Logging 303 `-. ,-' `-. _,-' \\ Data 304 `--'--' `--'-' || 305 ,-----. 306 ,' `-. 307 ( dCDN-3 ) 308 `. ,-' 309 `--'--' 311 ===> CDNI Logging Interface 312 ***> outside the scope of CDNI 314 Figure 1: Interactions in CDNI Logging Reference Model 316 A dCDN (e.g., dCDN-2) integrates the relevant logging information 317 obtained from its dCDNs (e.g., dCDN-3) in the logging information 318 that it provides to the uCDN, so that the uCDN ultimately obtains all 319 logging information relevant to a CSP for which it acts as the 320 authoritative CDN. 322 Note that the format of Logging information that a CDN provides over 323 the CDNI interface might be different from the one that the CDN uses 324 internally. In this case, the CDN needs to reformat the Logging 325 information before it provides this information to the other CDN over 326 the CDNI Logging interface. Similarly, a CDN might reformat the 327 Logging data that it receives over the CDNI Logging interface before 328 injecting it into its log-consuming applications or before providing 329 some of this logging information to the CSP. Such reformatting 330 operations introduce latency in the logging distribution chain and 331 introduce a processing burden. Therefore, there are benefits in 332 specifying CDNI Logging format that are suitable for use inside CDNs 333 and also are close to the CDN Log formats commonly used in CDNs 334 today. 336 2.2. Overall Logging Chain 338 This section discusses the overall logging chain within and across 339 CDNs to clarify how CDN Logging information is expected to fit in 340 this overall chain. Figure 2 illustrates the overall logging chain 341 within the dCDN, across CDNs using the CDNI Logging interface and 342 within the uCDN. Note that the logging chain illustrated in the 343 Figure is obviously only indicative and varies depending on the 344 specific environments. For example, there may be more or less 345 instantiations of each entity (i.e., there may be 4 Log consuming 346 applications in a given CDN). As another example, there may be one 347 instance of Rectification process per Log Consuming Application 348 instead of a shared one. 350 Log Consuming Log Consuming 351 App App 352 /\ /\ 353 | | 354 Rectification-------- 355 /\ 356 | 357 Filtering 358 /\ 359 | 360 Collection uCDN 361 /\ /\ 362 | | 363 | Generation 364 | 365 CDNI Logging --------------------------------------------- 366 exchange 367 /\ Log Consuming Log Consuming 368 | App App 369 | /\ /\ 370 | | | 371 Rectification Rectification--------- 372 /\ /\ 373 | | 374 Filtering 375 /\ 376 | 377 Collection dCDN 378 /\ /\ 379 | | 380 Generation Generation 382 Figure 2: CDNI Logging in the overall Logging Chain 384 The following subsections describe each of the processes potentially 385 involved in the logging chain of Figure 2. 387 2.2.1. Logging Generation and During-Generation Aggregation 389 CDNs typically generate logging information for all significant task 390 completions, events, and failures. Logs are typically generated by 391 many devices in the CDN including the surrogates, the request routing 392 system, and the control system. 394 The amount of Logging information generated can be huge. Therefore, 395 during contract negotiations, interconnected CDNs often agree on a 396 Logging retention duration, and optionally, on a maximum size of the 397 Logging data that the dCDN must keep. If this size is exceeded, the 398 dCDN must alert the uCDN but may not keep more Logs for the 399 considered time period. In addition, CDNs may aggregate logs and 400 transmit only summaries for some categories of operations instead of 401 the full Logging data. Note that such aggregation leads to an 402 information loss, which may be problematic for some usages of Logging 403 (e.g., debugging). 405 [I-D.brandenburg-cdni-has] discusses logging for HTTP Adaptive 406 Streaming (HAS). In accordance with the recommendations articulated 407 there, it is expected that a surrogate will generate separate logging 408 information for delivery of each chunk of HAS content. This ensures 409 that separate logging information can then be provided to 410 interconnected CDNs over the CDNI Logging interface. Still in line 411 with the recommendations of [I-D.brandenburg-cdni-has], the logging 412 information for per-chunck delivery may include some information (a 413 Content Collection IDentifier and a Session IDentifier) intended to 414 facilitate subsequent post-generation aggregation of per-chunk logs 415 into per-session logs. Note that a CDN may also elect to generate 416 aggregate per-session logs when performing HAS delivery, but this 417 needs to be in addition to, and not instead of, the per-chunk 418 delivery logs. We note that this may be revisited in future versions 419 of this document. 421 Note that in the case of non real-time logging, the trigger of the 422 transmission or generation of the logging file appears to be a 423 synchronous process from a protocol standpoint. The implementation 424 algorithm can choose to enforce a maximum size for the logging file 425 beyond which the transmission is automatically triggered (and thus 426 allow for an asynchronous transmission process). 428 2.2.2. Logging Collection 430 This is the process that continuously collects logs generated by the 431 log-generating entities within a CDN. 433 In a CDNI environment, in addition to collecting logging information 434 from log-generating entities within the local CDN, the Collection 435 process also collects logging information provided by another CDN, or 436 other CDNs, through the CDNI Logging interface. This is illustrated 437 in Figure 2 where we see that the Collection process of the uCDN 438 collects logging information from log-generating entities within the 439 uCDN as well as logging information coming through CDNI Logging 440 exchange with the dCDN through the CDNI Logging interface. 442 2.2.3. Logging Filtering 443 A CDN may require to only present different subset of the whole 444 logging information collected to various log-consuming applications. 445 This is achieved by the Filtering process. 447 In particular, the Filtering process can also filter the right subset 448 of information that needs to be provided to a given interconnected 449 CDN. For example, the filtering process in the dCDN can be used to 450 ensure that only the logging information related to tasks performed 451 on behalf of a given uCDN are made available to that uCDN (thereby 452 filtering all the logging information related to deliveries by the 453 dCDN of content for its own CSPs). Similarly, the Filtering process 454 may filter or partially mask some fields, for example, to protect End 455 Users' privacy when communicating CDNI Logging information to another 456 CDN. Filtering of logging information prior to communication of this 457 information to other CDNs via the CDNI Logging interface requires 458 that the downstream CDN can recognize the set of log records that 459 relate to each interconnected CDN. 461 The CDN will also filter some internal scope information such as 462 information related to its internal alarms (security, failures, load, 463 etc). 465 In some use cases described in [RFC6770], the interconnected CDNs do 466 not want to disclose details on their internal topology. The 467 filtering process can then also filter confidential data on the 468 dCDNs' topology (number of servers, location, etc.). In particular, 469 information about the requests served by every Surrogate may be 470 confidential. Therefore, the Logging information must be protected 471 so that data such as Surrogates' hostnames is not disclosed to the 472 uCDN. In the "Inter-Affiliates Interconnection" use case, this 473 information may be disclosed to the uCDN because both the dCDN and 474 the uCDN are operated by entities of the same group. 476 2.2.4. Logging Rectification and Post-Generation Aggregation 478 If Logging is generated periodically, it is important that the 479 sessions that start in one Logging period and end in another are 480 correctly reported. If they are reported in the starting period, 481 then the Logging of this period will be available only after the end 482 of the session, which delays the Logging generation. 484 A Logging rectification/update mechanism could be useful to reach a 485 good trade-off between the Logging generation delay and the Logging 486 accuracy. Depending on the selected Logging protocol(s), such 487 mechanism may be invaluable for real time Logging, which must be 488 provided rapidly and cannot wait for the end of operations in 489 progress. 491 In the presence of HAS, some log-consuming applications can benefit 492 from aggregate per-session logs. For example, for analytics, per- 493 session logs allow display of session-related trends which are much 494 more meaningful for some types of analysis than chunk-related trends. 495 In the case where the log-generating entities have generated during- 496 generation aggregate logs, those can be used by the applications. In 497 the case where aggregate logs have not been generated, the 498 Rectification process can be extended with a Post-Generation 499 Aggregation process that generates per-session logs from the per- 500 chunk logs, possibly leveraging the information included in the per- 501 chunk logs for that purpose (Content Collection IDentifier and a 502 Session IDentifier). However, in accordance with 503 [I-D.brandenburg-cdni-has], this document does not define exchange of 504 such aggregate logs on the CDNI Logging interface. We note that this 505 may be revisited in future versions of this document. 507 2.2.5. Log-Consuming Applications 509 2.2.5.1. Maintenance/Debugging 511 Logging is useful to permit the detection (and limit the risk) of 512 content delivery failures. In particular, Logging facilitates the 513 resolution of configuration issues. 515 To detect faults, Logging must enable the reporting of any CDN 516 operation success and failure, such as request redirection, content 517 acquisition, etc. The uCDN can summarize such information into KPIs. 518 For instance, Logging format should allow the computation of the 519 number of times during a given epoch that content delivery related to 520 a specific service succeeds/fails. 522 Logging enables the CDN providers to identify and troubleshoot 523 performance degradations. In particular, Logging enables the 524 communication of traffic data (e.g., the amount of traffic that has 525 been forwarded by a dCDN on behalf of an uCDN over a given period of 526 time), which is particularly useful for CDN and network planning 527 operations. 529 2.2.5.2. Accounting 531 Logging is essential for accounting, to permit inter-CDN billing and 532 CSP billing by uCDNs. For instance, Logging information provided by 533 dCDNs enables the uCDN to compute the total amount of traffic 534 delivered by every dCDN for a particular Content Provider, as well 535 as, the associated bandwidth usage (e.g., peak, 95th percentile), and 536 the maximum number of simultaneous sessions over a given period of 537 time. 539 2.2.5.3. Analytics and Reporting 541 The goal of analytics is to gather any relevant information to track 542 audience, analyze user behavior, and monitor the performance and 543 quality of content delivery. For instance, Logging enables the CDN 544 providers to report on content consumption (e.g., delivered sessions 545 per content) in a specific geographic area. 547 The goal of reporting is to gather any relevant information to 548 monitor the performance and quality of content delivery and allow 549 detection of delivery issues. For instance, reporting could track 550 the average delivery throughput experienced by End-Users in a given 551 region for a specific CSP or content set over a period of time. 553 2.2.5.4. Security 555 The goal of security is to prevent and monitor unauthorized access, 556 misuse, modification, and denial of access of a service. A set of 557 information is logged for security purposes. In particular, a record 558 of access to content is usually collected to permit the CSP to detect 559 infringements of content delivery policies and other abnormal End 560 User behaviors. 562 2.2.5.5. Legal Logging Duties 564 Depending on the country considered, the CDNs may have to retain 565 specific Logging information during a legal retention period, to 566 comply with judicial requisitions. 568 2.2.5.6. Notions common to multiple Log Consuming Applications 570 2.2.5.6.1. Logging Information Views 572 Within a given log-consuming application, different views may be 573 provided to different users depending on privacy, business, and 574 scalability constraints. 576 For example, an analytics tool run by the uCDN can provide one view 577 to an uCDN operator that exploits all the logging information 578 available to the uCDN, while the tool may provide a different view to 579 each CSP exploiting only the logging information related to the 580 content of the given CSP. 582 As another example, maintenance and debugging tools may provide 583 different views to different CDN operators, based on their 584 operational role. 586 2.2.5.6.2. Key Performance Indicators (KPIs) 588 This section presents, for explanatory purposes, a non-exhaustive 589 list of Key Performance Indicators (KPIs) that can be extracted/ 590 produced from logs. 592 Multiple log-consuming applications, such as analytics, monitoring, 593 and maintenance applications, often compute and track such KPIs. 595 In a CDNI environment, depending on the situation, these KPIs may be 596 computed by the uCDN or by the dCDN. But it is usually the uCDN that 597 computes KPIs, because uCDN and dCDN may have different definitions 598 of the KPIs and the computation of some KPIs requires a vision of all 599 the deliveries performed by the uCDN and all its dCDNs. 601 Here is a list of important examples of KPIs: 603 o Number of delivery requests received from End-Users in a given 604 region for each piece of content, during a given period of time 605 (e.g., hour/day/week/month) 607 o Percentage of delivery successes/failures among the aforementioned 608 requests 610 o Number of failures listed by failure type (e.g., HTTP error code) 611 for requests received from End Users in a given region and for 612 each piece of content, during a given period of time (e.g., hour/ 613 day/week/month) 615 o Number and cause of premature delivery termination for End Users 616 in a given region and for each piece of content, during a given 617 period of time (e.g., hour/day/week/month) 619 o Maximum and mean number of simultaneous sessions established by 620 End Users in a given region, for a given Content Provider, and 621 during a given period of time (e.g., hour/day/week/month) 623 o Volume of traffic delivered for sessions established by End Users 624 in a given region, for a given Content Provider, and during a 625 given period of time (e.g., hour/day/week/month) 627 o Maximum, mean, and minimum delivery throughput for sessions 628 established by End Users in a given region, for a given Content 629 Provider, and during a given period of time (e.g., hour/day/week/ 630 month) 632 o Cache-hit and byte-hit ratios for requests received from End Users 633 in a given region for each piece of content, during a given period 634 of time (e.g., hour/day/week/month) 636 o Top 10 of the most popularly requested content (during a given day 637 /week/month), 639 o Terminal type (mobile, PC, STB, if this information can be 640 acquired from the browser type header, for example). 642 Additional KPIs can be computed from other sources of information 643 than the Logging, for instance, data collected by a content portal or 644 by specific client-side application programming interfaces. Such 645 KPIs are out of scope for the present memo. 647 The KPIs used depend strongly on the considered log-consuming 648 application -- the CDN operator may be interested in different 649 metrics than the CSP is. In particular, CDN operators are often 650 interested in delivery and acquisition performance KPIs, information 651 related to Surrogates' performance, caching information to evaluate 652 the cache-hit ratio, information about the delivered file size to 653 compute the volume of content delivered during peak hour, etc. 655 Some of the KPIs, for instance those providing an instantaneous 656 vision of the active sessions for a given CSP's content, are useful 657 essentially if they are provided in real-time. By contrast, some 658 other KPIs, such as the one averaged on a long period of time, can be 659 provided in non-real time. 661 3. CDNI Logging File Format 663 As defined in Section 1.1 a CDNI logging field is as an atomic 664 logging information element and a CDNI Logging Record is a collection 665 of CDNI Logging Fields containing all logging information 666 corresponding to a single logging event. This document defines a 667 third level of structure, the CDNI Logging File, that is a collection 668 of CDNI Logging Records. This structure is illustrated in Figure 3. 669 The CDNI Logging File structure and encoding is specified in the 670 present section. 672 +------------------------------------------------------+ 673 |CDNI Logging File | 674 | | 675 | +--------------------------------------------------+ | 676 | |CDNI Logging Record | | 677 | | +-------------+ +-------------+ +-------------+ | | 678 | | |CDNI Logging | |CDNI Logging | |CDNI Logging | | | 679 | | | Field | | Field | | Field | | | 680 | | +-------------+ +-------------+ +-------------+ | | 681 | +--------------------------------------------------+ | 682 | | 683 | +--------------------------------------------------+ | 684 | |CDNI Logging Record | | 685 | | +-------------+ +-------------+ +-------------+ | | 686 | | |CDNI Logging | |CDNI Logging | |CDNI Logging | | | 687 | | | Field | | Field | | Field | | | 688 | | +-------------+ +-------------+ +-------------+ | | 689 | +--------------------------------------------------+ | 690 | | 691 | +--------------------------------------------------+ | 692 | |CDNI Logging Record | | 693 | | +-------------+ +-------------+ +-------------+ | | 694 | | |CDNI Logging | |CDNI Logging | |CDNI Logging | | | 695 | | | Field | | Field | | Field | | | 696 | | +-------------+ +-------------+ +-------------+ | | 697 | +--------------------------------------------------+ | 698 +------------------------------------------------------+ 700 Figure 3: Structure of Logging Files 702 The CDNI Logging File format is inspired from the W3C Extended Log 703 File Format [ELF]. However, it is fully specified by the present 704 document. Where the present document differs from the W3C Extended 705 Log File Format, an implementation of CDNI Logging MUST comply with 706 the present document. 708 A CDNI Logging File MUST contain a sequence of lines containing US- 709 ASCII characters [CHAR_SET] terminated by either the sequence LF or 710 CRLF. A CDNI Logging implementation consuming CDNI Logging Files 711 MUST accept lines terminated by either LF or CRLF. 713 Each line of a CDNI Logging File MUST contain either a directive or a 714 CDNI Logging Record. 716 Directives record information about the CDNI Logging process itself. 717 Lines containing directives MUST begin with the "#" character. 718 Directives are specified in Section 3.1. 720 Logging Records provide actual details of the logged event. Logging 721 Records are specified in Section 3.2. 723 3.1. CDNI Logging File Directives 724 An implementation of the CDNI Logging interface MUST support the 725 following directives (formats specified in the form <...> are 726 specified in Section 3.3): 728 o Version: 730 * format: . 732 * semantic: indicates the version of the CDNI Logging File 733 format. The value MUST be "1.0" for the version specified in 734 the present document. 736 * occurrence: there MUST be one and only one instance of this 737 directive. It MUST be the first line of the CDNI Logging file. 739 o UUID: 741 * format: 743 * semantic: this is Universally Unique IDentifier for the CDNI 744 Logging File as specified in [RFC4122]. 746 * occurrence: there MUST be one and only one instance of this 747 directive. 749 o Origin: 751 * format: 753 * semantic: this identifies the entity transmitting the CDNI 754 Logging File (e.g. the host in a dCDN supporting the CDNI 755 Logging interface) or the entity responsible for transmitting 756 the CDNI Logging File (e.g. the dCDN). 758 * occurrence: there MUST be zero or one instance of this 759 directive. This directive MAY be included by the 760 implementation transmitting the CDNI Logging file. When 761 included by the transmitting side, it MUST be validated or 762 over-written by the receiving side. When, it is not included 763 by the transmitting side, it MAY be added locally by the 764 receiving side. [Editor's Note if we include a non-repudiation 765 mechanism: discuss the fact that this will provide incentive to 766 dCDN to not cheat , as it can be detected] 768 o Record-Type: 770 * format: 771 * semantic: indicates the type of the CDNI Logging Records that 772 follow this directive, until another Record-Type directive (or 773 the end of the CDNI Logging File). "cdni_http_request_v1" MUST 774 be indicated in the Record-Type directive for CDNI Logging 775 records corresponding to HTTP request (e.g. a HTTP delivery 776 request) as specified in Section 3.2.1. 778 * occurrence: there MUST be at least one instance of this 779 directive. The first instance of this directive MUST precede a 780 Fields directive and precede any CDNI Logging Record. 782 o Fields: 784 * format: [ ], where the allowed list of 785 are specified for each Record-Type in Section 3.2. 787 * semantic: this lists the names of all the fields for which a 788 value is to appear in the CDNI Logging Records that are after 789 this directive. The names of the fields, as well as their 790 possible occurrences, are specified for each type of CDNI 791 Logging Records in Section 3.2. The field names listed in this 792 directive MUST be separated by a whitespace (" "). 794 * occurrence: there MUST be at least one instance of this 795 directive per Record-Type directive. The first instance of 796 this directive for a given Record-Type MUST precede any CDNI 797 Logging Record for this Record-Type. 799 o Integrity-Hash: 801 * format: 803 * semantic: This directive permits the detection of a corrupted 804 CDNI Logging File. This can be useful, for instance, if a 805 problem occurs on the filesystem of the dCDN Logging system and 806 leads to a truncation of a logging file. The Integrity-Hash 807 value is computed, and included in this directive by the entity 808 that transmits the CDNI Logging File, by applying the MD5 809 ([RFC1321]) cryptographic hash function on the CDNI Logging 810 File, including all the directives and logging records, up to 811 the Intergrity-Hash directive itself, excluding the Integrity- 812 Hash directive itself and, when present, also excluding the 813 Non-Repudiation-Hash directive. The Integrity-Hash value is 814 represented as a US-ASCII encoded hexadecimal number, 32 digits 815 long (representing a 128 bit hash value). The entity receiving 816 the CDNI Logging File also computes in a similar way the MD5 817 hash on the received CDNI Logging File and compares this hash 818 to the value of the Integrity-Hash directive. If the two 819 values are equal, then the received CDNI Logging File MUST be 820 considered non-corrupted. If the two values are different, the 821 received CDNI Logging File MUST be considered corrupted. The 822 behavior of the entity that received a corrupted CDNI Logging 823 File is outside the scope of this specification; we note that 824 the entity MAY attempt to pull again the same CDNI Logging file 825 from the transmitting entity. 827 * occurrence: there MUST be one and only one instance of this 828 directive. This field MUST be the last line of the CDNI 829 Logging File when the Non-Repudiation-Hash is absent, and MUST 830 be the one before last line of the CDNI Logging File when the 831 Non-Repudiation-Hash is present. 833 o Non-Repudiation-Hash: 835 * format: 837 * semantic: This hash field permits the non-repudiation of the 838 CDNI Logging File by the entity that transmitted the CDNI 839 Logging File. [Editor's Note: I need help for specifying the 840 appropriate hash - ie hash must be signed with private-key of 841 entity transmitting the CDNI Logging File] 843 * occurrence: there MAY be one and only one instance of this 844 directive. When present, this directive MUST be the last line 845 of the CDNI Logging File. 847 3.2. Logging Records 849 A CDNI Logging Record consists of a sequence of CDNI Logging Fields 850 relating to that single CDNI Logging Record. 852 CDNI Logging Fields MUST be separated by the "horizontal tabulation 853 (TAB)" character. 855 Some CDNI Logging field names use a prefix scheme similar to the one 856 used in W3C Extended Log File Format [ELF] to facilitate readability. 857 The semantics of the prefix in the present document is: 859 o c: refers to the User Agent that issues the request (corresponds 860 to the "client" of W3C Extended Log Format) 862 o s: refers to the dCDN Surrogate that serves the request 863 (corresponds to the "server" of W3C Extended Log Format) 865 o cs: refers to communication from the dCDN Surrogate towards the 866 User-Agent 868 o sc: refers to communication from the User-Agent towards the dCDN 869 Surrogate 871 [Editor's Note: see discussion with Rob about adding definition for 872 "r"] 874 An implementation of the CDNI Logging interface as per the present 875 specification MUST support the CDNI HTTP Delivery Records as 876 specified in Section 3.2.1. [Editor's Note": other types of delivery 877 records will be listed here if we specify other types for this 878 version eg Request Routing]. 880 The formats listed in this section in the form <...> are specified in 881 Section 3.3). 883 3.2.1. HTTP Request Logging Record 885 The HTTP Request Logging Record contains the following CDNI Logging 886 Fields, listed by their field name: 888 o date: 890 * format: 892 * semantic: the date at which the processing of request started 893 on the Surrogate. 895 * occurrence: there MUST be one and only one instance of this 896 field. 898 o time: 900 * format: