idnits 2.17.1 draft-bormann-t2trg-stp-01.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- == It seems as if not all pages are separated by form feeds - found 10 form feeds but 16 pages Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (June 30, 2018) is 2125 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Outdated reference: A later version (-08) exists of draft-ietf-netconf-notification-messages-03 == Outdated reference: A later version (-15) exists of draft-ietf-netconf-restconf-notif-06 == Outdated reference: A later version (-26) exists of draft-ietf-netconf-subscribed-notifications-13 == Outdated reference: A later version (-05) exists of draft-ietf-netconf-udp-pub-channel-03 == Outdated reference: A later version (-25) exists of draft-ietf-netconf-yang-push-17 -- Obsolete informational reference (is this intentional?): RFC 7230 (Obsoleted by RFC 9110, RFC 9112) -- Obsolete informational reference (is this intentional?): RFC 7232 (Obsoleted by RFC 9110) Summary: 0 errors (**), 0 flaws (~~), 7 warnings (==), 3 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group C. Bormann 3 Internet-Draft Universitaet Bremen TZI 4 Intended status: Informational K. Hartke 5 Expires: January 1, 2019 Ericsson 6 June 30, 2018 8 The Series Transfer Pattern (STP) 9 draft-bormann-t2trg-stp-01 11 Abstract 13 Many applications make use of Series of data items, i.e., an array of 14 data items where new items can be added over time. Where such Series 15 are to be made available using REST protocols such as CoAP or HTTP, 16 the Series has to be mapped into a structure of one or more resources 17 and a protocol for a client to obtain the Series and to learn about 18 new items. 20 Various protocols have been standardized that make Series-shaped data 21 available, with rather different properties and objectives. The 22 present document is an attempt to extract a common underlying pattern 23 and to define media types and an access scheme that can be used right 24 away for further protocols that provide Series-shaped data. 26 Status of This Memo 28 This Internet-Draft is submitted in full conformance with the 29 provisions of BCP 78 and BCP 79. 31 Internet-Drafts are working documents of the Internet Engineering 32 Task Force (IETF). Note that other groups may also distribute 33 working documents as Internet-Drafts. The list of current Internet- 34 Drafts is at https://datatracker.ietf.org/drafts/current/. 36 Internet-Drafts are draft documents valid for a maximum of six months 37 and may be updated, replaced, or obsoleted by other documents at any 38 time. It is inappropriate to use Internet-Drafts as reference 39 material or to cite them other than as "work in progress." 41 This Internet-Draft will expire on January 1, 2019. 43 Copyright Notice 45 Copyright (c) 2018 IETF Trust and the persons identified as the 46 document authors. All rights reserved. 48 This document is subject to BCP 78 and the IETF Trust's Legal 49 Provisions Relating to IETF Documents 50 (https://trustee.ietf.org/license-info) in effect on the date of 51 publication of this document. Please review these documents 52 carefully, as they describe your rights and restrictions with respect 53 to this document. Code Components extracted from this document must 54 include Simplified BSD License text as described in Section 4.e of 55 the Trust Legal Provisions and are provided without warranty as 56 described in the Simplified BSD License. 58 Table of Contents 60 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 61 2. Objectives . . . . . . . . . . . . . . . . . . . . . . . . . 4 62 3. A REST Series Transfer Pattern (STP) . . . . . . . . . . . . 4 63 3.1. Basic collections . . . . . . . . . . . . . . . . . . . . 4 64 3.2. Pagination and Observing linked lists . . . . . . . . . . 5 65 3.3. The "cursor" pattern . . . . . . . . . . . . . . . . . . 7 66 4. IANA considerations . . . . . . . . . . . . . . . . . . . . . 8 67 5. Security considerations . . . . . . . . . . . . . . . . . . . 9 68 6. Informative References . . . . . . . . . . . . . . . . . . . 9 69 Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . 10 70 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 10 72 1. Introduction 74 (TO DO: Insert an extended form of the abstract first here, expanding 75 the reference to [RFC7230] and [RFC7252] in the process.) 77 Examples for protocols that provide Series-shaped data are: 79 o The Atom Syndication Format [RFC4287] defines _feeds_ as Series of 80 _entries_ (links plus some metadata, which can often be much of 81 the content of an entry), where a feed is represented by a 82 collection resource that contains just a small number of the most 83 recent entries. By polling a feed, a client can contain a fresh 84 view of the Series, with a focus on recent items. If the client 85 does not poll often enough, it will _miss_ items. 87 o Messaging protocols such as XMPP or SIMPLE transfer series of what 88 is often called "Instant Messages". A publish/subscribe mechanism 89 allows a client to select sequences of messages that it is 90 interested in. 92 o Mail servers that provide interactive access to stored messages 93 present a Series to their clients. Obviously, loss of messages is 94 frowned upon. 96 o CoAP Observe allows a client to observe a resource as it changes; 97 the client can collect the changes into a Series. Observe is 98 focused on eventual consistency: a fresher data items simply 99 overwrites an older one. The present document uses the observe 100 pattern to build a more general Series Transfer Pattern. 102 o Syslog is an interesting case of a Series Transfer. 104 o [I-D.ietf-netconf-yang-push], 105 [I-D.voit-netmod-yang-notifications2], 106 [I-D.ietf-netconf-subscribed-notifications], 107 [I-D.ietf-netconf-notification-messages], 108 [I-D.ietf-netconf-restconf-notif]. 110 o An RTP stream can be viewed as an (somewhat extreme) case of a 111 Series; new items are just sent inside separate UDP packets. A 112 sequence number allows to detect (but not normally ask for 113 retransmission of) missing items. A timestamp as well as source 114 data (SSRC, CSRC) provide further common metadata that aid in the 115 processing of the Series items. 117 o An example of an ad-hoc design of a series transfer protocol is 118 [I-D.ietf-netconf-udp-pub-channel]. 120 o Server-sent events [sse] are a somewhat bizarre version of a 121 series transfer protocol. 123 o The Interface for Metadata Access Points (IF-MAP) specified by the 124 Trusted Computing Group and emerging derivatives of that protocol 125 create a series of updates to a graph representation of related 126 network-related security information. The requests created by IF- 127 MAP clients are bundled operations of updates to a MAP Graph, 128 which compose a Series Transfer Pattern of bundled atomic 129 operations that ensure the integrity of the MAP Graph. [Henk 130 Birkholz] 132 o netflow/IPFIX was defined to transfer a series of data items about 133 flows. Information about PDU flows accounted by network 134 interfaces of endpoints is emitted in a series of counter bundles 135 via the IPFIX protocol. Only a series of these continuous Flow 136 Records creates a meaningful bigger picture about the current 137 traffic in the network topology of an administrative domain. 138 Depending on the characteristics measured, loss of a Flow Record 139 can range from harmless to missing the only vital counter 140 measurement. [Henk Birkholz] 142 o TO DO: Add more items. 144 [I-D.birkholz-yang-push-coap-problemstatement] is a problem statement 145 that will require the design of another scheme to transfer Series- 146 shaped data. 148 2. Objectives 150 Series transfer applications may have rather different objectives. 152 o The completeness of the Series transfer may be of utmost 153 importance (e.g., if each item represents a sale), it may be 154 desirable but can be jettisoned in an overload situation, or it 155 may just be a likely outcome with a very active client (e.g., with 156 Atom). Note that there is never a way to _guarantee_ completeness 157 unless all of the rate and size of incoming new items, the 158 transfer capacity available, and the processing capabilities of 159 the client are controlled; however, system designs may want to 160 give the illusion of "reliability". 162 o Minimizing the latency of the transfer may be important, as may be 163 limiting it below a defined maximum (note that these are different 164 objectives). The latter can be supported in a polling system by 165 polling at least as often as that maximum latency; this may be 166 considered inefficient and "push" mechanisms may be developed. 167 Mail environments have developed "push" services to enable 168 minimizing the latency. Where latency requirements go below the 169 time that might be needed for an end-to-end retransmission, error 170 concealment may provide an acceptable user experience (e.g., in 171 RTP). 173 In general, minimizing latency and ensuring completeness are 174 competing objectives. 176 Series transfer environments sometimes centralize information 177 distribution functions, leading to "broker" architectures (often 178 combined with the "publish/subscribe" pattern). With brokers, Series 179 publishers may use an entirely different interface to the brokers 180 from that used by the receiving clients, or the interfaces can be 181 designed so they are similar for all the forwarding steps. 183 3. A REST Series Transfer Pattern (STP) 185 3.1. Basic collections 187 A series of items can be represented by a single collection resource: 189 _____________ 190 | | 191 | item 11 | 192 |_____________| 193 | | 194 | item 10 | 195 |_____________| 196 | . | 197 | . | 198 | . | 199 |_____________| 200 | | 201 | item 1 | 202 |_____________| 204 Figure 1: A collection of items 206 While this is adequate in many cases, it has a number of limitations: 208 o Each retrieval fetches the entire collection 210 * As long as the collection does not change, this can be 211 mitigated with ETags (Section 5.10.6 of[RFC7252], Section 2.3 212 of [RFC7232]). 214 o When the collection becomes too large, the server has to prune 215 older items. These then no longer can be retrieved, and there is 216 even no way for the server to indicate that there used to be older 217 items. 219 3.2. Pagination and Observing linked lists 221 In the Browser Web, it is usual to provide _Pagination_ for 222 collection resources that can grow large (e.g., search results): 224 _____________ _____________ _____________ 225 | | | | | | 226 | item 11 | +--->| item 9 | +--->| item 2 | 227 |_____________| | |_____________| | |_____________| 228 | | | | | | | 229 | item 10 | | | item 8 | . . . | item 1 | 230 |_____________| | |_____________| |_____________| 231 | | | | | | page M 232 | link -----------+ | link -----------+ 233 |_____________| |_____________| 235 Figure 2: A paginated collection of items 237 Without modification, this does not work well for resources that 238 actually change by themselves: Once a new page needs to be added, 239 what previously was page 1 now becomes page 2. Obviously, the naming 240 of pages better remains unchanged with new pages added a the front. 242 _____________ _____________ _____________ 243 | | | | | | 244 | item 11 | +--->| item 9 | +--->| item 2 | 245 |_____________| | |_____________| | |_____________| 246 | | | | | | | 247 | item 10 | | | item 8 | . . . | item 1 | 248 |_____________| | |_____________| |_____________| 249 | link -----------+ | link -----------+ 250 |_____________| |_____________| 252 Figure 3: Pagination with stable names 254 However, now the client has no idea what initial page to request to 255 get the freshest items and the head of the list. It is easy to add a 256 link to the freshest page: 258 _____________ _____________ _____________ 259 | | | | | | 260 | link --------------->| item 11 | +--->| item 2 | 261 |_____________| |_____________| | |_____________| 262 head | | | | 263 | item 10 | . . . | item 1 | 264 |_____________| |_____________| 265 | link -----------+ 266 |_____________| 267 page M 269 Figure 4: Pagination with stable names 271 The head of the linked list can now be simply observed; the addition 272 of pages will then be notified to the observer. 274 As usual in series transfer, the following considerations remain: 276 o When can the server decide to no longer retain older items? 278 * There may be a desire for an observer to be able to catch all 279 items in the series. 281 + How does the server know who are the observers? E.g., what 282 to do with newly joining observers? 284 + How does an observer signal that it has caught up (to a 285 specific item)? 287 o What to do when the decision to remove items from the list cannot 288 be made and there is no room for new items? 290 The link head can also include items that have so far not been added 291 to pages; this can be used to fill up pages evenly without them ever 292 changing. Obviously, the best number of items to prenotify in this 293 way as well as the best time to open a new page are different for 294 different applications. 296 3.3. The "cursor" pattern 298 A GET on a resource representing a Series may return a collection 299 item that contains the following pieces of information 301 o An array of Series items, either as an array of media-typed 302 objects in a suitable representation format (e.g., CBOR, MIME) or 303 by using an array-like media type (e.g., SenML). 305 * Items may be full items or limit themselves to some metadata 306 and a link; the client can then follow that link if it is 307 interested in the data (possibly basing that decision on the 308 metadata and/or a measure of load). 310 o A "cursor" that can then be used as a parameter in further GET 311 requests (see below) in order to receive only newer items than 312 those received with this transfer. 314 o A "more bit" that indicates whether such further items already 315 exist but could not be returned in the present response. 317 In Figure 5, the cursor is implemented as a URI that can be used as a 318 link to the next page. 320 _____________ _____________ _____________ 321 | | | | | | 322 | item 10 |<---+ | item 1 |<--------------- link | 323 |_____________| | |_____________| |_____________| 324 | | | | tail 325 | item 11 | . . . | item 2 | 326 |_____________| |_____________| 327 page M | | | 328 +----------- link | 329 |_____________| 331 Figure 5: Cursor pattern pictured as pagination 333 A GET may be enhanced with additional parameters (possibly turning it 334 into a FETCH): 336 o The cursor. 338 o A "wait bit" that indicates whether a (possibly empty) reply 339 should be given right away or the server should wait for new items 340 to become available. (To avoid the equivalence of the "silly 341 window syndrome", the wait bit may be enhanced by a minimum number 342 of items and a timeout after which even a smaller number is made 343 available.) In effect, this requests a form of "long polling"; 344 see [RFC6202] for some considerations for this in HTTP. 346 A server may implement a form of custody transfer by interpreting the 347 cursor as an acknowledgement that the client has received all data up 348 to the cursor. This is not necessarily acting as an unsafe request 349 ("destructive GET"), as other clients may be active that have not yet 350 received all these data. To implement a full custody semantics, the 351 server needs to be aware of all the clients that expect a full Series 352 Transfer (a classical group management problem). 354 (Explain how Observe can help. Can it?) 356 4. IANA considerations 358 This memo registers a number of media types: TO DO. 360 o A media type for FETCH selectors (Section 3): 362 * An alternative way to encode this information into the URI of a 363 GET should also be available. 365 o A Series media type as alluded to in Section 3. 367 5. Security considerations 369 TO DO 371 6. Informative References 373 [I-D.birkholz-yang-push-coap-problemstatement] 374 Birkholz, H., Zhou, T., Liu, X., and E. Voit, "YANG Push 375 Operations for CoMI", draft-birkholz-yang-push-coap- 376 problemstatement-00 (work in progress), October 2017. 378 [I-D.ietf-netconf-notification-messages] 379 Voit, E., Birkholz, H., Bierman, A., Clemm, A., and T. 380 Jenkins, "Notification Message Headers and Bundles", 381 draft-ietf-netconf-notification-messages-03 (work in 382 progress), February 2018. 384 [I-D.ietf-netconf-restconf-notif] 385 Voit, E., Rahman, R., Nilsen-Nygaard, E., Clemm, A., and 386 A. Bierman, "RESTCONF and HTTP Transport for Event 387 Notifications", draft-ietf-netconf-restconf-notif-06 (work 388 in progress), June 2018. 390 [I-D.ietf-netconf-subscribed-notifications] 391 Voit, E., Clemm, A., Prieto, A., Nilsen-Nygaard, E., and 392 A. Tripathy, "Customized Subscriptions to a Publisher's 393 Event Streams", draft-ietf-netconf-subscribed- 394 notifications-13 (work in progress), June 2018. 396 [I-D.ietf-netconf-udp-pub-channel] 397 Zheng, G., Zhou, T., and A. Clemm, "UDP based Publication 398 Channel for Streaming Telemetry", draft-ietf-netconf-udp- 399 pub-channel-03 (work in progress), July 2018. 401 [I-D.ietf-netconf-yang-push] 402 Clemm, A., Voit, E., Prieto, A., Tripathy, A., Nilsen- 403 Nygaard, E., Bierman, A., and B. Lengyel, "YANG Datastore 404 Subscription", draft-ietf-netconf-yang-push-17 (work in 405 progress), July 2018. 407 [I-D.voit-netmod-yang-notifications2] 408 Voit, E., Bierman, A., Clemm, A., and T. Jenkins, "YANG 409 Notification Headers and Bundles", draft-voit-netmod-yang- 410 notifications2-00 (work in progress), February 2017. 412 [RFC4287] Nottingham, M., Ed. and R. Sayre, Ed., "The Atom 413 Syndication Format", RFC 4287, DOI 10.17487/RFC4287, 414 December 2005, . 416 [RFC6202] Loreto, S., Saint-Andre, P., Salsano, S., and G. Wilkins, 417 "Known Issues and Best Practices for the Use of Long 418 Polling and Streaming in Bidirectional HTTP", RFC 6202, 419 DOI 10.17487/RFC6202, April 2011, 420 . 422 [RFC7230] Fielding, R., Ed. and J. Reschke, Ed., "Hypertext Transfer 423 Protocol (HTTP/1.1): Message Syntax and Routing", 424 RFC 7230, DOI 10.17487/RFC7230, June 2014, 425 . 427 [RFC7232] Fielding, R., Ed. and J. Reschke, Ed., "Hypertext Transfer 428 Protocol (HTTP/1.1): Conditional Requests", RFC 7232, 429 DOI 10.17487/RFC7232, June 2014, 430 . 432 [RFC7252] Shelby, Z., Hartke, K., and C. Bormann, "The Constrained 433 Application Protocol (CoAP)", RFC 7252, 434 DOI 10.17487/RFC7252, June 2014, 435 . 437 [sse] WHATWG, "HTML Living Standard -- 9.2 Server-sent events", 438 n.d., . 441 Acknowledgements 443 The need for a Series Transfer Pattern has been made clear by a 444 number of people that contribute to the IRTF Thing-to-Thing Research 445 Group (T2TRG), e.g. Matthias Kovatsch and Henk Birkholz (both of 446 whom also provided feedback on an early draft). Henk also 447 contributed further examples for the use of Series Transfers in 448 protocols. 450 Authors' Addresses 452 Carsten Bormann 453 Universitaet Bremen TZI 454 Postfach 330440 455 Bremen D-28359 456 Germany 458 Phone: +49-421-218-63921 459 Email: cabo@tzi.org 460 Klaus Hartke 461 Ericsson 462 Torshamnsgatan 23 463 Stockholm SE-16483 464 Sweden 466 Email: klaus.hartke@ericsson.com